Advanced Series in Neuroscience - Vol. 3
This page is intentionally left blank
Advanced Series in Neuroscience - Vol. 3
edited by
H Gutfreund The Hebrew University of Jerusalem Mount Scorpus, Jerusalem 91905, Israel
G Toulouse Laboratoire de Physique Statistique, Ecole Normaie Superieure 24 rue Lhomond, 75231 Paris Cedex 05, France
Ufe World Scientific Vb
Singapore • New Jersey • London • Hong Kong
Published by World Scientific Publishing Co. Pte. Lid. P O B o \ 128, Fairer Road, Singapore 9128 USA office: Suite I B , 1060 Main Street, River Edge. NJ 07661 UK office: 73 Lynton Mead. Totieridge, London N20 8DH
The editors and publisher are grateful to the authors and the following publishers for their assistance and their permission to reproduce the articles found in this volume: Academic Press Inc. The American Association for the Advancement of Science (Science) American Institute of Physics (Rev. Mod. Phys.) American Physical Society (Phys. Rev. A ) American Psychological Association {Psychological Review) American Scientist (Am. Sci,) Annua] Review Inc. (Anna. Rev. Neurosci) Blackwell Publishers Blackwell Scientific Publications. Inc. Cambridge University Press Cold Spring Harbor Laboratory Press Dover Publications Inc. Elsevier Science Publishers. BV Elsevier Science Publishers Ltd (UK) IOP Publishing Ltd Karger, Basel Kluwer Academic Publishers Lawrence Erlbaum Associates, Inc. Macmillan Magazines Ltd (Nature) Massachusetts Institute of Technology M I T Press National Academy of Sciences (Proc. Nat l Acad. Sci. USA) Pergamon Press Rockefeller University Press Rout ledge Springer Verlag Yale University Press
B I O L O G Y A N D C O M P U T A T I O N : A PHYSICIST'S C H O I C E Copyright © 1994 by World Scientific Publishing Co. Pte. Ltd. All rights reserved. This book, or pans thereof, may not be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording or any information storage and retrieval system now known or to be invented, without written permission from the Publisher. For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 27 Congress Street, Salem, M A 01970. USA.
ISBN ISBN
981-02-1405-7 981-02-1406-5 (pbk]
Printed in Singapore
C O N T E N T S
GENERAL INTRODUCTION
CHAPTER a)
1
xi
SETTING T H E STAGE
Forewords; Introductory Warnings
1. T h e Mechanical M i n d H. B . Barlow, Ann. Rev. Neurosci.
1 3 5
13, 15-24 (1990).
2. E s t h e t i c Analysis of Works of A r t H. L . F . Helmholtz, in On the Sensations of Tone, transl. by A. J . Ellis, pp. 366-371 (Dover 1954).
15
3. More is Different P. W . Anderson, Science 177, 393-396 (1972).
21
4. W h a t is Computational Neuroscience? P. S. Churchland, C . Koch and T . J . Sejnowski, in Computational Neuroscience, ed. E . Schwartz, pp. 46-55 ( M I T Press 1990).
25
b)
P h y s i c s , Biology, C o m p u t a t i o n
35
5. Life, Thermodynamics, and Cybernetics L . Brillouin, Am. Sci. 37, 554-568 (1949).
37
6. Physics, Biological Computation, and Complementarity J . J . Hopfield, in The Lesson of Quantum Theory, Proceedings of the Niels Bohr Centenary Symposium, eds. D. de Boer, E . Dahl and O. Ulfbeck, pp. 295-314 (North-Holland 1986).
52
7. Evolution and Tinkering F . Jacob, Science 196, 1161-1166 (1977).
72
8. Physics and Antiphysics V . Braitenberg, in On the Texture of Brains, Chap. 2, pp. 5-8 (Springer 1977).
78
9. Manifesto of B r a i n Science V . Braitenberg, in Information Processing in the Cortex, eds. A. Aertsen and V . Braitenberg, pp. 473-477 (Springer 1992).
82
c)
C o m p u t e r a n d Brain; Logic and Statistics
87
10. T h e Logical Structure of the Nervous System; Nature of the System of Notations Employed: Not Digital B u t Statistical; T h e Language of the Brain Not the Language of Mathematics J . von Neumann, in The Computer and the Brain, pp. 74-82 (Yale U. Press 1958).
89
11. Probability, Philosophy and Science: A Briefing for Bayesians A. J . M. Garrett, in Maximum Entropy and Bayesian Methods,
98
ed. J . Skirling, pp. 107-116 (Kluwer 1989). V
12. T w o Revolutions — Cognitive and Probabilistic; Is the Mind a Bayesian? G . Gigerenzer and D. J . Murray, in Cognition as Intuitive X I - X I I , pp. 147-162 (Lawrence Erlbaum 1987). d)
108 Statistics,
Some P e r c e p t u a l Facts a n d Issues
124
13. Perception M. Delbrfick, in Mind from Matter?, Chap. 8, pp. 109-119 (Blackwell 19S6).
125
14. Understanding Images in the Brain C. Blakemore, in Images and Understanding, eds. H. Barlow, C . Blakemore, M. Weston-Smith, pp. 257-283, 386-388 (Cambridge U. Press 1990).
136
CHAPTER 2
164
a)
BIOLOGICALCONCEPTS AND METHODS; COMPUTATIONAL GOALS AND MEANS
Mental Representations
166
15. Concerning Imagery D. O. Hebb, Psychological Review 75, 466-477 (1968).
168
16. Mental Rotation of Three-Dimensional Objects R. N. Shepard and J . Metzler, Science 171, 701-703 (1971).
180
17. Mental Rotation of the Neuronal Population Vector A. P. Georgoponlos, J . T . Lurito, M. Petrides, A. B. Schwartz and J. T . Massey, Science 243, 234-236 (1989).
183
b)
Information Theory and Perception
18. Whatever Happened to Information T h e o r y ? R. L . Gregory, in Odd Perceptions, pp. 187-194 (Methuen 1986).
186 188
19. Some Informational Aspects of Visual Perception F . Attneave, Psychological Review 61, 183-193 (1954). 20. T h e Magical Number Seven, Plus or Minus Two: Some Limits on O u r Capacity for Processing Information G . A. Miller, Psychological Review 63, 81-97 (1956).
196 207
21. Single Units and Sensation: A Neuron Doctrine for Perceptual Psychology? H. B. Barlow, Perception 1, 371-394 (1972).
224
22. Conditions for Versatile Learning, Helmholtz's Unconscious Inference, and the Task of Perception H. B. Barlow, Vision Research 30, 1561-1571 (1990).
248
c)
259
Neuroanatomy
23. Reading the Structure of Brains V . Braitenberg, Network 1, 1-11 (1990).
260
24. Axonal Trees and Cortical Architecture G . J . Mitchison, Trends Neurosci. 15, 122-126 (1992).
271
vi
d) 25.
A s p e c t s of B i o computation
276
Antibodies and Learning; Selection Versus Instruction N. K . Jerne, in The Newosciences: A Study Program,
278
eds. G . Quartern, T . Menelchuk and F . O. Schmitt, pp. 200-205 (Rockefeller U. Press 1967). 26.
E a r l y Vision and Focal Attention
284
B. Julesz, Reviews of Modern Physics 63, 735-772 (1991). 27.
Neuropsychology and the Nature of Consciousness L . Weiskrantz, in Mindwaves, eds. C . Blakemore and S. Greenfield, pp. 306-320 (Blackwell 1987).
CHAPTER 3
a) 28.
MODES OF COMPUTATION; PROCESSING AND LEARNING
Neural Networks
322
337
339
Collective Processing and Neural States J. J . Hopfield, in Modeling and Analysis in Biomedicine, ed. C . Nicolini, 369-389 (World Sci. Pub. 1984).
341
29. T h e Space of Interactions in Neural Network Models E . Gardner, J . Phys. A 2 1 , 257-270 (1988).
361
b)
375
Parallel Algorithms
30.
Optimization by Simulated Annealing S. Kirkpatrick, C . Gelatt and M. P. Vecchi, Science 220, 671-680 (1983).
376
31.
A n Analogue Approach to the Travelling Salesman Problem using an E l a s t i c Net Method R. Durbin and D. Willshaw, Mature 326, 689-691 (1987).
386
c)
Generalization; Learning a Rule
389
32.
Large Automatic Learning, Rule Extraction, and Generalization J. Denker, D. Schwartz, B . Wittner, S. Solla, R. Howard, L . Jackel and J . J . Hopfield, Complex Systems 1, 877-922 (1987).
391
33.
Confronting Neural Network and Human Behavior in a Quasi regular Environment M. A. Virasoro, article based on a paper presented at the Conference: "From Statistical Physics to Statistical Inference, and Back", Cargese, 1992.
437
34.
Statistical Mechanics of Learning from Examples H. S. Seung, H . Sompolinsky and N. Tishby, Phys. Rev. A 4 5 ,
447
6056-6091 (1992). 35. O n the Classification of Learning Machines G . Parisi, Network 3, 259-265 (1992).
483
d)
490
36.
E a r l y Sensory Processing Perceptual Neural Organization: Some Approaches Based on Network Models and Information Theory R. Linsker, Anno. Rev. Neurosci. 13, 257-281 (1990). vii
492
37. A n Information-Theoretic View of Analog Representation in Striate Cortex J . G . Daugman, in Computational Neuroscience, ed. E . Schwartz, pp. 403-423 ( M I T Press 1990).
517
38. W h a t Does the Retina Know About Natural Scenes? J. J . Atick and A. N. Redlich, Neural Compulation 4, 196-210 (1992).
538
e)
553
Neural Codes
39. Finding Minimum Entropy Codes H. B. Barlow, T . P. Kanshal and G. J . Mitchison, Neural Computation 1, 412-423 (1989).
554
40. Reading a Neural Code W. Bialek, F . Rieke, R. R. de Ruyter van Steveninck and D. Warland, Science 252, 1854-1857 (1991).
566
41. Spike Arrival Times: A Highly Efficient Coding Scheme for Neural Networks S. J . Thorpe, in Parallel Processing in Neural Systems and Computers, eds. R. Eckmiller, G . Hartmann and G. Hauske, pp. 91-94 (Elsevier 1990).
570
CHAPTER 4
574
a)
BRAIN AREAS, CIRCUITS AND DYNAMICS
Sensory a n d Motor Pathways
576
42. Deciphering the Brain's Codes M. Konishi, Neural Computation 3, 1-18 (1991).
578
43. Segregation of Form, Color, Movement, and Depth: Anatomy, Physiology, and Perception M. S. Livingstone and D. H. Hubel, Science 240, 740-749 (1988).
596
44. Separate Visual Pathways for Perception and Action M. A. Goodale and A. D. Milner, Trends Neurosci. 15, 20-25 (1992).
606
45. Behavioral Neurophysiology: Insights into Seeing and Grasping S. P. Wise and R. Desimone, Science 242, 736-741 (1988).
612
46. A Back-Propagation Programmed Network that Simulates Response Properties of a Subset of Posterior Parietal Neurons D. Zipser and R. A. Andersen, Nature 331, 679-684 (1988).
618
b)
Bridges Between Psychophysics a n d Physiology
624
47. Neuronal Correlates of a Perceptual Decision W. T . Newsome, K. H. Britten and J . A. Movshon, Nature 341, 52-54 (1989).
625
48. Subjective Contours — Bridging the G a p Between Psychophysics and Physiology E . Peterhans and R. von der Heydt, Trends Neurosci. 14, 112-119 (1991).
627
viii
c)
S t r u c t u r e s a n d Functions of Various B r a i n A r e a s
635
49.
Reciprocal L i n k s of the Corpus Striatum with the Cerebral Cortex and Limbic System: A Common Substrate for
636
Movement and Thought? W. J . H . Nauta, in Neurology and Psychiatry: A Meeting of Minds, ed. J . Mueller, pp. 43-63 (Karger, Basel 1989). 50. T h e Cerebellar Network. Attempt at a Formalization of Its Structure
657
V . Braitenberg, Network 4, 11-17 (1993). 51. Hippocampal Synaptic Enhancement and Information Storage W i t h i n a Distributed Memory System B . L . McNanghton and R. G . M. Morris, Trends Neurosci. 10, 408-415 (1987). d)
Representations of Space in the B r a i n
665
673
52. T h e Updating of the Representation of V i s u a l Space in Parietal Cortex by Intended Eye Movements J . R. Duhamel, C . L . Colby and M. E . Goldberg, Science 255, 90-92 (1992).
675
53. O n Listing's L a w K . Hepp, Commun.
678 Math. Phys. 132, 285-292 (1990).
54. T w o - R a t h e r T h a n Three-Dimensional Representation of Saccades in Monkey Superior Colliculus A . J . van Opstal, K. Hepp, B. J . M. Hess, D. Straumann and V . Henn, Science 252, 1313-1315 (1991).
686
e)
689
Oscillations and Synchrony
55. T e m p o r a l Coding in the V i s u a l Cortex: New Vistas on Integration in the Nervous System A . K . Engel, P Konig, A. K . Kreiter, T . B. Schillen and W. Singer, Trends Neurosci. 15, 218-226 (1992).
690
CHAPTER 5
699
DEBATES AND SPECULATIONS
a)
Theory-Experiment Interplay
701
56.
Opening the Grey Box R. J . Douglas and K . A. C . Martin, Trends Neurosci. 286-293 (1991).
702 14,
57. I n Defence of Single Electrode Recordings D. J . Amit, Network 3, 385-391 (1992).
710
b)
717
58.
R o l e s of R e t r o a c t i v a t i o n Function of the T h a l a m i c Reticular Complex: T h e Searchlight Hypothesis F . Crick, Proc. Natl. Acad. Sci. USA 81, 4586-4590 (1984).
ix
718
59. Functions of Neural Networks in the Hippocampus and Neocortex in Memory E . T . Rolls, in Neural Models of Plasticity, eds. J . H. Byrne and W. O. Berry, pp. 240-265 (Academic Press, 1989).
723
60. T h e Brain Binds Entities and Events by Multiregional Activation from Convergence Zones A. R. Damasio, Neural Computation 1, 123-132 (1989).
749
61. O n the Computational Architecture of the Neocortex: I — T h e Role of the T h a l a m o c o r t i c a l Loop D. Mumford, Biological Cybernetics 65, 135-145 (1991).
759
62. O n the Computational Architecture of the Neocortex: I I — T h e Role of the Cortico-Cortical Loops D . Mumford, Biological Cybernetics 66, 241-251 (1992).
770
c)
781
Computational Strategies
63. Shifter Circuits: A Computational Strategy for Dynamic Aspects of Visual Processing C . H. Anderson and D. C . van Essen, Proc. Natl. Acad. Sci. 84, 6297-6301 (1987).
783 USA
64. Olfactory Computation and Object Perception J. J . Hopfield, Proc. Natl. Acad. Sci. USA 88, 6462-6466 (1991).
788
65. A Theory of How the Brain Might Work T . Poggio, in The Brain, Cold Spring Harbor Symposium on Quantitative Biology, Vol. LV, pp. 899-910 (Cold Spring Harbor Laboratory Press 1990).
793
d)
805
Language and Consciousness
66. Intelligence, Guesswork, Language H. B. Barlow, Nature 304, 185-195 (1987).
806
67. T h e Biological Role of Consciousness H. B. Barlow, in Mindwaves, eds. C . Blakemore and S. Greenfield, pp. 361-374 (Blackwell 1987).
817
68. T h e Problem of Consciousness and Introspection D. O. Hebb, in Brain Mechanisms and Consciousness, ed. J . F . Delafresnay, pp. 402-417 (Blackwell 1954).
831
Index
847
GENERAL INTRODUCTION
T h i s book may simply be used as a collection of articles related to neural computation. I n this compound opus the reprinted texts are the prominent figures, and the editors' introductions merely provide some background. However, the book will acquire some additional meaning if the reader perceives our initial motivations, and the process along which it evolved toward its present organisation.
Purpose The purpose of this reprint volume is two-fold. The first is to help physicists entering the field of neural networks and brain studies, to obtain a broader view of the context of a domain, new for them but not for science. T l i e second, reciprocally, is to help scientists of other disciplines to reach a better understanding of physicists' contributions within a context of perspectives they can relate to. When we entered this field in the mid-80s, we first felt acutely the difficulty of locating and accessing relevant texts of high quality and lasting value, among a myriad of books, conference proceedings and journals. Subsequently, a second difficulty was experienced as we attempted to communicate to unprepared audiences what the statistical physics of neural networks had achieved. Eventually we came to think that these two difficulties might have a common cause, and a common cure. T h e different scientific subcultures have developed not only different methods and terminology but also different scientific attitudes. When they meet and merge on the brain-study arena, a joint effort of cross-disciplinary understanding and dialogue is required. It
xi
is to this goal that our volume is meant to contribute.
Boundaries I n the meantime, several useful reprint volumes have appeared. Mention should be made of the two earlier books in this "Advanced Series in Neuroscience" (World Scientific Pub. C o . , Singapore): • Brain Theory edited by G . L . Shaw and G . Palm (1988); heretofore referred to as Vol. 1. •
The Neurobiology of Learning and Memory edited by G . L . Shaw, J . L . McGaugh and S. P. R . Rose (1990); Vol. 2.
Quite naturally, we decided on a book which would be complementary to these two collections as well as avoid any duplication. With another worthy couple of books, endowed with insightful introductions: Neurocomputing and Neurocomputing 2, edited by J . A . Anderson et at. ( M I T Press, 1988, 1990), the overlap has been limited to 5 texts (our Refs. (10), (21), (30), (46), (58)), which we felt necessary for the general harmony of our selection. Some other general resource books, at the boundary of our domain, are The Oxford Companion to the Mind edited by R . L . Gregory (Oxford U . Press, 1987) and the twin volumes of Parallel Distributed Processing edited by J . McClelland and D . Rumelhart ( M I T Press, 1986). Several textbooks on modern neural network theory have appeared recently, among which we will mention three: Modeling Brain Function by D. J . Amit (Cambridge U . Press, 1989), Introduction to the Theory of Neural Computation by J . Hertz, A . Krogh and R . G . Palmer (Addison-Wesley, 1991), and An Introduction to the Modeling of Neural Networks by P. Peretto (Cambridge U . Press, 1992).
Organisation Somehow, the preceding references define the boundaries of our book, namely, what it does not attempt to cover, and what it is not. But then what is it? It is a sequence of 68 reprints, organised into 5 chapters and 22 sections. There is a certain arbitrariness at all levels: selection, grouping into sections, and successive ordering. The titles chosen for the chapters and sections are, so to speak, a guide for the eye. They were invented a posteriori, in order to emphasise some thematic relationships between papers placed in succession. But, besides the "local relations", provided by the grouping into sections, there are also many "long-range relations". It is the purpose of our
xii
introductions to the various chapters and sections to indicate the richness of these relations. Further demonstration of the many connections between the different papers may be derived from the Index. Our goal will be achieved, if the collection of reprints appears to ihe attentive reader as a connected, yet flexible, web.
Not too tight, because
the field is a rapidly evolving one. Not too loose, because the appeal of this scientific enterprise rests largely on the convergence of diverse approaches.
Contents T h e text selection process has evolved over a span of a year. We consulted many colleagues from many disciplines. Invaluable advice was received from a number of experts. It is a pleasure to acknowledge our particular gratitude to: Roger Balian, Horace Barlow, Elie Bienenstock, Valentino Braitenberg, Jean Bullier, Pierre Buser, Miri Dick, Klaus Hepp, John Hopfield, Claude Meunier, Jean-Michel Roy, Simon Thorpe and David Willshaw, among all those who were consulted. Two articles, (35) and (57), have been specially conceived for this volume; we are grateful to their authors. Although these colleagues bear no responsibility, of course, for the final selection — where much arbitrary pruning had to be done in order to keep the project within a reasonable size — their comments and suggestions along the way have considerably enriched the breadth and quality of this book. In the process, the editors were rewarded (for all their painful awareness of limited competence and risks of shortcomings) with a sense of achieving an intellectually significant experience. Chapter 1 sets the stage by providing historical landmarks and perspectives, and insights from various disciplines (biology, physics, mathematics, computer science, psychology, philosophy). T h e two perennial questions: W h a t is intelligence? Mind from matter?, are touched on by our authors, with a standard of quality and honesty that should decisively immunise the newcomer against much of the recurrently mediocre literature to which he may otherwise fall naive prey. Chapter 2 introduces biological ideas and approaches that are not part of the standard education of a physicist. In brief, it may be described as a presentation of biocomputation to physicists and other outsiders. Chapter 3 discusses neural networks that solve a variety of information processing and learning tasks. Several models and algorithms, taking inspiration from information theory and statistical physics, are presented in mutually enlightening succession.
xiii
Chapter 4 is a return to brain realities. It attempts to give a state-ofthe-art account of the structures and functions of various perceptual and motor areas. Finally, Chapter 5 is deliberately turned toward the future. So much remains unknown in this field, that there is space for debates on appropriate measurements, for speculations on the role of the pervasive retroactivation pathways, for suggestions about the existence of further unexplored computational strategies, and for a final round on language and consciousness.
Perspectives All over the world, new educational programmes and new research centres are created in the field covered by this book. They appear under a variety of names: "Computation and Neural Systems", "Brain Theory", "Cognitive Neurosciences", "Parallel Computation", etc. They involve collaborations between neurosciences and psychology, mathematics and physics (logic, probability theory, geometry, dynamical systems, statistical physics), computer science and engineering (information theory, signal theory, artificial intelligence, robotics), even evolution theory and philosophy. A n d this enumeration is not exhaustive. We do not think that a new discipline with a unified language will soon emerge from these interdisciplinary collaborations, but we do believe in multilingualism. In this sense, our book is a second generation endeavour, because it belongs to a set of initiatives that aim to overcome a previous "early" stage. The efforts that we invested in the edition of this reprint volume will not have been lost if they contribute to enhancing an educated mutual respect between physicists and biologists, experimentalists and theoreticians, and if they help to deepen a sense of awe for the mysterious unfolding of relations between the diverse sciences.
Index This book should be used as a companion volume to the conventional textbooks in the field; it is not meant to replace them. Their virtue lies in coherence and homogeneity, to which our book wishes to add a complementary dimension of historical and interdisciplinary depth. A book of this nature entails a special style of index. T h e index of this book is essentially a help for discovering further relations between the various reprinted articles. The number of entries is limited, and only the articles in
XIV
which they are discussed are indicated, not the page numbers. T h e reader is invited to look through the whole list in order to discover what terms have been selected. For instance, the term "saccade" does not appear, but the corresponding studies will be found at "eye movements". In contrast, we thought it helpful to distinguish between the different uses of "entropy" concepts. It would have been counterproductive to enter all technical terms from each discipline, specially when similar things are designated by different names in neighbouring disciplines.
X V
Chapter 1
SETTING T H E STAGE
As is fitting for a beginning chapter, attempts are made here to provide historical perspectives and insights from various vantage points. T h e reader will thus find here maximal diversity in terms of the dates of publications, the background of the authors, and the span of problems evoked. Here also, careful consideration has been given, in our selection process, to the following qualities of exposition: clarity, accuracy of thought, forcefulness of expression, and elegance of style. The personality of some of the authors is worth mentioning, because their lives may serve as existent proof of some of the purposes of this volume. Four of the authors have established epoch-making bridges between disciplines: H . Helmholtz, 19th century biologist and physicist {see picture and note in paper (67)); J . von Neumann, mathematician and physicist, pioneer of computer science; L . Brillouin, physicist and champion of early information theory; and M . Delbruck, physicist and one of the founders of modern biology. Also worth mentioning for their openness of mind, in addition to the brilliance of their writings, are three biologists: H . Barlow, neurophysiologist (recordman for the number of texts selected in this volume; see the short biography in paper (67)); V . Braitenberg, neuroanatomist (second on this account, ex-aequo with J . J . Hopfield); F . Jacob, molecular biologist; and two physicists: P. W . Anderson, noted in particular for his many contributions to condensed and disordered matter physics; and J . J . Hopfield, contemporary exemplar of a successful move from theoretical physics to theoretical biology. The division of this chapter into four sections is particularly arbitrary, for most of the texts are involved with the description of our domain, i.e. neurocomputation and its boundaries, and all lay the ground for the four
1
subsequent chapters. ordering.
Nevertheless, here are a few ideas that guided our
Section l a starts with a consideration of higher brain functions, to which return shall be made occasionally in the course of the book, particularly in the final section 5d. Definitions of level of study, and warnings against the dangers of level confusion, provide a common theme. Section lb offers perspectives on the rich historical relations between physics and biology, and sheds light on some interesting differences between physical laws and biological explanations. Section l c puts emphasis on mathematical (logical, statistical) and epistemological notions. Finally, section I d provides an adequate transition to the subsequent chapters of the book introducing some of the fundamental issues in perception.
2
la Forewords; Introductory Warnings
If the reader absorbs the contents of the first three texts in this section, he (henceforth, "he" is a short word for "he/she") will be equipped to skip 90%of the sterile recurrent controversies on the mind-body, the mind-brain and the mind-matter problems, and to avoid most of the false prophets and doctrinaires. T h u s , he will be prepared to face the real debates, for which there is no universal doctrinal panacea. One can hardly but be impressed by the high standard of serenity and modesty already reached by Helmholtz, the 19th century pioneer (a serenity that is perhaps only equaled in this century by the writings of Donald Hebb). T h e fourth text, written by one philosopher, Patricia Churchland, in collaboration with two biologists involved in theoretical and experimental neurosciences, introduces many concepts that are developed in all subsequent chapters. Apart from this survey of computational neurosciences and a thorough discussion of the computer-brain issue, the authors also review and criticise the influential David Marr's three-level theory. The distinction they present between "natural kinds" and "non-natural kinds" is indeed a point that needs to be stressed, especially because it is not so familiar to physicists. For many problems of perception (such as colour vision, pattern recognition, pain, etc.) and learning (generalisation, etc.), it is essential to distinguish between the properties of the physical stimuli and the biases of the perceiving apparatus. T h i s distinction between objective and subjective qualities is a theme that runs through the whole book, and that has many facets. A related question is: "How much is the brain externally (vs. internally) driven"? Many views on conscious-unconscious processes, reduction-construction, analysis-synthesis, are evoked in this section. Perhaps these discussions will help the reader to overcome barriers, and to discover that similar scientific tensions exist within physics and within biology. Historical and general perspectives are further elaborated in the three sections that follow, thus completing the setting of the stage for the whole
3
volume. Some noticeable returns to Helmholtz's insightful remarks on unconscious processes, and to the problem of consciousness, occur in papers (13), (22), (27) and (44) and in Chapter 5, Section 5d. It was tempting to extract some felicitous sentences from the four texts of this section, and bring them together in order to define an "enlightened" epistemology about the matter-life-mind trilogy, but we have resisted the temptation.
4
Reprinted with permission from Anna. Rev. Neurosci., Vol. 13, pp. 1S-24,1990 © 1990 Annual Reviews Inc.
THE MECHANICAL M I N D Horace Barlow Physiological Laboratory, Cambridge, C B 2 3 E G , England
Most neuroscientists accept the machine as a useful metaphor or model of the mind. It points our research in a direction that has been outstandingly successful for more than a century, namely the reductionist analysis of brain function in terms of simpler physical, chemical, and biological processes, and because we can understand all, or almost all, about machines, the metaphor encourages us to think we can discover all, or almost all, about the mind. I thought when I started writing this piece that the metaphor was sound and useful, though its uncritical acceptance made me uneasy, especially because I knew that many of my colleagues in subjects like mathematics and linguistics received it with something close to incredulity, while many others disliked it intensely. I therefore thought it would be worthwhile to examine the idea in more detail, to try to find if there was any justification for unease, incredulity, or dislike; if you read on you will find that I have been forced to the conclusion that the metaphor is misleading and potentially harmful, but that this results from prevalent ignorance and prejudice about machines rather than any gross defect of the metaphor.
IN DEFENCE OF THE ANALOGY I think the main purpose behind calling the mind a machine is to drive out demons. W e are saying, in effect, " L o o k , there is no more to the working of the brain than the physics and chemistry of its componets, just as there is no more to a machine than the physics and chemistry of its components." This is admirable as an invocation not to waste time on mental spirits and to study the physics and chemistry instead, so what are the objections? First there are three minor ones, namely that we don't actually know all about machines, that brains are made of totally different materials, and that they have come into existence in a strikingly different manner. By considering these objections we see where to be cautious about the meta-
5
16 phor. but they are not fatal to it. However, there are more serious problems that do justify mistrust and dislike; first, minds do some things that no current machines do, and if one is mainly interested in these particular tasks the analogy is not much help and could be misleading; second, it is the minds of other people one interacts with, so to treat minds as machines valued mainly for their usefulness has unpleasant ethical implications.
We Do Not Understand All About Machines The person who understands most about a machine is its designer, and no designer of a complex machine would claim that everything about it was perfectly understood. Engineers are really very different from scientists, for they use a body of knowledge to create something new. and provided the goal is reached they are not too concerned if their creation has unforeseen or unknown properties. Scientists make use of the same body of knowledge, but they must pay particular attention to anything unforeseen or unknown, since their goal is to extend the body of knowledge rather than to exploit it. In engineering, intuitive leaps and creative solutions are rightly admired when they make a machine work, but as scientists, we should admire creativity and intuition that enable us to understand what was previously unknown; we do not want to import the attitude "If it works, that's good enough" along with the concept that mind is a product of engineering. The current enthusiasm for neural networks may show that this niggle is partly justified. Here are techniques that enable machines to perform tasks that hitherto lay in the province of the mind, but whereas artificial intelligence and the previous styles of computer simulation led to a more detailed analytical knowledge of the requirements for performing the task, the network style of simulation does not; instead it uses some blind procedure, such as back-propagation, and the success of a simulation is taken as justification for the procedure, rather than for the programmer's analysis oTthe task requirements. T o be fair, the major proponents of the network approach do not regard this as an advantage and claim the method gives insight by, for instance, showing what intermediate elements or "hidden units" enable a task to be done; but one suspects the avoidance of detailed analysis is nonetheless a factor in the popularity of the method.
The Materials Are Very Different The objection that the materials are totally different need not be taken very seriously since, whatever the materials are, they still obey the rules of physics and chemistry. It just means more surprises for the neuroscienttst, since he will come across unexpected properties and methods: birds do not use propellers or jet engines, but they do obey the same laws of aerodynamics as machines that fly.
6
17 Perhaps one should be a little more cautious when it comes to computers and the brain, for the architecture as well as the materials are so different, but surely some at least of the problems encountered when performing a task on a serial computer will carry over to the performance of the same task by the brain. A s long as that is so, we can learn from the analogy.
Minds and Machines Differ in Origin Does it matter that brains are the product of millions of years of genetic evolution combined with a few months' ontogenesis and several years of teaching and experience, whereas machines are designed by those brains and made by human hand? I can't see why it should, and it could be claimed that the explicit knowledge required for design and construction gives deep insight into the nature of the mind. But once again there are cautions. Evolution is an unprincipled and conservative designer. A s a result, one rarely finds neat theoretical solutions embodied in brains, and in particular they often use parallel mechanisms for achieving a single goal. This fact tends to make life difficult for the experimentalist, because it frustrates attempts to do simple and effective controls. F o r instance, one might naively expect that if binocular vision enables people to judge distance, then blocking one eye would seriously impair this capacity, but motion parallax, knowledge of the normal sizes of objects, and other cues leave good distance judgment in one-eyed people. The use of alternative sensory cues for balance can cause confusion (see below), and the presence of many parallel methods has caused untold difficulty in the analysis of homing and other navigational feats in animals. It seems to be hard for us to avoid the automatic assumption that just one method is used to perform some difficult task, and this may partly result from the metaphor of the mechanical mind, for multiple alternative methods are rare in machines.
WHAT THE MIND DOES M a n y of the things the mind and brain do are also done by machines, and when this is the case understanding the machine obviously helps. T o understand how a man balances on one leg, one should understand the principles of servo-feedback, though to reinforce the point made above, one should also know that a man's ability to stand on one leg with his eyes shut does not prove that vision is irrelevant to the task, and the ability to remain upright after the destruction of his vestibular organs does not prove they are unimportant either. But the brain also does things that no machine does, and here we get into trouble.
7
18
Things No Computer Does T o start with, consider the fact that we constantly model the inanimate world around us, and also monitor it for changes. We know thoroughly the route from home to office, and notice when a section of road is being repaired, or when the office door is newly painted. M u c h of this modelmaking is quite automatic and unconscious and we only know it has been done when something changes; we do not consciously monitor the spectral composition of sunlight or the pitch of the front door bell, but it's pretty certain we would notice if they changed. Robots are of course beginning to record and model their environments in order to find their way around in it, but compared with us they are extraordinarily backward, and so far they have little to teach us. But it would be premature to say that, for this reason, the metaphor is wrong or misleading, because the need for modeling has only recently arisen. It is likely that the principles will soon be better understood, and when this happens they may give us new insight into the brain's methods of building useful models of the environment.
Modeling People M o r e serious problems arise when one considers the human brain's propensity to model people. This starts at a very early age, and parents quickly realize that they are not the only ones trying to run the household; a baby quickly becomes pretty expert at parent-control. It seems to me that this really is a most un-machinelike process, and likely to remain so. One reason is that machines are designed to do something useful for their owners, whereas babies are not. N o doubt one could incorporate a few tricks in a computer that would enable it to get the better of its user—in fact many of them appear to do this effortlessly, without deliberate design; but in such cases it is clear upon reflection that they are not really getting the better of us, they are simply failing to give the desired service. T o improve the man-machine interface, an engineer might program a computer to learn about the user's behavior, just as a baby learns about its parents' behavior; but there the analogy ends, for the engineer's purpose would always be to diminish the conflict of wills, while that is not the baby's purpose at all. A machine is by definition intended to be useful, and therefore it must be designed so that its user's will is unopposed as far as possible. Here, then, we have reached a point where the machine is not a good metaphor for the mind, simply because no machine has been designed to do what the mind does. I am not saying there is a theoretical reason that they should not be so designed, only that engineers are not likely to
8
19
explore the means of doing something that it would be useless or counterproductive to do, and still less likely actually to do it. We, as neuroscientists, can explore the problems of one brain out-guessing another, but we shall not at present get. much help in doing so by regarding the mind as a machine, simply because that is not at present the sort of thing machines do. Notice that this argument is only valid for the present; comparative and competitive interactions between computers can be modeled, and this may be very instructive for understanding human interactions. Hence the implications of the metaphor are not static, and it may mean something very different to future generations that have more understanding of the complexity of purely mechanistic interactions. All the same, for the present the metaphor is, at best, unhelpful on such problems.
TAKING THE METAPHOR TOO SERIOUSLY Perhaps this discussion has brought us to the reason for the unease and dislike aroused by the mind-machine metaphor. If we took it seriously, would we not regard other people's minds as machines, and hence be interested in them primarily for their utility to us? Machines do not and should not oppose our wills, and nor should other people, according to the metaphor. This would all be less worrying if the newspapers were not full of the deeds and misdeeds of individuals who behave as though the metaphor was their gospel, and such an attitude is not entirely unfamiliar even in academic circles. Of course the metaphor is not intended to be a moral exhortation saying: "Treat other people's minds as machines that could be useful to you." But when we use it aren't we in effect saying: "It's alright to treat minds as machines, because actually they are"? Thus the metaphor may not be entirely innocuous, since it might influence how people think about minds and consequently how they treat other people. To follow up this thought we need to see how people actually use the idea of the mind.
Distinguishing Minds from Brains So far, encouraged by the mind-machine metaphor, I have used the words "mind" and "brain" more or less interchangeably, but now we see that a distinction might be useful. The brain is what the metaphor properly applies to, while the mind is something different: it is the concept we use to describe the source of other people's, and our own, behavior. Since the concept of a particular person's mind is fashioned out of the observed behavior of that individual, it is a model one's own brain makes of the
9
20 other individual's brain. Thus minds are the brain's models of itself and other brains, and the important thing is that the vast majority of people attribute behavior to such mind-models. A s neuroscientists we believe it is the brain that controls behavior, but this is the belief and terminology of a small minority of experts; others attribute a person's behavior to his mind and care nothing for the beautiful nerve cells that we dedicate our lives to. T h a t may be a pity, but in justification of the majority's attitude, pause to think how many of the facts you learn from this volume will alter the way you treat your colleagues, bring up your chiidcrn. talk to the janitor, or vote at the next election. Y o u r beliefs about other people's minds, on the other hand, clearly do influence your way of life. ;ind it is because the metaphor may modify your attitude to minds that it is potentially obnoxious. One can be thoroughly mechanistic in believing that the brain controls behavior strictly within the laws of physics and chemistry without this distinction between mind and brain becoming a mockery. In most cases one has knowledge of only a minute fraction of the physical and chemical factors that are actually (we mechanistically believe) controlling the brain's output, but this does not stop us from making quite good and reliable judgments about the actions our own and other minds will initiate. There is nothing unusual in a model having such predictive power in spite of its use of incomplete data; in fact, good models stringently select the data they represent, both in the case of one's mental models of the physical environment with the people in it, and in the case of accurate scientific theories such as thermodynamics. The predictive power of a model depends on its correct identification of the dominant controlling factors and their influence, not upon its completeness. A n incomplete model is often more generally useful than a more accurate one, as for example with Newtonian laws of motion.
Minds Take Over the Control of Behavior Figure 1 is an attempt to persuade neuroscientists that minds are actually more important than brains, and to show how this comes about. In stage 1, the mechanistic brain is in sole command, but brain A observes other brains, B for example, also controlling their own behavior. Occasionally they interact, or their behaviors conflict, so brain A builds a model of the way brain B controls what B says and does, based on what A has seen and heard. This model of brain B inside the brain of A is B's mind, shown in step 2; brain A uses his idea of B's mind to predict what B will say and do, and this should be beneficial to A when living in the same environment as B . B of course has done the same and now has its internal model of A ' s brain, A's mind.
10
21 Brain A
Observations received
fj BehavioT~) -
Brain B
(Behavior
J
Interactions Stage 1. Brain A alone controls its own behavior, but observes B also controlling its own behavior. Bruin B docs likewise.
Brain A
( Behavior)
Observations received
Interactions
Brain B
( Behavior)
Stage 2. Brain A has made B's mind, an internal model of B's brain based on what A has observed B doing. This helps A predict B's behavior. Brain B has done the same.
Brain A
( Behavior)
( Behavior) Interactions
Stage 3. Brain A creates a model of itself, A's mind; the behavioral interactions between A and B can now be modeled within A's brain by the interactions of A's mind and B's mind. Minds now control social behavior.
Figure I Minds control behavior. We attribute o i h t r people's actions to their minds, which it therefore seems appropriate to regard as the models our own brains make o f the source or other people's and our o w n behavior. This diagram shows three stages i n the development within the brain of minds that take control of its social behavior, leaving the brain in charge o f die unconscious, automatic actions we regard as mindless.
11
22 At this stage brain A can predict B's solo behavior, but this will not enable him to predict the outcome of interactions between A and B, for these cannot be modeled, since there is nothing inside A ' s brain suitable for B's mind to interact with. Brain A needs to make a model of the way its own brain generates its own behavior; this is A's mind, and the interactions between A's mind and B's mind allow brain A to model the interactions between A and B. By this time the two minds in brain A have become dominant in controlling much of the behavior of A , especially the verbal and other exchanges with other individuals. O n the other hand, nonverbal and nonsocial behavior may still be generated predominantly by brain A, uninfluenced by the minds it has created within itself—these are, as we would say, mindless actions. O f course all the time similar processes will have been going on in brain B, so A ' s models will not have it all their own way when it comes to making accurate predictions. Furthermore, if these mind-models are realistic, they should include minds within themselves; B's mind in brain A should, for example, have a little A's mind inside itself, for by this stage brain B will have its own version of A's mind, as shown in the right half of the figure. This infinite regress can presumably be cut short after a few cycles without seriously affecting the predictive power of the models, but it makes the brain into something more like a hall of mirrors than any currently understood machine. Surely, however, this hall of mirrors accurately portrays the fact that two people's minds interact with each other in an extraordinarily complex and intimate manner. A t the very least, the interactions that this figure attempts to portray will have to be taken into account in any mechanistic accounts of higher human behavior, and I do not see how this can be done without giving the concept of the mind a very dominant role.
Why Minds are More Interesting Now that we have a distinction between mind and brain, we can see why most people are much more interested in one than the other. T h e mathematician, linguist, historian, or literary critic is not concerned with the nuts and bolts of brain mechanisms, but with mind—the abstract model that appears to be responsible for the works that he studies and perhaps produces himself. It is the same with cars, domestic appliances, word processors, and so on: except for the expert, people simply do not care how they work as long as they do the job expected. O f course when minds fail to work as expected we may call in the brain specialist, but until then it is minds we deal with. I think this distinction between the mechanistic brain and the mindmodels it makes has interesting implications for our understanding of pain.
12
23 pleasure, ethics, and consciousness itself. A s I have argued elsewhere, the part of the brain's functioning of which we are consciously aware seems to be confined to the inputs, outputs, and interactions of the m i n d s of Figure 1. Just as ordinary people are more interested in minds than brains, so should be philosophers, for it is the source of peoples' actions rather than their mechanism that is their primary concern. Minds are the conceptual sources of this behavior, and although minds are the product of a mechanistic brain, their actions and interactions are not going to be understood in terms of physics and chemistry alone, even if they are ultimately determined by physics and chemistry alone. Notice that within this entirely mechanistic framework one c a n ask interesting questions about minds: H o w is the stability of social interestion affected by how good a model of B's brain A has in the form of B's mind? Are inaccurate models unethical? O r might it be unethical to have too good a model? H o w about the same questions applied to A ' s m i n d , the model of his own brain? H o w far will taught precepts affect the f o r m of these models? D o we also have mind-models of deities? Surely these are the sort of questions appropriate for theoretical analysis by philosophers and others, and the mechanistic nature of the brain hardly affects them. It is worth adding that such ideas might also form the basis for experiments. T h u s , to neuroscientists, the machine-like aspects of the brain are immediately important, because they firmly direct our attention to the physics and chemistry of the brain. But the metaphor should not be taken to imply that there is no more to the brain than can be described in terms of physics and chemistry, and the concept of mind is certain to be important in any satisfactory account of the way the brain controls social behavior. F o r this reason, philosphers and ordinary people are properly more c o n cerned with minds than brains.
CONCLUSIONS T o conclude, for most people in most circumstances the physical a n d chemical causation of the brain's output is not as important as the observable behavior that actually occurs; minds are what we attribute this behavior to, and our common-se"nse understanding of them will almost always give more useful predictions than knowledge of the physical a n d chemical processes that underlie them. F o r neuroscientists trying to refine such mechanistic accounts, the metaphor is appropriate and encouraging, but the dislike it arouses in others seems fully justified by its implication that other people's minds are mere machines and can be treated as s u c h . What is needed to prevent the metaphor of the mechanical mind sanctioning unbridled egocentric behavior is to get rid of the prejudice that
13
24
machines are essentially simple and deterministic, and to gain an appreciation of the complexity and difficulties in predicting behavior produced by two or more minds interacting in the manner shown in Figure 1. There is no reason to doubt that the brain is entirely mechanical, but it is a wonderful mechanism that can generate and use the concept of mind. ACKNOWLEDGEMENTS
I would like to acknowledge considerable help toward clarifying these ideas from Ian Glynn, Graeme Mitchison, Kathy Mullen, and Miranda Weston-Smith.
14
Reprinted with permission from On the Sensations of Tone. pp.366-371. 1954 © 1954 Dover Publications Inc.
E S T H E T I C ANALYSIS O F WORKS O F A R T
The esthetic analysis of complete musical works of art, and the comprehension of the reasons of their beauty, encounter apparently invincible obstacles at almost every point. But in the field of elementary musical art we have now gained so much insight into its internal connection that we are able to bring the results of our investigations to bear on the views which have been formed and in modem times nearly universally accepted respecting the cause and character of artistic beauty in general. It is, in fact, not difficult to discover a close connection and agreement between them ; nay, there are probably fewer examples more suitable than the theory of musical scales and harmony, to illustrate the darkest and most difficult points of general esthetics. Hence I feel that I should not be justified in % passing over these considerations, more especially as they are closely connected with the theory of sensual perception, and hence with physiology in general. No doubt is now entertained that beauty is subject to laws and rules dependent on the nature of human intelligence. The difficulty consists in the fact that these laws and rules, on whose fulfilment beauty depends and by which it must be judged, are not consciously present to the mind, either of the artist who creates tbe work, or the observer who contemplates it. Art works with design, but the work of art ought to have the appearance of being undesigned, and must be judged on that ground. Art createB as imagination pictures, regularly without conscious law, designedly without conscious aim. A work, known and acknowledged as the product of mere intelligence, will never be accepted as a work of art, however perfect be its adaptation to its end. Whenever we see that conscious reflection has acted in tbe arrangement of the whole, we find it poor. "'
Man fiihlt die Absicht, und man wird verstimmt. (We feel the purpose, und it jars upon us.)
And yet we require every work of art to be reasonable, and we shew this by subjecting it to a critical examination, and by seeking to enhance our enjoyment and our interest in it by tracing out the suitability, connection, and equilibrium of all its separate parts. The more we succeed in making the harmony and beauty of all its peculiarities clear and distinct, the richer we find it, and we even regard as the principal characteristic of a great work of art that deeper thought, reiterated observation, and continued reflection shew ua more and more clearly the reasonableness of all its individual parts. Our endeavour to comprehend the beauty of such a work by critical examination, in which wa partly succeed, shews that we assume a certain adaptation to reason in works of art, which may possibly rise to a conscious understanding, although such understanding is neither necessary for the invention nor for the enjoyment of the beautiful. For what is esthetically beautiful is recognised by the immediate judgment of a cultivated taste, which declares
15
3°7 it pleasing or displeasing, without any comparison whatever with law or conception. But that we do not accept delight in the beautiful as something individual, but rather hold it to be in regular accordance with the nature of mind in general, appears by our expecting and requiring from every other healthy human intellect the same homage that we ourselves pay to what we call beautiful. At most we allow that national or individual peculiarities of taste incline to this or that artistic ideal, and are most easily moved by it, precisely in the same way that a certain amount of education and practice in the contemplation of fine works of art is undeniably necessary for penetration into their deeper meaning. The principal difficulty in pursuing this object, is to understand how regularity can be apprehended by intuition without being consciously felt to exist. And this unconsciousness of regularity is not a mere accident in the effect of the beautiful on our mind, which may indifferently exist or not; it is, on the contrary, most ^ clearly, prominently, and essentially important. For through apprehending everywhere traces of regularity, connection, and order, without being able to grasp the law and plan of the whole, there arises in our mind a feeling that the work of art winch we are contemplating is the product of a design which far exceeds anything we can conceive at the moment, and which hence partakes of the character of the illimitable. Eemembering the poet's words: Dn gleiohst dem Geiat, dec du begreifst, (Thou'rt like the spirit thou couceivest),
we feel that those intellectual powers which were at workin the artist, are far above our conscious mental action, and that were it even possible at all, infinite time, meditation, and labour would have been necessary to attain by conscious thought that degree of order, connection, and equilibrium of all parts and all internal relations, which the artist has accomplished under the sole guidance of tact and taste, and which we have in turn to appreciate and comprehend by our own tact and taste, long before we begin a critical analysis of the work. It is clear that all high appreciation of the artist and his work reposes essentially on this feeling. In the first we honour a genius, a spark of divine creative fire, which far transcends the limits of our intelligent and conscious forecast. And yet the artist is a man as we are, in whom work the same mental powers as in ourselves, only in their own peculiar direction, purer, brighter, steadier; and by the greater or less readiness and completeness with which we grasp the artist's language we measure our own share of those powers which produced the wonder. Herein is manifestly the cause of that moral elevation and feeling of ecstatic satisfaction which is called forth by thorough absorption in genuine and lofty works of art. We learn from them to feel that even in the obscure depths of a healthy and harmoniously developed human mind, which are at least for the present inaccessible to analysis by conscious thought, there slumbers a germ of order that ^ is capable of rich intellectual cultivation, and we learn to recognise and admire in the work of art, though draughted in unimportant material, the picture of a similar arrangement of the universe, governed by law and reason in all its parts. The contemplation of a real work of art awakens our confidence in the originally healthy nature of the human mind, when uncribbed, unharassed, unobscured, and unfalsified. But for all this it is an essential condition that the whole extent of the regularity and design of a work of art should not he apprehended consciously. It is precisely from that part of its regular subjection to reason, which escapes our conscious apprehension, that a work of art exalts and delights us, and that the chief effects of the artistically beautiful proceed, not from the part which we are able fully to analyse. If we now apply these considerations to the system of musical tones and harmony, we see of course that these are objects belonging to an entirely subordinate
16
368 Application to Music
and elementary domain, but nevertheless they, too, are slowly matured inventions of the artistic taste of musicians, and consequently they, too, must be governed by the general ruleB of artistic beauty. Precisely because we are here still treading the lower walks of art, arid are not dealing with the expression of deep psychological problems, we are able to discover a comparatively simple and transparent solution of that fundamental enigma of esthetics. The whole of the last part of this book has explained how musicians gradually discovered the relationships between tones and chords, and how the invention of harmonic music rendered these relationships closer, and clearer, and richer. We have been able to deduce the whole system of rules which constitute Thorough Bass, from an endeavour to introduce a clearly sensible connection into the series of tones which form a piece of music. A feeling for the melodic relationship of consecutive tones, was first developed, •J commencing with Octave and Fifth and advancing to the Third. We have taken pains to prove that this feeling of relationship was founded on the perception of identical partial tones in the corresponding compound tones. Now these partial tones are of course present in the sensations excited in our auditory apparatus, and yet they are not generally the subject of conscious perception as independent sensations. The conscious perception of everyday life ia limited to the apprehension of the tone compounded of these partials, as a whole, just aa we apprehend tbe taste of a very compound dish as a whole, without clearly feeling how much of it is due to the salt, or the pepper, or other spices and condiments. A critical examination of our auditory sensations as such was required before we could diacover the existence of upper partial tonea. Hence the real reason of the melodic relationship of two tones (with the exception of a few more or less clearly expreaaed conjectures, as. for example, by Rameau and d'Alembert) remained so long undiscovered, or at least was not in any respect clearly and definitely formulated. I believe that I have ^ been able to furnish the required explanation, and hence clearly to exhibit the whole connection of the phenomena. The esthetic problem is thus referred to the common property of all sensual perceptions, namely, the apprehension of compound aggregates of sensations as sensible symbols of simple external objects, without analysing them. In our usual observations on external nature our attention is so thoroughly engaged by external objects that we are entirely unpractised in taking for the subjects of conscious observation, any properties of our sensations themselves, which we do not already know as the sensible expression of some individual external object or event. After musicians had long been content with the melodic relationship of tones, they began in the middle ages to make uae of harmonic relationship as shewn in consonance. The effects of various combinations of tones also depend partly on the identity or difference of two of their different partial tones, but they likewise partly depend on their combinational tonea. Whereas, however, in melodic „ relationahip the equality of the upper partial tonea can only be perceived by remembering tbe preceding compound tone, in harmonic relationship it is determined by immediate sensation, by the presence or absence of beats. Hence in harmonic combinations of tone, tonal relationship ia felt with that greater liveliness due to a present sensation as compared with the recollection of a past sensation. The wealth of clearly perceptible relations grows with the number of tones combined. Beats are easy to recognise as such when they occur slowly; but those which characterise dissonances are, almost without exception, very rapid, and are partly covered by sustained tones which do not beat, so that a careful comparison of slower and quicker beats is necessary to gain the conviction that the essence of dissonance consists precisely in rapid beats. Slow beats do not create the feeling of disson&nce, which does not arise till the rapidity of the beats confuses the ear and makes it unable to distinguish them. In this case also the ear feels the difference between the undiaturbed combination of sound in the case of two consonant tones, and the disturbed rough combination resulting from a dissonance. But, as
17
3<>9
s general rule, the hearer is then perfectly unconscious of the cause to which the Unconscious disturbance and roughness are due. Sense of The development of harmony gave rise to a much rioheT opening out of musical Resemblance art than was previously possible, because the far clearer characterisation of related combinations of tones by means of chords and ehordal sequences, allowed of the use of much more distant relationships than were previously available, by modulating into different keys. In this way the means of expression greatly increased as well as the rapidity of the melodic and narmonic transitions which could now be introduced without destroying the musical connection. As the independent significance of chords came to be appreciated in the fifteenth and sixteenth centuries, a feeling arose for the relationship of chords to one another and to the tonic chord, in accordance with the same law which had long ago unconsciously regulated the relationship of compound tones. The relationship of compound tones depended on the identity of two or more partial tones, that of • chords on the identity of two or more notes. For the musician, of course, the law of the relationship of chords and keys is much more intelligible than that of compound tones. He readily hears tbe identical tones, or sees them in the notes before him. But the unprejudiced and uninstructed bearer is as little conscious of the reason of the connection of a clear and agreeable series of fluent chords, as he is of the reason of a well-connected melody. He is startled by a false cadence and feels its unexpectedness, but is not at all necessarily conscious of the reason of its unexpectedness. Then, again, we have seen that the reason why a chord in music appears to be the chord of a determinate root, depends as before upon the analysis of a compound tone into its partial tones, that is, as before upon those elements of a sensation which cannot readily become subjects of conscious perception. This relation between chords is of great importance, both in the relation of tbe tonic chord to the tonic tone, and in the sequence of chords. j The recognition of these resemblances between compound tones and between chords, reminds us of other exactly analogous circumstances which we must have often experienced. We recognise the resemblance between the faces of two near relations, without being at all able to say in what the resemblance consists, especially when age and sex are different, and the coarser outlines of the features consequently present striking differences. And yet notwithstanding these differ, ences—notwithstanding that we are unable to fix upon a single point in the two countenances which is absolutely alike—the resemblance is often so extraordinarily striking and convincing, that we have not a moment's doubt about it. Precisely the same thing occurs in recognising the relationship between two compound tones. Again, we are often able to assert with perfect certainty, that a passage not previously heard is due to a particular author or composer whose other works we know. Occasionally, but by no means always, individual mannerisms in verbal or f musical phrases determine our judgment, but as a rule we are mostly unable to fix upon the exact points of resemblance between the new piece and the known works of the author or composer. The analogy of these different cases may be even carried farther. When a father and daughter are strikingly alike in some well-marked feature, as the nose or forehead, we observe it at once, and think no more about it. But if the resemblance is so enigmatically concealed that we cannot detect it, we are fascinated, and cannot help continuing to compare their countenances. And if a painter drew two such heads having, say, a somewhat different expression of character combined with a predominant and striking, though indefinable, resemblance, we should undoubtedly value it as one of the principal beauties of his painting. Our ad. miration would certainly not be due merely to his technical skill; we should rather look upon his painting as evidencing an unusually delicate feeling for the
18
37°
Unconscious Sense of Tonal Relationship
significance of the human countenance, and find in this the artistic justification of his work. Now the case Is similar for musical intervals. The resemblance of an Octave to its root is so great and striking that the dullest ear perceives it; the Octave seems to he almost a pure repetition of the root, as it, in fact, merely repeats a part of the compound tone of its root, without adding anything new. Hence the esthetical effect of an Octave is that of a perfectly simple, but little attractive interval. The most attractive of the intervals, melodically and harmonically, are clearly the Thirds and Sixths,—the intervals which lie at the very boundary of those that the ear can grasp. The major Third and the major Sixth cannot be properly appreciated unless the first five partial tones are audible. These are present in good musical qualities of tone. The minor Third and the minor Sixth are for the most part justifiable only as inversions of the former intervals. The more complicated •' intervals in the scale cease to have any direct or easily intelligible relationship. They have no longer tbe charm of the Thirds. Moreover, it is by no means a merely external indifferent regularity which the employment of diatonic scales, founded on the relationship of compound tones, has introduced into the tonal material of music, as, for instance, rhythm introduced some such external arrangement into the words of poetry. I have shewn, on the contrary, in Chapter X I V . , that this construction of the scale furnished a means of measuring the intervals of their tones, so that the equality of two intervals lying in different sections of the scale would be recognised by immediate sensation. Thus tbe melodic step of a Fifth is always characterised by having the second partial tone of the second note identical with tbe third of the first. This produces a definiteness and certainty in the measurement of intervale for our sensation, such as might be looked for in vain in the system of colours, otherwise so similar, or in the estimation of mere differences of intensity in our various sensual If perceptions. Upon this reposes also the characteristic resemblance between the relations of the musical scale and of space, a resemblance which appears to me of vital importance for the peculiar effects of music. It is an essential character of space that at every position within it like bodies can be placed, and like motions can occur. Everything that is possible to happen in one part of space is equally possible in every other part of space and is perceived by us in precisely the same way. This is the case also with the musical scale. Every melodic phrase, every chord, which can be executed at any pitch, can be also executed at any other pitch in such a way that we immediately perceive the characteristic marks of their similarity. On the other hand, also, different voices, executing the same or different melodic phrases, can move at the same time within the compass of the scale, like two bodies in space, and, provided they are consonant in the accented parts of bars, without creating any musical disturbances. Such a close analogy consequently exists in If all essential relations between tbe musical scale and space, that even alteration of pitch has a readily recognised and unmistakable resemblance to motion in space, and is often metaphorically termed the ascending or descending motion or progression of a part. Hence, again, it becomes possible for motion in music to imitate the peculiar characteristics of motive forces in space, that is, to form an image of the various impulses and forces which he at the root of motion. And on this, as I believe, essentially depends the power of music to picture emotion. It is not my intention to deny that music in its initial state and simplest forms may have been originally an artistic imitation of the instinctive modolations of the voice that correspond to various conditions of the feelings. But I cannot think that this is opposed to the above explanation ; for a great part of the natural means of vocal expreesion may ba reduced to such facts as the following: its rhythm and accentuation are an immediate expression of the rapidity or force of the corresponding psychical motives—all effort drives the voice up—a desire to make a pleasant impression on another mind leads to selecting a softer, pleasanter quality of
19
37« tone—and so forth. An endeavour to imitate the involuntary modulations of the Expression voice and make its recitation richer and more expressive, may therefore very posof Motion sibly have led our ancestors to the discovery of the first means of musical expresin Music sion, just as the imitation of weeping, shouting, or sobbing, and other musical delineations may play a part in even cultivated music, (as in operas), although such modifications of tbe voice are not confined to the action of free mental motives, but embrace really mechanical and even involuntary muscular contractions. But it is quite clear that every completely developed melody goes far beyond an imitation of nature, even if we include tbe cases of the most varied alteration of voice under the influence of passion. Nay, the very fact that music introduces progression by fixed degrees both in rhythm and in the scale, renders even an approximative^ correct representation of nature simply impossible, for most of the passionate affections of the voice are characterised by a gliding transition in pitch. The imitation of nature is thus rendered as imperfect as the imitation of IT a picture by embroidery on a canvas with separate little squares for each shade of colour. Music, too, departed still further from nature when it introduced tbe greater compass, the mobility, and the strange qualities of tone belonging to musical instruments, by which tbe field of attainable musical effects has become so much wider than it was or could be when tbe human voice alone was employed. Hence though it is probably correct to say that mankind, in historical development, first learned the means of musical expression from tbe human voice, it can hardly be denied that these same means of expressing melodic progression act, in artistically developed music, without tbe slightest reference to tbe application made of them in the modulations of the human voice, and have a more general significance than any that can be attributed to innate instinctive cries. That this is the case appears above all in the modern development of instrumental music, which possesses an effective power and artistic justification that need not be gainsaid, although we may not yet be able to explain it in all its details. IT
Here J close my work. It appears to me that I have carried it as far as the physiological properties of tbe sensation of hearing exercise a direct influence on the construction of a musical system, tbat is, as far as the work especially belongs to natural philosophy. For even if I could not avoid mixing up esthetic problems with physical, the former were comparatively simple, and the latter much more complicated. This relation would necessarily become inverted if I attempted to proceed further into the esthetics of music, and to enter on the theory of rhythm, forms of composition, and means of musical expression. In all these fields the properties of sensual perception would of course have an influence at times, but only in a very subordinate degree. The real difficulty would lie in the development of the psychical motives which here assert themselves. Certainly this is the point II where the more interesting part of musical esthetics begins, tbe aim being to explain the wonders of great works of art, and to learn the utterances and actions of the various affections of the mind. But, however alluring such an aim may be, I prefer leaving others to carry out such investigations, in which I should feel myself too much of an amateur, while I myself remain on the safe ground of natural philosophy, in which I am at home.
20
Reprinted wilh ; .- i ::•-•> i on from Science. Vol. 177, pp. 393-396. 4 Aug 1972 ' 1972 The Ameriuin Association for the Advancement of Science
less r e l e v a n c e t h e y s e e m t o have t o t h e very
real
ence,
problems o f the
much
less
to
rest o f s c i -
those
of
society,
T h e constructionist hypothesis breaks down
More Is Different
when
confronted
with
the
twin
d i f f i c u l t i e s o f scale a n d c o m p l e x i t y . T h e behavior
o f large
and
gates o f e l e m e n t a r y
complex
particles,
turns
in
terms
a simple extrapolation o f the
prop-
o u t . is n o t t o be u n d e r s t o o d
B r o k e n symmetry and the nature of
of
the hierarchical structure of science.
aggreit
erties
of
a
few
particles-
Instead,
fit
each level o f c o m p l e x i t y e n t i r e l y new properties P. W .
ing
Anderson
appear, and the
o f The n e w
understand-
behaviors
requires
s e a r c h w h i c h I Chink is as in
its n a t u r e
as
seems t o m e
any
that
other.
one
re-
fundamental That
may
is, i t
array
the
sciences r o u g h l y l i n e a r l y i n a h i e r a r c h y , T h e reductionist hypothesis m a y
slill
be a l o p i c t o r c o n t r o v e r s y a m o n g
phi-
l o s o p h e r ^ b u t a m o n g [he g r e a t m a j o r i t y o f a c t i v e s c i e n t i s t s I t h i n k i t is a c c e p t e d without question. T h e w o r k i n g s of o u r m i n d s a n d bodies,
and
o f all the
ani-
m a t e Or i n a n i m a t e m a t t e r o f w h i c h
we
have
as-
any
detailed
knowledge,
are
s u m e d t o be c o n t r o l l e d b y the s a m e set of
fundamental
under
certain
laws, extreme
which
except
conditions
we
feel w e k n o w p r e t t y w e l l . Tt seems i n e v i t a b l e t o g o o n
uncrit-
i c a l l y t o w h a t a p p e a r s at first s i g h t be ism:
an
obvious
corollary of
reduction-
that i f e v e r y t h i n g obeys the
fundamental
laws,
then
to
the
same
only
sci-
e n t i s t s w h o are s t u d y i n g a n y t h i n g r e a l l y
plan ai ion o f phenomena in terms o f k n o w n fundamental laws. A s always, distinctions o f this k i n d are n o t unambiguous, but I hey are clear in most cases. Solid state physics, plasma physics, and perhaps also b i o l o g y are extensive. H i g h energy physics a n d a g o o d p a r i o f nuclear physics arc intensive. There is always m u c h less intensive research going o n t h a n extensive. Once new f u n d a m e n t a l laws are discovered, a large and ever increasing activity begins In order to apply the discoveries to h i t h e r t o unexplained phenomena. Thus, there are t w o dimensions t o basic research. T h e f r o n t i e r o f science extends all along a Jong line f r o m the newest and most m o d e r n iniensive research, over the extensive research recently spawned b y the intensive research o f yesterday, t o the broad and w e l l developed w e b o f extensive research activities based o n intensive research o f past decades.
to
some
astrophysicists,
some
and
other
mathematicians,
known
this
by
But [hat
this
hierarchy
does
not
imply
s c i e n c e X is " j u s t a p p l i e d ¥ , " A t
e a c h stage e n t i r e l y n e w l a w s , generalizations
are
concepts,
necessary,
re-
as great a d e g r e e as i n t h e p r e v i o u s One, P s y c h o l o g y is n o t a p p l i e d b i o l o g y , n o r
of
is b i o l o g y a p p l i e d
and
few
article
Weisskopf
physiology psychology
be i n d i c a t e d b y t h e f a c t t h a t f h e a r d i t
is e x p r e s s e d i n a r a t h e r passage
psychology social sciences
q u o t e d recently b y a leader i n the held materials
science,
"fundamental
of
Y elementary particle physics many-body physics chemistry molecular biology
elemen-
the
purpose
X solid state o r m a n y - b o d y physics chemistry molecular biology cell b i o l o g y
logicians
participants
main
science Y .
q u i r i n g i n s p i r a t i o n a n d c r e a t i v i t y t o just
o t h e r s . T h i s p o i n t o f v i e w , w h i c h i t is oppose,
elementary
T h e effectiveness o f t h i s message m a y
amounts
tary particle physicists, some
idea; T h e
e n t i t i e s o f science X o b e y t h e l a w s o f
and
f u n d a m e n t a l are t h o s e w h o are w o r k i n g on those laws. I n practice, that
a c c o r d i n g to t h e
to
well-
{!):
matter
who
urged
at a m e e t i n g d e d i c a t e d problems
physics"
to
in
accept
there
w e r e few o r no such p r o b l e m s a n d
that
science,
w h i c h he s e e m e d t o e q u a t e w i t h
device
engineering. The
main
fallacy
in
this
kind
of
t h i n k i n g is t h a t t h e r e d u c t i o n i s t h y p o t h esis does
not
by any
"constructionist''
one;
means The
tal
l a w s does n o t
start
from
imply
imply
ability
fundamen-
the ability
those laws a n d
a to to
reconstruct
t h e u n i v e r s e . I n f a c t , t h e m o r e t h e elem e n t a r y p a r t i c l e p h y s i c i s t s t e l l us a b o u t the nature o f the fundamental laws, the
21
chemistry.
I n m y o w n field o f m a n y - b o d y p h y s ics,
w e are, p e r h a p s , c l o s e r t o o u r
fun-
damental, intensive underpinnings
than
in
non-
any
other
science
in
which
t r i v i a l c o m p l e x i t i e s o c c u r , a n d as a sult
we
general
reduce everything to simple The author it a member • ! Ihe technical ElafT of the Bel] Telephone Laboratories Murray Hill. New JerMf 07974, and »i*IUns protewor oE IhEoretical prijiic* at Cavandiili Laboratory. Cambridge. England. Thi* article l i i n expanded uenton ot a Rescnta' Lecture prcc In 1967 at IhB UnJ-ErtiEy el California, La Jail*.
to
condensed that
n o t h i n g was left but extensive L o o k i n g at the development o f science in ihe T w e n t i e t h C e n t u r y one can dist i n g u i s h i w o trends, w h i c h J w i l l call " i n t e n s i v e " a n d "extensive" research, l a c k ing a better t e r m i n o l o g y . I n s h o r t : i n tensive research goes f o r the fundamental laws, extensive research goes f o r the ex-
the
from
have theory
begun
to
o f just
formulate
how
this
quantitative to qualitative
rea
shift differ-
entiation takes place. T h i s f o r m u l a t i o n , called
ihe
theory
of
"broken
sym-
m e t r y . " m a y be o f h e l p i n m a k i n g m o r e generally
clear
constructionist ism.
Che b r e a k d o w n converse
of
of
the
reduction-
I w i l l give an elementary
and in-
c o m p l e t e e x p l a n a t i o n o f these ideas, a n d t h e n go o n t o s o m e m o r e g e n e r a l speculative
comments
about
analogies
at
other levels and about similar phenomena. Before beginning this I wish to sort out two possible sources of misunderstanding. First, when I speak of scale change causing fundamental change I do not mean the rather well-understood idea that phenomena at a new scale may obey actually different fundamental laws—as, for example, general relativity is required on the cosmological scale and quantum mechanics on the atomic. 1 think it will be accepted that all ordinary matter obeys simple electrodynamics, and quantum theory, and that really covers most of what I shall discuss. (As I said, we must all star! with reductionism, which 1 fully accept.) A second source of confusion may be the fact that the concept of broken symmetry has been borrowed by the elementary particle physicists, but their use of the Term is strictly an analogy, whether a deep or a specious one remaining to be understood. Let me then start my discussion with an example on the simplest possible level, a natural One for me because 1 worked wilh it when I was a graduate student: the ammonia molecule. At that time everyone knew about ammonia and used it to calibrate his theory or his apparatus, and I was no exception. The chemists will tell you that ammonia "is" a triangular pyramid
with the nitrogen negatively charged and the hydrogens positively charged, so that it has an electric dipole moment (p), negative toward the apex of the pyramid. Now this seemed very strange to me. because I was just being taught ihat nothing has an electric dipole moment. The professor was really proving that no nucleus has a dipole moment, because he was leaching nuclear physics, but as his arguments were based on the symmetry of space and lime they should have been correct in general, 1 soon learned that, in facl they were correct (or perhaps it would be more accurate to say not incorrect) because he had been careful to say that no stationary state of a system (that is. one which does not change in time) has an electric dipole moment. If ammonia starts out from Ihe above unsymmelrical state, it will not slay in it very long. By means of quantum mechanical tunneling, the nitrogen can T
leak through the triangle of hydrogens to the other side, turning the pyramid inside oul, and, in fact, it can do so very rapidjy. This is the so-called "inversion," which occurs at a frequency of about 3 x 10 ° per second. A truly stationary state can only be an equal superposition of the unsymmetrical pyramid and its inverse. That mixture does not have a dipole moment. (I warn Ihe reader again that 1 am greatly oversimplifying and refer him to the textbooks for details.) l
I will not go through the proof, but the result is that the state of the system, if it is to be stationary, must always have the same symmetry as the laws of motion which govern it. A reason may be pur very simply: In quantum mechanics there is always a way, unless symmetry forbids, to get from one state to another. Thus, if we start from any one unsymmetrical state, the system will make transitions to others, so only by adding up all the possible unsymmetrical states in a symmetrical way can we get a stationary state. The symmetry involved in the case of ammonia is parity, the equivalence of left- and right-handed ways of looking at things. (The elementary particle experimentalists' discovery of certain violations of parity is not relevant to this question; those effects are too weak to aflecl ordinary matter.) Having seen how the ammonia molecule satisfies our theorem that there is no dipole moment, we may look into other cases and. in particular, study progressively bigger systems to see whether the state and the symmetry are always related. There are other similar pyramidal molecules, made of heavier atoms. Hydrogen phosphide, PH . which is twice as heavy as ammonia, inverts, but at one-tenth the ammonia frequency. Phosphorus trifluoride, PF . in which the much heavier fluorine is substituted for hydrogen, is not observed to invert at a measurable rate, although theoretically one can be sure that a state prepared in one orientation would invert in a reasonable time. 3
a
We may then go on to more complicated molecules, such as sugar, with about 40 atoms. For these it no longer makes any sense to expect the molecule to invert itself. Every sugar molecule made by a living organism is spiral in the same sense, and they never invert, either by quantum mechanical tunneling or even under thermal agitation at normal temperatures. At this point we must forget ahout the possibility of inversion and ignore the parity symmetry:
22
the symmetry laws have been, not repealed, but broken. If, on the other hand, we synthesize our sugar molecules by a chemical reaction more or less in thermal equilibrium, we will find that there are not, on the average, more left- than righthanded ones or vice versa. In the absence of anything more complicated than a collection of free molecules, the symmetry laws are never broken, on the average. We needed living matter to produce an actual unsymmetry in the populations. In really large, but still inanimate, aggregates of atoms, quite a different kind of broken symmetry can occur, again leading to a net dipole moment or to a net optical rotating power, or both. Many crystals have a net dipole moment in each elementary unit cell (pyroelectricity), and in some this moment can be reversed by an electric field (ferroelectricity). This asymmetry is a spontaneous effect of the crystal's seeking its lowest energy state. Of course, the state with the opposite moment also exists and has, by symmetry, just the same energy, but the system is so large that no thermal or quantum mechanical force can cause a conversion of one to the other in a finite lime compared to. say, the age of the universe. There arc at least three inferences to be drawn from this. One is that symmetry is of great importance in physics. By symmetry we mean the existence of different viewpoints from which the system appears the same, I I is only slightly overstating the case to say that physics is the study of symmetry. The first demonstration of the power of this idea may have been by Newton, who may have asked himself the question: What if the matter here in my hand obeys the same laws as that up in the sky— that is, what if space and matter are homogeneous and isotropic? The second inference is that the internal structure of a piece of matter need not be symmetrical even if the total state of it is. I would challenge you to start from the fundamental laws of quantum mechanics and predict the ammonia inversion and its easily observable properties without going through the stage of using The unsymmetrical pyramidal structure, even though no 'state" ever has that structure. It is fascinating that it was not until a couple of decades ago (2) that nuclear physicists stopped thinking of the nucleus as a featureless, symmetrical little hall and realized that while it really never has a dipole moment, it can become footballH
shaped or plate-shaped. This has observable consequences in the reactions and excitation spectra that are studied in nuclear physios, even though it is much more difficult to demonstrate directly than the ammonia inversion. In my opinion, whether or not one calls this intensive research, it is as fundamental in nature as many things one might so label. But it needed no new knowledge of fundamental laws and would have been extremely difficult to derive synthetically from those laws; it was simply an inspiration, based, to be sure, on everyday intuition, which suddenly fitted everything together. The basic reason why this result would have been difficult to derive is an important one for our further thinking. If the nucleus is sufficiently small there is no real way to define its shape rigorously: Three or four or ten particles whirling about each other do not define a rotating "plate" or "football." It is only as the nucleus is considered to be a many-body system—in what is often called the JV -* «i limit—that such behavior is rigorously definable. We say to ourselves: A macroscopic body of that shape would have such-and-such a spectrum of rotational and vibrational excitations, completely different in nature from those which would characterize a featureless system. When we see such a spectrum, even not so separated, and somewhat imperfect, we recognize that the nucleus is, after all. not macroscopic; it is merely approaching macroscopic behavior. Starting with the fundamental laws and a computer, we would have to do two impossible things —solve a problem with infinitely many bodies, and then apply the result to a finite system—before we synthesized this behavior. A third insight is that the state of a really big system does not at all have to have the symmetry of the laws which govern it; in fact, it usually has less symmetry. The outstanding example of this is the crystal: Buili from a substrate of atoms and space according to laws which express the perfect homogeneity of space, the crystal suddenly and unpredictably displays an entirely new and very beautiful symmetry. The general rule, however, even in the case of the crystal, is that the large system is less symmetrical than the underlying structure would suggest: Symmetrical as it is, a crystal is less symmetrical than perfect homogeneity. Perhaps in the case of crystals this appears to be merely an exercise in confusion. The regularity of crystals
could be deduced •-. :miempirically in the mid-19th century without any complicated reasoning at all. But sometimes, as in the case of superconductivity, the new symmetry—now called broken symmetry because the original symmetry is no longer evident—may be of an entirely unexpected kind and extremely difficult to visualize. In (he case of superconductivity. 30 years elapsed between the time when physicists were in possession of every fundamental law necessary for explaining it and the time when it was actually done. The phenomenon of superconductivity is the most spectacular example of tbe broken symmetries which ordinary macroscopic bodies undergo, but it is of course not the only one. Antiferromagnets, ferroelectrics. liquid crystals, and matter in many other states obey a certain rather general scheme of rules and ideas, which some many-body theorists refer to under the general heading of broken symmetry. J shall not further discuss the history, but give a bibliography at the end of this article {3).
1 do not mean to give the impression that all is settled. For instance. I think there are still fascinating questions of principle about glasses and other amorphous phases, which may reveal even more complex types of behavior. Nevertheless, the role of this type of broken symmetry in the properties of inert but macroscopic material hodies is now understood, at least in principle. In this case we can see how the whole becomes not only more than but very different from the sum of its parts. The next order of business logically is to ask whether an even more complete destruction of the fundamental symmetries of space and time is possible and whether new phenomena then arise, intrinsically different from tbe "simple" phase transition representing a condensation into a less symmetric state.
We have already excluded the apparently unsymmetric cases of Liquids, gases, and glasses. (In any real sense they are more symmetric.) It seems to me that the next stage is to consider the The essential idea is that in tbe so- system which is regular but contains called N —> « limit of large systems (on information. That is. it is regular irr our own. macroscopic scale) it is not space in some sense so that it can be only convenient but essential to realize "read out," but it contains elements that matter will undergo mathematically which can be varied from one "cell" sharp, singular "phase transitions" to to the next. An obvious example is states in which the microscopic sym- DNA; in everyday life, a line of type metries, and even the microscopic equa- or a movie film have the same structions of motion, are in a sense violated. ture. This type of "information-hearing The symmetry leaves behind as its ex- crystallinily' seems to be essential to pression only certain characteristic be- life. Whether the development of life haviors, for instance, long-wavelength requires any further breaking of symvibrations, of which the familiar exam- metry is by no means clear. ple is sound waves; or the unusual macKeeping on with the attempt to charroscopic conduction phenomena of the superconductor: or, in a very deep acterize types of broken symmetry analogy, the very rigidity of crystal lat- which occur in living things, 1 find that tices, and thus of most solid matter- at least one further phenomenon seems There is, of course, no question of the to be identifiable and either universal or system's really violating, as opposed to remarkably common, namely, ordering breaking, the symmetry of space and (regularity or periodicity) in the time time, but because its parts find it ener- dimension. A number of theories of life getically more favorable to maintain cer- processes have appeared in which regtain fixed relationships with each other, ular pulsing in time plays an important the symmetry allows only the body as role: theories of development, of growth a whole to respond to external forces. and growth limitation, and of the memory. Temporal regularity is very commonly observed in living objects. U This leads to a "rigidity," which is plays at least two kinds of roles. First, also an apt description of superconduc- most methods of extracting energy from tivity and superfluidity in spite of their the environment in order to set up a apparent "fluid" behavior. [In the for- continuing, quasi-stable process involve mer case, London noted this aspect lime-per in die machines, such as oscilvery early {4).\ Actually, for a hypo- lators and generators, and the processes thetical gaseous but intelligent citizen of of life work in the same way. Second, Jupiter or of a hydrogen cloud some- temporal regularity is a means of hanwhere in the galactic cooler, the proper- dling information, similar to informaties of ordinary crystals might well be tion-bearing spatial regularity. Human a more baffling and intriguing puzzle spoken language is an example, and it than those of superfluid helium. h
23
is noteworthy that all! computing machines use temporal pulsing. A possible third role is suggested in some of the theories mentioned above: the use of phase relationships of temporal pulses to handle information and control the growth and development of cells and organisms &).
possible; analysis, on the other hand, may be not only possible but fruitful in all kinds of ways: Without an understanding of the broken symmetry in superconductivity, for instance, Josephson would probably not have discovered his effect. (Another name for the Josephson effect is "macroscopic quantum-interference phenomena": interference efIn some sense, structure'—functional structure in a teleologies! sense, as op- fects observed between macroscopic posed to mere crystalline shape—must wave functions of electrons in superalso be considered a stage, possibly in- conductors, or of helium atoms in sutermediate between crystallinity and in- perfiuid liquid helium. These phenomformation strings, in the hierarchy of ena have already enormously extended the accuracy of electromagnetic meabroken symmetries. surements, and can be expected to play To pile speculation on speculation. T a great role in future computers, among would say that the next stage could be other possibilities, so that in the long hierarchy or specialization of function, run they may lead to some of the major or both. At some point we have to stop technological achievements of this dectalking about decreasing symmetry and ade (5)J For another example, biology start calling it increasing complication- has certainly taken on a whole new asThus, with increasing complication at pect from tbe reduction of genetics to each stage, we go on up the hierarchy biochemistry and biophysics, which will of the sciences. We expect to encounter have untold consequences- So it is not fascinating and, I believe, very funda- true, as a recent article would have it mental questions at each stage in fitting (7), that we each should "cultivate our together less complicated pieces into the own valley, and not attempt to build more complicated system and under- roads over the mountain ranges , standing the basically new types of be- between the sciences," Rather, we havior which can result. should recognize that such roads, while There may well be no useful parallel often the quickest shortcut to another to be drawn between the way in which part of our own science, are not visible complexity appears in the simplest cases from the viewpoint of one science alone. The arrogance of the particle physiof many-body theory and chemistry and tbe way it appears in the truly complex cist and his intensive research may be cultural and biological ones, except per- behind us (the discoverer of the positron haps to say that, in general, the rela- said "the rest is chemistry"), hut we tionship between the system and its have yet to recover from that of some parts is intellectually a one-way street. molecular biologists, who seem deterSynthesis is expected to be all but im-
24
mined (o try to reduce everything about the human organism to "only" chemistry, from the common cold arid all mental disease to the religious instinct. Surely there are more levels of organ i z a t i o n between human e t h o l o g y and DNA than there are between DNA and quantum electrodynamics, and each level can require a whole new conceptual structure. In closing, I offer two examples from economics of what I hope to have said. Marx said that quantitative differences become qualitative ones, but a dialogue in Paris in the 1920's sums it up even more clearly: F r r z o E R A L D : The rich are different from us. HEM [NOWAY: Yes, they have more money.
HclcKlKcl 1. V . F. WeiMkopt, In BtOvkhavtit Nor. Lob. PubL SSBT160 tiMS), Al*n fee Nuovo CImrn'o SUppl. Ser I 4. 46i (1966); Phys. Today 20 (No. 5). Z3 JI947). 2. A . Babe and B. R. MoHelson. Kgl. DAI. Vldflh. Srltk. htaU Fyt. Medd. 27. 16 (19511. 3. Broken lymmcliy Bad pbUc irUliliQD^ L . D. Landau. f * » % Sot-ietunloa U, i f i . 543 (1937]. Broken fymmcLrj and collective motion. £cDcralr J. t i . : . I - . A . Salam. S. Weinberg, rhys- Rev- 117, 965 (I961J: P. W. Andenon, Concepts In Solids (Benjamin. New York. 1963). pp. 175-IB2; B. D . JtHeptuon. Inesii. Trinity Colleae. Cambridae UniVEHJiy (1962). Special case*: an Illerro m agnei Ism. P. W . Anderson. Phyt- Rev. B6, 694 <1952); tuperHKiduflMTy, , Ibtf. B27 (1939); Ibid. 112. J9Q0 fl«6>; Y. "-" i Ibid. i l l , SOB (19*01. 4. P. London, Superflulds {Wiley. New York, 1950]. vol. 1. 5. M. H . Cohen, J. Theor. Biol. 31, 101
. 7. A . B. PLppaid. RttPntHbie Physics wilh ly (Cambridge Univ. P r t u , Loudon. 1972).
Reprinted with permission from Computational Nturvsttenee, pp. 4 6 - « . ]99fJ © 1990 M I T Press
The expression "Computational neuroscience" reflects the possibility of generating theories of brain function in terms of the information-processing properties of structures that make up nervous systems. It implies that we ought to be able to exploit the conceptual and technical resources of computational research to help find explanations of how neural structures achieve their effects, what functions are executed by neural structures, and the nature of representation by states of the nervous system.
What Is Computational Neuroscience? Patricia S. Churchland, Christof Koch, and Terrence J. Sejnowski
The expression also connotes the potential for theoretical progress in cooperative projects undertaken by neurobiologists and computer scientists. This collaborative possibility is crucial, for it appears that neither a purely bottom-up strategy nor a purely top-down strategy for explaining how the brain works is likely to be successful. With only marginal caricature, one can take the purely bottom-up strategy as recommending that higher-level functions can be neither addressed nor understood until all the fine-grained properties of each neuron and each synapse are understood. But if, as it is evident, some properties are network effects or system effects, and in that sense are emergent properties, they will need to be addressed by techniques appropriate for higher levels and described by theoretical categories suitable to that level. Assuming there are system properties or network properties that are not accessible at the single-unit level, then knowing all the fine-grained detail would still not suffice to explain how the brain works. A purely top-down strategy is typified, again with minimal caricature, by its dismissal of the organization and structure of the nervous system as essentially irrelevant to determining the nature of cognition (Pylyshyn 1950; Fodor 1975). Advocates of this strategy prefer instead to find computational models that honor only (or at least primarily} psychological and computational constraints. One major reason for eyeing skeptically the purely top-down strategy is that computational space is consummately vast, and on their own. psychological and engineering constraints do not begin to narrow the search
25
biological constraints has provoked theorists to reconsider what neuroscientists have said all along: the neuronal organization matters. Connectionists have begun to develop alternative strategies to GOFAI that are yielding models that are surprisingly powerful as well as more biologically plausible. {For example. Lehky and Sejnowski {1930j describe a neural network model that computes the shape of an object from its gray level intensities array; see also papers in Rumelhart and McClelland 1986 and Zipser and Andersen 19S8J
space down to manageable proportions. Unless we go into the black box, we are unlikely to get very far in understanding the actual nature of fundamental cognitive capacities, such as learning, perceiving, orienting and moving in space-time, and planning. There is an additional reason it may be inefficient to try to determine how the brain accomplishes some task, such as binocular depth perception, by taking a purely engineering approach: nervous systems are products of evolution. The brain s solutions to such problems may be radically different from what a clever engineer might design, at least for the reason that evolutionary changes are made within the context of a design and architecture that already is in place. Evolution cannot start from scratch, even when the optimal design would require that course. As [acob ( 1 9 8 2 ) has remarked, evolution is a tinkerer. and it fashions its modifications out of available materials, limited by earlier decisions- Moveover, any given capacity, such as binocular depth perception, is part of a much larger package subserving sensorimotor control and survival in general. From an engineering point of view, a design for an isolated stereoscopic device may look like a wonderfully smart design, but in fact it may not integrate at all well with the wider system and may be incompatible with the general design of the nervous system For these and other reasons [Churchland 1966). neurobiological constraints have come to be recognized as essential to the project.
Is the Brain a Computer? Ef we seek theories to explain in computational terms the function of some part of the nervous system—a network of neurons, or an individual neuron, or perhaps a system of networks—then that structure is seen as doing computation and hence as being a computer. But is this a justifiable perspective? If by "computer" we mean "serial digital computer," then the answer is No, For the analogy between the brain and a serial digital computer is an exceedingly poor one in most salient respects, and the failures of similarity between digital computers and nervous systems are striking. On the other hand, if we embrace a broader conception of computation (see below] then the answer is a definite Yes,
Although the name "computational neurosdence' may have emerged recently, the central motivation for the enterprise is by no means new. Even so, a great deal has happened between the publication of Cybernetics by Wiener in 1948 and the Computational Neuroscience meeting in Monterey in the spring of 1987. First, there has been a spectacular growth in anatomical and physioLogical information regarding nervous systems, and in techniques for extracting information. So remarkable and munificent have been the discoveries, that it has begun to seem that the time is ripe for generating empirically adequate computational theories at a variety of levels. No less dramatically, technological achievements in designing fast, powerful, and relatively Inexpensive computing machines have made it possible to undertake simulation and modeling projects that were hitherto only pipe dreams. Finally, disappointment in conventional A l strategies (Good Old Fashioned A l , or GOFAI, as Haugeland [ 1 9 8 5 ] calls it) for modeling cognition independently of neuro-
26
Of the dissimilarities between serial digital computers and nervous systems, perhaps the most important is this: nervous systems are not general purpose machines that are programmed up to be specialized machines; rather, they evolved to perform certain kinds of tasks. Unlike manufactured computers, nervous systems have plasticity—they grow, develop, leam and change. Nervous systems have a profoundly different architecture from serial digital computers. In particular, they have parallel structure, and nervous systems appear to use a system of information storage that is very different from that employed in serial digital computers. In conventional silicon chips one node is connected, on average, to three to four other nodes, while by contrast one cortical neuron may receive input from hundreds or thousands of neurons and in turn project to thousands of other neurons in cortex. It is also important in this context to note that events in nervous systems happen in the millisecond [10~ second) range, whereas events in silicon systems happen in the nanosecond (10~* second) range. The individual comH
3
ponents making up the brain and those making up serial digital machines work on vastly different time scales. This makes it quite obvious that the parallel architecture is not a trivial but an absolutely critical aspect of computational design in nervous systems. Finally, much of the computing in nervous systems undoubtedly is not manipulation of symbols according to rules explicit in the system, in contrast to conventional A l programs run on a serial digital computer. (Rumelhart and McClelland 1986)
and associated physical variables. More exactly, a physical system computes a function fix) when there is 11) a mapping between the system's physical inputs and x (2) a mapping between the system's physical outputs and y, such that (J) /(*) = y. This kind of arrangement is easily understood in a digital computer, though it is by no means confined to a machine of that configuration. A machine is taken to compute, for example, (PLUS 2,3) to give 5 as the output, by dint of the fact that its physical regularities are set up in such as way as to honor the These dissimilarities do not imply that brains are not computers, but only that brains are not serial digital abstract regularities in the domain of numbers. But that computers. Identifying computers with serial digital com- sort of property (physical states of a device mappable onto information-bearing states) can be achieved in many puters is neither justified nor edifying, and a more insightdifferent kinds of architectural arrangements. Notice that ful strategy is to use a suitably broad notion of compualthough two pieces of wood sliding over each other may tation that takes the conventional digital computer as only a special instance, not as the defining archetype. (For initially not be regarded as .• computer, two pieces of wood with the proper representations is a slide rule, and a different view, see Pylyshyn 1984.) a slide rule is a mechanical computer whose physical It is useful, therefore, to class certain functions in the structure is set up so that certain mathematical relations brain as computational because nervous systems represent are preserved. Given this basic account of computation, it and they respond on the basis of the representations. is also useful to hypothesize what computations are perNervous systems represent the external world, the body formed by neurobiological components. For example, it they inhabit, and, in some instances, parts of the nervous appears that the neurons in the parietal cortex of primates system itself. Nervous system effects can be seen as compute head co-ordinates on the basis of retinal cocomputations because at some levels the explanations of ordinates (Andersen and Mountcastle 1983; Zipser and state transitions refer to abstract properties and the Andersen. 1987,1988), and that neurons in area VI of the relations between abstract properties Thai is, the exvisual cortex compute depth of a stimulus on the basis of planations are not simple mechanical explanations that binocular disparity (Poggio et al. 1985). cite causal interactions between physically specified items: for example, the explanation of how a molecule is transA point that is perhaps only implicit in the foregoing ported down an axon. Rather, the explanation describes characterization of computation should now be emthe states in terms of the information transformed, repphasized. A physical system is considered a computer resented, and stored. when its states can be taken as representing states of f
some other system: that is. so long as someone sees an interpretation of its states in terms of a given algorithm. Thus a central feature of this characterization is that whether something is a computer has an interest-relative component, in the sense that it depends on whether someone has an interest in the device s abstract properties and in interpreting its states as representing states of something else. Consequently, a computer is not a natural kind, in the way that, for example, an electron or an protein or a mammal is natural kind. For categories that do delimit natural kinds, experimentation is relevant to determining whether a given item really belongs to the category, and there are generalizations and laws (natural laws) about the items in the categories and theories embodying the laws. Non-natural kinds differ in all these
The distinction between physical causation and computation also arises in other areas of biology. For example, the sequence of base pairs in DNA codes a wide variety of structures and functions, including the sequences of amino acids in proteins, regulatory signals, and developmental programs. We are still a long way from the complete account of how the information contained in the DNA guides regulation and development, but the activation and deactivation of different genes at different stages of development in different cells can be considered a very complex computational system whose goal is to produce a living creature In a most general sense, we can consider a physical system as a computational system just in case there is an appropriate (revealing) mapping between some algorithm
27
about the real nature of the device and how it works. Thus finding out what is computed by a sieve is probably not very interesting, and will not teach us much we did not already know. In contrast, finding out what is computed by the cerebellum will teach us a lot aboul the nature of that tissue, and how it works.
respects, and typically have an interest-relative dimension. For example, 'bee' is a natural kind, but 'gem', 'dirt', and 'weed' are not. Stones are considered gems depending on whether some social group puts special value of them; plants are considered weeds, depending on whether a given gardener happens to like them in his garden. Some gardeners cultivate Baby's Breath as a desirable plant; other gardeners fight it as a weed. There is no experiment that will settle for us whether Baby's Breath really is a weed or not, because there is no (act of the matter, only soda] or idiosyncratic conventions. Nor are there interestindependent generalizations about all weeds—there is nothing common to plants called weeds save that some gardener has an interest in keeping them out of his garden. Similarly, there is no intrinsic property common to all computers, just the interest-relative property that someone sees a value in interpreting a system's states as representing states of some other system, and the properties of the system support such an interpretation. (For more on natural kinds, see P. M . Churchland 1985.1 Desk-top von Neumann machines have become prototypical computers because they are so common just as dandelions are prototypical weeds, but these prototypes should not be mistaken for the category itself.
Their simplicity notwithstanding, simple computational arrangements can lead to interesting insights. For example, assuming that we can landscape a terrain appropriately, a marble rolling down a hill into a valley can find (compute) the local minimum of a nonconvex two-dimensional cost function just as well as the steepest descent algorithm working on a digital computer, and probably faster. There is an analogy between this simple computer and Hopfield networks, parallel networks that store information at local minima of energy functions similar to these landscapes (Hopfield 1962). Global optimization problems can be solved by adding noise to the system, effectively "heating up" the marble so that it can jump out of a local minimum (Hinton and Sejnowski 1983). Applied to a problem in vision, for example, states of a Boltzmann machine correspond to a three-dimensional interpretation of a twodimensional image, and states with the lowest energy correspond to the best interpretation [Kienker et al. 1986; Sejnowski et al. 1986) Further, this analogy between a ball rolling down to find a local minimum of a landscape and computation of minima by analog networks has been the basis for a proposal that most problems in early vision are carried out by analog networks minimizing convex and nonconvex cost functions (Foggio and Koch 1985). The inpul is given by currents injected into the nodes of a resistive network and the final solution is recovered by measuring the voltage at each node (Koch et al. 1986). In other words. Kirchhoff's voltage and current laws are exploited directly for the computation, in contrast to standard VLSI circuits. IF we do not know the exact mapping between a problem and the physical domain, however, it may seem rather strange to classify a marble rolling down a hill or a current flowing in a wire as computing.
it may be suggested as a criticism of this very general characterization of computation that it is too general. For in this very wide sense, even a sieve or a threshing machine could be considered a computer, since they sort their inputs into types, and in principle one could specify the sorting algorithm. While this observation is correct, it is not so much a criticism but an apt appreciation of the breadth of the notion of computation. Conceivably, sieves and threshing machines could be construed as computers, if anyone has reason to care about the specific algorithm reflected in their input-output functions, though it is hard to see what those reasons might be. Still, an appropriately shaped rock can function as a sundial, and though this is a very simple computer, we do have reason to care about the temporal states that its shadow-casting states can be interpreted as representing.
A distinguishing feature of certain computational systems is their ease of programmability, and electronic computers are justly esteemed for this virtue. While it is impossible to adapt the marble-on-the-landscape computer to three or higher dimensional optimization problems, this would be a simple task for an optimization program running on an electronic computer. On the other hand.
Nonetheless, there is perhaps a correct intuition behind the criticism that is this. Finding a device sufficiently interesting to warrant the description 'computer' probably also entails that its input-output function is rather complex and unobvious. so that discovering what the function is reveals something important and perhaps unexpected
28
different initial conditions can be achieved very simply in the marble-on-the-landscape computer, which demonstrates that its flexibility lies in a different domain. Ease of programrnability addresses only one practical aspect of the prototypical computer, and although this aspect is of great importance for many purposes, it is irrelevant to the fundamental theoretical question of what it is to be a computer.
tectures, though not necessarily with comparable speed, elegance, or equivalent procedures. In principle, any algorithm can be run on a Turing machine, which is the dominant formal model of computation, not to be confused with an actual piece of electronic hardware. Apart from telling us that these algorithms can in principle be implemented on computers, the observation about Turing machines tell us little else, since the structure of the algorithm and the actual operations needed to carry out these computations in the nervous system remain to be discovered. What it does tell us is that the brain does not perform any magical operations when it computes depth or moves an arm.
Given this preamble on the nature of computation, the question now is whether it is appropriate to describe various structures in nervous systems as computing- The summary answer is that it certainly is. The reason is that we need a description of what various structures are doing that specifies the algorithm and the abstract properties thai it maps. Ions flowing along concentration and electrical gradients compute ]ust as surely as do electrons and holes in an MOS circuit. In fact, both obey rather similar equations. What we do not yet know in most cases of interest in neurobiology is the relationship between the problem domain, for instance computing binocular depth, and the appropriate biophysical variables, for instance average firing frequency, timing of individual action potentials, and the occurrence of bursts.
It is important to emphasize that different architectures may be input-output equivalent, but nevertheless be radically different in procedures and in the speed of arriving at results. The matter of time is especially worth raising in a biological context, since for organisms making a living in a world "red in tooth and claw," timing is of the essence. The organism's nervous system cannot afford the luxury of taking several days or even several minutes to decide whether and where to flee from a predator. If serial digital computers running GOFAI programs to perform tasks such as motor control and visual recognition had to compete in the actual space-time world with actual fauna—even humble fauna—they would not stand a chance. There is enormous evolutionary pressure to come up with fast solutions to problems, and this is a constraint of the first importance as we try to understand the computational principles that govern a brain Biologically speaking, a quick and dirty solution is better than a slow and perfect one.
On this point there is of course a major contrast between manufactured and biological computers. Since we construct digital computers ourselves, we build the appropriate relationship into their design. Consequently, we tend to take this mapping for granted in computers generally, both manufactured and evolved. But for structures in the nervous system, these relationships have to be empirically discovered. In the case of biological computers this may turn out to be very difficult since we typically do not know what is being computed by a structure, and intuitive folk ideas may be very misleading.
Levels
The relationship between electronic computers and the neurobiologies] computers that can be simulated on electronic computers may be misunderstood. Although nervous systems appear to be very different from serial digital computers, once we discover what their computational principles are. the computations can be simulated on serial digital computers. This is a purely formal point that can be put another way; so long as there is a systematic way fa nonmagicaJ way) that a network performs its tasks, then in principle that way can be captured by an algorithm that formally specifies the relations between input and output. Since it is an algorithm, it can run on computers with a variety of different archi-
Marr's Three Levels Discussions concerning computational theories and models designed to explain some aspect of nervous system function invariably make reference to the notion of levels, A simple framework outlining a conception of levels was articulated by Marr and Poggio {1976) and Marr {1982), and provided an important and influential conceptual starting point for thinking about levels in the context of computation by nervous structures. Marr's ideas drew upon the conception of levels in computer science, and accordingly he characterized three levels: [1) the com-
29
putahcna] level of abstract problem analysis, wherein the important to see that the purely formal point cannot speak to the issue of how best to discover the algorithm in task (e.g., determining structure from motion] is decomfact used by a given machine, nor how best to arrive at posed into its main constituents; (2) the level of the algorithm, which specified a formal procedure by which, for the neurobiologically adequate task analysis (P M . Churchland 19A2). Certainly it cannot tell us that the a given input the correct output would be given, and the discovery of the algorithms used by nervous structures task could thereby be performed; (3) the level of physical will be independent of a detailed understanding of the implementation of the computation. nervous system. Moreover, it does not tell us that any A centra] element in Marr's view was that a higher implementation is as good as any other. And it had better level was largely independent of the levels below it, and not, since as discussed above, different implementations hence computational problems of the highest level could display enormous differences in speed, size, efficiency, be analyzed independently of understanding the algorithm elegance, etc. The formal independece of algorithm from that executes the computation. For the same reason the architecture is something we can exploit to build other algorithmic problem of the second level was thought to machines once we know how the brain works, but it is be solvable independently of an understanding of the no guide to discovery if we do not yet know how the physical implementation. Thus his preferred strategy was brain works. "top-down" rather than "bottom-up " At least this was the official doctrine, though in practice, downward glances figured significantly in the attempts to find problem analyses and algorithm solutions. Ironically, given his advocacy of the top-down strategy. Man's work was itself highly influenced by neurobiological considerations, and implementational facts constrained his choice of problem and nurtured his computational and algorithmic insights. Publicly, advocacy of the top-down strategy did carry the implication, dismaying for some and comforting for others, that in solving questions of brain function the neurobiological facts could be more or less ignored, since they were, after all, just at the implementation level. Two very different issues tended to become conflated in the doctrine of independence One concerns whether, as a matter of discovery, one can figure out the algorithm and the problem analysis independently of facts about implementation. The other concerns whether, as a matter of formal theory, a given algorithm that is already known to perform a task in a given machine [e.g., the brain) can be implemented in some other machine that has a distinct architecture. Answers to these two questions may well diverge, and, as we argue, the answer to the first is probably No, while the answer to the second is Yes. So far as the latter is concerned, what computational theory tells us is merely that formal procedures can be run on different machines; in that sense and that sense alone, the algorithm is independent of the implementation. The formal point is straightforward: since an algorithm is formal, no specific physical parameters [e.g.. "vacuum tubes", "Ca**") are part of the algorithm. That said, it is
30
The issue concerning independence of levels in topdown strategies marks a substantial conceptual difference between Marr and the current generation of connectionists and computational neuroscientists. In contrast to the doctrine of independence of computation from implementation, current research suggests that considerations of implementation play a vital role in the kinds of algorithms that are devised and the kind of computational insights available to the scientist. Knowledge of brain architecture, so far from being irrelevant to the project, can be the essential basis and invaluable catalyst for devising likely and powerful algorithms—algorithms that have a reasonabe shot at explaining how in fact the neurons do the job. Consequently, if we consider the problem of discovering the computational principles of the nervous system, it is clear that analyzing the task and devising the algorithm are not independent of the neurobiological implementation. Without benefit of neurobiological constraints, we will undoubtedly find ourselves exploring some region of computational space utterly remote from the region inhabited by biological machines. Such exploration could be fun for its own sake, and may lead to technological innovations, but given the vastness of computational space, it stands a negligible chance of helping explain how nervous systems work. Levels are not independent, and there is much more coupling and intercalation than was previously appreciated. This interdependence should not. however, be considered a disadvantage, but an advantage, because it means we can avail ourselves of guidance from all levels.
Levels of Organization in the Brain
molecules, membranes, synapses, neurons, nuclei, circuits, networks, layers, maps, and systems (figure I). A t each structurally specified stratum we can raise the computational question: what does that organization of elements do? What does it contribute to the wider, computational organization of the brain? In addition, there are physiological levels; ion movement, channel configurations, EPSPs, IPSPs, action potentials, evoked potential waves, behavior, and perhaps other intervening levels that we have yet to leam about and that involve effects at higher anatomical levels such as circuits or systems.
Three levels of Analysis: How Adequate? In the previous section we provisionally acquiesced in a background presumption of Marr's framework. That presumption treated computation monoltthtcally. as a single kind of level of analysis. In the same vein, the framework treats implementation and task description/ decomposition each as -s single level of analysis. The presumption should not pass unchallenged, however, for on a closer look, there are difficulties, and these difficulties bear on how we conceive of levels. The difficulties center around the observation that when we measure Marr's three levels of analysis against levels of organization in the nervous system, the fit is poor and confusing (Churchland and Sejnowski 1988). To begin with, there is organized structure at different scales:
10 c m
Note also that the same level can be viewed computationally (in terms of functional role) or impiementationally (in terms of the substrate on which the function runs), depending on what questions you ask, and on whether you look up or down. For example, an action potential, from the point of view of communication between neurons, might be considered an implementation, since the neurons really only care about the presence of a binary event. However, from a lower level—the point of view of ionic distribution—the action potential is computational, since it is the result of integrating many sources of information, and this information is carried by the train of action potentials. In the next section we shall explore in more detail the question of levels in the nervous system, and now this bears upon the project of discovering computational models to explain nervous system function.
Systems
Maps
1 mm
100
The range of structural organization implies, therefore, that there are many levels of implementation, and that each has its companion task description. But if there is a ramifying of task descriptions to match the ramified structural organization, this diversity will probably be reflected in the ramification of algorithms that characterize how a task is accimplished. This in turn means that the notion of the algorithmic level is as oversimplified as the notion of the implementation level.
Networks
Lim
Synapses
Molecules
Philosophical Observations on Levels of Organization Figure 1 .
Which really are the levels relevant to explanation in the nervous system is an empirical, not an a priori, question. The answers to the question will slowly emerge as neuroscience and computational theories co-evolve, as theory and experiment converge on robust hypotheses. As things stand, there is a variety of ways of addressing the question of levels of organization.
Structural levels o f organiiation in The nervous system. The spatial v a l e at which anatomical organization can be Identified varies over many orders o f magnitude, from molecular dimerLiiOftS to that of the entire central nervous system. The schematic diagrams on the right illustrate (bottom} a typical chemical synapse, (middle) a proposed circuit for generating cmeriied receptive fields in visual cortex, and (top) part o f the m o t o r system thai controls the production ui speech sounds.
31
aid us in understanding this level (Sejnowski and Lehky
The range of spatial scales over which the nervous
1988; Zipser and Andersen 1985),
system has been explored covers more than eight orders of magnitude, from molecular dimensions measured in
Temporal Aspects of Levels
angstroms to fiber tracts that span centimeters. Organizational principles emerge at each spatiaJ Level that are
Nervous
directly relevant to the function of the nervous system
systems
are
dynamic
and
physiological
observations at each structural level can also be arranged
(figure 1). A few of these principles will be summarized
in hierarchy of time scales. These scales reach from tens to
here to serve as concrete examples.
hundreds of microseconds in the case of gating of single
Sensory information tends to be organized in spatial
ionic channels to days or weeks for biophysical and bio-
maps. The image of the world on the retina is mapped
chemical events underlying memory, such as long-term
onto subcortical structures such as the superior colliculus
potentiation. Action potentials and synaptic potentials of
and lateral geniculate
single neurons can be measured on a time scale of milli-
nucleus, and from there to a
proliferation of more than twenty maps in visual cortex.
seconds
The generality of this principle throughout the somato-
Recently, optical techniques have been used to record the
sensory, auditory, and visual systems was noted by Adrian
activity in populations of neurons in cerebral cortex with
(1953), who suggested that even the olfactory system
a time resolution of about 100 msec and a spatial resolution
may be organized in this way. Often these maps are
of about 100 /im. The measurement of blood flow with
arranged in sheets, as in the superior colliculus where
PET scanning has been used to observe the averge activity
maps of the visual, auditory, and somatic spaces are
throughout the brain, averaged over minutes with 5 mm
using intracellular single-electrode
recording.
stacked in adjacent layers such that stimuli from corre-
resolution. Even longer-term changes of behavior can be
sponding points in space lie above each other.
monitored and correlated with changes in the biochemical state of the nervous system. The range of time scales that
Topographic maps and laminae are special cases of a more general principle, which is the exploitation
can be studied varies over 12 orders of magnitude from
of
microseconds to days.
geometry in the design of information processing systems (Mead 1967; Yeshurun and Schwartz, this volume). Spatial proximity may be an efficient way for biological systems
The Status of Levels
to get together the information needed to solve rapidly difficult computational problems. For example, it is often
Each of the structural levels—molecules,
important to compute the differences between similar
synapses, neurons, nuclei, circuits, layers, columns, maps,
membranes,
features of a stimulus at nearby locations in space. By
and systems—is separable conceptually but not detachable
maintaining neighborhood relationships the total length
physically. What is picked out as a level is actually a
of the connections needed to bring together the signals is
boundary imposed on the structure we study using such
minimized.
techniques as are available to us, in order to try to understand the phenomena at hand. In the brain, they are all
Most of our information about the representation of
part of one, integrated, unified biological machine. That
sensory information is based on recording from single
is, the function of a neuron depends on the synapses that
neurons. This technique is revealing but it is also coruSning
bring it information, and, in turn, the neuron processes
insofar as it biases us toward thinking about the cellular
information by virtue of its interaction with other neurons
level rather than the subcellular or circuit levels. A s we
in local circuits, which themselves play a particular role by
leam more about the complexity of subcellular structures,
virtue of their place in the overall geometry of the brain.
such as voltage-dependent processing in dendrites and spines
(see
the
contributions
of Koch
This is a cautionary, if obvious, observation, generated by
and Poggio,
the appreciation that in a sense the postulation of levels is
Shepherd, Perkel. and Rail to this volme). our view of the
artificial from the point of view of the functioning brain,
computational capability of the neuron may change. W e
even though it may be scientifically indispensable if we
are especially in need of techniques that would allow us
are to make progress. A n d until we understand how the
to monitor populations of neurons (Llinas 1985). Simu-
brain works, it is always an open question whether we
lations of information processing in neural networks may
have put our boundaries in the most revealing place.
32
a model that aims to reproduce as much of the nervous system as possible from the bottom up. The problem with this approach is that a genuinely perfect model, faithful in every detail, is likely to be as incomprehensible as the nervous system itself. Although such a model may mimic the brain, it is debatable wheather any true understanding has been achieved. However, as noted above, the exclusively top-down approach has its own special problems.
Concluding Remarks A scientific held is defined primarily by its problem space and its successful large-scale theories. Until there are such theories in computational neuroscience. the field is defined mostly by the problems it seeks to solve, and the general methodology and specific techniques it hopes will yield successful theories. Accordingly, within the framework outlined here, we can say quite simply that the ultimate aim of computational neuroscience is to explain how the brain works. Since nervous systems have many levels of organization, theories appropriate to these levels will have to be generated, and the theories should integrate, level by level, into a coherent whole.
At this stage in our understanding of the brain, it may be fruitful to concentrate on models that suggest new and promising lines of experimentation, at all levels of organization. In this spirit, a model should be considered a provisional framework for organizing possible ways of thinking about the nervous system. The model may not be able to generate a full range of predictions owing to incompleteness; some assumptions may be unrealistic simplifications, and some details may even be demonstrably wrong. Nevertheless, as long as the model captures some useful kernel of truth that leads to new ways of thinking and to productive experiments, the model will have served its purpose, suggesting an improved model even as it is itself disproved, A corollary to this is that a very elaborate and sophisticated model that does not translate well into an experimental program is a sterile exercise when compared to something rougher that leads to productive research directions.
No single neural mode! can be expected to span all the levels cited, and an essential feature at one level of organization may be an insignificant detail at another. For example, the identity of a neurotransmitter is an essential feature in studying receptor binding, less important when studying circuits, and of marginal significance at the systems level. The multiplicity of levels of organization is a feature not only of neuroscience. but also of physics and chemistry, where explanations of phenomena on distinct levels of organization are more developed. Assuming a comprehensive "theory of the brain" does emerge, it will involve establishing a successive and overlapping chain of reasoning from the lowest levels to the highest, encompassing the various spatial, temporal, structural, and computational levels. Only then will we have achieved our ultimate aim.
The convergence of research in computer science, neuroscience, psychophysics, and cognitive psychology makes these truly exciting times. The hope that some of Ihe major mysteries of mind-brain function will finally be explained has become less pie-in-the-sky romanticism and more a palpable project that can be actively and experimentally pursued. Even so, the brain is awesomely complex; experiments are typically very difficult to do; there are ethical barriers to studying the normal human brain, and there may be nontrivial individual differences at many levels of organization. The answers may well elude us for some time to come.
It is rare for a model to jump over many levels, and the most successful models typically link only neighboring levels. For example, the Hodgkin-Huxley ionic model of the initiation and propagation of action potentials related macroscopic current measurements to the microscopic kinetics of ion channels, although such channels were only hypothetical at that time. This example also demonstrates that it may be necessary to make assumptions beyond the data available at one level in order to reach a better understanding at that level. Jt is only with the advent of single channel patch clamp recording techniques developed decades after the Hodgkin-Huxley model that their assumptions regarding the nature of microscopic ion channels could be verified. As knowledge accumulates at the cellular and molecular levels, it is tempting to incorporate all that is known into
33
Poggio, C . B. C Motter. S. Squalrito. and Y . Trotter f 1955) Responses
References
of neuron* in visual corte* ( V l and V 2 ) of Ihe alert macaque (o dynamic random-dol stereograms. Vision Res. 25; 397-406.
Adrian. E. D. (1953) The mechanism ol olfactory stimulation In Ihe Poggio. T . and C- Koch (1985} lli-posed problems in early vision,
mammal. Mo. Sci. {London) 9: 417-420
from computational theory lo analogue networks. Proc. R. Soc. Land. Andersen, R, A . and V. B. Mounicaslle (1953) The influence of the
B 226; 395-323,
angle oi gaze upon ihe excitability ol the light-sensitive neurons of Pylyshyn. Z W. (19B4) Computation and Cognition. Toward a
the posterior parietal cortex. ]. N/vosei. 3: 53?~548,
Foundation far Cogniliot Science. Cambridge. M I T Press. QiuichJand. P. M. 11982) 'Is thinker a natural kind?" Dialogue Hi Rumelhart, D E. and McClelland. J- L £1986) Parallel Distributed
223-238-
r
Processing: Explorations in the Micioshfclure of Cognition. Carnbridge, Qturchjand. P. M. (1985 > Conceptual progress and word/world
MIT Press.
relations; In search of ihe essence oi natural kinds. Can. }. tsfPhibs. 15: Sejnowski. T | . P. K. Kienker. and G - E Hinion (1986) Learning
1-17 Churdu'and, P. S. (19*61 tJEuraphil/naphif:
symmetry groups with hidden units: Beyond the perception. Priyaito Toward a Unified Science of
22D;
260-275.
Ihe tvitnd-brain. Cambridge. MTT PressZipser. D., and R. Andersen (1958) A backpropagalion network thai Churchland, P. 5., and T . 3, Sejnowski (1956) Neural representaions
simulates response properties of a subset of posterior parietal neurons.
and neural compulations. In; Biological Computation, ed- L Nadel.
Nature 33 lr 629—654.
Cambridge. M I T Press Fodor. J. A. (1975) The Language of Though!. New York; Crowell. (Paperback edition, 1979. Cambridge, Harvard University Press.} Haugeland, J. (1955) Artificial intelligence; The Very idea. Cambridge, M I T Press Hinton. G . E.. and T . ]. Se}nowskJ (1963) Optimal perceptual inference. In Proceedings of ihe IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Hopfield, J. J. (1962) Neural networks and physical systems with emergent collective computational abilities. Proc Noll Acad. Sci. 79;
Jacob. F. (1952) Thr Possible and the Actual Seattle, University of Washington Press Kienker, P. K_ T. J, Sejnowski. G . E . Hinton. and L E . Schumacher (19*6) Separating figure from ground with a parallel network Perception 25- 197-215, Koch. C . |. Mairoqum. and A Yullle (1956) Analog neuronal networks for early vision. Proc. Acad. Nail Sri. 63; 4263-4267. Lehky, S . and T . ]. Sejnowski (1988) Network model of shape-fromshading: Neural function arises from both receptive and projective fields Nature 333:
452-454,
Uinas R. R. (1965) Etectrolomc transmission m the mammalian central nervous system- in Cap Junctions, ed. Michael E. L Bennett and David C . Spray. Cold Spring Harbor Labaralory. Marr. D. (1962) Vision. San Francisco, freeman. Man-. D . and T. Poggio (1976) Cooperative computation of stereo disparity. Science 194; 263-257. Mead, C (1987) Analog VLSI and Nturo! Systems. Reading, Mass.. Addi son-Wesley.
34
lb
Physics, Biology, Computation
These four authors, two physicists and two biologists, contribute to shed light on the main themes of this volume from a diversity of angles. The definition of levels of study and the description of different "attitudes" towards science are elaborated in a clear and elegant fashion. The notion of computation, as a link between physics and biology, is further discussed. So this section can be viewed as an unfolding of many of the ideas introduced by Anderson (paper (3)), and by Churchland et al (paper (4)). It seemed to us essential to show how these ideas have appeared historically, as a result of deep and brilliant reflections from the best scientific minds (sometimes misled in ways still instructive for us), and as a progressive adjustment and refinement of past guesses. T h e first paper, by Leon Brillouin, presents an educated synthesis, dating from the middle of the 20th century. It includes views on the past revolutions in physics (thermodynamics, statistical physics, quantum mechanics) and it attempts to make predictions of future scientific progress on the life-matter issues, at the time of the edification of information theory, and before the triumphs of molecular biology. It is quite interesting and fruitful to reexamine these forecasts from our vantage point, half a century later, in order to assess our own chances at guessing right about the mind-body problems. T h e second paper, by John Hopfield, provides a "reasonable", "modern" account of these issues. It is a lucid attempt at pointing out what is a mystery and what is trivial, and at focusing on promising lines of research. It contains also a pedagogical introduction to neural networks, used for contentaddressable memories or for optimisation tasks, and thus leads naturally to Chapter 3. T h e third paper, by Frangois Jacob, is a brilliant survey of biological evolution, with a now famous distinction between the engineer and the tinkerer approaches to design. It offers views that are coherent and complementary to those presented earlier in the papers by Anderson (paper (3)), Churchland
35
et al {paper (4)), and Brillouin (paper (5)). Increasing complexity, greater selection among the possible, the increased role of history, led to a necessity for an analysis of complex objects at all levels. T h e two short texts by Valentino Braitenberg are separated by a lapse of fifteen years. They reach almost poetic value, through accuracy and condensation of expression. They are also sharp and even occasionally polemical in their definition of adequate strategies to understand brain functions. The plea, in favour of "interpretative neuroanatomy", will be illustrated in subsequent reprints of the author and in other articles of Chapters 2, 4 and 5.
36
itepruiieu in>iii nm. ML, Vol. 37. pp. 554-568. 1949 © 1949 American Sciential
LIFE, THERMODYNAMICS, A N D CYBERNETICS By L . B R I L L O U I N Cruft Laboratory, Harvard University"
H
OW is it possible to understand life, when the whole world is ruled by such a law as the second principle of thermodynamics, which points toward death and annihilation? This question has been asked by many scientists, and. in particular, by the Swiss physicist, C . E . Guye, in a very interesting book. The problem was discussed at the College de France in 1938. when physicists, chemists, and biologists met together and had difficulty in adjusting their different points of view. We could not reach complete agreement, and at the close of the discussions there were three well denned groups of opinion: (A) Our present knowledge of physics and chemistry is practically complete, and these physical and chemical laws will soon enable us to explain life, without the intervention of any special "life principle." (B) We know a great deal about physics and chemistry, but it is presumptuous to pretend that we know all about them. We hope that, among the things yet to be discovered, some new laws and principles will be found that will give us an interpretation of life. We admit that life obeys all the laws of physics and chemistry at present known to us, but we definitely feel that something more is needed before we can understand life. Whether it be called a "life principle" or otherwise is immaterial. ( C ) Life cannot be understood without reference to a "life principle." The behavior of living organisms is completely different from that of inert matter. Our principles of thermodynamics, and especially the second one, apply only to dead and inert objects; life is an exception to the second principle, and the new principle of life will have to explain conditions contrary to the second law of thermodynamics. Another discussion of the same problems, held at Harvard in 19+6, led to similar conclusions and revealed the same differences of opinion. In summarizing these three points of view, I have of course introduced some oversimplifications. Recalling the discussions, I am certain that opinions A and B were very clearly expressed. As for opinion C, possibly no one dared to state it as clearly as I have here, but it was surely in the minds of a few scientists, and some of the points introduced in the discussion lead logically to this opinion. For instance, consider a living organism; it has special properties which enable it to resist destruction, to heal its wounds, and to cure occasional sickness. This is very strange behavior, and nothing similar can be observed about inert matter. Is such behavior an exception to the second principle? It appears so, at least superficially, and we must be prepared to 1
* Noto Director oi Electronic Education, International New York, N. Y. 1
Levolution
Business Machines
physico-chimiqne {Paris, E. Chiron, 1922).
37
Corporation,
555 accept a "life principle" that would allow for some exceptions to the second principle. When life ceases and death occurs, the "life principle" stops working, and the second principle regains its full power, implying demolition of the living structure. T h e r e is no more healing, no more resistance to sickness; the destruction of the former organism goes on unchecked and is completed in a very short time. T h u s the conclusion, or question: W h a t about life and the second principle? I s there not, in living organisms, some power that prevents the action of the second principle? The Attitude
of the
Scientist
T h e three groups as defined in the preceding section may be seen to correspond to general attitudes of scientists towards research: (A) strictly conservative, biased against any change, and interested only in new development and application of well established methods or principles; (B) progressive, open-minded, ready to accept new ideas and discoveries; ( C ) revolutionary, or rather, metaphysical, with a tendency to wishful thinking, or indulging in theories lacking solid experimental basis. I n the discussion just reviewed, most non-specialists rallied into group B. T h i s is easy to understand. Physicists of the present century had to acquire a certain feeling for the unknown, and always to be very cautious against over-confidence. Prominent scientists of the previous generation, about 1900, would all be classed in group A. Common opinion about that time was that everything was known and that coming generations of scientists could only improve on the accuracy of experiments and measure one or two more decimals on the physical constants. T h e n some new laws were discovered: quanta, relativity, and radioactivity. T o cite more specific examples, the Swiss physicist Ritz was bold enough to write, at the end of the nineteenth century, that the laws of mechanics could not explain optical spectra. T h i r t y years passed before the first quantum mechanical explanation was achieved. T h e n , about 1922, after the first brilliant successes of quantum mechanics, things came to a standstill while experimental material was accumulating. Some scientists (Class A) still believed that it was just a question of solving certain very complicated mathematical problems, and that the explanation would be obtained from principles already known. O n the contrary, however, we had to discover wave mechanics, spinning electrons, and the whole structure of the present physical theories. Now, to speak frankly, we seem to have reached another dead-end. Present methods of quantum mechanics appear not to be able to explain the properties of fundamental particles, and attempts at such explanations look decidedly artificial. M a n y scientists again believe that a new idea is needed, a new type of mathematical correlation, before we can go one step further. All this serves to prove that every physicist must be prepared for many new discoveries in his own domain. Class A corresponds to cautiousness. Before abandoning the safe ground of well established ideas, says the
38
556 cautious scientist, it must be proved that these ideas do not check with experiments. Such was the case with the Michelson-Morley experiment. Nevertheless, the same group of people were extremely reluctant to adopt relativity. Attitude B seems to be more constructive, and corresponds to the trend of scientific research through past centuries; attitude C, despite its exaggeration, is far from being untenable. W e have watched many cases of new discoveries leading to limitations of certain previous "laws." After all, a scientific law is not a "decree" from some supernatural power; it simply represents a systematization of a large number of experimental results. A s a consequence, the scientific law has only a limited validity. I t extends over the whole domain of experimentation, and maybe slightly beyond. B u t we must be prepared for some strange modifications when our knowledge is expanded much farther than this. M a n y historical examples could be introduced to support this opinion. Classical mechanics, for instance, was one of the best-established theories, yet it had to be modified to account for the behavior of very fast particles (relativity), atomic structure, or cosmogony. F a r from being foolish, attitude C is essentially an exaggeration of B; and any scholar taking attitude B must be prepared to accept some aspects of group C, if he feels it necessary and if these views rest upon a sound foundation. T o return to the specific problem of life and thermodynamics, we find it discussed along a very personal and original line in a small book published by the famous physicist E . Schrodinger. His discussion is very interesting and there are many points worth quoting. Some of them will be examined later on. I n our previous classification, Schrodinger without hesitation joins group B: 2
We cannot expect [he states] that the ''laws of physics" derived from it [from the second principle and its statistical interpretation] suffice straightaway to explain the behavior of living matter, . . We must be prepared to find a new type of physical law prevailing in it. Or are we to term it a non-physical, not to say a super-physical law?' T h e reasons for such an attitude are very convincingly explained by Schrodinger, and no attempt will be made to summarize them here. Those who undertake to read the book will find plenty of material for reflection and discussion. L e t us simply state at this point that there is a problem about "life and the second principle." T h e answer is not obvious, and we shall now attempt to discuss that problem systematically. The
Second
Principle
of Thermodynamics, Shortcomings
Its Successes
and
Its
Nobody can doubt the validity of the second principle, no more than he can the validity of the fundamental laws of mechanics. However, the question is to specify its domain of applicability and the !
E . Schrodinger. What is Lift? (London, Cambridge University Press, and New York, The Macmillan Company, 1945). " Ibid., p. 80.
39
557 chapters of science or the type of problems for which it works safely. We shall put special emphasis on all cases where the second principle remains silent and gives no answer. I t is a typical feature of this principle that it has to be stated as an inequality. Some quantity, called "entropy," cannot decrease (under certain conditions to be specified later); but we can never state whether "entropy" simply stays constant, or increases, or how fast it will increase. Hence, the answer obtained from the second principle is very often evasive, and keeps a sibyllic character. W e do not know of any experiment telling against the second principle, but we can easily find many cases where it is useless and remains dumb. L e t us therefore try to specify these limitations and shortcomings, since it Is on this boundary that life plays. Both principles of tkervwdynamics apply only to an isolated system, which is contained in an enclosure through which no heat can be transferred, no work can be done, and no matter nor radiation can be exchanged. T h e first principle states that the total energy of the system remains constant. T h e second principle refers to another quantity called "entropy," S, that may only increase, or at least remain constant, but can never decrease. Another way to explain the situation is to say that the total amount of energy is conserved, but not its "quality." E n e r g y may be found in a high-grade quality, which can be transformed into mechanical or electrical work (think of the energy of compressed air in a tank, or of a charged electric battery); but there are also low-grade energies, like heat. T h e second principle is often referred to as a principle of energy degradation. T h e increase in entropy means a decrease in quality for the total energy stored in the isolated system. Consider a certain chemical system (a battery, for instance) and measure its entropy, then seal it and leave it for some time. W h e n you break the seal, you may again measure the entropy, and you will find it increased. I f your battery were charged to capacity before sealing, it will have lost some of its charge and will not be able to do the same amount of work after having been stored away for some time. T h e change may be small, or there may be no change; but certainly the battery cannot increase its charge during storage, unless some additional chemical reaction takes place inside and makes up for the energy and entropy balance. O n the other hand, life feeds upon high-grade energy or "negative entropy." A decrease in highgrade energy is tantamount to a loss of food for living organisms. Or we can also say that living organisms automatically destroy first1
5
4
The fundamental definition must always start with an isolated system, whose energy, total mass, and volume remain constant. Then, step by step, other problems may be discussed. A body at "constant temperature" is nothing but a body enclosed in a large thermostat, that is, in a big, closed, and isolated tank, whose energy content is so large that any heat developed in the body under experience cannot possibly change the average temperature of the tank. A similar experimental device, with a closed tank containing a large amount of an ideal gas, leads to the idea of a body maintained at constant pressure and constant temperature. These are secondary concepts derived from the original one. Schrodinger. p. 12, 5
40
5S8 quality energy, and thus contribute to the different mechanisms of the second principle. I f there are some living cells in the enclosure, they will be able for some time to feed upon the reserves available, but sooner or later this will come to an end and death then becomes inevitable. T h e second principle means death by confinement, and it will be necessary to discuss these terms. Life is constantly menaced by this sentence to death. T h e only way to avoid it is to prevent confinement. Confinement implies the existence of perfect walls, which are necessary in order to build an ideal enclosure. B u t there are some very important questions about the problem of the existence of perfect walls. D o we really know any way to build a wall that could not let any radiation in or out? T h i s is theoretically almost impossible; practically, however, it can be done and is easily accomplished in physical or chemical laboratories. There is, it is true, a limitation to the possible application of the second principle, when it comes to highly penetrating radiation, such as ultra-hard rays or cosmic rays; but this does not seem to have any direct connection with the problem of life and need not be discussed here. Time and the second principle. T h e second principle is a death sentence, but it contains no time lijnit, and this is one of the very strange points about it. T h e principle states that in a closed system, S will increase and high-grade energy must decrease; but it does not say how fast. W e have even had to Include the possibility that nothing at all might happen, and that S would simply remain constant. T h e second principle is an arrow pointing to a direction along a oneway road, with no upper or lower speed limit. A chemical reaction may flash in a split second or lag on for thousands of centuries. Although time is not a factor in the second principle, there is, however, a very definite connection between that principle and the definition of time. One of the most important features about time is its irreversibility. T i m e flows on, never comes back. W h e n the physicist is confronted with this fact he is greatly disturbed. A l l the laws of physics, in their elementary form, are reversible; that is, they contain the time but not its sign, and positive or negative times have the same function. A l l these elementary physical laws might just as well work backward. It is only when phenomena related to the second principle (friction, diffusion, energy transferred) are considered that the irreversibility of time comes in. T h e second principle, as we have noted above, postulates that time flows always in the same direction and cannot turn back. T u r n the time back, and your isolated system, where entropy ( S ) previously was increasing, would now show a decrease of entropy. T h i s is impossible: evolution follows a one-way street, where travel in the reverse direction is strictly forbidden. T h i s fundamental observation was made by the founders of thermodynamics, as, for instance, by Lord K e l v i n In the following paragraphs:
41
559 // Nature Could Run Backward* If, then, the motion of every particle of matter in the universe were precisely reversed at any instant, the course of nature would be simply reversed forever after. The bursting bubble of foam at the foot of a waterfall would reunite and descend into the water; the thermal motions would reconcentrate their enerey and throw the mass up the fall in drops re-forming into a close column of ascending water. Heat which had been generated by the friction of solids and dissipated by conduction, and radiation with absorption, would come again to the place of contact and throw the moving body back against the force to which it had previously yielded. Boulders would recover from the mud the materials required to rebuild them into rheir previous jagged forms, and would become reunited to the mountain peak from which they had formerly broken away. And if, also, the materialistic hypothesis of life were true, living creatures would grow backward, with conscious knowledge of the fuiure but with no memory of the past, and would become again, unborn. But the real phenomena of life infinitely transcend human science, and speculation regarding consequences of their imagined reversal is utterly unprofitable. Far otherwise, however, is it in respect to the reversal of the motions of matter uninfluenced by life, a very elementary consideration of which leads to the full explanation of the theory of dissipation of energy.
This brilliant statement indicates definitely that L o r d K e l v i n would also be classed in our group B, on the basis of his belief that there is in life something that transcends our present knowledge. Various illustrations have been given of the vivid description presented by L o r d Kelvin. Movie-goers have had many opportunities to watch a waterfall climbing up the hill or a diver jumping back on the springboard; but, as a rule, cameramen have been afraid of showing life going backward, and such reels would certainly not be authorized by censors! I n any event, it is a very strange coincidence that life and the second principle should represent the two most important examples of the impossibility of time's running backward. This reveals the intimate relation between both problems, a question that will be discussed i n a later section. Statistical interpretation of the second principle. T h e natural tendency of entropy to increase is interpreted now as corresponding to the evolution from improbable toward most probable structures. T h e brilliant theory developed by L . Boltzmann, F . W . Gibbs, and J . C . M a x w e l l explains entropy as a physical substitute for "probability" and throws a great deal of light upon all thermodynamical processes. This side of the question has been very clearly discussed and explained in Schrodinger's book and will not be repeated here. Let us, however, stress a point of special interest. W i t h the statistical theory, entropy acquires a precise mathematical definition as the logarithm of probability. I t can be computed theoretically, when a physical model is given, and the theoretical value compared with experiment. When this has been found to work correctly, the same physical model can be used to investigate problems outside the reach of classical thermodynamics and especially problems involving time. T h e questions raised ;
6
William Thompson (Lord Kelvin) in the Proceedings of the Royal Society oj Edinburgh 8: 325-331. 1874. Quoted in The Autobiography of Science, F . R . Moulton and J . J . Shifferes, Editors (New York, 1945), p. 468.
42
S60 in the preceding section can now be answered; the rate of diffusion for gas mixtures, the thermal conductivity of gases, the velocity of chemical reactions can be computed. I n this respect, great progress has been made, and in a number of cases it can be determined how fast entropy will actually increase. I t is expected that convenient models will eventually be found for all of the most important problems; but this is not yet the case, and we must distinguish between those physical or chemical experiments for which a detailed application of statistical thermodynamics has been worked out, and other problems for which a model has not yet been found and for which we therefore have to rely on classical thermodynamics without the help of statistics. I n the first group, a detailed model enables one to answer the most incautious questions; in the second group, questions involving time cannot be discussed. Distinction between two classes of experiments where entropy remains constant. T h e entropy of a closed system, as noted, must increase, or at least remain constant. W h e n entropy increases, the system is undergoing an irreversible transformation; when the system undergoes a reversible transformation, its total entropy remains constant. Such is the case for reversible cycles discussed in textbooks on thermodynamics, for reversible chemical reactions, etc. However, there is another case where no entropy change is observed, a case that is usually ignored, about which we do not find a word of discussion in textbooks, simply because scientists are at a loss to explain it properly. T h i s is the case of systems in unstable equilibrium. A few examples may serve to clarify the problem much better than any definition. I n a private kitchen, there is a leak in the gas range. A mixture of air and gas develops (unstable equilibrium), but nothing happens until a naughty little boy comes in, strikes a match, and blows up the roof. Instead of gas, you may substitute coal, oil, or any sort of fuel; all our fuel reserves are i n a state of unstable equilibrium. A stone hangs along the slope of a mountain and stays there for years, until rains and brooklets carry the soil away, and the rock finally rolls downhill. Substitute waterfalls, water reservoirs, and you have all our reserves of "white fuel." U r a n i u m remained stable and quiet for thousands of centuries; then came some scientists, who built a pile and a bomb and, like the naughty boy in the kitchen, blew up a whole city. Such things would not be permitted if the second principle were an active principle and not a passive one. Such events could not take place in a world where this principle was strictly enforced. All this makes one thing clear. A l l our so-called power reserves are due to systems in unstable equilibrium. T h e y are really reserves of negative entropy—structures wherein, by some sort of miracle, the normal and legitimate increase of entropy does not take place, until man, acting like a catalytic agent, comes and starts the reaction. Very little is known about these systems of unstable equilibrium. N o explanation is given. T h e scientist simply mumbles a few embar-
43
561 rassed words about "obstacles" hindering the reaction, or "potential energy walls" separating systems that should react but do not. T h e r e is a hint in these vague attempts at explanation, and, when properly developed, they should constitute a practical theory. Some very interesting attempts at an interpretation of catalysis, on the basis of quantum mechanics, have aroused great interest in scientific circles. But the core of the problem remains. How is it possible for such tremendous negative entropy reserves to stay untouched? W h a t is the mechanism of negative catalysis, which maintains and preserves these stores of energy? T h a t such problems have a stupendous importance for mankind, it is hardly necessary to emphasize. I n a world where oil simply waits for prospectors to come, we already watch a wild struggle for fuel. How would it be if oil burned away by itself, unattended, and did not wait passively for the drillers? Life and Its Relations
with ike Second
Principle
We have raised some definite questions about the significance of the second principle, and in the last section have noted certain aspects of particular importance. Let us now discuss these, point by point, in connection with the problem of life maintenance and the mechanism of life. Closed systems. M a n y textbooks, even the best of them, are none too cautious when they describe the increase of entropy. It is customary to find statements like this one: "The entropy of the universe is constantly increasing." This, in my opinion, is very much beyond the limits of human knowledge. Is the universe bounded or infinite? W h a t are the properties of the boundary? Do we know whether It is tight, or may It be leaking? Do entropy and energy leak out or in? Needless to say, none of these questions can be answered. W e know that the universe is expanding although we understand very little of how and why. Expansion means a moving boundary (If a n y ) , and a moving boundary is a leaking boundary; neither energy nor entropy can remain constant within. Hence, it is better not to speak about the "entropy of the universe." I n the last section we emphasized the limitations of physical laws, and the fact that they can be safely applied only within certain limits and for certain orders of magnitude. T h e whole universe is too big for thermodynamics and certainly exceeds considerably the reasonable order of magnitude for which its principles may apply. This is also proved by the fact that the Theory of Relativity and all the cosmological theories that followed always involve a broad revision and drastic modification of the laws of thermodynamics, before an attempt can be made to apply them to the universe as a whole. T h e only thing that we can reasonably discuss Is the entropy of a conceivable closed structure. Instead of the very mysterious universe, let us speak of our home, the earth. Here we stand on familiar ground. T h e earth is not a closed system. I t is constantly receiving energy and negative entropy from outside—radiant heat from the sun, gravitational energy from
44
562 sun and moon (provoking sea tides), cosmic radiation from unknown origin, and so on. T h e r e is also a certain amount of outward leak, since the earth itself radiates energy and entropy. How does the balance stand? Is it positive or negative? I t is very doubtful whether any scientist can answer this question, much less such a question relative to the universe as a whole. T h e earth is not a closed system, and life feeds upon energy and negative entropy leaking into the earth system. Sun heat and rain make crops (remember April showers and M a y flowers), crops provide food, and the cycle reads: first, creation of unstable equilibriums (fuels, food, waterfalls, etc.); then, use of these reserves by all living creatu res. Life acts as a catalytic agent to help destroy unstable equilibrium, but it is a very peculiar kind of catalytic agent, since it profits by the operation. W h e n black platinum provokes a chemical reaction, it does not seem to care, and does not profit by it. Living creatures care about food, and by using it they maintain their own unstable equilibrium. T h i s is a point that will be considered later. T h e conclusion of the present section is this: that the sentence to "death by confinement" is avoided by living in a world that is not a confined and closed system. The role of time. W e have already emphasized the silence of the second principle. T h e direction of any"reaction is given, but the velocity of the reaction remains unknown. I t may be zero (unstable equilibrium), it may remain small, or it may become very great. Catalytic agents usually increase the velocity of chemical reactions; however, some cases of "anticatalysis" or "negative catalysis" have been discovered, and these involve a slowing down of some important reactions (e.g., oxidation). Life and living organisms represent a most important type of catalysis. It is suggested that a systematic study of positive and negative catalysts might prove very useful, and would in fact be absolutely necessary before any real understanding of life could be attained. T h e statistical interpretation of entropy and quantum mechanics are undoubtedly the tools with which a theory of catalysis should be built. Some pioneer work has already been done and has proved extremely valuable, but most of it is restricted, for the moment, to the most elementary types of chemical reactions. T h e work on theoretical chemistry should be pushed ahead with great energy. Such an investigation will, sooner or later, lead us to a better understanding of the mechanisms of "unstable equilibrium." New negative catalysts m a y even make it possible to stabilize some systems that otherwise would undergo spontaneous disintegration, and to preserve new types of energies and negative entropies, just as we now know how to preserve food. W e have already emphasized the role of living organisms as catalytic agents, a feature that has long been recognized. E v e r y biochemist now
45
563 thinks of ferments and yeasts as peculiar living catalysts, which help release some obstacle and start a reaction, in a system in unstable equilibrium. Just as catalysts are working within the limits of the second principle, so living organisms are too. I t should be noted, however, that catalytic action in itself is something which is not under the jurisdiction of the second principle. Catalysis involves the velocity of chemical reactions, a feature upon which the second principle remains silent. Hence, in this first respect, life is found to operate along the border of the second principle. However, there is a second point about life that seems to be much more important. Disregard the very difficult problem of birth and reproduction. Consider an adult specimen, be it a plant or an animal or man. This adult individual is a most extraordinary example of a chemical system in unstable equilibrium. T h e system is unstable, undoubtedly, since it represents a very elaborate organization, a most improbable structure (hence a system with very low entropy, according to the statistical interpretation of entropy). T h i s instability is further shown when death occurs. T h e n , suddenly, the whole structure is left to itself, deprived of the mysterious power that held it together; within a very short time the organism falls to pieces, rots, and goes (we have the wording of the scriptures) back to the dust whence it came. Accordingly, a living organism is a chemical system in unstable equilibrium maintained by some strange "power of life," which manifests itself as a sort of negative catalyst. So long as life goes on, the organism maintains its unstable structure and escapes disintegration. I t slows down to a considerable extent (exactly, for a lifetime) the normal and usual procedure of decomposition. Hence, a new aspect of life. Biochemists usually look at living beings as possible catalysts. B u t this same living creature is himself an unstable system, held together by some sort of internal anticatalystl After all, a poison is nothing but an active catalyst, and a good drug represents an anticatalyst for the final inevitable reaction: death. N . Wiener, in his Cybernetics, takes a similar view when he compares enzymes or living animals to Maxwell demons, and writes: " I t may well be that enzymes are metastable Maxwell demons, decreasing entropy. . . . W e may well regard living organisms, such as M a n himself, in this light. Certainly the enzyme and the living organism are alike metastable: the stable s&te of an enzyme is to be deconditioned, and the stable state of a living organism is to be dead. A l l catalysts are ultimately poisoned: they change rates of reaction, but not true equilibrium. Nevertheless, catalysts and M a n alike have sufficiently definite states of metastability to deserve the recognition of these states as relatively permanent conditions." 7
Living
Organisms
and Dead
Structures
In a discussion at H a r v a r d (1946), P . W . Bridgman stated a funda' Norbert Wiener. Cybernetics, or Control and Communication in the Animal and the Machine (New York, John Wiley and Sons, 1948), p. 72.
46
564 mental difficulty regarding the possibility of applying the laws of thermodynamics to any system containing living organisms. H o w can we compute or even evaluate the entropy of a living being ? I n order to compute the entropy of a system, it is necessary to be able to create or to destroy it in a reversible way. W e can think of no reversible process by which a living organism can be created or killed: both birth and death are irreversible processes. There is absolutely no way to define the change of entropy that takes place in an organism at the moment of its death. W e might think of some procedure by which to measure the entropy of a dead organism, albeit it may be very much beyond our present experimental skill, but this does not tell us anything about the entropy the organism had just before it died. T h i s difficulty is fundamental; it does not make sense to speak of a quantity for which there is no operational scheme that could be used for its measurement. T h e entropy content of a living organism is a completely meaningless notion. I n the discussion of all experiments involving living organisms, biologists always avoid the difficulty by assuming that the entropy of the living objects remains practically constant during the operation. T h i s assumption is supported by experimental results, but it is a bold hypothesis and impossible to verify. T o a certain extent, a living cell can be compared to a flame: here is matter going in and out, and being burned. T h e entropy of a flame cannot be defined, since it is not a system in equilibrium. I n the case of a living cell, we may know the entropy of its food and measure the entropy of its wastes. I f the cell is appaxently maintained in good health and not showing any visible change, it may be assumed that its entropy remains practically constant. A l l experimental measures show that the entropy of the refuse is larger than that of the food. T h e transformation operated by the living system corresponds to an increase of entropy, and this is presented as a verification of the second principle of thermodynamics. B u t we may have some day to reckon with the underlying assumption of constant entropy for the living organism. T h e r e are many strange features In the behavior of living organisms, as compared with dead structures. T h e evolution of species, as well as the evolution of individuals, is an irreversible process. T h e fact that evolution has been progressing from the simplest to the most complex structures is very difficult to understand, and appears almost as a contradiction to the law of degradation represented by the second principle. T h e answer is, of course, that degradation applies only to the whole of an isolated system, and not to one isolated constituent of the system. Nevertheless, it is hard to reconcile these two opposite directions of evolution. M a n y other facts remain very mysterious: reproduction, maintenance of the living individual and of the species, free will, etc. 8
A most instructive comparison is presented by Schrodinger when he points to similarities and differences between a living organism, such as a cell, and one of the most elaborate structures of inanimate matter, 8
Schrodinger, pp. 3 and 78.
47
565 a crystal. Both examples represent highly organized structures concontaining a very large number of atoms. B u t the crystal contains only a few types of atoms, whereas the cell may contain a much greater variety of chemical constituents. T h e crystal is always more stable at very low temperatures, and especially at absolute zero. T h e cellular organization is stable only within a given range of temperatures. F r o m the point of view of thermodynamics this involves a very different type of organization. When distorted by some stress, the crystal may to a certain extent repair its own structure and move its atoms to new positions of equilibrium, but this property of self-repair is extremely limited. A similar property, but exalted to stupendous proportions, characterizes living organisms. T h e living organism heals Its own wounds, cures its sicknesses, and may rebuild large portions of its structure when they have been destroyed by some accident. This is the most striking and unexpected behavior. T h i n k of your own car, the day you had a flat tire, and imagine having simply to wait and smoke your cigar while the hole patched itself and the tire pumped itself to the proper pressure, and you could go on. T h i s sounds incredible. It is, however, the way nature works when you "chip off" while shaving in the morning. There is no inert matter possessing a similar property of repair. T h a t is why so many scientists (class B) think that our present laws of physics and chemistry do not suffice to explain such strange phenomena, and that something more is needed, some very important law of nature that has escaped our investigations up to now, but may soon be discovered. Schrodinger, after asking whether the new law required to explain the behavior of living matter (see page 556) might not be of a superphysical nature, adds: "No, I do not think that. F o r the new principle that is involved is a genuinely physical one. I t is, in m y opinion, nothing else than the principle of quantum theory over again." T h i s is a possibility, but It is far from certain, and Schrodinger's explanations are too clever to be completely convincing. 5
10
There are other remarkable properties characterizing the ways of living creatures. F o r instance, let us recall the paradox of Maxwell's demon, that submicroscopicat being, standing by a trapdoor and opening it only for fast molecules, thereby selecting molecules with highest energy and temperature. Such an action is unthinkable, on the submicroscopical scale, as contrary to the second principle. H o w does it 11
8
The property of self-repairing has been achieved in some special devices. A selfsealing tank with a regulated pressure control is an example. Such a property, however, is not realized in most physical structures and requires a special control device, which is a product of human ingenuity, not of nature. Schrodinger, p. 81. Wiener discusses very carefully the problem of the Maxwell demon (Cybernetics, pp. 71-72). One remark should be added. In order to choose the fast molecules, the demon should be able to see them;- but he is in an enclosure in equilibrium at constant temperature, where the radiation must be that of the black body, and it is impossible to see anything in the interior of a black body. T h e demon simply does not see the panicles, unless we equip him with a torchlight, and a torchlight is obviously a source of radiation not at equilibrium. It pours negative entropy into the system. {Continued on following page) 1 0
1 1
48
566 become feasible on a large scale: M a n opens the window when the weather is hot and closes it on cold d a y s ! Of course, the answer is that the earth's atmosphere is not in equilibrium and not at constant temperature. Here again we come back to the unstable conditions created by sunshine and other similar causes, and the fact that the earth is not a closed Isolated system. T h e very strange fact remains that conditions forbidden on a small scale are permitted on a large one, that large systems can maintain unstable equilibrium for large time-intervals, and that iife is playing upon all these exceptional conditions on the fringe of the second principle. Entropy and Intelligence One of the most interesting parts in Wiener's Cybernetics Is the discussion on " T i m e series, information, and communication," in which he specifies that a certain "amount of information is the negative of the quantity usually denned as entropy in similar situations. T h i s is a very remarkable point of view, and it opens the way for some important generalizations of the notion of entropy. Wiener introduces a precise mathematical definition of this new negative entropy for a certain number of problems of communication, and discusses the question of time prediction: when we possess a certain number of data about the behavior of a system in the past, how much can we predict of the behavior of that system in the future? I n addition to these brilliant considerations, Wiener definitely indicates the need for an extension of the notion of entropy. "Information represents negative entropy"; but if we adopt this point of view, how can we avoid its extension to all types of intelligence? W e certainly must be prepared to discuss the extension of entropy to scientific knowledge, technical know-how, and all forms of intelligent thinking. Some examples may illustrate this new problem. T a k e an issue of the N e w Y o r k Times, the book on Cybernetics, and an equal weight of scrap paper. D o they have the same entropy? According to the usual physical definition, the answer is "yes." But for an intelligent reader, the amount of information contained in these three bunches of paper is very different. I f "information means negative entropy," as suggested by Wiener, how are we going to measure this new contribution to entropy? Wiener suggests some practical and numerical definitions that may apply to the simplest possible problems of this kind. T h i s represents an entirely new field for investigation and a most revolutionary idea. Under these circumstances, the demon can certainly extract some fraction of this negative entropy by using his gate at convenient times. Once we equip the demon with a torchlight, we may aiso add some photoelectric cells and design an automatic system to do the work, as suggested by Wiener. The demon need not be a living organism, and intelligence is not necessary either. The preceding remarks seem to have been generally ignored, although Wiener says: the demon can only act on information received and this information represents a negative entropy. Wiener, Chap. H I , p. 76. 1 E
49
567 M a n y similar examples can be found. Compare a rocky hill, a pyramid, and a dam with its hydroelectric power station. T h e amount of "know-how" is completely different, and should also correspond to a difference in "generalized entropy," although the physical entropy of these three structures may be about the same. T a k e a modern largescale computing machinery, and compare its entropy with that of its constituents before the assembling. C a n it reasonably be assumed that they are equal: Instead of the "mechanical brain," think now of the living human brain. D o you imagine its (generalized) entropy to be the same as that for the sum of its chemical constituents: It seems that a careful investigation of these problems, along the directions initiated by Wiener, may lead to some important contributions to the study of life itself. Intelligence is a product of life, and a better understanding of the power of thinking may result in a new point of discussion concerning this highly significant problem. Let us try to answer some of the questions stated above, and compare the "value" of equal weights of paper: scrap paper, New Y o r k Times, Cybernetics. T o an illiterate person they have the same value. A n average English-reading individual will probably prefer the New Y o r k Times, and a mathematician will certainly value the book on Cybernetics much above anything else. "Value" means "generalized negative entropy," if our present point of view be accepted. T h e preceding discussion might discourage the reader and lead to the conclusion that such definitions are impossible to obtain. T h i s hasty conclusion, however, does not seem actually to be correct. A n example may explain the difficultv and show what is really needed. L e t us try to compare two beams of light, of different colors. T h e human eye, or an ultraviolet photo-cell, or an infrared receiving cell will give completely different answers. Nevertheless, the entropy of each beam of light can be exactly defined, correctly computed, and measured experimentally. T h e corresponding definitions took a long time to discover, and retained the attention of most distinguished physicists (e.g., Boltzman, P l a n c k ) . But this difficult problem was finally settled, and a careful distinction was drawn between the intrinsic properties of radiation and the behavior of the specific receiving set used for experimental measurements. E a c h receiver is defined by its "absorption spectrum," which characterizes the way it reacts to incident radiations. Similarly, it does not seem impossible to discover some criterion by which a definition of generalized entropy could be applied to "information," and to distinguish it from the special sensitivity of the observer. T h e problem is certainly harder than in the case of light. Light depends only upon one parameter (wave length), whereas a certain number of independent variables may be required for the definition of the "information value," but the distinction between an absolute intrinsic value of information and the absorption spectrum of the receiver Is indispensable. Scientific information represents certainly a sort of negative entropy for Wiener, who knows how to use it for prediction, and
50
568 may be of no value whatsoever to a non-scientist. T h e i r respective absorption spectra are completely different. Similar extensions of the notion of entropy are needed in the field of biology, with new definitions of entropy and of some sort of absorption spectrum. M a n y important investigations have been conducted by biologists during recent years, and they can be summarized as "new classifications of energies." F o r inert matter, it suffices to know energy and entropy. F o r living organisms, we have to introduce the "food value" of products. Calories contained in coal and calories in wheat and meat do not have the same function. Food value must itself be considered separately for different categories of living organisms. C e l lulose Is a food for some animals, but others cannot use it. When it comes to vitamins or hormones, new properties of chemical compounds are observed, which cannot be reduced to energy or entropy. All these data remain rather vague, but they all seem to point toward the need for a new leading idea (call it principle or law) in addition to current thermodynamics, before these new classifications can be understood and typical properties of living organisms can be logically connected together. Biology is still in the empirical stage and waits for a master idea, before it can enter the constructive stage with a few fundamental laws and a beginning of logical structure. I n addition to the old and classical concept of physical entropy, some bold new extensions and broad generalizations are needed before we can reliably apply similar notions to the fundamental problems of life and of intelligence. Such a discussion should lead to a reasonable answer to the definition of entropy of living organisms and solve the paradox of Bridgman (pages 563-564). A recent example from the physical sciences may explain the situation. D u r i n g the nineteenth century, physicists were desperately attempting to discover some mechanical models to explain the laws of electromagnetism and the properties of light. Maxwell reversed the discussion and offered an electromagnetic theory of light, which was soon followed by an electromagnetic interpretation of the mechanical properties of matter. We have been looking, up to now, for a physicochemical interpretation of life. I t may well happen that the discovery of new laws and of some new principles in biology could result In a broad redefinition of our present laws of physics and chemistry, and produce a complete change in point of view. I n any event, two problems seem to be of major importance for the moment: a better understanding of catalysis, since life certainly rests upon a certain number of mechanisms of negative catalysis; and a broad extension of the notion of entropy, as suggested by Wiener, until it can apply to living organisms and answer the fundamental question of P . W . Bridgman.
51
Reprinted with permission from The Lesson of Quantum Theory, pp. 295-314, 1986 © 1986 Elsevier Science Publishers, BV.
Physics, Biological Computation and Complementarity John J. Hopfield California lnsiilule of Technology Pasadena, California, USA
Contents 1. The domain of physics in biology 2. Logical, physical and biological computers 3. Neural compulation 4. Classical neurodynamics 5. Beyond neurodynamics: complementarity Summary Postscript References Discussion
295 296 300 301 308 311 312 312 313
1. The domain of physics in biology Biology as we know it lies wilhin a restricted domain of physics. The laws of elementary particle physics and cosmology and the history of the universe serve merely to determine the nature of a planetary environment. The dynamical equations of quantum mechanics and quantum electrodynamics (and their classical equivalents when appropriate) are the essential elemental laws of physics which lead to biology. Some physicists make claims that "we shall never understand life until we understand the origins of the elementary particles". But the real mysteries of biology he in the way in which these dynamical laws of physics, and the substrate of electrons, photons and nuclei on which they operate, produce the complex set of counter-intuitive phenomena labeled with the term biology. Biology is a problem in dynamics—an organism functions by irreversibly preying on the available free energy which it finds in its environment in order to maintain its dynamic state. Driven (or non-equilibrium) physical systems of simple components already show complex and almost unpredictable behaviors. Turbulence in fluid flow, deterministic chaos and fractal forms in snowflakes are a few of the complex phenomena which arise from the same simple physical laws and substrates that rule biology. Some of the basic unsolved problems of biology, such as the generation of 52
296 complex forms in large systems, can already be seen in these simple systems. O u r understanding of such problems in physics is far from complete. T h e physics of large systems in equilibrium is unified and simplified through our understanding of the derived or secondary laws of statistical mechanics and thermodynamics. There is no comparably general theory of strongly non-equilibrium systems. The major conceptual problems in biology have the additional complication that the biological matter is itself very complex due to a long and selective evolutionary history. Biology is not a quantum-mechanical problem. Because the masses of nuclei are large compared to the masses of electrons, the adiabatic separation of electron and nuclear motion in the Schrodinger equation is usually adequate. The electrons can then be effectively removed from the problem, leaving a problem of only nuclear motion with effective interactions of some complexity between nuclei. This remaining problem of nuclear motion is partly in the classical regime, and partly a quantum-mechanical problem. T h e hydrogen stretching vibrations are of large quantum energy compared with kT at room temperature, and are rigid in the molecular dynamics. The rotational quantum numbers are large at room temperature, and the rotational motion is thus essentially classical, as is much of the translational motion. T h e molecular dynamics of liquid water at room temperature can be described in classical terms as the motion of rigid molecules having a complex set of two- and three-body forces between them. Accurate descriptions of the viscosity, rotational relaxation and dynamic neutron diffraction of liquid water have been obtained from the computed dynamics of such a model of water. Non-equilibrium problems such as the turbulence of water are in essence problems of classical physics, as is biology. This is not to deny that there are intrinsic limitations in the knowledge with which we can know positions and momenta of nuclei. In this regard, the classical approximation ignores a quantum-mechanical limitation. But the essential mysteries, phenomena and complexity of biology are not a problem of Planck's constant. They are a problem of the large size of Avagadro's number combined with the non-equilibrium nature of the system. There do not seem to be any important larger quantum coherent aspects to living matter, contrary to the romantic hopes of some physicists of the 1930s.
2. Logical, physical and biological computers Biology can be seen as a hierarchy of computations and computational devices. The translation of D N A into protein structure is a kind of computation. T h e construction of a complex organism from the instructions in D N A is also the following of an algorithm. The recognition of a familiar object is a neural computation. These diverse computations share some common characteristics because they share an evolutionary background. T h e species which exist in biology today are those which have survived the competition with other organisms and the cataclysms of weather and geology. A t the other end of the size scale, the proteins which exist within a given species have survived a competition with different molecules which might have performed the same functions, and with the alternative of simply not having this
53
297 function performed at all. The fundamental survival and competition problem for the organism or for the enzyme molecule is to develop an algorithm for predicting the future, or more accurately, to develop a behavior such that physical processes (or actions) taken now will be likely to be appropriate to the future environment in which the organism or molecule finds itself, and promote the survival or reproduction of the organism. The organism which survives best will be that which most clearly "sees" the consequences of its possible present actions. The prediction of the future from present information is a kind of computation. T o understand complex aspects of biology beyond the descriptive level, it becomes necessary to think about the computational aspects—what is computation and how does biological computation differ from that which we think about in conventional computers. Before delving into neurobiological computation, we will illustrate some of the issues. Computation has three conceptual elements: an input, an output and a "device", which reads the input and produces the output. The particular output, or range of outputs, is the consequence both of the particular input and the nature of the computing "device". The classic conceptual device of computational theory is the Turing machine. This machine has several internal states. It can read an input-output tape, shift the tape, write output on the tape and change the internal state of the machine. Universal computation can be performed by such machines. Turing machines are intrinsically digital (or logical) machines, having a small number of reading and writing symbols and a finite number of internal states. When computation is a physical process, as in a real computer or in biology, each of these elements has a physical manifestation. T h e description of computers as logical devices has been thoroughly developed in the past 50 years. A computer is also a physical dynamical system, which follows the laws and limitations of physics. The object of a computer designer is to develop a real dynamic system which behaves as similarly as possible to some given logical design. Physical systems have noise and imperfections not envisioned in the simple logical view of computer function. A s a result, considerable effort in hardware design is invested in a problem of no logical concern, namely trying to get reliable computation in a noisy and fault-filled world. Biology compounds this problem by adding a unique view of the nature of errors and of logic itself. We will examine these points by an example from protein synthesis. A growing cell is constantly producing more proteins [see, for example, Watson (1976)]. I n this process, the information in a strand of messenger R N A is used as an instruction for making a protein. The input tape consists of a single strand of R N A . The output is a protein, a linear polymer of amino acids which then folds into a functional three-dimensional structure. The m R N A is a polymer made up of four kind of units, A , U , G and C . The protein is a polymer made up of twenty different kinds of amino acids, glycine, alanine, tyrosine, AUGGGUCCAAAGAGCCUGUGG Met G l y Pro L y s Ser L e u T i p
UGA stop
mRNA protein
Protein synthesis is carried out at a polymolecular assembly of proteins and R N A called a ribosome, which is the "Turing machine" for the process. Many different
54
298 molecules participate in the protein synthesis on the ribosome, including t R N A , G T P and a host of co-factors. The ribosome reads the m R N A input tape and inserts the appropriate amino acids into the protein in sequence. The logical operation performed might be described as the following instruction set: (1) (2) (3) (4)
Read the next three bases on the m R N A molecule "tape". Look up the corresponding amino acid in the genetic code "dictionary". A d d lhat amino acid to the protein. Shift the input tape by three bases.
T h e program contains a possible "stop" instruction in the code dictionary. T h e signal for starting is more complex. T h e logical operation of reading a particular nexl codon in the m R N A is actually carried out in a chemical reaction which knows nothing about logic. T h e reaction might schematically be written: peptide-n + correct amino acid - * peptide-n + 1, Correct refers to the amino acid in correspondence with the next m R N A triplet. There is, however, also a competing reaction peptide-*! + incorrect amino acid - * incorrect peptide-n 4- 1. Within the logical view of computation this competing reaction simply "doesn't happen". F r o m the point of view of chemistry, the competing reaction can happen, but has a different energy barrier for taking place. The reaction is rather similar to the one desired, and any enzymatic system which allows the correct reaction to take place must also allow the incorrect reaction, albeit with rather slower rates. The equilibrium constant for adding the correct amino acid and for adding the incorrect one are essentially identical. Equilibrium processes—physical processes which take place sufficiently slowly—result in useless proteins being formed because a particular incorrect amino acid is as likely to be added as a correct one. T h e logic which we would like the biological system to display—correct amino acid only—is not possible without errors, and is possible with low but finite errors only by deliberately running the system out of equilibrium. In non-equilibrium processes, the choice between products can be made on the basis of rates. The biological process must be designed to make use of kinetics to obtain accurate logical calculation. The qualitative description of biochemistry—that reactions take place because molecules fit together, and other reactions do not because the corresponding enzymatic binding does not happen—consists of logical statements, and by being only logical overlooks the essential non-equihbrium element to real physical computation. Ordinary computers also make use of dissipation to obtain their accuracy. T h e course of a computation might be described as a motion in computer state space, or in a physical phase space. The initial data define a starting point in that space, and the computer is supposed to follow an appropriate and determinate path to the solution, represented by another point in that space. This is illustrated by the solid line with arrows in fig. 1. The effect of noise and imperfections is to cause the flow to deviate from the desired direction, and an accumulation of noise or statistical imperfections will result in a wrong answer.
55
299
COMPUTER
STATE SPACE Correct Result
Initial Information
Fig. I, The dynamical Irajectory of a computing machine from initial information to the appropriate answer. In a noiseless perfect machine, Ihe desired paih would be directly followed. Noise and imperfections cause increasing divergence from this path, while restoration returns the trajectory toward the ideal one.
T h e way to avoid this problem is to introduce a physical process which squeezes the system back down onto the proper track in state space. I f we can keep this compression going on, it will result in finding the correct answer in spite of noise. Compressing a bunch of trajectories in phase space down into a smaller volume is easily done by a dissipative process. This squeezing down is essential to avoiding errors in a system which has the physical possibility of making them. T h e reason that protein synthesis must be run out of equilibrium is exactly this necessity for compacting the occupied state space in a dissipative fashion to avoid errors. ( I n electronic digital computers, the electrical power necessary to do a computation is in principle bounded by this aspect, though in fact the power dissipation is far larger for reasons which have nothing to do with fundamental physics.) I n digital computers, this idea is called restoration (von Neuman 1952, Mead and Conway 1980), and the fact that a nominal digital "one" (really a particular voltage level) will recover to the appropriate level after a transient perturbation is given to the circuit is an illustration of the presence of restoration. In thinking about the computations which are performed in neurobiology, we must expect that the system will show such restoration in an obvious fashion, or else the system would be unable to compute. Restoration—which gives a robustness against noise and computer imperfections-is a universal necessity to physical computers, while an irrelevancy in logical computers. Even with restoration, biological computers are inaccurate. T h e intrinsic accuracy of protein synthesis of the simplest sort based on elementary chemical recognition,
56
300 is about 1 error in 100 in difficult cases (Pauling 1957, Hopfield and Y a m a n e 1980). T h e protein synthesis system also has a proofreading system (Hopfield 1974), which consumes more free energy and adds another layer of restoration, resulting in error levels on the scale of 1/3000, a drastic level from ihe point of view of electronic machines, but apparently good enough for biology. T h e one additional oddity about biological computers is thai they do nol perform as well as possible. Streptomycin resistant mutants of bacteria proofread belter (Yates 1979) and are more accurate than the normal wild-lype bacteria, but are an artifact of an evolutionary history with streptomycin present in the growth environment. In the absence of streptomycin, the bacterium reverts to the wild type—which is less accurate. T h e same general kind of less-than-best accuracy has been also demonstrated in D N A synthesis in T 4 bacleriophage by Muzyczka et al. (1972). I n biology, being as accurate as possible in computations may be a loosing proposition! Biology is not interested in perfect logic. The possibility of making progress through random accidents—called creative thought when ihey occur in neurobiology, or evolution and special ion when they occur in molecular biology—seems to be an essential part of biological computation.
3. Neural compulation W e turn next to computation in neurobiology. The object is to understand how a set of neurons makes decisions, generates actions, generalizes and leams and profits from past experiences. The emphasis here must be on our understanding. T o merely know the input-outpul relation of a set of neurons by exhaustive study is not satisfying. Worse, a clump of neurons with 100 input neurons and 100 output neurons might easily require more than 1 0 bits of information to characterize the input-output relation, making exhaustive study impossible. I i would be equally unsatisfying to know the neural hardware in sufficient detail to be able to simulate the hardware on a monsterous digital computer and predict correctly the behavior of a neural system. This would correspond to being able to simulate ihe behavior of a classical gas of complex molecules, without the conceptual understandings brought to such gases by statistical mechanics and ihermodynamies. 4 0
Perhaps the largest computational burden placed on our brains is involved in visual perception. The end result of this computation is our decisions about what we have seen. (It is appropriate to emphasize decisions, for decisions are the essence of computation. Strictly linear systems do not truly compute, although they can be very useful elements in a computational system.) Given a flash exposure to a typical visual scene, we note Ihe presence of a few familiar objects, some rough characteristics of each, such as color, general size, etc. T h e immense supply of almost non-meaningful information—more than 1 0 bits of information were processed by the retinal cells which begin this calculation—is compressed into significant perceptual information, of which there are probably only a few thousand bits. I n a digital machine such a computation would be done by making a very large number of sequential decisions, but somehow the essence of biological decisions seems to be rather more holistic, collective, or Gestalt. We want to understand how such decisions are made. 9
57
301 Aspects of neural computations in higher animals which are particularly puzzling from a physics viewpoint include: (1) The system makes very effective use of its computational resources, whether measured by speed of calculation or volume or energy considerations. (2) In such systems, the computations done by the system are very resistant to damage to the neural system (fail soft). (3) Emergent properties such as self-awareness seem to be present. (4) I n higher animals, neurobiology manages to function without a determined circuit diagram. There are two reasons to think that progress might be made in understanding neural function. First, the system is large, the connectivity between neurons is large, the behavior is somewhat insensitive to the destruction or misfunction of components, and the calculation seems to have a somewhat holistic character. This suggests that collective effects might be involved, and that a search for collective effects in neural networks [Little (1974); Little and Shaw (1978); Hopfield (1982)] and neural computation might be fruitful. (Note that this is not the way that most chip designers work—they do not make use of collective effects, and would attempt to suppress them if they ever were to be noticed.) Second, the system cannot be as complex as it might appear. It is true that a general module of neurons with 100 inputs and 100 outputs could require 1 0 bits to specify its behavior, and would require such a number if all we think of is to list them. But a general module would require also 1 0 bits of information to describe how to build it. A simple module, which can be described in perhaps 10000 bits cannot produce a general input-output relation. It must produce a very special kind of input-output relation, which can be only apparently complex, not truly complex. (In this same fashion, the random-number generators which are used in computers appear to provide highly random numbers, but in fact generate very special sequences because the generators can be described by short programs.) 4 0
4 0
4. Classical neurodynamics We must understand what computational or circuit facilities a nervous system has at its disposal in order to see what is a mystery and what is trivial. The anatomy of a "typical" cortical neuron is sketched in fig. 2. The morphological and functional diversity of such cells is very large. We will briefly review the electrophysiology of such cells, which is covered in detail in textbooks (see, for example, K a n d e l and Schwarz 1981). A small electrode can be inserted into a cell body, and the potential difference between the inside and the outside of the cell can be studied as a function of time. A typical result of such a recording is a baseline potential of about — 90 millivolts, on which is superimposed a set of more or less stereotyped voltage "spikes" called action potentials, rising to a potential of about + 50 millivolts, and being of about 1 millisecond duration. A n action potential propagates from a cell body down an axon by an active regenerative process, and can thus propagate long distances without attenuation.
58
302
Dendrites <
Soma Anon and Branches Fig. 2. A "typical" neuron from brain, showing Ihe dendritic arbor (chiefly inputs), the cell body or soma, and the axon or chief output path
T h e rate of generation of action potentials depends on the input to the dendritic arborization. Inputs to the dendrites produce current flows which depolarize the cell body, and if the cell body is depolarized enough, an action potential will be generated. This process can be readily simulated by passing a positive D C current into the cell body. The rate of firing (generation of action potentials) as a function of that positive current is sketched in fig. 3. Such curves have generally a sigmoid form, going from zero for large negative currents to a saturating maximal value of 100 to 1000 per second (depending on the neuron) for large positive currents.
Input Current •
-
Fig. 3. The mean firing rate of a lypical neuron as a function of the inpul current to the cell body. A sigmoid curve of this general shape is taken as the input-output relation V = g( u) for the model neurons.
59
303 The ends of branches of axons of a eel] are connected by synapses to the dendritic arborization of other cells. When an action potential reaches a synapse, a small amount of a chemical neurotransmitter is released at the axon terminal. Some of that transmitter becomes bound to receptors on the adjacent dendrite, and a current leak across the dendritic cell membrane is briefly produced. T h e detailed time history of this process depends on the type of cell, and on the particular neurotransmitter and receptor molecules involved, but is typically on the millisecond scale in simple motor neurons. This net current introduced into the cell body by such a process contains positive terms from synapses which excite the cell, and negative terms from synapses which inhibit (or suppress) the activity of that cell. T h e electrical capacitance of a neuron is not negligible—cell membranes are only about 50 A thick. The integrative time constant of a cell is appreciable, on the time scale of milliseconds. A n individual action potential arriving at a synapse is generally not sufficient to result in the production of an action potential by the next cell. More typically, many action potentials must arrive (generally at many different synapses) on the dendritic arborization of a single cell within an integrative time constant before the post synaptic cell will fire. A minimal set of properties which might be used to describe the functioning of a neural network might: (a) Regard the individual action potentials as irrelevant quantization, and replace the true output of a cell i (a set of action potentials occurring at particular times) by a smooth, continuous property V describing the short-term average of the number of action potentials per unit time being sent out by that neuron. (Similarly, the discrete electrons are replaced by continuous charges and currents when one is dealing with electronic circuit behavior, and the quantization of electrical charge is generally neglected except when a noise analysis is done.) t
(b) Describe the effective output V of a cell by a monotonic increasing sigmoid function I^ = g;(u,) (where u, is the input voltage of cell i) with g ( - o o ) = 0 and g( + oo) = 1. (The choice of 1 is only a convenient scaling.) (c) Linearize the input dendritic arborization. The connection from neuron j to neuron / can then be described by a transconductance T The input to cell i can then be described as T.jTyVj + I,. /, represents the input to a cell coming from sensory input (e.g. light falling on a light-sensitive neuron, resulting in a transduction from external light to an internal neural signal), external electrodes placed by a neurobiologist, or other neurons not being described at the moment. i
iy
(d) Represent the effect of the transmembrane resistance and capacitance of cell i by constants p, and C , . These approximations result in the following equations of motion for the state of the system of neurons:
C
'
at
L
"
1
R,
"
These equations (Sejnowskii 1981, Hopfield 1984), describe a classical neurodynamics—classical in the sense of classical physics. They omit the effect of the
60
304 quantization of the action potentials, the "pulse coding" by which many neurons communicate. They are also classical in treating the communication speed as infinite, neglecting the time necessary to propagate an action potential. These are equations both of extreme complexity and also of drastic oversimplification. T h e complexities of these equations can be demonstrated by the properties of simple circuits of these elements. F o r example, two neurons can be connected together to make a bistable flip-flop circuit. Three or five neurons with negative couplings can be connected in a ring, forming a ring oscillator or clock. B y choosing the correct offsets in the gain functions and the correct Z L , a neuron can be set up so that its output is very nearly zero unless at least two other neurons which are driving it have outputs near one. This makes a logical circuit which performs the "and" operation. Continuing in this fashion, we can construct all the elements necessary to build the sequential logical operations and memory necessary for a universal computer. A n y digital computer which can be built can be duplicated (as far as its logical operation is concerned) by appropriate choices of the set of g, T, I, R and C . T h e complexity of the behaviors which can be shown by these equations is beyond description, for they comprise all possible computations and computers. I n real brains, the synapses themselves change with time due to the activity of the network. Learning is believed to chiefly involve the modification of the synaptic strengths T . A complete set of neurodynamical equations must also describe the rate of change of the synaptic strengths, an equation of motion for the T themselves. Considerable experimental work and modeling of this procedure has been done, but cannot be reviewed here. The dynamic equations described above might describe the response of a brain to a new situation in the light of what the brain already knows, but does not include learning new facts or relations. u
u
These equations are, at the same time, a gross oversimplification of the realities known from experimental neurophysiology and neuroanatomy. Individual action potentials are known to be important in some parts of neurocomputation, especially in early sensory processing in the auditory system and in the visual analysis of motion. Cells in invertebrates often display bursting behavior even when driven by a constant input, generating a train of equally spaced spikes followed by a period of no action potentials. T h e description of inputs from axons to dendrites is an idealization of motor neurons, while neurons in brains often show connections from dendrites of one cell to dendrites of another cell, and some connections where three processes (e.g. two dendrites and an axon) make mutual close contact. Signals in the dendritic arborization do not simply add—for there are rather more complicated interactions between inputs, and connections made close to the cell body may "veto" the inputs coming from more distant parts of that dendritic arbor by forming a shunting impedance path. Some general neurotransmitters have relatively longer-lasting or modulatory effects on neurons, and several time scales of integration are present. T h e above list of emissions and oversimplifications could be easily extended to great length, making many neurobiologists feel that such a set of equations must be inadequate. But for looking at collective behaviors, it is essential to model a neuron as simply as possible, realizing that if any interesting computational properties occur in a simple system, the real system will certainly have these properties, and
61
305 perhaps more. A n excessively complex model of the preclude the possibility of understanding the collective are led to ask whether these very simple networks will collective fashion some of the kinds of computation uniquely well.
capabilities of a cell would aspects of the system. So we do in a natural, elegant, and which biology seems to do
In conventional computer hardware, memory is located by address. Information is stored somewhere, al a particular address, and can be retrieved by an instruction to read the contents at that address. This is most obvious in the case of a magnetic recording medium, where the information is stored as magnetization at a physical location on the recording disc, and the address is a description of that location. It is equally true of the chips used in a simple semiconductor memory like a R e a d Only Memory ( R O M ) , where there are two separate sets of signal wires to the chip. One is a set of address leads, into which information is sent about the locations to be read. The others are information leads, on which the information from the desired location is sent out. If we were in possession of part of the information stored at some address and wanted the rest of that information, but did not know the address, there would be no alternative but to examine each memory i n turn. Partial information is not a key to more information in ordinary machine memory hardware. Memory in animals seems to be of an entirely different nature. E a c h of us knows many individuals, and a particular acquaintance can be described by a large number of characteristics (eye color, height, face shape, accent, vocation, spouse, children, shared experiences, education, nationality, political v i e w s . . . ) . A l l these characteristics belonging to a single individual are somehow associated or linked in a memory. The entire set of features connected to one individual can be retrieved from an incomplete and partially incorrect set of information about that individual. " T h e short jovial tennis-playing physicist from M O T " is sufficient information to identify Prof. Feshbach, in spite of the fact that M I T has been misspelled and has hundreds of physicists and many tennis players. In this retrieval we have no notion of address. A n y substantial subset of the information can be used to access the memory. Psychologists refer to such linkages as associative memory. Associative memory is a general and important feature of memory i n all animals which leam, and the classical association paradigms of Pavlovian condition, often thought of in terms of higher animals, have been demonstrated by Gelperin (1983) and Sahley et al. (1981) on animals as simple as a garden slug {Limax maximus). Although associative memories seem natural to biology, they are foreign to conventional computer hardware. They turn out to be natural for collective biological hardware, because the neurodynamics we have given describes a dynamic system, not a logical calculation, and the idea of associative memory is a natural construct in a dynamical system. Consider an "information space" about people, with many dimensions. E a c h axis could be labeled with a characteristic such as height, weight, name of spouse, or age. I n such a space, each person you know is represented by a point, whose location describes all characteristics of the individual in question. Y o u know very few of all possible people, so the set of points which represent your memories is very sparse in this space. Suppose now that you are given partial information about some
62
306
Fig. 4. A simple motion map of the change of the state of a dynamic system, characterized by a set of stable states. The stable states represent memories (in ihe case of associative memory), or more generally, possible answers in a computation.
particular friend. This information represents an approximate location in the information space. Presumably this approximate location is closer to the position of this friend in the information space than to the positions of other individuals. A n associative memory functions by using this closeness to locate or move to the complete information. I f a dynamical system is described by a flow characterized by stable points, as i n fig. 4, the initial information could start the dynamical system at a location close to one of the stored memories. If the state-space motion is such that the motion goes to the most nearby stable point, the dynamical system would be functioning as an associative memory (Hopfield 1982). Many dynamical systems display state-space motions with multiple stable states. A n y such system is a candidate for an associative memory if in addition the stable states can be placed wherever they are desired. Thus associative memory is not an esoteric property of computational systems. It is natural behavior of many simple dynamical systems. I n the case of the neuronal dynamics model, when the matrix of connections is symmetric, Hopfield (1984) showed that the flow is simple, and has no cyclic or strange attractors. Somewhat less stringent conditions will suffice. T h e symmetry of the connections is natural to the associative memory problem because the idea of simple association is itself symmetric. The essential step in this proof is the construction of a Lyapunov function, the "computational energy", which decreases monotonically during the neurodynamical motion. This energy is minimized (at least locally) by the state-space motion. Of course, the system of neurons is itself unaware that it is doing anything so grand as minimizing some global function—it merely obeys its own neurodynamical equation. This computational energy is simply
63
307 our way of describing a global or collective property which enables us to understand the nature of the behavior of a large and complex system. The idea of restoration is intrinsic to the notion of computation in a world of noise and error. The entire computation of this neural dynamics is intrinsically restorative, for the motion of the system is downhill on an "energy" terrain. Minor noise, or perturbations in the shape of the energy surface will not change the valley to which the state-space trajectory is descending, except when it starts at or traverses a logically ambiguous location such as a saddle point in the terrain. Associative memories based on these principles of collective behavior and neurodynamics have been simulated on digital computers. The T,j construction is itself amazingly simple. The T can be built up by adding one memory at a time, as biology would require. The particular connections between i and j do not require any global information about the memory to adjust themselves, but only make use of information locally available from the "experience" of the network being held in a state to be remembered by the external input to it. Hebb (1949) postulated biological learning rules for synaptic connections which are of the nature that this associative memory would require. tJ
Because the neuronal equations also correspond to elementary (but highly connected) electrical circuits of novel form, they also suggest the utility of making devices of such a structure. John Lambe first constructed such circuits, and there is a growing effort to investigate such devices in both electrical and optical formats (Lambe et a l 1985, Sivilotti et al. 1985, Jackel et al. 1986, Psaltis and Farhat 1985). The secret to the simple flow pattern to stable states is the "computational energy" function which is always being decreased by the equations of motion. T h e operation of the collective circuit decision could be anthropomorphic ally described as the attempt of the circuit to find a minimum of this function. Computational problems can often be stated as minimization or optimization problems. When a mapping can be found which creates a correspondence between an energy function and the quantity which is to be optimized, then this kind of network may be effective in solving the problem. Many perceptual and reasoning problems from biology are of this minimization nature. What is the best route home, or the best pathway to move your hand in order to reach out and grasp the sandwich? H u m a n speech is normally a continuous sound stream. Given such a continuous stream, where is the best place to interpret the breaks between words? (If you think this is not a problem, try to remember the first time you heard French spoken if your native language is not French). I n the scene at which you are now looking, what is the best way of associating features together so that they will make reasonable objects? There is an immense field of optimization problems which biology must solve, and the fact that the neural networks are spontaneously capable of making a collective optimization decision suggests the way that a nervous system may go about doing such computations. I n an effort to understand the power of such networks i n difficult computations, Hopfield and Tank (1985) took a non-biological optimization problem which is computationally very difficult (NP-complete) and examined the kinds of solutions which a neural network that we designed would generate. The object was not to see how a brain might solve this problem, but rather to study the computing power
64
308 which an assembly of 100-1000 richly interconnected neurons could possess. The Traveling Salesman Problem investigated is defined as follows. Given a particular set of cities—Copenhagen, Stockholm, Oslo, Bergen, Malmo, Aarhus in what order should one fly between the cities such that each city is visited once, one returns to the starting point, and the total distance flown is as short as possible? T h i s problem is connected to biological computation of problems like word boundaries because in each case, there is a combina tori ally larger number of possibilities which must be considered. T h e simulation showed that a simple network of 900 "neurons" could find a good solution to a problem on 30 cities in a single convergence. The settling time for a set of neurons involved in such a computation would be 0.1-0.2 seconds, T h e network did not find the best solution. But of the more than 1 0 ° possible solutions, the network found one of the best 10 , rejecting poor solutions by a factor of 1 0 . This is an immense amount of computation for such a small set of elements. A microcomputer, with an intrinsic speed about 1 0 times faster and containing 100,000 times more transistors, would achieve a comparable result in a comparable time. The fact that the neuronal network found a very good solution but not the best one is typical of biological computation. A s was evident from the earlier molecular examples, biology does not insist on perfect solutions to computational problems. 3
7
23
s
T h e immense computing power of this small network comes from features which are explicitly those of neurobiology. First, large connectivity between neurons (or in engineering, amplifiers) is required. Brains have connectivities on the scale of 1000, while typical transistors in integrated circuits get inputs from only a few other transistors, and send outputs to only a few others. Second, the system operates in an analog mode, using the smooth, graded and nonlinear response of a neuron to its input. By contrast, the conventional digital machine emphasizes logical operations, ("on" or " o f f , 1 or 0), and suppresses as much as possible the fact that the fundamental hardware is itself an analog circuit. Third, a single collective decision is smoothly made by many neurons at once, rather than a sequence of minute logical decisions. The operation is as a dynamical physical system rather than as a logical one.
5. Beyond neurodynamics: complementarity I turn finally to topics which I do not usually write about. They are subjects about which Niels Bohr would have asked, and on this centenary I am obliged to try to give answers—such as they are—to some queries which clearly come forth from the legacy of Bohr's essays. Bohr's view of complementarity extended rather more broadly than the narrow confines of the wave-particle duality or the quantum-mechanical uncertainty relations. His view came from the understanding of quantum mechanics and its relation to classical systems, but did not fundamentally rely on quantum ideas. H e asked whether there were other contexts in which the conflicting demands of observer and of normal unperturbed system behavior produced an essential paradox, and inabil-
65
309 ity to know all about a particular system in the same fashion that we cannot ask simultaneously wave and particles questions. Such limitations on complete knowledge need not even necessarily arise from observer-experiment problems, but might arise in other situations. Bohr particularly raised this question with respect to the operation of biology, where he remarked (Bohr 1958, 1963): " The basis for the complementary mode of description in biology is not connected with the problems of controlling the interaction between object and measuring tool, already taken into account in chemical kinetics, but with the practically inexhaustible complexity of the organism." This clearly expresses his view that issues of complementarity and the limits of knowledge in biology are of the nature that I described tor turbulence, and not fundamentally involved with quantum mechanics in a profound way. Bohr's (1958) view toward complementarity in biology evolved during his lifetime, as biology and the understanding of quantum physics also rapidly evolved. His later essays themselves seem somewhat complementary. The next paragraph attempts to combine the views expressed chiefly in several assays written in the last five years of his life (Bohr 1963). Bohr viewed the action of the human brain as perhaps the most likely case of the occurrence of a complementarity limitation to the precision of knowledge of biological behavior. There appeared to Bohr to be intrinsic complementary aspects to the detailed study of the anatomy of a particular brain and the thought processes in that brain. Since the thought processes themselves change the microanatomy of the brain at the molecular level—how else could we remember our thoughts—observation of the function of the brain in a whole animal inevitably changes the anatomy of that brain. But an observation of the anatomy of the brain in sufficient detail to predict the functioning of that brain is so destructive as to preclude making the observations which would test the predictions. I n this description, microanatomy and psychology become complementary views of a brain, to be united by a description which tike the Schrodinger equation and its interpretation, should show why this limitation must forever remain. What does a generalization of the classical neurodynamic equations in order to take into account thermal noise, and more realistic anatomy and physiology, suggest about such questions? If. with such generalizations, an adequate number of precise measurements were made (including studies of the properties of the modification processes) we could with sufficiently large computers predict the state of the neuronal system for a short time, and thus be able to simulate its behavior. Adequately precise information about a mammalian brain for such purposes might well require 1 0 ° bits of information about a single brain, clearly a hopeless task. It might be too complex to ever simulate. This might be a fundamental limitation to knowledge of such systems. But while Bohr considered this aspect, it is not the complementarity which he viewed as most important. 2
His essential complementarity problem is whether even a complete set of electrophysiological measurements for simulation of neurons in complete detail can be connected to particular higher mental processes which are normally described in macroscopic or cognitive terms. There must be something observably different in
66
310 the microscopic behavior of my neurons which occurs between the time at which I do not understand a mathematical theorem and the time slightly later when I do understand it. That difference can he studied by physiologists. That difference can also be pursued by psychologists asking questions about the nature of my understanding. But even if we can observe all the physiological microdetails, these microdetails may not necessarily lead to any knowledge of what "to understand the theorem" means. There may be no way to deduce the information available to the psychologist from the data obtained by the physiologist. I n such a case, there would necessarily be parallel or complementary descriptions of mental phenomena. The analog in physics might be a phase transition so subtle that even though we can see macroscopically that the phase transition has occurred, the phase itself is so complex that it cannot be described microscopically by a list of symbols or words short enough to be read during a lifetime. I n that case, a complete Hamiltonian, the equivalent to the equations of neural dynamics, would still need to he augmented by the complementary description of macroscopic physics in order to describe observable phenomena. T h e complementarity issues just discussed have to do with the issue of computational complexity and complexity measures, a topic which has barely begun to enter physics. There is, in addition, one peculiar aspect of the mathematics of collective neuronal computation which has the feeling of complementarity. This involves the question as to whether we will be able to understand the logical process by which the brain computes. Computation is, in the usual description, a logical operation. But when we look at the way problems like the one of the traveling salesman are solved on a neural network (Hopfield and Tank, 1985), it does not turn out to be a logical procedure. Figure 5a shows a representation of the state of the neural network at an intermediate time during the settling to a solution to such a problem. Each point in the array of squares represents the output of a particular neuron. When the square is as large as the one at the lower right, that neuron has an output of 1. When the system has found a good solution to the problem, each of the outputs will be either 1 or 0. T h u s , in the final state of the system, the output will appear as in fig. 5b, with one 1 in each row and each column. I n this computational process, each of the neurons can be thought of as standing for a proposition. A n output of 1 means that the proposition is considered to be true, and an output of 0 means that the proposition is not true. I n fig. 5, the propositions (which are of the general form "the salesman should go to city D as the 7th city in the tour") are laid out in such a fashion that each proposition contradicts the other propositions in each row and column. A n appropriate final state must have no more than one 1 in each row and each column. What are we then to make of the intermediate state of the form shown in fig. 5a? At this time, the proposition that city D should be in position 8 has a weight of 0.25 while at the same time, the contradictory propositions that city D should be in position 7 and that city C should be in position 8 also have weights of about 0.25. T h e system might be characterized as simultaneously having partial belief in several mutually contradictory propositions. There is no obvious logical characterization of this state of partial belief in contradictory propositions. W e cannot view this partial
67
311 1
2
3
4
5
6
7
8
9
10
1
(o)
2
3
4
5
6
7
8
9
10
Ibl
Fig. 5. The linear dimension of each square represents the activity of that neuron. The neuron i n square A3 represents the proposition that city A should be visited 5th. (a) A n intermediate state of the calculation. City 3 has been arbitrarily placed i n location 10, so lhat the output of the corresponding neuron is of maximum size, (b) The final stable stale of the network, represent the path A E G F I H D B C J (and return lo A ) .
belief in terms of probabilities, for the entire system is deterministic. There is a logical space in which a proposition is true or false, a space which for Jv" propositions can be described as the vertices on an ^-dimensional cube. The neural network computes by moving on a trajectory in the interior of this cube, while only the corners have logical meaning. Thus an insistence on finding correct logic during the decision process of the network destroys the computation as done by the network. The ability to use the non-logical interior of this space seems to be an important part of the power of these collective "neuronal" networks. Is this a sensible way to go—to try to understand a brain by giving up simple logic and moving toward analog, analytic, and collective behavior? John von Neumann (1948) thought about the relation of the brain to the digital computers which he had been so influential in helping develop, and concluded: "AJ] this will lead to theories (of computation) which are much less rigidly of an all-or-none nature... than formal logic. They will be of a much less combinatorial, and much more analytical, character." He continued this theme in later lectures (von Neumann 1952), but did not find what he regarded as a satisfactory solution to the problem of how a brain computes.
6. Summary We have learned a great deal about the chemical and electrophysiological basis of brain since Bohr wrote the cited essays. The way in which important aspects of brain function and neural computation could be of a collective nature is beginning
68
312 to emerge. We see some simplified governing equations which are relatable to collective behaviors. These collective behaviors even in rudimentary form seem to show how associative memory and comb in atori ally difficult decision-making of the kind necessary in visual or auditory perception can be carried out with great effectiveness by a modest collection of neurons. It is not clear where the limits of such pursuits will lie. Will we be able to explain some of the phenomena of cognitive psychology on the basis of details of microscopic neuronal behavior? Or will the macroscopics never be microscopically understandable? I f Bohr's argument that a form of complementarity may be involved in brain turns out to be correct, the mathematical explanation for it will lie in the nature of computational complexity and the meaning of computation. W i l l there be a mathematical complementarity in brain? M y own guess is that the complications of the system are not intrinsic, and that we will be able eventually to go astonishingly far toward understanding the language of psychology from a microscopic viewpoint. While weather prediction on the basis of molecular modeling is impossible, we do understand why there is weather and what storms are on the basis of an atomic approach to atmospheric phenomena. In this same sense, new approaches to theory in neurobiology should lead us at least to an understanding of the existence of psychological constructs on a microscopic basis, and potentially lead much further.
Postscript T h e editors have expressed the hope that someone might read this book in connection to the second Bohr Centenary. The science described here will, 100 years from now, be amusingly naive. How science is done—who influenced whom, and how—may be of greater interest. I close on a personal note, which would otherwise be lost. I did not know Niels Bohr. John Wheeler was the strongest influence in my taking a position at Princeton, and M a x Delbruck strongly influenced the Caltech scene and my interest in moving there. Both of these men were in turn involved with Bohr and Copenhagen. I have deep admiration for the imaginative scientific spirit of each of them. Reading the essays of Niels Bohr and listening to this symposium, I became increasingly aware that many elements I find admirable in these scientists are related to the spirit in which Bohr worked. But whether this similarity is due to influence or due to mutual affinity is not obvious. Francis O. Schmitt and the Neurosciences Research Program did much to stimulate my interest in neurobiology. T h e significance and liveliness of the many discussions about biological and network computation with D a v i d W. T a n k is gratefully acknowledged.
This research was supported in part by N S F grant PCM-8406049.
References Bohr, N . . 1958, Atomic Physics and Human Knowledge (Wiley, New York). Bohr. N . , 1963, Essays (1958-1962) on Atomic Physics and Human Knowledge (Inierscience, New York)
p! £
69
313 Gelperin, A., 1983, i n : Neuroethology and Behavioral Physiology, eds F. Huber and H . Markl (SpringerVerlag, Berlin) p. 189. Hebb. D.O., 1949. The Organization of Behavior (Wiley, New York). Hopfield, J.J., 1974, Proc. Natl. Acad. Sci. USA 71, 4135. Hopfield. J.J., 1982. Proc. Natl. Acad. Sci. USA 79, 2554. Hopfield. J.J.. 1984, Proc. Natl. Acad. Sci. USA 81, 3088. Hopfield, J.J., and D.W. Tank, 1985, Biol. Cybem. 52, 141. Hopfield, J.J., and T. Yamane, 1980, i n : Ribosomes, eds G. Chamblis. G.R. Craven. J. Davies, K . Davies, L. Kahan and M . Nomura (University Park Press, Baltimore) p. 585. Jackel, L.D., R.E. Howard. H.P. Graf, B. Slraughn and J.S. Denker, 1986, J. Vac. Sci. Tech. B4, 61-63. Kandel. E.R., and J.H. Schwarlz, 1981, Principles of Neuroscience (Elsevier, New York). Lambe, J., A . Moopenn and A.P. Thakoor, 1985, Jet Propulsion Lab Publication, November. 85. Little, W.A., 1974, Math. Biosci. 19, 101. Little, W.A., and G.L. Shaw, 1978, Math. Biosci. 39, 281. Mead, C.A., and L. Conway, 1980, Introduction to VLSI Systems (Addison-Wesley, Menlo Park, C A ) p. 335. Muzyczka. N „ R.L. Poland and M.J. Bessman. 1972, J. Biol. Chem. 247, 7116. Pauling. L., 1957, i n : Feslschrifl Arthur Sloll (Birkhauser, Basel) p. 597. Psaltis, D „ and N . Farhat, 1985. Opt. Lett. 10, 98. Sahley, C . A. Gelperin and J.W. Rudy, 1981, Proc. Nat. Acad. Sci. USA 78, 640. Sejnowskii, T.J., 1981, in: Parallel Models of Associative Memory, eds G.E. Hinton and J.A. Anderson (Lawrence Erlbaum Associates, Hillside, NJ) p, 189. Sivilotti, M . , M.R. Emmerling and C A . Mead. 1985, in: Conf. on Very Large Scale Integration, ed. H , Fuehs (Computer Science Press, Rockville, M D } p. 329. von Neumann. J.. 1948. i n : Collected Works, Vol. 5, ed. A . H . Taub (Pergamon Press, New York. published in 1963) p. 304. von Neuman, J., 1952, ibid, p. 354. Watson. J.D.. 1976. Molecular Biology of the Gene (Benjamin, Menlo Park, C A ) . Yates, J.L.. 1979, J. Biol. Chem. 254, 11550.
Discussion, session chairman N.K. Jerne Zimanyi: Y o u have used a symmetric connectivity matrix Ta. But in a neuronal network we have about as many inhibitory neurons as excitatory ones. The part of the connectivity matrix representing these types of neurons should be antisymmetric instead of symmetric. Howe can one overcome that problem? Hopfield: Even networks which do not in detail look symmetric may be constructed equivalently to symmetric networks and possess the stability properties of symmetric networks. The symmetry assumed here serves to ensure viability of the mathematics. Asymmetric systems open a Pandora's box of totally uncontrollable mathematical behavior, but have a much richer computational structure. von Weizsacker: 1 have no quarrel with reductionism. Bohr used to explain complementarity by the difficulty of language. C a n you explain the complementarity structure in language by your network mode? A n example: My uncle, Victor Weizsacker, a medical psychologist, who reminded me in many respects of Niels Bohr, once said: "When I ask you: 'what are you thinking now?', and you start to answer, you are already lying." Hopfield: Y o u are absolutely right in referring to Bohr's pointing out the language as an intrinsicaUy complementary system; complementarity must be left out from microscopies when deriving collective results in a reductionist attitude.
70
314 Anderson: G o i n g back to one of your first sentences about the complications in biology arising not from the size of Planck's constant, I should like to point to the size of Avogadro's number, a biological constant in itself, as it represents a piece of matter approximately our size. This may be the amount of matter necessary to produce the required complexity. Berry: Let me return to your point concerning how tliinking causes microscopic changes in physiology and your skepticism whether microphysiology could tell us when a person has solved a problem. Strictly, I should say "when the person thinks the problem is solved." I suspect you don't really believe that, because you and everybody in this room knows how solving a problem makes you feel better, quite literally. A problem must be awfully easy and awfully trivial to generate no such feeling at its solution. Consequently, do you not think, the microphysiology should display some chemical signal when the person thinks he or she has found the solution? Hopfield: Emotional states are relatively easy to recognize by chemical signals. I think it could probably be done now. But you would still not be able to tell what the person's understanding of the answer was or whether it was solved from your point of view.
71
terest. T o produce a valuable o b s e r v a -
Reprinitfd wilh permission \nymSde"ce. VM. 196, pp. 1161-1166,10 June 1977 ) ] 977 Tbe America Association for ihe Advuiccr^cnl of Science
tion, one has first to have an idea of what lo observe, a preconception of what is possible. Scientific a d v a n c e s often c o m e from uncovering a hitherto unseen a s pect of things a s a result, no! so m u c h o f using some new instrument, but rather o f
Evolution and Tinkering
looking at objects from a different angle. T h i s look is necessarily guided by a certain idea of w h a l ihe so-called reality
Francois Jacob
might be. It a l w a y s involves a certain conception about the u n k n o w n , that is, about what lies beyond thai w h i c h o n e has logical or experimental reasons to
Some o f ihe 16th-century books devoted to zoology and botany arc illustrated b y superb drawings of the various animals I hat populate ihe earth. C e r t a i n contain detailed descriptions o f such creatures as dogs with fish heads, men with c h i c k e n legs, or e v e n w o m e n without heads. T h e notion of monsters that blend the characteristics of different species is not itself surprising: everyone h a s imagined or sketched such hybrids. What is disconcerting today is thai in [he 16th century these creatures belonged, not lo the world of fantasies, but to the real world. M a n y people had seen rhem and described them in detail- T h e monsters walked alongside Ihe familiar animals of e v e r y d a y life. T h e y were wiihin the limits of Ihe possible. W h e n looking at present-day science fiction books, one is struck by the same phenomenon: the abominable animals lhal hunt the poor qsironauf lost o n a distant planet are products o f recombinations between the organisms living on the earth- T h e creatures coming from outer space to explore the earth are depicted in Ihe likeness of man. Y o u can w a t c h them emerging from their unidentified flying objects ( U F G ' s k they are vertebrates, mammals without any doubt, walking erect. T h e only variations concern body size and the number of eyes. Generally these creatures have larger skulls rhan h u m a n s . To suggest bigger brains, and sometimes one or two radioantennae o n the head. To suggest very sophisticated sense organs. T h e surprising poinT here again is what Is considered possible. It is the idea, more than a hundred years after D a r w i n , that, if life o c c u r s a n y w h e r e , it is bound to produce animals not loo different from Ihe terresrrial o n e s ; and above all to evolve something like man.
The interest in I h c s r monsters is that •hey show how a culture handles the possible and marks its limits. It is a require¬ ment of the human brain to put order in the universe. It seems fair lo say lhal all cultures have more or less succeeded in providing their members with a unified and coherent view of the world and of the forces that run i l . O n e may disagree wilh the explanatory systems offered by myths or magic, but one cannot deny them unity and c o h e r e n c e . Tn fact, ihey are often charged with too much unity and coherence because of their capacity to explain anything b y the same simple argument. A c t u a l l y , despite their differences, whether m y t h i c , magic, or sci enlific, all explanatory systems operate on a common principle. In the w o r d s of the physicist J e a n Perrin. the heart of Ihe problem is a l w a y s "to explain the complicated visible by some simple invisible" ( / } . A thunderstorm c a n be viewed as a L-onsequence of Z e u s ' anger or o f a difference of potential between ihe clouds and the earth. A disease c a n be seen as (he result of a spell cast o n ihe patienl or of an infection by a v i m s . In all cases, h o w e v e r , one watches the visible effect o f some hidden cause related lo the whole sei of invisible forces that are supposed to run the world.
believe, in (he words of Peter M e d a w a r , scientific investigation begins b y the " i n vention of a possible world or o f a tiny fraction of that, w o r l d " (2). S o also begins mythical thought. B u t it stops there. H a v i n g constructed what it considers as the only possible w o r l d , it easily fits reality
into
its
scheme-
For
scientific
thought, instead, imagination is only a part of ihe game. At e v e r y step, it has to meet with experimentation and c r i t i c i s m . T h e best world is the one that exists and has proven to w o r k already for a long time. S c i e n c e attempts io confront the possible w i l h the a c t u a l . The price to be paid for Ihis outlook, however, turned out to be high. It w a s , and is perhaps more than e v e r , renouncing a unified world v i e w . T h i s from the very way science Most
other
mythic,
systems
results
proceeds.
of e x p l a n a t i o n -
magic, or religious—generally
encompass
everything. T h e y apply to
e v e r y domain. T h e y a n s w e r any possible question. T h e y account for the origin, the present, and the end o f the u n i v e r s e . Science proceeds differently. It operates by detailed experimentation w i l h nature and thus appears less ambitious, at least at first glance. I I docs not aim at reaching al once a complete and definitive explanation of the whole u n i v e r s e , its beginning, and its present form. I n s t e a d , i l looks for partial and provisional a n s w e r s
The W o r l d View c-f Science
about those phenomena that c a n be isolated and well defined. A c t u a l l y , the be-
Whether mythic or scientific, the view
ginning of modern science c a n be dated
of the world that man constructs is a l -
from the time when s u c h general ques-
ways largely a product of imagination.
tions as, " H o w w a s the universe created?
F o r the scientific process does not c o n sist simply in observing, in collecting data, and in deducing from them a theory. O n e can w a i c h an object for years and never produce any observation o f scientific in-
72
The author l% a professor at cell genetic* zi ihe
In&EiEul Pahleui. 2B Rue .1. DockEeui R D U > 75015.
Paris. Fiance. Thii jnidr is [he text of a lecture delivered Hi the University of California. Berkeley, in March 1977.
W h a t is m a i l e r made of? What is ihe es-
j e c t s , the problem is a l w a y s to explain
sence of l i f e ? " w e r e replaced by s u c h
the more complex in terms a n d concepts
limited! questions as " H o w d o e s a stone
applying lo the simpler. T h i s is ihe old
the human societies with w h i c h ethnol-
fall? H o w d o e s w a t e r flow in <• l u b e ?
problem of reduction, emergence, whole
ogy a n d sociology deal represent only a
H o w d o e s blood circulate in v e s s e l s ? "
and parts, and so forth. Ls it passible lo
restricted
T h i s substitution h a d an amazing result.
reduce chemistry to p h y s i c s , biology to
actions between human beings.
While asking general questions
led
proportion o f these 200 types. S i m i l a r l y ,
group o f all possible
inter-
1
to
p h y s i c s plus c h e m i s t r y , and so forth ?
limited a n s w e r s , asking limited questions
C l e a r l y an understanding of the simple is
turned out to provide more and more
necessary lo understand the more com-
general a n s w e r s .
p l e x , but whether i l is sufficient is ques-
At Ihe same l i m e , h o w e v e r , this scien-
rangement, in the n u m b e r , and in the
tionable.
Constraints and History Nature
functions
by
integration.
tific method could hardly avoid a parcel-
T h i s type of question has resulted in
W h a t e v e r the l e v e l , the objects a n a l y z e d
ing out of ihe world view. E a c h b r a n c h
endless arguments. O b v i o u s l y , the two
by natural sciences are a l w a y s organiza-
of science investigates a particular do-
critical events o f evolution—first the ap-
tions, or systems. E a c h s y s t e m at a given
main that is not necessarily
pearance of life and later that of thought
level uses as ingredients some systems of
wilh
the
neighboring
connected
thai
the simpler level, but some o n l y . T h e hi-
knowledge thus a p p e a r s lo consist of iso-
ones.
previously did not exist on the earth. T o
e r a r c h y in the complexity of objects is
lated islands. I n the history of s c i e n c e s ,
describe and to interpret these phenome-
thus a c c o m p a n i e d by a series of restric-
important a d v a n c e s
na,
tions and limitations. At e a c h level, new
often
Scientific
come
from
and
Language—led to phenomena
new concepts,
meaningless at Ihe
bridging Ihe gaps. T h e y result from Ihe
previous l e v e l , are required. What c a n
properties
recognition
separate
ihe notions of sexuality, o f predator, or
new constraints on the system. B u i these
observations c a n be viewed from a new
o f pain represent in p h y s i c s or c h e m -
are merely additional constraints. T h o s e
angle and seen lo represent nothing but
istry? O r the ideas of j u s t i c e , of increase
that operate at any given level are stilt
different
phenomenon.
in value or of d e m o c r a t i c power in biol-
valid at all more complex levels. E v e r y
T h u s , terrestria! and celestial mechanisms
ogy? A t the limit, tola! reduction ism re-
proposition that is true for p h y s i c s is also
became a single science with N e w t o n ' s
sults in absurdity. F o r Ihe pretention thai
true for c h e m i s t r y . biology, or sociology.
l a w s . T h e r m o d y n a m i c s and mechanics
every level can be completely reduced to
Similarly e v e r y proposition that is v a l -
w e r e unified through statistical mechan-
a simpler one would result, for example,
id for biology holds true in
lhal two
facets
of
hitherto
one
may
appear w h i c h
impose
sociology.
i c s , as w e r e optics a n d electromagnet ism
in explaining d e m o c r a c y in terms of the
Bui as a general rule, the statements o f
through M a x w e l l ' s theory of magnetic
structure a n d properties of elementary
greatest importance a l one level are of no
held, o r chemistry and atomic physics
particles; a n d this is clearly nonsense.
interest at the more complex ones. T h e
through quantum m e c h a n i c s . Simdarly different
same
different w a y . O n e c a n look a l the series
Ihe objects of biology or sociology than
laws,
were
of objects, moving from the simpler lo
for those of physics. It is simply irrelevant
s h o w n by biochemists lo compose
both
the more c o m p l e x . M o l e c u l e s are made
in the context o f the problems w i l h which
of atoms. T h e y therefore obey the laws
biologists,
thai determine ihe behavior of atoms.
gists, are c o n c e r n e d .
obeying
the
of
l a w of perfect gases is no less true for
the
atoms,
combinations
T h i s problem can be considered in a
same
the inanimate and the living worlds.
But, in addition, two statements can be made about molecules.
T h e H i e r a r c h y of Objeets
This
and e v e n
hierarchy
of
more so
sociolo-
successive
in-
F i r s t , they can
tegrations, c h a r a c t e r i z e d by restrictions
exhibit new properties, s u c h as isomeri-
and by ihe appearance of new properties
how-
zalion, r a c e m i z a i i o n , and so forth. S e c -
a l each l e v e l , has several c o n s e q u e n c e s .
e v e r , large gaps r e m a i n , some o f w h i c h
o n d , the subject matter of c h e m i s t r y , Ihe
T h e first is the necessity o f analyzing
probably will not be bridged for a long
molecules found in nature or produced in
complex objects a l all levels, Lf m o l e c u -
time, if e v e r . T o d a y , there exists a series
Ihe laboratory, represents only a small
lar biology, which presents a strong re-
of sciences thai differ, not only by the
fraction of all the possible interactions
ductionist altitude, yielded s u c h a suc-
nature of the objects that are studied, but
between atoms. C h e m i s t r y constitutes,
cessful analysis of heredity, it was main-
also by the c o n c e p t s and the
language
therefore, a special case of p h y s i c s . T h i s
ly because, at every step, the analysis
that are used. T h e s e sciences can be ar-
is e v e n more so with biology l h a l deals
was carried out simultaneously at the
ranged
wilh
Despite
such
in
a
generalizations,
certain
order—physics,
a complex
hierarchy o f
object*
the black box, the bacterial cell. T h i s ap-
an order that c o r r e s p o n d s to the hier-
e c o s y s t e m s - T h e objects w h i c h exist at
plies also lo recent developments in im-
a r c h y of complexity found in the objects
each level constitute a limitation of the
munology. A n d il seems likely that s u c h
of
total possibilities offered by the simpler
a convergence of analysis will play an
from p h y s i c s lo sociology, one goes from
level. F o r instance, the set of molecules
important role in the study of human
the simpler to the more complex objects
found in living organisms represents a
beings and their societies.
and also, for obvious r e a s o n s , from the
very restricted range of chemical
older to the younger s c i e n c e , from the
j e c t s . At the next l e v e l , the number of
ability. I s it possible to make predictions
poorer lo the richer empirical content, as
animal species amounts to several mil-
at one
well as from the harder to the softer sys-
lions; h o w e v e r , this is small relative to
known at a simpler o n e ? O n l y to a very
tem of hypotheses a n d experimentation.
ihe number thai could exist. All verte-
limited extent. T h e properties of a sys-
In order to obtain a unified world v i e w
brates are composed
tem can be explained by the properties o f
through s c i e n c e , the question has repeat-
number of cellular types, at most
edly been raised as 10 Ihe possibdily of
such
mailing bridges between
adjacent dis-
nerve c e l l s . T h e great diversity of verte-
laws of p h y s i c s , there is no w a y of recon-
c i p l i n e s . B e c a u s e o f Ihe h i e r a r c h y of ob-
brates results from differences in the ar-
structing the universe. T h i s means that a
these
biology,
sciences.
psychosociology—
F o l l o w i n g the
line
ranging from cells to populations
level of ihe molecules and at the level of
and
chemistry,
of a very limited
as muscle c e l l s ,
73
ob-
T h e second level
point c o n c e r n s predicton
the basis of what
is
200,
lis components. T h e y cannot be deduced
skin cells, and
from them. Slarting from fundamental
!• ii 111. ii system, say a cell, has only a certain probability of appearing. AM prc• I - - 11 • - • - aboui its existence can only statistical. Molecular biology has shown that ultimately the characteristics of a cell rest on the structure of its molecular components- But the appearance of life on the earth was not the necessary consequence of the presence of certain mo¬ lecular structures in prebiolic times. In fact, there is absolutely no way of estimating what was ihe probability for life appearing on earth. Il may very well have appeared only once. The third point concerns the nature or the restrictions and limitations found at every step of increasing complexity. Can one explain why. among all the possible interactions at one level, only certain are actually observed at the more complex one? How is il that only some types of molecular structures are present, for instance, in living organisms? Or only some interactions in human societies? There is no general answer lo such questions, and it seems doubtful that there will ever be a specific answer for any one particular level of complexify- Complex objects are produced by evolutionary processes in which two factors are paramount: the constraints that at every level control the systems involved, and the historical circumstances that control the actual interactions between the systems. The combination of constraints and history exists al every level, although in different proportions. Simpler objects are more dependent on constraints lhan on history. As complexity increases, history plays a greater pail. But history has always to be introduced into the picture, even in physics. According to presenl theories, heavier nuclei are composed of lighter ones and ullimalely of hydrogen nuclei and neutrons- The transformation of heavy hydrogen into helium occurs during the fusion process, which is the main source of energy in the sun as well as in hydrogen bombs. Helium and all Ihe heavier elements are thus ihe result of a cosmological evolution. According to present views, the heavier elements are considered as products of supernovae explosions. They seem to be very rare and not lo exceed I or 2 percent by mass of all mailer, while helium represents one-fifth and hydrogen four-fifths of all matter. The earth and the other planets of Ihe solar system have thus been made of very rare material under conditions that seem to be rarely encountered in the cosmos. The source of hydrogen itself is left to theories and Speculations concerning the origin of the universe.
Engineer and Tinkerer
NaluraL Selection be The constraints to which systems are subjected vary with the level of com* plexily- There are always some Constraints imposed by stability and thermodynamics. Bui as complexity increases, additional constraints appear—such as reproduction for living systems, or economic requirements for social systems. Consequently, there cannot be any general law of evolution, any recipe that accounts for increasing complexity at all levels. Since Darwin, biologists have progressively elaborated a reasonable, although slill incomplete, picture of the mechanism that operates in the evolution of Ihe living world, namely, natural selection. For many, it has been templing to invoke a similar mechanism of selection to describe any possible evolution, whether cosmological. chemical, cultural, ideological, or social. But this seems condemned to fail from Ihe outset. The rules of ihe game differ at each level. New principles have therefore, to be worked oul al each level. r
The action of natural selection has often been compared to lhal of an engineer. This, however, does not seem to be a suitable comparison. First, because in contrast to whal occurs in evolution, ihe engineer works according lo a preconceived plan in that he foresees ihe product of his efforts. Second, because of ihe way the engineer works: to make a new product, he has at his disposal both material specially prepared to that end and machines designed solely for thai task. Finally, because the objects produced by the engineer, al least by the good engineer, approach the level of perfection made possible by the technology of Ihe time. In contrast, evolution is far from perfection. This is a point which was repeatedly stressed by Darwin who had To fighi against the argument of perfecl creation. In the Origin of Species. Darwin emphasizes over and over again the structural or functional imperfections of the living world. For instance, when he discusses natural selection ;. . p. 472): ;
Natural selection is the result of two constraints imposed on every living organism: (i) the requirement for reproduction, which is fulfilled through genetic mechanisms carefully adjusted by special devices such as mutation, recombination, and sex to produce organisms similar, bul not identical, lo their parents: and • !'• the requirement for a permanenl interaction with the environment because living beings are whai ihermodynamic ists call open systems and persist only by a constant flux of matter, energy, and information. The first of these faclors generates random variations and produces populations in which all individuals are different. The interplay of the two factors results in differential reproduclion and consequently in populations that evolve progressively as a function of environmental circumstances, of behavior, and of new ecological niches. Bul natural selection does not act merely as a sieve eliminating detrimental mulalions and favoring reproductions of beneficial ones as is often suggested. In the long run, it integrates mulalions, and it orders them into adaptalively coherent patterns adjusted over millions of years, and over millions of generations as a response lo environmental challenges. Il is natural selection that gives direction to changes, orients chance, and slowly, progressively produces more complex structures, new organs, and new species. Novelties come from previously unseen association of old material. To create is lo recombine.
74
Nor ought we to marvel if all rhf contrivances in nature be not. AS far as we can judge, absolutely pelted- We need, noi marvel at the sling of the bee causing the bee's own death; at drones being produced in such vast numbed for one single act. and being then slaughtered by their sterile sisters: at the astonishing wnsie of pollen by ourfirtreesial the instinctive hatred of ihe queen bee for her own fertile daughters; at ichneumonidae feeding within the live bodies of caterpillars: and at other such cases. The wonder indeed is, on [he theory of natural selection, that more cues of the want of absolule perfection have nol been observed. There are innumerable statements of Ihis type in ihe Origin ofSptcies. In fact,one of ihe best arguments againsi perfection comes from exlinci species. While the number of living species in The animal kingdom can be estimated to be around a few million, the number of exiincl ones since life existed on earth has been estimated by Simpson (4) al aroundfivehundred million. Natural selection has no analogy with any aspect of human behavior. However, if one wanted co play with a comparison, one would have to say thai natural selection does not work as an engineer works. It works like a tinkerer—a linkerer who does not know exactly whal he is going lo produce but uses whatever hefindsaround htm whether it be pieces of string, fragments of wood, or old cardboards: in short il works Like a linkerer who uses everything at his disposal to produce some kind of workable object. For the engineer, the realization
of his (ask depends on his having the raw mate rials and the tools thai exactly fil his project. The tinkerer, in contrast, always manages with odds and ends. What he ultimately produces is generally related to no special project, and it results From a series of contingent events, of all the opportunities he had to enrich his stock with leftovers. As was discussed by LBvlStrauss O h none of Ihe materials at the tinkerer's disposal has a precise and definite function. Each can be used in a number of different ways. In contrast with the engineer's tools, those of ihe tinkerer cannot be defined by a project. What these objects have in common is "it might well be of some use." For whal? That depends on ihe opportunities.
Evolution as Tinkering This mode of operation has several aspects in common with the process of evolution. Often, without any well-delined long-term project, the tinkerer gives his materials unexpected functions to produce a new object. From an old bicycle wheel, he makes a roulette; from a broken chair the cabinet of a radio. Similarly evolution makes a wing from a leg or a part of an ear from a piece of jaw. Naturally, this lakes a long lime. Evolution behaves like a tinkerer who, during eons upon eons, would slowly modify his work, unceasingly retouching it. cutting here, lengthening there, seizing the opportunities to adapt it progressively to its new use. For instance, the lung of lerrestrial vertebrates was, according to Mayr 1,6), formed in ihe following way. Its development started in certain freshwater fishes living in stagnant pools with insufficient oxygen. They adopted the habit of swallowing air and absorbing oxygen through the walls of the esophagus. Under these conditions, en large men I of the surface area of ihe esophagus provided a selective advantage. Diverticula of the esophagus appeared and. under continuous selective pressure, enlarged into Jungs. Further evolution of Ihe lung was merely an elaboration of this theme—enlarging the surface for oxygen uptake and vascularization. To make a lung with a piece of esophagus sounds very much like tinkering. Unlike engineers, tinkerers who tackle Ihe same problem are likely to end up wilh different solutions. This also applies to evolution, as exemplified by The variety of eyes found in the living world [see (7)]. It is obviously a great advantage under many conditions to possess light receptors, and Ihe variety of photoreceptors in Ihe living world is amazing. The
most sophisticated are ihe image-forming eyes that provide information, not only on Ihe intensity of incoming light, but also on Ihe objects tight comes from, on their shape, color, position, motion, speed, distance, and ihe like. Such sophisticated structures are necessarily complex. They can develop only in organisms already complex themselves. One might suppose, therefore, that Ihere is just one way of producing such a structure. This is no! ihe case. Eyes appeared a greal many times in the course of evolution, based on at least three principles—pinhole, lens, and multiple lubes. Lens eyes, like ours, appeared both in mollusks and vertebrates. Nothing looks so much like our eye as the octopus eye. Both work in almost exactly the same way. Yet they did not evolve in the same way. Whereas in vertebrates photoreceptor cells of the retina point away from light, in mollusks they point toward light. Among all solutions found to Ihe problem of photoreceptors, these two are similar but not identical. In each case, natural selection did what it could with the materials at its disposal.
same and the translating machineries are very nearly so. The same coenzymes mediale similar reactions. Many metabolic steps remain essentially the same, from bacteria to man. Obviously, for life to emerge, a number of new molecular types had first to be formed. During chemical evolution in prebiotic limes and at Ihe beginning of biological evolution, all those molecules of which every living being is buill had lo appear. But once life had started in The form of some primitive self-reproducing organism, further evolution had to proceed mainly through alterations of already existing compounds. New functions developed as new proteins appeared. But these were merely variations on previous themes. A sequence of a thousand nucleotides codes for a medium-sized protein. The probability thai a functional proTein would appear de novo by random associaTion of amino acids is practically zero. In organisms as complex and integrated as Those That were already living a long time ago, creation of entirely new nucleotide sequences could not be of any importance in the production of new information.
Evolution does not produce novelties from scratch. Il works on what already exists, either transforming a system to give il new functions or combining several systems to produce a more elaborate one. This happened, for instance, during one of the main events of cellular evolution: namely, the passage from unicellular to multicellular forms. This was a particularly important transition because it carried an enormous potential for a specialization of The parts. Such a transition, which probably occurred several times, did not require the creation of new chemical species, for there are no major differences between molecular Types of uniand multicellular organisms. Lr was mainly a reorganization of what already existed.
The appearance of new molecular structures during much of biological evolution must, therefore, have rested on alteration of preexisting ones. This is exemplified by the finding lhal large segments of genetic information, thai is. of DNA, turn out to be homologous, nol only in the same organism, bul also among different organisms, even among those that are phylogeneticatly distant. Similarly, as more is known about amino acid sequences in proteins, it appears not only that proteins fulfilling similar functions in different organisms have frequently similar sequences, buf also that proteins wilh different functions oflen exhibit rather large segments in common. The hypothesis most generally envisaged to account for these similarities was proposed by Horowitz (£), by Ingram [9\. and by Ohno U0). A segment of DNA. corresponding to one or several genes, is assumed to be duplicated by some genetic mechanism. When a gene exists in more lhan one copy in a cell or a gamete, il is released from the contrainls imposed on functions by natural selection. Mutations can then accumulate more or less freely and result in modified protein structures, some of which can eventually fulfill new functions. Since natural selection exerts a continual pressure on organisms, an alteration in a protein can be further improved by other, later changes, IT can also lead to a perturbation in the interactions with other proteins and eventually favor modifications of these proteins. A large fraction
Molecular Tinkering It is al the molecular level that the tinkering aspect of natural selection is perhaps most apparent. Whal characterizes ihe living world is both its diversity and its underlying unity. The living world contains bacteria and whales, viruses and elephants, organisms living ai -2Q*C in polar areas and others at 70"C in hot springs. All these objects, however, exhibit a remarkable unity of chemical structures and functions. Similar polymers, nucleic acids or proteins, always made of Ihe same basic elements, the four bases and The 20 amino acids, play similar roles. The genetic code is the
75
of the genome of complex organisms might actually derive from a few antesiral genes. Biochemical changes do not seem, therefore, lo be a main driving force in the diversification of living organisms. The really creative part in biochemistry must have occurred very early. For the biochemical unity that underlies the living world makes sense only if most of the important molecular types found in organisms, that is. most of the metabolic pathways involved in the production of energy and in biosynthesis or degradation of the essential building blocks already existed in very primitive organisms such as bacteria. Once this stage passed, biochemical evolution continued as more complex organisms appeared, fiut it is not biochemical novelties that generated diversification of organisms. En all likelihood, il worked the other way around, tl is Ihe selective pressure resulting from changes in behavior or in ecological niches that Led to biochemical adjustments and changes in molecular types. What distinguishes a butterfly from a lion, a hen from a fly. or a worm from a whale is much less a difference in chemical cons til uents than in the organization and the distribution of these a n stifuents The few big steps of evolution required acquisition of new information. But specialization and diversification occurred by using differently the same structural information. Among neighboring groups, vertebrates for instance, chemistry is ihe same. What makes one vertebrate difTerenl from another is a change in the lime of expression and in the relative amounts of gene products rather than the small differences observed in the structure of these products. Il is a matter of regulation rather ihan of structure (see [ID]. After egg fertilization, embryonic development occurs in afixedorder and according to a precise schedule set by the genetic program contained in the chromosomes. This program determines when and where lines of differentiated cells will emerge, when and where different proteins wdl be made and in what amounts. Both the quality and quantity of ihe different proteins vary in time and space during development. Thus in The adult, the various types of cells or tissues contain different repertoires of molecular types in agreement with their functions. The genetic program is executed through complex regulatory circuits that switch the different biochemical activities of Ihe organism on or off. Very little is known as yet about the regulatory circuits that operate in fhe development of complex organisms. It is known, however, that.
among related organisms such as mammals, the first steps of embryonic development are remarkably similar, with divergences showing up only progressively as developmeni proceeds. These divergences concern much less the actual structure of cellular or molecular types than their number and position. 11 seems likely that divergence and specialization of mammals, for instance, resulted from mutations altering regulatory circuits rather than chemical structures. Small changes modifying the distribution in time and space of the same structures are sufficient to affect deeply the form, the functioning, and the behavior of the final product—the adult animal. It is always a mailer of using the same elements, of adjusting them, of altering here or there, of arranging various combinations to produce new objects of increasing complexity. It is always a matter of tinkering.
Con sequences of Tinkering Marks of this tinkering are thus found at every level thoughout the living world. Of course, they can be found in human beings as shown by the following few examples. In humans, as in many mammals, there exist very complex processes responsible for such functions as blood coagulation, inflammatory reactions against foreign bodies, and the immunological defenses mediated by the socalled complement system. These three processes have been independently analyzed in some detail during recent years. Each one exhibits an unexpected complexity. Each involves aboul ten proteins, none of which initially has enzymatic activity. Conversion of the first protein into a catalytkally active form [riggers a cascade of reactions. The first protein cleaves the second one at a specific point; a product of this reaction cleaves the third protein, and so on. In this series of reactions, the individual proteins are thus split in sequence and Ihe released fragments serve as activators, or inhibitors, in other reactions of Ihe chain. Furthermore, ihese three chains of reactions are noi wholly independent. A product of cleavage in one chain can suddenly become an active element in another chain or even play a role in a completely different process. These products may serve as signals to conned chemically unrelated, but physiologically dependent, systems. 11 is as though some protein molecules, which happened to be formed, were used here or there as a source of smaller but active peptides as new functions were taking shape. Recently, a number of peptides of
76
different sizes have been found to participate in a variety of physiological processes. Some of them, such as hormone peptides or brain peptides, are known not to be chemically transformed in the reaction they activate or inhibit. They appear just to bind to some protein lo favor an allosteric transition, thus acting as simple chemical signals. For the biologist, it is thus generally impossible to make a prediction, or even an inspired guess, about the nature of such molecules and their structural relations with other consliluents. All he can do is to detect them, purify them, and analyze them. Later, as ihe structures of more proteins become known, there wdl perhaps be a chance to define the functional interrelations and evolutionary relationship among such molecules. Anolher example of tinkering can be found in early human embryonic development. Embryonic development is a tremendously complicated process of which little is known al present. Studies of the past 10 or 20 years have revealed an amazing phenomenon. In various human populations, 50 percent of all conceptions are estimated lo result in spontaneous abortion |see U2)\. A large fraction of these abortions occur during Ihe first 3 weeks of pregnancy and generally pass unnoticed. Thus, in half of the total conceptions, something is wrong to begin wilh. Many of these spontaneous abortions appear lo be due to an odd number of chromosomes; Instead of having one set of chromosomes derived from its mother and one from its father, the embryo lacks a chromosome, or has an extra one, or even has three sets instead of two. As a result, some (unctions necessary To embryonic development are not performed correctly. The fetus dies and is expelled. Thus many potentially malformed fetuses disappear; not all, unfortunately, since some of them still come to term. This reveals the imperfections of a mechanism that is a| the very core of any living system and that has been refined over millions of years. A third example of tinkering which is very intriguing when one thinks about it is the association between reproduction and what is generally called pleasure. Sex is one of Ihe most efficient inventions of evolution. In lower organisms which apparently reproduce asexual I y by fission, the genetic program is scrupulously recopied at every generation. Within a population, it always remains the same, except for rare mutations. Division of the organism is an automatic process resulting from growth. When something resembling sexuality exists, as in bacteria, it is a luxury. In such pop-
ulations. adaptation necessarily involves
ter, a rat gives himself pleasure until U
ous and hormonal system partially, but
the selection of rare mutants under envi-
collapses from sheer exhaustion. E x p e r i -
not totally under the rule o f the
ronmental c o n d i t i o n s . I n contrast,
ments
cortex—strongly
ual
reproduction,
sex-
w h i c h probably
oc-
and
performed during brain surgery
descriptions
of
feelings
by
the
resembles
neo-
the
lin-
kerer's procedure. I l is somewhat
like
curred e a r l y in e v o l u t i o n , c o m p e l s reas-
patients leave very lilTle doubl as to Ihe
s o r t m e n l of genetic
programs in inter-
existence of s u c h centers in m a n and to
is not surprising, in either case, lhal a c c i -
breeding populations
A s a result, e v e r y
ils association wirh sexual activity. T h u s
dents, difficulties, and conflicts can oc-
genetic program (that is. e v e r y individ-
pleasure appears as a mere expedienl to
cur.
ual) is different from the others. T h i s per-
push individuals to indulge in sex and
manent
iherefore
reshuffling o f genetic
elements
to reproduce,
A rather s u c -
adding a j e t engine to an old horse cart. It
I l is hard
to realize that the living
world as w e k n o w it i s j u s l one
among
provides tremendous potentialities o f ad-
cessful expedient indeed, as j u d g e d by
many possibilities; that its actual struc-
aptation. B u t once sexuality had become
The stale of the w o r l d population.
ture
a n e c e s s a r y condition for reproduction,
recognize and meet each other and a second, driving them to unite. T h e first of these requirements has been fulfilled by a variety o f specific
signaling
systems—
from
the
history
of
the
earth. Vet living organisms are historical
il required special m e c h a n i s m s : o n e . allowing individuals of opposite sexes lo
results
structures: literally creations of history. T h e y represent, not a perfect product of
A F i n a l E x a m p l e of T i n k e r i n g :
engineering, but a patchwork of odd sets
The H u m a n Brain
pieced together when and where opporAlthough
our
brain
represents
The
tunities arose.
F o r the opportunism of
species,
natural selection is n o l simply a matter of
v i s u a l , auditory, or o l f a c t o r y — o f amaz-
main
adaptive
feaTure o f our
whaT it is adapted lo is not clear at all-
indifference to the structure and opera-
ing precision and efficiency. T h e s e c o n d
What is c l e a r , h o w e v e r , is (hat, like The
lion of its products. It reflects Ihe very
has been met through the development of
rest of our body, o u r brain is a product of
nature of a historical process full of con-
genetically
and v e r y rigid
natural selection, lhal is, of differential
tingency.
programs of behavior. F o r instance, in
reproductions accumulated o v e r millions
determined
A s S i m p s o n {4) pointed o u | . The in-
birds, at the proper season, Ihe v i e w of
of years under The pressure of various
terplay of local opportunities—physical,
an individual of ihe opposite sex initiates
environmental conditions- O u r brain has
ecological, and constitutional—produces
a whole p r o c e s s of rituals, c o u r t s h i p , and
therefore evolved at our gonad's s e r v i c e ,
a net
parade leading almost automatically
as already e m p h a s i z e d by F r e u d many
lum
copulation, nidation, and progeny c a r e . T h e course
to
hisTorical determines
opportuniTy which how
genetic
in
oppor-
years ago. But curiously enough, brain
tunities
is
development in mammals was not as in-
net
c h a r a c t e r i z e d by a trend to greater flexi-
tegrated a process a s , for instance. The
controls the direction and pace o f adap-
bility in the execution of the genetic pro-
transformation of a leg into a wing. T h e
tive evolution. T h i s is why the probability
gram.
human
As
of evolution,
this
however,
program b e c a m e
more
is
this
mainly
is practically z e r o lhal living systems,
position of new structures on old ones.
w h i c h might well exist elsewhere in ihe
To
lower
c o s m o s , would have evolved into some-
actions to sexual signals were no longer
m a m m a l s a neocortex
lhal
thing looking like human beings. E v e n if
completely automatic. In order to drive
rapidly, perhaps too rapidly, look a most
life in outer space uses the same material
the
important role in the evolutionary
se-
as on the earth, e v e n if the environment is
sexuality had therefore To be associated
quence leading lo man. F o r some neu-
not too different from ours, e v e n i f ihe
with s o m e other d e v i c e s . A m o n g these
robiologists,
nature of life and of its chemistry strong-
w a s pleasure. In the O x f o r d d i c t i o n a r y ,
these two types o f structures c o r r e s p o n d
ly limits the way to fulfill certain func-
pleasure is defined as "the opposite of
to iwo Types o f functions but have not
tions, Ihe sequence of historical opportu-
p a i n . " obviously, but also as "the condi-
been
nities there could not be the same
tion of c o n s c i o u s n e s s induced by the en-
toward
reproduction,
by
It lhat
less rigidly determined by the genes- Re-
individuals
formed
exploited.
super-
old
was
be
o p e n , so to s p e a k , the behavior became
the
brain
will
historical opportunity
rhinencephalon
was added
especially
completely
of
McLean
coordinated
[t3),
or hier-
as
a r c h i z e d . T h e recent one, the neocortex,
here. A different
joyment o f w h a l is felt or viewed as good
controls intellectual. cognitive
formed by different actors. Despite sci-
or d e s i r a b l e , " I I s e e m s likely that feel-
T h e old one, derived from ihe rhinen-
ence
ings of discomfort and pleasure m u s i al-
cephalon, controls emotional and visceral
us. A n d we might as well have
ready have e x i s t e d
activities. In contrast to the former, ihe
like one o f those 16th-century monsters.
complex
for a long time in
a n i m a l s . A n animal is
activity.
more
latter d o e s not seem to possess any pow-
likely to have progeny if a feeling of dis-
er of specific d i s c r i m i n a t i o n , or any c a -
comfort dissuades it from entering harm-
pacity for s y m b o l i z a l i o n ,
ful situations. It is clear thai the exist-
self-consciousness.
ence of n e r v o u s c e n t e r s , connected w i l h
w h i c h , in l o w e r m a m m a l s , was in total
sense organs and able lo correlate what
c o m m a n d has been relegated to the de-
is fell as
partment of emolions. In man, it consti-
pleasant or unpleasant
with
The
what is actually good or bad for s u r v i v a l ,
tutes w h a l M c L e a n c a l l s
language, old
- L
or
structure
i h e visceral
is of great selective value. In fact, s u c h
b r a i n . " Perhaps because development is
centers are now k n o w n to exist. S o m e 20
so prolonged and maturity so delayed in
y e a r s ago,
m a n . these centers maintain strong con-
neurobro legists delected
in
the brain, first in the rat and later in
nections with l o w e r autonomic
many vertebrates, the presence of two
and continue to coordinate such funda-
remarkable centers—one called Ihe center
mental d r i v e s as obtaining food, hunting
of a v e r s i o n and the other called the center
for a sexual p a n n e r . or reacting to an
of a ulosti mutation. F i l l e d with c o r r e c t l y
enemy.
implanted
ihe
ihe formation of a dominating neocortex
means o f activating al will i h e latter c e n -
coupled with Ihe persistence of a nerv-
electrodes
and
given
T h i s evolutionary
77
centers
procedure—
fiction.
play had to be per-
Marlians cannot look like looked
RcffirnMs u d Nom 1. i. Penip. Ui Arwirs tAkMi. Paris. 19I4J, 2. P. B. Mcdzwai, THt Hope of Pro teu (Doubleday, We* Yoik, 1973V 3. C Darwin, On thr OHuJn nfSprc'f (London. e
4. G. G. Sirtipfcon. Evolution 4. J42(t9S2V 5. C. Levi StrausE, Lo Pent* Swage iPJon. Pari*, L»Z). ft. E. Mayr. Fed. Prac. Ftd. Am. Sac. Etp. Biol. 23, I2)t (1964). 7. G. G. Simpson,The Mean™ of Evolution (Yjfc Univ. Prcs*. Ne* Ha-en, Conn. J»7). 8- N . H o r < " M l i . ^ . O « ^ i . J i a 9 S u ] V. V. M. Ingram, Hemoglobins In Genetics and Evolution (Columbia Oniv. Pren, New York, L961) 10. S. Ohno. EvKAuftH by Gtm DaplieatUrn (Spnusfr-VcrlBa. New Yorti. 1970). tl. M. C. King and A. C. W i h o n . J i W r IM, 107 (I97JJ 12. A. Hour and I. 0. Bour in Physiology and Genftrtt of PtntodKiipn. E. Ccutinho and F, Fuchi, Eds. (Plenum, New York. I975>. vnl. 4b, p. 117. 13. P. McLean. Psychoiom. Mrd 11,33H (1949). r
r
• P h y s i c s and Antiphysics
For a long lime it seemed as if in natural science the problem of language played only a secondary role. W. Heisenberg, i960 {2.1}
T h e r e are things that are difficult to recognize, not because they are too small for our eyes, but because they are too complex. I n coming to terms with a difficult mathematical treatise or a poem written in a concentrated style, a magnifying glass will not help. Studying the exact form of the letters does not contribute to an understanding of the text. E v e n if, at the next higher level, we understand the letters as phonemes, and study the statistical laws of their combinations in a certain language, we may l e a m something about speech mechanisms and the structure of the language, but we will learn nothing about the meaning of the text. A particular text cannot be explained from the rules that govern a language. This obvious fact has to be mentioned because it is in contrast with a habit of thought we have appropriated from physics, and which we sometimes use uncritically. Physicists tend to explain the variety of forms at the macroscopic level, i. e., at the level to which our observation has direct access, by descending to the next lower level where particles are discovered or postulated that escape macroscopic observation. If the postulated or observed interactions of the particles lead to theoretical constructs from which the macroscopically observed forms can be derived, then this is taken as the explanation of the macroscopic level. T h e shape of crystals is explained as a consequence of the geometry of the orderly network resulting from the interactions of neighboring atoms. I maintain that in studying brains, the strategy to adopt has more to do with the study of a text than with the analysis of a physical observation. Just as all of literature consists of strings of letters taken from an alphabet of about 30 signs, the same for all texts in a certain language, the hundreds of thousands of different animal brains (the number of
78
6
animal species that have a brain comes close to one million) consist of different combinations of very few cell types: neurons, glia cells, and some other elements concerned with the delivery and disposal of metabolic products (in blood vessels and tracheoles and in the membranes enveloping the nerve tissue). It may even be argued that nothing is lost if we describe brains only in terms of neurons. These "particles" are found with surprising uniformity in all brains. The uniformity is in their fine structure, the electrochemical properties of their cell membrane, the way they conduct signals and transmit them to other neurons. The variation is, rather, in the form and size of the fibrous cell processes with which they touch each other. The speciesspecific and individual blueprint of a brain is written in the language of the geometric relations of the neuronal cell processes. The particular tasks that one brain can accomplish, the peculiarities that distinguish the behavior of one animal from that of another, must be principally due to differences in the connective scheme and not in the characteristics of single cells. From this we may conclude that a specific brain cannot be explained from the general properties of neurons just as a particular text cannot be explained by starting with the transition probabilities of the letters in texts of a certain language. The transition probabilities in the case of the neurons correspond to the average distribution of their cell processes around the cell body; in the same analogy a particular word in the text corresponds to a particular scheme of the connections between the neurons. Even the question as to what types of brains are at all possible, and what brains are certainly impossible, can only be given a very vague answer on the basis of what we know about single neurons. Barring grotesque fantasies, we come closest to the truth by saying: everything thinkable is possible. Some neurons are found making connections over a distance equaling the diameter of the whole animal, while others are connected only to their nearest neighbors. A neuron can transmit its signals to just one other neuron or to many thousands. The signal transmitted from one neuron to another may be only a very small fraction of the excitation that causes the neuron to become active, or it alone may be enough. Above all, there seem to be all shades of possibilities ranging from an apparently purely statistical distribution of cell processes, not more regular than the distribution of branches on a tree, to very precise patterns of connections of neurons among each other. In other words, the language in which brains are written is a very free language. We will not be able, in most cases, to explain the peculiarities of a certain brain structure by invoking the
79
7
rules and constraints of the mechanisms that synthesize brains out of neurons, but will always have to consider explanations in terms of the function it performs. The distinction between crystals and texts, between objects that are explained at the level of the particles out of which they are composed, and those whose explanation lies in a context encompassing the objects themselves (as for example, the explanation of a statement from the social context between the speaker and listener) may seem artificial and unnecessary. It is certainly true that both kinds of explanations have something in common, namely, the extension of the observation, either in the microscopic or the macroscopic direction, until the enlarged field of view allows for the recognition of structures and regularities that were previously hidden. It is a question of taste whether or not we wish to see a profound philosophical polarity in the distinction between the two kinds of explanations or, in connection with this, between several kinds of causality. I would merely like to point out that the two kinds of explanations make different strategies necessary and create different general attitudes toward science. The discipline that is sometimes called cybernetics is the field in which both kinds of explanations are legitimately at home, causal explanations as well as explanations from the context. It is also the science of information carriers, or the science of complex things, which is another way of saying the same thing. Complexity, roughly, is measured by the number of elements of a situation that I have to mention in order to explain a macroscopic phenomenon. Here it is important to distinguish between the mention of a set of things and the mention of the individual things. A n example of the first case is if we explain the pressure of a gas by the movement of individual molecules in the container. Although we are talking about a large number of particles, we are considering them statistically. As far as our consideration goes, they are interchangeable; only the molecules in a collective sense are of interest, not the fate of single molecules. Although the gas represents a complicated situation and although the formulas of statistical mechanics that enable us to derive the pressure of the gas from the movement of molecules may look very complicated, I would like to reserve the word "complex" (as distinct from "complicated") for another kind of situation. A piece of skin from a stranger that is transplanted in place of my own releases reactions that result in a rejection of the transplant. Only my twin brother's skin can be grafted onto my body without difficulty. The difference can only be explained by detailed reference to the text written as an orderly sequence of submolecules within the thread-like
80
8 giant molecules. This is apparently variable from individual to individual. Such giant molecules are truly complex: the statistical description misses the point. I n order to account for the peculiar affinity between the substance of my twin brother and my own, I cannot avoid a detailed mention of a very large number of individual units. Perception is also complex. I recognize someone whom I last saw twenty years ago. T h e rhythm of his walk plays a part in this, the way his hair curls, the style of speech he uses in expressing himself, perhaps the odor of his tobacco or his clothes. T h e perception of this person is a complex affair and demands a complex brain: it is not a simple, one-dimensional measurement that is involved here, neither his weight, his size, the color of his hair, nor any such simple quantity, but a small region in the multidimensional space of my perception in which only he fits. A t the very root of the distinction between the objects of physics and those of biology we observe that the latter are essentially the result of (genetic or individual) memory processes. A physicist, or a biophystcist for that matter, will always try as best he can to isolate the things he is studying from the accidental environment in which he performs his experiment. H e is interested in those properties of matter that are repeatably the same whether he measures them here and now, or later somewhere else. O n the contrary, when we talk about the brain of the frog, about the behavior of white A m e r i c a n housewives, or about the markings on a butterfly's wings, we are dealing with things that are essentially historical, not invariant in time and space, and therefore not to be considered in isolation from their environment. L i v i n g matter is in a very fundamental sense always an image of the world. I hope these remarks will convince the reader that the antithesis mentioned in the title of this chapter does not reflect any antipathy toward the most beautiful of sciences, but rather an invitation to extend its canons and morals even to fields which, at first sight, seem to go in a direction opposite to that of traditional physics, such as the kind of interpretative neuroanatomy that I am proposing here.
References 2.1 Heisenberg, W.: Sprache und Wirklichkeit in der modernen Physik (1960). In: Schritte iiber Grenzen, gesammelte Reden und Aufsatze. Munchen: Piper & Co., 1971
81
© ]yy2 Springer-Verlag
Manifesto of Brain Science* Valentino Braitenberg Max-Planck-Institut fiir biologische Kybemelik, Spemannstrasse 38,7400 Tubingen, Germany
1. We believe that there are questions of s u c h p h i l o s o p h i c a l i m p o r t that they deserve being p u r s u e d without any consideration of u t i l i t y . O n e of them refers to the nature of thought processes, o r more g e n e r a l l y , of c o m p l e x behaviour. 2.
We
thought
are
convinced
that
ultimately
and behaviour will
be
given
a
satisfactory
in a language
explanation akin
to that
of of
p h y s i c s , i.e. i n m a t h e m a t i c a l terms. 3. T h e p h y s i c s of
l i v i n g organisms, w i t h
its s p e c i a l i z e d
b r a n c h e s of
p h y s i o l o g y and b i o c h e m i s t r y w i l l undoubtedly u n d e r l i e s u c h an e x p l a nation, but it w i l l not be s u f f i c i e n t by itself as a theory o f b e h a v i o u r . !n f a c t , w h e n boiled d o w n to p h y s i c a l f u n d a m e n t a l s , a mouse or e v e n a w o r m is not v e r y different f r o m m a n . B u t we a r e ultimately i n t e r ested i n thought and b e h a v i o u r of people, not of w o r m s .
4. T h e d i f f e r e n c e
in behaviour in different
species
reflect
different
w a y s o f c o p i n g w i t h the e n v i r o n m e n t , or w i t h distinct n i c h e s o f environment. These different
the
behaviours have their m a t e r i a l c o u n t e r -
part i n d i f f e r e n t
brains. T h e r e f o r e the p e c u l i a r nature of a n y a n i m a l
brain cannot
explained
explanation
be
necessarily
from
the
physical components
i n v o l v i n g causes
residing outside
alone,
the
the a n i m a l ,
i.e. i n f o r m a t i o n d e r i v e d f r o m the e n v i r o n m e n t .
5. C o n c e p t s s u c h as ( S h a n n o n i a n ) i n f o r m a t i o n ,
coding,
computation,
are outside of p h y s i c s ( e v e n i f the f o r m u l a s look l i k e p h y s i c s ) but a r e
* This was prepared for a meeting in Brussels where the European Commission considered the future of Neurobiology in Europe.
82
474
central to brain science. They are also fundamental concepts in electronics and computer science. This is the basis for a lively discourse between certain branches of engineering and brain science. Another common aspect is that both deal with functions directed to a goal, which physics does not. 6. Our object being the discovery of computer-like operations in brains, the hypotheses we test in our experiments are really inventions of computing schemes, and are therefore necessarily more complex than hypotheses in other branches of science. Thus we are inventors of computing devices much as our colleagues in Artificial Intelligence are and we can learn from them as much as they learn from us. Some ideas in brain science (e.g. neural networks) have already turned into eminently practical applications. 7. Computation in brains is a fine-grain operation, requiring a spatial resolution of about one to ten micrometers. The functional elements are fibers and neurons, and in most cases many neurons are involved in a significant functional pattern. This requires special techniques of observation which are being developed: multiple electrode recording, optical recording by means of voltage sensitive dyes. In most cases the vast amount of information produced by these recording techniques can only be stored and digested by means of large computers. Their theoretical analysis also requires sophisticated computing techniques, and often computer simulation. 8. We believe that the distribution of electrical signals observed at a spatial resolution of 1 pm and a temporal resolution of about 1 ms is all we need to know as the material counterpart of behaviour. Behaviour may be modified by hormones, or by pharmaca, or by pathological unbalance of some transmitter substance, but the actual effect of these is always expressed in terms of the occurrence or not occurrence of action potentials in neurons. A more fine-grain analysis, say at the molecular level, is relevant for the understanding of the individual cell, but probably unnecessary for the understanding of behaviour, or may even obscure the picture there, 9. Developmental neurobiology is a branch of general biology, not of brain science as defined here. "How do the fibers from the eye find
83
475
their way into the optic tectum?" The answer to this question will tell us much about how the arteries find their way into the kidney, or the roots of a tree into the earth, but very little about the problem of the. analysis of visual information in the optic tectum. 10. Efficient research groups in brain science are small. Everybody is groping for experimental techniques and for conceptual models. I f someone draws up a gigantic project in order to solve the problem by brute force, he is likely to have missed the complexity of the situation and is sure to produce much waste. To illustrate my points, I want to mention some successes of modern brain science in its interaction with information engineering. (a) The paper by McCulloch and Pitts (1943) proposed an equivalence between nets of neurons connected by synapses and propositions of Boolean logics. The terminology and the diagrams of this paper were immediately taken over by the designers of calculating machines (e.g. J . v. Neumann) and the basic ideas became the starting point for more than one theoretical development, e.g. Automata Theory and Chomskian linguistics (via the formalization proposed by Kleene, 1956). (b) Some ideas were exchanged repeatedly to and fro between brain science and engineering. The idea of the Perceptron (Rosenblatt 1962), a randomly connected image processor, was inspired by neuroanatomy and neurophysiology. It has admirable technological consequences even today (Sejnowski 1987). It received criticism on theoretical grounds by Minsky and Selfridge, who thought that a machine equipped with preset form detectors ("Pandemonium") would be much more efficient. The Pandemonium inspired neurophysiological work by Lettvin and others which turned into the seminal paper "What the frog's eye tells the frog's brain" (1959). Parallel to it, in the same intellectual environment, work on the cat (and monkey) visual cortex in the hands of Hubel and Wiesel produced splendid, Nobel prize winning results. The Hubel and Wiesel "feature detectors" in turn gave the engineers ideas, who produced several efficient image processing devices along the same theoretical lines.
84
476
(c) Filter theory, auto- and crosscorrelation, generalized Fourier analysis are theoretical concepts developed in electronics. Reichardt (1970) described the results of his (and Hassenstein's) experiments on optomotor reactions in insects in those terms. This was the first model of animal behaviour which made detailed predictions on the neurological wiring underlying the behaviour, and the predictions were in part confirmed anatomically (Braitenberg 1977). The technological fallout was a patent for a device designed to measure the direction and velocity of a moving panorama. (d) Associative memory was postulated by psychologists (Hebb) as a principle underlying cognition and behaviour. This idea turned into a variety of mathematical models, one of which, the neural net of the Hopfield kind, is hailed now as the cornerstone of a new age of computing technology. It was shown that the statistics of connections within the cerebral cortex (Braitenberg and Schiiz 1991) can best be explained as being specialized for associative memory. The global states of activity which the Hopfield theory postulates, a sort of resonant modes or "eigen-states" of the cortex, have been shown with computer analysis of multiple electrode recordings (e.g. Aertsen and Gerstein 1991; Vaadia and Aertsen, this volume).
References
Aertsen A , Gersiein G L (1991) Dynamic aspects of neuronal cooperativity: fast stimulus-locked modulations of effective connectivity. In: Kriiger J (ed) Neuronal Cooperativity, Springer Series in Synergetics, Springer, Berlin Heidelberg Braitenberg V (1977) On the texture of brains. Neuroanatomy for the cybernetically minded. Springer, Berlin Heidelberg New York Braitenberg V, Schiiz A (1991) Anatomy of the Cortex. Statistics and Geometry. Springer, Berlin Heidelberg New York Hebb D O (1949) The organization of behavior. Wiley, New York Kleene S C (1956) Representation of events in nerve nets and finite automata. In: Shannon C B , McCarthy J (eds) Automata Studies, Princeton University Press Lettvin J Y , Maturana H R , McCulloch WS, Pitts W H (1959) What the frog's eye lells the frog's brain. Proc Inst rad Engrs 47:1940-1951 McCulloch WS, Pitts W H (1943) A logical calculus of ideas immanent in nervous activity. Bull Math Biophys 5:115-133
85
Retchardt (1970) The insect eye as a model for analysis of uptake, transduction and processing of visual data in the nervous system. 34. Phystkertagung 1969, Salzburg, Plenarvortrage. B G Teubner, Stuttgart Rosenblatt F (1962) A comparison of several perceptron models. In: Yovits M C , Jacobi G T , Goldstein G D (eds) Self-organizing systems. Spartan Books, Washington Sejnowski T J , Rosenberg C R (1987) Parallel networks that learn to pronounce English text. Complex systems 1: 145-168 Vaadia E , Aertsen A (this volume) von Neumann, J (1956) Probabilistic logics and the synthesis of reliable organisms from unreliable components. In: Shannon C E , McCarthy J (eds) Automata Studies, Princeton University Press
86
lc Computer and Brain; Logic and Statistics
These three texts provide unusual views on mathematics, science and rationality (logic): these higher intellectual functions are examined here as products of the activity of human brain. In most of the bulk of this volume, the articles are directed toward somewhat lower brain functions, that are presently more easily amenable to neurobiological investigations. Nonetheless, the confrontation between the computer and the brain remains a permanent source of questioning (as already evoked by previous authors), and there are insightful analogies between the processes of perception and those of science itself ("man as a scientist"). It would be absurd to attempt to summarise the extract (10) from John von Neumann's posthumous book, The Computer and the Brain, because it is so dense, and also because the formulation itself of his ideas has much historical significance. Let us just signal two prophetic paragraphs that might otherwise pass unnoticed: one concerning the notion of logical depth, further mentioned in article (35) by Parisi, and another about the suggestion of a neural coding mechanism by cross-correlation of pulses, discussed later by Engel et al (article (55)). T h e whole field covered by this reprint volume, at the interfaces between physics, biology and computer science, is infused with probabilistic notions. Let us review briefly here four ways by which probability comes in. First, the probabilistic revolutions of Boltzmann and Darwin, in statistical physics and evolution theory. Second, information theory, giving a new meaning to the notion of entropy. T h i r d , the experimental observation that neural activity is noisy, and that the language of brain is statistical and not digital (as von Neumann puts it). Fourth, science and cognition (perception, reasoning) as inductive logic and statistical inference. This fourth level is evoked and illustrated in articles (11) and (12). The texts are polemical in style, but also somewhat complementary: article (11) with some brio, article (12) more
87
pedantic. No previous familarity with Bayesian theory is needed, because enough examples are presented in article (12). This last reference contains actually two extracts from the same book: its general introduction, and one piece from the chapter on thinking (inductive reasoning and rationality). The three items of this section are supposed to help the reader in becoming aware of several facets of the ancient question: "Are there universal laws of thought?", and of the probabilistic threads that run across many levels of brain study.
88
Reprinted with permission from The Computer and the Brain, pp. 74-82. 1958 © I95S Yale University Press
The Logical Structure of the Nervous System At this point, the discussion is best redirected toward another complex of questions. These are, as I pointed out previously, not connected with the problems of the memory or with the questions of complete and short codes just considered. They relate to the respective roles of logics and arithmetics in the functioning of any complicated automaton, and, specifically, of the nervous system.
IMPORTANCE
OF T H E
NUMERICAL
PROCEDURES
The point involved here, one of considerable importance, is this. Any artificial automaton that has been constructed for human use, and specifically for the control of complicated processes, normally possesses a purely logical part and an arithmetical part, i.e. a part in which arithmetical processes play no role, and one in which they are of importance. This is due to the fact that it is, with our habits of thought and of expressing thought, very difficult to express any truly complicated situation without having recourse to formulae and numbers. Thus an automaton which is to control problems of these types—constancy of temperature, or of certain pressures, or of chemical isostasy in the human body—will, i f a human designer has to for-
89
75
mulate its task, have that task defined in terms of numerical equalities or inequalities.
INTERACTION WITH
OF NUMERICAL
PROCEDURES
LOGIC
On the other hand, there may be portions of this task which can be formulated without reference to numerical relationships, i.e. in purely logical terms. Thus certain qualitative principles involving physiological response or nonresponse can be stated without recourse to numbers by merely stating qualitatively under what combinations of circumstances certain events are to take place and under what combinations they are not desired.
REASONS
FOR EXPECTING
HIGH
PRECISION
REQUIREMENTS
These remarks show that the nervous system, when viewed as an automaton, must definitely have an arithmetical as well as a logical part, and that the needs of arithmetics in it are just as important as those of logics. This means that we are again dealing with a computing machine in the proper sense and that a discussion in terms of the concepts familiar in computing machine theory is in order. In view of this, the following question immediately presents itself: when looking at the nervous system as at a computing machine, with what pre-
90
76
cision is the arithmetical part to be expected to function? This question is particularly crucial for the following reason: all experience with computing machines shows that if a computing machine has to handle as complicated arithmetical tasks as the nervous system obviously must, facilities for rather high levels of precision must be provided. The reason is that calculations are likely to be long, and in the course of long calculations not only do errors add up but also those committed early in the calculation are amplified by the latter parts of it; therefore, considerably higher precision is needed than the physical nature of the problem would by itself appear to require. Thus one would expect that the arithmetical part of the nervous system exists and, when viewed as a computing machine, must operate with considerable precision. In the familiar artificial computing machines and under the conditions of complexity here involved, ten- or twelve-decimal precision would not be an exaggeration. This conclusion was well worth working out just because of, rather than in spite of, its absolute implausibility. Nature of the System of Notations Employed: Not Digital but Statistical
As pointed out before, we know a certain amount about how the nervous system transmits numerical data. They are usually transmitted by periodic or
91
77
nearly periodic trains of pulses. An intensive stimulus on a receptor will cause the latter to respond each time soon after the limit of absolute refractoriness has been underpassed. A weaker stimulus will cause the receptor to respond also in a periodic or nearly periodic way, but with a somewhat lower frequency, since now not only the limit of absolute refractoriness but even a limit of a certain relative refractoriness will have to be underpassed before each next response becomes possible. Consequently, intensities of quantitative stimuli are rendered by periodic or nearly periodic pulse trains, the frequency always being a monotone function of the intensity of the stimulus. This is a sort of frequencymodulated system of signaling; intensities are translated into frequencies. This has been directly observed in the case of certain fibers of the optic nerve and also in nerves that transmit information relative to (important) pressures. It is noteworthy that the frequency in question is not directly equal to any intensity of stimulus, but rather that i t is a monotone function of the latter. This permits the introduction of all kinds of scale effects and expressions of precision in terms that are conveniently and favorably dependent on the scales that arise. It should be noted that the frequencies in question usually lie between 50 and 200 pulses per second. Clearly, under these conditions, precisions like the ones mentioned above (10 to 12 decimals!) are altogether out of question. The nervous system is a computing machine which manages to do its ex-
92
Nature of the System of Notations Employed
78
ceedingly complicated work on a rather low level of precision: according to the above, only precision levels of 2 to 3 decimals are possible. This fact must be emphasized again and again because no known computing machine can operate reliably and significantly on such a low precision level. Another thing should also be noted. The system described above leads not only to a low level of precision, but also to a rather high level of reliability. Indeed, clearly, i f in a digital system of notations a single pulse is missing, absolute perversion of meaning, i.e. nonsense, may result. Clearly, on the other hand, if in a scheme of the above-described type a single pulse is lost, or even several pulses are lost—or unnecessarily, mistakenly, inserted—the relevant frequency, i.e. the meaning of the message, is only inessentially distorted. Now, a question arises that has to be answered significantly: what essential inferences about the arithmetical and logical structure of the computing machine that the nervous system represents can be drawn from these apparently somewhat conflicting observations?
ARITHMETICAL
DETERIORATION.
ARITHMETICAL
AND LOGICAL
ROLES
OF
DEPTHS
To anyone who has studied the deterioration of precision in the course of a long calculation, the answer is clear. This deterioration is due, as pointed out before, to the accumulation of errors by superposition, and even more by the amplification of errors committed early in the calculation, by the
93
79
manipulations in the subsequent parts of the calculation; i.e. it is due to the considerable number of arithmetical operations that have to be performed in series, or in other words to the great arithmetical depth of the scheme.
The fact that there are many operations to be performed in series is, of course, just as* well a characteristic of the logical structure of the scheme as of its arithmetical structure. I t is, therefore, proper to say that all of these deterioration-of-precision phenomena are due to the great logical depth of the schemes one is dealing with here. ARITHMETICAL RELIABILITY,
PRECISION
OR L O G I C A L
ALTERNATIVES
It should also be noted that the message-system used in the nervous system, as described in the above, is of an essentially statistical character. In other words, what matters are not the precise positions of definite markers, digits, but the statistical characteristics of their occurrence, i.e. frequencies of periodic or nearly periodic pulse-trains, etc. Thus the nervous system appears to be using a radically different system of notation from the ones we are familiar with in ordinary arithmetics and mathematics: instead of the precise systems of markers where the position—and presence or absence—of every marker counts decisively in determining the meaning of the message, we have here a system of notations in which the meaning is conveyed by the statistical properties of the message. We have seen how this leads to a lower level of arithmetical precision hut to a higher level of
94
Nature of the System of Notations Employed
80
logical reliability: a deterioration in arithmetics has been traded for an improvement in logics. OTHER
STATISTICAL
MESSAGE
SYSTEM
TRAITS
THAT
OF T H E
COULD
BE
USED
This context now calls clearly for the asking of one more question. I n the above, the frequencies of certain periodic or nearly periodic pulse-trains carried the message, i.e. the information. These were distinctly statistical traits of the message. Are there any other statistical properties which could similarly contribute as vehicles in the transmission of information? So far, the only property of the message that was used to transmit information was its frequency in terms of pulses per second, it being understood that the message was a periodic or nearly periodic train of pulses. Clearly, other traits of the (statistical) message could also be used: indeed, the frequency referred to is a property of a single train of pulses whereas every one of the relevant nerves consists of a large number of fibers, each of which transmits numerous trains of pulses. It is, therefore, perfectly plausible that certain (statistical) relationships between such trains of pulses should also transmit information. I n this connection i t is natural to think of various correlation-coefficients, and the like. The Language of the Brain Not the Language of Mathematics
Pursuing this subject further gets us necessarily into questions of language. As pointed out, the nerv-
95
81
ous system is based on two types of communications: those which do not involve arithmetical formalisms, and those which do, i.e. communications of orders (logical ones) and communications of numbers (arithmetical ones). The former may be described as language proper, the latter as mathematics. It is only proper to realize that language is largely a historical accident. The basic human languages are traditionally transmitted to us in various forms, but their very multiplicity proves that there is nothing absolute and necessary about them. Just as languages like Greek or Sanskrit are historical facts and not absolute logical necessities, it is only reasonable to assume that logics and mathematics are similarly historical, accidental forms of expression. They may have essential variants, i.e. they may exist in other forms than the ones to which we are accustomed. Indeed, the nature of the central nervous system and of the message systems that it transmits indicate positively that this is so. We have now accumulated sufficient evidence to see that whatever language the central nervous system is using, it is characterized by less logical and arithmetical depth than what we are normally used to. The following is an obvious example of this: the retina of the human eye performs a considerable reorganization of the visual image as perceived by the eye. Now this reorganization is effected on the retina, or to be more precise, at the point of entry of the optic nerve by means of three successive synapses only, i.e. in terms of three consecutive logical steps. The statistical character of the message system used in
96
Question of Language
82
the arithmetics of the central nervous system and its low precision also indicate that the degeneration of precision, described earlier, cannot proceed very far in the message systems involved. Consequently, there exist here different logical structures from the ones we are ordinarily used to in logics and mathematics. They are, as pointed out before, characterized by less logical and arithmetical depth than we are used to under otherwise similar circumstances. Thus logics and mathematics in the central nervous system, when viewed as languages, must structurally be essentially different from those languages to which our common experience refers. It also ought to be noted that the language here involved may well correspond to a short code in the sense described earlier, rather than to a complete code: when we talk mathematics, we may be discussing a secondary language, built on the primary language truly used by the central nervous system. Thus the outward forms of our mathematics are not absolutely relevant from the point of view of evaluating what the mathematical or logical language truly used by the central nervous system is. However, the above remarks about reliability and logical and arithmetical depth prove that whatever the system is, it cannot fail to differ considerably from what we consciously and explicitly consider as mathematics.
97
Reprinted with permission from Maximum Entropy and Bayesian Methods, pp.107-116,1989 © 1989 Kluwer Academic Published
P r o b a b i l i t y , Philosophy and a b r i e f i n g f o r Bayesians
Science:
A.J.H. G a r r e t t Department of Physics S Astronomy U n i v e r s i t y of Glasgow GLASGOW G12 BCR Scotland, U. K.
Abstract The o b j e c t i v e Bayesian view i s considered i n r e l a t i o n to philosophy and philosophy of s c i e n c e . Camap's d i s t i n c t i o n between l o g i c a l and f a c t u a l p r o b a b i l i t i e s i s rejected, as i s an anti-Bayesian argument due to Popper. Reasons for the confusion are advanced. Inductive philosophy of science i s defended, and a t e n t a t i v e methodology proposed. E'our prominent a n t i - i n d u c t i v i s t s are analysed; Popper, Lakatos, Kuhn and Feyerabend. Popper i s shown to have been misled by the problem of improper p r i o r s , i n hypothesis space; h i s deductive doctrine of f a l s i f l a b i l i t y i s replaced by the inductive one of t e s t a b i l i t y . Kuhn's view that successive t h e o r i e s do not approach any k i n d of l i m i t i s c r i t i c i s e d , and i s traced to h i s r e j e c t i o n of induction. I t i s concluded that deductive methodologies of science are untenable, and that inductive methodology i s sound.
1.
Introduction
This paper d i v i d e s i n t o two p a r t s : a survey of how our p h i l o s o p h i c a l colleagues view p r o b a b i l i t y , and a c r i t i q u e of the prrminent d e d u c t i v i s t philosophies of science. S c i e n t i s t s concerned with probability often do not r e a l i s e that there has long e x i s t e d a p a r a l l e l e f f o r t i n philosophy. The two a r e r e l a t e d through the connection indeed i d e n t i t y - of p r o b a b i l i t y theory w i t h inductive l o g i c , a connection denied by one major school of thought. I n both parts we shall encounter a major f i g u r e of the 20th century! K a r l Popper (1902).
2. P h i l o s o p h e r s * views of
probability
I s h a l l present t h i s s e c t i o n from the objective Bayesian point of view. I s h a l l not attempt to defend i t here since that has been m a s t e r f u l l y done by others (Jaynes: c o l l e c t e d papers [ 1 ] ) . The objective Bayesian view i s that the p r o b a b i l i t y of an occurrence, conditional on information i n a given space, measures how l i k e l y one b e l i e v e s that occurrence to be, and t h a t the laws of p r o b a b i l i t y are laws of inference. This view i s objective to the extent that d e f i n i t e information corresponds to a d e f i n i t e probability. Anybody having the same information but assigning a d i f f e r e n t probability i s t h e r e f o r e g u i l t y of reasoning i n c o n s i s t e n t l y . I t I s anthropomorphic to the extent that different i n d i v i d u a l s often possess d i f f e r e n t information, and t h e r e f o r e (consistently) a s s i g n d i f f e r e n t p r o b a b i l i t i e s to the same event. Ttmt the o b j e c t i v e Bayesian view corresponds to the f a m i l i a r sum and product r u l e s ("Kolmogorov axioms") was
98
108 demonstrated by R. T. Cox [ 2 ] . A l t e r n a t i v e schemes may coincide w i t h the o b j e c t i v e Bayesian i n p a r t i c u l a r circumstances, for example the f r e q u e n t i s t view given an i n f i n i t e number of t r i a l s . These a l t e r n a t i v e s are then quite acceptable; but only the o b j e c t i v e Bayesian view works i r r e s p e c t i v e of context. (Have you ever a c t u a l l y seen an i n f i n i t e number of t r i a l s ? ) Philosophers do not s u f f e r as badly as some from F r e q u e n t i s t ' s Disease. The reason i s h i s t o r i c a l : philosophers have for c e n t u r i e s been concerned with that l o g i c to be used when there i s i n s u f f i c i e n t information for c e r t a i n t y , c a l l e d i n d u c t i v e l o g i c . They were d i s c u s s i n g the problem of i n d u c t i o n long before probability even became q u a n t i t a t i v e , and longer still before f r e q u e n t i s t s h i j a c k e d i t . Consequently philosophy never completely f e l l for t h i s a b e r r a t i o n . Of course, the frequentist view d i d influence philosophers, and i n f a c t Popper i n i t i a l l y advocated i t , i n Logik der Forschung ( t r a n s l a t e d as The Logic of S c i e n t i f i c Discovery [ 3 ] ) . He was concerned to oppose nono b j e c t i v i t y , but f a i l e d t o d i s t i n g u i s h properly between the o b j e c t i v e Bayesian view and the obviously c r a z y 'subjective' one that anyone may a s s i g n any p r o b a b i l i t y to anything. (True; but doing i t c o n s i s t e n t l y i s another matter.) Today, thanks t o Cox, we know that inductive l o g i c i s p r o b a b i l i t y theory, and v i c e v e r s a . I t I s a g e n e r a l i z a t i o n of the deductive l o g i c of c e r t a i n t y , Boolean algebra, from values 0 and 1 t o the i n t e r v a l i n between. This view i s inexorably gaining acceptance among p h y s i c i s t s . I n philosophy t h e r e i s a d i v e r s i t y of p o s i t i o n s ; one hugely i n f l u e n t i a l stance i s due to Rudolf Carnap (1891-1970). Carnap d i s t i n g u i s h e s two kinds of p r o b a b i l i t y , which he calls l o g i c a l and f a c t u a l [ 4 ] , TESTABLE INFORMATION HYPOTHESIS SPACE Symmetry principles
DATA
"PRIOR" PROBABILITY Maximum Entropy
•POSTERIOR" PROBABILITY]
t Baygs' Theorem
F i g u r e 1: Bayesian P r o b a b i l i t y Assignments Let us examine Carnap's c l a i m from an o b j e c t i v e Bayesian p e r s p e c t i v e . The Bayesian procedure f o r making inference i s s e t out i n Figure 1. We begin a t the l e v e l of the hypothesis space, { x } . The f i r s t stage i s to determine the measure m(x) on i t , using symmetry arguments such as i n v a r i a n c e under exchange of two elements, or transformation group theory [ 5 ) . Next, t e s t a b l e information i s incorporated. These are statements l i k e "the mean has value 2.T which can be checked, i n c o n t r a s t to data which are the outcome of measurements. ( I f the mean i s t o be estimated, a symbol stands for i t s value a t t h i s point.) The p r o b a b i l i t y ( d e n s i t y ) p(x) i s determined by maximising the information entropy /dx p ( x ) log (p(x)/m(x))
(1)
s u b j e c t to the c o n s t r a i n t s of normalisation and any t e s t a b l e information [ 1 ] . I f t h e r e i s none, p(x) « mix) without further ado and the entropy maximisation I s customarily omitted. F i n a l l y , data are brought i n to update the "prior" p r o b a b i l i t y u s i n g Bayes' theorem. Further data can be incorporated a t any time,
99
109 and the end r e s u l t i s independent of the order of i n c o r p o r a t i o n . This i s i n f a c t one of the consistency conditions from which Cox d e r i v e d the laws of p r o b a b i l i t y i n the Bayesian view [ 2 ] . The other i s that p r o b a b i l i t i e s s h a l l not depend on what has been marginalized out. Carnap's l o g i c a l p r o b a b i l i t y corresponds to the p r o b a b i l i t y before data are incorporated, for t h i s i s assigned using l o g i c a l arguments: syunetry and maximum entropy. But the r a t i o n a l e i s not given with anything l i k e the same c l a r i t y , needed to t a c k l e r e a l problems. "Factual" p r o b a b i l i t y corresponds to p r o b a b i l i t y assignments a f t e r data ( f a c t s ) a r e incorporated. Thus, although there i s a d i s t i n c t i o n i n the generation of these p r o b a b i l i t i e s , they both have f u l l y Bayesian interpretations as the only ones which can be consistently assigned fran the information a t hand, be i t data or t e s t a b l e . Philosophers often c i t e examples l i k e p(John i s left-handed | John i s a banker and 151 of bankers are left-handed) = 0.15
(2)
as l o g i c a l assignments of p r o b a b i l i t y . Whatever, t h i s i s an unfortunate example, because the information i s given i n the "handedness-space" of a l l bankers, not the required space of John's handedness. While Bayesians c e r t a i n l y emerge with the answer 0.15, the problem contains e x t r a complications. Philosophers who have c r i t i c i s e d Catnap's p o s i t i o n include W. V. 0. Quine (1908) and Popper. Indeed, e n t i r e books have been w r i t t e n on the PopperCamap controversy [ 6 ] . Popper has i n addition presented a t e c h n i c a l a n t i - B a y e s i a n argument [ 7 ] . Though erroneous, i t i s s t i l l doing damage today, and a r e f u t a t i o n i s i n order. The reader i s warned that philosophers o f t e n use a notation i n which the conditioning information follows a f t e r a conraa. Thus, what s c i e n t i s t s call p(AjB) i s w r i t t e n by Popper as p(A,B) - which unfortunately has the d i s t i n c t meaning to s c i e n t i s t s "the j o i n t p r o b a b i l i t y of A and B " . Here we use • s c i e n t i f i c " notation. Popper's argument, paraphrased, runs l i k e t h i s : suppose proposition B supports proposition A, given information I . This i s h e l d t o correspond to the inequality p(A|BI) > p ( A | l ) .
(3)
Since p(A|I) - p(AB|l) + p(A"ff|l), m a r g i n a l i z i n g over B (from the sum and product r u l e s ) , and since p(AB|I) can be decomposed using the product rule, f u r t h e r i n e q u a l i t i e s can be derived. Popper now supposes that the statement "B supports A to the degree z" corresponds to the Bayesian assignment P(A|BI) - z.
(4)
I t i s then easy to conjure up a c o n t r a d i c t i o n of the type " I am l i k e l y to drink tea. I an u n l i k e l y to drink coffee." (Both type (A).) "But, given a choice, I am more l i k e l y to drink coffee than tea." (Type ( 3 ) . ) Popper makes the point with dice. Popper concludes that the Bayesian view i s i n c o n s i s t e n t . But t h i s i s semantic confusion: the concept of support i s d i f f e r e n t i n (S) and ( 4 ) . Suppose
100
110
(3) i s taken as t h e d e f i n i t i o n o f w h a t i t means foe B t o support A; t h i s accords w e l l w i t h i n t u i t i o n . That done, t h e word means something else i n ( 4 ) , f o r no i n e q u a l i t y i s a t hand. L e t us i l l u m i n a t e t h i s by showing t h a t "supports" i n [4) may n o t coincide with intuition. Suppose t h a t A, B and I are such t h a t 0.99 - p ( A | B I ) < p ( A [ l ) .
(5)
There i s no problem i n a r r a n g i n g t h i s : l e t , f o r example A • " t h e r e w i l l be o t r a f f i c jam i n c e n t r a l London today", B = " t h e r e are no road works i n c e n t r a l London a t p r e s e n t " , and I = " i t i s a w o r k i n g day". Then B i s a n t a g o n i s t i c t o A; but a c c o r d i n g to Popper's q u a l i t a t i v e statement o f ( 5 ) , i t s u p p o r t s i t t o a degree o f 0 . 9 9 , i.e. v e r y s t r o n g l y . The l e s s o n i s t h a t assignment o f p r o b a b i l i t y i s d i s t i n c t from comparisons of p r o b a b i l i t i e s . Only p r o b a b i l i t i e s , not t h e i r differences (or d i f f e r e n c e s o f t h e i r l o g a r i t h m s ) s a t i s f y Cox's axioms. I n summary, I b e l i e v e t h e c o n f u s i o n p r e v a i l i n g over p r o b a b i l i t y i n p h i l o s o p h y i s due t o two f a c t o r s . F i r s t , the philosopher's d i s p o s i t i o n i s to ask "What i s p r o b a b i l i t y ? " , w h i l e t h e s c i e n t i s t seeks s o l u t i o n s to specific p h y s i c a l problems, asking i n s t e a d "How can p r o b a b i l i t y h e l p me?" The general i s always i l l u s t r a t e d by the s p e c i f i c . Second, i t c o u l d be t h a t p r o b a b i l i t y has had i t s day i n p h i l o s o p h y . Philosophy bore the t o r c h of W e s t e r n l e a r n i n g and e n q u i r y f o r c e n t u r i e s ; b u t , as more became known, s p e c i a l i s e d areas o f knowledge branched o f f from i t . Science itself i s the o u t s t a n d i n g example; u n t i l r e l a t i v e l y r e c e n t l y physics was known as N a t u r a l P h i l o s o p h y . But w i t h t h e u n d e r p i n n i n g o f Cox, r e c o g n i t i o n of the dominant r o l e o f the P r i n c i p l e o f Maximum E n t r o p y , and the beginnings (in quantum s t a t i s t i c a l mechanics) o f an o p e r a t o r - v a l u e d theory o f p r o b a b i l i t y , the day o f the amateur i n t h e b e s t sense, f o r p h i l o s o p h e r s are o f t e n eminent i n s e v e r a l branches o f t h e i r d i s c i p l i n e - may be a t an end.
3.
Modern p h i l o s o p h y
of
science
T h i s s e c t i o n c r i t i c a l l y s u r v e y s much 2 0 t h century p h i l o s o p h y ' s view o f s c i e n t i f i c methodology. For the b a s i s o f t h i s m a t e r i a l I am i n d e b t e d t o my former c o l l e a g u e a t the U n i v e r s i t y o f Sydney, David Stove, now r e t i r e d from its Department o f T r a d i t i o n a l and Modern P h i l o s o p h y . David holds t o t h e relevance of Carnap's d i s t i n c t i o n , b u t he i s a defender o f i n d u c t i o n , and a formidable c r i t i c of d e d u c t i v i s t philosophy o f science. Philosophy o f science i s b e s t d e s c r i b e d as t h a t which s c i e n t i f i c endeavours have i n common, b u t which n o n - s c i e n t i f i c s t u d i e s do n o t n e c e s s a r i l y share. We should not suppose, though, t h a t t h e r e i s a n y t h i n g magic about s c i e n c e : i t is s i m p l y a s u s t a i n e d a p p l i c a t i o n o f cannon sense. And since common sense i s consistent reasoning ( i n an a p p r o p r i a t e l y chosen space), we f i n d ourselves s t a r i n g s t r a i g h t a t Bayesian i n d u c t i v e p r o b a b i l i t y . Bayes n o t w i t h s t a n d i n g , modern p h i l o s o p h y o f science has grown i n t o a major p a t h o l o g y a s s o c i a t e d w i t h t h e names Popper, Lakatos, Kuhn and Feyerabend. D e s c r i b i n g i t i s my aim h e r e , and as a p r e l i m i n a r y arming I p r e s e n t a tentative f l o w c h a r t of how science i s done.
101
Ill
theoretical loop
theory
disfavoured
EVOLVE A THEORY
EXPERIMENT ATTESTING 1
theory favoured
experimental loop
REDUCE WO. OF UNSPECIFIED PARAMETERS
Figure 2:
A Broad Methodology For Science
I n i t s b a s i c form, the model i s given i n Figure 2. There i s no endpoint, and so no " f i n a l answer". Instead, one continually r e f i n e s theory and p r a c t i c e . The vexed debate over r e a l i s m i s circumvented by d e f i n i n g s c i e n t i f i c t r u t h a s the asymptote towards which t h i s process ( i n p r a c t i c e ) converges. Laws of Nature a r e unknown and never change (assuming, reasonably, that they e x i s t ) , but our approximations t o them improve as we l e a r n more. S i n c e the loops always contain a t l e a s t one inductive step, the whole process i s i n d u c t i v e . This i s no more than i t should be: one can be almost c e r t a i n t h a t the Sun w i l l continue to r i s e i n the e a s t , based on past observations (and the c e l e s t i a l mechanics constructed to e x p l a i n them), but c e r t a i n t y i s absent. The model can be fleshed out to varying degrees, and t h a t which I have found most i l l u s t r a t i v e i n p h y s i c a l science i s displayed i n F i g u r e 3. It is here t h a t philosophy can help s c i e n t i s t s , although they g e n e r a l l y pursue such a strategy i m p l i c i t l y . A new i n t u i t i v e leap can throw up a theory a t any time. The resulting flowchart i s welded to the old by applying the process to the union of the two t h e o r i e s . Unifying demonstrates d i r e c t l y t h a t science i s not dogmatically r e d u c t i o n i s t i c : reductionism i s simply a convenient way of implementing the s t r a t e g y of Figure 3. The i n t u i t i v e leap corresponds to a widening of the region of hypothesis space under consideration, a process f o r which there i s as yet no theory, even i n model problems. Meanwhile, the hotchpotch of guess, conjecture and imagination c a l l e d i n t u i t i o n i s p r e c i s e l y what d i s t i n g u i s h e s great s c i e n t i s t s from the r e s t . The i n d u c t i v e view of science goes a t l e a s t as f a r back as the 12th century scholar Roger Bacon, and thence forward to E l i z a b e t h I ' s c o u r t i e r F r a n c i s Bacon. (Cf course, William of Ockham's famous razor p r i n c i p l e " E s s e n t i a non sunt m u l t i p l i c a n d ^ praeter necessitatem" - e n t i t i e s should not be m u l t i p l i e d beyond n e c e s s i t y - i s e s s e n t i a l l y Bayesian.) The distinguished B r i t i s h e m p i r i c a l philosopher David Hume (1711-1776) argued against induction, and Stove t r a c e s today's movement to t h i s source [ 8 J .
102
112
INTUITIVE LEAP BASED ON PRIOR INFORMATION. OCKHAM-S RAZOR
FORMULATE HEM THEORY
REJECT THEORY"
INTERNALLY CONSISTENT? theory Strongly
disfavoured
FORMAL CHECK AGAINST P R I O R I N FORMATIOH inconclusive or strongly favoured
DEDUCE A TESTABLE CONSEQUENCE EXPERIMENTAL TESTING INFORMAL ANALYSIS OF RESULTS TIGHTEN PROTOCOL AND CONTINUE moderate *ti l _
strong
RECHECK. OTHER GROUNDS IN THEORY" S FAVOUR
theory -Strongly disfavoured
FORMAL STATISTICAL ANALYSIS OF RESULTS strongly inconclusive favouren
i
AFFIRM THEORY AS CURRENT MODEL
SEEK THEORY VTTH FEWER ARBITRARY PARAMETERS, SAME PREDICTIONS OF PAST OBSERVATIONS
AWAIT IMPROVEMENTS TO EXPERIMENTAL TECHNOLOGY
WITHOLE JUDGEMENT
SEEK FURTHER TESTABLE CONSEQUENCES
deductive process * w e l l - d e f i n e d inductive process *weakly d e f i n e d inductive process F i g u r e 3; A D e t a i l e d Methodology For P h y s i c a l Science More r e c e n t l y , Pierre Duhem (1861-1916) argued that theory and experiment never meet face-to-face, because i n r e a l science a p r o h i b i t i v e number of a u x i l i a r y assumptions are involved i n reaching the i n t e r f a c e [ 9 ] . Today t h i s i s c a l l e d the Quine-Duhem t h e s i s . On the inductive p i c t u r e , e x t r a assumptions a r e r e a d i l y incorporated by s e t t i n g up the p r i o r d i s t r i b u t i o n for t h e i r parameters, c a l c u l a t i n g the j o i n t p o s t e r i o r d i s t r i b u t i o n of these and the d e s i r e d q u a n t i t i e s from t h e data by using Bayes' theorem, and then marginalizing over the extra parameters to take them out.
103
Ill
K a r l Popper opened the modern era w i t h L o g i k der Forschung i n 1934. I t is f a r more about philosophy of science than p r o b a b i l i t y t h e o r y . C u r i o u s l y though, Popper i s regarded more by h i s p r o f e s s i o n , a t l e a s t o u t s i d e England, as a c e l e b r i t y than a p h i l o s o p h e r ' s p h i l o s o p h e r . Summarising him i s n o t easy: as w i t h most c u l t s , many meanings can be read i n t o i t . T h i s i s due t o such c o n t r a d i c t i o n s as h i s acceptance of p r o b a b i l i t y but r e j e c t i o n o f induction; David Stove, i n the f i r s t p a r t of Popper find A f t e r : Four Modern I r r a t i o n a l i s t s [ 8 ) exposes the devices by which Popper (among o t h e r s ! lays h i s smokescreen. Popper'" primary t e n e t s can nevertheless be d i s c e r n e d . One i s t h a t a l l observations are "theory-laden". I n Popper's own words, "sense-data. u n t h e o r e t i c a l items of o b s e r v a t i o n , simply do n o t e x i s t ' ' [ 1 0 ] . I t i s difficult f o r Bayesians t o express the d e p t h of t h e i r disagreement w i t h t h i s . Data are d a t a , be they a d i s t r a c t i o n ( n o i s e ) or t r a c k s i n bubble chambers photographed a t enormous i n g e n u i t y and c o s t . Whatever, they are i n c o r p o r a t e d i n t o theory u s i n g Bayes' theorem. Of course, t h e o r i e s suggest which data t o seek, b u t that i s n o t a t a l l the same t h i n g ; once found, data can be used t o update t h e p r o b a b i l i t y o f any h y p o t h e s i s whatsoever. Popper a l s o i n s i s t s t h a t science i s d e d u c t i v e r a t h e r than inductive. P a r t l y t h i s i s a t e r m i n o l o g i c a l d i s p a r i t y , r e f e r r i n g n o t t o t h e o v e r a l l process but t o a s i n g l e stage: d e d u c t i o n o f the consequences o f a h y p o t h e s i s p r i o r to testing. (Popper's scheme i s o f t e n d e s c r i b e d as h y p o t h e t i c o - d e d u c t i v e . ) But Popper does r e j e c t i n d u c t i o n ; we have seen already h i s r e j e c t i o n of the i n d u c t i v e v i e w of p r o b a b i l i t y i n favour o f o t h e r i n t e r p r e t a t i o n s . Indeed, Popper has a s s e r t e d t h a t no t h e o r y ever becomes more probable when evidence i n i t s favour i s discovered, and t h a t every s c i e n t i f i c theory n o t o n l y begins by being i n f i n i t e l y improbable, b u t always remains so [ 1 1 ] . The f i r s t of these statements d i r e c t l y denies Bayes' theorem. U n d e r l y i n g the second i s the i d e a , seldom r e c o g n i s e d , o f the space i n w h i c h p r o b a b i l i t i e s are d e f i n e d . This c o n t a i n s an i n f i n i t y of competing t h e o r i e s , and before l o o k i n g at t h e i r d i s t i n c t i v e f e a t u r e s we must a s s i g n each equal prior p r o b a b i l i t y 1/co, or zero. I t seems t h a t Popper i s c o r r e c t . But Bayesians recognise t h i s as the problem o f n o n - n o r m a l i s a b l e or improper p r i o r s , i n hypothesis space. The r e s o l u t i o n i s the same: though the p r i o r i s nonn o r m a l i s a b l e , Bayes' theorem g i v e s f o r the p o s t e r i o r r a t i o a w e l l - d e f i n e d limit o f 0/0 which may p e r f e c t l y w e l l be n o r m a l i s a b l e [ 5 ] , Bayesians can open The Logic o f S c i e n t i f i c Discovery, and the f i r s t volume o f i t s massive Postscript [12], expounding Popper's p o s t - f r e q u e n t i s t " p r o p e n s i t y " v i e w o f probability (much c l o s e r t o Bayesian, though never made p l a i n ) , almost at random and i l l u m i n a t e the problems exposed. The idea by which Popper i s best known, and one o f which most students are aware, i s the d o c t r i n e of f a l s i f i a b i l i t y . A hypothesis i s only s c i e n t i f i c i f i t i s capable o f being proved f a l s e by o b s e r v a t i o n . This i s an i m p o r t a n t idea, b u t i t i s b a l d l y d e d u c t i v i s t i n r e s t r i c t i n g the concept t o f a l s i t y b u t not truth; f o r i n d e d u c t i v e l o g i c a s i n g l e counter-example can f a l s i f y a t h e o r y b u t no number o f examples can prove i t . I n r e a l science though, t h e o r i e s are n o t proved f a l s e (or t r u e ) w i t h c e r t a i n t y . I n s t e a d , data are i n c o r p o r a t e d v i a Bayes' theorem i n t o the p o s t e r i o r p r o b a b i l i t y , which may approach zero or one. As i t gets s u f f i c i e n t l y c l o s e (a matter o f t a s t e ) , the t h e o r y i s r e j e c t e d o r adopted. So the c r i t e r i o n i s n o t f a l s i f i a b i l i t y , but t e s t a b i l i t y : t h a t one can conceive of data which a l t e r t h e p r o b a b i l i t y o f the h y p o t h e s i s . E q u i v a l e n t l y , the h y p o t h e s i s must not be e q u a l l y disposed t o every datum. Stove t r a c e s
104
114
Poppet's dictum of f a l s i f i a b i l i t y to h i s e a r l y d i s t a s t e , i n Vienna, with dogmatic claims that i d e a s such as Marx's and Freud's were " i r r e f u t a b l e " (11,13]. The word game began here, though Marx a t l e a s t understood the stakes long before. He wrote to Engels i n 1857, concerning a h i s t o r i c a l event: "One can always get out of [making an a s s of o n e s e l f ] with a l i t t l e d i a l e c t i c . 1 have, of course, so worded my p r o p o s i t i o n as to be r i g h t e i t h e r way" [ 1 4 ] . Popper does r e f e r to t e s t a b i l i t y although, deprived of i n d u c t i v e l o g i c , he fails to n a i l down the i d e a . (Any statement p e r t a i n i n g to a theory, but which i s not t e s t a b l e , i s part of t h a t theory's i n t e r p r e t a t i o n . ) He a l s o r e f e r s to degrees of f a l s i f i a b i l i t y , and attempts to r e l a t e them to p r o b a b i l i t y . This again i s word-play: the ease w i t h which a hypothesis i s tested i s a matter for t e c h n o l o g i s t s , not t h e o r e t i c i a n s . I t was the successor t o Popper's London C h a i r , Imre Lakatos (1922-1974), who pointed out a f r e s h t h a t i n r e a l s c i e n c e , t h e o r i e s are never disproved (or proved) with c e r t a i n t y . Lakatos c l e a r l y reached t h i s conclusion through c o n t r a s t i n g the n a t u r a l s c i e n c e s with mathematics, i n whose h i s t o r y he had worked [ 1 5 ] , But, f o l l o w i n g Popper i n renouncing induction, Lakatos was left w i t h no framework to hang h i s observation on. His own attempt to b u i l d one, a doctrine of r e s e a r c h programmes [ 1 6 ] , confuses philosophy with h i s t o r y of science. Like many ideas given b i r t h i n e a s t e r n Europe, deductivism has became popular i n America. L e t us t h e r e f o r e examine the work of Thomas Kuhn (1922¬ ) , another avowed a n t i - i n d u c t i v i s t and the author of the hugely influential work The S t r u c t u r e of S c i e n t i f i c Revolutions [ 1 7 ] . Like Lakatos, Kuhn i s a f i r s t - r a t e h i s t o r i a n of s c i e n c e , who has w r i t t e n on the Copernican revolution i n astronomy, and the black-body controversy which gave b i r t h to the e a r l i e s t quantum hypothesis. The S t r u c t u r e of S c i e n t i f i c Revolutions presents a c y c l i c view of how science evolves, beginning w i t h a mass of unordered observations and competing theories, going i n t o a quiescent stage a f t e r the triumph of one theory over the r e s t , followed by gradual breakdown i n t o chaos again under the accumulation of anomalies from more stringent t e s t i n g . Kuhn has bequeathed us one of today's f a s h i o n a b l e words, paradigm, to describe the model prevailing during the quiescent stage. The h i s t o r y of s c i e n c e abounds with examples of t h i s process; trouble again a r i s e s when i t i s combined w i t h a n t i - i n d u e t i v i s m as a philosophy. Close to the end of The S t r u c t u r e of S c i e n t i f i c Revolutions, Kuhn c l e a r l y echoes Popper's a s s e r t i o n that every theory i s i n f i n i t e l y improbable, when he says that "we may have to r e l i n q u i s h the notion that changes of paradigm carry s c i e n t i s t s . . . . c l o s e r and c l o s e r to the t r u t h " [ 1 8 ] . I n other words, Kuhn b e l i e v e s that t h e o r i e s come and go as a r b i t r a r i l y as fashions i n clothing. For a n t i - i n d u e t i v i s t s , the c l o s e r f i t to observation of r e l a t i v i s t i c mechanics than Newton's counts for nothing. T h i s s i n g u l a r ideology i s d e f l a t e d by applying i t to p r o g r e s s i v e l y simpler problems: i t can hardly be no more true that the Moon i s made of rock than green cheese. There i s no doubting, though, that Popper, Lakatos and Kuhn a l l appear r e s p e c t f u l t o science. Consider f i n a l l y the i d e a s of Paul Feyerabend (1924¬ ) [19,20]. Feyerabend d e s c r i b e s h i s approach as "epistemological anarchism", and h i s slogan i s "Anything Goes". Again, t h i s derives c l e a r l y from the idea that a l l t h e o r i e s a r e e q u a l l y i n v a l i d , though i t i s a l s o an accurate d i s t i l l a t i o n of the s u b j e c t i v e Bayesian stance. And Popper's o l d e s t c r i t i c i s m holds of i t : i t i s not f a l s i f i a b l e l
105
115 Feyerabend i s on r e c o r d as s t a t i n g t h a t normal science i s a f a i r y t a l e , and t h a t equal time should be g i v e n to " a s t r o l o g y , acupuncture and w i t c h c r a f t " [ 2 1 ) ( t h o u g h 1 do not know what unctions he seeks when i l l ) . He i s fond o f c a t e g o r i s i n g science w i t h " r e l i g i o n , p r o s t i t u t i o n and so on" [ 1 9 J . Feyerabend believes that science i s j u s t one of many i n t e r n a l l y c o n s i s t e n t views o f the world, and t h a t the consequent choice between them should be made on s o c i a l grounds. But w h i l e many systems are i n t e r n a l l y c o n s i s t e n t , o n l y one plugs c o n s i s t e n t l y i n t o the w o r l d of observations, and t o reason s y s t e m a t i c a l l y about t h a t w o r l d we must use i t : science. E t h i c a l and s o c i a l considerations may d i c t a t e w h i c h areas t o s t u d y , but t h a t i s a d i f f e r e n t m a t t e r . Feyerabend's ideas have been brought f o r t h by "the sleep o f reason", and w h i l e they c o u l d p r o b a b l y o n l y f l o u r i s h i n a society d i s i l l u s i o n e d w i t h s c i e n c e ( t h r o u g h i t s p e r c e i v e d misuse), they represent the l o g i c a l c u l m i n a t i o n o f the r e j e c t i o n o f induction. For t h a t i s what Popper, Lakatos, Kuhn and Feyerabend have i n coomon; and d e s p i t e much mutual r e p u d i a t i o n , i t f a r outweighs t h e i r d i f f e r e n c e s . The ideas of these f o u r comprise a major stream i n contemporary p h i l o s o p h y o f science. I t i s an odd f a c t t h a t s c i e n t i s t s o f t e n quote these p h i l o s o p h e r s f a v o u r a b l y [22]. Science magazine r e c e n t l y lauded Feyerabend's views as "a b r e a t h o f f r e s h air" [ 2 1 ] . The e x p l a n a t i o n i s undoubtedly a benign i g n o r a n c e . Being prepared t o r e v i s e hypotheses i n the l i g h t of fresh i n f o r m a t i o n ( a n a t t i t u d e p o l i t i c i a n s m i g h t heed) makes s c i e n t i s t s easy p r e y . What, by c o n t r a s t , could alter Feyerabend's o p i n i o n ? 4.
Conclusion
The o b j e c t i v e Bayesian v i e w i s as capable o f r e s o l v i n g problems concerning i n d u c t i v e l o g i c i n p h i l o s o p h y as i t i s i n science. Difficulties are not c o n c e p t u a l , but merely t e c h n i c a l : the huge spaces used i n r e a l problems, and t h e d e t e r m i n a t i o n of p r i o r p r o b a b i l i t i e s i n a wide v a r i e t y o f c o n t e x t s [1]. In particular, the Bayesian view, applied t o s c i e n t i f i c methodology, produces a coherent, i n d u c t i v e p h i l o s o p h y o f science. Non-inductive philosophies o f s c i e n c e i n v a r i a b l y lead t o a b s u r d i t i e s . References (1]
[2] [3] [4] [5] [6] [7] [8]
and
Notes
E . T . Jaynes. 1983. Papers on P r o b a b i l i t y , S t a t i s t i c s and S t a t i s t i c a l P h y s i c s , ed: R. D. Rosencrantz. Synthese s e r i e s v o l 158. R e i d e l (Dordrecht). R.T. Cox. 1946. Am. J . Phys. 14, 1 . K . R . Popper. 1959. The Logic o f S c i e n t i f i c D i s c o v e r y . Hutchinson (London). T r a n s l a t i o n o f : Logik der Fotschung ( S p r i n g e r , Vienna, 1 9 3 4 ) . R . Carnap. 1950. L o g i c a l Foundations o f P r o b a b i l i t y . U n i v e r s i t y of Chicago Press. E.T. Jaynes. 1968. IEEE Transactions i n Systems Science and C y b e r n e t i c s , SSC-4, p227. R e p r i n t e d as Chapter 7 of [ 1 ] . A . C . M i c h a l o s . 1971. The Popper-Carnap Controversy. M a r t i n u s N i j h o f f (The Hague). K . R . Popper. 1954. B r i t . J . P h i l o s . S c i . 5, 143. R e p r i n t e d i n r e f e r e n c e [ 3 ] , r e v i s e d e d i t i o n 1980. Appendix i x . D.C. Stove. 1982. Popper And A f t e r : Four Modem I r r a t i o n a l i s t s . Pergamon
106
116 (Oxford). P. Diihem. 1954. The Aim and S t r u c t u r e o f P h y s i c a l T h e o r y . Second e d i t i o n , P r i n c e t o n U n i v e r s i t y Press. ( T r a n s l a t i o n o f second French e d i t i o n . 1914.) [ 1 0 ] K . R . Popper. 1968. I n : Problems i n the P h i l o s o p h y o f Science, eds: I . Lakatos & A. Musgrave, p l 6 3 . N o r t h - H o l l a n d (Amsterdam). [ 1 1 ] see: D.C. Stove. June 1985. Encounter, p65-74. [ 1 2 ] K . R . Popper. 1982-3. P o s t s c r i p t t o The L o g i c o f S c i e n t i f i c D i s c o v e r y , Volume I : Realism and the Aim of Science ( 1 9 B 3 ) , Volume I I : The Open U n i v e r s e : An Argument f o r I n d e t e r m i n i s m ( 1 9 8 2 ) , Volume I I I : Quantum Theory and the Schism i n Phyics ( 1 9 8 2 ) . Ed: W.W. H a r t l e y 1 1 1 . Hutchinson ( L o n d o n ) . Mote the t i t l e o f volume I I I ; the schism occasioned by quantum theory i s between past and p r e s e n t , n o t i n physics as Popper a s s e r t s . [9]
[ 1 3 ] D.C. S t o v e . 1982. "How Popper's Philosophy Began'. P h i l o s o p h y . 57, 3 8 1 . The source i s an a u t o b i o g r a p h i c a l d e t a i l by Popper i n the summary paper 'Science: Conjectures and R e f u t a t i o n s ' , i n ; Conjectures and R e f u t a t i o n s , p u b l i s h e r : Routledge & Kegan Paul (London) 1963. [ 1 4 ] K. Marx. 1983. C o l l e c t e d Works, 40, 152. Lawrence & W i s h a r t (London). [ 1 5 ] I . L a k a t o s . 1963-4. V a r i o u s papers, c o l l e c t e d as; Proofs and R e f u t a t i o n s : The Logic o f Mathematical Discovery. Eds: J . W o r r a l l & E. Zaher. Cambridge U n i v e r s i t y Press, 1976. [ 1 6 ] I . L a k a t o s . 1978. P h i l o s o p h i c a l Papers, V o l I : The Methodology o f S c i e n t i f i c Research Progranmes. Eds: J . W o r r a l l & G. C u r r i e . Cambridge U n i v e r s i t y Press. [ 1 7 ] T . S . Kuhn. 1962. The S t r u c t u r e of S c i e n t i f i c R e v o l u t i o n s . U n i v e r s i t y of Chicago Press. (Second E d i t i o n , e n l a r g e d , 1970.) [ 1 8 ] Reference [ 1 7 ] , second e d i t i o n , p l 7 0 . [ 1 9 ] P.K. Feyerabend. 1975. A g a i n s t Method: O u t l i n e of an A n a r c h i s t i c Thee < of Knowledge. New L e f t Books (London). [20] P.K. Feyerabend. 1987. F a r e w e l l To Reason. Verso ( L o n d o n ) . [ 2 1 ] P.K. Feyerabend. 1979. Quoted i n : Science, 206, 534. [ 2 2 ] T. Theocharis k M. Psimopoulos. 19B7. N a t u r e . 329, 595, and 3 3 1 , 384 (1988).
107
Reprinted with permission from Cognition as Intuitive Statistics, pp. 147-162. 1987 © 1987 Lawrence Erlbaum Associates. Inc.
Two Revolutions — Cognitive and Probabilistic; Is the Mind a Bayesian? IS T H E M I N D A B A Y E S I A N ? After the inference revolution, a new question arose: A r e the long soughtafter laws of thought the laws of probability theory '' A n d , to answer the new question, a new kind of problem was posed. These problems were constructed so that they couid be answered by calculating probabilities, means, variances, or correlations. Restructuring was seldom necessary for the solution of such problems, and so this aspect of the theory of thought was held in abeyance. Consequently, the new vocabulary for understanding human reasoning was the vocabulary of the statistician; the new elements of thinking were numbers (probabilities), and the process of thinking itself was explained by statistical operations such as calculating likelihood ratios. T h e theoretical questions asked by experimenters, the problems posed to the subjects, and the explanations sought all reflected the fascination with probability theory and statistics. The new vocabulary made the study of inductive thinking into one of the fastest growing areas of the new psychology of thinking, revitalized by the cognitive revolution. O u r concentration here is on the link between inductive thinking and probability theory to the exclusion of other issues such as deductive thinking (e.g. J o h n s o n - L a i r d , 1983; Wason & JohnsonL a i r d , 1972) and computer models of problem solving (e.g. Dorner, 1983; Newell & Simon, 1972; Simon, 1979). In this section we deal with the question whether the mind is a Bayesian statistician.
Conservatism U r n s and balls have long been the stock-in-trade of the probabilists. F o r instance, when L a p l a c e (1774/1878— 1912> proved B a y e s ' theorem, he used the well-known urn filled with white and black balls as his illustration. In the 1960s, the urn-and-balls problems made their way into the laboratories of experimental psychologists, although often in a more contemporary terminology, as bookbag-and-poker chips problems (e.g. E d w a r d s , 1966; E d w a r d s . Lindman & Phillips. 1965). Consider for instance a typical problem posed to the subjects; Imagine yourself in the following experiment. Two urns are filled with a large number of poker chips. The first urn contains 70% red chips and 30% blue. The second contains 70% blue chips and 30% red. The experimenter flips a fair coin to select one of the two urns, so the prior probability for
108
148 each urn is .50. He then draws a succession of chips from the selected urn. Suppose that the sample contains eight red and four blue chips. What is your revised probability that the selected urn is the predominantly red one? (Peterson & Beach, 1967, p. 32).
If your answer is around .75, you agree with most of the subjects. C o m p a r e the new bookbag-and-poker chips problems with those of the Wurzburg school and Gestalt psychology. In Biihler's and Duncker's problems, calculation was seldom a useful tool: and if it w a s , it was not sufficient to find a solution. N o w the answer asked for is a number, and calculation rather than restructuring is sufficient to go from the problem to the answer. H o w is that calculation done? B a y e s ' theorem gives an answer. B a y e s ' theorem is an elementary consequence of the definition of conditional probability given a mutually exclusive and exhaustive set of hypotheses. In the foregoing problem, there are two hypotheses: / / , that the selected urn is the predominantly red one; and H that it is the predominantly blue one. T h e answer asked for is the posterior probability p(H,\D) that H, is true given the data D, that is. eight red and four blue chips. A c c o r d i n g to B a y e s ' theorem, the posterior probability is: 2
p{H,\D)
= p(HMD\H,)lp{D)
(5.1)
with p(D)
= p{HMD\H,)
+
pWMWHJ.
A s described in chapter 1, p ( / / , ) and p ( / / j ) are the prior probabilities of the hypotheses H and / / , , respectively, and p(D\H,) and p(D\H ) are called the likelihoods of the data D if H, and / / , , respectively, are true. T h u s , B a y e s ' theorem gives a rule for revising a prior probability (or, base rate, as it is often called) into a posterior probability after new data has been observed. T h e posterior probability will be greater than the prior, if the ratio p(D\H )lp{D) exceeds 1, which is the case if the data has a high probability if the hypothesis /V, is true. F o r the present problem, the likelihood of getting x = 8 red chips in a sample of n = 12 if / / , holds is given by t
2
t
p(D\H,)
= ("WO-P,)"-'
(5.2)
In words, p," is the probability of drawing a sequence of x red chips (p, is the percentage of red chips in the urn H,), and ( I - p , ) " ' is the probability of drawing a sequence of (n-x) blue chips. Therefore, the product of - pi)""* is the probability of a sequence of x red and ( n - j r ) blue chips. T h e number (;) denotes the number of different orderings in which x red and (n-x) blue chips can occur. T h u s , we can calculate the likelihoods: n
109
149 p(D\H ) = (tf ).7» x .3* = .231 p(D\H ) = (',' ).3 x .7 = .008. x
g
4
2
Inserting the likelihoods and the prior probabilities into B a y e s ' theorem, we calculate the posterior probability: p(//,IZ)) -
.5 x .231/(.5 x .231 + .5 x .008) = .967
T h u s , after having observed eight red chips in a sample of twelve, we should revise the prior probability from .50 to around .97. Compared with this, the average subject was more conservative, revising from .50 to only .75. T h i s "cautious" revision of the prior probability was called conservatism and was the main finding of that early research. Incidentally, the calculations for this simple bookbag-and-poker chip problem illustrate the complexity of the mental calculations that the reasoning mind is supposed to perform. B y the late 1960s, it was concluded that conservatism was a persistent phenomenon, although some variables, such as the sequential order of the data and the introduction of incentives, influenced the degree of conservatism (Peterson & B e a c h , 1967; Peterson & D u C h a r m e , 1967; Phillips & E d w a r d s , 1966). In E d w a r d s ' (1968) words, "It takes anywhere from two to five observations to do one observation's worth of work in inducing a subject to change his opinions" (p. 18). T h e question whether the mind reasons like a B a y e s i a n seemed to have found a consistent answer: T h e mind is a quasi-Bayesian, that is, a very conservative one. T h e mind mistrusts new data and gives greater weight lo the prior probabilities.
How to Explain
Conservatism?
Peterson and Beach (1967) proposed that the mind systematically miscalculates equation (5.2), but not (5.1). That is, the mind first miscalculates the likelihoods, but then correctly uses Bayes" theorem, although with the wrong likelihoods. E d w a r d s (1968) proposed Ihe opposite explanation— that the mind correctly calculates equation (5.2) but miscalculates (5.1). In other words, the likelihoods are correct, but the mind misaggregates them with the prior probabilities instead of combining them according to B a y e s ' theorem. T h e r e were other interesting attempts to explain the phenomenon, such as the idea that the mind considers as relevant data the ratio of red to blue chips rather than their difference. Bayes' theorem, however, implies that only the difference counts: eight red and four blue chips (a difference of four) in a sample of twelve give the same result as five red and one blue chips in a sample of six (see Manz, 1970). T h e important come from rather T h e psychological bility theory and.
point here is to see where the explanatory concepts than to speculate which explanation might be correct. explanations now come from the vocabulary of probain particular, from B a y e s ' theorem, like the research
n o
150 question and the problems posed. First, the facts that require explanation are the deviations from B a y e s i a n reasoning: H a d the subjects followed B a y e s ' theorem, no explanation would have been necessary. B a y e s ' theorem has become the frame of reference. S e c o n d , although human reasoning does deviate from B a y e s i a n reasoning, the explanation is still sought in the vocabulary of B a y e s ' theorem, such as •"miscalculating" likelihoods or •'misaggregation." L i k e textbook problems in probability theory, these explanations ignore the content and context of the specific problem: it is the mathematical structure that counts. It is ironic that the whole phenomenon of conservatism disappeared when in the early 1970s Daniel K a h n e m a n and A m o s T v e r s k y posed Bayesian problems with a content different from bookbags and poker chips. Subjects no longer seemed to reason conservatively about the new problems: indeed, they even seemed to neglect the prior probabilities.
B a s e Rate N e g l e c t : T h e K a h n e m a n a n d T v e r s k y Program There is an important difference between the kind of problems posed by E d w a r d s , on one hand, and K a h n e m a n and T v e r s k y , on the other. E d w a r d s ' problems were pure application of probability theory, whereas K a h n e m a n and T v e r s k y ' s problems, while still of the textbook sort, approximated real-life situations. T h e latter thus became simultaneously more interesting and, as we shall see, more ambiguous in their interpretation. K a h n e m a n and T v e r s k y (1973) started with the unquestioned belief that B a y e s ' theorem would give a single correct answer to the problems they investigated. T h e i r problems were typically presented in questionnaires and distributed in classrooms and other group settings; subjects were typically high school students and college undergraduates. One of the best known problems is the E n g i n e e r - L a w y e r Problem. Kahneman and T v e r s k y (1973) told the following cover story to a group of students: A panel of psychologists have interviewed and administered personality tests to 30 engineers and 70 lawyers, all successful in their respective fields. On the basis of this information, thumbnail descriptions of the 30 engineers and 70 lawyers have been written. You will find on your forms five descriptions, chosen at random from the 100 available descriptions. For each description, please indicate your probability that the person described is an engineer, on a scale from 0 to 100. The same task has been performed by a panel of experts, who were highly accurate in assigning probabilities to the various descriptions. You will be paid a bonus to the extent that your estimates come close to those of (he expert panel, (p. 241)
111
151 A second group of students received the same instruction with inverted base rates (prior probabilities), namely, 70 engineers and 30 l a w y e r s . A l l subjects
were given
the same
personality descriptions.
O n e o f these
thumbnail descriptions follows: Jack is a 45-year-old man. He is married and has four children. He is generally conservative, careful, and ambitious. He shows no interest in political and social issues and spends most of his free time on his many hobbies which include home carpentry, sailing, and mathematical puzzles. The probability that Jack is one of the 30 engineers in the sample of 100 is —
%. (p. 241)
Although
the
likelihoods—p(descriptionl E n g i n e e r )
and
p(descrip-
t i o n l L a w y e r ) — a r e not specified here, the experimental situation is set up in such a way that it is still possible for K a h n e m a n and T v e r s k y to use B a y e s ' theorem to compute the posterior probabilities: T h e y calculate the ratios of the odds in both groups, so that the likelihoods cancel
out.
1
B a y e s ' theorem predicts that the posterior probabilities should be different for the two groups. K a h n e m a n and T v e r s k y found, however, that the mean responses in the two groups were for the most part the same and 1
concluded that base rates were largely ignored.' T h e explanation w a s that
'The likelihoods are cancelled oui in ihe following way; The "prior odds" p(Engineer)/ p(Lawyer) are 30/70 in the first and 70/30 in the second group. The likelihood ratios p(descriptionlEnEineer)/p(descrip(iontLawyer) are assumed to be the same in bolh groups. If we use the symbol O, and O ; for the prior odds in the two groups, respeclively, and L for the likelihood ratio, then ihe ratio of the posterior odds Q, and ( I . in the two groups is given as follows:
This means that the posterior odds p( E ngi nee rldescription)/p( L a wye ride scrip! ion) in the first group should be only \ t% of those in the second, orequivalenlly. the odds in the second should be more lhan five limes as high as in Ihe first (see Kahneman & Tversky. 1973). •'In f a d . however. Kahneman and Tversky (1983) found a statistically signifkan! difference between the two groups (p < .01). Nevertheless, (hey subsequently ignored this and concluded lhal the manipulation of ihe information about base rates had only a "minimal effect" and therefore lhat base rales were "largely ignored." Of course, they were quite right to forgel aboul their null hypothesis lest, since null hypothesis testing in this situation, wilh two competing prediclions. is useless. Bolh Bayes' theorem and the "neglect of base rales" hypothesis specify prediclions, and i( is nonsensical to identify one wilh a null hypolhesis. In f a d . Kahneman and Tversky came lo their conclusion by eyeballing (Ihe results lie much closer lo ihe prediclions of ihe "base rate neglect" hypothesis than to those of Bayes' theorem): alternatively one might have adopted a symmetric hypothesis testing procedure such as Neyman & Pearsons'. This example, like that of cognitive algebra in chapter 3. shows that null hypolhesis testing not only may be useless, but may actually be misleading. In contrast to Anderson, Kahneman & Tversky realized this, bul it is still a question why they performed the null hypothesis testing ritual in the first place.
112
152 the subjects had arrived at the probability judgments by judging only the similarity between the description and the stereotype of an engineer, or, in order words, the degree to which the description is representative of the stereotype. T h i s strategy was called the representativeness heuristic. T h e five personality descriptions included the following nondiagnostic s k e t c h , which is particularly interesting because it was constructed to be totally uninformative with respect to the question whether the person is an engineer or a lawyer: " D i c k is a 30-year-old man. He is married with no children. A man of high ability and high motivation, he promises to be quite successful in his field. He is well liked by his colleagues." ( K a h n e man & T v e r s k y , I973, p. 242). T h e median probability judgments for Dick being an engineer were 50% in each group, and the authors concluded that the subjects had ignored base rates even when the description was totally uninformative. T h e representativeness heuristic again reveals the dream of universal laws of thought—if not Bayes' theorem, then at least such universal heuristics as representativeness. T h i s search for a few simple laws plays d o w n the importance of the task for structuring thought. F o r instance, G i n o s a r and Trope (1980) repeated the E n g i n e e r - L a w y e r study, but gave only one of the five personality descriptions (including the nondiagnostic sketch) to each subject. T h e y concluded that in this situation the nondiagnostic sketch was assessed by the base rate. It seems that one particular feature of K a h n e m a n and T v e r s k y s task, namely the presentation of diagnostic descriptions before the nondiagnostic sketch, directed the reasoning about the nondiagnostic sketch. A second feature of K a h n e man and T v e r s k y ' s task was that no subject was given different base rates. But FischhorT, S l o v i c , and Lichtenstein (I979) report that base rates have more impact if they are varied in the problem posed to a subject. T h e general structure of the task, and not only some general "law of thought," appears to determine performance.
Represen
tativeness
What is the explanation for the new phenomenon, the so-called neglect of base rates? K a h n e m a n and T v e r s k y {1973, T v e r s k y & K a h n e m a n , 1982a) offer two major explanations, the representativeness heuristic and causal versus incidental base rates. T h e first explanation is derived from the E n g i n e e r - L a w y e r Problem and similar problems. K a h n e m a n and T v e r s k y claim that their subjects arrived at their answer by a heuristic, or short-hand, strategy rather than by B a y e s ' theorem. This heuristic is called representativeness and is identified with the similarity between the description and the stereotype of an engineer. In their early writings, they
113
153 seemed to imply that the base rate fallacy necessarily results from the use of the representativeness heuristic. But soon it became clear that other problems, such as the C a b Problem (see below), could not be dealt with by a representativeness heuristic. T h u s , representativeness might be only one condition for the neglect of base rates. A s Kahneman and T v e r s k y (1972) put it, '"Representativeness, like perceptual similarity, is easier to assess than to characterize. In both cases, no general definition is available, yet there are many situations where people agree which of two stimuli is more similar to a standard" (p. 431). The term representativeness was used in the early writings with more than one meaning, as the authors themselves later realized (Tversky & K a h n e m a n , 1982b). We shall therefore confine our discussion to the use of the term representativeness in the context of Bayesian-type problems of probability revision such as the E n g i n e e r - L a w y e r Problem. K a h n e m a n and T v e r s k y (1973) distinguish between formal rules like B a y e s ' , which specify how we should determine probabilities, and nonformal heuristics like "representativeness," which describe how we actually determine probabilities. B a y e s ' theorem is correct and is about probability; heuristics are mostly misleading and not influenced by probability. But as we shall show, this distinction is illusory. In fact, the representativeness heuristic boils down to computing probabilities using only likelihoods, without prior probabilities. It is therefore just as formal and potentially quantitative as Bayes' theorem itself, though not normative. We submit that this formal nature of a purportedly informal, qualitative heuristic shows the extent to which probability theory has permeated contemporary theory construction in this area of psychology. In the following section we show that the concept of a representativeness heuristic can be reduced to the concept of likelihood in B a y e s ' theory.
The Skeleton
of a
Heuristic
H o w can we show that representativeness is a redescription of base rate neglect rather than an explanation? It seems to us that in most, if not all, cases, representativeness or similarity is synonymous with likelihood. If this is true, the explanation will turn out to be merely a redescription of the phenomenon in Bayesian terms. What basis do we have for this claim? L e t us consult T v e r s k y and Kahneman (1982b), who dissect their original use of the term representativeness into two different senses; judgments by representativeness, the heuristic used for inference and prediction; and judgments of representativeness, the finding that people judge and expect samples to be highly representative of their parent
114
154 populations. It is the first meaning we are concerned with. T h e y differentiate four basic cases of judgment by representativeness. F o r each, they c h a r a c t e r i z e representativeness as a directional relation between a model or population and an instance o r sample. T h i s means that a sample is more or less representative of a particular population, but not vice versa. This general characterization is in accordance with that of a likelihood, which is a directional relation, p(D\H) between a sample D and a population or hypothesis H. Although K a h n e m a n and T v e r s k y believe that their subj e c t s use representativeness as an alternative to probabilistic thinking, they themselves characterize at least the first three kinds of judgments by representativeness in terms of distributions (Tversky & K a h n e m a n , 1982b, p.87). L e t us look at the four basic kinds in which the concept of representativeness is invoked. In the first c a s e , representativeness is defined as a relation between a class H and the value D of a variable defined in this class. T h e given example speaks of (more or less) representative values of the income of college professors. T h e authors slate that representativeness is determined here mainly by what the subject knows about the frequency distribution, and that a value D will be most representative if it is close to the m e a n . In this first c a s e , representativeness is identical with the likelihood of an observed value D given a subjective distribution H, that is of a likelihood p(D\H), where H is a one-dimensional distribution and D is a single observation. In the second and third cases, representativeness is defined as a relation between a class and an element or subset, respectively. Since K a h n e m a n and T v e r s k y see the second case as a special case of (he third, we shall deal only with the latter. To use one of the authors' examples, students of astronomy are less representative of the enlire student body than are students of psychology. K a h n e m a n and T v e r s k y contrast this to Ihe first case by pointing out that we now deal with more than one attribute in the population and with more than one element in the sample. T h u s , the first can be regarded as Ihe unidimensional version of the third, as the authors themselves state. Therefore, representativeness is for case 3 identical with the notion of a likelihood p(D\H). where D is a sample rather a single event and H is an unknown, subjective multidimensional distribution rather than a unidimensional one. In the last case, representativeness is defined as a relation between a causal system and a possible consequence. Here, it is no longer a class but a system that may produce a consequence D. F o r example, let H be pneumonia and D be high fever, which is frequently associated with pneumonia. T h i s case seems to be equivalent to the unidimensional case 1, except that the subjective distribution of effects which specifies the likelihoods is now interpreted as " c a u s e d " by a system H. T h i s specific
115
155 interpretation does not, however, change the formal identity of case 4 with case 1 and can therefore be subsumed under case 1 To summarize, representativeness seems to be in all cases reducible to likelihood. T h u s , to say that the subject uses a representativeness heuristic for probability revision seems to be equivalent to saying that the subject uses the likelihood in B a y e s ' theorem, but not the prior probability. O u r conclusion, therefore, is that Bayes' theorem provides the vocabulary for the explanation of the phenomenon, as it also did for the phenomenon itself. T h i s clarifies our point that the explanation offered is but a redescription of the phenomenon. The phenomenon that is called "neglect of base rates" could just as well be called "neglect of base rates and nonneglect of likelihoods." T h e explanation, judgment by representativeness, can be now rephrased as "use of likelihoods and neglect of base rates." The explanation is a redescription of the phenomenon. Although Bayes' theorem has been dismissed by Kahneman and T v e r s k y as a fundamental law of human thinking, the new candidate, representativeness, has inherited major attributes, since it stems from the same framework, as we have shown. T h e s e attributes are typical of formal and calculational approaches to human thinking: it is held that the representativeness heuristic operates independently of the content and the context of the problem; the heuristic has nothing to say about the process of information search; and it emphasizes rationality rather than passion.
Independence
of Human Thinking from Content.
T h e assumption is
that human thinking should be directed by the formal structure of the problem, not by the content. Along with their conviction that B a y e s ' theorem should be used in all problems that seem to have a Bayesian structure, independent of the actual content, Kahneman and T v e r s k y (1972) have the parallel conviction that people use a representativeness heuristic for all contents: "Although our experimental examples were confined to well-defined sampling processes (where objective probability is readily computable), we conjecture that the same heuristic plays an important role in the evaluation of uncertainty in essentially unique situations where no 'correct' answer is available" (p. 451). E x a m p l e s that follow are the judgment of the probability that a particular 12-year-old boy will become a scientist, that a company will go out of business, and that a politician will be elected for office. However, in the recent past it has become more and more evident that if the structure of the problem
'The problem with Tversky and Kahneman's (1982b) distinction into four kinds of judgments by representativeness is that their own formal structure (p. 871 does not make sense in Ihe light of all the examples they give. Here we have concentrated on those examples which fit into their own formal structure.
116
156 posed is held constant, but the content is changed, responses differ. That content is crucial for the laws of human reasoning has been found in problems concerning both inductive and deductive thinking (e.g., see E i n h o r n , 1980; E v a n s , 1984; Griggs, 1983; Pollard, 1982; Wason, 1983). F o r instance, in their research on deductive thinking and the understanding of negations, Wason and J o h n s o n - L a i r d (1972) also started with the assumption that the content they invented to illustrate a structure, say a syllogism or a double negation, would not affect the operation of the laws of thought; " F o r some considerable time we cherished the illusion that this was the way to proceed, and that only the structural characteristics of the problem mattered. Only gradually did we realize first that there was no existing formal calculus which correctly modelled our subjects' inferences" (pp. 244f.). K a h n e m a n and T v e r s k y (1982) also made a retreat, realizing that content is crucial and that, consequently, human reasoning cannot be described by content-independent formal rules. However, what has not been clarified is that judgments by representativeness—because they are in essence judgments by likelihood—are as content independent as formal B a y e s i a n reasoning. Independence of Human Thinking from Context. To claim that human thinking is directed by a general-purpose heuristic such as representativeness implies that the context of the problem is not very important for a theory of thinking. F o r instance, the wording of the problem and the particular example given are contextual variables. A s we pointed out earlier, the proponents of the Wiirzburg and the Gestalt school concluded from their experiments that these contextual variables give an impulse or a direction to the flow of thought. Recently, the role of these '"irrelevant" variables has again come to be appreciated (e.g. Berkeley & Humphreys, 1982; C r o c k e r , 1981; E i n h o r n , 1980). F o r instance, consider the additional information in the E n g i n e e r - L a w y e r problem that a panel of experts were "highly a c c u r a t e " in the same task and that "you will be paid a bonus to the extent that your estimates come close to those of the expert panel." It seems that K a h n e m a n and T v e r s k y added this paragraph to increase the motivation of their students to fill out their questionnaires more carefully. In fact, the impact may be far greater. T h e subjects may understand from the success of the experts that the personality descriptions are highly informative if only one knows how to read them. T h u s they might conclude that there is only one strategy to win the bonus, namely, to concentrate on the description and to forget about the base rates. Whether this explanation of the base rate neglect in terms of contextual variables rather than general heuristics is true or not can be easily c h e c k e d by repeating the experiment with different information. T h e general point, however, is that the representativeness heuristic, like B a y e s ' theorem, seduces one into believing that thinking is directed by a
117
157 few universal laws of thought. But it is becoming increasingly evident that thinking, like perception (see e.g. Birnbaum, 1982), is directed by the specific context in w h i c h a problem is presented. The Blind Spot: Information Search. Statistical hypothesis testing does not start until the variables and numbers needed for the formulas are available. It is not concerned with the preceding measurement process, with the questions, what is the relevant information we shall look for? and how shall w e measure the variables which we consider relevant? L i k e statistical theories, the representativeness heuristic does not deal with the process of information search. We have shown that it stands in the framework of statistical problems, where thinking means mental work on prepackaged information. When the subject enters the laboratory, most if not all of the process o f information search has already been done by the experimenter. In contrast, the older meaning of heuristic tool in Duncker's theory w a s confined to search for new information and new functional relationships. Heuristic tools were "looking around," "inspection," and "selection," and the outcome of heuristic processes were "restructurings" or "judgments of relevance," as recently reemphasized by E v a n s (1984). Passionless Irrationality. T h e belief that Bayes' theorem is the manifestation of rationality has its mirror image in the link between representativeness and irrationality. Since the representativeness heuristic was assumed to be used in almost every content and context, it seemed to provide a cognitive explanation for many a human error, even those usually explained by people's emotions and passions. F r o m base rate fallacy to self-serving attributions to ethnocentric prejudices—such phenomena were now understood as the result of a relatively passionless mind, handicapped by its ignorance of the laws of probability (e.g. Nisbett & R o s s , 1980). T h e parsimonious heuristic user is seen as "governed by a consistent misperception of the world rather than by opportunistic wishful thinking. G i v e n some editorial prodding, he may be willing to regard his statistical intuitions with proper suspicion and replace impression formation by computation whenever possible" ( T v e r s k y & K a h n e m a n , 1971,p. 110).
Causal
versus Incidental
Base
Rates
K a h n e m a n and T v e r s k y posed a second type of problem to their subjects, also of a B a y e s i a n structure, but one in which the likelihood could not be easily interpreted as similarity. Therefore, the supposed base rate neglect could not be explained by representativeness. One of the best known examples is the C a b Problem. Since the publication of the first results around 1972, different versions have appeared; the present one is from T v e r s k y and K a h n e m a n (1980).
118
158 A cab was involved in a hit-and-run accident at night. Two cab companies, the Green and the Blue, operate in the city. You are given the following data: (i) 85% of the cabs in the city are Green and 15% are Blue. (ii) A witness identified the cab as a Blue cab. The court tested his ability to identify cabs under the appropriate visibility conditions. When presented with a sample of cabs (half of which were Blue and half of which were Green) the witness made correct identifications in 80% of the cases and erred in 20% of the cases. Question: What is the probability that the cab involved in the accident was Blue rather than Green?" (p. 162) T h i s problem has the same formal Bayesian structure as the urn-andpokerchip problems, only the content has changed. T h e predominantly blue urn became the Blue cab company, and the chips drawn became the witness report. In contrast to the pokerchip problem, the likelihoods are already numerically given (.80 and .20). F r o m B a y e s ' theorem, T v e r s k y and K a h n e m a n calculated a probability p(B\"B") = .41 that the cab was Blue (B) given the witness report " B l u e " ( " # " ) . If the subjects were " c o n s e r v a t i v e " B a y e s i a n s , they should answer about .25-.30, in between the prior probability (base rate) for Blue (.15) and the Bayesian solution. T v e r s k y and K a h n e m a n (1980), however, report that several hundred subjects gave a modal and median response of .80. Since this median answer coincides with the credibility of the witness (the likelihood p{"B"\B), they conclude that their subjects ignored the relevant base rates. T h e same conclusion was drawn in several studies (see T v e r s k y & K a h n e m a n , 1982a), for undergraduates as well as for professionals. F o r example, C a s s c e l l s , Schoenberger and G r a y b o y s (1978. cited in T v e r s k y & K a h n e m a n , 1982a) posed the following problem lo 60 students and staff at H a r v a r d Medical School: " I f a test lo detect a disease whose prevalence is '/moo has a false positive rate of 5%, whal is the chance that a person found to have a positive result actually has the disease, assuming you know nothing about the person's symptoms or signs?" (p. 154). If one assumes that the test correctly diagnoses this disease in everyone who has it, one c a n calculate the posterior probability by Bayes' theorem, which is 2%. O f the 60 students and staff members, 11 gave this answer, whereas the average answer (mean) was 56% and the most c o m m o n answer (mode) was 95%!. T h i s range amply illustrates the large interindividual differences often found in such studies. T h e conclusion drawn was that "even highly educated respondents often fail to appreciate the significance of outcome base rate in relatively simple formal problems" ( T v e r s k y & K a h n e m a n , 1982a, p. 154). What is the explanation of (his phenomenon? First, it is necessary to
119
159 be clear about what the phenomenon itself is. T h e phenomenon observed is the mean, median, or modal probability judgment. T h e phenomenon is not the neglect of base rates. L i k e "conservativism," the latter is already an interpretation of the observed judgments in terms of B a y e s ' theorem. Nevertheless, T v e r s k y and K a h n e m a n (e.g. 1980, pp. 61ff) talk about "the phenomenon of base-rate neglect." T h i s phrasing is quite interesting, suggesting that B a y e s ' theorem has become such a ubiquitous background to theorizing that the observed fact is Immediately interpreted in terms of B a y e s ' theorem. T h i s interpretation now appears as the fact itself; any observed median probability can be immediately located on a scale with the base rate and the likelihood on the extremes, where the observation appears as either neglect of base rates or overweighting base rates (conservatism). T h u s , the fascination with probability theory has already penetrated the facts observed.
The
"Explanation"Analyzed.
What is the explanation for the phenomenon called the neglect of base rates? Since the reliability of the witness can be hardly called a similarity, K a h n e m a n and T v e r s k y cannot explain this supposed neglect of base rates by a representativeness heuristic. Instead they offer a second explanation: "We propose that the phenomenon of base-rate neglect largely depends on whether or not the evidence is given a causal interpretation" (Tversky & K a h n e m a n , 1980). What does this mean? F o r illustration, the authors give the following variation of the C a b Problem, where information (i) in the previous C a b Problem is replaced by the following information: (i'> "Although the two companies are roughly equal in size, 85% of cab accidents in the city involve Green cabs, and 15% involve Blue c a b s " (p. 63). T h e difference concerns the question, base rate of what? In the original instructions, base rates of Blue cabs in the city were specified, whereas in the new instruction base rates of Blue cabs that were involved in accidents were also specified. With this new base rate, the median answer of subjects was .60, as compared with .80 previously. Intraindividual variability again was large. Since this median answer lay in between the reliability of the witness (.80) and B a y e s ' solution (.41), the authors concluded that the base rate was no longer ignored. T h e explanation given is that there are two different types of base rates, causal and incidental. T h e "city" base rate in (i) is called incidental because the greater base rate of G r e e n cabs in the city "does not justify a causal inference that makes any particular G r e e n cab more likely to be involved in an accident than any particular Blue c a b . " T h e "accident" base rate in (i') is "causal because the difference in rates of accidents
120
160 between companies of equal size readily elicits the inference that the drivers of the G r e e n cabs are more reckless and/or less competent than the drivers of the Blue cabs." ( T v e r s k y & K a h n e m a n I982a, pp. I57-158) But is there also a causal link between a driver's competence and the criminality of fleeing the scene of the accident? A lack of a causal link would imply that the "accident" base rate is also "incidental." One subject's incidental rate might be another's causal rate. We shall return to this question shortly. What does it mean to explain the neglect of base rates by saying that the base rate was incidental? A n urn analogy may be helpful here. Imagine there are Blue and G r e e n urns, each filled with a mixture of witness reports " B " o r " G . " There are different possible experiments, two of which are called "urns in the c i t y " and "urns in accidents." In each experiment, first one urn is randomly drawn from all urns, and then a witness report is randomly drawn from that urn. The experiments differ only in that different populations of G r e e n and Blue urns are used. Imagine you are the subject in the accident experiment. Your task is to guess the color of the urn from which the witness report has been drawn. We tell you that a report B was drawn, and we tell you the proportion of reports B and G in the Blue and G r e e n urns, respectively. We do not give you the base rates of Blue and G r e e n urns in the accident experiment, but, by mistake, we inform you about the base rates in the city experiment. T h e latter do not refer to your experiment. This is exactly the meaning of incidental base rates, and you may be well advised to ignore such base rates. We now repeat the accident experiment with another subject, correct our mistake, and give the subject information about the appropriate accident base rate. T h i s is the situation to which K a h n e m a n and T v e r s k y refer as causal base rates. T h u s , it becomes clear that the distinction between causal and incidental base rales boils down to whether the relevant base rates for one experiment or another are given. This simple analysis of the nature of the new explanation has an interesting consequence: T h e phenomenon disappears, and the subjects may even be rehabilitated as good B a y e s i a n s . The Phenomenon Disappears. T v e r s k y and Kahneman (1980) still believe that a posterior probability of .41 is "the correct answer" to the original version of the C a b Problem, where the city base rates were given. H o w e v e r , as we have argued, they did not specify the relevant base rate "cabs involved in hit-and-run accidents," but only those of a different "experiment." In this situation, a Bayesian might either use the base rates of the city experiment as the unknown prior probabilities and hope that both happen to be equal, or, for lack of better knowledge, fall back on the principle of indifference and assume that as many Green drivers as Blue
121
161 drivers may commit hit-and-run crimes. In the latter case, a B a y e s i a n would calculate a posterior probability of .80, w h i c h , surprisingly, is identical to the median answer given by the subjects. T h e conclusion from this would be that the subjects in fact behaved like Bayesians who were forced to apply the principle of indifference. T h e phenomenon therewith disappears. T h e point is, however, that there is no way to decide about the correct answer. L e t us turn to the revised C a b Problem, where the base rates of cab accidents were given. K a h n e m a n and T v e r s k y now insert the accident base rate instead of, as before, the city base rate into the formula and again calculate .41 as the correct answer (since the numbers of two base rates have been simply exchanged). Note again that the posterior probability asked for is concerned with the hypothesis that a Blue cab was in a hit-and-run situation, not in just any accident or in the city. The relevant hit-and-run prior probability is again unspecified. Why do the authors estimate the missing prior probability using the base rales of accidents rather than of cabs in the city, or by applying Ihe principle of indifference? T h e r e is no discussion of this choice. A s we pointed out earlier this choice means inferring lack of honesty from lack of competence. This may be true, or it may not be. F o r instance, drivers who constantly have accidents may be so well adapted as not to panic and drive off, whereas the driver who never had an accident may be nervous and excited and prone to speed away. In these cases, it would be unreasonable to insert the accident base rates into the formula. But we do not know. The point is not the plausibility of such alternative causal accounts, but rather the choice of which base rate to use must be defended—otherwise we risk mixing up experiments, as in the urn analogy. Consequently, there is no unique way to calculate the posterior probability in this experiment, since the prior probability is unspecified. Kahneman and T v e r s k y ' s calculation of .41 is only one alternative. Therefore, the phenomenon disappears again in this second study, since one cannot conclude that subjects neglect base rates or that they neglect them to a lesser degree, if the base rate itself is not specified. Finally, if we assume that the subjects should apply the principle of indifference, then the median judgment of .60 even appears as an instance of conservatism. T h e general lesson to be learned from this is twofold. First, the explanation of the neglect of base rates by incidental versus causa! base rates can be reduced to an elementary rephrasing of Bayes' theorem: To revise a prior probability into a posterior probability and to assume that this has a single correct answer implies that the relevant base rate is specified. I f not, as in "incidental" base rates, it is nonsensical to talk about a neglect of the base rate. Second, the more Ihe urns in the example take on a specific content, the more ambiguity arises about which base
122
162 rate is the relevant one and therefore the more doubtful it is that there exists a single correct B a y e s i a n answer. We shall return to this fundamental issue in the next section. To summarize, this new psychology of thinking is saturated with probability theory; B a y e s ' theorem gives the framework for the question posed, the problems constructed and even the vocabulary for the phenomenon "observed." Prior probabilities and likelihoods are the new elements of human reasoning. F r o m this perspective, if the mind is not a B a y e s i a n , there are only two possible alternatives; the neglect of base rates, o r the neglect of the likelihoods. T h e former is the result presented by K a h n e m a n and T v e r s k y ; the latter has been concluded from the earlier bookbag-and-poker chips studies. In fact. K a h n e m a n and T v e r s k y ' s explanations, although they appear informal, can also be reduced to these two new elements of the mind. To explain Ihe neglect of base rates by saying that the subject uses a representativeness heuristic is in essence saying that the subject uses the likelihood in B a y e s ' formula, but not the base rate. T h e r e f o r e , the supposed explanation is simply a redescription of the supposed phenomenon. T h e second explanation, causal versus incidental base rates, also boils down lo B a y e s ' formula: To explain the neglect of base rales by saying that the base rales specified in the problem were incidental rather than causal is in essence saying thai the relevant prior probabilities were not specified in the first place. We have repeatedly pointed out how the descriptive and explanatory conclusions hinge on B a y e s ' formula as ihe normative background. We have also repeatedly shown how this entire perspective depends on the assumption that Bayes" theorem is normative. Not only the explanations, but the descriptions of the phenomena themselves are colored by this assumption, as we have seen. H o w e v e r , it is only by neglecting content, context, and information search that this normative assumption is made tenable. In s u m . B a y e s ' theorem has become the framework for reasoning about reasoning.
123
Id Some Perceptual Facts and Issues
T h e two texts of this section axe so well conceived and clearly written that they stand by themselves as good reading. Furthermore, they provide here an appropriate conclusion to the introductory chapter, and a smooth transition to the next. Both texts address fundamental issues concerning the nature of the human mind that have long puzzled philosophers, and both bring neatly argued and convincingly simple — even though still incomplete — answers to these major questions. In article ( 1 3 ) , a chapter from his posthumous book, Mind from Matter? (a title that might well have been chosen as a subtitle for this anthology), Max Delbriick presents the remarkable phenomena of colour, position and size constancy in human perception. He shows that our perceptual apparatus constructs a useful model of the external world, so that primary sense impressions do not have access to consciousness. He stresses an essential distinction between phylogenetic and ontogenetic learning, and ends with a briUiant conclusion about science and perception. Article (14) is a pictorial, highly informative, introduction to brain structures and functions. Blakemore argues about the functional significance of the spatial arrangements of activity in the brain. Why have ordered maps? The existence of isomorphic and non-isomorphic maps is well documented and used as an explanatory clue. Many experimental facts are described; thus this review may serve as a precious outline for a large number of papers in the subsequent chapters. T h e conclusion deals, in a sober and sound fashion, with recurrent issues concerning the relations between science and language. During his progression through this volume, the reader will often find useful a return to the clear introductions provided by these two articles.
124
Reprinted with permission from Mind From Matter, pp.109-119. 1986 © 1986 Blackwell Scientific Publications. Inc.
Perception
One
might naively imagine that v i s u a l perception a m o u n t s to the con-
scious m i n d looking at the image on the retina. T h i s is obviously not so; between retina a n d consciousness there are m a n y processing steps that progressively boii d o w n the information p r o v i d e d by the pattern of neuronal excitation i n the retina. We will n o w m o v e to yet more abstract levels of the perception process, w h i c h illustrate some general principles of adaptive brain evolution. T h e s e levels reflect the capability of the h u m a n cerebral cortex to filter a n d process the v i s u a l input to give "objective," observer-independent information about a n object being observed. T h i s capability was not w o r k e d out by neurophysiologists with their electrodes but "psyched out" by perceptual psychologists. It is manifest in n u m e r o u s perceptual constancy p h e n o m e n a , of w h i c h w e will discuss three examples. I n each of these p h e n o m e n a the perceptual apparatus of the cortex extracts objectified information from the v i s u a l input.
I . T h e first example is the constancy of the perceived color of an object, irrespective of the color of the i l l u m i n a t i n g light. We perceive that a n object h a s the same color, w h e t h e r we see it i n the bluish light of m o r n ing,
i n the r e d d i s h light of e v e n i n g , or i n the y e l l o w light of a fire or of
an i n c a n d e s c e n t l a m p . ( O n e of the rare situations i n w h i c h the perceptual apparatus for color vision is grossly fooled is that of a scene illuminated by a monochromatic light source, s u c h as the y e l l o w s o d i u m lamps u s e d for lighting streets a n d h i g h w a y s . M o n o c h r o m a t i c light is, of course, highly unnatural and was not present i n the environment while the h u m a n
125
110
perceptual apparatus evolved.) T h u s , u n d e r natural conditions, the real color of a n object is perceived regardless of the color of the illuminating light. T h i s is possible because what w e actually perceive as the color of an object is its property of absorption a n d reflection of the r e d , green a n d blue components of polychromatic light, in relation to other objects in the scene. T h i s abstraction of the color of an object from its relative absorbance a n d reflectance of the spectral components of the illuminating light is performed preconsciously, by the intuitive use of the concepts of "white" (meaning light i n w h i c h all spectral colors are equally represented) a n d "complementary color" ( m e a n i n g a member of the color pairs r e d and green, or blue and yellow, w h o s e mixture is perceived as white light). In 1925, E w a l d H e r i n g p r o p o s e d that to assess the color of various objects in a scene, the perceptual apparatus s u r v e y s the whole field of vision a n d defines one object as w h i t e , that is, as reflecting equally all colors of the visible spectrum. T h e light reflected from all other objects is then interpreted relative to the spectral composition of the light reflected by the white object. To make that interpretation, the perceptual apparatus can be thought of as a d d i n g p h a n t o m light of a color complementary to that of the illuminating light so that the object defined as white is actually perceived as white. For instance, if the illuminating light is predominantly red, the perceptual apparatus adds phantom green light to it, w h i c h makes a white object look white rather than red. T h e addition of phantom light of complementary color to the actual light source then provides not only for the perception of the white object as white but also for the reasonably accurate perception of the real colors of ail other objects.
2. T h e second example of a perceptual constancy p h e n o m e n o n is the invariance of the perceived position of an object d u r i n g voluntary h e a d or eye movements. A s a p e r s o n shifts the gaze or turns the h e a d , the image of the objects i n the v i s u a l s u r r o u n d moves on the retina. T h i s m o v e m e n t is not, however, perceived as a morion of the objects; they are correctly perceived as being stationary. T h i s perceptual compensation for the motion of the image is so completely automatic that it is not even consciously registered as a motion of the h e a d or the eyes. It is not registered because the change in position of the image of the objects on the retina is filtered out by the perceptual apparatus. A simple experiment reveals h o w this filtering process w o r k s . If you close one eye and jiggle the other with your fingers, stationary objects are perceived as jiggling. Since in this experiment your eye is m o v e d passively (rather than actively by contraction of your head or eye muscles), the result
126
121 s h o w s that the nurmaily perceived constancy of spatial position during a voluntary m o v e m e n t is the result of y o u r m o v e m e n t being taken into account by your perceptual apparatus. A n o t h e r experiment reveals that what is taken into account here is not the actual occurrence of the voluntary m o v e m e n t itself, but the
command
to the muscles to perform it.
In this experiment the muscle that w o u l d carry out the m o v e m e n t is temporarily paralyzed by injection of a d r u g . U n d e r these conditions, w h e r e the intention to move the eye but not its motion can occur, comm a n d of a n eye m o v e m e n t causes the stationary image of a n object to be incorrectly perceived as a m o v e m e n t of the object. H e r e the perceptual apparatus proceeds as if the eye had been m o v e d voluntarily and adjusts the fixed position of the image to compensate for the intended movement. To inform the perceptual a p p a r a t u s that a c o m m a n d for a h e a d or eye m o v e m e n t has been i s s u e d , a duplicate of the n e r v e impulse pattern directed from the brain to the motor n e u r o n s that c o m m a n d contraction of the appropriate muscles is s i m u l t a n e o u s l y sent to the relevant cerebral neurons. It is this "efference copy" of the c o m m a n d that allows for compensation of the m o v e m e n t of the retinal image. T h e idea of the role of such a n efference copy in p r o v i d i n g for the brain a quantitative expectation of the change i n sensory input resulting from the animal's o w n m o v e m e n t s was first proposed by E r i c h v o n Hoist in the early 1950s. T h e r e is n o w
neurophysiological e v i d e n c e
that eye m o v e m e n t
com-
m a n d s do influence neurons i n the v i s u a l cortex i n a m a n n e r consistent with v o n Hoist's efference copy proposal.
3. T h e third example of a perceptual constancy p h e n o m e n o n is the invariance of the perceived size of a n object regardless of the distance at w h i c h it is seen. W h e n an object is m o v e d t o w a r d the eyes, the size of its retinal image increases, yet the object is not incorrectly perceived as increasing i n size. T h e perceptual a p p a r a t u s accomplishes this c o m p e n sation by evaluating distance information a c c o r d i n g to the principles of perspective. A variety of clues are available to the perceptual apparatus regarding the distance of the object. For instance as the object approaches, the curvature of the eye lens increases (by action of the ciliary muscles) to keep the retinal image of the object i n focus, a n d if both eyes are to stay trained on the a p p r o a c h i n g object, their optical axes must increasingly converge (by action of the t w o sets of extraocular muscles). Either of these clues can be p r o v i d e d by a n "efference copy" of the c o m m a n d to the eye muscles. But if both eyes r e m a i n trained on a stationary distant object, the images cast on their retinas by the a p p r o a c h i n g object will
127
112
Command to the Eye Muscles to Move the Eveball Message to the Brain
Efference Copy
Eve Muscles
>
Message from the Retina about Movement
Block diagram illustrating von Hoist's hypothesis that the cerebral command for an eye movement is added, in the form of an efference copy, to the eye's report of perceived morion. According to this hypothesis, this operation determines whether the perceived motion is interpreted as a result of the percipient's own movement, or of the movement of the perceived object or visual surround. [After Hassenstein, 1971]
become increasingly disparate (parallax). T h i s third clue can be p r o v i d e d by n e r v e cells in the visual cortex that are especially dedicated to computing the binocular disparity of retinal images, to subserve the stereoscopic depth perception made possible b y the evolution of frontal eyes. O n the basis of these clues about the closeness of approach, the perceptual apparatus adjusts the perceived size of the retinal image to y i e l d the correct view that an object of constant size is coming closer. It is to be noted that all the processes mentioned here as being r e s p o n sible for perceptual constancy p h e n o m e n a involve preconscious operations. Hence it might be said that sensations as s u c h do not have access to consciousness. T h e processes by w h i c h percepts are abstracted from the sensory input cannot be introspected by the percipient. T h i s point is often overlooked w h e n physicists discuss the nature of reality, since
128
113
they tend to equate sensation i n the s e n s o r y organs w i t h what is presented to the consciousness. T h e conscious m i n d h a s no access to raw data; it obtains only a highly processed portion of the input. F r o m the evolutionary viewpoint, s u c h processing is e n o r m o u s l y adaptive, since it a l l o w s the m i n d to cope w i t h the real w o r l d . F o r instance, D o n a l d M c K a y suggests that voluntary m o v e m e n t s of the eyes are really preconscious questions about the w o r l d : if u p o n m o v i n g the eyes the images of objects move, the perceptual apparatus infers that the objects are i n reality stationary. A l l these processes s e r v e to abstract information free of the vagaries of the sensory organs a n d thus allow us to construct an objective w o r l d from our sensations. T h e abstraction of constancy of object color, position, a n d size from the retinal image represents only a very low level of the entire process of perception. T h e next higher level is that of constancy of abstraction of form. That is, w h e n an object is seen u n d e r c h a n g i n g conditions, its perceived form remains the s a m e e v e n though entirely different sets of sensory receptors are stimulated at different times. T h e variability of the object's aspects simply does not reach the consciousness. What is abstracted is the form of the object, irrespective of the particular part of the sensory system that provided the information about the object. T h i s capacity for abstraction of form from diverse bits of information, s u c h as the illumination a n d the angle from w h i c h an object is seen, is what certain psychologists call Gestait perception, n a m e l y the ability to see an object as a w h o l e . T h i s capacity is a precondition to forming the category of object, w h i c h , as we shall soon see, is one of the early concepts that the developing intellect of the infant produces. H o w is it that the m i n d perceives a n object as a w h o l e , a n d the w o r l d as only one real w o r l d , in view of the fact that the brain consists of two h e m i s p h e r e s ? O r to put this question i n another way, h o w do the two h e m i s p h e r e s give rise to a single m i n d ? F r o m o u r previous d i s c u s s i o n , we k n o w that the right half of the visual field projects to the left v i s u a l cortex, and the left half of the v i s u a l field to the right cortex. A similar crossed pathway also obtains for the projection of the acoustic field to the auditory cortex a n d for the projections from the bilateral motor areas of the cortex, w h i c h issue their c o m m a n d s to the musculature o n the opposite side of the body. T h e existence of t w o halves of the brain is not in itself s u r p r i s i n g , since the h u m a n b o d y is, on the w h o l e , bilaterally symmetric. However, the two halves of the body must communicate, because they behave in a coordinated manner. It might be thought that this coordination or integration of the t w o halves w o u l d take place i n a n organ that is u n i q u e rather than bilaterally
129
114
Gestalt perception is an active contribution of our perceptive apparatus to the interpretation of these figures. In each case, structures are seen that are not actually present. They are called subjective contours. When the contours are examined closely, they disappear. One subjective contour even appears to pass under another that intersects with it (b). Optical illusions demonstrate that subjective contours have the same functional effect as real contours, as indicated by the Ponzo illusion shown in (d). Although both vertical lines are the same length, the effect of the subjective triangle is to make the line on the left appear longer. [After Kanizsa, 1976]
symmetric, reflecting the unity of the m i n d . Rene Descartes took this line of reasoning early in the seventeenth century. I n the course of h i s anatomical explorations of the brain, Descartes discovered the pineal b o d y — a single organ l y i n g near the center of the b r a i n — a n d designated it as the seat of the m i n d . M o d e r n investigations have s h o w n the inadequacy of the theory of the unity of the m i n d in general and of its seat in the pineal body in particular. A s for the pineal body, it appears to function, not as the seat of the m i n d , but as a component of the biological clock that controls the daily r h y t h m s of physiology and behavior.
130
235 N o , i n our quest for the m i n d , we m u s t focus on the cerebral cortex, w h i c h is the organ of consciousness and of language. Do consciousness a n d language refer to the s a m e function? C e r t a i n l y not, for otherwise we w o u l d never be at a loss for w o r d s to express o u r thoughts. But can we be conscious of something that w o u l d be impossible for u s to verbalize? H e r e the w o r k on lateralization of various cortical capabilities has revolutionized our insights. N o r m a l l y the two h a l v e s of the cortex are so intimately integrated that investigation of their i n d i v i d u a l function is v e r y difficult. Some years ago, however, surgeons introduced an operation for patients w i t h severe epilepsy. I n this operation the corpus callosum, a massive strand of n e r v e fibers connecting the two h e m i s p h e r e s , is severed. Superficial observation of s u c h patients after their operation indicated, very surprisingly, that they s e e m e d to have suffered no loss at all of their n o r m a l perceptions, their motor activities, or their speech. But more refined studies by Roger Sperry of "split brain" patients revealed a fantastic situation: v i s u a l , auditory, or tactile inputs can be so d e s i g n e d as to reach only one of the h e m i s p h e r e s , a n d w h e n this is d o n e the other h e m i s p h e r e literally does not k n o w about it. T h i s c a n be tested by asking the patient to identify—say, by touch w i t h the right h a n d , controlled by the left h e m i s p h e r e — a n object seen in the left half of the v i s u a l field, a n d thus w i t h visual input leading to the right h e m i s p h e r e . T h e patient cannot do it, even though the verbal i n s t r u c t i o n is given to both h a l v e s of the brain. E v e n more strikingly, it t u r n s out that the right half is incapable of verbalizing what it "knows," e v e n though this k n o w l e d g e is clearly present, since the right half is able to use it for solving complex mental tasks. H o w is this astonishing result to be explained? A s we saw in chapter 6, the cortical areas dedicated to the production of speech are present on only one side (usually the left), rather t h a n symmetrically distributed over both hemispheres. W h e n the corpus c a l l o s u m is cut, the left h e m i sphere, w h i c h contains the speech p r o d u c t i o n centers, has no idea of what is being presented to the right. N e v e r t h e l e s s , w h i l e the right h e m i sphere cannot produce speech, it is capable of a great deal of mental processing on its o w n . T h e cerebral d i c h o t o m y goes so far that the patient may s h o w a n emotional response, say by a smile, w h e n seeing a picture with the right brain, but is asked why he or she s m i l e s , the verbalizing left brain can only admit ignorance. T h e s e great discoveries s h o w that we have two m i n d s u n d e r one roof, t w o m i n d s n o r m a l l y so well integrated that their separation is inapparent: they talk to each other via the corpus callosum, a n d then talk to the outside w o r l d with one voice, controlled by the left m i n d .
131
Lateralization of function in the cerebral hemispheres of the human brain. The left hemisphere (in the majority of individuals) is specialized for language comprehension, speech, and computational abilities. The right hemisphere is specialized for spatial constructions and nonverbal ideation, and also possesses simple verbal abilities. These lateralized functions can best be demonstrated in individuals in which the neural connections (via the corpus callosum) between the two cerebral hemispheres have been severed in the surgical treatment of severe epilepsy. [After Sperry, 1968]
Does the right h e m i s p h e r e , the n o n s p e a k i n g hemisphere, have consciousness? It certainly has a m i n d , in the sense that it can hear a n d u n d e r s t a n d speech and rationally answer questions, not by speech, but by solving problems. W h e t h e r one wants to call this consciousness is a matter of terminology, a n d terminology i n this area is not settled at present. I n any case, the unity of the m i n d i n n o r m a l persons is evidently the result of an interhemispheric c o n s e n s u s mediated by the corpus callosum. We will n o w strive to take a broader view of h u m a n perception and consider the relevance of these neuropsychological findings for epistemology, that is to say, for the philosopher's quest for u n d e r s t a n d i n g how we come to k n o w what we know.
132
117
Until quite recently, it was one of the basic misconceptions of philoso p h e r s that the m i n d deals with p r i m a r y sense impressions a n d that the individual learns to make the abstractions that form the basis of perception. For instance, according to the e m p i r i c i s m of the early eighteenth century, as formulated mainly by D a v i d H u m e a n d the F r e n c h encyclopedists, the m i n d at birth is a clean slate o n w h i c h there is gradually sketched a representation of the real w o r l d , built on cumulative experience. T h i s representation is orderly, or structured, because, thanks to the principle of inductive reasoning, we can recognize regular features of our experience a n d infer causal connections b e t w e e n events that habitually occur together. T h i s viewpoint rejects as a logical absurdity the possibility of innate or a priori knowledge of the w o r l d , that is, k n o w l edge possessed prior to h a v i n g experienced the w o r l d , w h i c h was a central feature of the seventeenth c e n t u r y p h i l o s o p h y of rationalism advocated by Descartes. I n the latter part of the eighteenth century, h o w ever, I m m a n u e l K a n t demonstrated that empiricist p h i l o s o p h y a n d its rejection of the possibility of a priori k n o w l e d g e is g r o u n d e d on a n i n a d equate u n d e r s t a n d i n g of the m i n d and its relation to reality. K a n t pointed out that sensory impressions can become experience, that is, gain meaning, onlv after they are interpreted in terms of the a priori categories— s u c h as time, space and object—that w e bring to, rather than derive from, experience. Tacit resort to propositions whose validity is similarly accepted a priori, s u c h as "Some A are B; therefore all A are B" (induction) or "The occurrence of a set of conditions A is both necessary a n d sufficient for the occurrence of B" (causation by A of B), allows the m i n d to construct reality from that experience. K a n t referred to these a priori categories a n d propositions of cognition as "transcendental," because they trans c e n d experience a n d were thought by h i m to be beyond the scope of scientific i n q u i r y But is it not strange that if, as K a n t alleges, we bring s u c h categories as time, space, a n d object, as well as the notion of causality, to sensation a priori, that they h a p p e n to fit the real w o r l d so well? C o n s i d e r i n g all the bizarre ideas we might have had prior to experience, it s e e m s nothing short of miraculous that our a priori ideas h a p p e n to be those that fill the bill. T h e way to resolve this d i l e m m a opened w h e n C h a r l e s D a r w i n put f o r w a r d the theory of natural selection i n mid-nineteenth century. But few philosophers or scientists s e e m e d to have noticed this until K o n rad L o r e n z d r e w attention to it in the 1940s. L o r e n z pointed out that the empiricist a r g u m e n t that knowledge about the w o r l d c a n enter the m i n d only through experience is valid if we consider only the ontogenetic devel-
133
118
opment of m a n , from fertilized h u m a n egg to adult person. But once w e also take into account the phylogenetic development of the h u m a n brain through evolutionary history, it becomes clear that persons c a n k n o w something of the w o r l d innately, prior to a n d independent of their o w n experience. After all, there is no biological reason w h y such k n o w l e d g e cannot be passed o n from generation to generation via the ensemble of genes that determines the structure a n d function of our brain. For that genetic ensemble came into b e i n g through the process of natural selection operating on our remote ancestors. A c c o r d i n g to L o r e n z , "experience h a s as little to do w i t h matching of the a priori with reality as does the matching of the fin structure of a fish w i t h the properties of water." In other w o r d s , the K a n t i a n notion of a priori knowledge is not i m p l a u sible at all. Rather, Kant's claims of the "a-priori-ness" of s u c h categories as space, time, and object, as w e l l as of causality, as transcendental c o m ponents of cognition almost hit the nail on the head. T h e s e ideas are indeed a priori for the individual, but they did not fall from heaven; they are matters of evolutionary adaptation, designed for s u r v i v a l i n the real world. It appears therefore that two k i n d s of learning are i n v o l v e d i n our dealing w i t h the w o r l d . O n e is phylogenetic learning, in the sense that during evolution we have evolved very sophisticated m a c h i n e r y for perceiving and making inferences about a real w o r l d , of w h i c h the preconscious neurophysiological abstraction processes acting on visual input, the perceptual constancy p h e n o m e n a associated with vision, a n d the interhemispheric c o n s e n s u s of our two m i n d s reached via the corpus callosum are but a few examples. T h e y s h o w that, collectively a n d across history, the h u m a n species h a s learned to deal with signals c o m i n g from the outside world by constructing a m o d e l of it. I n other w o r d s , w h e r e a s in the light of m o d e r n u n d e r s t a n d i n g of evolutionary processes, w e can say that the individual approaches perception a priori, this is by no m e a n s true w h e n we consider the history of m a n k i n d as a whole. W h a t is a priori for individuals is a posteriori for the species. T h e second k i n d of learning involved in dealing with the world is ontogenetic learning, namely the lifelong acquisition of cultural, linguistic, a n d scientific knowledge. T h u s we see the world t h r o u g h multiple pairs of glasses: some of them are inherited as part of o u r physiological apparatus, others acquired from direct experiences as we proceed through life. I n a sense, the discoveries of science help us to see what the w o r l d is like without some of these pairs of glasses. A s K o n r a d L o r e n z has put it, every step of knowledge means taking off a pair of glasses—but we could never dispense with all of them.
134
119
REFERENCES Campenhausen, C . von. 1981. Die Sinne ties Menschett. Stuttgart: Thieme. Geschwind, N. 1978. Specialization of the human brain. Scientific American 241(3): 180-199. Hassenstein, B. 1971. Information and Control in the Living Organism. London: Chapman and Hall. Hubel, D. H . , and T. N. Wiesel. 1970. The period of susceptibility to the physiological effects of unilateral eye closure in kittens, journal of Physiology 206: 419 - 436. Kanizsa, G . 1976. Subjective contours. Scientific American 243(4):48-52. Lorenz, K. 1941. Kant's Lehre vom apriorischen im Lichte gegen war tiger Biologie (Kant's doctrine of the a priori in the light of contemporary biology]. Blatter fitr Deutsche Philosophie 15: 94-125. In General Systems, ed. Bertalanffy and Rapoport, vol. 7, pp. 23-35. A n n Arbor: Society for General Systems Research, 1962. . 1959. Gestaltwahrnehmung als Quelle wissenschaftlicher Erkenntnis (Gestalt perception as a source of scientific knowledge), Zeitschrift fitr Experimented und Angeimndte Psychologie 6: 118-165, In Studies in Animal and Human Behaviour, vol. 2, pp. 281-322. Cambridge: Harvard University Press, 1971. Also in General Systems, ed. Bertalanffy and Rapoport, vol. 7, pp. 37-56. Ann Arbor: Society for General 5ystems Research, 1962. McKay, D. M, 1971, Voluntary eye movements are questions. Bibliotheca Ophthalmologics 82; 369-376. Purves, D., and J. W. Lichtman. 1985. Principles of Neural Development. Sunderland, Mass.: Sinauer Associates. Sperry, R. W. 1968. Mental unity following surgical disconnection of the cerebral hemispheres. The Harvey Lectures 62: 293-323. . 1982. Some effects of disconnecting the cerebral hemispheres. Science 217: 1223-1226. Wiesel, T. N., and D. H . Hubel. 1963. Single cell responses in striate cortex of kittens deprived in one eye. journal of Neurophysiology 26: 1003-1017.
135
Reprinted with permission from Images and Understanding, pp. 257-283, 1990 © 1990 Cambridge University Press
Understanding images in the brain COLIN
liLAKHMORK
The homunculus
fallacy
If y o u h a v e n e v e r read L u d w i g Wittgenstein's great treatise Philosophical Investigations', I h a v e just a single w o r d of advice. Don't! O r , to be less facetious, do it with a n open m i n d and a stiff d r i n k , ft is a n e x t r a o r d i n a r y a n d puzzling book - a m i x t u r e of brilliant insight a n d frustrating c o n u n d r u m s . Wittgenstein's e n d u r i n g c o n c e r n w a s for the n a t u r e o f l a n g u a g e a n d w h a t it tells us about the m i n d s of language-users. 1 lis a r g u m e n t s led h i m lo one c o n c l u s i o n that seems trivial, maybe just plain w r o n g , but w h i c h
has
s p a w n e d w o r r y i n g criticism of o u r use of e v e r y d a y w o r d s lo describe h o w I he brain w o r k s . Wittgenstein wrote: 'Only of a living h u m a n being a n d w h a t resembles (behaves like) a living h u m a n being c a n one say: il h a s .sensations: 1
it sees; is blind; hears; is c o n s c i o u s or u n c o n s c i o u s ' ( W i t t g e n s t e i n ) . 1
A n t h o n y Kenny- , in his spirited defence of Tlie Legacy of Wittgenstein, laments w h a t he sees as the frequent rejection of this d i c t u m by psychologists of perception, by neurophysiologists a n d by those w h o a r e interested in m a c h i n e intelligence. He coined the t e r m h o m u n c u l u s fallacy' to describe s u c h false ascription of predicates that a r e valid only for whole h u m a n beings w h e n t a l k i n g about bits o f a n i m a l s or people (especially their brains), o r about electronic circuits. W h a t , exactly, is the problem, y o u m i g h t ask? After all, noone w o u l d suggest that a one-legged m a n c a n n o t hear (In the sense that w e use that w o r d to describe the auditory perceptions of a n intact h u m a n being). A n d is it really a gross c o n c e p t u a l b l u n d e r lo talk of a robot w i t h a n artificial eye as s m u g ? H o w complete m u s t a h u m a n being be a n d h o w h u m a n - l i k e m u s t the b e h a v i o u r of a m a c h i n e be in order for the predicates of sensation a n d c o n s c i o u s n e s s to be valid? T o clarify Ihe issue we need a n example, a n d the one Uiul A n t h o n y K e n n y develops is the question of h o w the brain perceives the v i s u a l world (il y o u will, for the m o m e n t , forgive me for deliberately c o m m i t t i n g the h o m u n c u l u s fallacy!). K e n n y t h i n k s that present-day v i s u a l neurophysiologists a n d psychologists h a v e still not escaped from a c o n c e p t u a l tangle first recoanized bv 136
258
Rene Descartes. In his description of the formation of the retinal image, in La Dioptrique of 1 6 3 7 . and his subsequent suggestion that images of a sort are formed in the brain. Descartes both invented and acknowledged the major problem. 'The things we look at', he wrote, 'form quite perfect images in the back of our eyes.' Furthermore, such an image is 'not only produced in the back of the eye but also transmitted lo the brain'. Descartes did not. of course, know about nerve impulses or synaptic transmission and he followed ancient Aristotelian views about the importance of fluids or 'spirits' in the body. The clear cerebrospinal fluid, which fills the cavities or ventricles of the brain, had long been thought of as a pure and perfect distillation of blood and air. ft was called animal spirit by the medieval philosophers, who believed it to be the source of the anima or soul. Descartes imagined that the myriad fibres in the optic nerves transmitted a coherent copy of the retinal image, in the form of a pattern of vibrations, on to the surface or the ventricles of the brain. Now. the traditional medieval Cell Doctrine had suggested that the ventricular system of the brain is divided into three cells, serving different aspects of mental function, from sensation (in the first cell) through thought and reason (middle cell) to memory or movement (last cell). Descartes' most famous addition to this scheme was his attribution of special status to the pineal gland, a tiny organ at the back of the ventricles (the true function of which is still obscure). He thought that the pineal gland secreted the animal spirit and acted as an interface between the soul and the mere physical machinery of the brain. In Les Passions de I'dme*. Descartes imagined that the two pictures of any particular shape in the outside world (one from each eye), formed on the lining of the ventricles, set up corresponding patterns of motion in the fluid, which 'radiate towards the little gland which is surrounded by these spirits. Thus the two images in the brain form only one image on the gland, which, acting directly on the soul, causes it to see the shape.' This single picture on the pineal gland not only fused the images of the two eyes into a single view of the world but even conveniently re-inverted itl But then comes the problem of the homunculus fallacy. How can simply having an image in the brain constitute seeing that image? Descartes had earlier acknowledged this problem for the picture on the walls of the ventricles. 'The picture to some extent resembles the objects from which it originates. .. but we must not think that it is by means of this resemblance that the image makes us aware of the objects - as If we had another pair of eyes inside the brain to see it.' What was different about the image on the pineal gland was, presumably, its direct communion with the soul, which was, in effect, just such an internal pair of eyes - indeed an internal perceiver, thinker. 1
137
259
;
t ig. 17.1.
Rene Descartes w a s n u l Ihe lirsl lo observe ihu Inverted Image formed
on the retina of the eye. but Descartes' description i n La Dioptrique of 16 S7 w a s the most complete at that time. This diagram shows the technique that Descartes used to reveal this image. He cut a window in the outer coat of the back of a n ox eye and covered it with paper. He then saw the liny inverted image formed on the paper screen.
138
260
Pig. 1 7.2.
D u r i n g Ihe medieval period, under the influence of the C h u r c h , very
little experimental observation w a s carried out. Theories ofbrain function rested Inrgely on the writings of the Greeks a n d the dissections of Galen of Rergama ( 1 2 9 - 1 9 9 A D ) . One dominant theme, w h i c h persisted until the 18th century, w a s that mental processes take place in Ihe fluid ('animal spirit') that tills Ihe chambers or ventricles or the brain. There are 3 s u c h chambers, the first being divided into a pairoflateral ventricles. According to the Cell Doctrine' of Ihe Karly Christian Fathers, this first pair received messages from the sense organs: ihe middle one w a s responsible for judgement, thought a n d reason; and the last was the seat of memory a n d (sometimes) movement. This illustration of the Cell Doctrine comes from the 1 5 0 6 edition of Philosophia pauperum of Alberlus M a g n u s ( 1 2 0 6 - 8 0 ) . published as Philosophia naturallsby M . Furter, Basle. T h e three ventricles are s h o w n as simple circles, each divided In two.
chooser and knower. That, of course, is the essence ofCartesian Dualism. The body, including the brain, is a machine; a delicate, complex, beautifully interconnected machine - but merely a machine. In the Traite de Vhomme of 1664 Descartes asks us to imagine that all the functions that I attribute to this machine such a s . . . waking and sleeping: the reception of light, sounds, odours . . ., the impression of ideas, . . . the retention of those ideas in memory; . . . appetites and passions; and finally the movements of all the external members... occur naturally in this machine solely by the disposition of its organs, no less than the movements of a clock.
139
261
Fig, 17.3.
I n this famous diagram from the Traite de I'homme af 1 6 6 4 , Des-
cartes got the anatomy very wrong but made a number of conceptual advances. He suggested that nerve fibres from each eye project into the brain in a coherent bundle, thus reproducing a n array of activity (2,4,6) on the wall of the ventricles corresponding to the retinal image (1,3,5). He also proposed that combination of signals from the two eyes might occur to form a binocular image (a,b,c) on the pineal body (H), w h i c h Descartes saw as the site of communication between the spiritual soul and the machinery of the brain. I n fact w e now k n o w that a binocular 'map' Is indeed set up, not in the pineal, but In the primary visual cortex at the back of Ihe cerebral hemispheres.
But, to Descartes, the essentially human features of existence and action consciousness, perception, decision and w i l l - w e r e the domain of the soul, a mysterious, silent agent that watched the pictures in the brain and poked its invisible fingers into the clockwork from time to time to guide the mere machine. If we resist the intellectual surrender of Dualism, is it possible to steer a course around the rocks of the homunculus fallacy on our way to a material explanation of behaviour and even of perception and consciousness?
Order and disorder in brains and machines Computers can do very clever things without forming pictures in their chips. In the most general sense of the word computer, the brain is a computer - in the immortal words of Marvin Minsky, it is a 'meat machine'. Then could the brain not work perfectly well without having to depend on Cartesian inner images? In June, 19 86. a steering committee for a System Development Foundation Symposium on Computational Neuroscience composed a 'formulation of
140
262
dialectics' i n w h i c h t w o opposed positions w e r e put f o r w a r d i n response to the q u e s t i o n 'Is p h y s i c a l locality a n essential p a r t of n e u r a l c o m p u t a t i o n ? ' : Position ) . . . : The anatomical structure of the brain has no more to do with its function than the shape of the cabinet of a Vax fa powerful computer), or the location of its pc boards. Brain function is determined by the logical and dynamic connection properties of Its neurons. The actual physical structure, location, architecture, and geometry is irrelevant compared to its logical, connectiontst aspects. Position 2: Significant recent work in brain research has been related to the discovery and elucidation of detailed forms of somatotopic mapping, laminar specialization in cortex, and columnar architectures representing sensory sub-modalities. These forms of functional architecture may represent a major mode of brain function: the formatting of sensory data in a manner which simplifies its further processing. One of the major differences in computational style of brain versus Vax may well be the indifference of the Vax to its geometry and the exquisite attention paid by the brain to its geometry'. W h e t h e r or not the p h y s i c a l , s p a t i a l a r r a n g e m e n t of activity i n the b r a i n h a s functional
significance is the issue that I w a n t to address. Undoubtedly
spatial o r d e r is a d o m i n a n t feature of the b r a i n , w h i c h is far from the r a n d o m n e t w o r k t h a t s o m e t h e o r e t i c i a n s w o u l d h a v e liked it to be. T h e h a n d f u l of jelly t h a t fills the h u m a n h e a d is i n fact a tightly p a c k e d c o n g l o m e r a t e of separate c l u m p s of n e r v e cells w r a p p e d i n h u g e bundles of a x o n s (nerve fibres). T h e s t e m of the b r a i n , the c e r e b e l l u m , the c e r e b r a l h e m i s p h e r e s - they a r e a l l distinct s t r u c t u r e s w i t h t h e i r o w n c h a r a c t e r i s t i c a p p e a r a n c e s i n m i c r o s c o p i c sections, a l l laid out a n d i n t e r c o n n e c t e d i n a h i g h l y ordered fashion. E v e n the c e r e b r a l cortex, the w r i n k l e d m a n t l e of grey m a t t e r that s w a d d l e s the c e r e b r a l h e m i s p h e r e s , is, i n fact, a vast p a t c h w o r k of distinct f u n c t i o n a l a r e a s . I n 1 6 7 2 , i n Oxford. T h o m a s W i l l i s first suggested, o n the basis of a n a t o m i c a l e v i d e n c e as w e l l as the effects of brain d a m a g e , t h a t m e n t a l f u n c t i o n s take place i n the c e r e b r a l cortex r a t h e r t h a n i n the fluid of the v e n t r i c l e s . B u t the first s u g g e s t i o n t h a t specific functions a r e localized i n different regions of the c e r e b r a l cortex c a m e from the bogus science of p h r e n o l o g y , w h i c h c l a i m e d t h a t v a r i o u s m e n t a l functions are performed in 'organs', distributed o v e r the b r a i n , w h o s e development is reflected i n the size of b u m p s o n the s k u l l . F r a n z Joseph G a l l , the A u s t r i a n p h y s i c i a n w h o founded p h r e n o l o g y at the start of the 1 9 t h c e n t u r y , w a s a n expert a n d respected anatomist, w h o
m u s t at least take credit for d r a w i n g attention to
the
distinctive folds a n d creases i n the cortex. E v e n the attempt to correlate i n d i v i d u a l differences i n m e n t a l faculties w i t h v a r i a t i o n s in the s t r u c t u r e of the b r a i n w a s r e m a r k a b l y o r i g i n a l a n d far-sighted. P h r e n o l o g y w a s scientific
141
263
Fig. 17.4.
This illustration, from the 1 6 6 4 edition of Cerebri Analomie of
Thomas Willis 0- Martyn and |. Allesiry, London), is perhaps the most famous 1 7 l h century diagram of Ihe h u m a n bruin. The original drawing w a s made by Christopher W r e n , w h o worked for Willis in Oxford. The brain is s h o w n from below, with the brainstem, surrounded by Ihe cerebellum (B). Many of Ihe cranial nerves, including Ihe optic nerve {li) are seen emerging from Ihe brainstem. T h e blood supply of Ihe base of Ihe brain is very clearly illustrated, as are the great cerebral hemispheres ( A ) , with their m a n y folds and grooves. Willis was the first to attribute mental functions lo the cerebral hemispheres.
n o n s e n s e not b e c a u s e of a defect in its u n d e r l y i n g logic b u t b e c a u s e of its d e p e n d e n c e on a n e c d o t a l o b s e r v a t i o n s a n d i n a d e q u a t e s a m p l e s . I r o n i c a l l y , a l t h o u g h the m e t h o d s a n d c o n c l u s i o n s of p h r e n o l o g y w e r e s o o n discredited a n d
ridiculed
by the scientific c o m m u n i t y , the c o n c e p t of f u n c -
t i o n a l l o c a l i z a t i o n s u b s e q u e n t l y b e c a m e the b e d r o c k of b r a i n r e s e a r c h , I n the
142
264
Fig. 17.5.
T h i s is one or the m a n y diagrams illustrating the bogus theory of
phrenology that were popular for more than a hundred years after the distinguished anatomist Franz foseph Gall (1 7 5 8 - 1 8 2 8 ) first suggested that mental faculties were localized in distinct brain organs and that character could be diagnosed by feeling the bumps of the skull.
1860s, neurologists, especially John Hughlings Jackson in London and Pierre-Paul Broca in Paris, started to chart the surface of the cerebral hemispheres, using as their evidence the effects on behaviour, perception, language, movement and personality of localized damage to the brain. Broca described an area in the frontal lobes, usually on the left side, that now bears his name and which is essential for normal speech. Damage here (through stroke or injury) robs people of their ability to say more than a word or two, but not of their understanding of the spoken or written word. In the context of images in the brain. Hughlings Jackson's findings were even more significant. He described the way in which epileptic seizures sometimes start with involuntary twitching in one particular group of
143
265
m u s c l e s , often those of the t h u m b , the c o r n e r of the m o u t h o r o n e of the toes, before spreading to other parts of the body. B y r e l a t i n g the starting point of these ' J a c k s o n i a n fits' to s m a l l regions of degeneration a n d i r r i t a t i o n , discovered post mortem i n the b r a i n . J a c k s o n c o n c l u d e d that a strip of cortex, r u n n i n g d o w n the middle of e a c h cerebral h e m i s p h e r e j u s t b e h i n d Broca's a r e a , is responsible for a c t i v a t i o n of the m u s c l e s on the opposite side of the body. W h a t is more, t h e motor strip on e a c h side is a r r a n g e d
topographically,
the m u s c l e s of the toes a n d leg being controlled by the top of the strip, w i t h the torso, a r m . h a n d a n d face represented s u c c e s s i v e l y a l o n g the strip. T h i s motor area of the cortex h a s n o w been explored i n a n i m a l s as well a s h u m a n s , by a variety of e x p e r i m e n t a l techniques, a n d they a l l c o n f i r m H u g h l i n g s J a c k s o n ' s opinion t h a t there is a k i n d of image of t h e body m u s c l e s i n t h e b r a i n . Since H u g h l i n g s J a c k s o n ' s time, the concept of f u n c t i o n a l sub-division a n d topographic representation h a s become a sine qua non of brain r e s e a r c h . T h e task of c h a r t i n g the b r a i n is far from complete b u t t h e successes of the past m a k e o n e confident that e a c h part of the b r a i n ( a n d especially the cerebral cortex) is likely to be organized in a spatially ordered fashion. Just a s i n t h e decoding of a cipher, the t r a n s l a t i o n of L i n e a r B or the r e a d i n g of hieroglyphics, all t h a t w e need to recognize the order i n t h e brain is a set of rules - rules that relate the activity of n e r v e s to events in the outside w o r l d o r in the a n i m a l ' s body. T h e most fully explored spatial representations a r e those i n t h e regions of cerebral cortex reserved for t h e three m a j o r senses - v i s i o n , h e a r i n g a n d bodily sensation ( t o u c h , p a i n , etc.). I n C h a p t e r 1, H o r a c e B a r l o w h a s already described these sensory a r e a s (see F i g . 1.2) a n d it is i m p o r t a n t to a d d that e a c h p r i m a r y sensory a r e a is n o w k n o w n to be s u r r o u n d e d by a n u m b e r of additional areas, e a c h probably c o n t a i n i n g its o w n distinctive form of s p a t i a l a r r a n g e m e n t of information-processing (but m o r e of this later). Faced w i t h s u c h o v e r w h e l m i n g evidence for topographic patterns of a c tivity i n the b r a i n it is h a r d l y s u r p r i s i n g that n e u r o p h y s i o l o g i s t s a n d n e u r o a n a t o m i s t s h a v e c o m e to speak of the b r a i n h a v i n g maps, w h i c h a r e t h o u g h t to play a n essential part in the representation a n d interpretation of the w o r l d by the b r a i n , j u s t a s the m a p s of a n atlas do for the r e a d e r of t h e m . T h e biologist J.Z. Y o u n g writes of the brain h a v i n g a l a n g u a g e of a pictographic kind: ' W h a t goes o n i n the brain m u s t provide a faithful representation of events outside, a n d the a r r a n g e m e n t of the cells in it provides a detailed model 6
of the w o r l d . It c o m m u n i c a t e s m e a n i n g s by t o p o g r a p h i c a l a n a l o g i e s ' . B u t is there a d a n g e r i n the m e t a p h o r i c a l use of s u c h t e r m s as 'language', ' g r a m m a r ' a n d m a p ' to describe the properties of the b r a i n ? Peter H a c k e r , the Oxford Wittgenstein s c h o l a r , sees it a s a n extreme e x a m p l e of t h e h o m u n c u l u s
144
266
Fig. 17.6. Hughlings Jackson's idea that there is a motor map In the human cerebral hemispheres received strong support from the observations of the neurologists W. Penfield and T. Rasmussen in the 1940s. They described the effects of local electrical stimulation of the surface of the cortex in conscious human patients undergoing neurosurgical operations under local anaesthetic. Here is Penfield and Rasmussen's diagram of the motor 'homunculus'. Imagine that the brain has been cut in half, in a vertical plane lying approximately through the two ears, and we are looking from behind at a cross-section through the motor cortex on the right side orthe brain. The labels show the various par is of the body that are caused to move by electrical stimulation at different points along the motor strip, from the tongue and face on the side of the hemisphere around to the toes on the part of the strip that lies buried in the cleft that runs along the middle of the hemispheres. The cartoon drawings Indicate the relative areas of motor cortex devoted in different parts of the body. Notice the way in which the jaws and hands are vastly over-represented.
145
267 fallacy'. He finds it a •startling, indeed amazing idea' that scientists should suggest that there are maps in the brain. He writes: there are no representing maps without conventions of representation. There are no conventions of representation without the use, by intelligent, symbol-employing creatures, of the representation. And to use a representation correctly one must know the conventions of representation, understand them, be able to explain them, recognize mistakes and correct or acknowledge them when they are printed out. Whether a certain array of lines is or is not a map is not an intrinsic feature of the lines ... but a conventional one (that is, the actual employment, by a person, of a convention of mapping)... the modern neurophysiologist... comes perilously close to saying that when a person sees an object there is a map, a representation of the object, not on the pineal gland, but on the visual cortex. But now he must explain who or what sees or reads the map. Frankly. I think that this concern is misplaced. I cannot believe that any neurophysiologist believes that there is a ghostly cartographer browsing through the cerebral atlas. Nor do I think that the employment of common language words (such as map, representation, code, information and even language) is a conceptual blunder of the kind that Peter Hacker imagines. Such metaphorical imagery is a mixture of empirical description, poetic licence and inadequate vocabulary. Yet an important question remains. If the existence of spatially ordered neural images is useful to the brain, what benefit does that order bestow? Indeed, is it possible that the presence of order {convenient though it may be to the neuroscientist) is no more than an epiphenomenon - perhaps an inevitable but useless consequence of the fact that nerve fibres, growing from the sense organs to their targets in the brain, tend to preserve the same spatial pattern as that of the receptors in the sense organ - rather like the individual glass fibres in a coherent fibre optic bundle?
Isomorphic maps A tendency of nerve fibres to maintain spatial order as they grow would, of itself, generate isomorphic distributions in the brain - patterns topographically directly related to the array of receptor cells in the sense organ. Many cerebral maps are of this kind. Perhaps the map of the visual field on the primary visual cortex (described by Horace Barlow in Chapter 1) is simply a consequence of the fact that there is an orderly input of nerve fibres maintaining the neighbourly relations of the cells in the retina (though, as Barlow points out, the partial crossing-over of fibres from the two eyes as they enter the brain is a striking, and functionally appropriate, departure from strict isomorphism). In the same terms, the somatic sensory area has on it a 'picture' of the
146
268 opposite half of the body because sensory nerve fibres entering the spinal cord form an array corresponding to their origin on the body surface. Even the 'tonotopic' map in the primary auditory cortex (in which nerve cells are distributed in an orderly sequence according to the tone frequency to which they respond best) is isomorphic. The sound receptor cells are distributed along a membrane in the cochlea (ihe sense organ in the inner ear) and different frequencies of sound set up different patterns of vibration of this membrane, the point of maximum vibration (and thus maximum nerve activity) shifting along the membrane as the tone changes in pitch from high to low. So. as long as nerve fibres in the auditory pathway maintain the same general pattern as that of the array ofreceptor cells in the cochlea, a tonotopic map is inevitable. It is well known that these cortical maps have spatial distortions, which appear as if they might be functionally important. For instance, the representation of the most acute, central fovea of the retina occupies a hugely disproportionate fraction of the visual cortex; and the most sensitive areas of the body surface - the fingers, toes and lips in humans, the snout in a pig have expanded representations in the somatic sensory cortex. At first blush, it is tempting to think of these enlarged sub-maps as being like the large-scale inserts of major cities in a road atlas - a functional device to provide greater detail about things of most importance. In fact, though, even these apparently significant peculiarities of cortical maps might also be a simple consequence of patterned nerve growth. The fovea of the retina, the fingers, toes and so on all have a much denser innervation of sensory nerve fibres than the rest of the retina and skin. Hence, over-representation of those parts is an inevitable consequence of uniform topographic distribution of nerve fibres into the brain. One of the most remarkable isomorphic maps is found in the somatic sensory area of the cortex of a mouse, or indeed any rodent that has large whiskers on its face, which it uses to explore its environment. The mouse has five rows of such whiskers and the sensory nerve that innervates them is much larger than the optic nerve from the eye. It is hard to imagine the perceptual world of a mouse but the sensations from its whiskers must surely play a major role in it. No surprise, then, to discover that a huge proportion of the somatic sensory area on the side of the cerebral hemispheres of a mouse (Fig. 17.7) is devoted to signals from the whiskers. But the isomorphism goes much further than the existence of a map of the body. Tom Woolsey and Hendrik Van der Loos looked at microscopic sections of this whisker area, cut parallel to the flat surface of the cortex. The sections through the fourth cortical layer (where most of the incoming sensory nerve fibres terminate) 8
147
269 have the remarkable appearance shown in Fig. 1 7 . 7 - a pattern of rings, each ring consisting of densely packed cortical nerve cell bodies surrounding a bunch of incoming fibres. They called these structures 'barrels' because their three-dimensional appearance is like rows of barrels. There are the same number of barrels as there are whiskers, and the barrels are distributed in a pattern just like that of the whiskers on the face. The obvious conclusion that each barrel of cells is a cortical 'organ' (a barrel organ?) corresponding to a single whisker has been amply confirmed by electrophysiologists, who have recorded activity from cortical cells in this area and have shown that the cells of each barrel respond when one particular whisker is touched. Figure 17.8 is a further, elegant demonstration of this barrel/whisker isomorphism, performed by Margaret Kossut and Peter Hand at the University of Pennsylvania'. It is a computer-processed image of a cross-section (not a surface-parallel section) through the somatic sensory cortex of the left hemisphere of a rat (which also has barrels). The anaesthetized animal was given an injection of a radioactively-labelled analogue of glucose (2deoxyglucose), which is taken up by any highly active nerve cells in the brain. After the injection, just one whisker on the opposite side of the face was vibrated gently, and the computer image (in which the colour scale indicates the level of radioactive labelling) shows one strongly labelled barrel in the middle of the cortex. In a trivial sense, these brain maps do now have readers - the anatomists and physiologists who have discovered them. But the question remains: does the brain itself make use of its maps in a functional sense? Or are they all accidents of nerve growth? In the case of the whisker barrels there is good evidence that the map is not an inherent property of that bit of cortex but is imposed on it by the ingrowing sensory fibres. Van der Loos and Woolsey have shown that the cortical cells do not become assembled into barrels until about five days after birth in the mouse; if, before this age. one row of whiskers is plucked out, the corresponding row of barrels fails to form and the neighbouring barrels are larger than normal. This strongly suggests that the topographic order in the brain is determined by the pattern of incoming fibres. if all maps in the brain were of this isomorphic form, it would be difficult to argue for their functional value. But if we were to find that the brain contained new forms of mapping, topologically related to some important uspecl of the sensory stimulus, but not simply rellectltig an anatomical pattern of ingrowing nerve fibres, that would surely indicate that maps have meaning.
148
270
Fig. 17.7. The somatic sensory area in the cortex of a mouse occupies a central strip down each side of the cerebral hemispheres. Each strip receives sensory information from the skin and other tissues or the opposite side of the body. The pattern of projection forms a crude, distorted map or Ihe half-body, with the muzzle representation vastly magnified. The upper diagram shows the right side or the mouse brain, with the general arrangement of this body map. There is a . huge area devoted to the whiskers. Below is a photomicrograph (kindly supplied by Hendrik Van der Loos), taken at low magnification, of a single thin section through the cortex in the whisker area, cut parallel to the surface of the cortex. This particular section has glanced through the fourth layer, where the incoming sensory fibres arrive. It is stained lo show the individual cortical nerve cells as small blue-green dots and they clearly form a distinct pattern of rings or 'barrels', with the same spatial arrangement as the array of whiskers on the muzzle.
149
271
Fig. 17.8.
Here we see a computer-processed image of a microscopic section cut
in a vertical plane through the whisker area of the left hemisphere of a rat. T h e 1 m m scale bar indicates the size of this piece of cortex. T h e rat had received a small injection of a radioactive substance similar to glucose, w h i c h is taken up by active nerve cells. D u r i n g Ihe uptake period, one whisker on the right side of Ihe face was gently stroked to stimulate its sensory nerve endings. T h e colour scale indicates the levels of radioactivity in the brain. T h e red spot in the middle layers of the cortex is presumably a n individual 'barrel' of nerve cells activated by the stimulated whisker. (Computer image kindly supplied by Margaret Kossut a n d 9
Peter H a n d . )
Non-isomorphic maps T h e r e a r e representations i n the b r a i n that a r e topographic in relation to the outside w o r l d , or to some aspect of the sensory s t i m u l u s , b u t w h i c h a r e not isomorphic, i n the sense that they do not h a v e the s a m e spatial a r r a n g e m e n t as the distribution of receptors. H o r a c e B a r l o w , i n C h a p t e r 1 of this v o l u m e 10
and e l s e w h e r e , h a s already discussed the principles b e h i n d ( a n d a d v a n tages ol) s u c h forms of m a p p i n g . A few examples will m a k e clear w h a t I m e a n . T h e first e x a m p l e - one already s h o w n i n F i g . 1.6 - is the w a y i n w h i c h the orientations of the edges of shapes i n the v i s u a l w o r l d a r e represented by a local system of s u b - m a p p i n g w i t h i n the v i s u a l cortex. I n d i v i d u a l n e r v e cells i n
150
272 the primary visual cortex are selectively sensitive to the angles of lines and edges in the visual field: some cells respond best (produce the highest rate of impulses) when shown a vertical line, others a horizontal line and so on, for every possible angle of line. Moreover, these nerve cells are clustered together in 'columns' or 'slabs', so that if a microclcctrode is put into the cortex perpendicular to the surface and recordings are taken from several cells in succession, then all of them, from the surface down to the bottom of the grey matter, respond best to roughly the same orientation. But if an electrode moves obliquely through the cortex, the preferred orientation shifts, usually in very small steps, from one cluster of column of cells to the next. It seems, possible, then, that the primary visual cortex is 'decomposing' images into the orientations of their component contours and classifying shapes by the angles of their edges. For such a representation to be complete, in every part of the visual field, there are certain formal requirements for the form of mapping. Obviously, all orientations must be represented for each position on the visual field, otherwise the animal or human being would be blind to certain edges at certain points in space. This is achieved by superb efficiency of representation. First, there is sufficient sloppiness or scatter in the simple isomorphic map of the visual field to ensure that each point in space is represented over a patch of visual cortex about 2 mm in diameter. In turn, the orientation columns are laid down in such a way that a full sequence of all orientations is represented across about 0 . 7 5 - 1 . 0 0 mm. This means that there is a 'safety factor' of about 2 in the mapping' . Any particular edge in the visual field causes activity in at least two different columns of orientationselective cells and the chance is very low that any particular orientation will be missed completely. 1
Now. orientation is a one-dimensional, cyclical variable - from vertical to oblique to horizontal to the other oblique, back to vertical and so on. On the other hand the visual field is two-dimensional. To represent both properties smoothly across a single two-dimensional surface (the cortex) poses a topological problem. Figure 17.9 shows the way in which it seems to be solved in the primary visual cortex of the cat. We are looking down on the surface of a small patch of cortex explored by the neurophysiologist Klaus A l b u s ; each short line indicates the preferred orientation of the column of nerve cells lying beneath that point. (Some of these orientations were obtained empirically, by recording with a microelectrode at that point; the rest were filled in by inspired guesswork.) The pattern has a number of interesting features; first, the preferred orientation usually, but not always, shifts in small steps from one column to all its neighbours; second, each complete cycle of orientations occupies 1 mm or less across the cortex; third, there is an element of lJ
151
273
/ /
/
y — —
—
'
—
• - ~ —
*•
y
/ /
1 \
\ % \ \ \ * \ \ % \ \ \ \ \ \ \ "S. m
—
/ y —' 4 / y / / / / / / / 1/ \ 4 / /
>
1 / / •1 1 •1 \ \
—
--. —
N V
* \ X
* \
^
—
^
y
.
x-
/ / /
/ •
—
"
/
S
/ / ' // * / / M
\ \ 1 s / / / 1 1 \ \ \ 0 / 1 * l \ \ ^ —« / 1 1 \ \
—
l-'ig. 1 7.9.
4 .
^-«e
—
y
—
y / —- y
- 1
*' ^ \ S / /1 V
\
\
\
\
\
\ s H\
\ \
—
;
This diagram, based on experiments by K l a u s H i n t s ' , represents a
small part (about 1mm * 0.5 mm) of the surface or the primary visual cortex of a cat. Each short line shows the angle at w h i c h a n edge must appear in the appropriate part of Ihe visual field to stimulate the liny column of nerve cells lying below l h a l point on the surface. T h e lines w i l h dots In the middle are actual experimental results, showing the preferred orientation of the tirst nerve cell recorded with a microeleclrode introduced into the cortex at that point. T h e other orientations were added on the basis of the experimental observation that, on average, Ihe preferred orientation of adjacent columns of neurons changes by about 1 0 ° fora movement of 51) ,iin across the cortical surface, 'litis piece of visual cortex receives information from a single patch of visual held. T h u s each edge of any object appearing in this region would set up activity in a number of columns of cells 'tuned' lo the appropriate angle.
152
274 randomness, manifest in occasional reversals in sequences of orientation and in sudden large jumps. Compare this pattern with Figure 1 7.10, a graphic exercise by the artist Bridget Riley. She has apparently set herself the same task as that of the map of orientation in the visual cortex, namely representing steady progressive shifts of angle of lines on a two-dimensional surface. And she arrives at much the same solution. My other example of a non-isomorphic map comes from the auditory system. We can use our ears not only to distinguish tones (and hence more complex sounds, including speech) but also to judge the position in space of a sound source - surely an important skill to an animal attacking or under attack. Now, the pitch of a sound, as I have already described, is reflected in the position of maximum vibration in the organ of hearing, and hence the place at which nerve cells are most active in the isomorphic tone map of the primary auditory cortex. But distinguishing the position of a sound source is a much more subtle affair. It depends on a comparison of differences in the intensity and time of arrival of the sound wave at the two ears (for the horizontal position) or of the relative intensities of different components of
•, i
s
v
I
'
•
.
\
I
'
•.
I
I
-v
.fc
* -
N
I
\
v
\
s
-
«.
-
•.
~
'
'
I
'
I
V .
N
f -
-
-
<
*: '
'
t
~
\
»
-
V
I
'
/ '
'
"• • -
—
»
*
<
I
I
/
/
.
-
•
i
I
/
•
,
-
-
\
1
I
I
,
/
~
*
'
-
1
'
'
'
\
I
'
*
-
•* \
|
J
»: > .
t
i
> >
\
%
*
-
\
V
•»
-
\
\
I
H.
V
\ -
.1
./
—
»
•«
%
V
A
-
> *
i l
/ /
1
-
t
-
.*
•*
.
-
'
t.
\t
>
-
-
/
/
-
.>•
>i
*
-
> >
r.
/
•/
i
\
I
l
I
|
/
I
I
I
* -, ..y I
/
>
>
\
I
I
/
/
\
\
I
f
/
\
V, /
•/
I
I,
t
\
I
•/
-
\
\
i
#
, ,
1
•» »
-
S
>•• . -
\
:
> s
-
* '
I
»
•>
»•• ,v- .V. :i
. / • ./
I
_ ,
\
~
.- : v
, /
s
-
-• -
t /
N
v I
-
\ V
\
-
.
r I
•. • " '
>
\
t I
>
.
\
'
\ ~
/ ' V
-
-
if..
«
Fig. 17.10.
~
t-
-
-
-
*
*.
> I.
I t
\
~
>
-
, f J f
i V
,
I /
\ .
'
\
)
i
i
i I
|
|
'
I
/
/
-
i
y'
*
*
"Study for Static' (1966). Pencil on paper by Bridget Riley.
153
275 complex sounds reflected from the crevices of the outer ear (for vertical as well as horizontal position). No simple isomorphism between sound receptors and nerve cells in the brain could create a map of auditory space, yet such maps do exist in the brains of birds and mammals. On the back of the midbrain of mammals (the top of the brainstem, underneath the cerebral hemispheres) there are two pairs of little bumps, the superior and inferior colliculi, both of which have input from the ears. The inferior colliculus has a beautiful isomorphic map of tone frequency (like that in the primary auditory cortex); each individual nerve cell responds preferentially to a particular pitch but is almost insensitive to the position of sound in space. The situation is just the opposite in the deeper layers of the superior colliculus. Here the nerve cells respond best to complex 'natural' sounds rather than pure tones, and most of them need the sound source to be placed at a particular position in space to give their best response. Moreover, the best position shifts progressively from cell to cell, forming a 2-dimensional map of auditory space across this brain structure . 11
It turns out that the superior colliculus is primarily a visual centre, involved in the control of movements or the eyes, head and body. The incoming visual fibres are distributed in an isomorphic map of the visual field, which is neatly in register with the non-isomorphic map of auditory space. And both of these sensory maps are aligned with a motor map, such that activation of one group of nerve cells in this structure (whether through the occurrence of a sound or a visual stimulus at a particular point in space) causes the eyes to move across to look in the direction of that stimulus. A perfect little neuronal machine, whose actions depend on those two, superimposed sensory maps, sharing the same motor output. The construction of a non-isomorphic map of auditory space requires much more than the simple ordered growth of bundles of nerve fibres. In fact, Andrew King and his colleagues at Oxford * have evidence that the map of auditory space in the superior colliculus of the ferret is not purely innately determined, but is partly dependent on interplay between auditory and visual signals early in life, leading to the establishment of an auditory map aligned with the visual one. This kind of construction of a spatial representation that bears a relationship to the world but not to any simple feature of the sense organ seems to me strong evidence that maps, as such, are useful to the brain. 1
Why have maps? The operations performed by a nerve cell are determined not only by its inherent electrical and chemical properties but also by the connections that it receives, just like any component in an electronic circuit. What value could
154
276
Fig. 17.1 1. In the deep layers or the superior colliculus (a visual centre in the brainstem) of mammals there are nerve cells that respond to sounds at particular positions in space, and they are distributed to form a map of auditory space aligned with the visual map formed by input from the eyes. Here we see a ferret viewed from above. The inset diagram represents the surface of the midbrain, consisting of the inferior colliculi (IC). which have purely auditory inpul, and the superior colliculi (SC). The numbered dots show the positions at which nerve cells were recorded in the deep layers. Each cell responded best to sound presented at the position on the opposite side of space, shown in the diagram on the left. Each curve (enclosing a striped area) represents the relative strength of response or the nerve cell for sounds presented at different positions around the animal's head. (Based on results of A.). King and M.E. Hutchings".)
there be i n h a v i n g these u n i t a r y n e r v e cells a r r a n g e d i n topographic order? P e r h a p s it h a s s o m e t h i n g to do w i t h the fact t h a t m o s t of the c o n n e c t i o n s i n the b r a i n a r e short, local fibres, s u c h that cells a r e either excited o r inhibited by t h e i r close n e i g h b o u r s . T h e n a t u r e of a n y c o m p u t a t i o n a c c o m p l i s h e d by s u c h local c o n n e c t i o n s w i l l obviously depend o n the r e l a t i o n s h i p b e t w e e n the properties of neighbouring
n e r v e cells,
155
277 C o n s i d e r the k i n d of processing that is k n o w n to go o n i n the p r i m a r y v i s u a l cortex. I n d i v i d u a l n e r v e cells, in a single c o l u m n , respond to a line of a p a r t i c u l a r orientation. T h i s very property of orientation selectivity s e e m s to depend, at least i n part, o n s h o r t - r a n g e inhibitory c o n n e c t i o n s from cells i n n e i g h b o u r i n g c o l u m n s ( w h i c h respond best to o t h e r o r i e n t a t i o n s ) . A d m i n i s tration of a d r u g that blocks inhibition i n the cortex m a k e s cortical cells partly or completely lose their selectivity for the angle of a l i n e " . S o m e c o r t i c a l cells respond best to a n edge of a p a r t i c u l a r angle that moves across a region of the v i s u a l field a n d they m a y a c h i e v e this property by r e c e i v i n g excitatory c o n n e c t i o n s from a n u m b e r of n e a r b y cells i n the s a m e c o l u m n , w h i c h all prefer lines of a p a r t i c u l a r angle, but e a c h at a slightly different position i n the field. Of course, i n t e r c o n n e c t e d elements in the b r a i n , j u s t as i n a computer, need not necessarily lie close to e a c h other. O n e could i m a g i n e a b r a i n i n w h i c h a l l the cells a r e redistributed r a n d o m l y , but m a i n t a i n i n g a l l the s a m e c o n n e c tions. W o u l d it not w o r k just as well? I n at least one respect it w o u l d not, Nerve fibres c o n d u c t impulses at very slow speeds (between a b o u t 0 . 5 a n d 1 0 0 m / s ) , c o m p a r e d w i t h the speed of t r a n s m i s s i o n i n electronic c i r c u i t r y . So, the inevitable l e n g t h e n i n g of n e r v e fibres introduced by s c r a m b l i n g a b r a i n w o u l d severely slow d o w n the operations of the b r a i n a n d w o u l d introduce differences a n d errors of t i m i n g b e t w e e n the v a r i o u s i n p u t s to e a c h cell, w h i c h m i g h t seriously interfere w i t h the process it performs. A n o t h e r a d v a n t a g e of topography is that it simplifies the problem of getting some other i n p u t into register w i t h the sensory a r r a y . I m a g i n e a v i s u a l a r e a i n the cortex t h a t receives not o n l y a n i n p u t of fibres c a r r y i n g the v i s u a l messages from the eyes forming a m a p of the v i s u a l field but also a m u c h coarser a r r a y of 'activating' fibres that adjust the excitability of n e r v e cells so as to t u r n v i s u a l attention o n o r off selectively i n different parts of the v i s u a l field. ( A c t u a l l y this is not w i l d speculation: almost c e r t a i n l y there a r e s u c h a c t i v a t i n g inputs.) A s long as there is a m a p . a n y other spatially distributed i n p u t to the a r e a will h a v e its influence topologically patterned i n relation to the coordinates of the m a p . If there w e r e a n o n - i s o m o r p h i c m a p i n w h i c h all red objects ( w h a t e v e r their position i n the v i s u a l field) w e r e represented i n one sub-division of the m a p (or all birds, all m i c e , all h u m a n faces, etc.), t h e n a set of 'activating' fibres distributed topographically across s u c h a m a p c o u l d confine attention to one sub-division of experience. W h a t e v e r the m e c h a n ism o r s t o r i n g memories, that loo m i g h t depend o n a system of "this-is-worlhr e m e m b e r i n g fibres' distributed across the m a p s of sensory experience. F i n a l l y there is the o v e r r i d i n g problem of e c o n o m y i n the specification of instructions for building the b r a i n . T h e c h r o m o s o m e s of a h u m a n being,
156
278 which have to be sufficient to specify the structure of the entire body, may contain as few as 50.000 functional genes. Some of the genetic instructions for building the brain may, however, be very simple, such as 'make all nerve cells inhibit their immediate neighbours on either side', or 'link together nearby cells that tend to fire off at the same time'. The theoretical modellers of brain function have shown that such simple rules for local Interaction replicate certain properties of real nervecells (such as selectivity for elongated lines at particular orientations) but only if the input to the network is topographically arranged. Indeed it is a very attractive notion that the basic circuitry of each local 'module' of cortex might be identical, or at least very similar, in every cortical region. The very different functions of different cortical areas may depend mainly on the nature of the inputs received and their topographical distributions. If this is true, the intrinsic circuitry of the whole cerebral cortex may be specified by a surprisingly small number of genes. Meaning from maps? 10
Horace B a r l o w has already argued that the value of maps may be to bring together new associations of activity that can reveal interesting properties of the original image. The idea that the existence of a map in a brain structure allows local circuitry to discover useful relationships between the activity of neighbouring nerve cells may resolve a recent puzzling finding. The somatic sensory area of the cerebral cortex of a fruit bat, the flying fox, has a map of the body surface, like that in other mammals, but with one surprising difference. Figure 17.12 shows, on the left, the typical arrangement, in the somatic sensory cortex of the rat. Both hindlimbs and forellmbs are represented at the front of the strip of cortex, while the animal's back is represented behind. The map in the bat (shown on the right) has the representations of the head, hind limb and back disposed much as in other mammals, but the forelimb (wing) representation lies at the back of the strip. Mike Calford and his colleagues' , who discovered this odd pattern, pointed out that the disposition of the parts of the body in the map mimicks the posture of this animal, which habitually hangs with its wings folded over its back. They inferred that the very existence of this topographic arangement may have meaning for the animal in terms of its body posture. But how could this be without a Cartesian inner eye to look at it? A n alternative possibility is that the arrangement allows local interactions between signals from parts of the body surface that usually lie close to each other and are therefore likely to be touched simultaneously or sequentially. Imagine, for instance, a branch brushing against the bat as it hangs in a tree. It is likely to touch both the fur of the animal's back and its wing 6
157
279
B
A
Rat
Bat
Pig. 1 7.12. There are certain fundamental similarities in the nature of the map iif the budy of the somatic sensory cortex in virtually all mammals so far studied. The somatic cortex ties in the middle of the cerebral hemisphere, wilh the head represented al the bottom of the strip (see Pig. 17.7). The diagram on the left shows the typical arrangement of the body map on the left side of the hemispheres of a rat. The representations of the Torelimbs and the hindlimbs point forwards. In the fruit bat, however, Mike Calford and his colleagues" found that there is a small but significant variation on this general theme. As shown on the right, the representation of the forelimb (the wing) points backwards, close lo the representation of the animal's back. In other words, the map reflects the habitual posture of the bat as il roosts, hanging with its wings folded over its back.
successively.
P e r h a p s local interconnection between the closely
spaced
regions of cortex representing these t w o regions of s k i n a l l o w s some n e r v e cells to detect a n d respond selectively to s u c h a s t i m u l u s . Multiple sensory m a p s 1 h a v e already mentioned that e a c h p r i m a r y sensory a r e a is s u r r o u n d e d by additional areas, i n w h i c h the s a m e basic sensory i n f o r m a t i o n m a y be re-cast o n different spatial coordinates. For instance, in the auditory region of the cortex of the m o u s t a c h e d bat
158
280 (which uses echo-location to guide its flight and hunting), the primary area has a simple (isomorphic) map of tone frequency, with a much enlarged representation (an auditory 'fovea') for frequencies around the strongest harmonic component of the bat's own echolocating cries. This area is flanked by others in which important functional characteristics of the sound are mapped out. One consists of nerve cells that respond only when the locating cry is followed by a delayed echo, and they are arranged in a map in which the required delay (in other words the required distance of a target) varies across the cortex. Another area contains cells that respond preferentially to the first harmonic of llie emitted cry plus anollier pitch, slightly shifted from elllier (lie second or the third harmonic. These cells would respond best as the bat approaches a target which reflects an echo that has all its harmonic components Doppler-shifted to a slightly higher frequency (like the siren of a police car as it approaches you). In this area there are two maps, each with one axis corresponding to a small range of frequency around one of the higher harmonics. This arrangement results in the velocity of approach being represented systematically along each sub-area". The monkey's cerebral cortex is now known to be filled with a profusion of sensory areas"- " . F i g u r e 17.13 gives a 1985 version of the cerebral atlas of the rhesus monkey" (though new areas are being discovered all the time). The many separate visual areas contain complete or partial representations of the visual field but the assumption is that each of them is serving a special purpose in the interpretation of the visual image. Semir Zeki at University College London has evidence that one of them, the fourth visual area (V4), is especially interested in the colour of visual stimuli , and another one (V5 or 'MT'), described by Tony Movshon in Chapter 8, is concerned with the analysis of movement in the visual scene. 16
Interestingly, the number of such additional cortical areas appears to have increased up the evolutionary scale. A monkey may have as many as twenty extra visual areas, whereas a mouse probably has only two. Indeed, the ballooning expansion of the cerebral hemispheres that has occurred during mammalian evolution may have been achieved largely by the addition of more and more sensory areas, each with its own map of a particular sensory property or properties. The conventional wisdom of comparative psychology is that the intelligence of animals (i.e. the richness and sophistication of their behavioural repertoires) is correlated with the size of their cerebral hemispheres. Could it be that each additional sensory area gives a quantal increment of understanding of the world? It is surely significant that visual areas occupy more than half of the entire surface of the cerebral hemispheres of a monkey. For we now know that,
159
281
Fig. 1 7 . 1 3 .
One of Ihe mast exciting discoveries of the last few years is that there
are m a n y different sensory and motor areas in the cerebral cortex, In the monkey brain, illustrated here, a large number of visual areas fill up more than half of the entire surface of the cerebral hemispheres. T h e small diagram at the top left is a view or ihe right side of the cerebral hemispheres of a monkey, showing the major folds and creases, with colours representing the various areas visible on the surface. The rest of this diagram is a n imaginary unfolded view of the entire cortical surface, revealing many other areas that are buried out of sight i n the clefts of the cortex. Motor areas appear grey, somatic sensory areas blue a n d auditory areas green. All the other colours, shades of red and yellow, represent individual areas concerned with vision. Some of these are thought (o be specialized for the analysis of particular aspects of the visual s c e n e - m o t i o n i n the area 1
labelled M T , form and colour in V 4 " . (Kindly supplied by David V a n E s s e n " ) ,
a l t h o u g h v i s i o n feels as if it is a s i m p l e a n d effortless process, it is e x t r a o r d i n a r i l y c o m p l i c a t e d i n c o m p u t a t i o n a l t e r m s . W r i t i n g c o m p u t e r p r o g r a m s to do t h i n g s t h a t w e t h i n k of a s b e i n g intellectually d e m a n d i n g , s u c h a s p l a y i n g c h e s s or d o i n g a d v a n c e d c a l c u l u s , h a s proved r e l a t i v e l y s i m p l e c o m p a r e d w i t h the m o n s t r o u s p r o b l e m of m a k i n g a m a c h i n e t h a t c a n see.
160
282 The ultimate enigma of perception After all of this, how do we stand with Wittgenstein? There are topographically arranged sensory areas in the brain. Some of them have spatial coordinates that bear no simple relationship to the pattern of sensory receptors. In these terms, the brain does contain images of the outside world and it seems likely that these images arc of functional value in Ihe brain's taskof analysing sensory signals. With deference to Wittgenstein, is it really misleading to call such things maps? We are making slow but steady progress towards an account of how the structure and organization of the brain allow the owner of that brain to respond appropriately to the visual world. One day we might even be able to build a machine that simulates vision by having the same kind of structure as the brain. But will it see? Will it have conscious awareness of its visual world? According to Wittgenstein, it is appropriate to use the language of the mental only when talking of people or of other animals (or even, presumably, machines) whose behaviour is indistinguishable from that of human beings. As far as the privy world ofconscious perception is concerned, this argument is flawless. Indeed one could go further: how can one be truly confident about the nature of the perceptual experiences of any creature but oneself? The nature of perceptual experience is rightly a question for philosophy (see Chapter 22 by Nelson Goodman). There is every reason for philosophers to be interested in such classical conundrums as when 1 say that I see red and you say that you see red, are we having the same experience?'; and 'is "pain" a property of noxious stimuli or a creation of the conscious mind?' But the mechanisms by which a creature understands the world around it (i.e. responds appropriately to things and events) is surely a matter for empirical investigation. The brain is a machine that accomplishes behaviour. The challenge faced by brain research is to provide a plausible account, backed by experimental evidence, of the workings of this machine. It should be no surprise to find that science - especially a branch of science that is rapidly throwing up new discoveries - runs out of words. Everyday language is an invention to deal with everyday events. It works well as a means of communication between people who share a common view of the world and of their own conscious experiences and motives. It is, in its simplest form, at the level of individual words, a convention of categorical descriptions. Therefore language, even everyday language, has grown and changed as new views about the world have developed and as the need for new categories has arisen. The thing I am sitting/kneeling on, as 1 write this, is neither a chair nor a stool. It is one of those new-fangled 'contraptions for sitting/kneeling on', which is supposed to be good for your back. I need a new word for it but I carry on calling it a chair, until a better noun comes along.
161
283 Science is constantly generating new categories and new notions, beyond previous experience. Sometimes it invents neologisms for them (Supernova, protoplasm, quark, molecule, entropy, isotope, gene); sometimes it sticks to everyday words, beating them into new meanings (Black Hole, mass, nucleus, power, cell, current, inertia). Is it really a greater conceptual confusion for brain researchers to call the distribution of activity in the visual cortex a map' than for me to call the thing that I am sitting/kneeling on a 'chair' (or to call the black birds that swim on the Swan River in Western Australia 'swans')? One of the most profoundly enigmatic aspects of science is that it often has to use everyday language to formulate questions and concepts concerning a world beyond everyday experience. It is true that some areas of intellectual effort (such as mathematics, logic and music) have devised new systems of notation because everyday language has proved inadequate as a medium of communication of questions and ideas in these disciplines. But most areas of science stumble along, using ordinary language to 'bootstrap' themselves up to new concepts. Nowhere is the problem of language greater than in brain research, but the difficulty is not so much a deep conceptual confusion as an inadequacy of vocabulary and notation. Ultimately, it should be the objective of brain research to provide an account of the function of the nervous system that even includes the mechanism of consciousness. But that is. I fear, a long way off (not least because the everyday language of consciousness gives little clue as to how to formulate questions about its mechanism). In the meantime, it would be a major achievement to give an empirical description of the mechanisms in the brain that allow animals and people to respond as if they understand the visual world.
162
Notes to Chapter 17 1. Wittgenstein, L . (1953), Philosophical Investigations. Basil Blackwell, Oxford. 2. Kenny A . (1984), The Legacy of Wittgenstein. Basil Blackwell, Oxford. 3. Descartes, R. (1637), La Dioptrique (see Philosophical Writings transl. and ed. E . Anscombe & P.T. Geach (1954) London, Nelson). 4. Descartes. R. ( 1 6 4 9 ) , Les Passions de I'Ame (see The Philosophical Works of Descartes I, transl. & ed. E.S. Haldane & G.T.R. Ross ( 1 9 1 1 ) Cambridge University Press. Cambridge). 5. T h e minutes of the (one. 1 9 8 6 . meeting of the steering committee fur the SDF Symposium on Computational Neuroscience were prepared a n d circulated by Professor Eric S c h w a r t z of the New York University Medical Center in September 1986. 6. Y o u n g . J.Z. ( 1 9 7 8 ) , Programs of the Brain. Oxford University Press, Oxford. 7. Hacker. P. ( 1 9 8 7 ) . Languages, brains and minds, i n Mindwaves. ed. C . Blakemore & S. Greenfield. Basil Blackwell. Oxford. 8. Woolsey, T . A . & V a n der Loos. H . ( 1 9 7 0 ) . T h e structural organization of layer IV in the somatosensory region (SI) of mouse cerebral cortex. T h e description of a cortical Held composed of discrete cytoarc hi tec tonic units. Brum Res., 17, 2 0 5 - 4 2 . 9. Kossut. M. & Hand. P. ( 1 9 8 4 ) . Early development of changes in representation of vibrissae following neonatal denervation of surrounding vibrissae receptors; a 2-deoxy glucose study in the rat. Neuroscience Letters, 4 6 , 7 - 1 2 . 10. Barlow, H . B , ( 1 9 8 1 ) . Critical limiting factors in the design of the eye and visual cortex. T h e Ferrier lecture 1 9 8 0 . Proc. Roy. Soc. B 2 1 2 , 1-34. Barlow, H.B., ( 1 9 8 6 ) . W h y have multiple cortical areas? Vision Research. 26, 81-90. 11. Hubel, D . H . & Wiesel, T . N . ( 1 9 7 7 ) . Functional architecture of macaque monkey visual cortex. Proc. Roy Soc. B 1 9 8 , 1 - 5 9 . 12. Albus, K . ( 1 9 7 5 ) . A quantitative study of the projection area of the centra! and the paracentral visual field In area 17 of the cat. II. T h e spatial organization of the orientation domain. Experimental Brain Research, 2 4 . 1 8 1 - 2 0 2 . 1 3. Harris, L . R . . Blakemore, C . & Donaghy, M.J. ( 1 9 8 0 ) . Integration of visual a n d auditory space in the m a m m a l i a n superior colliculus. Nature. 2 8 8 , 5 6 - 9 . King, A . j . & Hutchings. M.F.. ( 1 9 8 7 ) . Spatial response properties of acoustically responsive neurons in the superior colliculus of the ferret: a map of auditory space. /. Neurophysiol., 57, 5 9 6 - 6 2 4 . 14. King, A . ] . . Hutchings. M.E.. Moore, D.R. & Blakemore. C . ( 1 9 8 7 ) . Developmental plasticity i n the representations of visual a n d auditory space. Nature, 3 3 2 , 7 3 - 6 . i 5. Rose, D. & Blakemore. C. ( 1 9 7 4 ) . Effects of bicuculline on functions of inhibition in visual cortex. Nature. 2 4 9 . 3 7 5 - 7 . Sillito, A . M . ( 1 9 8 4 ) . Functional considerations of the operation of GABAergic inhibitory processes in the visual cortex. In The Cerebral Cortex, vol, 2 A , ed. A , Peters & E . G . (ones. Plenum Publishing Corporation. New York. pp. 9 1 - 1 1 7. 16. Calford, M.U., Graydon. M.L., Huerta. M.F., Kaas, J.F. & Pettigrew, J.D. ( 1 9 8 5 ) . A variant nf the m a m m a l i a n somatotopic map In a bat. Nature. 31 i. 4 7 7 - 9 . 17. Suga. N . . Kuziral. K . & O'Neill, W.F.. ( 1 9 8 1 ) . How biosonar information is represented in the bat cerebral cortex. I n Neuronal Mechanisms of Hearing, ed. J. Syka & L . Aitkin. Plenum Publishing Corporation, New York. 18. Zeki, S. ( 1 9 8 3 ) . Colour coding in the cerebral cortex: the reaction of cells in monkey visual cortex to wavelengths a n d colours. Neuroscience. 9, 7 4 1 - 6 5 . 19. V a n Essen, D . C . (1985). Functional organization of primate visual cortex. In Cerebral Cortex, vol. 3, ed, A . Peters & E . G . Jones. Plenum Plublishing Corporation, New York.
163
Chapter 2
BIOLOGICAL CONCEPTS AND METHODS; C O M P U T A T I O N A L GOALS AND MEANS
This chapter introduces notions that a physicist, or any other outsider, is advised to learn about, before entering the field of biocomputation, if he wishes to avoid looking like a beotian or a tourist. In contrast with the previous chapter, where half of the author's names would be familiar to the average physicist, the authors in this chapter may well be unknown to him. But the newcomer should soon recognise in the late Canadian psychologist Hebb a major figure. Together with the two American psychologists Miller and Shepard, he contributed to the demise of behaviourism that had dominated psychology in the North American continent, and to the advent of the so-called "cognitive revolution" in the fifties and the sixties. The notion of mental representations, the suggestion that they might be embodied in the brain as cell assemblies, and the psychophysical experiments on mental images, constitute some of the main foundations of cognitive sciences. These foundations are evoked in section 2a, along with some recent investigations of their neurophysiological substrate. Section 2b is devoted to information theory as well as to an evaluation of its strong impact on psychology and brain theory. Quantitative estimates of sensory information flow and of the span of short-term memory, and emphasis put on the attention bottleneck and on the need for redundancy reduction, belong to this scientific tradition originating in the forties. Miller's "Magical number 7-plus-or-minus-2" a n d Barlow's "Cardinal cells"
164
are so often quoted and misquoted, that it is both a surprise and a pleasure to rediscover their seminal and masterful papers. T h e theoretical tension between Hebb's cell assembly and Barlow's sparse coding is still a major issue in neurocomputation. "Reading the structure of brains", as well as inferring the "how" and "why" of brain areas from neuroanatomy, is a challenging exercise evoked in section 2c. It requires qualities of "esprit de finesse", unfamiliar to physicists. But this synthetic approach is essential to brain theory. I n brief, the three articles of section 2d are mainly united by their differences. They illustrate quasi-orthogonal viewpoints. T h e immunologist Jerne presents the old debate between the concepts of selection and instruction — a tension pervasive in biological thought — with a sharpness that has been too often missing in the follow-up literature. The psychobiologist Julesz, creator of the random-dot stereograms, reviews early vision, pre-attentive and attentive. T h e neuropsychologist Weiskrantz is representative of the ancient branch of neuropsychiatry, that has been drawing precious information from lesions of the human brain.
165
2a Mental Representations
The mental representation of a percept or a concept in the brain is a central theme of cognitive science which will recur time and again throughout this volume. The three texts of this section report on three complementary fronts of research: psychology, psychophysics and neuroscience. The first paper (15) by Hebb focuses on imagery, i.e. the production or processing of mental images without sensory stimuli, which is among all thought processes the one that appears to be most easily accessible to scientific investigations. This paper has been chosen, because it was written shortly before the landmark experiments of Shepard and Metzler {article (16)); it illustrates beautifully how far psychology was able to go, using a mixture of scattered observations and educated intuitions. Some twenty years earlier (in 1949), Donald Hebb had written a book, The Organisation of Behaviour, that has since remained a classic, and that introduced two essential ideas for the field of neurocomputation: the notion that a percept or a concept is represented in the brain by an assembly of cells firing together, and the so-called Hebb learning rule for synaptic modification. These prophetic paragraphs have been so often reproduced elsewhere that we did not feel useful to do it again here. Article (15) offers a distinction between sensation and perception, and it emphasises the importance of the motor component in perception. Moreover, this article presents an additional value, in that it tries to relate Hebb's early intuitive ideas about cell assemblies to the then recent (in the sixties) neurophysiological findings of Hubel and Wiesel. T h e reader will discover many intriguing reports and speculations about phantom limbs, hallucinations, eidetic and hypnagogic imagery, and will also enjoy the free style of a great scientist. In contrast, the letter from Shepard and Metzler (article (16)) may appear as a dry technical report on reaction time measurements. But its method and findings played a key role in proving beyond all doubt that scientific investigation of mental phenomena was possible, and this paper is at the origin of the "mental rotation paradigm" in cognitive psychology.
166
T h e basic finding is that the time needed to recognise that two pictures are rotated aspects of the same object, is proportional to the angle of rotation. W h a t may come as a surprise is the fact that this recognition time does not depend on the orientation of the axis of rotation in space. Intuitively one could expect that mental detection of a rotation within the picture plane should be easier. T h e third paper (17), written about twenty years later, reports on electrode recordings from the motor cortex of monkeys which were trained to perform a specific task. Somehow, it bridges the gap between Hebb's intuitions about cell assemblies and Shepard-Metzler's psychophysical measurements on mental rotations. It shows that the cognitive operation of mental rotation may be represented by a varying activity pattern of a population of directionally tuned neurons. T h e weighted sum of their preferred directions defines the "neural population vector", which rotates through the shortest angle from the initial to the final position in the task.
167
Reprinted wilh permission from Psychological Review, Vol. 75, No. 6, pp. 466-477,1968 © 1968 American Psychological Association
CONCERNING IMAGERY ' D. O. H E B B McGill University, Montreal, Canada An attempt is made to analyze imagery in physiological terms. It is proposed (a) that eye movement has an organizing function, (_b) that Ist-order cell assemblies are the basis of vivid specific imagery, and (c) that higher-order assemblies are the basis of less specific imagery and non re presentational conceptual processes. Eidetic images, hallucinations, and hypnagogic imagery are compared with the memory image, and certain peculiarities of the memory image are discussed. T h i s paper concerns the content and mechanisms of imagery. T h e topic has received only sporadic attention, partly because of the positivistic temper of modern psychology and partly, one may suppose, because of the difficulties of dealing with thought processes in general. I propose to see what sort of analytical treatment can be made of the image and, equally, of its relation to sensation, perception, and thought. The occasion for such treatment is mainly my interest in thought—one can hardly turn round in this area without bumping into the image—but also the recent work on the place of imagery in paired-associate learning (Paivio, in press) and the convincing demonstrations of eidetic imagery made by Haber and his colleagues (Haber & Haber, 1964; Leask, Haber, & H a b e r ) . I have also in mind the hallucinatory activity reported in conditions of monotony, perceptual isolation, and loss of sleep (Bexton, H e r o n , & Scott, 1954; Melvill Jones, cited by Hebb, 1960, p. 741; Malmo & Surwillo, 1960; Morris, Williams, & L u b i n , 1960; Mosely, 1953). 1
1
Preparation of this paper was supported by the Defence Research Board ot Canada, Grant No. 9401-11. • J . Leask, R. N. Haber, and R . B. Haber. Eidetic imagery in children: I I . Longitudinal and experimental results, in preparation.
T H E P L A C E O F IMAGERY I N O B J E C T I V E PSYCHOLOGY
Let me first dispose of what seems to be a misconception, that reporting imagery, or describing it, is necessarily introspective. T h e point has been made elsewhere (Hebb, 1966) but I repeat it here for those not addicted to introductory textbooks. A n excellent example to begin with is the phantom limb, which is clearly a case of somesthetic imagery. After an arm or leg has been amputated there is, apparently in every case (Simmel, 1956), a hallucinatory awareness of the part that has been cut off. I n some 10-15% of the cases the patient also reports pain, the fingers or toes being curled up with cramp. Is this a report of introspection? T h e argument might be: T h e pain is in the right hand, but the patient has no right hand; so the pain is really in his m i n d ; so he is describing his mental processes, which is introspection : "looking inward." But the argument is faulty. W e are still dealing with a mechanism of response to the environment, though the mechanism (because a part is missing) is now functioning abnormally. Figure 1 represents a right hand connected schematically with brain and speech organs, before amputation. When the fingers are burnt or cramped the subject (S) says " O u c h " or "My hand hurts." T h i s is a normal mode of
168
467
response to the environment, involving (a) sensory input, (b) excitation of the central processes of perception and consciousness, and (c) motor output determined by the central activity. It is obvious that in such reactivity— when I burn my fingers and say "Ouch"—no question of looking inward arises. My verbal response is no more dependent on introspection than a dog's yelp when his tail is trod on. The same conclusion holds after an amputation. No excitation can originate in the missing hand, but the same excitation in principle can arise higher in the pathway by spontaneous firing on the neurons at level X in Figure 1. U S now reports pain in his imaginary or imagined hand we are not dealing with any different mechanism, in brain function, than when a normal 5" reports pain. Report of "sensation" from a phantom limb is not introspective report. The ordinary memory image can be understood in much the same way. The central processes here may be excited associatively (i.e., the cell assemblies are excited by other assemblies instead of spontaneouslyfiringafferent neurons), but in both cases we are dealing with a short circuiting of a sensory-perceptual-motor pathway. The .£ on holiday, seeing the ocean for the first time, remarks on tbe size of the breakers; reminded of the scene later he may say, "I can still see those waves." Though there is now no sensory input, the same central process, more or less, is exciting the same motor response—more or less. (What the differences may be we will consider later.) It is the same outward-looking mechanism that is operative, not introspection. At least, it is not introspection in the sense of a special in ward-looking mechanism of self-knowledge. Any-
F i a . 1. To illustrate the relation between normal sensation and the phantom limb.
one may define the term to suit himself, and may use it when reports of private events such as endogenous pain and imagery are in question. My point is that such report does not transcend the rules of objective psychology, in which mental processes are examined by inference and not by direct observation. The primary basis of inference is the relation of overt behavior to present and past stimulation, but there is also a basis of inference about oneself from the appearance of the external world: I may, for example, conclude that I am color-blind if surfaces that others call green and red look alike to me. I also make inferences about the functioning of my visual system when I oh serve positive and negative afterimages though my eyes are shut. It is important to say also, with regard to a report of imagery, that one is not describing the image but the apparent object. This becomes clear if one observes the apparent locus of what one is describing. One does not perceive one's perceptions, nor describe them; one describes the object that is perceived, from which one may draw
169
468 eye, head, or hand to see, hear, or feel better; ( c ) the resulting feedback; (d) further motor output, further feedback, and so on. A s we will see later, this is not a trivial point but must affect our understanding both of percept and of image.
inferences about the nature of the perceptual process. I n the case of imagery, one knows that the apparent object does not exist, and so it is natural to think that it must be the image that one perceives and describes, but this is unwarranted. T h e mechanism of imagery is an aberrant mechanism of exteroception, not a form of looking inward to observe the operations of the mind. So understood, the description of an imagined object has a legitimate place in the data of objective psychology. WORKING
Physiologically there is a discontinuity in the mode of operation of the afferent pathway to the sensory cortical area and the structures that lie beyond. T h e afferent transmission is highly reliable, whereas cortico-cortical transmission, the higher activity that includes perception as defined, occurs only in favorable circumstances. A n evoked potential in sensory cortex is obtainable in coma or under anesthesia, but any transmission past this point is not sufficient to break up the synchronous E E G activity. T h u s "anesthesia," meaning literally a lack of sensation, is a misnomer; we are dealing instead with a failure of transmission at a higher level.
DEFINITIONS
I n what follows it will be necessary to distinguish between sensation and perception, without supposing that there is a sharp separation between them. T h e distinction is based primarily on physiological considerations but the psychological evidence is in agreement. Sensation is defined here as the activity of receptors and the resulting activity of the afferent pathway up to and including the cortical sensory area; perception as the central (cortico-diencephalic) activity that is directly excited by sensation as defined. F o r the purposes of this analysis, then, sensation is a linear input to sensory cortex, perception the reentrant or reverberatory activity of cell-assemblies lying in association cortex and related structures. The term perception itself has two meanings in ordinary usage. Which of the two is intended is usually clear from the context, but when necessary I distinguish perceivintj, as the process of arriving at a "perception," from a percept, the end product, the brain process that is the cognition or awareness of the object perceived. Except with very familiar objects, perceiving is not a one-stage, single-shot affair. It usually involves (a) a sensory event; ( 6 ) a motor output, the adjustment of
A s Teuber ( I 9 6 0 ) has pointed out, perception cannot be identified with an activity of sensory cortex, so the physiological basis of a distinction between sensation and perception is clear. Sensory systems are organized with fibers in paratlel. providing for lateral summation and hence reliability of transmission at each synaptic junction. T h e divergent course of fibers from sensory cortex onward lacks this feature, and transmission here requires supporting; facilitation from the brainstem arousal system, which is absent under anesthesia. T h e selectivity of response to sensory stimulation even in the normal conscious state strongly indicates that supporting facilitation is also needed from the concurrent cortical activity; except when there is a sharp increase of arousal, due to pain stimulation or certain unfamiliar events, we "notice," perceive, or respond to only those events in the nor-
170
469
mal environment that are related to what we are thinking about at the moment. Finally, another relevance of the distinction between sensation and perception from a psychological point of view is the fact that different sensations or sensory patterns can give rise to the same perception, as in the perceptual constancies; and the fact that the same sensory pattern can give rise to quite different perceptions, as in the ambiguous figure (even with fixation of gaze). In this latter case, the only explanation that has been given is that different cell assemblies are excited by the input at different times.
With the printed word before him, spelling the word backward can be done nearly as quickly as forward, but this is not true of the image and the who tries such a task for the first time is apt to be surprised at what he finds. There is a sequential left-toright organization of the parts within the apparently unitary presentation, corresponding to the order of presentation in perception as one reads English from left to right and from top to bottom.
T H E P A T T E R N OF A C T I V I T Y
Both the ordinary memory image and the eidetic image arise from perception. As we will see, this does not mean that the memory image is identical with perception (though eidetic imagery may be), but it does have implications that have not been recognized. The percept of any but tbe simplest object cannot be regarded as a static pattern of activity isomorphic with the perceived object but must be a sequentially organized or temporal pattern. The same statement, it seems, applies to the memory image. This has been well established for the image of printed verbal material (Woodworth, 193S. p. 42, citing Binet and Fernald). The S with good visual imagery may be asked to form an image of a familiar word of medium length ("establish" or "material" would be suitable). When he has done so, he is asked to read off the letters backward ; or if 5 is one who reports that when he has memorized verse he can see the words on the page, lie may be asked to recall a particular stanza and then to read ihe last words of each line going from bottom to top.
Something of the same sort applies with imagery of nonliteral material, though now the order of "seeing" or reporting is less rigid. If the reader will form an image of some familiar object such as a car or a rowboat he will find that its different parts are not clear all at once but successively, as if his gaze in looking at an actual car shifted from fender to trunk to windshield to rear door to windshield, and so on. This freedom in seeing any part at will may make one feel that ail is simultaneously given: that tbe figure of speech of an image, a picture "before the mind's eye," in the old phrase, does not misrepresent the actual situation. But Binet (1903) drew attention to a surprising incompleteness in certain cases of imagery, which suggests a different conception. Let us consider the question more closely.
171
First, consider the actual mechanics of perceiving a complex visual object that is not completely strange but not so familiar that it can be fully perceived at a glance. Figure 2 represents a slightly off-beat squirrel or chipmunk. The eye movements made in perceiving it must vary, but assume that there are four points of fixation, A, B, C, and D. After fixating these points, perhaps repeatedly, the object is perceived with clarity: one percept is arrived at. But how are the separate visual impressions integrated ?
470 tions were reinstated at the same time. T h e effect must be the same as if, in perception, one saw four copies of Figure 2 superimposed to make Points A, B, C, and D coincide, in a mishmash of lines. Instead, Activities A, B, C, and D must be reinstated one at a time, the transition from one to the next mediated by a motor activity corresponding to the appropriate eye movement.
FIG. 2. To illustrate the role of eye movement in perception and imagery. (A, B, C, D, fixation points.) W e must take account of the fact that these four part-perceptions are all made in central vision, more or less on top of each other, though they are separated in time by eye movements. E a c h of the four is an excitation of a small group of cell-assemblies, which I will call for the moment Activity A, Activity B, and so on. These activities must take place in the same tissues, more or less intertwined. Activity A is separated from a following Activity B by an eye movement to the right and slightly downward; if Activity D occurs next, it is preceded by an eye movement downward and to the left; and so on. These movements are mechanically necessary in scanning the object, but they may have a further role. I n other words, the motor process may have an organizing function in the percept itself and in imagery. The image is a reinstatement of the perceptual activity, but consider the result if all four of the separate part-percep-
W h e n looking at the actual object each part-perception is accompanied by three motor excitations (assuming these four fixation points) produced by peripheral stimulation. One of them becomes liminal and the result is eye movement followed by another partperception. If the image is a reinstatement of the perceptual process it should include the eye movements (and in fact usually does); and if we can assume that the motor activity, implicit or overt, plays an active part we have an explanation of the way in which the part-images are integrated sequentially. I n short, a part-image does not excite another directly, but excites the motor system, which in turn excites the next part-image. T h a t there is an essential motor component in both perception and imagery was proposed earlier (Hebb, 1949, pp. 3 4 - 3 7 ) with some informal supporting observations that as far as I can discover are still valid. It is easy to form a clear image of a triangle or a circle when eye movement is made freely (not necessarily following the contours of the imagined figure), harder to do with fixation of gaze while imagining the eye movement, but impossible if one attempts to imagine the figure as being seen with fixation on one point. Though such informal evidence cannot carry great weight, it does agree with the idea that the motor accompaniments of imagery are not adventitious but essential.
172
471 A B S T R A C T I O N AND
HIGHER-ORDER
ASSEMBLIES
One of the classical problems of imagery is the generality of the image. Another is its relation to abstract thought. A hypothetical clarification of such questions emerges from a consideration of the relation of secondary and higher-order assemblies to primary ones. T h e r e is a classical view going back to Berkeley that an image must be of a specific object or situation and cannot have generalized reference, but Woodworth (1938, p. 4 3 ) cites an early paper by Koffka to the contrary, and it seems that the view is more a consequence of theory (regarding the image as reinstated sensation) than of observation. But how can an image be general, or abstract? Again, Binet (1903, p. 124) reports an opposition between thought and imagery. The image, a representative process, is by definition an element in thought: H o w can we understand such an opposition ? T h e present status of the theory of cell assemblies is paradoxical, since it has a way of leading to experiments that both support and disprove it. A n impressive confirmation from the phenomena of stabilized images ( P r i t c h ard, H e r o n , & Hebb, 1960) is matched by quite definite evidence, from the same set of experiments, that the theory is unsatisfactory as it stands (Hebb, 1963). Some of the difficulty for the treatment of perception becomes less with the proposal of Good (1965) that an assembly must consist of a number of subassemblies that enter momentarily now into one assembly, now into another. T h e assembly itself need no longer be thought of as all-or-nothing in its activity. Fading, for example, may be a function of the density of subassemblies active in a given region, and a strong stimulation may excite all the subassemblies that are available for a
173
given assembly while a weaker stimulation excites fewer of them. A subassembly, conceivably, might be as small as one of Lorente de No's closed loops consisting of only two or three neurons. It is, however, another aspect of the theory that concerns us now. T h i s is the varying degree of directness of relation between a sensory stimulus and an assembly activity. T h e old idea that an image must always be of a specific object was the result of thinking ( a ) that an image is the reinstatement of a sensory-central process, and ( 6 ) that the central part of the process corresponds exactly to the sensory stimulation. T h e epochal work of Hubel and Wiesel (epochal certainly for understanding perception) shows that this may be true for some components of the central process but is not true for others. A "simple cell" in the cortex responds to a specific retinal stimulation, its receptive field permitting little variation; but "complex cells" respond to stimulation in any part of their larger receptive fields, upwards of half a degree of visual angle in extent ( H u b e l & Wiesel, 1968). A subassembly made up of simple cells, or controlled by them, will thus be representative of a very specific sensory event, but one made up of complex cells will incorporate in itself some degree of generalization or abstraction. Assembly activities accordingly may be more or less specific as perceptual or imagina! events. The superordinate assembly (Hebb, 1949) takes the process of generalization or abstraction further. T h e primary or first-order assembly is one that is directly excited by sensory stimulation. T h e second-order assembly is made up of neurons and subassemblies that are excited, farther on in transmission, by a particular group of primary assemblies ; the third-
472 order made up of those excited by second-order assemblies. T h e theoretical idea is very similar to what Hubel and Wiesel have demonstrated experimentally for simple, complex, and hypercomplex cells; simple cells being those on which a number of retinal cells converge, complex cells those on which simple cells converge, and hypercomplex those on which complex cells converge. F r o m this it may be concluded that the first-order assembly is predominantly composed of or fired by simple cells.
nizations, and many more "ideas" {considered as temporary organizations of neuron groups) are possible than the total number of neurons in the brain; an ideational element may be a phase in a constant flow of changed groupings of neurons. The second point is that there are limits on the process of elaboration. A secondary assembly may be the limit of capacity of the rat brain as far as triangles are concerned, for all the complexity of that small brain. T h e tertiary assembly, it seems, calls for a bigger brain. T h e rat is doubtfully capable of perceiving a triangle as a whole, even when it has a fixed orientation, and is not capable of recognizing a triangle when it is rotated from the position in which he was trained to respond to it. T h e young chimpanzee, however, or the 2-year-old child, recognizes the rotated figure easily (Gellerman, 1933).
A n artificial example to make this specific: L e t us say an infant has already developed assemblies for lines of different slope in his visual field. H e is now exposed visually to a triangular object fastened to the side of his crib, so he sees it over and over again from a particular angle. Looking at it excites three primary assemblies, corresponding to the three sides. A s these are excited together, a secondary assembly gradually develops, whose activity is perception of the object as a whole—but in that orientation only. I f now he has a triangular block to play with, and sees it again and again from various angles he will develop several secondary assemblies, for the perception of the triangle in its different orientations. Finally, taking this to its logical conclusion, when these various secondary assemblies are active together or in close sequence, a tertiary assembly is developed, whose activity is perception of the triangle as a triangle, regardless of its orientation.
Another example: T h e baby repeatedly exposed to the sight of mother's hand in a number of positions would develop subassembly and assembly activities corresponding to perceptions of parts of the hand, and then the whole hand, as seen in these varied orientations. A s the hand is seen in motion, these assemblies would be made active in close sequence. T h e i r combined effects, at a higher level in transmission, would be the basis for forming a higher-order assembly whose activity would be the perception of a hand irrespective of posture.
H o w realistic is this proposal of complex processes developing further complex processes, in brain function? A heavy demand is made on the brain in the large number of neurons needed for what seems a simple perception. T w o comments are in order. As Lashley (1950) observed, the same neuron may enter into different orga-
Although with present knowledge such proposals must be made in very general terms, they are not unreasonably complex in the light of the anatomical and physiological evidence that is available, and they do offer an approach to the otherwise mysterious abstractions and generalizations of thought. Human thought consists of
174
473 abstraction piled on abstraction, of generalizations themselves based on generalizations, and if we are to accept the notion that thought is an activity of the brain we must explore the speculative possibilities of how this may occur. T h e actual perception of an object, following this line of thought, i n volves both primary and higher-order assemblies. T h e object is perceived both as a specific thing in a specific place with specific properties, and as generalized and abstracted from—but not all of this simultaneously. In imagery, only part of this activity may be reinstated. First-order assemblies, directly excited by sensation, must be an essential feature of perception, but need not be active when the excitation comes from other cortical processes. A memory image, that is, may consist only of second- and higher-order assemblies, without the first-order ones that would give it the completeness and vividness of perception. E I D E T I C IMAOERY W e are now in a position to consider a hypothesis of the nature of eidetic imagery. T h e eidetic image has been regarded with skepticism, I think, because as described it seems to have incompatible characteristics of both afterimage and memory image. Its occurrence only after stimulation, its transience, and its vividness in detail make it seem like an afterimage; but its apparent independence of eye movement, its failure to move as the eyes move and the possibility of "looking at" its different parts, to see them with equal clarity, means that it cannot be an afterimage. T o the skeptic it sounds like an image that has got stuck to the viewing surface, which is unlikely to say the least. Part of the difficulty of understanding disappears if we first assume, with Allport ( 1 9 2 8 ) ,
that the eidetic image is in the same class with the memory image, and if we then recognize that eye movement has a positive integrating role in memory images. N o w the scanning of the viewing surface becomes intelligible: A s the eidetiker changes fixation from one point to another the motor activity helps to reinstate the corresponding part-percept. It remains then to account for the detailed vividness of the eidetic image and my hypothesis proposes in short that the eidetic image includes the activity of first-order cell assemblies that are characteristic of perception but absent from the memory image. The idea finds support in the observation of Leask, Haber, and Haber (see Footnote 2 ) that the eidetic image may be strictly monocular when formed with one eye open, disappearing when that eye is closed and the other opened. I t was proposed above that the first-order assembly is composed of (or controlled by) first-order cells most of which have monocular function (Hubel & Wiesel, 1968). Since the eidetic image occurs only for a brief period following stimulation, one thinks first of the hypothesis as meaning that there is some after-discharge in the first-order assemblies. But this would imply the continued activity of all of them at the same time, whereas—as we have seen—the activity must be sequential and motor-linked. The hypothesis instead must be that the eidetiker has first-order visual assemblies which for some reason remain more excitable, for a brief period following stimulation, than those of other ,9s. It is possible, as Siipola and H a y den (1965) and Freides and Hayden (1966) have suggested, that the difference may be due to some slight brain damage. Other perceptual anomalies suggest, in turn, that one effect of brain damage may be to im-
175
474 pair the action of inhibitory neurons in the cortex: neurons whose function is to "turn off" one perceptual (or imaginal) process when it is replaced by another (Hebb, 1960, p. 7 4 3 ) . T h i s inhibitory function, however-—assuming it exists—is not entirety absent in the S"s of Leask et al., in view of the ease with which the eidetic imagery could be prevented or disrupted. T h e disruptive effect of new stimulation (as 5* looks away from the viewing surface) is intelligible if the subassembly components of the first-order assemblies are now excited in new patterns. Leask et al. also report that "thinking of something else" interferes with the formation of the image, and that the same effect results from verbalization in the attempt to memorize the picture's content. T h e theoretical implication is that higher-order assembly activity tends to interfere with some lower-order activity, if not by direct inhibition then possibly because the higher-order assembly utilizes some of the components of the lower-order one and thus breaks up its organization. The central fact in this area is Haber's brilliant experimental analysis of eidetic imagery. T h e present speculation does nothing to extend his results, though it may help to reduce skepticism (for example, in showing how S's eye movement may actively help in retrieving detail). Instead, his work serves here to provide solid experimental data whose import extends to a wider field in which trustworthy data are sparse indeed.
H A L L U C I N A T I O N , HYPNAGOGIC AND M E M O R Y I M A G E
IMAGE,
I wish therefore to conclude by taking account of hallucination and hypnagogic imagery, and relating them and the memory image to thought (or, properly, to other forms of thought).
176
T h e term hallucination is used here to include any spontaneous imagery that might be taken for a perception, even if S knows that he is not perceiving. A phantom leg, for example, is so convincing that the patient may not realize that the leg has been amputated ; at this point it meets the criterion of hallucination in the narrower sense, since the patient is deceived, but it continues to have the same convincing character after the patient is informed of his loss and can see the stump of the limb. T h e nature of the process has not changed; if it was hallucination before, it is hallucination now. Similarly, the imagery of some 5 s in perceptual isolation (Bexton, Heron, & Scott, 1954) was such that they would have thought they were looking at moving pictures if they had not known they were wearing the occluding goggles. T h i s must be hallucination also. It was proposed above that the memory image lacks vivid detail because it is aroused centrally instead of sensorily. Hallucinations have a central origin also, but their vividness is not inconsistent with the above conclusion. If the cause of hallucinations is spontaneous firing by cortical neurons, the spontaneous firing may occur in firstorder as well as in higher-order assemblies. I n its vividness, and in its implication of activity by first-order assemblies, the hallucination is like the eidetic image though it is at the opposite pole in its relation to sensory stimulation, since it seems to depend on a failure of sensation. I n norma! waking hours there is a constant modulating influence of sensory input upon cortical activities, helping both to excite cortical neurons and to determine the organization of their firing. When this influence is defective for any reason—pathological processes, or habituation resulting from monotony
475 or "sensory deprivation"—there is still cortical activity. Neurons fire spontaneously if not excited from without. T h e activity may be unorganized, and in the isolation experiments 5 s in fact were in a lethargic state much of the time, unable even to daydream effectively. B u t when by chance the spontaneous cortical firing falls into a "meaningful" pattern—when the active neurons include enough of those constituting a cell assembly to make the assembly active and so excite other assemblies in an organized pattern—S may find himself with bizarre thoughts or, if first-order assemblies are among those activated, with vivid detailed imagery. T h e hypnagogic image, like the eidetic image and unlike hallucination, is an aftereffect of stimulation but there may be a gap of hours, instead of seconds, between stimulation and the appearance of the imaginal activity. It is characteristic of the period before sleep, but on rare occasions may happen at other times. K . S. Lashley once said that after long hours at the microscope watching paramecia he found himself, as he left the laboratory, walking waist-deep through a flowing tide of paramecia (somewhat larger than life-size!) F o r myself, true hypnagogic imagery is of the same kind though it occurs only before sleep, and is quite different from the ordinary slight distortions of visual imagery at the onset of dreaming and sleep. I t depends on prolonged experience of an unaccustomed kind. A day in the woods or a day-long car trip after a sedentary winter sometimes has an extraordinarily vivid aftereffect. As I go to bed and shut my eyes—but not till then, though it may be hours since the conclusion of the special visual stimulation—a path through the bush or a winding highway begins to flow past me and continues to do so till
sleep intervenes. T h e scenes have a convincing realism, except in one respect. F i n e detail is missing. I see bushes with leaves on them, for example, but the individual leaf or bush becomes amorphous as soon as I try to see that one clearly, at the same time that its surroundings in peripheral vision remain fully evident. T h e phenomenon must be very much like the eidetic image, except in its time properties and its lack of fine detail.
177
T h e memory image does not share the peculiarities of these other forms of imagery, but it may still be more peculiar than is generally recognized. W e have already seen that it lacks detail, due apparently to an inefficiency of associative mechanisms of arousal. W e must now observe that the memory image is typically incomplete in gross respects as well. It frequently lacks even major parts of the object or scene that is imagined—-though if one looks for them they show up at once and so, unless the question is made explicit, one may have the impression that the whole was present all along. Binet's 14-year-old daughter had the advantage of not being psychologically trained and not realizing how improbable her reports would sound. Asked to consider the laundress, she reported seeing only the lady's head; if she saw anything else it was very imperfect and did not include the laundress's clothing or what she was doing. For a crystalline lens, she saw not the lens but the eye of her pet dog, with little of the head or the rest of the animal; and for a handle-bar, all the front part of her bicycle but missing the seat and the rear wheel (Binet, 1903, p. 126.) T o think of the memory image as the reinstatements of a single unified perceptual process makes such reports fantastic, but they are not all fantastic when the image is regarded
476 as a serial reconstruction that may terminate before the whole perceptual process has been reinstated. Incomplete imagery has a special relevance for ideas of the "self" ( a mixture of fantasy and realism discussed elsewhere: Hebb, 1960). It is comprehensible of course that one can, with deliberate intention, imagine what one would look like from another point in the room—that is, one can have imagery of oneself as seen from an external point—but a less complete imagery may occur unintended and without recognition. Memory of floating in water commonly includes some visual imagery of water lapping about a face (if one recalls floating face up) or of wet hair about the back of the head (if face down). A long time ago I could introspect with ease and did so freely. Becoming aware that there were theoretical difficulties about introspection, I began to look at the process critically. Eventually I discovered to my astonishment that it included some imagery of a pair of eyes with the upper part of the face (my eyes and face) somehow embedded in the back of a head (my head) looking forward into the sort of gray cavern R y l e (1949) has talked about. Unfortunately this seemed so ridiculous that I rapidly lost my ability to introspect and now can no longer report on the imagery in detail. But such fantasy in one form or another may be a source of the common conviction that one's mental processes are open to inspection. T h e imagery is fleeting and unobtrusive and not likely to be reported even to oneself, being so inconsistent with one's ideas of what imagery is and how it works, but it may nonetheless be a significant determinant of thought. T h e theoretical analysis earlier in this paper, in terms of lower- and higher-order assemblies, implies a con-
tinuum from the very vivid imagery of hallucination through the less vivid memory image to the completely abstract conceptual activity that has nothing representational about it. ( T h i s includes of course auditory—especially verbal—imagery as well as somesthetic imagery, and it must be wrong to make a dichotomy between visual imagery and thought, or to identify abstract ideas with verbal processes.) T h e ordinary course of thought involves an interaction of sensory input with the central processes—one looks at the problem situation directly, if it is available, makes sketches, talks to oneself— and the activity of the lower-order assemblies, in imagery, may have the same "semi-sensory" function of modulating the concurrent activity of higherorder assemblies. T h e relative efficacy of concrete nouns (names of imaginable objects as stimulus-words in paired-associate learning (cf. e.g., Paivio, 1969), together with the fact that pictures of such objects are still better, suggests something of the sort. Once it has had its effect on higher activity the image may cease; it would be reportable only when it is persistent, tending to interrupt the ongoing thought process, or reinstated later without reinstatement of the whole thought process of which it was part. In this way it is possible to understand how bizarre imagery, of the kind involved in my introspection, might occur without being recognized, or how visual imagery might form an essentia! part of the cognitive map (Tolman, 1948) of a man driving a car through familiar territory, even for the man who believes that visual imagery plays no part in his planning. T h e difference between those who have little imagery and those who have much may be not a difference of the mechanism of thinking, but a difference in the retrievability of the image.
178
477 REFERENCES G . W. The eidetic image and the after-image. American Journal oj Psychology, 1928, 40, 418-425.
ALLPORT,
BEXTON,
W.
H.,
HERON,
W . ,
4
SCOTT,
T.
H . Effects of decreased variation in the sensory environment. Canadian Journal of Psychology, 1954, 8, 70-76. B I N E T , A. L'ltude experimcntatc de ['intelligence. Paris: Schleicher, 1903. F R E T D E S , D., St H A Y D E N , S . D. Monocular testing: A methodological note on eidetic imagery. Perceptual and Motor Skills, 1966, 23, 88. GELLERMAN, L . W. Form discrimination in chimpanzees and two-year-old children: I. Form (triangularity) per se. Journal oi Genetic Psychology, 1933, 42, 3-27. G O O D , I. J . Speculations concerning the first ultraintelligent machine. Advances in Computers, 1965, 6, 31-88. HABER, R. N., & H A B E R , R. B . Eidetic imagery: I. Frequency. Perceptual and Motor Skills, 1964, 19, 131-138. HEBB, D, O. Organisation of behavior. New York: Wiley, 1949. HEBB, D. O. The American revolution. American Psychologist, 1960, 15, 735-745. H E B B , D. O. The semi-autonomous process, its nature and nurture. American Psychologist, 1963, 18, 16-27. HEBB, D. O. A textbook of psychology. (2nd ed.) Philadelphia: Saunders, 1966. H U B E L , D. H . , & W I E S E L , T . N. Receptive fields and functional architecture of monkey striate cortex. Journal oj Physiology, 1968, 195, 215-243. K. S. In search of the engram. In Symposia oj the Society oj Experi-
LASBLEY,
179
mental Biology. No. 4. Cambridge: Cambridge University Press, 1950. M A L M O , R . B.. & S U K W I L L O , W. W. Sleep deprivation; Changes in performance ami physiological indicants of activation. Psychological Monographs, 1960, 502 74(15, Whole No. 502). MORRIS,
G . 0.,
WILLIAMS,
H.
L., &
LUBIN,
A . Misperception and disorientation during sleep deprivation. A.M.A. Archives oj General Psychiatry, 1960, 2, 247-254. MOSELEY, A . L . Hypnagogic hallucinations in relation to accidents. American Psychologist, 1953, 8, 407. PAIVIO, A . Mental imagery in associative learning and memory. Psychological Review, 1969, in press. PRITCHARD,
R.
M.,
HERON,
W.
P
&
HEBB,
D. O. Visual perception approached by the method of stabilized images. Canadian Journal of Psychology, 1960, 14, 67-77. RYLE, G. The concept oj mind. New York: Barnes & Noble, 1949. S I I P O L A , E . M., & H A Y D E N , S. D. Exploring eidetic imagery among the retarded. Perceptual and Molar Skills, 1965, 21, 275-286. S I M M E L , M. L . Phantoms in patients with leprosy and in elderly digital amputees. American Journal oj Psychology, 1956, 69, 529-545. TEUBER, H . L . Perception. In J . Field, H. W. Magoun, & V. E . Hall (Eds.). Handbook of Physiology: Neurophysiology. Vol. 3. Washington, D. C . : American Physiological Society, 1960. T O L M A N , E . C. Cognitive maps in animals and man. Psychological Review, 1948, SS, 189-208. W O O D W O R T H , R. S. Experimental psychology. New York: Holt, 1938. (Early publication received July 10, 1968)
Reprinwd wiili pcnniss ion from Science. VW. 1,1. pp. 701-703. 1971 J1971 The American Association for Ihe AdvATicemeni of Science
Mental Rotation of Three-Dimensional Objects Abstract. The time required to recognize that two perspective drawings portray objects of the some three-dimensional shape is found to be (i) a linearly increasing function of the angular difference in the portrayed orientations of the two objects and (i0 no shorter for differences corresponding simply to a rigid rotation of one of the two-dimensional drawings in its own picture plane than for differences corresponding to a rotatio/i of the three-dimensional object in depth. Human subjects are of [en able to determine that two two-dimensional pic lures portray objects of tbe same three-dimensional shape even [hough the objects are depicted in very different orientations. The experiment reported here was designed to measure [he time that subjects require to determine such identity of shape as a function of the angular difference in the portrayed orientations of the two three-dimensional objects. This angular difference was produced either by a rigid rotation of one of two identical pictures in its own picture plane or by a much more complex, nonrigid [ransformation, Of one of the pictures, that corresponds to a (rigid) rotation of the three-dimensional object in depth. This reaction time is found fi) to
increase linearly with the angular difference in portrayed orientation and (ii) to be no longer for a rotation in depm [han for a rotation merely in the picture plane. These findings appear to place rather severe constraint on possible explanations of how subjects go about de[ermining identity of shape of differently oriented objects. They are, however, consistent with an explanation suggested by the subject themselves. Although introspective reports must be interpreted with caution all subjects claimed (i) that to make the required comparison they first had to imagine one object as rotated into the same orientation as the other and that they could carry out this "mental rotation" at no greater than a certain limiting rate; and (ii) that, since they perceived the two-dimensional pictures as objects
180
J
7W in three-dimensional space* they could imagine the rotation around whichever axis was required wilh equal ease. In the experiment each of eight adult subjects was presented with 1600 pairs of perspective line drawings. For each pair the subject was asked to pull a right-hand lever as soon as he determined thai the two drawings portrayed objects that were congruent with respect to three-dimensional shape and to pull a JetE-hand lever as soon as he determined thai the two drawings depicted objects of different three-dimensional shapes. According to a random sequence, in half of the pairs (the "same" pairs) Ihe two objects could be rotated into congruence with each other (as in Fig. I , A and B ) . and in the other half (the "different" pairs) the two objects differed by a reflection as well as a rotation and could not be rotated into congruence (as in Fig. 1C). The choice of objects that were mirror images or "isomers" of each other for the "different" pairs was intended lo prevent subjects, from discovering
_
A
_
some distinctive feature possessed by only one of the two objects and thereby reaching a decision of noncongruence without actually having lo carry out any menial rotation. As a further precaution, the ten different three-dimensional Objects depicted in the various perspective drawings were chosen to be relatively unfamiliar and meaningless in overall three-dimensional shape.
presented twice. The remaining 800 pairs, randomly intermixed with these, consisted of 400 unique "different" pairs, each of which (again) was presented twice. Each of these "different" pairs corresponded to one "same" pair (of either the "depth" or "picture-plane" variety) in which, however. One of the three-dimensional objects had been reflected about some plane in three-dimensional space. Thus the two objects in each "different" pair differed, in genera), by both a reflection and a rotation.
Each object consisted of ten solid cubes attached face-lo-face to form a rigid armlike structure with exactly three right-angled "elbows" (see Fig. 1). The sel of all ten shapes included two The 1600 pairs were grouped into subsets of five: within either subset, no blocks of not more than 200 and preshape could be transformed into itself sented over eight to ten 1-hour sesor any other by any reflection or rota- sions (depending upon the subject). tion (short of 360 ), However, each Also, although it is only of incidental shape in either subset was the minor interest here, each such block of presimage of one shape in the other sub- entations was either "pure," in that alt set, as required for the construction of pairs involved rotations of the same the "different" pairs. type ("depth" or "picture-piane ) , or "mixed." in that the two types of rotaFor each of the ten objects. IS different perspective projections—corresponding to one complete turn around the vertical axis by 20° steps—were generated by digital computer and assoA 1Plr.luie-pi.ine paif*> ciated graphical output [1). Seven of the IS perspective views of each object were then selected so as (i) to avoid any views in which some part of the object was wholly occluded by another part and yet (ii) to permit the construction of two pairs that differed in orientation by each possible angle, in 20° steps, from 0° to 18D . These 70 line drawings were then reproduced by photooffset process and were attached to cards in pairs for presentation to the subjects. 1
D
1h
6
B
_
Fig. L Examples of pairs of perspective line drawings presented to the subjects. (A) A "same" pair, which differs by an SO' rotation in tbe picture plane; (B) a "same" pair, which differs by an 80" rotaiion in depths and (C) a "different" pair, which cannoi be brought into congruence by any rotation.
Half of the "same" pairs (the "depth" pairs) represented two objects that differed by some multiple of a 20" rotation about a vertical axis (Fig. IB). For each of these pairs, copies of two appropriately different perspective views were simply attached to the cards in the orientation in which they were originally generated. The other half of the "same" pairs (the "picture-plane" pairs) represented two objects that differed by some multiple of a 20" rotation in the plane of the drawings themselves (Fig. JA). For each of these. One of Ihe seven perspective views was selected for each object and two copies of this picture were attached to the card in appropriately different orientations. A l together, the 1600 pairs presented to each subject included 300 "same" pairs, which consisted of 400 unique pairs (20 "depth and 20 "picture-plane" pairs at each of the ten angular differences from 0" to 180*). each of which was ,h
181
24
*2
90
90
I M liQ 140
IU
IH
Angle of rotation (degrees) Fig. 2. Mean reaction limes lo two perspective line drawings portraying object of the same three-dimensional shape. Times are plotted as a function of angular difference in portrayed orientation: (A) for pahs differing by a rotation in the picture plane only; and (B) for pairs differing by a rotation in depth. (The centers of the circles indicate the means and. when, they extend far enough to show outside (hese circles, the vertical bars around each circle indicate a conservative estimate of the standard error of that mean based on the distribution of the eight component means contributed by the individual subjects.)
703 lion were randomly intermixed within the same block. Each trial began with a warning tone, which was followed half a second Jater by the presentation of a stimulus pair and the simultaneous onset of a timer. The lever-pulling response stopped the timer, recorded the subject's reaction lime and terminated the visual display. The line drawings, which averaged between 4 and 5 cm in maximum linear extent, appeared at a viewing distance of about 60 cm. They were positioned, with a center-to-center spacing that subtended a visual angle of 9 , in two circular apertures in a vertical black surface (see Fjg. 1. A to C).
ner for the "different*" pairs. The overall mean reaction time for these pairs was found, however, to be 3.8 seconds— nearly a second longer than the corresponding overall means for Ihe "same" pairs. (In the postexperimental interview, the subjects typically reported that they attempted to rotate one end of one object into congruence with the corresponding end of ihe other object; they discovered that the two objects were different when, after this "rotation," the two free ends still remained no n congruent.)
a
The subjects were instructed to respond as quickly as possible while keeping errors to a minimum. On the average onfy 3.2 percent of the responses were incorrect (ranging from 0.6 to 5.7 percent for individual subjects). The reaction-time data presented below include only the 96.g percent correct responses. However, the data for the incorrect responses exhibit a similar pattern. In Fig. 2, the overall means of the reaction times as a function of angular difference in orientation for all correct (right-hand) responses to "same" pairs are plotted separately for the pairs differing by a rotation in the picture plane (Fig. 2 A ) and for the pairs differing by a rotation in depth (Fig. 2B). In both cases, reaction time is a strikingly linear function of the angular difference between the two three-dimensional objects portrayed. The mean reaction times for individual subjects increased from a value of about 1 second at 0° of rotation for all subjects to values ranging from 4 to 6 seconds at 180° f rotation, depending upon ihe particular individual. Moreover, despite such variations in. slope, the linearity of ihe function is clearly evident when the data are plotted separately for individual threedimensional objects or for individual subjects. Polynomial regression lines were computed separately for each subject under each type of rotation. In all 16 cases the functions were found lo have a highly significant linear component {P < .001} when tested against deviations from linearity. No significant quadratic or higher-order effects were found (P > .05, in all cases).
Not only are the two functions shown in Fig. 2 both linear but they are very similar to each other with respect to intercept and slope. Indeed, for the larger angular differences the reaction times were, if anything, somewhat shorter for rotation in depth than for rotation in the picture plane. However, since this small difference is either absent or reversed in four of ihe eight subjects, il is of doubtful significance. The determination of identity of shape may therefore he based, in both cases, upon a process of the same general kind. If we can describe this process as some sort of "mental rotation in threedimensional space," then the slope of the obtained functions indicates that the average rate at which these particular objects can be thus "rotated" is roughly 60° per second. Of course the plotted reaction times necessarily include any times taken by the subjects to decide how to process
0
The angle through which different three-dimensional shapes must be rotated to achieve congruence is not, of course, defined. Therefore, a function like those plotted in Fig. 2 cannot be constructed En any straightforward man-
182
the pictures in each presented pair as well as the time taken actually to carry out the process, once it was chosen. However, even for these highly practiced subjects, the reaction times were still linear and were no more than 20 percent lower in the "pure" blocks of presentations (in which (he subjects knew both the axis and the direction of the required rotation in advance of each presentation) than in ihe "mixed" blocks (in which the axis of rotation was unpredictable). Tentatively, this suggests that SO percent of * typical one of these reaction times may represent some such process as "mental rotation" itself, rather lhan a preliminary process of preparation or search. Nevertheless, in further research now underway, we are seeking clarification of this point and others. ROGER N.
SHEPARD
J A C Q U E L I N E MBTZLBB Department Stanford Stanford,
of
Psychology,
University, Califarnia
94305
References and IN'nlei L. H n . Jlh-Jie Chana of the Bell Telephone Lahor'tQrtej generated Ihe ISO perspective projections tor us by means al the Bell Laboratories' Strombera-t •: • • .1 microfilm retorder and tbe computer program for constructing tuch projections developed there by A- M . Noll. See. tor example, A . M . Noll. Computers Automation 14. 20 (1965). 1. We Ehonk Mrs. Chana l * » (I)]', and WC also Ibank Dr. J. D. ElMhoft tot her liigBfljtion concerning the statistical analyses. Assistance In 1be computer graphics was provided by the Bell Telephone Laboialo.net. Supported by N S F grant GS-5?S3 to R . N . S . 9 Maieh 1970; replied 9 September 1970
•
Reprinted with permission tram Science. vol. 243 pp. 234-236. 1939 © l9S9The American Association for the Advance meni of Science L
Mental Rotation of the Neuronal Population Vector
weighted
vector
sum
of
contributions
("votes") o f d i r e c t i o n a i l y t u n e d
neurons:
each neuron LS assumed t o vote i n its o w n
A P O S T O L O S P. G E O R G O P O U T . O S , * JOSEPH T. L U R I T O , M I C H A E L P E T R I D E S , A N D R E W R. S C H W A R T Z , JOE T. M A S S E Y
preferred d i r e c t i o n w i t h a strength that depends o n h o w m u c h the activity o f the neuron changes Tor (he m o v e m e n t
under
A rhesus m o n k e y w a s ( r a i n e d t o m o v e its a r m i n a d i r e c t i o n t h a i was p e r p e n d i c u J a r ( o
consideration. T h i s vectorial analysis
and counterclockwise f r o m the d i r e c t i o n o f a target light that changed i n position f r o m
proved useful tn visualising (he directional-
t r i a l t o t r i a l . S o l u t i o n o f t h i s p r o b l e m was h y p o t h e s i z e d t o i n v o l v e t h e c r e a t i o n a n d
ity o f the p o p u l a t i o n i n t w o - and
m e n t a l r o t a t i o n o f an i m a g i n e d m o v e m e n t v e c t o r f r o m (he d i r e c t i o n o f (he l i g h t ( o the
dimensional space d u r i n g the reaction time
d i r e c t i o n o f t h e m o v e m e n t . T h i s h y p o t h e s i s was tested d i r e c t l y b y r e c o r d i n g (he
{7) and d u r i n g an instructed delay p e r i o d
a c t i v i t y o f cells i n Che m o t o r c o r t e x d u r i n g p e r f o r m a n c e o f the task a n d c o m p u t i n g (he
(8).
n e u r o n a l p o p u l a t i o n v e c t o r i n successive time i n t e r v a l s d u r i n g t h e r e a c t i o n time. T h e
Given
the
mental
rotation
has
three-
hypothesis
p o p u l a t i o n v e c t o r r o u t e d g r a d u a l l y c o u n t e r c l o c k w i s e f r o m t h e d i r e c t i o n o f (he l i g h t t o
above and the neuronal p o p u l a t i o n vector as
t h e d i r e c t i o n o f t h e m o v e m e n t at an average t a t e o f 7 3 2 ° per s e c o n d . T h e s e results
a neural representation o f the
p r o v i d e d i r e c t , n e u r a l evidence f o r the m e n t a l r o t a t i o n h y p o t h e s i s a n d i n d i c a t e ( h a t rhe
direction^ a s t r o n g test is as f o l l o w s : i f a
n e u r o n a l p o p u l a t i o n v e c t o r is a u s e f u l t o o l f o r ' r e a d i n g : o u t " a n d i d e n t i f y i n g c o g n i t i v e
m o n k e y performs in (he above-mentioned
operations o f n e u r o n a l ensembles.
task and the neuronal activity i n the m o t o r cortck.
A
is recorded
during
movement
performance,
FUNDAMENTAL PROBLEM IN COG-
d i r e c t i o n o f a stimulus. U n d e r these c o n d i -
w o u l d the p o p u l a t i o n vector rotate i n t i m e ,
l i t i v c neuroscience is the idcntilica-
tions
the
as the hypothesis l o r a mental r o t a t i o n o f an
d o n a n d e l u c i d a t i o n o f b r a i n events
angle, w h i c h suggests that the subject may
imagined movement vector w o u l d predict?
The
solve this p r o b l e m b y a m e n t a l r o t a t i o n o f an
Because the appropriate m o v e m e n t dircC-
technique o f r e c o r d i n g the activity o f single
imagined movement vector f r o m the direc-
t i o n can be arrived at by either a counter-
cells in the brain o f behaving animals (2)
t i o n o f the stimulus r o rhe d i r e c t i o n o f the
clockwise or a clockwise r o t a t i o n , w h i c h o f
provides a direct t o o l
purpose.
actual movement ( 5 ) . N o w , ihe d i r e c t i o n o f
these t w o rotations w o u l d be realized by the
Indeed, a wcatrh o f k n o w l e d g e has accumu-
i n u p c o m i n g m o v e m e n t i n space seems to
p o p u l a t i o n vector? O f course, there is n o
lated d u r i n g rhe pasr I S years c o n c e r n i n g
be represented i n the m o t o r cortex as rhe
reason rhat rhe p o p u l a t i o n vector s h o u l d
the activity o f cells i n several b r a i n areas
neuronal p o p u l a t i o n vector ( 6 ) w h i c h is a
rotate ar ail, a n d i f i t rotates* there i$ n o a
underlying c o g n i t i v e operations
f o r that
(I).
the reaction time increased w i t h
h
d u r i n g performance b y monkeys o f complex tasks. A major f i n d i n g o f chese studies has been that the activity o f single cells in specific areas o f the cerebral cortex changes d u r i n g performance
ol
particular
tasks;
these
changes are t h o u g h r t o reflect the participat i o n o f rhe area under study in the c o g n i t i v e function i n v o l v e d i n the task ( J ) . H o w e v e r , a direct visualination o f a c o g n i t i v e operat i o n in terms o f neuronal activation i n the brain is lacking. W e chose as a test case f o r this p r o b l e m the c o g n i t i v e o p e r a t i o n o f mental r o t a t i o n . I m p o r t a n t w o r k in experimental psychology d u r i n g the past 20 years { J ) has established die mental r o t a t i o n p a r a d i g m as a standard in c o g n i t i v e psychology and as a p r i m e t o o l in investigating c o g n i t i v e operations o f the " a n a l o g " type. W e adapted t h i s procedure in a task that r e q u i r e d m o v e m e n t o f a handle in a d i r e c t i o n (hat was at an angle w i t h (he
A- P- GeQrRQpoulDS and |. T. LurvtO, The Philip Bard Laboraiones of Heuiophysiology, DtpJumem of Neunoetcnf( The Johns llopkuu UniverMry School of Meeutlne. 725 Nonti Wolf* Street, Qalrifnore, M£> 21205 M . FOnds, Deparanenr. of Psychology McGiU University 1205 Dr. Penfield Avenue, Montreal, Quebec, Canidl H3A J R | A. B. Schwann, Division o f Neurobiology, Sr. Joseph's Hospital and Medieal Center. BarrOv. Neurological Institute 3SO Wesc Thomas Road, fhucni-. A £ S S 0 I 3 . J. T. MiUcy, Deparanenr. of Neurowicn" and Department of Bioniedjral tncuieennE. The lohjis Hopkins University Sf hoot of Medicine. 725 N o r * Wolfe Srreei Baltimore. MD 21205 h
h
L
T o •^hom correspondence should be addressed.
Fig. 1. Result, from a direct [left] and rotation (right) movement. (A) Task. Unfilled and filled circles indicate dim and bright Light, respective lv. Interrupted and continuous, lines with, arrows indicate stimulus IS) and movement {M) direction, respectively. (B} Neuronal population vectors calculated every l O r m f r o m (he onset of the stimulus [S] at positions shcuwn in (A) unl.il after the onset o f the movement ( M ) . When the population vector lengthens, for the direct case (left} Lt poults in the direction o f the movement, whereas for the rotation case it points initially i i i the direcuon o f the stimulus and then rotates counterclockwise irrom 12 o'clock to 9 o'clock) and points in die direcuon o f the movement (C) Ten successive population vectors from (B) are shown in a spaual plot, starting from the first population vector that increased signuicandy in length Notice the counterclockwise rotatiim ol" the pnptilauon vector fright panel). (Dj Scatter plots- o f rhe direction of the population vectiir as a function trf tune, starting from the first population vector that increased significantly in length after stimulus onset (Sf. For the direct case (left panel) the direction o f the population vector is in the direction o f the mnvement (•-ISO"); fur the rotation case (right panel) the direction o f the population vcaor rotates counterclockwise from the direction o f the stimulus ( - v O " ! D the direcuon o f the movement (-1B0-).
183
A
Dlruci
Rotation S • W
g
c
D
~ £ c •= 8" J
"
J Q I )
_ 10O £00 300 Tin* (me}
•00 SOD 300 Urn* (Ml
235 priori reason char it should rotate in one or die other direction; for all we know, any of these alternatives is possible. The activity of single cells in rhe motor cortex was recorded (9) while a rhesus monkey performed in the mental rotation taskIn the beginning of a trial, . light appeared at the center of * plane in front of the animal, which moved its arm toward the light with a freely movable handle (10). After a variable period oftime(0.75 to 2.25 s). the center light was rumed off and turned on again, dim or bright-, at one of eight positions on a circle of 2-cm radius (If). The monkey was trained to move the handle in the direction of the light when it came on dim fdirecT. trials) or in a direction that was perpendicular (90") to and counterclockwise from the direction of the light when it came on bright (rotation trials) (T2) The movements of the animal were in the appropriate direction for both kinds of trials. The neuronal population vector was calculated every 10 ms starting from the onset of the peripheral light (that is, at the beginning of the r
reaction time). The preferred direction of stimulus onset. At this point its direction each cell (n = 102 cells) was determined was close to the direction of the stimulus; from the cell activity in the trials in which the average angle between the direction of the animal moved toward the light (direct the population vector and that of the stimutrials). For the calculation of the population lus was 17" counterclockwise (the average vector, peri stimulustimehistograms [10-ms absolute angle was 29"). The population bin width) were computed for each cell and vector stabilized in direction at 225 ± 50 each of the 16 combinations (classes) used ms after stimulus onset. At this point its [eight positions and two conditions (direct direction was close to the direction of the or rotation), see (11) above] with counts of movement; the average angle between the fractional interspike intervals as a measure of direction of the population vector and that the intensity of cell discharge. A square root of the movement was 0-5" clockwise (the transformation was applied to these counts average absolute angle was S ). Finally, the to stabilize the variance (13). For a given movement began 260 = 30 ms after stimutime bin, each cell made a vectorial contribu- lus onset, thai is, 35 ms after ihe direction of tion in the direction of the celTs preferred the population vector became relatively stadirection and of magnitude equal to the ble; this difference was statistically significhange in cell activity from that observed cant (P < 0.02, paired t rest). during 0.5 s preceding the onset of the These results support the hypothesis that peripheral stimulus (control rale, that is, ihe directional transformation required by while the monkey was holding the handle at the task was achieved by a cotmtcictockwise the center of the plane). The population rotation of an imagined movement vector. vector P for the j ' class andfc time bin is This process was reflected in the gradual change of activity of motor cortical cells, m which led to the gradual rotation of the Pj^-Su'.j.fcCi vectorial distribution of the neuronal ensemble and the population vector. The average where C, is the preferred direction of the slope of the rotation of the population veci" cell and iv\ is a weighting func- tor (7327s, see above) was comparable to tion M'ij.t = t^ij.fc) — -"ij "here 4j.k is but higher than that observed when human the square root-trarisfbrTned (IJ) discharge subjects performed a similar task (—4007s) rate of the r* cell for the f class and k' rime (5) and that observed in a task that involved bin, and a,j is the similarly transformed mental rotation of rwondimertsional images control rate of the r* ceil for the class. (-4007s) (14). It is likely thar all three Figure 1 illustrates the results obtained experiments involved a process of mental when the movement direction was the same rotation which, in the present case, was (toward 9 o'clock) but the stimulus was reflected in the motor cortical recordings of cither at 9 o'clock (direct trials, left panel] or this study and identified by using the popuat 12 o'clock (rotation trials, right panel). In lation vector analysis. Of course, other brain ihe direct trials die population vector point- areas are probably involved in such complied in the direction of the movement (which cated trarisformations; for example, recent coincided with the direction of the stimulus) (Fig. \, left). However, in the rotation trials experiments with measurements of regional the population vector rotated intimecoun cerebral blood flow (15) suggested that tetclockwise From the direction of the stimu- frontal and parietal areas seem to be in¬ lus to rhe direction of the movement (Fig. 1, volved in the mental rotation task of Shepright). Another example is shown in Fig. 2 ard and Metzler (16), whereas frontal and and illustrated in the cover photograph. The central areas seem to be involved in a line working space is outlined in blue. The time orientation task (f5); in both of these tasks axis is the white line direcred upwards. The there was a greater increase in blood How in population vector is shown in green, as it the right than in the left hemisphere. rotates during the reaction time from the The rotation of the neuronal population stimulus direction (between 1 and 2 o'clock) vector is of particular interest because there to the movement direction (between 10 and was no a priori reason for it to rotate at all. 11 o'clock). The population vector was cal- It is also interesting that the population culated with a 20-ms bin sliding every 2 ms. vector rotated consistendy in the counterThe red lines are projections of the popula- clockwise direction: this suggests that the tion vector onto the working space. spatial-motor rransfotmation imposed by the task was solved by a rotation through The rotation of the population vector was the shortest angular distance. Given that die a linear function of time with an average mental roration is time consuming, this soslope (for the eight positions of the Ughi lution was behaviorally meaningful, for it used) of 732 ± 4567s (mean * SD). The nuniiruzed both the time for the animal to population vector began to change in length get the reward and the computational effort 125 ± 28 ms (mean ± SD, rr = &) after ihe which would have been longer if the rotaB
h
111
1
JM
h
^
L
90
Fig. 2- Rotauon of the population vector for a different set of rotation trials. The stimulus and movement directions arc indicated by the interrupted and continuous lines at the top. The population vector in the two-dimensional space is mown for successive time frames beginning 9D ms after stimulus onset. Notice its rotation counterclockwise from the direction of the stimulus to rhe direction of the movement.
184
h
236 •on had been through 270" clockwise (77). Finally, these results were obtained from one animalr because cognitive problems could be solved in different ways bv different subjects, it is important that techniques for reading out brain operations be sensitive enough to be applied to single subjects. Indeed, the findings of our study indicate that rhe population vector is a sensitive tool by which an insight can be gained into the brain processes underlying cognitive operations in space.
its movement exceeded 3 em and slaved within =25" of the dirteOon. requited!. The average direction of ihe actual movement rja|ectoriCi wai wirJun =5" of die direcuon required. Performance was over 70% correct trials. 13 The square roar transformation was used as a variance-stabilizing Irani forma don for counrs [G. W. Snedetor and W G Cochran. Stattiliwl Mtihtdi (Iowa Slate TJniv. Press, Ames. Iowa, ed. 7, 1980.}. pp. 298-290.] Although the results obtained withour this iraniformiiion were similar, rhe transformation is mare appropnare because of ihe im JII size nf the time him (10 ms), and, therefore, the small number of counts.
1.....-.J..' P;.!-,-! 7.20(1975). 15. G. Deutseh. W. T. Bourbon, A. C Papanicolaou, H M Eiscnberg tovt°pty
4
ri
IK. We rhank Brandt ind N . Toner for help during some of the •penmencs Supported by USPHS 14 L A Cooper and ft N Shepard. in VHml fopwUgrants N517413 and NS20868 iron Prfrmifff, w . G Chase, Ed- {Academic Press, New York. 1973), pp 7S-176; L A. Cooper. 1 August 1983i accepted 1 November lYfifl
REFERENCES AND NOTES 1. See, for trample, V. B. rVlQunicascJe, Tnndi Mfuroin 9, SOS (.I486); M . A Arbib and M . B Hesse, The Cfl-urnirtiim of Aar% {Cambridge umv Press. New V o t t L986};. ] . 7, Young, PhJotaphy W rtr Brain {Orfbjd UnW Press, New York, 1987} 2. B. N . Lemon, Methods Jar fvtfuniiuiJ Btiotdirtg at Ce-utioiu Afimah (Wiley. Chisester. 1984} 3 Set for example, HWfcuot of PHyjietBgy, Section I , Thl Nervous System, High" F-n^lron ojthr Brain, pins I ind 3, V. B. Mrxinrcastle, F. Plum, % R. Geiger. Eds. (Americin Physiological Society, Brthrsda. MD, 1987), vol, 5. 4. R. N . Shepard and ). Metzler. $arw< 171. 701 (1971); B_ N . Shepard and L. A. Cooper, Mtittit Pmajfi JOJ TV"r TMnr^nFmJHiu (MIT Press, Cambridge, MA, 1982) 5. A. P. Georgopoulos and T T, Massey, Exp. Brain Per 65, 361 ,1987). 6. A. P. Gcorgopoulcrt, p. Cuniniti, ]• f • KalaskaJ. T. Massev, ifriJ. Suppt. 7, 327 {1953); A. P. Gcorgopoulos, A. E. Schwartz. R. E. Keirner, Srfmcr 233. M16{r986); f N u « r « 8, 2923(1988» 7. A. P GcorgopouJos, | E. Kajatka, JVt. D. Cnitcher. Et Caminln, |. T. Massey, in Dynamic Aiptits ef Niaonia! FHHW. G. M Edelman, W. E. Gall, W. M Cowan. Eds. (Wiley, New YonV, 1984). pp 501-524L A P Georgopoulos, A fl. Schwartz. R. E.Kenner,/ N f m M 8. 292B (1988>. fl. A_ P. Grorgopoukn, M . D. Cnitehcr, A. BSchwartt £»p. fl"r« fl=J.. in press. 9. The elecmcal sigrn of iCtivJi^' of Individual cells in the irTn area of The motor conei conirabteral to the pcrfonntng arm were recorded utracelluliriy |A. P. GeorgDpoulos, 1. F KzJaska. EL Cimlnld, )- T. Missey, j . 2, 1527 J1982)] All surgical •pflTJlionj [A. P. Georgopoulos, f, F. Kalaika. B_ CauunlTi, f, T. .Massey. / .\ •-•... 2, 1527 (1982)1 for the prepzradon of ihe animal for electrophysiological nccordingJ. •vers perfotmed under general pentobarbital anesthesii. BehaviDral control and data eollecnon and analysis were performed wilh a laboritory minicomputer. 10. The apparatus -"is udesenbed ui A. P Georgapoulo* and |. T Mfcucy I ^ T Brain Pts. 65, 361 (1987)]. Briefly, it consisted of a 25 cm by 25 cm planar working surface made of frosted plenigtzss •nro which a HcNc lalCt beam was back-projecled with a sysrem of manrors and rwa gzlvanomcters The monkey (5 kg} sac comfortably On 1 primate chair and grasped a freely movable, artieuTarcd handle at its distal end, net ro a 10-mm diameter rraruparCnr plexiglass circle wirhin which the animil capnired the center light. 11 The eight positions were equally spaced on the circle, that is. IE angullf intervals of 45°, and were the same thjOUghout the operiment. The brightness amditjon (dim or bright) and the pojipon of rhe light v^ere muted. The resulting lfi hnghtness-pOJi•on combinations were randomized. Eight Tepeu•ons of these 16 COmbinations were orocmed in a randomized block design. 12. The Term "eouiircidockwise" i l limply descriptive^ no counteiclockwise Or elotkwise direetioni were indicated to the animal. The direction tn which rhe animal was required io move can be desenbed equivalent^ 31 either 90" countcretockwise or 270° clodtwue The animal received 1 liquid reward wher*
185
2b Information Theory and Perception
Perhaps an introduction to this section could be dispensed with because the first text, (18) by Gregory, provides an almost ideal stage-setting for the four others. Nevertheless, here follow a few side remarks. Computation is information-handling, processing of information. The edification of the mathematical theory of information, to which the name of Claude Shannon is prominently associated, gave an adequate tool for engineering communication systems. It had also considerable impact on psychophysics and cognitive sciences, and raised more hopes and expectations than did materialise in subsequent developments. Gregory points out some difficulties of applying information theory to brain studies, yet conveys a feeling of expectation for a further revival. Essential notions such as information content, information entropy, channel capacity (measured in bit/s), span or repertoire (measured as a dimensionless number, or in bits), redundancy reducing code, etc., have been used to quantify the perceptual and motor capacities of biological brains i.e. their performances at processing the huge flow of sensory data. The following four papers will help the reader to assess the contributions and the limitations of information theory. The early paper of Attneave (paper (19)) expresses the excitement of the fifties with this new scientific tool. Its insistence on "perception as economical description", an early formulation of the "redundancy reduction" principle, prepares the way for two subsequent papers, (21) and (22) by Barlow, as well as for article (38) in Chapter 3. In this section are reproduced two major classics, papers (20) and (21), which we felt necessary in this selection because they express essential, and occasionally subtle, ideas with unrivalled clarity. The article of George Miller (20), inspired by the channel capacity concept, attempts to discover and quantify various information bottlenecks in our perceptual apparatus (such as span of absolute judgement, span of immediate memory) and to characterise the ways by which we manage to overcome
186
these innate limitations (such as chunking by learning). T h e first text of Barlow (paper (21)) sets itself a programme so ambitious, that success would have meant solving almost all the fundamental mysteries still present today in the field of neurocomputation. The "single neuron doctrine" is directly inspired by the information-theoretic principles of redundancy reduction and economy of representation. It involves a coding scheme based on decorrelation processes, requiring at each stage fewer and fewer neurons, leading finally to highly selective "cardinal cells". These issues will be evoked again in section 3d. T h e reader is invited to ponder on the differences between this line of thought, and that exposed in the previous section ("cell assemblies"). A t this point, the reader is supposed to begin drawing much benefit from the cross-sights between the various sections of this volume, and also to acquire a sense of historical depth provided by the chronology of the articles. T h e second text of Barlow (paper (22)), written about twenty years later, presents several useful additional remarks and suggestions concerning the interplay between perception and learning, the versatility of learning in higher mammals, synaptic mechanisms, neural coding, and correlation/ decorrelation mechanisms. The reader is invited to examine the compatibility of the requirements on neural coding, as exposed in papers (21) and (22): Is the impulse frequency of a neuron coding for subjective certainty, or for surprise? Finally, he will come to appreciate the antiquity of the definition of perception as unconscious inductive interference, and to construe his own insightful analogies between perceptual processes, thought processes and scientific processes.
187
Reprinted with permission from Odd pp, 187-194, 1986 © 1986Routledge
Perceptions.
WHATEVER HAPPENED TO INFORMATION THEORY?
The notion of information bridges the external world of physical events and our perception and understanding, for information of events and things is separate from us in space and time. How to think about and to measure information are still controversial matters, and there are snags, or at least limitations, in all theories and methods. The lead has come from engineers, but until recendy at least their concern has been with communication-transmission systems, rather than with brains or minds. With developments in artificial intelligence this is now changing, and we may look for further leads from the engineers who have to grapple with these difficult problems. The issues are technical, and we cannot altogether escape technical detail here. Twenty or thirty years ago the great hope for quantifying perceptual decision processes, andfindingout what goes on in skills, was the mathematical theory of information, which was developed for engineering communication systems and most elegandy worked out and presented by Claude Shannon and Warren Weaver of the Bell Telephone Laboratories in 1949. This promised to be able to measure amounts of information, rather as there are measures for costing energy, for estimating efficiencies and charging for messages conveyed by telegraph and telephone lines. It also promised to quantify the information-handling of the nervous system and so to give basic insights for a functional philosophy of perception and behaviour. Some of this came true, but we hardly hear of information
188
theory now in accounts of perception - though it is extremely important for communication and control engineering. Why isn't it as useful for physiology and psychology? Has this something to do with the subjective nature of perception? Does this prevent objective measurements? Conveying information by gestures and by speech is pre-human in origin, and indeed essential for all intelligent life. By writing, and much later by printing, information has been transmitted over space and stored in time with incalculable benefit, but attempts to transmit information very fast over distances beyond the range of speech are quite recent, and the means for costing or charging for information transmission have been devised more recendy still. There is, however, a considerable history before Shannon and Weaver. Suggestions for telegraphs, using magnetism or electricity, go back to Roger Bacon's suggestion in 1267 that a 'certain sympathetic needle might be used for distant communication'. T h i s was the lodestone, or rather a needle magnetized by rubbing with a lodestone, as used in the mariner's compass. T h e n there is a seventeenth-century description of a telephone working with wires by the inventive genius Robert Hooke, in the Preface to his Micrographia of 1664. After a discussion of optical instruments that 'must watch the irregularities of the Senses, but . . . not go before them or prevent their information', Hooke wrote: T i s not impossible to hear a whisper a furlong's distance, it having already been done; and perhaps the nature of the thing would not make it more impossible, though that furlong be ten times multiplied. And though some famous Authors have affirm'd it impossible to hear through the thinnest plate of Muscovy-glass; yet I know a way, by which it is easy enough to hear one speak through a wall a yard thick. It has not yet been thoroughly examin'd, how far Otocousticons may be improved, nor what other ways there may be of quickening our hearing, or conveying sound through other bodies than the Air: for that this is not the only medium, I can assure the Reader, that I have, by the help of a distended wire, propagated the sound to a very considerable distance in an instant, or with as seemingly quick a motion as that of light, at least, incomparably swifter than that, which at the same time was propagated through the Air; and this not only in a straight line, direct, but in one bended in many angles. Just what was this seventeenth-century otocousticon telephone? Was it a 'string telephone' - a wire joining distant diaphragms? If so, how, exacdy, did it work round comers? O r can it have been an electric telephone? Surely this was beyond even Hooke's ingenuity at that time. T h e r e were several suggestions for electric telegraphs a
189
century later, following Steven Grey's discovery in 1729 that electricity could be carried on insulated wires. By the 1780s there began to appear working single-wire telegraph systems using sounders with a code. Samuel Morse (1791-1872) turned from painting to building telegraph transmitting and receiving equipment, and to inventing his code. This was clever, because it used the fewest dots and dashes for the most frequently used letters - which showed how the code must be optimized for efficient transmission of information. It was first demonstrated on Saturday 2 September 1837, at Washington Square in New York, over 1700 feet of wire. But wire was expensive and information was in demand, and rapid rates of transmission of messages soon became economically vital. This need became all too clear with the first, incredibly expensive, transadantic cable. This supreme triumph of Victorian engineering, as it turned out to be, was very nearly a disaster, as fundamental principles were not available or understood. Itfirstoperated in 1858, but through misuse it broke down after only a few hundred messages had been transmitted and received. The next cable was successfully laid in 1865 by Brunei's huge ship the Great Eastern, which also recovered and repaired the earlier cable. At first messages could only be sent at the uneconomic rate of eight words per minute - not because of the human limitations of the senders and receivers but because of the inductive inertia (or low bandwidth) of the cable. The signal strength was low, which led to the invention of relays, amplifiers and repeaters which solved the problems for the long-distance telegraph cables as they do in the nervous system; but atfirstthe brute-force method of applying very high voltages to the cable was tried - and damaged it. Separating concepts of information from power and appreciating their statistical nature were the essential steps for the understanding and designing of information systems. The best scientists of the time - Michael Faraday, Charles Wheatstone, William Cooke, Lord Kelvin and, as a young man whose great inventive achievements came later, Thomas Alva Edison - contributed to this extraordinarily important technical development. Apart from the practical importance of rapid international communication, it focused attention on information as a concept, and it led to notions of channels and coding and so on that generated insights into all kinds of communication, including language, which deeply influenced philosophy as well as providing suggestions for how information is handled by the nervous system. The notion of a minimum necessary bandwidth (frequency range) for transrm'tting signals at a required rate was appreciated by Lord Kelvin, from his work on the transadantic cable; but it was not formulated until 1924 by H . Nyquist in America and K. Kumpfuller in Germany, who independendy stated the law that was developed to
190
190 its general form by R.V.L. Hartley in 1928-mat me transmission of a 'given quantity of information' is limited by the product of bandwidth and time. Hartley went further, to define information as the successive selection of signs, or words, from a given list or ensemble of possibilities. For this definition he rejected meaning as 'subjective' (although we usually think of the meanings of messages as what matters), for it is signals, not meanings, that are transmitted. But the word 'information' remained in use though it no longer referred to meanings, which can be confusing. Here we get more technical. Hartley showed that a message of N distinguishable signs (such as letters or dots and dashes) selected from a repertoire or ensemble of S signs, has Sn possibilities, and that the 'quantity of information' is most reasonably and usefully defined in logarithmic units, to make information measures additive. Hardey quantified information thus: H=
N\o% S 2
An essential notion for quantifying information is that the less likely the symbol or event, the more information it conveys. Information rate is not defined simply in terms of the number of symbols that can be transmitted, but also in terms of the probability (or the surprise value) of their occurrence. T o modify a favourite example of Bertrand Russell's, 'Dog bites man' does not convey much information - but '"Man bites dog" is news'. The second costs more to transmit, for to be sure such an unlikely message is correct we must know with confidence that it is free of error, and this requires expensive reliability - which is difficult to achieve in engineering, or for the nervous system. Since the simplest choice is yes or no (or on or off, for a switch) the information unit of a bit (binary choice) is useful. Norbert Weinerand Claude Shannon developed the Hartley approach by examining the statistical characteristics of signals, including the values of waves for analogue signals. They reinterpreted Hartley's Law, to define the average information of long sequences of n symbols as n
H=-'2p \ogp i
i
i
(The minus sign makes H positive, since it involves logarithms of p which is fractional.) It is well worth playing the dictionary game of finding a word with the 'Twenty questions' technique of asking, for example, 'Is it before K?' and, if it is, 'Is it before E?', and so on. Any word in a dictionary can usually be located within twelve such binary decisions, which is quite remarkable. This is surprising because we are not used to thinking in terms of powers of two. The number of binary choices
191
191 enabling selection of say a word, from a set of words, increases as follows (values rounded off): Binary decisions required
t 1 4 8 ifi 3* 64 128 256
Size of dictionary from which any word can be found (no. of headwords)
2 4 [6 (1.6X10) 256 (2.6 x c o ) 65000 (6.5x10*) 4300000000 (4.3 X I o ) 1g 000 000 000 000 000 000(1.9X10") 340 000 000 000000000000000000000000000000 (3.4X]o > 160000000000000000000000000000000000000 (1.6x10") 000000000000 000 000000000000000000000000 1
2
9
3a
Surely the power of this strategy has implications for how we recall memories. It is not possible, however, to use the amazingly efficient 'Twenty questions' technique except for items which are ordered in some way, such as words arranged alphabetically in dictionaries. Memories may be filed for our almost instant recall in some orderly way that is still beyond computer technology T h e greatest quantity of information that can be transmitted through a channel, with bandwidth u> over time r, in the presence of disturbing random noise, was shown by Claude Shannon to be * t log* | i +j^J (bits), where P and Ware mean signal and mean noise powers respectively. T h i s represents a definite limit which no channel (including the visual channel) can exceed. I f however the coding of the signals is non-optimal, the information rate may be very much lower, so we need to know the coding to assess physiological efficiency. There is something odd about information as described by Shannon's theory, which is now universally accepted as the best account, for information is quite different from anything in the natural sciences. Unlike normal causes, it depends not only on what has been, and what is, but also on alternatives of what might be. It is often said that information theory can give no account of meaning, but the situation is not quite so bleak - because the selected alternatives may have, or be, meanings. In order to apply information theory rigorously it is necessary to know the number of alternatives from which selections are made. Unfortunately we seldom ii ever know just what these are for humans, so information theory can seldom be rigorously applied outside purely engineering situations where we have full knowledge of the system and especially its range of alternatives. A law for human
192
192 response time was, however, established at Cambridge by Edmund Hick in 1952, which showed that in situations where there are clearly defined choices, such as pressing response keys to lights, the mean decision time, (, increases with the number of the possibilities, n, by t = K\o% (n + i )
The decision or choice time (the disjunctive reaction time minus the simple reaction time) is very nearly proportional to the number of bits per stimulus {where a 'bit' is a binary choice). As a matter of history, I was the only subject apart from Edmund Hick for this experiment, which went on one hour a day for months. Hick did not complete it himself, so Hick's Law is based on my (very) nervous system. The relation was later found by R. Hyman (1953), who showed that with each added possibility the choice time increases by hist over a tenth of a second; and that this same increase in choice occurs no matter how the information is increased - by changing the relative frequencies of the alternatives, their sequential dependencies (introducing redundancy, so that some tended to follow others in predictable sequences), or by increasing the number of possibilities. The maximum rate of information (bit rate) of even the most skilled human operator looks surprisingly small: about 22 bits per second for an expert pianist; speech does not exceed about 26 bits, and silent reading may reach 44 bits per second. This seems low in engineering terms, where bit rates of thousands per second may be achieved, but we have to remember that far more is going on - with more choices to be made - in the richness of the brain's circuits. At present, information-handling computer systems are not comparable to us because they cannot do anything like as much as we can. This takes us to the Big Snag with information theory when it is applied to us. Unlike engineering systems, organisms are not stricdy limited to the set of choices allowed by the experimenter's situation or task. Thus, ideally, doing Hick's experiment I should only have been able to respond to the ten little lights, which were pinned in a random arrangement on cork mats, each lit with a particular key from the row under one's fingers. But when less than all ten were being used, to restrict the value of n possibilities, I was not, strictly speaking, blind to the others. And I could respond to a knock on the door, or to Edmund Hick asking me why I was not doing better, or saying that it was lunch time. One was not deaf or blind to all except the allowed alternative choices of the experimental condition, in spite of the long practice with each number of lights in use - so the total bit rate must have been far greater than measured. Although Hick's Law works beautifully in some conditions (interestingly when there is not too much practice), it cannot truly quantify the information we handle because we cannot
193
set or assess the number of alternatives, n, we may choose from. The trouble is that, unlike a computer system, a human being's range of alternatives cannot be strictly limited by the situation he is in, or by an experimenter, or by his most concentrated attention. In a much quoted paper, bearing the splendid tide: 'The magic number seven plus or minus two', the American psychologist George Miller (1956) suggested that there is an absolute limit of around seven similar items for the immediate span of apprehension. Thus we can estimate at a glance, without counting, up to about seven dots spaced randomly. But the effective information in a perceptual span (the 'specious present') can be gready increased by 'chunking' bits into larger units. This is a form of coding of data, requiring decoding of course. The most powerful coding appears to be language. Coding can set the number of bits per chunk; but the number of chunks that can be retained in immediate memory is limited to around seven. Presumably Chinese ideogram characters are efficient chunks, each conveying a lot of information. Much of learning is chunking bits of information into large units which can be stored in memory and recalled as a unit, rather like a rich ideogram. Are perceived objects memory-chunked bits of information? Again, it is commonly said that information theory has nothing to say about meaning, but this is not entirely true. Donald MacKay (1960a) produced an interesting way of looking at this. He suggested that meaning was related to conditional readiness, as the meaning of a sentence (or of a perceived event) changes the partem of possibilities for future action. He gave a working definition of meaning as the 'selective function on the range of the recipient's states of conditional readiness for goal-directed activity; so the meaning of a message to you is its selective function on the range of your states of conditional readiness'. Defined in this way, meaning is clearly a relationship between message and recipient, rather than a unique property of the message alone. So there is a 'subjective' side to the engineer's information. Can this be measured? It might be, if we knew the selective function of a message. Then we could apply information theory to quantify meaning, at least in some cases. MacKay went on in a later paper (1960b), 'What is a question?', to suggest that states of readiness for organisms are large numbers of conditional probabilities. Asking a question is a means of changing the conditional probabilities Of the questioner's states of readiness. This notion can be expressed in computer-progranuning terms. If there remains a 'problem of meaning' the problem lies somewhere beyond the adequacy of this notion of selections from, or changes of, states of readiness. These can be described in computer terms, and be implemented by computers with no special problems. Surely discussions of meaning could well start at this point.
194
194 It is sometimes said that true statements have more meaning than false statements. And philosophers frequently deny meaning to logical (and even to contingent) impossibilities. T h u s '2 + 2=5', 'She is a dark-haired blonde', might be said to be meaningless. Such logical impossibilities are internal inconsistencies of the state of readiness, such that they are program-stoppers, preventing changes in states of readiness. It is an interesting question how general program-stoppers must be before we call them 'logical errors'. Is it possible that here lies the distinction between 'logical' and 'contingent'? o
r
Reaction times are of continuing interest and an acknowledged source of useful data for teasing out processes of perception and decision-taking. But, given that reaction times depend on the repertoire of possibilities, which cannot be established - which is what destroyed the effective use of information theory-how can reactiontime experiments work? Why do they not suffer from the same deep trouble? It is sad that information theory is virtually dead in current thinking on perception. Perhaps, like poor mistaken Juliet it deserves awakening from cryptic sleep: but if so - wherefore art thou Romeo?
REFERENCES
Hick, W . E .
11-26.
(1952) 'On the
rate of gain of information', Q. J. Psychol.
4,
Hyman, R. (1953) 'Stimulus information as a determinant of reaction time', J. Experimental Psychol 45, 188-96. MacKay, D.M. (1960a) 'Meaning and mechanisms', reprinted as chapter 3 in Information, Mechanisms and Meaning (1969), Cambridge, Mass., M I T Press. MacKay, D . M . (1960b) 'What is a question?', reprinred as chapter 4 in Information, Mechanisms and Meaning {1969), Cambridge, Mass., M I T Press. MacKay, D . M . (1969) Information, Mechanism and Meaning, Cambridge, Mass., M I T Press. Miller, G.A. (1956) "The magical number seven plus or minus two: some limits to our capacity for processing information',/'g'fAo/. Rev. 63, 81-97.
195
Reprinted with permission from Psychological Review, Vol. 61, No. 3, pp. 183-193, 1954 © 1954 American Psychological Association
SOME
INFORMATIONAL
ASPECTS
O F VISUAL
PERCEPTION
FRED A T T N E A V E Perceptual and Motor Skills Research Laboratory, Human Resources Research Center' The ideas of information theory are at present stimulating many different areas of psychological inquiry. In providing techniques for quantifying situations which have hitherto been difficult or impossible to quantify, they suggest new and more precise ways of conceptualizing these situations (see Miller [12] for a general discussion and bibliography). Events ordered in time are particularly amenable to informational analysis; thus language sequences are being extensively studied, and other sequences, such as those of music, plainly invite research. I n this paper I shall indicate some of the ways in which the concepts and techniques of information theory may clarify our understanding of visual perception. When we begin to consider perception as an information-handling process, it quickly becomes clear that much of the information received by any higher organism is redundant. Sensory events are highly interdependent in both space and time: if we know at a given moment the states of a limited number of receptors (i.e., whether they are firing or not firing), we can make better-tban-chance inferences with respect to the prior and subsequent states of these receptors, and also with respect to the present, prior, and subsequent states of other receptors. T h e preceding statement, taken in its broadest im-
plications, is precisely equivalent to an assertion that tbe world as we know it is lawful. I n the present discussion, however, we shall restrict our attention to special types of lawfulness which may exist in space at a fixed time, and which seem particularly relevant to processes of visual perception.
T H E NATURE OF REDUNDANCY IN V I S U A L STIMULATION: A DEMONSTRATION Consider the very simple situation presented in F i g . 1. With a modicum of effort, the reader may be able to see this as an ink bottle on the corner of a desk. L e t us suppose that the background is a uniformly white wall, that the desk is a uniform brown, and that the bottle is completely black. T h e visual stimulation from these objects is highly redundant in the sense that portions of the field are highly predictable from other portions. I n order to demonstrate this fact and its perceptual significance, we may employ a variant of the "guessing game" technique with which Shannon (17) has studied the
40
10
» 1
The experimental w o r k for this study was performed as part of tbe United States A i r Force Human Resources Research and D e velopment Program. The opinions and conclusions contained in this report are those of the author. They are not to be construed as reflecting the views or indorsement of the D e partment of the Air Force.
196
10
COLUMNS
F I G . 1. Illustration of redundant visual stimulation
184 redundancy of printed English. We may divide the picture into arbitrarilysmall elements which we "transmit" to a subject ( S ) in a cumulative sequence, having him guess at the color of each successive element until he is correct. This method of analysis resembles the scanning process used in television and facsimile systems, and accomplishes the like purpose of transforming two spatial dimensions into a single sequence in time. We are in no way supposing or assuming, however, that perception normally involves any such scanning process. If the picture is divided into SO rows and 80 columns, as indicated, our S will guess at each of 4,000 cells as many times as necessary to determine which of the three colors it has. If his error score is significantly less than chance [2/3 X 4,000 + 1 / 2 ( 2 / 3 X 4,000) = 4,000], it is evident that the picture is to some degree redundant. Actually, he may be expected to guess his way through F i g . 1 with only I S or 20 errors. I t is fairly apparent that the technique described, in its present form, is limited in applicability to simple and somewhat contrived situations. With suitable modification it may have general usefulness as a research tool, but it is introduced into the present paper for demon strat ion al purposes only. L e t us follow a hypothetical subject through this procedure in some detail, noting carefully the places where he is most likely to make errors, since these are the places in which information is concentrated. T o begin, we give him an 80 X SO sheet of graph paper, telling him that he is to guess whether each cell is white, black, or brown, starting in the lower left corner and proceeding across the first row, then across the second, and so on to the last cell in the upper right comer. Whenever he makes an error, he is allowed to guess a second and, if necessary, a third time until he is corr e c t H e keeps a record of the cells he
has been over by filling in black and brown ones with pencil marks of appropriate color, leaving white ones blank. After a few errors at the beginning of the first row, he will discover that the next cell is "always" white, and predict accordingly. T h i s prediction will be correct as far as Column 20, but on 21 it will be wrong. After a few more errors he will learn that "brown" is his best prediction, as in fact it is to the end of the row. Chances are good that the subject will assume the second row to be exactly like the first, in which case he will guess it with no errors; otherwise he may make an error or two at the beginning, or at the edge of the "table," as before. He is almost certain to be entirely correct on Row 3, and on subsequent rows through 20. O n Row 21, however, it is equally certain that he will erroneously predict a transition from white to brown on Column 21, where the corner of the table is passed. Our subject's behavior to this point demonstrates two principles which may be discussed before we follow him through the remainder of his predictions. I t is evident that redundant visual stimulation results from either (a) an area of homogeneous color ("color" is used in the broad sense here, and includes brightness), or (b) a contour of homogeneous direction or slope. I n other words, information is concentrated along contours (i.e., regions where color changes abruptly) , and is further concentrated at those points on a contour at which its direction changes most rapidly (i.e., at angles or peaks of curvature) . 2
2
Our "scanning" procedure introduces a certain artifact here, in that a particular subject will make errors at a linear contour only the first few times he crosses it. It is fairly obvious that if the starting point of the sequence and the direction of scan were varied randomly over a large number of subjects, summated errors would be distributed evenly along such a straight contour.
197
185 which the dots represented. A good sample of the results is shown in F i g . 2: radial bars indicate the relative frequency with which dots were placed on each of the segments into which the contour was divided for scoring purposes. I t is clear that 5s show a great deal of agreement in their abstractions of points best representing the shape, and most of these points are taken from regions where the contour is most different from a straight line. This conclusion is verified by detailed comparisons of dot frequencies with measured curvatures on both the figure shown and others. F I G . 2. Subjects attempted to approximate the closed figure shown above with a pattern of 10 dots. Radiating bars indicate the relative frequency with which various portions of the outline were represented by dots chosen. Evidence from other and entirely different situations supports both of these inferences. The concentration of information in contours is illustrated by the remarkably similar appearance of objects alike in contour and different otherwise. The "same" triangle, for example, may be either white on black or green on white. Even more impressive is the familiar fact that an artist's sketch, in which lines are substituted for sharp color gradients, may constitute a readily identifiable representation of a person or thing. An experiment relevant to the second principle, i.e., that information is further concentrated at points where a contour changes direction most rapidly, may be summarized briefly. Eighty Ss were instructed to draw, for each of 16 outline shapes, a pattern of 10 dots which would resemble the shape as closely as possible, and then to indicate on the original outline the exact places
Common objects may be represented with great economy, and fairly striking fidelity, by copying the points at which their contours change direction maximally, and then connecting these points appropriately with a straightedge. Figure 3 was drawn by applying this technique, as mechanically as possible, to a real sleeping cat. The informational content of a drawing like this may be considered to consist of two components: one describing the positions of the points, the other indicating which points are connected with which others. The first of these components will almost always contain more information than the second, but its exact share will depend upon the precision with which positions are designated, and will further vary from object to object. L e t us now return to the hypothetical subject whom we left between the corner
8
s
This study has been previously published only in the form of a mimeographed note: "The Relative Importance of Parts of a Contour," Research Note P&MS 5 1 - 8 , Human Resources Research Center, November 1951,
F I G . 3. Drawing made by abstracting 3 8 points of maximum curvature from the contours of a sleeping cat, and connecting these points appropriately with a straightedge.
198
186 of the table and the ink bottle in F i g . 1. H i s errors will follow the principles we have just been discussing until he reaches the serrated shoulders of the bottle. ( A straight 4 5 ° line would be represented in this way because of the grain of the coordinate system, but we shall consider that the bottle is actually serrated, as it is from the subject's point of view.) O n the left shoulder there are 13 right angles, but these angles contain considerably less than 13 times the information of an angle in isolation like the corner of the table. This is true because they fall into a pattern which is repetitive, or redundant in the everyday sense of the term. T h e y will cease to evoke errors as soon as S perceives their regularity and extrapolates it. This extrapolation, precisely like 5's previous extrapolations of color and slope, will have validity only over a limited range and will itself lead to error on Row 38, Column 48. At about the same time that he discovers the regularity of the stair-step pattern (or perhaps a little before), our 5 will also perceive that the ink bottle is symmetrical, i.e., that the right contour is predictable from the left one by means of a simple reversal. As a result he is very unlikely to make any further errors on the right side above Row 32 or 33. Symmetry, then, constitutes another form of redundancy.* I t should be fairly evident by now * The reader may be comforted to know that six subjects have actually been run on the task described. Their errors, which ranged in number from 13 to 26, were distributed as suggested above, with a single interesting exception: 4 of tbe 6 5s assumed on Row 1 that the brown area would be located symmetrically within the field, and guessed "white" on Column 61. B y tbe use of Shannon's formulas (17) it was estimated that the field contains between 34 (lower limit) and 1S6 (upper limit) bits of information, in contrast to a possible maximum of 6,340 bits. The redundancy is thus calculated to be between 97.5 and 99.5 per cent.
that many of the gestalt principles of perceptual organization pertain essentially to information distribution. T h e good gestalt is a figure with some high degree of internal redundancy. T h a t the grouping laws of similarity, good continuation, and common jate all refer to conditions which reduce uncertainty is clear enough after the preceding discussion, and we shall presently see that proximity may be conceptualized in a like manner. It is not surprising that the perceptual machinery should "group" those portions of its input which share tbe same information: any system handling redundant information in an efficient manner would necessarily do something of the sort. Musatti (20) came very close to the present point when he suggested that a single principle of homogeneity might subsume Wertheimer's laws as special cases. All of our hypothetical S's extrapolations have involved some variety of homogeneity (or invariance), either of color, of slope, or of pattern. The kinds of extrapolation that have been discussed certainly do not exhaust the repertory of the human observer. For example, if the brightness of a surface were changed at a constant rate along some spatial extent, an observer could probably extrapolate this change with a fair degree of accuracy (given an appropriate response medium, such as choosing from a set of Munsell color patches). Likewise, we may reasonably suppose that a contour, the direction of which changes at a constant rate (i.e., the arc of a circle), could be extrapolated. Any sort of physical invariance whatsoever constitutes a source of redundancy for an organism capable of abstracting the invariance and utilizing it appropriately, but we actually know very little about the limits of the human perceptual machinery with respect to such abilities. A group of psychophysi-
199
187 cal studies determining the accuracy with which observers are able to extrapolate certain discrete and continuous functions of varying complexity must be carried out before we can usefully discuss any but the simplest cases." A troublesome question arises in this connection: where does perception leave off and inductive reasoning begin? T h e abstraction of simple homogeneities from a visual field does not appear to be different, in its formal aspects, from the induction of a highly general scientific law from a mass of experimental data. Certain subjective differences are obvious enough: thus reasoning seems to involve conscious effort, whereas perception seems to involve a set of processes whereby information is predigested before it ever reaches awareness. When extrapolations are required of a subject in an experimental situation, however, it is difficult or impossible for the experimenter to be certain whether the subject is responding on an "intuitive" or a "deliberative" basis. I do not know any general solution to this problem, and can only suggest that a limited control may be exercised by way of the establishment of a desired set in the subject.
T H E ABSTRACTION OF STATISTICAL PARAMETERS
Although F i g . 1 presents a situation much simpler, or more redundant, than the visual situations which ordinarily confront us, the reader need merely look around the room in which he is sitting to find that the principles illustrated apply to the real world. Further, it may be argued on neurological grounds that the human brain could not possibly utilize all the information provided by states of stimulation which were not highly redundant. According to Polyak's (14) estimate, the retina contains not less than four million cones. At any given instant each of these cones may be in either of two states: firing or not firing. Thus the retina as a whole might be in any one of about 2».'">°-°°° or 1QI.2OO.OTO states, each representing a different configuration of visual stimulation. Now, if by some unspecified mechanism each of these states were to evoke a different unitary response, and if a unitary response consists merely of the firing of a single unique neuron, then i t ) ' ' of such response-neurons would be required. T h e fantastic magnitude of this figure becomes somewhat apparent when one calculates that only about 10 neurons could be packed into a cubic light year. T h e fact that the number of patterns of response-neurons might plausibly equal the number of retinal configurations simplifies matters only if there are certain one-to-one connections between cones and responseneurons, in which case the response is to some degree merely a copy of the stimulus. 1
3 0 0
0 0 0
54
s
There is, however, a great deal more that can be said about the simplest cases. Vernier acuity demonstrates that, under optimal conditions, error of extrapolation may be less than the "minimum separable." It has been found by Salomon (16) that the error made in "aiming" a line at a point some constant distance from its end is a decreasing, negatively accelerated function of the line's length. This may be taken to mean that increasing the length of a tine adds information about its extension, but at a decreasing rate, somewhat as increasing the length of a passage of English text adds decreasing increments of information about the next letter (13, 17). Dr. Karl Zener, under whose direction the Salomon study was done, is at present conducting a program of related psychophysical experiments which may answer some of the questions raised above.
We may nevertheless ask: how would an observer respond to a situation in which the retinal receptors were stimulated quite independently of one another? This situation would be in practice very difficult to achieve (even more difficult than its diametric opposite, the
200
188 tion of the random field suggested above might look at some particular instant. Perhaps the most striking thing about the figure is the subjective impression of homogeneity that it gives: the left half of tbe figure seems, at least in a general way, very much like the right half. T h i s is remarkable because we have previously associated homogeneity with redundancy, and F i g . 4 was constructed to be completely nonredundant. Now, in psychological terms, it is fairly clear that the characteristic with respect to which the figure appears homogeneous is what Gibson (6) would call its texture. I n physical terms, two invariant factors may be specified: (a) the probability (.50) that any cell will be black rather than white, and (b) the size of cells. Both of these factors probably contribute to perceived texture, which is undoubtedly a multidimensional variable, though the latter may be somewhat the more important. I f the figure is viewed from a sufficient distance, these two parameters become identifiable with (a) the central tendency, and ( 6 ) the dispersion, of a continuous brightness distribution in two dimensions.
F I G . 4. A "random field" consisting of 19,600 cells. The state of each cell (black vs. white) was determined independently with a p of .50. Ganzfeld), particularly if we demanded that the stimulation at a given moment (which might be supposed to have a duration of about 100 msec, [see Attneave and McReynolds, 1 ] ) be entirely independent of the stimulation at any other moment. I n an effort to get some notion of what such a random field would be like, F i g . 4 was constructed. E a c h of the 140 = 10,600 small cells of the figure was either filled or not filled according to the value of a number obtained from a conversion of Snedecor's (18) table of random numbers from decimal to binary. I f the figure is viewed from a distance such that the angle subtended by a cell is of the order of the "minimum separable" (about 1'), it illustrates roughly how a small porz
8
* This laborious task was carried out by Airmen 1/C W. H. Price and E . F . Chiburis. Unfortunately, a slight distortion of the relative sizes of black and white cells was introduced in the photographic copying process. The figure was constructed not only for demonstration purposes, but also to serve as a source of random patterns for experimental use. It may also be used wherever a table of random binary numbers is needed, facilitating, for example, the selection of random "draws" from a binomial distribution.
I t appears, then, that when some portion of the visual field contains a quantity of information grossly in excess of the observer's perceptual capacity, he treats those components of information which do not have redundant representation somewhat as a statistician treats "error variance," averaging out particulars and abstracting certain statistical homogeneities. Such an averaging process was involved in drawing the cat for Fig. 3. I t was said earlier that the points of the drawing corresponded to places of maximum curvature on the contour of the cat, but this was not strictly correct; if the principle had been followed rigidly, it would have been necessary to represent the ends of individual hairs by points. I n observ-
201
189 ing a cat, however, one does not ordinarily perceive its hairs as individual entities; instead one perceives that the cat is jurry. Furriness is a kind of texture; the statistical parameters which characterize it presumably involve averages of shape and direction, as well as size, of elements. T h e perceived contour of a cat (e.g., the contour from which the points of F i g . 3 were taken) is the resultant of an orthogonal averaging process in which texture is eliminated or smoothed out almost entirely, somewhat as if a photograph of the object were blurred and then printed on high-contrast paper (cf. Rashevsky, 15, and Culbertson, 5 ) . The sense in which a surface of a particular texture may be said to provide redundant stimulation has perhaps been adequately indicated. T h i s sort of redundancy might be demonstrated by the guessing-game technique, with a suitable modification in the level of prediction required, i.e., by increasing the unit area to be predicted and requiring the subject to select from a multidimensional array of samples the texture (i.e., the statistical parameters) which he believes the next unit will have. I n view of Gibson's (6) convincing argument that a physical edge, or contour, is as likely to be represented in vision by an abrupt texture change as by an abrupt color change, I have considered it important to show how texture may be substituted for color without materially altering the principles derived from F i g . I . PERCEPTION AS ECONOMICAL DESCRIPTION
I t is sometimes said that the objective of science is to describe nature economically. We have reason to believe, however, that some such process of parsimonious description has its beginnings on a fairly naive perceptual level, in
scientists and their fellow organisms alike; thus the difficulty, mentioned earlier, of distinguishing between perception and inductive reasoning. I t appears likely that a major function of the perceptual machinery is to strip away some of the redundancy of stimulation, to describe or encode incoming information in a form more economical than that in which it impinges on the receptors. If this point of view is sound, we should be able to generate plausible hypotheses as to the nature of specific perceptual processes by considering rational operations which one might deliberately employ to reduce redundancy. The approach suggested, as it applies to the perception of a static visual field, is equivalent to that of a communications engineer who wishes to design a system for transmitting pictures of real things over a practically noise-free channel with the utmost economy of channel time and band width, but in a manner designed to meet standards such as human observers are likely to have. Some of the reduction principles which he might usefully employ in such a system are listed below. I t will be found that these principles serve to summarize and integrate ideas which have been developed somewhat informally in the foregoing sections, as well as to introduce new considerations. T h e principles may be grouped according to the forms of redundancy with which they are concerned: thus 1-4 deal with varieties of continuous regularity; 5 and 6 with discontinuous regularity, or recurrence; 7-9 with proximity; and 10 with situations involving interaction. 1. An area of homogeneous color may be described by specifying the color and the boundaries of the area over which it is homogeneous. ( I t is assumed that limits of error tolerance on relevant dimensions have been agreed upon, e.g., that there is some definite number of
202
190 colors from which the receiving mechanism may be directed to choose.) 2. Likewise, an area of homogeneous texture may be described by specifying the statistical parameters which characterize the texture and the boundaries of the area over which these parameters are relatively invariant. Thus, if F i g . 4 represented a part of the upholstery of a sofa, it would probably be satisfactory simply to instruct the receiving mechanism to reproduce the texture by filling in cells of a certain size from any table of random numbers. I t is true that this process would result in the complete loss of 19,600 bits of information; the essential point is that we are dealing here with a class of stimuli from which such a huge information loss is perceptually tolerable. 3. A n area over which either color or texture varies according to some regular function may be described by specifying the function and the boundaries of the area over which it obtains (cf. Gibson's [6] texture gradient). This principle actually implies both 1 and 2 as special cases. 4. Likewise, if some segment of an area boundary (i.e., contour) either maintains a constant direction or varies according to some other regular function, it may be described by specifying the function and the loci of its limiting points. Figure 3 illustrates a special case of this principle. 5. I f two or more identical stimulus patterns (these might be either successive portions of a contour, or separate and discrete objects) appear at different places in the same field, all may be described by describing one and specifying the positions of the others and the fact that they are identical (cf. similarity as a grouping l a w ) . 6. I f two or more patterns are similar but not identical, it may be economical to proceed as in S, in addition specifying either (a) how subsequent patterns dif-
fer from the first, or else (b) how each pattern differs from some skeleton pattern which includes the communalities of the group (cf. the "schema-with-correction" idea discussed by Woodworth [ 2 0 ] ; also Hebb's [7] treatment of perceptual schemata). 7. When the spatial loci of a number of points are to be described in some arbitrary order, and the points are arranged in clusters or proximity groups (as in F i g . 5 ) , it may be economical to describe the points of each group with respect to some local origin ( 0 ' or 0 " ) , transmitting as a separate component the positions of the local origins with respect either to each other or to some arbitrary origin, whichever is required. Since the points occupy a smaller range of alternative coordinates on the local axes than on arbitrary axes, less information is required for their specification. I f the amount of information thus saved is greater than the amount needed to specify the positions of the local origins, a net saving will result. What is redundant in the present case is the approximate location of points in a cluster: this component is isolated out when a local origin is described (cf. the con¬ Y
:
!
i *
M t •
r'
F I G . 5. A functional aspect of proximitygrouping is illustrated. Tbe loci of clustered points may be described with choices from a smaller set of numbers if local origins are used.
203
191 cepts of "within" and "between" variance). T h e local origin principle may also be used in conjunction with some regular scanning procedure if tbe order in which the points are to be specified is not predetermined (but see also 9, below). T h e relevance of these considerations to proximity as a perceptual grouping law is evident. 8. The preceding principle may be generalized to apply to dimensions other than spatial ones; e.g., brightness, coarseness of texture, etc. A "local origin" on such a continuum would appear to have essentially the characteristics of Helson's ( 8 ) adaptation-level, in terms of which constancy phenomena and a variety of other psychophysical findings may be accounted for. This generalized principle is closely similar to 6 above, the chief difference being that 6 is applicable to combinations of discrete variables, or to situations of ambiguous dimensional organization. 9. I f the loci of a number of points are to be described, and the order in which they are taken is immaterial, they may be arranged in a sequence such that the distances between adjacent points are minimized, and transmitted with each point serving as origin for the one following it. This procedure will result in some saving if the points are clustered, as in F i g . 5, but it is most clearly applicable when the points are "strung-out" in some obvious sequence. In the latter case, a further economy may be achieved by the use of special coordinates such as distance from a line passing through the two preceding points (or from an arc through the three preceding points, etc.; cf. 4 above). 10. Certain areas and objects may be described in a relatively simple way, by procedures of the sort suggested above, if they are first subjected to some systematic distortion or transformation. Consider the case of a complex,
204
symmetrical, two-dimensional pattern viewed from an angle such that its retinal or photographic image is not symmetrical. I t will be economical to transmit a description of the pattern as if it were in the frontal plane, and thus symmetrical (eliminating the redundancy of symmetry by means of 6 a ) , together with a description of the transformation which relates the frontal aspect described to the oblique aspect in which the pattern is viewed (cf. Gibson's [6] discussion of perspective transformations; also the "Thompsonian coordinates" of D'Arcy Thompson [ 1 9 ] ) . Koffka ( 1 0 ) and other gestalt psychologists have held that many objects have some "preferred" aspect, and that this aspect has the characteristics of a "good gestalt." T h e present principle supports this view on functional grounds, since the perceptual transformation of a figure to an aspect in which similarities among parts are maximized may be interpreted as the initial step in an efficient information-digesting process. I t should be clearly recognized, however, that an over-all economy is achieved only if the amount of information required to describe the transformation is less than the amount of information saved by virtue of the transformation; thus a transformation must be relatively simple to be considered useful, at least by the present criterion. Let me indicate briefly how these considerations may be integrated with others of a more general nature. Interdependencies among sensory events may exist either in space or in time, or they may cut across both space and time. I n studying the redundancy of spoken English ( 1 1 ) , for example, one is dealing with interdependences which may be considered purely temporal. The present discussion has been restricted, quite arbitrarily, to relationships in space: to forms of redundancy and informationdistribution which may obtain in the
192 visual field at a particular instant, and which a computer of conceivable complexity might evaluate from a photograph. T h e extension of the visual field in time, which I propose to discuss in a subsequent paper, introduces new v a rieties of redundancy involving the temporal continuation or recurrence of spatial configurations which may be nonredundant at any instant considered in isolation. Any individual learns a great deal, over his life span, about whatgoes-with-what. Thus, if an ear is disclosed in a situation like that illustrated by F i g . 1, the observer can predict that a mouth, nose, eyes, etc. are also present, and approximately where they are. This sort of redundancy is spatiotemporal in its basis; predictions are not possible merely on the basis of the present visual field, but depend also upon previous fields which have contained faces. Principle 6 above suggests the approach to economical description which might be extended to such cases. Further, as Brunswik (2, 3, 4 ) has pointed out in some detail, ecological principles of very broad generality may be derived from experience.* F o r example, the frequency with which an observer has encountered symmetrical objects in his past may certainly affect the point at which, in predicting successive ceils of F i g . 1, he "assumes" that the ink bottle is symmetrical. Likewise in terms of economical encoding; each of the varieties of spatial redundancy suggested above will itself occur with some determinate frequency over any given set of fields (e.g., the r Brunswik ( 2 ) , Hebb (7), and the Ames group at Princeton (9) have advanced views concerning the role of experience i n perception which have much in common w i l h one another and w i t h m y o w n position i n the matter. I t appears to me, however, that they have i n general tended to underestimate (as the gestalt psychologists have somewhat overestimated) the importance of lawful relationships which may exist w i t h i n the static and isolated visual held.
set of pictures which a computer-transmitter might have been required to handle over some period of past operation), and a knowledge of this and related frequencies may be used in determining the optimal assignments of actual code symbols. As a result of factors such as these, spatial and spatiotemporal redundancy (or entropy) are difficult to separate empirically, but the distinction remains a conceptually convenient one. T h e foregoing reduction principles make no pretense to exhaustiveness. I t should be emphasized that there are as many kinds of redundancy in the visual field as there are kinds of regularity or lawfulness; an attempt to consider them all would be somewhat presumptuous on one hand, and almost certainly irrelevant to perceptual processes on the other. I t may further be admitted that the principles which have been given are themselves highly redundant in the sense that they could be stated much more economically on a higher level of abstraction. This logical redundancy is not inadvertent, however: if one were faced with the engineering problem suggested earlier, he would undoubtedly find it necessary to break the problem down in some manner such as the foregoing, and to design a multiplicity of mechanisms to perform operations of the sort indicated (some principles, e.g., 6, would require further breakdown for this purpose). Likewise, the principles are frankly intended to suggest operations which the perceptual machinery may actually perform, and accordingly the types of measurement which are likely to prove appropriate in the quantitative psychophysical study of complex perceptual processes.
R E F E R E N C E S 1. A T T N E A V E ,
F., :
MCRETNOLDS,
P. W.
A
visual beat phenomenon. Amer. J. Psy-
chol.,
205
1950, 6 3 , 1 0 7 - 1 1 0 .
193 2.
E . Systematic and representative design of psychological experiments: with results in physical and so-
( E d . ) , Handbook oj experimental psy-
BHUNSWIK,
cial perception. Berkeley:
Univer. of
chology. New Y o r k : Wiley, 1951. Pp. I04O-1OJ4. 12. M I L L E R , G . A. What is information meas-
California Press, 1947. 3.
BRUNSWIX, E .
urement?
The conceptual framework
of psychology. Chicago: Chicago Press, 1952. 4. B R O N S W I K , E . , & K A M I T A , J .
Univer. of
Amer. Psychologist, 1953, 8,
3-11. 13. N E W M A N ,
E . B . , & GERSTMATJ, L . S.
A
new method for analyzing printed EngEcological
cue-validity of "proximity" and of other gestalt factors. Amer. J. Psychol., 1953, 56, 20-32.
lish.
J. exp. Psychol, 1952, 44, 114¬
125.
S. L . The retina. Chicago: Univer. of Chicago Press, 1941. 5. CiTLBEKTsou, J . T . Consciousness and be15. R A S B E V S K Y , N . Mathematical biophysics. havior. Dubuque: Wm. C . Brown, Chicago: Univer. of Chicago Press, 1950. 1948. 6. G I B S O N , J . J . The perception of the visual 16. SALOMON, A. D . Visual field factors in world. Boston: Houghton Mifflin, 1950. tbe perception of direction. Amer. J. 7. H E B B , D . O. Organization of behavior. Psychol., 1947, 60, 68-88. New York: Wiley, 1949. 17. SHANNON, C . E . Prediction and entropy 8. H E L S O N , H . Adaptation-level as frame
of reference for prediction of psycho-
14.
POLVAZ,
of printed English.
Bell Syst. tech. J.,
1951, 30, 50-64. 18. SNEDECOR, G . W . Statistical methods. 60, 1-29. Ames: Iowa State Coll. Press, 1946. 9. I T T E L S O N , W . H . The Ames demonstra19. T H O M P S O N , D , W . Growth and form. tions in perception. Princeton: Princeton Univer. Press, 1952. New York: Macmfflan, 1942. 10. K O F F K A , K . Principles of gestalt psychol- 20. WOODWORTH, R . S . Experimental psyogy. New York: Harcourt, Brace, 1935. chology. New York: Holt, 1938.
physical data.
Amer. J. Psychol., 1947,
11. L I C K L I D E R , J . C . R „ * M I L L E R , G . A.
perception of speech.
The
I n S. S. Stevens
206
(Received June 26, 1953)
Reprinted with permission from Psychological Review. Vol.63, No.2, pp. 81-97,1956 © 1956 American Psychological Association
THE PSYCHOLOGICAL REVIEW THE
MAGICAL N U M B E R SEVEN, PLUS O R MINUS TWO: SOME LIMITS ON O U R CAPACITY F O R PROCESSING
INFORMATION'
G E O R G E A. M I L L E R
Harvard V nsverni j My
judgment. Historical accident, however, has decreed that they should have another name. We now call them experiments on the capacity of people to transmit information. Since these experiments would not have been done without the appearance of information theory on the psychological scene, and since the results are analyzed in terms of the concepts of information theory, I shall have to preface my discussion with a few remarks about this theory.
p r o b l e m is t h a t I h a v e b e e n perse-
c u t e d b y a s integer. t h i s n u m b e r has
F o r seven
years
followed m e a r o u n d , has
intruded i a m y most private data, and has
a s s a u l t e d m e f r o m t h e pages of o u r
most public journals.
T h i s number as-
s u m e s a v a r i e t y o f disguises, b e i n g sometimes a little
little larger a n d
smaller
than
sometimes
usual,
c h a n g i n g so m u c h a s to be able.
The
a
never
unrecogniz-
persistence w i t h w h i c h t h i s
number plagues
me
a random accident. a
but
famous
senator,
is far more
than
T h e r e i s , to q u o t e a design
behind
INFORMATION MEASUREMENT
it,
some p a t t e r n g o v e r n i n g its a p p e a r a n c e s . E i t h e r there r e a l l y is s o m e t h i n g u n u s u a l a b o u t the n u m b e r o r else I a m suffering f r o m d e l u s i o n s of p e r s e c u t i o n . I s h a l l begin m y c a s e h i s t o r y b y telling you
about
some
experiments
that
tested h o w a c c u r a t e l y people c a n assign n u m b e r s to t h e m a g n i t u d e s
of
aspects
the
tradi-
psychology
these
tional
of
a
stimulus.
language
of
In
various
w o u l d be c a l l e d e x p e r i m e n t s i n a b s o l u t e ' This paper w u first read u an Invited Address before the Eastern Psychological Association in Philadelphia on April IS, 1955. Preparation of the paper w u supported by the Harvard Psycho-Acoustic Laboratory un¬ der Contract N5ori-76 between Harvard U n i versity and the Office of Naval Research, TJ. S. Navy (Project N R 1 4 J - I 0 1 , Report P N R - 1 7 4 ) . Reproduction for any purpose of the U . 5. Government is permitted.
The "amount of information" is exactly the same concept that we have talked about for years under the name of "variance." The equations are different, but if we bold tight to the idea that anything that increases the variance also increases the amount of information we cannot go far astray. The advantages of this new way of talking about variance are simple enough. Variance is always stated in terms of the unit of measurement— inches, pounds, volts, etc.—whereas the amount of information is a dimensionless quantity. Since the information in a discrete statistical distribution does not depend upon the unit of measurement, we can extend the concept to situations where we have no metric and we would not ordinarily think of using
207
82
the variance. And it also enables us to compare results obtained in quite different experimental situations where it would be meaningless to compare variances based on different metrics. So there are some good reasons for adopting the newer concept. The similarity of variance and amount of information might be explained this way: When we have a large variance, we are very ignorant about what is going to happen. If we are very ignorant, then when we make the observation it gives us a lot of information. On the other hand, if the variance is very small, we know in advance how our observation must come out, so we get little information from making the observation. If you will now imagine a communication system, you will realize that there is a great deal of variability about what goes into the system and also a great deal of variability about what comes out. The input and the output can therefore be described in terms of their variance {or their information). If it is a good communication system, however, there must be some systematic relation between what goes in and what comes out. That is to say, the output will depend upon the input, or wUl be correlated with the input. If we measure this correlation, then we can say how much of the output variance is attributable to the input and how much is due to randomfluctuationsor "noise" introduced by the system during transmission. So we see that the measure of transmitted information is simply a measure of the input-output correlation. There are two simple rules to follow. Whenever I refer to "amount of information," you wUl understand "variance." And whenever I refer to "amount of transmitted information." you will understand "covariance" or "correlation." The situation can be described graphically by two partially overlapping cir-
cles. Then the left circle can be taken to represent the variance of the input, the right circle the variance of the output, and the overlap the covariance of input and output. I shall speak of the left circle as the amount of input information, the right circle as the amount of output information, and the overlap as the amount of transmitted information. In the experiments on absolute judgment, the observer is considered to be a communication channel. Then the left circle would represent the amount of information in the stimuli, the right circle the amount of information in his responses, and the overlap the stimulusresponse correlation as measured by the amount of transmitted information. The experimental problem is to increase tbe amount of input information and to measure the amount of transmitted information. If the observer's absolute judgments are quite accurate, then nearly all of the input information will be transmitted and will be recoverable from his responses. If he makes errors, then the transmitted information may be considerably less than the input. We expect that, as we increase the amount of input information, the observer will begin to make more and more errors; we can test the limits of accuracy of his absolute judgments. If the human observer is a reasonable kind of communication system, then when we increase the amount of input information the transmitted information will increase at first and will eventually level off at some asymptotic value. This asymptotic value we take to be the channel capacity of the observer: it represents the greatest amount of information that he can give us about the stimulus on the basis of an absolute judgment. The channel capacity is the upper limit on the extent to which the observer can match bis responses to the stimuli we give him. Now just a brief word about the bit
208
83 a n d we can begin t o look a t some data. One b i t of i n f o r m a t i o n is the a m o u n t of i n f o r m a t i o n that we need to make a decision between t w o equally l i k e l y a l ternatives. I f we m u s t decide whether a m a n is less than six feet tall or more t h a n six feet t a l l a n d i f w e k n o w t h a t t h e chances are 50-50, then we need one b i t of i n f o r m a t i o n . N o t i c e t h a t this u n i t of information does n o t refer i n a n y w a y to the u n i t of length t h a t we use—feet, inches, centimeters, etc. However y o u measure the man's height, we still need j u s t one b i t of i n f o r m a t i o n . T w o b i t s o f i n f o r m a t i o n enable us to decide among four equally l i k e l y alternatives. Three bits of information enable us to decide among eight equally Likely alternatives. Four b i t s of inform a t i o n decide among 16 alternatives, five among 3 2 , a n d so o n . T h a t is to say, i f there are 32 equally l i k e l y alternatives, we must make five successive b i n a r y decisions, w o r t h one b i t each, before we k n o w w h i c h alternative is correct. So the general rule is simple: every t i m e the number of alternatives is increased b y a factor of two, one b i t of i n f o r m a t i o n i s added. There are t w o ways we m i g h t i n crease the amount of i n p u t i n f o r m a t i o n . W e could increase the rate a t w h i c h we give i n f o r m a t i o n t o the observer, so that the amount of i n f o r m a t i o n per u n i t time w o u l d increase. Or we could ignore the timp variable completely a n d increase the a m o u n t o i i n p u t information b y increasing t h e number of alternative s t i m u l i . I n the absolute j u d g m e n t experiment we are interested i n the second alternative. W e give the observer as m u c h time as he wants to make his response; we s i m p l y increase the number of alternative s t i m u l i among w h i c h he must discriminate and look t o see where confusions begin to occur. Confusions w i l l appear near the point that we are calling his "channel c a p a c i t y . "
ABSOLUTE JUDGMENTS o r UNIDIMENSIONAL STIMULI N o w l e t us consider w h a t happens when we make absolute judgments of tones. Pollack ( 1 7 ) asked listeners to i d e n t i f y tones b y assigning numerals to t h e m . T h e tones were different w i t h respect t o frequency, a n d covered the range f r o m 100 to 8000 cps i n equal logarithmic steps. A tone was sounded a n d the listener responded b y giving a numeral. A f t e r the listener h a d made his response he was told the correct identification o f the tone. W h e n o n l y t w o o r three tones were used tbe listeners never confused them. W i t h four different tones confusions were q u i t e rare, b u t w i t h five or more tones confusions were frequent. W i t h fourteen different tones the listeners made m a n y mistakes. These data are plotted i n F i g . 1. Along the b o t t o m is the amount of i n p u t information, i n b i t s per stimulus. As the number of alternative tones was increased f r o m 2 t o 14, the input information increased from 1 t o 3.8 bits. O n the ordinate is plotted the amount of
INPUT
INFORMATION
FIG. 1. Data from Pollack (17, 18) on the amount of information that is transmitted by listeners who make absolute judgments of auditory pitch. As the amount of input information isincreased by increasing from 2 to 14 the number of different pitches to be judged, the amount of transmitted information
209
approaches as its upper limit a channel capacity of about 2.5 bits per judgment.
84 transmitted information. T h e amount of transmitted information behaves i n much the way we would expect a communication channel to behave; the transm i t t e d information increases linearly u p to about 2 bits and then bends off t o ward an asymptote at about 2.5 bits. T h i s value, 2.5 bits, therefore, is what we are calling the channel capacity o f the listener for absolute judgments o f pitch. So n o w we have the number 2.5 bits. W h a t does i t mean? First, note that 2.5 bits corresponds t o about six equally likely alternatives. T h e result means that we cannot pick more than six different pitches that the listener w i l l never confuse. O r , stated s l i g h t l y differently, no matter how many alternative tones we ask h i m t o judge, the best we can expect h i m t o do is t o assign them t o about six different classes w i t h out error. Or, again, i f we know that there were .V alternative stimuli, then his judgment enables us to narrow down the particular stimulus t o one o u t of If/6,
INPUT INFORMATION
Frc. I . Data from Garner (7) on the channel capacity for absolute judgments of auditory loudness.
N e x t y o u can ask bow reproducible this result is. Does i t depend o n the spacing o f the tones o r the various conditions o f judgment? Pollack varied these conditions i n a number of ways. The range o f frequencies can be changed b y a factor of about 20 w i t h o u t changing the amount o f information transm i t t e d more than a small percentage. Different groupings of the pitches decreased the transmission, b u t the loss was small. F o r example, i f y o u c a n M o s t people are surprised that the discriminate five high-pitched tones i n number is as small as six. O f course, one series and five low-pitched tones i n there is evidence that a musically so- another series, i t is reasonable t o exphisticated person w i t h absolute p i t c h pect that y o u could combine a l l ten into can identify accurately a n y one o f 50 a single series a n d still t e l l them a l l or 60 different pitches. Fortunately, I apart w i t h o u t error. W h e n y o u t r y i t , do n o t have time t o discuss these re- however, i t does n o t w o r k . T h e chanmarkable exceptions. I say i t is for- nel capacity for p i t c h seems t o be about tunate because I do n o t know how t o six and t h a t is the best y o u can do. explain their superior performance. So W h i l e we are o n tones, l e t us look I shall stick t o the more pedestrian fact next a t Garner's ( 7 ) w o r k on loudness. that most of us can identify about one Garner's data for loudness are sumout of only five or six pitches before we marized i n F i g . 2. Garner went to some begin t o get confused. trouble t o get the best possible spacing I t is interesting t o consider t h a t psy- of his tones over the intensity range chologists have been using seven-point f r o m 15 t o 110 d b . H e used 4 , 5, 6, 7, rating scales for a long time, o n the 10, and 20 different stimulus intensities. intuitive basis that t r y i n g t o rate i n t o T h e results shown i n F i g . 2 take i n t o finer categories does not really add much account the differences among subjects t o the usefulness of the ratings. Pol- and the sequential influence o f the i m lack's results indicate that, at least for mediately preceding judgment. Again pitches, this intuition is f a i r l y sound. we find that there seems t o be a l i m i t
210
85 and Gamer (8) asked observers to interpolate visually between two scale markers. Their results are shown in Fig. 4 . They did tbe experiment in two ways. In one version they let the observer use any number between zero and 100 to describe the position, alTASTES though they presented stimuli at only JUDGMENTS Of SALINE CONCENTRATION 5, 10, 20, or 50 different positions. The results with this unlimited response 1 2 5 4 technique are shown by the filled circles MPUT INFORMATION on the graph. In the other version the FIG. 3. Data from Beebe-Center, Rogers, observers were limited in their reand O C o n n r i l ( I ) on the channel capacity for sponses to reporting just those stimuabsolute judgments of saltiness. lus values that were possible. That is to say, in the second version the numThe channel capacity for absolute judg- ber of different responses that the obments of loudness is 2.3 bits, or about server could make was exactly the same five perfectly discriminable alternatives. as the number of different stimuli that Since these two studies were done in the experimenter might present. The different laboratories with slightly dif- results with this limited response techferent techniques and methods of analy- nique are shown by the open circles on sis, we are not in a good position to the graph. The two functions are so argue whether five loudnesses is signifi- similar that it seems fair to conclude cantly different from sue pitches. Prob- that the number of responses available ably the difference is in the right direc- to the observer had nothing to do with tion, and absolute judgments of pitch the channel capacity of 3.25 bits. are slightly more accurate than absolute The Hake-Garner experiment has been judgments of loudness. The important repeated by Coonan and Klemmer. Alpoint, however, is that the two answers though they have not yet published are of the same order of magnitude. their results, they have given me perThe experiment has also been done mission to say that they obtained chanfor taste intensities. In Fig. 3 are the nel capacities ranging from 3.2 bits for results obtained by Beebe-Center, Rogers, and O'Connell (1) for absolute / / 5 judgments of the concentration of salt . 3.25 solutions. The concentrations ranged — ^ • BITS from O J to 34.7 gm. NaCI pe I c c tap water in equal subjective steps. 0 2 u They used 3, 5, 9, and 17 different conIPO'NTS ON A LINE centrations. The channel capacity is o N,'N, I 1.9 bits, which is about four distinct • N, >O0 2 4 concentrations. Thus taste intensities a. i 2 i 4 a seem a little less distinctive than auditory stimuli, but again the order of INPUT INFORMATION magnitude is not far off. On the other hand, the channel caFIG. 4. Data from Hake and Gamer (S] on the pacity for judgments of visual position channel capacity for absolute judgments of the seems to be significantly Larger, Ha' position of a pointer in a linear interval. r
1
0
0
1
/
211
.
86
very short exposures of the pointer position to 3.9 bits for longer exposures. These values are slightly higher than Hake and Garner's, so we must conclude that there are between 10 and 15 distinct positions along a linear interval. This is the largest channel capacity that has been measured for any unidimensional variable. At the present time these four experiments on absolute judgments of simple, unidimensional stimuli are all that have appeared in the psychological journals. However, a great deal of work on other stimulus variables has not yet appeared in the journals. For example, Eriksen and Hake ( 6 ) have found that the channel capacity for judging the sizes of squares is 2.2 bits, or about five categories, under a wide range of experimental conditions. In a separate experiment Eriksen ( 5 ) found 2.8 bits for size, 3.1 bits for hue, and 2.3 bits for brightness. Geldard has measured the channel capacity for the skin by placing vibrators on the chest region. A good observer can identify about four intensities, about five durations, and about seven locations.
for the long exposure. Curvature was apparently harder to judge. When the length of the arc was constant, the result at the short exposure duration was 2.2 bits, but when the length of the chord was constant, the result was only 1.6 bits. This last value is the lowest that anyone has measured to date. I should add, however, that these values are apt to be slightly too low because the data from all subjects were pooled before the transmitted information was computed.
Now let us see where we are. First, the channel capacity does seem to be a valid notion for describing human observers. Second, the channel capacities measured for these unidimensional variables range from 1.6 bits for curvature to 3.9 bits for positions in an interval. Although there is no question that the differences among the variables are real and meaningful, the more impressive fact to me is their considerable similarity. If I take the best estimates I can get of the channel capacities for all the stimulus variables I have mentioned, the mean is 2.6 bits and the standard deviation is only 0.6 bit. In terms of One of the most active groups in this distinguishable alternatives, this mean area has been the Air Force Operational corresponds to about 6.5 categories, one Applications Laboratory. Pollack has standard deviation includes from 4 to been kind enough to furnish me with 10 categories, and the total range is the results of their measurements for from 3 to 15 categories. Considering several aspects of visual displays. They made measurements for area and for the wide variety of different variables the curvature, length, and direction of that have been studied, I find this to lines. In one set of experiments they be a remarkably narrow range. used a very short exposure of the stimuThere seems to be some limitation lus—y second—and then they re- built into us either by learning or by peated the measurements with a 5- the design of our nervous systems, a second exposure. For area they got limit that keeps our channel capacities 4v
2.6 bits with the short exposure and 2.7 bits with the long exposure. For the length of a line they got about 2.6 bits wilh the short exposure and about 3.0 bits with the long exposure. Direction, or angle of inclination, gave 2.8 bits for the short exposure and 3.3 bits
in this general range. On the basis of the present evidence it seems safe to say that we possess a finite and rather small capacity for making such unidimensional judgments and that this capacity does not vary a great deal from one simple sensory attribute to another.
212
87 ABSOLUTE JUDGMENTS or MULTIDIMENSIONAL STIMULI You may have, noticed that I have been careful to say that this magical number seven applies to one-dimensional judgments. Everyday experience teaches us that we can identify accurately any one of several hundred faces, any one of several thousand words, any one of several thousand objects, etc. The story certainly would not be complete if we stopped at this point. We must have some understanding of why the onedimensional variables we judge in the laboratory give results so far out of line with what we do constantly in our behavior outside the laboratory. A possible explanation lies in tbe number of independently variable attributes of the stimuli that are being judged. Objects, faces, words, and the like differ from one another in many ways, whereas the simple stimuli we have considered thus far differ from one another in only one respect. Fortunately, there are a few data on what happens when we make absolute judgments of stimuli that differ from one another in several ways. Let us look first at the results Klemmer and Frick (13) have reported for the absolute judgment of the position of a dot in a square. In Fig. 5 we see their re-
INPUT
REFORMATION
FIG. 5. Daca from Klemmer and Frick (13) on [he channel capacity for absolute judgments of the position of a dot in a square.
suits. Now the channel capacity seems to have increased to 4.6 bits, which means that people can identify accurately any one of 24 positions in the square. T h e postion of a dot in a square is clearly a two-dimensional proposition. Both its horizontal and its vertial position must be identified. Thus it seems natural to compare the 4.6-bit capacity for a square with the 3.25-bit capacity for the position of a point in an interval. The point in the square requires two judgments of the interval type. I f we have a capacity of 3.25 bits for estimating intervals and we do this twice, we should get 6.5 bits as our capacity for locating points in a square. Adding the second independent dimension gives us an increase from 3.25 to 4.6, but it falls short of the perfect addition that would give 6.5 bits. Another example is provided by BeebeCenter, Rogers, and O'Connell. When they asked people to identify both the saltiness and the sweetness of solutions containing various concentrations of salt and sucrose, they found that the channel capacity was 2.3 bits. S i n c e the capacity for salt alone was 1.9, we might expect about 3.8 bits if the two aspects of the compound stimuli were judged independently. A s with spatial locations, the second dimension adds a little to the capacity but not as much as it conceivably might. A third example is provided by Pollack (18), who asked listeners to judge both the loudness and the pitch of pure tones. Since pitch gives 2.5 bits and loudness gives 2.3 bits, we might hope to get as much as 4.8 bits for pitch and loudness together. Pollack obtained 3.1 bits, which again indicates that the second dimension augments the channel capacity but not so much as it might. A fourth example can be drawn from the work of Halsey and Chapanis (9) on confusions among colors of equal
213
88 luminance. Although they did not analyze their results in informational terms, they estimate that there are about 11 to IS identifiable colors, or, in our terms, about 3.6 bits. Since these colors varied in both bue and saturation, it is probably correct to regard this as a twodimensional judgment. I f we compare this with Eriksen's 3.1 bits for hue (which is a questionable comparison to draw), we again have something less than perfect addition when a second dimension is added. I t is still a long way, however, from these two-dimensional examples to the multidimensional stimuli provided by faces, words, etc. T o fill this gap we have only one experiment, an auditory study done by Pollack and Ficks (19). They managed to get six different acoustic variables that they could change: frequency, intensity, rate of interruption, on-time fraction, total duration, and spatial location. E a c h one of these six variables could assume any one of five different values, so altogether there were 5', or 15,625 different tones that they could present. The listeners made a separate rating for each one of these six dimensions. Under these conditions the transmitted information was 7.2 bits, which corresponds to about 150 different categories that could be absolutely identified without error. Now we are beginning to get up into the range that ordinary experience would lead us to expect. Suppose that we plot these data, fragmentary as they are, and make a guess about how the channel capacity charges with tbe dimensionality of the stimuli. The result is given in F i g . 6. I n a moment of considerable daring I sketched the dotted line to indicate roughly the trend that the data seemed to be taking. Clearly, the addition of independently variable attributes to the stimulus increases the channel capacity, but at a
a
HUMStH OF W i a B L t
iSPCCTS
F i c . 6. Tbe general form of the relation between channel capacity and the number of independently variable attributes of the stimuli.
decreasing rate. I t is interesting to note that the channel capacity is increased even when the several variables are not independent. Eriksen (5) reports that, -when size, brightness, and hue all vary together in perfect correlation, the transmitted information is 4.1 bits as compared with an average of about 2.7 bits when these attributes are varied one at a time. B y confounding three attributes, Eriksen increased the dimensionality of the input without increasing the amount of input information; the result was an increase in channel capacity of about the amount that the dotted function in F i g . 6 would lead us to expect. T h e point seems to be that, as we add more variables to the display, we increase the total capacity, but we decrease the accuracy for any particular variable. I n other words, we can make relatively crude judgments of several things simultaneously. We might argue that in the course of evolution those organisms were most successful that were responsive to the widest range of stimulus energies in their environment. I n order to survive in a constantly fluctuating world, it was better to have a little information about a lot of things than to have a lot of information about a small segment of the
214
39 environment. I f a compromise was necessary, the one we seem to have made is dearly the more adaptive. Pollack and Ficks's results are very strongly suggestive of an argument that linguists and phoneticians have been making for some time ( 1 1 ) . According to the linguistic analysis of the sounds of human speech, there are about eight or ten dimensions—the linguists call them distinctive features—that distinguish one phoneme from another. These distinctive features are usually binary, or at most ternary, in nature. F o r example, a binary distinction is made between vowels and consonants, a binary decision is made between oral and nasal consonants, a ternary decision is made among front, middle, and back phonemes, etc. T h i s approach gives us quite a different picture of speech perception than we might otherwise obtain from our studies of the speech spectrum and of the ear's ability to discriminate relative differences among pure tones. I am personally much interested in this new approach ( 1 5 ) , and I regret that there is not time to discuss it here. I t was probably with this linguistic theory in mind that Pollack and Ficks conducted a test on a set of tonal stimuli that varied in eight dimensions, but required only a binary decision on each dimension. With these tones they measured the transmitted information at 6.9 bits, or about 120 recognizable kinds of sounds. I t is an intriguing question, as yet unexplored, whether one can go on adding dimensions indefinitely in this way. I n human speech there is clearly a limit to the number of dimensions that we use. I n this instance, however, it is not known whether the limit is imposed by the nature of the perceptual machinery that must recognize the sounds or by the nature of the speech machinery that must produce them. Somebody will have to do the experiment to
find out. There is a limit, however, at about eight or nine distinctive features in every language that has been studied, and so when we talk we must resort to still another trick for increasing our channel capacity. Language uses sequences of phonemes, so we make several judgments successively when we listen to words and sentences. T h a t is to say, we use both simultaneous and successive discriminations in order to expand the rather rigid limits imposed b y the inaccuracy of our absolute judgments of simple magnitudes. These multidimensional judgments are strongly reminiscent of the abstraction experiment of Kiilpe ( 1 4 ) . As you may remember, Kiilpe showed that observers report more accurately on an attribute for which they are set than on attributes for which they are not set. F o r example, Chapman ( 4 ) used three different attributes and compared the results obtained when the observers were instructed before Ihe tachistoscopic presentation with the results obtained when they were not told until after the presentation which one of the three attributes was to be reported. When tbe instruction was given in advance, tbe judgments were more accurate. When the instruction was given afterwards, the subjects presumably had to judge all three attributes in order to report on any one of them and the accuracy was correspondingly lower. T h i s is in complete accord with tbe results we have just been considering, where the accuracy of judgment on each attribute decreased as more dimensions were added. T h e point is probably obvious, but I shall make it anyhow, that the abstraction experiments did not demonstrate that people can judge only one attribute at a time. They merely showed what seems quite reasonable, that people are less accurate i f they must judge more than one attribute simultaneously.
215
90
two dimensions of numerousness are area and density. When the subject can subitize, area and density may not be the significant variables, but when the subject must estimate perhaps they are significant. In any event, the comparison is not so simple as it might seem at first thought. This is one of the ways in which the magical number seven has persecuted me. Here we have two closely related kinds of experiments, both of which point to the significance of the number seven as a limit on our capacities. And yet when we examine the matter more closely, there seems to be a reasonable suspicion that it is nothing more than a coincidence.
SUB1TIZING
I cannot leave this general area without mentioning, however briefly, the experiments conducted at Mount Holyoke College on the discrimination of number (12). In experiments by Kaufman, Lord, Reese, and Volkmann random patterns of dots were flashed on a screen for 1/5 of a second. Anywhere from 1 to more than 200 dots could appear in the pattern. The subject's task was to report how many dots there were. The first point to note is that on patterns containing up to five or six dots the subjects simply did not make errors. The performance on these small numbers of dots was so different from the performance with more dots that it was given a special name. Below seven the subjects were said to subitize; above seven they were said to estimate. This is, as you will recognize, what we once optimistically called "the span of attention."
T H E SPAN OP IMMEDIATE MEMORY
T h i s discontinuity at seven is, of course, suggestive. Is this the same basic process that limits our unidimensional judgments to about seven categories? The generalization is tempting, but not sound in my opinion. The data on number estimates have not been analyzed in informational terms; but on the basis of the published data I would guess that the subjects transmitted something more than four bits of information about the number of dots. Using the same arguments as before, we would conclude that there are about 20 or 30 distinguishable categories of numerousness. This is considerably more information than we would expect to get from a unidimensional display. It is, as a matter of fact, very much like a two-dimensional display. Although the dimensionality of the random dot patterns is not entirely clear, these results are in the same range as Klemmer and F r i c k ' s for their two-dimensional display of dots in a square. Perhaps the
Let me summarize the situation in this way. There is a dear and definite limit to the accuracy with which we can identify absolutely the magnitude of a unidimensional stimulus variable. I would propose to call this limit the span oj absolute judgment, and I maintain that for unidimensional judgments this span is usually somewhere in the neighborhood of seven. We are not completely at the mercy of this limited span, however, because we have a variety of techniques for getting around it and increasing the accuracy of our judgments. The three most important of these devices are (a) to make relative rather than absolute judgments; or, if that is not possible, (4) to increase the number of dimensions along which the stimuli can differ; or (e) to arrange the task in such a way that we make a sequence of several absolute judgments in a row. The study of relative judgments is one of the oldest topics in experimental psychology, and I will not pause to review it now. The second device, increasing the dimensionality, we have just considered. It seems that by adding
216
01
more dimensions and requiring crude, binary, yes-no judgments on each attribute we can extend the span of absolute judgment from seven to at least ISO. Judging from our everyday behavior, the limit is probably in the thousands, if indeed there is a limit. I n my opinion, we cannot go on compounding dimensions indefinitely. I suspect that there is also a span oj perceptual dimensionality and that this span is somewhere in the neighborhood of ten, but I must add at once that there is no objective evidence to support this suspicion. T h i s is a question sadly needing experimental exploration. Concerning the third device, the use of successive judgments, I have quite a bit to say because this device introduces memory as the handmaiden of discrimination. And, since mnemonic processes are at least as complex as are perceptual processes, we can anticipate that their interactions will not be easily disentangled. Suppose that we start by simply extending slightly the experimental procedure that we have been using. Up to this point we have presented a single stimulus and asked the observer to name it immediately thereafter. We can extend this procedure by requiring the observer to withhold his response until we have given him several stimuli in succession. A t the end of the sequence of stimuli he then makes his response. We still have the same sort of input-output situation that is required for the measurement of transmitted information. B u t now we have passed from an experiment on absolute judgment to what is traditionally called an experiment on immediate memory. Before we look at any data on this topic I feel I must give you a word of warning to help you avoid some obvious associations that can be confusing. Everybody knows that there is n finite span of immediate memory and that for
a lot of different kinds of test materials this span is about seven items in length. I have just shown you that there is a span of absolute judgment that can distinguish about seven categories and that there is a span of attention that will encompass about six objects at a glance. What is more natural than to think that all three of these spans are different aspects of a single underlying process? And that is a fundamental mistake, as I shall be at some pains to demonstrate. T h i s mistake is one of the malicious persecutions that the magical number seven has subjected me to. M y mistake went something like this. W e have seer, that the invariant feature in the span of absolute judgment is the amount of information that the observer can transmit. There is a real operational similarity between the absolute judgment experiment and the immediate memory experiment. I f immediate memory is like absolute judgment, then it should follow that the invariant feature in the span of immediate memory is also the amount of information that an observer can retain. I f the amount of information in the span of immediate memory is a constant, then the span should be short when the individual items contain a lot of information and the span should be long when the items contain little information. F o r example, decimal digits are worth 3.3 bits apiece. We can recall about seven of them, for a total of 23 bits of information. Isolated English words are worth about 10 bits apiece. I f the total amount of information is to remain constant at 23 bits, then we should be able to remember only two or three words chosen at random. In this way I generated a theory about how the span of immediate memory should vary as a function of the amount of information per item in the test materials. The measurements of memory span in the literature are suggestive on this
217
92 SO
question, but not definitive. And so it was necessary to do tbe experiment to see. Hayes (10) tried it out with five different kinds of test materials: binarydigits, decimal digits, letters of the alphabet, letters plus decimal digits, and with 1,000 monosyllabic words. T h e lists were read aloud at the rate of one item per second and the subjects had as much time as they needed to give their responses. A procedure described by Woodworth (20) was used to score the responses. The results are shown by the filled circles in F i g . 7. Here the dotted line indicates what the span should have been if the amount of information in the span were constant. T h e solid curves represent the data. Hayes repeated the experiment using test vocabularies of different sizes but all containing only English monosyllables (open circles in Fig. 7 ) . This more homogeneous test material did not change the picture significantly. With binary items the span is about nine and, although it drops to about five with monosyllabic English words, the difference is far less than the hypothesis of constant information would require. I r L l " ( « » BCItS
COKS'SNT NO Of ITCMS-
LtTTERS
I
50 20
t I
•OOO
I
( CONSTANT
tnroftMATiati i
\
i % \
10 10
3
*
5
IZ
INFORM An ON PER ITEM IN BITS
Fin. 7. Data from Hayes (10) on Ihe span of immediate memory plotted as a function of the amount of information per item i n the test malerals.
•
SITS
F I G . 8. Data from Pollack ( 1 6 ) on amount of into m a t ion retained alter presentation plotted as a function of amount of information per item in the materials.
50 *0
Z
INFORMATION PER ITEM IN
the cne the test
There is nothing wrong with Hayes's experiment, because Pollack (16) repeated it much more elaborately and got essentially the same result. Pollack took pains to measure the amount of information transmitted and did not rely on the traditional procedure for scoring the responses. His results are plotted in F i g . 8. Here it is clear that the amount of information transmitted is not a constant, but increases almost linearly as the amount of information per item in the input is increased. And so the outcome is perfectly clear. I n spite of the coincidence that the magical number seven appears in both places, the span of absolute judgment and the span of immediate memory are quite different kinds of limitations that are imposed on our ability to process information. Absolute judgment is limited by the amount of information. I m mediate memory is limited by the number of items. I n order to capture this distinction in somewhat picturesque terms, I have fallen into tbe custom of distinguishing between bits of information and chunks of information. T h e n I can say that the number of bits of information is constant for absolute judgment and the number of chunks of informa-
218
93 tion is constant for immediate memory. T h e span of immediate memory seems to be almost independent of the number of bits per chunk, at least over the range that has been examined to date. T h e contrast of the terms bit and chunk also serves to highlight the fact that we are not very definite about what constitutes a chunk of information. F o r example, the memory span of five words that Hayes obtained when each word was drawn at random from a set of 1000 English monosyllables might just as appropriately have been called a memory span of 15 phonemes, since each word had about three phonemes in it. Intuitively, it is clear that the subjects were recalling five words, not 15 phonemes, but the logical distinction is not immediately apparent. We are dealing here with a process of organizing or grouping the input into familiar units or chunks, and a great deal of learning has gone into the formation of these familiar units. RZCODESG
I n order to speak more precisely, therefore, we must recognize the importance of grouping or organizing the input sequence into units or chunks. Since the memory span is a fixed number of chunks, we can increase the number of bits of information that it contains simply by building larger and larger chunks, each chunk containing more information than before. A man just beginning to learn radiotelegraphic code hears each dit and dah as a separate chunk. Soon he is able to organize these sounds into letters and then he can deal with the letters as chunks. T h e n the letters organize themselves as words, which are still larger chunks, and he begins to hear whole phrases. I do not mean that each step is a discrete process, or that plateaus must appear in his learning curve, for surely the levels of organization are
219
achieved at different rates and overlap each other during the learning process. I am simply pointing to the obvious fact that the dits and dahs are organized by learning into patterns and that as these larger chunks emerge the amount of message that the operator can remember increases correspondingly. I n the terms I am proposing to use, the operator learns to increase the bits per chunk. I n the jargon of communication theory, this process would be called recoding. T h e input is given in a code that contains many chunks with few bits per chunk. T h e operator recodes the input into another code that contains fewer chunks with more bits per chunk. There are many ways to do this recoding, but probably the simplest is to group the input events, apply a new name to the group, and then remember the new name rather than the original input events. Since I am convinced that this process is a very general and important one for psychology, I want to tell you about a demonstration experiment that should make perfectly explicit what I am talking about. T h i s experiment was conducted by Sidney Smith and was reported by him before the Eastern Psychological Association in 1954. Begin with the observed fact that people can repeat back eight decimal digits, but only nine binary digits. Since there is a large discrepancy in the amount oi information recalled in these two cases, we suspect at once that a recoding procedure could be used to increase the span of immediate memory for binary digits. I n Table 1 a method for grouping and renaming is illustrated. Along the top is a sequence of I S binary dibits, far more than any subject was able to recall after a single presentation. In the next line these same binary digits are grouped by pairs. Four possible pairs can occur: 00 is renamed 0, 01 is renamed 1, 10 is renamed 2, and 11 is
94 TABLE 1 WAYS
or
Binary D i o n (Bits)
RECODING SEQUENCES
1 0 1 0 0 0
2:1
Chunks Recoding
10 2
3:1
Chunks Recoding
101 5
4:1
Chunks Recoding
1010 10
Chunks Recoding
10100 20
5:1
10 2
00 0
or B t N u y D i o n s I
0 0
10 2
000 0
100 4 0010 2
01001 9
renamed 3. T h a t is to say, we recode from a base-two arithmetic to a basefour arithmetic. I n the recoded sequence there are now just nine digits to remember, and this is almost w i t h i n the span of immediate memory. I n the next line the same sequence of binary digits is regrouped into chunks of three. There are eight possible sequences of three, so we give each sequence a new name between 0 and 7. N o w we have recoded from a sequence of 18 b i n a r y digits into a sequence of 6 octal digits, and this is well w i t h i n the span of immediate memory. I n the last t w o lines the binary digits are grouped b y fours and b y fives and are given decimal-digit names f r o m 0 to IS and from 0 to 3 1 . I t is reasonably obvious t h a t this k i n d of recoding increases the bits per chunk, and packages the binary sequence i n t o a f o r m that can be retained w i t h i n the span of immediate memory. So Smith assembled 20 subjects and measured their spans for binary and octal digits. The spans were 9 for binaries and 7 for octals. T h e n he gave each recoding scheme t o five o i the subjects. T h e y studied the recoding u n t i l they said they understood i t — f o r about 5 or 10 minutes. T h e n he tested their span for binary digits again while they tried to use the recoding schemes they bad studied.
I
I
1 0
01 11 1 3 111 7 0111 7
0
00 0 001 1 0011 3
11001 25
1 1 I
11 3
0
10 2 no 6 10 no
T h e recoding schemes increased their span for binary digits i n every case. B u t the increase was not as large as we had expected on the basis of their span for octal digits. Since the discrepancy increased as the recoding ratio increased, we reasoned t h a t the few minutes the subjects h a d spent learning the recoding schemes h a d n o t been sufficient. A p p a r e n t l y t h e translation f r o m one code to the other must be almost automatic or the subject w i l l lose part of the next group while he is t r y i n g to remember t h e translation of the last group. Since the 4 : 1 a n d S: 1 ratios require considerable study, Smith decided t o imitate Ebbinghaus and do the experiment on himself. W i t h Germanic patience he drilled himself on each recoding successively, and obtained the results shown i n F i g . 9 . Here the data follow along rather nicely w i t h t h e results y o u w o u l d predict on tbe basis o f his span for octal digits. H e could remember 12 octal digits. W i t h t h e 2 : 1 recoding, these 12 chunks were w o r t h 24 b i n a r y digits. W i t h the 3 : 1 recoding they were w o r t h 36 b i n a r y digits. W i t h the 4 : 1 a n d 5 : 1 recodings, they were w o r t h about 4 0 binary digits. I t is a l i t t l e dramatic to watch a person get 4 0 binary digits i n a r o w a n d then repeat them back w i t h o u t error. However, i f y o u t h i n k of this merely as
220
95 so g s>
40 a
Jo
I
"
in I
to
PREDICTED
mat w > n* OCTH. OWITI
ONE HIGHLY
2:i
5:i
RECODING
PRACTICED SUBJECT
mi
ti
RATIO
Flo. 9. T h e ipan of immediatt memory for binary digits is plotted i s a function of tbe recoding procedure used. Tbe predicted function is obtained by multiplying tie span for oculs by 2, 3 and J J for recoding into base 4, base 8, and base 10, respectively.
a mnemonic trick for extending the memory span, you will miss the more important point that is implicit in nearly all such mnemonic devices. T h e point is that recoding is an extremely powerful weapon for increasing the amount of information that we can deal with. I n one form or another we use recoding constantly in our daily behavior. I n my opinion the most customary kind of recoding that we do all the time is to translate into a verbal code. When there is a story or an argument or an idea that we want to remember, we usually try to rephrase it "in our own words." When we witness some event we want to remember, we make a verbal description of the event and then remember our verbalization. Upon recall we recreate by secondary elaboration the details that seem consistent with the particular verbal recoding we happen to have made. T h e well-known experiment by Carmichael, Hogan, and Walter ( 3 ) on the influence that names have on the recall of visual figures is one demonstration of the process. The
inaccuracy of the testimony of
221
eyewitnesses is well known in legal psychology, but the distortions of testimony are not random—they follow naturally from the particular recoding that the witness used, and the particular recoding he used depends upon bis whole life history. Our language is tremendously useful for repackaging material into a few chunks rich in information. I suspect that imagery is a form of recoding, too, but images seem much harder to get at operationally and to study experimentally than the more symbolic kinds of recoding. I t seems probable that even memorization can be studied in these terms. T h e process of memorizing may be simply the formation of chunks, or groups of items that go together, until there are few enough chunks so that we can recall all the items. T h e work by Bousfield and Cohen (2) on the occurrence of clustering in the recall of words is especially interesting in this respect. SUMMARY
I have come to the end of the data that I wanted to present, so I would like now to make some summarizing remarks. First, the span of absolute judgment and the span of immediate memory impose severe limitations on the amount of information that we are able to receive, process, and remember. B y organizing the stimulus input simultaneously into several dimensions and successively into a sequence of chunks, we manage to break (or at least stretch) this informational bottleneck. Second, the process of recording is a very important one in human psychology and deserves m u c h more e x p l i c i t attention than it has received. In particular, the kind of linguistic recording that people do seems to me to be the very lifeblood of the thought processes. Recording procedures are a constant concern to clinicians, social psycholo-
96
gists, linguists, and anthropologists and yet, probably because recoding is less accessible to experimental manipulation than nonsense syllables or T mazes, the traditional experimental psychologist has contributed little or nothing to their analysis. Nevertheless, experimental techniques can be used, methods of receding can be specified, behavioral indicants can be found. And I anticipate that we will find a very orderly set of relations describing what now seems an uncharted wilderness of individual differences.
[or absolute judgment, the seven objects in the span of attention, and the seven digits in the span of immediate memory? For the present I propose to withhold judgment. Perhaps there is something deep and profound behind all these sevens, something just calling out for us to discover it. But I suspect that it is only a pernicious, Pythagorean coincidence.
Third, the concepts and measures provided by the theory of information provide a quantitative way of getting at some of these questions. The theory provides us with a yardstick for calibrating our stimulus materials and for measuring the performance of our subjects. In the interests of communication I have suppressed the technical details of information measurement and have tried to express the ideas in more familiar terms; I hope this paraphrase will not lead you to think they are not useful in research. Informational concepts have already proved valuable in the study of discrimination and of language; they promise a great deal in the study of learning and memory; and it has even been proposed that they can be useful in the study of concept formation. A lot of questions that seemed fruitless twenty or thirty years ago may now be worth another look. In fact, I feel that my story here must stop just as it begins to get really interesting. And finally, what about the magical number seven? What about the seven wonders of the world, the seven seas, the seven deadly sins, tbe seven daughters of Atlas in the Pleiades, the seven ages of man, the seven levels of hell, the seven primary colors, the seven notes of the musical scale, and the seven days of the week? What about the sevenpoint rating scale, the seven categories
222
REFERENCES 1. B E E B E - C E N T E K , J .
G.,
Roc ESS,
M.
S,
*
D. N. Transmission of information about sucrose and saline solutions through the sense of taste. J. Psychol., 1955, 39, 157-160. O'COXKEZL,
2. BOUSTTEXD, W . A.,
a
COBEK, B . H .
Tbe
occurrence of clustering in the recall of randomly arranged words of different frequencies-of-usage. / . gen. Psychol, 1955, 52, 83-95. 3. C A S M J C B A E L , L , H C C A S , H . P . , S W A L T E R ,
A. A. An experimental study of the effect of language on the reproduction of visually perceived form. J txp. Psychol., 1932, 15, 73-86. 4. C H A F S U N , D. W . Relative effects of determinate and indeterminate Aufgaben. Amer. J. Psychol.,
5.
1932 , 44, 163-174.
Multidimensional stimulus differences and accuracy of discrimi-
EFIESEK, C . W .
nation.
USAF,
WADC
Tech.
Rep.,
19S4, No. $4-165. 6. E W X S E N , C. W , k H A E E , H . W . Absolute judgments as a function of the stimulus range and the number of stimulus and response categories. J. txp. Psychol, 1955, 49, 323-332. 7. GAfLSEft, W . R. An informational analysis of absolute judgments of loudness. /. tip. Psychol., 1953, 46, 373-3SO. 8. H A K E , H . W . , a G A H C E K , W . R.
The
ef-
fect of presenting various numbers of discrete steps on scale reading accuracy. /. txp. Psychol^ 1951, 42, 358-366. 9. H A L S E Y , R. M., k C H A P A N I S , A. Chroma licit y-confusion contours In a complex viewing situation. J. Opt. Soc. Amer., 1954, 44, 442-454. 10. H A Y E S , J . R. M. Memory span for several vocabularies as a function of vocabulary siie. In Quarterly Progress Report, Cambridge, Mass.: Acoustics Laboratory, Massachusetts Institute of Technology, Jan.-June, 1952.
97 11. J A X O B S O N , R . , FAJTT,
C. C. M
n
*
HALLE,
M. Frelimmarit ] to speech analysis. Cambridge, Mass.: Acoustics Laboratory, Massachusetts Institute of Technology, 1952. (Tech. Rep. No. 13.) 12. E A T J F U W , T.
E . L „ LORD,
M.
W , a VOLXKANH, J .
W.,
REESE,
The discrimi-
nation of visual number. Amer. J. Psychol, 1949, 62, 498-S2S. 15. K U M I P J , E . T , a F a i c x , F . C . Assimilation of information from dot and matrix patterns. J. t*p. Psychol, 1 9 5 3 , 45,
15-19.
14. S U L P E , O . Versuche uber Abstraction. Ber. Q. d I Kor.gr. /. tzper. Psychol, 1904, 15.
56-63.
M n . i . E B , G . A-, a N I C E L Y , P . E . A n analy-
sis of perceptual confusions among some
English consonants. J. Acouit. Soc. Amer., 195S, 27, 33S-352. 16. P O L L A C E , I . The assimilation oi sequin* tially encoded information. Amtr. J. Psychol., 1953, 66, 421-435. 17. P O L L A C K , I . Tbe information of elementary auditory displays. / . Acoust. Soc. Amer, 1932. 2 4 , 745-749. 13. P O L L A C K , I . The information of elementary auditory displays. I I . J. Acoust. Soc. Amer, 1953, 2 5 , 765-769. 19.
POLLACK,
I , a FICKS,
L.
Information of
elementary multi-dimensional auditory displays. / . Acoust. Soc. Amer, 1934, 2 6 , 155-158. 2 0 . W O O D W O K T H , R . S. Experimental psytkoloty. New York: Holt, 1933. (Received M a y 4 , 1955)
223
Reprinted from Perception, Vol. 1. pp. 371-394,1972 © 1972 PlONLtd.
Single units and sensation: A neuron doctrine for perceptual psychology? H B Barlow Department of Physiology-Anatomy. University ol California. Berkeley, California 94720 Received 6 December 1972
Abstract. The problem discussed is the relationship between Ihe firing of single neurons in sensory pathways and subjectively experienced sensations. The conclusions are formulated as the following five dogmas: I, To understand nervous funclion one needs to look at interactions at a cellular level, rather Lhan either a more macroscopic or micioscopic level, because behaviour depends upon the organized pattern of these intercellular interaclions. Z. The sensory system is organized lo achieve as complete a representation of the sensoiy stimulus as possible with the minimum number of active neurons. 3. Trigger features of sensoiy neurons are matched to redundant patterns of stimulation by experience as well as by developmental piocessei. 4. Perception corresponds lo the activity of a small selection from the very numerous high-level neurons, each of which corresponds to a pattern of externa] events of the order of complexity of the events symbolized by a word. 5. High impulse frequency in such neurons corresponds to high ceitainty that the trigger feature is present. The development of Ihe concepts leading up lo these speculative dogmas, their experimental basis, and some of their limitations are discussed. 1 Introduction In this article I shall discuss the difficult but challenging problem of the relation between our subjective perceptions and the activity of the nerve cells in our brains. Results obtained by recording from single neurons in sensory pathways have aroused a lot of interest and obviously tel! us something important aboul how we sense the world around us; but what exactly have we been told? In order to probe this question, ideas that fit current knowledge as well as passible must be formulated, and they must be staled clearly enough lo be tested to see if they are right or wrong; this is what I have tried to do. The central proposition is that our perceptions ate caused by the activity of a rather small number of neurons selected from a very large population of predominantly silent cells. The activity of each single cell is thus an important perceptual event and it is thought to be related quite simply to our subjective experience. The subtlety and sensitivity of perception results from the mechanisms determining when a single cell becomes active, rather than from complex combinatorial rules of usage of nerve cells. In order to avoid vagueness, 1 have formulated this notion in five definite propositions, or dogmas, and the reader who wishes to see the trend of this article can glance ahead (to page 380). Some of the dogmas will be readily accepted by most people who hope to find a scientific basis for human thought processes, but 1 felt they required statement and discussion in spite of their widespread tacit acceptance. Others are more original, will be challenged by many, and have the nature of extrapolations from the current trend of results rather than conclusions reasonably based upon them. Before these dogmas are stated the developments that have led to them will be briefly reviewed. The literature is extensive, and much of it will have been incorporated into the reader's common knowledge. My aim, therefore, is to pick out the conceptual turning points in order to show the direction we are headed.
224
372
After stating the dogmas, criticisms and alternatives will be discussed in an attempt both to justify them and to clarify them further. 2 Recording from single neurons
2.1 Peripheral nerves In the twenties and thirties methods were developed for amplifying and recording the weak transient electrical potentials associated with the activity of nerve fibres, and Adrian and his colleagues used these methods to record the all-or-none impulses of single nerve fibres connecting the sense organs to the brain (Adrian, 1926a, 1926b; Adrian and Zotterman, 1926a, 1926b; Adrian, L928}. They showed. Tor example, that each fibre coming from the skin responded to a particular type o f stimulus, such as pressure, temperature change, or damage, applied to a specific region or receptive field. The frequency o f the impulses depended upon the intensity of the stimulus, but it was clear that the character of the sensation (touch, heat, or pain) depended upon the fibre carrying the message, not the nature of the message, since this consisted of trains of similar impulses in all fibres. Nerves had long been recognized as the link between physical stimulus and sensation, so these results provided physiological flesh and blood to the skeleton that anatomical studies had revealed a long time earlier. Most of the results confirmed another ancient idea, namely Miiller's doctrine o f specific nerve energies: the specificity of different sensations stems from the responsiveness o f different nerve fibres to different types of stimulus. The chemical senses proved to be a little different (Pfaffman, 1941, 19SS; Ganchrow and Erickson, 1970), but i n spite o f the fact that they did not quite fall in line, the concept that resulted from two decades o f recording from peripheral fibres and following their connections i n the brain was o f a simple mapping from sense organs to sensorium, so that a copy o f physical events at the body surface was presented to the brain (Bard, 1938; Marshall el al., 1941; Adrian, 1941, 1947). Some modification was recognized to occur, for sensory nerves usually adapt to a constant stimulus, and therefore signal sudden changes o f stimulus energy better than sustained levels. Neighbouring receptive fields and modalities were also known to overlap, but when the activity of neurons at higher levels in sensory pathways was recorded it became obvious that something was happening more complex and significant than could be fitted into Ihe concept of simple mapping with overlap and adaptation,
2.2 Sensory neurons of Ihe retina Starting with Granit (Granit and Svaetichin, 1939; Granit, 1947) and Hartline (1938; 1940a, 1940b) in the retina, and Galambos and Davis (Galambos and Davis, 1943; Galambos, 1944; Galambos and Davis, 1948) at the periphery of the pathway for hearing, a generation of physiologists has studied sensory neutons in the central nervous system; all this obviously cannot be reviewed here, but we shall concentrate on the results that expanded the conceptual frame built on the earlier work. Previously it was passible for physiologists to be satisfied with describing how the sense organs and their nerves present a picture a( Ihe external world to the brain, and they were happy to leave it to the psychologists l o discuss what happened next; but these next things started to happen around the physiologist's micro-electiodes, and he has to Join the discussion. The realization that physiological experiments can answer questions o f psychological interest first dawned on me personally when 1 was working on the frog's retina. A vigorous discharge can be evoked from retinal ganglion cells by stimulating the appropriate region of the retina—the ganglion cell's 'receptive field' (Hartline, 1940a); but i f the surrounding region is simultaneously stimulated the response of the cell is diminished or completely abolished (Barlow, 1953). This phenomenon is
225
373
called lateral inhibition, or peripheral suppression, and such a physiological mechanism had already been postulated in order to account for simultaneous brightness and Mach bands (Mach, 1886; Fry, 1948). Thus the physiologicai experiment was really providing evidence in support of a psychological hypothesis. The invasion of psychological territory did not stop at this point. I f one explains the responsiveness of single ganglion cells in the frog's retina using hand-held targets, one finds that one particular type of ganglion cell is most effectively driven by something like a black disc subtending a degree or so moved rapidly to and fro within the unit's receptive field. This causes a vigorous discharge which can be maintained without much decrement as long as the movement is continued. Now, i f tl,c stimulus which is optimal for this class of cells is presented to intact frogs, the behavioural response is often dramatic: they turn towards the target and make repeated feeding responses consisting of a jump and snap. The selectivity o f the retinal neurons, and the frog's reaction when they are selectively stimulated, suggest that they are 'bug detectors' (Barlow, 1953) performing a primitive but vitally important form of recognition. This result makes one suddenly realize that a large part of the sensory machinery involved in a frog's feeding responses may actually reside in the retina rather than in mysterious 'centres' that would be too difficult to understand by physiological methods. The essential lock-like property resides in each member of a whole class of neurons, and allows the cell to discharge only to the appropriate key pattern of sensory stimulation. Lettvin et at. (1959) suggested that there were five different classes of cell in the frog, and Levick, Hill and I (Barlow et at., 1964) found an even larger number of categories in the rabbit. We called these key patterns 'trigger features', and Maturana et at. (1960) emphasized another important aspect of the behaviour of these ganglion cells: a cell continues to respond to the same trigger feature in spite of changes in light intensity over many decades. The properties of the retina are such that a ganglion cell can, figuratively speaking, reach out and determine that something specific is happening in front of the eye. Light is the agent by which it does this, but it is the detailed pattern o f the light that carries the information, and the overall level of illumination prevailing at the time is almost totally disregarded. It is true that Ingle (1968, 1971), Griisser and Griisser-Comehls (1968), and Ewert (1970) have shown that it is too simple to suppose that feeding automatically and inevitably follows the activation of a certain class of retinal ganglion cells by their trigger features; higher coordinating mechanisms are also involved. Just as light is only an intermediate agent allowing a retinal ganglion cell to detect its trigger feature, so these optic nerve impulses must doubtless be regarded as intermediate agents enabling the higher centres to perform their tasks. We shall proceed to discuss these problems, but we have gained two important concepts from the frog's retina: it transmits a map, not of the light intensities at each point of the image, but o f the trigger features in the world before the eye, and its main function is not to transduce different luminance levels into different impulse frequencies, but to continue responding invariantly to the same external patterns despite changes of average luminance.
2.3 Sensory neurons of the cerebral cortex The function of the visual area o f the mammalian cerebral cortex is obviously more relevant to the problem of our own subjective perceptions than is the frog's retina, and Hubel and Wiesel (1959) early discovered examples of selectivity for pattern in the responsiveness of cells in the visual cortex o f cats. They found that a light or dark line, or a dark-light border, was required to evoke a vigorous response even in the simplest first-order cells. Furthermore the stimulus had to be at a rather precise
226
374
orientation and position i n the visual field and in addition it usually had lo be moving, often in a specific direction. Hubel and Wiesel (1962) also made a distinction between these cells and other classes wilh more elaborate stimulus requirements, which they believed corresponded lo cells al Inter stages of information processing. They called these 'complex' and 'hypercomplcx' units, and showed that they had properties suggesting that the input to each was from the simpler category of cells. The fascination of this analysis depends to a large extent upon successfully following the way units become selective for more and more complex properties at each stage. Some doubts have been cast on their hierarchical scheme (Stone, 1972), but it certainly gave new insight into how higher levels of categorization are developed from lower levels. As well as the hierarchical concept, this work provided evidence for a new typu of invariance. In the cat, as in the frog, the retina is mainly responsible for ensuring that the message sent to the brain is not much perturbed by changes in ambient illumination. In the cortex Hubel and Wiesel (1962) found that some of the higher level neurons responded to the same trigger feature over a considerable range of positions. The modality specificity of peripheral neurons indicates how one can, for instance, detect warmth at any point on the body surface, and we now see that the organized pattern specificity of a set of cortical neurons can in the same way produce positional invariance for pattern perception. This was previously one of the great puzzles, and, although we certainly do not understand how recognition is invariant for position, size, and perspective transformations, at least a start has been made. Later experiments have shown that the primary neurons o f the visual cortex are more specific in one respect than Hubel and Wiesel originally thought. They showed that most neurons are fed by inputs from both eyes, and they emphasized that the dominance of ipsi- or contra-lateral eye varied from cell to cell. Now it can be shown that a binocular stimulus often has to be very precisely positioned in both eyes in order to evoke the most vigorous response (Barlow el a t , 1967; Pettlgrew el al., 1968), and a more important variable than dominance emerges from Ihe exact relative positions in the two eyes. Consider what must happen when the eyes are converged on some point in front of the cat and appropriate visual stimuli arc presented; il is easy to position this stimulus correctly for either eye by ilself, b u l , i f i l is to be correctly positioned for both, it will have to be at some specific distance from the cat. When the precise positioning for different units is studied, it is found that this specific distance for optimal response vanes in different units in Ihe sanif curlcx. and among units serving the same region of visual space. Conversely, the selection of units which are activated provides the cat with some informalion about the distances of the various stimulus objects. In uncovering this aspect of the pattern selectivity of sensory neurons we again get the sense that a central neuron is reaching out to discover something important about what is happening in the real objective world. One even wonders i f the line and edge detectors of Hubel and Wiesel may not luivc. as their main function, the linking together o f information about the same object in the two retinal images i n order to determine the object's most important c o o r d i n a t e its distance from the animal. At all events, as in the case of the frog's bug-detector, the importance of the information abstracted from the retinal images gives some insight into the purpose or direction of the physiological mechanisms. Something is known about these first steps of information processing in the visual cortex; what about the later stages? Results suggesting greater and greater specificity of response requirements have been obtained, and a nice example is the unit described by Gross el al. (1972) in the uifero-temporal cortex of macaques; this responded best to stimulation by a figure with many of the specific characteristics of a monkey's hand, and the requirements o f one such unit are well documented.
227
375
Work in this area is not easy to repeat, for one can readily see that it is largely a matter of chance to find a trigger feature of this order of complexity. Also, the possibility that cells may retain to adulthood the modifiable properties of Immature cells that will be described later makes the prospect of investigating the sensory association areas an intimidating one. Cortical neurons receive selective excitatory and inhibitory inputs from other neurons and thereby possess selective responsiveness for some characteristics and invariances for changes in other characteristics. This seems to have the potentiality of being a powerful information processing system. 3 Single units and psychophysics The neurophysiological discoveries outlined above of course made a deep impression on those investigating sensation psycho physically, but although there are many superficial points of contact it has not proved easy to link sensations securely to specific patterns of neurophysiological activity. The topics 1 have chosen are again ones which seem to have implications about how we conceptualize this neuropsychic relationship.
3.1 Lateral inhibition and simultaneous contrast The relation between lateral inhibition in the retina and simultaneous contrast has already been mentioned, but there is a large gap between the physiological level and the subjective effects shown in textbook illustrations, and it is too big to be bridged by a single simple statement. It is quite easy to show that frog and cat retinal ganglion cells demonstrate relevant effects, since their antagonistic surrounds (Barlow, 1953; Kuffler, 1953) make their responses depend upon contrast rather than absolute luminance. Hence on-centre cells respond to spots we would call white, off-centre cells to spots we would call black, even when the so-called black spot has a higher luminance than the white spot (Barlow et al., 1957). But subjective contrast effects also hold for conditions where one cannot make such easy comparisons, for instance at the centre of an area which is much too large to fill the centre of a retinal receptive field. Of course, one can postulate some Tilling in' process (Yarbus, 1965), but the necessity oT introducing ad hoc assumptions makes many explanations of subjective effects in terms of single units unconvincing. The concept that enables one to escape this difficulty is to concentrate on the informational flow rather than on the direct subjective-physiological comparison. Information discarded in a peripheral stage of processing cannot be accurately added back centrally, and in the present case it helps to talk about 'attenuating low spatial frequencies' instead or 'signalling spatial contrast'. To say that some of the lowfrequency attenuation of the whole visual system is performed by the opposed centre-surround organisation of the retinal ganglion cell (Campbell and Green, I96S; Enroth-Cugel) and Robson, 1966) is more accurate than to say that all simultaneous contrast effects originate there.
3.2 Colour in the field of colour vision De Valois has looked for relationships between various psychophysically measurable aspects of colour and the properties of single unit responses recorded at the level of the lateral geniculate nucleus. The main tesults provided a startling confirmation of Hering's long-standing hypothesis about the reciprocal organisation of colour systems (Svaetichin and MacNicholl, 1958; De Valois, I960; Hurvich and Jameson. 1960; Wagner et al., 1960), but the details are important. He has been able to establish neuro-psychic parallels using what may be called the 'lower envelope' or 'most sensitive neuron' principle. A monkey's ability to discriminate hue and saturation (De Valois et al., 1966, 1967) is very close
228
376
to whal one would expect if the monkey only pays attention to the most sensitive of the optic nerves conveying information about these qualities of the stimulus. Thus the psychophysical performance follows the lower envelope of the performance of individual fibres. It is particularly interesting to see that a continuous psychophysical function, hue discrimination as a function of wavelength, is served by a different type of neuron in different ranges; over the long wavelength range the red-green opponent system was much more sensitive to wavelength shift, whereas the blue-yellow system was more sensitive at short wavelengths. This result again fits in with the concept that neurophysiology and sensation are best linked by looking at the flow of information rather than simpler measures of neuronal activity. For instance it might be suggested that sensation follows the average neural activity, and it would be easy to justify this on the neurophysiological grounds that post-synaptic potentials are usually additive. However, this oversimple suggestion is proved false by the fact that psychophysical hue discrimination does not follow the average response of the red-green and blue-yellow systems, but instead follows the lower envelope. Now when two noisy channels are both conveying information about a signal, the channel with the highest signal/noise ratio dominates the situation; the low signal/noise ratio channel can be used to improve performance slightly, but it is a very small contribution except where its signal/noise ratio is nearly as high as that of the more sensitive channel. Thus the 'most-sensitive neuron' principle again fits the concept that, to link neurophysiological activity and sensation, one should look at the flow of information.
3.3 Touch Another example is given by the work of Mountcastle and his colleagues (Talbot et at., 1968), in which they studied the responses at a number of levels to vibratory stimuli applied to the glabrous skin of the hand. First they recorded from cutaneous afferents in the monkey, then the cortical responses in the same species, finally they made psychophysical measures of sensory responses in humans to the same stimuli, As with the work on colour, they established that the sensory response depends simply upon the category of nerve fibre with the lowest threshold. The fact that the subjective sensation in both the colour and touch systems seems to follow the lower envelope of the responses of the various types of sensory neuron may give an important clue lo the way in which these neurons represent. sensations. It is as If the screen on which sensations appear is completely blank until a sensory pathway is activated, but when this happens a point lights up and becomes instantly visible. This is nol what one would expect i f there was a lot of ongoing activity in all pathways, or if the magnitude of the signal was a linear function of intensity, nor is it what one would expect if sensation depended in a complex combinatorial way upon the activity of many units. Rather it suggest the concept that Ihe magnitude of the signal directly represents the signal/noise ratio, for then the insignificant signals will automatically be small, and the neuron firing most will automatically be (he most sensitive. This concept receives some support in the next section and is taken up in the fifth dogma and its discussion.
3.4 Adaptation after-effects The fact that one is almost unaware oT the constant pressure applied tu the skin by the chair one is sitting on presumably results, at least in part, from the rapid decline in frequency of the volley of sensory impulses initiated by contact (Adrian. 1928), Central neurons that respond to specific patterns of sensory input also give a decreased response when the pattern is sustained or repeatedly presented, though there have actually been surprisingly few investigations of this effect. These
229
377
adaptation, habituation, oi fatigue effects lead to plausible explanations for many well-known sensory illusions. For example the rate of discharge in the directionally selective neurons of the rabbit retina declines if a stimulus is continuously moved through (he receptive field in the preferred direction, and following cessation of movement the maintained discharge is found to be suppressed (Barlow and Hill, 1963). The resulting imbalance between neurons signalling opposite directions seems to provide a ready explanation of the apparent reversed movement of stationary objects following prolonged inspection of moving objects (the so-called 'waterfall effect'), and provides another example of an ancient psychophysical hypothesis (Wohlgemuth, 191 J) being confirmed neurophysiologically. One must bear in mind that these neural effects were described in the rabbit's retina, whereas in the human, as in the cat and monkey, neurons are probably not directionally selective until the level of the visual cortex (Barlow and Brindley, 1963), but the same type or explanation may well apply to neurons at this level. It has been suggested that one can make inverse inferences from the existence of an after-effect to the presence of neurons with particular selective responses. This is no place to argue whether the after-effects of adaptation to gratings imply a Fouriertype analysis (Blakemore and Campbell, 1969), or whether they can be satisfactorily accounted for by families of different-sized neurons with conventional Hubel-Wieseltype receptive fields, but there is certainly room for argument, and this makes selective adaptation a difficult tool to use to discover later stages of information processing. Instead, 1 think the importance of sensory adaptational effects, and of the corresponding neurophysiological phenomena, lies in the support both these phenomena lend to the concept put forward at the end of the last section. If sensory messages are to be given a prominence proportional to their informational value, mechanisms must exist for reducing the magnitude of representation of patterns which are constantly present, and this is presumably the underlying rationale for adaptive effects. 3.5 Noisiness or reliability of single units It used to be commonly held that nerve cells were unreliable elements, much perturbed by metabolic or other changes and perhaps also by random disturbances of more fundamental origin (McCulloch, 1959; Burns, 1968). The fairly high degree of reliability that the nervous system achieves as a whole was explained by the supposed redundancy of neural circuits and appropriate rules for averaging and combining them. Developments in the study of human vision at the absolute threshold and of the absolute sensitivity of retinal ganglion cells in the cat now indicate that nerve cells are not intrinsically unreliable and that noise often originates externally. Signal detection theory has familiarized psychologists with the problem of detecting signals in the presence of noise (Tanner and Swets, 1954; Green and Swets, 1966), and I think the assumed prevalence of internally generated noise was a major reason why this was thought to be an important new approach. But psychophysical studies have actually shown that the senses and the brain can operate with astonishing intrinsic reliability. Noise may always be present, but to an amazing extent it originates outside the nervous system. This was originally implied by the results of Hecht et al. (1942) on the absolute threshold of vision; they showed that about 100 quanta at the cornea, leading to 10 or less absorptions in the retina, were sufficient to give a sensation of light. But their most revolutionary finding was that the frequency-of-seeing curve, describing the breadth of the threshold zone, is mainly accounted for by quantum fluctuations, not internal sloppiness or random variations of the threshold criterion as had previously been thought. That is not to say that
230
378
Intrinsic retinal noise' or 'dark light' is non-existent or unimportant, for it is probably the main factor determining how many quanta are required Tor reliable detection (Barlow, 1956). It now appears probable that this originates in the photoreceptors and, in some subjects at least, is low enough to allow the conscious detection of the sensation caused by absorption of a single quantum (Sakitt, 1972); similar sensations occur in the absence or light stimuli, but at a lower frequency. In addition, the subjects can apparently discriminate between the sensory messages resulting from 2, 3, 4 , etc. quantal absorptions, each being detected progressively more clearly and reliably. This psychophysical work shows that the human brain, acting as a whole, can distinguish between the disturbances caused by small numbers of quantal absorptions. These must of course originate from single molecular events in single cells, but possibly the disturbance is thereafter diffused through many cells and abstracted in some way from a redundant neural representation. It therefore becomes very interesting to go into the neurophysiology and find how the absorption of a few quanta is signalled. A sensitive example of a retinal ganglion cell of the cat, with its associated bipolar cells, receptors, amacrine and horizontal cells, will give a readily detectable discharge of impulses to as few as 2 or 3 quanta of light absorbed in the retina (Barlow al., 1971). Such a stimulus will give rise to an average of 5 to 10 extra impulses. Thus a single quantal absorption causes as many as 3 extra impulses, two quanta cause about 6 impulses, and so on. The addition of 3 impulses to the maintained discharge is detectable on average, though, like the absorption of a single quantum in the human, it cannot be reliably detected on a single trial. There is of course some intrinsic noise, as shown by the maintained discharge, but its level is extraordinarily low when one considers that a single ganglion cell is connected to more than 100 rods containing a total of some l O molecules of rhodopsin, each poised ready to signal the absorption of a quantum. The important point is that quantitative knowledge of the noise level and reliability of single retinal ganglion cells enables one to see that the performance of the whole visual system can be attributed to a single cell: averaging is not necessary. 1 0
Individual nerve cells were formerly thought lo be unreliable, idiosyncratic, and incapable of performing complex tasks without acting in conceit and thus overcoming their individual errors. This was quite wrong, and we now realise their apparently erratic behaviour was caused by our ignorance, not the neuron's incompetence. Thus we gain support from this neuropsychical comparison for the concept of a neuron as a reliable element capable of performing a responsible role in our mental life, though we need not of course go to the other extreme and assume thai mental errors arc never caused by malfunctioning, ill-educated, or noisy neurons. 4 Mod inability of cortical neurons The most recent conceptual change about the neural basis of our sensations has arisen from a reinspection of the origin of the selective responsiveness of cortical neurons, 4.1
Evidence
for
modi/lability
Hubel and Wiesel (1963) at first thought they had shown thai the whole of [he elaborate organization responsible for the selectivity of neurons in the primary visual cortex was developed solely under genetic control. They reported that they found cortical neurons with normal adult-type specificity of responsiveness in young kittens which had not opened their eyes, or which had been deprived or visual experience by suture of their eyelids. In later investigations (Wiesel and Hubel. 1963, 1965; Hubel and Wiesel, 1965) they found that abnormal visual experience, such as unilateral
231
379
eye-suture, or prevention of simultaneous usage of the eyes by alternating occlusion or surgically induced strabismus, caused the development of an abnormal population of cortical cells. In accordance with their earlier findings they attributed this to a disruption of the preformed organization, and they discovered the very important fact that abnormal experience only modifies the cortex if it occurs during a particular •sensitive' period—about 3 to 12 weeks in cats (Hubel and Wiesel, 1970). Recent developments have extended these seminal findings, but they lead to somewhat different conclusions about the relative importance of experience and genetic factors in determining the selectivity of cortical neurons. First it was shown that kittens brought up with the two eyes exposed to different stimuli, one to vertical stripes, the other to horizontal, had a corresponding orientation selectivity of the receptive fields connected to each eye (Hirsch and Spinelli, 1970, 1971). This was confirmed in kittens exposed only to vertically or horizontally striped environments; these had no neurons sensitive to horizontally or vertically oriented stimuli respectively (Blakemore and Cooper, 1970). Evidence has been obtained that cats raised with a vertical displacement of the images in one eye induced by prisms also have abnormal vertical disparities of the pairs of receptive fields of cortical neurons connected to both eyes (Shlaer, 1971). Again, the cortex of a kitten exposed only to bright dots, with no contours or edges, contained units of an abnormal type responding well to small spots of light and showing little of the customary preference for lines (Pettigrew and Freeman, forthcoming). Furthermore it appears that a very brief period of exposure, as little as an hour, can have very pronounced effects on the subsequent selectivity of neurons in the visual cortex (Blakemore and Mitchell, 1973). Such results could still possibly have been explained by disruption of the innately-determined highly-specific connections that were originally thought to underlie response specificity, but a reexamination of the properties of cortical neurons of kittens with no visual experience shows that they do not actually have fully-developed adult-type specificity (Barlow and Pettigrew, 1971). This is certainly Ihe case with regard to disparity selectivity and, although there is directional preference and may be some weak orientation selectivity, they are not as narrowly selective as adult cells (Pettigrew, forthcoming). The anatomy or the developing cortex shows that only a small Traction of the normal complement or synapses is present before the critical period, and it is hard to believe that the cells could have adult properties (Cragg, 1972). It will take more work to determine the limits within which the pattern selectivity of cortical neurons can be modified, but the results already make it impossible to believe Hubel and Wiesel's original claim that many cells of the visually inexperienced kitten have the full adult-type selectivity, 4.2 Type of modification caused It is instructive to look at the way in which experience modifies selectivity. In all cases the cortex of animals whose visual experience has been modified lacks neurons selectively responsive to patterns of excitation which a normal animal receives, but which have been excluded by the experimental modification. Thus unilateral lid suture led to a cortex with very few neurons excitable from the lid-sutured eye; likewise, alternating occlusion or strabismus, which decreases the probability of simultaneous excitation of corresponding neurons in the two eyes, decreased the proportion of neurons responding to both eyes. The same is true of the kittens reared in striped environments, or with a vertically deviating prism over one eye, or in an environment with point sources but no lines; in alt these cases the rule holds that neurons are found for patterns of excitation that occur in the modified environment, but normally occurring types of selectivity are rare or absent if the patterns they would respond to have not been experienced in the modified environment.
232
380
This rule seems to amount to a striking confirmation of the speculation (Barlow, I960) that a prime function of sensory centres is to code efficiently the patterns of excitation that occur, thus developing a less redundant representation of the environment. Previous examples of redundancy-reducing codes could be explained as genetically determined features of neural connectivity, but the above discoveries are definite examples of a modified code developed in response lo a modified environment. i f on this page we have begun the correct story for simple cells of area I 7. one can see lhal a book has been opened with regard lo the properties of cells higher in the hierarchy, which are presumably themselves experience dependent and are fed by information from these experience-dependent neurons at the lower cortical levels. Even a small degree of modifiability would be extraordinarily significant in a hierarchically organized system, just as, in evolution, weak selection pressure is effective over many generations. 5 Current concept of the single neuron The cumulative effect of all the changes I have tried to outline above has been to make us realise that each single neuron can perform a much more complex apd subtle task than had previously been thought. Neurons do not loosely and unreliably map the luminous intensities of the visual image onto our sensorium. but instead they detect pattern elements, discriminate the depth of objecls. ignore irrelevanl causes of variation, and are arranged in an intriguing hierarchy. Furthermore, there is evidence that they give prominence to what is inrormationally Important, can respond with great reliability, and can have their pattern selectivity permanently modified by early visual experience. This amounts to a revolution in our oullook. I l is now quite inappropriate to regard unit activity as a noisy indication of more basic and reliable processes involved in mental operations; inslead, we must regard single neurons as the prime movers of these meehanisms. Thinking is brought about by neurons, and we should not use phrases like 'unit activity reflects, reveals, or monitors thought processes', because Ihe activities or neurons, quite simply, OW thought processes. This revolution stemmed from physiological work iiritl makes us rcali/.e lhal ihe activity of each single neuron may play a significant role ill perception. I think thai more clearly staled hypotheses are now needed llbutil llu.se roles in order lo allow our psychological knowledge and intuitions about our perceptions to help us plan future experiments. 6 Five propositions The following five brief statements are Intended lo define which aspect of the brain s activity is i m p o r t J I I I for understanding tt.i main limclion. lo suggest the way that single neurons represent whal is going on around us. and lo say how this is related lo our subjective experience. The statements are dogmalie and incautious because it is important that they should be clear and testable.
6.1
First dogma
A description of that activity of a single nerve cell which is iransimilcd l o and influences other nerve cells, and or a nerve cell's response lo such influences from other cells, is a complete enough description for runctional understanding o f the nervous system. There is nothing else 'looking a l ' or controlling this activity, which must therefore provide a basis for understanding how ihe brain controls behaviour.
6.2 Second dogma At progressively higher levels in sensory pathways information about the physical stimulus is carried by progressively fewer active neurons. The sensory system is
233
331
organized to achieve as complete a representation as possible with the minimum number of active neurons.
6.3 Third dogma Trigger features of neurons are matched to the redundant features of sensory stimulation in order to achieve greater completeness and economy of representation. This selective responsiveness is determined by the sensory stimulation to which neurons have been exposed, as well as by genetic factors operating during development.
6.4 Fourth dogma Just as physical stimuli directly cause receptors to initiate neural activity, so the active high-level neurons directly and simply cause the elements of our perception.
6.5
Fifth dogma
The frequency of neural impulses codes subjective certainty: a high impulse frequency in a given neuron corresponds to a high degree of confidence that the cause of the percept is present in the external world. 7 First dogma: Significant level of description This dogma asserts that a picture of how the brain works, and in particular how it processes and represents sensory information, can be built up from knowledge of the interactions of individual cells. At the moment single-unit electrical recording is the only tool wilh temporal and spatial resolution adequate to locate the effect of a particular sensory stimulus in a particular cell. Other tools (biochemical, electron microscopy, etc.) can obviously provide essential information about these interactions, but the dogma may be criticized more fundamentally; it may be suggested that the whole problem should be approached at a different level. One could attack from either side, suggesting either that one should look at grosser signs of nervous activity, such as the weak extracellular potentials that result from the activity of many neurons, or that one should approach the problem at a more microscopic level, studying synaptic and molecular changes. Interest in evoked potentials and electroencephalography has waned partly because their study led to slow progress compared with single-unit recording, but also because the rationale for their use was undermined. A prime reason for attending to these macroscopic manifestations of nervous activity was the belief that individual cells were too unreliable to be worthy of attention singly, and hence it was better to look at a sign of activity that resulted from many of them. Here, it was thought, may be a property of a group of cells analogous to temperature or pressure as a property of a collection of molecules that individually behave randomly. The demonstration that single nerve cells have diverse and highly specific responsiveness to sensory stimuli, and are astonishingly reliable, showed the fallacy of this analogy. The search for a molar property of a mass of working nerve cells is certainly not worthless. Physiologists, and all biologists for that matter, tend to be emotionally divided into globalists and atomists. The globalists are amazed at the perfection of functioning of the whole animal, and they observe that the atomists' analytical investigations of living matter always leave unexplained many of the most remarkable attributes of the intact animal. As a result the globalist can play a crucially important role in pointing out where the atomists'explanations are incomplete. Now the brain does much more interesting things than produce weak extracellular potentials: it controls behaviour, and this is surely the global product that, at our present state of understanding, really does appear greater than the sum of its parts. It would be no use looking at single neurons if it will be forever impossible to explain overall behaviour in terms of the actions and interactions of these subunits;
234
382
if that were so, the globalists' despair would be justified. On the other hand il is precisely because rapid progress has been made lhal this article is being written; il no longer seems completely unrealistic to attempt to understand perception al I lie atomic single-unit level. The second criticism, that one should approach the problem at a more microscopic level, is really only answerable by saying. "Go ahead and do it", for undoubtedly there is much to be learned at a synaptic and molecular level. But the important question here is whether lack of this knowledge will impede a major advance in our conception of how the brain works. The dogma asserts that it is the intercellular actions and interactions that possess the elaborate organization responsible Tor behaviour; hence it asserts that knowledge at a more-microscopic intracellular level is not a prerequisite for understanding such organization. S Second dogma: The economical representation of sensory messages The main task in this section is to discern the principles that underlie the changes in characteristic responsiveness of single units at successive levels in sensory pathways. The aim is to understand how sensory information is represented or 'displayed'. The successive levels to be considered will be peripheral photoreceptors and cutaneous afferents; retinal ganglion cells of the cat, frog, or rabbit, the latter of which seem to exemplify a more complex type of processing; and the visual cortex of cats. Obviously these are not an ideal series for comparisons and extrapolations, but they are the best we can do. The discussion initially revolves around three issues: changes in the degree of specificity and generality of the stimuli to which the cells respond; changes in the number of parallel categories of selectively sensitive cells that carry the information; and changes in the number of the cells that one may expect to be activated by normal visual scenes. What emerges is that, at the higher levels, fewer and fewer cells are active, but each represents a more and more specific happening in the sensory environment. 8.1 Specificity and generality of responsiveness The pattern specificity of sensory neurons is the aspect that is most widely emphasized; it was spectacular to discover single neurons in the retina responding to movement of the image in a specific direction, cortical neurons responding only to slits of light at a particular orientation, and a unit in the inTero-temporal cortex that responds best to a monkey's hand. But the invariance of the response to changes in the stimulus is equally remarkable. A retinal unit continues to respond to direction of motion in spite of many decades of change in input luminance or contrast, in fact in spite of reversal of contrast (Barlow, 1969a). At the cortical level a complex cell insists that a stimulus is appropriately oriented, but will respond in spile of wide variations of position (Hubel and Wiesel, 1962), And the monkey-paw unit similarly retains its pattern specificity over a large part of the visual field (Gross et al,, 19721. In talking about these properties of sensory neurons actual examples are perhaps more informative than the words specific and general. A single receptor containing a red-sensitive pigment is specific in the sense lhal long-wave length light must be present at a particular part of the image in order to excite it, and it is general in the sense that all images with this property will excite it. In contrast to this type of specificity and generality, the high-level neurons are no longer limited to purely local attributes of the image. They are selective for pattern, which requires that a considerable region of the image is taken into account. But there are other aspects of their specific selectivity that also need to be considered.
235
383
8.2 Number of selective categories At Ihe level of receptors there are a small number of different sensory modalities picking up. in parallel, information from different positions. This is Ihe case both for the half dozen types of cutaneous Sensation, and for the smaller number o f retinal receptor types responding to the visual image. A t the level o f ganglion cells the number of sub-modalities or selective categories has greatly increased. Consider the rabbit, where there are the following (Barlow et at., 1964; Levick, 1967); two concentric types (on- and off-centre); four o n - o f f type directionally selective (for movements up, down, antero-posterior, and postero-anterior); three directions for slow, on-type, directionally selective; one type sensitive l o fast movement; one type sensitive to 'uniformity'; and, confined to the visual streak, two types of orientationselective neurons, neurons selective for slow-moving small objects, and neurons selective for edges. This makes a total o f 15 different selective categories. In addition there must be units signalling colour, since the rabbit shows behavioural evidence for it, but these have not yet been found in the retina. Now move up l o the simple cells in area 17 o f cat cortex. These vary in position, orientation, disparity, and size, as well as being selective for light bars, dark bars, or edges. The evidence is not sufficient to say how many distinct selective categories these form, but for each of the first three variables the resolution o f a single neuron is good, in the sense lhal small departures from the preferred position, orientation, or disparity cause large decreases o f response amplitude (Bishop, 1970). These variables already define four dimensions, and we have not yet considered size specificity, velocity specificity, nor the additional complexities o f light, dark, or edge detectors, and of course colour. There are certainly several orders o f magnitude more neurons in the primary projection area than there are input fibres, or resolvable points in the visual field, and it is abundantly clear that the number o f selective categories has increased enormously. Activity o f a particular neuron signifies much more than the presence o f light at a particular locus in the visual field; its activity signifies a great deal about the nature o f the pattern o f light at that locus. The fact that many parallel communication channels are used in Ihe nervous system has been widely recognised, but here we see an enormous expansion of the number of parallel paths, and this occurs without much redundant reduplication o f channels, for each neuron seems to have a different specific responsiveness. It is as if, at high levels, the size of the alphabet available for representing a sensory message was enormously increased. Perhaps it would be betler to say that, i f the activity o f a low-level neuron is like the occurrence of a letter, that of a high-level neuron is like the occurrence of a word—a meaningful combination o f letters. But to understand this better we must look at the third aspect o f the way sensory messages are represented at different levels, namely the proportion of neurons that are usually active. I f the pattern of activity caused by a visual scene has, on average, AT neurons active out of the total of N neurons, then we have seen above that N increases at high levels; can one say anything about how K changes?
8.3 Number of active celts If one considers the retinal cones under typical photopic conditions, the vast majority must be partially active. They may be nearer the depolarized than the hyperpo la riled limit of their dynamic range, but the majority will be somewhere well within i t . For Ihe relinal ganglion cells of a cat the situation is a little different; while a few units, those corresponding to the brightest and dimmest parts o f the scene, will be vigorously active, the majority, corresponding to the parts o f the scene near the mean luminance, will be discharging at rates close to their maintained discharge level, which in its turn is near the low-frequency end of their dynamic range. Thus there will be a lot o f units with low degrees o f activity and a few which
236
334
are vigorously active. Recoding in the retina changes the distribution oi' activity so that low impulse frequencies are common, high impulse frequencies rare. Now consider Ihe rahbil, wilh its more elaborate retinal processing, and greater richness of pattern-feature signalling neurons. IL is characteristic of the more specific of these neurons that they have a very low maintained discharge, and are extremely hard to excite until their exact trigger feature has been found. One flashes lights, waves wands, and jiggles 'noise figures' in the appropriate part of the visual field for many minutes, maybe hours, before finding the right combination for excitation. It is reasonably certain that the right combination does not occur often in the natural environment either, and therefore Ihese units must spend only a small fraction of the time in an active state. Low impulse frequencies are even commoner, high impulse frequencies even rarer, than in cat retina. For the cat cortex this trend is carried further, and one can see another aspect emerging. If one takes a small region of the visual field, it either does contain a bright bar, dark bar, or edge, or, much more likely, it does not. Thus, like the rabbit units, the cells with these specific responsivities must be only infrequently active. But in addition, on the rare occasions when one of the appropriate trigger features is present, it is one of a set which tend to be mutually exclusive: a bright bar cannot be a dark bar, and it can have only one orientation and disparity. The stimulus selects which cell to activate from a range of many possible cells, and it is pretty well impossible to activate simultaneously more than a small fraction of this number The picture developing is that at low levels visual information is carried by the pattern of joint activity of many elements, whereas at the upper levels of the hierarchy a relatively small proportion are active, and each of these says a lot when it is active. But, although we clearly see that the proportion active, KIN, decreases, we cannot tell whether it decreases as rapidly as N increases, and thus we still do not know how K itself changes. The second dogma goes beyond the evidence, but it attempts to make sense out of it. it asserts that the overall direction or aim of information processing in higher sensory centres is to represent the input as completely as possible by activity in as few neurons as possible (Barlow, 1961. 1969b), In other words, not only the proportion but also the actual number oi active neurons, K, is reduced, while as much information as possible about Ihe inpul is preserved. By now much can one reasonably expeel K to be reduced? One requires Ihe concepts of channel capacity and redundancy from information Ihcory (Shannon and Weaver. 1949; Woodward, I9S3) to make a rough estimate. Some reduction can be accomplished without any loss of information simply by the increase of N. KIN is the probability of a fibre being active, and, if it is the same for all neurons, the information capacity of a sel of N neurons, either active or not active, is -Aflogi(iSr/A/)-{/V- K)log,((/V- K)IN\. If KIN is small, the second term contributes little; the capacity then is, approximately, the number of active neurons times the information provided by each active neuron, and this increases directly as the negative logarithm of the probability of it being active, -\og(K/N). Hence the number active can be reduced as N increases without loss of information capacity, but by itself this does not allow K to be reduced very much: for instance, if we suppose that 1 of the 2 x 10 optic nerve fibres are active and that there are 10' cortical neurons receiving this information, then one finds that I -5 x 10 cortical neurons must, on average, be active in order to have the same information capacity as the 5 x 10' active optic nerve fibres. But this applies only to capacity, and a substantial reduction in K is possible on the basis of another principle. s
s
Visual information is enormously redundant, and it has been suggested previously that sensory coding is largely concerned with exploiting this redundancy to obtain
237
385
more economical representation of the information. If the argument is correct, the number of active neurons can be reduced, but it is very difficult even to guess how big a reduction in K such recoding can achieve; if it is rii up to the cortex, and another factor of achieved in visual I , II and III, one would end up with about 1000 active fibres carrying the information provided by 5 x 10 active optic nerve fibres; though the reductions might be substantially greater or less, this is the order of magnitude oT the reduction contemplated. J
According to dogma, these 1000 active neurons represent the visual scene, but it is obvious that each neuron must convey an enormously larger share of the picture than, say. one point out of the quarter million points of a television picture. Perhaps a better analogy is to recall the 1000 words that a picture is proverbially worth; apparently an active neuron says something of the order of complexity of a word. It seems to me not unreasonable to suppose that a single visual scene can be represented quite completely by about 1000 of such entities, bearing in mind that each one is selected from a vast vocabulary and will in addition carry some positional information. 9 Third dogma: Selectivity adapted to environment
9.1 Evolutionary adaptation Some economies of the type indicated above can be achieved by exploiting forms of redundancy which are present in all normal environments. Levels of sensory stimulation do not range at random over the whole scale of possible values, and it makes sense to regard adaptation of peripheral receptors as a measure to achieve economy by signalling changes from the mean instead of absolute values. Similarly in most situations neighbouring points on a sensory surface are more likely to be similar than distant points, and it thus makes sense to regard contrast enhancement by lateral inhibition as another economy measure. The argument can be carried on to cover the redundancy-reducing value of movement, edge, or disparity detectors (Barlow, 1969b), but, if these are genetically-determined redundancy-reducing codes, they must be fixed once and for all during development, and they could only work for redundant properties of all sensory environments. The hypothesis becomes more interesting when one considers the possible mechanisms Tor achieving economy by exploiting the redundancy of particular sensory scenes, for this requires storage of information and plasticity of the neural structures involved.
9.2 Reversible adaptation The neural changes of dark and light adaptation may be regarded as a simple example of reversible plasticity achieving this end. The luminance corresponding to zero impulses is affected by the past history of illumination and by the surrounding luminances in such a way that the majority of fibres are responding at low frequencies. But, even though this involves definite changes in the synaptic transfer properties of retinal neurons, the statistical characteristic of visual images that enables this to achieve economy is always the same, namely the fact that the distribution of luminances is grouped around local and temporal mean values, so that small deviations from the mean are commoner than large deviations (Barlow, 1969a). Hence the most commonly occurring luminances require fewest impulses.
9.3 Permanent adaptation The effects permanently impressed on the visual system during the sensitive period are the first example of plasticity for a particular type of redundancy. The distribution of orientational selectivity of primary neurons is biased in favour of the orientations the individual experienced during this critical time. If the analogy of a neuron's signal resembling the utterance of a word is recalled, this result suggests that the kitten's cortex only develops words for what it has seen. This could be brought
238
386
aboul by either selection or moilificalion: are Ihe dictionary words there, only ihe ones experienced becoming permanently connected; or do ihe cells themselves determine that a frequently experienced pattern, sucli sis lines of a particular range of orientations, are events for which words are desirable'! The evidence favours modification, and the idea to which i l leads of the successive hierarchical construction of a dictionary of meaningful neurons has enormous appeal. For the present we can only justify the third dogma by saying the evidence suggests such a dictionary may be built up. though we are far from being able to look into its pages by physiological methods, hi the ncxi section we turn to the subjective view of this dictionary. 10 Fourth dogma: Origin of perceptions
10.1 Persona! perception In order to delimit more accurately what this dogma does and does not say it may be useful to define and separate three mysteries of perception. The first is the personal, subjective, aspec! of my experience of. say. the red pencil with a blue eraser in my hand. There does not seem to be anything that could be said about the activity or nerve cells accompanying this experience lhat would in any way 'explain' the aspect of it that is mysterious, personal, and subjective. I think this part of the experience is something that one must be content to leave on one side for the moment, but it is important that this part of subjective experience almost always accompanies electrical stimulation of a peripheral sensory nerve, and usually accompanies electrical stimulation of the sensory areas of the brain, for Ihis implies that the Tull subjective experience, including this mysterious personal element, accompanies the neural events of sensation, however these arc caused. This fact strongly suggests that it is no waste of time to look into these neural events: beauly is a mysterious attribute of a work of art, but lhat does not imply thai you cannot create a beautiful painting by non-mysterious material means.
10.2 Conscious perception The second mystery is thai we are not consciously aware of much that goes on in our brains, so the inverse of the fourth dogma is certainly not true: not every cortical neuron's activity has a simple perceptual correlate. Even at high mental levels much neural business is conducted without conscious awareness, and my own belief is lhat (he conscious pari is ijimfjiKjd lo experiences one communicates to other people, or experiences one is contemplating communicating lo other people. This immediately introduces a social clement inlo individual consciousness, for communication is impossible without a channel being open to a recipient. However, for present purposes we need only point out that interesting aspects of consciousness of Ihis sorl are by no means incompatible with Ihe fourth dogma. An clement of perception can possess a simple neural cause without i l necessarily being the case lhat all simple neural eveuls cause perception. There is Iherel'ore plciily ol room lor social, historical, or moral influences on perception, because these can Influence ihe selection o f the neural events that enter conscious perception.
10.3 Validity of perceptions The third mystery about perceptions is why they arc generally 'Irue': why are they so extraordinarily useful in guiding our actions and helping us to make decisions? This is the aspeel that the second and third dogmas help one to understand. The economical and fairly complete representation of visual scenes by a reasonably small number of active neurons makes it much easier to visualize how they can be used for these purposes. The key point is that the active neurons carry the bulk of the information, and the vast number o f inactive ones need not be laken into consideration. The difficulty of detecting among our sense impressions the entities
239
387
we use for rational thought has always been baffling: 'water', 'men', 'sheep', and even the simple letter ' A ' represent particular logical functions o f activity among the sensory neurons, but the number or possible logical functions is so vast that we are mystified how particular ones are realized, or why particular ones are selected for realization. The representation suggested by the second and third dogmas would allow relatively simple logical combinations to have properties approaching those required for the literal symbols of Boole (1854), the subjects o f our conceptions. By using such symbols together with operational signs he founded mathematical logic, but the title of his major work, "The Laws of Thought", clearly states his claim that his inquiries had "probable intimations concerning the nature and constitution o f the human m i n d " It is gratifying to approach closer t o an intuitively plausible neural realization of what he symbolized. The notion that what we sense is a point by point representation of the physical signals impinging on our body has been rejected for psychological and philosophical reasons (see Boring, 1942), and more recent physiological evidence clearly supports this rejection. But this same evidence suggests that i t should be replaced, not by return to a subjectively constructed phenomenology (Dreyfus, 1972), nor by the notion that we sense the world in terms o f rigidly preordained 'structures', but by the deeper and more adaptive ideas of dogmas two and three; our sensorium is presented with a fairly small number of communications, each representing the occurrence of a group of external events having a word-like order of complexity, and, like words, having the special property that they lead to an economical representation o f these physical events. 11 Fifth dogma: Signalling subjective certainty There is one way in which the properties of neurons do not match up to the way Boole used symbols, for he insisted on their binary nature, or 'duality'. This is the property Aristotle called the principle of contradiction, and without it Boole's symbolic representation of logic would have been impossible. In contrast to this duality the response of a sensory nerve cell to its trigger feature consists o f a volley of impulses lasting TO to I second, during which time the neuron can discharge any number of impulses between zero and nearly 1000. Therefore the response is graded, and it is not legitimate to consider it as a Boolean binary variable. The essential notion expressed in the fifth dogma is that a neuron stands for an idealization of reality whose complement can be formulated as a null hypothesis, and it is this that has the required Boolean logical property o f duality. The idealizations should not be thought of as Kantian or Platonic, but rather as abstractions that model reality in the manner suggested by Craik (1943). The ideal populations of a statistician come even closer, for the parameters of such distributions model reality, and they are used to calculate whether or not a particular sample belongs within i t . The process o f idealizing the complement o f the trigger feature will be clarified by a simple example. Suppose we have a sensory neuron whose trigger feature is a simple physical event, such as the increase of light intensity at a specific position in the visual field. We are examining the suggestion that the graded responses this neuron gives to visual stimuli represent some function of the degree of certainty that the light did in fact increase, estimated from the physical events accessible to the neuron. For simplicity assume that a record is available of the total numbers o f quanta absorbed in the receptive field of the sensory neuron during successive periods of about rb second duration up to and including the period about which it is to signal centrally. It is i n principle possible to calculate the probability o f occurrence o f the observed number of quantal absorptions on the hypothesis that there was no change i n ihe light intensity, and this is the test a statistician would apply to determine whether or not
240
:
u
Ihe trigger feature was present. On ihis view impulse frei|Ucit e signals some rtintluj'n of Ihe sign i lie m i l . hvel at a lust i>f Ihis sort, low probabilities corresponding lo high impulse frequencies. Notice that Ihe trigger feutilrc is "an increase o f light', the idealization is 'there was no Increase', and this idealization is based on observation o f whal has recently happened and therefore incorporates a model o f Ihe recent past. The responses actually obtained to varying intensities of incremental stimulus fit quite well into this scheme (Barlow, 1969a), as do some less obvious features. The low-frequency maintained discharge could well represent the results, oT low significance, obtained by testing the null hypothesis when no stimulus has been applied. Although individual fibres would not reach significance, information about quantal absorptions would be retained, and changes insignificant singly could be combined centrally to reach significance. In the above example a high value o f P and low impulse frequency would result from a shadow falling on the receptive field. The detection of this shadow might be of great survival significance for the animal, but the on-centre unit's trigger feature, null hypothesis, and statistical tests would be a poor way of detecting and signalling this important event. A different type of unit is required whose null hypothesis should be "There has been no decrease in the quantal absorption rate"; these would fire when the hypothesis is disproved by the quantal absorption rate dropping below the normal range of variation. Obviously these are the off-centre units, and i l seems that the existence of complementary 'on' and ' o f f systems fits the notion quite well. When there are a large number of neurons with trigger features that cannot coexist, as in area 17, these correspond to a large number of mutually exclusive hypotheses to be tested. The fifth dogma clearly requires more development and testing, but it provides a possible answer to the question "Whal variable corresponds to impulse frequency in a high level sensory neuron ' " Furthermore the answer lies i l to a rather definitely felt subjective quantity—the sense of certainty. 12 Criticisms and alternatives Single-unit recording hints al this probabilistic adaptive, many Ii villi. U bygli.-iii lor processing and displaying sensory information, bul can we believe lhat what we perceive is the activity of a relatively small selection of upper-level units oi' this hierarchy'.' This is certainly a big jump beyond the present physiological evidence. We do not know how perspective transformations are disregarded, enabling us to perceive Ihe same object irrespective of our angle of view, nor do we understand the mechanisms underlying size constancy, yet these mechanisms must intervene between the highest neurons we know aboul and quite simple perceptions. I think we have seen enough of whal can be achieved in a few stages of neural image processing lo believe lhat a lew more stages could reach the point where a single neuron embodies, by virtue oT its peripheral connections and their properties, an elementary percept, but let us examine an alternative and the evidence adduced in its support. 12.1 Combinatorial or holographic represenlutitms The key suggestion aboul the organization of sensory processing thai Ihe second dogma asserts is that the information is carried by progressively fewer active neurons at progressively higher levels in Ihe hierarchy. The brain receives complex patterns or activity in nerve fibres from the sensory receptors, and it generates complex patterns o f outgoing commands to the muscles. It could be held thai the patterns are equally complex al all the intervening stages as well, and Ihis would mean lhal Ihe significance o f a single unit's activity would be virtually undecipherable without knowing what was going on in a host of other units. Certainly one would make
241
389
little progress in understanding a computer's operation by following the status of a single bit in its central processor, so this criticism is partly met by pointing to the success lhat has been achieved in the visual system by looking at the activity o f units singly, one at a time. But we should also look critically at the main evidence advanced in favour of the combinatorial or holographic scheme.
12.2 Mass action and resistance to damage The main argument that has been levelled against the view that individual cells play an important role in perception and in favour o f a holographic representation is the reported fact that large parts of the cortex can be damaged with only minor resultant changes in behaviour or learning {Lashley, 1929, 1950), This led to Lashley's doctrine of 'cerebral mass action', but repetition o f the original experiments and refinements in methods of testing, some of it by Lashley himself, have considerably weakened the original evidence in its favour (Zangwill, 1961). However, it certainly is remarkable that a mechanism with as much interdependence between its parts as the brain can function at all after it has been extensively damaged. A computer would not usually survive brain surgery or gunshot wounds, and it is therefore worthwhile considering the implications of the fact that the cortex is relatively immune to quite extensive injury. The whole of a visual scene can be reconstructed from a small part o f a hologram, with only slight loss of resolution and degradation of signal/noise ratio (Gabor et al., 1971), so it has been claimed that the cortex must operate by some analogous principle in order to account for its resistance to damage. What is not widely appreciated is the fact that holography differs from ordinary image-recording photography not only in principle, but also in the materials used, for it requires photographic emulsions with resolutions of the order of the wavelength of light (Gabor et a!., 1971). With such materials a good quality 35 mm picture could easily be reproduced and repeated in every 1 m m of the plate, and a plate containing such a reduplicated image would have to be pulverized into tiny pieces to prevent reconstructibility of the original from every fragment. The mass-action-like resistance to damage of the hologram is partly due lo the enormous informational capacity o f the materials lhal are required; immunity tu damage is easy to achieve when such high redundancy is permissible, and this argument carries little weight in favour of holographic views of nervous operation. 1
Codes can be given error-correcting properties much less wastefully. and the argument can be turned around to favour the representation hypothesized in dogmas two and three. Because the few active cells have a fixed significance, and because the inactive ones are thought to carry so little information that they can be neglected, the only result of removing part o f the cortex would be to eliminate some of the active neurons, and hence some of the perceptual entities, when a given scene is viewed. The 'meaning' of the active units in the undamaged cortex would remain the same, and might provide a sufficient basis for decision and action. This is very different from the situation where a neuron's activity has totally different significance depending upon the pattern of activity of which it forms a part, for i f any of this pattern was in a damaged region the significance of activity i n the undamaged part would be altered. Hence damage immunity is really an argument for neurons having an invariant meaning not dependent upon the activity of other neurons. It should also be pointed out that very limited replication of 'percept neurons' would give considerable damage immunity: i f a given neuron is replicated half a dozen times in different cortical regions there is a good chance of at least one o f them surviving an extensive cortical ablation. This sixfold redundancy is enormously less than the holographic scheme possesses, and it can be concluded that the m a s
242
390
action argument rebounds against the extensive combinatorial usage o( neurons and actually favours the hypothesis o f this article,
12.3 Pontifical celts Sherrington (194.1) introduced the notion of "one ultimate pontifical nerve-cell, ... lite climax of the whole system o f integration" and immediately rejected the idea in favour of the concept of mind as a "million-fold democracy whose each unit is a cell" Those who like the notion of perception as a cooperative or emergent property of many cells dismiss the suggestion lhat the activity or a single neuron can be an important element o f perception by saying that, carried to its logical conclusion, it implies there must be a single 'pontifical cell' corresponding to each and every recognizable object or scene. First, notice thai the current proposal does not say that each distinct perception corresponds to a different neuron being active, i f perception is taken to mean the whole o f what is perceived at any one moment; it says there is a simple correspondence between the elements or perception and unit activity. Thus the whole of subjective experience at any one time must correspond to a specific combination o f active cells, and the 'pontifical cell' should be replaced by a number o f 'cardinal cells'. Among the many cardinals only a few speak at once; each makes a complicated statement, but not, o f course, as complicated as that of the pontif i f he were to express the whole of perception in one utterance. Two important difficulties arise from the nolion of pontifical cells; first, i f a separate neuron is needed for each of our perceptions, there are not enough to account for their almost incredible variety; second, the activity of a single isolated element would not convey anything o f a perception's great richness, the connection between one perception and others. The 'grandmother cell' might respond lo all views of grandmother's face, but how would that indicate thai it shares features In common with other human faces, and that, on a particular occasion, it occurs in a specific position surrounded by other recognizable objects? Our pcrceplions simply do not have the properly of being isolated unique events as one would expect i f each corresponded to the firing of a unique neuron. Instead, they overlap with each other, sharing parts which continue unchanged from one moment lo another, or recur al later moments in different contexts. I think the 'cardinal cell' representation surmounts these problems without any difficulty; i f a critic can say how many different pcrceplions we arc capable of, and how rich a network of relalcdness exists between these perceptions, then one might be able to cslimalc how many cardinals' voices were required to represent these perceptions. But there is a misleading feature of the ecclesiastical analogy. Most organizational hierarchies are pyramids: there are many members of the church, fewer priests, only a select number oT cardinals, and a single pope. The hierarchy of sensory neurons is very different. It is true that there are more retinal receptors than ganglion cells, but the number of cortical neurons in area 17 is certainly orders o f magnitude greater than the number of incoming fibres. The numbers at succeeding levels may be somewhat fewer, but a high proportion of the nerve cells in the brain must be capable oT being influenced by vision, so i f the hierarchical organization is pyramidal it is inverted rather lhan erect, divergent rather than convergent. I f one uses the term 'cardinal cell', one musl be sure lo remember lhat the college of these cardinals outnumbers the church members and must include a substantial fraction of the 1 0 ' ° cells of the human brain. After-though ts It is sufficiently obvious that these propositions are incomplete, that there are aspects of the sensory problem left untouched, and lhat Ihe dogmas go considerably beyond the evidence. 1 have said, in essence, that the cells of our brain are each capable of
243
391
mote than had previously be more simply related to previously been thought. program with its recursive simple hierarchy of clever perception.
been supposed, and that what their activities represent may the elements of our conscious perceptions than had But clever neurons are not enough. The simplest computer routines and branch points has more subtlety than the neurons that 1 have here proposed as the substrate o f
I think one can actually point to the main element that is lacking. We have seen that some properties of the environment can be represented, or modelled, in a system of the type proposed; 1 feel that a corresponding model is also needed for our own motor actions and their consequences. Such motor and sensory models could then interact and play exploratory games with each other, providing an internal model for the attempts of our ever-inquisitive perceptions to grasp the world around us. A higher-level language than that o f neuronal firing might be required to describe and conceptualize such games, but its elements would have to be reducible t o , or constructible from, the interactions o f neurons. The five dogmas do not impede developments i n this direction. My claim for them is that they are a simple set o f hypotheses on an interesting topic, that they are compatible with currently known facts, and that, i f any are disproved, then knowledge in this field will be substantially advanced. Acknowledgements. This essay was started many years ago when Gerald Westheimer suggested Lo me thai, if a single-neuron dogma of the power and generality of 'DNA codes protein' could be found, il might speed progress of neuropsychology as much as Crick and Watson speeded up molecular biology. Since Ihen I have been helped by the discussion of these ideas with a group of neurophysiologists and psychologists organized in Berkeley by M. F. Land, and by many useful suggestions from B. Sakill, J think Ihe single neuron revolution is having a powerful effect in sensory psychology, bul I slill wish it could be expressed in a single dogma. References Adrian, E. D., 1926a, "The impulses produced by sensory nerve-endings", Pt.l, /. Physiol., 61, 49-71. Adrian, E. D., 1926b. "The impulses produced by sensory nerve-endings, Pt.4, Impulses from pain receptors",J. Physiol.. 62, 33-51. Adrian, E. D . 1928. 77ie Basis of Sensation (Christophers, London; also Haftier. New Yoik, 1964). Adrian, E. D.. 1941, "Afferent discharges to the cerebral corlex from peripheral sense organs", J. Physiol.,
100. 159-191.
Adrian, E. D., 1947, The physical background of perception (Clarendon Press, Oxford). Adrian, E. D., Zotteiman, Y., 1926a, "The impulses produced by sensory nerve-endings, Pi.2, The response of a single end-organ", J. Physiol., 61, 151-171. Adrian, E. D., Zotterman, Y., 1926b, 'The impulses produced by sensory nerve-endings, Pt,3, Impulses set up by touch and pressure", / Physiol, 61, 465-493. Bard, P., 1938, "Sludies on the cortical representation of somalic sensitivity", Harvey Lectures 1938 (Academic Press, New York), pp.143-169. Barlow, H. B., 1953, "Summation and inhibition in the frog's retina", / Phydol., 119, 69-88. Barlow, it. B., 1956, "Retinal noise and absolute threshold", / Opt. Soc, Amer., 46, 634-639. Barlow, H, B., I960, 'The coding of sensory messages" in Current Problems in Animal Behaviour, Eds, W. H. Thorpe, O. L. Zangwill (Cambridge University Press, Cambridge), pp33l-360. Barlow. H. B,. 1961, "Possible principles underlying (he transformations of sensory meissgei" in Sensory Communication, Ed. W. A, Rosenblilh (MIT Press. Cambridge, Mass. and John Wiley New York), pp.217-234. Barlow. H. B., 1969a, "Pattern recognition and the responses of sensory neurons", Arm, N, ¥. Acad Sci., 156, 872-881. Barlow, H. B., 1969b, 'Triggei fealuies, adaptation, and economy of impulses", in Information Processing in the Nervous System. Ed, K. N. Leibovic (Springer-Verlag, New York), pp.209-226. Baslow, H. B., Blakemore, C, Pelligrew, J, D,, 1967, 'The neural mechanism of binocular depth discriminalion", /. Physiol., 193, 327-342. Bailow, H. B., Brindley, C. S., 1963. "Interooular transfer of movement after-effects during pressure binding of Ihe stimulated eye", Nature, 200, 1346-1347.
244
392
Barlow, H. B„ FilzHugh, R„ Kuffler, S. W.. 1957, "Change of organization in ihe receptive fields or the cat's retina during dark adaptation", J. Physioi, 137, 338-354. Barlow, H. B„ Hill, R. M., 1963, "Evidence for a physiological explanation of the waterfall phenomenon andfigure!aricr-effects", Nature, 200. 1345-1347. Barlow, II B.. Hill. R. M., Levick. W. R.. 1964, "Retinal ganglion cells responding selectively 10 direction and speed of image motion in the rabbit", /. Physiol., 173. 377-407. Barlow. H. B., Levick, W. k., Yoon. M., 1971, "Responses lo single quanta of light in retinal ganglion cells of the cat", Vision Research, 11, Suppl. 3, 87-102. Barlow, H. B., Peitigrew, 3.D., 1971, "Lack of specificity of neurones in ihe visual corlex of young kittens",/. Physiol, 218, 98-100. Bishop, P, 0,, 1970, "Beginning of form vision and binocular deplh discrimination in corlex". in TTre Neurosciences: Second Study Program, Ed. P.O. Schmiti (Rockefeller University Press. New York), pp.471 -485. Blakemore, C, Campbell, F, W., 1969, "On the existence of neurones in the human visual system selectively sensitive lo the orientation and size or retinal images". J. Physioi, 203. 237-260. Blakemore, C , Cooper, C. P., 1970, "Developmeni of ihe brain depends on Ihe visual environment", Nature. 228. 477-478. Blakemore, G , Mitchell, D. E., 1973, "Environmental modification uf the visual cortex and the neural basis of learning and memory",Nature. 241, 467-468. Boole, C , 1854, An Investigation of the Laws of Thought (Dover Publications Reprint, New York). Boring. E.G., 1942, Sensation and perception in the history of experimental psychology (Appleton Crofts, New York). Burns, B., 1968, The uncertain nervous system (Edward Arnold. London), Campbell. F. W., Green, D.G., 1965, "Optical and retinal faclors affecting visual resolution", /. Physiol..
181. 576-593,
Cragg, B. G., 1972. "The development of synapses in cat visual corlex", investigative Ophthalmology, 11,377-385. Craik, K. 3 W., 1943. The Nature of Exphnaiion (Cambridge University Press. Cambridge) De Valois, R, L,, I960, "Color vision mechanisms in the monkey", J. Gen, Phvsiai.. 45, Suppl., 115-128. De Valois, R. L.. Abramov. I . . Jacobs, G. H.. 1966. "Analysis of response patterns uf LGN cells", J. Opt. Soc. Am.. 56, 966, 977. De Valois, R, L.. Abramov, I . , Mead, W. R,, 1967, "Single cell analysis of wavelength discrimination at Ihe lateral geniculate nucleus in the macaque", J. rJcitropliysiai. 30. 415-433. Dreyfus, H. L., 1972. Wlial Computers Can't Do (Harper and Row. N e w Yorkl Enrolh-Cugell. C . Robson. J . G . 1966. "The contrast sensitivity of retinal ganglion celts ol Ihe r
at". J. Physioi.
187.517-552.
Ewert, J. P.. 1970. "Neural mechanisms nf piey-calcliine. and avoidance lieiiavinr In the ton J {nufii bufa /, ) " , Helm. Era!., 3,36-56. Fry, G. A., 1948, "Mechanisms subserving simultaneous brightness contrast", Am. J. Qplitnt,. 25. 162-178, Gabor, D., Kock, W. E.,Siroke, G, W.. 1971, "Holography". Science. 173, 11-23 Galambos, R.. 1944, "Inhibition of aclivity in single auditory nerve libers by jenuslie sliinulaiinn", J. Neurophysiol.,
7, 287-303.
Galambos, R., Davis, H., 1943, "The lesponse of single audi Wry-nerve fibres lu iCQUItiC stimulation",./. Ncuniphyitoi. 7. 287-303. Galambos, R., Davis, H., 1948. "Action potentials from single auditory-nerve fibres?" Science, 108, 513. Ganchrow, J. R.. Erickson. R. P., 1970, "Neural currelales of gustatory intensity and quality", J. Neurophysiol..
33. 768-783.
Granit, R., 1947. The Sensory Mechanisms of the Retina (Oxford University Press. Oxford). Granit, R., Svaetichin, G. 1939, "Principles and lechnique of ihe electrophysiological analysis ut colour reception with Ihe aid or microelecnodes". Upsala Lakaref Farh., 65, 161 -177. Green, D. M,, Swets, J. A., 1966. Signal Detection Theory and Psych'iphviics (John Wiley. New York). Cross, C. G., Rocha-Miranda, C. E., Bender, D. B., 1972. "Visual properties of neurons in inferotemporal cortex of Ihe macaque", /, Neurophysiol., 35, 96- I I I . Griisser, O.-J., Griisser-Cornehls, U., 1968, "Neurophysiologische Grundlagen visueller angeborencr AuskSsemechanismen beim Frosch", Zeitschrift fir vergteichende Physiologic, S9, I -24. Hartline, H. K,, 1938, "The response of single optic nerve fibres or the vertebrate eye te illumination or the retina". Am. J. Physiol., 121, 400-415, T
245
393
Hartline, H. K., 1940a, "The receptive fields of optic nerve fibers", Am. J. Physiol.. 130,690-699. Hariline, H. K„ 1940b, 'The effects of spatial summation in the retina on the excitation oflhe fibers of the optic nerve", Am. J. Physiol. 130, 700-711. Hechi, S.i, Shlstr, S., Pirenne. M.. 1942. "Energy, quanta, and vision",/. Gen. Physiol. 25, 819-840. Iliisch, II. V. B„ Spinclli. D. N.. 1970, "Visual experience modifies distribution of horizontally and vertically oriented receptive fields in cats". Science. 168. 869-871. Iliisch, II. V, B„ Spinelli. D. N.. 1971, "Modification of the distribution of receptive field orieuiation in cats by selective visual exposure during development". Exp. Brain Res., 13, 50°-537. Iluhel, 0. H., Wiescl.T. N., 1959, "Receptive fields of single neurones in the cat's striate cortex", / Physiol..
148.
574-591.
Hubel, B. I I . . Wiesel. T. N.. 1962. "Receptive fields, binocular interaction, and functional architecture in (he cat's visual corlex",/ Physiol.. 160, 106-154. Hubel, D. H., Wiesel, T. N., 1963, "Receptive fields of cells in striate cortex of very young, visually inexperienced kittens", / Neurophysiol., 26, 994-1002. Hubel, D. H., Wiesel, T, N„ 1965, "Binocular interaction in striate cortex of kittens reared with artificial squint", / Neurophysiol.. 28, 1041-1059. Hubel, D, H., Wiesel. T. N., 1970. "The period of susceptibility lo the physiological effects of unilateral eye closure in kitlens",/ Physioi, 906, 419-436. Hurvich. L. M., Jameson, D., I960, "Perceived color, induction effects, and opponent-response mechanisms", / Gen. Physioi. 43, Suppl., 66-80. Ingle. D., 1968, "Visual release of prey-calehing behaviour in flogs and loads", Brain, Behaviour and Evolution,
1, 500-518.
Ingle. D., 1971, "Prey-caiching behaviour of anurans toward moving and stationary objects", Research, Suppl, No.3, 447-456, Kuffler, S. W., 1953, "Discharge patterns and functional organization of mammalian retina", /. Neurophysiol.,
Lashley,
Vision
16. 37-68,
1929, Brain Mechanisms and Intelligence: a Quantitative Study of Injuries to the Brain (University of Chicago Press, Chicago), Lashley, K. S,, 1950, "In search of Ihe Engiam physiological mechanisms in animal behaviour", in Symposium of Ihe Society for Experimental Biology, Ed. J. F. Danielli and R. Brown (Cambridge University Piess, Cambridge). Lettvin. J. Y,. Maturana, H. R., MeCull^h, W, S., Pills, W. H„ 1959, "What (he frog's eye tells (he frog's brain", Proc. Inst. Rod Eng.. 47, 1940-1951. Levick. VI, R., 1967. "Receptive fields and trigger features of ganglion cells in the visual streak of ihe rabbu's retina",/ Physiol. 188, 285-307. Mach, E.. 1886, The Analysis of Sensations, and the Relation of Ihe Physical to the Psychical Translation of first edition (1886) levised from fifth German edition by S. Waterlow (Open Court. Chicago and London, 1914) Ed. C. M.Williams, (Also Dover Publications, New York, 1959.) Marshall, W. H., Woolsey, C. N., Bard, P., 1941, "Observations on cortical somatic sensory mechanisms of cat and monkey",/ Neurophysiol, 4, 1 -24. Maiurana. H. R., Letivin, J. Y., McCulloch, W.S., Pitts, W. H., 1960, "Anatomy and physiology of vision in the frog {Rana Pip\ens)",J. Gen. Physioi.. 43, Suppl. No.2, Mechanisms of Vision. 129-171. McCulloch, W. S., 1959. "Agatha Tyche: of nervous nets—the lucky reckoners", in Mechanisation of
K. S.,
Thought
Processes:
Proceedings
of a Symposium
Held
at
the
National
Physical
Laboratory
Vol.2 (HMSO, London), pp.611 -634. Pettigrew, J. D, (forthcoming), "The effect of visual experience on the development of stimulus specificity by kitten cortical neurons". Pettigrew, J. D,, Freeman, R, (forthcoming), "Visual experience without lines; Effect on developing conical neurones" pciiigTCw. 1, D,, Nikaia, T,, Bishop, P. O., 1968, "Binocular interaction on single units in cat siriale C U I I C K ; simultaneous stimulation by single moving slit with receplive fields in correspondence", Exp.
Ilralnftes..
6. 39!
-410.
1'lafiman. C 1941, "Gustatory aflcrcnt impulses", / Cell. Comp. Physioi, 17, 243-258. Pfaffman, C. 1955. "Gustatory nerve impulses in rat. cat, and rabbit",/. Neurophysiol,. 18, 429-440. Sakitt, B.. 1972. "Counting every quantum", / Physioi, 111, 131-150. Shannon. C. E., Weaver, W., 1949. The mathematical theory of communication (University of Illinois Press, Urbana).
246
394
Sherrington, C. S., 1941, Man on His Nature {Cambridge University Press, Cambridge). Shlaet. R., 1971, ••Shift in binocular disparity causes compensatory change in the cortical structure of kittens", Science, 173, 638-541. Stone, J., 1972, "Morphology and physiology of the geniculocorlical synapse in the cat: The quest inn or parallel input to Ihe striate cortex", Invest. Ophthai, 1 1, 338-346. Svaelichin, G., MacNichoi, E. K . Jr., 1958. "Retinal mechanisms for chromatic and achromatic vision", Ann, N. Y. Acad. Sci, 74, 38S-404. Talbot, W. H., Darian-Smilh, 1., Kornhuber, H. H., Mounlcaslle, V. B., 1968. 'The sense of flutter-vibration: Comparison of human capacity with response patlerns of mechano-receptive afrerenls from ihe monkey hand". J. Neurophysiol., 31, 301 -334. Tannei, W. P., Jr.. Swcls, J, A., 1954, "A decision making theory of visual detection", Psychol. Review,
61, 401
-409.
Wagner. H, G„ MacNichoi, E. R., Wolbarshi, M. L., I960, "The response properties or single ganglion cells in Ihe goldfish retina", J. Gen. Physioi, 43. Suppl., 115-128. Wiesel, T. N., Hubel, D. II., 1963, "Single cell responses in striate cortex of kittens deprived of vision in one eye",/. Neurophysloi. 26, 1004-1017, Wiesel, T. N., Hubel, D. H., 1965, "Comparison or the effects of unilateral and bilateral eye closure on cortical unil responses in kittens"./. Neurophysiol., 28, 1029-1040. Wohlgemuth, A., 1911, "On the after-effect of seen movement", Brti. J. Psychol. Monograph. Suppl., I , 1-17. Woodward, P. M . , 1953, Probability and Information Theory with Applications to Radar (Pergamon Press, Oxford). Yirbus, A. L., 1965, Eye Movements and Vision. Translated from Russian by Basil Haigh (Plenum Press, New York). Zangwill, O. L., 1961, "Lashley's concept of cerebral mass aclion", in Current Problems In Animal Behaviour, Eds. W.H.Thorpe. O. L. Zangwill {Cambridge University Press, Cambridge).
247
Reprinted with permission from Vision Research, Vol.30. No. 11, pp. 1561-1571, 1990 © 1990 Pergamon Press
CONDITIONS
F O R VERSATILE
LEARNING,
HELMHOLTZ'S
UNCONSCIOUS
INFERENCE,
AND
T H E TASK
O F
HORACE
PERCEPTION
BARLOW
Physiological Laboratory, Cambridge, CB2 3EG, U.K. (Received
10 August
1989;
in revised form
1 March
1990)
Abstract—It is a mistake to consider perception and learning separately because what one learns is strongly constrained by what one perceives, and what one perceives depends on whal one has experienced. I shall propose the hypolhesis lhal perception is the compulation of a representation thai enables us to make reliable and versatile inferences aboul associations occurring in the world around us—that is, perception prepares the ground for learning. The statistical problem in learning is lo determine whether a compound eveni such as "C followed by U " is a random co-occurrence or a significant association, for if it is the former it would be a mistake lo pay any particular attention lo C, whereas if il is the Latter C is a conditional stimulus for U and a useful predictor for it. Now you cannot decide whether the association is random or not without knowledge of the prior probabilities of C and U: hence on my hypolhesis when you perceive an object or event the representation must nol only signal "it's there" or "it's happened", but musl also make evident (or rapidly accessible! ihe prior probability of what has been signalled. Furthermore it must de Ihis for all the objects or events that can acl as conditional slimub,and Ihis implies lhal Ihe representative elements should be statistically independent (or approximately so) in Ihe normal environment. Forms of coding that would do this, and ihe relationship with Helmholtt's unconsicous inference, will be discussed. These considerations imply lhal Ihe task performed in perception has been overlooked bolh by Learning Iheorists and by connectionisls working on associative and adaptive networks. Coding for independence may be particularly imponani in understanding ihe developmental processes during the sensitive period; it may be Ihe operation that leads ontogeneticatly-Limed. activity-dependent, connections to imprint appropriate codes i f the animal has experience, but inappropriate codes without experience. Coding Conditioning Cortex Representation Sensitive period
Inference
Helmhollz
Prior probability
Perception
THE RELATION OF PERCEPTION TO LEARNING
subsequent learning mechanisms to be versatile and reliable.
I decided to talk on this topic with some trepidation because Gerald knows his Helmholtz so much belter than I do and does not, I suspect, trust a non-German speaker to get him right. However Helmholtz expressed himself with unrivalled clarity and the ideas I shall propose are directly descended from his well-argued proposal that percepts represent unconscious inferences, so I cannot avoid bringing him in and musl risk Gerald's criticisms.
I shall assume that learning is based on what we perceive, and that cerebral cortex is where the representation we perceive is computed, even though neither assumption is 100% certain: McCormick, Lavond, Clark, Kettner, Rising and Thompson (1981) and Y e o , Hardiman and Glickstein (1985) have shown that conditioning of the nictitating membrane response in the rabbit occurs in the cerebellum, and I am sure that many other forms of learning can occur without the learner consciously perceiving the sensory stimulus that is learnt. Nevertheless our perceptions certainly provide much of the information from which we learn, and the cerebral cortex must create the representations used for this purpose. This is a sufficient basis for my argument, though one should be aware that other types of representation and learning do occur.
My argument is, briefly, that to understand perception one must view it as a prologue to learning. Acquiring new knowledge of the world is among the most important things our brains do for us, and for most people at this meeting it is probably the most interesting thing it does. So I shall try out on you the idea that perception is the process of preparing a representation of the current sensory scene in a form that enables
248
1562
Most people are familiar with the idea that representations vary in their completeness and accuracy, for these determine the results of tests of resolution, Weber fraction, and so forth. And the idea of a transformed representation, such as that provided by the coefficients of a Fourier transform, is now almost too familiar. ] want to examine a completely different question, namely "what properties should a representation have in order to make it suitable for use by subsequent learning mechanisms?" It is often taken for granted that any complete representation would do, but this is not the case and I shall start by considering what information is needed simply to establish that an association exists. A model of efficient learning should allow access to all this information, but associative network models based on Hebbian synapses can only access some of it. Because of this they do not account for the versatility of learning, and it seems to me lhat this is the aspect that is most remarkable in higher mammals. We have an astonishing store of knowledge about the associative structure of the world around us, and can recognise changes very readily; I think Helmholtz's unconscious inference results from automatic access and use of this store, and it is this that makes perception so effective for learning. These ideas also suggest a new hypolhesis about the puzzling defects that result from deprivation of experience during the sensitive period. THREE REQUIREMENTS FOR RELIABLY DETECTING PREDICTIVE ASSOCIATIONS
The formation of a conditioned reflex may be taken as a paradigm for the detection of a predictive association. To do this reliably the brain must determine that the conditional stimulus C precedes the unconditional stimulus U significantly more often than would be expected from the overall probabilities with which the two events have occurred in the past. This obviously requires knowledge of the occurrences of the sequence (U following C ) and some means of estimating how often this happens, but it also requires knowledge of the past occurrences of U and C and estimates of the rates they have occurred. One might question whether these three requirements must really be met, but I think it can be seen at once that predictive associations derived from less complete information would
be less reliable, and that an animal using an inefficient method would be at a disadvantage compared with one that made the correct computation; it would either detect fewer of the associations that were genuinely present, or it would attach importance to accidental associations, or it would make more errors of both kinds. Detecting predictive associations can bring enormous advantages, so it must be a very competitive business; of course no brain does the computation perfectly, but when considering the methods brains may use it is sensible to have in mind the requirements for the correct operation. Now consider the implications of these requirements: the site at which learning takes place must be influenced by the number of times C and U have each occurred previously, and also by the frequency of their joint occurrence in the correct temporal relation. Suppose these numbers are used to form estimates of the probabilities P(C). P(U). and P(U.C) where (U.C) symbolises a compound event, namely the joint occurrence of U and C in the correct temporal relationship: then the coincidence is significant and not random if N x P(U.C) significantly exceeds N x P(U) x P{C), N being the number of possible occurrences of (U.C) in the period under consideration. Questions naturally arise about the time scales over which these counts and estimates are made, for learning can occur in seconds, or may require years. It is probable that an efficient system would need to make estimates in parallel over several different times, and this is certainly a topic we need to know more about, but the logic of inferring that C predicts U requires some form of probability estimate so let us concentrate on this aspect. Figure 1 shows in outline how Hebb (1949) suggested that these requirements might be met by a nerve cell. He postulated that joint preand post-synaptic activity strengthens synapses, and the main merit of this suggestion is that it identifies the synapse between the input for the conditional stimulus and the output neuron as the site where the conjunction of U and C produces lasting effects. This is widely accepted, even though we do not yet know just what these lasting effects are, nor even whether they affect the presynaptic terminal, post-synaptic mechanisms, or both. The unconditional stimulus U and its frequency of occurrence could also produce lasting effects at this synapse because it is assumed that U by itself fires the post-synaptic
249
1563 Unconditional stimulus U
elements of the representation is an important part of perception. Always tlrsB pofll-Hynaptlc neuron
Conditional stimulus C
Hebb only postulated an increase in synaptic efficacy with joint pre- and post-synaptic activity as specified in the top line of the table at the bottom of Fig. 1. but a decrease of the transmission across the synapse when C or U often occur separately is needed if the mechanism is to identify predictive stimuli correctly. It is also needed to model the extinction of a conditional response when reinforcement is withheld, or when reinforcement occurs too often without the conditional stimulus.
Rules required tor weight changes Pre-synapilc Active Active Inactive Inactive
Pc-sl-ayneptlc Weight change Active Inactive Active I • i: • •:: I-. • -
Increase Decrease Decrease No change
Fig. I . Diagram showing Hebb's proposal that associations are detected and stored at the junction between synapses from afferent? carrying information about Ihe conditional stimulus on to a post-synaptic neuron. All Ihe required information is available al this site, and his proposal is now widely accepted.
neuron, and that the membrane at the site of the modifiable synapse is depolarised when this occurs; again we do not know what these lasting effects are. The occurrence of C is obviously signalled at the pre-synaptic terminal, and again it could produce lasting effects there, or on postsynaptic mechanisms But it is worth introducing immediately a rather different possibility for the way that P(C) is computed and signalled. Sensory messages often show habituation: they decrease in strength when a stimulus is repeated many times at short intervals. It is tempting to regard this as the means by which the prior probability of C is taken into account, strong signals with many impulses being given for rare events and weak signals with few impulses when the event signalled has happened frequently in the recent past. Although habituation occurs over time scales of seconds or minutes in the examples familiar from neurophysiologies! recording, one cannot exclude the possibility that much slower forms also occur, for they would be hard to observe in such experiments. One knows that sensory stimuli that have become familiar over much longer times are ineffective as conditional stimuli in learning experiments, which implies that P(C) can be estimated over these longer times, so it is tempting to suppose that this is caused by much slower habituation mechanisms that we do not yet know about physiologically. This would fit well the notion I shall develop shortly that the provision of estimates of P(C) for the
The most satisfactory development of the Hebb-type model is that of Sutton and Barto (1981), which includes suggested mechanisms for ensuring that C and U have the appropriate temporal relationship. Furthermore that paper showed the connection between this type of model, the learning theory of Rescorla and Wagner (1972]. and the Widrow and Hoff (1960), L . M . S . . or delta rule of adaptive networks. Here I want to go off on a different tack and consider how to make more versatile models based on the spirit of the Hebbian principle, for I think both learning theorists and those working on adaptive or associative networks have failed to consider some of the essentia! features that make this possible.
MAKING THE MODEL MORE VERSATILE In Fig. I it was assumed that the conditional signal C was already known and the problem was simply to find whether or not it predicted the unconditional stimulus U . The natural way of extending it would by by adding conditional inputs in parallel, as shown in Fig. 2; the post-synaptic neuron might then be conditioned to respond to many alternative stimuli such
Fig. 2. Possible extension of Hebb's scheme lo make learning more versatile. One difficulty is lo provide afferenls of word-like specificity; another is thai all ihese possible conditional stimuli would have to be known in advance in order lo be wired in.
250
1564 as "bell", "whistle", "tuning fork", " G # " , "flashing light", "foot-pinch" etc. But for learning in an advanced mammal it is totally unrealistic to assume prior knowledge of all possible conditioning stimuli, and it must be the versatility of our learning, based in my view on our perceptual capacities, that makes us preeminent in this way. We must therefore seek other ways of extending the model that will enable a large number of previously unknown conditional stimuli to be used, bearing in mind the requirement for estimates of P{C) for all such stimuli if learning is to be done efficiently. As well as the assumption that all possible conditional stimuli are known in advance there are other things wrong with Fig, 2. We have labelled the inputs to a neuron with words, but the fact that an appropriate word exists does not make it reasonable to postulate that the dog's brain has an input line with appropriate selectivity. Our facility with words has tricked us into making this step, but it isn't a simple one, and labelling the lines as in F i g . 2 just evades the problem. The model is inadequate in yet other ways, for it would not show any initial generalisation of conditioning from one stimulus to others sharing similar qualities, nor is there any obvious mechanism for subsequent narrowing of the class of effective stimuli. But although this extension of F i g . 1 does not provide what we are looking for, it does illustrate the enormous gulf that exists between simple cellular models of learning and the real thing. Perhaps it also points to the nub of the problem, namely how to create a representation whose elements would give the same versatility as is provided by labelling the input lines of Fig. 2 with words. George Boole (1854) thought that his logical functions composed with logical variables satisfactorily formalised the relation between a word and the set of sensations that it symbolised. It might be reasonable to suppose that the tnputs to the nerve cell of F i g . 2 correspond to individual items of sensation, so to approach the versatility of word-labelling we need to find a learning system that can use logical functions of its input lines; such logical functions could then be analogous to words. O f course no-one has ever claimed that there is a word for every possible logical function of a set of sensations, so we need not make the impossible demand that our learning model be capable of using every possible logical function of its input lines as a conditional stimulus, but it should be
B1
B)
ES t 4 91
11 17 I I
Fig, 3. An associative net (from Longuet-Higgins, Willshaw & Buneman, 1970). The input lines (A1-A8) run horizontally; the output lines (BI-B8) run vertically. « and (O) represent synapses that have or have not been turned on. Four associations have been recorded: A i , A2, A3 with B4, B6, B7; A2, A5, A8 with B l , B5, B7; A2, A4, A6 with B2, B3, B6; and A l . A3. A7 wilh B3. B4, B8. This is an efficient way of storing associations between input and output vectors, provided the probability of any input line being on is not too high, but it is not clear how the prior probabibties of input vectors could be taken account of. and this is necessary for efficient association formation. able to use a reasonable number of them: for instance it should be able to use as a conditional stimulus some at least of the possible conjunctions of inputs. This is precisely what is claimed for associative nets, so we must take them seriously as candidates for achieving versatility.
ASSOCIATIVE NETWORKS Figure 3 shows a neural network which associates inputs on the horizontal A lines with outputs on the vertical B lines. In discussing the problem of constructing " . . . an associative information store which can learn to associate very many pairs of conditional and unconditional stimuli . . . " Longuet-Higgins, Willshaw and Buneman (1970) claimed that this provided an "entirely satisfactory" solution. But there are snags. As we have seen, to do the job of detecting associations properly one should have access to the prior probabilities of the possible conditional stimuli, for without this one cannot tell whether an association is random or genuine; hence in this case the mechanism should have access to the prior probabilities of all the input vectors it can use, not just the probabilities of their components. It is perfectly reasonable to suppose that each synapse should have access to the prior probability of firing of its presynaptic input, but it is impossible to obtain the
251
IS6S
probability of a vector from the probabilities of its components unless there are statistically independent. The inescapable conclusion is that, if associations with conjunctions of inputs are to be discovered efficiently, these inputs must be statistically independent of each other under normal conditions of use. Note that independence does not ensure that the prior probabilities of vector inputs are made use of; it merely prevents this being impossible.
SHOUTING FOR ATTENTION
Thirty-one years ago Oliver Selfridge (1959) proposed a learning model called Pandemonium (Fig. 4). It had computational demons tuned to detect features of a stimulus, each of which shrieked with an intensity dependent on how closely the actual stimulus resembled the feature to which it was tuned; a set of cognitive demons combined these shrieks with weights adjusted according to the resemblance of the set of Note also that although ignorance of prior shrieks to a paradigm such as a letter of the probabilities makes efficient learning impossible, alphabet, and a decision demon then selected this does not imply that these networks cannot the loudest shrieking cognitive demon as the leam at all. The proofs that nets and perceptions recognised letter. I think the key element here is converge on the right solutions (e.g. Rosenblatt, that the demons shriek with a loudness that is 1959; Minsky & Papert, 1969; Longuet-Higgins supposed to indicate directly the importance of et al., 1970; Anderson, Silverstein, Ritz& Jones, their message for the next stage of processing; 1977) are not cast in doubt, but because they they not only signal, they also attract attention. cannot have access to prior probabilities they Since I am arguing that the prior probability of will be slow and error-prone compared with the current scene is one factor that determines efficient methods of association detection that its importance for association formation we make use of this additional information. need a Probabilistic Pandemonium in which the I think we need to separate the mechanism shrieks signal definite attributes of the stimulus that makes P(C) available for all usable con- as in Seifridge's model, but their loudness has ditional stimuli from the learning mechanism a probabilistic interpretation; they signal how that changes connectivity when there is evidence unexpected the occurrence of an attribute is on for a genuine association between C and U. The the evidence given by past history and the first mechanism corresponds roughly to percep- current presence of other attributes. When an tion, and I think it is this that is responsible for unconditional stimulus U arrives, it is the the versatile behaviour of higher mammals. The demons which have just shrieked loudest that second corresponds to the simplest forms of should be searched for possible conditional learning, and on this view there is nothing very stimuli, for their low prior probability means surprising in the suggestion that this is equally that the expected number of co-occurrences good in all subhuman vertebrates (Mcphail, with U is also low and there is therefore likely 1982). to be a genuine new causal factor behind the coincidence. If this distinction has anything to it the name Rosenblatt (1959) chose for his associative network, the perceptron, is singularly unfortuAnother idea relevant to this line of thought nate, for the insight it gives does not apply to is the noveltyfilter,as described, for instance, by perception but to learning. Kohonen (1984). Such a device learns the set of To summarise, learning an association is a usual images entering its input, and is able after definable statistical task and it is possible to a time to suppress those that have previously specify what is required to do it efficiently. The occurred. This idea has an initial appeal, but difficulty that arises is to make accessible the one does not want to suppress completely the prior probabilities of all input patterns that can non-novel inputs; what is needed is a filter that act as conditional stimuli; this seems to require makes novelty salient, but which continues to the elements of the representation should be transmit an image that is usable for ordinary independent of each other in the normal purposes. Seifridge's model used fixed feature environment, and this is not a problem that the filters, and there is plenty of scope for ontocurrent generation of associative networks genetically determined structure in the connectackle. Without it I think they do less than half tions of sensory pathways in addition to the the job; they may model learning, but they do plastic component for which there is evidence, not begin to model perception. There are, how- as described below (p. 1568). But the point of ever, some other early ideas that are relevant to interest here is the kind of outputs a model of perception should have in order to enable this problem.
252
1566 COGNITIVE DEMONS
Fig 4. Seifridge's Pandemonium (1959). as depicted by Lindsay and Norman (1977). This scheme for pattern recognition has many interesting adaptive aspects, but the feature of interest here is that each demon shrieks with loudness determined by Ihe fit of Ihe data lo rhe patlem.s Ihcy rcpreseni. so they signal the importance of their messages as well as the presence of the pattern. In a probabilistic pandemonium the shrieks would he proportional to — log P. where P is Ihe probability of occurrence of Ihe feature the demon detects. Normal sensory Inpuu
associative nets to be more versatile and efficient at learning; this does not depend on the preformed or plastic nature o f the connections that precede the output. Probabilistic
Response NirJeeelr
Reflponeea
Pandemonium
Figure 5 is a somewhat fanciful diagram illustrating the complexity o f Perception compared w i t h Learning. The perceptual outputs which act as inputs to the associative process are logically binary, that is they signal the presence o f a particular feature o f the i n p u t when they fire, but let them also be capable o f a graded discharge i n which the number o f impulses depends upon the p r o b a b i l i t y P o f the feature being present, based upon the past history o f excitation and also upon the other features that are present in the current input. F u r t h e r m o r e let us assume that this graded response approximates t o —log P, so the perceptual output neuron shrieks l o u d l y when its feature is unexpectedly present, softly when it is present but in circumstances such that this is not. surprising. Finally let us suppose that each o f these perceptual o u t p u t elements fires independently o f the others as long as the system is i n an environment to which it had adapted. I ' l l say a little more about h o w this might be achieved later, but first
Fig. 5. Diagram illustrating the complexity of perception and the simplicity of learning. The task of providing a representation whose elements are independent, and of signalling — log P for all of them, is much more difficult than Ihe simple associative task of learning. Note, however, thai the two are probably noi completely separable, for the results of Ihe associative process are likely to influence the demons of ihe probabilistic pandemonium through feedback.
253
1567 look at how desirable the output of such a (Barlow, 1989; Barlow & Foldiak, 1989). It is perceptual representation would be to act as the a plausible process physiologically because it could be achieved by having mutually inhibitory input to the next, associative, stage. The impulse frequency on any active percep- connections between the outputs whose strength tual output line gives —logP, where P is the increases when these ouputs are correlated, probability of the event or happening signalled along the lines earlier suggested by Wilson by that line; in other words it signals how (1975); thus it would do what ordinary lateral unexpected the feature or event is, given the past inhibition does, but instead of being fixed, the strengths of all the inhibitory interconnections history of its occurrence and the other sensory would be increased until the outputs were no events that are occurring. A s we have seen, this is just what is needed in order to assess whether longer correlated. Instead of such a negative the co-occurrence of this event with another feedback process, decorrelation could result event U is random or not. But this represen- from regulated positive feedback between outtation does more than provide this for each puts, the strength of which diminishes when the single output line: because the outputs are stat- feedback connections help to fire a neuron— istically independent, the probability of a com- anti-Hebbian positive feedback. Adaptation to bination of them is the product of their patterns, and contingent adaptation as in the individual probabilities, so the prior probability McCollough effect, are thought to result from of all the outputs together or of any subset of such mechanisms. ihem is simply the sum of the number of impulses each is firing. This idea of a probabilistic pandemonium is obviously informal and preliminary and is based on the fact that likelihoods can be multiplied or their logs added. The use of likelihoods to decide statistical problems goes back to Fisher (1925), and Kullback (1978) relates likelihoods to modern information theory and develops the methods further. I f one considers a plausible physiological realisation, the requirement that the demons shriek with loudness proportional to —log/ would not be too hard to approximate, for any mechanism of adaptation or habituation discounts frequently repeated events and thus leads to something like the desired "unexpectedness" signal. The requirement for independence is harder. 1
INDEPENDENCE Independence of the output signals implies that the presence of other outputs does not affect the significance (i.e. the prior probability) of any given one; positive or negative correlations among the sensory inputs thus have to be taken into account. I f this approximate independence of the representative elements is achieved, then the proposed representation can show how unexpected a signal is, given the other sensory stimuli that are present. Two methods of approximating independence can be suggested.
Decorrelation One method is to change the coordinate system used to represent sensory variables
Decorrelating networks of this sort could only handle small subsets of the sensory input, for it would be too vast a job to handle much of it at once. In addition, note that it only handles pairwise correlations, and would be insensitive to triples or larger groups of inputs that might be associated.
Minimum entropy coding This goes about the task of obtaining a set of independent representative elements by imposing two constraints on the code: first it must be reversible, and second the entropy calculated from the probabilities of the representative elements must be as low as possible (Barlow, 1989; Barlow, Kaushal & Mitchison, 1989). This entropy is always greater than the true entropy of the output calculated from the probabilities of all the output states, unless the representative elements are completely independent of each other; hence by finding a reversible code that reduces the entropy calculated from the representative elements one can diminish the mutual dependencies between these elements. It is a much more general method than decorrelation, but although it has the flavour of a problem suitable for a synthetic neural net it is hard to image a real neural mechanism for generating these codes.
Multi-stage recoding The above recoding methods are done in a single stage, though they could be repeated in the sort of way that the organisation of visual areas in the cortex suggests (Zeki, 1978; Barlow, 1981). By multi-stage recoding one might obtain
254
1568 representations which decorrelated associations between orientation and colour, or between stereo and motion depth cues; illusions to be mentioned below suggest that such mechanisms are present.
usually vanishes in a few minutes with normal use of the eyes. It is true that very long-lasting and powerful adaptation can produce effects persisting for days or more, but one cannot be sure this does not involve other mechanisms.
At first one might think that such recoding mechanisms would make it impossible to keep track of the prior probability of newly generated pattern elements: how would one know the prior probability of an element that received inputs, some inhibitory and some excitatory, from motion, colour and disparity selective units at earlier levels? This is not a serious problem, because once the pattern selectivity of an element has been established its prior probability can be determined afresh, simply by waiting to see how often it fires. Admittedly this would be inefficient, because it would not make use of experience ante-dating the establishment of the pattern selectivity, but it would avoid the need for working out the probability of what might be a very complicated logical function from the probabilities of its components. Thus it seems quite possible that the repeated application of a principle as simple as decorrelation would lead to a representation that was a good approximation to that postulated in the Probabilistic Pandemonium.
The other form of plasticity is that, long known to ophthalmologists, whose physiological basis was revealed by Hubel and Wiesel (1970) in their celebrated experiments on visual deprivation in kittens. In contrast with the first form of plasticity this requires a longer period to induce, it occurs mainly during a restricted sensitive period early in life, and the results persist indefinitely. Note in particular that plasticity in the sensitive period increases sensitivity to the inducing experience, whereas the other process actively desensitises the system to the adapting stimulus. Foldiak (1989, 1990) has developed a network of elements that receive inputs through simple Hebbian synapses and interconnect with each other through antiHebbian synapses. Possibly the rapid antiHebbian decor relating mechanism and the slower Hebbian mechanism of the sensitive period may work synergistically in the following manner. During the sensitive period pathways that are active become permanently connected, but the decorrelating mechanism (if it works the same during the sensitive period as in adult life) influences which pathways are active and can thereby determine the permanent pattern of connections that is established. T o caricature the suggested process, decorrelation ensures that different commonly occurring patterns of sensory stimulation each stimulate different cortical neurons, because if they failed to do so commonly occurring patterns would cause correlated outputs from two or more neurons. The Hebbian mechanism then ensures that the pattern of connections made to an activated neuron comes from the inputs that successfully activated it; thus the decorrelation mechanism helps to determine the pattern of connectivity that is permanently laid down. If the animal is deprived of experience during the sensitive period the Hebbian process presumably still operates, but there are no correlations in the spontaneous maintained activity of the inputs resulting from patterns in the outside world, so there is nothing for the decorrelating mechanism to work on. The consequence will be a somewhat disordered pattern of connectivity guided only by the ontogenetic mechanisms, without any adaptation to the
Figure 5 shows some of the tasks perception must achieve in order to give associative networks the versatility that has been claimed for them. O f course the diagram only poses the problem, but I hope it suggests to neuroscientists and psychophysicists the important role perception plays in giving higher mammals, especially humans, their intellectual pre-eminence, and I hope it reminds connect ion ists how much preprocessing of sensory input is required before their adaptive networks will function efficiently.
THE ROLE OF PLASTICITV AND THE SENSITIVE PERIOD I think there is good experimental evidence for two types of plasticity in the physiological mechanisms of perception. First there is the rapid adaptation or habituation that I have so far talked about, and which we think tends to make the representative elements uncorrected in the recently experienced environment (Barlow, 1989; Barlow & Foldiak, 1989). The illusions which provide the psychophysical evidence for such a process require adaptation times of the order of a minute or so to produce quite marked effects, and following such exposures the illusion
255
1569
statistical characteristics of the normal sensory input. This seems a very intriguing possibility that might reconcile the conflicting views about mechanisms, consequences, and purpose of the sensitive period (Movshon Sc. Van Sluyters, 1981), and it could turn out to be a model for the influence of experience on connectivity elsewhere in the brain.
Table 1 Inductive inference Major
All men are mortal
Minor
Perception
premiss
Stimulation of the temporal retina always results from luminous objects in the nasal field
premiss
Caius is a man
The temporal retina is being stimulated
Conclusion
UNCONSCIOUS INFERENCE
Therefore Caius is mortal
The link between the suggestions I have made and Helmholtz's views about unconscious inference or induction will be obvious to anyone familiar with his writings, but one cannot take it for granted, even with this audience, that everyone has read the Treatise on physiological optics (Helmholtz, 1925) from cover to cover, and Vol. iii is the part most likely to have been skipped. Warren and Warren (1968) have collected together Helmholtz's main writings on perception, and I think it is surprising that his views about unconscious inference are so rarely quoted or discussed. In his own writings one can perhaps detect a note of disappointment that the philosophers, whom he always took seriously even when he disagreed with them, seem to have dismissed the idea of unsconsious inductive inference for the apparently trivial reason that induction is a process necessarily conducted with the conscious use of words. Helmholtz argued, using examples from a wide field, that our percepts have a status analogous to the conclusions that are drawn by the process of inductive inference, the sole difference lying in the use of words to express major premiss, minor premiss, and conclusion in the latter case. I think his argument is correct, and it is a major inspiration for the views developed here. Thanks to Fisher (1925) and his followers the logic of induction is now understood very much better than in Helmholtz's day, and I have tried to use this understanding to draw conclusions about the operations that must necessarily occur in perception if it is to do what Helmholtz said it did, so let us examine what he said more closely. A very simple example of the analogy he draws is given in Table 1, which shows how straightforward syllogisms following the acceptance of the major premisses lead to the conclusion that Caius is mortal, or that there is a luminous object in the temporal visual field. Helmholtz used this example to explain why one refers the excitation to the nasal visualfieldeven
Therefore there is a luminous object in the nasal held
when it is actually caused by mechanical pressure on the temporal retina. However to "see" retinal stimulation in the position in the visual field that normally causes such excitation seems such a straightforward phenomenon that one is initially unwilling to attach much importance to it, let alone to call it an inference. But when you realise that a few days wearing inverting glasses changes the "always" condition in the major premiss, and correspondingly changes the position to which the excitation is referred, then it becomes difficult to call it anything else. Stratton (1897) published his account of the effects of wearing inverting glasses just after Helmholtz had died and long after he had formulated his arguments, but [ do not believe he made any clear reference to Helmholtz's theory, even though it is hard to conceive any more dramatic verification of its predictions. Of course it is still a mystery how the major premiss is changed as a result of experience, and how most of the complex inferential structure of perception is preserved, becoming adapted to the new conditions simply by this change in the accepted major premiss; perhaps we can dimly foresee a day when the hallowed subject of logic will be recognised as an idealisation of physiological processes that have evolved to serve a useful purpose. The extent to which our perceptions depend upon normal experience as a reference point does not need emphasising to this audience, but it comes as a continual surprise (even to me) to realise how the principle applies to minute associative details, as proved by a host of illusions which I can do no more than mention. Motion and tilt after-effects; the McCollough effect; micropsia with accomodative effort induced by minus lens or drugs; ditto with the convergence effort induced by prisms; the "toytown" effect induced by an unnaturally large
256
1570
range of disparities in a stereo scene; the reverse apparent motion when stereo parallax is present but the motion parallax expected from a movement is absent. All these illusions and many more can be explained as inferences that become false through the failure of a previously valid perceptual major premiss of the type shown in Table I, They show what an astonishingly deep knowledge of the normal patterns of associated activation our visual system possesses and automatically uses. The decorrelation model for approximating independence of the representative elements of perception would provide a means of storing this information in the form of the "antiHebbian" coefficients of interaction required to decorrelate. At the moment we know that cortical neurons show pronounced adaptation or habituation to patterned stimuli, but we cannot be sure that this interpretation of pattern adaptation is correct. The attempt to prove or disprove the hypothesis might bring some new life to cortical neurophysiology for it might bring out a common feature of cortical processings—the detection of new causal factors in the environment from the new associations they cause among cortical afferents. CONCLUSION
I think we have neglected the important role that perception must play in providing a representation that promotes the efficient learning of predictive associations. Ignoring this necessary preprocessing of the input is a serious defect both in current learning theory and in work on adaptive networks. But the argument I've given here probably does not go far enough, for it still leaves perception and learning as two processes almost as separate as they seem to be in current thinking. In reality it is pretty certain that what we learn influences what we perceive; in other words the demons of the probabilistic pandemonium are not only sensitive to the statistical structure of the input, but must also be influenced by fear of flogging and hopes of bribery administered on the basis of the results they have delivered. A composite, multistage, process, exploiting direct instruction as well as the statistical structure of the input, might begin to model the astonishingly versatile and useful representation that real perception gives us,
REFERENCES Anderson, J, A.. Silverstem, I . W., Ritz, S. A. & Jones, R. S. (1977). Distinctive features, categorical perception, and probability learning: Some applications of a neural model. Psychological Review, 84, 413-451. Barlow, H. B. (1981). Critical limiling factors in the design of the eye and visual cortex, (the Ferrier Lecture 1980). Proceedings
oj ihe Royal
Society,
London,
B 212,
1-34.
Barlow, H. B. (1989). A theory about the functional role and synaptic mechanisms of visual after-effects. In Blakemore, C. (Ed.). Vision: Coding and efficiency. Cambridge: Cambridge University Press. Barlow. H. B. & Foldiak. P. F. (1989). Adaptation and decorrelation in the cortex. In Durbin, R . Miall, C. & Mitchison, G. J. (Eds,), The computing neuron. Mass.: Addison-Wesley. Barlow, H. B., Kaushal, T. P. & Mitchison, G. J. (1989). Finding minimum entropy codes. Neural Compulation, l, 406-416. Boole, G. (1854). An investigation of the lam of thought. New York: Dover pubbcalions reprint. Fisher, R. A. (1925). Statistical methods for research workers. Edinburgh: Oliver & Boyd. Foldiak, P. F. (1989) Adaptive network Tor optimal linear feature extraction. International joint conference on neural networks 1989. Washington. D.C. Vol. I , pp. 401-405. Foldiak, P. F. (1990). Forming sparse representations by local anti-Hebbian learning. Biological Cybernetics (In press). Hebb, D. O. (1949). The organisation of behaviour. New York: Wiley. Helmholtz, H. von (1925). Treatise on physiological optics. Translated from ihe 3rd German ediiion (1910), Southall, J. P. C. (Ed.). Washington: Optical Society of America. Hubel, D. H. & Wiesel, T. (1970).The period or susceptibility to the physiological effects of unilateral eye closure in kittens. Journal of Physiology, London, 206. 419-436. Kohonen, T.
(1984). Self-organisation
and
associative
Berlin: Springer.
memory.
Kullback, S. (1978). Information
theory
Gloucester.. Mass.: Smith. Lindsay, P. H. & Norman, D. A. (1977). processing:
An introduction
and
Human
lo psychology
statistics. information
(2nd edn, p.
260).
New York: Academic Press. Longuet-Higgins, H.C., Willshaw, D. J. & Buneman, O. P (1970). Theories of associative recall. Quarterly Review of Biophysics,
3, 223-244.
McCormick, D. A., Lavond, D. G., Clark. G. A., Kettner, R. E., Rising, C. E. & Thompson, R. F (1981) The cngram found? Role of the cerebellum in classical conditioning of the nictitating membrane and eyelid response. Bulletin
of the Psychonomic
Society.
18, 103-105.
McPhail, E. (1982). Brain and intelligence Oxford: Oxford University Press.
in
vertebrates.
Minsky. M.&Paperl.S. (1969). Percepirons:
An
insroduction
Cambridge, Mass.: MIT press. Movshon, J. A. & Van Sluyters. R. C. (1981). Visual neural development. Annual Reviews by Psychology, 32,477-522. Rescorla, R. A. & Wagner, A. R. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and non-reinforcement. In Block. A. H. & Prokasy. W. F. (Eds.), Classical conditioning II: Current research and theory (pp. 64-99), New York: ApplelonCentury-Crorts,
257
to computational
geometry.
1571 Rosenblatt, F. (1959). Two theorems of statistical separability in the perceptron. In proceedings of a symposium
on
Ihe mechanisation
of
though!
processes
(pp. 421-456). London: Her Majesty's Stationary Office. Selfridge, O. G. (1959). Pandemonium: A paradigm for learning, in Proceedings isation
of
thoughi
of a symposium
processes
on the
mechan-
(pp. 3-16). London: Her
Majesty's Stationary Office. Straiten, G. (1987). Vision without inversion of the retinal image. Psychological Review, 4, 341-360 and 463-481. Suttoa. R. S. & Barto, A. G. (1981). Towards a modern theory of adaptive networks: Expectation and prediction. Psychological Review. 88. 135-170. Warren. R. M . & Warren, R. P (1968). Helmholtz on
258
Its physiology and development. New York: Wiley. Widrow, G. & Hoff, M . E. (1960). Adaptive switching circuits. Institute of Radio Engineers, western electronic show and convention, convention record. I960, Part 4, pp. 96-104. Wilson, H. R. (1975). A synaptic model for spatial frequency adaptation. Journal of Theoretical Biology. 50, 327-352. Yeo, C. H.. Hardimau, M . J. & Glicksttia, M . (1985). Classical conditioning of the nictitating membrane response of the rabbit (3 papers). Experimental Brain Research. 60, 87-98; 99-113; 114-125. Zeki. S. (1978). Functional specialisation in the visual cortex of the rhesus monkey. Nature, London, 274, 423-428. perception:
Neuroanatomy
T h e two articles in this section illustrate an approach to biocomputation, that lays higher emphasis on neuroanatomical data than is done in the other sections of this chapter. Their common strategy stems from the tenets that evolution has incorporated knowledge about the world into brains, and that observation of brain structure might thus reveal a good deal about its functional goal. T h e caveat is that historical and developmental constraints may prevent optimisation of adaptation. T h e article of Braitenberg (23) starts with an admirably compact definition of a brain, and surveys the lessons that can be drawn from anatomical observations about several invertebrate (fly's eye) and vertebrate (cerebral cortex and cerebellum) structures. T h e reader will notice that the author seems to favour Hebb's cell assemblies and Hopfield's associative memory networks in comparison to Barlow's single unit doctrine, in the case of the mammalian cerebral cortex. His ideas on the cerebellum receive further elaboration in paper (50) of Chapter 4. This brief survey of various brain structures S H O U L D B E demonstrative enough to convince the reader of the diversity that exists in biological nervous systems, and thus of the danger of sweeping statements about biological neural networks. Both articles in this section can be viewed as building on the introduction provided in Chapter 1, by the paper by Blakemore (paper (14)). Perhaps, for the part of the article of Mitchison (24) that concerns stripes and blobs, the reader will find it useful to give an early glance at the survey of Livingstone and Hubel (paper (43)) in Chapter 4. Mitchison attempts to derive, from a principle of economy in neuronal wiring, answers to questions like: W h y have multiple cortical areas? W h y have separation of grey and white matter? W h y have structures within areas? The two texts in this section are exercises in educated guesswork that we found as enjoyable to read as detective novels.
259
Reprinted with permission from Network, Vol. l.pp. 1-11, 1990 © 1 9 9 0 IOP Publishing Lid.
Reading the structure of brains Valentino Braitenberg Max-Planck-Institute for Biological Cybernetics, Spermannsirasse 38. 7400 Tubingen, Federal Republic of Germany Received 23 August 1989 Abstract. It is a fashionable philosophical tenet to consider Darwinian evolution as a process which incorporates knowledge into brains. We ask ourselves: can this knowledge about the world be recognised in the structure of brains? The present article gives a partial answer to this. Mechanisms of information handling and storage may well be related to the impressive major cortices of the vertebrate brain. Ihe cerebral and the cerebellar cortices. The structure of the first fits (he idea of an associative memory while the second strongly suggests computation of movement in terms of velocities. In some insect brains the mechanisms of visual perception can be related to detailed neuroanatomical structure, and one such network incorporates knowledge about the optics of a camera-lype eye. Another one provides ihe wiring that would be expected in a set of velocity deleclors using the principle of crosscorrelation of neighbouring inputs. Knowledge acquired during a lifetime is also laid down in brains but the search for Ihe 'engram' in the structure of brains has not yel been very successful.
I . Introduction 20
A brain is a complex spatiotemporal affair perhaps describable as the occurrence of 1 0 action potentials in 10'° neurons in the course of a human life. The structure of a brain as studied in neuroanatomy is a projection of this complex affair onto the spatial coordinates, with loss of the temporal aspects, except for the very low resolution analysis, which the study of ageing or of comparative anatomy may provide. How much of the original spatiotemporal complexity can be recovered in this projection? In other words, how can we make physiological sense out of the histological information? There is information from three sources embodied in brain structure: (i) inborn knowledge from evolution, (ii) engrams acquired during a lifetime and (iii) leftovers from the phase of construction during embryogenesis. The first two are legitimate sources of information about the complex spatiotemporal solution of Ihe problem of coping with the environment, while the third is related to this problem only in a very devious way and may introduce misleading clues. I will neglect embryotogical considerations and act as if every part of the adult brain could eventually be understood in terms of the computation for which it is responsible. 1 will ask three questions. First question. C a n we relate brain structures to specific mechanisms or abstract schemes of computation? Second question. C a n we recognise inborn ideas (in the ethotogists' sense of this term) in some aspects of the anatomy of the nervous system? Third question. C a n we see anatomical traces of acquired knowledge or ideas? 260
2 2. The cerebral corlex Let us consider first the most impressive, quantitatively prevailing organ in the (higher) vertebrate brain, the cerebral cortex (Braitenberg 1978a, b). A quantitative analysis of the mouse cortex, which has occupied us for the past few years, has revealed a few remarkable properties which can most probably be extended to the cortex of larger animals as well. In Golgi preparations it is easy to distinguish a number of different types of neurons in the cerebral cortex (figure 1) but on the basis of characteristics revealed by electron microscopy the variety can probably be reduced to three fundamental types, although each of these types may be subject to extensive morphological variation: pyramidal cells, smooth stellate cells and Martinotti cells. The important finding is that most (perhaps 80%) of the synapses in the cerebral cortex connect neurons of one type only—the pyramidal cells to each other. These synapses, judging from their microscopical appearance and from some indirect physiological evidence, are excitatory. The skeleton of the cortex (figure 2) is thus given by a vast system of elements of one kind richly connected by fibres mediating positive feedback. This is a situation which is not very compatible with the picture that many have in mind, i.e. hierarchies of neurons fed at the basis by sensory input, structured by systems of logical relations akin to
Pyromidnl
Figure L Three different types of neurons in the cerebral cortex: pyramidal cell (about 80% of all cortical neurons), with excitatory axon; stellate cell, almost certainly inhibitory; Martinotti cell wilh ascending axon, probably inhibitory. The thin axon can easily be distinguished, in all three types, from the other cell processes, the dendrites. It is loosely ramified in Ihe pyramidal cell, very densely ramified in the stellate cell and ascending in the Martinotti cell.
261
3 M
Figure 2. The skeleton cortex containing only pyramidal cells. M is the motor region where axons leave the cortex to reach other parts of the brain. S is a sensory region, with tactile, auditory or visual sensory fibres terminating in a middle layer of the cortex. O is the olfactory region, where the sensory input enters mainly the uppermost layers of the cortex. There are short-range (SR| and long-range |LR) connections between pyramidal cells. The fibres from region B are seen leaving the cortex and re-entering again in region C. (After Braitenberg 1977.|
expressions of the calculus of propositions and culminating in individual neurons which represent by their activity situations of arbitrary complexity in the sensory spaces. If this perceptron-like picture were taken literally, one would expect a more unidirectional array of fibres in the cerebral cortex oriented along the direction defined by the sensory and motor areas. Also, one would rather expect a near equal frequency of excitatory and inhibitory synapses corresponding to the near equal use of negated and non-negated terms in the logical expressions. Another fact in cortical neuroanatomy which makes the connectivity of cortical neurons very different from that of elements in conventional computers, and thus does not fit the logical theory of nerve nets very well, is the vast divergence (and consequently convergence) embodied in this system. Each cortical pyramidal cell in the mouse participates actively and passively in five to ten thousand synapses, and these are distributed among almost as many different pyramidal cells. Multiple connections between any pair of pyramidal cells, which would reduce the convergence factor, must be quite rare. We may ask which theoretical idea, among those proposed for the function of the cortex, fits the realistic picture most closely. We conclude that it is the idea of the cortex as an associative memory in which temporary correlation of activity in sets of neurons leads to their being connected by strong excitatory contacts so that each such set, a so-called cell assembly, represents within the brain a situation or a thing of the external world characterised by a certain internal consistency. The special bonus which the idea of these cell assemblies (Hebb 1949) boasts is the property of pattern completion intrinsic in the excitatory connections. Any sufficiently large part of a cell assembly may ignite the whole assembly and it is important to realise that the part that triggers the recall does not have to be specified in advance in this sort of memory, setting it apart from other types of associative memory. For those who do not have direct experience of neuroanatomy, it is important to stress another aspect of the cerebral cortical network. It is quite obvious to anybody who has had the opportunity to look at many individuals of one sort of neuron, e.g. pyramidal cells (figure 1), that their morphological characteristics are only given in outline, the
262
4 details being filled in by largely random processes of growth. The apical dendrite has a tendency to distribute itself in a roughly conical region on top of the neural cell body, and the basal dendrites in a spherical region centred around it, but no two pyramidal cells have quite superposable dendritic arbors. The same goes of course for their axonal ramifications. If the genetic specification of the shape of neurons is in terms of some kind of geometrical statistics, then it would be wise to cast their neuroanatomical description in the same language in order to render the essential information. This may not be so in all parts of the vertebrate brain and is certainly not so in some well known cases of insect brains (see a later section of this article) but most likely applies to a large extent to the cerebral cortex where only the sensory input fibres as well as some types of stellate cells are likely to be somewhat better defined in their target neurons. This essential randomness argues very strongly against the idea of a wiring as precise as that of electronic circuits, as some continue to suspect in the cerebral cortex, and provides a counter argument against the very pessimistic note which the impossibility of unravelling precise circuitry for an enormous number of neurons would imply.
3. The cerebellum Another piece of nerve tissue whose structural analysis leads directly to the suggestion of a mechanism is the cerebellar cortex (Braitenberg 1961, 1967a, 1987). This network is radically different from that in the cerebral cortex. First of all, in the cerebellum we find no evidence of loops of excitatory connections either in anatomy or in the rather well worked out synaptic physiology. The only intrinsic excitatory elements within the molecular layer of the cerebellar cortex are the granular cell-parallel fibre neurons which relay their signals exclusively to inhibitory interneurons and to the inhibitory output cells of the cerebellar cortex, the Purkinje cells. Whatever signals are fed into the excitatory neurons come back with a reversed sign. Thus positive feedback does not seem to be prominent there, unless we want to construct complicated schemes of double or quadruple reversal of signs. Consequently, if memory plays a role in the cerebellum, which it probably does, it is not likely to be of the associative memory kind just described. Another striking characteristic of the cerebellar cortex is its essentially onedimensional layout (figure 3). Although the macroscopical relations are a bit more involved due to the three-dimensional curvature of the cerebellar cortical sheet, within any small region transmission of signals in the parallel fibres (the elements which numerically vastly outpower all the others) happens in one direction only, that direction being defined by the transversal axis of the cortical sheet. The cerebellar cortex in its synaptic layout is not a truly two-dimensional structure, but the sum of a very large number of one-dimensional strips corresponding to what the physiologists call 'beams of parallel fibres'. This idea is even more convincing when one realises that the connections at right angles to these strips are inhibitory and therefore may be thought of as insulating neighbouring strips from each other by preventing the signals from spreading from one strip to another. There is more in the system of parallel fibres to make it quite unique (figure 4). The connectivity of most pieces of the nervous system, especially of other sheet-like structures called cortices, seems to be defined only in terms of neighbourhood relations, with the distance of the elements and consequently the length of the connecting fibres often being greatly distorted by the macroscopical bending and stretching of these structures.
263
5
| / i l
1 1
f
\ \
1
\
z
—
-J. '.
*
_
Inpul
Beam of parallel fibres
• _
_
£
i
'
\
1
1
/
A W
•
I\
\
S
J I i
•
Purkinje Stellate Figure 3. Surface view of the cerebellar corlex (very schematic). The horizontal lines are Ihe 'parallel fibres', contacting the elongated dendritic trees of Purlcinje cells (open) and stellate cells (batched shading). The black dots are the places where the inpul reaches the parallel fibres in order to be propagated in both directions (arrows). Purkinje ceils are the output elements. The stellate cells (including the so-called basket cells) inhibit Purkinje cells on both sides of the beam of parallel fibres on which they are located. The inhibitory regions are shown for one stellate cell on the right (broken circles).
264
6 However, in the cerebellum the folds of the cortex seem to respect the system of intrinsic coordinates of the tissue, being always parallel to the direction of the parallel fibres. It is as if distortion of the lengths of the parallel fibres in different sublayers of the molecular layer (such as would be produced by bends in the other direction) is carefully avoided. The most likely explanation for such metric invariance, as opposed to the merely topological invariance prevalent in other neuroanatomical structures, is that the times of signal transmission and reception in different points of the parallel-fibre beam play a crucial role in the cerebellum. This idea led to a reinterpretation of the role of the cerebellum in movement and posture which is still subject to experimental and physiological analysis.
4. Visual movement detectors in the fly The third example which I have in mind for the detection of mechanisms of computation in the nerve tissue is taken from the realm of the much more schematic insect nervous system. Vision in the fly has been analysed in behavioural experiments to an unusual level of precision by Reichardt (1970) and his school. At the basis of the impressive system which these studies reveal are the postulated individual elementary movement detectors situated between neighbouring individual channels of the compound eye (figure 5). They transform the parallel input of temporal fluctuation in the raster of input points into a representation of local movement all over the visual field. The localisation of these movement detectors in the tissue has not yet been precisely defined by electrophysiological means. However, neuroanatomy provides some very strong clues. At one level of the visual system of the fly (in the first ganglion, the lamina ganglionaris) the picture of the environment, cut up into small overlapping and inverted subregions by the optics of the compound eye, is reconstituted in its original order. We will come back to this point later, when talking about inborn concepts. Here another point is important (figure 6) (Braitenberg and Hauser-Holschuh 1972). Signals from each point in visual space are relayed through synapses of exactly the same sort and number to two different secondorder neurons which in turn relay them to the same column, but at two different levels, of the next visual ganglion (I neglect the existence of other second-order neurons for clarity). It is not clear at first sight why the nervous system does not in this case apply the usual trick for the reproduction of the same signal in different places, namely the branching of axons. If two different neurons are being used in this case, we must suppose that they have different transmission properties; one may be excitatory and the other inhibitory or the time constants of their responses may be different. The latter idea is
Figure S. Simplified movement detector according to Reichardt (1970) comprising sensors (S). low-pass filter (F| and multiplying unit ( M | , For a given pattern moving from right to left, the integral of the output of the multiplying unit is a measure of the velocity.
265
7
o b
b t
c d
de
Figure 6. Schematic view of the fly's eye. The upper part of the drawing shows how the geometrical optics of the lenses is compensated by the fibre pattern between the compound eye and the brain. The lines of sight corresponding to the three light-sensitive elements underneath each lens (in reality there are seven) are parallel to the optical axes of the corresponding lens (the central one) or of neighbouring lenses (the other two). Note that a distant object in a certain position (a, b, c, d, e) is therefore seen by three (in reality seven) lenses The fibres corresponding to each of these positions are brought together in one of the compartments a to e of the visual brain. The lower part of the picture shows how the input to each of these compartments is relayed to two different elements L , and L which, because of their different shape, may be hypothetical!y identified with the different filters in the input to Reichardt's movement detectors (figure 5}. :
indeed corroborated by the finding that the two second-order neurons, and L , are of different thicknesses and, moreover, of a thickness varying in a systematic way in different parts of tbe ganglion. It is very appealing to think that these fibres are the filters interposed between the input and the movement detectors, since in a scheme proposed by Reichardt (1970) each input line feeds different movement detectors, one through one sort of filter and one through the other, the difference being mainly in the time constants. This interpretation is all the more appealing since some experiments seem to show that the effectiveness of the movement detectors in guiding the fly's behaviour is different in different parts of the visual field, and the quantitative variation of the size of the L , and L fibres in different parts of the ganglion may be related to this. Of course, if it turns out that the two fibres produce signals of different signs, excitatory and inhibitory, again we will have a different model of the movement detector to fit the picture (Barlow and Levick 1965), but this interpretation would not provide an explanation for the gradients of size found in anatomy which make such good sense if we suppose the fibres to be filters with different time constants. 2
3
266
8
5. Band-pass niters for spatial frequency In some nerve nets the size and shape of the neuronal arborisations can immediately be interpreted in terms of known concepts of information technology. Take as an example the varying sizes of the dendritic trees of pyramidal cells contained in the visual cortex. Their diameters vary by a factor of about ten, the smaller cells being completely contained within the confines of the dendritic trees of the larger cells and all the cells very strongly overlapping with their neighbours {figure 7). Very likely, many of these neurons receive the 'specific input', the set of about one million fibres projecting point-to-point the image of the visual environment onto the plane of the cortex. Neglecting some claims that have been made over the years of much more complicated properties of the dendritic membrane, we may assume that each neuron receives as input some sort of average of the individual input points within its compass. Thus the smallest neurons represent the picture with its original fine grain, while the sets of progressively larger ones, up to the largest, constitute low-pass filters for spatial frequency, each with a bandwidth inversely proportional to the size of its elements. Thus, if we assume that sets of neurons of different sizes add their output separately, then we have a system capable of something akin to Fourier analysis. It is easy to convince oneself, studying one's own visual perception, that we are well able to use different sets of such filters when scrutinising the visual scene, both consciously and unconsciously, depending on the level of detail at which the picture makes most sense. Thus spatial frequency is in a way an inborn concept, at the same level that contours and the slope of contours are inborn concepts if we want to interpret the results of Hubel and Wiesel (1977) in these terms.
Figure 7. Pyramidal cells of greally varying size, some so large that only a small part of their dendritic arbor is shown, collect the same input in the visual area of the cortex. They may be understood as sets of different band-pass filters which provide different views of the visual input. (From Ramon y Cajal 1911.)
267
9 6. Bilateral symmetry: an inborn category of perception Another example evident in visual perception is more impressive since it shows the discrepancy between the simplicity of some anatomical structures and the high-order analysis which it performs. It is well known that we have a very acute perception of bilaterally symmetric forms, even if they are embodied in considerable visual noise (Barlow and Reeves 1978). The detection of such symmetrical patterns (figure 8) is best when the fixation is on the axis of symmetry so that the two symmetrical halves of the picture are projected onto symmetrical positions of the right and left brain. With the fixation to one side, acuteness of the perception falls off very rapidly. Apparently there are fibre systems in the brain which are able to compare and establish the identity or non-identity of symmetrical points of the visual field. Such fibre systems are of course well known; the 200 million fibres of the corpus callosum and several other commissural bundles may well serve this purpose. (The fact that the primary visual area does not participate in the commissural system, except for a small part of it, is only an apparent counter argument since the perception of form is ascribed more to the secondary and tertiary cortical representations of the visual field which have abundant callosal connections.) The philosophical counterpart of this very simple wiring scheme is surprisingly complex. In a natural environment, bilaterally symmetrical shapes in the visual field mostly mean one thing: another animal facing the observer, with friendly or unfriendly intention as the case may be. This is in any case a situation which justifies a
268
10
special detector, be it at the expense of 200 million long fibres reaching from one side of the brain to the other. In a way, the corpus callosum is an inborn idea of a sociological or at least interactive behavioural nature.
7. Geometrical optics embedded in a nerve set Another example of inborn knowledge is interesting because it provides a counterexample to the common opinion (to which I myself also subscribe when talking about the cerebral cortex) that most of the wiring in the brain is statistical. A n argument which is often adduced in support of this idea is that genetic information cannot possibly be sufficient to specify with absolute precision the origin and termination of each fibre in a piece of nerve tissue. In smaller brains, for example in insects, the absolute order of fibres is nevertheless occasionally obtained (Braitenberg 1967b). We have already mentioned the optics of the compound eye of the fly with its 6000 individual lenses which project 6000 partial images of the optical environment onto as many small retinas. The optics being of the photographic-camera kind, these partial images are inverted and they are partly overlapping (figure 6). The fibres that re-establish the original order of the points in visual space find their target neurons in the ganglion with absolute precision according to a scheme which can be quite easily computed on the basis of the geometrical optics of the individual lens camera ('ommatidium') and of the divergence angle between the axes of neighbouring ommatidia. Obviously the first thing to do for this system of fibres is to rotate the little bundle of fibres emerging from each small retina ('retinula') by 180° in order to compensate for the rotation of the image due to the lens and to fit the orientation of the small images to the overall orientation of the image in the eye, which, being made of separate radially arranged channels, is non-inverting. And this is indeed the case. Thus it is no exaggeration to say that the system of these bundles contains the knowledge of the geometrical optics of the camera eye—a piece of physics already incorporated into the building of the insect eye while in the pupal stage, even before it has ever been used. I know that some people object to the use of the term 'knowledge' in this connection but, stripping it of ideological and introspective connotations, it is appropriate.
8. Acquired concepts I think it is safe to say that nobody has ever seen a specific memory trace ('engram') in the brain. O n the other hand, it is very likely that many of the details which we observe in electron microscopy or even at the level of ordinary light microscopy have developed on the basis of experience. The trouble with finding an engram is that, from all we know about tbe function of the higher nervous centres, it would consist of distributed changes submerged in a near infinity of details which perhaps represent changes connected with other engrams. T o find the trace of an event in perception, we would have to know first exactly how the event is represented in the brain. The very brutal changes in the environment ('deprivation experiments') which sometimes do lead to visible alteration of the normal anatomy are not really comparable to the events that are remembered in ordinary life. Clearly, only such changes in the anatomy that continue throughout life are good candidates for the anatomical counterpart of engrams. It is difficult to separate the
269
LI processes of embryology which set up the brain from the later changes which subserve memory in those animals, like mice, rats, cats and humans, where the embryology is not completed at the time of birth. In some other animals, like guinea pigs, however, birth happens late in embryology and
the animal immediately after birth faces the environment with a completely
developed brain. I n such animals we may look for ongoing changes, possibly related to memory. Schiiz (1981) found in guinea pigs that the number of vesicles increases considerably in some synapses after birth and so does the size of dendritic spines. Both are good candidates for suspected anatomical changes underlying memory.
References Barlow H Band Levick W R 1965 The mechanism of directionally selective units inrabbil retina J. Physiol. 178 447-506 Barlow H B and Reeves B C 1978 The versatility and absolute efficiency of detecting mirror symmetry in random dot displays Vision Res. 19 783-93 Braitenberg V 1961 Functional interpretation of cerebellar histology Nature 190 539-40 Braitenberg V 1967a Is the cerebellar cortex a biological clock in the millisecond range? The Cerebellum {Progress in Brain Research 25) (Amsterdam: Elsevier) 1967b Patterns of projection in the visual system of thefly.1. Retina—lamina projectons Exp. Brain Res. 3 271-98 1977 On the texture of brains Neuroanatomy
for the Cybernetically
Minded
(Berlin: Springer)
1978a Cortical architectonics: general and areal Archilectonics of rhe Cerebral Cortex ed M A B Brazier and H Petsche (New York: Raven) pp 443-65 (1978b) Cell assemblies in the cerebral cortex Lectures Notes in Biomatkematics vol 21, ed R Heim and G Palm (Berlin: Springer) pp 171-88 1987 The cerebellum and the physics of movement: some speculations Cerebellum and Neuronal Plasticity ed M Glickstein, Ch Yeo and J Stein (New York: Plenum) pp 193-207 Braitenberg V and Hauser-Holschuh H 1972 Patterns of projection in the visual system of the fly. I I . Quantitative aspects of second-order neurons in relation lo models of movement perception Exp. Brain Res. 16 184-209 Hebb D O 1949 The Organisation of Behavior (New York: Wiley) Hubel D H and Wiesel T N 1977 Functional architecture of macaque monkey visual cortex Proc. R. Soc. B 198 1-59 Ramon yCajal S 1911 Histologie du system* nemeux deThomme el des vertebres(]972 Translated by L Azoulay, (Madrid: instituto Ramon y Cajal)) Reichardt W 1970 The insect eye as a model for analysis of uptake, transduction and processing of oplical data in the nervous system Physikertagung [Salzburg, 1969) (Stuttgart: Teubnerl Plenorvortrage 34 Schiiz A 1981 Prenatal maturation and postnatal changes in the guinea pig corlex: a histological sludy of a natural deprivation experiment. I t Postnatal changes (Pranatale Reifung und postnatale Veranderungen im Cortex des Meerschweinchens: Auswertung eines natiirlichen Deprivationsexperimenls. I I . Postnatale Veranderungen) J. Hirnforsch. 22 113-27
270
Reprinted wilh permission Ironi Thuttb in fieuni^ientei Vol. 15. No. t. pp. 122-126. 1992 © 199! Elsevici Science Publishers Ud. (UK!
Axonal trees and cortical architecture Graeme Mitchison Graeme Milcbison isIn modem computet design considerable care is taken to arrange would Ihe be to examine different spatial arrangements it the Physiologicalcomponents in such a way lhat winng is kept to a minimum. Certain of neurons in the cortex that keep all the connecLaboistoiy. teatores ot cortical strudute- the mappings, stnpes ind blobs within tions of each neuron unchanged (rearranging and Cambridge C82 KG.areas, andaieas themselves-are somewhat reminiscent of tbe layout stretching the axons if necessaryl. This would show UK oi computer components, and suggest thai the codex may alsowhether be the present configuration comes close to organized so as to economize on neuronal 'winng' One important achieving the lowest possible wiring volume. difference between the brain and a computet if lhat Ihe wiring mHowever, tbe this is an impossible program because so brain lakes tbetornot elaborate branched structures, namely atonal many configurations could be obtained in this way bees. In this article, it il argued lhat an assessment oltbe efficiency butof at least it is possible lo ask how altering certain cortical winng must take account oi the branching rules of these trees. key features of cortical anatomy affects the wiring
volume. A large part of ihe mammalian brain is 'wiring'. In the mouse cortex, about 30% of the volume of grey matter is taken up by axons, and dendrites occupy an approximately equal volume ; similar proportions are found in cat visual cortex . It would be reasonable to expect that the cortex has been organized so as to keep this wiring to a. minimum*" , since a wasteful arrangement of neural processes could significantly increase the volume of the cortex, and hence of the whole brain, and this would presumably carry a considerable selective penalty. Both axons and dendrites could be regarded as 'wiring', but axons are much longer and thinner than dendrites and have more elaborate patterns of connectivity. It seems reasonable, therefore, to think of them as the basic wiring of the cortex. Dendrites, which are densely studded wilh synapses, tan be regarded for the present purposes as extensions of the cell body, designed to receive as many synapses as possible without occupying too much volume. How can it be decided whether the cortex is efficiently wired up by its axons? One approach 1
5
7
271
Scrambling the maps withm areas would clearly be very deleterious. Even il this was done only within areas, the effect of shuffling connections randomly, so lhat they would have lo extend over an enlire area rather than being grouped in a neighbourhood of a few hundred microns, would clearly greatly increase the wiring volume. It is more rewarding lo ask what would happen if the order within areas was preserved but the areas were merged into one structure. As we shall see, this suggests some guiding principles for wiring design. It may also be more closely relaled to the process whereby new cortical areas evolve. Cortical areas Suppose that two cortical areas ", assumed for simplicity to be the same size, were merged inlo a single composite area. The cells from the two areas could be interleaved, as far as possible without destroying their original order, in such a way as to produce a sheet of the same thickness as the original areas (Fig. 1). The intermixing of the two types of 3-
neuron means that an axon has to spread further to make the same pattern of connections. Assuming that the volume of dendrites, cell bodies and other components remains unchanged, the total volume of cortex will increase by about 16% (see Bo« 1) If more than two areas were combined in this way the increase in cortical volume would be greater. For instance, if ten areas were lumped together the total volume would increase by a factor of two; for 100 areas this becomes a factor of ten.
Box 1. Relative volumes of different cortical layout? Suppose an area ha; linear dimensions s (in a surface view of the cortex), and thickness X. iMis the volume of axons and & the volume of other elements (such as dendrites, cell bodies and blood vessels), then equating volumes gives 14
2
s \ =
A+B
If n such areas (assumed equal in size) are merged lo give one area of linear dimension t. Ihe volume of each component wiN be increased by a factor n (assuming that axon arbors from different areas are simply superposed, rather than being more efficiently combined in some manner; see section on agonal branching patterns). Furthermore, the axons have lo extend further to reach their targets, by a scale factor t/s. Thus, the volume of the merged area Is given by
Against this increase of volume a possible gain can be set. Suppose neurons in one area send axons to another area, so that each neuron has arbors in both areas (fig. 2A). If these connections preserve the topography of the areas, then they can be merged in such a way that the two arbors have the same centre, so that no connecting axon is needed (Fig. 2B). This clearly reduces the wiring volume, and the question is whether this decrease could r*-n(f+s) balance or even outweigh the gain due to the greater spread of arbors within the combined area. Given n, and assuming axons constitute 30% of the A semi-empirical answer lo this can be given by cortex and that A = 0.3 and 8 = 0.7. these two noting that the total volume of the white matter, equations can be solved for r/s. The ratio of the volume which contains the association fibres (the connecof the merged area to that of the rr separate areas, which tions between areas), is considerably less than the is (f/s> /n, can then be calculated. This is given below for some values of n. expected increase due to merging areas. In humans, for example, where estimates suggest there may be at least 100 separate cortical areas", the volume volume of merged areas would increase by a factor of more than ten on volume of n separate areas merging, and the white matter occupies a volume 2 that is approximately equal to that of the grey 1.16 10 2.1 matter' In the mouse, where 26 cortical areas have 100 10.4 been identified' , merging would increase the volume by a factor of about 3.5; yet the white matter occupies only about 13% of the volume of the grey matter . So, for both species, the associBy using empirical estimates for relative volumes ation fibres take up much less volume than would be of white and grey matter the question of how the occupied by the expanded connections in a merged connections in the white matter are arranged has structure. been side-stepped. Are the areas laid out on the cortex in such a way that the volume of association fibres is about as low as it could be. given the pattern of connections between areas? This is an intriguing question, which the rapidly amassing anatomical data on primate cortex may soon allow us to answer J
3
4
13
15
Anon a! branching patterns The preceding calculations suggest that dividing the cortex into areas allows large savings in wiring volume. However, there is a loophole in the argument, for it ignores the possibility that two axonal arbors might be combined in some fashion that makes the volume of the combined arbor less than that of the two original arbors. It will usually be possible, for instance, to make an arbor with a larger number of short branches (Fig. 2C). Whether this will have a volume smaller than the two onginal arbors depends on the way the cross-sectional area of axons varies.
I + 11 Ffg. 1. Two cortical areas, I and U (in cross-section), are merged fl+lt), by intercalating neurons from the two areas in an ordered fashion. The anon of a neuron has to extend further in the merged area than in the separate areas to make the same set oi connections.
If the cross-sertional area is constant for all orders of branching, then it turns out that an efficiently wired arbor has a volume proportional to the square root of the number of synapses it makes'. This means that two arbors of equal size could be combined to make a single one of V2 times the volume instead of twice the volume. A simple
272
A
B
sectional area appears to be conserved at branches; that is, the sum of the areas of the two daughter branches equals that of the parent axon . It is not easy to determine if the same law applies to pyramidal ceil axons, because they are so much thinner than dendrites. There is the suggestive observation that the number of microtubules in an axon is conserved at a branch, though this does not necessarily imply that the total area is conserved. Suppose the assumption is made that the number of synapses made by an axon increases in some fashion as the cross-sectional area of that anon increases. Although there is little direct evidence for this, it is consistent with the general behaviour of axons; for example, the larger magnocellular axons from the geniculate make many more synapses than the thin parvocellular ones' . If it is also assumed Fig- 2. (A) Diagrammatic surface view of cortex showing Ihe axon of a neuron that area is preserved at axon branches, the hypothramifying in two areas (square outlines labelled I and II}. the two arbors being esis can be proposed that the number of synapses connected by an association fibre. In (B) Ihe two areas are merged (1+11) in supported by an axon is proportional to its crosssuch a way as to preserve topographic order. The two arbors can be combined (the arbor originally from area II is shown dotted}, and the association fibre sectional area. If correct, this hypothesis would dispensed with. Two ways of combining the arbors are shown in (C): by imply that the commonly observed tendency for groups of axons to run nearly parallel over some superposition, and by a more finely divided tree. distance (Fig. 3A) is not so neglectful of wiring economy as it might at first appear to be. Suppose calculation shows that, using this rule, the result of two axons make synapses in the same region (Fig. combining 100 areas would be to increase the 3B|. If they are replaced by a single axon that makes volume by a factor of about sii instead of ten. This is all their connections (Fig. 3Q, the cross-sectional still comfortably in excess of the factor of two that area of this axon would be the sum of the areas of results from including the volume of white matter. the two original axons, which implies that its volume It is clear, however, that axon diameters do not would roughly equal that of the two original axons. remain constant, but decrease at branches. In the This argument can be extended to show that, given case of the dendrites of pyramidal cells the cross- our hypothesis, superposing arbors is an efficient way of constructing a combined arbor, provided that the original arbors are themselves efficiently A constructed, in this case, therefore, the assumptions made in the preceding section can be justified. 4B 16
17
6
Stripes and blobs Within certain cortical areas there is a finer level of structure-the stripes* " - = ' , b l o b s " " and other kinds of patches. In the primary visual cortex, neurons often make clustered connections"'" (see Fig. 4A), where the distance between the clusters is approximately the period of ocular dominance stripes (Fig. 4B). This suggests that neurons make connections predominantly to others of the same ocular dominance type. A similar phenomenon is observed with blobs . Therefore, the stripe or blob systems have some of the properties of areas; namely, they allow neurons to form more localized connections at the cost of making a few long connections, which, in the case of stripes and blobs, are the long axons connecting the clusters. 26
Fig. 3. (A) An axon entering the visual cortex. Many of its branches run approximately parallel and make synapses in overlapping or neighbouring regions. This might seem uneconomical, because a single axon could supply the same region. However, with conservation of cross-sectional area the volume of the two axons depicted in (B) would be approximately equal to that of the single axon in (C), which replaced them. Numbers refer to cortical layers.
Abbreviation: WM, white matter. [Part (A) taken from Ref. 18.1
273
The wiring volume of stripes or blobs can be analysed by the same method used for areas. However, there is a significant difference, because the long connections between stripes run through the grey matter rather than occupying the white matter (as in the case of the association fibres). If the long fibres share the same space as the arbors of other neurons, they spread these arbors apart, thereby increasing their wiring volume. How large this effect is depends on the way axonal size changes at branches. If axon diameter is
(airly constant, it can be shown that, even when the axons mix with the arbors, a stripe pattern gives a lower wiring volume than a uniform mixture of neuron types. In (act, there is an optimum width to the stripes, which would lead to each neuron having four or five clusters on average . However, if crosssectional area is conserved, so that the long axons are thick relative to the clusters they supply, then the decrease in volume achieved by making more compact arbors within stripes is almost exactly cancelled by the volume of the long axons . This would not be true if the long axons were segregated from the arbors, within, for example, a separate layer of the grey matter. The stria of Gennari, which give the striate cortex its name, consist in part of myelinated axons running long distances , and may provide an example of the kind of structure that can exploit the wiring advantages of a patchy organization. Given such a structure, the volume gains for making stripes would be the same as for an equivalent number of areas. In the striate cortex, two types of ocular dominance stripe and four types of orientation patch might be distinguished, which would give a gain equivalent to eight areas, that is, a factor of about 1.9 in volume. 7
7
need to make specific patterns of connections, like the serial synapses made by basket cell axons along apical dendrites' , or the 'corkscrew' formations of chandelier cells''. As pointed out earlier, it is not necessarily inefficient for axons to run parallel over some distance, since a single axon replacing them might have to be proportionately thicker to support the larger number of connections it makes. However, there is no denying the wastefulness of some of the tricks axons get up to, like the backtracking seen in Fig. 3A. This does not mean that the idea of wiring economy should be abandoned, but rather that we should only look for robust gains and losses. If tidying up the axonal tree could halve its volume, then (assuming as usual that 30% of the cortex comprises axons) the gain in volume for the whole cortex would be about 16%. Against this, the gain in making 100 cortical areas is a factor of ten in total volume, and the gain in making a system of stripes could amount to a factor of two or so. In essence, the idea proposed here is that a patchy cortical organization allows an axon to make localized clusters of short connections, and thereby 8
Concluding remarks There seems little doubt that subdividing the cortex into areas confers a considerable advantage in wiring economy, at least when contrasted with the somewhat hypothetical alternative scheme where areas are merged into larger structures with as little possible destruction of their topographic order. It is less clear that the next level of patchiness, the stripes and blobs within areas, offers significant advantages in wiring economy. Much depends on the way the thickness of axons changes at branches. If the higher order branches are much thicker - in particular if the cross-sectional area is preserved at branches, as happens with some pyramidal cell dend rites - then the axons that link localized arbors in different stripes may have a large enough volume to cancel the gains from making more compact arbors.
lOOiim
,s
A further question arises here: do the linking axons occupy the same region of grey matter as the axonal arbors of the same, or other, neurons? This is pertinent because fibres not only occupy volume but also fill out the space that must be traversed by other axons in order to reach their targets. It is this 'filling out' that eventually weighs against the merging of areas and makes it more efficient to segregate the association fibres in the white matter. We can rephrase our question, therefore, and ask whether there is a substructure within the grey matter that serves as a kind of white matter' for linking stripes and blobs within areas The stria of Gennari in primary visual cortex may furnish an example of this. If economy of wiring is important for the cortex it would be reasonable to expect to see evidence of efficient design in individual axonal trees. In fact, axonal branches often reach their targets in a very roundabout way. This may be due in part to the
S
SB|4B 4A 3 SA
4 Fig. 4. (A) A neuron in the visual cortex, the axon of which shows a very marked set of clusters at a spacing that is approximately that ol ocular dominance stripes. (B) A representation of ocular dominance stripes, in a surface view of the cortex, showing how the clusters may relate to the stripe system. Numbers refer to cortical layers. Abbreviation; WM. white matter.
(Part (A) taken from Ref. 25.)
274
Acknowledgements I
Ibank A. Schiir and
P. Somogyi to< helpful suggaliom.
economizes on wiring. II is not difficult to think of other reasons why a patchy structure might be advantageous, although some of these can be discounted readily enough. For example, the gain in the propagation time of spikes obtained by having shorter connections is probably insignificant {a few hundred microseconds", with the most favourable geometry). A more attractive possibility is that patchiness might improve the efficiency of the search for synaptic targets by a growing axon, supposing that the patches contain a higher concentration of appropriate cell types. This cannot be dismissed lightly, especially as the existence of circuitous paths taken by many axons hints that making the correct connections may be a difficult task. However, the criteria arrived at in this article should help to distinguish this goal from economy of wiring, and this in turn may give some insight into the engineering problems that have constrained cortical evolution. Selected references 1 Braitenberg, V. and Schiiz, A. (1591) Anatomy of the Cortex. Spnneer-Verlag 2 Foh, £., Hang. H., Kbnig. U. and Rait. A. ( 1 9 7 3 ) Microsc Acta 75. 148-168 3 Cowey. A. (1979) Q. J. Exp. Psychol. 341, 1-17 4 Mitchison. Q. J. and Durbin, R. M. (1986) SAM ISoc. Ind. Appl. Math.) ) . Alg. Discos!. Methods 7. 571-583 5 DufUn, R. M. and Mitchison. C . J. (1590) Nature 343. 644-647 6 Nelson. M. E. and Bower, J. M. (1990) Trends Neurosci. 13. 403-408 7 Mitchison. C . J. 11991) Proc. P. Soc. London Sei 8 746. 151-158 8 Hubel, 0. H. and Wiesel. T. N. (1977) Proc. S. Soc. London Ser. B 1 9 8 . 1 - 5 9 9 Brodman, K. (1909) Vtrgleichende Lokatisahonsleh/e der Crosshimnnde in ihren Pnnzipien dargestellt suf Crund des
275
10 11 12
13
14 15
Zellenhaues tPnnciples ot comparative localization in the cerebral corlex presented as the basis of cytoarchitecture). Leipzig. Earth Zeki. S. M (1978) Nature 274. 473-428 Van Essen, D. C. (1985) In Cerebral Cortex (Vol. 3) (Peters. A. and Jones, E. G., eds), pp. 259-379, Plenum Press Crick. F. and Asanuma, C. (1987) in Parallel Ditfnbuted Processing (Vol. 2) (McClelland, J. L. and Rumelharl, O. E., eds), pp. 333-371, MIT Press Braitenberg, V. (1978) in flrcfl/recronics ot Cerebral Cortex (Brarier, M. A. B. and Petsche, H„ eds), pp. 443-465, Raven Press Caviness. V. S (1975) I, Comp. Neurol. 164, 747-264 Felleman. D . J . and Van Essen. D. C . (1991) Cereb. Cortex 1, 1-47
16 HiHman. D. E, (1979) in The Neurosciencei. Fourth Study Program (Sehmitt, F. O. and Wofden, F. O , eds). pp 477-498, MIT Press 17 Weiss, P. A. and Mayr, R. 11971) Acta Neuropathology (Suppl. 5), 19B-206 18 Freund. T. F.. Marlin. K. A. C . Soltesz. I.. Somogyi, P. and Whitteridge. D. (1989) J. Comp. Neurol. 289. 315-336 19 Livingstone. M. S. and Hubel. D, H. (1982) Proc. Natl Acad. Sci. USA 79, 6Q9B-6101 20 Tootell, R. 6. H., Silverman. M. S., DeValois. fl. L. and lacobs. C. H. 09B3) Science 220. 737-739 21 Slasdel. C . G. and Salama. C . (1986) Nature 321. 579-585 22 Humphrey. A. L. and Hendiickson. A. E (1980) Soc. Neurosci. Abstr 6. 315 23 Norton, J. C. and Hubel. D. H. (1981) Nature 292. 762-764 24 Gilbert. C . D. and Wiesel, T. PJ. (1983) /. Neurosci. 1. 1116-1133 25 Martin. K. A. C . and Whitteridge. D. (1984) J. Physiol. 353. 463-504 76 Livingstone, M. S. and Hubel, D. H. (1984) I Neurosci. 4, 2830-7835 27 Valverde. F. (1985) in Cerebral Cortex (Vol. 3) 1 Pelers, A. and Jones, F. C . eds). pp. 207-257, Plenum Press 38 Kisvarday, I. F„ Martin, K. A. C , Friedlander. M. J. and Somogyi. P. (1987) i Comp. Neural 260, 1-19 29 Somogyi. P. (1979) I. Physiol. 296. 1B-19 30 Wesman. 5. G . and Bennett. M V. L (1972) Nature 238. 217-219
2d Aspects of Biocomputation
T h e unity of this section is not manifest a priori] it is to be found in the diversity, i.e. in the complementarity of the approaches presented by the three authors, who are experts respectively in immunology, psychophysics and neuropsychology. Within a human body reside two conspicuously intelligent systems, with a capacity for learning: the nervous system and the immune system. T h e article by Niels Jerne (25) exposes elegantly some similarities, and also differences, between these two smart systems. (Note in passing how a capacity for learning appears naturally as an important criterion for intelligence; the liver does a great processing job, but it does not hold a high reputation as an intelligent organ). Jerne's section on "cellular dynamics in immunology" is a bit technical, but the following one on "instruction versus selection in learning" addresses issues that are central in biocomputation. One fact that puts brains (specially human brains) apart, is that neuronal cells cease to reproduce early in life. Thus the notion of reproductive success, which is one main ingredient of Darwinian evolution theory, is missing in brain life. Intercell connection strengths change, not cell population sizes. T h i s is one sufficient reason to take doctrines of "neural Darwinism", as advocated by some in analogy with evolution and immunology, with a grain of salt. Jerne's thoughtful analysis of the selection-versus-instruction debate should also alert the reader against statements that describe perception and learning as "resonance" between external objects and internal "prerepresentations". The whole point in the selection-versus-instruction debate, as explained by Jerne, lies in the accurate assignment of the selective level — there is always one selective level, and it need not be the most interesting one. The review of Bela Julesz (article (26)) on early vision and focal attention was specially written for physicists who are novices in brain research; thus, it provides appropriate historical perspectives, and a commented list of references. The author's invention of random dot stereograms, used for the study of binocular vision, has opened a vast new field for quantitative
276
psychological investigations, and it has led to numerous fundamental and applied results. T h e study of stereopsis, followed by a discussion of preattentive texture discrimination and of the serial process of focal attention, provides ground for a wealth of casual side remarks that touch on almost all the issues evoked in the subsequent Chapters 4 and 5. T h e text of Larry Weiskrantz (paper (27)) on neuropsychology and the nature of consciousness, provides a brilliant and entertaining survey of some of the confusion that continues to surround the word "consciousness". He succeeds, though, in clarifying some core components of this compound concept, with a presentation of amnesia and blindsight syndromes in humans, and of several pertinent animal observations. His "monitoring" theory of consciousness is germane to Hebb's views, as exposed in the last article (68) of this volume.
277
Reprinted with penrussiori from The Neuwicicnets: A Study Program,
pp. 200-205, 1967 © 1967 Rockefeller University Press
Antibodies and Learning: Selection versus Instruction NIELS KAJ J E R N E
U N T I L LESS T H A N ten years, ago, there was an almost (man*
imous consensus among immunologists that antibody formation was equivalent to a learning process in which the antigen played an instructive role. T h e main basis for this belief was that the number of different antibody specificities, or the number of different antibody molecules that one animal can produce, is so large that it would be impossible for a cell nucleus to accommodate genes for this entire range of potentialities for protein synthesis. The number of different antigens is immense. Every species o f animal, for example, must have several species-specific antigens. Against any one of these millions of antigens the immune system of one individual animal can produce a specific antibody. Therefore, the argument went, the number of different antibodies that an animal can produce
must be virtually unlimited. Furthermore, the work o f Landstcjncr and his school had shown that an animal produces antibodies even against artificially synthesized substances (haptens) that were made in a chemical laboratory and had never before existed in the world. The immune system of an animal could not possibly have anticipated the arrival of such antigens and must therefore have been "instructed" by the antigen itself in the formation of antibody. The instructive mechanism proposed was that the antigen, after having entered a competent cell, guides rhe tertiary folding of polypeptide chains into globulin molecules, thereby imposing upon those molecules a conformation complementary to a surface region on the antigen. " 1
1
1
Instruction versus selection in antibodyformation Paul Ehrlicli Institute, University of Frankfur,. Frankfurt am Mam, West Germany F U E L S K A; jBJitJE
In contrast to this view, a selection mechanism was proposed, based on a logical argument concerning "recognition. The precision of recognition by the immune system
278
201 Can be illustrated by examining ihe antigenic properties of the "constant" parr of the kappa light chain of human immunoglobulin .This constant portion comprises 107 amino acids (numbered 108 to 214) at the carboxyl end of the light chain, which have been found to be identical in sequence in all individual cases so far examined, except for amino acid number 191. In some individuals, this is valine; in others it is leucine. This difference is known as an allotypic difference. Those individuals that have valine at the 191 position belong to allotype lnv b . whereas those that have leucine at this position belong to allotype lnv a*. This allotypic difference was detected by immunological methods involving the formation of allotype Specific antibodies.* The immune system is thus capable of recognizing the replacement of one amino acid within a long sequence of amino acids of a protein molecule. It is a characteristic fcarurc of the immune system rhat an animal does not normally appear to produce antibodies against its own circulating antigens, which do elicit antibody formarion when injected into a different animal. How docs the animal recognise that an antigen arriving in its tissues is. in fact, an antigen against which antibody should be produced and not one of its own antigens to which no response is desired? +
The immune system, then, must not let itself be stimulated to produce antibodies before having recognized that the anrigen with which it is confronted differs from its own antigens. In order to recognize its own antigens, which differ among themselves in innumerable ways, the animal would have to possess a large set of self-recognizing molecules, and it would not be able to decide that a given antigen is its own berorc this entire set had been applied—a scrutiny, moreover, that would have to be interminably repeated. It would therefore seem impossible for the animal to recognize its own antigens. The recognizing agent must recognize foreign antigens, and the obvious molecules to accomplish this task are antibody molecules. It follows that an animal cannot be stimulated to make specific antibodies, unless it has already made antibodies of this specificity before the antigen arrives. It can thus be concluded that antibody formation is a selective process and that instructive theories of antibody formarion are wrong. Many immunologists were not convinced by the logical argument presented above, and only because of direct experimental evidence, accumulated during the last three or four years, have instructive theories of antibody formarion finally been abandoned. Antibody molecules have been shown io consist of two identical heavy polypeptide chains and two identical light polypeptide chains. It has also been shown that these polypeptide chains are assembled on ribosomes and that the specificity of an antibody 7-10
11
l?
molecule is determined by the primary structure of its polypeptide chains. This leaves no room for instructive action by the antigen. Furthermore, it has been demonstrated rhat certain antibody-producing cells, which turn our more than 1000 antibody molecules per second, contain no antigen, The experimental methods would have detected the antigen if mote than ten molecules of antigen had been present per cell. At one ribosomic site, a light or heavy polypeptide chain cannot be synthesized in less than 15 seconds. Therefore more than 100,000 ribosomcs in an anribody-produeing cell must simultaneously be able to turn out specific chains in the absence of antigens. Although it is thus clear that the antigen plays a selective and amplifying role, we do not yet know by what mechanism the selective stimulus is transmitted. All we know with certainty is rhar, one day and later after having injected an antigen into an animal, we can find, in its spleen or lymph nodes, cells that are both multiplying and producing specific antibody. The simplest assumption would seem to be that an animal contains, among its population of 10* to 10 lymphocytes (depending on species and age), a very large number of subpopulations, each of which is capable of being stimulated by certain antigens to grow, divide, and produce antibody-secreting cells among their offspring {perhaps because on their surface the cells display the antibodies they can synthesize). 11
11
Cellular dynamics in immunology This picture can be illustrated by experiments that make use of an agar plaque method for counting, in a cell suspension obtained from a mouse spleen, the number of cells secreting a certain antibody. A certain amount of such a suspension—say IO mouse spleen cells, as well as 4 X 10* sheep red blood cells(SRC)—is added to 2 milliliters of fluid 0.7 per cent agar ar 45° C. The mixture is immediately poured into a pctri dish, where it solidifies into a Layer less than 1 mm thick. Each spleen cell is now surrounded by many SRC in a fixed position in the agar layer. I f a spleen cell, during incubation of the petri dish at 37 C, secretes antibody molecules directed against an antigen of the SRC surface, the molecules diffuse into the agar and become fixed to the SRC in the immediate surroundings. Red blood cells, to the surface of which an antibody has attached, are said to be sensitized. Such sensitized cells will lyse in the presence of a scrum factor called complement. By flooding the petri dish with complement after onehour incubation, the sensitized SRC will lyse and lose their hemoglobin. Thus, around each mouse spleen cell that secretes hemolytic antibodies against sheep red blood cells, a pale plaque, visible to the naked eye, appears. Microscopic observation reveals the antibody-producing
279
6
202 lymphocyte, Oi plasma cell, in the precise center of each plaque. The spleen of an untreated eight-weeks-old inbred mouse contains about BO plaque-forming cells (PFC) among a total of about 1.5 X 10 spleen cells. These SO PFC produce antibody against sheep red blood cells, although the mice have never experienced sheep antigen. As always i n immunological observations, there is a great variation among individual animals. The normal level of 80 PFC is an average. Among a group of 50 apparendy identical mice, the normal level could range from, say, 10 to 500 PFC. 16
17
s
We now give each of a few hundred mice one injection of 4 x 10 SRC into a tail vein. Every day we sacrifice 20 mice and determine the total number of plaque-forming cells in each spleen. Twenty-four hours after the antigen injection, the number of PFC Starts to rise above the n o r mal level, proceeding exponentially to reach an average of 10 PFC per spleen at four days, after which there is a rapid decline. T
6
The following t w o observations, appear to support the assumption rhat the exponential rise in PFC between day one and day four reflects cell multiplication. If, on day two, an animal is given one microgram of colcimid per gram body weight and is killed three hours later, about 20 per cent of the PFC in its spleen is found to have been arrested in metaphase of mitosis. If, on day three, the spleen cells are suspended for 30 minutes in vitro in a medium containing tritiated thymidine, about 55 per cent of the PFC can be shown by autoradiography to have synthesized DNA, whereas the remainder have not. The rale of appearance of PFC after an intravenous dose of 4 X 10 SRC corresponds to a cell-doubling rime of seven hours. Although smaller doses of SRC evoke smaller responses, it is not possible to obtain a larger response than that elicited by 4 X 10 SRC, even i f the dose is increased to 4 X 10* or 4 X 10* SRC. Each dose within this hundtedfold range produces the same maximum response, indicating that all cells capable of being stimulated by sheep red blood cell antigen are maximally engaged. T
T
The experiments described above can be repeated with similar results i f rabbit ted blood cells are used as antigen. The antibodies produced by mice against SRC and tabbit red blood cells do not cross-react. Also, the PFC that arise after a mouse has been injected with SRC do not form plaques in agar with rabbit red blood cells, and vice versa. Furthermore, the following experiment shows that the class of cells in the mouse spleen initially stimulated by an SRC injection is different from the class of cells initially stimulated by a rabbit red blood cell injection. The number o f PFC against rabbit red blood cells appearing after a single injection of 10* rabbit red blood cells is the same.
whether or not 10* SRC are injected simultaneously. The two types of antigen clearly do not compete for rhe same target. In summing up, we can conclude that the mouse spleen possesses, among its more than 10* cells, small classes of cells that can be stimulated to grow and divide by particular antigens, and that anribody-secreting cells arise from these subclasses by cellular proliferation. We can now try to estimate the number of cells in a mouse spleen that belong to the class char can respond to SRC antigen. We have reasons to believe that these cells are nor the PFC that fotm the normal level in nonstimulared animals. First, the normal level PFC are mostly plasma cells, whereas the cells that respond to a primary antigen stimulus are probably small lymphocytes. Second, the magnitude of rhe response of individual mice appears unrelated to their normal level of PFC. We therefore do not believe that the normal PFC belong to the class of cells that can respond to a primary stimulus of SRC antigen, nor that they become the ancestors of the PFC arising after a stimulus. The exponential curve describing the appearance of PFC extrapolates below the notmal level of PFC at less than 24 hours after the sheep red blood cell stimulus, suggesting rhat the average size of the class of responding cells might be less than 80. Other experiment appear ro leave little doubt, however, that the size of the class of initially responding cells is of the order of several thousand. These experiments involve (1) the exponential decay of this class of cells after increasing exposure of nonstimulated mice to X-radiarion, and (2) the transfer of a small fraction of the cells from a normal mouse spleen ro the spleen of a mouse that has been rendered immunologically incompetent by X-irradiation. Thus, only a small fraction of the immediate descendants of the initially responding cells arc PFC, i.e., secrete an antibody that can cause lysis of sheep red blood cells. The initially responding cells and most of their immediate descendants might then display or secrete antibodies of a degree of specificity that enables the antigen to stimulate these cells to divide, but which is not serologically recognizable- This might also explain the finding that certain serologically unrelated antigens, such as different Salmonella flagellar antigens and different protein subunits of lactodehydrogenase, can induce immunological tolerance with respect to each other. In both of these cases, tolerance may be due to the removal of a class of initially responsive cells that display cross-specific antibodies not detected by serological methods. The picture that emerges for the initial stages of antibody formation is that the antigen first selects a class of initially responding cells. These are stimulated to dif-
280
J3
l,,!l
11
H
203 ferentiatc and to divide. For each division they may require a new antigenic stimulus. All daughter cells do not necessarily produce the same antibody, and the antigen preferentially stimulates those descendant cells thai produce rhe antibody best fitting the antigen. This brings me to another relevant immunological phenomenon, rhe increase in "avidity" of the antibodies produced during the lime following an antigenic stimulus, in the early days of immunology, the term avidity was introduced to express rhe degree of firmness of the bond formed between serum antibody and antigen in vitro. I f the antigen dissociated easily from the antigen-antibody complex on ddution, the antibody was said to be of low avidity. Avidity is thus a measure of the goodness of fit o f antibody toward the antigen. It was shown that the antibody present in ammal scrum is a heterogeneous population of molecules varying widely in avidity, and that the average avidity of the antibody present in an animal increases after repeated antigenic stimulation, or even with time after a single primary stimulus, *Recent studies have confirmed that antibody molecules produced by one animal shortly after a single primary antigenic stimulus have a lower avetage association constant with tespcel to the antigen than later, and that this applies to a single class of antibody (IgG) with respect to a single antigenic determinant.** This indicates that a further selection of cells producing better-fitting antibody takes place among the descendants of the cells first stimulated after the initial antigenic stimulus. Avidity does not increase as quickly after a large dose of antigen as after a small dose. As might be expected, a large dose of antigen is less selective, and in extrapolation we might relate tolerance following excessive doses of antigen to the absence of progressive selection. The immunological tolerance observed after repeated minimal doses of antigen might be caused by stimulation of the entire class of available responding cells, followed by a decay of their descendants because of the absence of antigen needed for further stimulation. Trie assumption underlying the above discussion, that the descendants of the initially stimulated cells require further antigenic stimulation in order to continue to multiply, is supported by the finding that the exponential rate at which PFC appear in the spleens of mice after one i n travenous injection of SRC decteases with decreasing antigen doses. Thus the doubling rimes of PFC per spleen after one dose of 4 X 10', 4 X 10*, 4 X 10 , and 4 X 10* SRC are 7, 9, 21, and 36 hours, respectively. The simplest explanation of this remarkable fact is that the cells require, and must therefore wait for, a new antigenic stimulus for each division. In the picture developed above, a continuing selective 5
16
s
role has thus been assigned to the antigen. Among the general population of antigen-responsive cells, a particular antigen first selects the small class of cells that can respond to the primary presence of that antigen. These cells are stimulated to grow and divide. A secondary selection by the antigen then takes place among the differentiating descendants of these cells, resulting in the production of increasingly better-fitting antibodies. Instruction versus selection in learning In accordance with this general picture of rhe selective role of antigen in antibody formation, the view that antigen acts instructively at the intracellular level has been abandoned. It may be useful, therefore, to examine in a broader biological context whether there exist situations in which instructive mechanisms operate or to which the term "instruction" is applicable, k would seem rhat an answer to this question tequires a specification of rhe organizational level at which a process is described. Thus, although the mechanism by which an antigen brings about antibody formation in the tissues of a mouse must be purely selective, the antigen does not select a mouse. When viewing the situation at the level of the entire mouse, we may still say that the antigen "instructs" the animal to produce an adequate antibody. Similar reasoning can be applied to examples from other areas of biology. For instance, in the case of the selection by si re pio my cin of streptomycin-resistant mutants among a population of bacteria, it is clear that the streptomycin molecules did not cause these mutants to arise. They were already present before the streptomycin arrived; no instructive role can therefore be assigned to streptomycin. On the level of the entire bacterial culture, however, we may still say that streptomycin instructs the transition to streptomycin resistance. Let us consider a more complicated example of Darwinian selection, A large population of brown moths spend a major part of their time sitting on a factory wall of the same color. These moths are rhe prey of certain birds. Now the wall is repainted white. One or two years later we observe that the moths sitting on the wall ate likewise white. In this case, the signal that entered into the system, t.c,, the color change, was not even received by the moths, but by the birds. The mechanism by which the color change in the moths came about was obviously selective, in that moths of lighter color were already present among the original population before the signal arrived- Again we might say. however. On rhe level of the entire system, that the signal "instructed" the population of moths to mimic the color change, A clear example of an instructive process would be the role of messenger RNA in protein synthesis. The mes-
281
204 scngcr RNA molecules arriving in their ribosomal habitat do not select already existing protein molecules and may therefore be said, at the organizational level of protein, to play an instructive role. The messenger RNA docs recognize and select, however, already available su bun its, namely, species of amino acid-charged rtansfer RNA- At this lower level, therefore, the process is a selective one. J will finally turn to the question of the analogies between the immune system and the centra] nervous system. Both systems have a history that develops during the lifetime of the individual. Each antigen that mates its appearance irreversibly changes the immune system. In the same way, the state of the central nervous system reflects the experience of the individual. The immune system appears to be learning by responding to antigens entering from rhe outside world. The central nervous sysrem also appears ro learn in response to sensory signals. Like the central nervous system, the immune system appears lo have a memory that enables it to benefit from previous experience. It produces more and better antibodies i f the antigen enters a second time, or repeatedly. The experience gathered by the immune system of an individual cannot be transferred to its progeny. As with the central nervous system, each newborn must start, so to speak, from scratch. In the remaining, specularive part of this paper, I shall try ro make the most of these analogies. Let us consider the kappa light chains of the antibodies of mice and man. Each of these light chains consists of a "constant" sequence of 107 amino acids and a "variable" sequence of 107 amino acids. The Constant part of human light chains is identical in all individuals and m all antibodies they produce. It differs, however, from ihe constant part of mouse light chains by some 40 amino arid substitutions that have obviously arisen by mutation during phylogeny. The variable part of human light chains, on the contrary, differs berween different antibody molecules of one individual, and the differences are similar in nature to the differences between mouse and man in the constant part. This is reminiscent of the old saying that ontogeny mimics phylogeny: phylogenic differences between species in the constant part of the light chain are mimicked by the ontogenic plasticity of the variable part." 30
Similarly, in the central nervous system, instincts arc fixed in one species, but each individual (particularly man) has also a plasticity in learning capacity, which mimics the total of all phylogenically developed instincts of different species. In the immune system, rhe constant part of the light chain is obviously laid down in rhe DNA of the zygote, and it is equally cleat that there is DNA in the
zygote that represents the variable part of the light chain, although, ontogcnically, this DNA may exhibit an immense plasticity. In the central nervous system, instincts are also obviously encoded in the zygote, most probably in the DNA. But if DNA acrs only through transcription into RNA and translation into protein, and if the phenotypic expression of instincts is based on particular a rrangemenrs of neuronal synapses, then DNA through RNA and protein must govern the synaptic nerwork in the central nervous system. Analogous to the utilization of the diversity of the variable part of the antibody light chain in the immune system, it would seem probable to me that, in the central nervous system, learning from experience is based on a diversity in certain parts of the DNA, or to plasticity of its translation into protein, which then controls the effective synaptic network underlying the learning process. I would, therefore, find it surprising if DNA were not involved in learning, and envisage that the production by a neuronal cell of certain ptoteins, which I might call "synap[obodies,' would permit thaE cell to enhance or depress certain of its synapses, ot to develop othets. Pursuing these analogies even further, we might now ask whether one can distinguish berween instructive and selective theories of learning in the central nervous system. Looking back into die history of biology, it appears that wherever a phenomenon resembles learning, an instructive theory was first proposed to account for the underlying mechanisms. In every case, this was later replaced by a selective theory. Thus the species were thought to have developed by learning or by adaptation of individuals to the environment until Darwin showed rhis to have been a selective process. Resistance of bacteria to antibacterial agents was thought to be acquired by adaptation, until Luria and Delbrilck showed the mechanism to be a selective one, Adaptive enzymes were shown by Monod and his school to be inducible enzymes arising through the selection of pre-existing genes.^ Finally, antibody formaEion that was thought to be based on insttuction by Ehe antigen is now found Eo result from the selection of afteady existing patterns. It ihus remains to be asked i f learning by the central nervous system might not also be a selective process; i.e., perhaps learning is not learning either. Several philosophers, of coutse, have already addressed themselves this point. John Locke held that the brain was to be likened to white paper, void of all characters, on which experience painEs with almost endless varicEy," This represents an instructive theory of learning, equivalent to considering the cells of the immune system void of all characters, upon which antigens paint with almost endless variery.
282
h
11
205 Contrary to this, the Greek Sophists, including Socrates, held a selective theory of learning. Learning, they said, is clearly impossible. For either a certain idea is already present in the brain, and then we have no need of learning it, or the idea is not already present in the brain, and then we cannot learn it cither, for even if it should happen to entet from outside, we could not recognize it. This argument is dearly analogous to the argument for a selective mechanism for antibody formation, in rhat the immune system could not recognize the antigen i f the antibody were not already present. Socrates concluded that all learning consists of being reminded of what is pre-existing in the brain. * 3
Summary In concluding this analysis, it would seem that selection refers to a mechanism in which the product under considetation is alteady ptesent in the system prior to the arrival of the signal, and is thus recognized and amplified. Each system that is capable of receiving a signal, however, is subject lo instruclion by this signal. Thus, at the level of an cntite systemj all such processes are instructive, whereas all instructive processes al a lower level imply selective mechanisms. In learning, and in all processes resembling learning, a discussion of instruction versus selection serves only to determine the organizational level of the elements upon which selective mechanisms operate. During recent years the belief that antigen plays an instructive role in antibody formation by intracellular guidance of the formation of the tertiary structure of globulin
molecules has been replaced by the idea that antibody formation is based on a selective process in which antigen selects pre-existing patterns and causes molecules representing these patterns to be produced at increased rates. The logical arguments for selection have been enforced by experimental evidence showing that the general mechanism of protein biosynthesis also applies to antibody production, rhat primary polypeptide structure determines antibody specificity, and that plasma cells can produce antibody in the absence of intracellular antigen. The antibody response appears to depend On multiplication of cells of the immune system. Attempts are being made to describe the cellular dynamics involved and to understand the nature of the antigenic stimulus. The teplacement of instructive by selective theories appears to be a general trend in the development of biology. A number of analogies arc drawn between the central nervous system and the immune system, and the question is posed whether a selective mechanism may also underlie the learning process. An analysis of this question leads to the conclusion lhat the terms instruction and selection can apply to descriptions of the same process at different levels. Each system that is capable of receiving a signal is subject ro instruction by rhis signal. Thus, at the level of an entire system, all such signals are instructive, whereas all instructive processes at some lower level imply selective mechanisms, through which products that were already present in the system prior to the arrival of the signal ate selected and amplified. In learning, as in all processes resembling learning, a discussion of instruction versus selection serves only to determine the organizational level of the elements upon which selective mechanisms operate.
283
Reprinted wilh permission from Reviews of Modem Physics, Vol. 63. No. 3, pp. 735-772. July 1991 © 199] The American Physical Society
Early vision and focal attention Bala Juiesz Laboratory end Division
of Vision Research, of Biology,
Rutgers
California
University—Kilmer
institute
of Technology,
Campus, Pasadena,
New Brunswick, California
New Jersey
08903,
91125
At the Ihirly-year anniversary of ihe introduction of the technique of computer-generated random-dot stereograms and random-dot cinemalograms into psychology, the impact of rhe technique on brain research and on the study of arlhicial inlelligence is reviewed. The main finding—thai stereoscopic depth perception (slereopsis). motion perception, and prsallenlive lealure discrimination are basically bottomup processes, which occur withoul the help of ihe lop-down processes of cognition and semantic memory—greatly simplifies the sludy of these processes of early vision and permits the linking of human perception with monkey neurophysiology. Particularly interesting arc the unexpected findings thai stcrcopsis (assumed lo be local} is a global process, while tenure discrimination (assumed to be a global process, governed by stBlisticsl is local, based on some conspicuous local features (lextonsl- !t ia shown •hat the lop-down process of "shape (depth! from shading" does not affeel stcrcopsis. and tome of the models of machine vision are evaluated. The asymmetry effect of human lenlurc discrimination is discussed, together with recent nonlinear spatial filler models and a novel extension of the '.'••'<• i theory lhat can cope with the asymmetry problem. This didactic review allempts to introduce ihe physicist (0 (he Acid of psychobiology and Us problems—including metascienlific problems of brain research, problems of scientific creativity, ihe stale of artificial intelligence research (including conncctiomst neural networks! aimed at modeling brain activity, and the fundamental role of focal attention in menial events.
SOME INTRODUCTORY READING FOR PHYSICISTS WHO ARE NOVICES IN BRAIN RESEARCH
CONTENTS Some Introductory Reading for Physicisls Who are Novices in Brain Research I. Introduction A.
On strategic and metascienlific problems in psycho' biology and the doctrine of "specific nerve energies"
B.
Bottom-up versus top-down processes, parallel versus
C-
serial processes, early vision, and focal attention Structuralist H n U Gestall theories
II. On the Creative Process and Scientific Bilingualism
ill
A.
The manifold view of scientific interaction and conjugacy. an approach TO discovery
B.
Scientific bilingualism, random-dol stereograms, and
735 737 737 738 740 742 742
ihe false-target problem Neurophysiology Versus Artificial Intelligence
743 746
A. B.
Global stereopsis and iIs linking lo neurophysiology Global slereopsis is a bottom-up and cooperative process
746
C.
Cooperative and noncooperative models of slereopsis
730
D.
The role of artificial intelligence model?, including Spin-glass, and conneclionisl models, in psychobiology
751
E.
A strategic problem of 3D. mathematical models of perceptual constancies, and Lie germs
751
On "maturational windows" and slereo blindness.
752
F-
IV. Tenure Discrimination and Focal Allention A. B.
A brief note on motion perception Freartenrive ientuie discrimination end focal attention
C.
Human tenure segregation^ when the whole is less than the sum of its parts
Y. From Tenons to Nonlinear Spalial Filters
743
(
753 75J 754 754 757
A.
A brief outline of ihe tenon theory of tenure discrimination
757
B.
The asymmelry problem of tenure segmentation and nonlinear spatial filters
760
C . Recent psychological and neurophysiological findings in tenure discriminalion D. Learning cfTccts in early vision VI. Conclusion Acknowledgments References
This review is intended for the physicist who is interested in vision research. It is written by an experimental psychologist who works at the interface between psychophysics and neurophysiology and who has a background in physics- It is hoped that the essenlial concepts in this review can be followed by the physicist without any prior knowledge of psychobiology, and lhat some of the problems raised, and their solutions, will appeal Lo physicists. As will be explained in the Introduction and Conclusion, Ihis review is highly partisan and is restricted to a subfield of vision research called early vision. The advantages of studying early vision are twofold. First, it is possible to avoid the daunting problems of semantics and of the higher symbolic processes by concentrating on stereoscopic depth perception (stereopsis) motion perception, and effortless texture segmentation which can be elicited by computer-generated stochastic displays (dynamic noise) of well specified geometrical, topological, and statistical properties. Second, these psychological phenomena can now be neurophysiologically studied in the input stages of the monkey cortex, for which, most importantly, the early neural processing stages of the visual system are practically identical to that of homo sapiens. The many dozens of higher cortical stages devoted to visual processing in the monkey are not well known, and their relationships to perception and cognition in the monkey, as well as in humans, are still a mystery. It is these enigmatic phenomena of form recognition, Gestalt organization, and the many unsolved problems of mental states (which are at present metascientific questions, as will be explained in the Introduction! that the naive reader would like most to read about, but that, unfbr-
764 764 765 767 767
284
(
736
tunately, will be skipped in Ihis review. Although there are interesting speculations—even by Nobel laureates — about the nature of dreams, free will, consciousness, and so on, early vision can be studied without knowing what these concepts are, and very few references will be provided that relate to these subjective ideas. The main reason why early vision, with its bottom-up processes, is still an interesting subfield of vision research is the inclusion of focal attention. Focal attention is a top-down process that permits us both to inspect different areas of the visual field without eye movement and to scrutinize different cortical areas. While the parallel processes of preattentive vision detect feature and texture gradients and tell the attentive system where to inspect, the searchlight of attention can scrutinize these gradients in order to identify what the features are (Sagi and Julesz, 1985). Focal attention superficially resembles some phenomena of consciousness, but is a simpler mental phenomenon and luckily can be scientifically studied with the available psychological and neurophysiological methods! In this review, much space is devoted to focal attention and to the preattentiveattentive dichotomy in general. For an up-to-date review on focal attention, see the article by neurophysiologists Desimone and Llngerleider [1989) and the review by Heilmaner al. (I9B7).
land 41986) gives a good introduction to elementary neuroscience, although the author is not a practicing neurophysiologist, but a philosopher. Those interested in the principles of neuroscience could benefit from the excellent textbook Principles of Neural Science, edited by Eric Kandel and James Schwartz (1985). Richard Gregory (1990) in Eye and Brain and Irvin Rock (1984) in Perception give a colorful introduction to perceptual phenomena of vision, although from a cognitive point of view. The introductory textbook Perception, particularly on visual perception, by Sekuler and Blake (1985), is more neurophysiologically inclined and is highly recommended. A remarkable textbook, The Senses, on sensory and perceptual processes, edited and written by Horace Barlow and John MoJlon (1982) gives a balanced introduction lo early vision and to the other modalities. (The first author is a pioneer in visual neurophysiology, the second an expert in color perception.) Those interested in parallel computers and their possible analogies to the brain will benefit from the monograph by Daniel Hillis (1985), The
Connection
Machine;
the author is the creator of a
massive parallel computer with the same name.
Early vision exhibits many interesting mental phenomena, from cooperative interactions and order-disorder transitions to learning and memory. It is the author's belief that it serves as an excellent introduction to the vast field of psychobiology. Because of the link between psychology and neurophysiology, the reader has to be somewhat familiar with some rudimentary concepts of brain research. Much effort has been expanded to make this article self-contained and to define new concepts as they are first mentioned; for more details the relevant references are cited. I should like to offer here some suggestions for introductory reading for the novice in psychobiology. The less ambitious reader might skip to the Introduction. Of course, there is no royal road to psychobiology, as there is no such road to physics. One approach might be lo read this review with an encyclopedia of brain research nearby, such as The Oxford Companion to The Mind (Gregory, 1987). Another approach might be to read this review and then to browse through the recent monograph by Resnikoff (1987a) entitled The Illusion of Reality, which introduces a large variety of concepts and facts on biological information processing, such as spatial niters, the analysis of subjective contours, the duality between aliasing and noise, visual illusions, and even an early version of my texton theory. After reading Resnikoff, the reader might find my review clearer, the more so since we cover similar topics and I start where Resnikoff ends. The reader unfamiliar with the rudimentary concepts of brains and neurons should turn to one of the pioneers in the field and read the brief and clear work by David Hubel (19881, Eye, Brain, Vision. Patricia Church-
There are many other interesting introductions to vision. The reader could benefit from the 25th anniversary issue of Vision Research (a leading journal of brain researchers working in vision), also published as a book edited by Boynton (1986), in which a dozen pioneers give a historical review of their specialties, including one by this author. A somewhat earlier, but still relevant, collection of articles on perception and visual adaptation phenomena, that benefitted considerably from the careful editing of Charles Harris (1980), Visual Coding and Adaptability, contains many introductory articles in advanced perception, including one by this author. A similar collection, but with emphasis on cortical neurophysiology, is the informative volume Dynamic Aspects of Neocorticai Function, edited by Edelman et al. (19841. For more advanced texts, many references are given in this article. I highly recommend the monumental Handbook of Physiological Optics by the physicist-physiologist Hermann von Helmholtz (1896; the 1924 English edition by the Optical Society of America was reprinted by Dover in 1962). i t is somewhat outdated, yet surprisingly, after a century, still fresh and stimulating. It is interesting to compare it with the Handbook of Perception and Human Performance, edited by Boff et al. 41986) a century later. The progress in knowledge is remarkable, but the many specialists in the later work cannot convey the unified and stimulating outlook of Helmholtz. The posthumous monograph of David Marr (1982), Vision, gives a unified and stimulating account of problems in computational vision from a leading figure of artificial intelligence. For a followup on Marr's legacy, the collection Natural Computation, edited by Whitman Richards (1988), is highly recommended. While Marr's view of computational vision is basically bottom-up, a recent short monograph by Roger Watt (1988), Visual Processing:
285
Computational,
Psychophysical,
and
Cognitive
737 argues for algorithms in early vision that are under the control of high-level processes and memory. Finally, the reader who has difficulties in fusion randomdot stereograms (RDS) and would like to view more than are shown in this review can turn to Julesz (1971) Foundations of Cyclopean Perception, which contains many RDS printed as red-green anaglyphs; these can be easily fused with the provided red-green goggles. There are several perceptual problems that seem elementary, do not require semantic memory, can be solved effortlessly, and yet are beyond our understanding. For example, knowing whether a point is inside or outside a bounded region—with rather meandering boundary lines—is such a problem, even though humans can solve it instantaneously. In this review, these problems of "visual cognition" will not be covered, and the interested reader should consult the insightful work of Shimon Ullman (1984). I can also recommend the two textbooks by Bruce and co-workers. Visual Perception by Bruce and Green 11985) and Visual Cognition by Humphreys and Bruce 11989). 1 wish to draw attention to the interesting work of Steven Grossberg and his collaborators, the more so because it has been widely ignored by the artificial intelligence community, although it is based on sophisticated mathematical tools. A good sample is Grossberg (1988), Neural Networks and Natural Intelligence. The physicist D. J. Amit's monograph Modeling Brain Function (1989) discusses connections between neural networks, spin glasses, Ising models, solid-state physics, etc., and might be of interest to physicists. There are many other excellent introductory texts to psychobiology, and the reader can always turn to friends for further advice. Here I only define the important concepts of a neurophysiological "receptive field" and of a "cortical column." In current neurophysiology, a single microelectrode is placed near a neuron in order to record electrical activity from the neuron. The cat or monkey is anesthetized and immobilized, and some visual stimulus (e.g., an oriented bar) is moved across the visual field until the selected neuron is activated. [Recently monkeys with a sealed opening in their skulls (installed weeks prior to experimentation) have been trained to fixate on a marker. Since the brain is devoid of pain detectors, the neurophysiologists can penetrate the brain with their microelectrodes hundreds of times (until the dura hardens) and measure neural activity in the alert monkey.) The specific local area of the luminance gradient with proper orientation and aspect ratio that elicits the largest activity is the receptive field of the selected neuron. Some neurons require quite complex trigger features to become activated, and the spatial concept of the receptive held is inadequate to describe the behavior of these complex and hypercomplex feature analyzers. Nevertheless, as a first approximation, a receptive field is an important concept. After all, who might have believed before the epochmaking findings of Barlow, Hartline, Hubel and Wiesel, Kuffler, or Lettvin and Maturana, to name a few pioneers of modern neurophysiology, that one could look for some Research,
specific feature extractors at a well defined locale in the brain and find neurons that would respond selectively to local stimulus features, such as the color, orientation, velocity, direction, flicker rate, etc., in the image. The psychologically defined local features (and the textons, to be defined later) are in most cases similar to the neurophysiological trigger features, and this correspondence between fundamental units of mind and brain makes the field of early vision so exciting. The cortical column, first discovered by Vernon Mountcastle in 1957 (for details, see Mountcastle, 1979) in the somatic sensory cortex of the cat, consists of narrow columns (slabs) that are perpendicular to the surface of the cortex and made up of neurons that perform similar local computations. For instance, neurons in the visual cortex of the monkey that are tuned to similar (but not identical) orientations of the same retinal location form such a column. Another example are those aggregates of neurons that are activated by left or right eye stimulations, respectively. These slabs are the columns of binocular dominance and can be made visible by radioactive dioxyglucose stains (Sokoloff, 1984); they can also be directly seen in viuo (after electronic enhancement) under the microscope (Ts'o et al., 1990). For the concept of the "hypercolumn," see Hubel and Wiesel (1974).
I. INTRODUCTION A. On strategic and metasci entitle problems in psychobiology and the doctrine of "specific nerve energies" During the spring and summer of 1990, while 1 was preparing this manuscript for physicists, by coincidence 1 worked on and finished three other manuscripts, one written for experimental psychologists, that is, colleagues in my own field, the second for philosophers interested in visual perception, and the third for neurophysiologists. In the first, just published in Trends and Tacit Assumptions in Vision Research (Julesz, 1991a), I pose about 40 strategic questions in vision research, in addition to half a dozen metascientific questions. Although this paper is written for active workers in psychobiology, it might give the physicist reader additional glimpses of my field. The second article is an open peer review in which I answer, among others, the philosopher John Searle on his recent idea that the brain cannot have unconscious processes. This paper (Julesz, 1991b), will be published in the journal Behatiioral and Brain Research, and I shall quote from it in a few places here. The third article (Julesz, 1990c) intended for neurophysiologists, will appear soon in the volume Symposium #55: The Brain, celebrating the 100th anniversary of the Cold Spring Harbor Laboratory, These three articles, together with this one, span a large audience belonging to four different disciplines, and
286
7M
while working on ihem I was forced to ponder the state of psychobiology from four different perspectives. The exercise confirmed yet again the broad appeal of problems in perception and their accessibility to workers in many disciplines. The human brain, the most complex structure in the known universe, with its visual cortex is many orders of magnitude more complex than the objects customarily studied by physicists and chemists. Therefore the psychobiology of vision appears to be in a premature state comparable to that of physics prior to Galileo, or molecular biology prior to the discovery of the double helical structure of D N A by Watson and Crick. Indeed, workers in visual psychophysics, neurophysiology, and neuroanatomy cannot agree on strategic problems; what is more, some would even deny that wc have adequate knowledge at present to formulate a strategic question. Just take the problem of "sensations" (also called "qualia"). Wherever the neurophysiologists probe the brain with microelectrodes, they seem lo record similar histograms of neural spike activity, regardless of whether the corresponding sensations are brightness, color, pitch, itch, temperature, pain, pleasure, anxiety, hunger, or contentment. Johannes Muller (1844), in his doctrine of "specific nerve energies," was already aware of the problem of sensations and stated that specific sensations arise in specific brain areasWhile this doctrine might seem somewhat unhelpful, I challenge any philosopher or brain researcher to say more with certainty at present than Johannes Muller said 150 years ago. (The interested reader is referred to a debate between John Searle and the author (Julesz, 1991b, in press).] Sensations are only one of the many metascicntific problems in brain research. The problems of "what is consciousness?, "is there free will?," "how are memories encoded?," "why do higher organisms sleep?," "what is the purpose of dreams?"—to name a few intriguing ones—cannot be tackled by the scientific methods currently available to us. Obviously these metascicntific questions are not all equally complex. Perhaps the riddle of short- and long-term memory will be solved in the not-too-distant future. However, it could be that "consciousness" belongs to a Godel-like problem, which might exclude neural nets to inspect (solve) certain complex states of their own. ["Is the human brain equivalent to, or more powerful than, a Universal Turing Machine (UTM)?" seems to be another metasci en tific problem, and the fact that the relatively simple "halting problem" cannot be solved (computed) by a UTM should instill in us a certain humility about tackling the really tough metasci en tific problems of our mental states. Returning to the question of the human brain's being more powerful than a machine, Penrose (19B9) thinks he can discuss this question scientifically by claiming that the conscious human mind can grasp (create and solve) mathematical problems that are not algorithmic, and even states that the essence of mathematics is not described by algorithms.] 11
B. Bottom-up versus top-down processes, parallel versus serial processes, early vision, and local attention In spite of this somewhat pessimistic introduction, I have witnessed the extraordinary progress in vision research during the last 30 years, and at the Editors' request I shall try to review some of the progress that was made in my own specialty, "early vision," a subfield of the experimental psychology of visual perception. I shall also glance at visual neurophysiology and neuroanatomy. Before I define the discipline of early vision, I have to explain some of my assumptions: I regard as "scientific psychology" only those subfields of psychology which emulate thermodynamics in the sense that higher (level I) phenomena can be explained by lower (level [1-1]) phenomena. In analogy to the phenomena (concepts) of temperature, pressure, entropy, enthalpy, etc., that arise from interactions between atoms and molecules as they collide with each other and the wall of the container, I want to define psychological phenomena [percepts) of depth and motion perception, textural segmentation (discrimination), focal attention, etc., as excitatory and inhibitory interactions between pools of neurons tuned to specific features (i.e., binocular disparity, motion disparity, texture gradients, etc.). This "structuralist" (reductionist) quest can be pursued only in vision (and even there within limits) and not in speech perception, as I shall now explain. Obviously, human speech perception cannot be explored by the current neurophysiological methods used on animals. (Although noninvasive methods of studying brain activity already exists, nevertheless the spatial and temporal resolution of PET scans and similar techniques are still inadequate.) Furthermore, the clinical literature on speech defects in stroke victims—though intriguing—by its very nature is more anecdotal than scientific. Because only homo sapiens possesses speech, my quest for a scientific psychology can only be pursued in vision. The visual system of the monkey is practically identical to ours and is intensively studied by neurophysiologists and neuroanatomists. At present only the first stages of the central nervous system of the monkey are studied with microelectrodes, and we have to restrict our stimuli so as to stimulate only the first retinal and cortical input stages (also called "bottom-up" processing stages). Thus we have to reduce or eliminate the influence of the higher (and enigmatic) cortical stages of semantic memory and symbolic processing (also called "top-down" stages). Figure 1 depicts such a bottom-up/top-down view of the central nervous system. Of course, both the bottomup and the top-down processes might have subprocesses with feedback loops, but in general the two main information streams flow as depicted. The reader should note that focal attention is regarded as a separate mechanism that can inspect many processing stages at will. Indeed, the reader, while ruminating over my last sentence, could easily turn his attention to countless thoughts and
287
73ft MacAdam, 1970), any color can be matched perceptually to three primary colors of given weights. While focal attention may inspect certain cortical stages, such as V4 (conjectured to be the color center by the neurophysiologist Semir Zeki, 1980), it does not have individual access to the red, green, and blue retinal receptors. In summary, focal attention can be drawn to many perceptual stages, bul not to the earliest ones, and probably even most of the higher perceptual stages and states cannot be a I tended. The fundamental problem of attention, particularly focal attention, means different psychological phenomena to different schools of psychologists and neurophysiologists. Here I stress the metaphor of focal attention as a shifting searchlight, as illustrated by Fig. 2. Here, two orthogonal line segments form elements of textural arrays. The X's pop out from the L's as a parallel process, i.e., almost instantaneously, independent of the number of elements. However, it requires time-consuming element-by-element scrutiny to detect an array composed of T's among the L's. These serial shifts of focal attention underlying scrutiny can take place tapidly without eye movements, as first observed by Helmholtz (1896) and elegantly studied by Posner 11978). [An English translation of this pioneering contribution by Helmholtz is provided by Nakayama and Mackenben (1989) from the second German edition after a century delay, because all existing English editions are based on the first German edition and Helmholtz added this important observation only to the second edition.] While evidence is in-
FIG. I . Bottom-up vs [op-down processes in the visual system and focal attention. After Jules; (1990b).
without effort could return to my next sentence. However, no introspection would permit the reader to discriminate a yellow color from a matched red and green color. That is, according to the theory of trichromacy, known since George Palmer (who stated it in 17771 and later rediscovered by Thomas Young in 1802 (whose publications are reprinted in a historical collection by
i f ' »-' '*• r ** f r S - » j j r <, > '"•> > > v f. f»r' i j r r »' - » j r -«-< r c j i M"V i ' J - ' " " N ' - v * - r J ' > <s j < - -v r -» i ' . 2 T t " ' » - v< -' > - j " * V j "i j r s
4
1
V
v
x
i-
t _t', r
>r r r <
r
r
_i
j r v ' *, * - 1 - ' >
iV*
v
r
>
r ^ ai±
J J < u.i
u
\ r
I^V
<. ^
j j
j
r
r
J J -> r „, j •>>*
( V « • » * * + «* j r i v< r i j i ^ ^ \5 j c +* + * * * * • <S »- J ' t f t j " ' V i¬ > r * * * ** • * • > ' * ^ "i *. / <. V's T - A \' T \' ^/ <,
A
,
v
v
1
x
/>
j
i / A . - ^
, v
s
r
^
j
-» ,
j
r
$
r
>
',. j '\"»i«\** 'f\ r / , _ / /* >f f ' i ~ -n* * " \ r J < A 1 ^ - c VA "»*> F j > " ^ -» » J ' y '"^ V J J " -iij i -i • -)i-i' >ji.i.!,ri.r T Jv*t /•> c •"• j /' j -> v f *"vt * r 1 _ r* ' - 1 - r v i>-'V v j >t t , j<*•,- v . * \ j - r >\ , ^ * v
l
- , i,
v
v
v
1
(s
/ V
s
,
v
>
%
/
A
v
t
< "T-'V
f - i r *
(
i > ' Y ^ J
FIG. 2. Preattenlive (parallel) I C X I U T E discrimination vs serial scrutiny by focal attention. The X's among the L's pop out effortlessly, while finding tbe T's among the L's requires an element-by-element search. From Julesz and Bergen (1983).
288
740
direct, depending on which model is used to interpret the experimental results, the "searchlight of attention" scans about 30-60 msec/item (Sternberg. 1966; Treisman and Gelade. 1980; Bergen and Julesz, 1983; Weichselgartner and Sperling. 19871. Saarinen and Julesz (1991) have recently measured directly the scanning speed of focal attention by briefly presenting (with masking) numbers at random locations, and though observers had difficulty correctly reporting the order in the sequence, they could follow and identify as many as four consecutive numbers at rates of 30 msec/item with orders of magnitude above chance. Obviously this rate depends on the visibility of the texture gradients, and some parallel mechanism seems to facilitate serial search (Krose and Julesz, 1989; Wolfe and Cave, 1990). Because during our waking stales we are bombarded by countless visual objects and patterns in a scene and can only make a few eye or limb movements in a given instance, much of the unwanted information must be filtered out centrally. Furthermore, we cannot store at a given moment more than a limited number of items (called "chunks") in memory, (usually 7±2 chunks), which is another reason why we have to filter the amount of incoming information. It appears lhal. in order to inspect objects and events thoroughly, focal attention is needed. It seems as i f there are only a very few processes (perhaps only one) of the highest level that can count as well as observe consciously and in great detail objects, patterns, and events, and i l is focal attention that does the preselection and presents the selected patterns and events item-by-item to the highest levels. Indeed, we found (Sagi and Julesz, 19851 that observers were able to find the positions of texture gradients without scrutiny—they just popped out effortlessly land did not depend on the number of elements in the arrays). However, to identify the features on ihe two sides of the texture boundaries (whose differences yielded the lexture gradients) required serial search by focal attention (and the search increased monotonically with the number of elements). The question of what can we perceive preattentively while our focal attention is engaged (e.g., our ability to identify at some specific position a character and to identify as well some characteristic feature at a texture gradient) is a strategic problem, currently being studied by Braun and Sagi (1990). Preliminary results indicate that, for instance, some crude features of position can be perceived preattentively, bul fine positional information requires scrutiny by focal attention. Besides focal attention, which scans objects about five times faster than one can do with eye movements, there is another way to increase information intake. This is by learning increasingly complex chunks. This kind of learning is a high-level process, beyond the scope of my interest; it is in the domain of cognitive psychologists to study this important mental phenomenon. Here I only explain the main idea of "chunking." If ihe reader is very briefly shown, say. five pebbles, probably their number can be guessed without error, while wilh more peb-
289
bles mistakes will be made. However if the pebbles are arranged in chunks of regular pentagons, the reader can count up to five pentagons without errors, and thus can count 25 pebbles. Whether one can extend Ihis learning of pentagons to increasingly complex chunks of pentagons is a typical question cognitive psychologists like to study. The literature on attention is vast. The previous paragraphs emphasize the single searchlight metaphor that originated with Helmholtz (1896) and was neglected afterwards by cognitive psychologists for some time, as I alluded lo. Another prominent figure of psychology, the American psychologist William James (1890) asked ". . . how many ideas or things can we attend to at once, . . . how many entirely disconnected systems or processes of conceptions can go on simultaneously; the answer is, not easily more than one, unless Ihe processes are very habitual, but then two, or even three, without very much oscillation of the attention." James's question opened up an entirely new field of inquiry called divided attention. This is characterized by the metaphor of finite resources (or limited capacity). According to this melaphor the resources can be regarded as a "fuel," and one can change Ihe amount of fuel allocated to various multiple (usually double) tasks. In -i way, the question of what resources are left for perceiving certain features when focal attention is engaged belongs to the problem of divided attention, bul in a more concrete sense than usually asked by cognitive psychologists. (The Saarinen and Julesz experiment discussed above—though conceived in the spiril of the searchlight metaphor—can support equally well the melaphor of divided attention.) The interested reader might consult [he book edited by LeDoux and Hirst (1986), which contains a critical debate between psychologists and neurophysiologists, with four articles devoted to attention. The end result is somewhat disappointing, since there is no real consensus between workers in the two disciplines. However, the articles review the many theories of attention, including the bottleneck and variable filler theories by Broadbent (19581 and his many followers. For a recent review on some outstanding findings in cortical neurophysiology of attention, sec the article by Desimone and Ungerleider (1989). C. Structuralist versus Gestalt theories Having introduced the basic concepts above, we return to the definition of early vision. Conceptually defined, "early vision" should be identical to the pure bottom-up visual processes depicted in Fig. J, without being influenced by the top-down stream of semantic information. Neurophysiologically defined, "early vision" should correspond lo the first neural processing stages in the retina and the visual cortex. Psychologically defined, "early vision" should encompass a range of perceptual phenomena that can be experienced by humans in the absence of higher cognitive and semantic cues. In the next
741 sec lion, we shall see thai with the techniques of computer-generated random-dot stereograms (RDS), random-dot cinematograms (RDC), and texture pairs with controlled statistical properties (Julesz, 1960, 1962), it is possible to show that stereoscopic depth perception, motion perception, and preattentive texture segregation can occur in humans without the mysterious cues of form and Gestalt. This in turn permits us lo link these mental phenomena to the neurophysiological findings obtained in the last three decades by probing with microelectrodes individual neurons in the early processing stages of the monkey cortex. My structuralist approach of treating early vision in a thermodynamic fashion does not mean that T regard human vision in its entirety as amenable to such treatment. In real-life situations, bottom-up and top-down processes are interwoven in intricate ways, and the slogan of the Gestalt psychologists that "Ihe whole is more Ihan Ihe sum of its parts" — a negation of the structuralist view of science—is probably true. Indeed, in Fig. 3 it is obvious that in the right upside-down image the eyes and mouth of the face have been manipulated. Because we are not familiar wilh inverted faces, the original face (left side) and the right one appear quite similar. Turning the page upside-down, so that the faces are now correctly seen, we
experience a dramatic difference; (he untouched face appears normal while the manipulated face looks grotesque! Obviously the mouth, the eyes, etc., are not simple building blocks of a perceived face; instead some global and highly complex interactions between them exist, and the concatenations of these parts into a Gestalt make the study of form recognition so frustrating at present. What 1 am suggesting is not that Gestalt phenomena be overlooked, but instead that an entire subfield of vision— early vision—be experimentally isolated and studied by the proven structuralist methods of the physicists. [For an epistemological review on the limits and merits of structuralist models in psychobiology, see Uttal (1990).] While I assure the reader that I stick to the structuralist (reductionist) paradigm throughout this review, 1 acknowledge the important contributions that Gestalt ("configuration") psychologists have made in the first part of this century, from the principles underlying figure-ground organizations to perceptual grouping. One reason why their popularity waned was lhat their belief in electric brain fields whose convergence loward minimum-energy states was not supported by results gained from single-mieroelectrode neurophysiology. However, some of the Gestalt ideas have resurfaced recently under the guise of connectionist neural networks;
FIG. 3. Demonstration of Gestalt- The upside-down pictures appear rather similar in appearance, in spite of the fact that in one picture the eyes and mouth seem 10 be inverted. When the page is turned upside down, the two faces reveal a dramatic difference as a result of Gestalt organization. From Julesz (1984) after an idea of Thompson (1980!.
290
742
the interested reader should see the brief and clear article of Rock and Palmer (1990), My own view on connectionist neural networks in early vision is given in Sec. H I D; however, I would not be surprised to see a neo-Gestalt revival explaining higher visual functions in the not-toodistant future.
II. ON THE CREATIVE PROCESS AND SCIENTIFIC BILINGUALISM A. The manifold view of scientific inletaction and conjugacy, an approach to discovery From the Introduction it is quite cJear that the study of the human cortex is so complicated that a gamut of disciplines—including sensory, perceptual and cognitive psychology, neurophysiology, neurology, neuroanatomy, embryology, neuropharmacology, mathematics, engineering, information theory, neural network theory, physics, and so on—is necessary to study its workings. The question arises of how any individual can cope with such a variety of different fields. A good answer is given by Michael Polanyi 11969), as follows: Even mature scientists know little more than the names of most branches of science.. . . The amplitude of our cultural heritage exceeds ten thousand times the carrying capacity of any human brain, and hence we must have ten thousand specialists to transmit it. To do away with Ihe specialization of knowledge would be lo produce a race of quiz winners and destroy our culture in favor of a universal dilettantism.. . , Bui how can anybody compare the scientific value of discoveries in, say, astronomy wilh ihose in medicine? Nobody can. but nobody needs to. All that is required is that we compare these values in closely neighboring fields of science. Judgments extending over neighborhoods will overlap and form a chain spanning the entire range of sciences.
well ask whether there are ways to make this slow accumulation of knowledge more directed or conscious, thus accelerating progress. This question is intimately related to the essence of scientific creativity. While there must be many ways to create a new paradigm or get a novel insight, here 1 shall briefly discuss "conjugacy" and particularly "scientific bilingualism" as Two approaches lhat can be used to advance science in general and brain research in particular. The first approach, conjugacy, is the "trick" of establishing an equivalence relation between a difficult or unpected explored task (operation) and a familiar one, whose solution is already known. This is depicted in Fig. 4, where the difficult task O is to transport an object from point A to B through an impenetrable obstacle (wall). A possible way to complete this task is to drill a shaft S from point A , drill a tunnel T under the wall (assuming that drilling is a routine operation), and finally drill an inverse shaft S " lo point B . One case in point is the facilitation of the operation O of multiplication (division) by introducing the logarithmic transformation S and its inverse S lhal reduces the task to Ihe much simpler operation T of addition (subtraction). Similarly, cross-correlation O of two functions can be reduced to Ihe simple multiplication of the Fourier transforms S of Ihe two functions, followed by taking the inverse Fourier transform. As a mallet of fact, when neurophysiologists discovered Mexican-hat-shaped (Laplacian of a Gaussian) receptive field profiles of a concentric circular kind in retinal ganglion cells of ihe cal (KufHer, 1953) and elongated field profiles in some orienlalion in the visual cortex of the cat and the monkey (Hubel and Wiesel. I960, 1968), it was apparent that these spatial filter responses had to be cross-correlated with Ihe visual image (brightness distribution cast on the retina). [The 1
- >
CONJUGACY 0=STS"
The spread of knowledge between overlapping scientific disciplines according to this "manifold view," occurs naturally, similarly to the cooperation between members of a beehive. The only requirement is that specialists working, say, on the brain not have too narrow specializations, so they can indeed communicate with other specialists in overlapping areas of shared knowledge. Such cooperation can results in novel applications or technical breakthroughs. For instance, in the forties, electrical engineers started to develop special low-noise amplifiers that enabled neurophysiologists to record spike potentials in individual neurons. Another example is the development of surgical techniques to reduce epileptic seizures by splitting corlical hemispheres; these techniques enabled Sperry (19B2) and his collaborators to study the mental competence of each separate hemisphere. For an up-to-date review of hemispheric localization, see Gazzaniga (1989). One may
291
T
FIG. 4. The equivalence relation "conjugacy." or how to solve a difficult task by transforming il lo an already familiar lask.
743
Spaml
Racapliva
MT F
Field
I
Spalm I F r a q u s n c y
Spatial
c<0«g
Potition
Otgim
FIG. 5. Illustration of conjugacy in brain research. The neurological!y determined double-Gaussian-shaped receptive field profiles and Ihe psychophysically determined modulation transfer functions Iby measuring Ihe visibility thresholds to spatial luminanca gratings wilh changing spatial frequencies) are related by the Fourier transform. From LeDoux and Hirst (1986),
reader unfamiliar wilh spatial filters might read Ihe text and figure caption related to Fig. 15 below]. However, neurophysiologists were not at home in the cross-correlation domain, and it took the psychologists (Blakemore and Campbell, 1969) to show that sinusoidal luminance gratings presented at long duration would selectively fatigue the detection of test gratings when the test grating had spatial frequencies similar to the adapting grating. It is apparent from Fig. 5 that the visibility function of sinusoidal gratings, called the modulation transfer function (MTF), is a decomposition of perceptually weighted Dirac functions in the Fourier domain, which in turn is the Fourier transform of the neurophysiologically measured receptive held profiles. Had researchers been more familiar either with Fourier analysis or with cross-correlation techniques, they could have discovered the luminance grating adaptation paradigm years earlier, when the first neurophysiological findings were obtained. 1 do not want to dwell on the use of conjugacy further, since the reader can make a long list of his or her own of novel unexplored tasks that have been transformed by some ingenious application of conjugacy to an already familiar one, (The interested reader should consult the inspiring mathematical monograph by Melzak. 1976.) However, an equally effective way to gain novel insight based on scientific bilingualism is less appreciated, and I turn now to this topic, after an analogy Lo linguistic bilingualism.
B. Scientific bilingualism, random-dot stereograms, and the false-target problem The reader who is irritated by personal anecdotes should skip the first two paragraphs of this section. The reader who speaks foreign languages in the same language family as his native tongue, say, Russian and French besides English, cannot appreciate the intellectual thrill one experiences when learning another language in a different language family. For a Hungarian child it was customary to learn an Indo-European language at an early age (since Hungarian belongs to the Finno-Ugric language family). I learned German and was quite surprised by the existence of a passive voice. In Hungarian one has to know the name of, say, a lecturer, in order to announce a lecture, since saying "someone will give a lecture on stochastic processes at noon" sounds comical even in Hungarian, while in all Indo-European languages one can easily announce that "a lecture will be given on stochastic processes at noon." This noncommittal passive voice is a great invention and is taken for granted by speakers of Indo-European languages, while Hungarians, Finns, Estonians, etc., have to struggle along with awkward situations that require them to know concretely who did what. In 1959, after I arrived at Bell Laboratories as a young communications and radar engineer, I started to learn psychology, since 1 had joined a group dedicated to re-
292
744
ducing Ihe visual information in pictures without perceptual deterioration and I thought that some familiarity with human vision would be beneficial. Obviously, pictures are redundant, and the human visual system does not require half-tone images in their full detail; it can, for instance, recognize objects merely from their outlines, a fact exploited by cartoonists. But what is an outline (contour)? Could one design a machine [algorithm) that could emulate a skilled cartoonist by extracting meaningful contours from patches of various grays? The idea of using the second spatial derivative [the Laplacian) for extracting contours was proposed and discussed by Ernst Mach (1886, reprinted 3959) and revived by David Marr (1982) and his followers, who averaged the image luminance with a Gaussian niter followed by the Laplacian. An image convoluted with the Laplacian of a Gaussian is almost identical to the double Gaussianshaped (or Mexican hat function) receptive held profiles found by neurophysiologists in concentric form in retinal ganglion cells, and in elongated form in the striate cortex (VI), as portrayed on the right-hand side of Fig. 5. [These receptive field profiles are also quite similar to the cosine and sine functions weighted with a Gaussian envelope now called Gabor functions, in honor of Dennis Gabor (19+6), who derived them as the optimal solution for the carrying of information simultaneously in both spatial and temporal domains.] Such simple linear operators applied to an image are surely inadequate to find contours and separate overlapping objects. For example, the image of a rabbit sitting before a fence has ears that could belong either to the fence or to the rabbit, and a great amount of familiarity with rabbits and fences is needed to separate the face from the background. Furthermore, the existence of "subjective contour" phenomena, discovered by Schumann (1904) and further elaborated by Kanizsa (1976), as demonstrated in Fig. 6, shows clearly that perceptually perceived contours exist in uniform areas where no luminance gradients occur, based on the linear continuation of quasicollinear line segments.
*
v
In the cognitive psychology literature, subjective contour perception was traditionally regarded as a complex task of completing line segments by some higher-order reasoning. Actually, as neurophysiologists have recently shown (von der Heydt el a!., 1988), the completion of subjective contours occurs in V 2 (an early cortical stage in the monkey cortex). In essence, as 1 became acquainted with the psychological literature in 1959, it became apparent that the extraction of contours was intimately linked with the segmentation of objects, which in turn was based on the little-understood cues of semantic memory. So when I suggested using stereoscopic depth perception (stereopsis), based on horizontal disparity, to segment objects, it was pointed out that in order to find in a crowd the corresponding faces in the left and right retinal projections, one would first have to recognize the faces—a most complex feat, whose execution is shrouded in mystery. Already with four targets, not knowing which reti¬ nal projections correspond to each other, as depicted in Fig. 7, sixteen possible localizations can be made, out of which only fout are correct and the rest are false (phantom) targets. With N targets, the number of phantom targets increases as (AT —AH. Thus, according to prevailing beliefs, monocular form cues and contours were essential for stereopsis, a belief shared by the leading expert of binocular depth perception of this period, Ken3
Left eye
Right eye
It
-
J
vv
Dots in right eye view
Dots in left eye view
A = Right eye
A = Left eye
R
FIG. 6. Subjective contours as first described by Schumann (1904) and elaborated by Kanizsa (1976). One sees a triangle with a sharp boundary even though no luminance contours exist at most places. After Kanizsa (1976).
293
L
FIG. 7. Binocular matching, or the problem of false-target elimination. The four identical targets (dots) yield 16 possible localizations, out of which only four are correct. Without monocular labeling only global constraints (suoh as crosscorrelation) can eliminate the false matches. From Julesz (1971).
735
nelh Ogle (1964). As a former radar engineer, I knew (hat this view of Ihe psychologists could not be valid. After all, in order to break camouflage in aerial reconnaissance, one would view aerial images taken from two different positions (the use of parallax) through a stereoscope, and the camouflaged target would jump out in vivid depth. Of course, in real life there is no ideal camouflage, and after stereoscopic viewing one can detect with a single eye a few faint cues that might discriminate a target from its surroundings. So I used one of the first big computers, an IBM704 that had just arrived al Bel) Labs, to create ideally camouflaged stereoscopic images as shown in Fig. B(a). Since the binocular disparity is an integer multiple of the randomly selected but correlated black and white dots (pixels) that make up Ihe left and right arrays, these images, when viewed monocularly, are only aggregates of random dots with no break or gap between areas of different disparities. However, when binocularly fused, the correlated areas segregate in vivid depth according to these monocularly invisible disparities (Julesz, 1960, 1971), For a personal history with more detail, see Bernstein (1984), or
my review in Ihe 25th anniversary Silver Jubilee issue of Vision Research (Julesz, 1986a), and also Julesz (1990a, 1990b). Figure Bib) is similar to Fig. 8(a), except that the density of the random-dot stereogram (RDS) is reduced to a few percent. It is as easy to fuse as the dense RDS, and it is interesting to observe that the visual system interpolates a surface in depth where there are no dots present (as i f it were a "convex hull"). [This "tilling i n " phenomenon inspired many model builders of early vision; the interested reader might consult Grossberg and Mingolla (1985).] I have dwelt on this historical account merely to stress Ihe poinl that the introduction of the RDS to psychology was just a lucky realization by a radar engineer that Ihe common knowledge in aerial reconnaissance thai camouflage cannot exist in 3D was unknown to psychologists. It was this scientific biiingualism that led to the discovery of RDS, although there must have been dozens of stereophotographs laken from balloons or airplanes of densely textured scenes that emulated random-dot stereograms quite well. (Babington-Smilh, 1977, reprints a
FIG. &. A computer-generated random-dot stereogram (RDS) that, when monocularly viewed, appears as a random array of black and white dots (pixels), yet when binocularly fused twith the aid of a prism or by crossing the eyes) reveals a center target, which jumps out in vivid depth, (al RDS with 50% density of black pixels: (b) RDS with 5% density of black pixels. From Julesz (1971).
294
748
British aerial reconnaissance photograph of Cologne taken from two different vantage points during W W I I , in which one can see the floating ice on the River Rhine. When binocularly fused, the pieces of ice portray interesting surfaces in depth, particularly al the vicinity of the piers of a bridge, where the ice flow is altered.) Of course, the arrival of computers permitted ideal camouflage with absolute stimulus control, and nowadays one can generate dynamic RDS at a rate of 60 frames/sec or faster, portraying moving surfaces in depth; when these are viewed monocularly, only dynamic noise (snowstorm) is perceived. [I referred above to the influential work of David Marr (1982) and his followers. In this historical recollection I want to point out that their ideas, particularly ihe close relationship between Ihe early contour extraction in 2D and the 3D information obtained through binocular disparity, was strongly influenced by the findings with RDS a decade earlier and led lo their "2.5-D sketch model." In the next section 1 shall discuss some of their contributions to the field of global stereopsis and give a critical evaluation of the role of the artificial intelligence approach in brain research.] The essence of this somewhat personal note is thai science usually progresses haphazardly through overlapping fields or disciplines, bul more directed, conscious contributions can be made i f a scientist is willing to learn two remote disciplines and apply this scientific bilingualism. This is nowhere more true than in brain research. For instance, physics and psychology, or physics and neurophysiology in my opinion are excellent backgrounds for attacking problems in psychobiology. III. NEUROPHYSIOLOGY VERSUS ARTIFICIAL INTELLIGENCE A. Global stereopsis and its linking to neurophysiology
1 now return to my original purpose, which was to study stereopsis — and psychological phenomena in general—by emulating thermodynamics. The reader is able lo see the "cyclopean" square in Fig. 8 after binocular fusion (aided by using an optical wedge to cross eyes, and wearing prescription glasses to correct for astigmatism—assuming, of course, that he or she does not belong to the 2% who are stereoblind); from this ability it becomes apparent that stereopsis is a bottom-up process that must occur before form perception, since the left and right images are devoid of all monocular cues, including shapes and their contours. Furthermore, it can be shown in the laboratory (Julesz, 1964; Julesz and Chang, 1976) by masking the stimulus 60 msec after the stimulus onset—and thus terminating the availability of the stereogram — lhat one can easily fuse and perceive
295
complex surfaces in vivid depth during such a brief presentation, provided Ihe maximum binocular disparity is within a limit, called Panum's fusional area. With dynamic RDS (which nowadays can be easily generated on personal computers and can be stored on video tape recorders), each subsequent frame contains uncorrelated random dots (but similar binocularly correlated areas), and these uncorrelated frames erase (mask) the previous frames. Therefore, in addition to the fact lhat the RDS are devoid of all familiarity cues, quick masking prevents the top-down processes from penetrating down in time and influencing the bottom-up processes of stereopsis. There are many demonstrations that stereopsis of RDS (which I often refer to as either "global stereopsis" or "cyclopean perception"! must be an early process, based on some correlationlike process prior to object (form) recognition. Here, I only lake one example from my monograph (Julesz, )97l) showing that optical illusions must occur after binocular combination of information, that is, after several synaptic processing stages in the retina, lateral geniculate nucleus, and cortex. This technique of process localization without a scalpel, which I called "psychoanatomy," enables us to trace the information flow in the visual system by portraying visual information, not with the usual luminance gradients [Fig. 9(a)], bul with binocular disparity gradients [Figs. 8(b) and 8(c)), using RDS (Papert, 1961; Julesz, 1971). Figure 9(a) shows the classic Ebbinghaus illusion, in which identical center disks appear to have different sizes. When the reader inspects the cyclopean version of [his illusion, by fusing the RDS of Fig. 9(b), it is apparent that the illusion is the same. Therefore the many processing stages prior in the "cyclopean retina" (an assumed stage at which binocular correlation first occurs) do not affect the illusion, and the optical illusion has to arise afterwards. With the technique of RDS one gains a new degree of freedom and can portray the center target at a different depth from the inducing figures, as shown in Fig. 9(c). When fusing Fig. 9(c), the reader can verify that the illusion is greatly reduced, if not gone. This suggests that optical illusions are the result of lateral interactions between neural pools tuned to the same binocular disparity, while pools tuned to different disparities do not interact with each other (Julesz, 1971). Thus the RDS, since its inception in 1960, suggested lhal stereopsis must be an early process and much simpler than form recognition, It was Ihe neurophysiologists who first realized these implications and switched ftom studying form lo exploring binocular vision (Hubel and Wiesel, 1962, 1970; Barlow. Blakemore, and Pettigrew, 1967; Bishop, 1969; Blakemore, 1969). The perceptual finding that one can adapt to selected depth planes in a RDS by prolonged viewing and, as a result of fatiguing neural pools tuned to the corresponding binocular disparities, modify the amount of petceived depth (Blakemore and Julesz, 19711 suggested the existence of neurons tuned to specific binocular disparities. Nevertheless, only in 1984 did Gian Poggio find neurons as ear-
747
FIG. 9 Classically and cyelopeanly portrayed Ebbing ha us illusion, (al The classical illusion portrayed by luminance gradients The testfiguresin the center are actually identical, but due to differences in Che inducingfigures,they appear of different size, lb) The same illusion cyclopeanly portrayed, by ihe method of RDS. Test and inducingfiguresare in the same depth plane when binocularly fused, &nd the perceived illusion is similar to the classical one, (c) Same as ib\ except Lhat the test and inducingfiguresare in different depth planes, and the perceived illusion is much reduced, if seen al all. From Julesz (1971). r
296
746
ly as layer 1VB in area V I (the input stage to the visual cortex of macaque monkeys) that selectively responded to binocular disparities in dynamic RDS (Poggio, 1984). Since his stereograms were presented at rates of 100 frames/sec, there was no monocular contrast between the correlated areas, and therefore these early processing stages were able to solve the binocular matching problem (shown in Fig. 7) of performing correlationlike processing. While in area V1 only 20% of the probed units were cyclopean, as Poggio et al. (1988) probed higher areas, V2. V3, and V3A, the majority of neurons were found to be cyclopean (i.e., Bred for dynamic RDS). Poggio's discovery of cyclopean neurons in the input stages of the cortex confirms the psychological predictions that global stereopsis fi.e., stereoscopic depth perception of RDS) is an early process and neurophysiologically determines the first stages of the "cyclopean retina." That global stereopsis is such an early process has another interesting implication. Since in 3D there is no camouflage, stereopsis probably evolved in our insectivore primate predecessors (e.g., lemurs) rather late in the evolutionary lime scale, in order to counteract the freeze response of insects, who would blend inlo the foliage at the sign of danger. [That in general there is no camouflage in three dimensions is the main insight gained by RDS; nevertheless, in a few rare cases, insects evolved such that their 3D shape mimics the 3D shape of leaves or reliefs of tree branches.J One would expect that such a late development would be relegated lo an "attic" at some unused area of the cortex. Yet Ihe emerging mechanisms of stereopsis were important enough to push aside existing machinery and grab the input stages of the visual corlex! Perhaps the most important aspect of these developments is the linking of a rather complex menial evenl — global stereopsis—-to some neurophysiological evenl — the firing of neural pools tuned to specific binocular disparities. As I mentioned previously, such a link was sought between neurophysiology and Ihe sensory psychology of color vision years earlier, bul Ihis is ihe first time in the more complex field of perceptual psychology that such a link between mind and brain was found. (When it comes lo linking hypotheses between mind and brain, different schools of psychobiology have rather different criteria. 1 found a relevant essay by Davida Teller (1980) most entertaining. While writing Ihis article, 1 came across a rather novel linking criterion by Salzman et al. (1990), who applied cortical microslimulation to an extraslriaie area (MT or V5) of the monkey that plays a prominent role in extracting motion information. This electrical microstimulalion biased the animals' perceptual judgments of motion. This finding implies that physiological events at ihe neuronal level can be causally linked to a specific aspect of perceptual performance. While the authors used motion in their study, they pointed out that they could also have applied the same experimental paradigm lo color or depth perception.]
297
B. Global stereopsis is a bottom-up and cooperative process One might wonder whether computer-generated RDS with rich disparity cues might not be a special case, and whether top-down processes might still interfere with global stereopsis in natural scenes when higher cognitive cues are present. An example might be depth from shading, noted by astronomers when viewing Ihe craters of the Moon. When viewed upside-down, Ihe concave craters appear to be convex mounds. Such a depth reversal is demonstrated in Fig. 10 when each of the slereo half-pairs is viewed monocularly, depending on whether the illumination comes from above or below. This "shape from shading" or "monocular depth from shading" mighi be construed as a top-down phenomenon interfering with the botlom-up processes of stereopsis. Indeed, Ramachandran (1988) exploited this phenomenon in his study of apparent motion perception and suggested that Ihe "shape-from-shading" process operates prior to motion perception. The motion perception studied by Ramachandran is of Ihe long-range kind lhat often disambiguates false matches by higher-order top-down processes, yet it is of interesl lo know whether the bottom-up process of short-range motion perception, which occurs for random-dot cinematograms and particularly for global stereopsis, is also influenced by shape from shading. After all, this phenomenon must be of a high-level kind, based on the fact Lhal Earth has only one Sun that shines from above. I f global stereopsis were lo utilize such a complex top-down process, based on some learned or genetically inherited information, we would have a counterexample of global stereopsis' being based on early visual processing alone. In order to test this "counterexample," Jih Jie Chang and I (Chang and Julesz, 1990) constructed a randomly speckled egg-crate pair portraying the convex (or concave) depth-fromshading phenomenon when monocularly viewed (Fig. 10). However, 30% of Ihe egg-crate pair is speckled with a RDS having a crossed disparity (if Ihe reader views the slereogram wilh crossed eyes). Whether Ihe reader views these figures right side up or upside-down, it becomes apparent that the convex shape determined by stereopsis will dominate depth from shading. Variations of this experiment, including ambiguous EDS (with two cyclopean shapes at different depth, but only one of them perceptible at a given instant), demonstrate that the monocular cue of depth from shading is ralher weak (since observers perceive depth according to their natural bias) and therefore does not influence global stereopsis (Chang and Julesz, 1990). In summary, it is most likely lhal global stereopsis is mediated by early visual processes without top-down influences. It is, in the usage of Fodor (1983), a "cognitively impenetrable module." In the previous paragraph, Ihe problem of shoriversus long-range processes in motion perception and slereopsis was briefly mentioned. My original use of "global" stereopsis and molion perception in RDS and
749
F I G . 10. M o n o c u l a r l y convex (concave) sphere due io depth (shape) shading, in w h i c h a random-dot stereogram (RDS) is mixed in. W h e n binocularly fused, depth from stereopsis dominates perceived depth. convex sphere, but appears concave when page is turned upside-down
When viewed monocularly, depth from shading yields a
Because the RDS has crossed binocular disparity, the binocu-
larly fused image appears strongly convex. F r o m C h a n g and Julesz. (1990),
RDC, respectively, referred to the solving of the falsetarget problem globally (by cross-correlation)'* since the thousands of individual random dots could be paired in billions of incorrect ways. With increased disparities between the left and right half-pairs in a RDS, or between successive monocular arrays in a RDC, the number of possible false matches increases. Thus for global stereopsis and motion perception, the finding of matches within a short range is relatively easy, whereas over a long range it rapidly becomes difficult. However, for patterns lhal can be locally identified {by their color, shape, etc., as in a neon sign), the problem of correct matching is virtually independent of disparity, and stereoscopic depth or apparent motion can be obtained over a long range. Hence "local stereopsis' and "local motion perception" can also be called "long-range motion' or "long-range slereopsis," My use of local versus global is discussed in Julesz (1978) and Chang and Julesz (1983)] the use of short- versus long-range motion is also discussed by Braddick {1974, 1980) and Snowden and Braddick (1990), as well as in a critical paper evaluating these dichotomies by Cavanagh and Mather (1989), 1
1
Lei us turn now to the problem of artificial intelligence 1AI) models. One could assume that, instead of neurophysiology, A l modeling could assume the role of an (I¬ D-level description. While this is a theoretical possibility, it turns out that A l models are too robust—i.e., di-
ametrically opposite models can explain perceptual phenomena equally well—and therefore one has to perform psychophysical and neurophysiological control experiments to check the biological plausibility of these models. Besides the binocular matching problem (or falsetarget elimination problem) discussed in Fig, 7 there are three basic psychological findings of global stereopsis that model builders have to take into account. First is the finding the global stereopsis of RDS is cooperative — exhibiting multiple stable states, disorder-order transitions, and hysteresis. (For details, see the monograph and reviews by Julesz 1971, 1978, 1986a, I990a1, Here, 1 only mention an experiment with ambiguous RDS in which a bias of a few percent of unambiguous dots with given binocular disparities can perceptually affect the perceived depth (Julesz, 1964, 197B; Julesz and Chang, 1976}—a typical cooperative effect. The second finding is that global stereopsis utilizes spatial-frequency (SF)filtered channels (Julesz and Miller, 1975). The spatial frequency spectrum of a left and right stereo half-image pair must overlap to some extent to obtain fusion. The third finding relates binocular disparity gradients to fusion and postulates "forbidden zones" where fusion cannot occur because the gradient exceeds a critical value (Burt and Julesz, 19801. That article demonstrates how nearby objects "warp" the disparity space, creating such forbidden zones.
298
750
C. Cooperative and noncooperative models of stereopsis The brief (60 msec) processing time for global stereopsis and its cooperative behavior suggests a parallel mechanism. Furthermore, when one of the states of an ambiguous RDS is biased by 4% (in favor of the other statel, a serial search mechanism would often stop at an almost perfect 96% binocular correlation, whereas, in fact, the stereopsis mechanism always chooses the perceptual state that yields 100% correlation, proving that not only the correlation mechanism but also the search mechanism for the best binocular match is parallel (Julesz and Chang, 1976). That perceptual phenomena are based on parallel processing is not a great surprise. Not only do 30 years of neurophysiological evidence point to this conclusion, but it is borne out by the following Gedanken-experiment: It takes about 500 msec or less to perform routine perceptual tasks leg., to recognize a face in a complex scene or to identify a carloon), whereas a synaptic event (computation) requires 2-4 msec. Therefore the brain mighl utilize al most 100 stages of computational layers (instructions, or codesl, which is in striking contrast to the thousands of codes necessary for A I algorithms running on the ubiquitous serial computers, which in addition perform perceptual tasks rather poorly. My AUTOMAP-l mode] was the first computer algorithm that could explain in successive iterations some of the cooperative phenomena of global stereopsis (Julesz, 1964). In order to give a better insight into the workings of such a model, I intentionally introduced a "springcoupled-magnelic-dipole-model" (Julesz, 1971, 1978) that explained the pulling effect and plasticity, i.e., the fact that the left image of a RDS can be zoomed up or down in size by 10% with respect to the right without loss of fusion. (The reader interested in the history of stereopsis models might consult Julesz, 1990b.) Simultaneously to my cooperative spring-coupled-dipole model, Sperling (1970) proposed an "energy-well model" lhal had a heuristic appeal too. while Dev (1974) and Nelson (1975) offered cooperative models based on spreading inhibition between disparity neurons tuned to different disparities, and facilitation between neurons tuned to the same disparity. Marr and Poggio (1976) also published a cooperative model that resembles lhat of Dev and Nelson, in which inhibition occurs between disparity detectors tuned to different disparities, but ihese detectors fall on the same lines of sight. (This requirement seems important, since it has a heuristic appeal, but in fact is not, so the model is very similar to those mentioned above.) Perhaps the greatest contribution of the Marr and Poggio (1976) model is its easy implementation as a computer algorithm, even on PCs. Recently, Tomaso Poggio (1989) at M I T implemented a very fast stereo algorithm on the Connection Machine, and Mohowald and Delbriick (19891 at Caltech designed a one-dimensional version of the Marr and Poggio 11976) model on a chip lhat works
very fast and solves Ihe false-largel problem in milliseconds. Cooperalivity underlies even Ihe computation of stereoscopic acuity (which is an order finer than visual acuity), as suggested by Westheimer (1979). While most models of stereopsis are cooperative, Marr and Poggio (1978) invented a second, noncooperative model of stereopsis. Their model is based on Ihe use of ihe spatial frequency channels found by Julesz and Miller (1975). According lo this model, coarse SF channels in the two retinal projections—which tolerate large disparity searches without false matches—are aligned first by vergence (convergence or divergence) movements of the eyes, and Ihen by finer and finer matches of high-SF channels by consecutive vergence movements — under the assumption that increasingly fine image detail has a decreasing amount of disparity. Indeed, it is a reasonable assumption that only extended objects protrude in depth [except for thin wires, which we often trip over), and fine surface textures in relief have shallow depth. Unfortunately, this ingenious scheme is not used by the human visual system. Among others, Mayhew and Frisby (1979) and Mowforth el at. (1981) have shown that high-SF RDS, with low and medium SF filtered out, can still elicit large vergence movements and yield fusion at large disparities during a brief Hash (when vergence movements are prevented). The finding that the Marr and Poggio (1979) noncooperative stereo algorithm based on vergence eye movements is not used by the visual system does not mean that a modified version could not be used, based on neurological couplings such that low-SF channels would monotonically reduce the disparity deviations of the high-SF channels with increasing SF. Such a model, based on a neuronal shifter hierarchy, appears much more complicated than a model based on a connected neural net exhibiting cooperative phenomena, but it could, nevertheless, exist. For instance, Anderson and Van Essen (19S7) have proposed models for such neural shifter networks in the eatly stages of the visual cortex, but until now no such shifter nets have been found. The fact that we see many objects al different velocities as equally sharp (without using a "shutlet" as in a movie camera)—even though we cannot follow simultaneously all these moving objects by eye movement—also argues for a cortical shifter mechanism that can compensate for the displacements of these drifting objects within limits. Of the many A l models of stereopsis, I mention a recent one by Lehky and Sejnowsky (1990) mainly because they discuss Ihe interesting debate in brain research between those who favor local and distributed representations. Retinal position is locally represented by a very large number of neurons thai fire when specific very narrow channels (positions) are stimulated. On the other hand, the many shades of color sensations are not conveyed by thousands of individually tuned neurons, but by the population of some broadly tuned channels for the three principal colors. Global stereopsis contains both systems of narrowly and of broadly tuned binocular disparity channels, as found by Gian Poggio (1984). I l
299
7S1 happens that Lehky and Sejnowsky postulated a model based on the distributed representation of the broad disparity channels. Here again one can see the robustness of A l models, since one could easily imagine that a dual model, based on local representation, might have worked equally well.
and to learn this logical function from scratch requires over 500 trials. Now, let us assume that Ihe number of trials for learning a task increases monotonically (linearly, polynomially, or exponentially) with the number of gates. A special mode of genetic learning (evolution) is to change the connectivity of neurons in the brain of a given species al each mutation. So, to develop the human brain with its 10 gates (synapses) might require eons, and even at a high mutation rate, the time of life on Earth is too short to have accomplished this feat by PDP techniques. This criticism of mine is based on Ihe monotonic increase of learning time with the number of synapses in a neural net; it would be groundless if one could prove that learning time stays invariant with the size of the network—an unlikely event. Indeed, Judd (1987) gives a proof that, in general, supervised training in connectionist networks is NP-coraplete; thus learning time goes up exponentially with the number of synapses (gates or nodes) in a network. Judd shows thai, even for very restricted networks, leaching (loading) time lo set Ihe connectivity of the gates requires polynomial time. Based on the learning (loading) time's being a polynomial or exponential function of I 0 gates, and thus orders of magnitude larger than the age of our universe, where the unit of each learning step is "one tick of the clock of mutation" for a given species, Ihe unlikelihood of Ihe evolution of connectionist neural networks seems to be correct- Because humans are particularly bad at Logical tasks of the exclusive OR kind, one could argue lhat Ibis criticism of PDP is irrelevant, since neither people nor machines are good at this task. Of course, I could have selected hundreds of tasks of form recognition in which people excel and PDP algorithms fail miserably. For a more extensive critique, see Pylyshyn (1984) and Fodor and Pylyshyn (19881; for discussion of the "top-down versus bottom-up paradigm" in neurophysiology, cognition, and PDP, see Churchland (1986). For an interesting discussion of evolution, brain research, and A l models, see Edelman(1989).] l!
D. The role ol artificial intelligence models, including spin-glass and connectlonist models, in psychobiology 1 have presented the noncooperative Marr and Poggio 11979) stereo algorithm to show that, after a psychological problem is clearly posed (i.e., the false-target problem in RDS), both cooperative and noncooperative models can offer a solution in a robust way. Furthermore, additional bottom-up processes (modules) based on occlusion, shape from shading, limits on disparity gradients, opacity, etc., help to disambiguate these already robust stereo models even more. Ultimately, only psychological and neurophysiological evidence can tell these models apart—that is, which kinds of models are certainly not used by the human visual system and which remain potential candidates. Since the human visual system evolved over millions of years, it is advisable that model builders, including practitioners of connected adaptive networks (e.g., Rumelhart and McClelland, 1986), be keenly aware of psychobiological insights before they invent some ad hoc schemes. By the same token, some insights from A l , like Ihe influential work of David Marr (1982) or the unexpected analogies belween human memory and spin glass drawn by Hopfield (1984), might inspire psychobiologists to search for some hidden structures or organizational principles in the brain. However, these insights by A l researchers are rather metaphors than deep analogies, and although they may be inspiring, the physicist should be warned Lhat the purpose of vision is most likely not to look for generalized cylinders ("stick figures' ) as Marr claimed (after all, vision evolved prior lo animals with axial symmetries), nor is the human brain—particularly the modules (transducers) of early vision—as similar to a spin glass or a parallel computer as devotees of parallel distributed processing (PDP) might like us to believe! However, it is intellectually exciting that the higher association areas (outside the scope of Ihis review) might be modeled by neural computation lhat exhibits flow to attractors of the network dynamics. Connect ion ism, particularly as represented by neural networks of the PDP kind (Rumelhart and McClelland, 1986), is popular among physicists who are not familiar with brain research but understand Hamiltonians, "strange attractors," thermal annealing, and so on. These physicists should be particularly on the alert. While Ihe invention of "hidden layers" and convergence theorems of learning through "back-propagation" have extended the processing power of "perceptrons" considerably, it should be noted that the simplest network with a hidden layer is the "exclusive OR" (with three gates), 1
IS
E. A strategic problem of 3D. mathematical models of perceptual constancies, and Lie germs Perhaps Ihe success of global slereopsis is owing in part to the likelihood that the problem "how can the visual system best
reconstruct
possible
3D
from
two
2D
representation
of
retinal
projections
ihe
a
environment?"
might prove to be a strategic question. Similar variants of this question, for example, "how can the visual system reconstruct
a 3D relief from
containing
the reflectance
a single
distributions
2D retinal
projection
of Lambertian
sur-
have been successfully attacked by Horn and Brooks (1985), Pentland (19861, and Bulthoffand Mallot (19901, among others. The reader may have noticed thai, instead of superficially reviewing thousands of topics in vision research, I have concentrated on a few, with emphasis
faces from
300
a specified
light source?,"
752
on, say, Ihe false-target problem of slereopsis, and through such select problems 1 have introduced the current state of my field. Of course, even in stereopsis I have had lo skip dozens of topics, from problems of stereo acuity to absolute depth. Because the latter is quite importani, 1 end this section with a brief discussion of absolute depth perception. Stereopsis, that is, depth perception based on binocular disparity, utilizes only the horizontal component of the disparity, and slight misalignments in the vertical direction cause a rapid loss of perceived deplh (Ogle, 1964; Julesz. 19711. Furthermore, slereopsis yields only relative depth. In machine vision one could use vertical disparities together with horizontal ones lo compute absolute deplh; human stereopsis, however, processes only horizontal disparities. Sensing the convergence angle of Ihe eyes is a very coarse approximation and cannot be used lo compute absolute depth by triangulation. However, with increasing convergence angles, the perceived depth also increases. [Fusing Fig. S wilh different prism angles will influence the perceived depth of the hovering square.] Here 1 should like lo mention the pioneering work of Hermann von Schelling (39561, who performed a very simple experiment (by looking at his outstretched palm and observing that, as he moved his head around the pivot of his neck, the perceived distance of the palm did not change) and made a sophisticated mathematical argument. He showed (by modifying Cay ley's invariant metric under projective transformations! that a unique metric exists, that is, a distance can be defined that stays invariant under any affine transformation, provided the space has a constant negative curvature (i.e., the space is hyperbolic}. I f F is a point in space lhat is fixed (a finger, or our palm), and R and L denote the focal points of the right and left eyes, and O marks the pivot of the neck, the tetrahedron FRLO should be the subjectively fixed frame for binocular depth perception, independent of head and eye movements. Since an afnne transformation changes any tetrahedron into another one, Schelling's assumption states that binocular vision should be invariant in a hyperbolic space, an insight whose truth has been verified by elaborate psychophysical experiments carried out by Luneburg (1950) and his followers. E have chosen this example to illustrate that a good mathematical background permits researchers in vision to design simple experiments, while with less sophistication the experiments in question appear so complex that the researcher avoids doing them or gets entangled in troubles. Similar perceptual invariance assumptions can yield important insights into the mathematical constraints with which Ihe visual system has to cope when viewing solid objects bolh wilh eye movements (Hadani el al., 1978, 1980) and wilh object movement (Ullman, 1977, 1979). Above I stated that stereopsis (alone) yields only relative depth, Absolute depth can, however, be perceived by humans as a result of internal feedback from the oculomotor system together with the visual system devoted to stereopsis and to monocular depth cues, an interesting problem beyond the
scope of this review. A much ignored although interesting theory thai tries to link global psychophysics to local neural mechanisms is the Lie transformation group model of neurophysiology, first proposed by W. C. Hoffman (1970) and developed by the psychologist P.C. Dodwell (1983). This theory is based on the observation that during the act of seeing the visual cortex annuls, to a great extent, the effects of affine transformations, phenomena called size and shape constancies in visual perception. Indeed, as we bring our outstretched arm closer to our eyes, the perceived size will not appear to change much, in spite of large changes in retinal size. Solid objects in rotation will appear unchanged in shape (within limits, since faces are poorly recognized upside-down). Circular disks at oblique viewing angles do not appear elliptical, but continue to be perceived as circular, and so on. Hoffman's theory is based on ihe mathematical theory of continuous groups (using Lie algebras, created by Soph us Lie in Ihe last century] to account for the perceptual constancies. He proposed lhat the visual cortex has built-in local Lie differential operators, called Lie germs. For details and criticisms the reader is referred lo a volume of papers edited by Paillard (1977). Whether the human brain uses Lie germs is a controversial topic. We have tried successfully on the Connection Machine (a massive parallel computer) an algorithm incorporating Lie germs (in each of the 64k parallel subunits) of several spatiotemporal affine transformations (Papathomas and Julesz 1989). Here is a typical case in which machine vision might use tools that solve the problem of perceptual in variances in space and lime, but the tools could very well differ from those utilized by brains. F. On "maturalional windows" and stereo blindness The topic of stereopsis leads us to the concept of "maturalional windows," one of the most importani discoveries in developmental psychology. It seems thai the genetic code is inadequate to carry all the information a highly developed organism needs, including Ihe fine-tuning to a particular niche of the environment. The best example is language acquisition. While the "deep structure" for learning a language is provided at birth — whether one is English or Chinese —the actual learning has to take place within a rather narrow "maturational window" in time. The missing of this window has dramatic influences, as attested by Ihe many stories of "wolf children" who could not be fully rehabilitated to learn, lhat is to understand and utter, speech after a "critical period-" One assumes that Ihe reason why we are not born wilh the syntax and semantics of a given language is the huge amount of information such a hereditary transfer might necessitate. Therefore it is important that even such a low-level process as detecting the orientations of contours can be dramatically influenced by environmental
301
753 manipulations within a specific critical period in the cat and monkey. For instance, Blakemore and Cooper 119701 and Hirsh and Spinelli 11970) restricted the visual input of young kittens to stripes of just one orientation. In contrast to normal kittens (for whom all orientations elicit neural responses), these kittens that underwent "environmental surgery" had functioning neurons mainly for the orientation they were exposed to during their critical period. For details on the effects of early visual experience, see Mitchell (1980) and the excellent textbook edited by Barlow and Mollon (1982). Humans with strong astigmatism that has not been corrected in infancy will have acuity defects at their original orientational defect even after their astigmatism is corrected by lenses (see Mitchell, 19S0). Experiments of this kind easl new light on the centuries-long debate concerning heredity and environment or "nature and nurture." Stereo blindness, which afflicts about 2% of the human population (Julesz. 1971), often has its origin in strabismus (cross-eyedness), which prevents binoculardisparity-luned neurons from receiving correct stimulation. This incorrect stimulation leads to a condition of "amblyopia ex anopsia" ("lazy eye"), which means That one eye becomes dominant, while the other becomes practically nonoperative. After surgical intervention to correct strabism, the child usually remains stereo blind, so the surgery is merely a cosmetic intervention. The reason why surgery does not, in most cases, restore stereopsis is that it is done too late, after the critical period during which stereopsis becomes functional. In order to determine the critical period for stereopsis in human infants, two steps have been taken. First, the technique of random-dot stereograms was extended to a new class of stimuli called random-dot correlograms (Julesz and Tyler, 1976), which are quite different from random-dot stereograms. A dynamic random-dot correlogram alternates between binocularly correlated and uncorrelated visual noise. A special case of uncorrelation is negative correlation (where the left and right images are complements of each other). The advantage of these stimuli is that, regardless of the position of the observer's head, the stimuli are either correlated (stimulating binocular neurons) or are not correlated. Furthermore, humans can detect changes from binocular correlation (order) into binocular uncorrelation (disorder) in as brief as 2 msec, while the reverse, detecting the change from disorder into order takes 20 msec. We named this entropylike behavior "neural entropy" or "neurontropy" (Julesz and Tyler, 1976). The second step was the use of dynamic random-dot stereograms and correlograms to evoke visual potentials on the human skull (Lehmann and Julesz, 1978; Julesz el al., 1980). Dynamic random-dot correlograms are a robust method, which was used by Miezin et at. (19SI) to evoke visual potentials in the monkey. More importantly, dynamic RDS and correlograms were used to determine the onset of functional stereopsis in human infants by Braddick et al. (1980) and Petrig et al. (1981). These studies re-
veal that evoked potentials appear first in normal human infants at 3.5 months after birth. Similar results were obtained using RDS but with behavioral techniques by Fox el al. (1980). In adults, the evoked potentials to a dynamic RDS portraying a cyclopean checkerboard that oscillates in depth at a rate of / Hz appear as a square wave with a period of f Hz, while a similar dynamic correlogram elicits a square wave at a frequency of If Hz (Julesz et al., 1980). This doubling of frequency appears in human infants at about 6 months, as shown by Petrig 11980). It seems that functional binocularity appears at 3,5 months, while some more sophisticated binocular process (perhaps stereopsisl emerges at 6 months in human infants. For a discussion of electrical activities in human brains, including evoked potentials in infants, I recommend the thorough and informative monograph by Regan (1989). Gian Poggio (personal communication) attributes the frequency doubling for correlograms to the existence of two classes of binocular neurons, which fire either for correlation or uncorrelation, respectively. Stereo deficiencies, including stereo blindness, are not serious handicaps, but there are several professions for which stereopsis is a must. For instance, fiuoroscopists routinely perform Ihe catheterization of the heart while looking at 3D screens of x-ray machines, they have for decades used the unfakeable RDS tests to weed out inappropriate applicants for Ihis specialty. For astronauts, as for quality inspectors (who inspect VLSI chips under stereo microscopes for imperfect ions), good stereopsis is a job-related requirement. The combination of dynamic RDS with evoked potentials now permits a fast detetmination of functional binocularity in human infants. It is hoped that such an early diagnosis will lead to early surgery to correct for strabismus before the maturational window closes, which in turn will result, in many cases, in the restoration of stereopsis.
IV. T E X T U R E DISCRIMINATION AND F O C A L ATTENTION
A. A brief note on motion perception The advances in global slereopsis made possible by RDS were paralleled by advances in motion perception using random-dot cinematograms (RDC). Since motion phenomena require demonstrations for the nonexpert that do not lend themselves to the printed page, I shall skip them here and turn to human texture discrimination. The reader interested in motion perception can find excellent reviews by Anstis (1978), Van de Grind et al, (1983), Hildreth (1984), Adelson and Bergen (1985), Nakayama (1985), Regan and Beverley (1985). Watson and Ahumada (1985), and Cavanagh and Mather (1989), among others. Since Ihis review is intended for physicists, I draw special attention to the pioneering work on motion perception in the By by the physicist Werner
302
754 Reichardt (19611 and co-workers (Reichardt and Egelhaaf, 19881. He found that correlation like mechanisms are used in motion perception in the fly's visual system; it turns out that similar Reichardt detectors are used in the human visual system loo (van Santen and Sperling, 19851. R D C exhibit the same false-target problem as R D S (see Fig. 7) except that, for the latter, one has to search only horizontally, whereas for the former, disparity is a vector and the search has to be carried out in two dimensions. Therefore RDC present a more complex problem. Reichardt and Egelhaaf (19881 pointed out mathematically that the ambiguity of motion in an aperture (called "the aperture problem"), contrary to common belief, can be locally solved by two correlation-type motion detectors. Similarly to stereopsis, motion perception exhibits cooperative phenomena (particularly hysteresis), as shown by Chang and Julesz (1984) and Williams el al. (1986). With RDS several motion phenomena can be produced that differ from classically portrayed apparent (stroboscope) motion, thus demonstrating that binoculardisparity-tuned neurons of motion have different time constants from monocular neurons of motion (Julesz and Payne, 1968). B. Preattentive texture discrimination and focal attention I turn now to preattentive texture discrimination (also called texture segmentation), a problem I posed almost simultaneously with the RDS paradigm (Julesz, 1962). We discussed how global stereopsis facilitates object segmentation based on binocular disparity differences. For objects in the same deplh planes, segmentation can be aided by differences in surface textures. We restrict ourselves Lo textures in two dimensions (i.e., the monocular retinal projections of real-life textures in relief, such as bark on a tree, grass or plowed fields viewed from a distance, textiles, microscopic slices of biological tissues, etc., and artificial 3D arrays of repeated or randomly scattered elements such as wallpapers). Effortless or preattentive texture discrimination (segmentation) was illustrated by Fig. 2. There we discussed the difference between preattentive pop-out of certain textures from among others by a parallel neural mechanism (which is independent of the number of texture elements) versus serial element-by-element scrutiny by the "searchlight" of focal attention. This dichotomy of pre attentive-attentive processing in visual perception is very helpful in explaining texture discrimination. As we have shown (Sagi and Julesz, 1987), texture gradients pop out only i f the element density is above a critical value. Even then, one can determine only the locations of the gradients in a brief Hash (e.g., where are the horizontal or vertical line segments embedded in a diagonal array of line segments); to identify them (i.e., whal they are: horizontal or vertical) requires scrutiny by focal attention, which depends on the
number of texture gradients (Sagi and Julesz, 1985). We also determined the aperture of focal attention ("searchlight"), which increases with eccentricity (Sagi and Julesz, 1986). In the Introduction we discussed how the searchlight of attention scans rapidly, without eye movements, at 30-60 msec/item depending on the visibility of the texture gradients, and how some parallel mechanism seems to facilitate serial search (Krose and Julesz, 1989; Wolfe and Cave, 1990). Particularly interesting are the findings of the cognitive psychologist Anne Treisman and co-workers (Treisman and Gelade, 1980; Treisman and Paterson, 1984), who postulated a "feature integration theory" of vision. According to this theory, "disjunctions" of features pop out. but "conjunctions" of features require serial search. For example, red horizontal line segments pop out from green ones, but in a mixture of red horizontal and green vertical line segments Lo find, say, red vertical line segments requires serial search (which depends on the number of elements). The psychological finding that color and line orientation are not coded simultaneously is quite unexpected from classical neurophysiological evidence [although recently Livingstone and Hubel (1987) found nonoriented blobs luned to color surrounded by the usual orientation-tuned neurons that were not sensitive lo color in area V I of Ihe monkey cortex]. Interestingly, if a third feature, such as depth or motion, is added to color and orientation, this "conjunction" of three features pops out preattenlively fNakayama and Silverman. 1986). Treisman's theory, though superficially resembling my "texton theory" of preattentive vision, differs from it essentially. While Triesman does not specify here stimuli and often uses stimuli whose elements are distant from each other, my lexton theory can only be applied for textures, which in turn requires that the elements be dense. IFor critical distances within texture elements and critical distances between texture elements see Julesz, 1986b. For olher work on texture density see Nothdurft, 1985, and Sagi, 1989.) I am skeptical that any theory can be postulated for distant elements at present, because they are influenced by Gestalt organizations in unknown ways, while the perception of textures is much simpler.
C. Human texture segregation: when the whole is less than the sum ot its parts Before 1 discuss the recent developments in the segmentation of textures by spatial filters, a brief survey on the evolution of the texton theory might be of interest to the physicist. In 1962 I realized that effortless texture discrimination might be a much less complex problem than the perception of form (Julesz, 1962), though it exhibits challenging properties. When the pair X and L is briefly presented, its discrimination is somewhat stronger than that of the pair L and T, bul not by much, so it is not obvious why aggregates of X's and L's do pop out while aggregates of L's and T's do not. Texture elements
303
755 thai segregate in isolation and remain segregated when they are pooled to form textures comprise a far less interesting case than elements that become harder to discriminate when pooled. Indeed, it is not obvious how aggregates of texture elements that segregate in isolation become less and less discriminable as the number of elements in the aggregates increases, Clearly, it is not interesting when a pair of elements in isolation is indistinguishable and remains so in aggregates. It is more interesting when a pair that is distinguishable in isolation becomes more and more discriminable as the number of elements increases in the two aggregates. For example, Caelli and Julesz (1979) took textures composed of dipoles (i.e., two nearby dots) as elements; dipoles in one texture had orientations in 0 1 range, while in the other range of 02. We showed both theoretically and experimentally that i f A = 0 2 - 0 I , and N is the number of dipoles, the psychophysical function at the visibility threshold of texture discrimination is r
logA = logA-1og(c-A)+fc where c and A; are experimental constants. Thus texture discrimination is a global function that increases monotonically with the number of texture elements and with the difference between the two dipoles' orientation range. It would be nice to develop a model that would predict why and how, say, discrimination between one T and L decreases with increasing numbers of elements, a problem to be discussed next. There are several possible reasons for elements that are strongly discriminable in isolation lo become less and less discriminable as Ihe number of elements in the two aggtegates increases. As of now 1 have identified three, based on the type of texture pair. One is shown in Fig. 11, Here an S- and a ID-shaped element can be easily discriminated from each other [Fig. 11(a)], but the tex-
ture pair formed by their aggregates is indistinguishable [Fig. 1Kb)]. The reason for this phenomenon is probably lateral inhibition between adjacent elements. We have to shift our small disk of focal attention to remove this lateral inhibition, thus enabling scrutiny. Furthermore, subjective contours close gaps (Williams and Julesz, 1991), to be discussed later. The second kind of indistinguishable texture pair is between, say, an R and its "mirror image dual," as shown in Figs. 12(a) and 12(b). In isolation the R and its mirrored dual can be discriminated in Fig. 12(a); however, in the laboratory this discrimination requires timeconsuming "mental rotation" (Shepard and Metzler, 1971), which depends on the amount of angular difference between the pair. It appears that texture pairs like this [Fig. 12(b)] cannot be rotated all at once, only element by element. Indeed, one can discriminate the elements of the texture pair only by inspecting each element one at a time and trying to decide whether it is an R or its mirror image. A third kind of indistinguishable texture pair is depicted by Fig. 13, where in isolation the T-shaped four-disk pair seems very different from the rectangle-shaped fourdisk pair. Yet their aggregates yield indistinguishable texture pairs (Julesz, 1975). This texture pair is generated by the "four-disk method," which yields textures with identical second-order statistics [hence with identical autocorrelations and Fourier power spectra (Julesz er al., 1978)]. The reason for our not being able to discriminate between these texture pairs appears to be the many irrelevant shapes that are formed by disks at the boundaries between adjacent texture elements, which in turn mask the dual elements. These demonstrations already show that the "law of superposition" does not apply for preattentive texture segmentation (since aggregates of discriminable element
Jl & %>$iS3s^^ei^(?'
ID %
%
^\ % & t
(a)
^
(j. |
ss (J W *
*5
0
0
\^ & % *
^
<2 $$s %
Oi §s
&
IQ
6f & 0 & si m
(b)
% % Oi \
m
%
%
(O
FIG. 11. Demonstration of a texture pair cnmposed of S- and 10-shaped elements, which are discriminable in isolation, as shown in (a), but indistinguishable in an array, as shown in (b). (c) is similar to (b), except that only one target is presented, and to detect this target ekmenl-by-elemem scrutiny is required. From Julesz (19B1).
304
7SS
* * *r
* * * ** * * *
4
* * * * s.
"*» •» »
**
4
+• t
* *
4
•*
"
h
«
*
*
t* * 4
*
f
***
* *
4
**
* 4
4 *
4
4
4
#
**
***
*
* 4
«•
*
4
* 4
<-
t
(b)
(a)
F I G 12 Iso-second-order texture pair, composed of / f t and their m i r r o r image duals; in isolation (a) the elements a r t discriminable, while in an array ft) ihey appear indistinguishable. F r o m Julesz 11931).
(a)
v / • 7 / * / r i • / > t •:. s* A v i • :•• X V ;? T ^ v >> r v •/•/!•> < ^ >vv< v YY' f • * X M
i . N r .
<.:. 71 i * v * 0 ?
i < /»o::
\
•
-v-w >
/•A .vr > * * * c • «*«*^ * <
F I G , 13. Iso-second-order texture pair generated by the "four-disk method". indistinguishable in lb>.
305
F r o m Juiesz I1975J illustrated in ial. T h e pair becomes
757 1978; Julesz, 1981, 1984). [It can be mathematically proven that the four-disk method of Caelli, Julesz, and Gilbert (1978) is the only one using identical disks that can generate iso-second-order texture pairs in the Euclidian plane, thus permitting a thorough search for dual elements whose aggregates might pop out.] Figure 14(a) demonstrates the first iso-second-order discriminable texture pair we found, using the four-disk method. This figure, together with 14(b), and 14(c), which were generated with the help of the "generalized four-disk method" (in which the disks are replaced one by one by specific symmetric shapes), depicts iso-second-order texture pairs that are preattentively discriminable. Discrimination is based on local features which we named
pairs become indiscriminable), and therefore no linear spatial filters can model human texture discrimination. This essential nonlinearity of human texture discrimination will be demonstrated later in Fig. 17. Before 1 review recent work with nonlinear spatial filters, 1 briefly discuss theoretical studies aimed at generating stochastic textures with identical second- and third-order statistics, which led to the concept of texlons—the basic perceptual units (perceptual quarks?) of preattentive texture discrimination.
V. FROM TEXTONS TO NONLINEAR SPATIAL FILTERS
"quasicoliineariiy,"
A. A brief outline ot the lexton theory of texlure discriminalion In 1962 I asked a combined mathematical and psychological question that has kept many mathematicians and psychologists busy ever since (Julesz, 1962). Because 1 knew that texture pairs that differed in their first-order statistics would be effortlessly segregated (based on differences in tonal quality) and assumed that differences in second-order statistics could be distinguished from each other (based on differences in granularity), I wanted to study textures with identical Mh-order statistics but different W + Dth-order statistics. [Here I define M i l order statistics as the probability that the vertices of an 'iV-gon" (e.g., a hexagon, pentagon, etc.) thrown randomly on a texture fall on certain iV colors.] I wanted to determine the highest N that still yielded texture segmentation and wanted to know what perceptual quality would accompany such discrimination. For example, would texture pairs with identical second-order statistics (hence identical first-order statistics and identical Fourier power spectra) be discriminable, and what would the perceptual difference be? Surprisingly, at that time mathematicians did not know how to create such constrained stochastic textures, but from 1962 to 1975, Slepian, Rosenblatt, Gilbert, Shepp, and Frisch were instrumental in creating iso-second-order random texture pairs whose elements in isolation appear conspicuously different, yet as textures cannot be told apart. The indiscriminable texture pairs depicted in Figs. 12 and 13 were obtained by these efforts. It seemed that isosecond-order textures were so severely constrained globally that the visual system could not tell them apart. However, in 1977 and 1978, colleagues T. Caelli, E. Gilbert, and J. Victor helped me to invent stochastic texture pairs with global constraints of identical second-order (and even identical third-order) statistics that yielded preattentive texture discriminalion based on some focal conspicuous features, which I later called textons. Luckily, now that we know whal textons are and their role in vision has been clarified, the reader need not take the tortuous mathematical path that led to their discovery (for details see Caelli, Julesz, and Gilbert, 1978; Julesz el a!., L
"corner,"
and
"closure"
(Caelli,
Julesz, and Gilbert, 1978). Figure 14(d) shows iso-thirdorder textures (Julesz, Gilbert, and Victor, 1978) with tbe property that any triangle thrown on these textures has the same probability of its vertices falling on the same colors (however, the vertices of probing 4-gons will have different probabilities). As the reader can verify, discrimination is effortless and is obviously not due to computing differences in fourth-order statistics, but rather to elongated blobs of different aspect ratios and orientations. What these textons really are is hard to define. For instance, in Fig. 14(a), besides quasicollinearity there are also more white gaps between these elements, giving rise to anlitextons. As I pointed out (Julesz, 1986) it is not only the black (white) textons whose gradients yield texture discrimination but also the white (black) spaces between them, which act as textons too. In essence, we found that texture segmentation is not governed by global (statistical) rules, but rather depends on local, nonlinear features (textons), such as color, orientation, flicker, motion, depth, elongated blobs, and coJIinearity. to name the most conspicuous ones that are both psychophysical^ and neurophysiologically accepted as being fundamental. Some less clearly defined textons are related to ends of lines or terminators, which occur in the concepts of "corner" and "closure" and are hard to define for halftone blobs. Particularly important is the realization that—contrary to common belief—texture segmentation cannot be explained by differences in power spectra. On the other hand, it became obvious that instead of searching for higher-order statistical descriptors, the visual system applies some local spatial filtering followed by some nonlinearity, and the results must be averaged again by the next spatial filter stage. This is depicted in Fig. 15, which illustrates how a Kuffler-type unit (instead of a Mexican-hat-function profile, a simpler spatial filter of 2X2 pixel center addition with a 2-pixel-wide surround annulus of subtraction, as shown in the inset) acts on the iso-third-order texture pair of Fig. 14(d), followed by a threshold-taking device (Julesz and Bergen, 1983). When viewing the output of this nonlinear spatial filter in Fig. 15, our visual system performs a second spatial filtering by separating the two areas of different lumi-
306
75a
FIG, 14. Preattentively discriminable iso-second-order and iso-ihird-order texture pairs, (a) Iso-second-order texture pair that is indiscriminable due to the local conspicuous feature (lexton) of "quasicoliineariiy." (b) Iso-second-order texture pair Thai is di&crimmablc due to the local conspicuous feature (texton) of "corner," (ci Iso-second-order texture pair that is discriminable due to Ihe local conspicuous feature (lexton) of "closure." From Caelli, Julesz, and Gilbert (1978). (d) Iso-third-order texture pair that is discriminable due to the local conspicuous feature (texton) of "elongated blobs of specific orientation, width, and length." From Mes2, Gilbert, and Victorfl978).
307
758 nance distributions (that were obtained by the threshold taking). One problem with such nonlinear simple filters is their inability to account for the asymmetry problem of human
F I G . 14.
308
texture segmentation. It is known (Julesz, l98l;Gumsey and Browse, 19S7; Treisman and Gormican, 1988) that very often a given texture A pops out more strongly from a background of B than B does from a background of A ,
{Continued).
760
as shown in Fig. 16(a). For many years 1 was worried rhat the asymmetry problem of preattentive texture discrimination might depend on top-down processes and complex figure-ground phenomena. Therefore I am glad to report that the asymmetry effect in Fig. 16 can be explained (Williams and Julesz, 1990, 1991) by assuming that the nonlinear operation is the subjective contour phenomenon that "closes the gaps." When we discussed Fig. 6, 1 pointed out that subjective (also called illusory) contours are extracted in V2 and therefore belong to early visual processes. Figure 16(b) yields a similar asymmetry of texture discrimination to that of Fig. 16(a), even though the orientation of the gaps is not jittered. We shall return to this demonstration in the next section. B The asymmetry problem of texture segmentation and nonlinear spatial filters In 1988, several texture segmentation algorithms were developed based on the use of linear spatial filters fol-
io F I G . 15. Demonstration of how a simple local linear filter followed by a nonlinearity (threshold-taking) can segment Ihe isothird-order texture pair of F i g . 14(d). F r o m J u l e s ; and Bergen (1983).
lowed by squaring or some other nonlinear operation (Bergen and Adelson, 1988; Voorhees and Poggio, 1988). (For a definition of spatial filters, e.g., the Laplacian of a Gaussian, or Gabor filters, see Sec. II.B.) For a critique showing that linear spatial filters cannot segment textures, see Julesz and Krose (1988) and Julesz (1990c), Recently Williams and Julesz (19911 showed the nonlinear behavior of human texture discrimination as depicted in Fig. 17. Here the nondiscriminable iso-secondorder texture pair, invented by Caelli, Julesz, and Gilbert (1978), is shown on the right side. (This texture pair is one of the rare cases that have iso-second-order statistics without having to rotate the texture elements randomly.) We were able to decompose this texture pair into the sum of a highly discriminable texture pair and a nondiscriminable texture pair, as shown on the left side. The fact that a discriminable texture pair becomes nondiscriminable when a nondiscriminable texture pair is linearly added shows convincingly the violation of the law of superposition for lexture discrimination. More recently Fogel and Sagi (1989) and, independently, Malik and Perona (1990) developed texture segmentation algorithms based on the use of local spatial filters (oriented Gabor filters) followed by a quasilocal nonlinear operation (simple squaring in Fogel and Sagi's version and some inhibition between neighboring elements in the Malik and Perona algorithm), with a second spatial filter for final segmentation. It was most impressive that this approach emulated human texture discrimination performance as measured by Krose (1987), but still could not account for the asymmetry effects. Therefore it is of great significance that Rubenstein and Sagi (1990) extended their mode) by determining the variances of the local lexture elements' distributions after the nonlinear stage and found these variances asymmetric, particularly when the orientation of the elements was jittered, mimicking human performance. Their model could account for the textural asymmetries reported by Gumsey and Browse (1987) and probably will be able to handle some other asymmetries of the kind shown in Figs. 18(a) and 18(b). In Fig. 18 a typical input-output pattern of the Rubenslein and Sagi (1990) algorithm is presented as it segments a lexture pair (A in B) and its dual (B in A). It is mosl heartening lhat even the textural asymmetry effects lhat seemed to be based on figure-ground reversals—which in turn depended on unknown topdown processes—can be successfully explained by boltom-up processes modeled by relatively simple nonlinear spatial filters. The Rubenstein and Sagi (1990) model can account for the asymmetry problem by assuming that jitter of line orientation accounts for increase in variance of their filter's output, hence increase in texture discrimination asymmetry, However, the demonstration of Figs. 16(a) and 16(b) clearly shows that in general the asymmetry problem of Lexture discriminalion does not depend on orientalional jitter. Indeed, recently. Williams and Julesz (1991) extended the texton theory Lo include illusory con-
309
?6" psychology. The theory of trichromacy states that any color can be matched to a combination of three basic colors, red, green, and blue, such that the boundary between the selected color and the combination colors becomes minimum lor disappears without scrutiny). (As a matter of fact this theory, postulated by George Palmer in 1777, can be regarded as the first scientific atom theory. years before Dalton introduced atoms into chemistry.)
lours and "fill-in" phenomena between gaps and nearby elements, which could be regarded as antitextons. The filling in of the gaps by subjective contours can account for the asymmetry effect shown in Figs. 17Ial and 17(b), and the fill-in phenomenon between tenure elements can explain many other asymmetries, Antitextons together with the textons extend the theory of trichromacy, the only real scientific theory in
oooooooo oooooooo oooooooo oooooooo oooooooo oooooooo oooooooo oooooooo
oooooooo oooooooo oocooooo oooooooo oooooooo oooooooo oooooooo oooooooo (a)
oooooooo oooooooo ooooccoo oooooooo oooooooo oooooooo oooooooo oooooooo
cccccccc cccccccc cccccccc cooccccc cooccccc cccccccc cccccccc cccccccc (b)
FIG. 16. Asymmetry of texture discrimination. Ill The perception of gapped octagons among closed octagons yields weaker discrimination than vice versa, lb) Similar lo (a) but the position of the gaps is not jittered. This does not reduce Ihe asymmetry effect. From Williams and lulesi ( I N I , in press).
310
702 V
"1:
^
\
S
\
\
\
\
T(
\
\
\
\
S
•>, it
S
1,
\
\
Il
>l
^
^
V
'\
f.
%:
%
%
%
L.
L
«i
"H "S
*
y
L
L
*
>
ii
V
V
L» 1.
s
s
^,
"S
%•».<.
>i
•>. "S
:
t,
*\ ^
1,
-t,
%
^
\ ^ \ \ \
^
\
\
^
l
\ \ \ \ \
*\
\
^
*\
\
'\
L
i
L
i l
^
\
\ \ \ \ \ *\ l
\ \ ^
*\
*\
F I G . 17. D e m o n s . r a t i o n of [he nonlinearity of h u m a n Texture discrimination. A d d i n g a nondiscriminable Texture pair Co a highly discriminable texture pair renders the latter nondiscriminable. thus violating the law of superposition.
F r o m W i l l i a m s and Julesz
( 1 9 9 l in press). f
When I introduced textons into psychology I wished to extend trichromacy to encompass textures as well as colors- 1 wanted to know whether any texture could be matched to a finite (and not too large) number of textons such that the boundary between any textural array and an array containing a mixture of textons would perceptually disappear without scrutiny. It seems now that this can be achieved. The fact that the gamut of colors can be matched by just three colors is in itself amazing. The finding that the infinitely richer variety of 2D textures could be matched lo a mixture of a finite number of textons is even more unexpected! In Fig. 17 it was shown that the law of superposition does not apply for texture elements. Whether the combination of textons themselves is a linear or nonlinear operation is not yet known. Howard L . Resnikoff (1987a) in his interesting monograph The Illusion of Reality, devotes an entire chapter lo an early version of my texton theory and argues For the linear superposition of textons. Whether linear superposition still holds for the new texton theory, incorporating antitextons, remains to be seen. For the author, who spent much oF his scientific career in search of the elusive texton, it is anticlimactic, yet most satisfying, to find that quasilocat spatial filters can extract texton gradients without having to specify complex concatenation rules between adjacent textons. (I have no doubt that in the near future such filters will mimic human preattentive texture discrimination by incorporating several perceptual operations from the formation of subjective contours to the filling-in of gaps.) The reader familiar with speech research will recognize the similarity between "phonemes" and "textons."
311
While phonemes were never well specified, and complex computer algorithms are now used to cope with the many rules at their various concatenations in order to segment speech, nevertheless, the rudely defined phonemes permitted the development of phonetic writing, one of the great discoveries of human civilization. Had the development of phonetic speech coincided millennia ago with the invention of supercomputers that could automatically segment speech and talk, the skill of writing might never have developed. Of course, the fact that our voice organs limit the number of phonemes to a few dozen contributed to their universal acceptance. Similarly, the main insight from the texton theory was that, of the infinite variety of 2D textures, only a limited number of textons have perceptual significance and are evaluated quasihcalty in effortless texture discrimination. 11 use the term "quasilocal" instead of "local," because line segments, closed loops, corners, etc., have some finite dimensions.) Even though superfast computers will soon perform automatic texture segmentation, practitioners of visual skills—painters, designers of instrument panels or advertisements, directors of movies or TV shows—can benefit from the texton theory by the enhanced ability it gives them to manipulate the viewer's eye. Indeed, some of the great artists have instinctly known how to create a strong texton gradient to capture attention or create a texton equilibrium for which time-consuming scrutiny is needed to discover the hidden images. I end this section with a mention of the work of Enns (19E6), who made up arrays of little 2D perspective cubes with targets (cubes) biased to be perceived in one kind of 3D depth organization amidst cubes biased in the dual
763
312
764
3D organization. This depth from 3D perspective cues yields preattentive texture segmentation ("pop-out"), with the implication that, in addition to the textons of brightness, color, orientation, and aspect ratio of elongated blobs, flicker, motion, and stereopsis, even perceived depth in 2D perspective drawings might act as a texton. In my belief, some simple, quasilocal rules of 3D perspective, occlusion, transparency, etc., are probably utilized, without the need to invoke top-down processing. C. Recent psychological and neurophysiological tin dings in texture discrimination Perhaps the most important implication of the texton theory was its division of human vision into preattentive and attentive modes of action. Certain text u re-gradientlike detections could be performed in parallel without scrutiny, while some other tasks that required identification needed serial search by attention. Recently, Braun and Sagi (1990) lent support to the "twovisual-system" concept by showing that while an observer's attention was loaded (by being asked to identify a letter) it was possible for the observer to carry out simultaneously the detection of texture gradients. Other recent neurophysiological studies seem to support the texton-gradient notion of our perceptual studies. Van Essen el al. (1989) studied the responses of single units in visual areas V I . V2, and MT of the macaque monkey to stationary and moving patterns. In V I and V2 the presence of a static texture surround (e.g., an array of parallel needles) somewhat decreases the response to a central texture element (single needle) within the classical receptive field i f the orientation of the needles in the surround is perpendicular lo that in the center. I f they are parallel, the neural response is greatly reduced. Another neurophysiological finding is that of Robert Desimone and co-workers (Moran and Desimone, 1985; Desimone and Ungerleider, 1989), who located neurons in V4 whose firing for certain trigger features changes in accordance with the focal attention of the monkey. D. Learning effects in early vision One of the main themes throughout this review has been the phenomenological richness of early vision. Without cognitive and semantic cues, rather complex feats of false-target elimination can be performed in stereopsis and movement perception, asymmetry of texture discrimination takes place, subjective contours are formed, and so on. Even long-term memory effects occur in early vision. We have discussed some of the hysteresis effects that accompany the cooperative phenomena of global stereopsis and motion perception. Hysteresis is one of the simplest memory effects, in which some action modifies the outcome of a later response. For instance, Fender and Julesz (1967), using binocular retinal stabilization (by giving the subjects close-fitting contact lenses with mirrors attached), showed that a RDS had to be
brought within Panum's fusional area (i.e., within 6 min.arc alignment) lo obtain fusion. But after fusion, the left and right images could slowly be pulled apart by as much as 120 min.arc without breaking fusion. So, the fusional area depends strongly on the prior perceptual states. Another learning effect can be experienced when one first tries to fuse a RDS with large binocular disparities (Julesz, 1971 gives several demonstrations). At first, it might take minutes to achieve fusion, hut even years later one can do it quite easily. This is not real perceptual (cortical) learning, but tat her a ptocedural (cerebellar) Learning. When fusing RDS with large disparities, the novice is trying large convergence movements to bring the corresponding areas of the RDS into Panum's fusional area, and it is this unconscious learning of proper vergence movements that is remembered years later (Julesz, 1986a). Some other teal cortical Learning phenomena of global stereopsis are also reviewed in the previous reference. Here I give only two examples of Learning effects in p real ten five texture discrimination: First, when one presents some indistinguishable isosecond-order texture pairs that, however, are composed of element pairs with different convex hulls, after several hundred trials they can be effortlessly discriminated (Julesz, 1984). Second, and even more interestingly, Kami and Sagi (1990) report a remarkable long-term learning effect in simple texture discrimination tasks were learning seems to be local in a retinotopic sense. Whal has been learned must be relearned for each different area of the visual field. Surprisingly, [hough learning is specific for target location, it is not specific for target orientation, but rather for background element orientation. These authors briefly presented, in an array of horizontal Line segments, a few targets of adjacent line segments that were tilted from the horizontal orientation. Wilh small lilts, it took several sessions lo deled these targets correctly. This improvement was retained for the next sessions in the same retinal quadrant but was not transferable to other retinal quadrants. Changing the orientation of the targets (from left oblique to right oblique) had no effect on the learned performance. However, chaning the orientation of the background array (from horizontal to vertical) obliterated learning. These plasticity effects are of great interest, and it seems thai perceptual learning in early vision might be a useful tool in understanding the mysteries of human memory. I conclude this review with some perceptual phenomena that are not just bottom-up, bul require semantic memory and other top-down processes for their explanation. I have emphasized throughout this review that most of the perceptual processes in global stereopsis, motion perception, and lexture discrimination are essentially bollom-up; their linking to present neurophysiological results obtained in the early cortical areas of V I , V2, V3, V4, or MT is now possible. This is also in agreement with David Marr's view of computational vision's being basically boltom-up. However, in a recent short mono-
313
IBS
graph
Visual
Processing:
Computational
Psychophysical,
Roger Watt (1988) argues for algorithms in early vision that are under the control of high-level processes and memory. Indeed, in cognition there are many perceptual phenomena that depend on high-level processes, including semantic memory. A well-known example in cognition is the "wordsuperiority effect," in which the recognition of certain letters is superior when they are contained in an English word to when they are in a nonsense word or presented in isolation. This makes a good deal of sense, since recognition of letters and words is surely a high-level semantic process. However, as Naomi Weisstetn and Charles Harris (1974) have shown, the same phenomenon exists in visual perception, where they call it the object-superiority effect. The detection of a line segment of certain orientation was greatly improved if the segment belonged to a line drawing that portrayed a 3D object; it deteriorated i f the segment belonged to a random line drawing and was the worst if the line segment was shown in isolation. and
Cognitive
Research,
Of coutse, object and form recognition are more complex, high-level processes in which semantics and Gestalt organization play a prominent role. Therefore the experiments of Gorea and Julesz 11990) are of special interest. We convened the object-superiority effect from an identification paradigm into a detection paradigm, as follows: We presented an array of oblique line segments into which three horizontal and one vertical line segments were inserted- These four nonoblique line segments were clumped either in random fashion or representing a primitive human face (two horizontal lines representing the eyes, the vertical line segment between the eyes representing the nose, and the bottom horizontal line segment portraying the mouth). Observers were not aware that occasionally a face was presented, and were only asked to detect any line segment that was not oblique. Surprisingly, observers detected the horizontal and vertical line segments significantly better when they belonged to the face than when they belonged to a random clump (or to a four-line-segment symmetric pattern) that was not a face. I have always assumed that the detection of a line segment in a texture (based on a texture (texton) gradient between adjacent orientation differences) was a simple parallel bottom-up process. And here is a case in which even such a simple perceptual task might depend on top-down processing! I say "might" because the effect is very small (though statistically significant) and only four observers were tried. Because of the importance of this experiment, 1 would welcome the attempt of others to repeat this study of ours with more observers and perhaps some other experimental design!
VI. CONCLUSION A physicist reader who only glanced through this article might be surprised by the Jack of explicit mathematical equations. Obviously I did not want to bore the
314
reader with the difficult proofs of iso-Mh-order texture generation, including their ergodicities. Furthermore, the internal structure of the cooperative computer algorithms modeling stereopsis or texture segmentation are much more complex in detailed mathematical notation than the usual differential equations of the Maxwell or Schrodinger kind. Furthermore, physicists have a knack of ignoring "dirty" problems, such as computing the shape of a puddle of spilt milk on a kitchen floor (a favorite example of George Sperling, 1978). [When they are forced to do so, to compute, say, the shape of a plasma in a magnetic bottle, they are confronted with the same difficulties as their colleagues in psychobiology.] Indeed, <• "thought" might correspond to the "shape of a puddle" of cooperating neural pools of a certain activity. I hope I have made it clear that progress in psychobiology is not hampered so much by the lack of mathematical tools, as by our inability to find the proper levels of complexity for describing mental phenomena. It is my opinion that whenever a canonical problem is raised in brain research, mathematical problems become simplified or vanish. Strategic insights might well be gained, not so much as the result of a new mathematical tool that will enable us lo handle many-body nonlinear systems, but as the result of a novel way of looking at some emerging property at a proper level of complexity [hat we have missed so far! Indeed, in modern molecular biology much of the action can be understood al a phenometiological level as proteins fold and unfold, without having to compute the many-body problems of van der Waals forces. Even the genetic code has turned out to be a redundant three-letter code, instead of some complex error-correcting code envisaged by information theorists. This does not mean that some insights from theoretical physicists, particularly from experts in complex adaptive systems, might not be crucial for psychobiology. Luckily in early vision we do not have to wait for such insights. The modular nature of early vision enables us to attack problems with adequate psychophysical, neurophysiological, and mathematical tools at our disposal. Obviously there are many interesting problems—from perceptual learning to the kind of information one can collect without focal attention—that will keep us busy for many years to come. The only unanswered question is whether such a reduced psychobiology of early vision is an adequately interesting subfield of human vision for intensive study. In my opinion, i l is a promising beginning until novel techniques—such as behavioral studies following micro ablations of targeted brain tissues using monoclonal antibodies, or direct observations of optical changes in firing neurons [as pioneered by Grinvald and co-workers (Ts'o et al., 1990)]—give us insight into the workings of Ihe higher brain centers. The physicist who wants to contribute to vision research can either learn psychobiology from scratch and apply his or her knowledge of mathematics and physics where it is applicable or try to suggest some novel principles to brain researchers regardless of whether these are
789
neurophysiologically plausible or not. After all, what is not plausible in psychobiology now might become probable decades later! The latter kind of contribution can be useful as an A l model or can be applied in robotics or machine vision. For both approaches there are many famous examples. The best known physicists and engineers who have directly contributed 10 psychobiology are Hermann von Helmholtz. Ernst Mach, Georg von Bekesy, and Werner Reichardt, among others. The physicists and engineers who indirectly contributed to brain research by introducing fundamental concepts, such as the programmable digital computer, information theory, cybernetics, analogies to memories. A l models, and so on are Jonn von Neumann, Leo Szilard, Allan Turing, Claude Shannon, Norbert Wiener, Dennis Gabor, David Marr. and John Hopfield, to name a few of the best known. As an example of the latter approach, I should like to draw attention to the delightful monograph by Valentino Braitenberg (19841. The author's main message, lhat "analysis is uphill, synthesis is downhill," is illustrated by building a succession of simple miniature vehicles of increasing complexity. The vehicles have two motors and a few sensors. Already such a simple operation as crossing the left and right photodetcctors to be connected to the right and left rear motors (that drive the wheels) results in behavior that an outside observer might interpret as "fear" and "aggression." The adding of inhibitions between sensors and motors produces behavior that an anthropomorphic observer might construe as "love." Some nonmonotonic connections result in behavior lhat resembles "decision making." The inclusion of wires that exhibit hysteresis yields behavior resembling "concepts." Introducing some random connections (mutations) and selection-of-t he-fit test conditions (lethal mutations cause some vehicles to fall off the tablelop), which the author calls "(he impersonal engineer," results in vehicles of ever-increasing sophistication, with concepts of "forms," "ideas," "rules and regularities," "trains of thought," "foresight," "egotism," and "optimism." This wide range of complex behavior is merely the outcome of manipulating the interconnections between a few sensors and motors and observing the outcome. I f instead of this synthetic approach (the essence of the A l paradigm), a designer were asked to construct a machine with some given complex behavior (the analytic approach of science), usually this could not be done at present. Of course, to regard such crude behavior as, for instance, "the circling of an object and periodically returning to i t " as "love" borders on the comical. Nevertheless, thirty years ago theorists of behaviorist psychology still had considerable influence on psychologists, and their restriction of studying mental phenomena by behavioral manifestations alone still has followers who might take the philosophy of vehicles quite seriously. Similarly, one can treat the transducers of brightness and color as an "ideal observer" and ask how well the visual system emulates such a hypothetical observer. For
315
instance, Buchsbaum and Gottschalk (1983) utilized the idea I-observer approach to extract maximum information from the red, green, and blue color mechanisms of given spectral characteristics. After solving the eigenfunctions of the connectivity matrices, they found a system of connections lhat in essence are the Hering (1878) color system of red-green and yellow-blue opposing mechanisms. By the same token, I am curious as to why nobody has found, so far. Hadamard transforms that would optimize signal-noise ratios, which might be a desirable requirement for the visual system that can detect a few quanta. [On quantum efficiency of vision, L recommend the beautiful textbook by Cornsweet (1970).] I selected this example, out of many, to illustrate the point that interesting mathematical and physical ideas might have been stumbled upon by "the impersonal engineer" during hundreds of millions of years of evolution; nevertheless, it is almost as interesting when human ingenuity suggests solutions lhat are better than those found by trial and error in Nature (such as the Hadamard fillers nowadays routinely used in measuring devices of all kinds). Another approach to applying mathematical ideas to brain research is provided by Resnikoff (1987b). He assumes some similarity between graph theory and brains by regarding the human brain as being randomly connected (a rather unlikely hypothesis) and applying ihe stochastic theorem of Erdds and Renyi (1959) [that estimates the minimum number of connections (synapses) between neurons that would permit • random graph (neural network) to be completely connected]. He argues that the human brain consumes about IS watts (thai is. the equivalent of a dim light bulb, so a "bright idea" is not really very bright) and assumes that it has 10" neurons. Assuming further that each neuron has about 10' synapses and using the Erdos-Renyi stochastic theorem, he finds that the brain must have at least 14 connections per neuron to be totally connected with a probability of 0.988, while with fewer connections the brain would fall into disconnected parts. For the physicist, who is accustomed to dealing only with local interactions, early vision must be a familiar discipline- Indeed, much of the interaction occurs between adjacent neurons often exhibiting cooperative perceptual phenomena. When the stimulus moves fast, reaching neural propagation velocities (which are quite slow for nonmyelinated axons), one can actually observe some of the relativislic foreshortening phenomena of special relativity theory (Caelli et al. 1978b), where instead of the speed of light one has lo postulate the maximum neural propagation velocity as the ultimate limit in the Lorentz transformation. For instance, disks moving increasingly faster might appear as shrinking ellipses in the direction of travel. However, there are processes lhat travel much faster through myelinated axons, which appear as "tachyons" representing "action at a distance" among the slowly propagating non-myelinated neural networks. Here is a fascinating world of relativislic and non relativislic phenomena mixed together in sophisticated ways.
787
The theoretical physicist fascinated with problems of "complexity," "self-similarity," "scale," "fractals," and "chaos" will find the human brain the proper structure for studying such problems. As an example, the reader can easily draw some connected and disconnected complex line drawings, that cannot be perceived as such by the human visual system. On the other hand, by reducing the complexity of these line drawings (e.g., as in Fig. 11(a)], we can effortlessly perceive connectivity. However, these same simple line drawings in aggregates do not yield texture discrimination without scrutiny [as illustrated by Fig, J Kb)]. Our limitations in perceiving connectivity in complex line drawings and in aggregates of simple line drawings is related to the self-similarity of perceiving connectivity at different scales. From simple and complex "mazes" to "impossible figures," one can juxtapose local and global rules of connections such that, at some scale, the solution of the maze or the impossibility of an object pops out, while at a finer scale these properties become hidden. Finally, the concept of "chaos" in deterministic systems is not very new to me, Over decades I used to get the largest (unclassified) pseudo-random-number generators with the longest cycles of repetition, 1 used the same pseudo-random generator to create the left and right images of a dynamic RDS, respectively, and even after millions of such stereograms composed of 1000X 1000 pixels, one could stereoscopically fuse them provided the computer did not make some error. This should drive home my point that computers are deterministic systems. They can exhibit complex behavior that resembles chaos, but cannot in fact generate randomnessl Nevertheless, I am impressed by the chaotic behavior, exhibited as fractals, at the boundaries between stable domains in the phase space of complex adaptive systems. It is suggested by Stewart Kauffman, among others, that learning (e.g., the evolution of life) does take place at "the edge of chaos." In such a condensed review, I have had to omit hundreds of important contributions and concentrate on authors who share my outlook, I have no doubt that if other psychologists had the opportunity to write a review for this journal, a very different story might be told, probably with a completely different list of references. Indeed, workers in visual perception belong to many camps. The majority believe in the "direct vision" paradigm of J. J. Gibson (1966). Gibson—whose ideas had a great effect on me in my youth—assumed that mental processes "resonate" to certain hidden structures in the environment, as a tuning fork might resonate to a specific tone. [However, metaphors of this kind, according to Peter Medwar, as quoted by Ramachandran (1990), are "mere analgesics; they dull the ache of incomprehension without removing the cause."] Others adhere to the view that perception is based on "unconscious inferences" as originally posited by Helmholtz (1867). [Interestingly, Helmholtz introduced the concept of the "unconscious" many years before Sigmund Freud.] Researchers in A l
316
believe in the paradigm of "natural computation," Several leading psychologists cling to old paradigms, from behaviorism to Gestalt theories, while others embrace radically different paradigms, such as mental imagery. [That paradigms die hard has been convincingly discussed by Kuhn (1972), and indeed Newton's theory of gravitation is still taught, although Einstein's general relativity theory has "falsified" it,] Some others, particularly Ramachandran (1990), assume that visual perception is based on a "bag of tricks." This last-mentioned paper by Ramachandran contains much useful information and several perceptual demonstrations. I end this review with a quotation from Ramachandran (1990, p. 22): "[The early visual system] . . . lakes advantage of constraints rather
semantic system
knowledge may
movement, matian
that incorporate
than top-down
but
properties
that depend
of specific
objects.
haoe built-in etc.,
general
influences
knowledge
not about
aboul
umbrellas,
of the on
Thus
the
surfaces, chairs,
world
high-level visual deplh, and
dal¬
dogs."
ACKNOWLEDGMENTS I thank Dr. Gerard Toulouse for inviting me to write this review on early vision and advising me to assume that the readers have no previous knowledge of psychobiology. He also advised me to give a set of introductory references before the Introduction, I thank Dr. Douglas Williams, Dr. Stanley Klein, Dr. Jochen Braun, and Dr. Francis Crick for their helpful comments while reading this manuscript, and Ms. Carol Esso for helping me with the word processing, particularly with the references. REFERENCES
Adelson, E. H„ and J. K. Bergen, 1985, "Spatio-temporal energy models for the perception of motion." J. Opt. Soc. Am. 2, 284-299. Amit, D. J., 1989, Modelling Brain Function: The World ofAtiraclor Neural Networks (Cambridge University, Cambridge/New York). Anderson, C. H., and D. C. Van Essen, 1987, "Shifter circuits: A computational strategy for dynamic aspects of visual processing," Proc. Nat. Acad. Sci. USA 84, 6297-6301. Anslis, S. M., 1978, "Apparent movement." in Handbook of Physiology. Vol. VIII. Perception, edited by R. Held, H. W. Leibowitz, and H. L. Teuber (Springer, Berlin), pp. 655-673. Babington-Smith, B., 1977, "A wartime anticipation of random-dot stereograms," Perception *. 233-234. Barlow, H. B.. C. Blakemore, and J. D. Pettigrew, 1967, "The neural mechanism of binocular depth discrimination," J. Physiol. 193,327-342. Barlow, H. B., and J. D. Mollon, 1982. Eds., 77ie Sense! (Cambridge University, Cambridge, England). Bergen, 3. R., and E. H. Adelson, 1988, "Early vision and tex-
788 ture perception," Nature 333, 363-364. Bergen, J. R., and B. Julesz, 1983, "Parallel versus serial processing in rapid pattern discrimination," Nature 303,696-698, Bernstein, Jeremy, 1984, Three Degrees above Zero: Bell Lobs in ihe Information Age (Scribner, New York), Chap. 3, "Bela Julesz," pp. 39-48. Bishop, P. O., 1963, "Neurophysiology of binocular single vision and stereopsis." in Handbook of Sensory Physiology, edited by R. Jung (Springer, Berlin). Vol. 7, pp. 255-303. Blakemore, C, 1969, "Binocular depth discrimination and the nasotemporal division," I . Physiol. 205, 471-497, Blakemore, C and F. W. Campbell, 1969. "Adaptation to spatial stimuli," J. Physiol. 200, 11-13. Blakemore, C and G. F. Cooper, 1970. "Development of Ihe brain depends on the visual environment." Nature 228, 477-47!. Blakemore, C, and B. Julesz, 1971, "Stereopsis depth aftereffect produced without monocular cues," Science 171. 386-388. Boff, K. R., L. Kaufman, and J. P. Thomas, 1986, Handbook of Perception
and Performance.
Vat. 1. Sensory processes and
per.
(Wiley. New York). Boynton, R., 1986. Ed., Reprinted version of Ihe 25th Jubilee Issue of Vision Res. 26(9), 1601-1619, Braddick, O.. 1974, "A short-range process in apparent motion." Vision Res. 14, 519-527. Braddick, O. J., 1980, "Low-lcvcl and high-level processes in apparent motion," Philos. Trans. R. Soc. London, Ser. B 290, 137-151. Braddick. O., J. Atkinson, B. Julesz, W. Kjopfl, t BodisWollner. and E. Rabb, 1970, "Cortical binocularity in infants," Nature 2SS, 363-365. Braitenberg, V 1984, Vehicles; Experiment in Synthetic Psychology (MIT, Cambridge. MA). Braun, J.. and D. Sagi, 1990, "Vision outside the focus or attention." Percept. Psychophys. 48.45 - 58. Broadbent, D. C, 1958, Perception and Communication {Pecgamon, Oxford). Bruce. V., and P. Green. 1985. Visual Perception: Physiology, psychology, and ecology (Erlbaum, Hillsdale, NJ). Buchsbaum, G.. and A. Gottscbatk, 1983, "Trichromacy, opponent colours coding and optimum colour information transmission in Ihe retina." Proc. R. Soc. London, SIT. B 220, 89-113. Bolthoff, H. H., and H. A. Mallot, 1990, "lotegration of stereo, shading, and texture," in A l and the Eye, edited by A. Blake andT. Troscianko (Wiley. New York), pp. 1)9-146. Burt, P., and B. Julesz, 1980, "A disparity gradient limit for binocular fusion." Science 208,615 -617. Caelli, T. M.. and B. Julesz, 1979, "Psychophysical evidence for global fearore processing in visual lexture discrimination," J. ception
r|
Opt. Soc. Am. 69(5), 675 - 678.
Caelli, T. M„ B. Julesz. and E. Gilbert, 1978, "On perceptual analyzers underlying visual lexture discrimination: Part I I , " Biol. Cybernetics 29, 201-214. Caelli, T. M., W. C. Hoffman, and H. Lindman, 1978, "Subjective Lorentz transformations and Ihe perception of motion," J.
Opt. Soc. Am. 68,402 -417.
Cavanagh, P., and G. Mather, 1989, "Motion: The long and short of it." Spatial Vision 4, 103-129. Chang. J. J., and B. Julesz, 1983. "Displacement limits, directional anisotropy. and direction versus form discrimination in random-dot cinematograms," Vision Res. 23, 639 - 646. Chang. J. I., and B. Julesz. 1984, "Cooperative phenomena in apparent movement perception," Vision Res. 24. I781-178S.
317
Chang, J. J., and B. Julesz, 1990, "Low-level processing of disparity-tuned binocular neurons lakes precedence of shape from shading," Invest. Ophthalm, Visual Sci. 31(4), 525 (Proceedings of ARVO Meeting, May, 1990). Churchland, P. S., 1986, Neurophitosophy. Toward a unified science of the mind-brain IMIT, Cambridge, MA). Cornsweet, T. N., 1970. Visual Perception (Academic, New York). Desimone, R-, and L. G. Ungerleider, 1989, "Neural mechanisms of visual processing in monkeys," in Handbook of Neurophysiology. Vol. 2, edited by F. Boiler and J. Grafman (Elsevier, New York/Amsterdam), pp. 267-299. Dev, P.. 1974, "Segmentation processes in visual perception: a cooperative neural model," COINS Tech. Rep. 74C-5, Computer Information Science. University of Massachusetts. Dev. P.. 1975. "Perception of depth surfaces in random-dot stereograms: A neural model," tnt. J. Man-Mach. Stud. 7, 511-528. DeValoiS. R. L., D, G. Albrecht, and L G. Thorell, 1978, "Cortical cells: Bar detectors or spatial filters^" in Frontiers of Visual Science, edited by S. J. Cool and E. L. Smith (Springer, Berlin). Dodwell. P. C, 1983, "The Lie transformation model of visual perception," Percept. Psychophys. 34, I -16. Edelman, G. M. 1989, The remembered
Present.
A biological
(Basic, New York). Edelman, G. M., W, E. Gall, and W. M. Cowan, 1984. Eds., Dynamic Aspects of Neocorlical Function (Wiley, New York). Enns, J., 1986, "Seeing textons in context," Percept. Psychophys. 39, 143-147, Erdos, P., and A. Renyi, 1959, "On random graphs. I . " Publ. Math. Debrecen 6. 290-297. Fender, D., and B. Julesz, 1967, "Extension of Panum's fusional area in binocularly stabilized vision." J. Opl. Soc. Am. 57(6), 819-830. Fodor, J.. 1983. Modularity of Mind IMIT. Cambridge, MA). Fodor. J. A., and Z. W. Pylyshyn, 1983, "Connect] on ism and cognitive architecture: A critical analysis," in Connections and Symbols, edited by S. Pinker and J. Mehler (MIT, Cambridge, MA), pp. 3-71. Fogel, L, and D. Sagi, 1989, "Gabor Alters as texture discriminators," Biol. Cybernetics 61. 103-113. Fox, R., R. N. Aslin, S. L. Shea, and S. T. Dumais, 19B0, "Stereopsis in human infants," Science 207. 323-324. Gabor, D., 1946, "Theory of communication," J. Inst. Eleclr. theory of consciousness
Eng. (London) 93,429-457. Gazzaniga, M. S., 1939. "Organization of the human brain," Science 245.947-952. Gorea, A., and B. Julesz, 1990, "Context superiority in a detection task with a line-element stimuli: A low-level effect," Perception 19, 5-16. Gregory, R. L., 1937. Ed., Tbe Oxford
Companion
to the Mind
(Oxford University, Oxford/New York). Gregory, R. L.. 1990. Eye and Brain:
The Psychology
of Seeing
(Princeton University, Princeton, NJ). Grossberg, S., 1938, Ed., Neural Networks and Natural Intelligence (MIT, Cambridge, MA). Grossberg, S., and E. Mingolla, 1985, "Neural dynamics of perceptual grouping: Textures, boundaries, and emergent segmentations," Percept. Psychophys. 38, 141-171. Gurnsey, R., and R. Browse, 1987. "Micropattem properties and presentation conditions influencing visual texture discrimination," Percept. Psychophys. 41, 239-252. Hadani, 1., G. Ishai, and M. Gur, 1978, "Visual stability and
769
Julesz, B . , 1962, " V i s u a l pattern discrimination," I R E Trans.
space perception in monocular vision: A mathematical mod-
Inf. T h e o r y I T - S . 8 4 - 9 2 .
el," Biomedical Engineering T e c h n i c a l Report: T S I (Israel I n stitute of Technology) M a y 1978.
Julesz, B „ 1964, " B i n o c u l a r depth perception without familiarity cues, Science 145, 3 5 6 - 362.
Hadani, I . , G . I s h a i , a n d M . G u r , 1980, " V i s u a l stability and space perception in monocular vision: mathematical model,"
Jolesz, B . , 1971. Foundations
J . O p t . Soc. A m . 7 0 . 6 0 - 6 S .
of Cyclopean
Perception (Universi-
ty of C h i c a g o , C h i c a g o ) . (Erl-
Jolesz, B . , 1975, " E x p e r i m e o l s in the visual perception of tex-
H e i l m a n , K . M . , R . T . Watsoo, E . Valenstein. and M . E . G o l d -
Julesz, B . , 1978, " G l o b a l stereopsis: Cooperative phenomena in
H a r r i s , C . S., 1980, E d . , Visual Coding and Adaptability
ture," Sci, A m . 232, (April), 3 4 - 4 3 .
baum, Hillsdale, N J ) . berg, 1987, "Attention: Chapter 11 in Handbook System,
Behavior and neural mechanisms," of Physiology, Section
I: The
of
Physiology,
edited by R . H e l d , H . W . Leibowitz, and
Julesz, B . , 1980. "Spatial frequency channels in one-, two-, and
461-481.
Ihree-dimensional vision: Variations on an auditory theme by
Helmholtz, H . von, 1896, Handbucb Dritter Abscbnitt,
der Physiologischen
Bekesy," in Visual Coding
Optik.
in E n g l i s h as Helmholtz's
Treatise on Physiological
Optics, by J.
edited by C . S.
Julesz, B . , 1981, "Textons, the elements of t e n u r e perception Bnd their interactions," Nature
lished by Dover, N e w Y o r k , 1962.] H e r i n g , E . , 1878, Zur Lehre
and Adaptability,
H a r r i s [ E r l b a u m , Hillsdale, N J ) , pp. 2 6 3 - 3 1 6 ,
Zweite Auflage (Voss. H a m b u r g ) . [Published
P. C . Southall, 1924 (The O p t i c a l Society of A m e r i c a ) ; repub-
290.91 - 9 7 .
Julesz, B . . 1984. " T o w a r d a n axiomatic theory of preattentive
vam Lichtsinn
vision," in Dynamic
(Gerold's Sohn,
Aspects of Neoconical
Function,
edited by
G . M . E d e l m a n . W E . G a l l , and W. M . C o w a n (Wiley, New
Vienna). H i l d r e l h , E . , 1984, Tbe Measurement
Y o r k ) , pp. 5 8 5 - 6 1 2 .
of Visual Motion ( M I T ,
Julesz, B „ 1986a, "Stereoscopic
Cambridge, M A ) . H i l l i s , D . W . , 1985 , 77ie Connection
Machine
vision," V i s i o n R e s . 26(9),
1601-1612.
I M I T , Cambridge,
MA).
Julesz, B . , 1986b, "Texton gradients: T h e texton theory revisited." B i o l . Cybernetics 5 4 , 2 4 5 - 2 5 1 .
H i r s c h , H . V . B . , and D . N . SpineHi. 1970, " V i s u a l experience modifies distribution of horizontally and vertically oriented re-
Julesz, B . , 1990a, " A l and early vision," in Al and the Eye, edit-
ceptive fields in cats." Science 168. 8 6 8 - 8 7 1 .
ed b y A . Blake and T . T r o s c i a n k o (Wiley, N e w Y o r k ) , pp. 9-20.
Hoffman, W . C , 1970, "Higher visual perception as prolongations of the basic L i e transformation group," M a t h . Biosci. 6,
Julesz, B . , 1990b, "Concepts in early vision," in Synergetics
437-471.
Cognition,
with emergeot collective computational abilities," Proc. Nat.
Julesz. B . , )990c, " E a r l y vision is bottom-up, except for focal
Aqad. S c i . U S A 79, 2 5 5 4 - 2 5 5 8 .
attention," in Symposium
H o r n . B . K . P . , and M . J . Brooks, 19B5, " T h e variational approach lo shape from shading," M I T Artificial Intelligence M e m o N o . 813,
#55:
The Brain (Cold Spring H a r -
bor Laboratory Press. L o n g Island, N Y ) . Julesz, B . . 1991a, "Some strategic questions in visual percep-
1-32.
H u b e l , D . H . . 1988, Eye, Brain,
in
edited by H . H a k e n and M . Stadler (Springer, Ber-
lin), pp. 2 1 8 - 2 3 2 .
Hopfield, J . J - , 19B4, "Neural networks and physical systems
Lab
Perception,
H . T e u b e r (Springer, Berlin), pp. 2 1 5 - 2 5 6 .
V o l u m e V , edited by V . B . Mountcastle, F . P l u m , and
S. R . G e i g e r ( A m e r i c a n Physiological Society. Bethesda. M D ) , pp.
stereoscopic depth perception." in Handbook Vol. VIII,
Nervous
and Vision (W. H . F r e e m a n .
New York).
tion," in Representations
in Vision:
tions in Vision Research,
edited by A . G o r e a (Cambridge U n i -
Trends and Tacit
Assump-
versity, C a m b r i d g e . England), pp. 3 3 1 - 3 4 9 .
Hubel. D . H . . and T . N . Wiesel, 1960, "Receptive fields of optic
Julesz, B „ 1991b, "Consciousness and focal attention," Behav,
nerve fibres in the spider monkey," J . Physio). 154. 5 7 2 - 5 E 0 .
Brain Sci. (in press).
H u b e l , D . H . , and T . N . Wiesel, 1962. "Receptive fields, binocu-
Julesz, B „ and J . R . Bergen, 19B3, "Textons, the fundamental
lar interaction and functional architecture in the cat's visual
elements in preattentive vision and perception of textures,"
cortex," J . Physiol. 160. 106-154.
Bell Syst. T e c h . J . 62(6), 1619-1645.
Hubel. D . H . , and T . N . Wiesel, 196E. "Receptive fields and
Julesz, B . , and J . J . C h a n g , 1976, "Interaction between pools of
functional architecture of the monkey striate cortex," J . P h y -
binocolar disparity detectors tuned to different disparities,"
siol. 195, 2 1 5 - 2 4 3 .
Biol. Cybernetics 22, 1 0 7 - 1 1 9
Hubel, D . H . , and T . N . Wiesel, 1970, "Cells sensitive to bino-
Julesz, B . , E . N . Gilbert,
and J . D . Victor,
1978,
"Visual
cular depth in area 18 of Ihe macaque monkey cortex: Stereo-
discrimination of textures with identical third-order statis-
scopic vision in the macaque monkey," N a t u r e 225,
tics," B i o l . C y b e r n e t i c s 31(3), 1 3 7 - 1 4 0 .
41-42.
H u b e l , D . H . , and T . N . Wiesel, 1974, "Uniformity of monkey striate cortex:
A parallel relationship between
scatter, and magnification
field
Julesz, B . . W . K r o p f l , and B . Petrig, 19B0, "Large evoked poten-
size,
tials to dynamic random-dot correlograms and stereograms
factor," J . C o m p . N e u r o l . 158.
permit quick determination of stereopsis." Proc. Nat. A c a d .
295-306.
Sci. U S A 77(4), 2 3 4 8 - 2 1 5 1 .
H u m p h r e y s . G . W . , and V . Bruce. Computational,
Experimental,
1989. Visual
Cognition:
and Neuropsychological
Julesz, B . . and B . K r o s e .
Perspec-
tives ( E r l b a u m . Hillsdale, N J ) . James, W . , IE90, Tfte Principles
(republished in
cy tuned channels in binocular fusion and rivalry," Perception 4, 125-143.
Judd. S., 1987, " L e a r n i n g in networks is h a r d , " in IEEE Conference
perception:
Julesz, B . , and J . E . Miller, 1975, "Independent spatial frequenof Psychology
1925 by D o v e r . N e w Y o r k ) . International
1988, " V i s u a l texture
Features and spatial filters," N a t u r e 3 3 3 , 3 0 2 - 3 0 3 .
on Neural
Networks,
First
Julesz. B . , and R . A . Payne, 1968, "Differences between mono-
V o l . I I , pp.
6B5 - 692.
cular
and binocular s t r o b o s c o p e
movement
perception,"
Vision R e s . 8 . 4 3 3 - 4 4 4 .
Julesz, B . , I960, "Binocular depth perception of compuler-
Julesz, B . , and C . W . T y l e r , 1976, " N e u r o t r o p y , a n entropy-
generated patterns," Bell Syst. T e c h . J . 39, 1125-1162.
like measure of neural correlation in binocular fusion and
318
770 rivalry," Biol. Cybernetics 23,25-32.
Res. 21, 177-179. Mitchell. D. E., 1980, "The influence of early visual experience on visual perception." in Visual Coding and Adaptability, edited byC S. Harris (Erlbaum, Hillsdale, NJ), pp. 1-50. Moran, J-, and R. Desimone, 1985. "Selective attention gates visual processing in the extrastriale cortex," Science 229. 782-784. Mountcastle. V. B., 1979, "An organizing principle for cerebral function: The unit module and the distributed system," in The Neurosciences: Fourth Study Program, edited by F. O. Schmidt and F. G. Worden (MIT, Cambridge. MA), pp, 21-42. Mowforth, P., J. E W. Mayhew, and J P. Frisby, 1981, "Vergence eye movements made in response to spatial-frequencyfiltered random-dot stereograms," Perception 10(3), 299-304. Muller, J., 1844, Handbuch der Physiologie des Menschen, 4th Edition (J. Holscher. Coblenzl. Nakayama, K., 1985, "Biological image motion processing: A review," Vision Res. 25, 625-660. LeDoux, J. E., and W. Hirst, 1986, Eds., Mind and Brain. DiNakayama, K , and M. Mackeben, 1989, "Sustained and tranalogues in Cognitive Neuroscience {Cambridge University, sient components of focal visual attention," Vision Res, 29, Cambridge, England). 1631-1647. Lehky, S. R. and T. J. Sejnowsky, 1990, "Neural model of Nakayama, K., and G. H. Silverman. 1986. "Serial and parallel slereoacuity and depth interpretation based on a distributed processing of visual feature conjunctions," Nature 320, representation of stereo disparity." J. Neurosci. 10, 264-265. 2281-2299. Nelson. J. 1,, 1975. "Globality and stereoscopic fusion in binoLehmann, D., and B. Julesz. 1978, "Laieralized cortical potencular vision," J. Theor. Biol. 49, 1-88. tials evoked in humans by dynamic random-dot stereograms/' Nothdurft. H. C, 1985. "Sensitivity for structure gradient in Vision Res. 18. 1265-1271. texture discrimination tasks," Vision Res. 25. 1957- 1968. Lettvin, J. Y., H- R. Maturana, W. S. McCulloch, and W. H, PiOgle, K. N., 1964. Researchers in Binocular Vision (Hafner. tts, 1959. "What the frog's eye tells to rhe frog's brain," Proc New York). Inst. Radio Engrs. 47. 1940-1951. Pat Hard, J., 1977, Ed.. "The Lie transformation group model for perceptual and cognitive psychology." Cah. Psychol. 2 (entire Livingstone, M. S.. and D. H. Hubel, 1984, "Anatomy and issue). physiology of a color system in the primate visual cortex," J. Neurosci. 4,309-356. Papathomas, T. v., and B, Julesz, 1989, "Stereoscopic illusion Luneburg, R. K.. 1950. "The metric of binocular visual space." based on the proximity principle," Perception 18(5,, 589-594. J. Opt, Soc. Am. 40, 627 - 642. Paperl, S,, 1961, "Centrally produced geometrical illusions," Nature 191, 733. MacAdam, D. L., 1970, Sources of Color Science (MIT, Cambridge, MA). Penrose, R., 1939, The Emperors New Mind: Concerning ComMach, E-, 1886, The analysis of sensations and Ihe relation of the puters, Minds, and the Laws of Physics (Oxford University, physical io Ihe psychical. First German edition in 1886; repubNew York/Oxford). lished by Dover (New York, 1959), using Ihe 5lh German ediPcntland. A. P., 1986, "Shading into texture" Artif. Intell. 29, tion revised and supplemented by S. Water!ow. 147-170. Petrig, B., 1980. "Nachweis von Slereopsis bei Kindern milters Mahowald, M. A., and T. Delbriick, 1989. "Cooperative stereo stochasischer Punk [Stereogram me und der zuhorigen evozierte matching using static and dynamic image features," in Analog Potenziale," Ph.D. thesis, ETH, Zurich. ELSI Implementation of Neuro Systems, edited by C Mead and M. Ismail (Kluxer Academic, Norwell, MA), pp. Petrig, B., B. Julesz, W. Kropfl. G. Baumgartner, and M. An lit213-23B. er, 1981, "Development of slereopsis and cortical binocularity in human infants: Electrophysiological evidence," Science 213, Malik, J. M. A., and P. Perona, 1990, "Preatientive texutre 1402-1405. discriminalion with early vision mechanisms," J. Opt. Soc. Am. 7(5), 923-932. Poggio. G, F.. 1984, "Processing of stereoscopic information in primate visual cortex," in Dynamic Aspects of Neocortical Marr. D., 1982, Vision (Freeman, San Francisco). Function, edited by G. M. Edelman, W. E, Gall, and W. M Marr, D., and T. Poggio, 1976, "Cooperative computation of Cowan (Wiley, New York), pp. 613-635, stereo disparity," Science 194, 283-287. Poggio, G. F,, F. Gonzalez, and F. Krause, 1988, "Stereoscopic Marr. D., and T. Poggio, 1979. "A theory of human stereopsis," mechanisms in monkey visual cortex: Binocular correlation Proc. R. Soc. London. Ser. B 204, 30U328, and disparity sensitivity," J. Neurosci. 8, 4531-4550. Mayhew. J. E. W., and J. P. Frisby. 1979, "Convergent disparity Poggio, T., V. Torre, and C. Koch, 19S9, "Computational discriminations in narrow-band-filtered random-dot stereovision and regularization theory," Nature 317, 314-319. grams," Vision Res. 19{l), 63-71. Polanyi, M., 1969. Knowing and Being (University of Chicago, Melzak, Z. A., 1976, Mathematical ideas, modeling & applies* Chicago). lions. Vol. H of Companion to Concrete Mathematics (Wiley, Posner, M. L , 1978, "Orienting of attention," Q . J. Exp. New York/Londonh Psychol. 32, 3-25. Miezin, F. W . J Myerson. B. Julesz, and J. M. Allman, 1981, Pylyshyn, Z- W., 1984, Computation and Cognition. Toivard a "Evoked potentials to dynamic random-dot correlograms in Foundation of Cognitive Science {MIT. Cambridge, MA). monkey and mam A lest for cyclopean perception." Vision
Kandel, E. R., and J. H. Schwartz, 1985, Principles
of
Neural
Science, 2nd edition (Elsevier/North-Holland, New York/Amsterdam). Kanjzsa, G., 1976, "Subjective contours." Sci. Am. 234,48-52. Kami, A., and D. Sagi. 1990, "Where practice makes perfect in texture discrimination," The Weizmann Institute of Science Technical Report No. CS90-02, Jan. 1990. Also in Proc. Nat. Acad. Sci. USA {1991, in press). Krose, B. J. A., 1987, "Local structure analyzers as determinants of preattentive pattern discrimination," Biol. Cybernetics 55, 289-298. Krose. B. J. A., and B. Julesz, 1989, "The control and speed of shifts of attention," Vision Res. 29(11), 1607-1619. Kuffler, S. W., 1953. "Discharge patterns and functional organization of mammalian retina," J. Neurophysiol. 14, 37-68. Kuhn, T,, 1972, The Structure of Scientific Revolutions. New edition {University of Chicago, Chicago).
r
r
319
771
Ramachandran, V. S., 1988, "Perception of shape from shading," Nature 163-166. Ramachandran, V. 5., 1990, "Visual perception in people and machines," in Al and tbe Eye, edited by A. Blake and T. Troscianko (Wiley, New York!, pp. 21-77. Regan, D., 1989, Human Brain Elecfophysiology (Elsevier, New York/Amsterdam). Regan, D., and K. D. Beverley, 1985, "Visual responses to verticity and the neural analysis of optic How," J. Opt. Soc. Am. 2.280-2B3. Reichardt, W., 1961, "Autocorrelation, a principle of evaluation of sensory information by the central nervous system," in Sensory Coding, edited by W. A. Rosenbluth (Wiley, New York), pp. 303-317. Reichardt, W., and M. Egelhaaf, 1988, Naturwissenschaften 75. 313-315. Resnikoff, H. L , 1987a, The Illusion of Reality (Springer. New Yorkl. Resinkoff, H. L., 1987b, "Concurrent computation and models of biological information processing," in Advances in Cognitive Science, edited by M. Kochen, and H. M. Hastings (Westview, Boulder, CO), pp. 78-98. Richards. W. A.. 1988. Ed., Natural
Compulation:
Selected
(MIT, Cambridge, MA). Rock. [„ 3984, Perception (Sci. Am. Library, Freeman, San Francisco). Rock, I . , and S. Palmer, 1990, "The legacy of Gestalt psychology," Sci. Am. I l l Dec.), 84-90. Rubenstein. B., and D. Sagi. 1990, "Spalial variability as a limiting factor in textore-discrimination tasks: Implications for performance asymmetries," J. Opt. Soc. Am. A 9, 1632-1643. Rumelhart, D. E., and J, L. McClelland, 19g6, Parallel DistriReadings
buted Processing:
Explorations
in the Microstructure
of Cogni-
(MIT. Cambridge. MA). Saarinen. J.. and B. Julesz. 1991. "The speed of attentional shifts in the visual Held," Proc. Nat. Acad. Sci, USA 88. 1812-IE14. Sagi, D., 199], "Detection of an orientation singularity in Gabor textures: Effect of signal density and spatial-freqoency," Vision Res. (in press). Sagi. D., and B. Julesz, 1985, "'Where' and 'what' in vision," Science 228. 1217-1219. Sagi, D., and B. Julesz, 3986, "Enhanced detection in the aperture of focal attention during simple discrimination tasks," Nature 321, 693 - 695. Sagi. D-. and B. Julesz, 1987, "Short-range limitation of detection of feature differences." Spatial Vision 2, 39-49. Saltzman, Z. D., K. H. Britlen. and W T, Newsome, 1990, "Cortical micros!imulation influences perceptual judgments of motion direction." Nalure 346, 589. Schomann, P., 1904, "Einige Beobachtungen uber die Zusammenfassung von Gesichtseindmcken zu Einheiten," Psychol. Stud. 1, 1-32. Sekuler, R., and R. Blake, 1985, Perception (Knopf, New York). Shepard, R, N., and J, Metzler, 1971, "Mental rotation of three-dimensional objects," Science 171, 701-703. Snowden, R. J., and O. J. Braddick, 1990, "Differences in the processing of short-range apparent motion at small and large displacements, "Vision Res. 30, 1211-1222. Probes of Central Nervous
System
in Experimental Animals and Man (Sinauer Associates, Sunderland. MA). Sperling, G., 1970, "Binocular vision: A physical and neural theory." Am. J. Psychol. 83,461-534. Activity
r
Ullman. S., 1979. Tbe Interpretation
tion
Sokoloff, L., 1984, Metabolic
Sperling, G., 1978, "The Goal of Theory in Experimental Psychology." Bell Teleph. Lab. Tech. Memo #78-1221-12. Sperry, R. W, 1982, "Some effects disconnecting the central hemispheres, "Science 217. 1223-1226. Sternberg, S., 1966, "High-speed scanning in human memory," Science 153, 652-654. Teller, D. Y. 1980. "Locos questions in visual science," in Visual Coding and Adaptability, edited by C. S. Harris (Erlbaum, Hillsdale, NJ),pp. 151-176. Thompson. P., 1980, "Margaret Thatcher: A new illusion," Perception 9,483-484. Treisman. A., and G. Gelade, 1980, " A feature-integration theory of attention," Cognitive Psychol. 12,97-136. Treisman, A., and S. Gormican, 1988. "Feature analysis in early vision: Evidence from search asymmetries," Psychol. Rev. 95, 15-48. Treisman, A , andR. Palerson, 1984, "Emergent features, attention and object perception," J. Exp. Psychol. 10. 12-31. Ts'o, D. Y., R. D. Frostig, E. E. Lieke, and A. Grinvald, 1990, "Functional organization of primate visual cortex revealed by high-resolution optical imaging," Science 249,417-420. Ullman. S.. 1977, "The Interpretation of Visual Motion." Ph.D. thesis. Department of Electrical Engineering and Computer Science, MIT. of Visual Motion (MIT,
Cambridge, MA). Ullman. S., 1984, "Visual routines." Cognition 18. 97-159. Uttal. W. R., 1990. "On some two-way barriers between models and mechanisms." Percept. Psychophys. 48, 188-203. Van de Grind. W. A., A. J. Van Doom, and J. J. Koenderink, 1983, "Detection of coherent movement in peripherally viewed random-dot patterns," J. Opt. Soc. Am. 73. 1674-1683. Van Essen. D. C. E. A. DeYoe, J. Olavarria, J. Knierim, J. Fox. D. Sagi, aod B. Julesz. 1989, "Neural responses lo static and moving texture patterns in visual cortex of the macaque monkey." in Neural Mechanisms of Visual Perception, edited by D. M. K. Lam and C. D. Gilbert (Portfolio, Woodlands, TX), pp. 137-154. van Santen. J. P. H , and G. Sperling, 1935, "Elaborated Reichardt-deteclots," J. Opt. Soc. Am. A2,451-473. von der Heydt, R., E. Peterhans,. and G. Baumgartner, 1988. "Illusory contours and cortical neuron responses," Science 224, 1260-1262. von Schelling, H-, 1956, "Concepts of distance in affine geometry and its applications in theory of vision," J. Opt. Soc. Am. 46,309 -315. Voorhees, H., and T. Poggio, 1988. "Computing texture boundaries from images," Nature 333, 364-367. Watson, A. B., and A. J. Ahumada. 1985, "Model of human visual-motion sensing," J Opt. Soc. Am. A2, 322-342. Watt, R., 1988, Visual Processing: Computational, Psychophysical, and Cognitive Research (Erlbaum. Hillside, NJ).
Weichselgartner, E.. and G. Sperling, 1987, "Dynamics of automatic and controlled visual attention," Science 238. 778-779. Weisstein, N., and C. S. Harris, 1974, "Visual detection of line segments: An objecl-superiority effect," Science 186,752-755. Westheimcr, G., 1979, "Cooperative neural processes involved in stereoscopic acuity," Exp. Brain Res. 36. 585 - 597. Williams. D. W., and B. Julesz, 1990, "Perceptual asymmetries in textore discrimination can be explained by bottom-up processing alone," In OSA Annual Meeting Technical Digest 1990. Vol 15 of the OSA Technical Digest Series [Optical Society of America, Washington, D.C. 1990), p. 51.
320
772 Williams, D. W. and B. Julesz, 1991, "Fillers versus lentous in human and machine texture discriminatioo," in Neurai Networks for Human and Machine Perception, edited by H. Wechsler (Academic, Orlando, FL1. in press. Williams, D. W„ G. C. Phillips, and R. W. Sekuler, 19B6, "Hysteresis in the perception of motion direction: Evidence for r
neural cooperativity," Nature 324,253-255. Wolfe, J. M., and K. R. Cave, 1990, "Deploying visual attention: The guided search model," in A l and ihe Eye, edited by A. Blake and T. Troscianko (Wiley, New York), pp. 79-103, Zeki, S, M .. 1980, "The representation of color in the cerebral cortex," Nature 184,412-41B.
321
322
Reprinted with permission from Mindwaves. pp. 306-320. 1987 © 1987 Blackwell Publishers
Neuropsychology and the Nature of Consciousness Larry Weiskrantz Larry Weiskrantz took his first degree in Swarthmore, USA, in 1949. After a period of postgraduate research in Oxford he returned to Harvard where he completed a PhD. He subsequently occupied various senior posts at Oxford and Cambridge and for the last 20 years has held the Chair of Psychology in Oxford. In 1980 he was elected to the Royal Society. Anyone who presumes to address the question of the nature of consciousness soon finds that it is a topic about which it is very difficult to say anything new, so often and for so long has it been searchingly pursued, but it is also almost impossible to say anything entirely old, because everyone has his own particular angle on it. Various people have tried to exorcize the word, but it is very much part of our vocabulary and it remains as a persistent challenge to neuroscientists to try to think of just what aspects of neural organization are related, and how, to changes in whatever the ordinary person means by levels or qualities of consciousness. Crudely speaking, and leaving aside- for now the complex questions of neural organization, historical approaches to consciousness have taken three routes. Firstly, it is said we all in some sense know what we mean by our own consciousness, we have our own awareness or self-awareness and through a complicated process of inference we assume that other creatures who appear to behave more or less the way we do must also have consciousness. So it is one kind of inferential approach based on a complex kind of social interaction. 'I think, therefore I am. As others also seem to exist, therefore they think.' But there is a second kind of view of old and distinguished vintage that asserts that consciousness of other creatures is a property more or less directly present in ordinary perception, as part of our phenomenal observation, or at least readily derived it. Darwin, for example, said that 'we have seen the senses and intuitions of various emotions and faculties such as love, memory, attention, curiosity, imitation, reason, etc. of which man boasts may be found in an incipient, or even sometimes in a well-developed condition, in the lower animals.' And Jennings, who in a classical work at rhe turn of the century wrote with enormous knowledge about the way (mainly) invertebrate creatures work, was quite convinced from his direct observation: 'If Amoeba were a large 1
Animal training sometimes results in behaviour that can all luo easily he anthropomophically interpreted in terms ofconscious intention. Photograph of a rabbit playing a piano by Mike Salisbury.
323
308
Figure 21.1 An irreverent view ofCharles Darwin from ikl Wljjll itU Vllliljf Fair.il lAf
lime ofgreat public inlerat in his theory ofevolution (1871).
324
309 animal, its behaviour would at once call forth the attribution to it of states of pleasure and pain, of hunger, desire, and the like, on precisely the same basis as we attribute these things to the dog.' Along similar lines, he later warns us that if Amoeba 'were as large as a whale, it is quite conceivable that occasions might arise when the attribution to it of the elemental states of consciousness might save the unsophisticated human being from destruction that would result from lack of such attribution'. 2
3
Jennings's views also provide a transition to the third route, namely, one that considers the evolutionary and adaptive value of consciousness. T h u s he says, 'we usually attribute consciousness . . . because this is useful; it enables us practically to appreciate, foresee, and control |an
Figure 21.2 The zoologist and physiologist George Romanes (1848-94). photographed in 1891. His claim thai consciousness depended simply mi a capacity to learn would not satisfy nuts! iieuroscienlisls or engineers today (Mary Evans Piclnre Library).
325
310 3
animal's] actions much more readily than we could otherwise do'. T h e question remains, rehat is it we are attributing, what do we mean when we assign this kind of property to another human or to any other creature? And what is it about this property that makes it useful? And this is where the difficulty arises, because the criteria are as broad as those for adaptive behaviour itself, and it is very difficult to find any in modem discussions that were not also put forward at least 100 years ago, although it is possible to find some from that period that would no longer satisfy us today. Romanes, for example, said that it was based on the capacity to learn, which would not satisfy most neuroscientists or engineers today as a sufficient criterion, because quite simple mechanical toys can be constructed with such a property. Lloyd Morgan said it was capacity for choice, which would also raise similar doubts in a computer age, although Morgan's usage probably was deeper than that involved in the design of computer software. More recently, possession of ethical values has been advanced as a criterion, which might then admit the dolphin and some other creatures; but there are, of course, difficulties about defining 4
5
Figure 21.3 Wiliiamjames (1842-I9I0),American philosopher and psychologist, who concluded thai even a headlessfrogmusl be conscious (Mary Evans Picture Library/ SocietyfirPsychical Research).
326
311
or evaluating ethical values, even in man. William James had a rather complex view. Hefirstof all said that the cerebral cortex was the organ of consciousness, but then when he was pressed for a criterion he said that it was based on 'the pursuance of future ends and the choice of means for their attainment'. And this led him to conclude quite logically, on the basis of evidence from Pfluger and Lewes and the other contemporary physiologists of the end of the nineteenth century, that the headless frog must be conscious, because it shows adaptive behaviour: it was reported that if it was stimulated on one limb and lhat limb was prevented from removing the source of irritation, another limb spontaneously substituted itself to carry out the purposeful activity. So he concluded that the spinal cord of a frog has conscious intelligence. When 1 put this to a physiological colleague he said, 'Well it may well be true but it is very difficult to address the question to the frog's head, now detached.' But it does raise precisely the question of Just why a frog with a head should be considered to be more conscious than one without on the basis of precisely defined criteria. 6
A view that has become increasingly popular in recent years links evolution with an emergent property, which assumes that as nervous systems evolve they become more and more complicated, and that at some stage, not exactly clear when, somehow this mysterious property of self-awareness or consciousness becomes an attribute of the nervous system. But exactly how one recognizes it when it happens remains the problem. At the other extreme (and this species of philosophical argument still exists, sometimes but not always linked with reductionist views) is the position that consciousness is just an epiphenomenon - what T . H . Huxley called the steam-whistle on the locomotive: it has no effect on the engine of the locomotive but is just there as an ancillary and powerless gadget. Lloyd Morgan challenged Huxley very direcdy and asked, if it is useless, why did it ever evolve? And any evolutionist has to answer that argument. 7
Well why did it evolve? One view often discussed these days is that advanced earlier by Michael Chance and Alison Jolly," and more recently by Nicholas Humphrey,'' that somehow the pressures from social complexity made it useful, perhaps necessary, for consciousness to evolve. The fact that wc arc complicated social creatures produces this extraordinarily strong need for each of us to attribute consciousness to other creatures so that wc can try to predict their behaviour by role-playing and other forms of thought. And being able to do so generates important advantages for survival. I would like to take just the opposite point of view, namely that creatures are indeed socially complicated throughout the whole vertebrate kingdom, but that most other creatures except man have very good ways of dealing with it. Man, in fact, is the only creature that perversely gets into social difficulty of any really serious kind, and one reason for this is because he is conscious and can think about all the social complications that he might confront, or deviously try to exploit for gain or for protection. And so I think the social complications are a consequence of the powers of consciousness rather than the other way around. It
327
312
is the fact that we have the capacity to think and be aware and consider what other people might be thinking about us that produces, I think, the uniquely human condition of paranoia. I do not know that paranoia, as a really damaging social-psychiatric condition, exists in any creature other than man, and human beings can have it only because their consciousness is so highly developed. 1 would like to stress a view about one essential property of attributed consciousness, and the one from which powerful evolutionary advantages derive, namely the view that Lloyd Morgan himself put. Lloyd Morgan is often cited as being a rather austere critic of people who are a little bit loose with their terminology, and is well known for his canon of parsimony, but in fact 1 think he had a broad range of quite creative insights, and he is very rarely quoted in full. The position I would like to adopt was put by him (quoting Mivart) this way: 'If a being has the power of thinking "thing" or "something", it has the power of transcending space and time. Here is the point where intelligence ends and reason begins.'" For my purposes, 'conscious awareness' can be substituted for 'reason' in this sentence. 1
The Evidence from Damaged Brains What 1 would like to do now is turn to some observations of neurological patients which, while they relate to just one restricted aspect of the problem of consciousness, raise the issue of the existence and the importance of knowledge of what it is one is doing, that is, knowledge of one's own capacities and one's own acts. Such observations, because they are concerned with the effects of damage to known neural structures, also provoke us to think about those neural structures and the type of organization in which they are embedded. I will deal with these observations of patients briefly, and then return to Lloyd Morgan's theme. There are patients with brain damage, and they arc clinically very striking, who have the capacity to learn quite adaptive skills and to carry out discriminative acts. If we were carrying out the very same acts, we would certainly be able to report and to communicate about those acts not only to others but to ourselves, that is, to think about them. These patients, in contrast, seem to have the capacity to leam and to perform in various ways but they are detached from awareness of doing so. We all know that a great deal of what we do in everyday life has just that character. In fact it would be a terrible waste of cerebral tissue and processing capacity to spend one's time thinking about all the skills that one has acquired. It would be a waste for us to recite, every time we spoke, the fact that we have indeed learnt every single word we were uttering; we no longer have a conscious memory of rhat; we have a skill. If we had to think, 'well now, that is mteresting; I can remember learning the English language and I can remember when I acquired a new word', and so forth, we would never get through a conversation. Or we would never be able to drive a car or indeed efficiently carry out any complicated skill. And there are a huge number of bodily responses of which we have no awareness at all. After all, the pupils of our eyes are responding enormously vigorously throughout our entire day, and I do not know 328
313 anyone who has ever reported an awareness of what his pupil is doing. In fact it doesn't behave like a mere photocell, changing only in relation to the level of light entering the eye: it also responds to the sight of pretty things, to beauty, to touch, and to all sorts of emotional disturbance. It has even been used as an index in marketing research, because it has been reported that, when a picture is taken secretly of someone's pupil when he is looking at goods in the supermarket, the diameter of the pupil correlates with the attractiveness of the product, and thus one can predict the sales values, say, of certain types of packaging. There is nothing inherently strange in the fact that we do lots of complicated things without awareness. What is strange is finding that there are patients who do things that we are normally aware of but of which they are not. There are two main types that I would like to review briefly.
The 'Amnesic Syndrome' The first are patients who suffer from the 'amnesic syndrome'. It can be caused by a large number of different agents, such as herpes simplex encephalitis - which is a very devastating form of it - by chronic alcoholism, by various kinds of poison. Surgeons can induce it if they ablate both medial temporal lobes of the brain, or just one lobe if the other one happens to be defective. We know the brain structures that must be damaged if the amnesic state is going to occur. You can also see this syndrome in an extreme form in very elderly or pre-senile patients, such as those with Alzheimer's Disease (although they have other problems as well), but it can occur in a pure form in young people and to a very extreme degree. The most striking clinical feature is lhat these patients, despite retaining normal intelligence and perceptual and cognitive skills, report not remembering anything at all from minute to minute; it really is a crippling disorder. You can be talking to such patients, then leave the room and come back in a minute: the patient docs not recognize you at all and admits of no memory of having seen you. So devastating is this clinical condition that for a long time it was thought that these patients were incapable of learning anything at all. But gradually it has become clear that in fact they arc capable of learning a large variety of new things, and of doing so quite efficiently. For example, they can be subjected to classical Pavlovian conditioning of a sort that each of us would surely report remembering, which they seem to acquire at normal rates. They show verbal learning, they show perceptual teaming, they show motor skill learning, and they retain these over long periods of time, sometimes for weeks or months." But the only person who does not acknowledge that anything has been Icamt or display any recognition of this fact is the patient himself. So there is a very clear dissociation berween the capacity to learn and to retain, on the one hand, and the patient's knowledge of that fact, on the other.
'Blindsight' The other type of patients I wish to describe suffer from lesions in the
329
314 first way-station in the cerebral cortex at which visual inputs are normally received, the striate cortex or first visual cortical area ( ' V - I ' ) . There is a very orderly projection of the visual field mapped on to this region of cortex. For a long time it was thought that such patients were rendered blind, absolutely blind, in that part of the visual field corresponding to the damaged portion of the projected map in the striate cortex. But in some patients, at least, it has proved possible to demonstrate that they have a very good capacity to discriminate, to detect visual events in this "blind' area, to locate them in space, to make judgements about the orientation of lines or gratings, to detect the onset and termination of movement, and even to do simple pattern discriminations. And yet there is no acknowledgement of any such capacity at all: subjects say when questioned that they are just guessing and plaving the experimenter's game when he insists on their saying 'yes' or 'no' when a light is flashed briefly in their 'blind' field, or insists on their saying whether a grating is horizontally oriented or non-horizontally oriented, or insists on their reaching to the spot where the 'invisible' flash occurs on a screen, none of which they can actually see. Indeed with the first patient that my associates and 1 tested, we did not (and still do not) give the patient knowledge of results during the test; when he was told his results afterwards he just expressed open astonishment. He thought he was just guessing. We called the phenomenon 'blindsight'. There are various gradations of this capacity which 1 cannot review in detail here but which have been recorded elsewhere. For example, under some conditions the patient will say that he does not 'see', but has a 'feeling' that something is there, and sometimes he may even achieve a strange kind of visual expriencc. A similar kind of phenomenon has been reported with damage in another sensory system — the somatosensory system - which has been called 'blindtouch'. T h a t is, the patient can locate touch stimuli on his skin, although he has no awareness of actually being touched.
12
13
Criticisms Recently it has been claimed that this evidence is based on artefact, and because the matter has been raised rather heatedly I want to write just a word or two about diis attack. One argument is that we do not know where the patient's lesion is precisely, or whether the cortex has been only incompletely damaged. T h i s is true: none of these patients (fortunately) has died yet, and so post-mortems have not been done; but I shall later mention some evidence that suggests that one can make pretty good guesses about this point. T h e other argument is rather more serious: it is that the investigators who reported 'blindsight', and there are now a goodly number of such reports, have not controlled for diffusion or scatter of light in the eye or stray light in the environment that reaches an intact part of the visual field, and it might be that the object is responding to such subde cues. I think the evidence against that argument is really very strong. There are several lines of evidence, but perhaps the best is based on a natural built-in control provided by a structure in each eye called the optic disc. This is truly and absolutely blind because it is where the optic nerve fibres penetrate the retina and hence where there are no 14
330
315
90%
90°
0°
180°
97% 93%
90%
77% 270°
Figure 21.4 Experiments on 'blind-sight'. Here we set a plot of the visualfieldofthe left eye nfa patient who had suffered damage In a large pari ofihe primary visual corlex(thestriare cortex) in his right hemisphere. The patient staredfixedly at ihe centre of a hemisphere (shown here as the large circle) while a small light was presented al different pmnls on ihe hemisphere to determine the limits ufthe visualfield ofihe left eye, shown as a dashed line. The brain damage has caused a scoroma or area ofblindness, shown by the dolled region. Visual stimuli presented within this region were no/ consciously 'seen' (although the hatched region indicates an area within the scotoma where presentation ofa stimulus produced a'feeling'that something had happened). Then a small spot of light was briefly flashed, 30 limes each, at the positions within the area ofblindness marked by spots. Under controlled conditions, where guessing would produce only SO percent correcl judgement, the patient hail lo say whether the lighl had been presented or not. even though he never tlmsiioiisly saw it. The figures show that he was correct between 77 per cent and 93 per cent of the lime - wellubtme chance performance. On ihe other hand, the score was only 43 per cent(indistinguishable from pure guesswork), if the lighl was deliberately presented so lhat lis image fell in the middle ofthe optic disc (where the optic nervefibres pass oul through a hole in ihe back oj the eye and which is therefore literally blind). The conclusion is lhat ihe visual ability ofihis patient wilhin the 'blind'area, though very limited and unconscious, was genuine and could not bettttomtedfurby sralleringnf lighl on lo the seeing part of the retina, since that shouldhave happened loan equal extent for the light falling on the optic disc.
receptors and no vision. This natural 'blindspot' is small, only 5° by 7° in angular size, which gives one a well-defined blind area within which to estimate how much effective light scatter there can be with a particular stimulus. If you have a spot of light that absolutely disappears when its image falls on the optic disc, you can really be sure that there is no discriminable scatter beyond about 2°. Now it usually happens that the optic disc, the natural blindspot, in one eye falls within the 'blindsight' field of a patient with damage in the visual cortex. A small spot of light therefore can be directed randomly either to the true 'blindspot' or to the apparently blind region immediately surrounding it. When studied in this way, the patient 331
316 shows chance performance when asked to detect the spot when projected on to the true 'blindspot' but very good detection performance when the light is projected into the 'blindsight' field outside it. But as far as he is concerned, he does not know which is which: he is guessing all the way through. There are various other compelling control experiments, and so I do not have any doubt about this point. 15
In the context of criticism of the observations on human patients it is illuminating to consider the original reason why 'blindsight' was ever discovered and pursued. Originally, there was evidence that monkeys could still discriminate certain types of visual events after damage to the striate cortex, and by and large they show just about the same capacity, in quantitative terms, that human patients with 'blindsight' can show. It was just because the evidence from experiments on animals suggested a residual function after conical damage that human subjects, who had always been thought to be quite blind, were reassessed using the same kinds of techniques that wc had used to demonstrate the residual capacity in animals. What came as a genuine surprise was not only that such a capacity could be demonstrated in a 'blind' field caused by striate cortex lesions in humans, but that the patients themselves were disconnected from any awareness of the stimuli they were discriminating. 3
In the primate wc have relatively good knowledge about what part of the brain is important for the residual visual function, and we know that the residual capacity can be detected even in the complete absence of striate cortex. And, as it happens, some of the patients in whom positive evidence of'blindsight' has been reported are also highly unlikely to have any striate cortex whatever left in one hemisphere, because they had their entire cerebral cortex surgically removed from that hemisphere (to alleviate otherwise uncontrollable epilepsy or other disruptive pathology). There is, incidentally, no inherent mystery about how the residual 'blindsight' might be mediated in the nervous system in the absence of striate cortex: the eye sends six or seven different outputs to various regions of the brain, especially to areas called the tectum and pretectum in the midbrain, that bypass the striate cortex. Presumably these areas, which would survive damage to the cerebral cortex, are responsible for the visual function in such patients. 16
Well, what are the implications of these two classes of patients, those with 'amnesic syndrome' and those with 'blindsight'? Both demonstrate really quite striking capacities in the absence of the patient's own conscious knowledge. T h e person can process information if it leads to a straightforward and unambiguous route from stimulus to response, in the absence of 'thought'. What I think has become disconnected is a monitoring system, one that is not part of the serial informationprocessing chain itself, but which can monitor what is going on. I think that is the kind of neural organization one is looking for in order to explain awareness. T h e presence of such a system might be assessed in the following way: imagine that we had to discriminate berween, say, a horizontal and a vertical grating, and had to press one of two keys to indicate which orientation had been presented; we would have no difficulty also in indicating on a third key whether we were 'seeing' the difference or merely 'guessing'. T h i s is where we would differ from the
332
317 'blindsight' patient who might be able to make the judgement of orientation quite accurately but deny seeing the pattern at all. Similarly, the amnesic patient could make a response based on stimulus information to which he had previously been exposed, but his third key would indicate 'guessing' and not 'remembering'.
W h e r e D o e s 'Monitoring' O c c u r ? Where is the brain structure involved in monitoring? I do not know. But I think that is the kind of organization that one has to look for, and 1 think that the point in the nervous system at which it is likely to emerge is not in the striate cortex, because that is too early in the chain, but at the first place at which information from the whole of the visual field comes together. ( T h e striate cortex in each hemisphere has input from only one half of the visual field.) We know where that coming together is in the monkey, and therefore we have some guesses where it occurs in man. In the monkey it is part of the cortex that lies in the inferior regions of the temporal lobe - several synapses beyond the striate cortex of each hemisphere - that first combines information from both half-fields, and therefore can deal with the whole visual world as an entity. It is within this region, in fact, that we have reason to think that the neural representations of external objects takes place. 17
It is all very well to speculate on the basis of such observations of these patients, with whom we can converse about their experiences, or lack of them, in relation to their impressive residual capacities. I have said that a monkey with comparable brain lesions also demonstrates comparable residual capacities; indeed, it was because of the work on animals that much of the human clinical work was pursued. But how could we know whether the sort of'monitoring system' that I have suggested constitutes consciousness is connected or not in a non-verbal animal? We have no way of conversing with the monkey, or at least no one has solved that problem yet. But I think there may be an interesting lead which has come from an ingenious idea put forward by some Canadians, Beninger and colleagues. They have, in effect, allowed laboratory rats to do what they like and then asked them if they knew what they were doing. Actually they did not let them do quite anything they wished, but four possible things that rats do quite frequently and naturally without training: (1) facewashing; (2) rearing up; (3) walking; or (4) remaining immobile. So the animals had a choice of one of four things to do when they wished, and they could do any one of the four in any order that they wished. But in order to get food reward, after they performed any one of these acts they had to press one of four different levers, so as to indicate what it was they had done. And the rats could do that. Now I do not know whether a sea slug or crab could do that - I very much doubt it - or a frog, or a pigeon. I would be very surprised if a monkey could not do it. And I know if we gave that kind of test to an amnesic patient he would fail if there was even a short delay between the first act and the key-press. So 1 think there may be a methodology which could be developed to ask animals to indicate what it is they know about their own behaviour; or, for that matter, about
333
318
Figure 21.5 Does a rat know whal il J doing? According lo the experiments ofR. Bellinger another animal's behaviour; and beyond thai, in principle at least, about and his colleagues ihis rat docs their own behaviour in relation to another animal's behaviour. know that it is washing its face Even here, there is a difficulty that is, in itself, quite revealing. One I© Arthur Christiansen, Frank highly adaptive feature of everyday life is the strong tendency to transform Lane Pit) u re Agency).
as much of our behaviour as possible into automatic routines. Achieving this, after all, will leave a cognitive system of limited capacity free to deal with those problems that still require thought. Quite complicated conversations can become automatic and, in a sense, unconscious, as in sleepwalking or in verbal exchanges about the weather. Many highly skilled persons, such as aircraft controllers, carry out highly complex but routine verbal exchanges without thinking and probably would have difficulty in reporting them after a delay if there were not some ancillary form of recording. And so the same thing might happen with any highly practised sequence, even in a rat. Tony Dickinson and others have invented one approach to trying to decide whether a sequence has reached this automatic state for a r a t . T h e strategy is to see whether the animal can adapt to a disturbance in a remote part of the chain or whether, alternatively, it continues to reel off its responses in an automatic way. For example, the value of the reward can be sharply increased or decreased to see if the animal nevertheless still persists in performing by 'force of habit', even though the end goal is no longer the same. In practice no doubt one will need a whole set of criteria, but conceptually I think there are ways of supporting and investigating the distinction between a 19
334
319 capacity or a skill and a monitoring of that skill.
T h e Adaptive V a l u e of'Monitoring' L e t me finish by asking what might be the evolutionary significance of the 'monitoring' function that creates conscious experience. 1 think its adaptive significance is just the one that Lloyd Morgan put forward, that it enables the animal to think about his own behaviour and others' behaviour, to communicate if it has words, as we do, so as to link objects and events in time and space that would otherwise be separated; in other words, to think and to reflect. T h e 'blindsight' patient, lacking the awareness of the discriminative skills that the 'blind' field actually possesses, has no image of his current visual experience to link to any other visual events. H e cannot compare or match by 'picturing'. Nor can the amnesic patient relate his current behaviour - even though it is affected by learning - to other 'memories'. This is one reason why amnesic patients are so severely incapacitated, despite evidence of such impressive residual capacities, and why they need chronic custodial care. ('Blindsight' patients are very rarely ever 'blind' in more than a half-field of vision, and so they can readily adjust by using their intact half-field.) T h e 'monitoring system', to paraphrase Lloyd Morgan, is where intelligent processing ends and consciousness begins. But also'implicit in this formulation is that action must be capable of redirection as a result of 'monitoring'; the monitor is not just another of Huxley's epiphenomenal stcam-whisdes. 'Monitoring' reflects a form of neural organization with a hierarchical capacity for control. Also implicit is the fact that sheer complexity of the nervous system, per se, is not a sufficient condition for such an organization to be generated as an emergent property, although it may be that only the more complex nervous systems can be so organized as to allow 'monitoring' and hierarchical control. Indeed, the very specific and circumscribed character of the neuropathological condition that dissociates a capacity from an acknowledged awareness, in itself, speaks against sheer complexity of the nervous system being a sufficient condition for emergence. T h i s is especially so given that other discrete sites of pathology give rise to quite different and dissociable disorders of cognition lhat arc, as it happens, just as revealing about other aspects of the structure of the mind. T h e r e is just one final point that ought to be touched on, so that I can fire the first shot before standing in the firing line of others. We all know the problem with the little homunculus in the brain: it is the problem of Ihe infinite regress. Who looks in on him, and so forth and so forth? I embrace the infinite regress; I have no worry at all about having another monitor looking at a set of other monitors, in thinking about thinking, and having an elaboration of hierarchical levels of organization, and hence of varying levels of abstraction in thought. It is not a little man doing the looking, but a pan of the nervous system itself that monitors other neural networks, rather than being firmly enmeshed as a link in a closed serial chain of processing. It is a matter of organization rather than of peeping Toms.
335
320 References 1 Darwin, C . (1871) The Descent of Man. London: John Murray (pp. 125-6 of 1894 edn). 2 Jennings, H . S. (1904) Behaviour of the Lower Organisms. Bloomington: Indiana Univ. Press. 3 Ibid. 4 Romanes, G . J . (1882) Animal Intelligence. London: Kegan Paul, Trench. 5 Morgan, C . Lloyd (1890) Animal Life and Intelligence. London: Edward Arnold. 6 James, W. (1890) The Principles of Psychology. London: Macmillan (p. 8 of 1901 edn). 7 Quoted by Morgan, C . Lloyd (1900) Animal Behaviour, p. 308. London: Edward Arnold. 8 Chance, M. R. A. and Jolly, A. (1970) Social Groups of Monkeys. Apes. andMen. London: Cape. 9 Humphrey, N. (1983) Consciousness Regained. Oxford: Oxford Univ. Press. 10 Morgan, C . WXtyjA, Animal Life and Intelligence, p. 375. 11 Warrington, E . K . and Weiskrantz, L . (1982) Amnesia: a disconnection syndrome? Neuropsycltologia, 20: 233—48. 12 Weiskrantz, L . , Warrington, E . K., Sanders, M. D. and MarshallJ. (1974) Visual capacity in the hemianopic field following a restricted occipital ablation. Brain, 97: 709-28, 13 Weiskrantz, L . (1980) Varieties of residual experience. Q. J. exp. Psychol. 32: 365-86. 14 Campion, J., Latto, R. and Smith, Y. M. (1983) Is blindsight an effect of scattered light, spared cortex, and near-threshold vision? Brain Behav. Sci. 6: 423-86. 15 Weiskrantz, L . (1983) Evidence and scotomata. Brain Behav. Sci. 6: 464—7. 16 Perenin, M. T . and Jeannerod, M. (1978) Visual function within the hemianopic field following early cerebral hcmidccortication in man. I. Spatial localization. Neuropsychologia, 16: 1-13. 17 Weiskrantz, L . and Saunders, R. C . (1984) Impairments of visual object transforms in monkeys. Brain, 107: 1033-72. 18 Beninger, R, J„ Kendall, S. B. and Vandcrwolf, C . H. (1974) The ability of rats to discriminate their own behaviours. Canad.J. Psychol. 28: 79-91.19. 19 Dickinson, A. (1985) Actions and habits: the development of behaviour autonomy. In L . Weiskrantz (ed.), Animal Intelligence, pp. 67-78. Oxford: Oxford Univ. Press.
336
Chapter 3
MODES OF COMPUTATION; PROCESSING AND L E A R N I N G
T h i s chapter is the most technically theoretical, in the sense of analytical treatments of computational and neural network models. However, an equilibrium has been reached between descriptions and equations, abstractions and applications, exactly soluble models and heuristic algorithms. Furthermore, the chapter attempts a blending of ideas coming from logic, information theory, algorithmics, statistical physics, and other sources. The famous 1982 paper of John Hopfield (reproduced in Vol. 1 of this Series) focused on self-coupled associative memory nets. Distantly influenced by spin glass physics, it opened wide an entrance to the world of attractor neural networks. Spin glasses are disordered magnetic materials, for which new analytical and numerical tools were developed in the late seventies and early eighties. Among our authors, the late E . Gardner, S. Kirkpatrick, G. Parisi, H . Sompolinsky and M . A . Virasoro are physicists with a strong background in the statistical physics of spin glasses. It is worth remarking that the two main spin-offs of spin glass physics in the eighties have been to analytical studies of neural networks {section 3d) and to numerical methods for combinatorial optimisation (section 3b). A few years later, these two apparently divergent contributions reconverged gracefully in the new domain of the "thermodynamics of learning", as described in section 3c. Section 3a contains two papers that cover important steps in the theoretical progress of the eighties: first, the notion of computing-with-attractors; and second, the notion of statistical analysis within interaction space (or, space of the repertoire of networks compatible with a given architecture). These are two key ideas for processing and learning in neurocomputation.
337
Section 3b is concerned w i t h h e u r i s t i c a l g o r i t h m s , i n s p i r e d by p h y s i c s or biology, t h a t confront h a r d issues of a l g o r i t h m i c c o m p u t a t i o n a l complexity (i.e. the size-scaling p r o b l e m ) and solve p r a c t i c a l t a s k s e n c o u n t e r e d i n engineering. S e c t i o n 3c addresses the issue of generalisation, v i z . l e a r n i n g a rule f r o m e x a m p l e s . T h i s l e a r n i n g task is different f r o m t h a t of a m e m o r y net, w h i c h stores a c t i v i t y configurations till it reaches s a t u r a t i o n (as evoked i n section 3 a ) . G e n e r a l i s a t i o n theory is another d o m a i n w h e r e p h y s i c a l m o d e l s of neur a l networks have m a d e a m a j o r i m p a c t (the t h i r d one i n our c o u n t ) , l e a d i n g to novel extensions of the notion of entropy (entropy of a l e a r n i n g m a c h i n e , f u n c t i o n a l e n t r o p y ) a n d thereby p r o v i d i n g useful confrontation w i t h o t h e r sets of ideas c o m i n g from logic a n d inference theory. Section 3d deals w i t h information-theoretic a p p r o a c h e s to e a r l y sensory processing, p r i m a r i l y vision. T h e psychological notion of c o m p u t a t i o n a l goal a n d the p h y s i c a l notion of o p t i m i s a t i o n are brought to b e a r together, i n various w a y s . F i n a l l y , section 3e offers a bundle of i n s p i r i n g ideas a b o u t o p t i m a l encodi n g and t e m p o r a l aspects i n perception (time-dependent spike t i m i n g ) .
338
sensory s t i m u l u s ,
3a Neural Networks
T h e first article (28) of this section, by John Hopfield, has been chosen among other papers, perhaps more commonly quoted, because it is a revealing document about the nascent stage of attractor neural networks in the first half of the eighties. This text is remarkable in at least two respects. F i r s t , it shows the heuristic power of numerical simulation that allowed the author to raise and explore a considerable number of issues, which were later tackled with the analytical tools of the physics of disordered systems (replica method, etc.). Second, it is noteworthy that many of the numerical estimates, obtained on small size networks (N = 20, 30 or 100), turned out later to be close to the exact results calculated in the thermodynamic limit, where the number of neurons A' tends to infinity. T h e last sentence of the paper mentions the word "spin glass". Indeed, in addition to Hopfield's clever combination of modeling and numerical simulation, the breakthrough in attractor neural networks is due to the fact that his presentation of neural net dynamics was couched in a language that made visible the connection with domains of statistical physics, where much relevant advance had occurred during the previous decade. T h e second article (29), published four years later, bears testimony to the progress accomplished in the meantime. The reader is referred to the treatise of Daniel A m i t , quoted in the general introduction, for a pedagogical coverage of the statistical physics of neural networks. But it will anyway be manifest that the notations used by Elizabeth Gardner are close to the language of magnetic phase transitions. A state of activity of a network of neurons corresponds to a configuration of spins. In this paper, the author introduces a new concept of considerable import: the notion of fractional volume of the space of interactions that is compatible with a set of constraints (such as the stability of a set of given spin configurations). This fractional volume vanishes when the number of spin configurations, chosen for storage, becomes too large, and this allows for a calculation of the maximum storage capacity in a very general manner. As the author remarks, her
339
s t r a t e g y defines an inverse p r o b l e m of o r d i n a r y s t a t i s t i c a l m e c h a n i c s ( w h e r e the i n t e r a c t i o n s a r e fixed a n d the spins are allowed to v a r y ) . H e r f o r m u l a t i o n of the p r o b l e m , a n d the d e m o n s t r a t i o n of its power a n d generality, c o n s t i t u t e thus a second b r e a k t h r o u g h t h a t m a y be d e s c r i b e d as a r e v o l u t i o n , not only for the s t u d y of n e u r a l n e t w o r k s , but also w i t h i n the c e n t u r y - o l d discipline of s t a t i s t i c a l m e c h a n i c s . Moreover, this strategy for the s t u d y of l e a r n i n g syst e m s h a s allowed for the establishment of bridges w i t h various other
fields
of science, u s i n g m e t h o d s of s t a t i s t i c a l inference as evoked in sections a n d 3c.
340
lc
ReprintedfromModeling
and
Analysis
in
Biomedicine,
pp. 369-389, 1984
© 1984 World Scientific Publishing Co Pte. Ltd.
C O L L E C T I V E PROCESSING AND N E U R A L STATES
J.J. HOPFIELD* California Divisions
Institute
of
of Chemistry
Pasadena.
Technology and
California
V. S.
Biology 91125
A.
and Belt Murray
Hill.
Laboratories New
V. S.
Jersey
07974
A.
Contents
1.
introduction
371
2.
Plausible Properties of Neurons
375
3.
Results for a Random 7^
379
A.
Why There Should be Limit "Points"
379
5.
Inserting Chosen Memories
380
6.
The Sense of Time and Time-order
384
7.
Prospects
387
References
389
'Research al Caltech suppoited in patt by the National Science Foundation, grant number DMR-8107494 Contribution Number 6802 from the Chemistry Department
341
1.
Introduction
The brain is a large physical system, and the principles of its operation must ultimately be describable in physical terms. There are enormous obstacles to accomplishing this. Unlike the physicist's large systems, which normally consist of a huge number of essentially identical simple objects, the brain is very complex. Biological evolution has endowed a brain with tens or even hundreds of different types of cells. The brain has a rich structure of regions, and cellular connection architecture within each region. If understanding the functioning of a brain entails knowing in complete detail the neuroanatomy of a brain, the task is hopeless for mammalian brains of different individuals are far from identical as well as extremely large. A typical central processor chip of a modern digital computer has between 10 and 10* individual gates on a silicon chip. It also has 10' to l u input and output lines. Given the circuit diagram of such a chip bul not told ihe way the chip was organized or the inputs arranged, it is essentially impossible to figure out how to use the chip. There are simply too many different ways that the inpul and output leads and sequences of signals could have been planned. Thus, even if the neuroanatomy and electrophysiology of a brain were given in detail, one would still no! be able to understand the processing basis of higher nervous function. Thus, the only way we are likely lo be able to understand the basic principles of higher nervous function — roughly, thinking, organized or content addressable memory, associations, etc. — is through modeling. The object of such modeling is to discover the kinds of "behaviors" that large and organized systems of "neurons" are capable of producing. In such modeling certain essences of neural behavior and neuroanatomy must be kept. At the same time, irrelevant details must be stripped away, or else all the computational time and computer programs involved will be devoted to irrelevant detail. We are not al this stage interested in precise modeling. An analog might be drawn with weather forecasting programs. If it is desired to predict tomorrow's weather based on today's, the details of actual geography — the Mediterranean Sea, the Maiterhorn, etc. — must be included, and a detaded geographical description of ihe temperature, water vapor, content of the air, solar flux, etc. must be given. If, on the other hand, we are still groping with questions like why there exist clouds, hurricanes, etc., the computer mode! could have a rather unearthly geography as long as it included the idea of oceans, continents, mountains, etc. It could work with only two kinds of molecules, one condensible or waterlike, and one inert but important for the absorption of infrared radiation. Il could thus give to single molecules composite properties of actual molecules. Our modeling of a large collection of neurons is similar in spirit to the weather modeling above. In neurobiology we are trying to understand the origin of the analogs of clouds, storms, etc. and to do so, we will abstract what we view lo be essential features of neurons and repress the rest. 4
:
342
372 The modeling to be described lies in the general area of memory, which is conceptually one of the simpler aspects of higher nervous function. Three questions about memory based on higher-level functioning which might be kept in mind are: How do memories stay intact, as entities, rather than being mixed with others? flow does the ability to recover an entire memory from any reasonable sub-part arise? Where does the strong time-order sense of many memories arise? (A simple and familiar melody whistled backward is utterly unfamiliar.) In thinking about such questions, we also want to keep in mind that the brain is in many ways unlike a modern computer. This point is exemplified by the disparity between the tasks which brain and computer do well. A brain is relatively poor at arithmetic compared to a computer. On the other hand, given a problem in pattern recognition such as the identification of a face, a computer is disastrously weak compared to a four-year old child. It is also exemplified by the behavior of brain and computer when devices fail. The failure of 1% of the gates in computer circuitry is an absolute disaster. About 1% of the nerve cells (neurons) in a brain die each ten years you get older. These are never replaced, and yet when as many as 5% of the cells have died, you continue to function amazingly well. The brain achieves this fail-soft behavior by operating in a far more holistic fashion than a computer does. In modeling, we must thus look for more collective phenomena than are generally used in present computers. In fact, one of the rewards of the neuromodeling of brain functions may be an understanding of how to do certain computer tasks in such a way that the impact of device failure is minimized and operation is of a more collective character. We begin by relating some ideas from computation with the general properties of content-addressable memory, and relating those general properties to a physical view of the world. In that way we will see that in principle simple physical systems can spontaneously achieve the essential aspects of a content-addressable memory (CAM). A typical CAM problem is how to recall a memory about a conversation I had six months ago. The memory includes a vague date, a location, the name and physical description of the individual with whom I conversed, the topic, the subject of the meeting I was attending, and so on. If you now ask me what Poggio told me at lunch al the meeting in Philadelphia in March, I would reply that the meeting was in Boston and that he told me about the effect of dendrite spines on . . . Any reasonable subset of the information would suffice — it is as though one had a computer or card catalog with all possible indexing schemes. It is also errorcorrecting — inconsistent information in the initial cues can be over-ridden.
343
373 1
Error-correction is much used in computers and in data transmission . A typical error-correction scheme is based on the following idea. A binary number of length n is to be encoded. The space of all binary numbers of length n is indicated by s, and particular such numbers are Sj. Consider a larger space 7" of binary numbers of length N, N > n. The elements of this space are 7",-. The encoding describes a 1 •* 1 map of S onto T, with a correspondence S, -+ T,\ This encoding map is called E. If the space 7" is much larger in size than space S, the elements I J can be chosen to be very far apart. (The distance scale used is the Hamming distance between binary numbers. Two binary numbers of length N areadistance d apart if they differin d of their bits.) The decoding map D is many-to-one, and maps a region r, in T near 7} onto Sj. The regions in T are compact and non-overlapping. These mappings are illustrated in Fig. 1. If the space T is sufficiently large, mistakes in one or even several digits of 7", will result in a point near Tjj and stilt in T(. The operation of the error correcting code is then very simple. When we want to remember Sj we encode it as ESj = Tj, If one or a few errors are made while it is stored, the result will be that a different element T will now be stored, but Tj will still be in v Thus the decoding map will yield DTj — Sj, and the correct number is thus retrieved from memory. The essence of all error-correcting codes is a mapping of the general type shown in Fig. 1. This mapping should be continuously updated if errors which occur at random in time are to be kept down. Thus, while the memory is stored, one should occasionally use the mapping ED of T onto T to keep the stored memories i from wandering very far from Tj. This product mapping ED introduces a flow in space 7 toward the stable memories T, as illustrated in Fig. 2.
Fig. 1 Tlx mapping of binary q»cei which encode! and decodes data in an error-correcting code.
344
374
Space T
The map ED of T onto T Fig, 2 item>ing
The flow pictured i n Fig. 2 contains the essence o f a CAM. The stable p o i n t s o f the flow are the " m e m o r i e s " o f the s y s t e m
1 , 3
. I f the system is started w i t h
partial i n f o r m a t i o n about one o f the memories, this starting p o i n t is represented by a p o i n t i n phase space near t o that particular m e m o r y and farther from
the
others. When the kinetics o f the system produce the flow illustrated, a system started near some particular m e m o r y w i l l later be at the memory state itself, and
thus the entire m e m o r y w i l l be generated. Thus the essence o f a CAM is a
dynamical system whose phase space contains a set o f fixed p o i n i s , the memories o f the system. N a t u r a l l y , i f the m e m o r y is t o be useful the previous experience o f the system must be capable o f locating ihe m e m o r y state vectors wherever desired. Many physical systems display dynamical behavior such that the natural phase space flow o f the system has stable l i m i t points (or small l i m i t regions w i t h tluctuations w i t h i n a region). Sometimes such a flow is difficult to understand. I n many cases, however, there is an u n d e r l y i n g structure to the problem w h i c h makes the reason for the existence o f stable points clear. The flow map o f Fig. 3 has no
345
375
Fig. 3 A flow map in state space for a ample system exhibiting a content-addressable memory. This map does not indicale why the memory states are stable. obvious meaning. B u l Fig, 4 shows a contour map o f a terrain, w i t h mountain tops (maxima) and lake b o t t o m s indicated. A drop o f water placed o n such a terrain w i l l proceed d o w n h i l l in a "steepest decent" path u n t i l i t reaches a lake b o t t o m . The paths o f all such drops on this terrain w o u l d produce a flow map, and Fig. 3 corresponds t o the flow map o f the terrain o f Fig. 4 . Thus the existence o f a terrain, coupled w i t h the idea ( o r law o f physics! that natural water flows dectease the height w i t h time, explains w h y the flow map o f Fig. 3 has stable points. We w i l l couple together a highly abstracted
sel o f neurons, give them a
plausible dynamics, and demonstrate the existence o f such a flow and the underlying " t e r r a i n " or reason for this kind o f f l o w . The capabilities o f this k i n d o f system to do content-addressable memory tasks w i l l be brielly summarized. 2.
Plausible Properties of Neurons The resting potential inside a typical axon or nerve process is aboul - 1 0 0
millivolts w i t h respect l o the outside. Such axons are w e l l - k n o w n to be capable o f solitary waves or "spikes" o f electrochemical activity propagating along an axon. These spikes are actively propagated
w i t h o u t loss, the energy c o m i n g from the
electrochemical gradients across the cell membrane.
346
376
l
{ ' ' * } ; solt lake
o
w
contours
Fig. 4 The contour mip underlying the How pattern of Ftg. 3. WiMr dropleti filling an juch a terrain would yield the flow pattern of Fig. 3. A cell body initiates action potentials in axons at a frequency or 'Tiring rate" which depends on the mean potential of the cell. That mean potential depends i n turn on the ion leaks of the cell, which in turn depend on the chemical signal from other
cells. Thus a particular cell may crudely be characterized by an "output"
firing rale which is caused by Ihe sum o f all the inputs to a cell. The typical experimental input-output curve (Fig, 5) is sigmoid, with firing rates less than zero ot greater than a maximum of about
500 sec"
1
not possible. This description sup-
presses all the details of the time structure of the firing pattern. We will presume lhat the dominant information in the system is contained in an appropriate short-term mean of the firing rate. There are circumstances in which the quantal nature of
347
377
t OUTPUT
INPUT
*-
Fig. 5 The input-output (oi voltage firing rate) telation fot a typical neuron and the jtepfunction approximation to be used in the modeling. spikes and their time separations will be essential. But just as classical electromagnetic theory, which neglects the quantal properties of light, is awfully useful in an appropriate realm, (such as understanding how television works) we believe the firing rate description will also have a useful domain* The input to a given neuron conies from the other neurons which make synapses with it. We will write Input to i = X TijVj S
where Vj is the firing rate of neuron / and 7» is ihe strength of synaptic connection from neuron / to neuron r. The actual linearity of this relation has not been well characterized. In real life, both signs of synapses are possible, but each neuron / tends to specialize in one sign of connection. At present, ihis is a needless complication, and our model neurons will each be allowed to have Ty of either sign for all i and all /. A linear sysiem does not ttuly compute. (There are at ihe same time, ideas in the field of linear associative "* nets which have some relevance to the nonlinear analysis.) Given a set of information, the linear systems cannot generate more. If we are trying to construct a system which reconstructs a total memory from a partial input, it is necessary that the system be nonlinear. Of course, the inputoutput relation of Fig, 5 is nonlinear. In modeling we will emphasize this non5
348
378 linearity by replacing the smooth input-output relation by a step function, (We now know from unpublished computer modeling by John Piatt that replacing the step by a smooth but steep sigmoid curve could have no major effect on the conclusions reached.) Thus at any particular time each neuron will be characterized as "on" or "off", V— 1 or 0, in the fashion of McCollough and Pitts . 10
To describe a flow in state space, a time dynamics of the neural states must be given. We use a delayed, stochastic, dynamics in which each neuron i stochastically but at mean rate W examines the sum of its inputs. If at that instant the sum of the inputs Input to f =
= £ TijVj I
is greater than the threshold /,', then neuron will switch to the state V, = 1, or remain in that state if already there. Similarly, if the sum of the inputs is less than Ii', the neuron / will switch its state to 0, or remain there if there. Given Tjj this algorithm completely defines the stochastic flow of the state of the system. The algorithm seems very arbitrary. Its choice was dictated by a desire to include noise, asynchronous operation, and propagation delay effects, while at the same time keep the mathematics and simulation problem tractable. The algorithm can be argued to represent qualitatively such effects without the need to include irrelevant details. Some assumption must be made about the matrix T,j in order to see whether the algorithm yields a flow with stable limit points. In a brain or even a simple insect ganglion, typical cells are connected to about 1,000 to 10,000 other nearby cells. While there is also some long-range connectivity, the dominant connectivity is to nearby cells. One might take a collection of about 1,000 cells, each capable of getting signals from each (or many) of the others as a plausible fundamental processing unit. The short-time behavior of such a finite and densely interconnected unit should have relevance to brain functions. Within such a unit, the possibilities of which cell connects to which other is greatly restricted by the development of the brain. In addition, both the particular synapses which are made and the strength of those synapses are believed to be influenced by experience. We begin by suppressing all such details, and consider a collection of neurons all with /,•' = 0 and with uniformly randomly chosen synapses strengths in the range 1 < Tj/ < I. Such a beginning is a confession of ignorance of detailed structure and experimental history. Bul just as a model of magnetism wilh one kind of magnetic ion and some particular exchange interaction is capable of examining whether ihe phenomenon of magnetism exists, our test case is capable of demonstrating that flow patterns with stable minima result. —
349
379 3.
Results for a Random '/',.-,
Simulations were carried out on a digital computer for JV = 10 and JV « 30 neurons, with varying numbers of neurons connected to others. Ten turns out to be too small a number to demons!rate collective behaviors well. For N = 30, a randomly chosen would produce a flow with about three dominating limit "points". Sometimes the limiting states were not single states, but instead small cycles like ABAB. . , Some of the limit "points" turned out to be small limit regionsin which the state wandered stochastically over a small region of state space. If the closeness of two digital states, each being a word of N bits, is defined as the Hamming distance between them (die number of digits in which they differ), all the states seen in the limit region were within about 4 units of each other. (Two random states would be about IS units apart for the case N = 30.) The dominating limit points were the end points of typical runs from random initial states. There are, in addition, other limit points which are very unlikely to be reached from random starting states. There may actually be many of them, but they would be of little processing significance. Thus the dynamics defined on the state space of JV-bit words by a random Tij and the stochastic algorithm produces the kind of flow in state space which is essential to content-addressable memory. There were no disasters like flows which wandered chaotically through space in an ergodic fashion. 4.
Why there should be Limit "Points" Consider the function H=-
TTijViV, i.i The change in H due to changing P j while keeping all other Vj fixed is 0
= - ( I ^ / K ^ A ^ ;'
-
According to the algorithm, A P can be positive only when X J V j Vj > O.and can be negative only when this sum is less than zero. Thus, following the algorithm, A / / is always negative. (It is assumed throughout that there is no self-coupling, i.e. T = 0.) ( o
i o
H
For the special case of a symmetric synapse matrix 7",y = Tu, one immediately notes that Atf, = A / f / . Thus for a symmetric matrix the algorithm is minimizing a bounded function, and must converge on a minimum which is at least locally stable. 0
a
350
380 When Tij is not symmetric, no such constraint exists, A / / , less random change at each step while &h',
a
is a more or
0
is monatonic. Such a mode of
behavior would be present for a symmetric system which was also in contact with a thermal reservoir, so that a "free energy" function is what was truly being minimized, and the energy tends to a compromise between lowering itself as much as possible and the phase space available (at finite temperature) for low energy excitations. Simulations show that when T is not symmetric, a system started in a random state (with N near zero) rapidly "cools", attaining a large negative H, then shows fluctuations in H around that value while wandering in a small final region. 5.
Inserting Chosen Memories In the literature of linear associative memory, the problem of constructing
or learning a matrix which will generate a given set of vector input-output relations has been extensively dealt with. An understanding of this literature suggests how lo build Tij in our problem in order that specific vector states i V'\V.
m
. (naturally
Vj = 1 or 0 for all i and s) be the stable memories. The following explanation is applicable for the special case / ' = 0 and for states with, on the average V\ = \• • Obvious simple alterations are necessary when this average is smaller. The biological idea behind this modeling is the modifiable synapse at the basis of 3
memory""' . In addition, we will pick the memories as random vectors, thus random A'-bits digital words. (This supposition of randomness is appropriate if the information being stored has been optimally coded. Thus, for example, coding DNA sequences appear
roughly
random. It could also be taken as representing our lack of
knowledge of how memories are stored.) We guess
m - *$m - wi - !> s The stability of a particular state s can be treated by examining
ZT„vf= iZ% &
a
I sfs'
(2Vf
- 1)1(217' - Ipf i
whose sign determines whether V ; should be 0 or 1. T h e first term, the double sum, has an average value of zero, and its value has no correlation with V* . In the second term, the sum has an expectation value of JV/2. Thus on the average, the net second term is A /2 if V'' is +1 and -JV/2 if V - - 0. The expectation value of ihis second term precisely stabilizes the state s'. T h e first term represents r
s
351
381 a noise term, and can be shown by statistical analysis to be much smaller than the second if the number of memories is not loo large. For a number of memories n, < O.liV, noise is not serious. Results of simulations are shown in Fig. 6. In each case, a set of 100 neurons was examined to see where the systems found a stable final state beginning from a nominal memory slate. For n = 5, essentially all memory states were exactly stable. For n = 10, the majority of memory states were stable with few errors, but some became unstable. For n = 15, only about half the nominal memory states are stable, and the remembered memories are typically a little in error. The memory saturates when too many vectors are simultaneously stored, and few of the nominal memories are then stable. The incremental rule s
s
s
AT„ = ( 2 K f - 1)(2 V - s
1)
when a new memory V* is to be inserted is very attractive, since it does not depend on what memories have been previously stored. In addition synapse 7// needs only information on K j and which is locally available to il, and needs no information about the global nature of a memory. A modification of ihis rule is essential lo forget old memories in order lo incorporate new ones. Many such modifications are possible, but a particularly simple one is ClTij = (2Vf
-
\)(2V\
-
1)
if such an increment leaves \TH\ <
T
m31i
if not. AT/; - 0 . m
If Tmax 4, then new memories are written into memory with almost the usual efficacy, while old memories become unstable and disappear. Associated with this perhaps surprising jesult is the amazing robustness of Ihe memory storage to changes in T. When a matrix is constructed in the usual way, the new matrix = sign(r ) y/
has the same memory states as T, wilh only a slight increase in noise or errors. This fact allows ihe use of a 7/,- matrix consisting only of two symbols 1 and - I , or by appropriate transformations, 0 and I . The major operation in calculating
352
382
Number of Errors in State Fig. 6 Statistics of the storage of memories by a network of 100 neurons, A particular bar, say at an abscissa of 1. gives the probability lhat an actual stable state of Ihe system had one error compared to a starting state of the system taken as a particular memory.
353
383 /, is Ihe multiplications and additions in
E i
m
When both T and V consist only of 0 or l's this can be done with logical "and", no multiplications, a small lookup table, and a few additions. In that way, a single computer word can be used to store 30 elements of T or of V. T h e computation time reduces to about 1/50 of that needed for implementing a full Tff of the usual sort. When the system is started very near a stable memory, that particular memory is found. As the distance between a starting state and the memory sought increases, the probability of reaching the correct memory decreases. This fact is illustrated in Fig. 7 for the case of 30 neurons with known stored memories. The system has I and -1 as the two states of V, and has ten stored memories, five randomly assigned and their complements formed by the replacement +1 ** ± 1, Sirs ol Bias Informalion
h
0.5 _
Fig. 7 The probability or going to a particular Halting state starting it i n Initial state containing
a number n of errors. In this 30 neuron system, a Tandom starting slate would have 15 errors. There are ten stable assigned memories in this model, and the chance tesult is indicated by Ihe horizontal dashed line. The erratic nature of the curve arises from the small size of the statistical sample collected in the modeling.
354
384 6.
The Sense of Time and Time-Order
For a random matrix, a sense of the flow of time is generated by Ihe tendency for H to decrease in time. Occasionally behaviors are seen in which the system settles into a metastable state, remains there a while, then finds a pathway out and falls to a lower H state or region. An example is shown in Fig. 8 which displays the value of H as a function of time for a particular run. H "decreases" to a "stable" plateau, then decreases to another plateau around which it wanders because Tjj Tp. These two plateus were found lo be distinct regions in Hamming space. One is then led to ask whether the anti-symmetric part of 7",, could be systematically used to result in an ability of the system to remember a time sequence of states, and to reproduce the sequence given the starting state. 1
The essential change in Tu to recall a sequence V^.V ,... to add a lerm A
S
t
1
(2V f*
-l)(2Vj
in order is
- 1)
to Tjj. This term has Ihe same kind of locality at site ij as the usual one. and is a trivial generalization. If before we were to think of Tjj as a time average Tu
- 4L(r,(r) -^){v,(t)
-i)
* experience time the additional term is a time delay, replacing V/(t) by Vfif - T ) while leaving C/(() unchanged. Any biochemical hardware conceptually capable of producing the previous Ty could equally easily be induced to generate the new prescription. By inserting in the anti-symmetric part of T,y the direction of where to go next is clearly indicated. It proves to be much more difficult to persuade the collection of neurons to move from one state to another. If A is too small, ihe minima are stable. If A is too large, the system moves rapidly through state space and never settles into the particular memories very well. The algorithm can be modified so that a small fraction of lime state changes of the wrong sense are made, and this helps ihe system move from stable point to stable point with a small A. Careful adjustment of parameters ultimately permitted following a sequence of four memory states, but performance was not very good. It could be improved chiefly by laking a much longer time to wander between minima, and there were major uncertainties in the step lime under these circumstances. More seriously, the delicate balance of parameters necessary to achieve this result suggested that an essential ingredient was lacking. In computet terms, what is clearly needed is a mechanism to "clock" the system from one state to the next in the sequence. The antisymmetric part of
355
356
386 T defines Ihe sequence order or direction adequately well; the lacking element in Ihe present description is the timing aspect. One artificiality of most of the simulation described so far is lhat on the average half of Ihe neurons are in the active firing state at any lime. Cells should be on a much smaller fraction of the time, so fot simulations one might better use memories in which say O . I A ' (instead of 0.5iV) of the neurons are on the average "on". O f course, for a given /V more such smaller memories can be stored in Tjj. In addition, it now makes sense to include another common fealute of neurons, namely that there are elements of regulation and fatigue in the activity of an individual neuron. F o r a given internal voltage, the firing rate tends to drop with time, and the spike shape changes. Whether synaptic efficacy changes with time is not known, but it would be expected to drop. After a cell stops firing it tends to relax back lo iis initial condition. Within the algorithm we have used, Ihe essence of such effect is lo change the effective thresholds of a neuron with time. While a neuron is "on", its threshold will rise, which il will decay back toward a resting level while the neuron is off. This additional hardware serves as a "clock" which shuts off a stable state afler a while. Tn such simulations on 240 neurons a variety of behaviors have been seen. With a symmetric T, a low value of the intrinsic threshold produces a behavior in which one stable state turns on and remains for a while, than switches " o f f and is followed at random by another stable state. A higher value of threshold results in a stable state appearing for a while and then being replaced by the quiescent state (0000 . . .) which is now a very stable state. jNoie the interesting effect of threshold level even when it is noi timedependent.
A low threshold means the quiescent state is not stable. Any initial
state will rapidly converge to one of the stored memories. One could call this forced categorization — all inputs are presumed to fit into an existing category, A high threshold makes the quiescent state (0000 . . .)very stable. Under this circumstance, an initial state quite near some particular memory state will rapidly converge to that stale, while an initial state which is not close to any memory will rapidly lead to the quiescent state. The system categorizes the familiar but rejects the unfamiliar. The state (00000) becomes a positive identification of unfamiliatiiy.] When the anti-symmetric part of Ty is added, sequences are followed much better than before. The length of time "at" each state is no longer grossly erratic, and can be independently adjusted. During the residence " a l " at stale, thai state is produced wilh little error. There are still some errors in switching between slates, but following sequences is no longer a delicate business. It is certain that further refinements in neural "hardware" would result in excellent sequence-following.
357
387 The important lesson to be drawn is that simple additions to the repertory of the neural "hardware" can be essen'ial for generating additional classes of behavior. In ihis example, an effect which is usually thought of as a "regulatory effect" ot perhaps passed off as mundane "fatigue" may be essential lo information processing in a totally unexpected fashion. 7.
Prospects
Under this rubric, I will comment on some things which need to be done, on speculations on what the system might do, and anecdotally on other results already obtained. The replacement of the sigmoid firing rate versus input by a step seemed a drastic oversimplification. The structure of the mathematics is such as to allow the analyses near points of stability when the true sigmoid shape is used. This has not been carried out in detail, but what has been done shows that the step can be replaced by a smooth sigmoid with no sudden change in stability. Single neurons have been allowed lo make bolh excitatory and inhibitory synapses. If there are two classes of cells, each of which makes only one type of synapse, it alters the signal-to-noise a little but does not change the basic behavior. While 100-1,000 strongly interacting celts may represent a small patch of cortex or insect ganglion, neurons which are further away are less likely to receive inputs or outputs. The spatial range of connectivity is an important potential addition, particularly including the difference in typical length of excitatory and inhibitory connections. New phenomena and new time scales will be produced. We have done little analysis in this exciting direction. Preliminary work has been done with Richard Palmer on systems with a timedependent synapse strength, and a synapse modification algorithm which depends on the present neuronal slate and state of ihe synapse. The aim of this line of research is to produce a memory system functioning in an environment and making decisions and categorizations for a model organism. The work suggests the importance of the control of learning. A system in which each synapse is a twoterminal device (sees only the slate of its two neurons) has fundamental limitations. It is, we believe, important TO have external control of part of the learning algorithm, probably by some non-specific regional effector which signals whether it is appropriate that a given spatial region be cutrently in a learning (or perhaps forgetting!) mode. The analysis of the accessability of memories should be possible. The question here is how many "bits" of correct information should be essential to recall a given memoty. The interesting computational point is how this number scales with A', Information theory shows that the number n of correct bits (the rest being random) must be n > )V ln n . David Feinstein has shown that n < eN for any finite ( is more than sufficient. 7
s
358
388 The "fail-sort" aspect of this memory when synapses or neurons ate arbitrarily eliminated is readily seen from the general reasons for the existence of the stable minima. Computer simulation and detailed analysis bear out the gentle degtadation of all memories by such failures. If a memory chip were made on this basis the yield problem is very different. If the error rate in fabricating individual gates is 10 , it should in principle be possible to make a memory with about 10 gates and 10* memory bits, containing 10 bad gates, and still have a memory with content addressability which functions essentially perfectly. The way this scales with JV is expected to allow a single chip memory of scale 0,01 JV , where N is the average number of gates which can be fabricated with a 50% probability of a single malfunction. As JV gets larger, the size of a fail-soft CAM can become truly immense. s
10
J
The relation of the modeling to processes which go on in real biological structures is somewhat remote. One cannot necessarily search in too elementary organisms for the aspects of CAM described here, for there is neither evidence that simple nervous systems have a reconstructive memory nor an essential need for such a reconstructive process in simple systems. An insect or mollusc might simply have a large repertory of fixed categories for its pattern recognition, and merely switch its consequences of an identification from learning or conditioning. Or it might have flexible categories — for example, single cells which tune themselves to a particular stimulus — without having collective aspects necessary. Thus, whether collective CAM will exist at the level of a mollusc or insect is far from clear. Nonetheless, to try to relate this system to one of the elegant simple learning systems such as Gelperin and coworkers" have studied is an obvious challenge. One reason for the attempt to model a simple biological system is the need to understand what the behavioral symptoms of such a system would be, and whether they have any clearly identifiable hallmarks. A major biological reason for hoping that collective aspects will be important processing tools in higher nervous function concerns the evolution of intelligence. If it is necessary to evolve a computer-brain connection by connection, and to have the connections all completely organized like a Cray in order to function, it is hard to understand how intelligence evolved. If on the other hand large assemblies of cells have important computational properties when their cellular properties are correctly chosen, then the evolutionary problem is much simpler. 1 do not mean to imply that a structureless nervous system could compute usefully, but rather that a system with gross structure and rules at a cellular level might, independent of details of the micro structure of this or that particular brain, manage to compute. There is considerable modeling lore in neurobiology which is less directly related to issues raised here, but which the interested reader may find fruitful reading. Some of these are listed as Refs. 12-29. In addition, some of the mathematics has connection with ideas from the spin glass . 30
359
389 References 1.
See foi example, C, E. Shannon and W. Weaver, The Mathematical Theory of CommuniUniversity of Illinois Press (1959), R, I . McEiiecc, The Theory of Communication and Coding. Addison-Wesley (1977). J. J. Hopfield, Proc, Natl. Acad. Sci. USA, 79, 2554 (19B2). J. J. Hopfield, Engineering and Science ICattech/46, 2 (1983). D. H. Perkeland T. H, Bullock, Neurosci. Res. Symp. Summ. 3, 405-527 (1969). cation,
2. 3. 4. 5.
L . N. Cooper, Physical
6. 7. 8. 9. 10. 11.
12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30.
Systems,
in
Proceedings
of
the
Nobel
Symposium
on
Collective
Properties
of
edited by Lundqvisl, B. and Lundqvist, S., Academic Piess, New York,
2S2-264 (1973). L. N. Cooper, F. Libeiman, F. and E. Oja, Biol. Cybern. 33, 9-28 (1979). 1. C. Longuel-Higgins,Proc. Roy. Soc. L o n . BI71, 327-334 (1968). J. C. Longuet-Higgins. Nature 217, 104-105 (1968). T. Kohonen, Associative Memory — A System-theoretic Approach. Springer-Verlag, New York (1977). G. Palm, Biol. Cybem. 36, 19-31 (1980). C. Sahley, J. W. Rudy, and A. Gelperin. / Comp. Physiol. 144, 1 (1981); C. Sahley, A. Gelperin, and J. W. Rudy, Proc. Natl. Acad. Sci. USA 78, 640 (1981); A. Gelperin, S. J. Weiland, and S. R. Barry, in Primary Neural Substrates of Learning and Behavioral Change, ediled by D. Alkon and J. Farley, Cambridge University Press (1983). J. G. Eccles, 7)ie Neurophysiological Basis of Mind, Clarendon. Oxford (1953). . E. R. Kander.Sct. Am. 241, 61-70 (1979). D. R. Smith and C. H. Davidson, Assoc. Comput. Mach. 9, 268-279 (1962). L. D. Harmon, Neural Theory and Modeling, edited by Reiss, R. F., Stanford University Press, Stanford, California, pp. 23-24 (1964). S. -1. Amari, Biol. Cybem. 26, 175-185 (1977). S.-1. Amari and T. Akikaiu, Uot. Cybem. 29, 127-136 (1978). W. A. Little. Math, Biosci 19, 101-120 (1974). J . M i r r , / , Physiol. 202, 437-470(1969). T. Kahonta, Content Addressable Memories, Springer. New York (1980). W. A, McCulloch and W. Pills, Bull.,Math Biophys. 5. 115-133 (1943). M. Minsky and S. Papert, Perceplrons: An Introduction to Computational Geometry, MIT Press, Cambridge, Massachusetts (1969). F. Rosenblatl, Principles of Perceplrons, Spartan Books, Washington DC (1962). G. Willwachei, Biol. Cybern. 24, 181-198 (1976). J. A. Anderson,Psych. Rev., 84, 413-451 (1977). C . L. Shaw and K . J . Roney. Phys. Rev. Lett. 74A, 146-150 (1979). D. Willshaw, in Associative Memory, edited by 1. Hinton, Erbaum Press (1981). T.J.Sejnowki. Biol. Cybem.. 22, 85 (1976); 22, 203 (1976); J. Math. Biol. 4, 303 (1977), D. O. Hebb, The Organization of Behavior, Wiley, New York (1949). S. Kirkpatrick and D. Sherrington, Phys. Rev. 17, 4384-1403 (1978).
360
Reprinted with permission from J. Phys. A: Vol. 21, pp. 257-270, 1988 © 1988 lOP Publishing Ltd.
Math.
Gen..
The space of interactions in neural network models E Gardner Department of Physics. Edinburgh University, Mayfield Road. Edinburgh EH9 3JK, U K Received 13 May 1987, in Final form 37 July 1987 Abstract. The lypical fraction of the space of interactions between each pair of N Ising spins which solve Ihe problem of storing a given set of p random paiterns as N-bil spin configurations is considered. The volume is calculated eiplicitly as a function of the storage ratio, a = p / N , or the value K{>0) of the product of the spin and the magnetic field at each site and or the magnetisation, m. Here m may vary between 0 (no correlation! and I (completely correlated). The capacity increases wilh the correlation between patterns from o = 2 for correlated patterns with « - 0 and tends to infinity as m lends lo 1. The calculations use a saddle-point method and. ihe order parameters al the saddle point are assumed lo be replica symmetric. This solution is shown to be locally stable. A local iterative learning algorithm for updating the interactions is given which will converge to a solution of given • provided such solutions exist.
1. Introduction There has been a lot of recent interest in McCulloch-Pitts (1943) neural networks (Hebb 1949, Little 1974, Hopfield 1982). Analytic results (Amit et al 1985a, b, 1987a, b, Kanter and Sompolinsky 1987, Mezard et al 1986, Bruce et al 1987, Gardner 1986) have been obtained for thermodynamic and dynamical quantities using particular storage prescriptions for the coupling strengths. The storage capacity for the Hopfield model for random patterns is p = 0.14N, while the pseudo-inverse (Kohonen 1984, Personnaz et al 1985, Kanter and Sompolinsky 1987) stores N linearly independent patterns. For very correlated patterns, each with magnetisation m, where 1 — m~ In N / N , there is a prescription (Willshaw et al 1969, Willshaw and Longuet-Higgins 1970) which stores of the order of N /(\n N) patterns. However, the maximum storage capacity of Ihese networks can be larger. In the random case, the maximum number of patterns is 2 N (Cover 1965, Venkatesh 1986a, b, Baldi and Venkatesh 1987) and we will show that this increases for correlated patterns. 2
2
The network is defined as follows. Ising spins, S j — * 1 , are defined on each site i, i = 1 , . . . , N . They are updated according to the rule S,(/ + l ) = s g n ( M / ) - 7 - )
(1)
where S,(f) is the Ising spin at time 1 and the internal magnetic field M O at time t and site i is given by
MO =-77? E W )
<> 2a
where J,j is the interaction strength for the bond from site j to site i. The interactions J and Jj, need not in general be equal. The field T, is a local threshold at the site i s
361
258 which is fixed in lime and Ihe interactions J„ are defined so thai {2b) at each site I The configuration {S,} is thus a fixed point of the dynamics (1), provided the quantity (3)
This paper follows a recent letter (Gardner 1987a) and will be concerned with the problem of choosing interaction strengths J,j such that p — a TV prescribed N-bit spin configurations or patterns. H = \,...,p\
# = ±1
i=
l,...,N
will be stored as fixed points of the dynamics defined in (1). It will turn out, however, that the requirement that each pattern is a fixed point is not sufficient to guarantee a finite basin of attraction and the stronger condition (4)
tfrW.ff})-!,)^
where K is a positive constant, will be imposed at each site r and for each pattern u.. Larger values of K should imply larger basins of attraction. The quantity of interest will be the density of states or the typical fractional volume of the space of solutions for the couplings {J^} to (2i>) and (4) and this will first be calculated. The volume vanishes above a value a of a which depends on the stability K and this determines the maximum storage capacity of the network. Secondly, a local iterative algorithm will be given which will converge to a solution of given K provided such solutions exist. In § 2, the volume will be calculated for uncorrelated patterns, where the thresholds T| are set equal to zero. For K = 0, the volume vanishes as a increases towards 2 and this determines the maximum storage capacity in agreement with the known results (Cover 1965, Venkatesh 1986a, b). The upper storage capacity Q ( K ) is calculated and decreases with n. In § 3, the calculation is repeated for patterns with a fixed magnetisation m and it is shown that the storage capacity increases with the correlation m between the patterns and, in particular, that a tends to infinity as m tends to 1 (for K=0). The network, therefore, can store more patterns if the constraints (4) are correlated. However, correlated patterns contain less information than random patterns and the information capacity of the network will turn out to decrease slightly with m. e
C
2
c
The calculation of the typical fractional volume of the space of interactions {J } which solve (4) is done by introducing replicas in this space while the prescribed patterns remain quenched. This is the inverse of what is done in the spin-glass problem (Edwards and Anderson 1975, Sherrington and Kirkpatrick 1975) where the interactions are quenched and the spins are allowed to vary. Since all pairs of spins are connected, the fractional volume can be obtained exactly using a saddle-point method. The integration is over variables, tj
in
362
259 and ihe replica-symmetric ansatz
Mf = M
B
q" = q
is assumed at the saddle point. The physical interpretation of the order parameter q is similar to that of the Edwards-Anderson order parameter in spin glasses and characterises the typical overlap between pairs of solutions for the couplings. As a increases, different solutions to (4) become more correlated and q increases. In particular, the fractional volume vanishes as q tends to its maximum value which is 1 (by equation (2b)) and the condition q = 1 therefore determines a . The local stability of the replica-symmetric solution is proved in the appendix. c
Since explicit solutions for the optimal ) are not known, it is necessary to have an algorithm for constructing solutions. In § 4 a local iterative learning algorithm will be defined which is a generalisation of perceptron learning (Rosenblatt 1962, Minsky and Papert 1969) to many threshold functions and to non-zero values of K necessary in order to obtain finite basins of attraction. The advantage of this kind of algorithm is that a convergence theorem exists. Provided solutions to the problem of storing the patterns with fixed K > 0 to equation (4) exist, the algorithms are guaranteed to converge to one such solution. v
2. C a l c u l a t i o n of the f r a c t i o n a l volume o r interactions for uncorrelated patterns with zero l o c a l threshold
In this section, the threshold T, in equation (1) will be set equal to zero and the f f will be taken to be random patterns. Since the quantity (5)
n
is one if the patterns can be stored and zero otherwise, the fraction of phase space V
T
which satisfies (2b) and (4) is given b y
for a given realisation of the random patterns {£f}.
The fractional volume V may be r
written
V r - f t V, where V, is the fractional volume in the space of interactions {/„} for fixed i. In the thermodynamic limit, we therefore have to study
We now assume that this quantity is self-averaging and it is necessary only to calculate (In V ) , the average of In V, over the quenched distribution of the patterns {(?; u. 1 , . . . , p}. This is done using the replica method,
(8
363
)
260 The method assumes the validity of the analytic continuation from positive integer to zero values of n. The expectation (V") is given by
1
*[n{n»!'(z y;) -")]'
w
i
where a = 1 n is the replica index and J° is the realisation of the J,, for replica a. The mean-held calculation of (9) is done by introducing integral representations of the 6 functions for each pattern fx and each replica a.
The average over the random patterns fj* in (9) at sites j
i gives
e x p [ I! l n c o s ( l x ; ^ / W ^ ] .
(11)
Neglecting terms which are of order 1/7V relative to the leading term, equation (11) becomes
B
The calculation of (9) can be done by introducing a variable q" ,
JV
f,iH
e
B
and a momentum F" conjugate to q" , in order to impose the constraint (13). The variable /•" will also be introduced for each a in order to impose the constraint (5). (V") can then be written
x e x p ^ N ^ G ^ t +G ^ . E J -
x(
J n d£a
£
tarf.M«m, £ „ ) + j £
^ + £ l f i , ) ]
B
04)
)])
where G,{q )
= In f]
o0
G (F !
B S
\
dx
a
\
^exp(iIx.A -iEx£e
, £ J = ln ft J d 7 ° e x p ( - 5 l F , , / i +
J
1 x.x )
1
olt
B
(15) (16)
F + J ' j A
because the integrals over x and A factorise over the patterns / i and the integrals over the J factorise over the s i t e s / In the large-/V limit (V") is given by taking the saddle point over the variables F , q and E of the function ag
G<,q ,F , ,EJ aB
< e
= G (q ) a
l
ag
ag
a
+ G (F ,E )2
aB
£
a
o<0
364
qF aS
aB
+1£
E„. a
(17)
261 in order to find this saddle point, the replica-symmetric ansatz ap
q
B
f
=q
a
= F
a
E" = E
<
3
l
foralla
(18)
will be assumed. This assumption is reasonable because the space of the solutions to (4) is connected; any solution of (4) can be continuously deformed into any other solution. In the appendix it will be shown that this solution is locally stable. The saddle-point equations for F and E are algebraic and so these variables can be eliminated and, as n tends to zero, (V) is given by ( V ) = exp[N (minG(q) + 0{l/rV))]
(19)
n
in the large-N limit where G(q) = a | Of l n / f ( ^ ^ j ) + J l n ( l - , j ) + U / ( l - g )
(20)
3
D
'=
expH/ )
,V_,.V (2ir)'
//(x) = |
d
'
<21)
Dz.
(22)
The maximum of G over the variable q is given by the saddle-point equation
The physical interpretation of the variable q at the replica-symmetric saddle point can be found by differentiating with respect to F , ••<»*
<24)
q is therefore the typical overlap between pairs of solutions to (4) and is similar to the Edwards-Anderson order parameter of spin glasses. As a -* 0, q -»0 from equation (23); for a = 0 all J, solve (4) and the typical overlap is equal to the most probable overlap between random pairs of configurations in the space of interactions. As a increases, solutions become more correlated and q increases. As q -* I the number of solutions tends to zero and the typical volume tends to zero. The upper storage capacity of the network is therefore given by taking the limit q-» 1 in equation (23). t
As q-*l, the integral in (23) is dominated by values of / > —
a = ( J ~ D/(f + ) ) '. ?
c
(25)
K
Taking the limit K -+0 in equation (25) gives a = 2 in agreement with the known results (Cover 1965, Venkatesh 1986a, b, Baldi and Venkatesh 1987), As the stability « increases, the constraints (4) become stronger and the optimal value a of a decreases. a ( i c ) is plotted in figure 1. c
c
c
365
262 t
\m=OS
o
i
2
Figure 1. The critical storage ratio, o , as a function of * for values of m =0, 0.5 and 0,8. t
3. Correlated patterns In this section, the calculation of § 2 will be repeated for correlated patterns including the local threshold term, 7",. A simple way of imposing a correlation between the patterns is to choose all of them to have the same magnetisation m. The are independent random variables with distribution P(^)
+ l).
= s(l + m ) 5 ( ^ - D + j ( l - m)8{it
(26)
The expectation (V") of equation (9) can be found by averaging over the distribution (26) and using the integral representation for the 9 functions in equation (10). This gives a term
exp{j^ln^(l + m )exp(Ml^x^r)+l(l-m)exp(iS^:^^) } .
I <j,*^)l
(27)
Expansion of the logarithm up to second order in £ „ J"j x ' / V N gives I
2
exp[-iI(mM„-T)x^r-^(l-m )(l(x:) +2 where q
a&
af
(28)
is given by equation (13) and
*^$4
m
Higher-order terms in the expansion vanish as JV-*co. The constraints (13), (29) and (5) are imposed by introducing order parameters F" , K", E", respectively. In the large-N limit, however, the effect of K" is of order 1//V relative to the other terms. e
366
263 in this limit, (V)
can be written
lim — ln
=
i t #H{J
fi
dMa
dE
d
dF
° J J ^ "°
x e x p ^ a G l ^ . M J +G ^ F ^ E J -
x(J"
ft
2^
d £ „ e x p [ r V ( G ( 0 , £ J + ^E„)]) ;
( 3 0
)
where G is again given by equation (16) and 2
/
f»
n
["<*> J^o
/
- i d - ^ I ^ - d "
E
o
oi)
where < ) means an average over the variable f with the distribution (26). In the large-N limit, ( 1 / j V ) ln(V") is given by taking the saddle point over the variables F , q , E and M„ of the function a0
a0
a
G(q ,M F ,E ) aB
ai
BB
= G\{q ,M )
o
o
a0
+ G {F ,E )-
a
2
BB
£
a
r*<:
q F af)
aB
+ iY, E
a
(32)
a
0
and the replica-symmetric ansatz (18), together with the condition M
a
= M
(33)
will be assumed in order to find a saddle point. The local stability of the solution is checked in the appendix. Elimination of the variables F and £ as in the previous section gives, for the limits n-*0, /v~-*co, (V") = exp[fVn(ext (q, G
0(1/N)\\
M, T) +
(34)
where the extremum ext means a maximum with respect to the variable M and a minimum with respect to the variables q and where G(q,
M, T) = G(
+ i(l-m)Jp/lntf[(
(
1
"^
/
7
+
^/)(l-,)-^]
+ ^ ) ( l -
g
) - ^ ]
Hln(l-q)+\q/(i-q))
(35)
v = M-T/m.
(36)
and where
The threshold T can therefore be eliminated. Any local external field can be compensated for by variation of the order parameter M. The physical interpretation of M at the replica-symmetric saddle point is obtained from equation (29) and is the typical ferromagnetic bias in the couplings.
367
264 In order to find the storage capacity as a function of a and m, the limit q-* 1 is taken in equations (36) and (37). The equation for cr (m, K ) is c
D
<37)
'(7r^')1
m-**r ,
J(- m- <>/(1— >"
where v is given by Ki+m)
+
\{l-t»
1
(
B
D,
)
/ J
+/
L-.,o- .." ((^ ) m
The storage capacity increases with correlation m, as one would expect, since the constraints in equation (4) become more correlated. In particular, for K = 0 and small values of m, (37) and (38) give a = c
2(l+2m7ir + 0 ( m ) ) 4
(39)
and as m tends to 1, a diverges as c
I
(40)
(l-m)ln(l-m)
K = 0. For general values of K and m, equations (37) and (38) can be solved numerically. In figure 1, a ( « ) is plotted for m = 0, 0.5, 0.8 and in figure 2, o ( m ) and u(m) are plotted for « •- 0. It is interesting to compare these optimal results with those of specific
for
c
e
0
I
M
1
a
Figure 2. The critical storage ratio a , the typical ferromagnetic bias M (for zero thresholds 7j) and the information capacily I as functions of m for « =0. c
368
265 storage prescriptions for the interactions. The divergence as m tends to 1 in equation (40) is obtained for a model of patterns which are very correlated (Willshaw et al 1969, Willshaw and Longuet-Higgins 1970). The number of different spins in the patterns is of order In N, implying 1 - m - l n N/N and the storage capacity is of order N / ( l n N ) instead of order N. This relation between m and a agrees with equation (4), although the largest value of the coefficient of 7VV(ln N) is a factor 2(ln if ~0.96 smaller than the optimal result (40) with K = 0. 2
!
1
Although the storage capacity increases with the correlation between patterns, the amount of information per pattern decreases. The total information capacity is the total number of bits stored in the patterns I =—
a ( m H i ( l - m ) l n [ i ( l - m ) ] + i ( I + m ) l n [ i ( l + m)]}.
(41)
e
For random patterns (m = 0) we have 7
I = 2N
(42)
or twice the number of bonds. The information capacity I , however, decreases slightly with m. For small m, 2
2
I = 2N [\-(2/tr-l/2
2
In 2 ) m ] - 2 N ( l - 0 . 0 S W )
(43)
and, as m tends to 1, J
/ = N V 2 1 n 2 = 0.721N .
(44)
In figure 2, / is plotted as a function of m for K =0. 4. Local iterative learning algorithm In this section, a local learning algorithm for updating the couplings which, provided solutions to (4) of given K exist, is guaranteed to converge to one such solution, will be given. The algorithm is a gradient descent and its convergence follows from a generalisation of the perceptron convergence theorem (Rosenblatt 1962, Minsky and Papert 1969). It is a generalisation of the algorithm for K = 0 (Wallace 1985, 1986, Bruce ei al 1986). The algorithm is defined as follows (for zero thresholds T,). Let be any set of couplings with the diagonal term J set equal to zero. A mask ef is defined at each site i and for each pattern u.. u
and the couplings are updated according to the rule AJ„ = . f f f f f .
(46)
The algorithm must be done in series over the patterns but can be done either in series or in parallel for the sites and is iterated until ef vanishes for each site i and pattern ft. Equation (46) is similar to the Hebb rule (Hebb 1949) except for the presence of the E ^ ; changes are made to enhance the recall of pattern it only at sites which are in error according to condition (4). The convergence theorem is stated as follows. Suppose a solution J % exists such that £? lJ*tf>(K
2
+ 8)(l(Jrj) )'
n
(47)
369
266 where 5 is some positive number for each pattern ii and each site i. Then the algorithm of (45) and (46) will terminate in a finite number of steps. Before proving the theorem, some notation will be introduced. The scalar product of a pair of interaction matrices J and U at the site t is defined by
=% W
(48)
and the norm of J at the site i by 2
Ph^iV-J),)*' .
(49)
Let {./•/'} be the set of interactions after n applications of (46) at the site i and let X P be defined by
J
O
0
'
-
<
»
>
n
" %\\J*l The theorem will be proved by assuming that the algorithm does not terminate after n steps, and that this requires X\ to become greater than 1 if n is sufficiently large. Since X\" is bounded above by L, by the Schwarz inequality this is impossible and the algorithm must terminate. At time step n, the numerator of (50) changes to N>
]
A ( / " - / * ) , . = f £ 0if j* ,
l
e
1/2
(* + « )
(51)
because of equation (47) and, therefore, at time step n the numerator of (50) is bounded below U
• i » ) > | / ' | i ( « + *)n + ( J
M
, 0 )
(
• J*)i.
(52)
The change in the denominator comes from the change in the norm of
if/""-./""), = 2e? I J^rtf
+ Ne?
}+* i
< U2K\\J "%
+ N)
e
(53)
since only wrong bits have e = 1 by equation (45) and so l
A\\J "%
+ N/2\\f%
(54)
for e = 1. Suppose the algorithm has been iterated n times (i.e. E?*Q has occurred n times) and has not terminated. The X)"' must be less than one at each step. Therefore, by (52),
\n>rn(
K
+
8)S-±jj^
(55)
for each m < n and so, by (54), TV
"-• T
/
( i
m
•
r*i
370
\~\~<
267 and so ^ > l * * t + 0 ( t / i t ) , ( K + ^ ^ T ) + 0(l/«))
'.
(57)
1
Therefore X j " becomes larger than one for sufficiently large «, contradicting the hypothesis that the algorithm does not terminate. The algorithm (45) and (46) can be generalised to include learning of the local threshold term % by defining a new site i = N + l which has spin f K , , = + l for all values of fi and letting J = Tj, Another generalisation is to the construction of a symmetric J . The change in J„, equation (46), is replaced by +
( N + 1
v
ty,~(s?+e?$0tf.
(58)
In this case, the convergence theorem can be proved only if the algorithm is done in parallel in the sites. The proof is similar to that of the asymmetric algorithm except that the scalar product at site i (48) is replaced by J - V = 1 W .
(59)
"i
5. Conclusions In this paper, a calculational method has been introduced which allows the maximum storage capacity of neural networks to be determined. In particular, if the patterns are correlated in the sense that they all have an equal magnetisation m, the capacity increases with the correlation between the patterns from a -2 for random patterns and diverges as m tends to one. This increase in capacity allows for the possibility that neural networks can be more efficient than comparison algorithms. If a is restricted to be less than one, as in the Hopfield model or in the pseudo-inverse, the recognition can be done more efficiently by simply comparing the noisy initial vector with each input pattern in order pN steps, whereas one step of parallel iteration in a neural network involves multiplying the N x N interaction matrix J by a vector and involves N steps. In this sense, provided the number of iterations to stability is not too large, neural networks can be more efficient if a is sufficiently larger than one. This relative efficiency therefore increases with the correlation m. Basins of attraction, however, are likely to be smaller in the neural network compared with recognition with nearly 100% noise for comparison algorithms. 2
h
Since no explicit expressions for the optimal couplings exist, it is necessary to have a method for constructing them. The algorithms of § 4 are proved to converge to a solution of given K provided such solutions exist. Other algorithms with convergence theorems similar to those of perceplrons also exist. For example (Gardner et al 1987) training with noisy initial vectors can also lead to finite basins of attraction. There are also algorithms like those of § 4 (Krauth and Mezard 1987) and algorithms which are similar but exclude the scaling of « by the norm of J at the site i (Diederich and Opper 1987). The algorithms are also similar to the back propagation algorithms of Rumelhart ei al (1985) used in hidden unit models, although in this case no convergence theorem exists.
371
268 The methods used in §§ 2 and 3 can be generalised to many other situations. If, for example, one is interested in the storage of patterns allowing for a fraction of the bits to be in error, the upper capacity can be increased (Gardner and Derrida 1988). This can be thought of as an optimisation problem with cost function equal to the total number of wrong bits, (60) where e? is defined in equation (45). For a < 2, the minimum cost function is zero, while for a > 2 this value increases. It is also possible to generalise to different distributions of the interactions; for example, J = ± 1 (Gardner and Derrida 1988). Associative memory and other properties of the learned models can also be determined using similar methods. In particular, the content-addressability as a function of K has been calculated for a diluted version of the model (Gardner 1987b). In this model finite values of K do lead to finite basins of attraction whose size increases with the parameter K. Numerical evidence (Forrest 1988) using the algorithms of § 4 for the fully connected model also suggests that finite values of K lead to finite content addressability. v
There are many other possible generalisations. In particular, the above calculations have been done with asymmetric J,j and it would be interesting to understand the effect of imposing the symmetry — J$ on the interactions. It would also be interesting to generalise the calculations to other properties of typical solutions, to cycles of patterns, (Kanter and Sompolinsky 1986) and to models with hidden units (Rumelhart etal 1985).
Acknowledgments I would like to thank B Derrida, H Gutfreund, D J Willshaw and the Groupe de Physique Theorique at Ecole Normale for useful discussions. I also thank the University of Edinburgh for the award of a Dewar Fellowship.
Appendix In this appendix we will show that the replica-symmetric solution of §§ 2 and 3 is locally stable. The stability is determined from the signs of the eigenvalues of the matrix of quadratic fluctuations in the rr(n + 1 ) variables M " , Q" , E" and F° at the replica-symmetric saddle point (23), (36) and (37) of equations (17) and (32). Because the solutions are unique in the replica-symmetric subspace, it should be necessary only lo consider transverse fluctuations to this space. The eigenfunctions of G ] and G whose eigenvalues are not degenerate with the longitudinal eigenvalues span an n(n -3)-dimensional subspace of the full n(n +1 )• dimensional space and their structure is the same as for the spin-glass problem (de Almeida et al 1978). I f it and t are the fluctuations in q and F"", respectively, for a
B
2
aB
a?
ap
2
a or B=a ot 0
372
Q
0
(Al)
269 while the fluctuations in M" and E" vanish. The values of c,, d,, i = 1, 2, 3, are chosen so that these eigenfunctions are orthogonal to the degenerate scalar and the vector eigenfunctions and span an n(n - 3)-dimensiona! space. 1
2
There are therefore two £ n ( n - 3 ) - f o l d degenerate eigenvalues of 3 G/d (q, (equations (17) and (32)} which are eigenvalues of the matrix iP-lQ'
+ R'
\
1
1 P-2Q
\
F)
(A2)
+ RJ
where !
p =
2
S G\ tq e
dq
a
dG
2
P'
SF*
oP
2
d G\
2
3q
a0
Q'
SF
SF
a0
oy
2
oy
3
d G\ a00
O0
dG
dq
Sq„ Sq
SF 2
Sq dq s
6> G
R'
3
Uridyl SF d
ySy
o0
yS
At a = 0, the solution to the mean-field equations (23) and (37) is q = 0, P'-2Q'+ R'-O and so the product of the eigenvalues of (A2) is - 1 . The solution is stable in this limit because it is simply an integral over the phase space of couplings. The sign -1 is due to change of variable F-* \F in equation (14) from its introduction as the variable conjugate to q. In this limit a-*a , q-*l and c
i " - 2 ( ? + / i - » — ^ - j A d + m) f
Dt + ( l - m ) [
Dt]
2
F-lQ'+R'^iX-q?
(A4)
and using equations (37) and (38), the product of eigenvalues is
[ '
K
(
1
+
m
J ™ , - ^ ( n ^ - ' ) x
[
m + m ,
+i
D
' ] +
L^u-^i' o^)
P
l D f
Dt
1
J^W^) T
which is negative provided « is positive. The sign of the product of eigenvalues therefore does not change as q increases from zero to one and a increases from zero to a , and the replica-symmetric solution is therefore stable. A vanishing eigenvalue occurs only in the limit a -* a and K -* 0. c
c
References de Almeida J R and Thouless D J 1978 J. Phys. A: Math, A m i l D J. Gulfreund H and Sompolinsky H 1985a Phys. 19850 Phys,
Rev.
1987a 1987b
Ann.
Phys..
Phys.
Rev.
A 32
1007
NY 173 30 A in press
373
Gen. Rev.
11 983 S5 1530
Lett.
270 Baldi P and Venkatesh S 1987 Phys. Rev. Leu. 58 913 Bruce A D. Canning A. Forresl B, Gardner E and Wallace D J 1986 Proc. Conf. on Neural Computing, Snowbird, UT (AlP Conf. Proc. 151) ed J S Denker (New York: A1P) p 65 Bruce A D, Gardner E and Wallace D J 1987 J. Phys. A: Math. Gen. 10 2909 Cover T M 1965 IEEE Trans. Electron. Comput. EC-14 326 Diederich S and Opper M 1987 Phys. Rev. Lett. 58 949 Edwards S F and Anderson P W 1975 J. Phys. F: Met. Phys. 5 965 Forresl B 1988 J. Phys. A: Math. Gen. 11 245 Gardner E 1986 J. Phys A: Math. Gen. 19 LI047
Networks
for
1987a Europhys. Leu. 4 481 19S7b in preparation Gardner E and Derrida B 1988 J. Phys. A: Math. Gen. 21 271 Gardner E, Stroud N and Wallace D J 1987 Edinburgh preprint 87/394 Hebb D O 1919 77ie Orgonirotion of Behaviour (New York: Wiley). Hopfield J J 1982 Proc. Natl Acad. Sci. USA 79 2554 Kanter I and Sompolinsky H 1986 Phys. Rev. Lett. 57 2S61 1987
Phys.
Rev. A 35
Kohonen T 1984 Self
380
Organisation
and Associative
Memory
(Berlin: Springer)
Krauth W and Mezard M 1987 / Phys. A: Math. Gen. 10 L745 Little W A 1974 Math. Biosci. 19 101 McCulloch W S and Pitts W A 1943 Bull. Math. Biophys. 5 115 Mezard M , Nadal J P and Toulouse G 1986 J. Physique 47 1457 Minsky M L and Paper! S 1969 Perceplrons (Cambridge, MA: M I T Press) Personnaz L. Guyon 1 and Dreyfus G 1985 1. Physique Lett. 46 L359 Rosenblatt F 1962 Principles of Neurodynamics (New York: Spartan Books) Rumelhart D E, Hinton G E and Williams R J 1985 Parallel Distributed Processing: Explorations in the Microsiructure of Cognition vol 1, ed D E Rumelhart and J L McClelland (Cambridge, MA: M I T Press) Sherrington D and Kirkpatrick S 1975 Phys. Rev. Lett. 31 1792 Venkatesh S 1986a Proc. Conf. on Neural Networks for Computing, Snowbird, UT (AlP Conf. Proc. 151) ed J S Denker (New York: A l P ) 1986b PhD thesis California Inslitule of Technology Wallace D J 1985 Advances in Lattice Gauge Theory D W Duke and J F Owens (Singapore: World Scientific) p326 —— 1986 Lattice Gauge Theory: A Challenge to Large Scale Computing ed B Bunk and K H Mutter (New York: Plenum) p 313 Willshaw D J, Buneman O P and Longuet-Higgins H C 1969 Nalure 122 960 Willshaw D J and Longuet-Higgins H C 1970 Machine Intelligence 5 351
374
3b Parallel Algorithms
T h e first article (30) of this section, by Scott Kirkpatrkk et al, is a classic. Moreover, it plays a central role in this chapter for several reasons. It contains lucid introductions to thermodynamics and statistical mechanics, to the Monte-Carlo (also called Metropolis) algorithm, and to spin glasses — on the one side — and to combinatorial optimisation, to computational complexity and worst-case analysis — on the other. T h e invention of the simulated annealing algorithm is a beautiful example of a breakthrough between separate well-established fields. It evokes an analogy with the annealing (prior heating and subsequent slow cooling) of a physical system to reach an equilibrium, assigning a "cost function" ("energy") to all configurations of the parameters describing a complex system, and a "temperature" to define a probability of transitions in the space of these configurations. T h u s , a mathematical procedure of minimising a function of many variables is formulated and studied in terms of concepts borrowed from the physical process of relaxation by annealing. Since then, this widely applicable algorithm has stimulated considerable follow-up work in both fundamental and applied directions. Note how computers, which perform as problem-solving devices, appear also in this article as problem-raising objects, because of the complexity of their design. Note also the interesting analogies between stochastic heuristic algorithms and natural intelligence processes. T h e second article (31) by Durbin and Willshaw reports on another algorithm, the elastic net method, that was inspired by problems arising in developmental biology (topographic mappings between different geometrical areas) and also in psychophysics {the matching problem in stereopsis). It is applied here to the prototype of all algorithmically complex problems — the travelling salesman problem. Both texts of this section are original articles that convey vividly the motivating circumstances that led to invention of efficient algorithms.
375
Reprinted with permission from Science. Vol 22Q. No. 4598, pp. 671-680, 0 1983 The American Association for the Advancement uf Science
Optimization by Simulated Annealing S. Kirkpatrick. C. D. Gelatt. Jr., M . P. Vecchi
In Ihis article we briefly review [he central constructs in combinatorial optimization and in statistical mechanics and then develop The similarities between ihe two fields. We show how ihe Metropolis algorithm for approximate numerical simulation of the behavior of a manybody system al a finite Temperature provides a natural Tool for bringing the Techniques of statistical mechanics to bear on optimization i We have applied this point of view io a number of problems arising in optimal design of computers. Applications to partitioning, component placement and wiring of electronic systems are described in This article. In each context, we introduce the problem and discuss the improvements available from optimization. Of classic optimization problems, the •raveling salesman problem has received the most intensive study. To tesi ihe power of simulated annealing, we used ihe algorithm on traveling salesman problems with as many as several thousand cities. This work is described in a final section, followed by our conclusions. r
Combinatorial Optimisation The subject of combinatorial optimization U) consists of a se( of problems that are central lo (he disciplines of computer science and engineering. Research in Ihis area aims at developing efficient techniques for finding minimum or maximum values of a function of very many independent variables (?), This function. usually called the cost function or objective function, represents a quantitative mea-
sure of the "goodness" of some complex system. The cost function depends on ihe derailed configuration of the many parts of that system. We are most familiar with optimization problems occurring in the physical design of computers, so examples used below are drawn from
with / V . SO that in practice exact solutions can be attempted only on problems involving a few hundred cities or less. The traveling salesman belongs lo the large class of NP-comptete (nondeierministic polynomial time complete) problems, which has received extensive study in the past 10 years U). No method for exact solution wilh a computing effort bounded by a power of N has been found for any of These problems, but if such a solution were found. It could be mapped inlo a procedure for solving alt members of the class. Lt is not known what features of the individual problems in the NP-complete class are the cause of their difficulty. Since the NP-complete class of problems conTains many situations of practical interest, heuristic meThods have been developed with computational require*
Summary. There is a deep and useful connection between statistical mechanics (The behavior of systems with many degrees of freedom in thermal equilibrium at a Unite temperature) and multivanate or combinatorial optimization (finding the minimum of a given function depending on many parameters), A detailed analogy with annealing in solids provides a framework for optimization of the properties of very large and complex systems. This connection to statistical mechanics exposes new information and provides an unlamiliar perspective on traditional optimization problems and methods.
that context. The number of variables involved may range up into the tens of thousands. The classic example, because it is so simply slated, of a combinatorial optimization problem is the traveling salesman problem. Given a list of /V cities and a means of calculating the cost of traveling between any two cities, one must plan the salesman's route, which will pass through each city once and return finally to the starting point, minimizing the total cost. Problems with ihis flavor arise in all areas of scheduling and design. Two subsidiary problems are of general interest: predicting the expected cost of the salesman's optimal route, averaged over some class of typical arrangements of cities, and estimating or obtaining bounds for the computing effort necessary to determine (hat route. All exact methods known for determining an optimal rou(e require a computing effort that increases exponentially
376
ments proportional To small powers of A'. Heuristics are rather problem-specific: there is no guarantee that a heuristic procedure for finding near-optimal solutions for one NP-complete problem will be effective for anotherThere are two basic strategies for heuristics: "divide-and-conquer" and iterative improvement. In the first, one divides the problem inlo subproblems of manageable size, then solves the subproblems. The solutions to The subproblems must then be patched back together. For this method to produce very good solutions, the subproblems musl be naturally disjoint, and the division made must be an appropriate one. so lhal errors made in patching do not offset the gains S. Ktopittfck ind C . D. Gdatt. Jr., arc rcECJirch staff members ind M. P. Vrcchi was a visiting scienriEr ZL IBM Thomas J. Waisc
672 obtained in applying more powerful methods to The subproblems W). In iterative improvement [5, 6). one starts with the system in a known configuration. A standard rearrangement operation is applied to all parts of the system in turn, until a rearranged configuration that improves the cost function is discovered. The rearranged configuration Then becomes the new configuration of the system, and the process is continued until no further improvements can be found. Iterative improvement consists of a search in this coordinate space for rearrangement sieps which lead downhill. Since this search usually gets stuck in a local but not a global optimum, il is customary to carry out the process several times* starting from different randomly generated configurations* and save the best result. There is a body of literature analyzing the results to be expected and the computing requirements of common heuristic methods when applied to the most popular problems [ I S ) . This analysis usually focuses on the worst-case situation—for instance, attempts lo bound from above the ratio between the cost obtained by a heuristic method and the exact minimum cost for any member of a famUy of similarly structured problems. There are relatively few discussions of the average performance of heuristic algorithms, because the analysis is usually more difficult and the nature of the appropriate average to Study is not always clear. We will argue that as the size of optimization problems increases, Ihe worst-case analysis of a problem will become increasingly irrelevant* and the average performance of algorithms will dominate the analysis of practical applications. This large number limit is the domain of statistical mechanics.
Statistical Mechanics Statistical mechanics is the central discipline of condensed matter physics, a body of methods for analyzing aggregate properties of the large numbers of atoms to be found in samples of liquid or solid matter (7). Because the number of atoms is of order 10" per cubic centimeter, only the most probable behavior of the system in thermal equilibrium at a given temperature is observed in experiments. This can be characterized by the average and small fluctuations about the average behavior of the sysiem. when the average is taken over the ensemble of identical systems introduced by Gibbs. In this ensemble, each configuration, defined by the set of atomic positions, {r,}. of the
sysiem is weighted by its Boltzmann probability factor. exp(-if({r;}j/A 7"). where £({•",}) is the energy of Ihe configuration, kg is Boltzmann's constant, and T is temperature. A fundamental question in statistical mechanics concerns what happens to the sysiem in the limit of low temperature— for example, whether the aloms remain fluid or solidify, and if they solidify, whether they form a crystalline solid or a glass. Ground stales and configurations close to them in energy are extremely rare among all the conftgurations of a macroscopic body, yet they dominate its properties al low temperatures because as T is lowered Ihe Boltzmann distribution collapses into the lowest energy state or states. B
As a simplified example, consider the magnetic properties of a chain of atoms whose magnetic moments. are allowed to point only "up or "down." states denoted by u,, = - 1 . The interaction energy between two such adjacent spins can be written / ^ u ^ j . Interaction between each adjacent pair of spins contributes - J to the total energy of the chain. For an jV-spin chain, if all configurations are equally likely the interaction energy has a binomial distribution, wilh the maximum and minimum energies given by - J Y / and the most probable state having zero energy. In this view, the ground slate configurations have statistical weight exp(-W/2) smaller than the zero-energy configurations. A Boltzmann factor, expt-EttaT"., can offset this if keT is smaller than J. If we focus on the problem of finding empirically The system's ground state, this factor is seen to drastically increase the efficiency of such a search. 11
In practical contexts, low temperature is not a sufficient condition for finding ground slates of matter. Experiments that determine the low-temperature state of a material—for example, by growing a single crystal from a melt—are done by careful annealing, first melting the substance, then lowering the temperature slowly, and spending a long time at temperatures in the vicinity of the freezing poini. If this is not done, and the substance is allowed to get out of equilibrium, the resulting crystal will have many defects, or the substance may form a glass, wilh no crystalline order and only metaslable, locally optimal structures.
the systems being optimized. We will introduce an effective temperature for optimization, and show how one can carry out a simulated annealing process in order to obtain better heuristic solutions to combinatorial optimization problemsIterative improvement, commonly applied to such problems, is much Like the microscopic rearrangement processes modeled by statistical mechanics, with the cost function playing the role of energy. However, accepting only rearrangements that lower the cost fund ion of the system is like extremely rapid quenching from high temperatures to T = 0, so it should not be surprising that resulting solutions are usually metastable. The Metropolis procedure from statistical mechanics provides a generalization of iterative improvement in which controlled uphill sleps can also be incorporated in ihe search for a better solution. Metropolis ei al. (8). in ihe earliest days of scientific computing, introduced a simple algorithm that can be used lo provide an efficient simulation of a collection of atoms in equilibrium al a given temperature. In each step of this algorithm, an atom is given a small random displacement and the resulting change, A£. in ihe energy of the sysiem is computed. If A£ £ 0. ihe displacement is accepted, and ihe configuration with ihe displaced atom is used as the starting point of ihe next step. The case L\E > 0 is treated probabilistically: the probability lhat the configuration is accepted is /*(A£) = cxp(-a,£/k T). Random numbers uniformly distributed in the interval 40,1) are a convenient means of implementing the random part of the algorithm. One such number is selected and compared with flAf). If ii is less Than P(,£LE). the new configuration is retained; if noi. the original configuration is used to start the next step. By repeating ihe basic step many limes, one simulates the thermal motion of atoms in thermal contact wilh a heal bath at temperature I. This choice of PihE) has the consequence that the system evolves inlo a Boltzmann distribution. B
Using the cost function in place of the energy and defining configurations by a set of parameters { i j . it is straightforward wilh the Metropolis procedure to generate a population of configurations Finding the ;. ^ -temperature state of a of a gfven optimization problem al some sysiem when a prescription for calculat- effective temperature. This temperature ing its energy is given is an optimization is simply a control parameter in the same problem not unlike Ihose encountered in unit5 as the cost function. The simulated combinatorial optimization. However, annealing process consists of first "meltihe concept of the temperature of a phys- ing" ihe system being oplimized at a ical system has no obvious equivalent in high effective temperature, then lower-
377
673 ing ihe temperature by slow stages until [he sysiem "freezes" and no further changes occur. At each temperature, the simulation must proceed long enough for the system to reach a steady state. The sequence of temperatures and ihe number of rearrangements of the {J,] attempted to reach equilibrium at each temperature can be considered an annealing schedule. Annealing, as implemented by the Metropolis procedure, differs from iterative improvement in lhal the procedure need not gel stuck since transitions out of a local optimum are always possible at nonzero temperature. A second and more important feature is ihat a son of adaptive divide-and-conquer occurs. Cross features of the eventual state of the system appear at higher temperalures; fine details develop al lower temperatures. This will be discussed with specific examples. Statistical mechanics contains many useful tricks for extracting properties of a macroscopic system from microscopic averages. Ensemble averages can be obtained from a single generating function, the partition function. 2.
z - r . ^ )
(i.
in which ihe trace symbol. Tr. denotes a sum over all possible configurations of ihe aioms in the sample system. The logarithm of Z. called the free energy. FIT), contains information about the average energy. <E(T)>, and also the entropy. 5(D, which is the logarithm of the number of-con figurations contributing to the ensemble at J*: - k T In Z = F\Ti = <E(D> - TS (2> B
Boltzmann-weigh led ensemble averages are easily expressed in terms of derivatives of F. Thus the average energy is given by <EiT)>
-fllnZ
=
[3)
d{Uk Ti Q
and the rate of change of the energy with respect lo the control parameter, T. is related to the size of typical variations in ihe energy by C[T)
=
d<E{T)> dT 2
[<E(T) >
~
2
]
kP B
In statistical mechanics CI 7) is called ihe specific heat. A large value of C signals a change in the state of order of a sysiem, and can be used in the optimization contexl to indicale that freezing has be-
This process is usually divided into gun and hence lhal very slow cooling is required, ft can also be used lo deter- several stages. Firsl. the design must be mine the entropy by Ihe thermodynamic partitioned into groups small enough to fit the available packages, for example, relation into groups of circuils small enough to fit asm = cm into a single chip, or into groups of chips <5> dT T and associated discrete components that can fit onto a card or other higher level Integrating Eq. 5 gives package. Second, the circuits are as_ ( 'C{T) dT signed specific locations on the chip. 16) Sffl = S{T,) This stage is usually called placement. h T where 7"i is a temperature at which S is Finally, the circuits are connected by known, usually by an approximation val- wires formed photolithographic ally out of a thin metal film, often in several id al high temperatures. layers. Assigning paths, or routes, to the The analogy between cooling a fluit wires is usually done in two stages. In and optimization may fail in one imporrough or global wiring, the wires are tant respect. Tn ideal fluids all the atoms assigned to regions lhat represent scheare alike and the ground stale is a regular matically the capacity of the intended crystal. A typical optimization problem package. In detailed wiring {also called will contain many distinct, noninlerexact embedding), each wire is given a changeable elements, so a regular soluunique complete path. From the detailed tion is unlikely. However, much rewiring results, masks can be generated search in condensed matter physics is and chips made. directed al systems with quenched-in randomness, in which the atoms are nol At each stage of design one wants to all alike. An important feature of such optimize the eventual performance of the systems, termed "frustration." is that system without compromising ihe feasiinteractions favoring different and in- bility of the subsequent design stage s. compatible kinds of ordering may be Thus partitioning musl be done in such a simultaneously present {9}. The magnet- way that the number of circuits in each ic alloys known as "spin glasses," which partition is small enough to fit easily into exhibil competition between ferromag- the available package, yet the number of netic and antiferromagneiic spin order- signals that must cross partition bounding, are the be si understood example of aries {each requiring slow, power-confrustration (10). It is now believed thai suming driver circuitry) is minimized. highly frustrated systems like spin glass- The major focus in placement is on minies have many nearly degenerate random mizing the length of connections, since ground slates rather than a single ground this translates into the time required for state with a high degree of symmetry. propagation of signals, and thus inio the These systems stand in the same relation speed of the finished system. However, to conventional magnets as glasses do to the placements with the shortest implied crystals, hence the name. wire lengths may not be wirable. because of the presence of regions in which the The physical properties of spin glasses wiring is too congested for the packagat low temperatures provide a possible ing technology. Congestion, therefore, guide for understanding the possibilities should also be anticipated and minimized of optimizing complex systems subject during the placemen! process. In wiring, to conflicting (frustrating) constraints. it is desirable to maintain the minimum possible wire lengths while minimizing sources of noise, such as cross talk bePhysical Design of Computers tween adjacent wires. We show in this and ihe next two sections how these The physical design of electronic sys- conflicting goals can be combined and tems and the methods and simplifica- made the basis of an automatic optimization £ employed to automate this process tion procedure. have been reviewed (//. 12). We first provide some background and definiThe tight schedules involved presenl tions related to applications of ihe simu- major obstacles to automation and optilated annealing framework to specific mization of large system design, even problems that arise in optimal design cf when computers are employed to speed computer systems and subsystems. up the mechanical tasks and reduce the Physical design follows logical design. chance of error. Possibilities of feedAfter the detailed specification of the back, in which early stages of a design logic of a system is complete, it is neces- are redone to solve problems that besary to specify ihe precise physical real- came apparent only at later stages, are ization of the sysiem in a particular tech- greatly reduced as the scale of Ihe overnology. all system being designed increases. OpJ
1
378
674 The objective function /has precisely Ihe form of a Hamiltoman. or energy function, studied in the theory of random magnets, when the common simplifying assumption is made lhal the spins. yi have only two allowed orientations (up or down)* as in the linear chain example of the previous section. Il combines local, random, attractive ("ferromagnetic") interactions* resulting from the asfis, with a long-range repulsive ("aniiferromagnetic") interaction due to X. No configuration of the can simultaneously satisfy all the interactions, so the system is "frustrated." in the sense formalized by Toulouse {9). lm
2000 3000 4000 5000 Sum oi P"is on I t a two chips
6000
Fig. 1. Distribution of iofa( numbtr of pins required in iwo-way pa nil ion of a microprocessor at various temperatures. Arrow indicates best solution obtained by rapid quenching as opposed to annealing.
limitation procedures lhal can incorporate, even approximately, information aboul the chance of success of later stages of such complex designs will be increasingly valuable in ihe limit of very large scale. System performance is almost always achieved at the expense of design convenience. The partitioning problem provides a clean example of this. Consider N circuits thai are lo be partitioned between two chips. Propagating a signal across a chip boundary is always slow, so the number of signals required to cross between the two must be minimized. Putting all the circuits on one chip eliminates signal crossings, but usually there is no room. Instead, for later convenience, il is desirable lo divide the circuits about equally. If we have connectivity information in a malrix whose elements \a ) are ihe number of signals passing between circuits f and / , and we indicate which chip circuit i is placed on by a two-valued variable | i j = ± 1, then N . the number of signals that must cross a chip boundary is given by I ^ / f l ^ K ^ . , - p.,)-. Calculating l,y. gives the difference between Ihe numbers of circuits on the two chips. Squaring this imbalance and introducing a coe£ficient \- to express ihe relative costs of imbalance and boundary crossings, we obtain an objective function,/, for the partition problem; ri
e
r
H
<7)
If the are completely uncorrelated. it can be shown (JJ) thai Ihis HamilIonian has a spin glass phase at low temperatures. This implies for Ihe associated magnetic problem lhal there are many degenerate "ground stales" of nearly equal energy and no obvious symmetry. The magnetic state of a spin glass is very stable at low temperatures {14), so the ground states have energies well below the energies of the random high-temperature states, and transforming one ground stale into another will usually require considerable rearrangement. Thus this analogy has several implications for optimization of partition: 1) Even in the presence of frustration, significant improvements over a random starting partition are possible. 2) There will be many good near-optimal solutions, so a stochastic search procedure Such as simulated annealing should find some. 3) No one of the ground states is significantly better than the others, so it is not very fruitful lo search for the absolute optimum. In developing Eq. 7 we made several severe simplifications, considering only two-way partitioning and ignoring the fact that most signals connect more than two circuits. Objective functions analogous to/that include both complications are easily constructed. They no longer have the simple quadratic form of Eq. 7. but the qualitative feature, frustration, remains dominant. The form of the Hamiltonian makes no difference in the Metropolis Monte Carlo algorithm. Evaluation of the change in function when a circuit is shifted to a new chip remains rapid as the definition of / becomes more complicated. It is likely lhat the a are somewhat correlated, since any design has considerable logical structure. Efforts to understand the nalure of this structure by analyzing the surface-lo- volume ratio of components of electronic systems las in "Rent's rule" {15)} conclude that the u
Reasonable values of X should satisfy \ 35 zil. where z is the average number of circuits connected lo a typical circuit (fan-in plus fan-out). Choosing X = zJ2 implies giving equal weight to changes in the balance and crossing scores.
379
• •
• •
•
••
•
Fig. 2. Construction of a horizontal nel-cross• ng histogram. circuits in a typical system could be connected wilh short-range interactions if they were embedded in a space wilh dimension between two and three. Uncorrelated connections, by contrast, can be thought of as infinite-dimensional, since they are never short-range. The identification of Eq. 7 as a spin glass Hamillonian is not affected by the reduction lo a two- or three-dimensional problem, as long as \ N = z/2. The degree of ground state degeneracy increases with decreasing dimensionality. For the uncorrelated model, there are typically of order N" nearly degenerate ground states (M). while in two and three dimensions. 2 , for some small value. a, are expected {16). This implies that finding a near-optimum solution should become easier, the lower the effective dimensionality of the problem. The entropy, measurable as shown in Eq. 6. provides a measure of the degeneracy of solutions. S[T) is the logarithm of the number of solutions equal to or better than the average result encountered al temperature T. 2
As an example of the partitioning problem, we have taken the Logic design for a single-chip IBM "370 microprocessor" {17) and considered partitioning it into two chips. The original design has approximately 5000 primitive logic gates and 200 external signals (the chip has 200 logic pins). The results of ihis study are plotted in Fig. 1. If one randomly assigns gales to ihe two chips, one finds the distribution marked J = « for the number of pins required. Each of ihe two chips (wilh about 2500 circuits) would need 3000 pins. The other distributions in Fig. I show the results of simulated annealing. Monte Carlo annealing is simple to implement in this case. Each proposed configuration change simply flips a randomly chosen circuit from one chip to the other. The new number of external connections* C. to the two chips is calculated (an external connection is a net with circuits on both chips, or a circuit
675 connected lo one of [he pins of the original single-chip design^ as is ihe new balance score, B . calculated as in deriving Eq. 7. The objective function analogous ro I - J 7 is
where C is The sum of the number of external connections on ihe two chips and 8 is the balance score. For ihis example. \ = 0.01. For the annealing schedule we chose lo start at a high "temperature." To - 10. where essentially all proposed circuil flips are accepted, then cool exponentially. T = iT,/T )"T . wilh the ratio 7,/Ta = O.S". Al each temperature enough flips are attempted that either there are ten accepted flips per circuit on the average (for this case. 50.000 accepted flips al each temperature), or the number of attempts exceeds 100 times the number of circuits before ten flips per circuit have been accepted. If ihe desired number of acceptances is not achieved at three successive temperan
n
u
tures, the sysiem is considered "frozen" and annealing stops. The finite temperature curves in Fig. I show the distribution of pins per chip for the configurations sampled at T - 2.3. 1.0, and 0.1. As one would expect from the statistical mechanical analog, the distribution shifts to fewer pins and sharpens as ihe temperature is decreased. The sharpening is one consequence of the decrease in the number of configurations thai contribute lo the equilibrium ensemble at the lower temperature. In ihe language of statistical mechanics, the entropy of the system decreases. For this sample run in the low-temperature limit, the two chips required 353 and 321 pins, respectively. There are 237 nets connecting the two chips (requiring a pin on each chip] in addition lo the 200 inputs and outputs of The original chip. The final partition in this example has ihe circuits exactly evenly distributed between the two partitions. Using a more complicated balance score, which did not penalize imbalance of less than
100 circuits, we found partitions resulting in chips with 271 and IS3 pins. Tf. instead of slowly cooling, one were to start from a random partition and accept only flips that reduce the objective function (equivalent to setting T = 0 in the Metropolis rule). Ihe result is chips with approximately 700 pins (several such runs led lo results with 677 to 730 pins). Rapid cooling results in a system frozen into a metastable slate far from the optimal configuration. The besl result obtained after several rapid quenches Is indicated by the arrow in Fig. L
Placement Placement is a further refinement of The logic partitioning process, in which ihe circuits are given physical positions ill. 12, iS. 19). In principle, the two stages could be combined, although this is not often possible in practice. The objectives in placement are to minimize
niilal position
•
242 *21
rn
•
E3 •
0
El
'ii.
"
ASA 6i4 757
ii*i
EE]
"
6(2
i las
E3
416
4?6
55$
ssa
7&9
472
Ei
}
[so]
CD
iw
b
7.10
0
1
0
"
ST
1 I 70 01B
10 6
m
1
1 175
an.
a
Si
1 17ft
71 166
m
0
m 7.5
• EE
m
SI
M
ED if "'>
10
EH
ii • 515
HQ 1091 12*6
1127 11 &7
flSfl
501
J-0
T•l?SO 99
405
36 ft 91
itl
760 •a
559
5Z1
560
700 at
27
17
79
15
11
1
10
49
710 it 7|7 ^45 695 4"
si
(sjjj
IB
LI
» 0 CO '
•
79
595
•
542
16 If
D
91
7
m
m
5 SB
B II ii 0
•
19 5
i!
tt
1
» 0 0 Ql EH i i •
it.
591
S
'jtj
ts
0 i
•
y 0 E3 ED
611 5S0 625 HZ 5*0 13' t?7 700 S9E 737. S i n 932 »09 71s Fig. 3. Ninciy-cighr chips on a ceramic module from the I B M 3031. Chips are identified by number < I to 100. with 2D and 100 absent) and function. The dark squares comprise an adder, the three types of squares w . i h ruJed lines are chips that control and supply data to the adder, iha Lightly dotred chips perform logical arithmetic rbitwise A N D , O R , and so onf, and the open squares denole genera I-purpose registers, which serve both drithmeiic units. The numbers ai ihe left and lower edges of ihe module Image are the vertical and honzonial net-crossing histograms, respectively, u i Original chip placement: ibi a configuration at T = LtJ.OOO: IcJ T = 1250: tdh a zero-temperature result.
380
676 signal propagation limes or distances while satisfying prescribed elect ricaJ constraints, without creating regions so congested that there will not be room later to connect the circuits with actual wire. Physical design of computers includes several distinct categories of placement problems, depending on the packages involved {20). The larger objects to be placed include chips lhat must reside in a higher level package, such as a printed circuit card or fired ceramic "module" {21), These chip carriers must in turn be placed on a backplane or "board." which is simply a very large printed circuit card. The chips seen today contain from tens to tens of thousands of logic circuits, and each chip carrier or board will provide from one to ten thousand interconnections. The partition and placement problems decouple poorly in this situation, since the choice of which chip should carry a given piece of logic will be influenced by the position of that chip.
time as wire length, we use a convenient intermediate analysis of the layout, a net-crossing histogram Its construction is summarized in Fig. 2. We divide the package surface by a set of natural boundaries. In this example, we use the boundaries between adjacent rows or columns of chip sites. The histogram then contains the number of nets cross¬ ing each boundary. Since at least one wire must be routed across each boundary crossed, the sum of the entries in the histogram of Fig. 2 is the sum of the horizontal extents of the rectangles bounding each net. and is a Lower bound to the horizontal wire length required. Constructing a vertical net-crossing histogram and summing its entries gives a similar estimate of the vertical wire length. The peak of the histogram provides a lower bound to the amount of wire that must be provided in the worst case, since each net requires at least one wiring
The simplest placement problems arise in designing chips with structured layout rules. These are called "gate array" or "masier slice" chips. In these chips, standard logic circuits, such as three- or four-input NOR's. are preplaced in a regular grid arrangement, and ihe designer specifies only the signal winng. which occupies the final, highest, layers of the chip. The circuits may all be identical, or they may be described in terms of a few standard groupings of two or more adjacent cells. As an example of a placement problem with realistic complexity without too many complications arising from package idiosyncrasies, we consider 98 chips packaged on one multilayer ceramic module of the IBM 3081 processor {21} Each chip can be placed on any of 100 sites, in a 10 H 10 grid on the top surface of the module. Information about the connections to be made through ihe signal-carrying planes of the module is con¬ tained in a 'netlfSt." which groups sets of pins that see the same signal. The state of the system can be briefly represented by a list of the 9& chips with •heir x and y coordinates, or a list of the contents of each of the 100 legal locations. A sufficient set of moves lo use for annealing is interchanges of the contents of two locations. This results in either the interchange of two chips or the interchange of a chip and a vacancy. For more efficient search at low temperalures, it is helpful to allow restrictions on the distance across which an interchange may occur. To measure congestion at the same
'in'
io'
IO laimrtttuww
1
io'
Fig. 4. Specific heat as a function o( temperature for the design of Fig. 3. a to d.
Fig. 5. Examples of
381
channel somewhere on the boundary. To combine this information into a single objective function, we introduce a threshold level for each histogram—an amount of wire that will nearly exhaust the available wire capacity—and then sum for all histogram elements that exceed the threshold the square of the excess over threshold. Adding this quantity to the estimated length gives the objective function that was used. Figure 3 shows the stages of a simulated annealing run on ihe 9&-chip module. Figure 3a shows the chip locations from Ihe original design, with vertical and horizontal net-crossing histograms indicated. The different shading patterns distinguish the groups of chips that carry out different functions. Each such group was designed and placed together, usually by a single designer. The net-crossing histograms show that the center of the layout is much more congested lhan the edges, most likely because the chips known lo have the most critical timing constraints were placed in the center of the module to allow the greatest number of other chips to be close to them, Heaiing the original design until the chips diffuse aboul freely quickly produces a random-looking arrangement. Fig. 3b. Cooling very slowly until the chips move sluggishly and the objeclive function ceases to decrease rapidly with change of temperature produced ihe result in Fig. 3c. The net-crossing histograms have peaks comparable to the peak heights in the original placement, bul are much flatter. Al this ''freezing point," we And that the functionally related groups of chips have reorganized from the melt, but now are spatially separated in an overall arrangement quite different from ihe original placement. In ihe final result. Fig, 3d. the histogram peaks are about 30 percent less than in the original placement. Integrating them, we find that total wire length, estimated in this way, is decreased by about 10 percent. The computing requirements for this example were modest: 250.000 interchanges were attempted, requiring 12 minutes of computation on an IBM 3033. Between the temperature at which clusters form and freezing starts (Fig. 3c) and the final result (Fig. 3d! there are many further local rearrangements. The functional groups have remained in the same regions, but their shapes and relative alignments conlmue to change throughout the low-temperature part of the annealing process. This illustrates lhat the introduction of temperature to the optimization process permits a controlled, adapiive division of the problem
677
through the evolution of natural clusters at the freezing temperature. Early prescription of natural clusters is also a central feature of several sophisticated placement programs used in master slice chip placement (22. 23). A quantity corresponding to the thermodynamic specific heal is defined for this problem by taking the derivative with respect lo temperature of the average value of the objective function observed at a given temperature. This is plotted in Fig. 4. Just as a maximum in the specific heal of a fluid indicates the onset of freezing or the formation of clusters, we find specific heal maxima at two temperatures, each indicating a different type of ordering in the probtem. The higher temperature peak corresponds to the aggregation of clusters of functionally related objects, driven apart by the congestion term in the scoring. The lower temperature peak indicates the further decrease in wire length obtained by local rearrangements. This sort of measurement can be useful in practice as a means, of determining the temperature ranges in which the important rearrangements m the design are occurring, where slower cooling with be helpful.
Wiring After placement, specific legal routings must be found for the wires needed to connect the circuits. The techniques typically applied to generate such routings are sequential in nature, treating one wire at • time with incomplete information about the positions and effects of the other wires [ii. 24). Annealing is inherently free of this sequence dependence. En this section we describe a simulated annealing approach to wiring, using the ceramic module of ihe last section as an example. Nets with many pins must first be broken into connections—pairs of pins joined by a single continuous wire. This "ordering" of each net is highly dependent on the nature of the circuits being connected and the package technology. Orde rings permitting more than I wo pins lo be connected are sometimes allowed, but will not be discussed here. The usual procedure, given an ordering, is first to construct a coarse-scale routing for each connection from which the ultimate detailed wiring can be completed. Package technologies and structured image chips have prearranged areas of fixed capacity for the wires. For the rough routing to be successful, it must not call for wire densities that exceed this capacity.
Random Grid Alza TO
M.C. Z-parhe Qrld a'ta ic
Fig. 6 (left). Wire density in the 98-chip module wirh the connections randomly assigned to perimeter roures. Chips are in the original placement. Fig. 7 (right). Wire density after simulated annealing of the wire routing, using Z-shaped moves. We can model ihe rough routing problem land even simple cases of detailed embeddingi by lumping all actual pin positions into a regular grid of points, which are treated as the sources and sinks of all connections. The wires are then to be routed along the Jinks that connect adjacent grid points. The objectives in global routing are to minimize wire length and. often, the number of bends in wires, while spreading the wire as evenly as possible to simplify exacl embedding and later revision. Wires are to be routed around regions in which wire demand exceeds capacity if possible, so lhat they will not "Overflow." requiring drastic rearrangements of the other wires during exact embedding. Wire bends are costly in packages that confine the north-south and easl-wesl wires to different layers, since each bend requires • connection between two layers. Two classes of moves lhat maintain the minimum wire length are shown in Fig. 5. In the Lshaped move of Fig, 5a. only the essential bends are permitted, while the Zshaped move of Fig. 5b introduces one extra bend- We will explore the optimization possible with these two moves. For a simple objective function that will reward the most balanced arrangement of wire, we calculate the square of ihe number of wires on each link of Ihe network, sum the squares for alt Jinks, and term the result F. If there are rV links and Nw wires, a global routing program that deals with a high density of wires wi|] attempt to route precisely the average number of wires, N fNi_. along each link. In this limit F is bounded below by S^'IN\_. One can use the same objective function for a low-density (or high-resolution) limit appropriate for deL
w
382
tailed wiring. In that case, all ihe links have either one or no wires, and links with two or more wires are illegal- For this Limit the best possible value of F will be A V t f . For the L-shaped moves. F has a relatively simple form. Let - +| along the links lhat connection i has for one orientation. - 1 for the other orientation, and 0 otherwise. Lei a, be I if the connection can run through the vth link in either of its two positions, and 0 otherwise. Note that a,, is just Then if ft," =1 indicates which route ihe .:!• connection has taken, we obtain for the number of wires along the vth Link, L
u
n, - L ~
^
~
+
<9)
where t (0j is the contribution from straight wires, which cannoi move without increasing their Length, or blockages. Summing the • / gives v
P =
2jj>p-il4 S^iAj
+ constants [101
which has the form of the Hamillonian for a random magnetic alloy or spin glass, like thai discussed earlier. The "random field." hi, felt by each movable connection reflects the difference, on the average, between the congestion associated with the two possible paths: k = J, f.A2f*J.G)
'
+ Z a ,\
i
rt
CM)
The interaction between two wires is proportional to ihe number of links on which the two nets can overlap, its sign depending on their orientation conventions:
678 BothJy and h, vanish, on average, so ii is [he Hue [nations in [he terms that make up F which will control ihe nature of ihe low-energy stales. This is also true in spin glasses. We havenoi tried to exhibit a functional form for the objective function with Z-moves allowed, but simply calculate it by first constructing the actual amounts of wire found along each Jink. To assess the value of annealing in wiring this model, we studied an ensemble of randomly situated connections, under various statistical assumptions. Here we consider routing wires for the 98 chips on a moduJe considered earlier. First, wc show in Fig. 6 the arrangement
of wire lhal results from assigning each wire to an L-shaped path, choosing orientations at random. The thickness of rhe links is proportional TO the number of wires on each link. The congested area that gave rise TO the peaks in the hisiograms discussed above is seen in the wiring just below and to the right of the center of the module. The maximum numbers of wires along a single Link in Fig, 6 are 173 U direction! and 143 ty direction), so the design is also anisotropic. Various ways of rearranging the wiring paths were studied. Monte Carlo annealing with Z-moves gave the besl solution, shown in Fig, 1. In This exam-
200
ple, the largest numbers of wires on a single link are 105 [xi and 96 tyi. We compare the various methods of improving the wire arrangement by plotting CFig. 8) The highest wire density found in each column of i-Jinks for each of Ihe methods. The unevenness of the density profiles was already seen when we considered net-crossing histograms as input information To direct placemen!. The lines shown represem random assignment of wires wiih L-moves; aligning wires in the direction of least average congestion—That is. along fcj—followed by cooling for one pass al zero T\ simulated annealing with L-moves only; and annealing wilh Z-moves. Finally, ihe light dashed line shows the optimum result, in which the wires are distributed with all links carrying as close to Ihe average weight as possible. The optimum cannot be attained in this example without stretching wires beyond their minimum lenglh. because the connections are too unevenly arranged. Any method of optimization gives a significant improvement over the estimate obtained by assigning wire routings at random. All reduce ihe peak wire density on a link by more than 45 percent. Simulated annealing with Z-moves improved the random routing by 57 percent, averaging results for both x and y links.
Traveling Salesmen Quantitative analysis of the simulated annealing algorithm or comparison between it and olher heuristics requires problems simpler than physical design of computers. There is an extensive literature on algorithms for the traveling salesman problem (J. 4), so il provides a natural context for this discussion. If the cost of travel between two cities is proportional to the distance belween them, then each instance of a traveling salesman problem is simply a list of the positions of N cities. For example, an arrangement of A points positioned at random in a square generates one instance. The distance can be calculated in either the Euclidean metric or a "Manhattan" metric, in which the distance between two points is the sum of their separations along the two coordinate axes. The laller is appropriate for physical design applications, and easier to compute, so we will adopt il. 1
We let the side of the square have Length flP , so that the average distance belween each city and its nearest neighbor is independent of N. K can be shown that this choice of length units leaves the oplimal tour Length per step independent of t\, when one averages over many 1
383
instances, keeping N fixed (25). Call rhis average optimal step length a. To bound a from above, a numerical experiment was performed with the following "greedy" heuristic algorithm. From each city, go to the nearest city not already on The Tour, From the M h city, return directly to the first. In the worst case, the ratio of the length of such a greedy tour to the optimal tour is proportional To Jn{/V) (26), but on average, we find that its step lengih is about 1.12. The variance of the greedy step length decreases as N~ , so the situation envisioned in the worst case analysis is unobservably rare for large /v\ 112
To construct a simulated annealing algorithm, we need a means of representing the tour and a means of generating random rearrangements of the tour. Each tour can be described by a permuted list of the numbers 1 to N. which represents the cities. A powerful and general set of moves was introduced by Lin and Kernighan (27. 28). Each move consists of reversing the direction in which a section of the tour is traversed. More complicated moves have been used to enhance the searching effectiveness of iterative improvement. We find with the adaptive divide-and-Conquer effect of annealing at intermediate temperatures that the subsequence reversal moves are sufficient (29). An annealing schedule was determined empirically. The temperature at which segments flow about freely will be of order N . since that is the average bond length when the tour is highly random. Temperatures less than I should be cold. We were able to anneal into Locally optimal solutions with a ^ 0.95 for N up to 6000 sites. The largest traveling salesman problem in the plane for which a proved exact soLulion has been obtained and published (to our knowledge! has 318 points (30). Real cities are not uniformly distributed, but are clumped, with dense and sparse regions. To introduce this feature inio an ensemble of Traveling salesman problems, albeit in an exaggerated form, we confine The randomly distributed cities to nine distinct regions with empty gaps between them. The temperature gives the simulated annealing method a means of separating out the problem of the coarse structure of the tour from the local details. At temperatures, such as T = 1.2 (Fig. 9a). where the small-scale structure of the paths is completely disordered, the longer steps across the gaps are already becoming infrequent and steps joining regions more than one gap are eliminated. The configurations studied below T = 0.8 (for instance. Fig. 9bl had the minimal number of long steps. {IZ
expect similar free zing effects to limit ihe effectiveness of ihe common device of employing iterative improvement repeatedly from different random starling configurations. Simulated annealing extends two of The mosl widely used heuristic techniques. The temperature distinguishes classes of rearrangements, so that rearrangements causing large changes in ihe Summary and Conclusions objective function occur at high temperaImplementing the appropriate Metrop- tures, while the small changes are deolis algorithm to simulate annealing of a ferred until low temperatures. This is an combinatorial optimization problem is adaptive form of the divide-and-conquer straightforward, and easily extended to approach. Like most iterative improvenew problems. Four ingredients are ment schemes, the Metropolis algorithm needed: a concise description of a con- proceeds in small steps from one configfiguration of the system; a random gener- uration to the next, but the temperature ator of "moves" or rearrangements of keeps [he algorithm from getting stuck Ihe elements in a configuration: a quanti- by permitting uphill moves. Our numeritative objective function containing the cal studies suggest lhal results of good trade-offs [hat have to be made: and an quality are obtained with annealing annealing schedule of the temperatures schedules in which the amount of comand length of times for which the system putational effort scales as /V or as a small is to be evolved. The annealing schedule power of N. The slow increase of effort may be developed by (rial and error for a with increasing N and the generality of given problem, or may consist of just the method give promise that simulated warming the system uniil it is obviously annealing will be a very widely applicamelted. Then cooling in slow stages until ble heuristic optimization technique. diffusion of [he components ceases. InDunham (5) has described iterative venting the most effective sets of moves improvement as the natural framework and deciding which factors lo incorpo- for heuristic design, calling it "design by rate into the objective function require natural selection." (See Lin (6) for a insight inlo The problem being solved and fuller discussion.) In simulated annealmay not be obvious. However, existing ing, we appear to have found a richer methods of iterative improvement can framework for the construction of heuprovide natural elements on which to ristic algorithms, since the extra control base a simulated annealing algorithm. provided by introducing a temperature but the detailed arrangement of the long steps continued to change down to 7 = 0.4 (Fig. 9c). Below 7 = 0.4. no further changes in the arrangement of the long steps were seen, but the small-scale structure within each region continued to evolve, with the result shown in Fig. 9d.
The connection wilh statistical mechanics offers some novel perspectives on familiar optimization problems. Mean field theory for the ordered stale ai low temperatures may be of use in estimating the average results (o be obtained by optimization. The comparison with models of disordered interacting systems gives insight info the ease or difficulty of finding heuristic solutions of the associated optimization problems, and provides a classification more discriminating than the blanket "worst-case" assignment of many optimization problems ro the NP-complete category. It appears lhat for the Large optimization problems thai arise in current engineering practice a "most probable" or average behavior analysis will be more useful in assessing the value of a heuristic than ihe traditional worst-case arguments. For such analysis to be useful and accurate, better knowledge of the appropriate ensembles is required. Freezing, at the temperatures where large clusters form, sets a limit on the energies reachable by a rapidly cooled spin glass. Further energy lowering is possible only by slow annealing. We
384
allows us [o separate out problems on different scale s. Simulation of the process of arriving al an opiimal design by annealing under control of a schedule is an example of an evolutionary process modeled accurately by purely stochastic means. In fact, it may be a better model of selection processes in nature than is iteralive improvement. Also, it provides an intriguing instance of ''artificial intelligence." in which the computer has arrived almost uninstructed at a solution That might have been thought To require the intervention of human intelligence. Rrfcrcncra i n d Nolta ] . E, L . Law lor. Combinatorial Onilmliniicn (Holl Rmehart & '-. in - in, New York. 1976). 1 A. V. Ahc-. J. E- Hapcroft. J- D- Unman, The Design and Analysis of Computer Algorithms (AddImn-We*ley, Reading, M • 1974). 3. M . R. Garey acid rj. S. Johnson. Computert and Intractability: A Garde It? the Theory cf NPCampltttneii (Freeman. San Francisco, r979). 4. R. K i r p , Math. Optr. Res. 1, J09119771 J B. Dunham. Svnlneie 15. 754 (19631. 6. S. Lin. Neiworkt 5. 33 11973). 1. For a concise and elegant presentation of ihe basic ideas ot flailstical mechanics, see E. Shrodmger. Statistical Thermodynamics (Cambndgr Univ. Presi, London, 1946). R, N . Metropolis, A. Rosenbllith, K . ftasenbluth. A. Teller. E. Teller. J. Chem. Phyt. Q , I0B7 9. G. Toulouse. Common. Phys. 1. 115 tl9T?}.
680 10. Fur review articles. s t e C . Caslellam, C. DlCiEtro. L . Peliii. Eds., Disordered StWm ond localization (Springer. New York. 19*1). 11. J Soukup, Pfot /ft"£49. 1281 £i98l) 12. M. A. Breuer. Ed.. Design Autumaiiin of QriJlal Svstems lPrentice-ha||. Engelwood Cliff*. N J . . IS72J. 13 D Shemnglon and S. Kirkpainck. Phvs. Rev. Leu. 35, 1792 (IWSJ; 5. Kirkpainck and D Shemnglon. Phvs. Rev. B 17.1364 C197&). 14, A P. Young and S. Knkpatnck. Ptni. Rev. B 25 . 440 rfSBZ). 15- B. Mandelbrot. Fractals: Form. Oflni-e. and Dimeniion fFreemirj, Sim FianfiscD. 1979), pp. 237-239. 16. S- Kirkpainck, Phvs. Rev. 8 16. 4630 IWTJ), 17. C. Davis. 0 Malty R. Simmons. H, Siollei. R. Warren. 7 Wohr, In Praceedings of the IEEE Intemotional Conference on Circuits and Co-"' outers. N. B. Guy Rabbal, E d . I I E E E , New Vert. 19BQJ, pp 669-673
IS H A. Hanan. P. K Wgltf. B. J AgOle.J. Des Amen. Fault-Tolerant Compnt. 2. 145 <197B|. 19. M. Breuer. ibid. I. Mi 11977). in. P. W. Case. H . Curreia. W, GianopulOE. W. R. Heller. H Olek. T C Raymond. R, L - Simek, C B- Sreighr;. VfiAf J . £>^i. 15. 631 (I9&I] 21. A. J. Blodgett and D R. Barbout, ibid. 26. 30 L1962K A. 1- Blodgeir. in Proceedings of the Electronics and Compu'eH Conference t I E E E . New York. 19S0J, pp. 7B1-785 22. K. A. Chen, M. Feuer, K. H. KhokhanJ, N. Nan, Sr Schmidi. in Proceedings of the 14th IEEE Design Automation Conference (New Orleans. L a . . 1977), pp. 258-302. 23. K W. LaJlier.J.B.HicksOn.Jr , R K. lackson. paper presented at the European Conference on Design Aulomalion, SepLember 1981. 24. D. Hijhlower. in Proceedings of the 6th IEEE Design Automation Workshop (Miami Beach. F a . , June 19691, pp. 1-24. 25. J. Bcardwood. J. H. Haliun. J. M. Hammers-
385
ley. Proc. Cambr-dge Philas. Soc. S i . 299 (1959). 26 D. J. Resenkraniz. R. E . Sicarns. P. M. Lewi* SIAMISot .ind. Appl. Malh.iJ Comout. 6. 563 (19771. 27 S Lin. Bell Syst. lech.J.M, 2245 (I96SI 2S. and; B. W. Kemighan. Qptr. Res. 31.495 (1973J. 79. V. Cemy has described an approach lo Ihe traveling salesman problem similar io OUTE in a manuscnpl received after this article was submitted pu bl icaiion. 30- H Crowder and M. W. Padberg. Manage. So 26. 495 (l9SO|. 31. The experience and tollaboralive efforts or many of our colleagues have been essential io Ihis -ft>rk. In panicular. we ihank J. Cooper, W. Donath. B. Dunham. T. Enger. W. Heller. I HickEOn, G . Hsi. D. Jcpsen, H Koch. R Lmsker, C Mchanlan. S. Rolhman. and L . Schulu. r
Reprinted with permission from Nature, Vol. 326, No 6114. pp.689-491, 1987 0 1967 Macmillan Magazines Lid.
An analogue approach to the travelling salesman problem using an elastic net method
N . Because perfect solutions are unattainable for problems of any appreciable size, there has been interest i n heuristic methods lhat And near-optimal solutions in a reasonable t i m e . Hopfield a n d T a n k have recently discussed how problems such as the T S P can be represented in the d y n a m i c a l b e h a v i o u r of a network o f simple model neurons. T h e activity o f ihe network evolves into a stable configuration w h i c h represents a good solution to the problem. T h i s procedure was s h o w n to be equivalent to gradient descent of an energy function w h i c h contains the tour length as a major term. F o r the 30-city problem discussed, the best tour found was only 19% longer than the shortest k n o w n tour, A related analogue algorithm ( G . T e s a u r o , personal c o m m u n i c a t i o n ) obtains shorter tours for 30 cities, but does not. in general, produce v a l i d solutions for larger problems. 1
1 3
Richard Durbin' & David Willshawt • M R C laboratory of Molecular Biology. Hills Road, Cambridge C B 2 2 Q H and King's College Research Centre. Cambridge C B 2 1 5 T . U K t Department of Zoology. University of Edinburgh. West Mains Road, Edinburgh EH9 3JT, U K
1
T h e (ravelling salesman problem is a classical problem in the field of combinatorial optimization, concerned w/itb efficient methods For ma xi mixing or minim I i log a function of many independent variables. Given tbe positions of JV cities, which in ihe simplest rase lie in the plane, what is the shortest closed tour in which each city can he visited once? W e describe how a parallel analogue algorithm, derived from a formal model " Far the establishment of topographically ordered projections in Ibe brain'"''", can be applied to the travelling salesman p r o b l e m ' U s i n g an Iterative procedure, a circular closed p i t h is gradually elongated nonu n i f o r m ^ until it eventually passes sufficiently near to a l l the cities (o define a tour. T h i s produces shorter tour lengths than another recent parallel analogue algorithm > scales well with the size of the problem, and is naturally extendable lo a large class of optimization problems involving topographic mappings between geometrical structures' . Although easy to state, the travelling salesman problem ( T S P ) is hard to solve; the number of possible tours increases exponentially with the number of cities, h is representative o f the an important class of optimization p r o b l e m s ' , called N P c o m p t e t e . w h i c h are all interconvertible, bul the computing effort needed to solve them increases faster than any power of 1
1
1
IJ
4
1
11
12
O u r a p p r o a c h is more geometrical. A lour c a n be viewed as a mapping from a circle to the plane so that each city in the plane is m a p p e d to by some point on the circle. W e consider mappings from a circular path o f points to the plane in which neighbouring points on the circle are m a p p e d as close as possible on the plane. T h i s is a special case of the general problem o f best preserving neighbourhood relationships when mapping between different geometrical spaces. O u r algorithm was developed from the lea-trade m o d e l ' for the establishment o f topographically ordered, neighbour-preserving projections between neural structures with matching g e o m e t r i e s .
1
4-10
T h e algorithm is a procedure for the successive recalculation of Ihe positions of a number of points in [he p l a n e in w h i c h the cities lie. T h e points describe a closed path w h i c h is initially a small circle centred on Ihe centre o f the distribution o f cities and is gradually elongated non-uniformly to pass eventually near all the cities a n d thus define a tour a r o u n d them ( F i g . 1). E a c h point on the path moves under Ihe influence of two types o f force. T h e first moves it towards those cilics to w h i c h it is nearest; the second pulls it towards its neighbours on the path, acting to m i n i m i z e the total path length. By Ibis means, each
Fig, 1 Example of the progress oF the M elastic net method For 100 cities randomly distributed in the unit square a. The initial path. To break symmetry, •his was sel up lo be a ring of radius 0.1 about the centroid of Ihe cities. A starling configuration with each poini on the path placed at a small, randomly chosen distance away from the centroid (see rcf. 13) was found lo be adequate as Jong as a large number of iterations at a high value or K were K= 0 . 1 3 K = 0 OS allowed, b, c and d. Paths generated as the value oF K was gradually lowered, e. The lour of length 7.7B deduced from the final configuration shown in d / . The shortest lour SO Far Found by us by any method (simulated annealing ' * with eight million trials, see Table 1 For details). This lour had a length oF 7.70. The parameter values used FOE the computations described in this paper which used the clastic net meihod wcie: M=2.5 Af, a =0.2. B = 2.0; the initial value of K was 0.2, and was reduced by 1% every n iterations to a final value in the range 0.01-0.02. In the calculation shown heie. K was reduced 10 0-01 in 7,000 iterations by taking rr= 75. The caie at which K can he reduced is limited by situations such as lhal shown by the arrow in c. where ihe sysiem must be sufficiently relaxed to prevent the occurrence of crossovers, which would increase the lour length. The Function
a**
1
1
!
K
I
+
r
I
lhal by, = -K dE/ay,, which means thai any change in y, according to equation ( I ) results in a reduction in the value of E and. because E Is hounded below, local minima of £ will eventually be reached.. In the limil where K lends lo zero, For £ to remain bounded the limiting path must pass through all the ciltes (for every i there must be a j such that | x , - y | tends to zero). As M gets large, ihe second term in ihe expression for E is then minimized by spacing the M path points equally around the path a| a distance D/lvf apart, where D is ihe palh lengTh. This term ihen likes ihe value D /M. which is minimized by minimizing ihe tour length D. p
i
386
2
city b e c o m e s associated with a particular section of the path. T h e tightness o r the association is determined by how the force contributed from a city depends on distance, a n d the nature o f this dependence changes as the algorithm progresses. Initially all cities have a roughly equal influence on each point on the path. Subsequently* Longer distances become less favoured, a n d each city g r a d u a l l y becomes more specific for the points on the path closest to it. T h i s gradual increase o f specificity (informally equivalent to the l o w e r i n g o f the 'temperature' in optimal simulated a n n e a l i n g ' * ) is controlled by a reduction of the length parameter K. W e call i h i s algorithm the elastic net method because o f the w a y in w h i c h the initial path is gradually deformed to p r o d u c e the final t o u r
tour was 19% longer. A comparison of o u r algorithm with m e r e conventional algorithms is s h o w n in Fig. I a n d T a b l e I . T h e length of the best tour a r o u n d 100 cities distributed at random in the unit square was 7,78, w h i c h is within 1% o f the best lour length (7.70) we have found by any method ( F i g , I ) . W h e n applied to 5 sets of SO randomly distributed cities, the best tour length was within 3.0 ± 1.4% ( m e a n ± s . d . ) o f the best tour length we have found by any method, a n d within 1.5 ± 0.7% o f the average length w e found by simulated annealing. *'' in a p p r o x i mately the same amount of computing time ( T a b l e 1). 1
,ia
Let the coordinates of the position of a typical city i be denoted by the vector x, a n d those of a typical point j on the path by Yt, T h e n the rule for the change Ay, i n the coordinates y o f point j at e a c h iteration is a
w, (K,-y > + /JJC(y *i-2y + y,_ ) /
l
J
J
jl)
1
where the constants a and 3 determine the relative strengths of ibe forces from the cities and the forces from its neighbours on the path. T h e coefficient Mfo specifies the influence o f city i on path point j , and c a n be thought of as the strength o f the connection between i a n d j . T h i s is a function o f the distance \n,— y | between • a n d j a n d o f rhe length parameter K. It is normalized s o that ihe total influence of each city is equal. r
», = * ( k - y , U ) / I * ( | . , - y , | . K )
in
l l
J
T a k i n g 4>[d K) as the gaussian function e x p ( - d / 2 K ) . an energy function can be defined which has the property that successive changes in the path calculated according to equation (1) lead to a stable state w h i c h represents a local m i n i m u m of energy (see F i g . 1 legend)- I n the limit where K lends to zero and the n u m b e r o f points M on the path tends to infinity, the global m i n i m a o f the energy function are optimal solutions of The T S P . T h i s suggests that even when these limits are not achieved* good tours will be obtained. t
T h e elastic net method was applied lo the distribution of 30 cities used by Hopfield a n d T a n k " . It generated the shortest k n o w n t o u r " in 1,000 iterations; Hopfield a n d T a n k ' s shortest
Table I
Tour lengths obtained from comparative studies on five sets of 50 randomly positioned cities
Cii? set
Elastic nel method"
I
5.98
2 i 1
603 5.74(5.;0]
s
6.49
5.90 [5.863
Simulated annealing!
Ail algorithms]
5M 6.01 US in 6.33
1
1 1
where 4>{d, K) is a positive, b o u n d e d decreasing function o f d that approaches zero for d > K , J
T h e elastic net method operates by integrating a set of simultaneous fir it-order difference equations, an essentially parallel operation, a n d it therefore naturally lends itself lo implementation in parallel hardware. E v e n when implemented on conventional computers, its complexity scales well with the number of cities N. h produces shorter tours than the parallel atgorirhm of Hopfield a n d T a n k a n d ii is not as sensitive to the choice o f parameter values. T h e m a i n computational cost is in calculating the distances at each iteration a n d is therefore proportional to the product o f the number o f iterations required with the number o f significant connections (connections with near-zero strength c a n be ignored). T h e number of connections is of the order N (given lhat the number of points on the path is of order TV), but this c o u l d be reduced by starting with a small number o f path points and gradually interpolating additional points in the path to keep a constant average number o f significant connections from each city. T h e number of connections required by the algorithm o f Hopfield and T a n k is of the order o f N * . although these are fixed in value. T h e number of iterations required seems to increase more slowly Than a Linear function o f N , a n d ihis number c o u l d probably be signiiicanTly decreased by optimizing The schedule for decreasing K. Because an energy function c a n be defined, a more sophisticated a p p r o a c h to m i n i m i z a t i o n " might also reduce the computational time required. 1 1
r
Ay, = t ,
6
O u r algorithm is fundamentally geometrical. It can be extended to the more general T S P with cities in any euclidean space, but not TO the case where an arbitrary matrix of distances is given. However, it is applicable to a wide range o f geometrical optimization problems. F o r instance, we can alter the dimension and topology of either space (by* for example* replacing the ring by a n y other graph), and also remove the condition that points in the 'ring' space arc to be evenly spaced (by varying the strengTfi of the force between neighbours). T h e algorithm therefore provides a general method for matching a set o f arbitrarily connected points to a second set o f points located in a geometrical space of any dimensionality. It m a y Thus be applicable lo a variety of matching a n d representational problems , s u c h as those that arise in early visual process tng" ". O u r interest in this problem originated in an interest in m e c h a n i s m s - ' for the establishment of ordered neural projections belween structures o f s i m i l a r geometry. T h e r e are other neural projections which involve topographical mappings belween spaces of different dimensionality"" *, for which several formal models have been proposed* "'*. T h e elasTic-net a p p r o a c h suggests a new w a y of Looking at these phenomena and we are currently investigating how simple parallel algorithms o f the type discussed here c o u l d give rise to the projection patterns observed. 1 , 1 6 , 1 9
5.84,1
5.99.3 5.57.6 5.70.4 6.17.1
Qj
1
1
5
"For each sci of cities, the algorithm was run for 1.250 iterations. *iih n = 5- For sets 3 and 4. a slower schedule for reducing the value of K resulted in shorter tours, shown in brackets. T At each trial. Ihree culs in the lour were made at random and rearrangement which produced shorter lours were accepted, in the absence of a shorter tout, the rearrangement which lengthened the [our least was acccpied with probability e x p { - A D / T ) * , where AD is the increase in tour length and the parameter 7*corresponds lo the parameter K of [he elastic nei method- T was reduced from 0.5 lo 0.003 (5 x 10 steps, n = 1,000). which look i0% more computer time than Ihe elastic net method. Each figure is the average of 5 simulations. 1 5 , 1
1
t T o u r optimization was also attempted by systematic trial of all possible three-cut rearrangements and accepting only those rearrangements which shortened the tour, ihe procedure being repeated from many different randomly generated inilial configurations. This Strategy has been shown to be efficient in ihe use of time for this size of problem . Each pair of figures In rhe finat column represents respectively the shortest lour obtained from any of ihese methods and the number of times ihis tour was obtained.
387
We thank the participants o f the C o n n e c t i o n i s t Summer School, held a l C a r n e g i e - M e l l o n University. U S A in M a y 1 ° 8 6 . for their interest a n d their help, particularly D r G . T e s a u r o . We also thank D r A . Y u i l l e for the suggestion o f a form of the function (d. K) which permits the construction of an energy function. D r D . W . T a n k kindly supplied the coordinates of the 30-city problem discussed.
Received 71 hJ0'«m6cj
u t t p L c d IB Febmary IMT.
I. La*.ef. £. L . L t i i i t n , } . X.. Hlmnny KJiin. A. H G. A Slimoy*, t>. B. ( c i i f Tie Tt^HnHl S O T U T U I Pn&\t*n CWtlcy. Neu York*, I9IS> L V o i d e r M t l & b u q . C I l A WLHih-w, O.J. fWm nflM Atsi So. USA ?OiT£-il7FJ l\¥>T\ r
3
1 D. f . 4 Voader M i M h i i t . CB. A 4 Trfi A b e RffT, 20J-H J11970, 4. t y f c * . S *. A h l n U L W- H 4 a J O U * l 24. i m . l J W ( l H U i C j T . l t M T V ' i n u f t c * vf Htrwt ( I W I I I H I I A40ICBK. Lcrikd4-• HipUm ] | | - 2 J l (1939J • KAUHIITTI. E l * Ktmiihtl. H fthnu, » , J 9 M 9 J (L»Hj. • D m i i r . U . C . A Hun*L, D . K. I. Stuwcph^ioL J * . * M - T [ ) <|fl71|. 10. WmtL ft H. 4 AlbuiD. J EL A. R*a Snwa 1 WMUdtlQ] 11. flllfi.M I t A lohriioik, D S ' d / n ^ L ' f • " . ] ] - • . . . I . , ' „ . I , i hr-Tir.jn.-i.n Frjn.nca • I. Pipidimilnaii. C. H. TVdr. Con-fur. £fi : " J44 i [-" .i I I Hopticld.J J. A Tank. D W . M Cf*#r* J. 1 4 1 - l f ] n t l ) ) . M. H H M f H t O. I . J Durbin.,ft.SMAf 7. JJJJ. Juc M f M 7, I I I . 1 1 3 ( L f l l l j . I i XJripurirfc. S.. G r i m . C D * Vtethl. M . P. S f l r w 11*, 671 - U H 1 SB >] l b fcjrlpiinrt, 5. A >"">" l y . i 1*. • ••• i • J - .. Tn
L
388
n rja, 5. A n*ffc4hu. i * ( h a h i i i . « i - m < i » 7 i j . 11. I n t A b f f f t r , D G I v n A r t a * t * H t awT .VMJWM- taj/raunLf 1 4 d d | 4 « » f » « T , ftttdm|_ U u m t . i r l l i . L W > I f C V d i n U r i h in T V r W A f l f 5*b^Kku> froMr* [ c d i L n * k f , £ L . L c u n J K_ R i i f M T KAIB. A. ri 0 . * I M V n , D BMJ1-444 f W J r , , few York, 19*31 H JriktZ. B • — ']•?••••" • t Crtlefti- Y'U'Vn , U 4 r T n » l i Or Ctaeajo. n i l ] JL D A '• - . j . : r- . - •• . • ] « , : (1979). 17. H u b * . D H I W i n d , I « . H H . * B I H . I-W,I»T7J. JJ L i » , M . L A CoiuiuiUDi'hLO'L M . t. Sntstl T. 7< I-? *9 H HBlf 24. F « « r r u I W. A W i l h l i a - . D. J Entire Jfc. ) ! 0 - J H M M 2 I H ; Vtnnlrr M i l i a r , . Cli. A W
L
Generalisation; Learning a Rule
T h i s section contains two long and detailed articles, (32) and (34), and two short and elegant texts, (33) and (35), which all revolve around the issue of generalisation, i.e. learning a rule from examples (approximating a function from a discrete set of points is one simple case). Article (32) is a bit verbose and somewhat untidy (the reader will have to discover by himself the systematic error in the numbering of figures!), but it is a mine of sound remarks, illuminating analogies and demystifying criticisms. Its casual presentation is representative of an uninhibited engineering style. Human cognition is often confronted with quasi-regular environments (think of regular and irregular verbs in many languages). Article (33) examines critically whether neural networks, with only few hypotheses about their initial structures and learning processes, can reproduce observed human properties, such as default-role behavior. This article brings lucid arguments, about the interplay between rule simplicity and frequency of examples, into the debate between classical cognitivism and connectionism. Article (34) makes a telling contrast with article (32). B y focusing on a more circumscribed set of problems, with simple network architectures —• and by making use of the tools of statistical mechanics and of the thermodynamic (large size) limit — the authors are able to harvest a large number of results on exactly soluble models. T h e bulk of this paper consists of step by step exposition of mathematical procedures and may look a bit frightening to the nonphysicist. Nevertheless, the reader is encouraged to appreciate the power of statistical mechanics. Some of the highlights are the demonstration of discontinuous transitions in learning processes, and insights on possible roles of noise in learning. Among a collection of insightful remarks, article (35) attempts a classification of learning machines, and a definition of intelligent behaviour. His proposal for resolving the ambiguity of the generalisation problem (much discussed in article (32)) is that intelligent generalisation is the simplest rule that fits the data (simplicity being defined through the algorithmic informa-
389
tion content, a notion that is, unfortunately, defined only in an asymptotic (large-size) limit). Note the similarity between the "entropy of an architecture", and the "functional entropy" defined in article (32). Common themes among the papers of this section are the concepts of weight space or parameter space (equivalent to the "interaction space" introduced in article (29)), and extended uses of the notion of entropy, defined in parameter space, that are appropriate measures to quantify the gain of information during a learning process. Many ideas evoked in Chapter 1 concerning Bayesian analysis, inductive logic and statistical inference, should now take on a more tangible meaning for the reader interested in neural network and machine learning.
390
Reprinted from Complex Systems, Vol. 1, pp. 877-922,1987 © 1987 Complex Sysiems Publications, Inc.
Large Automatic Learning, Rule Extraction, and G eneralizat ion John Denker Daniel Schwartz Ben Wittner S a r a So I la Richard Howard Lawrence Jackel AT&T Bell Laboratories, Hotmdel, NJ 07733, USA J o h n Hopfield AT&T Bell Laboratories, Murray Hill, NJ 07974, USA and
California Institute of Technology, Pasadena, CA 91125, USA
Abstract.
Since antiquity, m a n has dreamed of building a device
that would "learn from examples", "form generalizations", and "discover the rules" behind patterns in the data. Recent work has shown that a highly connected, layered network of simple analog processing elements can be astonishingly successful at this, in some cases. I n order to be precise about what has been observed, we give definitions of memorization, generalization, and rule extraction. T h e most important part of this paper proposes a way to measure the entropy or information content of a learning task and the efficiency with which a network extracts information from the data. We also argue that the way in which the networks can compactly represent a wide class of Boolean (and other) functions is analogous to the way in which polynomials or other families of functions can be "curve fit" to general data: specifically, they extend the domain, and average noisy data. A l a s , finding a suitable representation is generally an ill-posed and ill-conditioned problem. Even when the problem has been "regularized", what remains is a difficult combinatorial optimization problem. When a network is given more resources than the minimum needed to solve a given task, the symmetric, low-order, local solutions that humans seem to prefer are not the ones that the network chooses from the vast number of solutions available; indeed, the generalized delta method and similar learning procedures do not usually hold the "human" solutions stable against perturbations.
391
Fortunately, there are
8T8
ways of "programming*' Into the network a preference for appropriately chosen symmetries.
1.
O v e r v i e w of the contents
Section 2 gives several examples that illustrate the importance of automatic learning from examples. Section 3 poses a test-case problem ("clumps") which will be used throughout the paper to illustrate the issues of interest. Section 4 describes the class of networks we are considering and introduces the notation. Section 5 presents a proof by construction that a two-layer network can represent any Boolean function, and section 6 shows that there is an elegant representation for the clumps task, using very few weights and processing units. Sections 7 and 8 argue that the objective function E(W) has a complicated structure: good solutions are generally not points in W space, but rather parameterized families of points. Furthermore, in all but the simplest situations, the E surface is riddled with local minima, and any automatic learning procedure must take firm measures to deal with this. Section 9 shows lhat our clumps task is a very simple problem, according to the various schemes that have been proposed to quantify the complexity of network tasks and solutions. Section 10 shows that a general network does not prefer the simple solutions that humans seem to prefer. Section 11 discusses the crucial effect of changes of representation on the feasibility of automatic learning. We prove that "automatic learning will always succeed, given the right preprocessor," but we also show that this statement is grossly misleading since there is no automatic procedure for constructing the required preprocessor. Sections 12 and 13 propose definitions of rule extraction and generalization and emphasize the distinction between the two. Section 14 calculates the entropy budget for rule extraction and estimates the information available from the training data and from the "programming" or "architecture" of the network. This leads to an approximate expression for the efficiency with which the learning procedure extracts information from the training data. Section 16 presents a simple model which allows us to calculate the error rate during the learning process. Section 17 discusses the relationship between rule extraction in general and associative memory in particular. In section 18, we argue that when special information is available, such as information about the symmetry, geometry, or topology of the task at hand, the network must be provided this information. We also discuss various ways in which this information can be "programmed" into the network. Section 19 draws the analogy between the family of functions that can be implemented by networks with limited amounts of resources and other families of functions such as polynomials of limited degree. Appendix A contains details of the conditions under which our data was taken.
392
879 2.
W h y l e a r n from e x a m p l e s ?
Automatic learning from examples is a topic of enormous importance. There are many applications where there is no other way to approach the task. For example, consider the problem of recognizing hand written characters. The raw image can be fed to a preprocessor that will detect salient features such as straight line segments, arcs, terminations, etc., in various parts of the field. But what then? There is no mathematical expression that will tell you what features correspond to a "7" or a "Q". The task is defined purely by the statistics of what features conventionally go with what meaning—there is no other definition. There is no way to program it; the solution must be learned by examples [6,11). Another example is the task of producing the correct pronunciation of a segment of written English. There are patterns and rules of pronunciation, but they are so complex that a network that could "discover the rules" on its own would save an enormous amount of labor (37]. Another example concerns clinical medicine: the task of mapping a set of symptoms onto a diagnosis. Here the inputs have physical meaning—they are not purely conventional as in the previous examples—but we are still a long way from writing down an equation or a computer program that will perform the task a priori. We must learn from the statistics of past examples [41]. Other examples include classifying sonar returns [10], recognizing speech [5,16,30,23], and predicting the secondary structure of proteins from the primary sequence [42]. In the foregoing examples, there was really no alternative to learning from examples. However, in order to learn more about the power and limitations of various learning methods and evaluate new methods as they are proposed, people have studied a number of "test cases" where there was an alternative— that is, where the "correct" solution was well understood. These include classifying input patterns according to their parity [33], geometric shape [33,35], or spatial symmetry [36].
3.
E x a m p l e : tvo-or-more clumps
The test case that we will use throughout this paper is a simple geometric task which an adaptive network ought to be able to handle. The network's input patterns will be A'-bit binary strings. Sometimes we will treat the patterns as numbers, so we can speak of numerical order; sometimes wc will also treat them as one-dimensional images, in which false bits (/'s) represent while pixels and (rue bits ( T s ) represent black pixels. A contiguous clump of 7's represents a solid black bar. We then choose the following rule lo determine the desired output of the network, as shown in table 1: if the input pattern is such that all Lhe T s appear in one contiguous clump, then the output should be F , and if there are two or more rlnmps, I.I ion the
393
880
Output F T F T F
Input pattern ffftttffff fffttftfff fttttttttt tttffttfft ffffffffff
Interpretation 1 clump 2 clumps 1 clump 3 clumps no clumps
Table 1: Examples of the two-or-mors clumps predicate. 1
output should be J". We call this the two-or-more clumps predicate. We will consider numerous variations of this problem, such as t h r e e - v e r s u s - t u o clumps and so forth. The one-versus-tvo clumps version is also known as the c o n t i g u i t y predicate [25]. Questions of connectedness have played an important role in the history of networks and automatic learning: Minsky and Papert devoted a sizable portion of their book [27] to this sort of question. There are a host of important questions that immediately arise, some of which are listed below. In some cases, we give summary answers; the details of the answers will be given in following sections. Can any network of the type we are considering actually represent such a function? (Yes.) This is not a trivial result, since Minsky and Papert [27] showed that a Perceptron (with one layer of adjustable weights) absolutely could not perform a wide class of functions, and our function is in this class. Can it perform the function efficiently? (Yes.) This is in contrast, say, to a solution of the parity function using a standard programmable logic array ( P L A ) [26], which is possible but requires enormous numbers of hardware components (0(2 ) gates). N
Can the network learn to perform this function, by learning from examples? (Yes.) How quickly can it learn it? (It depends; see below.) Ilow many layers are required, and how many hidden units in each layer? How do the answers to the previous questions depend on the architecture (i.e. size and shape) of the network? Ilow sensitive are the results to the numerical methods and other details of the implementation, such as the analog representation of T and F , "momentum terms", "weight decay terms", step size, etc.? Does the solution (i.e. the configuration of weights) that the network finds make sense? Is it similar to the solutions that humans would choose, given the task of designing a set of weights to perform the assigned task? Is its solution logical and systematic, or is it a kludge? 'A predicate is a Tloo I can-valued function; that is, its range is the set {T,F). domain can be anything you like.
394
The
881
Does the network discover the rule? T h a i is, when it is trained on a subset of the possible input patterns, does il find a configuration that yields the "correct" output for the remaining input patterns? 4.
Basics
The purpose of this section is to establish the notation. We assume that the reader has a basic familiarity with layered analog networks. This is a brutally brief review of a standard, vanilla model; there are many possible refinements and extensions that will not be mentioned here. Information on the history of these ideas can be found in reference [29]. A good survey of many successful applications of adaptive networks can be found in reference [39], A sophisticated and detailed analysis, dealing mainly with one-layered networks, appears in reference [27], A collection of recent work on parallel networks and learning from examples appears in reference [32], Papers on related topics, including the feasibility of building such networks, can be found in reference [8], Figure 4 shows a typical layered analog network. It has two layers of weights (/- = 2), and, cquivalently, two stages of processing. There are /V' ' processing units in the output layer, and W' ' units in the hidden layer; we use a superscript in square brackets to indicate layer number. There are JV' ' input wires, which are not considered processing units, although we sometimes refer to them as the 0 layer. The input bits have value v/ ' The weights in layer ( have values Wf}, for / = 1 to L . The empty triangles are amplifiers with a sigmoid transfer function g{) such as g(x) = tanh{i). The amplifiers do not have a separate "bias" or "threshold" input; the bias function is implemented by the weights connected to the "stuck bits". The stuck bits are provided by the triangles with no inputs; a "+" or "—" inside indicates that the output is stuck at T or F respectively. In each layer /, we number the unstuck bits from 0 to — 1, and designate the stuck bit as number N™ for a total of /V" + 1 connections to each unit in layer / + 1. 2
1
0
th
0
The input to each amplifier in layer t is given by:
and the corresponding output is:
V}1 = g(uf)
(4.2)
Equations 4.1 and 4.2 define the input-output relation of the network, depending on the W values of course. Denoting the input or the network as Jj = V-® and the actual output as A = Vfc , we write this overall inputoutput relation as: k
4
A = A{W;I).
3
(-)
395
882
Output
Figure 1: Typical network. Suitable W values could be chosen by hand, but we are mainly interested in choosing them automatically—learning from examples. The training examples are ordered pairs: (input, desired output), denoted {I°,D ), where a runs over all examples. The traditional measure of how well the network has memorized the training examples is: a
E(W)
= Y, \V
- A(W\ / * ) ! ' ,
(4-4)
s that is, what you wanted minus what you got, norm squared. This makes the task of learning equivalent lo the task of searching W space for a minimum of E{W), hence the name "least mean squares" ( L M S ) learning method. In the case when E{W) is dilTerentiable, there are several powerful methods for descending the E surface. Notable among these is the "generalized delta method," which can be efficiently calculated by the "back propagation" algorithm. 5.
Representability
Here is a proof that a two-layer network can implement any Boolean function. Note that we arc discussing representability, not learning, so it suffices to construct by hand a representation. The discussion here essentially parallels the proof in reference [27J that any predicate can be expressed in terms of masks. Also, for those readers wilh a background in digital circuits, there is a one-line proof: the two-layer network contains the P L A as a special case, a P L A contains a R O M as a special case, and an /V-input ROM can implement an arbitrary Boolean function of A' bits.
396
883
First Layer
Hidden
°
u
,
p
u
t
Inputs Figure 2: Network to represent an arbitrary predicate.
In fancy network language, the proof goes like this: construct a twolayer network with JV = inputs and 2 hidden units. There are exactly 2 possible input patterns, and we dedicate one hidden unit to each input pattern. You can call the hidden units "match filters" or "grandmother cells" if you like. The weight matrix W' ' for the first layer is chosen as follows: the weight connecting input bit t to hidden unit j will be called Wj\\ for t = 0 to N - 1 and j = 0 to 2 * - 1. wf} is set to +b if the i bit is set in the binary representation of the number j , and is set to —6 otherwise. This ensures that when input pattern number j is present at the input, the j ™ hidden unit will receive a contribution of strength +Nb from the matrix, and all other units will receive lesser contributions (by steps of 26). We give each hidden unit a threshold of [N — 1)6. We make b very large, which IS equivalent to making the gain of the transfer function very large (so the transfer function becomes a step function). Then the j unit will be turned on, practically to saturation, and all the other hidden units will be turned off, practically saturated in the negative direction. N
N
1
t h
l h
T h e rule for choosing the second layer of weights is similarly straightforward: if the fc output bit is supposed to be a T when the j input pattern is presented, then the weight WM +P> ' ~P otherwise. The bias of output unit k is set to §*> algebraic sum of all the weights feeding that unit. th
l h
t
,S
s
h
e
e
t
t
o
397
a n c
s e t
t
o
881
The circuit diagram in figure 5 is practically a picture of the t r u t h table of the chosen input-output relation. The weights in the first layer are an enumeration of the possible input patterns, and the weights in the second layer arc the corresponding o u t p u t patterns. T h e device can also be thought of as a fully decoded Read Only Memory ( R O M ) . T h e first layer of weights forms the decoder, and the weights i n the second layer are the stored memories. This demonstrates that the two-layer network can represent any Boolean function. This is i n stark contrast with the single layer network, which can only represent linearly separable functions [27], which are not very interesting. T h e term refers to the fact that a one-layer network can only separate the input space into simple categories: the separation boundary can only be a function linear in the input variables, i.e., a hyperplane. This solution is quite general, but i t is also quite inefficient. T h e number of weights in the network grows exponentially as the number of input bits is increased. The astute reader may have noticed that in the case of a single o u t p u t b i t , the number of hidden units can be reduced by a factor of at least two, by using a wired-or (or wired-and). This trick doesn't work (or A 3> 1. 1
6.
Efficient representations
For many tasks of practical interest, there exist efficient representations. Figure 6 shows an efficient representation for our clump-counting task. T h e solution capitalizes on the idea that clumps have edges. T h e first layer of weights is used as an array of (falling, right-hand) edge detectors, and the second layer counts the number of edges that were detected. Figure 6 is laid out in a non-standard way in order to clarify a special case: what happens if the edge of a clump coincides w i t h the edge of the input field? To eliminate this, we embed the N real input bits using JV + 2 wires, where the rightmost and leftmost wires are stuck in the F state. Now every edge of every clump can be seen a transition from T to F or vice versa. The N real inputs are labeled 0 to N — 1; the special edge wires are labeled — 1 and N. The ordinary bias line is stuck at T and is labeled JV + 1. O f course, when actually building such a network, the function of the edge bits would be combined with the ordinary bias line. The weights are assigned as follows: WjJ is set to + 6 if j = i, —b if j — i -f- 1 , and zero otherwise. The biases are all set to —6. This ensures that hidden unit j will be " h i g h " if and only if the corresponding input bit is in We arc quite aware that in many practical cases, the inputs are not Boolean, but can take on a range of values. That situation is quite a bit more complex. Consider a network with a single input wire, taking values x in the half-open interval (0, lj. The simple function which yields tbe parity of the largest integer less than 1/r will obviously require an arbitrarily large number of hidden units. Furthermore, Lippmann (private communication; see also [23]) lias shown that in the case of continuous-valued (non-Boolean) output levels, there are functions that cannot be represented wilh two layers, no matter how many hidden units are used. J
398
885
First Layer
Hidden
0
u
,
p
u
t
Inputs
Figure 3: Efficient network for clump counting. the T state and its neighbor to the right is in the F state. Specifically, the input to hidden unit j will have one of three possible values: +6
if ( / , . „ / ; ) = ( T , F ) if ( / ; _ , , / , ) = T,r) or ( f - , F )
-36
if (I -i,I ) j
(6.1)
= (F,T)
j
The weights in the second layer are +p, and the bias unit in the second layer is q. Hence, for a pattern containing C clumps, the input signal to the output unit will be U - Cpg(+b)
+ Cpg(-b)
4 {N+l~2C)pg{-3b)
+ g.
(6.2)
We want the network to return T when there are C clumps and F when there are C — 1 clumps. This leads to the following equation, which determines the optimal value of p:
l
9~ (T)-g-\F)
= ~
p (+6) + 0
6
P ? ( + ) - P9{-b)
(-6)-2 (-36)
ra
ra
(6.3)
for large 6
This construction shows that it is possible to represent the clump-counting problem using a two-layer network with only M = N + 1 hidden units. This proves that at least one efficient representation exists, for all N; we do not claim that this is the only such representation or even the best representation. We call this the geometric solution. We also refer to it as the human
399
880 solution, since when asked to design a network to solve this task, people tend to choose this solution. We emphasize that in the case where all the training patterns contain either C or C — 1 clumps, this is an exact, E = 0 solution. Since g() is bounded and monotonic, 6 —* oo implies g( — b) —t g(—3b). One consequence is that in this limit, the output of the j = 0 hidden unit is nearly constant. That hidden unit can be eliminated entirely and its output weight lumped in with the ordinary bias unit, simplifying the construction of the network. The problem of searching for a minimum of £{W) must be stated properly in this case, since for finite 6 there is no solution with E = 0, or even with E equal to a minimum. Yet in all situations that we care about, E = E„^„ is not required; E < E^ + e is the proper criterion for the solution, and e can be made as small as you like by increasing 6. We will return to this point in section 8 (see also [40j). a
The clumps predicate can also be represented by a one-layer network of sigma-pi units, which has been explored by reference [25]. 7.
T h e s t r u c t u r e of weight space
Solutions should not be thought of as a single configuration of weights, but as a class of equivalent configurations, differing only by certain symmetries. One important symmetry comes from our freedom to choose b\ we call this the it-symmetry. Also, in the large 6 limit it is possible to choose a different bj for each hidden unit j ; we call this the & j - 5 y m m e l r y . This means that the solution set includes a large region of a //-dimensional subspace of weight space. This phenomenon is not confined to our clumps example. Clearly, whenever a unit is being used as a linear threshold element, uniformly increasing its input weights leaves the result unchanged. Another important symmetry comes from the fact that the ordering of the hidden units is arbitrary. In each hidden layer, we can relabel the units in / / ! ways, where H is the number of units in that layer. Specifically, if we ,] set W'$ = WjL, and Wk^ = JV^j], for any hidden layer ( and for any permutation P, then the behavior of the network is absolutely unchanged. Yet another symmetry comes from the fact that in most networks that have been considered the polarity of the hidden units is arbitrary. In the representation where the transfer function is odd (which is typical when T = 1 and F = —1), this symmetry is very simple and easy to visualize: Tor each hidden unit, if we change the sign of every weight on its input and every weight on its output, then the behavior of the network is unchanged. In other representations where the transfer function is an odd function plus a constant (which is typical when T = 1 and F = 0), the symmetry is still there, but requires an adjustment to the bias terms in the next layer. The number of equivalent configurations in each hidden layer is 2 HI. Whenever 3
H
3
One can construct networks in which the transfer function has no particular symmetries, but we have seen no advantage in doing so.
400
887
one wishes to compare two solutions, one must remove these symmetries by converting the configuration to some standard form. These symmetries imply a certain periodicity in W space. This leads us to visualize the £ ' ( H ) surface as resembling a sombrero, or as a phono record that has been warped in certain symmetric ways: near the middle (W = 0) all configurations have moderately bad E values. Radiating out from the center are a great number of ridges and valleys. The valleys get deeper as they go out, but asymptotically level out. In the best valleys, E is exactly or asymptotically zero; other valleys have higher doors. This picture is, of course, an oversimplification. /
8.
Search methods
When the weights are very large (or g() is a sharp threshold function), the task of "learning" the best set of weights is an unvarnished combinatorial optimization problem, and the network formulation does not offer any discernible advantage over conventional combinatorial optimization procedures. However, by varying the gain (steepness) of the transfer function, Hopfield and Tank were able lo improve the results of a similar search. It is possible for the system to express temporary compromises, moving through weight space along complex paths such that many components (i.e. the weights) have intermediate values. In favorable situations, this allows the system to circumnavigate small barriers in the F,(W) surface, exploiting the high dimensionality of W space. In order to promote this sort of behavior, one can introduce an additional term in the learning process which we refer to as deterministic weight decay. (Other types of weight decay will be discussed shortly.) This additional term effectively defines a new surface £'(W) = E(W) + £ ( W ) , where Ei is large when | W | is large. Of course, searching the new surface is a different task than searching the original surface. If E is too large, then the solutions to the new task do not correspond to the solutions of the original task, and we have accomplished nothing. Indeed, if Ei is extremely large, it is possible to have only one solution, the parasitic solution W = 0. On the other hand, if E is very small, it doesn't do any harm, but then it doesn't do any good either. A better procedure is to use a term Ej(W,t), which is an explicitly decreasing function of lime. Then, the real solutions will gradually emerge out of the parasitic solutions. It is hoped that the best solution will be the first to emerge, whereupon the system will find it and follow it as it evolves to its final form. This is one example of a class of techniques we call simulated ironing, because they remove wrinkles from the E surface. It is a nice vision of what might happen, but there is no guarantee that it will happen in a particular case. Il is not a bad heuristic, but it is not guaranteed, either. Weight decay also improves the performance of certain learning methods by preventing the method from wasting time seeking the 6 —* oo solutions discussed in section 6. Most versions of the generalized delta rule learn very slowly when the weights are large. Weight decay is a rather artificial way of 5
2
3
401
888 introducing stable local minima at finite values of 6. We have shown that a better choice of objective [unction (i.e. other than LMS) greatly reduces the 6 - * oo problem [40]. Another heuristic that is very easy to implement involves using Euler's method with a step size that is a little too large. This situation was analyzed by Jeffrey and Rosner [19]. Basically, i t can be thought of as introducing into the equation of motion some higher-order terms, notably terms proportional to the curvature of the E-surface. This penalizes narrow basins of attraction, relative to wide basins of comparable depth. The resulting trajectory in W space is very complex, and E(W(t)) is not monotonic. We call this behavior "thrashing". I t is a quick and dirty expedient that helps when a clean, accurate descent of the E surface would have gotten stuck i n narrow local minima. There are several methods for escaping local minima which depend on introducing an element of randomness into the search procedure. T h e classical way is to simply restart the search many times from different random initial conditions and choose the best of these trial solutions [31]. Another well-known technique is to use simulated annealing [20], which systematically allows the system to take steps uphill on the E surface, w i t h a probability depending on the "temperature", which is an explicitly decreasing function of time. One can introduce a free energy surface, F, and show that temperature smoothes the F surface just as simulated ironing smoothes the E surface. We wish to emphasize a point that is widely misunderstood: the two procedures, although similar, are not completely equivalent. Specifically, a probabilistic system will eventually cross a small barrier i n the F space by activated hopping, while a deterministic system that always goes downhill can never cross a barrier, however small, in the E space. 4
Another way in which an element of randomness can be injected is by using "incomplete smoothing". Recall that the E function is defined w i t h respect to (a sum over) all the items in the memorization set M. I t is possible, however, to consider partial contributions to E, defined w i t h respect to subsets (or even single elements) of M. This leads one to compute motions in W space that minimize the various partial contributions, and the choosing of the subsets can introduce enough randomness to help the search escape from local minima. Global minima will remain stable if E = 0 there; otherwise, they might not. Another technique is known as ''stochastic weight decay" [43], Rather than decaying all the weights a little bit, one may decay a (random) subset of the weights somewhat more. This, too, allows the system to escape local minima and encourages solutions that are robust against this sort of perturbation. Wc use the term "jostling" to refer collectively to ironing, annealing, *Tbis depends somewhat on the order of limits: here we have taken the limit of long time before taking the limit ofsinall fluctuations and averaging over large ensembles. We are conducting "slow" annealing. To do otherwise would defeat much of the purpose of annealing.
402
889
Measure Layers Hidden Units Order of Predicate Number of Weights Dynamic Range Bits Required 5
Parity 2 N N N N N log N 2
2
Clumps 2 JV 2 JV 2 JV log N
Table 2: Comparison of complexity. stochastic weight decay, and similar procedures. There are a number of questions associated w i t h such procedures; for example, i t is difficult to decide how quickly the amplitude of the random forces should be decreased. This scheduling problem first arose in the context of simulated annealing and has been extensively studied. Perhaps i t should not be called the "annealing schedule" problem, but rather the "jostling schedule" problem, since i t crops up i n numerous methods that are distinctly not equivalent to annealing. AH of these methods for escaping local m i n i m a incur an enormous time penalty, and the penalty increases as the size [N) of the problem increases. This leads to the strong suspicion t h a t the learning process is formally intractable in general. The similar problem of finding the m i n i m u m number of minterms for a PLA is known to be NP-complete [9], but we know of no proof one way or the other for networks. 9.
Measures of complexity
In the general case, a network having H = JV hidden units would require roughly II x N = N weights, but our geometric solution (section 6) is very sparse. The number of (non-zero) weights is proportional to N to the first power only. In fact, each hidden unit has only two input weights, so this is what Minsky and Papert [27] called a second-order predicate. (We know that no first-order solution exists.) Also, note t h a t the weights need only take on small integer values, so the number of bits needed to specify the entire configuration is very small, proportional to A'logJV. Finally, note that the input weights have the property of short range, or locality: the two nonzero weights are not distributed at random, but are in fact adjacent. We w i l l argue t h a t not all of these concepts are accurate measures of the complexity or difficulty of a task, but in this case, the preponderance of the evidence indicates that this is a very cany task. The order of the predicate, the number of layers, the number or weights, the number of bits of specification, and the range are all small. IT they got much smaller, the problem would be reduced to a triviality. Table 2 contrasts this task with the p a r i t y predicate [33], which is often cited as a demonstration of the power of this sort oT network. The network can indeed learn to solve the clumps task, learning from examples. Table 3 shows how the learning time depends on the w i d t h of the 7
403
890
N 6 6 6 6 6 6 10 10 10
H 2 3
4 5 6 7 6 10 11
Median # Passes 1484 422 390 393 414 439 1747 1180 1072
Table? 3: Learning time versus network size and shape. hidden layer for two different sizes of input. Figure 9 is similar, showing how the learning lime depends on the w i d t h of the hidden layer for fixed input size. The data reported in the table and figure represent the median number of passes, where each pass consists of one presentation of each of the patterns in the training set M . Note that the median is a better measure than the mean, because sometimes the network fails to memorize the M data perfectly, even after arbitrarily long training time. It would be impossible to compute a mean in such cases, yet the median is still well behaved. The discussion of intrinsic complexity and learnability will be continued below in section 18.
10.
Stability of the h u m a n , geometric solution
It is interesting t h a t table 3 indicates that the network was able to solve the task i n those cases where the number of hidden units was less than the number of input bits. This answers one of the questions posed in section 3: the network is quite happy to find solutions that don't correspond to human designs. Humans seem to have a very hard time designing a solution in which // < N. Actually, our observations go further: even i n the case where H > Nj the network does not find the geometric solution. To confirm and extend this result, we conducted a perturbation analysis as follows: we constructed a network w i t h the geometric solution as the initial condition and then proceeded to further train it with the generalized delta method. Since the network was already at a solution, E == 0, no further adaptation took place. We then perturbed the system, moving the weights to a point W$ in weight space, and re-trained i t . We found that the system was quite able to re-solve the task, returning to E = 0, but did not do so by undoing llie perturbation. In fact, i t moved in some other direction and settled on a new point Wp. The learning procedure can he considered a dynamical system, describing the motion of the point W in weight space. Each local m i n i m u m of E is an
404
4.0
0.0
0.0 10.0 Hidden Unila
12.0
14.0
F i g u r e 4: L e a r n i n g t i m e versus w i d t h of the hidden layer.
405
18.0
892
"lO~*
IO
- 3
IO"
3
10"'
Start Distance
Figure 5: Perturbation of the geometric solution.
406
10°
893
o
10"*
10~
3
10"
z
10"
1
Start Distance
Figure 6: Perturbation wilh more symmetries removed.
407
10°
891 attractor, and each wilt have its own basin of attraction. The "6 symmet r y " means that the attractor associated with the geometric solution is not pointlike but is at least one-dimensional. To make the data in figure 10 more meaningful, we "projected o u t " this dimension by using the following slightly peculiar distance measure: first, we ignore the output weights and all the bias weights, so that only / f x N weights remain. Then, we normalize by dividing by the biggest weight. Finally, we calculate the distances, using what we call the RMS metric: square each component of weight-difference, divide by the number of weights, and take the square root. We denote Ds = \Ws — Wo\ and Dp = \ WF — Wol, where W represents the geometric solution. In this projected space, the geometric solution is not a sole point, but a set of isolated points, because of the discrete symmetries discussed above. A
If (in the projected space) the geometric solution had consisted of isolated pointlike attractors, figure 10 would have shown a characteristic flat region: Dp — 0 for all Ds < D', where D" represents the radius of the basin of attraction. Since the figure shows no sign of a flat, zero region, we conclude that either the basin of attraction is exceedingly small, or that the attractor is not pointlike in the projected space, i.e. the attractor i n the full W space has dimensionality greater than one. To say i t another way, the b symmetry is not the only continuous transformation that leaves E invariant. 8
Since we know that the bj symmetry exists, we repeated the above experiment using a metric that projected out this larger symmetry. Specifically, for eac/r hidden unit we normalized its weights to make the largest one equal to unity. This experiment checked the stability of a wider class of solutions, which have the same topology but not the same translational symmetry of the human solution. The results are shown in figure 10. Once again, there is no indication of any basin of attraction, and so we conclude that the E = 0 solution set is multidimensional in the projected space, and more than H dimensional in the original W space. The slope in figures 10 and 10 are remarkably close to unity. We conjecture that this is essentially a consequence of the law of similar triangles. T h a t is, the learning procedure can be seen roughly as a projection operator, which projects the whole W space onto the solution set. T h a t would explain why doubling the perturbation Ds would double its projection Dp. We believe the intercept in these figures reflects the dimensionality of the solution set, relative l o the number of dimensions collapsed by the learning-projection. These experiments show that the human solution does not have positive s t a b i l i t y — i t is a neutrally unstable subset of a larger attractor. This is evidence against an important conjecture [42], namely that " t h e network is lazy; i t will find the lowest-order predicate that is consistent with the data," We showed in section 6 that a second-order solution exists, yet the "W( reserve the word symmetry lo refer to guaranteed universal symmetries such as Ihe ones discussed in section 7. We use the broader term invariance to refer to anything that leaves E unchanged, including things which depend critically on the training data for this particular task.
408
895 generalized delta method does not show any sign of preferring this or any other second-order solution. I n fact, i t seems greatly to prefer A" -order solutions (perhaps because they are more numerous). We do not mean to imply that networks never find human-type solutions. Rumelhart, Hinton, and Williams [33] report that a back propagation network of precisely the type we are considering found a sensible " h u m a n " solution to the p a r i t y task, and Maxwell et al. [25] report that a one-layer network of sigma-pi units found a sensible " h u m a n " solution to the clumps task. The concept of the order of a predicate is useful for some purposes but not all. For a task w i t h (uniformly) bounded order, the number of connections required is less than i t would be in a task of order N, and this could affect the practicality of building a system to solve the task. On the other hand, if a network is given the resources to build representation of a given order, we see no reason why i t should not use all its resources. Perhaps one should use the notion of order to discuss restrictions on the resources available (or resources needed), rather than the resources used. h
11.
Preprocessors,
representations, and feasibility
Although the network's ability to memorize and recall data is impressive, the thing that really stirs the imagination is the hope that the network could extend this behavior to "similar" data it had never seen. Indeed, in the early days of network research, i t was hoped that the network would be able to generalize in several dramatic ways. Some of these powers have already been demonstrated; others remain topics of research, and others we believe to be unachievable. I t might be hoped that: The network builds a sensible internal representation. The network serves as a "rule-finding" system (in contrast to conventional A l programs, which are referred to as "rule-following systems"). The network behaves "as if i t knew the rules.'' The learning process is largely unhindered by local minima. The network is good lor "discovering hidden symmetries." The network generalizes. In discussing these ideas, i t is important l o be clear about the meaning of the terms. Also, if we claim to be designing a network, we must be careful to specify in advance just what the network is expected to do. A l l too often, people build a network and then retrospectively discover what i t is good for. This sort of analysis is useful, but should not be confused with synthesis. It is also i m p o r t a n t not to make intemperate claims. I t is clear that some networks can discover exceedingly general solutions to some tasks. For instance, let the input consist of binary numbers x and consider the predicate
409
896
" i is an odd number." Even a one-layer network can learn this predicate, using a very modest number of examples. It does in fact find an approximation of the "human" solution—it connects the output to the low-order input bit and ignores the other input bits. The network will then generalize from those few examples to every representable integer, which is a high ratio of generalization indeed. On the other hand, consider a different predicate, namely "the number x has an odd number of prime factors." We can't prove that there is no network that can learn this predicate, but such a thing seems too good to be true. Factorization is considered an intractable computer-science problem, and the network is surely no more powerful than a standard computer. Obviously, blatantly superficial structural properties (like the oddness of binary numbers) are easier to learn than deep abstract properties (like primality). But note lhat the question of what is superficial and what is deep is very sensitive to change in representation. In the base 3 representation, oddness is not such a simple structural property. Similarly, it is easy to imagine a representation in which a number is stored in terms of its factors, which would make factoring easy (and make addition hard). In many cases, the practicality of solving a problem hinges on constructing a preprocessor that transforms the data into a reasonable representation. Indeed, here is a proof that "automatic learning will always succeed, given the right preprocessor": let the preprocessor accept the raw input and "tag" it with the desired answer. (By this we mean combining the two using the Cartesian product, i.e. concatenating the bit strings.) Feed this processed input into a one-layer perceptron. It is guaranteed to learn to ignore the raw data part of its input and copy the desired output to the actual output, just as it learned to copy the low-order bit in the odd-even task. This settles the debate concerning the importance of preprocessors; they are all-important. It is, alas, completely wrong to conclude from this that "automatic learning will always succeed," since we have no automatic procedure for generating the required preprocessor. The questions of generalization, learnabifity, and representability are also sensitive to the architecture of the network. For example, the solution to the p a r i t y task using H = N hidden units depends on the analog accuracy of the weights. If the weights could take on only binary values (which is the way it is in standard digital V L S I processes), then a two-layer network would need 0 ( 2 " ) hidden units. That would be a very silly representation, since a digital solution using log A layers of N units is much more efficient. For that matter, there is a digital solution using only a total of A' X O R gates (although no simple learning method is known for such an architecture). The point is that the resources needed to do a good job on a given task are very sensitive to the shape of the network, the accuracy of the weights, the form of the objective function E(), and other architectural specifications. Therefore, we emphasize that the question of generalization must be answered quantitatively, not categorically. Some network architectures, some learning methods, and some representations are suitable for some problems. 7
410
897
Figure 7: Rule extraction.
12.
D e f i n i t i o n of "rule e x t r a c t i o n "
We hereby propose the following definition of rule extraction. This is, we believe, what most people mean by "finding the rules" or "discovering the symmetries" or "induction", and what some people mean by generalization. Reference works include Angluin and Smith [2], Holland et al. [14], Packel and Traub [28], and references therein. Let / be the set of all possible inputs to the network, and let A be the set of all possible outputs. Let the set U = I x A be the universal set of all possible ordered pairs (input, output}. All functions and other relations are subsets of U. As indicated in figure 12, let us choose a particular function, a rule R C U, and see how well the network can discover it. In our specific example, the two-or-more clumps predicate defines the rule of interest. Now we identify a subset of R which we call the memorization set, M, and another set X which is disjoint from M, i.e. X C R — M. We call X the extraction set or the extension set, since the idea is to extract the rule from the data M, and extend it to the testing data X. The network adaptively learns the data in the M set. T h e error function E which the generalized delta method seeks to minimize is defined as a mea7
*The tHO-or-more clumps predicate is, mathematically speaking, a Function, sinrc Tor each input pattern there is a unique output. The same is true for parity and most other predicates considered in the literature. Since all networks in the class we are considering are deterministic, the network's actual input-output relation is a function, too. In a realworld situation, the training data M may contain a certain amount of nois<-. Hy thai., we mean two things. For one, we extend our definitions to include the case where a few of the elements of M may be inconsistent with the rule we are trying lo find (i.e., they lie outside R). Second, in the presence of noise, M could easily be a relation thai is not a function. Learning from noisy data is perhaps the most important use of networks that we can foresee. We will return to this topic below.
411
898 1
sure of how accurately the network's actual input-output relation matches the desired relation Af. We define the network's extraction score (for the rule R) to be the the accuracy with which the network's input-output relation agrees with the X relation—that is, the data that it did not see during learning. Occasionally, kibitzers suggest that we could improve the extraction score of our networks by including a "few" of the points from X in the learning process; we emphasize that to do so would by definition defeat the idea of rule extraction. The network's ability to produce the desired output, given input data that it saw during training, we call memorization; the ability to produce the desired output corresponding to input data that it has never seen before, we sometimes call generalization of the rule R, but we prefer to call it rule ex-traction. We emphasize that rule extraction is a rather slippery concept, since it is possible to change a network's extraction score (without changing the network) simply by changing one's mind about what rule was "supposed" to be extracted. In order for rule extraction to make sense, we require the property we call represent at ivity; that is, the M and X sets must be representative samples oi R. It generally suffices to construct M and X by the same random process. (Figure 12 should not be taken to mean that M and X are systematically different.) The point is to rule out nonsense of the following sort: suppose we were seeking to extract the odd-even predicate. Imagine that the subsets were manipulated so that the abscissas in M were all equal to 0 or 7 mod 8, while the abscissas in X were equal to 1 or 6 mod 8. That is, in one set there is a positive correlation between the bit we care about and its two neighbors, while in the other set there is a negative correlation. In such a perverse case, we should not expect good rule extraction. We have performed experiments to see how well the generalized delta method and variants thereof can extract the clumps rule. In some circumstances, the rule extraction score is quite disappointing, and in other circumstances, il is rather good. Our struggle to reconcile these two results led to most of the ideas in the following sections. For smallish networks (// ss JV a 10 or less), and even using a substantial fraction of all available data for training, we observed rather poor rule extraction. To confirm this result, we performed a perturbation analysis similar to the one described above in section 10. The network used in these experiments had H = N, not H = N + 1, so the best performance (measured with respect to the testing set X = R — M) would occur for large 6; specifically, b > 4 suffices. For lesser values of b, the initial condition does not constitute a solution, but is only a hint, having the same symmetry and topology as the true solution. Given an exceedingly strong hint, the performance was 100 percent, even with no training data, as can be seen in figure 12. Given no hint at all, the performance was poor even after training. Given a moderately strong hint, training with one or two hundred examples led to good performance. The amount of hint required
412
899
Figure 8: Rule extraction wilh weak hints.
413
900
°< 2 0 0
training
o 100
training
patterns
0 training
patterns
•
0.0
0.0
patterns
1.6 CauHlan
2-0
Noire A m p l i t u d e
Figure 9: Rule extraction with noisy hints.
414
2.E-
3.0
901 decreased dramatically when the size of the training set was increased. In similar experiment, the network was always given a strong hint (6 = 3.5), but the weights were perturbed by adding random noise. This is shown in figure 12. The data in figures 12 and 12 shows a distinct basin of attraction. T h a t is, there is a special region in W space surrounding the geometric solution. When started from any point w i t h i n this region, the learning procedure leads to a final point w i t h a rather good rule extraction score (X score). Starting outside t h a t region produces a final point with a rather poor X score (even though i t has E as 0). This is in contrast to figure 10, which did not show any basin of attraction. From this, we conclude that the geometric solution, indeed the whole multidimensional solution set related to the geometric solution, is a very small subset of the region of W space that has a good X score. On the other hand, figures 12 and 12 show that beyond a certain distance, the learning procedure does not lead to a point with a good X score, geometric or otherwise. This is the basis for our strongest negative conclusion: wc have reason to believe that our implementation of the learning procedure is not faulty (since i t does exhibit learning and rule extraction for some starting points), yet for general starting points i t does not find a set of weights w i t h a reasonable rule extraction score. This result applies under the stated conditions of //, N> m , etc.; we emphasize that under other conditions we have achieved a very good X score, as w i l l be discussed below.
13.
Definition of "generalization"
To pursue these ideas, we must now define what we mean by true generalization (as distinct from rule extraction). As before, let U be the universe of relations, and let M be the memorization data, a subset of U. Now consider the sets Gi, G3, etc. (For simplicity, let us restrict M and G, to be lunctions for now.) We say that Gi is a generalization of M if M is a proper subset of Gi, as shown in figure 13. T h a t is, the relation G has a larger domain than A:/, and the two relations agree wherever their domains overlap. Note that we do not speak of the generalization, but one of many generalizations. There are, in fact, an enormous number of generalizations, and it is interesting to calculate the number. Consider a network, a definite network that is not, for now, undergoing training. This network performs some definite Boolean function, and we can tabulate this function in a t r u t h table. Since there are N input bits, the t r u t h table w i l l have 2 rows. We now ask how many distinct functions can exist. If there are a output bits, there arc 2" possible output symbols for each row of the t r u t h table and a total of 2 x a independent bits on the output side of the table. We choose values for those bits in all possible ways and calculate how many different t r u t h tables exist; namely, = ( 2 ) " (where T denotes the set of all functions, i.e. t r u t h N
N
0
2
415
902
Figure 10: Many generalizations are possible. tables). In the case of a single output b i t , this simplifies to # ( ^ ) = 2 " possible Boolean functions. It is sometimes useful to imagine the space of all possible t r u t h tables as a hypercube with 2 dimensions. Each dimension is a row of the t r u t h table, and each corner is one of the 2 distinct functions. Now in quest of generalizations, we ask which functions are consistent with our training data M. Let m = be the number of data items (ordered pairs) in the M set. This data dictates the output values for m of the rows in the t r u t h table, freezing out m of the dimensions in the t r u t h hypercube. There remain (2^ — m) x a undetermined bits, and hence the number of possible generalizations is 8
5
N
3
#(t7) =
J
(2°) ~/(2T
(13.1)
where Q denotes the set of all (?,. This calculation is similar in spirit to the definition of Kolmogorov complexity [22]. The numbers in this equation are fantastically large. As a modest example, take a single output bit (a = 1), thirty input bits (N = 30), and a thousand training examples (TO = 1000). Then there are #(6) = 2'°'/2 °' = 2'°' generalizations. These are all perfectly valid generalizations, in the sense that they are perfectly consistent with the training data M. ,
14.
E n t r o p y and i n f o r m a t i o n
Wc can use these ideas of function-counting to discuss rule extraction, and at first glance, the numbers seem very discouraging. The learning procedure "Wc will henceforth discuss only the a = I case; tbe interested reader will have no difficulty in deriving the corresponding general expressions.
416
903
can result in any one of a vast number of possible generalizations. The data does not provide any reason to prefer any of these over any other. The set of networks that compute the rule we are seeking is a vanishingly small fraction of the set of all possible networks. It is inconceivable that any automatic learning procedure would be able to stumble onto the "correct" extension. Therefore, we must not search the space of all possible [unctions or all possible networks. We must perform a constrained search. There are many ways of doing this. One crucial idea is to search the space of all efficient networks. In some cases, we may know a priori that the task that has been posed can b e solved by an efficient network. This special knowledge may come, for instance, from considerations of the geometry, symmetry, or intrinsic complexity of the task. In other cases, we may not have such special knowledge, and we must in principle search all possible functions. These are what Abu-Mostafa [1] calls "random problems". However, all is not lost, because we have a choice: the crucial idea is that we choose to search the efficient network functions first. We have powerful search techniques for such functions. If, after efficiently searching the efficient networks, we still do not have a solution, we can then decide whether or not further searching is worth the trouble.
14.1
Preview
We are now almost ready to derive the main results of this paper. The discussion is a bit complex, so we will briefly list the main ideas and then proceed to derive them in detail. 1. We will be searching through the space of networks (i.e. weight space). Note that the previous section discussed searching through the space of functions; we will exploit the connection between the two spaces. 2. Since we will be calculating a probability, we need some notion of a prior distribution. We call this "throwing darts at weight space." It allows us to assign a probability P[W)dW to each volume element dW in weight space. 3. We will extend the idea of exactly correct rule extraction by accepting nearly correct extractions wilh some e r r o r tolerance / . You can of course set / = 0 if you want to recover strict rule extraction. 4. We use the training data M and all other information at our disposal to restrict as much as possiblethe portion of weight space that we need to consider; call this portion W. We then calculate the probability that a point in W will meet our /-acceptance criterion. II this probability is near 100 percent, we conclude that this network can be expected to perform rule extraction with the required accuracy / . On the other hand, if this probability is small, we conclude that the network can not be relied upon to perform rule extraction.
417
901
We remind the reader that one is not allowed to search W space to find the "correct" rule extracting network. T h a t cannot be done w i t h o u t using data from the testing set X, which defeats the purpose, by definition. T h a t would be like betting on the winning horse after the race is over. We are only allowed to play the probabilities i n W space. 14.2
Derivation
The task of choosing a probability distribution in W space is a b i t tricky. The choice depends on just what method is used for "learning", i.e. for searching W space. Fortunately, the exact form of the distribution is not important for our argument. You could, for instance, use a probability density proportional to e'™V j for some "radius"' w. We w i l l for most purposes use a distribution that is uniform inside a hypercubical volume (w on a side) and zero elsewhere. We choose w to be big enough to enclose reasonable weight values, but not too much bigger than that. w
We can map weight space onto function space as follows: for each configuration of weights, W, build a network with those weights. Present it all possible binary inputs. Observe the corresponding outputs, and convert to binary. This mapping associates a definite t r u t h table, i.e. a definite Boolean function, with each point in W space. To say i t the other way, the inverse image or a function is a region in weight space. By integrating over weight space, we can assign a probability P, to each function. If w is large enough, and if there are enough hidden units {H on 2 ), there will be non-zero probability assigned to every function, according to the discussion in section 5. On the other hand, we are particularly interested in the case where there are very few hidden units, perhaps H oc N or JV . In that case, we expect many functions to have zero probability. N
2
3
It is interesting to consider the quantity we call the "functional entropy", namely $••= ^ - P d o g P i
(14.1)
where 7 is the set of all functions. A l l logarithms i n this paper are base 2, so entropy is measured in bits. I t has its maximal value when all functions are equally likely, in which case S = 2 . If some functions are less likely than others (or ruled out completely), the value of S is reduced. We define S to be the initial value of S, measured before any training has taken place. A large S means that the network is capable of solving a large class of problems; a small So means that the architecture has restricted the class of problems that this network can handle. Now we get to use the training data M. The training data applies directly to function space, but we can use the mapping to "black o u t " the regions of W space l h a t are (inverse images of functions that are) inconsistent w i t h the training data. We can also define a reduced functional entropy, N
0
0
418
905
S= m
£ -P-HP' lie**
(14.2)
where the "*" indicates that we have removed from consideration all those functions that are inconsistent with the training data M and normalized the remaining probabilities. If all functions were equally likely, we might hope that each element of training data would reduce the entropy by one bit, so that 5 = So — rn. Alas, we forsee that this will not always be the case, so we define r/ to be the average efficiency with which the learning procedure extracts information from the training data, defined by the expression m
9
,„,, It is sometimes useful to treat m as an independent variable and define the local efficiency, n
m
= -o\S /din.
(14.4)
m
There are a number of possible reasons why i\ might be less than unity. The most obvious possibility is that the training data might contain duplicate points. The first copy of that point would freeze out one axis of the truthhypercube, reducing the entropy by one bit, but succeeding copies of that point would contribute nothing. We have obscured this issue by referring to the training data (Af) as a set-—and a set cannot have duplicate points. In the real world, however, the training data is treated as a list, not a set, so it can have duplicate points. We also point out that in the case where M is not a function, the phrase "rule out" is too strong a term. Perhaps "provide evidence against" would be better. Conflicting evidence, like duplicate evidence, will lower the efficiency. A more profound and interesting case occurs when a training item cuts the truth-hypercube across an axis such that the entropy is not equally distributed among the two halves. It is well known that the optimum strategy for playing the game of twenty questions is to use questions that divide the number of possible objects in half each time. This strategy returns one bit of useful information per question; any other type of question returns less. In the other direction, it might seem possible to construct a network with an artificially large r), even greater than unity, by hindsightfully choosing a rule R that coincides with whatever function the network initially produced. In such a case, the network could seem to "learn" the function with no data whatsoever! We insist, however, that this cannot be considered a rule extraction. Any claim of rule extraction which implies an efficiency greater than unity must be viewed with extreme skepticism. It is possible, of course, every so often to stumble onto a good rule extraction based on very little data, just by dumb luck and waiting for coincidences. 9
T h i s is not to be confused with other forms of efficiency, such as efficiency of representation mentioned in section 6.
419
906 For instance, consider the following unusual game of 20 questions: Q: "Is your name Rumplestiltskin?" A: "Yes." This outcome is technically possible, but any strategy t h a t asks such questions is a very poor one. I t can only search 20 names, whereas a good strategy can search 2 names. A n apparent n > 1 will only occur if the rule R is very special, and/or the initial value of tbe weights W is very special. This sort of nonsense can be detected and prevented by evaluating I J for a number of different, sensible rules and/or using an ensemble of randomized initial weights. 2A
15.
C a l c u l a t i n g the efficiency
By introducing some approximations and restrictions, we can arrive at a convenient way of estimating the initial entropy in practical situations. In the network of figure 4, each hidden unit has =a N weights connected to its input wire. If a signal of size u is significant t o the hidden unit's input wire, a roundoff error of size xt/y/N per weight would add up t o be significant, if the errors added in quadrature, which is characteristic of random numbers. (Errors of size u/jV would add up t o be significant if they all added in the same direction.) From this, we conclude that each weight needs roughly ,51og/V (or perhaps logJV) bits of precision. The fact that our solution, figure 6, has only three non-zero weights per hidden unit does not count, since we don't know that fact until after learning is completed. Let us calculate the number of bits B needed t o specify the configuration of the network. A network with H ?z N hidden units has 0 { N ) weights, i n which case B = .5N log N. Therefore, there are of order 2* » " different networks lhat we can build. T h a t is a large number, but i t is quite a reduction from 2 ". In general, to obtain a value (or B, one must know L , the number of (significantly) different levels that each weight t in the network can take. Specifically, 2
2
NL
h
2
;
B =
Y,
leg**
(15-1)
weights
It is difficult t o obtain an exact value for L;, b u t i t easy t o obtain a good estimate, and only its logarithm matters anyway. T h e estimate can be performed by iteration: start w i t h a small £ and increase i t , building a series of networks. Stop when you get a network that is capable of learning. Our emphasis on networks with the m i n i m u m number of specification bits B is important not only (or information-theoretic reasons, but also because there are important technological limits to the precision w i t h which weights can be fabricated in real-world networks. The notion of an efficient representation, which was introduced back in section 6, can now be made precise: we require that B grow no faster than some polynomial in /V, the size of the problem. We take as our prior distribution the notion t h a t all of our 2 networks are equally likely. s
420
907 We tried to arrange by construction that each of the B bits is "significant". Imagine for a moment that each bit was sufficiently significant that each t i m e we change a bit the network implements a different Boolean function. I n that case, there would be 1 different functions, all equally likely, and the functional entropy would be So = B. B
This is clearly an overestimate for 5 , for a number of reasons. For one t h i n g , the actual number of different functions implemented by our B-bit network w i l l be less than 2 , because of the various symmetries discussed in section 7. Furthermore, there w i l l most likely be some low-order bit in the network specification somewhere that does not change the input output relation A{). T h i s means that the 2 imaginary networks are "folded" onto some smaller number of actual networks. We can get an expression for S by taking i n t o account as many of these foldings as possible. 0
B
B
2 " s
=
n
(15.2)
2»
<
/7!2
H
The first factor in the denominator accounts for the permutation symmetry of the hidden units. The second factor accounts for the polarity symmetry. The factor F, represents the fact that although our construction of B guarantees that most of the bits will be significant, quite likely not all of them will be. The fudge factor F represents the 6-symmetry; we do not know how much of this symmetry survives the coarse-graining involved in our construction of B. The factors F --represent any other symmetries that have simply escaped our attention. Setting these unknown factors to 1 gives us an upper bound on SoWe believe that whenever the final rule extraction score is significantly better than 50/50, the final entropy must be very small, for reasons given in section 16. In that case, we can estimate the efficiency as TJ So/m, using the value of S estimated from equation 2. 2
3
0
Other
Viewpoints
Equation 14.3 defines an average efficiency n that was actually achieved in a particular case. Another viewpoint would be to imagine that there is a definite expected efficiency rj that the network is capable of. We can then consider the expression (15.3)
6S = So-S -rlm m
If 6S is positive, then the number of consistent generalizations (consistent w i t h the training data M) is large compared l o the number of /-acceptable extractions, and we would expect that the rule extraction score would be very poor. On the other hand, i l we increase m (the amount of training
421
908 data), then we can presumably winnow the number of valid generalizations down to the point (6S < 0) where they are all /-acceptable extractions. Similarly, we could hold 6S = 0 and hold rj fixed, and solve equation 15.3 for the expected error rate / as a function of TO. Yet another possibility would be to consider everything but So fixed. This equation would then tell us how much we would need to restrict the space of a priori likely functions, by limiting the number of hidden units or their connectivity, etc. Our experiments indicate that the network can learn the 2 v e r s u s 3 clumps predicate with reasonably high efficiency, as shown in table 4. For instance, using N — 16, the network was able to extract the three-or-more clumps rule with a 18 : 1 gain factor (< = 5.4 percent), and 97 percent correct rule extraction (/ = 3.3 percent). This gives n = 56 percent (assuming S = 0). m
ji Inputs N 16
Data Used
Error Rate
TTl
/
800
033
Efficiency .56
Table 4: Rule extraction score versus N.
16.
A model system
There is a model system which captures the our main ideas about rule extraction, yet is simple enough to be soluble. Consider the following feat: a particular card in a deck of 52 cards is marked, and a blindfolded mentalist offers to deduce which one it is. Now the simplest way is for him to ask a series of questions: Is the marked card in the top half of the deck? Is it in the first or third quarter? Is it in an odd-numbered eighth? and so on. Alter six questions, there is no doubt as to which of the cards is the marked one. The analogy to automatic learning is this: the deck is an ensemble of functions—imagine that a truth table is written on the back of each card. The distribution is weight space is such that it gives each of these functions equal probability and all other functions zero probability. The questions (with answers) are the training data. This version of the model is not very realistic, since in real applications the training data generally has not been constructed to be optimal and orthogonal. Therefore, consider a second version of the model, in which the questions divide the cards in a random way. It is easy to visualize randomizing the cards, which is equivalent to randomizing the questions. The mentalist first asks iT the marked card is in the top half of the deck. He then punches a hole through every card in the half that does not contain the marked card. Then the deck is shuffled. Again, he asks if the card is in the top half, and punches all the cards in the excluded half. This continues, with the deck being shuffled before each question. In this version, six questions do not suffice, but
422
909 after many questions have been asked it becomes exponentially unlikely for any card except the marked one to remain unpunched. We now come to the third and final version of the model. We must take into account the fact that we do not expect the questions to divide the deck in a nice 50/50 way—the shuffle will not be perfect. What really happens is that each training item (i.e. ordered pair) focuses attention on a particular row of the truth tables that are written on the cards, namely, the row that matches the abscissa of that training item. The deck is then divided into two pieces, according to whether the ordinate on the card agrees or disagrees with the marked card. The ones that disagree get punched. The fraction that disagree is denoted <j>; this is the principal free parameter of this model. A perfect shuffle would give 4> = 1/2, Let the expected number of unpunched cards (after m questions) be Z; then, £
Z = l+(2 °-l)n(l-<M-
(16.1}
m
The first term, 1, represents the marked card, and the second term represents all the other unpunched cards, which are reduced by a factor of (1 — <£ ) each time. For simplicity, we will treat <j> as a constant from now on; it is easy to generalize the formulas. m
If we pick a card at random from the set of unpunched cards, the probability of not picking the marked card is ( Z — 1 ) / Z , and therefore the total probability of picking up a card lhat disagrees with the marked card is
(lfi-2)
f = ^~-
We have used the principle of representativity (discussed back in section 12) to connect tj>, which is a property of the M set, with / , which is a property of the X set. Now, suppose that we wish to train the network until it does better than some specified error rate / * . How much data do we expect it to require? We assume that the network learns as efficiently as possible given adverse training data. Within this model, the worst 4 that could be produced by any training set would be $ «i /", because if <j> were smaller than this, equation 16.2 would imply lhat / was less than / ' , and the task would be complete, while if 4> were larger than this, the network would learn faster according to equation 16.1. For small / " , we expect m
^5oln2//-.
(16.3)
The network can learn faster than this if the error rate stays up near 50/50 until near the end of training, and it can, of course, learn more slowly if it does not live up to its information-theoretic potential. The entropy is
423
910
0
SO
100
130
ZOO
250
Size s i Training Sel
Figure I I : Entropy and error rate versus m.
S
m
= logZ.
(16.4)
It is useful to introduce the pre-efficiency p = — log(l — and recall that the local efficiency rj = — dS /dm. We assume <j> and hence p will be constant, or at least relatively constant. Initially, / = ^ and rj = p., but / and r) decrease exponentially after the entropy has been squeezed out of the network [pm > S ). Figure 16 shows the entropy and error rate for So = 50, p = 1/3. Changing S would rigidly shift the curves horizontally, and changing p would just rescale the horizontal axis by a uniform factor. m
m
m
m
0
0
17.
Associative memory and clustering
The ability of massively parallel networks to perform associations, and to categorize the inputs into clusters, has received enormous amounts of attention— for a review, see reference [32]. Sometimes people get carried away and take associative memory as the definition of what a network ought to do, or even as the definition of computation in general. We take the opposite view, 10
'"The T\iring machine, although not a practical device, is widely used as a formal, theoretical model of "computation in general." An associative memory is clearly less
424
911
1
A fl
\ i \
o at *? V
Layered Network o Clustering - W i t h Weight Decay
\
c m to
—A
a
|
i
V
^ 6
•
-€>
a 0.0
10.0
30.0
110.0 Si70
of
40.0
50.0
80.0
70.0
Network
Figure 12: Rule extraction under various conditions. namely that clustering is a particular case of rule extraction in which the rule is (roughly) of the form "nearby inputs should produce nearby outputs." The definition of "nearness'' depends on the specific task in question. A general learning machine can learn to do clustering, b u t the reverse is not true: there are huge categories of tasks a specialized cluster analysts machine cannot do. A t worst, the general machine's task can be discussed in two phases: first the network discovers that clustering is appropriate, and then i t learns where the cluster boundaries should be. A cluster analysis machine has less work to do, since i t already "knows" that clustering is the way to go. There are a number of quite efficient clustering methods ([21] and references therein), and they should be used whenever they are appropriate. More general rule extraction procedures must be used in the remaining cases. The upper trace figure 17 shows the rule extraction score for a very simple layered network—no particular effort was made to optimize its performance. The lower trace shows the analogous results for the most powerful clustering analysis we could think of, as described in appendix A. The layered network was clearly superior in this case. The figure also displays the data from table powerful than a Turing machine; for starters, it cannot perform loops.
425
912
4, showing the importance of stochastic weight decay. 18.
S y m m e t r y , easiness, and p r o g r a m m i n g
At first glance, the amount of data required for learning, as indicated in table 4, seems rather large. Humans seem to "discover the rule" using far fewer training examples. To understand this phenomenon, consider the following modification of the task: let us permute the bits of the input pattern, leaving unchanged the numerical value of the output. (There is only one permutation, applied to all patterns.) An example of this is shown in table 5, in which the permutation (0123456789) - • (3120459786) was applied. This is a very simple permutation (two pairs are exchanged: 0 with 3, and 6 with 9), but it has a drastic effect on the appearance of the input patterns. Old Input ffftttffff fffttftfff fttttttttt tttffttfft ffffffffff
Output F T F T F
Permuted Input tfffttffff tffftftfff tttftttttt ftttftttff ffffffffff
Table 5: Permutation of the input. This change has a similarly drastic efTect on the rate at which humans learn the rule. First of all, the humans must guess that the task requires treating the input as a geometric pattern, which is not an obvious assumption—for all they know, the task might require treating the inputs as binary numbers and factoring them! Even if they guess that permuted geometry is important, they must guess what particular permutation has been applied and then discover the clump-counting or edge-counting rule. The humans would require a stupendous number of examples in order to achieve this. Permutation of the input bits is an exact symmetry of the architecture; lhat is, the first step in the operation of the network is a calculation of the form £»V.
(18.1)
i
in which the symbol t is a dummy index. This is in addition to the similar symmetry of the hidden units, described above. Humans have a very strong prejudice in favor of geometric solutions. They consider the original task easy and the permuted task hard. The network has no such prejudice. It has no built-in notion of geometry or topology. For the network, the original task is just as difficult as the permuted task. We feel that this difficulty is artificial. It arises when one asks an overgeneral network lo solve an under-specified task. In real-world situations, the data does not exist in a vacuum, but exists together with important
426
913 ancillary information such as symmetry, geometry, topology, and so forth. When we presented the t v o - o r - m o r e clumps predicate hack in section 3, we presented the topological information to the reader, along w i t h the data. T h r o w i n g away the ancillary information makes the problem more abstract and certainly makes i t less tractable. W h y should we ask the network to solve a very hard general task, when all we cared about was a rather easier special task? Learning from examples and related techniques will never replace programming— they w i l l supplement i t . Someone who understands a task will always do better than someone who does not. It is important to realize that search techniques are useful when you have an intermediate amount of knowledge about a task. There are some tasks (such as finding the m i n i m u m of a parabola) t h a t are so well understood t h a t the answer is obvious, or obtainable by by conventional analysis. A t the other extreme, in the case of a t r u l y random function, sophisticated procedures will not do any better than simple procedures. (You use grandmother cells to memorize the M data, and guess at the rest.) In an artificial example such as the t h r e e - o r - m o r e clumps predicate, i t js a question of taste as to what constitutes "giving ancillary information" and what constitutes "giving away the whole answer." Real-world tasks are so much more complicated t h a t giving all available ancillary information still leaves plenty of work for the network to do. Having decided l o provide ancillary information to the network, we need sensible techniques for doing so. These techniques can be thought of as a strange sort of "programming language" for networks. One way is to change the architecture of the network, restricting the "receptive field" of each of the i n p u t units. For example, we could require that Wij = 0
whenever \i — j \ > p
(18.2)
where p represents the radius of the receptive field. A gentler and more general way to implement limited receptive would be t o add to the E function a term of the form
fields
In the case where the multiplier A is large and the kernel K is a suitable step function, this method becomes effectively identical l o the previous restriction. The idea of "programming" the network by adding terms to the E function is traceable to the traveling salesman network paper of Hopfield and Tank [15]. Either of these methods provide hints that the solution should be sparse, topological^ one-dimensional, and local along that dimension. They break the permutation symmetry of i. To say i t another way, we give the network a notion of neighborhoods (each containing w 2p input bits), and these neighborhoods induce a topology on the input space. 3
427
914
The work on "feature spin" systems |24,3,4,21] used carefully constructed input and output spaces to perform automatic clustering. Hinton [43] used a network with three layers, one of which had one-dimensional limited receptive fields, to good advantage. Our studies of the representations found by networks when given such hints wilt be reported elsewhere [34]. A different type of ancillary information can be provided. We notice that the solution presented in section 6 has a high degree of translational invariance. It is easy to concoct a term analogous to equation 18.3 that penalizes solutions that are not translationally invariant. It is also possible to impose a rigorous restriction, analogous to equation 18.2, but one must be careful since the solution is not exactly invariant. The human prejudice in favor of geometric solutions is not an accident; it is the result of thousands of centuries of accumulated information about the world. For any particular geometric task, there may be a very efficient ad hoc, non-geometric representation, which has nothing in common with the solution to any other task. The geometric solution is useful for many, many tasks. In order to test this idea, we are checking the stability of the human solution when a network with several output units is required to perform the combined 3 / 4 / 5 / 6 / . . . clumps task [34].
19.
N E R F s . r e g u l a r i z a t i o n , and c u r v e fitting
Once again, let us consider a collection of data and assume we have a reason to believe that it adheres to some simple rule. For instance, the data might be obtained by measuring some very simple physical system. This does not give us any reason to believe that the data can be represented efficiently by a network (i.e. by a network with H <SC 2 hidden units). What's more, in cases where a representation exists, the solution may well not be unique, and (as discussed back in section 8) the learning process may be NP-complete, in which case there is no general automatic learning procedure that will find the solution appreciably faster than an exhaustive search of weight space. N
It is quite important to reconcile these sobering points with the observation that automatic learning procedures in general, and layered networks in particular, are capable of doing real-world tasks, sometimes astonishingly well (see examples and references listed in section 2). It is also imperative that we understand what the limitations are. Fortunately, there is a powerful analogy that sheds considerable light on this situation, namely the analogy to curve fitting. Consider the task of fitting a smooth curve through the four data points shown in figure 19a. One way to do this would be to minimize the error function (19.1) where (*j,ffrj are the experimental data points, a is the uncertainty in yi, and / ( ) is the theoretical function (actually a family of functions, dependt
428
915
Figure 13: Curve
filling.
ing on some adjustable parameters) to be fitted so we can extrapolate and interpolate over some interval D. For concreteness, let us consider the case where /() is restricted to be a parabola. This is the most elementary form of curve fitting. T h e resulting fit function /() can be considered a generalization of the data points in the sense of extending the domain. The data can be considered a function (or relation) on a very small domain—just the isolated points {ii}. The function /() is defined on the whole interval D. That is a fantastic degree of generalization, a huge extension of the domain. Curve fitting is also generalization in the sense of averaging noisy data. The data could be quite unsmooth, yet a low-degree polynomial that fits the data cannot possibly have very much curvature on the interval D. Curve fitting can, however, be very tricky. In figure 19, the data has uncertainties in r , as well as Only small horizontal shifts, much less than the length of an error bar, would be needed to make the four points form two pairs, each pair having a single abscissa. In that case, there would be an infinite number of parabolas that could be fit through the points, all having the same E value or "chi-square" or "goodness of fit". Also, when the data is nearly paired, small changes in the data can lead to arbitrarily large changes in the parameters of the "best" parabola, as illustrated by going from figure 19a to figure 19b. This is a double-purpose example of what Hadamard [12] classified as ill-posed or ill-conditioned problems; that is, the solution is not uniquely determined by the data, or the solution depends discontinuously on the data. Tikhonov |.'!8] showed how to deal with such problems; in this case, wc just need to remember that what we originally wanted was a smooth curve through the data, i.e. a function w i l h low curvature. This is expressed in the
429
916
following formula by the second term", which explicitly and precisely penalizes functions that have a large curvature.
Of course, this is just an example of a possible Tikhonov regularization term. One chooses a penalty term that is appropriate to the actual task. The idea is that the penalty term expresses an estimate of the a priori implausibility of each possible solution / ( ) . There is a vast literature on maximumlikelihood estimation—spe. Press et al. [31]. and references therein. Note that we have invoked two subtly different ways of limiting the curvature. One is what we call structural stabilization, in which we explicitly limit the degree of the polynomial so that it cannot have much curvature. The second case we call formal stabilization, such as equation 19.2, in which it is not really necessary to require that / ( ) have a small degree. The Tikhonov regularization term will automatically select a function of low curvature, whether or not it can be expressed as a low-order polynomial. We emphasize that neither stabilization by structural restrictions nor formal, Tikhonov-style regularization is the exclusive, universal solution. The latter is more elegant, and it provides a mathematical language for discussing things that would otherwise be "teleological and anthropomorphic", as George Furnas put it [44]. On the other hand, there are important practical reasons why structural restrictions are often more appropriate; for instance, they can greatly reduce the number of free parameters, making the search more efficient and making the answer more readily describable and understandable. Practically all of these ideas can be applied equally well to layered networks. Just as a polynomial with high enough degree can closely approximate most reasonable functions, a network with enough hidden units can represent any Boolean function. Low-degree polynomials can only represent relatively smooth functions, and small networks can only represent a certain class of relatively simple Boolean functions. We call these particular functions network efficiently representable functions, or N E R F s . If the training data is noisy, and only defined on a sparse set of points, fitting a N E R F to it will average out the noise and provide us an extension that covers the entire input space. If the data is inadequate to specify a unique N E R F , we can add to the optimization equation a stabilization term such as in equation 18.3, which is quite analogous to the second term in equation 19.2. It expresses the designer's preference among solutions that would otherwise have equivalent E-functions. The question immediately arises: what is so special about N E R F s ? For that matter, what is so special about polynomials? Polynomials are special for several reasons. Perhaps the main reason is that polynomials occur in nature. To a good approximation, the path of a thrown stone draws a parabola in the sky. A second reason is that polynomials are reasonably easy to evaluate by elementary methods. Third, a
430
917 polynomial has a small number of adjustable parameters, so even if the fit is purely phenornenological and fortuitous, i t is a handy way of describing the data. Finally, people just have a lot of experience dealing with polynomials and can readily visualize what they look like and how (as the degree is increased) they come to resemble other functions. NERFs have some analogous virtues. It is hard to say just how common NERFs are in nature, b u t i t is not easy to concoct a meaningful, natural function that is not a N E R F . This is in contrast to single-layer perceptrons, which Minsky and Papert (27) showed could not solve the p a r i t y problem, T/C problem, or any of various connectedness problems. For the /V-layer networks, which are much more powerful, it would be nice to have an analogous pons asinorum which the network cannot solve, as well as a simple analytic concept (analogous to linear separability) to help us understand the limitations. It might be said the NERFs exist in nature in another, quite different sense: recall l h a t the reason that this family of functions came originally to be considered is l h a l they seem to be a modest model of the computational operations that are implemented by the neural circuits in the brain. As for the second virtue, there is no doubt that NERFs are especially easy to evaluate. Analog integrated circuits have been built which perform the required products and sums using far less time and space than would be required for other, seemingly similar, functions [17,18], As for the third and fourth virtues, NERFs are not widely used for phenornenological data reduction, precisely because most people do not have a keen i n t u i t i v e understanding of whal they look like. To a certain extent, this defines the present and future task of everyone in this field: our job is to understand NERFs, until they become as familiar as polynomials. Of course, neither polynomials nor NERFs are a panacea. Suppose that rather than the points in figure 19, the data consisted of a thousand points w i t h very small error bars, closely fitting ten cycles of a sine wave. It would be insane to fit that data w i t h a high-degree polynomial; it would be much more sensible to use a low order Fourier series. The set of low-order Fourier series and the set of low-degree polynomials are both subsets of the set of all smooth functions. Any given smooth function might belong to one, or the other, or neither of these subsets. Similarly, the set of low-order NERFs is an important but still limited subset of the set of all "reasonable" Boolean functions. Surely for some applications we must t u r n l o other families of functions to smooth and extrapolate our data. Various extensions to the basic network models, such as sigma-pi networks and higher-order terms (numerous references in [32]; also 1,7]) are analogous to the way in which rational functions are an extension of the family of polynomials.
431
918
20.
Summary
A generalization is a function that extends the domain of the training data. This is very useful for averaging noisy data. Practically all of the problems (and the power) of this sort of generalization can be understood by comparing it with the well-studied discipline of curve fitting. Given enough training data, the generalization will be unique. More commonly, though, there are an enormous number of perfectly valid generalizations, i.e. functions that are perfectly consistent with the training data. The amount of data needed to determine the result allows us to define a measure of efficiency with which the network extracts information from its training data. Giving the network too many resources (e.g. too many hidden units) increases the initial entropy and hinders rule extraction. Rule extraction involves comparing the generalization found by the network with some prechosen rule. The distinction between the memorization set and the extraction set must be scrupulously respected. The language of regularization theory is important because it allows us to speak quantitatively of "good" generalizations and "bad" generalizations. A layered network is capable of extracting the two-or-more clumps predicate from the training data. Indeed, it extracts it with rather high efficiency. The internal representation that it uses does not have any discernible degree of symmetry, sparseness, or locality; the symmetric, low-order, local solution that humans prefer is of course a fixed point of the training process, but is not stable against perturbations. A general, tabula rasa network is a fine subject for abstract, formal studies, but one should not try to use it to solve practical problems. Automatic learning will never replace programming—it will supplement it. One should pre-program the network with all available information about the structure of the problem, especially information about the symmetry and topology of the data.
21.
Acknowledgements
We thank Arno Penzias for calling our attention to the two-or-more clumps predicate and the questions of generalization and "human" representation that it raises. Our preliminary results were intensively discussed on the computerized bulletin board "[email protected]" created by David Touretzky. Contributors to this discussion include Alexis Wicland, Christof Koch, David J . Burr, Terry Sejnowski, Thomas Maxwell, Scott E . Fahlman, GeofT Hinton, George W. Furnas, Jerry Feldman, Steven Nowlan, and Richard Golden. We are particularly indebted to David Burr for providing us with relevant "back issues" of the bulletin board. Wc thank Lee Giles for making available the results of his group [25] prior to publication. It is a pleasure to acknowledge important discussions with Bill Jeffrey. Our friend and colleague Erik deBennedictis created the hypercube hardware and software that was indispensible to our work.
432
919
Some of us carried out part of tins work at the Santa Fe Institute (JSD) and at the Institute for Theoretical Physics, U C Santa Barbara ( J S D and SAS). We are grateful to these institutes and their participants for their hospitality, support and interest.
Appendix A.
C o n d i t i o n s of the e x p e r i m e n t s
The data in table 3 was taken using the L M D objective function [40]. The predicate was 2 v e r s u s 3 clumps. The memorization set was exhaustive, using complete smoothing. The logical levels were T = 1 and F — -I, and the convergence criterion was 100 percent categorically correct (i.e. closer to the right target than the wrong target). The initial distribution of weights was uniform on the interval [—.2,.2], Each data point is the median of an ensemble of fifteen trials. All hidden units in all the trials reported in this paper were fully connected. The data in figure 9 was taken using the L M S objective function. The predicate was 1 v e r s u s 2 clumps. The memorization set included 40 items (20 of each class), and there was extreme incomplete smoothing—one by one. The logic levels were T = 1 and F = 0, and the convergence criterion was R M S error < .005. The initial distribution of weights was Gaussian, with unit variance. Each data point is the median of an ensemble of ten trials. For figures 12 and 12, the memorization data was the same as In the previous paragraph, except that the predicate was 2 v e r s u s 3 clumps, and the size of the memorization set was varied as indicated in the figure (always half in each class). The extraction set consisted of all the remaining patterns, so the rule extraction gain ratio (1 — c)/c is a strong function of /V, and the extraction set was not half in each class. Note: the number of patterns with C clumps is jf^J')- The criterion for correctness during the testing phase was being with .2 of the target. In figure 17, the predicate was 2 v e r s u s 3 clumps. The memorization set included 50A items and the extraction set consisted of another 50A' patterns (half in each class). The data in the upper curve was taken using the L M S objective function, using the generalized delta method with incomplete smoothing (one by one). The criterion for correctness during the testing phase was being with .2 of the target. The high isolated point represents the data from table 4, which was taken under the same conditions, with the addition or stochastic weight decay, according to the following scheme: For each weight, with probability P, multiply by (1 - o ) , where K is the number of completed passes through the data, and in this case P — 1/250 and a — 0.1. 7
K
The lower curve in figure 17 was generated as follows: All elements of the M set were used as "prototypes", i.e. the centers of clusters. Each clement of the X set was compared with each of the prototypes, and a histogram of distance was made. All prototypes in the minimal bin (closest distance) voted on what class the X element should be in.
433
920
References [I] Y . Abu-Mostafa, article in Complexity in information Theory, Y . AbuMostafa, ed., (Springer-Verlag, New York, 1986); see also article in Neural Networks for Computing, (Snowbird 1986), AIP Conference Proceedings #151, (American Institute of Physics, New York, 1986). [2] Dana Angluin and Carl Smith, Computing Surveys, 15:3 (1983) 237. [3] David J . Burr, "A dynamic model of image registration", Computer Graphics and Image Processing, 16 (1981) 102. [4] David J . Burr, "Matching elastic templates", in Proceedings of the Royal Society Symposium on Physical and Biological Processing of Images, (Springer Verlag, 1982). [5] David J . Burr, "A Neural Network Digit Recognizer", Proceedings of the IEEE Conference on Systems, Man, and Cybernetics, (Atlanta, 1986) 1621. [6] David J . Burr, "Experiments with a Connectionist Text Reader", to appear in Proceedings of the First International Conference on Neural Networks, (San Diego, 1987). [7] H. H. Chen, Y . C . Lee, G . Z. Sun, H. Y . Lee, Tom Maxwell, and C . Lee Giles, in Neural Networks for Computing, (Snowbird 1986), AIP Conference Proceedings #151, (American Institute of Physics, New York, 1986). [8] John S. Denker, ed., Neural Networks for Computing, (Snowbird 1986), AIP Conference Proceedings #151, (American Institute of Physics, New York, 1986). [9] Michael R, Garey and David S. Johnson, Computers and Intractability— A Guide to the Theory of NP-Completeness, (W. H. Freeman, New York, 1979). [10] P. Gorman and T. H. Sejnowski, article in J P L Document #D-4406, (1987) 224-237. [II] II, P. Graf, W. Hubbard, and L . D. Jackel, to be published. [12] J . Hadamard, Lectures on the Cauchy Problem In Linear Partial Differential Equad'ons, (Yale University Press, 1923). [13] G. E . Hinton and T. J . Sejnowski, "Learning and Relearning in Boltzmann Machines", in Parallel Distributed Processing—Explorations in the Microstructure of Cognition, (MIT Press, 1986). [14] J . II. Holland, K. J . Holyoak, R. E . Nisbett, and P. R. Thagard, Induction, (MIT Press, Cambridge, MA, 1986). [15] J . J . Hopfield and D. W. Tank, Biological Cybernetics, 52 (1985) 141. [16] J . J . Hopfield and D. W. Tank, to appear in Proceedings of the IEEE International Conference on Neural Networks, (San Diego, 1987).
434
921
[17] R . E . Howard, Proceedings of the 13th European Solid State Circuits Conference, (1987). (18] L . D . Jackel, H . P. Graf, and R. E . Howard, "Electronic Neural Network C h i p s " , Applied Optics (1987). [19] Jeffrey W . and R . Rosner, Astrophysics J., 3 1 0 (1986) 473; see also article in Neural Networks for Computing, (Snowbird 1986), A I P Conference Proceedings # 1 5 1 , (American Institute of Physics, New York, 1986). [20] S. Kirkpatrick, C . D . Gelatt, and M . P. Vecchi, "Optimization by Simulated Annealing", Science, 220 (1983) 671. [21] Teuvo Kohonen, Self-Organization and Associative Memory, Verlag, New York, 1984).
(Springer-
[22] A . N . Kolmogorov, "Logical basis for information theory and probability theory", IEEE Transactions on information Theory, I T - 1 4 (1968) 662. [23] Richard P. Lippmann, IEEE
ASSP, 4 (1987).
[24] Christof von der Malsburg, "Self-organizing of orientation sensitive cells in the striate cortex", ifybernetifc, 1 4 , (19873) 85-100. [25] T o m Maxwell, C . Lee Giles, and Y . C . Lee, "Generalization in Neural Networks: T h e Contiguity Problem", to appear in the Proceeding of the IEEE Internationa! Conference on Neural Networks (San Diego, 1987). [26] Carver Mead and Lynn Conway, Introduction to VLSI Systems, (Addison Wesley, Reading, M A , 1980). [27] Marvin Minsky and Seymour Papert, Perceplrons, ( M I T Press, Cambridge, M A , 1969). [28] Edward W . Packel and J . F . T r a u b , "Information-Based Complexity", Nature, 3 2 8 (1987) 29. [29] David Parker, to appear in the Proceedings of Ihe 1987 AAAI Conference. [30] R . W . Prager, T . D . Harrison, and F . Fallside, Computer Speech and Language, 1, (1986) 3. [31] W i l l i a m H . Press, Brian P. Flannery, Saul A . Teukolsky, and W i l l i a m T . Vetterling, Numerical Recipes—The
Art of Scientific Computing, (Cambridge
University Press, 1986). [32] David E . Rumelhart and James B. McClelland, Parallel Distributed Processings Explorations in the Microstructure of Cognition, ( M I T Press, 1986). [33] D . E . Rumelhart, G . E . Hinton, and R . J . Williams, "Learning Internal Representations by Error Propagation" in Parallel Distributed Processing— Explorations in the Microstructure of Cognition, ( M I T Press, 1986).
435
922
[34] S. Ryckebusch, S. A. Solla, J . S. Denker, B. S. Wittner, D. B. Schwartz, to be published.
[35] R. Scalettar and A. Zee, I T P preprint (1987). [36] Terry J . Sejnowski, P. K. Kienker, and G. E . Hinton, "Learning symmetry groups with hidden units: beyond the perceptron", Pbysica, D22 (1986). [37] Terry J . Sejnowski and C . R. Rosenberg, "NETtalk: a parallel network that learns to read aloud," Johns Hopkins technical report J H U / EECS-86/01 (1986). [38] A. N. Tikhonov and V. Y . Arsenin, Solutions of El-Posed Problems, (Wiley, New York, 1977). [39] Bernard Widrow and Samuel D. Stearns, Adaptive Signal Processing, (Prentice-Hall, Englewood Cliffs, NJ, 1985). [40] B. S. Wittner, J . S. Denker, et al., to be published. [41] Yann LeCun, private communication, [42] Terry J . Sejnowski, private communication. [43] Geoffrey Hinton, private communication. [44] George Furnas, private communication.
436
This article is based on a paper presented at the Conference:
"From Statistical Physics to Statistical Inference and Back". Cargese 1992
CONFRONTING NEURAL NETWORK AND HUMAN BEHAVIOR IN A Q U A S I R E G U L A R E N V I R O N M E N T
M.A.Virasoro
Dipartimenio di Fisica Universila di Roma "La
Sapienza".
A B S T R A C T . W e compare neural network behaviour with h u m a n behaviour in a quasiregular environment. W e find both interesting simlie.tities and essential differences. T h e most striking difference is that a generic neural network does not use the simplest rule as a default rule. O u t analysis applies to large classes of networks.
1. I n t r o d u c t i o n Modelling the brain as a neural network conveys on us the importance of estimating and explaining the amount of organisation the brain has. Organisation is a resource that is not for free in a biological system. It is well known and widely quoted that an important event in the history of the species has been the rapid growth of the volume of the Homo Sapiens brain. The sheer growth of an organ poses no burden. The growth of the giraffe neck is a text-book example on how selection works and conceals no mystery. On the other hand the brain's case is different because the additional volume comes with new special organisation: the Homo Sapiens brain is not simply a duplication of the Homo Habilis brain. Its organisation and specialisation are rather complex and had to be reached in a relatively short time. Could there be a problem here? When model building is done from a pure cognitivist perspective, and therefore at the functional level, any constraint indicating that certain resources are limited remains hidden. Models may then become infinitely complicated. We believe this is happening and will discuss one example in the following section. O n the other hand as soon as we begin to use neural networks we are forced to reconsider how much information there is in a particular configuration and how this information can be transfered into an amorphous system. The theory of optimisation suggests that this transfer will encounter obstacles. I f we want a system with JV degrees of freedom to reach a specific optimal configuration one generically needs exp(JV) steps. 437
I f we consider t h a t the h u m a n b r a i n h a s 1 0
1 1
neurons and 1 0
1 4
s y n a p s e s it
seems obvious t h a t n a t u r a l e v o l u t i o n h a s p r o d u c e d its present c o n f i g u r a t i o n i n a m u c h s m a l l e r n u m b e r of t r i a l s . T h e simplest e x p l a n a t i o n c o u l d be to a s s u m e t h a t m a n y different configurations of n e u r o n s a n d s y n a p s e s p e r f o r m s i m i l a r l y . M o r e precisely, we suggest t h a t a " t y p i c a l " configuration, one t h a t c a n be the result of a r a n d o m choice, a l w a y s falls near a p e r f o r m i n g one, or, i n a m o r e p r o v o c a t i v e way, t h a t our b r a i n is not very different f r o m a t o t a l l y r a n d o m one. A s i m i l a r point of view h a s been a d v o c a t e d b y S. K a u f f m a n c o n c e r n i n g biological o r g a n i z a t i o n i n g e n e r a l ( l ) . F o r c o m p l e t e n e s s we s h o u l d also r e m e m b e r the r e l a t e d a g g r e g a t e f i e l d h y p o t h e s i s p r o p o s e d b y K . S . Lashley. T h i s scenario suggests t h a t one s h o u l d s t a r t b u i l d i n g a m o d e l f r o m t o t a l r a n d o m n e s s . T h i s m e a n s t h a t one s t a r t s w i t h a n e n s e m b l e of configurations w i t h a flat p r o b a b i l i t y m e a s u r e denned o n it. O n e t h e n i n t r o d u c e s m o n i t o r e d i n f o r m a t i o n (for i n s t a n c e t h a t it c o r r e c t l y p e r f o r m s a p a r t i c u l a r t a s k ) . T h i s i n f o r m a t i o n will concentrate the p r o b a b i l i t y d i s t r i b u t i o n i n a s m a l l e r subset of configurations thereby forcing the entropy to decrease. A t the e n d the final p r o b a b i l i t y d i s t r i b u t i o n m a y be u s e d to m a k e e s t i m a t e s of various q u a n t i t i e s . T h i s is a n e x a m p l e of a s t a t i s t i c a l inference, m a x i m u m l i k e l i h o o d , m a x i m u m entropy or B a y e s i a n one. I n every case one h a s to s t a r t b y denning the p r i o r p r o b a b i l i t y d i s t r i b u t i o n o n the space of n e u r o n s / s y n a p s e s
configura-
tions to m a k e sense of the r a n d o m choice. I n our case the s y s t e m is large a n d then the l a w s of large n u m b e r s i m p l y that a large r a n g e of i n i t i a l d i s t r i b u t i o n w i l l l e a d to b a s i c a l l y e q u a l final e s t i m a t e s . S t a t i s t i c a l inference is simple a n d powerful. H o w e v e r it is i m p o r t a n t to realize t h a t it is j u s t a recipe. I f one finds t h a t the p r e d i c t i o n s do not agree w i t h the d a t a it s i m p l y m e a n s t h a t there is s o m e t h i n g b i a s i n g t h e p r i o r dist r i b u t i o n a n d / o r a d d i t i o n a l c o n s t r a i n t s r e s t r i c t i n g the posterior d i s t r i b u t i o n . One t h e n h a s to discover these biases, take t h e m i n t o account a n d repeat the p r o c e d u r e . O n the other h a n d it is a m e t h o d specifically t a i l o r e d to keep a detailed account of the a m o u n t of i n f o r m a t i o n c o n t a i n e d i n the s y s t e m . C o n s i s t e n t l y we prefer the m o d e l w i t h m i n i m u m i n f o r m a t i o n content a n d m a x i m u m disorder t h a t is capable of p e r f o r m i n g as the r e a l t h i n g .
2. T h e q u a s i r e g u l a r e n v i r o n m e n t W e w a n t to discuss the a d a p t a t i o n of a n e u r a l n e t w o r k i n a q u a s i r e g u l a r e n v i r o n m e n t as denned, for i n s t a n c e , i n ( 2 ) . Q u a s i r e g u l a r i t y m e a n s t h a t the m a j o r i t y of e x t e r n a l s t i m u l i presented to the s y s t e m r e q u i r e the a p p l i c a t i o n of a s i m p l e r u l e b u t i n some e x c e p t i o n a l cases the response h a s to be different a n d m o r e c o m p l e x . E v e n for these l a t t e r cases a r u l e m a y exist b u t it m u s t be m o r e c o m p l e x (for i n s t a n c e it m u s t require a longer d e s c r i p t i o n i n a u n i v e r s a l computer). T h i s e n v i r o n m e n t is m u c h m o r e c o m m o n t h a n one i n w h i c h a l l s t i m u l i follow the s a m e r u l e w h e t h e r simple or c o m p l e x . H o w e v e r there h a v e b e e n few s y s t e m a t i c a n a l y s i s of n e u r a l n e t w o r k s l e a r n i n g a n d generalising i n s u c h s i t u -
438
ations. We find it particularly suggestive that even man-made environments, like for instance natural languages, are quasiregular. In fact the example we will now concentrate on concerns the ability to read english. There is a large amount of data gathered on reading coming both from experiments with normal individuals and studies of cases of acquired dyslexia due to brain damage(3). Fig 1, from ref. (4), is an example of a model formulated at the functional level in terms of information processing modules. What is the meaning of the boxes in this figure? When a certain function can be analyzed, from a purely logical point of view, in a series of more elementary operations performed in series or in parallel, these operations are represented by independent boxes i f there is evidence that they can be affected, modulated or impaired independently of each other. Notice that this definition is at a pure phenomenological level. It does not make any assumptions on how these boxes are realised in the brain. Some examples may help to understand how such a model is generated from the data (for a deeper discussion see (4). In the model reading may go through route A, the semantic route, where the meaning of the word is identified before pronunciation, or route B, the phonological route, where certain graphemephoneme pronunciation rules are applied. They are distinguished because there are patients whose pattern of dysfunction looks as if one of the routes is much more damaged than the other one: Phonological dyslexia, where Early
visual
processing
L e t t e r Form Analysis
Filter Control
Letter Identification Visual,Explicit
visual word-Form Analysis
Phonological Correspondence Processing (look-up r o u t i n e s )
Semantic System
Output word-form system (Phonological data base)
Pre-articulatory assembly and ordering
Fig. 1. (From Shallice and McCarthy (1985)) Model of the word-form multiple levels approach to the reading process. A = the semantic "route", B = the phonological "route" and C = the compensatory strategy route. B l , B2... represent different levels of operation of the multiple level system. The elliptic shape represents the operation of the hypothetical reversed spelling procedure.
the phonological route is damaged, is characterized by the fact that words are much better read than non-words (meaningless neologisms); Semantic dyslexia where reading is preserved is characterized by lack of comprehension of the text meaning. Route C is another possibility used when both A and B are destroyed (for instance because the visual word-form analyzer has been damaged). There is no evidence that normal, healthy readers use this route: when it is used, patients identify painfully every letter, then through some kind of reverse spelling identify the word before being able to pronounce it. The proliferation of boxes in models of this kind is the symptom that at this purely functional level we are not sensitive to limitations of resources and to the origin of organization in the brain. The program presented in the introduction suggests checking whether some of the features in F i g . 1 emerge spontaneously from a disordered initial state. We will concentrate on a small part of the model: the multiple parallel routing (the Bi, B ,... arrows). Phonological pronunciation is an example of a quasiregular task. Words pronunciation differ in degree of regularity. Thus boost, crow, vase are ordered in terms of decreasing regularity(5, Table 5.2 page 93). It is known that dyslexic patients that use the phonological route make more mistakes when the pronunciation is more irregular. More precisely when asked to read a list of 20 irregular words and 20 regular ones, matched so that other factors like frequency, length etc are statistically equal, the patient will make more mistakes on the irregular list. 2
The model in F i g 1 explains these data assuming the multiplicity of routes B\, B2,.... Furthermore it assumes that the Bi route is more robust that the Bit route if i < k. Following our method we will show that the fine structure in the B route is unnecessary. Any generic system that has learned responses of different degree of regularity, when damaged will make errors correlated with the regularity very similar to the patients data. But we will also show that the network does not learn to apply the simplest rule as a default-rule. As a consequence there are many aspects in human cognition that these models do not reproduce. 3. T h e g e n e r a l i z a t i o n i n a q u a s i r e g u l a r e n v i r o n m e n t A Neural Network is an adaptive system. Part of its architecture remains fixed while a large number of parameters (synaptic weights) are adjusted during the learning process. In many cases the N N defines a mapping from an input space to an output space. If the synaptic weights are adjusted so that the network reproduces a prescribed output in P examples, in the next example it will extrapolate in a well defined way. It is obvious that in so doing the network is using more information than the one contained in the examples: mathematically there is no unique way to derive the P + 1 example from the first P ones. Therefore the network is biased. Paradoxically this fact explains why these models have aroused so much interest in Psychology. It was one of the leit-motives of the so called Cognitive revolution the realization that human cognition had to be innately 440
biased. It is as if human beings and machines alike had an inner definition of r e g u l a r i t y of a mapping and that in front of specific instances they would assume the most regular hypothesis. Here we would like to push this comparison further to see whether these two built-in biases are similar or not. To focus sharply on one issue we assume, as in most of the literature, that the Neural Network built-in information is contained in the fixed architecture and on the variability range of its parameters and not in the learning process. Specifically we will explore the case in which the N N predicts the output by randomly selecting a set of parameter values that performs well on the training examples. This type of learning has been extensively studied in the literature under different names: statistical learning (6; 7; 8), minimum biased learning (9; 10), Boltzmann-Gibbs algorithm (11) etc. The precise meaning of random selection assumes the definition of a prior probability measure in parameter space previous to the onset of the learning process. T h i s measure is then restricted when the system learns following a classical Bayesian reasoning. The prior distribution is the one responsible for the bias. From this point of view, a mapping is more or less regular if the prior probability measure of the set of parameters such that the network implements the mapping is larger or smaller (12). We are going to show that this precise characterization of the built-in bias explains the correlation between the regularity factor and the probability of error but is not sufficient to explain other important aspects of human behavior. Rule inference by a N N is demonstrated through its ability to generalize. I n a quasiregular environment the system has to learn different input-output mappings with different degrees of regularity. One must then take into account the frequency of occurrence of each class. Quite often, the more complex the behavior the smaller the frequency, but both factors, regularity and frequency, are conceptually very different and though correlated are largely independent . We will show that during learning NNs trade their built-in definition of regularity for frequency information. I n a sense the regularity of a mapping becomes mixed with the frequency factor. This fact is at the origin of pecularities in N N behavior in which they seem to differ from humans: the inability to eliminate confusing information (9); the fact that rule behavior does not become default-rule behavior; the fact that "regularization" errors do not predominate after a lesion(13). 1
If we call djj.p(W) the probability distribution in synaptic weights space after P examples had been presented, Bayes reasoning implies that after
!
R e g u l a r i t y is directly connected to the algorithmic complexity of the mapping. W h e n there is coexistence of mappings with different regularity one can distinguish two different steps in the calculation of the reponse. F i r s t a decision is made on whether the stimulus requires the more or less complex response. Once this is done one calculates the appropriate reponse. T h e complexity of the first step may in many cases be related with the frequency (in fact, the smaller the frequency, the larger is the description necessary). O n the contrary the complexity of the response c o m p u t a t i o n is completely independent
441
p r e s e n t a t i o n of the P + 1 e x a m p l e :
dHP+l(W) = -—dppWe-WW)
(1)
w h e r e $ is zero (one) if the c o m p u t e d o u t p u t of the n e t w o r k w i t h weights W
o n p r e s e n t a t i o n of the i n p u t £ is equal (not e q u a l ) to a. 0 is a p a r a m e t e r
t h a t controls how m u c h the p r o b a b i l i t y is c o n t r a c t e d b y the n e w e x a m p l e s . When 0
—* oo the e x p o n e n t i a l factor k i l l s the subsets of W
t h a t do not
p e r f o r m correctly. A finite 0 o n the other h a n d m e a n s t h a t the p r o b a b i l i t y o n s u c h a subset is r e d u c e d by a factor
0
e~ . Yp+i is a n o r m a l i z a t i o n factor.
I t depends on
a c c o r d i n g to the d i s t r i b u t i o n
dfip(W) the p r o b a b i l i t y of g e n e r a l i z a t i o n is defined as the p r o b a b i l i t y t h a t
P + 1 e x a m p l e I n t e r m s of Yp+i
the n e t w o r k reponds correctly o n the
P r o b a b i l i t y .jof .correct . g e n e r a l i z a t i o n =
_^—
(2)
T h e i n f o r m a t i o n about the r e g u l a r i t y of a h y p o t h e s i s is solely c o n t a i n e d i n
Yp+i value. I t depends on the i n p u t vector. T h e l a t t e r m a y belong to different r e g u l a r i t y classes a n d it is chosen a c c o r d i n g to a p r o b a b i l i t y d i s t r i b u t i o n inside e a c h class. Reflecting t h i s fact the
Yp+t b e c o m e different r a n d o m
variables for each r e g u l a r i t y class. W e define:
Z
P
f[Y
=
(3)
i
P r o d u c t s of r a n d o m variables are better h a n d l e d t h r o u g h their l o g a r i t h m s :
F =h (Z ) P
1
P
= ( ^ ^ { Y
l
^
(4)
where the b r a c k e t s s t a n d for the average over the r a n d o m p a t t e r n s . W e now m a k e the c r u c i a l a s s u m p t i o n t h a t AF becomes independent
= F. P
(5)
- F
P
of P for large P a n d fixed P' — P.
T h i s is t r u e i n
m o d e l s t h a t have been a n a l y z e d in detail a n d in their context is c o n n e c t e d w i t h the existence of the t h e r m o d y n a m i c a l l i m i t (we are not
considering
discrete weight s p a c e ) . W e t h e n a s s u m e that the P' — P e x a m p l e s c a n be classified i n K
different
classes a c c o r d i n g to their regularity. T h e i n d e x a — 1,..,K i n d i c a t e s these classes a n d / „ their frequencies of o c c u r r e n c e . T h e n :
^ p = tf.(^) l
0=1
442
(6)
F r o m a thermodynamic point of view -0AF is the (quenched) averaged free energy variation due to the P' - P examples. I n the limit 8 -* co it is the entropy decrease. ^ l n y f ° ' ^ can be interpreted as the corresponding function for class a examples. The posterior probability distribution is such that F is maximal. This fact immediately implies that AF^ will adjust during learning. A numerical example clarify this statement. Let us assume that before learning A f W / A F ^ = 0.001 and f /fi = 0.5 so that the two corresponding terms in the sum are in the ratio 0.002. Behaviour (1) is more regular and more frequent but the regularity ratio is much larger than the frequency one. Maximizing AF implies an equation between the two derivatives multiplied by their frequencies. The two terms must therefore be of the same order. It will be an unnatural accident that the functions and the derivatives are of a very different order. Therefore we expect: 2
~ 0.5
AF^/AFW 1
(7) 2
which means that A i K ) has to grow and A F ' ' has to diminish. After learning a certain number of examples the N N does not see the regular class as regular as at the beginning. This argument has been carefully analysed for the perception (10) . I n this case one can accomodate only two classes and a single binary output. Furthermore:
Y
W
= l + e~
0
(8)
because every input pattern is classified in one of the two classes. The variation in free energy specific to each class is again a random variable with respect to the input vector choice. Its distribution can be explicitely calculated using Replica methods (10; 11; 14) or the Cavity approach (13). It can be expressed parametrically through:
where 2 is a Gaussian variable with variance q that reflects the random choice in the input patterns.The average over z is the average over the input patterns.The function H(u) -
/
e"
l^dt. M,q are (order) parameters in
—oo
the posterior probability distribution. They are therefore determined maximizing the total free energy. I n particular M is derived maximizing: (hAF^
+ f AFW) 2
(10)
For small q the fluctuations with the input patterns can be neglected. Formulae simplify enormously and we obtain:
443
This result confirms the expectations derived from the more general argument. For general values of q the equations are more complicated but the conclusions are similar. What should, we conclude from these arguments? First of all it is clear that the more regular response continues to be more robust. A damage to the model will in general mean that each configuration can be displaced randomly up to a fixed distance. The probability distribution, as in all diffusion processes will becomes flatter. Information will get lost and entropy will increase but the state so obtained will be a continuous deformation of the undamaged one. For small damages the information lost on a word cannot be worst than if such word had never been learned. Therefore the percentage of correct answers will be bound from below by the generalization probability. As the latter is larger for more regular words the differential in the probability of error follows. Interestingly enough there is another feature in patients errors that is not reproduced in models (15; 13).When patients make a mistake it is frequently a r e g u l a r i s a t i o n , that is they pronounce an irregular word regularly. The models behave differently. The reason is that the information about the irregular character of the word is not totally lost and therefore the network does not act in front of it as in front of a new word. The generalization that in other cases would have favored regular prononciation is here biased by the original information (the damage is supposed to be small) The crucial difference between human and neural network behavior is that the network is trying to use all the information it has even if deteriorated. Our strategy seems different, if the information has been deteriorated, we can disregard it and use exclusively a simpler default behavior. Similar divergent behavior occurs at a critical moment during learning when the amount of information gathered on the regular cases is sufficient to infer the simpler rule while at the same time the number of irregular examples gives only a partial incomplete picture of the more complex response. Humans may then apply the simpler rule to a l l new cases. This is what we mean by saying that the rule becomes the default rule. This is a type of behavior that is not reproduced in models. Again our explanation is that models cannot judge the quality of their own information. In this context, an extremely interesting evidence concerns the so called o v e r r e g u l a r i z a t i o n behavior. It refers to the fact that children that are learning to speak English pass through a period in which previously learned irregular forms of the past tense of verbs are disregarded when the regular form has been apprehended (16). N N models will not make this type of mistakes. It should be clear that there is no way in which we could accuse N N of a wrong strategy. First of all the learning procedure is totally consistent with Bayes logic. Furthermore the NN strategy is probably the best one to infer eventually the more complex behavior. The real question (as stressed elsewhere in this volume by Toulouse) is whether the way humans infer a rule is different. 444
On this point our conclusions are rather stringent. I n many cases, when we learn a rule we deal with exceptions in a completely different way from the way N N deal with them. We apply the rule as a default rule. We memorize the exceptions and do not try to generalize them. I f we do not know the example we apply the rule. If the exceptions are not generalizable (for instance, they are really random) this strategy guarantees the smaller generalization error but the situation is more subtle if exceptions are really instances of a higher complexity rule. T h e possibility of default rule behavior implies that at any moment we have the information on the simplicity of the rule, more specifically the knowledge that we can generalize it. I n N N it is generally assumed that this information is contained in the volume of a subset of parameter space. We have shown in this paper that this information gets mixed with frequency information when the N N learns the exceptions. We are not saying that default rule behavior is universal in human cognition. I n fact experts in the field (19) have pointed out that there are "Classical Categories" and "Family Ressemblence Categories". Only the former can lead to default classification. These authors argued on different grounds that N N could not learn Classical Categories. Their essential argument is that they are not denned through associations. Although our conclusions are similar we differ in two points. First of all, we do not believe that N N can only deal with associations. Second, our point concerns more the way the innate bias is incorporated rather than the general ideas of N N modeling. It has been claimed in the literature (17; 18) that temperature is useful to ehminate noise in the input. In fact, when 0 ^ oo , the N N makes errors in the training set. It makes relatively more errors on exceptions than on regular examples and therefore goes a bit in the right direction. I n fact, as can be seen in our formulas, the trade of regularity information for frequency information is done at a slower rate. But this fact can not be a strategy because eventually one should learn all examples without error. T h a t means that the temperature has to be modulated. Its control would again require information that is not in the model as it is. T h e easiest way to modify this state of affairs, although not the cheapest, would be to include a second N N that observes the first one, in particular its relative success in generalization. I n the language of Game theory we need a system such that: i)it can infer the rule; ii)it knows it can infer the rule; iii)it knows it cannot generalize the exceptions.
A ck n o w l e d g e m e n t s It is a pleasure to thank the organisers of this workshop for the nice atmosphere and the excellent choice of subject.
446
References [1] KaufFman S. A. "The Origins of Order: Self-Organization and Selection in Evolution" Oxford University Press. Oxford 1992. [2] Seidenberg M.S. and McClelland J . S . A distributed developmental model of visual word recognition and pronunciation; acquisition, skilled performance and dyslexia. In From Reading to Neurons: Towards Theory and Methods for Research on Developmental Dyslexia. M I T Press/ Bradford Books, Cambridge Mass 1988. [3] From Reading to Neurons: Towards Theory and Methods for Research on Developmental Dyslexia. E d . Galaburda. M I T Press/ Bradford Books, Cambridge Mass 1988. [4] Shallice, T . and R. McCarthy, Phonological reading: from patterns of impairment to possible procedures. In Surface Dyslexia: neuropsychological and cognitive studies of phonological reading (eds. K . Patterson, J . C . Marshall and M. Coltheart) Earlbaum, London 1985. [5] Shallice T . (1988) From Neuropsychology to Mental Structure. Cambridge University Press, Cambridge. [6] Gardner E . The phase space of interactions in Neural Net works, Journal of Physics A: Math. Gen.,21A (1988) 257 [7] Denker J . , Schwartz D-, Wittner B . , Solla S., Hopfield J . J . , Howard R. and Jackel L . Complex Systems,1, (1987)877. [8] Sompolinsky H . , N. Tishby and H. S. Seung Phys Rev. Ie(t.,vol 65, 1683 (1990) [9] Del Giudice P., S.Franz, and M.A.Virasoio J. Phys. ("Paris),vol 121 (1989) [10] Virasoro M.A., Analysis of the effects of lesions on a Perceptron. Journal of Physics A; Math. Gen.,22 (1989) 2227- 2232 11 Opper M. and D. Haussler. Phys Rev. Zett.,66, 2677, (1991) 12 Carnevale P. and Patarnello S. Europhysics Letters'! , (1987)1199 13 Virasoro M.A. On studying the Brain as an Adaptive Disordered System. 97-121 in "Nonlinear Phenomena in Fluids Solids and other complex systems" ed. by P.Cordero and B. Nachtergaele. North Holland. Amsterdam (1991). [14] Opper M. and D. Haussler. in Proc on Computational Learning Theory, "Colt 91", Santa Cruz, C a . [15] Patterson K . , Seidenberg M.S. and McClelland J.S. Connections and Disconnections: Acquired Dyslexia in a Computational Model of Reading Processes. In Parallel Distributed Processing: Implications for Psychology and Neurobiology R . G . M . Morris (ed) Oxford University Press 1989. [16] Pinker S and A. Prince 1988 On language and connectionism; Analysis of a Parallel Distributed Processing model of language acquisition, Cognition, 28,73-193 17 Hansel D. and H. Sompolinsky, Europhys. left.,vol 11, 687 (1990) 18 Gyorgy, G . Phys. Review Lett, 64,2957 (1990) 19 Pinker S. and A. Prince 1989. The nature of Human Concepts: Insight from an Unusual Source. M I T preprint.
446
Repnnlc4 wilh permissions from Physical Review A. Vol 45. No B, 6056-6091, 1992 © 1992 The American Physical Society P P r
Statistical mechanics of learning from
examples
H. S, Seung and H. Sompolinsky Racah
Institute
oj Physics
and AT&T
ond
Bell
Center
for
Laboratories,
Neural
Computation,
600 Mountain
Avenue,
Hebrew Murray
University,
Jerusalem
Hill, New Jersey
91904,
Israel
07974
N. Tiahby* AT&T
Bell
Laboratories,
600 Mountain
Avenue,
Murray
Hill, New Jersey
07974
(Received 16 May 1991; revised manuscript received 23 September 1991) Learning from examples; in feedforward neural networks is studied within a statistical-mechanical framework. Training is assumed to be stochastic, leading to a Gibba distribution of networks characterized by a temperature parameter T. Learning of realisable rules as well as of unrealizable rules ia considered. In the latter case, the target rule cannot be perfectly realized by a network of the given architecture. Two useful approximate theories of learning from examples are studied; the high-temperature limit and the annealed approximation. Exact treatment of the quenched disorder generated by the random sampling of the examples leads to the use of the replica theory. Of primary interest is the generalization curve, namely, the average generalization error t versus the number of examples P used for training. The theory implies that, for a reduction in c that remains finite in the large-N limit, P should generally scale as oN, where N is the number of independently adjustable weights in the network. We show that for smooth networks, Le,, those with continuously varying weights and smooth transfer functions, the generalization curve asymptotically obeys an inverse power law In contrast, for nonsmoolh networks other behaviors can appear, depending on the nature of the nonlinear! ties as well as the readability of the rule. In particular, a discontinuous learning transition from a state of poor to a slate of perfect generalization can occur in nonsmooth networks learning realizable rules. We illustrate both gradual and continuous learning with a detailed analytical and numerical study of several single-layer perceptron models. Comparing with the exact replica theory of perceptron learning, we find that for realizable rules the high-temperature and annealed theories provide very good approximations to the generalization performance. Assuming this to hold for multilayer networks as well, we propose a classification of possible asymptotic forms of learning curves in general realizable models. For unrealizable rules we find that the above approximations fail in general to predict correctly the shapes of the generalization curves. Another indication of the important role of quenched disorder for unrealizable rules is that the generalization error is not necessarily a monotonically increasing function of temperature. Also, unrealizable rules can possess genuine spin-glass phases indicative of degenerate minima separated by high barriers. PACS number(s): &7,10+e, D2.SQ+s 05.20-y B
9
r
(
I. INTRODUCTION In recent years, many attempts have been made to train layered feedforward neural networks to perform computational tasks, such as speech recognition [1] and generation [2], handwriting recognition [3], and protein structure prediction \4). These networks have also been used as models for neurobiological systems [5, 6], and have been employed as metaphors for cognitive processes such as learning, generalization, and concept formation
Pi Learning in neural networks, as well as in other para¬ metric models [8J, has also attracted considerable theoretical interest. The activity in this area has centered on two issues. The first is the question of representation, or reaiiwbiiity. Given a network of some architecture and size, is there a set of weights that makes the network perform the desired task? The second is the question of learning. Given that such a network exists, can its structure and parameters be found with a reasonable amount of time, computational resources, and training data?
Here we focus on the question of learning. We further restrict our scope to supervised learning from examples, which relies on a training set consisting of examples of the target task. The training algorithm uses the examples to find a set of network weight values that perform the Lask well. The most widely used class of training algorithms works by optimizing a suitable cost function that quantifies the error on the training set. Such learning algorithms have several potential difficulties. The algorithms may become trapped in local minima that are far from optimal. Furthermore, finding good minima may require prohibitively long convergence times. Finally, there is no guarantee that good performance on a training set also leads to good performance on novel inputs. This last issue, the ability of adaptive systems to gentrsliie from a limited number of examples, is the focus of the present work. Understanding the determinants of generalization ability is crucial for devising machine learning strategies, as well as for obtaining insight into learning processes in biological systems.
447
Our study is based on a statistical-mechanical (SM)
6057 formulation of learning in neural-networks. The training procedure is assumed lo be stochastic, leading to a Gibbs distribution of network weights. The performances of the system on the training set as well as on novel inputs are calculated as appropriate thtrmal averages on the Gibbs distribution in weight space and quenched averages on the sampling of examples. These averages provide an accurate account of the typical behavior of large networks.
random systems. Although the AA simplifies enorntously the theoretical analysis of these complex systems, in most interesting cases it is known to be unreliable, sometimes even in its qualitative predictions. The second problem is that no attention has been given to the dependence of performance on system size. In fact, the behavior of large systems may be quite different from that of smallsize ones, and its analysis is more involved.
The currently dominant approach in computational learning theory is based on Valiant's learning model and on the notion of probably almost correct (PAC) learning [9,10]. The main achievements of this approach are general bounds on the probability of error on a novel input for a given size of the training set [11,12], as well as classification of learning problems according to their time complexity [9. 13]. Most of these (sample complexity) combinatorial bounds depend on the specific structure of the model and the complexity of the task through only a single number, known as the Vapnik-Chervonenkis dimension [14-16]. Generally, they are independent of the specific learning algorithm or distribution of examples. The generality of the PAC approach is also its main deficiency, since it is dominated by the worst case, atypical behavior. Our statistical-mechanical approach thus differs considerably from the PAC learning theory in that it can provide precise quantitative predictions for the typical behavior of specific learning models.
In the present study we attempt to characterize the generic behaviors of learning from examples in large layered networks. In particular we investigate the expected rate of improvement of the generalization with an increasing number of examples, denoted by the jcniraliia(isn carve. The PAC theory bounds the generalization curve by an inverse power law. Such a gradual improvement has also been observed in computer experiments of supervised learning [20,29]. In other cases, however, one observes a rather sharp improvement when a critical number of examples is reached [20,28,30].
The SM formalism can also be applied to certain learning models for which few PAC results are yet known. Despite recent works which extend the original PAC framework [12, 17], most PAC theorems apply to realizablt tasks, namely tasks that can be performed perfectly by the network, given enough examples. In many real life problems the target task can only be approximated by the assumed architecture of the network, so the task is unrealizable. In addition, many of the PAC learning results are limited to networks with threshold decision elements, although in many applications analog neurons are used. The SM approach is close in its spirit, though not in its scope and results, to the Bayesian informationtheoretic approach, recently applied also to continuous networks [17,18]. A SM approach to learning from examples was first proposed by Carnevali and Patarnello [19], and further elaborated by Tishby, Levin, and Solla [20,21]. Del Giudice, Franz, and Virasoro, and Hansel and Sompolinsky applied spin-glass theory to study perceptron learning of a classification task [22], Gardner and Derrida [23] and Gyorgyi and Tishby [24,25] have used these methods for studying learning of a perceptron rule. Related models have been studied in Refs. [26,27], However the extent of applicability of results gained from these specific toy models to more general circumstances has remained unknown. Recently an interesting attempt to characterize generic generalization performance has been put forward by Schwartz et al. [28]. This work suffers from two basic deficiencies. First, the analysis relies on an approximation whose validity has not been addressed. In fact this approximation is closely related to the well-known an¬ nealed approximation (A A) in the statistical mechanics of
These seemingly conflicting behaviors have analogies in psychological studies of animal learning. The dichotomy between gradual and sudden learning is at the heart of the Long-standing controversy between the behaviorist [31] and the gestalt [32] approaches to learning in the cognitive sciences. In this debate the underlying assumption has been that a Learning process that is based on incremental modifications of the internal structure of the system can yield only gradual improvements in performance. The sudden appearance of concept understanding was therefore related to preexisting strong biases towards the learned concept, or to mysterious holistic learning mechanisms. In the present study we show that in large systems, a sudden emergence of good generalization ability can arise even within the framework of incremental microscopic training algorithms. We analyze the conditions under which such discontinuous transitions to perfect learning occur. Also, we study the asymptotic forms of learning curves in cases where they are smooth. Other issues addressed in this work include (i) the consequences of the annealed approximation for learning in large networks and the scope of its validity, (ii) the properties of learning unrealizable rules, (iii) the possible emergence of spin-glass phenomena associated with the frustration and randomness induced by the random sampling of examples, (iv) how the non linearities inherent in the network operation affect its performance, and (v] the effect of stochastic training (noise in the learning dynamics) on generalization performance.
448
We address these issues by combining general results from the 5M formulation of learning with detailed analytical and numerical studies of specific models. The specific examples studied here are all of learning in a single-layer perceptron models, which are significantly poorer in computational capabilities than multilayer networks. Even these simple models exhibit nontrivial generalization properties. Indeed, even the realization of random dichotomies in a perception with binary weights is a hard problem both theoretically and computationally (see, e.g., Krauth and Mezard [33] and also [34]) Here we study learning from examples in a perceptron wilh
6058 real-valued weights as well as with binary weights. Some of the results found here for the perceptron models have been recently shown to exist in two-layer models also [35, 36]. Furthermore, a perceptron with strong constraints on the range of values of its weights can be thought of as representing a nonlinearity generated by a multilayer system. In Sec. I I we present two useful approximations to the SM of learning from examples: a high-temperature theory, and the above-mentioned annealed approximation. Several general consequences of these approximations as well as their range of validity are discussed. We then present the full theory, based on the replica method of averaging over quenched disorder, and derive from it some general results. In Sec. I l l we derive an inverse power law for the learning curves in the case of smooth networks, where the training energy is a different!able function of the weights. Learning curves of nonsmooth networks do not have a single universal shape. In order to elucidate the possible behavior of such networks, we study in Sec. IV perceptron learning models where both the target rules and the trained networks are single-layer perceptrons. In Sec. V we focus on specific examples of realizable perceptron rules. We study in detail the case of perceptrons with binary weights, where discontinuous transitions in learning performance occur, i n addition, we investigate the spin-glass phases that exist in these models at low temperatures and small number of examples per weight. The annealed approximation has proved to yield qualitatively correct predictions for most of the properties of the realizable perceptron models. In Sec. V I we show that this is not the case for unrealizable rules. We investigate two models of unrealizable perceptron rules where the architecture of the trained perceptron is not compatible with the target rule. Spin-glass phases are found in the unrealizable models, even at large number of examples per weight. Also the generalization error as a function of temperature may have a minimum at nonzero T, demonstrating the phenomenon of overtraining. Section V I I summarizes the results and their implications. A preliminary report on some of this work appeared previously in Ref. [37],
target function oofS], One way of achieving this is to provide a set of examples consisting of P input-output pairs (S ,o- fS')), with I = \ , . . . , P . We assume that each input S is chosen at random from the entire input space according to some normalized a prion measure denoted d/i(S). The examples can be used to construct a 1
D
Irii;-i
P 1
EtWH^IWiS ),
(2.1)
where the error function e(W; S) is some measure of the deviation of the network's output a~(W; S) from the tar¬ get output £7 (S). The error function should be zero whenever the two agree, and positive everywhere else, A popular choice is the quadratic error function 0
'(WlSI^KWiSI-^S)]
1
(2.2)
Training is usually accomplished by minimizing the energy, for example via gradient descent = -V £(W)
(2.3)
W
The training energy measures the network's performance on a limited set of examples, whereas the ultimate goal is to find a network that performs well on all inputs, not just those in the training set. The performance of a given network W on the whole input space is measured by the generaUzation function. It is defined as the average error of the network over the whole input space, L,e. ?
E
(W) = y ^ ( S M W ; S ) .
(2.4)
We distinguish between learning of realizable rules and mles. Realizable rules are those target functions ffo( ) that can be completely realized by at least one of the networks tn the weight space. Thus in a realizable rule there exists a weight vector W " such that unrealizable
s
(
I I . G E N E R A L THEORY
energy
( W , 5 ) = 0 for all S ,
or, equivalently, *(W*) = 0. An target function for which
(2.5) unrealizable
rule is a
A . Learning from examples We consider a network with M input nodes Si [i = 1 M ) , N synaptic weights Wi ( i = 1 , . . . , N ) , and a single output node a — o-(W;S). The quantities S and W are M - and N-component vectors denoting the input states and the weight states, respectively. For every W , the network defines a map from S to a. Thus the weight space corresponds to a class of functions, constrained by the architecture of the network. Learning can be thought of as a search through weight space to find a network with desired properties. In supervised learning, the weights of the network are tuned so that it approximates as closely as possible a
449
c
m i n
= min<W)>0,
(2.6)
Unrealizable rules occur in two basic situations. In the first, the data available for training are corrupted with noise, making it impossible for the network to reproduce the data exactly, even with a large training set. This case has been considered by several authors, including Refs. [24] and [25j. Here we will not address this case explicitly. A second situation, which will be considered, is when the network architecture is restricted in a manner that does not allow an exact reproduction of the target rule itself.
6059 B. Learning at finite: temperature
cal mechanics may now be applied to calculate thermal averages, i.e., averages with respect to / ( W ) . They will be denoted by {)r- In the thermodynamic limit, such average quantities yield information about the typical performance of a network, governed by the above measure, independent of the initial conditions of the learning dynamics. Even after the thermal average is done, there is still a dependence on the P examples S'. Since the examples are chosen randomly and then fixed, they represent quenched disorder. Thus to explore the typical behavior we must perform a second, quenched average over the distribution of example sets, denoted by (()) = / f l i °V(S'). 3
We consider a stochastic learning dynamics that is a generalization of Eq. (3*3)- The weights evolve according to a relaxational Langevin equation ^
= - V w S ( W ) - V V ( W ) + n(t), W
(2.7)
where T) ia a white noise with variance
<«(0«(O} = 2r«o«p-o. (2.8) We have added also a potential V ( W ) that represents possible constraints on the range of weights. This term depends on the assumptions about the a prion distribution of W and does not depend on the examples. The above dynamics tends to decrease the energy, but occasionally the energy may increase due to the influence of the thermal noise. At T = 0, the noise term drops out, leaving the simple gradient descent equation (2.3). The above equations are appropriate for continuously varying weights. We will also consider weights that are constrained to discrete values. In such cases the analog of (2.7) is a discrete-time Monte Carlo algorithm, similar to that used in simulating Ising systems [38]. In simulated annealing algorithms for optimization problems, thermal noise has been used to prevent trapping in local minima of the energy [39]. The temperature is decreased slowly so that eventually at T si 0 the system settles to a state with energy near the global energy minimum. Although thermal noise could play the same role in the present training dynamics, it may play a more essential role in achieving good learning. Since the ultimate goal Is to achieve good generalisation, reaching the global minimum of the training energy may not be necessary. In fact, in some cases training at fixed finite temperature may be advantageous, as it may prevent the system from overtraining, namely finding an accurate fit to the training data at the expense of good generalization ability. Finally, often there are many nearly degenerate minima of the training error, particularly when the available data set is limited in size. In these cases it is of interest to know the properties of the ensemble of solutions. The stochastic dynamics provides a way of generating a useful measure, namely a Gibbs distribution, over the space of the solutions.' In the present work, we study only long-time properties. As is well known, Lq (2.7) generates at long times a Gibbs probability distribution. In our case it Is
The
average
training
and
generahmtian
errors
are
given by e,(T.P) = p - ^ ( E ( W ) ) ) } , r
(2.11) (2.12)
The free energy F and entropy S of the network are given by F{T,P)
-r((lnZ)),
=
;(T.P) = -(Jjy d , i ( W ) 7 > ( W ) l n P ( W )
(2.13) (2.14)
They are related by the identity F
= Pc,
-
TS
.
(2.15)
Knowing F , the expected training error can be evaluated via
d{0F)
1
(2.16)
and the entropy by b
~
dr
(2.17)
The graphs of c,(T, P ) and c,(T, P) as functions of P wiU be called learning curves. Formally our results will be exact in the thermodynamic limit, i.e., when the size of the network approaches infinity. The relevant scale is the total number of degrees of freedom, namely the total number of (independently determined) synaptic weights N. For the limit N —• oo to be well defined we envisage that the problem at hand as well as the network architecture allow for a uniform scaleup of A . However, our results should provide a good approximation to the behavior of networks with a fixed large size. The correct thermodynamic limit requires that the energy function be extensive, i.e., proportional to A'. The consequences of this requirement can be realized by averaging Eq. (2.1) over the example sets, yielding 1
w
P(W) = Z - ' e - ^ > ,
(2.9)
where the variance of the noise in tbe training procedure now becomes the temperature T = 1/0 of the Gibbs distribution. The normalization factor Z is the partition function Z = J d^(W)exp[-/J£fW)J,
(2.10)
( £ ( W ) ) ) = .P<(W).
and we have incorporated the contribution from V ( W ) into the a prion normalized measure in weight space,
(2.18)
Hence, assuming that e(W) is of order 1, the number of examples should scale as
450
P =
oN,
(2.19)
6060 where the proportionality constant a remains finite as N grows. This scaling guarantees.that both the entropy and the energy are proportional to N. The balance between the two is controlled by the noise parameter T, which remains finite in the thermodynamic limit. A formal derivation of this scaling is given below using the replica method. Using the definitions Eqs. (2.11) and (2.12) and the convexity of the free energy, one can show that
version of the statistical mechanics of learning in the high-r limit. Equation (2.23) can be written as 2
=
0
/
dcexp[-N0af(c)]
(2.24)
,
where the free energy per weight of all networks whose generalization error equals ( is (2.25)
s
/(0 = < - r < 0 (2.20)
(,(T,a)<(,lT, ) a
for all T and a (see Appendix A ) . We will show below that, as the number of examples P increases, the deviations of the energy function from its average form, Eq. (2.18), become increasingly small. This implies that for any fixed temperature, increasing a leads to f, — frnit,,*! — 'mill,
a
—oo.
The function s(<) is the entropy per weight of all the networks with e(W) = c, i.e.. (2.26) In the large-// limit the expected generalization error is simply s( ) given = N~by In j ^ ( W ) * ( e ( W ) - e) l
(
ds
C. High-temperature limit
(2.27)
A simple and interesting limit of the learning theory is that of high temperatures. This limit is defined so that both T and a approach infinity, but their ratio remains constant: 3a — finite ,
ct — oo ,
T — oo .
(2.21)
In this limit E can simply be replaced by its average Eq. (2.18), and the fluctuations S E , coming from the finite sample of randomly chosen examples, can be ignored. To see this we note that S E is of order yfP. The leading contribution to j3F from the term 0&E in Z is proportional to p?(((5E)' )) & i V o f . This is down by a factor of 0 compared to the contribution of the average term, which ia of the order Na0. Thus, in this limit, the equilibrium distribution of weights is given simply by 2
Thus the properties of the system in the high-T limit are determined by the dependence of the entropy on generalization error. From the theoretical point of view, the high-T limit simply characterizes models in terms of an effective energy function *(W) which is often a rather smooth function of W . The smoothness of the effective energy function also implies that the learning process at high temperature is relatively fast. One does not expect to encounter many local minima, although a few large local minima may still remain, as will be seen in some of the models below. Another feature of learning at high temperature is the Lack of a difference between the expected training and generalization errors, i.e,, t — c . From Eq. (2.22) and the definitions Eqs. (2.11) and (2.12) it follows that Ci| = < in the high-T limit. Of course the price that one pays for learning at high temperature is the necessity of a large training set, as a must be at least of order T. B
t
g
1
PofW) ~ Z-
ex [-JV/3tt((W)] , P
(2.22)
where D. The annealed approximation Z
a
= fdptyV)
exp[-W/7c, (W)] . e
(2.23)
The subscript 0 signifies that the high-temperature limit is the zeroth order term of a complete high-temperature expansion, derived in Appendix B. [n the high-r limit, i t is clear from Eq. (2.22) that all thermodynamic quantities, including the average training and generalization errors, are functions only of the effective temperature T/ct. It should be emphasized that the present case is different from most high-temperature limits in statistical mechanics, in which all states become equally likely, regardless of energy Here the simultaneous o — "•- limit guarantees nontrivial behavior, with contributions from both energy and entropy. In particular, as the effective temperature T/a decreases, the network approaches the optimal ("ground state") weight vector W * , which minimizes c(W). This behavior is similar to the T = finite, a — oo limit of (2.58) below.
Another useful approximate method for investigating learning in neural networks is the annealed approximation, or AA for short It consists of replacing the average of the logarithm of Z, Eq (2.13), by the logarithm of the average of Z itself. Thus the annealed approximation for the average free energy F is r
ari
= ln«Z»
Using the convexity of the logarithm function, it can be shown that the annealed free energy is a lower bound for the true quenched value, f« < F
451
(2-29)
Whether this lower bound can actually serve as a good approximation will be examined critically in this work. Using Eqs. (2.10) and (2.1) one obtains (Z))
It is sometimes useful to discuss the microcanonicat
(2.28)
=
jdu{W),
-FG..(W)
(2.30)
6061 this relation could be used in actual applications to estimate the generalization error from the measured training error.
(2.31) The generalization and training trail errors are approximated by
2.
-PG.„(W)
dGan(W) _ P S . . | W I
t.
S i n g l e Boolean
The
annealed
(2.33)
otttpat
approximation in example
(2.32)
ae a
dynamics
apace
The above annealed results only approximate the learning procedure described in Eq. (2.7). However, they can be viewed as the exact theory for a dynamic process where bolh the weights and the examples are updated by a stochastic dynamics, similar to Eq. (2.7), involving the same energy function, i.e..
A particularly simple case is that of an output layer consisting of a single Boolean output unit. In this case c(W;S) = 1 or 0 only, so that G„(W) = -]n[l-(l-e-")c(W)] .
exp W[G„(c) - « C „ ( ( ) ] ,
(2.35)
S
Here £ is a function of both W and S'. This dynamic process leads to a Gibbs probability distribution both in weight space and in input space B
n
G (<) 0
I
] n j
d^(W)Ht
-
f(W)) .
(2.37)
The function JVGo(e) is the logarithm of the density of networks with generalization error c. At finite temperature, it is different from the annealed entropy S S - f l f W & T , which is the logarithm of the density of networks with training error c. However, since t, — t, in the high-temperature limit, jVGo approaches as T —• 00. In the thermodynamic limit (N —» oo), the integral (2.35) is dominated by its saddle point. Thus at any given a and T the value of the average generalization error is given by minimizing the free energy / ( ( ) , where -(if = Go — oGan. This leads to the implicit equation M
SGo l-(l-e-<%
de
(2.38)
which is analogous to the high-T result Eq. (2.27). It is interesting to note that in this case the A A predicts a simple relation between the training and generalization errors. Using Eq. (2.33) above, one obtains
' '
=
i -
(
i - ; - ' v
(
2
3
9
)
where e is the average generalization error given by (2.38). or, equivalent^, s
/ d„(W)
[
exp[-0E(W)}
fld^S')
W
W
,
(2.44)
which is exactly the annealed partition function. From the perspective of Eqs. (2.41) and (2.42) the A A represents the behavior of a system with a distorted measure of the input space. The fact that we will find it to be a good approximation in several nontrivial cases reflects the robustness of the performance of the networks in these cases to moderate distortions of the input measure. The effect of reducing temperature is also clear. The larger 0 is, the larger the distortion of the a prion input measure due to the Gibbs factor. Consequently, one expects that deviations from the AA may be important at low T. 3. Hovr
good
it the annealed
( 2
'
approximation?
First we note that — 0f(W) as 0 - * 0. Thus the AA is valid at high temperatures, since it reduces to the high-T limit described above. At lower temperatures the AA does seems to incorporate some of the effects of quenched disorder, in that c, is generally less than ( , , in accord with Eq. (2.20). This is in contrast to the high-T limit, in which ( , = f,. On the other hand, the results of the AA are in general not exact at finite temperature. To obtain some insight into the quality of the AA at finite temperatures we examine its behavior in the limit of large a. From Eq. (2.34) it follows that in the AA the asymptotic value of the generalization error is lim ( j ( T , c ) = c ( W ' ) ,
=
(2.43)
l
(2.36)
]
S N -
s ,
where =
-h [l-(l-c-''M
w
JWW;S') = Z - V " < i
where G„ (.;) =
(2.42)
^ 1 = _ V , E + „,(!).
(2.34)
Since GM, depends on W only through e(W), which is of order 1, we can write a microcanonical form of the AA, analogous to what was done for the high-T limit in Eqs. (2.24)-(2.27). The annealed partition function ((2)) takes the form [ 2)) = jdt
(2.41)
(2.45)
4 0 )
where W ' minimizes G . In general, this vector is not necessarily the same as the vector W*, which minimizes M
To the extent that the annealed approximation is valid,
452
6062 <(W). Hence there is no guarantee that the A A correctly predicts the value of the optimal generalization error or the values of the optimal weights, except for two special cases. One is the case of a realizable rule for which f ( W * ; S ) = 0 for all inputs S. Clearly the minimum of G „ in Eq. (2.31) then occurs at C „ ( W ) = 0. The second is the case of a network whose output layer consists of a single Boolean output unit, as discussed above. From (2.34) it is evident that the minimum of C „ , in this case, coincides with the minimum of c and hence W = W B
B
9l
With respect to the training error, the AA for unrealizable rules is also inadequate: the correct limit E — c j„ is typically violated, even for the Boolean case, and the limit t , — c, does not hold either. In particular, in the T —* 0 limit the annealed training error approaches (
lim c.tT.a) 1 1 T-o
'
=
m
'
(2.51) The simplicity of the replica formulation lies in the fact that only the number of examples remains as a simple prefactor in Eq. (2.48). All other example dependence has been removed, so that the replicated Kamiltonian Q, depends only on the form of *(W; S) and on the nature of the a prion measure on the input space d/i(S). Equations (2.48) and (2.49) also make explicit the scaling of the problem. Since ( ( W ; S ) is defined to be of order 1, Q, itself is of order 1 times n. Thus, as the integral on the weight space in Eq. (2.48) is nN dimensional, where N is the number of degrees of freedom in weight space, P must scale as N.
P
since annealing both the weights and examples at zero temperature minimizes the training energy with respect to al) variables. Often the right-hand side is zero, so that the AA predicts spuriously (t(T = 0,o) = 0 for all cr.
a n
t.
Replica
theory
and
Ihe high-T
limit
The replica theory provides a simple derivation of the high-T limit described in Sec. I I C . Since Q is an intensive quantity independent of P , the high-T limit can be derived by simply expanding it in powers of 0. The leading terms are
E- The replica method
r
To evaluate the correct behavior at all T one has to evaluate quenched averages such as Eq. (2.13) and its derivatives. Such averages are commonly studied using the replica method [40]. The average free energy is written as - J 3 F = ((InZ)) = l i r n ^ I n f l Z " ;
Pt7.[W"] = N[
o-^cfW)
[2.47)
One first evaluates Eq. (2.47) for integral n and then analytically continues to n = 0. Using Eqs. (2.1) and (2.10) we obtain
(252) where ff(W,W)
W
-NoS,[w"l
The AA can be obtained from Eq. (2.47) by setting n — 1 instead of taking the limit n —» 0. The replicated Hamiltonian £ [ W ] with n s= 1 reduces to the annealed expression G ( W ) , Eq. (2.31).
(2.46)
m i n 1( ( W ; S )
w,s
p
» = / f[ d u ( W ) e x p ( - A ' c , r j [ W ' ] ) ,
-((,W)c(W) .
(2.48)
r
= yd^(S)i(W;S)<(W°;S) (2.53)
(2.49)
Note that g measures the correlations in the errors of two different weights on the same example. The general form of Eq. (2.52) is similar to that of a spin-glass replica Hamiltonian (41, 42]. The first term is the one that survives the high-T limit. It represents the nonrandom part of the training energy- Taking into account only this contribution leaves the different replicas uncoupled, and hence F reduces to its high-T limit
The average generalization error (2J2) can be rewritten using replicas as
(2.54)
where the replicated H ami Hon i an is Sf,[W«] = - l n Jdu(S)exp
E
, ( T , o ) = lim & Z-'
(-0 £
e ( W ; S)J .
,B
w
J «Vi(W)((W)e- < >
= lim / J ] a > ( W > ( W V ' ' " ' and the average training error (2.11) as C
| W
(2.50) '
in which the training energy becomes proportional to the -SF - Infunction, j a><W)e generalization i.e., £ ( W ) — P c ( W ) . As T decreases the second term of Eq. (2.52) becomes important. This term is a coupling between different replicas which originates from the randomness of the examples.
453
6063
2.
Spin
glasses
and
replica
symmetry
breaking
* -f£min,
In some cases, the coupling between replicas produce only minor changes in the learning curves. In others, such terms can lead to the appearance of qualitatively different phases at low temperatures. These phases are conveniently described by the properties of the matrix (9.66) which measures the expected overlap of the weights of two copies of the system. Since the replicated Hamiltonian (2.49) is invariant under permutation of the replica indices, one naively would expect that - q for all Ii jS v. The physical interpretation of q would then be
! = tt-'«(w>.(w> ;
(2.58)
r
Cl—E in,
(
It is known as the Ed wards-Anderson parameter in spinglass theory [40]. The high-temperature phase indeed possesses this replica symmetry. However, as the temperature is lowered, a spontaneous replica symmetry breaking (RSB) can occur, signaling the appearance of a spin-glass phase. In this phase, the expected values of correlations among different replicas do depend on the replica indices. Formally, the spin-glass phase is characterized by a nontrivial dependence of quantities such as <7„,. on the replica indices. Physically, tbe spin-glass phase is marked by the existence of many degenerate ground states of the energy (or free energy) which are well separated in configuration space. The different values of Q represent the distribution of overlaps among pairs of these ground states. This degeneracy is not connected with any simple physical symmetry, but is a result of strong frustration in the system. Furthermore, these degenerate ground states occupy disconnected regions in configuration space that are separated by energy barriers that diverge with A*. Such barriers are important in the context of learning, since they lead to anomalously slow learning dynamics [43-45].
m
a — cx>.
(2.58)
In the following section, a will be used as a control parameter in a saddle-point expansion to calculate the approach to the optimum for smooth networks.
I I I . SMOOTH NETWORKS We define smooth networks to be those with continuously varying weights and error functions *(W;S) that are at least twice differentiable with respect to W in the vicinity of the optimal weight vector W " According to this definition, whether a network is smooth depends on both the smoothness of the weight space and the smoothness of the error function f ( W ; S ) . In a smooth network, neither the output neuron nor the hidden neurons are saturated at the optimal W " . We now use the replica formalism to derive the asymptotic shape of the learning curves in these networks. As stated above, the integrals over the weight space are dominated, as a —• oo, by the optima] weight vector W , which minimizes both Q (in the n - • 0 limit) and <(W). At finite large a, the leading corrections to ( , come from the immediate neighborhood of W " In a smooth network we can expand Q, in powers of r
6W°
= W ? - W t
(3.1)
The linear terms vanish since W " is a minimum of Q . The leading corrections are r
ov
(3.2) where
Al? = W'i " ~ PVii s
(«*)
The matrix l\ is the Hessian of the error function at the optima] weight vector W \ Le,, s
u = j <<^(s)aa(w-,s). iS
jf
(3.4}
The symbol d; denotes 8/8W,. The matrix V;j is
F. The large-cr limit The replica formalism can also be used to investigate the behavior at a large number of examples, i.e., the a —• oo limit. From Eq. (2.48) it is clear that the free energy and tbe training and generalization errors are all weight space integrals that are dominated by the minimum of Q as a —• oo. Denoting this minimum by Vi° — W*, we find
Vii = J d u f S J i V ^ S l V f W - . S ) .
(3.5)
Since NoQ, defines a Gaussian measure in weight space, it is straightforward to calculate the average deviations of the weights from W They are
ri
(2.57) = ^n((W*) + 0(n ). gmir. _ j expt-^ncf W ; S)] This implies that W " minimizes both the generalization error e(W) and Q, in the n —i 0 limit. Hence we conclude that for any fixed temperature the training and generalization errors both approach the optimal value E in — S j W ' j as a —• 0 0 ,
(6W)
= 0,
(3.6)
s
=
m
[ n
(3.7) where we have already taken the n —* 0 limit. Equations (3.6) and (3.7) have a simple meaning in terms of the physical system. Equation (3.6) reads
454
6064 « ( W ) ) ) = { ( ( W ) ) ) - W = 0. o
r
(3.8}
T
<mb = c ( W ) = j d ( S ) e ( W - ; S ) = 0 . u
The diagonal element (in the replica indices) of Eq. (3.7) yields the average correlations
= -^lnv-%+{U-
(3.9)
ij
The first term, which is proportional to T , represents the contribution of the thermal fluctuations about W " . The second term represents the quenched fluctuations due to the random sampling of examples. This interpretation is confirmed by inspecting the off-diagonal element of Eq. (3.7), which is K(W )T(W,)r» -
1
-J^cz-'vi/- ),,
(
This is equivalent to the statement that i ( W * ; S ) — 0 for all input vectors S, because the error function was defined to be non-negative. Since W minimizes f ( W ) , we can also assume c V ( W " ) = 0, as long as W " lies in the interior of the weight space. We have e ( W ; S ) < e|W;S), since the left-hand side is zero, and the right-hand side is non-negative. This implies that ft<(W;S)>0
(3.10)
• j = tn,m + ^ T r [ / C
(3.11)
Substituting Eq. (3.9) one obtains +
(f
+
2 ^ ) i
+
(3.17)
for all S, but also
To evaluate the corrections to the generalization error, we expand Eq. (2.12) in powers of 6 W , yielding
,
(3.16)
0(
j aV(S)cV(W-;S) = c V ( W ' ) = 0
Equations (3.17) and (3.18) together imply that a , f ( W ; S ) = 0 for all S, which in turn implies Y% = 0. Finally, we have the result
o. (3.12)
(3 18)
i
+ 0(o- ).
(3.19)
The same holds for At aero temperature, the coefficient of 1/a vanishes for t and t,. Furthermore, for a smooth network learning a realizable rule, the higher-order terms also vanish at T = 0. This implies that there is a finite value cx for which £j = f[ = 0 for all a > a , at T = 0. An example of such behavior will be presented in Sec. V A below. Such a state we call a state of perfect teaming. It should be noted, however, that a realizable rule with a smooth network is an unrealistic situation. The smoothness requirement, as defined above, implies that the measure of error involves equalities and not inequalities. Therefore to realize a rule would necessitate infinite precision in determining the optimal weights. Unlike the case of discrete problems, learning tasks in smooth networks are generic ally unrealizable. t
The 1/a expansion for the average training error can be evaluated by calculating first the corrections to the average free energy. Substituting Eq (3.2) in Eq. (2.48) yields, after taking the n —• 0 limit
OF = N*0c
mm
+ i[Tr \r,{0U) - 0 Tt VU'']
Using Eq. (2.16) one obtains (3.14) The above results predict an important relationship between the expected training and generalization errors at T — 0. According to Eqs. (3.12) and (3.14) both errors approach the same limit e j with a 1/a power law. The coefficients of 1/a in the two errors are identical in magnitude but different in sign yielding m
|i = - i * ,
dot
dot
• - . » ,
n
T = 0.
(3.15)
This result can be used to estimate the expected generalization error from the measured training error in smooth networks. A special case occurs when the rule to be learned by the smooth network is realizable. This means that there exists a weight vector W within the allowed weight space that has zero generalization error, i.e. h
455
c
e
I V . L E A R N I N G OF A PERCEPTRON RULE A. General formulation The perceptron is a network which sums a single layer of inputs Sj with synaptic weights Wj, and passes the result through a transfer function a
* = '
^ " "
2
E^i^j
2
= 9 (N~" W
• S)
(4.1)
where g(x) is a sigmoidaE function of x. The normalization l / \ / j v in Eq. (41) is included to make the argument of the transfer function be of order unity. Learning is a search through weight space for the perceptron that best approximates a target rule. We assume for simplicity that the network space is restricted to vectors that satisfy the normalization
606S
(4.2) The a priori distribution on the input space is assumed to be Gaussian, N
s>(S) =
Has, ,
(4.3)
where Dx denotes the normalized Gaussian measure Dx
=
- %
(4.4)
- ' ' *
C
overlap with the teacher, i.e., <(W) - t [ R ) . Learning can be visualized very easily since R — cosfl, where 9 is the angle between W and W The generalization function goes to zero as tbe angle between the student and the teacher weight vectors vanishes. Perfect learning corresponds to an overlap R = 1 or 6 = 0. In the following we discuss perceptrons with either linear or Boolean outputs, and weights that are either binary or obey a spherical constraint. For the linear perceptron, the transfer function is g[x) — c. The error function, Eq. (4.6), is in this case a quadratic function in weight space,
We consider only the case where the target rule is another perceptron of the form
"°
(S)= S
W
(^ °
(4.5)
and W ° is a fixed set of N weights W?. We assume that the teacher weights W ° also satisfy the normalization condition (4.2). Training is performed by a stochastic dynamics of the form (2.7) with the training energy function (2.1). For each example the error function is taken to be !
, I
(w,s) = 5 [ (r" w-s) -s(w- ' w°.s)]'
l
f
(4.6)
f
(
X
(4-9)
(410)
in accord with Eq. (4.7) with a linear g . A second output function to be considered is the Boolean output g(x) = sgn(i), which corresponds to the original perceptron model studied by Rosenblatt [46). The Boolean perceptron a- = sgn(W . S) separates the input space in half via a hyperplane perpendicular to the weight vector. The error function, Eq. (4.6), is (up to a factor of 2) S)(W
0
S))
(4.11)
which is 1 when the student and teacher agree, and 0 otherwise The generalization error
i[ ( ^TRi
f D y
(W) = 1-/1,
e(W;S) = e ( - ( W
( W ) = j OSe(W;S) = j D
3
r
Averaging this function over the whole input space, we find
The generalization function is (
(W;S) = ^ [ ( W - W ' ) - S ]
9
rv
+ v R )
_
<(W) - - cos"
(4.12)
(4.7) where R is the overlap of the student network with the teacher network, i.e., R =
—
W
W'
(48)
(see Appendix C). The relationship between (4 6) and (4.7) is plain, since in both cases the arguments of g are Gaussian random variables with unit variance and cross correlation R . It is important to note that in perceptron learning, the generalization function of a network depends only on its
-pf
B. The annealed approximation for perceptron learning The annealed free energy of perceptron learning is shown in Appendix C to be
= G (R) - c.G„(Jt) ,
(4.13)
A
G«(fi) = - l n
GQ(R)
is simply proportional to the angle between the student and teacher weight vectors.
= A>"'
j j Dx
In J
( - | [, (JVI -
Dyexp
dti(Vif)6{R
-
N->Yi
W°)
rt= + yfi)
-g(y)}^
,
(4.14)
(4.15)
.
456
6066 The function NG (R) is the logarithm of the density of networks with overlap R, so we will sometimes refer to it as the "entropy," even though it is not the same as the thermodynamic entropy s — —df/OT. The properties of the system in the large- N limit are obtained by minimizing the free energy / , which yields a
dGg(R) _ OR
dG^R)
~°
(4.IS)
BR
Solving for R one then evaluates the average generalization error via (4.7). Likewise the average training error is evaluated by differentiating G with respect to 0, as in Eq. (2.33). We will consider the case of a spherical constraint and that of an Ising consiraini In the perceptron with a spherical constraint, the a priori measure dfi(W) is uniform on the sphere of radius \J~N, given by the normalization condition, Eq. (4.2). We may write the measure more formally as a n
MW)sTT^*(w.w-w)
(4.17)
i=i v2ire which is normalized to f dVi(W) = L In this case, the fraction of weight space wilh an overlap R is simply the volume of the (N — 2)-dimensional sphere with radius VI ~ H e n c e the entropy GQ(R), Eq. (4.15), is (in the limit of large IV)
tremely simple. The properties of the system can be expressed in terms of a single order parameter, namely, the overlap R. The stochastic fluctuations in the value of R can be neglected in the limit of large iV. Hence the system almost always converges to a unique value of R given by the minimum of the free energy / ( R ) . Depending on the details of the problem, / ( H ) can have more than one local minimum. I f this happens, the equilibrium properties of the system are determined by the unique global minimum of / . The local minima represent metastable states. Starting from a value of R near one of these minima the system is likely to converge rapidly to the local minimum. It will remain in this state for a time that scales exponentially with the network size. Hence for large networks the local minima of / can be considered as stable states.
C. The replica theory of perceptron learning The calculation of Q,, (2.49), for perceptron learning is presented in Appendix D The dependence of Q, on the weights is through the order parameters . Eq. (2.55), and R„ = i w - W
0
,
(4.21)
which measures the overlap of the networks with the teacher. The values of these order parameters are obtained by extremizing Q In general, evaluating the saddle-point equations for Quy and A requires making an ansatz about the symmetry of the order parameters at the saddle point. The simplest ansatz is the replica symmetric (RS) one, r
2
G (S) = ^ l n ( l - R ) ,
(4.18)
0
p
a result that is derived in more detail in Appendix C. The entropy diverges as R —• 1. as the fraction of weight space with overlap R approaches zero. Such a divergence is typical of a continuous weight space. The Ising perceptron corresponds to a network with binary valued weights Wi = ± 1 , or JV
dfi(W) s J l dWi[6(Wi •=i
- 1) + 6{Wi + 1)]
(4.19)
G (R)
=
a
1 + R, 1 + R In — ,
2
,. „„. (4.20)
a result derived in Appendix C. It approaches zero as R —• 1, meaning that there is exactly one state with R = 1. This nondivergent behavior is typical of discrete weight spaces. To conclude, the picture emerging from the AA is ex-
Go^--^(l-g)i-
G
r
= -jDt
RR + j
Dp\nj
J o z l n
Dzexp
The RS free energy of perceptron learning is (see Appendix D) -0f
/
457
0
- «G (q, r
R) , (4.25)
i ~ 9 + yR + W,~ 7
= ^ ( ( I n Z ) ) = G (o, R.g.R)
where
v
v
(4.24)
0
fl-^«(W)ri)W
^(W)exp[W(z '? +
[3 ( 1
(423)
In this case the order parameters of the replica theory have simple meanings in terms of thermal and quenched averages. The order parameter £ is given by Eq, (2,56), and R is the expected value of the overlap with the teacher,
The entropy of Ising networks with an overlap R is given by 1-R, 1-R — 1" —
(4.22)
Q„.. = S„, + < i - V ) s . R„ = R.
W°fl)] ,
2
«)
- »(!/)]
(4.26)
(4.27)
6067 and z is a vector of Gaussian variables {zi}?-, with Di = r j f Dzir The free energy has to be extremized with respect to the order parameters q and JE, and their ''conjugate" counterparts q and i t . Differentiating with respect to q and R yields the saddle-point equations ,=xy
particularly Bimple. Krogh and Hertz [48] have done a complete analysis of the training dynamics for this model. Here we derive the equilibrium properties using the replica theory. Applying Eqs. (4.25)-(4.27), yields - B f = G - aG , with 0
r
(4.28)
DZ(W),.{W),,
G = iA 0
i
+
0 !
-RR-iln(A
+
d)
+
i
^
-
i
,
(4.29) (5.1) The definition of tbe average ( W ) j in these two equations reveals the~ meaning parameters q and R , .R = j Dz {of W the ),-W ° (dllCYf) W exp[W •
(5.2)
W°ft)]
( W ) , = 1— _ / d^(W) exp[W • (zV? + W ° R ) ] (4.30) In this equation, the local field z\/? + W R acting upon W consists of two parts. The first is a Gaussian random field with variance q originating from the random fluctuations of the examples. The second is the bias towards the teacher weights W ° , with an amplitude R . In general, we know from Eq. (2.58) that W must approach the optimal weight vector W " as a —» oo. For a realizable perceptron rule ( W — W ° ) , this means that R —* 1. If W " is unique, the Gibbs distribution in weight space contracts about oo, which means that ] —» 1, The approach to the optimum is reflected in a competition between the two terms of the local field: the strength of the ordering term diverges (R —• oo) and the relative strength of the disorder goes to zero (y?/fi — • 0). One criterion for the validity of the RS ansatz is the local stability of the RS saddle point. Often one finds that the RS solution becomes locally unstable at low temperatures, and hence invalid, fn the phase diagram, the tine at which the instability appears is known as the AlmeidaThoulcss fine {47]- To find the true solution beyond this line, one must break replica symmetry. For systems with discrete-valued degrees of freedom, a simpler diagnostic for RSB is available, based on the fact that such systems must have non-negative entropy. Below the Jtro-enfropj fine, the entropy is negative and hence the RS ansatz must be incorrect. Hence the zeroentropy line provides a lower bound for the temperature at which USB first occurs. Since the zero entropy line is easier to calculate than the Almeida-Thouless line, we will rely on it to estimate the location of the RSB regiou. Since it is generally extremely difficult to find the correct RSB solution, we will consider only the RS solutions. The only exceptions are the models of Sees. V D and V I C below, for which we analyze the first step of RSB. Otherwise, we expect that the RS solution will still serve as a good approximation in the RSB region. 0
V. PERCEPTRON L E A R N I N G OF R E A L I Z A B L E RULES A . Linear- output with continuous weights The case of a perceptron with the quadratic error function Eq. (4.9) defined on a continuous weight space is
The additional order parameter X is the Lagrange multiplier associated with the spherical constraint. Extremizing / with respect to the order parameters and eliminating A yields (5.3) 2
q =
(q + R ) { \ - q )
l + „, * -
a
0
2
(5.4)
:
(5.5)
W-9)' 0-2R+1 [i +
(5.6)
W-OP"
First we consider the simple case of zero temperature. Only those weight vectors with zero training energy are allowed, i.e., those that satisfy ( W - W ° ) • S' = 0 ,
1=1,...,/'.
(5.7)
For P < N these homogeneous linear equations determine only the projection of W — W ° on the subspace spanned by the P random examples S This implies that the subspace of ground states of /:' has a huge degeneracy; it is N - P dimensional. As P —• N this degeneracy shrinks and for P > N there is a unique solution to Eq. (5.7), W = W ° , for almost every random choice of examples. At T — 0 the saddle-point equations reduce to J
For a < 1, the fact that q < 1 reflects the degeneracy of the ground states, according to the definition Eq. (2.56). When cr reaches the critical value = 1,
(5.9)
the degeneracy is finally broken (n — 1), and the training energy possesses a unique minimum W — WQ (R = 1), in agreement with the simple arguments presented above. Thus there is a continuous transition to perfect learning at a = 1. However, this transition does not exist at any finite temperature, because of thermal fluctuations about the global minimum. From the saddle-point equations, one can calculate that the asymptotic generalization curve is given by
458
6068 where
(5.10] in accord with the general result (3.19) for smooth networks learning realizable rules. This means that at any finite temperature, perfect generalization is attained only in the limit of infinite a. The above results are in agreement with the dynamic theory of Krogh and Hertz [48]. It is interesting to compare the above exact results with those of the A A . Evaluating Eq. (4.14), with g ( i ) — x yields =
In [1 + 2 0 ( 1 - * ) ]
(5.18) and H{x) is defined as in (5.54). The saddle-point equations are R =
(5.19)
R ( l - q ) ,
2
q=(q+R )i,l-q)'
(5.20)
,
(5.11) R
=
-l)-'+tf(v)
Adding this to (4.18) yields the annealed free energy of Eq. (4.13), 5
- / ? / ( « ) = i ln( 1 - R ) - | In [1 + 2/3(1 - R ) \ ,
'[( s e
1
+ (5.22)
(5.12) where u is defined in Eq. (5.18) above, and
which is extremized when R
1-
a8
1 + 20(1-
R?
(5.13)
R)
The training error is given by 1 - R a
d0
(5.21)
(5.14)
1 + 23{1-R)
"
The asymptotic behavior of < = 1 — R agrees with the correct result Eq. (5.10). At T = 0, the AA predicts s
2-2cr
(5.15)
in contrast to the true quenched result t* ~ 1 — or, a < 1. Although the value of c is incorrect, the second-order transition to perfect learning at a = 1 is correctly predicted.
The solution of these equations leads to a learning curve with a l / o tail for all T. Note that this power law is not a consequence of the general 1/a law derived in Sec. H i , since a network with a Boolean output is not a smooth network. At T = 0, the asymptotic learning curve is , = 2 (J DtH-'(t/V2^j
i + 0(o-')
0.625
(5 24) (5.25)
?
c
B- Boolean output with continuous weights The Boolean perceptron with continuous weights corresponds to the original perceptron studied in [46,49], At zero temperature, weight vectors with zero training energy satisfy the inequalities (5.18)
(WS')(W°-S')>0.
Since these inequalities do not constrain the weight space as much as the equalities (5.7), this model requires more examples for good generalization than did the preceding linear model. The quenched theory of this model has been previously studied in detail [24], We present below a few of the results for completeness. Since the a priori measure d)i(W) is the same as in the linear-continuous model of Sec. V C above, Go is again given by (5.1). For a Boolean output, C, equals Gr = -2 f " Dy j
fl
Dt In [e" + (1 - e"")/. (u)j , (5,17)
459
Unlike the previous models, there is no transition at finite a to perfect learning at T = 0. In fact, there is no phase transition at any T and a. The AA for this model is [Eq, (4.14)] (5.26)
G„(/l) = - l yielding for the free energy -8f=
| l n ( l - R ' ) + a l n ( l - - "J
- cos~' R ^ (5.27)
Evaluating R by minimizing the free energy, we find that fs =
T3W • 0
+ 0|a
!,
" '
5 28
<' >
This agrees with the correct power law, but does not predict correctly the prefactor; see Eq. (5.24). Finally, it has been recently shown using the replica method that the above 1/a law can be improved by at most a factor of
6069 C- Linear output with discrete weights Imposing binary constraints on the weights of a perceptron with a linear output modifies its learning performance drastically. Let us first consider the zerotemperature behavior. The weight vector must satisfy the same P homogeneous linear equations as before, Eq. £o.7), but now it is constrained to Wi = i l . For almost every continuous input S, the equation ( W - W " ) S = 0 cannot be satisfied unless W = W ° Hence just one example guarantees perfect learning at zero temperature, i.e.,
o = 0.
(5.29)
c
However, this argument does not exclude the existence of local minima of E In this case the main effect of increasing the number of examples is to smooth the energy surface, In order to make the unique global minimum dynamically accessible for large networks. Jn order to investigate these aspects, study of the finite-X version of the problem is extremely useful. 1.
Annealed
approzimation
As the phase diagram of the model at low T is rather complex, it is instructive to first analyze the relatively simple AA, which captures most of the qualitative behavior. Using Eqs. (4.14), (4.15), and (4.20) the annealed free energy is f
-PS=
-^o!n[l + 20(l-fl)} 1-R, l - R In —:: 2 2
l + R ,
1 +R
z— In-
(5.30)
which is extremized when <*3
tanh"'R -
1+ 20(1-R)
(5.31)
For large a, the learning curve t, = l - R s 2 e S
2
a
'
T
FIG, 1. Phase diagram of the linear-discrete model in the (o,T) plane. The bold dashed lines are from the replica symmetric theory, and the solid thin lines from the annealed approximation. The spinodal fines, marked a and b, demarcate the region where there are two metastable phases. The Line running between them is the thermodynamic transition line, at which the two phases have equal free energies. In the R5 phase diagram there is a fourth line, running fiorti the origin to spinodal 6. This is the RS zero-entropy Line of the low-R metastable state.
For a < 0.730 there is only a single local minimum. At Q = 0.730 another local minimum appears at higher R . This is called a spinodal pcinJ of the full spinodal line a in Fig. 1. At a > 0.756 the high-R minimum becomes the global minimum of / . However, the local minimum of lower R still exists for 0.756 < a < 0.781. Only above a = 0.781, a second spinodal point, does the low R minimum vanish, leaving only the high-R one. If the system can be truly equilibrated then the transition to a state with high generalization will occur along the middle line in Fig. 1, which is therefore the thermodynamic transition line. Note that this line starts from
(5.32)
has an exponential tail. This decay is much faster than the inverse power law of the linear or continuous model (5.10), reflecting the severe constraints on the weight space. Also, the prefactor of a in the exponent diverges as T —• 0, signaling the fact that t„ =
0
,
T =
0
(5.33)
for all a > 0, in agreement with Eq. (5.29). Evaluating Eqs. (5.30) and (5.31) one finds that at low T and small a there are two solutions for R . The full AA phase diagram is drawn in Fig. 1 with solid thin lines. The lines marked a and i bound the region where there are two locally stable states. These lines are called sptnodaf fines. The thermodynamic transition line in the middle marks the points where the free energies of the two solutions are the same. All three lines terminate at T — 0.40, Q — 0.87. The appearance of two free-energy minima is demonstrated in Fig. 2 where graphs of the free energy for T = 0.3 and various values of a are displayed.
FIG. 2. Annealed free energy of the linear-discrete model as a function of the overlap R at T — 0.3 and o = 0.730 (spinodal), a = 0.756 (thermodynamic transition), and a 0.78) (spinodal).
460
6070 the origin a = T = 0, implying that for any a as T —• 0 the equilibrium state is the high-A state. Since in this state R — 1 as T — 0 the equilibrium state at T -» 0 is always R = 1, in agreement with Eq. (5.29). However, the line approaches the origin as
the saddle-point equations for q and R , Eqs. (5.36) and (5.37), behave like
(5.34)
Hence the generalization curve has the same exponential tail t j zs 2 e ~ ' as given by the AA in Eq. (5.32). The RS phase diagram is drawn with bold dashed lines in Fig. 1. The similarity of the RS phase boundaries to those of the A A (thin solid lines) is remarkable. Between the spinodal lines marked a and b, there are two locally stable solutions of the saddle-point equations. The thermodynamic transition line runs between the two spinodals. The line running from the origin to spinodal b is the RS zero entropy line of the low R metastable state. At T = 0 the thermodynamic transition line and the spinodal line b intersect the or axis at
1
R ~ tanhRes tanh(a0) , 1
l a
This implies that for a small number of examples even a small noise in the dynamics will generate a transition to the low-K state. For training in large networks the most important transition is, in general, not the thermodynamic one, but rather the spinodal line b. This is because starting from initially random weights (R s 0), the system converges quickly to the nearest metastable state, which is the state with low R as long as such a state exists. The time required to cross the free-energy barrier to the thermodynamic high-R phase is prohibitively large, scaling as i =s §f*™t where A / is the height of the free-energy barrier (per weight) between the two states. It is important to note that, unlike the equilibrium transition Line, the spinodal line terminates at T - 0 at a finite value of a, a = 0.556. This implies that in spite of Eq. (5.29), a finite value of cr is required to learn in a finite time. According to the A A the minimal value of a for learning at T = 0 in finite time, denoted by a , is a = 0.556. e
2. Replica
symmetric
- ^ l . l - l ) q - R R
+
jVumerical simulations
theory
is given by
(5.36) (5.37)
<*0 (5.38) 1 + 0(1-?) " a °-2R+i (5.39) * * * * * [1 + 0 ( 1 - ? ) P For any fixed temperature, R —> 1 and q —• 1 as a —• oo. To investigate this approach to the optimum, we note that for large or R =
J
(5.45)
e
r
tanh'tv^* + £) .
(5.44)
a . = 0.48, f
The replicated Hamiltonian G , Eq. (4.27), which depends on the error function but not on the weight constraints, remains the same as in Eq. (5.2). The resulting sad die-point equations are
a=j
a = 0 ,
respectively. The result a = 0 implies that for any a > 0, the training energy possesses a unique global minimum R = 1. However, the training energy may still possess low-R metastable states. These states vanish above the spinodal point or = 0.48.
j Diln2coBh(vfc + R) .
J Di t a n h ( v ^ + R ) •
(5.43)
T
c
(5.35)
R -
(5.42)
a
The replica symmetric free energy Eq. (4.25) where Grj, Eq. (4.26), is Go =
!
q ~ tanh R a tanh (o0) .
,-~o0 [2(l-ft)-(l-o)] ,
(5-40)
R ~ c B .
(5.41)
Clearly there is a divergence of R, the strength of the ordering term in the local field of Eq. (4.30). At the same time, Vq/R —* 0, so that the relative strength of the disordering term is going to zero. This means that
We have used the Metropolis Monte Carlo algorithm to simulate learning in the linear-discrete perceptron. This algorithm is a standard technique for calculating thermal averages over Gibbs distributions [38). The simulations were performed for multiple samples, i.e., different training sets drawn randomly from a common input distribution, Here the inputs were chosen to be S , — ± 1 at random, i.e., S was drawn randomly from the vertices of the N-dimensional hypercube. This discrete input distribution allowed us to take advantage of the speedup offered by integer arithmetic, yet leads to the same learning curves as the Gaussian input distribution (4.3) in the thermodynamic limit (see Appendix C) The quenched average was performed over these samples, and error bars were calculated from the standard error of measurement of the sample distribution. In the figures of this paper, when a Monte Carlo data point lacks an error bar, it means that the error bar would have been smaller than the symbol used to draw that point. In general, fewer samples were required for larger N, because of self-averaging. In Fig. 3 we present the numerical results as well as the RS theoretical predictions for the training and generalization errors as a function of or. The results of the RS theory are in very good quantitative agreement with Monte Carlo simulations of the model at least for T > 0.2. At T — 0.2 [Fig. 3(a)] the prominent feature is the rapid transition to R as 1 near a a 0.65. This is in agreement with the spinodal point a = 0.66 for this temperature, which can be read from line b in Fig, 1. The location of the thermodynamic transition is shown by a dotted vertical line, and the first spinodal (corresponding to line a
461
6071 a in Fig. 1) is marked by an arrow on the or axis. The roundness of the transition in the simulations is consistent with the expected smearing of the discontinuity in a finite system. At T = 1.0 [Fig. 3(b)] the generalization curve decreases smoothly to zero with an exponential tail. The dependence of ( , on A' is also shown; the training error did not vary appreciably. It is interesting that the finite-size effects are much more noticeable for the simulations at T = 1.0 than at T = 0.2. 4-
Spin-glass
phase
The above theoretical results, which were based on the replica symmetric ansatz described by Eqs. (4.22) and (4.23), are not exact, at least for sufficiently low T and a. This is indicated by the fact that the RS entropy of the metastable state with low R becomes negative in the region marked S G in Fig. 1. Tbe zero entropy line is a lower bound for the temperature below which the metastable
1J)
\.
(o) • G o - (i r j . i
OS
u •• • n
" Tralninfl
0.8
state must be described by a theory with replica symmetry breaking. The interpretation of the RSB is that for small cr the energy surface far away from the optimal overlap R = 1 is rough. On the other hand, we expect that the energy surface in the neighborhood of the optimal state is rather smooth. Hence tbe high-ft phase is probably described correctly by the RS theory, and does not exhibit spin-glass properties. This is substantiated by our calculation of the number of local minima in Sec. V C 5 . Because of RSB, we expect the true location of the spinodal line at low T and a to differ from the RS results, but this difference, and in particular the value of a , may not be large. 0
5.
minima
Local
The above finite-T statistical-mechanical results account for the equilibrium state, as well as for metastable states that are separated by barriers that diverge as N -nx>. However, the system may in addition possess states that are local minima of the energy (2.1), but are separated by barriers that remain finite in the JV —* CO limit. Although these states are washed away by thermal fluctuations at any finite temperature, they may dominate the dynamics at T = 0. Even at finite low T these barriers may be high enough to prevent equilibration in a reasonable amount of time. Following Gardner [52], we calculate an upper bound for the number of local minima as a function of the overlap R . Defining the outer product matrix of the examples by
E UJ
0.* 03
,
(5.46)
0.0 0.0
0-2
t °-S
0.4
OS
1.0
the energy can be written as
a
fv
s=
w T^WiW,
- £T„HW/
+
ft* (5.47)
with Wi — ± 1 . This has the standard form of an Ising Hamiltonian. The condition for a local minimum is hfWi
> 0
(5.48)
for all i , where the local fields hi are defined by N h
< = T ,
T
W
^
i
- WiJ + oWl •
(5.49)
The number of local minima is then OX
OS
1.0
1,5
!0
zs
tf
FIG. 3. Learning curves for the linear-discrete model, (a) Monle Carlo simulations at T = 0.2 for N = 100, averaged over 64 training sets. The generalization and training curves are from the replica symmetric theory, and portray the transition at spinodal b in Fig, 1. The thermodynamic transition is marked by the dotted vertical line, and spinodal a by the arrow on the a axis, (b) Monte Carlo simulations at T = 1 for JV = 20, 50, and 60S.
(5-50)
= T v,{[e(h W,) r
i
For a typical sample of examples we expect the logarithm of the number of local minima lnA^JV.fi) to be extensive and hence self-averaging. We instead perform the simpler annealed average NF(R, j a
=
W(tf{N,R)))
,
(5.51)
which yields an upper bound for the typical number of such states ((In A/(N. R ) ) ) . A saddle-point expansion
462
6072 yields F{R
= 0
=
(5.52}
confirmed by numerical simulations of the model at T = 0. We have found that the system converges rapidly to R = 1 from almost all initial conditions for
arid (for R < 1) a 5 1.0.
(5.55)
D. Boolean output w i t h discrete weights
1 - ft, 1 - f i ; — In—=—
l + R .
1 +
R
(5.53)
—^— I " —=—
Here we have defined H{x)
mj "
Di,
This Boolean-discrete model, first studied by Gardner and Derrida [23], exhibits a first-order transition from a state of poor learning to a state of perfect learning [37,53]. Unlike the linear-discrete perceptron discussed above, this model's transition persists at all temperatures. The occurrence of this remarkable transition can be understood using the high-temperature limit.
(5.54)
and F has to be extremiied with respect to r and y. Figure 4 shows graphs of F(R,a) as a function of R for various values of ot, obtained by solving the saddlepoint equations for r. and y numerically. Wherever F is negative, then there are no local minima in the thermodynamic limit, since the annealed average is an upper bound. As seen in the figure, F is negative near R = 1 for all values of o. In fact, it can be shown that as R — 1, F — - a ( * , l n 2 - J ) = -0.09680. This implies that there ate no local minima in the neighborhood of the optimal W , and the energy surface there is smooth. Note that at R = 1, F = 0, i.e., M = 1 as expected. Figure 4 also shows that as a increases the size of the hole in the number of local minima increases, until a — 2.39, above which F is everywhere negative, so that there are no local minima at all. The implication for learning dynamics is that there exists some a < 2.39 above which learning is fast even for T = 0. This prediction has been
1. The hign-femperaJure limit In the high-T limit the energy of the system is given simply as Nat(R)Hence, using Eq. (4.12) for i{R), the free energy is simply -Pf
=
cos
R
— In —
1 + fl,
l +
R
(5.56) This free energy is shown in Fig. 5 for various values of a/T. In contrast to previous models, the state R = 1 is a local minimum of / for all values of T and a. For small values of a/T the state R = 1 is only a local minimum of / , as can be seen in Fig. 5. The global minimum is given by the solution of the saddle-point equation (5.57)
R = tanh
c
This state of poor learning R < 1 is the equilibrium state for
0£ OA 0 1
1-0 ~ J.JB
*
\ *
\
\ \
1
^ V
aO
0-2
04
06
0M
14 0-0
FIG. fl. Annealed upper bound for the logarithm of the density of local minima in the linear-discrete model as a function of Ii For a = 1.0 there is * small gap around R = 1.0. AM a increases, the density of local minima decreases, until above a = 2,39 there are na local minima at all, except the isolated minimum at R = 1 (marked with a solid dot).
0-2
0.4
0.6
0.0
\A
FIG, 5- High temperature limit of the free energy 0f of the Boolean-discreLe model as a function of the overlap R fox a/T • 1,4) 1-7 (theimodynamic transition), 2.08 (spinodal), and 2.5. The vertical dashed line at R = 1 marks the upper bound of the allowed range of R.
463
6073 T > 0.59c
3.
(5.58)
In this regime the optimal state R - 1 is only metastable. If the initial network has R which is close to 1 the learning dynamics will converge fast to the state R — 1. However starting from a random initial weight vector R 0 the system will not converge to the optimal state. For T/a < 0.59 the equilibrium state is R - 1, although there is still a local minimum, i.e., a solution of Eq. (5.57) with R < 1. Finally for T < 0.48a ,
(5.59)
there is no solution with i i < 1 to Eq. (5.57). In this regime (beyond the spinodal), starting from any initial condition the system converges fast to the optimal state. The collapse of the system to the energy ground state at finite temperature is an unusual phenomenon. The origin of this behavior is the square-root singularity of c(ft) at R':•:= i. This singularity implies that a state characterized by 6ft = 1 - R <£ 1 has an energy which is proportional to
Replica
symmetric
theory
Because of the unusual features of the transition in this model we will analyze the quenched theory in some detail. We first study the replica symmetric theory and then investigate the replica symmetry breaking in this system. The RS free energy is given by combining Ga of the perceptron with discrete weights, Eq. (5.35), with G for a Boolean output, Eq. (5.17), yielding r
-gf=-l(l-q)q-RR
+2«
+ j Dzln2cosh(\/9i + R)
[°°Dy
D ( l n [ - * + (l-e-")(Y(u)] ,
f°°
e
(5.66) where the functioD H(x) is as defined in (5.54). The saddle-point equations are R-J
D i t a n h l v ^ i - r ft) .
(5.67)
(5.60)
EOLNVTR
!
q = jDitanh (v^3
+ ft) ,
(5.68)
This big increase in energy cannot be offset by the gain in entropy, which is proportional to R
6NGa{R)
at N{SR)\n(6R)
.
(5.61)
This effect can be nicely seen using the microcanonical description. According to Eq. (2.27) above, a smooth low-temperature limit exists provided that lim
^
= 00.
(5.62)
On the other hand, Eq. (4.20) implies that in the present case ^ss-clnc, eft
(-0.
(5.63)
Thus the rate of increase in entropy is too small to give rise lo thermal fluctuations below some critical temperature. It is instructive to apply the above argument to the case of states that differ from the ground state by a flip of a single weight. According to Eq. (5.60) the energy of such states is E oc -JN ,
6NG {R) a
ot
]nN
(5.65)
I I should be emphasized, however, that examining the spectrum of the first excitations is not generally sufficient for determining the thermodynamic behavior at any finite T, where the relevant states are those with energy of order NT.
( £-l)-i e
*=*p^ii
+H(v)
Dy
(5.69)
Dt
L [(S^D-'+W (5.70)
where u and v are given by Eqs. (5.18) and (5.23) above. At T = O^the equations simplify somewhat, since q — R and q = ft. For o less than o - 1.245 , t
(5.71)
there are two solutions, one with ft = 1 (perfect generalization), and one with ft < 1 (poor general ization). The ft < 1 saddle point has the lower free energy, and is therefore the equilibrium phase. Upon crossing this critical a, the balance of free energy shifts, and there is a first-order transition to the ft = 1 state. Hence at a — 1.245 there is a discontinuity in the generalization curve, a sudden transition to perfect learning. However, the R < 1 state still remains as a metastable phase until the spinodal point c
a, - 1.492 .
(5.04)
whereas the entropy associated with such an excitation is only
=
(5.72)
At any fixed T, there is the same sequence of thermodynamic transition followed by spinodal transition with increasing or. The RS phase diagram is shown in Fig. 6To the left of the dashed thermodynamic transition line, tbe state of poor generalization ft < 1 is the equdibrium state, and the state of perfect generalization ft = 1 is metastable. In the region between the dashed line and the solid spinodal line, the situation reverses, with ft = 1 becoming the equilibrium state and ft < 1 the metastable state. To the right of the spinodal line, there is no low ft metastable state.
464
6074
n = NPoor Generalization
(5.73)
The parameter qrj represents the average overlap between a pair of two different states a and 6, i.e.,
/
(5.74)
1
= A'- (((W )r(W )r;
9 0
o
1
When the n - . 0 limit is taken, the size m of the step in 5(E) (see Appendix E) must be determined variationally, like qo and qi- Denoting the Gibbs probabilities of the different states by P = e x p ( - 0 F ) , it can be shown [45] that
Partner Generaiijauon
Q
o
(5.75)
FIG. 6. RS and one-step RSB phase diagram for the Boolean-discrete model. To the left of the dashed thermodynamic transition line, the equilibrium state has R < 1 (poor generalization), while the perfect generalization stale (R = 1) is metastable. Between the dashed line and the solid spinodal line the f7 = J state is the equilibrium one, while the R < I state IB metastable. To the right of the spinodal line, there is no R < 1 phase. In the region marked "SI].' the one-step RSB calculation predicts a metastable spin-glass state.
Hence m is the probability of finding two copies of the system in two different states. In tbe present model, a one-step RSB solution exists in the regime marked SG in Fig. 6, below the line T {a). This SG phase is special in that q\ — 1, independent of both T and a. From Eq. (5.73) it follows that each of the different valleys is completely frozen, i.e., there are no fluctuations in W within each valley. Indeed, the entropy of this phase is zero (to linear order in flf), For fixed a, the order parameter qo does not vary with temperature within the SG phase; it is frozen at its value on the phase boundary T ( a ) . The same holds true for (, and R, i.e., 9
a
SoP',a) = 3.
Replica
symmetry
breaking
o r
The line T (o) in the T-a plane, where the RS " p ° generalization" state j R < 1) has zero entropy, is the upper border of the region marked SG in Fig. 6. It should be noted that this line T ( o ) is to the right of the thermodynamic transition line and coincides with it only at T = 0 (at a = 1.24). This implies that the RS theory is invalid for this metastable phase only. To find the correct metastable state (with poor generalization) at low T we must search for saddle-point solutions to the replica theory that break the replica symmetry. fl
r
(5.76)
(T,(o;), ) a
= t,(T (o,),c,)
,
(5.77)
R{T,c,)
=
,
(5.78)
t
and merasrabfe spin-gfass phase
0 0
t (T,c,)
s
R(T„( ), ) a
a
everywhere in the SG phase. The values of fi(a) and tj(or) in this phase are shown in Fig. 7. The parameter m is linear in T, m = T/T (o). Near the transition temperature T^, the degeneracy is very severe so that ui s 1. At T = 0 the degeneracy is broken, and the Gibbs weight concentrates in the SG metastable state with minimal energy, resulting in m = 0. t
To gain a better understanding of this metastable state, we have studied a solution to the rephca meanfield theory with a one-step replica symmetry breaking ansatz. Our study is based on the work of Krauth and Mezard (33] concerning the problem of loading random dichotomies on a perceptron with discrete weights. The formal derivation of the mean-field equations is presented in Appendix E. Here we present the main results and their physical meaning. In the one-step ansatz of replica symmetry breaking, the n x n order parameter matrix Q „ , acquires two offdiagonal values q and gi arranged in a block structure of m x m submatrices (see Appendix E). This block structure reflects the existence of many, almost degenerate, spin-glass (SG) states [45]. These states are valleys in the free-energy surface that are separated by barriers that diverge with the system size [43]. The parameter q, represents the overlap of each state with itself, i.e., it Is the order parameter q, Eq. (2.56), measured within a single valley denoted a, 0
F1G- 7. Training and generalization errors of metastable spin glass phase in the Boo Lean-discrete model, according to the one-step RSB ansa L 2. In this phase, the errors are independent of temperature, and are given by their RS values on the zero-entropy Line T [a) of Fig. 6.
465
g
6075 A similar frozen SG phase exists in the perception model of Krauth and Mezard [33], and was first discovered by Derrida in the random-energy model [54] How¬ ever, unlike these other examples, the SG phase in the present model is only a metastable phase, as evidenced by the nonzero u in Fig. 7 Hence the one-step RSB theory does not alter the RS prediction that the zero temperature thermodynamic transition is at a = 1.245. However, it raises to a* = 1.628
(5.79)
the spinodal line for the vanishing of the metastable SG phase in Fig. 6¬ 4- Numerical
004
0 06
m
simulations
FIG. 9. Graph of o va l/N at T = 0 for the Booleandiscrete model from exhaustive search. Each data point is the average of roughly 10* or 10" random samples of examples. The error bars are the standard error of measurement over the sample distribution, A I east-squares fit to a quadratic function yields the extrapolation o (JV = oo) = 1,30 ± 0.03. The replica symmetric prediction tt = 1,24 is also marked. c
Figure 8 shows learning curves for Monte Carlo simulations with N = 75j averaged over 32 samples. At T = 1, there is quite a good fit if the transition Is taken to occur at the spinodal line. At lower temperatures, the Metropolis algorithm tends to become trapped in local minima, making equilibration difficult. Hence we cannot use it to check the 7" = 0 predictions of the quenched theory. The Metropolis algorithm produces a random walk through weight space that samples the Gibbs distribution. For small system siaes, we do not have to sample the weight space; we can explore it exhaustively. Gardner and Derrida [23] applied this idea to compute a at T = 0, Starting with all 2^ possible student vectors, an example is chosen at random, and all student vectors that "disagree" with the teacher are eliminated. Eventually some number of examples P is reached such that the addition of one more example produces perfect learning. Then a (N) = P / N for this sample. The procedure is then repeated with a different set of examples, so that a can be sample averaged. c
t
t
e
e
We have extended the results of Gardner and Derrida up to the size N = 23. With the discrete inputs Si = ± 1 , only odd N were used, so as to avoid the situation W - S = 0. Figure 9 exhibits a as a function of \/N, with each data point an average of from 10 (large IV) to 10 (small N) samples. Fitting to a quadratic function and extrapolating to N = oo yields the estimate a = 130 ± 0.03, which is fairly close to the prediction 1,245 of the quenched theory. It is possible that the quadratic function assumed here may be a poor approximation to the true finite-size scaling. This would account for the remaining disagreement between the numerics and the theory. See Ret (55) for more simulations and an attempt at addressing the question of finite-size scaling. c
s
6
c
V I . PERCEPTRON L E A R N I N G OF U N R E A L I Z A B L E RULES So far we have studied examples of perceptron learning of realizable rules. In this section we study three rules that are unrealizable because of architectural mismatch. In the first model, the student and teacher perceptrons are mismatched due to their different transfer functions. In the second and third models, the teacher's weight vector is not included in the weight space of the student. A. Linear-continuous perceptron with unrealizable threshold
FIG. 8- Learning curves for the Boolean-discrete model. Monte Carlo simulations at T = 1,0 and N = 75, averaged over 32 samples. The solid line and dashed lines are the RS generalization and training curves for the spinodal transition. The thermodynamic transition is marked by the dotted vertical line.
In this model, the perceptrons of the student and teacher both have linear outputs and continuous weights, as in the model of Sec, V A. Bowever, in the present case the transfer function of the teacher has a threshold, i.e., the rule is given by Eq. (4 5) with a transfer function = 9a[x) where
466
r
(6,1)
6076 The output of the trained network is given by Eq. (4.1) with ff(i) =
i .
(6.2)
The realizable linear-continuous model of Sec. V A corresponds to the case $ = 0. The average of tbe error function
In the T —> 0 limit there is a critical a below which the examples can be loaded with zero training error. This value is
2 + e2 - 6VT+W
3.12)
For a < or , q is less than 1 and the values of the order parameters are given by the saddle-point equations c
< ( W ; s ) =
t(W
w
(6.3)
s
2>7 - ° > " -
((w) = i-fl+-e
1
(6.4)
A t R = 1, the generalization error takes on its minimum value ( = 0 /2. Recall that according to Eq. (2.58), the average training and generalization errors both approach c as a — oo, at all T. As in the linear-continuous model, the T = 0 behavior can be understood from geometric arguments. Weight vectors with zero training error must satisfy P linear equations 7
m i n
mjn
( W - W ° ) • S' =
,
I = 1...
P
(6.5)
and the spherical constraint W • W = N. There exists an o < 1 such that for a < or , these equations have solutions. For or > a , the equations have no solution, so that the training error rises from zero, and asymptotically approaches f j as c. —» oo. The RS free energy is e
c
c
m
n
i
aoi
2
1 + 3(1-a)
•
(6.C)
where fg a is the RS free energy of the realizable case; see Eqs. (5.1) and (5.2). The saddle-point equations are the same as (5.3)-(5.6), except that Eq. (5.6) for q must be modified to =
g - 2fl + 1 + 9
(6.13)
R = a ,
over the input distribution, Eq. (4.3), yields the generalization error
3
(67)
q = a
+
afl 1-
2
(6.14)
a
In this regime the minimum of (, is highly degenerate with an extensive zero-temperature entropy. Note that for any threshold 9, the critical a satisfies the bounds 1/2 < a, < 1, For a > a „ q -* 1 as T — 0, with 0(1 - q) approaching a finite value. Thus at T - 0 the order parameters are given above a by c
3
2R
= (2 + a + $
1
) R * - a
(6.15)
,
R
(6.16)
«l-4) =
The zero temperature a of this and other unrealizable models is similar to the a defined for the problem of loading random patterns onto a network. In both cases a is the limit of storage capacity above which no weight vectors can achieve zero training error. This is in contrast to the a that we defined for realizable models, above which there is exactly one weight vector which can achieve zero training error. e
e
c
c
As in the zero-threshold linear-continuous model, we expect that in this linear model the RS theory is exact for all T and a. Furthermore, the learning dynamics should be quick, since there are no spurious Local minima in the training energy. In Sec. II D we noted that the AA may yield the wrong a —* oo limit, for unrealizable, non-Boolean rules. The present model is a simple example of this phenomenon. In the present case, Cm,, Eq. (4.14), is
For fixed temperature and a —• oo, we find the asymptotic learning curves T + & etf-CBiin +
— — h O t a "
(6.8) 3
(6,9)
To compare with the general results Eqs. (3.12) and (3.14) for smooth networks, we note that in the present case ft(fW°,S) = -N'^HSi and 6>,cV(W°, S) = N~'SiSj, so that the matrices denned in Eqs. (3.4) and (3.5) are =JiT%
The resulting annealed free energy, Eq. (4.13), is
) ,
T — ffl
Vij
(6.17)
2
-8f(R)
!
= | l n ( l - fi ) - | 2
a
Bff
2 1 + 20(1
1
0
R = { 1- i(fl'-T), (6.13) 1
Hence the coefficient W - ' T r V C / " in Eqs. (3.12) and (3.14) equals f> , in agreement with Eq. (6.8) above. 2
467
5.18)
- R )
In the a —* co limit R is determined by minimizing yielding
(610)
,
In [1 + 20(1 - R)\
-1,
T +
4 < 9
T<0
(6.19)
. a
According to these results, when 0 > 4 and T = 0, the weight vector W approaches - W as a —* oo. G
6077
_ 1
'
1
Before we present the replica solution we note that both the high-T and the annealed approximations predict for this model c,(a) - ( „ ct a~', a — oo. As we shall see below, this is not the correct asymptotic behavior. ml
J.
symmetric
= -;(1-JW-|TZ7
(6.20)
, / N 1 + 2(9(1-1)
Replica
theory
Here tbe replicated Hamiltonian G, is that of the previous linear models, Eq. (5.2). From Eq. (4.26) Go may be calculated for the case of mismatched weight spaces Making the change of variables q + R ? — q, and then eliminating R altogether, one obtains the free energy -W
06(1-R)
(6.24)
= ! - / £ „ = 0.202 .
To understand the origin of this gross failure of the annealed approximation we recall from Sec. IE D that the annealed approximation can be viewed as the exact theory for a stochastic training dynamics in both weight and input space, leading to the GibbB distribution (2.43) in both W and 5 space. The magnitude of the resulting distortion in the posterior distribution of the inputs relative to their tt priori one determines the quality of the approximation. In the present case we can obtain a measure of this distortion by calculating the average of an input vector, e.g., (S'.,., In the a prion Gaussian distribution of inputs (4.3), the average value of an input is of course zero. On the other hand, evaluating this average using the posterior Gibbs distribution (2.43), implied by the AA, we find
implying that the inputs are biased towards —flW° It is important to note that the magnitude of the bias is small—down by from the magnitude of S. However, it is enough to push W towards - W ° , leading to overlaps as low as R = —1. Finally, note that in the realizable case 9 = 0, the average of S is zero. However, even in this case the second moments of S are in general distorted, except when This is consistent with the general expectation that the distortions of the input distribution, implied by the AA, are relatively small in realizable rules. But they vanish only m the limit of o — oo.
+
/ Dzlnacoshlv^r)
•**+»---i45^ <«•> The saddle-point equations are 3 R
=
(l-?) 1 +
1
P
[l + 0 ( l - e ) l
(6.26)
2
(6.27) 0(1
-
qj '
1 = jDiUnh'iv^z) •
(628)
To obtain the asymptotic form of the learning curves we have expanded the solutions in powers of a keeping T fixed. We find - 1
U. Linear output with weight mismatch In this model, the unreadability is due to a mismatch in weight space between teacher and student. We assume that the weights of the teacher network W ° are real valued whereas the trained network is restricted to Wi = ± 1 . For simplicity we consider here the case where the individual teacher weights W ° are drawn from a continuous Gaussian distribution P ( W ° ) , ,
7
PiW?) = (2T)- "e-<>» >W>'
,
.
^min
Ron
-
it
i 3
-j+0(o- ),
(6.29)
(6.21) a
As in the previous Linear perceptron models the error function is given by Eq. (4.9), and the generalization error for a given network by Eq. (4.10). In the present model the optimal weights for the restricted architecture of the trained network are
+
fl»Q
+
i-«„
+
^
)
±
+
0
(a-,. (6.30)
Thus for any fixed T the leading behavior of the errors is an a power law. This power law is not a consequence of the general results (3.1!) and (3.14), since the present network is not smooth. Note that in the high-T limit, only the term T'a~ survives. Thus for this unrealizable rule the behavior predicted by the high-T limit (as well as by the annealed approximation) does not reflect the correct behavior at large a for any fixed T. - 1
(6.22)
W; = sgnlW?) , which corresponds lo the maximal overlap
3
s
j
In the T — 0 limit with 1 — q finite the saddle-point equations reduce to 0
dW P(W")\W''\
m
v^T"
(6-23)
(6.31) (6.32)
The second equality holds in the thermodynamic limit. The minimal generalization error, achieved in the limit a —• co, is
468
6078
(6.33) Criticality is reached when q — oo, and 1 - q —• 0, which happens at a
f
c
2 \ '
n
= l - I 1 - - J
=0.397
(6.34)
Above o , the limit 0 -. oo must be taken with 0(1 - q) constant, yielding t
0 = 2 f l - (2 + c)R? + — , 3
0(1
-
,)
=
(6.35)
R c-R
(6.36)
A t o , 0(1 - q) diverges, indicating that the finite 1 - q solution takes over. t
E.
Optimal
temperature
We define the optimal temperature T (a) as the temperature that minimizes e (T, a). For our models of realizable rules, the optima] generalization error was at T = 0 for all values of a. But in general, T^, may be greater than zero, although the convexity of the free energy guarantees that the training error is a nondecreasmg function of T . This is the case in the present model, as can be seen from Eq. (6.29), which implies that apl
s
6ff«
Toptfc)
= 0.485
(6.37)
as a —• oo. Note, however, that the leading term of Eq. (6.29) is independent of T, implying that in the present model the effect on f of optimizing with respect to T is relatively small.
FIG. 1 0 , The RS zero-entropy Line for the linearmismatched model, which is a lower bound for ihe temperature at which RSB occurs. The line approaches the limit T = 0 . 2 4 2 (dashed line) I S Q - M . temperature gives a lower bound on the temperature at which replica symmetry is broken. Below this temperature, a spin-glass phase with replica symmetry breaking occurs. As T —- 0, the zero-entropy line approaches the origin, i.e., the RS entropy is negative for all a at T = 0. This means that the RS solution is incorrect at T = 0 for any or, and in particular that the R5 prediction (6.34) for o is incorrect, which is consistent with our prediction, Eq. (6,38). c
Below T = 0.242, the system never escapes from the spin-glass phase even as a —• co (see Fig. 10). Note, however, that the entropy (6.39) approaches zero from below. This suggests that the effects of RSB become less severe as s — oo.
g
43.
Replica
symmetry
breaking
The above RS theory is not exact at low T. First, the prediction of a finite value of a is probably incorrect. As in the corresponding realizable discrete model of Sec. V B , we expect that c
(6.38)
Q< = 0 ,
i.e., there is no vector of discrete weights that satisfies order N real, random linear equations. Second, the entropy of the RS solution becomes negative at low T, as shown in Fig. 10. The asymptotic form of the zero-entropy line can be calculated by expanding the entropy in powers of
Numerical
aimuiatious
Figure 11 exhibits Monte Carlo simulations at T = 0.5 and 0.1 with N = 100 and 64 samples. Even at T = 0.1 [Fig. 11(b)] the fit to the RS theory is very good, even though for a > 0.76 the curves are in the RSB region. This suggests that the effects of RSB in this system are weak. The RS T = 0 learning curves are also plotted for comparison. Note that above a > 0.6, the generalization error for T — 0-1 is less than that at T — 0. At least in this range, we may thus conclude that the optimal generalization temperature T > 0, assuming that the RS theory is a good approximation for the true behavior in the RSB region. o p [
1
or .
C- Boolean output w i t h weight mismatch (6.39)
The s = 0 temperature approaches the finite limit = 0242
(6.40)
as a —• oo. The full line ia drawn in Fig. 10 for all a. This
As in the previous model, the teacher weights are again drawn from a continuous Gaussian distribution, whereas the student weights are constrained to ± 1 . However, here we consider the case where the perceptron transfer functions of both the teacher and the student are Boolean. The maximal overlap is still RQO = but the optimal generalization error is now
469
6079
e„ mi
=
-cos"'
Re,
=
0.206 .
(6.41]
where the function H(i) is as defined in (5.54) above. The saddle-point equations are
of Dt ~ * J-,
ft
As in the previous model, both the high-T and the annealed approximations predict t, (a) - ( i „ « a" , a — 00. We shall see below that the true asymptotics are quite different. 1
m
,-.v>
(6.43)
J . Replica symmetric theory (6.44)
For this model, G is the same as that of the previous section, and G is that of the previous Boolean models Eq. (5.17). Hence the RS free energy is given by 0
r
-W =
4 '"5T~^ (1_9)
+
/ "
Dzln!cosh( /50 v
B
+2a j f ° D y j™ Dt In [ < - " + (1 - c~ )H
(u)] ,
(6.45) The quantities u and u are given by Eqs. (6.18) and (5.23) qabove. = jDitanhJ(\/?z) . The large-c limit of the RS theory is derived by expansion in l / i / o keeping 0 fixed,
(6.42)
(S.46) where we have defined the integral (6.47) 1 1
As 0 - oo, 1(0) a (20) ' /3,
l (
(T=0. ) = a
! n
so that
„ +| ^ ( f - i j ' ^ - ^ n
1
+ Olo- ). (6.48)
Thus for any fixed temperature, the RS generalization curve possesses a tail. At T = 0, one solution of the RS equations is obtained by taking the limit 0 —• oo with 1 — q finite. This solution ceases to exist when • I . which happens at the point R.
a,
1
*
, — =. — =. cot — \ / l - ff QC
(6.49)
a = 1.99,
(6.50)
2
t
R , =0.534 .
Above a,, the limit T — 0 must be taken with 0(1 - q) finite, yielding
•A FIG. 11. Learning curve* for the iLnear-miamatchco' model, (a) Monte Carlo simulations at T = OS with N = 100 and 64 samples, with lines from RS theory. The dotted hoiiiontal line is the asymptotic error *,nin. (b) Simulations at T = 0.1. Note that the RS curves (solid lines) fit the data even beyond the zero-entropy point a — 0.76, where there is RSB. The dashed lint is the T = 0 RS curve, shown for comparison.
£ = -
R}
,
,
= -(j- -< i'-«)/(i-« >')
(6.51)
e
TT \
)
Jo
•
IvT^F/ (6.52)
The RS theory predicts that the optimal temperature for generalization T i ( o ) is nonzero for a above a,n = 1.27. The shape of T pt(ct) for large a can be derived using the high-temperature expansion of Appendix Dp
0
470
6080 F. The dominant terms are
which leads to the power law w s
r o~0.315a , j =
(6.53) The term comes from expanding the coefficient of the quenched result (6.46) in 0, Thus it represents the result of first taking the T/a - * 0 limit and then the 0 —• 0 limit. The l / a term is just the leading term from the high-T limit, which amounts to taking first the 0 — 0 limit and then the T/a — 0 limit. The optimal T at large a is obtained by minimizing with respect to T, yielding !
T
3
o p t
~ 0.239a '
5
Spin-glass
g
?
opl
g
For any fixed T, the a — co limit enters the RSB regime. Hence the large a limit of < for any fixed T is given by substituting Eq. (6.57) in Eq. (6.53), yielding
Tbe RS entropy vanishes on the line shown in Fig. 12. Here again the s = 0 line provides a lower bound for the spin-glass transition temperature. Comparing with Fig. 10 two differences should be noted. First, the line intersects the a axis at a finite value, at
(
( , ( T , a ) - ( , „ ~ 0.185o"V«
(6.58)
m
(6.55)
Second, T o ( a ) diverges with a. The asymptotic behavior can be calculated using a double expansion in the variables 0 and T / a . In Appendix F, we discuss the expansion, and how to locate the dominant terms through power counting. They are
(6.57)
s
phase
Cf, = 1.106 .
,
a
(6.54)
However, as we will see below, this RS optimal temperature lies below the zero temperature line, i.e., in the regime where the RS solution is unstable. 2.
a-.ec
Equation (6.56) also reveals that at fixed T, tbe RS entropy goes to -oo as Q —• co, indicating that replica symmetry is violated more and more severely. Like the realizable Boolean model with discrete weights, this model possesses a frozen one-step RSB solution (see Appendix E). However, in contrast to the Boolean or discrete model, this one-step solution is the equilibrium, not the metastable phase, and exists for arbitrarily high a. According to this RSB solution there is spin-glass phase everywhere below the s = 0 line. The training and generalization errors in the SG phase are given by their values on the phase boundary T, — T,at the same a [see Eq. (5.76)). This is shown in the plot of e in Fig. 13. Since T ( a ) is above the RS T ( a ) , there is no minimum in i (T) for any a. Instead there is a whole regime of temperatures where c does not change with T, implying that optimal generalization can be obtained anywhere on or below the zero entropy line
which is independent of T. This power-law decrease is faster than the RS prediction l/^/a, Eq. (6.48). S.
1=
Numerical
simulations
Figure 14 shows Monte Carlo simulations at T = 0.5 and 1.0 with N — 75, averaged over 64 samples, Below T = I the training curve is significantly different from the RS result. These deviations for T = 0.5 are significantly larger than those observed for the linear-
1.6
///
o
at
os
i
T
as
1
5 a
10
60
FIG. 12. RS sero-entropy line for the Boolean-mismatched model. The line intersects the o axis at the finite value o = 1.100.
FIG. 13. Generalization error for fixed a = 1.2, 1.6, 2.Q, 2,4, 2.8, and 3.2 as a function of T for the Booleanmismatched model. Each curve is a combination of the onestep RSB solution below T {a) (marked with a dot on each curve) and the RS solution above. The light dotted lines are the continuation of the RS curves below T (a). Note that the minimum of each of the RS curves is below g
9
471
6081
in certain circumstances to train the system at finite T because convergence times may be prohibitively long at T = 0. This is particularly true for highly nonlinear models, such as the Boolean perceptron with discrete weights. Although the critical number of examples per weight o ( T ) increases with T in this model, we have found in our simulations that the time i t takes to converge to the optimal state in this model, increases dramatically as T is lowered. It should he stressed however that we have used only a simple Monte Carlo algorithm. Recently several heuristic training algorithms for perceptions with binary weights have been proposed [34,56]. I l would be interesting to study their dynamic and generalization properties. t
0.1 OO i
,
,
,
,
•
I
4
•
•
l
10
B. The high-temperature and annealed approximations
0.0
L
.
,
,
,
!
0
2
4
•
•
It
o FIG. 14- Learning curves for the Boolean, mismatched model. |a) Simulation! al T = 1.0 with = 100 and 12 samples. Each learning cutve is a combination of the RS solution below the zero-entropy point a = 8.4 and the one-step RSB solution above. The light dotted lines are the continuation of the RS curves above a = 6.4. (b) Simulations at T m O.S.
mismatcbed model, indicating the importance of RSB for this model. The RSB solution fits fairly well at large a, but in the vicinity of the transition, there are still some deviations between the theory and simulations. These deviations may be the result of difficulties in equilibrating the system near the transition. V I I . DISCUSSION A. Learning at Unite temperature In this paper we have studied the process of learning from examples with a stochastic training dynamics. The level of noise in the dynamics is denoted by the temperature T. One of the most important results of our analysis is that learning at finite temperature is possible, and sometimes advantageous. For any finite T, as the number of examples increases, the network weights approach their optimal values, namely the values that minimize the generalization error. Thus even when the generalization error increases with T it may be profitable
We have presented two approximate theoretical approaches to the problem of learning from examples in layered networks. The first approximation replaces the training energy by the number of examples times the generalization error, and becomes exact in the limit of learning with high thermal noise level. The dependence on T and a is only through the effective temperature T/a. Even in this simple approximation, perceptron models exhibit a rich spectrum of generalization behaviors. The second approximation, the annealed approximation, reduces to the proper high-T limit, but deviates from it significantly at finite a and T, where the behavior ia no longer a function of only the ratio of the two parameters. In all four realizable perceptron rules studied here, these approximations have predicted correctly the shapes of the learning curves at finite T and large a. Furthermore, the AA has yielded interesting results at finite T and a that are qualitatively correct. For instance, the first-order transitions predicted by the AA for the perceptrons with discrete weights are clearly observed in the simulations, Figs. 3 and 8, and are in agreement with the full quenched theory. On the basis of our genera] arguments in Sees. I I and I I I , we expect that these approximations will also hold for realizable rules in the more complex cases of multilayer networks This is borne out in recent studies of two-layer networks of local feature detectors [35], and other multilayer systems [36] Thus these approximations provide powerful theoretical tools for the study of learning from examples, at least for realizable tules. Our treatment should be contrasted with the difficult problems of the capacity of single- and multilayer networks [57-59]. The capacity problems usually deal with loading random sets of data In this case the system is highly frustrated and one has to employ the complex methods of spin-glass theory, such as replica symmetry breaking [33,60]. In the learning problems of the present work, the training set consists of very structured data, generated by a well-defined rule. The underlying structure is represented in our formulation by the generalization function ( ( W ) , Eq. (S.4), which dominates the
472
6082 behavior of the system at least when tbe number of ex¬ amples is large. This function would be completely flat, and therefore meaningless, in the case of random training sets. In fact, the high-T limit ignores completely the fluctuations due to finite sampling of the rule. It is thus useful in studying how learning is affected by the nature of the target rule [e.g., through the function c(W)] as well as by the network architecture [e.g., the entropy s(c). Eq. (2.26)]. As has been explained in Sec. I l l the AA does take into account the effect of randomness induced by the examples, though only approximately. The reasonable results generated by the AA even at finite a and T imply that this randomness may not have major effects (e.g., frustration and other spin-glass phenomena), at least in models where the chosen architecture is compatible with the rule to be learned. This may explain why, in many applications of supervised learning, simple local gradient algorithms seem to yield good results. I t should be emphasized that the above approximations may be useful not only for studying specific "toy" models, but also in generating general approximate predictions that could perhaps be useful in applied research on neuraJ networks. An example is Eq. (2.40), which provides a way of estimating the generalization error from the observed training error for a multilayer network with a Boolean output.
D. Learning realizable rules The results of the specific models studied here indicate that the shapes of learning curves may be very different from the PAC learning bound of an inverse power law. The shape of the generalization curve depends strongly on the degree of constraint imposed on the network space. In the case of a linear output, the imposition of binary constraints changes a T/a tail [Eq. (5.10)] into an exponential exp(-2et/T) [Eq. (5.32)]. In addition, at low T there is a discontinuous transition at finite a from poor to good generalization; see Figs. 1 and 3. The superior generalization ability of the constrained network is not surprising Since the target rule itself was assumed to be realizable in the constrained architecture, imposing the constraints is essentially using prior knowledge (or assumptions) of the nature of the rule to restrict the space of possible networks. The most dramatic effect of constraining the network is found for the conventional perceptron rule, i.e., one with a Boolean output. In the unconstrained network the generalization error again behaves as an inverse power law [this time even at T = 0, Eq. (5.28)]. On the other hand, in tbe case of binary weights there is a discontinuous transition at a critical ct from poor to perfect learning (see Fig. 6). This transition is unique in that i t exists even in learning at high temperatures. The collapse of a thermodynamic system at finite T to its ground state above some a stems from the singular spectrum of excitations above this state, as discussed in Sec. V D, c
C. Inverse power law for smooth networks The most important general result is the inverse power law for the asymptotic behavior of both the generalization and training errors, Eqs. (3.12) and (3.14) of Sec. I I I . These results also provide a simple relationship between the two errors, namely Eq. (3.15). This power law is consistent with the general bound obtained within PAC learning theory. However, our theory is not distributionfree and holds only for smooth networks On the other hand, it holds for general unrealizable rules, whereas the PAC bounds are essentially for realizable rules. The results regarding smooth networks were derived using a perturbative approach, i.e., assuming that essentially all the components of W deviate only slightly from W* for sufficiently large a. Of course in reality there can also be contributions coming from W far away from W " , leading to nonperturbative terms t o t , . However, we expect that for sufficiently large a the nonperturbative terms will be negligible (e.g., exponentially dependent on a) relative to the power-law contribution of the smooth fluctuations. On the other hand, these localized nonperturbative errors may be important for the dynamics of the training, since their relaxation may be extremely slow compared with the continuous fluctuations. In nonsmooth networks the gene r alii at ion performance depends on the nature of the task as well as the network architecture, as the results of Sees. V and V I demonstrate. These results alBo indicate that there is a qualitative difference between the learning of realizable and unrealizable rules, as discussed below.
In all the realizable models studied in this work, the quenched RS behavior at finite T is qualitatively similar to that given by the AA. Furthermore, the asymptotic shapes of the RS learning curves in the different models agree with those of the AA. This suggests that for realizable rules the effect of disorder is minor for large a. The main qualitatively different result of the quenched theory is the appearance of spin-glass phases, as shown in Figs. 1 and 6. These phases result from the randomness and frustration associated with optimizing for a particular realization of examples, and cannot be predicted by the high-T or the annealed approximations. Two special features distinguish the SG phases of realizable models. First, it exists only at low T and in restricted regime of a, typically a < 1. Second, it is only metastable, i.e., its free energy (as well as training energy) is higher than lhal of the optimal state, which is therefore the true equilibrium state in that regime of or. The absence of SG phases at large a indicates that as ce increases, the relative scale of fluctuations in the training energy becomes smaller, i.e.. the training energy surface gets smoother as the number of examples increases. This ia indeed demonstrated by the analytical bounds of Sec. V Bon the number of local minima for the linear perceptron wilh discrete weights (Fig. 4). These bounds exhibit two important, and possibly general, features. First, the energy surface in the neighborhood of the optimal state I i • smooth. Second, the number of local minima is exponentially large for small a, but decreases monotonically with a. In the linear-discrete model there are no local minima above a critical value of
473
a. Fontanari and Koberle [61] have studied numerically the local minima of the Boolean-discrete model for small system sizes. The number of local minima was found to scale exponentially with JV and to decrease monotonically with a. No critical value of a for the disappearance of Local minima in the Boolean model was found. However, a definitive conclusion would require the investigation of larger sizes. Finally, in all our realizable models the generalisation error decreases monotonically with 1 '. BO that the optimal temperature for learning is T„p, = 0 .
(7.1)
Of course this refers to the equilibrium properties, whereas from dynamic point of view the optimal temperatures may be finite even in realizable rules, as has been pointed out above.
in the two-layer model network for the contiguity problem [35]. The above classification requires knowledge of the behavior of s(<) near c — 0. In general, evaluating s may be difficult even in relatively simple models, since the primary interest is its behavior in the limit N —* co. For this reason, the simple existence of a gap in the spectrum of c is an insufficient basis for classification. At the very least, the scaling of the gap with N must be determined (see Sec. IV F). Finally, it should be stressed that tbe classification of the asymptotic shapes of the learning curves according to the properties of s(() has been justified so far only in the high-T limit. I t may be, however, that in many classes of realizable rules these results apply also to finite T or even T = 0. I f this is true then our results may provide useful hints for understanding or even predicting the behavior of some real-world learning problems.
E. Classification of learning curves for realizable rules The discontinuous behavior of the Boolean perceptron with discrete weights calls for an understanding of the general conditions which determine whether a given learning task will be achieved gradually or not. Insight into this question Is provided by the high-T limit of
F. Unrealizable rules
(7,2]
For unrealizable rules the optimal weight vector W * is reached only in the limit a —• 00. This is true at all temperatures. As T —* 0 the generic shape of the learning curves is roughly that shown in Fig. 11(b). There is a critical value a which marks the loading capacity, i.e., the point below which the examples are memorized perfectly. Above a the training error increases, and approaches the same limit as the generalization error, i.e., approaches c ^ from below as a —• 00. Also, in some unrealizable models (not encountered in this work, but see Refs. [22,24]) there is a local maximum in t near a .
For a smooth weight space we expect s to diverge logarithmically to -co, i.e., it = 0- This naturally leads, via Eq. (2.27), to the inverse power law. For a discrete weight space, we expect the entropy to approach a finite value, i.e., x > 0. When 0 < x < 1 the generalization curve obeys a nontrivial power law
One of the important results of our study is that for unrealizable rules the learning curves are substantially different from the high-T and AA predictions at any fixed T. For example, for the models with weight mismatch, these approximations predict o tails for fixed T, whereas the RS theory yields [see Eqs. (6.29)]
Sec. 1 I C According to Eqs. (2.2b)-(2.27)
r
a system can
be classified according to the behavior of its entropy s(*}. Recall that B ^ ' M I S defined as the number of networks that yield a given generalization error (. The general form of s(t) for small t is expected to be s
s(c)(x( ln(,
t - 0 .
c
c
m
s
1
1
^(a/?)" '* -*',
op^oo,
0 < r < l .
(7.3)
up to logarithmic corrections. When x = 1 this law turns into an exponential, hw
fn/? —* co, i = l .
fl
t
3
= 0,
a&>
{Q&) , C
X > 1 .
- J
fjCT.Ql-c^.^a.-'
(7.6)
for the case of a linear output. In the Boolean output case the full spin-glass theory predicts
(7.4)
An example of this case is the perceptron with discrete weights and linear output; see Eq. (5.32). When r > 1, dsfdt remains finite at small E , S O there must be a discontinuous transition at some {afi) to £j = 0, f
e
(75)
The Boolean perceptron with discrete weights is an example of an entropy with x = 2; see Eqs. (5.62) and (563) In our perceptron models, nontrivial shapes of learning curves resulted only when the weights were constrained to discrete values. In contrast, for multilayer networks we expect that the exponent x in Eq. (7.2) may be nontrivial even when the weights are allowed to vary continuously, possibly leading to Eq. (7.4) or (7.5). This will occur, for example, if the optimal solution involves discrete internal representations of the hidden neurons, as demonstrated
474
^ J
(T,c,)-
- 1
,
f m
i ~ •- / n
Q
5
(7.7)
7
An a ' tail in < has also been found in Boolean perceptrons with continuous weights where the rule is unrealizable due to corruption of the examples by noise [24] or due to the random nature of the rule itself [22], s
The RS theory of perceptrons with unrealizable rules predicts that, for sufficiently large a, E , is nonmonotonic with T, but possesses a nonzero optimal temperature. Similar results have been obtained in [22,24]. In the case of a Boolean perceptron with weight mismatch, where the corrections to RS theory could be calculated, it has been found that rather than having a minimum at finite T, e, is independent of T at low T; see Fig. 13. Whether correction to the RS theory will modify substantially the conclusions regarding T of the RS theory in the other models remains to be studied. o p l
Another feature of unrealizable rules is the prevalence
6084
APPENDIX Ai TRAINING-GENERALIZATION INEQUALITY
of spin-glass phases even at large a. In the present work we have evaluated the T at which the entropy vanishes. This provides a lower hound for the onset of the spin-glass phase. Here again the linear and Boolean perceptrons differ. In the linear case this temperature levels off at a finite value as a increases, as shown in Fig. 10. In the Boolean case i t grows with the same power law as T t ; see Eq. (6.57) and Fig. 12. These results suggest that spin-glass effects are strong in unrealizable cases.
Let £rj denote the average value of the training energy P ( ( W ) , as in Eq. (2.18) and SE the difference E - EQ. Define the function
op
E
B
jd^(W)e-' '-^ 6E
(Al)
The fact that the spin-glass phase exists also for large a suggests that the fluctuations in the training energy do not necessarily shrink as a increases. Nevertheless, we believe that their relative scale does indeed vanish. This is because the energy itself grows linearly with a. This conjecture is supported by the observation that the temperature that marks the onset of spin-glass phenomena grows only sublinearly with a [see Eq. (6.57)] - The effect of increasing the number of examples on the roughness of the training energy surface (particularly in unrealizable rules) is an important issue which deserves further study.
J d f ( W ) -yEa-gSB e
This can be interpreted as the energy of a system at temperature 1/0 with HamUtonian i E and measure d j i ( W ) e " ' . By the convexity of the free energy, the energy is a decreasing function of 0. Hence A is bounded above, lEo
, E
/tfii(W)e- °4£ 4.(0,7) < A(i? = 0 , ) = 1— 7
(A2)
E
/
dpCW)e-' >
We now take the quenched average of both sides. In the integrals, the average can be applied directly to 6 E , since Eo is independent of the quenched disorder. But ((6E)) — 0, so tbe right-hand side vanishes, and we have
G- Uniqueness of the optimal solution Throughout this paper we have assumed that the optimal network weights, i.e., the components of the weight vector W " that globally minimizes the generalization function c ( W ) , are unique. If they are nonunique, we assume that the degenerate global minima are at least widely separated in network space, forming a discrete set. This is related to our assumption that the training dynamics searches for the values of the network weights within a well-defined architecture. Under these conditions it is reasonable to expect that the optimal solution is generally unique up to obvious symmetries. For instance, in multilayer networks the solution may be unique only up to permutation of the hidden neurons [62], Other symmetries may result from the nature of the rule itself. An example is the up-down degeneracy of the optimal solution of the two layer network of edge detectors in Ref. [35]. This uniqueness does not hold if the architecture of the network, e.g., the number of neurons or the number of layers, is allowed to vary significantly. The important issue of learning under these circumstances is planned to be discussed elsewhere.
(A3)
;AOS, )»
In particular, for 1 — 0. ((A)) = f ( < . - £ ) , so that we finally obtain s
(A4) which was stated without proof in Eq. (2.20).
A P P E N D I X B: H I G H - T E M P E R A T U R E EXPANSION In this appendix, we show using a cumulant expansion that the free energy (2.13) can be written as a power series in 0 with coefficients that are functions oia0. The zeroth-order term of this series is the high-temperature limit discussed in Sec. I I C . The first step is to separate the energy into random and nonrandom parts. The nonrandom part is S
So s ( ( £ ) ) = Pe(W) = Arc*(W) ,
(BI)
and the random part is p
ACKNOWLEDGMENTS SB = E -
We would like to thank E. Baum, D. Hansel, D. Haussler, and D. Huse for helpful discussions. We are grateful to G. Gyorgyi for his comments on an earlier version of the manuscript and for drawing our attention to the RSB phase in the Boolean model with mismatched weights. We also thank the Aspen Center for Physics for host ing a Workshop on Neural Networks where part of this research was conducted. H. S. S. acknowledges support from Harvard University.
E
a
= J 2
6
c
«
•
<
B 2
)
where S
FU
= {W-S")-i(Yr).
(B3)
t
We now treat 6 E as a perturbation to the "free Hamiltonian" Ea- The partition function takes the form Z = Z (e0
s , E
) . 0
(B4)
Here ( ) denotes the average with respect to the distri-
475
0
6085 l
aE
bution Za t~ ' Z (3c<)
The factor E
= j
e
C j - A ^ ' / ^ a f l ,
dp[V/) -^ '=
jd (W)e-
e
f l
, v
<
5 !
to leading order in a. In the expansion for the free energy, the term containing C, only contributes to terms of order $»<^ or higher in the ultimate high-temperature expansion. Hence the free energy can be written in the form
w
"< >
(B5) is the high-T partition function introduced in Sec. 11C. Taking the logarithm of both sides and performing the quenched average, we obtain -
3F = In Z (aB)
+
B
C,)),
j=i
(B6)
3
where the Cj are from the eiimutant expansion of ln(e-<" ) E
u
C,-{i£)o, C
2
(Bill
- S F ^InZc + f^-f.a-/?), f=i
(B12)
where Ft contains contributions from the finite number of cumulants C\,•.., Cjj, In general, any quantity A that is finite in the high-T limit possesses a high-temperature expansion of the form
(B7)
= ((SEY} -(SE)l
A(/9 T/a) = f ; / 3 M ( r / ) .
(B8)
0
C» = --.
1
i
(B13)
0
.=0
(B9)
If ft£ were a quantity of order unity; the cumulants would be functions only of <*&, the only patameter governing the distribution E~ . Hence Eq. (B6) would be the desired power series in $ In fait, the situation is somewhat more complicated because 6 E scales like a. To investigate the scaling of the cumulants with a, we write them as sums over connected correlation functions
In the high-temperature limit (3 -• 0, T/a = const), A(S,TM^A (T/a)
SEO
,
B
(BH)
which depends only on the effective temperature T/a, In general, the functions At can be nonanalytic functions of T/a. In particular, the high-T limit term A can be nontrivial and lead to such behavior as the first-order transition in the Boolean-discrete modela
p
}
Counting the P terms in this sum the naive estimate would be that Cj ™ P . However, the quenched average makes any term containing an unrepealed index vanish, since ffffd U = 0. In other words, if an index appears, it must appear at least twice for the term to be nonvanishing. This means that the cumulants can scale no faster than Cj ~ At the same time, there are also factors of N which make the cumulant extensive, so that it behaves like
A P P E N D I X C: A N N E A L E D A P P R O X I M A T I O N TOR PERCEPTRON L E A R N I N G
h
J
.
I
- j DS((W;S)
=
In this appendix, we give a fuller account of the results that were outlined in Sees. IV A and IV B. We begin with a derivation of Eq. (4.7) for the average generalization error, which illustrates many of the calculational techniques of this paper. Integrating the error function (4.6) over the a priori input measure (4.3), we obtain
j d z jdy\\3(x)
3
- ff(y)]
j
(CI) DSS
(x - W " " W • s) !
S (y
-
N
^
W •
s) , (C2)
- ?<# J SSexp [-iJV-V'(Wi + W S ) . s] I
The two auxiliary variables • and y are introduced to remove S from the argument of the g functions, and z and y are introduced to transform the 6 functions into exponentials using the identity
* =/S '" (r,
E
(C3)
simple Gaussian integral in i and y is left. These variables can in turn be integrated out, leaving C ( W )
- / 277CT
f , P
*^[s(*)-S
When the Gaussian average over S is now performed, a
476
l!
2(1-g)
) (C4)
This result has a simple interpretation. I t is the average
60B6 3
of £[j(z)-s(!/)l . "here i and y, like W S/VN and W S/VNia (4.6) , are Gaussian variables with unit variance and cross correlation R . A simple change of variables in Eq. (C4), a shift followed by a reseating, yields the form (4.7).
We can write the stationarity equations in a more revealing form by proceeding further in the evaluation of Go- The 6 function in (C8) can be expanded by introducing another order parameter R ,
To derive the annealed approximation for perceptron learning, we begin by evaluating Eq. (2.31) for G ( W ) . The calculation is essentially the same as the previous one for * ( W ) , and results in a similar formula, U
„(W) = - In J
2
txp
xexp(-0[j,( )- (y)]V2) r
(CIO) In the thermodynamic limit, this reduces to Go -
(C5)
s
Again, the answer depends on W only through the over, lap R . A change of variables in the integral (C5) yields the form (4,14), Evaluating the integral for the cases of g(x) — x (linear perceptron) and g(x) — sgn(i)/\/2 (Boolean perceptron), we obtain
-RR
+ | j In j
- B f = extr j , i-RR + ±] j a
Since G „ ( W ) , Eq. (2.31), depends only on the overlap R , the annealed partition function (2.30) can now be rewritten as an integral over R ,
(Cll)
d/ifW) (C12)
3G„
(C13)
-a-
OR
/i._.
{
W
)..W
0
(C14)
where ^ /d^W)Wexp(flW W°) *
/dV(W)exp(flW.W°)
(C15)
The order parameter R has a natural interpretation: it is the strength of a local field pushing W in the direction of W ° . Since it increases with a, it forces W toward W as ct —* co. Upon eliminating R , these equations reduce to Eq. (4.16). Equation ( C l l ) can be evaluated quite easily for the case of the Ising constraint W; = ± 1 . Then we can make the replacement 0
(C7)
,
r
' ,
The saddle-point equations are R=
Although derived using the Gaussian a priori input distribution (4.3), the above results for f ( W ) and G ^ f W ) apply also to the case of the discrete inputs Si — ±1 in the thermodynamic limit. This insensitivity to input distribution is explained by the central limit theorem, which guarantees that W • S/\/j7 and W ° • S / i / W are Gaussian variables (in the N —• oo limit) with very weak assumptions about the distribution of S. This assertion may be verified by a straightforward calculation for discrete S, which yields Eqs. (C6) as the leading terms in a saddle-point expansion in 1/N.
= jdRcxpN[Go{R)-aG, ,{.R)]
w
a
(C6) , - In [1 - (1 - e " * ) T - ' cos"' R] (Boolean).
[Z))
w
-oG „(R)
- i | n [ l + 2 0 ( l - / i ) ] (linear) {
e«
where the right-hand side must be stationary with respect to the saddle-point parameter R . The free energy can now be written as a saddle point over two order parameters
fl
C„ -
RW W°
2
x + y - 2xyR 2(1 -R?)
dxdy
2*1/1-fl?
+ ln / d(i(W)
aV(W) where
(C16) w,=±t
NG (R) B
= In j
0
n>fW)*(fl-Ar'W-W )
which leads finally to
(C8)
GQ(R)
is the logarithm of the density of networks with overlap R . In the thermodynamic limit (N —t co), the integral can be evaluated as -0f(T,c,)
= ^ln((Z)) =extr [Go(R)-«G^(fi)] ,
= -RR + In2coshR ,
(C17)
assuming that the teacher weights also satisfy the Ising constraint. Extremizing with respect to R , we obtain the equation R = tanh/?
(CIS)
R
(C9) Hence the thermodynamic free energy f(T, a) is determined by extremizing the free-energy function -0f(R) = G (R) - oG^(R). Differentiating }{R) with respect to R , we obtain the stationarity condition Eq. (4.16). a
477
Eliminating R , one can finally derive the result (4-20), which is purely a function of R , and is the familiar result for the entropy of the Ising model as a function of the magnetization recalculating G o for the spherical distribution is somewhat more complicated. We rewrite the a prion spherical
6087
distribution (4.17) as
Eliminating and A finally yields the result (4.18), which was justified previously by geometric arguments.
(CIS)
QVI(W) t
(=1 > ™ / - * »
2 t
' A P P E N D I X D: REPLICA THEORY OF PERCEPT RON L E A R N I N G
and then Go is determined as a saddle point over R and
Go = - i l n ( 2 i e ) + f?.fl + A + l n
j
d W s
l
w
' '
i
<
The starting point of the replica calculations is the derivation of the replicated Hamiltonian (2.49), which resembles the derivation of the annealed C described in Appendix C. We introduce auxiliary variables i „ i „ , If, and y in order to simplify the average over S,
w
u
=
_ i
RR
+
+
A-iln(2A) +
(C20)
= jl\dz, j dyexp (-^'^^-mff) fm
X J DSexp -iN'i
W * i , + W°yj
ft* ( * .
-
N-'^W-s)
6
(
v
- W ^ W "
s)
(DI)
S
The average over S is now a simple Gaussian integral and yields
(D2)
Since Q depends on the weights only through the order parameters Q and R that were defined in Eqa. (2.55) and (4.21), the replicated partition function can be written as an integral over these order parameters r
<( "» z
A
gfi
=
/
/II
dK***V<{-NaG \Q.,.R.\) t
x /j]^(W
f ,
1
)f]i(g„ -jV- W'' 0
W»)J]*(/t
1
l r
0
-A'- W'.W ),
(D3)
(D4) where
So = - g
U
- £
+ 4 In f
du(W") exp [ £ fl, W
478
W° + £
3„ W
-W ]
(D5)
6088 is the logarithm of the density of replicated networks with the overlaps Q „ and R . In the thermodynamic limit, the integral (D3) over the order parameters is dominated by the saddle point in Qffo and R . The free energy is obtained by analytically V • ... .,, . . . „ * ' continuing this saddle point to n = 0, 0
(08} (DO)
c
a
w
w
(D10) i% = R . nr.. L .-. *t r .1 , L P With this substitutions, the free energy takes the form
w
-8!
- exlT\G {q,R,q,R) 0
- aC ( ,H)) , r
9
(Dll)
9.ft
where we have defined G, S lim — , n-*0 n «* — IGD = lim So —
(D61 According to the RS ansati, the saddle point takes the form
n —0
(012) (D13)
rt
To calculate G , we substitute the RS ansatz into Eq. (D2) and perform the integral over j), leaving P
(D7)
xexp l-hl-q)Y^il
+ i£f,(i,-%))
/Dlexp ( - . t ^ ^ f l = £ >
(DM)
2
The auxiliary variable ( has been introduced via the identity J Dt e" = e x p ( ± t ) . Performing the integrals over the x , and shifting and rescaling the i„ integrals, we finally obtain a
= j'otjog
Dzexp (-^3
\g (t^/l^
+ yR + t~/q~^W) - -(»)] ')] "
(D15)
Taking the limit n — 0 gives Eq. (4.27) for G,. The RS result for Qo is
C = n(-flft0
| n ( n - l) d - i d ) + I I n / Dz ?
(/
^(W)
exp[W • ( z \ / ? + W » B ) ] )
The n — 0 limit of this expression yields Eq. (4.26) for Co.
(Die)
A P P E N D I X E: ONE-STEP RSB FOR PERCEPTRONS Following the Parisi theory of RSB [45] we make the following "one-step" ansatz for the form of the order parameter matrix Qft \ V
/If!
jo SD • • •
qo qc •••
<3*»=
V
qo qo • • • qo qo • • •
1 Ii Ii 1
(El)
/
Each block in this structure is an m x m matrix. All off-diagonal blocks consist of Q = qo- The diagonal block has Q = 1 and Q„„ = q for / i 7? 1/. The conjugate matrix (?„„ has a similar block structure. The order parameters R pr
n
v
479
6089 and R are symmetric at the saddle point, i.e., R = R and R ? — R , as before. Upon replacing Eqs (D7)-(D10) by tbe RSB ansatz and taking the appropriate n -» 0 limit, the free energy (D6) is given by a
u
-3f
= Gofjo.fli, !o, ! L , R , R , m) - cG (oD, «i, R , m) ,
(E2)
r
iv here
C
= ^{moo^
0
+ [ { l -
m
)
q
- l ]
i
A
\ } - R R + ^ - j D x
l
a
n
J
(J d,,(W) exp(W • Z ) )
Dz,
,
(E3)
Z = z \/fa + zi Vii - ?o + WR,
(E4)
B
= -Aj&j, j
0
r
t = ioV'qa
-R"
j
Dk\n
+ t, VkT^
Di,
-
|7i^^.(-|^}-ff(i^j ,
IBS)
yR
(E6)
The free energy has to be minimized with respect to go, ( i , q , i i . R . R , and m. Note that after the n —• 0 limit has been taken the allowed range of m is 0 < m < 1. Also, the physical meaning of go, 4i• and m is explained in Sec. V D 2 (see Eqs. (5.73)-(5.75)]. Upon substituting m - 0. tbe free energy / reduces to the RS result, Eq. (5.66). Specializing to the case of Ising constrained weights in G and Boolean output in G i , we obtain a
D
GD=
i(meovo + [ ( l - m )
+77™
, T
-
£
faa-R
la\
/
3
D
!
°
l
n
/
¥ l
D
:
/q, - ao +in/n
l]q,)-RR i
[
2 c 0 3 h
yR ,,
z
( °
+ ii
Vii
-
io
+ W?R)]",
,„ . (E9 n
•
We now search for a solution to the saddle-point equations at finite temperature with the property
G, = - m
Il = 1,
5i = o= •
(E7)
[°° Dy fDt\a[eJo
Bm
+ (1 - e -
f l m
)fl(u)j
,
J
(E12)
(E10)
According to Eq. (5.73), for such a solution each pure phase consists of a single (or a few almost identical) microscopic state. Thus the system collapses at finite temperatures to phases with zero entropy. This is not unlike the collapse of the system to the ground state at higher values of o (i.e., the perfect generalization state). However, it occurs in the metastable, spin-glass regime. To find * solution with the property (E10) we take the limit q^ — 1, j , — oo of Eqs. (E7) and (E9) while keeping 0 finite. We obtain 2
Go = — im jo(so - 1) - mRR) m
1 - qo
(E13)
i / l - 5o
Comparing Eqs. (E11)-(E13) with Eq. (5.66) one obtains 2
/ R S B ( 9 o , f o , f t , f l , m , 0 ) = ^/Rs(m,m 9o.fl,mrc,(3m) . (E14) This structure is similar to the result of Krauth and Mezard [33) for the case of training a perceptron with random input-output mappings. Stationarity with respect to q , Jo, R and R results in B
q (T,m,a)
= qns(T/m,a)
R(T,a,m)
= R^T/m.a)
B
(Ell)
480
(E15)
, .
(E16)
W90 Finally, stationarity with respect to m yields * (r/m,o) = 0,
(Et7)
R S
which implies
o p
m(T,c.) = T/T,(a) ,
fa
(£18)
where T,(a) is the s = 0 tine of the RS solution. We have not attempted lo search for other RSB solutions or to cheek the stability of this solution. However, this solution is probably exact both here and in the random perceptron problem of Krauth and Mezard. Similar completely frozen SG phases are known to exist in the REM, the "simplest spin glass" [54], and the large-P Potts glass. Recently, they have also been found in learning of random mappings by two-layer networks 160,62,63]. A P P E N D I X F: POWER C O U N T I N G I N T H E H I G H - T E M P E R A T U R E EXPANSION For the Boolean-mismatched model, the optimal generalization and zero entropy lines both follow power laws of the form T ~ a', with 0 < P < L The determination of this exponent requires the balancing of the two dominant terms in the asymptotic expansion. This is somewhat tricky, because determining which are the two dominant terms in turn depends on the exponent. Since both T and a are diverging, it would seem natural to perform a double expansion around (1/T, 1/a) = (0,0). Such an expansion is in fact ill defined, since the existence of a nontrivial high-temperature limit shows that the T, a —• oo limit depends on the ratio T/a. As discussed in Appendix B, the proper hightemperature expansion for a quantity A that is finite in the high-T limit is A(3.T/*)
= f^S
(Fl)
1
i=0
= - £
l-r
f
The problem thus reduces to linear programming. Given a set of pairs (ai,fr,), find the r that maximizes minora;+ (1 -r)t>,-]. The problem looks difficult because there are an infinite number of pairs i /,: to consider, but in fact most of the pairs can be eliminated from consideration. We define a partial ordering of the set of pairs: (a,A) £ means a, < ay and t, < 6j. A term (a, b) is minimal if (a, 6) > (c,d) implies (a, b) = (c, d) for all (c, d). Only the subset of minimal pairs (which cannot be ordered) are relevant, because (a;,*;) < ( a j , ( j ) implies ra, + ( l — r)o, < raj + (1 — r ) i j . The linear programming problem thus only includes the finite minima] subset of pairs (a*, 6*). For the Boolean-mismatched model, we calculated the asymptotics of the generalization error as a series in e t - " at fixed T, 1
OO
•,lT,a) - t ,„ = £ e , ( j J ) o - - ' " j=i m
)
A
- ' ( T ) °
ii;)'
|
F
2
)
(F3)
!
To convert this to the form (F2) of the high-T expansion, we expand the ( j in powers of 3. Only the leading term from each e, need be retained, since the higherorder terms arc irrelevant. From the terms of order a ' through o r we obtain the (1,1/2), (1,1). (1,3/2), and (0,2) terms in the high-T expansion (F2). From these four terms, wc finally extract the minima] subset (1,1/2) and (0,2). Terms in (F3) of order n " ' and higher need not be considered, since they are all bounded below by (0,2) and are thus irrelevant. - 1
3
1
5
If one proceeds to expand the A,, one obtains a double expansion about ( l / T . T / e . ) = (0,0) AWT,T/C,)
that B < > < 1, so that both 1/T and T/a approach zero as a — ' co. Assuming now that we have an expression for ( , of the form (F2), let us find the power law T i *" a' such that t9 decreases at the fastest rate. In the series (F2) each term a, b scales like l / a * " * ^ , s o that the exponent is some weighted average of a and tj. The power of the dominant term in the sum is min,(ra, + (1 - r}b,\ This exponent must be maximized, to ensure the fastest decrease of t .
2
The maximin problem, which involves only (1,1/2) and (0,2), has r = 3/5 as its solution, justifying the result Tap, ~ a " quoted in Eq. (6.54). The calculation of the zero-entropy line is similar, except that the minimal terms must add up to zero. 3
This high-T expansion is the proper tool for investigating power laws of the form T ~ a' We must only assume
" Present address: Department oi Computer Science, Hebrew University. Jerusalem 91904, Israel. [1] R. P. Lippman, Neural Comput. 1 , 1 (1989). (2] T. 3. Sejnowski and C. Rosenberg, Complex Syst. 1, 143 (1987). [3] Y. LeCun, B. Bosei, J. S. Denker, D Henderson. R. E. Howard, W. Hubbard, and L. D. Jackel. Neural Comput. 1, 541 (1989). (4] N. Qian and T. J. Sejnowski. J. Mol. Biol 202, 865 (1998). [5] D. Zipser and R. A. Andersen, Nature 331, 679 (1988). (S] S. R. Loekery, G. Wittenberg, Jr., W. B. Kristan, and
[7)
W. G. Cornell, Nature 340, 468 (1989). Parallel Distributed Processing: Explorations CToatructure
oj Cognition,
in the Mi-
edited by D- E- Rumelhart and
J. L. McClelland (MET Press, Cambridge, MA, 1986). [8] T. Poggio and F. Girosi. Proc. IEEE 78, 1481 {1990). [9] L.G. Valiant, Common ACM 27. 1)34 (1984). (10) A. Blumcr, A. Ehrtnleuchl. D. Haussler, and M. K. War[11] [12] [13] [14]
481
mulh, J. ACM 38. 929 (1989). E. Bautn and D. Haussler, Neura] Comput. 1, 151 (1989). D. Haussler, Inf. Compulation (to be published). S. Judd, J. Complexity 1, 177 (1988). V. N. Vapnik and A. Y. Chervonenkis, Theory Probab.
6091 Appl. 16, 264 (1971). [15) V N. Vapnik, Estimation r
of Dependences Bated on Em-
pirical Data (Springer-Verlag, New York, 1962). [16] Y. S. Abu-Mostafi, Neural Comput. 1, 312 (1989). [17] A. R. Banna, in Proceedings of the 28th Conference on
Decision and Control (IEEE Control Systems Society, New York, 19B9), Vol. t, pp. 280-285. (18] L. Devioye, IEEE Trans. Pattern Anal. Midi. Intell. 10, S3D (1988). [19] P. Catnevali and S. Patarnello, EurophyB. Lett, 4, 1199 (1987). [20] N. Tishby, E. Levin, and S. Solla, in Proceedings of the International Joint Conference on Neurol Networks (IEEE,
New York, 1989), Vol. 2, pp. 403-409. [21] E. Levin, N. Tishby, and S. A. Solla. Proc. IEEE 78, 1568 (1990). (22) P. del Giudict, S. Pram, and M. A. Virasoro, J. Phys. (Paris) 80, 121 (1989); D. Hansel and H. Sompolinsky, Europhys. Lett. 11, 687 (1990). [23] E. Gardner and B. Deirida, I . Phys. A 22, 1983 (1989). [24] G. Gyorgyi and N. Tishby, /VeuTUl A'elurort) and Spin Glosses, edited by W. K. Theumann and R. Koberle (World Scientific, Singapore, 1990), pp. 3-36. ]25] G. Gyorgyi, PhyB. Rev. Lett. 64, 2957 (1990). [26] W. Krauth, M. Mezard, and J.-P. Nadal, Complex Syst. 2, 387 (1988).
Lett. 85, 1663 (1990). [36] K. Binder and D. W. Heerman, Monre Carlo 5imulaiion in SrOiisIical Mechanic! (Springer-Verlag, Berlin, 1966). [39] S. Kirkpatrick, Jr., C, D. Gelatt, and M. P. Vecchi, Science 220, 671 (1983). [40] S. F. Edwards and P. W. Anderson, J. Phys. F 5, 965 (1975). [41] D. Sherrington and S. Kirkpatrick, Phys. Rev. Lett. 35, 1792 (1975). [42] S. Kirkpatrick and D. Sherrington, Phys. Rev. B 17, 4384 (1978). [43] H. Sompolinsky, Phys. Rev. Lett. 47, 935 (1981). [44] K. Binder and A. P. Young, Rev, Mod. Phys. 58. 801 (1986). [45] M. Mezard, G. Parisi, and M.A. Virasoro, Spin Gloss Theory and Beyond (World Sdentific, Singapore, 1967). [46] F. Rosenblatt, Principles of Neurodynamics (Spartan, New York, 1962). [47] J. R. L. de Almdda and D. J. Thouless, J. Phys. A 11, 129 (1978). [48] A. Krogh and J. A. Hertz, in Advances in Neural Information Processing Systems, edited by R. P. Lippman, J.
[49]
[27] J, A, Hertz, in Statistical Mechanics of Neural Networks: Proceedings of the Eleventh Sitges Conference, edited by
[50]
L. Garrido (Springer, Berlin, 1990), pp. 137-153. [26] D. B. Schwartz, V. K. Saroalam, J. S. Denker, and S. A. Solla, Neural Comput. 2. 374 (1990). [29] 1. Shrager, T. Hogg, and B. A. Huberman, Science 242. 414 (1966). [30] J. Denker, D. Schwartz, B. Wittner, S. Solla, R. Howard, L. Jackel, and J. Hopfield, Complex Syst. 1, 677 (1987), [31] E.L. Thorndike, Fundamentals of Learning (Columbia University Press, New York, 1932). [32] W. Kohler, Geslalr Psychology (Liveright, New York, 1929). [33] W. Kiiuth and M. Mezard, J. Phys. (Paris) 50, 3057 (1989) . [34] S. S. Venkatesh, in Proceedings of the Fourth Annual
[51]
Workshop on Computational
[52] [S3] [54] [55] [56] [57] [S6] [59] [60]
Learning Theory, edited by
M.K. Warmuth and L.G. Valiant (Kaufmann, San Mateo, CA, 1991), pp. 257-266. [35] H. Sompolinsky and N. Tishby, Europhys. Lett. 13. 56T (1990) . [36] I . Kochei and R. Monasson (unpublished). [37] B. Sompolinsky, N. Tishby, and H. S. Stung, Phys. Rev,
482
[61] [62] [63]
E. Moody, and D.S. Touretzky (Kaufmann. San Mateo, CA, 1991), pp. 897-903; J Pliys. A (to be published). M. L. Minsky and S. Papert, Perceptrons (MIT Press, Cambridge, MA, 1988). M. Opper and D. Haussler, Phys. Rev, Lett. 66, 2677 (1991). M. Opper and D. Haussler. in Proceedings of the Fourth Annual Workshop on Computational learning Ttnsc-ry (Ref. [34]), pp. 75-87, E. Gardner, J. Phys. A 19, L1047 (1986). G. Gyfirgyi, Phys. Rev. A 41, 7097 (1990). B. Derrida, Phys. Rev. B 24, 2613 (1981). B. Derrida, R. B. Griffiths, and A. Prigel-Bennett, J. Phys. A 24, 4907 (1991). J, F. Fontanari and R. Meir, Network 2, 383 (1991). T. M. Cover, IEEE Trans. Electron. Comput. 14, 326 (1965). E. Gardner, J. Phys. A 21, 257 (1968). E. Gardner and B. Derrida, J. Phys. A 21, 271 (1988). E. Barkai, D. Hansel, and I . Kanlei, Phys. Rev Lett. 85, 2312 (1990). J. F. Fontanari and R. Koberle, J. Phys. (Paris) 81, 1403 (1990). E. Barkai, D. Hansel, and H. Sompolinsky, Phys. Rev. A 45, 4146 (1992). E. Barkai and I , Kanter, Europhys, Lett. 14, 107 (1991).
Reprinted with permission from Network, Vol. 3. pp. 259-265, 1992 © 1 9 9 2 IOP Publishing Ltd.
On the classification of learning machines Giorgio Parisi Dipartimento di Fisica and 1NFN sezione T o r Vergaia', I I Universila di Roma T o r Vergata', Via E Camevale, Roma 00173, Italy
Received 23 March 1992
Abstract. In this paper we define the entropy of an architecture, which quantifies the propensity of a machine to learn a rule by examples. The distance in Ihe learning propensities of two architectures is also defined, A possible prototype of intelligent behaviour is defined.
1. Introduction The possibility of having machines which learn rules from examples has been the subject of intensive investigations in recent years (see, for example, [1]). A very interesting question is to understand under which conditions the machine is able to generalize from examples, i.e. to learn the whole rule from a partial set of instances. The propensity to generalize is often considered to be a sign of intelligent behaviour (indeed a typical I Q test consists in finding the rule which generates a sequence, given the first few elements of the sequence). The aim of this paper is to discuss how different machines may be classified in relation to their propensity to generalize, and how to distinguish one machine from another in this framework. At the end of the paper we will also present a possible definition of intelligence, i.e. the propensity for arriving at the correct generalization. For simplicity we will discuss only the case where all the variables are discrete (the generalization of the results to the case of continuous variables should not be too difficult).
2. Definition of the framework Before presenting the results it is better to give precise definitions of most of the concepts. In this field the same words have often been used with slightly different meanings. In the following I will consider rules which assign to each input vector of JV Boolean variables (a, = 0 or 1 for i = 1,..., N) an output which consists of a single Boolean value. In other words, a rule is a Boolean-valued function defined on a set of 2 elements (i.e. the set of all the possible values of the variables a); it will be denoted by RfjrT. Some examples of simple rules are the following: N
(a) R[«r] = 0
(ft) (c)
RO]= 1 RO] =
Id) R O ] =
ffl
°i
(e) R O ] = 0 i X O R
rr X O R 2
tr - X O R tr„ 3
483
260 where a X O R b denotes the exclusive or of a and b. A rule may be specified either by some analytic formula as in the previous examples or by explicitly stating the output for all the possible inputs. The number of different rules increases very fast with N: it is given by 2 , where N is the number of different possible input vectors, i.e. 2". In the following we will always consider N to be a large number: terms proportional to l/N will be neglected relative to unity. Nl
2
An intelligent learning machine should be able to learn a rule using only the information represented by the output of the rule in a partial set instances of the input. A learning machine is fully specified if we know its architecture and the learning algorithm. We suppose that the computations that the machine performs depend on M Boolean variables ( J , , k = 1,..., M). In a nutshell, the architecture is a Boolean function A[p, /}, which gives the response of the machine to the input tr's for each choice of the control parameters J . Typical architectures are the perceptron [2] or a neural network, with discretized synaptic couplings. F o r each rule R and the choice of each J's, the machine makes some errors with respect to the rule R. The total number of errors ( E [ R , J ] ) depends on the rule R and on the J's; it is given by
E[R,J]
= I(R[O-]-^[CT,J]) . 2
(1)
W
F o r a given architecture the machine may learn the rule R without errors if and only if there exists a set of J's such that E [ R , J ] = 0. Simple counting arguments tell us that there are rules which cannot be learned without errors by any machine with M smaller that 2" In a learning session we give the machine the information on the values of R [a] for L instances among the
Re[R,J] = I ( R M - ^ [ , J ] )
J
0
(2)
W where the sum is only over the L instances in the tr's. A learning algorithm tries to find the J's which minimize Re[R, J ] . This can be done using explicit formulae for finding the J's (when available) or using a minimization algorithm. In this latter case, very often the learning algorithm reaches only a local, not a global minimum (the computation of the global minimum may be very time consuming). We denote by J * the J's found by the learning algorithm. In order to study the propensity of the machine to generalize it is crucial to consider the number of errors that the machine makes for the unlearned configurations of the tr's (i.e. the global error): Ge[R,J*] = E [ R , J ' ] - Re[R,J*].
(3)
In order to see the effects of the training it is important to define the standard error (Se[R]) as the typical number of errors G e [ R , J ] for a generic choice of the J . For example we could define ln{Se[R]} = l n { G e [ R , J ] }
(4)
484
261 where the bar denotes the average over the J's. If G e [ R , J * ] / S e [ R ] is much smaller than 1, we could say that the number of errors on the n's which have not been presented has decreased as an effect of having learned to apply the rule in a limited set of instances. In this case we say that the machine is able to generalize, at least to a certain extent. Perfect generalization is achieved when G e [ R , J * ] = Oand R e [ R , J * ] = 0 . F o r a given machine the propensity to generalize depends on the rule, and not all rules will be generalized by the machine. O u r aim is to understand how the propensity to leam different rules changes when we change the machine; in this paper we are only interested in the effect of changing the architecture.
3. The Carnevali-Patarnello approach It was suggested by Carnevali and Patarnello in a remarkable paper [3] that, if we suppose that the learning algorithm is quite efficient (e.g. it finds the real minimum of Re[R, J J ) , the propensity of the machine to generalize a given rule (p ) depends only on the architecture. The propensity (ppj may be approximated by the number of different sets of control parameters J for which the total number of errors is zero (apart from an overall normalization factor). In other words R
M
Pj = 2 " I ^ ( E [ R , J ] ) (J)
(5)
t
where E [ R , J ] is given by (1) and the function d is denned as
It is obvious that p
R
depends only on the architecture.
According to Carnevali and Patarnello, rules for which the machine has very small propensity cannot be learned by generalization, while rules with higher propensity will be easier to generalize. In this approach, the propensity of a given architeciure to generalize is summarized by the values of the function p„ for all the 2 arguments (the 4. criteria p„*sPropensity depend only on the architecture, not on the learning algorithms). N !
Our aim is to find some propensity criteria to study in detail the properties of p . Considerations based on statistical mechanics will turn out very helpful. The p are normalized to 1 (£„p = 1); indeed £„t>(E[R, J ] ) = 1 for any J . Indeed there is one and only one rule which corresponds to each choice of the control parameter J. The quantity p can be interpreted as the probability that the rule R is realized after a random choice of the J's. In this probabilistic framework is natural to introduce the entropy of the architecture A B
R
R
R
SLA} = -XPRlntp*).
<7)
485
262 The entropy S M ] is a non-negative number smaller than or equal to ln(2) m i n ( 2 \ M ) . I n order to understand the meaning of entropy in this context let us consider two extreme cases: (a) S[_A] = 0; (b) S[A] = 2". In case (a) the function A {which represents the architecture ,4) does not depend on J and only one rule may be represented: the machine performs always the same computation. In case (b) all rules are equally represented ( M ^ 2 ) and the machine is essentially a digital RAM (random access memory). In both cases it does not perform interesting generalizations. s
We could say that if the entropy is finite for large JV, the machine is able to represent essentially a finite number of rules, while if the entropy is too large, too many rules are acceptable. As a warm-up exercise let us compute the entropy of the perceptron (without hidden unities). In this case (for JV odd) the function A can be denned to be
*\*. J] = ^ { l +
I
sgn^_
^ (1 + M l + J.) ~ | ] }
(8)
where both the tr's and the J's take values 0 or 1. I n this case (N = M) a simple computation shows that all the 2 choices of the J's lead to different rules (i.e. two different J's produce a different output at least in one case) and therefore S[A] = ln(2)JV. The computation of the entropy for the perceptron with hidden units would be much more complicated and we do not attempt it here. It is quite possible that there are other architectures for which the entropy may be computed analytically, but we do not investigate this point any further. We note that the previous definition of entropy could be generalized by introducing a partition function Z(p") denned as follows N
Z W = £ e x p ( J l n ( p ) ) = 5;pS. f
R
(9)
J1
X
We could introduce the entropy S(/J) associated with the partition function Z(/3) of (9): S(p) s d{ln[Z(<3)//3]}/d^.
(10)
The previous defined entropy coincides with S(l). It would be very interesting to find out if there are models in which there is a phase transition in ft and if there is a low-temperature phase in which the replica symmetry is broken. 5. Relative entropies and distance between architectures The value of the entropy tells us something about the propensity of the architecture t o generalize; more precisely, it gives information on how the p 's are distributed. However, many more questions may be posed; in particular we would like to be able to distinguish between architectures which have similar entropies. For example, we could consider two different architectures (e.g. a layered perceptron and a symmetric neural network) with N R
486
263 inputs and one output which have the same entropy (this can be achieved, for example, by adjusting the number of internal layers or hidden neurons). It is natural to ask if these two different architectures are able to generalize the same rules, or if their propensity to generalize is concentrated on rules of very different nature. O u r aim is to define a distance between two architectures, which will help us to compare the different performances. Let us consider two architectures A and B. A first step may consist in defining the entropy of B relative to A as
SIB/A]
- - VpM^M/PdM
(ii)
R
It can be shown that S[B//4] is a non-negative quantity that becomes zero if and only if p (A) = p (B). The relative entropy is not symmetric (i.e. S[fl/,4] * S[_A/BJ). However, if p {A) — p {B) is small (let us say of order e), the leading contribution (of order e ) is symmetric and a non-symmetric term appears at order e . We can define the distance between A and B as R
R
2
R
R
3
d(A, B) = ±{SIB/A-]
+ SIA/B-]}.
(12)
Many similar definitions of the distance are possible. We stress that the name distance for the RHS of (10) is misleading in so far as it does not satisfy triangular inequalities. Using a more precise terminology d{A, B) should be called the difference between A and B. The introduction of a distance allows us to give a precise mathematical answer to the following question. Let us consider two different architectures with the same generalization propensity (i.e. with the same entropy); do they generalize the same rules or are they able to generalize only different rules? It is quite possible that the first architecture performs better on one class of problems, while the second architecture is more efficient on another class of problems. The opposite scenario is also possible: the two architectures perform equally well on the same problems. We can translate this extremely interesting question as follows: if (he entropies of the two architectures are similar, is the distance much smaller than the entropy, or is it of the same order? The question is now well-posed and the answer is given by the function d(A,B). Unfortunately, the explicit computation of this function may be very long and difficult.
6. Intelligence of architectures Having a definition of the distance between two architectures, we can now come to the more speculative question we posed in the introduction: how to define the intelligence of an architecture. If it were possible to define a most intelligent architecture l(o, J), then the intelligence of A can be defined, relative to /, as — d[A, /]/S[yt] (the factor S[,4] has been introduced for normalization purposes). The definition of the intelligent architecture / is the real problem. We suggest that a sequence of the most intelligent architectures is provided by a Turing machine (roughly speaking a general purpose computer) with infinite amount of time for the computation with a code of length L. More precisely the J's are the L bits of a code for the Turing machine (written in a given language) which uses the a as inputs. The
487
264 function I(o, J) is 1 if the program coded by J stops after some time and it is 0 if the program never stops. With this architecture we can compute the function s(R) (i.e. the simplicity of a rule) [4], defined as I3
<>
*R)=pm It may be possible to estimate the function s(R) using the relation s{R)*
2-
C ( R 1
(14)
where C{R) is the algorithmic complexity [ 5 - 8 ] , i.e. the length of the shortest codes which compute the function R [ c ] (equation (12) holds only if C(R) < L). Indeed, if the shortest length of a code which computes R is C(R), we expect that the number of programs of length L which compute R is approximatively 2 ~ " ": we can fill the last L ~ bits of the code by arbitrary bits without affecting the results. It is a common experience that the addiction to arbitrary instructions usually destroys a program, unless such instructions are put after the S T O P instruction.
c
c
t
m
The definition of the architecture / and of the algorithmic complexity clearly depend on the language we use to write the codes. However, the value of the complexity (if it is large) is not very sensitive to the language we use: the compiler of one language can also be written in another language using a finite number of instructions. Therefore, if a rule has a very large algorithmic complexity using a given language, its algorithmic complexity using another language may differ from the previous one by a amount which remains bounded, when the complexity of the rule goes to infinity. F o r very complex rules the algorithmic complexity is an objective quantity which depends weakly on the language chosen. The intelligence of an architecture A can now be defined as the distance from a Turing machine with approximately equal entropy. Recall that the entropy of a Turing machine with a code of length L is asymptotically ln (L), neglecting additive constants. We should compute the distance between the two architectures. We note that if we consider two Turing machines with the same code length {L) using two different languages, their distance remains finite when L goes to infinity, while their entropy diverges. The definition of intelligence becomes independent of the details of the Turing machines for architectures whose entropy is very large. 2
Another very interesting quantity is the logical depth of a rule [9], i.e. the computing time (the number of operations) that is needed to compute the output for all possible inputs by a Turing machine which uses the shortest possible code which defines the rule. Roughly speaking, the minimum number of bits needed to identify the rule is the algorithmic complexity and the time needed to use this information is the logical depth. The logical depth and the algorithmic complexity are in some sense complementary: most of the rules with short (compared to 2") algorithmic complexity have very long logical depth. F o r example, the program for computing the first 1 0 ' ° digits of n can be relatively short (a few lines), but the computer time may be extremely large. The previous observations imply that algorithms which compute the ruie in a relatively short time are likely to be unable to implement many rules with low algorithmic complexity and high logical depth. Many of the most common algorithms (perceptron with or without hidden units, symmetric neural network) belong to this category because the number of operations needed to compute the output for a given input is proportional to a power of JV. For other algorithms (e.g. an asymmetric neural
488
265 network, in which we require the computation of a limit cycle) the computer time needed may be much larger (e.g. proportional to 2 ). It is natural to suppose that this last class of algorithms will be more intelligent than the previous one and will learn more easily rules with low algorithmic complexity and high logical depth [10]. The previous arguments may be summarized as follows. The complexity of a rule is the minimal amount of words we must use to define the rule. Simple rules may be stated very briefly and complex rules have a very long definition. The point of view expressed in this paper is that an intelligent generalization consists in picking the simplest rule which fits the data. The intelligence of an architecture may be measured by its propensity to make this choice. It seems that this definition of intelligence is in good agreement with the intuitive meaning of this word. L
Aeknowledgments It is a pleasure to thank E Brezin, S Cabasino, S Caracciolo, P Paolucci and M A Virasoro for interesting discussions on the subject of this paper, and D Amit and A Sagnotti for a very careful reading of the manuscript.
References [ 1 ] Amit D J 1989 Modeling Brain Functions (Cambridge: Cambridge University Press) Denker J, Schwartz D. Wittner B, Solla S, Horward R. Jackel L and Hopfield J J 1987 Complex Systems 1 877 Somplinsky H and Tishby N 1990 Phys. Rev. Lett. 6S 1683 Seung H S, Sompolinsky H and Tishby N 1991 Statistical mechanics of learning from examples Phys. Reo. A submitted [ 2 ] Minsky M and Papert S 1969 Perceptrons (Cambridge, M A : M I T Press) [ 3 ] Carnevali P and Patarnello S 1987 Europhys. Leu. 4 1199 [ 4 ] Chaitin G J 1975 / . A C M 22 329 [ 5 ] Kolmogorov A N 1965 Problemy Peredachi Informatsii I 3 [ 6 ] Chaitin G J 1966 J . A C M 13 547 [ 7 ] Chaitin G J 1969 J. A C M 16 145 [ 8 ] Chaitin G J Perspectives on Biological Complexity (IUBS Monograph Series no 6) ed O T Solbrig and G Nicolis (available from IUBS secretarial, SI Boulevard de Montmorency. 75016 Paris. France) [ 9 ] Bennet C H 1982 Int. J. Theor. Phys. 21 90S [10] Parisi G and Slanina F 1991 Europhys. Lett. 17 497
489
3d Early Sensory Processing
The three articles in this section share information-theoretical concepts and themes, viz. redundancy reduction, data compression, decorrelation, optima] encoding, mutual information, noise effects. In article (36), the emphasis is on a discussion of information transmission between successive layers of cells. In article (37), it is on image analysis and spatial visual representations. Article (38) focuses on the earlier stage of visual processes at the retinal level. AH three articles emphasise redundancy reduction and decorrelation as underlying principles in the early stages of sensory processing. Thus, this section may be seen as a continuation of section 2b. It has also ties with the previous section (statistical analysis) and the following one (encoding efficiency). The first article (36) of this section, by Linsker, is a review that addresses two important topics. One is the search for simple rules of self-organisation that might explain the development of anatomical structures in brains (areas, stripes, topographic and non-topographic maps; see articles (14), (23), (24), (31), (43)). Much of the modelling about the construction of topographic maps was initiated in the seventies. Linsker reviews the early work and further theoretical and experimental progress. The author's own work in this context, quoted in the article, deserves special mention. He has shown that a layered neural network, with Hebb-like connections between the layers, develops spontaneously feature-detecting cells (centre-surround cells, edge detectors), as a result of a maturation process. Unlike in the case of learning of input-output pairs (cf, section 3c and article (46)), the "learning" process here is unsupervised. The second topic, addressed in this review, concerns the attempts to understand these self-organising patterning rules, in terms of information theoretical optimisation principles. Article (37), by Daugman, reviews several aspects of image encoding: orthogonality and completeness of spatial representations, properties of various basis sets of feature detectors, Gabor functions, and self-similar wavelets. The article mixes engineering considerations (and demonstrations) with neu-
490
rophysiological and psychophysical observations related to cortical processing. T h e discussion is illustrated pictorially. Article (38), by Atick and Redlich, is more mathematically ambitious. It emphasises the adaptation of retinal ganglion cell activities to the statistical properties of natural images, and attempts to find an optimisation principle that combines the two goals of redundancy reduction and noise suppression. T h e theory has probably not yet reached final form, but it is advanced enough to raise sharp questions and to incite confrontations with various biological data on the visual system of primates.
491
Reprinted with permission from Annu. Rev. Neurosci.. Vol. 13, pp. 257-281, 1990 © 1990 Annual Reviews Inc.
PERCEPTUAL NEURAL ORGANIZATION: SOME APPROACHES BASED ON NETWORK MODELS AND INFORMATION THEORY Ralph Linsker I B M Research Division, T . J . Watson Research Center, Yorktown Heights, New Y o r k 10598
INTRODUCTION T o understand neural processing we need to study structure and function al many levels of organization, from subcellular to systemic. W e also need to understand the linkages between levels. First, what are the mechanisms at a lower level that generate structures at a higher one? Second, of all the possible structures that could be formed from the given constituents, only some are in fact generated by the lower-level mechanisms. Are the generated structures optimal, or favored over the other structures, with respect to some property? I f so, we may be able to describe the lower-level mechanisms as implementing an organizing, or optimization, principle. Third, can we account for such putative organizing principles in terms of their adaptive value to the animal? This review explores linkages between lower-level mechanisms and functional architecture in the processing of sensory information. It brings together two lines of study. The first of these is the investigation of how lower-level mechanisms can generate the types of neural structures that are found in the early processing stages of perceptual systems. This approach involves modeling the formation and modification of neuronal connections by simple rules (for example of Hebb type), expressing these rules as a mathematical procedure or algorithm, and using computer simulations or mathematical analysis to determine what structures the rules generate. The
492
258 types of structures or patterns whose formation has been studied include topographic maps, orientation-selective receptive fields, and ocular dominance and orientation columns. T h e second line of study consists of a set of general ideas about how data may be encoded and transformed in a perceptual system, and the informational purposes that these transformations may serve. I review both approaches, then discuss recent results that suggest how the approaches may be unified—that is, how lowerlevel mechanisms may be used to create perceptual processing stages that implement certain types of optimal encoding principles.
FROM SALIENT EXPERIMENTAL FEATURES TO MODELS OF SELF-ORGANIZATION How do specific patterns of neural connectivity and functional architecture develop, how may they be plastically altered, and how do these patterns subserve perceptual functions? A great deal of experimental progress has been made concerning these issues. Theoretical work in this area has several purposes: to seek common rules and principles that may account for a range of observations, to predict new features of neural organization and cell response, and to provide a view of biological information processing that integrates several levels of organization. Experimental evidence indeed suggests that common rules and principles may underlie important aspects of sensory processing. First, cortical regions subserving different processing functions share similar intrinsic structure (Mountcastle 1978). Second, by altering the character of the input to a sensory processing region, one can induce patterns of organization and response properties that differ from those normally found in that region, but in an apparently lawful way (e.g. Constamine-Paton & L a w 1978, K a a s et al 1983, Merzenich et al 1984, Metin & Frost 1989, Rauschecker 1987, S u r e t al 1988). The biological systems of interest exhibit immense complexity. The models to be discussed are by comparison extremely simple. The purpose of this simplicity is to allow us to gain understanding of how underlying rules can generate structure and function, so that essential complexities can then be added in an insightful way.
Types of Pattern-Generating
Models
When a structure or organized pattern is found in nature, various types of patterning models may account for it. T w o extreme types are patterning by "explicit specification' and patterning by a process of so-called "selforganization." I n the first type of model, the pattern is directly determined or strongly influenced by a pre-existing pattern in an underlying substrate. A n example of this type is Sperry's (1943, 1963) chemoaffinity model of 1
493
259 topographic map formation, in which each cell has a specific chemical "address." When such a model is used, the problem of explaining the original pattern's emergence is replaced by the problem of explaining how the underlying pattern of "addresses" itself arises from lower-level rules. I n some models, an explicit specification may determine the pattern in detail; in others, it may determine only certain features of the pattern, such as its overall orientation or its coarse-grained arrangement. In a model of self-organization, on the other hand, the pattern develops from an initially homogeneous structure as a result of processes that incrementally change each element of the system according to a relatively simple set of rules (Turing 1952). Typically such rules are local; that is, each incremental change in one element depends only upon the state of a few other elements. In the generation of certain properties such as topographic maps, both self-organization and a partial form of explicit specification appear to play a role (Udin & Fawcett 1988). This review focuses mainly on the self-organizing aspects of patterning models.
Basic Structure of the Models The models discussed here can all be understood with reference to a common basic structure and set of patterning rules, although the details vary and not all of the components are present in each model. The structure consists of a "source" and a "target" layer of cells, with feedforward connections from source to target and lateral connections between target cells. E a c h connection is characterized by a number called its "strength." The patterning process consists of repeatedly modifying the connection strengths (and in some cases creating or destroying connections) according to the rules until a final configuration develops. Each connection strength is modified in a way that depends upon the "state" of the connected cells. In "marker-based" models the state of a cell is defined as the amount of a marker substance of some type (there may be more than one type of marker). In "activity-dependent" models the state is defined as a measure of neuronal signaling activity such as a firing rate. The patterning rules consist of three parts: (a) a "transmission" rule that determines how each target cell's state depends upon the states of the source cells connected to it (the target cell's state is typically related to an average of the source cells' states weighted by their connection strengths); (b) a "lateral interaction" rule that describes how the states of nearby target cells are modified by interactions within the target layer; (c) an "update" rule that modifies each feedforward (and in some cases lateral) connection according to the degree to which the states of the connected cells are similar.
494
260 I n activity-dependent models, the update rule typically changes each connection strength in a way that depends upon the presynaptic signaling activity and the postsynaptic firing rate or depolarization potential. In this case, the change usually depends upon the degree of correlation between the pre- and postsynaptic quantities, with stronger correlation causing an increase in strength. I refer to this type of neural activity-dependent rule as "Hebb-like" (see B r o w n et al 1990 for review), while recognizing that Hebb's original proposal (Hebb 1949) referred to cell firing, not depolarization, and that it provided no statement of the conditions under which strengths could decrease (cf. Stent 1973). Note that both Hebb-like rules and marker-based rules play similar roles in the patterning process, although the mechanisms to which they refer are very different. In each case a connection is strengthened when the pre- and postsynaptic cells are correlated with each other—either in their signaling activity or in the possession of similar amounts of a marker. It is striking that many salient features of perceptual neural organization emerge in models containing the basic elements described ablve; a positive feedback process (whether marker-based or Hebb-like) in which large connection strength causes the states of the connected source and target cells to be more similar, and greater similarity tends further to increase the connection strength; and a lateral interaction process that*causes the states of nearby target cells to tend to become either more or less similar to each other, depending upon the interaction. Models of neural self-organization developed since the early 1970s have emphasized different aspects of these rules and invoked different constraints or assumptions concerning connectivity and input and output signaling activity. They have been directed toward various goals such as elucidating basic principles of pattern formation, showing how specific features emerge, or modeling specific underlying mechanisms in greater detail. T h e remainder of this section reviews some key ways in which these models have increased our understanding of how simple rules lead to complex structures that resemble those found biologically.
Self-Organizing
Models of Topographic Map Formation
In a topographic map, source and target cells are connected in such a way that positional ordering is preserved. Experimental results on topographic map formation have been recently reviewed (Udin & Fawcett 1988). How are the main features of topographic maps generated in self-organizing models? N E I G H B O R - P R E S E R V I N G MAPS
F o r marker-based models, the essential idea
495
261 is described by the so-called "tea trade analogy" (von der Malsburg & Willshaw 1977, Willshaw & von der Malsburg 1979). In this model, one assumes that the states of nearby source cells are similar. E a c h target cell acquires a state that is the average of the source cells' states weighted by their connection strengths. The lateral interaction then "blends" the states of nearby target cells so that these states become more similar to each other. The update rule strengthens connections between source and target cells having similar states. The result of this process is that nearby source cells map to nearby target cells. How can a topographic map arise in an activity-dependent model (Willshaw & von der Malsburg 1976)? Start with initially random nonnegative feedforward connection strengths. ( F o r simplicity, all pairs of source and target cells can be assumed to be connected. A strength that declines to and remains at zero corresponds to an absent connection in a more realistic model.) Many patterns of signaling activity in the source layer are presented to the network, one at a time. E a c h input presentation consists simply of a localized region, or "spot," of activity against a quiet background. If certain constraints (specific to the model) are met, each such presentation will cause a localized group of target cells to be active. (In particular, the lateral interactions are set up so that the active target cells tend to form a localized region, rather than being dispersed.) The connections between pairs of active source and target cells are then strengthened by a small amount (this is the Hebb-like rule), and compensating reductions are made in the strengths of other connections. After many such spot patterns are presented, the resulting pattern of connection strengths will be topographic; that is, nearby source cells will make their strongest connections to nearby target cells. Kohonen (1982a,b) has described a related algorithm in which a simple geometric computation is substituted for the more detailed properties of the Hebb-like rule and lateral interactions, and a topographic map emerges. F o r each presentation of an input activity "spot," this algorithm finds the particular target cell that fires most strongly, and then changes the target cells' response properties so that the maximally responding target cell and its neighbors will respond more strongly to a spot at the same input location in the future. The algorithm also applies more generally to cases in which the input patterns are ordered according to properties other than spatial position. In both marker-based and activity-dependent models, the degree of selectivity of a target cell—that is, the range of different input patterns to which the cell responds strongly—can depend upon (a) the update rule (e.g. Bienenstock et al 1982); (f>) constraints on total connection strength, or on some other function of the strengths, for each source or target cell
496
262 (Willshaw & von der Malsburg 1976); and (c) the particular form of lateral interaction (Grossberg 1976). L e t us consider several important experimental and theoretical questions concerning topographic maps that go beyond the basic issue, of mapping neighbors to neighbors, discussed above. O V E R A L L M A P O R I E N T A T I O N T h e self-organizing model rules do not by themselves favor one map orientation over another. In biological systems, temporal or chemical mechanisms that favor certain regions to be connected first, or preferentially, may induce an orientation bias. This roughly corresponds to weakly biasing either the initial connection strengths (Willshaw & von der Malsburg 1976,1979) or the patterning rules (Whitelaw & C o w a n 1981, Fraser 1985). Also, the shapes of the source and target layers may favor a particular orientation, owing to boundary effects. More generally, source and target layer boundary conditions can influence map formation in significant ways that are not limited to overall orientation (Schwartz 1977). Models having such biases can be thought of as hybrids of self-organizing and explicit-specification models. Experimental manipulations (reviewed by U d i n & Fawcett 1988) are important for determining the extent to which the biological mechanisms fit either type of model. L O C A L L Y O P T I M A L M A P S T H A T ARE N O T G L O B A L L Y O P T I M A L
I f separate
regions of the network independently become organized, the resulting maps may be out of register along their boundaries or may have conflicting orientations. T o avoid this, one can start with a coarse topographic map that is subsequently refined by a local process. A s an example, a two-step process in which the refinement stage is activity-dependent is found during map regeneration in goldfish (Schmidt 1985). Alternatively, the lateral interaction distance in the target layer can be made initially large, then decreased during development (Kohonen 1982a,b). In a biological system, growth of the target layer relative to the lateral interaction distance could achieve a similar result. Y e t another approach is to limit early map formation to a single region of nucleation (Willshaw & von der Malsburg 1976). C O N T I N U O U S vs. DISCRETE MAPS Some self-organizing models generate topographic maps in which the receptive fields of a sequence of target cells either (a) shift in a continuous and overlapping manner, as described above, or (b) form clusters with discontinuous jumps as one crosses from one column-like target region to another (Takeuchi & Amari 1979). Discontinuous mappings also can emerge in models with more complex dynamics (Pearson et al 1987). M A P M A G N I F I C A T I O N FACTORS
In certain models (Amari 1980, K o h o n e n
497
263 1982a,b, Pearson et al 1987, Linsker 1988b), i f one region of the source layer is stimulated by an activity "spot" more often than another, that source region is mapped onto a larger region of the target layer. F r o m one theoretical standpoint (see A P R I N C I P L E OF M A X I M U M PRESERVATION below)
it can be desirable for the magnification factor of each source region to be proportional to the frequency of "spot" stimulation, so that each target region of given size is activated an equal fraction of the time. Although the Kohonen algorithm described above was originally thought to have this property (Kohonen 1982a,b), it has since been shown not to (Ritter & Schulten 1986, Kohonen 1988). A model that does generate maps having this proportionality property is described by Linsker (1989b), and has been generalized to cases in which the activity patterns are more complex and the magnification factor does not depend simply upon frequency of stimulation.
Ocular Dominance Banded regions of ocular dominance are found in cat and monkey striate cortex ( L e V a y et al 1975, Hubel & Wiesel 1977, Shatz & Stryker 1978) and in frog tectum following implantation of a third eye (ConstantinePaton & L a w 1978). Their formation depends upon correlated electrical activity (Stryker & Harris 1986, Reiter & Stryker 1988). Two questions addressed by self-organizing models are: 1. How does the process of segregation into ocularity domains interact with the process of topographic map formation? 2. Why do patterns ranging from regular stripes to less regular bands or patches form, and what determines their characteristic dimensions? The interaction between topographic map and ocularity domain formation is studied in a marker-based model (von der Malsburg 1979) having three classes of markers: one for ocularity and one each for horizontal and vertical position. The multiple-marker model is an extension of topographic map models to a case in which similarities in properties other than position affect the mapping. The overall "similarity" between the marker content of two cells is defined as a combined measure of the similarities of the markers in each class. The overall effect is to map source to target in such a way that the similarity between nearby target cells is maximized on average. This type of mapping criterion is either explicitly introduced or emergent in other self-organizing models, both marker-based and activitydependent, as well (see for example K o h o n e n I982a,b, 1988, Linsker 1988b, Durbin & Mitchison 1988). Models that incorporate a more detailed interaction between topog-
498
264 raphy and ocularity in map formation include an activity-dependent model (von der Malsburg & Willshaw 1976) in which same-eye input activities are locally correlated and opposite-eye activities are anticorrelated, and a model (Fraser 1985) involving markers that mediate an adhesive interaction, activity-dependent modification, and global positional biasing effects. W h y do ocular dominance stripes and related patterns form? Suppose we ignore visual field position, and simply distinguish between inputs from the two eyes. E a c h target cell receives connections of initially random strength from each eye. N o w let the target cells interact, and their connection strengths change incrementally, according to the following assumed rule (Swindale 1980): E a c h cell's ocularity preference changes to be more nearly like the average preference of its near neighbors, and to be more nearly opposite the average preference of its midrange neighbors. The result is that regions of each ocularity preference form and segregate into (a) locally parallel stripes (with forks and bends) if the lateral interaction rules are unbiased between the two eyes, or (b) islands of one ocularity in a "sea" of the other if the rules favor the latter ocularity to a sufficient degree. Stripes tend to run into the layer boundary at a right angle, consistent with the observed perpendicularity of stripes at the area 17/18 border in, for example, macaque (Hubel & Wiesel 1977). In addition to these simulation results, mathematical analysis of the onset of segregation shows that as stripes start to form, their width has a particular preferred value that depends upon the scale of the lateral interaction (Swindale 1980). Stripe formation has also been analyzed and simulated in a more detailed activity-dependent model in which each target cell receives inputs from source cells lying in a fixed topographic arrangement (Miller et al 1986, 1989).
Orientation-Selective
Cells and Columns
Cells selectively responsive to edges or bars having a particular orientation, and arranged in columns containing cells of similar orientation preference, were discovered by Hubel & Wiesel in cat (Hubel & Wiesel 1962, 1963, reviewed by Fregnac & Imbert 1984) and macaque (Hubel & Wiesel 1968, 1977). In macaque, they are present at birth (Wiesel & Hubel 1974) prior to structured visual experience. Recent advances in optical imaging of electrical activity (Blasdel & Salama 1986) have provided a more detailed picture of the columnar structure in macaque. The detailed mechanisms that mediate orientation-selective response are still unclear (see review by F e r s t e r & K o c h 1987). T h e early experimental findings motivated attempts to explain how orientation selectivity could be generated by activity-dependent self-organ-
499
265 izing models. T o see what has been learned from these models, it is useful to think of the problem in two parts. 1. When the input patterns presented to the network (during the connection modification process) differ from one another with respect to only one property, how does each target cell learn to respond selectively to patterns having a particular value (or range of values) of that property? That is, how does selectivity per se emerge when there is only one pattern selection criterion? 2. When the input patterns differ from one another in many respects, what determines the particular property, or combination of properties, to which each target cell will learn to respond selectively? I n particular, how can orientation selectivity emerge in self-organizing models either in the absence of structured visual experience, corresponding to the prenatal development of well-formed selectivity in macaque, or under the influence of realistic postnatal experience? HOW
DOES S E L E C T I V I T Y
EMERGE?
I n the activity-dependent
models
of
topographic map formation discussed above, topographic ordering arises when each input activity pattern consists of a localized spot. T w o input activity patterns are similar to each other if their spots are at similar positions. The update rule, and in some models the lateral interactions, cause each target cell to become selective for a subset of similar patterns, and the lateral interactions cause nearby target cells to have similar response properties. In the first self-organ izing models to address orientation selectivity (von der Malsburg 1973, Bienenstock et al 1982), each input pattern consists of an arbitrarily oriented bar of activity against a quiet background. (There can be added random noise that does not affect the results.) The bar's shape and center position are the same for all the input patterns. The only measure of pattern similarity is thus bar orientation similarity. Von der Malsburg (1973) considers a layer of target cells with fixed center-surround lateral connections. E a c h source cell is connected to each target cell (topographic mapping is neither present initially nor does it emerge). The result is that most of the target cells become selectively responsive to bars lying at or near some orientation, and nearby target cells tend to become selective for bars of similar orientation. Bienenstock et al (1982) consider a single target cell whose synaptic inputs can be thought of as forming a circular ring whose center is the same as that of each bar pattern. A version of a Hebb-like rule is used in which the change in strength depends upon the difference in value between the postsynaptic activity and a n adjustable threshold. This threshold is
500
266 importani in their model for ensuring that the target cell becomes selective for one bar pattern or a subset of similar patterns. Because of the simple centered-bar form chosen for the input patterns, the emergence of orientation selectivity in both these models closely parallels the emergence of positional selectivity in the formation of topographic maps. In fact, the activity pattern on a semicircular portion of the ring of synapses in the latter model consists of an interval (or onedimensional "spot") of activity whose position is given by the bar orientation. The development of orientation selectivity in this model thus corresponds exactly to the development of positional selectivity in a network having a one-dimensional source "layer" (that has been "wrapped around" so that its endpoints meet). C O L U M N F O R M A T I O N We have noted that von der Malsburg's (1973) model generates target cells whose orientation preferences are similar within a region of the target layer; that is, the cells form orientation "columns." I n the model studied by Swindale (1982), each target cell is labeled by a vector whose magnitude and direction are modified by lateral interactions with the labels at other sites. T h e approach is the same as that used for ocular dominance stripe formation (Swindale 1980), in which each celt was labeled by its degree of left-eye and right-eye preference. The same type of center-surround lateral interaction rule is used, so that each label incrementally changes to become more nearly like its near neighbors and unlike its midrange neighbors. The result is lhat irregularly banded regions of similarly labeled cells emerge. In this model, the questions of why orientation-selective receptive fields arise, and how the particular lateral interaction between labels may arise, are not addressed. By invoking a similar center-surround type of lateral interaction, von der Malsburg & C o w a n (1982) show that if one postulates that different groups of target cells have, for example, horizontal and vertical orientation preference, then the lateral interactions induce the formation of a sequence of cells having the intermediate orientation preferences. W H Y O R I E N T A T I O N SELECTIVITY? Linsker ( I 9 8 6 a - c ) has analyzed the development of a network in which random uncorrelated signaling activity is preprocessed by a layer of cells having topographically arranged centersurround receptive fields. T h e output from this "source" layer of cells is in turn provided as input to a target layer via Hebb-modifiable connections. Even in the absence of lateral connections, a layer of orientation-selective target cells emerges under certain conditions. Since the architecture and development rules possess no orientational bias, Ihe emergence of orientation selectivity is an example of a "symmetry-breaking" process (Linsker 1986b, 1988c). Under different conditions a layer of center-surround cells
501
267 emerges. The two parameters that determine which cell type develops are the radius of the source region that provides input to each target cell, and the degree of correlation needed to cause strength increase via the Hebblike rule. I f the parameter values lie in a range that leads to orientation-selective cell formation, then adding weak lateral connections of fixed strength to the model causes orientation columns to form. In the model, feedforward connections suffice to generate orientation-selective cells. The simulations discussed do not address the question of the extent to which lateral interactions can influence the formation of orientation-selective cells (as opposed to their columnar organization). The emergence of orientation-selective cells and columns in this model is explained by analyzing how the Hebb-like rule creates geometric patterns of connection strengths when the pairwise signal-activity correlations in the source layer depend upon the relative positions of the source cells (Linsker 1986b, 1987). These activity correlations arise because of the center-surround preprocessing. E v e n if one uses a Hebb-like rule different from the particular one used in this model, orientation-selective cells can emerge (see for example K a m m e n & Yuille 1988). The mathematical reason is that the orientation-selective solutions are formed from combinations of the first few eigenfunctions (those having the largest eigenvalues) of the covariance matrix of activity in the source layer (Linsker 1987). Experimentally, one finds a series of several layers of cells having substantially center-surround receptive field properties in retina and L G N of cat and macaque monkey, and layer I V of macaque striate cortex, followed by the onset of well-tuned orientation selectivity. Although the feedforward connections in the model can in general be of both excitatory and inhibitory types, orientation selectivity emerges even when the connections from the center-surround layer are constrained to be excitatory (Linsker 1987, and unpublished results). This is of interest since the geniculocorttcal connections which immediately precede the first well-tuned orientation-selective stage in cat are thought to be exclusively excitatory. In Linsker's model, the center-surround cells responsible for the preprocessing themselves emerge by a self-organizing process, given only random input and topographically arranged connections to each cell from a neighborhood of cells of the previous layer (Linsker 1986a). The means by which the center-surround layer is generated—whether by an activitydependent self-organizing process as in the model, or by retinal interactions mediated by anatomically complex connections—does not affect the emergence of orientation selectivity in the later processing stage of the model.
502
268 T h e model as a whole shows that a sequence of feature-analyzing cell types of progressive complexity can emerge via a Hebb-like rule in a multistage feedforward network. Structured rather than random uncorrelated input can also be provided to the first layer, and the developmental results studied (Linsker 1988a). Models relating to orientation selectivity that are not self-organizing models of cell formation are outside the scope of this review; two examples are (a) a nonadaptive model of cell formation (Braitenberg 1985) and (b) a demonstration, using a "learning by error correction" algorithm, that orientation-selective elements perform useful intermediate processing functions in a network computation of curvature in a scene (Lehky & Sejnowski 1988).
NETWORK MODELS, OPTIMIZATION PRINCIPLES, AND INFORMATION THEORY The results of the preceding section provide a stimulus for investigating why a Hebb-like rule, with or without lateral connections, generates a sequence of feature-analyzing cell types in a layered network. This section explores several senses in which the types of patterning rules studied can create structures that are optimal from an information processing standpoint.
What Does a Hebb-like Rule Optimize? First consider a single cell with feedforward but no lateral connections. T o understand intuitively the effect of a Hebb-like rule, an analogy is useful; Imagine a committee whose recommendation ("output activity") on any issue is the weighted average of its members' opinions ("input activities"), each opinion being weighted according to the member's voting strength. A member who consistently agrees (disagrees) with the committee's opinion receives an increase (decrease) in voting strength. The operation of this Hebb-like rule transforms a committee of members having random voting strengths, whose averaged output is rarely a strong recommendation (either positive or negative), into a committee whose members of high voting strength tend to agree more often, and whose recommendations tend to be strong (in either direction) more of the time. What does this result mean for a mode! cell (Linsker 1988a)? Consider a cell whose output activity is a linear function of its input activities. A histogram of the output values over a range of input presentations will have a spread, or variance, that depends upon the connection strengths. I f the Hebb-like rule has a term that changes each strength by an amount proportional to the product of (a) the input activity at that connection
503
269 (minus its mean value) and (A) the cell's output activity (minus its mean value), this term will tend to increase the cell's output variance. Depending upon the other terms in the update rule, the net result can be to maximize the variance subject to some constraint, for example on the sum of the strengths (Linsker 1986b, 1988a,c) or the sum of the squares of the strengths (Amari 1977, O j a 1982). A cell with low output variance tends to be nonselective—its responses to different input presentations are similar. High variance tends to correspond to greater selectivity. Even when the cell's output variance is itself constrained (e.g. by a maximum allowed firing rate), or when the cell's response is nonlinear, a suitable Hebb-like rule can act to maximize the cell's responsiveness to statistical structure (or "features") in the input presentations, and to minimize its responsiveness to uncorrelated inputs or random processing noise (Linsker 1988a). Under certain conditions, a Hebb-like rule generates cells that are optimal in several ways. A s we have seen, such a rule produces cells that respond selectively to statistically significant properties of the input presentations. In particular, a suitably constructed rule (Oja 1982) produces a cell that performs a standard computation in statistical feature extraction called "principal component analysis." [Watanabe (1985) discusses in detail the application of this and related statistical methods to pattern recognition.] A cell produced by such a Hebb-like rule also has an "optimal inference" property: The average error incurred when using the cell's output value to estimate its input values is less for such a cell than for any other linear response cell. Finally, under certain conditions, the output from such a cell conveys maximum information about its input activity values, compared with cells having arbitrary connection strengths (Linsker 1988a).
The Role of Lateral Interactions If a Hebb-like rule can cause individual target cells to develop so as to optimize certain properties, can the addition of lateral interactions allow an assembly, or an entire layer, of target cells to develop in an optimal or near-optimal way with respect to some property that is important for information processing? When both feedforward and lateral connections are present, target cells not only can develop feature analyzing properties by means of a Hebblike rule (as shown above), but can "cooperate" or "compete" with each other in the formation of these properties (see, for example, von der Malsburg 1973, Grossberg 1976, Kohonen 1982a,b, Rumelhart & Zipser 1985, Linsker 1986c, 1988a,b, 1989b, Pearson et al 1987, Hinton 1989). A n optimization method (Durbin & Willshaw 1987) based on the "tea trade" model (von der Malsburg & Willshaw 1977) has been developed.
504
270 In some of these models the process of modifying the connection strengths corresponds closely to a statistical method for classifying or detecting regularities in data sets (see Lippmann 1987 for review). F o r example, a network algorithm for forming clusters of related input presentations, developed in connection with the "adaptive resonance theory" (Carpenter & Grossberg 1988), is related to the "sequential leader clustering" algorithm in statistics. Kohonen's (1982a,b) algorithm for "feature map" formation, discussed above, is related to a version of the '"k-means" algorithm (MacQueen 1967) for partitioning a set of data points into a number of groups such that the members of each group are similar to each other according to a particular measure. This type of partitioning is useful for extracting common features from raw data, and also for the data compression problem of communicating an item of data, by stating which group it belongs to, in such a way that the original datum can be inferred with minimum error. [Cf. the "optimal inference" result of the previous section, the literature on "vector quantization" (reviewed by G r a y 1984), and the work of Watanabe (1969, 1985) on feature extraction.] I discuss the choice of "similarity" measure further in the next section. Might some type of statistical algorithm be playing an important roie in the biological processing of sensory data? I f so, what property if any is being optimized, how can the optimization process be realized by a biological network, and of what value is the process to the animal? Although these questions are necessarily speculative, I discuss here a concrete approach to them, in which some basic ideas from information theory play an important role (Linsker 1988a). T o place recent connections between information theory and perceptual network algorithms in perspective, it is important to review the history, since the 1950s, of the idea that information theory may relate to the organization of sensory and perceptual processing. The relevant information-theoretic ideas are discussed here in an informal way. F o r the classic development of information theory see Shannon (1949); a tutorial treatment of some of the ideas as they relate to sensory processing is given in Linsker (1988a).
Information Theory and Perception REDUNDANCY REDUCTION Sensory input, such as a time sequence of visual scenes, is not random and uncorrelated, but contains structure and regularities. K n o w i n g certain aspects of the input allows us to infer other aspects; the input is partially redundant. Attneave (1954, p. 189) proposed that A major function of the perceptual machinery is lo strip away some o f the redundancy of stimulation, to describe or encode incoming information in a form more economical than lhat in which it impinges on the receptors.
505
271 The suggested approach is "'equivalent to that of a communications engineer" who exploits regularities in an input scene, in order to encode the input signals so that they may be sent along an essentially noise-free channel of limited capacity and reconstructed by the recipient with high fidelity (low distortion). [For examples of how the computations performed by cells during early visual processing can be analyzed from the standpoints of data compression and reconstruction, see Daugman (1988) and Oguztoreli & Caelli (1986).] Attneave's recognition that encoding for redundancy reduction is related to the identification of specific features such as edges or corners in a scene, and that a redundancy-reducing strategy might be important for perceptual processing, is an important insight. The idea is related to Craik's (1943) view of the brain as building a model of the external world that incorporates the world's lawful regularities and constraints. Attneave's idea of sensory processing as data compression, however, leaves some crucial questions unanswered: What role if any is played by the "recipient" in the communications analogy? I f the original scene is never "reconstructed" by the brain (and why should it be?), what is the meaning of the fidelity criterion? That is, if not all input information can be preserved, what criterion determines what to discard? Finally, how do biological constraints and costs influence what types of encoding are "economical," and how does a biological system carry out the encoding? In a series of papers. Barlow has studied several of the physiological correlates of redundancy reduction as a perceptual processing strategy: 1. In a model having discrete (e.g., binary) inputs, with no processing noise to induce coding or transmission errors, one can encode the input "message" by removing correlations (a type of redundancy) and thus produce an output that is more compact than the input, yet is able to regenerate the input in a completely reversible way (Barlow 1959, 1961). The compaction process can be driven by a constraint limiting the average firing rate to a low value in later processing stages (Barlow's principle of "economy of impulses"). Note that as long as the code is fully reversible, the question of what fidelity measure to use does not arise. 2. When the input signals are continuous-valued (or can take on a very large number of values), it is important to treat processing elements having limited resolution (Barlow 1969). This, however, leaves open the question of when two stimuli should be considered "similar" enough not to be resolved by the network, and of how "similarity" with respect to different properties should be compared. The issue is related to the choice of appropriate fidelity measure (above), and recurs in connection with Marr's work (below). 3. Building on Hebb's (1949) discussion of the importance of identifying
506
272 correlated inputs, one can regard a Hebb-like rule as forming cells that detect "suspicious coincidences" (Phillips et al 1984, Barlow 1985). These "coincidences" are sets of events that signal the existence of structure (e.g. coherent motion of an object), since they would not jointly occur in the absence o f such structure. Legendy (1970, 1975) discusses a similar idea, that of forming cells that respond selectively to "surprising" combinations of stimuli, in the context of a suggestion that individual cells may possess complex memory and processing capabilities to carry out the requisite computations. 4. T h e existence of multiple cortical areas with different mappings allows different types of regularities to be detected, i f one assumes that a constraint on connection length requires cells to be relatively close together for correlations in their activity to be detected, and consequently for redundancy to be reduced (Barlow 1986). T H E I N F O R M O N M O D E L A n information-theoretic idea is explicitly used as the motivation for a synaptic learning rule in the "informon" model of Uttley (1970, 1979). T h e "mutual information" (Shannon 1949) between two messages is, informally, a measure of the information that either message "conveys about" the other. F o r example, if two signals tend to co-occur, their mutual information is high. I f knowing one signal has no effect on one's expectation of what the other signal is, their mutual information is zero. I n Uttley's model both variable and fixed strength synapses provide input to a cell. E a c h variable strength is incrementally adjusted so that the cell's output becomes as nearly decorrelated with the synaptic input as possible. T h e update rule is motivated by the idea of minimizing the mutual information between input and output at each synapse, although in practice a function different from the mutual information is used. This learning rule is very similar to the Rescorla-Wagner conditioning rule and to the Widrow-Hoff " L M S " algorithm for supervised learning (Widrow & Stearns 1985). The rule adjusts the strengths so that the net effect of the inputs at the variable-strength synapses is to cancel, as nearly as possible, the net reinforcement or "teacher" signal at the fixed-strength synapses. T h e output of the "informon" cell is then equal to the residual difference between the two sets of signals. This rule differs from a Hebb-like rule, in which correlations between input and output cause synaptic strength to increase. Nonetheless, the rule causes the cell to detect structure among the inputs to the variable-strength synapses, and to use such structure to match and cancel the "teacher" signal most effectively. I return to the use of a mutual information criterion below (Linsker 1988a) in a different context. I N F O R M A T I O N THEORY A N D BIOLOGICAL U T I L I T Y
507
Marr's (1970) discussion
273 of redundancy reduction in sensory processing addresses an issue (introduced above) that is crucial for any connection between informationtheoretic or statistical classification schemes and perception. T o form generalizations, a system must be able to decide when two events (input presentations) are "similar"—that is, belong in the same class. This "lumping process," however, discards information about the events. By what criterion should this, rather than other, information be discarded? M a r r notes that system reliability offers one such criterion: I f two events correspond to sufficiently similar sets of signal activities, the events could be confused by the network, and therefore should not be classified differently. He does not, however, consider this a fundamental criterion. Marr seeks to integrate information-theoretic ideas with biological utility in the following way. His criterion for discarding information is that the loss of that information should not impair the system's ability to diagnose whether the event possesses features of a certain type. The type of feature that is favored is a feature whose presence or absence (in an event) is not sensitive to small changes in the set of other, already-diagnosed features that are present (in the same event). Thus the presence of a feature (or type of redundancy) as such is not sufficient ground for classification; the feature must tend to co-occur with a sufficiently large class of other features (see the "Fundamental Hypothesis," Marr 1970, p. 182). It is intuitively plausible that the detection of a feature is likely to have greater biological utility if that feature tends to be associated with others. Marr's (1970) theory is developed substantially beyond this point to make neurophysiologic predictions concerning the roles of various cell types. Important questions that are left open include: C a n the theory be used to make specific predictions of feature-analyzing properties? I f not, would the incorporation of additional biological constraints of some type suffice to allow one to make such predictions? Is the proposed feature classification criterion definite enough that it can be used, even in principle, to construct a perceptual system (or the early stages of one) whose functioning might be compared with what is biologically observed? A PRINCIPLE OF M A X I M U M I N F O R M A T I O N PRESERVATION
Motivated by
the
finding (discussed above) that a Hebb-like rule under certain conditions generates a cell whose output conveys maximum information about its input activities, Linsker (1988a) has proposed an information-theoretic principle for the organization of a biological sensory processing stage having source and target cell layers. This principle of "maximum information preservation" states that the transformation of sensory signals from source to target layer should be chosen such that the target cell activities jointly convey maximum information about the source cell activi-
508
274 ties. The quantity to be maximized is the average (over time) of the mutual information between the two sets of activities. The choice of transformation is subject to biological "hardware" constraints. These may include constraints on (a) the type of function each cell can compute (e.g. a linear combination of inputs, or a sigmoid function of the linear combination, or some more complicated function), (b) the spatial extent of lateral connections, and (c) the reliability of signal transmission and processing. The concept of maximizing average mutual information between input and output can be made more concrete by an analogy to a "guessing game" (cf. Shannon 1949) such as "Twenty Questions." In that game, the questioner's proper strategy is to ask, at each stage, a question whose answer will convey, on average, the maximum possible amount of information (in that case, one bit of information) about the unknown object. In this analogy, the "questioner" is the cell, the "question" is the computation being performed by the cell upon its inputs (e.g. a center-surround cell "asks" the "question": "What degree of contrast is present between the central and peripheral regions of my receptive field?"), and the "answer" is the output activity of the cell. A n unreliable "answer" corresponds to the effect of noise associated with the processing stage. We want to know what set of processing functions, or computations, emerges for an entire layer of cells, according to the principle of maximum information preservation. This is analogous to asking: How should a set of questioners, asking questions in parallel about the same object (the input presentation), choose their questions optimally? I f the activity values could take on only discrete values, and if noise could be ignored, the proposed principle would lead to data compression with redundancy reduction along the lines discussed by Attneave and Barlow (above). Processing noise is unavoidable, however, and it has important consequences. It helps to determine which types of information about the input pattern are preserved or discarded, and how redundancy is to be introduced—as well as removed—during processing. The latter point is important if one is to account for the similar response properties of nearby cells in the context of an "optimal encoding" principle. The optimal choice of cell response properties, according to the principle of maximum information preservation, depends upon the types of statistical regularities present in the set of input presentations, and on the processing constraints. The principle has been applied to simple model networks under various conditions. It generates features that are qualitatively similar to those found in biological systems. These features include topographic maps (Linsker 1988b, 1989b; also see M A P M A G N I F I C A T I O N FACTORS above), center-surround cells and cells sensitive to temporal variations in
509
275 input activity (Linsker 1989a), orientation-selective cells, and column-like assemblies of cells having similar response properties (Linsker 1988a,b, 1989b). In each case the generic quantity being optimized—the average mutual information between source and target layer—is the same. Note that the quantity being maximized is a function of the processing being carried out by many target cells. This global optimization criterion stands in contrast to other proposed criteria that refer only to an information-theoretic quantity at a single connection (Uttley 1970,1979) or cell (Pearlmutter & Hinton 1986). A different global information-theoretic quantity is optimized in the "Boltzmann machine" network discussed by Hinton & Sejnowski (1983). The idea of choosing system parameters so as to maximize an appropriately defined mutual information has been used in several nonbiological contexts, including the choice or placement of sensors in physical systems (e.g. Phua & Dillon 1977 and references therein, Luttrell 1985, Fraser & Swinney 1986) and the choice of acoustic-processing models in speech recognition (Bahl et al 1987). M u c h remains to be learned about the patterns of information processing that can emerge from this type of in form at ion-theoretic principle, particularly in nonlinear systems having multiple processing stages. The potential value of cross-fertilization between neuroscience and nonbiological fields, in advancing our understanding of these issues, is great. Although the average mutual information between two layers of cells is a global property having a complex mathematical form, a local algorithm that implements the proposed principle has recently been developed for certain types of model networks having feedforward and lateral connections (Linsker 1989b). By a local algorithm we mean one that modifies connection strengths in a way that depends only upon signals available at that connection or cell. This algorithm, or learning rule, was derived by asking: How should the response properties of each cell be incrementally adjusted, so as to increase the average mutual information between the source and target layer activities by the greatest amount? (It is an example of a "gradient ascent" learning rule.) It is striking that the resulting rule (Linsker 1989b) exhibits properties of Hebb-like modification and cooperative and competitive learning, combined in a particular way, even though no assumptions were made concerning the form of the learning rule or its component properties. The principle of maximum information preservation has some theoretically attractive features (Linsker 1988a, 1989b), and the existence of local algorithms increases our confidence that biological networks may be capable of implementing it. It does not follow from this that nature in fact makes use of the principle. It will be necessary to see what detailed pre-
510
276 dictions the principle generates in biologically realistic situations, and to subject these to experimental test. The proposed principle may be usefully extended in various ways. F o r example, the principle as stated uses processing noise as the criterion for classifying two patterns as "similar." (That is, "similar" patterns are those which cannot be reliably discriminated from each other.) It can be biologically useful to introduce a different criterion of pattern "similarity." A suitable choice of criterion can facilitate the learning of generalizations or allow resolution to be varied as the focus of attention is shifted. The type of local algorithm mentioned above can still be used to generate nearoptimal processing stages provided the "similarity" criterion satisfies certain conditions (Linsker 1989b). [For examples of how the choice of resolution or "similarity" measure is handled in other approaches to learning and perception, see Carpenter & Grossberg's (1988) use of an adjustable "vigilance" parameter, and the "regularization" method for reconstructing aspects of a scene from its image (Poggio & K o c h 1985).]
SUMMARY AND FUTURE DIRECTIONS This review has focussed on a set of ways in which experimental work, modeling, and theory have interacted in advancing our understanding of the organization of sensory processing systems. These interactions are not of a rigid "bottom-up" or "top-down" type. They involve interplay among—rather than isolated study of—the "computational task," algorithmic, and hardware levels of description (Marr 1982). The path traced here comprises the following steps: 1. T h e finding of specific salient experimental features motivates the search for models, especially self-organ izing models, that can generate these features. 2. C o m m o n elements of architecture and patterning rules are found to underlie a variety of these models. 3. These elements are found, in certain cases, to be associated with optimization principles that appear significant from an information-processing standpoint. I n essence, the models are treated as objects of study in their own right, and are found to have properties that might have been difficult or impossible to infer directly from observation of the much more complex biological system. This study leads to links with principles and methods in statistical analysis and information theory. 4. T o be able to test whether these putative principles are biologically relevant, it is necessary to develop biologically plausible algorithms that can implement them, and to generate explicit predictions of cell
511
277 properties and neural maps that can be compared with experiment. This step has been taken for simple model cases. More work is required to extend the principle and algorithms to networks and sets of input presentations that are of biologically realistic complexity. If some of the theoretical ideas discussed withstand future experimental tests, it will become important to extend the work in two directions. First, new experimental findings concerning neuronal dynamics (e.g. G r a y & Singer 1989) could allow "'optima! encoding" strategies to be implemented in new ways. Second, even if the principles discussed are found to describe some of the main effects of feedforward and lateral interactions within a processing stage of a sensory system, it will be important to understand how these principles relate to higher levels of neural organization. The study of attentional mechanisms, other effects of feedback from later processing states, integration across sensory modalities, and sensorimotor integration are examples of such higher-level organizational issues to which the type of approach explored here might be fruitfully extended. More sophisticated experimental techniques (e.g. for recording and analyzing complex spatial and temporal patterns of activity), increased computational power, and refinement of theoretical ideas are likely to lead to closer and more fruitful contact between experimental and theoretical work. F o r the theorist, it is important to use experimental findings to (a) constrain the essential features of biological models, (A) inspire and inform the choice of key questions, and (c) enable comparisons with predicted structure and behavior. F o r the experimentalist, model predictions may suggest appropriate candidate stimuli for studying receptive field properties and neural maps, and may help to guide the analysis and interpretation of complex activity data. Such guidance becomes valuable as increasingly complex sets of response properties are found, and as the characterization of a cell as a detector or analyzer of a well-defined "feature" may become more difficult or untenable. More generally, if we are to gain an understanding of how and why neural systems are organized in particular ways, it is important to supplement descriptive principles with a developing theoretical framework that can be used to generate testable predictions and to relate structure and function at a variety of organizational levels. Multifaceted interactions among experiment, modeling, and theory will play an important role in achieving these goals. ACKNOWLEDGMENTS
I thank D r s . E r i c Schwartz, Roger Traub, and Christoph von der Malsburg for their comments on the manuscript.
512
278 Literature
Cited
A m a r i , S.-I. 1977. Neural theory of association and concept-formation. Biol. Cybern. 26: 175-85 A m a r i , S.-I. 1980. Topographic organization of nerve fields. Bull. Math. Biol. 42: 339¬ 64 Attneave, F . 1954. Some informational aspects of visual perception. Psychol. Rev. 61: 183-93 Bahl, L . R., Brown, P. F.. de Souza, P. V . , Mercer, R. L . 1987. Speech recognition with continuous-parameter hidden Markov models. Comput. Speech Lang. 2:219¬ 34 Barlow, H . B. 1959. Sensory mechanisms, the reduction o f redundancy, and intelligence. In Mechanisation of Thought Processes, Natl. Phys. Lab. Symp. 10: Paper4-1. London: H M S O Barlow, H . B. 1961. Possible principles underlying the transformations of sensory messages. I n Sensory
Communication,
Craik, K . J. W. 1943. The Nature of Explanation. Cambridge: Cambridge Univ. Press Daugman, J. G . 1988. Complete discrete 2¬ d Gabor transforms by neural networks for image analysis and compression. IEEE
ed.
W. A . Rosenblith, pp. 217-34. Cambridge: M I T Press and New Y o r k / L o n don: Wiley Barlow, H . B. 1969. Trigger features, adaptation and economy of impulses. I n Information
Processing
in the Nervous
Sysiem,
ed. K . N . Leibovic, pp. 209-26. New Y o r k : Springer-Verlag Barlow, H . B. 1985. Cerebral cortex as model builder. I n Models of the Visual Cortex, ed. D . Rose, V. G . Dobson, pp. 37^16. New Y o r k / L o n d o n : Wiley Barlow, H . B. 1986. Why have multiple cortical areas? Vision
Res. 26:
81-90
Bienenstock, E. L . , Cooper, L . N . , M u n r o , P. W . 1982. Theory for the development of neuron selectivity: Orientation specificity and binocular interaction in visual cortex. J.
Neurosci.
2: 32^18
Blasdel, G . G . , Salama, G . 1986. Voltagesensitive dyes reveal a modular organization in monkey striate cortex. Nature 321: 579-85 Braitenberg, V. 1985. Charting the visual cortex. In Cerebral Corlex, ed. A . Peters, E. G . Jones, 3: 379-414, New Y o r k / London: Plenum Brown, T. H . , Kairiss, E. W., Keenan, C. L . 1990. Hebbian synapses: Biophysical mechanisms and algorithms. Annu. Rev. Neurosci. 13: 475-511 Carpenter, G . A . , Grossberg, S. 1988. The A R T o f adaptive pattern recognition by a self-organizing neural network. Computer 21(3): 77-88 Const ant ine-Palbn, M „ Law, M . t 1978. Eye-specific termination bands in tecta o f three-eyed frogs. Science 202: 639¬ 41
513
Trans.
Acoust.
Speech
Signal
Process.
36:
1169-79 Durbin, R., Mitchison, G. 1988. Dimension reduction models for topographical maps in
the cortex. Neural
Networks
l(Suppl.
l):491(Abstr.) * Durbin, R., Willshaw, D . J. 1987. A n analogue approach to the travelling salesman problem using an elastic net method. Nature
326: 689-91
Ferster, D . , Koch, C. 1987. Neuronal connections underlying orientation selectivity in cat visual cortex. Trends Neurosci. 10: 487-92 Fraser, A . M . , Swinney, H . L. 1986. Independent coordinates for strange attractors from mutual information. Phys. Rev. A 33: 1134-40 Fraser, S. E. 1985. Cell interactions involved in neuronal patterning: A n experimental and theoretical approach. In Molecular Bases
of Neural
Development,
ed. G . M .
Edelman, W. E. Gall, W. M . Cowan, pp. 481-507. New Y o r k : Wiley Fregnac, Y., Imbert, M . 1984. Development of neuronal selectivity in primary visual cortex of cat. Physiol. Rev. 64: 325^134 Gray, C. M . , Singer, W . 1989. Stimulusspecific neuronal oscillations in orientation columns o f cat visual cortex. Proc. Natl. Acad. Sci. USA 86: 1698-1702 Gray, R. M . 1984. Vector quantization. IEEE
ASSP
Mag.
1(2): 4-29
Grossberg, S. 1976. On the development of feature detectors in the visual cortex with applications to learning and reactiondiffusion systems. Biol. Cybern. 21: 145— 59 Hebb, D . O. 1949. The Organization of Behavior. New York: Wiley Hinton, G. E. 1989. Connectionist learning procedures. Artif. Intel!. 40: 185-234 Hinton, G. E., Sejnowski, T. J. 1983. Optimal perceptual inference. Proc. IEEE Conf. Comput. Vision, pp. 448-53 Hubel, D . H , , Wiesel, T. N . 1962. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. J. Physiol.
160: 106-54
Hubel, D . H . , Wiesel, T. N . 1963. Shape and arrangement o f columns i n cat's striate cortex. J . Physiol. 165: 559-68 Hubel, D . H . , Wiesel, T. N . 1968. Receptive fields and functional architecture o f monkey striate cortex. J . Physiol. 195: 215-43 Hubel, D . H . , Wiesel, T. N . 1977. Functional
279 architecture of macaque monkey visual cortex (Ferrier lecture). Proc. R. Soc. London Ser. B 198: 1-59 Kaas, J. H „ Merzenich, M . M . , Killackey, H . P. 1983. The reorganization o f somatosensory cortex following peripheral-nerve damage in adult and developing mammals. Annu. Rev. Neurosci. 6: 325-56 Karnmen, D . M . , Yuille, A . L . 1988. Spontaneous symmetry-breaking energy functions and the emergence of orientation selective cortical cells. Biol. Cybern. 59: 23-31; (erratum) 59: 430 Kohonen, T. 1982a. Self-organized formation of topologically correct feature maps. Biol. Cybern. 43: 59-69 Kohonen, T. 1982b. Analysis of a simple self-organ izing process. Biol. Cybern. 44: 135^J0 Kohonen, T. 1988. Self-Organization and Associative Memory. Berlin/Heidelberg/ New York: Springer-Verlag. 2nd ed. Legendy, C. R. 1970. The brain and its information trapping device. I n Progress of Cybernetics, ed. J. Rose, I : 309-38. London/New York/Paris: Gordon & Breach Legendy, C. R. 1975. Three principles of brain function and structure. Int. J. Neurosci. 6:237-54 Lehky, S. R., Sejnowski, T. J. 1988. Network model of shape-from-shading: neural function arises from both receptive and projective fields. Nature 333: 452-54 LeVay, S., Hubel, D . H . , Wiesel, T. N . 1975. The pattern of ocular dominance columns in macaque visual cortex revealed by a reduced silver stain. J. Comp. Neurol. 159: 559-76 Linsker, R. 1986a. From basic network principles to neural architecture: Emergence of spatial-opponent cells. Proc. Natl. Acad. Sci. USAM: 7508-12 Linsker, R. 1986b. From basic network principles to neural architecture: Emergence of orientation-selective cells. Proc. Natl. Acad. Sci. USA 83: 8390-94 Linsker, R. 1986c. From basic network principles to neural architecture: Emergence o f orientation columns. Proc. Natl. Acad. Sci. USA 83: 8779-83 Linsker, R. 1987. Towards an organizing principle for perception: Hebbian synapses and the principle of optimal neural encoding. IBM Res. Rep. RCI2830. I B M Res., Yorktown Heights, N Y Linsker, R. 1988a. Self-organization in a perceptual network. Computer 21(3); 105-17 Linsker, R. 1988b. Towards an organizing principle for a layered perceptual network. In Neural Information Processing Systems, Denver, CO. 1987, ed. D . Z . Anderson, pp. 485-94. New York: A m . Inst. Phys. Linsker, R. 1988c. Development of feature-
analyzing cells and their columnar organization in a layered self-adaptive network. In Computer Simulation in Brain Science, Copenhagen, 1986, ed. R. M . J. Colterill, pp. 416-31. Cambridge/New York: Cambridge Univ. Press Linsker, R. 1989a. A n application o f the principle of maximum information preservation to linear systems. I n Advances in Neural Information Processing Systems, Denver, CO, 1988, ed. D . S. Touretzky, 1: 186-94. San Mateo, Calif: Morgan Kaufmann Linsker, R. 1989b. How to generate ordered maps by maximizing the mutual information between input and output signals. Neural Comput. 1: 402-11 Lippmann, R. P. 1987. A n introduction to computing with neural nets. IEEE ASSP Mag. 4(2): 4-22 Luttrell, S. P. 1985. The use of transinformation in the design o f data sampling schemes for inverse problems. Inverse Problems I : 199-218 MacQueen, J. 1967. Some methods for classification and analysis of multivariate observations. In Proc. 5lh Berkeley Symp. Math. Stat. Prob., ed. L . M . LeCam, J. Neyman, I : 281-97 Berkeley: Univ. Calif. Press Marr, D . 1970. A theory for cerebral neocortex. Proc. R. Soc. London Ser. B 176: 161-234 Marr, D . 1982. Vision. San Francisco: Freeman Merzenich, M . M „ Nelson, R. J., Stryker, M . P., Cynader, M . S., Schoppmann. A . , Zook, 1. M . 1984. Somatosensory cortical map changes following digit amputation in adult monkeys. J. Comp. Neurol. 224: 591-605 Metin, C , Frost, D . O. 1989. Visual responses of neurons in somatosensory cortex o f hamsters with experimentally induced retinal projections to somatosensory thalamus. Proc. Natl. Acad. Sci. USA 86: 357-61 Miller, K . D „ Keller, J. B., Stryker, M . P. 1986. Models for the formation o f ocular dominance columns solved by linear stability analysis. Soc. Neurosci. Abstr. 12: 1373 Miller, K . D . , Keller, J. B.. Stryker, M . P. 1989. Ocular dominance column development: Analysis and simulation. Science 245: 605-15 Mountcastle, V. B. 1978. A n organizing principle for cerebral function: The unit module and the distributed system. I n The Mindful Brain, ed. G. M . Edelman, V . B. Mountcastle. Cambridge: M I T Press Oguztoreli, M . N . , Caelli, T . M . 1986. A n inverse problem in neural processing. Biol. Cybern. 53: 239-45
514
280 Oja, E. [982. A simplified neuron model as a principal component analyzer. J. Math. Biol. 15: 267-73 Pearlmutter, B. A . , Hinton, G . E. 1986. Gmaximization: A n unsupervised learning procedure for discovering regularities. I n Neural Networks for Computing, ed. J. S. Denker, A I P Conf. Proc. I S l . N e w Y o r k : A m . Inst. Phys. Pearson, J, C , Finkel, L . H . , Edelman, G. M . 1987. Plasticity in the organization o f adult cerebral cortical maps: A computer simulation based on neuronal group selection. J. Neurosci. 7: 4209-23 Phillips, C, G „ Zeki, S., Barlow, H . B. 1984. Localization of function in the cerebral cortex. Brain 107: 327-61 Phua, K . . Dillon, T. S. 1977. Optimal choice of measurements for state estimation. I n 1977 Power Industry Computer Applic. Conf, Toronto, pp. 431-41. New York: IEEE Poggio, T., Koch, C. 1985. Ill-posed problems in early vision: From computational theory to analogue networks. Proc. R. Soc. London Ser. B 226: 303-23 Rauschecker, J. P. 1987, What signals are responsible for synaptic changes in visual cortical plasticity? I n Imprinting and Cortical Plasticity, ed. J. P. Rauschecker, P. Marler, pp. 193-219. New York: Wiley Reiter, H . 0 „ Stryker. M . P. 1988. Neural plasticity without postsynaptic action potentials: Less-active inputs become dominant when kitten visual cortical cells are pharmacologically inhibited. Proc. Natl. Acad. Sci. USA 85: 3623-27 Ritter, H . , Schulten, K . 1986. On the stationary state of Kohonen's self-organizing sensory mapping. Biol. Cybern. 54: 99-106 Rumelhart, D . E., Zipser, D . 1985. Feature discovery by competitive learning. Cogn, Sci. 9: 75-112 Schmidt, J. T. 1985, Formation of retinotopic connections: Selective stabilization by an activity-dependent mechanism. Cell. Mol. Neurobiol. 5: 65-84 Schwartz, E. L . 1977. The development of specific visual connections in the monkey and the goldfish: Outline of a geometric theory of receptotopic structure. J. Theor. Biol. 69: 655-83 Shannon, C. E. 1949, In The Mathematical Theory of Communication, ed. C. E. Shannon, W. Weaver. Urbana: Univ. Illinois Press Shatz, C. J., Stryker, M . P. 1978. Ocular dominance i n layer I V of the cat's visual cortex and the effects o f monocular deprivation. J. Physiol. 281: 267-83 Sperry, R. W. 1943. Visuomotor coordination i n the newt (Triturus viridescens) after regeneration of the optic nerve. J. Comp. Neurol. 79: 33-55
Sperry, R. W . 1963. Chemoaffinity i n the orderly growth of nerve fiber patterns and connections. Proc. Nail. Acad. Sci. USA 50: 703-10 Stent, G . S. 1973. A physiological mechanism for Hebb's postulate of learning. Proc. Natl. Acad. Sci. USA 70: 997-1001 Stryker, M . P., Harris, W. A . 1986. Binocular impulse blockade prevents the formation of ocular dominance columns in cat visual cortex. J. Neurosci. 6: 2117¬ 33 Stir, M , Garraghty, P. E., Roe, A . W. 1988. Experimentally induced visual projections into auditory thalamus and cortex. Science 242: 1437^11 Swindale, N . V . 1980. A model for the formation o f ocular dominance stripes. Proc. R. Soc. London Ser. B 208: 243-64 Swindale, N . V. 1982. A model for the formation of orientation columns. Proc. R. Soc. London Ser. B 215: 211-30 Takeuchi, A . , Amari, S.-I. 1979. Formation of topographic maps and columnar microstructures in nerve fields. Biol. Cybern. 35; 63-72 Turing, A . M . 1952. The chemical basis o f morphogenesis, Philos. Trans. R. Soc. London Ser. B 237: 37-72 Udin, S. B , Fawcett, J. W. 1988. Formation o f topographic maps. Annu. Rev. Neurosci. 11:289-327 Uttley, A . M . 1970. The informon: A network for adaptive pattern recognition. J. Theor. Biol. 27: 31-67 Uttley, A . M . 1979. Information Transmission in the Nervous System. London/ New York/San Francisco: Academic von der Malsburg, C. 1973. Self-organization o f orientation sensitive cells in the striate cortex. Kybernetik 14: 85-100 von der Malsburg, C, 1979. Development of ocularity domains and growth behaviour of axon terminals. Biol. Cybern. 32: 49-62 von der Malsburg, C , Cowan, J. D . 1982. Outline of a theory for the ontogenesis of iso-orientation domains i n visual cortex. Biol. Cybern. 45: 49-56 von der Malsburg, C , Willshaw, D , J, 1976. A mechanism for producing continuous neural mappings: Ocularity dominance stripes and ordered retino-tectal projections. Exp. Brain Res. I : 463-69 (Suppl.) von der Malsburg, C , Willshaw, D . J. 1977. H o w to label nerve cells so that they can interconnect in an ordered fashion. Proc. Nail. Acad. Sci. USA 74: 5176-78 Watanabe, S. 1969. Knowing and Guessing. New York: Wiley Watanabe, S. 1985. Pattern Recognition: Human and Mechanical. New York: Wiley Whitelaw, V. A . , Cowan, J. D . 1981. Specificity and plasticity of retinotectal con-
515
281 nections: A computational model. / . How patterned neural connections can be set up by self-organization. Proc. R. Soc. Neurosci. I: 1369-87 London Ser. B 194: 431 -45 Widrow. B., Stearns, S. D. 1985. Adaptive Signal Processing. Englewood Cliffs, NJ: Willshaw, D. J„ von der Malsburg, C. 1979. A marker induction mechanism for the Prentice-Hal! establishment of ordered neural mapWiesel, T. N., Hubel, D. H. 1974. Ordered pings; Its application to the retinotectal arrangement of orientation columns in problem. Proc. R. Soc. London Ser. B 287; monkeys lacking visual experience. J. 203-43 Comp. Neurol. 158: 307-18 Willshaw, D. J., von der Malsburg, C. 1976.
516
Reprinted with permission from Computational 403-423, 1990 0 t990 MIT Press
Ntaroscitnce,
Writing from his deathbed John von Neumann predicted 30 years ago that eventually both brain theory and the theory of computation would come to resemble the physics of statistical mechanics and thermodynamics. His prediction was based partly on the stochastic character of neural activity and on the apparent inevitability of statistical descriptions for neural interactions; perhaps one might hope that emergent behavior in such statistical systems could be understood in somewhat the same way that the macroscopic laws of thermodynamics emerge from the microscopic kinetics of statistical mechanics. A second inspiration for von Neumann's vision was the maturing of the mathematical theory of information, which in the hands offigurestike Wiener, Shannon, Kolmogorov, and Gabor (building on earlier work by Hartley, Szilard, Nyquist, Boltzmann, and Maxwell) had emerged from statistical physics with a probabilistic definition of information and had revealed a set of mathematical laws governing its representation, communication, and transformation. These developments fostered the remarkable view that one might actually think of information as resembling in certain respects a substance, insofar as its movements and transformations, like those of matter, turn out to be constrained by some fundamental laws. A generic example of such a principle is the relationship berween the information capacity tin bits/second) of a generalized "channel," its bandwidth, and its signal-tonoise ratio whether the medium in question happens to be an undersea telegraph cable, a broadcasting band of
A n Information-Theoretic View of Analog Representation in Striate Cortex
1
John G . D a u g m a n
J
H
L Von Neumann diKUUed the need tor a statistical theory of brain funchon on pp. 79-^2 of hij uncompleted manuscript tot the St Li man Letfuf ej which he was lo deliver at Yale m l9Se. His IUPCM and death prevented hi, delivering ihe ledum, but the partial manuscript may br found In the von Neumann 1950 reference. r
Zflf^n/J[fcrr/*njifiraWg 36a: S0-11211941); and |. D. Farmer, 'InfomuHon dimension and Ihe probabilistic structure oi chaos." Z/iHthtift fur Nafurfouthung 3fm U O l - J JZ5U942). The history olthe reluctant acceptance ol the idea thai there are formal and abstract constraints, on the movement! of information, regardless of its physical instantiation, was evocatively described by Gabor ( l " f i l in the following lermi: "aj the principle of coruervation of energy emerged from the slowly hardening conviction of the impossibility of a p*rpfhturn nubile, this fundamental principle -.. arose mam the refutation of ingenious attempts to break the as yet unformulated law" (p. 429). Q
2. Two elegant recent pipefj that illuihate 'he relationship, bolh metaphorical and concrete, between the properties ai information rod the • i :•.. • of fluids, are Robert Shaw. "Strange attractOrV chaotic behavior, and information flow,"
517
404
the electromagnetic spectrum, or a bundle of nerve fibers.' Information theory also contributed to the foundations of Ihe theory of computation by providing insights into corcputability and complexity, not to mention supplying the basic unit of symbolic manipulation (Ihe bit), and imparting its power to signify Perhaps information theory, the child of statistical physics, might accomplish foe brain theory what it had done for electrical engineering As with many of von Neumann's ideas, his deathbed vision of an eventual nexus among statistical mechanics, brain theory, and computational theory appears likely lo come In.'- Among the theoretical constructs from statistical physics that are now often invoked bolh in formal models of neural assemblies and in certain novel computational algorithms are stochastic relaxation and selforganization, the dynamics oi variational energy functions, cooperative and competitive elementary phenomena, stochastic annealing, the emergence of order from chaos, critical phenomena, domain formation in Markov random fields, dissipative trajectories toward attractors in phase space, and so on. A general feature of many of these approaches is the idea that physics is computation. In some ways, this trend represents a departure from recent decades of cognitive science, in which the principle of the formal equivalence of all universal Turing machines provided a justification for cognitive scientists to disregard the hardware implementation as functionally irrelevant With the current shift of focus from symbol manipulating frameworks for cognitive science in favor of analog neural networks, the rediscovery that the laws of physics can be the basis of "analog computation'' may generate renewed interest in physical processes, or simulations of them, which can sometimes implement nonaigorithmic solutions to very complex problems. In the spirit of von Neumann's intuition that key factors aboul brain mechanisms and architecture might reflect the essentially probabilistic nature of information, the goal of this chapter is to analyze some characteristics of spatial image coding mechanisms in mammalian visual systems from the standpoint of statistical information theory. We begin with the basic property of nanotthaganakty of neural visual codes and the question of their com-
3. Other eiarnale) of irkTofmalion-theorellc pnncplei uvlkioV the nrletioruiiipt among aampling rile. band*..i:'• JJ ., prediction: the relationship between fource enlropy. channel entropy, and iignal correlation and Ihe reliTiomtup berween Ihe enjemble probabilities of Ihe ttaltt ol lleliitkai veiiemi end
followed by demonstrations that receptive field structure serves significantly lo dicorrelatl ihe spatial information in images and hence reduces the entropy, or statistical complexity, of its neural representation We may compare Ihe observed degrees of decorrelation with the exhaustive decorrelation produced by a KarhunenLoeve expansion, in which image structure is represented in terms of the eigenfu net ions of the diagonaliied covariance matrix, leading to a statistical interpretation of the strategies adopted in early visual coding.
pletincss.
In considering the mutual nonorthogonality llarge inner products) of overlapping visual neural receptive held profiles, a paradox becomes apparenl in the classical interpretation of whal il is that such neural firing rates signify. {A possible resolution of Ihis paradox, by a relaxation neural nelwork. is suggested in the Appendix.) Subsequent sections review bolh physiological and psychophysical evidence about the form of Ihe oriental ion-selective spatial visual code, and aboul the values of some dimensronless parameters in the code lhat reveal the strategies adopted in the face of inescapable tradeoffs in the allocation of coding resources. Coding and sampling issues associated with polar spectral self-similarity of the oriented, multiscale visual primitives are then discussed in relation to recent mathematical work in wavelet theory, Finally, simulations are presented that illustrate compressive self-similar coding oi real images using these biologically based code primitives, with variations in orientation sampling rates and variations in code quantization, followed by illustrations of orien la lion-based segmentation oi stochastic images by anisotropic visual codes.
Completeness No one would seriously suggest lhat the purpose of early vision is to reconstruct the retina] image within the brain, as in the familiar derogatory "movie screen' analogy. On Ihe other hand, it is undeniable lhat there ii some cortical representation of image structure (or several representations) such that, up to Ihe relevant limits of resolution in space, time, contrast, and chrominance, retinal image structure is captured, represented in some neural basis.
The amount? of information JiroclaTcd wilh Ihe appearance ol Iheie l l * t « . Eircniive treatment* of the theory of Information may be found in Ihe died eronu by Abrimion 11floJr. Callagor
518
405 and analyzed. The elementary tad that wilhin these limits, any stimulus can be seen, is neither trivial nor without significant implications. The neural representations, within their dimensional limits of resolution, their inhomogcneities. and obviously with transformations in Eormat, are apparently algebraically complete-, there do not appear to exist large equivalence classes of different, resolvable patterns that are either invisible or which under scrutiny all look the same. The question of completeness Eor theories of perception, although it rarely seems to be raised, is somewhat analogous to asking whether a given linguistic theory is capable of parsing and representing all possible wellformed ("resolvable') sentences. Defined operationally, perceptual completeness would imply that within the resolution limits there is no incoming pattern or shape that cannot be uniquely represented. Such a state of affairs is not an automatic consequence of adequate resolutionit depends on Ihe number of independent degrees of freedom in the representation and on other mathematical properties, some of which can be very subtle. For example, the "muttiscale blurred zero crossings" theory of early visual representation popularized by Marr (1982) invokes a very powerful and subtle theorem of B. F. Logan (1977), which shows that bandlimited signals can be trjrwpktily represented (up to a scale constant) by their zero crossings alone, provided that (I) their bandwidth does not exceed one octave, and (2) they have no complex zeroes in common with their Hilbert transform. As d test of this theory of human vision, Daugman [19BSa) generated families of visual textures that should all look identical for human observers if this model of early vision were correct, because the textures all have identical Laplacian zero crossings at all scales of filtering and analysis. The texture family was designed to violate the second condition of Logan's Theorem concerning their Hilbert transforms, and hence zero crossings representations would not distinguish among them, yet human observers do so easily. These stimuli (seen in figure 2 of Daugman 19fl8a) thus constitute counterexamples for this theory in the matter of completeness, because they reveal that the human visual system can distinguish among signals which Ihe Marr model could not.
imagine an auditory system that is fully responsive over the normal frequency range except that it is incapable of representing any sounds of intermediate frequency, say in the vicinity of 800 Hz (Missing basis functions cannot be constructed from, nor can their work be done by, any olhet members of the basis set.) In the case of vision, one might ask, for example, whether Hubel s and Wiesel's classical description of receptive held properties in terms of "bar" and "edge' detection is rich enough to form an image basis. Can a face, for example, be completely represented by bars and edges? The Iwo types of theoretical commitments outlined in the above paragraphs regarding completeness, namely (1) nonreconstruction yet (2) sufficient richness to capture at least implicitly all information, require that theories of biological visual coding walk a kind of epistemological tightrope. The two poles of the tightrope result from demanding that the representation be informationaEly complete over its domain, while at the same time avoiding Ihe implication lhat what is entailed by completeness is a cortical image reconstruction (as if on behalf of cortical movie-viewing hcmunculi}. It is easy to fall from this tightrope. Most theories about biological image coding strategies incorporate at least an implicit stance on completeness. This can be algebraically formulated as the question of whether all relevant signal structure can be represented by weighted sums of the proposed visual code's primitives. For an encoded signal y), a given set of candidate primitives {G,(.r,y)} constitutes a complete code if there exists a set of coefficients {a, \ (representing the relative strength or presence of each of the primitives in the code for the signal) such that ]|>G,(.i,y} = iU,y..
(1)
In vision models, the code's coefficients {a,} are the firing rates of the neurons whose corresponding response selectivities are the {G,(i. y)} code primitives. Even if the code primitives do constitute a complete code, if they happen to be nonorthogonal primitives, then it may be extremely difficult to arrive at the coned set of coefficients [a,} for a given signal (although Eheir existence for any signal is guaranteed by virtue of completeness). Complete codes can be composed of either orthogonal or nonorthogonal sets of primitives, bul orthogonality of the code primitives does not ensure completeness. Whatever the level of Ihe visual res presentation in question, and whether or not
In a mathematically noncompete scheme of representation, some pallems may be well within resolution limits and yet be invisible because of an inadequate number of degrees of freedom. By way of illustration, one might
519
4(16
offset positions, in general would not satisfy the above criterion for orthogonality; they cannol signify nonredundarii, or independent, facts about the retinal image * Similarly, cortical simple cells responding to a given paich of visual space are in general nonorthogonal and hence have correlated responses, although if they differ in their preferred orientation, then their inner producl (a measure of their nonorthogonality, given in the lefl-side of equation 2) will be much reduced. Quadrature phase offsets will also reduce the measure of nonorthogonaliIy; paired simple cells having the quadrature phase relation discovered by Pollen and Ronner (1981) will be exactly orthogonal if their symmetries are purely even or odd.
the corresponding code primitives (given by Ihe neural response selectivities) are orthogonal, the completeness of the image code in a given area of visual space/ up to the appropriate resolution limit there, is implicitly determined by whether or not the number of Linearly independent degrees of freedom in the code is at least as great as the dimensionality of the incoming sampled signal.
Nonorth ogonal ity A ubiquitous property of biological sensory and motor control systems appears to be their use of nonorthogonal coordinates. For example, as emphasized by Pellionisz [1988; Pellionisz and Graff 19871. the three semicircular canals of the vestibular system, which funclions as an accelerometer. lie in planes inclined to each other at roughly 120°; thus there is no physical direction of bodily acceleration that projects onto only one axis. Other examples include the nonorthogonal attachments of the eye muscles, as well as Ihe skeletal musculature, and the mutually nonorthogonal action spectra of retinal color pigments. These properties presumably have greater functional significance than mere redundancy; Pellionisz (1988; Pellionisz and Graff 1987) has proposed that Ihey reveal an undertying tensor geometry embedded pervasively in the nervous systems of higher animals.
It is important to note that the orthogonality or nonorthogonality of spatial visual code primitives in Ihe twodimensional (2-D) space domain (as equation 2) implies the same characteristic for the 2-D spectral selectivities of the code primitives. If we left F,(u, v) and Fj(u, u) be the 2-D Fourier transforms of Ihe spatial code primitives G,(i,y| and Gjtx.il). respectively, then by invoking (1) the fact that the area, or volume, under a function equals its Fourier transform evaluated at the origin; (2] the fact that the Fourier transform of the product of Iwo functions equals the convolution of their respective Fourier transforms; and (3) the fact that the convolution of two functions, when evaluated at the origin, is simply the area, or volume, under their product; we see that
An important but rarely discussed fact about visual neurons responding lo a given patch of visual space is that, in analogy with the examples cited above, their receptive field profiles are not mutually orthogonal, in the mathematical sense that their inner product (the integral of the product of any two receptive field profiles) is in general nonzero. A given set of visual receptive field profiles (G,i.'. would comprise an orthogonal code if and only if "wliv*/!,
JjG fcy)G tey)»Wv = 0. l
J
||G (r,y)G,(r.yl .rdy = ^..r.{G,(r,y)G,(T. jr)}|,o, :
t
01
(3)
= F,{H.V) » F (u, u)l, ,!
W)
= | | f i t M . B j F / - ! . . -«)d«do.
(5)
y
0ll
Thus the orthogonality, or nonorthogonality. of code primitives in the 2-D space domain implies likewise the orthogonality, or nonorthogonality, of the 2-D spectral selectivities (for orientation and spatial frequency) of the code members. In general, except between quadrature cell pairs, visual neurons have response selectivities rendering them nonorthogonal as code primitives As we shall see laler. this fact has significant consequences for how their firing rates
(2)
This is clearly not the biological case. For example, two retinal ganglion cells in the same region of the relina with, say, the usual concentric center/surround slructure. with perhaps somewhat different sizes and perhaps somewhal
a. If a light Itpnulus cuctly marched Ihe receptive field profile oi onr of these two cells, then il the two cells were orthogonal the fipjl cell done should
520
respond. Obviously, in Ihe biological situation described, in general both cells ivould respond significant I y: Ihis reflects I heir nonarlhogonjlity.
407
can be interpreted at subsequent levels. If cells' firing rates, given by their receptive field inner product with the image, were interpreted as signifying the relative presence of image structure resembling their receptive field shape {the classical view), then a great deal of spurious and misleading image structure would be collectively implied by their ensemble responses. In addition to the apparent theoretical paradox, these observations are interesting and important because for systems of high dimension such as vision, the complexity of computing the correct values in a visual code which is nonorthogonal becomes enormous. (Formulated as a problem of linear algebra, it grows as the cube of the number—perhaps a million—of linearly independent dimensions!) Complicated mechanisms would have to be invoked to sort out which parts of the signified image structure are real from those that are the spurious consequences of neural nonorthogonality.
Entropy, Decorrelation, and Principal-Components Analysis Image coding is optimally efficient if the coefficients of the representation are nonredundant, in the Shannon-Weaver (1949) sense that their mutual information is zero. In that case, the code coefficients are aecoireiated and they form a statistically independent ensemble, so that the conditional probability P(r|y) of one coefficient's value (X = i), given any other coefficient's value ( Y = y), is the same as its unconditional probability P(x); P(r|y) = P(i). In order for each coefficient to capture a unique property of the image that cannot be captured by any other coefficient, the expansion functions employed in the code must be linearly independent. In order for the code coefficients to have zero mutual information, H(X; Y) = £ t , Pfe y) log,lP(i, y)/P(x) P( y)l = 0, the code primitives must be orthogonal so that their projections onto each other are always zero (because then Pfi.y) = P(l)P(y) inside the logarithm!
image code however, the exact eigenvectors and their eigenvalues would depend on the image. But if the image has stationary statistics at least locally, then the optimal Karhunen-Loeve expansion is equivalent (Pratt 1978; Bossomaier and Snyder 1986) to a windowed Fourier analysis in those local regions. possible
These facts about statistics, decorrelation, and Karhunen-Loeve principal components are very interesting biologically, because converging evidence from diverse sources (reviewed in Daugman 1980, 1984, 1985) suggests that the 2-D receptive field structure of oriented visual neurons in mammalian striate cortex lends itself to carrying out a local (Gabor-like) 2-D spectral analysis of the image. Various psychophysical findings about local orienlation-tuned and spatial-frequency-tuned visual channels also support this view. The fact that the resulting visual representation is conjoint, defined simultaneously in 2-D spatial and 2-D spectra! terms, can therefore be interpreted in terms of statistical information theory as satisfying the condition of stationary statistics within local 2-D spatial patches. (Image statistics tend to be locally stationary because of object and textural homogeneities,) Under these circumstances, the 2-D Gabor representation [Daugman 1985, 1968b) approximates in an image-invariant way the optimal possible coding scheme that would be obtained by principal-components anaiysis. As noted by Barlow (1961, 1972), Buchsbaum (1987). and others, the basic and ubiquitous "center/surround" structure of a sensory neuron, in diverse modalities, can be described as a deccrrelator of the relevant signal. Examples of such decorrelating mechanisms include spatial lateral inhibition among invertebrate ommatidia. chromatic opponency in retinal ganglion cells, temporally transient "on-off " cells, disparity-tuned cells, motion-contrast cells, and so on. In these diverse visual hierarchies, the mechanisms respond to change, to epochs of the signal where it is no longer the same (across space, time, wavelength, depth, velocity, etc.) This decorrelating property and its consequences for image coding can be seen graphically by comparing figures 1 and 2, which are histograms showing the statistical complexity of two different kinds of complete image codes. Figure I is a histogram of coefficients in the usual point-by-point image representation, which one might associate with retinal photoreceptors; the histogram plots the number of pixels in the image having each gray level between 0 (black) and 255 (white). If the image had been uncorrelated random noise, then
r
For any given image, such a representation can always be found by its Karhunen-Loeve expansion (Karhunen 1947; Loeve 19481. This form of principal-components analysis (Hotelling 1933; Oja 1982; Oja and Karhunen 1985; Linsker 1986) generates a complete representation in terms of the eigenvectors of the diagonalized covariance matrix of the image, all of which are, of course, orthogonal. It may be shown [Pratt 19781 lhat this is the mosi efficient
521
408
Figure 1. :. i-. TI-.
Figure 2. v,-v of a standard image when represented point-
Histogram of Ibe coefficients comprising a complete 2-D Gabor
by-polnr. as if by retina, photoreceptors In ihis pixel histogram Ihe
representation ol the same image as described in figure I* with the
entropy, or mutual self-Information, is $ = 7.57 for ihe ensemble. A
same number o! degrees of freedom 165.53o) and the same
point-by-point image represent..lion dues not extract the Intrinsic
quantization [& bits]-. Compared with the statistical complexity of the
redundancies; and correlation., thai distinguish structure from noise.
distribution in figure 1. this distribution has an entropy reduced by five binary log units to S = 2.55. but It nonetheless represents a
•his histogram would be flat [all gray levels being equally probable) and the entropy would be maximum, i.e., 5 = 8 At the other extreme, if all pixels had the same value, regardless of how many possible values there were, then the entropy would be zero. The entropy of a statistical ensemble is defined as
complete encoding of the Image. Nearly all the image structure is carried by a small fraction of the 2-D Cabot coefridenls. namely those that are far out on the tails of the distribution. (The actual coefficients are plotted in figure 3 as an image, with the average gray denoting lero.J Assuming that 2-D Cabot functions are a good model for cortical simple cell receptive fields profiles. Ihe implication for biological image coding is thai for any g.vm image, most of the information load is carried by a very small subset of the Cells, while
(6)
given that for the ensemble of available states r with associated probabilities p,-. (71 Clearly, Ihe image pixel histogram in figure 1 is broad and multimodal; its entropy is 5 = 7.57, dose to the maximum possible value of S-bit images (namely S = 8). If the same image is represented instead by coefficients on 2-D Gabor elementary functions (Daugman 1980. 1984, 19851, then the complete representation has the histogram shown in figure 2, which has obviously much reduced entropy. The actual values of these 2-D Gabor coefficients are displayed as luminance in figure 3. orga-
522
the majority are essentially quiescent.
409 domain representation; this reduction in entropy was obvious from the two histograms of figures 1 and 2. Demonstration of the fact that the 2-D Gabor image transform displayed in figure 3 still contains complete information about the original image is provided in figure 4, which shows an actual reconstruction of the original image from the 2-D Gabor coefficients plotted in figure 3. (The black points indicate the lattice centers for the global space-domain distribution of 2-D Gabor elementary functions.) Without loss of information, the statistical complexity of the image has been compressed by five (binary) log units, from originally 7.57 bits to 2,55 bits per pixel in the 2-D Gabor representation with the same number of degrees of freedom. (Larger compression factors approaching 2 0 : 1 will be demonstrated later for self-similar visual codes that sacrifice completeness.) The reason for the large entropy reduction, and hence great increase in the statistical coding efficiency, is that the 2-D Gabor elementary functions, which are based on empirically measured simple cell 2-D receptive field profiles (Jones and Palmer 1987; Daugman 1985, 1989). serve to represent the image in deconelated form.
Figure ) Complete 2-D Gabor representation ol the Lena portrait described in figures 1 and 2. plotting coefficient values as luminance. The Coordinate sysiem is globally still tbe spatial (i, v) Image domain, but embedded within ihis are local 2-D spectral parameters, which may be interpreted as locally windowed spatial frequency and orientation. The strategy of embedding angular and spectral variables within
A Paradox in the Classical View
global relinotcpic coordinates resembles the functional architecture of slriate carte*. Only a small fraction of the "cells" in the 2-D Cabor
The classical view of early visual representation is that the kinds of receptive field structure possessed by neurons in early vision constitute the primitives of the biological image code. Depending on the theory, these might be elongated blobs, edges {blurred at various scales), center/ surround structures, spatial frequency wavelets, or the components of Boolean conjunctions forming "textons." The firing rates of neurons in the early visual code are thought to signify the relative presence of such localized features in the image. Thus, in the classical view, the original image is represented as a set of coefficients (firing rates) on a set of expansion functions (the neurons' local receptive held profiles], whose completeness as a family accounts for the fact that within resolution, any pattern can be encoded. The paradox in this view is that since the response selectivities tend, at all relevant Levels of the visual system, to be significantly nonorthogonal, such a visual representation would be erroneous and misleading; it would be seriously compounded with false and spurious image structure, (in image processing simulations with such nonorthogonal visual primitives, one can see falsely introduced image structure including multiple
representation are significantly stimulated by any given image, yet •his encoding scheme completely and exactly captures all image st-nichire. as demonstrated by its reconstruct ion in figure 4.
nized as an image, in which the global coordinates are still the spatial £t, y) image domain but the local coordinates systematically map out windowed 2-D spectral parameters for each local region of the image, (Full details are given in Daugman 1988b.) The pixel brightness value at each location in figure 3 signifies the coefficient value associated with a 2-D Gabor elementary expansion function, which is spectrally tuned both for orientation and spatial frequency but at the same time localized to that specific image region. The average gray value in figure 3 corresponds to a 2-D Gabor coefficient value of zero, while lighter values signify positive coefficients and darker values negative coefficients. It is obvious that most of the transform image representation in figure 3 is at the uniform gray value, with only sporadic significant departures to white or black. Consequently, the entropy of this image is only 5 = 2,55, a full five log units smaller than the space2
523
410 £ ,G,( . )jf/{i,y). fl
I
M
u
I
As a purely ad hoc theoretical solution to the biological fact of nonorthogonal visual primitives, a "relaxation" network is described in the Appendix and illustrated in figure 11. which could correct for the misrepresentation of image structure that results from neural nonorthogonality.
Primitives of the Spatial Visual Code: Physiological Evidence
figure flDemomtntion of r o m p l e t e n K i a l (he ?-D C*bor ntpretcftliTion. t h e original Lent portrait a n be recovered without | O H horn Ihe 2-D Gjbor coefficient shown in figure 3. o n p i l r Ihe r i l r e n v )p>rsrne*i ol Ihe repiesentjlion.
echoes of edges and other features, spurious zero-crossings or other extrema of derivatives, and so on: a further consequence, if degrees of freedom are also lost, can be the failure to encode real image structure.) These implications oi the nonorthogonality of the response selectivities of neurons, for theories of biological image coding, seem not lo have been recognized. The only circumstance under which the firing rates of the ensemble of neurons comprising a visual code actually correctly represent the image in teims of their own response selectivities. is if the complete ensemble of receptive field profiles {G,(j.y)) are orthonotmal. Then, given that a cell's firing rate (a,) depends on the image iU.y) and on its own 2-D receptive field profile G,(r,y) according to
Figure 5 shows representative measurements by Jones and Palmer (1987) of the 2-D receptive field profiles of cat simple cells (top row). Ever since the pioneering studies of Hubel and Wiesel (1962. 1974) on cortical functional architecture and receptive field organization, it has been known that critical tuning variables governing the response selectivities of visual cortical cells include 2-D location, size, and orientation: more recent studies revealed associated luning also for spatial frequency and phase (Pollen and Ronner 1961). These several spatial variables can be combined naturally into a single framework that describes cortical simple cell receptive field profiles as 2-D Gabor elementary functions (Daugman 1960, 1985): 1
Gd.y) = e x p l - n l d - T o l ' / c t + (y -
y |'/('|] 0
" exp[ — 2ni(n„i + u„y)}.
(10)
The parameters of such expansion functions simultaneously capture the physiologically observed neural tuning properties bolh in 2-D spatial terms flocation ( r , y ) , width c, length /I) and in 2-D spectral terms |so that if a cell's preferred orientation is B and its preferred spatial frequency is tu, then the 2-D spectral parameters in equation (10) are given by ti incos(f) and s tusinlfl). with the orientation bandwidth and spatial frequency bandwidth determined by the relationship among a, /I, ji . and u }. An illustration of a canonical 2-D Gabor elementary function introduced as a model for 2-D neural receptive field profiles may be found in Daugman (1960). figure 6, simultaneously with its corresponding 2D tuning surface for orientation and spatial frequency. See also lacobson and Wechsler 1988. 0
m
t
0
which really constitutes Ihe classical definition of what a receptive field profile is. then if the primitives are orthogonal and complete, they have conectly encoded Ihe image. But given the fact that the biological primitives are nonorthogonal, then the code coefficients (firing ratesl are wrong:
524
a
m
0
Q
The 2-D elementary functions defined above are complex rather than real-valued: the real and imaginary parts of these analytic expansion functions correspond respectively to even- and odd-symmetric profiles. The above
dll
Difference
Figure 5. Top row; illustration* ol cat simple tell 2-D receptive held profiles, measured by Jones and Palmer (1967). Middle row: best-fitting 2-D
Gabor wavelet for each neuron, described by equation 10. Bottom row: residua] error ol the tit, which for 97% of Ihe cells studied was indistinguishable from random error in tbe sense.
analytic form can be regarded as representing not one cell but a module of two cells, in quadrature phase, which together constitute a local phasor.* This is easily understood by interpreting the second term in equation 10 with Euler's formula;
Euler relations for expressing a real modulation term as the sum of two complex exponentials:
exptrttu) = cos(riii) + isin(uJi).
(II)
Alternatively, one could regard a single cell as being described by the sum of two (complex-valued) 2-D Gabor functions in conjugate form, again with the help of the
!
r "'
(12)
sin(oii) = j , e x p ( — icui)
(13)
—
Jiexp(ju>x).
Different phases or symmetry relations in the 2-D receptive field profiles can be projected out of the above functional form by noting that the implicit amplitude constant multiplying the elementary functions is itself not real but complex, and hence it determines not only amplitude but also phase. In phasor form, this arises from the fact that if A and ^ are real constants and C is the complex quantity
The quadrature phase relation discovered by Pollen and Ronner (1901) among pairs of simple cells thus fits naturally within this Gabor framework, provided one considers two cells at a time as comprising a Gabor phasor.
5. A -i.: of tuch •vulytir
cos(tur) = ^exp( — iinx) + ^exp(icur},
C = Aexp(i$l
rfdJ-valued jrceptive field profiles u e prelected out fur-CLiOns may be found on p. Ilfi2 of Daugman
19AS.
525
U4)
412 then by multiplying the above phasor in equation 11 with the complex constanl C we can simultaneously rotate the phase of the elementary function and independently scale its amplitude: Cexp(iBix) = Acos(tui + t) + Aisin(u)i +
ft.
(15)
For reasons given earlier, the sum of two such complex functions in conjugate form wilh complex coefficients C will give a single real-valued modulation term with any desired phase, amplitude, and spatial frequency: JCexpl — iioi)
+
^Cexp(nur) =
Acos{ioz +
(16)
When expressed in 2-D form and multiplied by a 2-D Gaussian of appropriate width, length and 2-D location, this (now real-valued) sum of two complex 2-D Gabor elementary functions becomes a good general-purpose model for simple cell receptive field profiles in the 2-D space domain as well as capturing their 2-D spectral properties of selectivity for spatial frequency and orientation. The middle row of figure 5 shows best-fitting numbers of this functional family for each of the corresponding empirically measured receptive field profiles in the top row. the bottom row shows the residual errors of fit. For 97% of the cat cortical simple cells whose 2-D receptive field profiles were thus measured by |ones and Palmer (1987) and fitted to equation 10. the residuals between empirical data and Ihis model were indistinguishable, in the i' sense, from random error.
In his classic paper, Gabor (1946) pointed out that there exists a "quantum principle'' for information, which he illustrated through the construction of a spectrogramlike iJi/urmuliori diagram. The infoimation diagram, a plane whose axes correspond to time and to frequency, must necessarily be grainy, or quantized, in Ihe sense that it is not possible for any signal or filter (and hence any carrier or detector of information! to occupy less than a certain minimal area in this plane. This minimal or quanta! area, which reflects the inevitable trade-off belween time resolution ( i l l and frequency resolution (AID) and equals the lower bound on their product, is expressed by the uncertainty relation (QtKiui) 2 1/4*
(17)
Gabor noted that Gaussian-modulated complex exponentials offer the optimal way to encode signals (of arbitrary spectral content) or to represent them, if one wished the code primitives to have a well-defined epoch of occurrence in time as well as a well-defined spectral identity The same principle is true for the spectral and temporal structure of tuned filters that are to delect, or resolve, information in terms that include both spectral content and lime of occurrence. One would like a filter to have an impulse-response function (iff) that is as short as possible, and simultaneously a frequency selectivity that is as sharply tuned as possible. This optimal form thus represents a compromise between a pure time-domain representation of information (in which each component has unlimited spectral content) and a pure frequency domain, or Fourier, representation (in which each component has unlimited duration). Gabor proved that one could completely represent any arbitrary signal by expanding it in terms of such optimally jointly localized "elementary functions/'comprising a conjoint time/frequency representation, although be could not actually prescribe a way to find the required coefficients of this expansion (since the elementary functions are not orthogonal).
Information-Theoretic Significance of 2-D Receptive Field Structure From the standpoint of information theory, the reason it is interesting that this model provides such a successful fit lo the measured neural 2-D receptive field profiles is that the Gaussian-windowed complex phasor described in equation 10. regardless of its 2-D spectral modulation or size, is the optimal solution to an inescapable uncertainly principle for the resolution of information simultaneously in 2-D spatial and 2-D spectral terms. The uncertainty principle for 1-D functions that emerged importantly in the formulation of quantum mechanics through the work of Hermann Weyl and Werner von Heisenberg in the 1920s was applied by Dennis Gabor in the 1940s to the theory of signal encoding, and it became one of the milestones of information theory.
Marcelja (19B0) pointed out that Gabor's 1-D elementary functions offer a good model for simple cell 1-D receptive field cross-sections. Simultaneously, Daugman U980) used a 2-D generalization of these elementary functions to model the 2-D receptive field profiles of cortical simple cells and at Ihe same lime to account for their 2-D spectral tuning curves for orientation and spatial frequency (which are not polar-separable). Twodimensional signals or filters are constrained by a 4-D
526
413 uncertainly principle, in which resolution for the two coordinates of visual space (Ai) and (Ay) enters into an inescapable tradeoff against resolution for the two coordinates of spatial frequency (AH) and (Au). This may be expressed in terms of a generated uncertainty relation (Daugman 1985): (Ai)(Ay)(Ai.)(Ai>) > 1/I6IT
1
method can provide Ihe coefficients for a se(/-similar, or maueltl. set of 2-D Gabor expansion functions all of which are dilations, translations, and rntnfions of a common underlying 2-D template. With the demonstration of hypothetical mechanisms that could yield correct coefficients in a biological visual code based on a set of self-similar (while nonorthogonal) 2-D Gabor spatial primitives, the classic tuning properties of cortical simple cells can be put in a unifying framework of coding theory. Their well-known selectivities for 2-D spatial location, orientation, spatial frequency, and phase can be represented in terms of an underlying functional form whose parameters are precisely these. More importantly, the trade-offs observed physiologically among the selectivities (sharpness of tuning bandwidths) for orientation, spatial frequency, and 2-D location can be understood in terms of generalized form of an inescapable uncertainty relation for information encoded conjointly in these variables. It is interesting, and perhaps no concidence. that the functional family thai provides a good fit to the great majority of simple cells' 2-D receptive field profiles (as seen in figure 5) also happens to constitute an optimal sel of coding primitives for extracting the most information possible in a 4-D information space (xlocation, y-localion. orientation, and spatial frequency) that is inherently, and necessarily, quantised.
(18)
For 2-D functions, such as images or neural receptive held profiles, each of the (A) terms in the above joint uncertainty relation is defined by a Cartesian second moment, given by the 2-D integrated dispersion of the signal or filter irf from the corresponding principal axis (defined by the corresponding component of the first moment). It should be noled that the Cartesian spectral coordinates (H, B) may be expressed in polar coordinates in terms of orientation and spatial frequency, which allows the uncertainties (Aw) and (Arj) to be interpreted in terms of the more physiologically familar measurements of orientation bandwidth (At)) and spatial frequency bandwidth (Au>), (Full details about conjoint uncertainty relations involving tbe variable of orientation, as well as the empirically observed regressions among these various tuning bandwidths, may be found in Daugman 1985,) Bastiaans (1980) provided Ihe first method for actually the correct coefficients needed for Gabor's expansion of 1-D signals. 34 years after Gabor's classic proposal, Bastiaans' method requires that all terms in the signal expansion use a fixed Gaussian envelope, regardless of the term's modulation frequency: this restriction of course contradicts the biological situation. If anything, as Kulikowski and Bishop (1981), among others, have pointed out. the population of cortical simple cells tend (to first order) to have a common template: this requires the Gaussian width to scale proportionally with modulation wavelength, rather than being strictly independent of it. Subsequently, Daugman (1988b) developed a relaxation method for obtaining the correct coefficients in 2-D Gabor expansions of 2-D signals without the restrictive, biologically unrealistic, requirement of a fixed Gaussian window. The relaxation process converges to the exact coefficients in the case of a complete expansion, and it converges to optimal coefficients (in the LMS sense) if the expansion is incomplete because of having fewer degrees of freedom than the number of sampled 2-D signal values. Of particular interest for the case of biological visual primitives, as will be illustrated later in this chapter, the
finding
Psychophysical Evidence on the Code Primitives, and Information Tradeoffs Although psychophysical experiments obviously involve the aggregate nervous system rather than just a given cell or layer, it is sometimes possible to gain useful information from the "intact observer" regarding overall signal processing strategies, bottlenecks, and limits. In recent decades, important insights about human vision have emerged from such studies of absolute detection and discrimination, preattentive vision, the channeling of information and its interactions, and perceptual organization generally. The strength of interaclion between two orthogonal signals, measured in terms of the degree to which Ihe threshold for detection of one signal is influenced by the simultaneous presence of Ihe other signal, afford an indication of the bounds of independence of the mechanisms involved in signal representation and detection. En spatial visual research, such "masking" effects have revealed the
527
414 existence of mechanisms, or "channels," that are tuned both for spatial frequency and for orientation (as well as for spatial position). Although these tuning properties were originally interpreted in terms of separate mechanisms (on the one hand orientation channels, on the other hand spatial frequency channels, etc.) it is possible (Daugman 1960, 1954) to represent these various psychophysical tuning curves and their interdependences through a single construct, namely the 2-D spectral consequences of the 2-D spatial organizational principles of local visual mechanisms. In this way. psychophysical masking effects in spatial vision can be understood in the same unifying framework of conjoint 2-D spatial/2-D spectral representation of information as was presented in Figure 5 for single cortical neurons.
20
FREQUENCY DOMAIN
H
Parametric studies of the 2-D spectral specificity of spatial visual masking of one signal by another are summarized in figure 6 (from Daugman 1964. where full detail on methods may be found). Figure 6a shows the elevation of contrast detection threshold distributed over Ihe 2-D Fourier plane tinterpretable in polar coordinates {r, 9) with r representing spatial frequency and & representing orienlation), for the detection of a 2-D Fourier component in the presence of a given masking component (r = 8 cycles/degree, 0 = 0°), Clearly, the threshold elevation effect is localized to the 2-D spectral vicinity of the mask coordinates, and one might interpret the 2-D threshold elevation data of figure da as revealing the 2-D spectral tuning surface of the masked neural 2-D visual mechanism Such an interpretation emerges from the observation that visual cortical neurons are selective for orientation and size (or spatial frequency); perhaps the population that is most responsive to the fi cycles/degree vertical mask is overdriven or fatigued by the mask and hence less able to detect the second signal when its 2-D spectral content is similar. The measured threshold elevation surface of figure 6a might then be interpreted as the mechanism's 2-D spectral tuning surface. Figure 6b shows the iso-half-amplitude contours of the threshold elevation data surface seen in figure 6a. The ellipses in the 2-D Fourier plane have a width/Length aspect ratio of about 2.1, which reveals how Ihe tradeoffs necessitated by the generalized uncertainty relation {equation IB) are negotiated. This empirical 2:1 spectral tuning aspect ratio can be interpreted as meaning that resolution for orientation is favored, in a sense, over
528
( o l THRESHOLD EICVUTION SURFACE
ftlHU.F-4MPLITUDE CONTOURS
2D SPACE DOMAIN
t e l EVEN STMMETHr KERNEL
Figure 6. (a] Psjchoph)... ,i' nielSureiri«nli •.< Ihe 2-D spectral jlructure of human visual encoding mechanisms, bawd on signal detection masking around a point in the 2-D Fourier pJane. (b Iso-halfamplitude slice through Ihe spectral mechanism shown in (a) over the 2-D Fourier plane, revealing elliptical spectral specificity with i l l 1 speciral tradeoff ratio. 4c) The inferred 2-D spatial weighting function, or receptive field profile, computed by numerical 2-D Fourie' I'anjformation of the data surface in (a). Ensembles of such primitives would give rise to the observed psychophysical specificity.
413 resolution for spatial frequency. A given amount of 2-D spectral area times the corresponding 2-D spatial area, {AuKAuHAjKAyk that specifies a fixed quantity of conjoint information, can be occupied by filters of many different shapes. The empirical 2:1 ellipses demarcating the iso-half-amplitude contours in figure 6b suggest a particular ratio for allocating the spectral part of the available information "budget" into the variables of spatial frequency resolution versus orientation resolution.
spaced in octave, not linear, terms fPollen and Feldman 1979). These two spectral properties are suggestive not of a Cartesian distribution lattice in the 2-D Fourier plane but rather of a log-polar distribution Lattice, as illustrated in figure 7. There are two important properties reflected in figure 7, First, the lattice centers of the ellipses obey a log-polar distribution rule. The center of any ellipse, (r, 6), falls on a preferred spatial frequency r that increases in geometric, not linear, sequence; and the ellipse centers have orientation coordinates lying on a polar grid, so that across all spatial frequencies (r) there are a fixed number of orientations (8) sampled. Second, the ellipses just touch each other in both angular and radial coordinates These two properties of the log-polar grid, when combined with the 2 i 1 spectral trade-off ratio seen both physiologically and psychophysical I y (as observed, for example, in figure 6b), require that the spatial frequency bandwidth of each mechanism be 1.5 octaves, and that its orientation bandwidth be + 15°. Any two of these three properties (a given 2-D spectral tradeoff ratio, a given orientation bandwidth, or a given spatial frequency bandwidth) suffice to determine the third property for the 2-D spectral logpolar distribution lattice. Thus the various classes of data on tuning curves can be placed in a self-consistent 2-D spectral framework, reflecting the inescapable tradeoffs lhat constrain the conjoint encoding and representation of information.
We might call this measure of the allocation of information resolution resources the spectral tradeoff ratio, (ones and Palmer [1967) measured this spectral ratio for 36 cortical simple cells, as well as fitting their 2-D receptive field profiles with equation (10) in order to obtain best estimates for the corresponding parameters a and 0 in both the spatial and spectra! domains. The measured aspect ratios ranged roughly from I ; l to 4:1, but the most common value was about 2:1. The same approximate value (1.7) for this spectral tradeoff ratio was reported by Movshon £1979). Thus the spectral tradeoff ratio appears to be the same in orientation-selective simple cells as that which can be inferred from the psychophysical measurements shown in figure 6a and b. Finally, figure 6c shows the numerical 2-D Fourier transform of the psychophysical data from figure 6a {assuming even symmetry); the resulting profile can be interpreted as the 2-D space-domain mechanism thai would account for the psychophysical 2-D spectral masking data presented in figure 6a. This psychophysical^ inferred mechanism bears close resemblance to physiologically measured 2-D profiles of oriented cortical simple cells.
Finally, we note that the elliptical spectral tuning surfaces seen in figure 7 correspond to the iso-amplitude contours of 2-D Gabor elementary functions, since the 2-D Fourier transform of a 2-D Gabor function (as specified in equation 10} with modulation spatial frequency t j , orientation 8. and width/length aspect ratio n'-i. will be a pair of elliptical Gaussians centered al polar Fourier plane coordinates {<••• + 0 ) and having a spectral tradeoff ratio of fffa (in the present case 2: 1J Obviously, an elliptical Gaussian will have elliptical iso-ampfitude contours, although if the spectral bandwidths are too broad, then the fact that the transform of a single real 2-D Gabor mechanism is really the sum of two conjugate elliptical Gaussians, rather than one, will produce some distortion in the ellipticity. Further analysis of these properties and their mutual relationships, and a review of the relevant neurophysiological evidence, may be found in Daugman (1965) tables 1, 2, 3, and 4, and figure 2,
A Self -Similar Polar Pyramid of Spatial Visual Primitives Both neurophysiological and psychophysical data suggest that spatial visual mechanisms have 2-D spectral coding properties that are organized in log-polar form. That is, their response selectivities tend to be defined in terms of an approximately constant angular bandwidth (say, + 1 5 ° ) corresponding roughly to the changes in preferred orientation between orientation columns, and their spatial frequency bandwidths tend to be roughly constant in octave terms (e,g. 1-2 octaves) rather than in Linear bandwidth terms. Moreover, the distribution of preferred spatial frequencies among cells also tends to be
529
416 Compact Image? Coding by Self-Similar Neural Wavelets In recent years, there has been a great deal of mathematical activity in Wavelet Theory, largely made possible by the work of the French mathematician Yves Meyer (see references by Meyer 1986; Morlet et al. 1982, Mallat 1908: Grossmann and Morlet 1964; Crossmann et al. 1985; Daubechies et al. 1986; Holschneider 1988). Wavelets are families of Localized, undulalory expansion functions all of whose members are dilations and translations of each other. Thus any 1-D wavelet f ^ U ) can be generated from any other wavelet in its family through the following generating function, parameterized by integers (m K) for octave dilations in size (m) and uniform shifts in position fit): VJW
=- 2—T(2—i - n).
(19)
The generating function captures the sftf-simitar.tif property, since whatever the shape of ¥Cr>, all other members of the set having different sizes and locations will iust be scaled versions of the same underlying shape. We may generalize to two-dimensional wavelet functions T U , y) that are anisotropic (oriented) by adding another degree of freedom to the generating function, namely, rotations through angle 0: Figure; 7 .
(20)
Spectra] self-similarity of the 2-D Cibor primitive* In the polar wavelet code, viewed in ihe 2-D Fourier plane. All members of this
where the substituted variables n'. / i capture dilations in size (m). 2-D translations in position (p. q). and changes in orientation {6) as follows:
visual code, described in either 2-D spatial or 2-D spectral terms, (an be generated by dilations, rotations, and translations ol each olher. or of a basic underlying complex member. {See equation 10 and 20-22 ) Moreover. Ihe self-similar sef of code primitives possessing these spectra offeis a good description ol both phyfiologLcaJ data (figure 5 :• and psychophysical data (figure 61. including the 2 . 1 spectral tradeoff raho governing orientation selectivity.
»' = 2""ticosW) + ysinWll - p
(21)
y' = 2 " - | - r s i n [ 9 ) + ycos(r)|| - q.
(22)
Figure 8 displays as luminance primitives several selfsimilar enamples of 2-D Gabor elementary functions, all of which can be generated from a single complex member of equation [10) by dilations, translations, and rotations, using the above 2-D wavelet generating functions (equations 20-22). The five e»amples in figure B differ in site and spatial frequency by factors of 2 (hence m in equations 20-22 equals 0, L 2, i, 4. although amplitudes here are constant), and five different orientations 6 are represented. Since the underlying 2-D Cabor elementary function that enters into the 2-D wavelet generating equa-
530
417 able loss of image inform at ion. For all schemes shown, the coefficients computed for the coarsest 2-D Gabor elementary functions have 2 bits greater quantization accuracy than those for the finest ones (Following psychophysical evidence on the detectability of quantization error). Thus the top row across figure 9 uses 8 bits for the coefficients on Ihe coarsest terms and 6 bits for the finesh the bottom row uses 6 bits and 4 bits, respectively Entropy rates are computed using equations (6) and (7). and displayed as the bits-per-pixel (bpp) figure above each recovered image.
Sampling Theory and Oriented Wavelet Codes
Figure a. Five members oF a self-similar 2-D Gabor wavelet code viewed in the space domain. All members ol This visual code, described in eilher 2-D spatial or 2-D spectral ierms, can be generated by dilations, rotations, and translations of each other, or of a basic underlying complex member. (See equations 10 and 2 0 - 2 2 }
tion (equation 20) as T U , y) is complex, the implicit complex constant multiplying each 2-D elementary function specifies not only contrast but also phase. Two quadrature phases are represented among the five examples shown in figure 8. We have seen that several physiologically and psychophysical^ measured properties of spatial visual mechanisms are captured by a self-smilar, log-polar ensemble of 2-D Gabor elementary functions. Can such an ensemble be the basis of an image code? Demonstrations of several different 2-D Gabor wavelet codes are provided in the 12 panels of figure 9. The images are reconstructed from coefficients on the 2-D Gabor wavelets, obtained despite the nonorthogonality of these elementary functions, using the relaxation network described in the appendix. The four columns represented different choices for the number of distinct orientations 6" used for generating the wavelets (equations 2 0 - 2 2 ) , namely, from left to right, 6 orientations, 4, 3, and 2. The rows represent different quantization depths for the computed 2-D Gabor coefficients, in order to show how very compact the codes can be made without unaccept-
531
It is interesting that compete recovery of image structure is possible with these wavelet codes, despite a number of properties that may seem to violate traditional intuitions about samplying requirements. First, because of the logpolar lattice for the distribution of 2-D spectral parameters, the only spatial frequencies represented follow the geometric sequence, 1, 2, 4, 8, 16. rather than the linear sequence 1, 2, 3, 4, ... familiar From Nyquist sampling theory. Second, and related to this, the orientations represented in any given scheme are always from a fixed set of angles (e.g., 0, 30, 60, 90, 120, and 150 degrees), regardless of spatial frequency. By contrast, familiarity with Nyquist sampling theory might lead to the expectation that the higher the spatial frequency, the larger the number of orientations that must be sampled, as in the familiar case of covering the 2-D Fourier plane with <• Cartesian sampling grid. Both of these unusual properties can be understood in light of the locaiimUon of the code primitives, as opposed to the global expansion functions used in a complete 2-D Fourier representation. Crudely speaking, Ihe fact that each 2-D Gabor wavelet is localized creates additional degrees of freedom, which compensate for the apparent undersampling in the spatial frequency and orientation parameters It seems remarkable that as few as three orientations (in 60° steps, column 3 of figure 9), or even two orientations (in 9 0 " steps, column 4) yield such rich and accurate representations of the image. For example, the recovered image seen in the third column, second row from top of figure 9. is reproduced with only three orientations and with quantization depths ranging from 7 bits to 5 bits per coefficient, to yield a remarkably compact image code
•UN
Figure 9. , n
Compact image representation 2-D Gabor wavelet codes. The column* correspond to different numbers of onenN t ion<, used in the code, ranging from e orientations (at l*H) to only 1 orientations (al right). The rows represent different levels of quantization accuracy lor each 2-D Gabor coefficient, with |h? cousesl terms [ranging from
A bits al top to 6 bill at bottom) having 2 bits more A c c u r a c y than the finest terms [A bits to 4 bits). The resulting codes' compression factors are revealed by comparing the original image (ft bits/pixel) with the encoded rep«s«»i*l«ons here, ranging From 2,87 bits/pixel (upper left! to only 0 J B bits/pixel (lower right).
532
<19 noise, with different regions of the image containing textures characterized by different anisotropic second moments. Since the 2-D Gabor elementary functions, when used for representing any image, essentially segregate the image structure in terms that are selective for size or density (first moment) as well as orientation or dipole (second moment), such a representation performs an efficient segmentation of the texture collage. The remaining three panels of figure 10 show the parts of the image that "lighl up" in dipole clustering of the computed 2-D Gabor expanson coefficients. The resulting segmentation of the texture field into regions of differing anisotropic statistics seems to mimic our perceptual organization of the original held.
Summary It has been suggested in this chapter that statistical information theory can illuminate some aspects of biological spatial visual representations. In particular, coding criteria such as decorrelation, completeness, compression, extraction of redundancy, minimization of conjoint 2-D spatial/ 2-D spectral uncertainty, self-similarity of primitives, and sensitivity to anisotropic statistics in image structure may have shaped the evolution of visual codes. Considerations that lie at the heart of information theory lead to the derivation of an optimal Family of 2-D expansion functions that, while not mutually orthogonal, generate selfsimilar primitives that constitute a complete visual code. This 2-D "wavelet" family offers a good description of both neurophysiological data and psychophysical data, on both the 2-D spatial structure and the 2-D spectral structure of visual codes. Moreover, this particular code leads to Large data compression factors by eliminating redundancy, since the code primitives are effective anisotropic decollators of image structure. At the same time, this code reorganizes spatial visual information in a way that can subserve textural image segmentation based on anisotropic statistics
Figure i n . Illustration oF statistical image segmentation by * visual code comprised ot 2-D Gabor primitives. The dipole clustering of the code primitive; isolates different image regions, in a manner resembling our perceptual organization of tbe original stochastic image (upper left panel) into anisotropic regions.
with a cost of only 1,03 bits/pixel. Again, the fact that only two or three distinct orientations for the'wavelets, at all sizes, can still capture so much image structure, must be understood in the context of the compensating degrees of freedom afforded by the 2-D position of each 2-D Cabor elementary function. These observations about the representational richness of wavelet ensembles of 2-D Gabor elementary functions, even when so sparsely sampled in orientation, render all the more compelling the still unanswered question about why the visual cortex has such a comparatively dense and crystalline sampling architecture for orientation.
Segmentation by Anisotropic Visual Coding Finally, it is interesting to note that the type of reorganization of spatial visual information discussed here in terms of 2-D Gabor elementary functions can subserve some elementary kinds of statistical image segmentation. Figure 10 displays in the upper left panel a collage of textures constructed from anisotroptcally filtered white
Appendix: A Relaxation Network for Computing Nonorthogonal Codes A theoretical "neural network" is illustrated in figure 11 that can correctly find the coefficients for nonorthogonal visual codes, such as those observed in biological visual
533
410
[mage l[x,y]
Ec.it-.ii
F i i c d Weights Layer
Figure 1 1 . A hypoEheEieal Ehree-layered rrUxalion "neural network" Ehjt : m correctly hud the coefhaenli far nonorthogonal visual codes, such a those observed In biological visual system* This network, which was used lo generate the image representations described In figures 2, 3. 4. and 9. operates by rela»alion of an energy surface as described in the Appendix. The result Is analogous to orthogonaliaalion of the representation without oilhogonaliiing Ihe visual code primitives. • •ii-. retaining [heir redundancy.
{G.li.yl,. These could be regarded, for example, as 2-D receptive field profiles, whatever their structure may be. It can be useful to think of a given image f | i . y\ comprising n pixels as a single vector in an n-dimensional vector space; the elements of the vector constitute the individual pixel values that comprise the image. In the (nonbiological) case of image representations based on complete orthonormal expansions, a given orthonormal expansion set would constitute a basis for this vector space. Examples of such orthonormal bases include the unit basis (point-by-point image representation), and the Fourier basis (in which each basis vector is a 2-D discrete Fourier component). Here we will describe a relaxation network for representing Ihe sampled image fli.yj by a set of coefficients (firing rates) on nonorthogonal code primitives (linearly dcpindtnt vectors), which may or may not completely span the vector space. The relaxation network implements constraint satisfaction by "relaxing" into a minimum energy state. Our problem is to represent y| either exactly or in some optimal sense by projecting it onto some given set of code primitives G,\i. y\. This requires finding projection coefficients \a ) such that the resultant vector H|r. y| t
systems. It was noted above that spurious and misleading image representations would result From visual coding by tbe direct inner products of receptive held profiles with the retinal image, because o( the mutual nonorthogonality of these primitives. Presumably, some mechanisms to compensate for this would exist in biological visual systems, since they clearly involve mutually nonorthogonal receptive fields. The network shown in figure 11. which was used to generate the image representations described in figures 2. 3, 4. and 9. operates by relaxation to arrive at coefficient values that equate at] of the mutual projections among the weighted primitives wilh their weighted projections onto the input signal. The resull is analogous to orthogonaliiation of the representation without orthogonal izing the visual code primitives, thus retaining their redundancy. For Ehe present purposes we shall treat Ihe original retinal image that is sampled by Ihe photoreceptor lattice as a discrete 2-D signal I j i , y\ with the nolational convention that square brackets signify a discrete rather than continuous signal. This 2-D signal is to be represented in the early visual nervous system by an ensemble of neural firing rates that comprise a set of expansion coefficients {*,} on some set of two-dimensional code primitives.
ffix,y]=
U3)
J>C,|r,yl
is either identical to l\x. y| (the complete case) or generates a difference vector J|i. y] — H|z. y] of minimal length (the optimization case). If the primitives [G^lLyl} comprise a complete orthogonal code, then the representation in Ftfi.y] is exact (the difference vector is zero) and the solution for [a,] is simple. _I..,iAI-.yl'[*,y|) E.-.tfUy] •
(24)
But if the primitives do not constitute a complete orthogonal code, then in general the representation H[x.y] will be inexact and the desired set of code coefficients [n,| must be determined by satisfying a constraint, such as minimizing the squared norm of the difference vectors: E=
UU.y\-
HU.y)l>
1
- £«U,y|-HU,y|) .
(25)
The norm £ will be minimized only when its partial derivatives with respect to all of the t coefficients {a,} equal zero:
534
All
2
C
Z[ (J>'^M) .I^|] = 0.
(261
Satisfying this condition for each of the fl, then generates a system of n simultaneous equations in n unknowns, where n is equal to the total number of pixels in the image |on the order of 100,0001: I
tftoVlGfcyl)
- £ [ ( £- . ( i f c ^ C ^ y j J .
(27)
Thus the solution that minimizes the squared norm oi the difference vector {equation 25) amounts to finding the set of coefficients {a,} such that the inner product of each code primitive G |i, y] with the entire linear combination of code primitives ^a^C [x, y) is the same as its own inner product with the original image / | i , y l . It should be noted that in the case when the {G,[r,y|} comprise a complete orthogonal code, then the inner products in Ihe righthand side of equation 27 are nonzero only for J: = i. and so each of the n equations then has only a single unknown and it is immediately apparent that the minimal-difference vector solution for each a, is identical to that given earlier in equation 24 as the familiar orthogonal case. (
k
The relaxation network shown in figure 11 converges to the correct values of {oj} (without a teacher) by gradient descent along the i-dimensional E{a,) surface, which expresses the quadratic energy function's dependency on all of the {rij} coefficients. A common feature of theoretical neural network architectures is the combination of some layers that have adjustable (or adaptive) synaptic weights and some layers with fixed weights. The present scheme begins with a layer of fixed connection strengths that are specified by a particular set of nonorthogonal code primitives {Gjlr.y]}. By summing the different image pixels through these weights, the output of the i'th neuron in this layer is simply the inner product of the |
535
of the middle layer constitute the new image representation as the set of coefficients {a,). The interlaminar control signal adjusts each of the weights by an amount A,, given by Ihe difference between a feedforward signal and a feedback signal. The feedforward signal is the level of activity of the neuron from the first layer, and the feedback signal is the inner product of the weighting function of the corresponding neuron in the third layer with the weighted sum of all the other neighboring neurons in that layer with which it is connected- Thus the weight adjustment is given by 4 = ^(Q .y|iUy] -z[G L,»l( i; 1
I
>
1
i
and the iterative rule for adjusting the value of each coefficient is Q = a, - f A,. It should be noted that the network does not require a "teacher" that generates the weight adjustment signal by comparing the current representation with a separate copy of the desired pattern. Rather, the adaptive control signal A, arises only from interlaminar network interactions. {
It can be seen by inspecting equations 26 and 28 that the weight adjustment rule is equivalent to
2 da.
1291
In the continuous-time case, the network becomes a dynamic system, each of whose nodes is described by the differential equation: da, _ _ 1 £E dl
™
(30)
Ida,'
It should be noted that the minus sign implies that the weight adjustment is always in the downhill direction of the energy surface and that the adjustment is proportional to the slope of the energy surface at this point The equilibrium state of the network that is reached when all da,/dl = 0 is the state in which the energy function E representing the difference-vector squared norm llfli.yl — H f e y l l has reached its minimum. This is the point at which the partial derivative of E with respect to all of the adjustable weights is nil: 1
(31)
422 Thus in the stable state, the middle layer of the network has weights that represent the optimal coefficients {a,} for the projection of the signal l[x,y] onto any set of code primitives {G [i,yl}, which, as noted eaHier, need comprise neither an orthogonal nor complete code. This relaxation network was used in the coding simulations that produced the images seen in figures 4, 9, and 10, based on visual coding with nonorthogonal primitives. f
References Abramson. N-11963) Information Theory and Coding. New YorV McGraw-Hill, Barlow. H. 0. (1961) The coding of sensory messages. In Cufrmi Problems in Animal Behavior, ed- W. H Thorpe and O . L. Zangwill Cambridge University Press, Barlow. H. B. (1972) Single units and sensation: A
neuron* doctrine
lor perceptual psychology? Perception 1: 371-394.
Acknowledgments Supported by N.S.F. Presidential Young Investigator Award 1RI-B853619 and by AFOSR University Research initiative F49620-37-C-OOlfl,
Bastiaans. M . (1980) Gabor's expansion of a signal inlo Gaussian elementary signals. Pre*. IEEE
66: 5 J 0 - 5 3 9 ,
Bossomaier, T , and Syndcr. A. W. 11966) Why spatial frequency processing in [He visual cortex? Vision Research 26: 1307-1309. Buchsbaum. G . • I
[mage coding in the visual system. Optics
B; 16-19. Daubechies. L, Grossmann. A. and Mayer, Y. (19961 Painless nonorthogonal expansions, Journal a} Mathematical Physio 27: 1 2 7 ] -
Daugman, f. (I9SU' Two-Dinien*Lor.a] spectral analysis of cortical receptive Field profiles. Vision Research 20: 647-656. Daugman. J. (1964) Spatial visual channels in the Fourier plane Vision Research 24(9}: 691-910. Daugman, J. (1965) Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical Filters, journal of Ihe Optical Society of America, A 2(7): 1160-1169. Daugman. J. (1963a) Pattern and motion vision without Laplacian rero-erossings. journal of ihe Optical Society of America A 5(7):
Daugman. f (1986b) Complete discrete 2-D Gabor transforms by L
neural networks for Lmage analysis and compression. I.E.E.E. Transactions Acoustics. Speech, and Signal Processing i6(7): 1369-1 J?9, Daugman. J, (1969) Entropy reducl ion arid dKomlation in visual coding by oriented receptive fields. IEEE Transactions an Biomedical tnginerring 36(1): 1 0 7 - T H . De Croats, 5., and Mazur. P. (1984) No*-Equilibrium Thtrrnpdi/namics. New
York
Dover.
Farmer. |. (1982) Information dimension and the probabilistic structun? of chaos Zritsthrift fur rJatnrfotschung 37{a . 1304-1325. r
Cabar, D. (1946) Theory of eorrifriun.calion. journal of the Institute of Electrical Engineers 93: 429-457. Gallagher, R. 11968) Information Theory and Reliable Comtrntnicniio'i New Y o r k Wiley, Grossmann. A . and Morlet. J, (1984) Decomposition of Hardy funcIjons into square integrate wavelets oE constant shape. Si AM of Mathtnrttiini Analysis 15; 723-736.
536
journal
423 Grossmann. A . Muriel. J . and Paul. T. (1965) Transforms associated
Oja. E., and Karhunen, }. (1985) O n stochastic approximation ol the
to squaie inlegrable group representations. I. General results, foumal
eigenvectors and eigenvalues of the expectation of a random matrix
of Mathematical Physics 26r 2473-2479.
journal of Mathematical ^rtaJysii and Applications 106: 6 9 - 8 4 .
Hoi Schneider. M. 0966). O n the wavelet transformation of fractal
Pellionisz, A. (1988) Tensor geometry: A language of brains and
objects, journal of Statistical Physics 50(5/6}: 963-991.
neurocompulers Generalized coordinates in neuroscience and robotics. In Neuronal CnrrTpirfera ed. R. Eckmiller and G. von Malsburg.
Hotelling. H. 11933) Analysis oF a compter of statistical variables into
Berlin: Springer-Verlag.
principal components. Journal of Educational Psychology 24: 417-441.
Pellionisz. A . and Graff, W. (1987) Tensor network model of the
Hubel. D . and Wiesel, T . 11962) Receptive fields, binocular interac¬
"ihree-ncuron uestibulo-oculai reflex-arc" in cat. Journal of Theoretical
tion, and functional architecture in the cat's visual cortex, journal of
Neurobiology 5: 127-151.
Neurophysiology llondonj 160: 106-154.
Pollen. D . and Feldman. S- (1979) Spatial periodicities of periodic
Hubel. D . and Wiesel. T . (1972) Sequence regularity and geometry
complex cells in the visual cortex cluster al one-half octave intervals
of orientation columns in the monkey striate cortex. Journal of Com-
Investigative Ophthalmology and Visual Science 18: 429-433
parative Neurology 156: 267-293.
Pollen. D . and Ronner. S. (1981J Phase relationships between adja-
lacobson. L . and Weehsler, H (1966) loint spatial'spatial-frequency
cent simple cells in the visual cortex. Science 212: 1409-1411
representation. Signal Processing 14: 37-66.
Pratt. W K 11978) Drgital Image Processing- New York: Wiley
Jones, J.. and Palmer, L. 11967) A n evaluation of the two-dimensional Gabor filter model of simple receptive fields in cat striate cortex.
Shannon. C . and Weaver. W. (1949) Mathematical Theory of Com
Journal of Neurophysiology 56(6): 1233-1256.
rtrrfriicflrfoJi.
Karhunen. H. (1947) O n linear methods in probability theory. {En-
Shaw. R. (19811 Strange attractors. chaotic behavior, and information
glish trans, by L Selin.) Rand Corp.. D o c T-1J1. I960
flow, Zeirschrift fur Nalurforschung 36fa>: 80-1T2.
Kuiikowski, ].. and Bishop. P. O - (1961)' Fourier analysis and spatial
von Neumann. 1.11958) The Computer and The Brain. New Haven;
representation in the visual cortex. Eipcricnlia 37: 160-163.
Yale University Press
Linsker. P. (I960) From basic network principles to neural architecture; Emergence of spatial-opponent cells, of onentation-selective cells, and of orientation columns. Proceedings of the National Academy of Science 83: 7506-7512. 8390-6394,
8779-8793.
Loeve. M_ (1948) Fomtions Aleatoires de Seconde Ordre. Paris: Hermann Logan. B. F. (1977) Information in the zero-crossings of bandpass signals. Bell Sysiem Technical journal 56; 467-510. Mallal. S. (1969} A theory for mulliresolutLon signal decomposition: The wavelet representation. l.E.E.E. Transactions on Pattern Analysis and Machine Intelligence, in press. Marcetja, S. 11980) Mathematical description of the responses ol simple cortical cells. Journal of Ihe Optical Society of America 70: 1297-U00 Marr, O. (1962) Vision-. A Computational Investigation into the Human Representation and Processing of Visual Information. New York: Freeman. Meyer, V. (1986) Principe d'incertitude. bases Kilbertiennes. et algebres d'operateurs. Seminairi Bourbaki. No. 662. Morlet. J . Arens, G., Fourgeau L and Giard. D. 11962) Wave pro pagation and sampling theory. Geophysics 47: 203-236, Movshon. J. (1979) Two-dimensional spatial frequency tuning of cat striate cortical neurons. Society for Newosticnct Abstracts 9: 799. Oja. E, (1962) A simplified neuron model as a principal component analyzer, foumal of Mathematical Biology 15: 267-273.
537
Grbana: Unrversity of Illinois Press.
Reprinted with permission from Neural Compulation, Vol. 4, pp. 196-210, 1992 © 1992 Massachusetts Institute of Technology
What Does the Retina Know about Natural Scenes? Joseph J . Atick* A. Norman R e d l i c h School of Natural Sciences, Institute for Advanced Study, Princeton, N] 08540 USA By examining the experimental data on the statistical properties of natural scenes together w i t h (retinal) contrast sensitivity data, we arrive at a first principles, theoretical hypothesis for the purpose of retinal processing and its relationship to an animal's environment. We argue that the retinal goal is to transform the v i s u a l input as m u c h as possible into a statistically independent basis as the first step i n creating a redundancy reduced representation i n the cortex, as suggested by Barlow. T h e extent of this whitening of the input is limited, however, by the need to suppress input noise. O u r explicit theoretical solutions for the retinal filters also show a simple dependence on mean stimulus luminance: they predict an approximate Weber law at low spatial frequencies and a D e V r i e s - R o s e l a w at high frequencies. A s s u m i n g that the dominant source of noise is quantum, we generate a family of contrast sensitivity curves as a function of mean luminance. T h i s family is compared to psychophysical data.
1 T h e Retina and the V i s u a l Environment . ... A n animal must have knowledge of its environment. A s Barlow (1989) has emphasized, one important type of knowledge that needs to be stored in the brain is knowledge of the statistical properties of sensory messages. This provides an animal with data about the regular structures or features in its environment. N e w sensory messages can then be compared to expectations based on this background data; for example, the background data can be subtracted. In this way, one can argue, the brain is able to discover unexpected events and new associations. Here we explicitly explore the possibility that even the retina knows some of the statistical properties of visual messages. O u r prejudice is that discovering how this information is used in the retina will not only help explain retinal processing but will be invaluable in applying this idea to the cortex. To discover what the retina knows about the statistics of its environment, it is first necessary to find out just what characterizes the ensemble "Address after July 1, 1992: The Rockefeller University, 1230 York Ave., New York, NY 10021, USA.
538
197
of visual messages i n a natural environment. A n important step in this direction has been taken by Field (1987), who has been analyzing pictures of "natural" scenes, such as landscapes without human-made objects as well as pictures of human faces. A s Field has argued, these represent a very small subset of all possible images: all possible arrangements and values of a set of pixels. What he found is that natural images have unique and clearly defined statistical properties. The first statistical measure Field calculated is the two-dimensional spatial autocorrelator (1.1)
R ( x , y ) =
which is defined as the average over many scenes (or the average over one large scene assuming ergodicity) of the product of luminance levels L(x) and L ( y ) at two spatial points x and y. Actually, by homogeneity of natural scenes the autocorrelator is only a function of the relative distance: R(x — y ) . One can thus define the spatial power spectrum, which is the Fourier transform of the autocorrelator R(f) = } dxe' R(x). This is the quantity that Field directly measured. What he found is ,x
R(f) ~ 7^5
w
which corresponds to a scale invariant autocorrelator: under a global rescaling of the spatial coordinates x —• ax the autocorrelator R(a\) —> R(x). Although this scale invariant spatial power spectrum is by no means a complete characterization of natural scenes, it is the simplest regularity they possess. The retina, being the first major stage in visual processing, is not expected to have knowledge beyond the simplest aspects of natural scenes and hence for understanding the retina the power spectrum may be sufficient. The question at this stage is what is the relationship between this property of the visual environment and the observed visual processing by the retina? To answer this, let us explore what happens to the spatial power spectrum of the visual signal after it is processed by the retina. The output of one major class of retinal ganglion cells is known to be related to the light input approximately through a linear filter: 2
Ofy)
= jdx
K{xj - x) L(x) =KL
(1.2)
where L(x) is the light intensity at point x, 0(x,) is the output of the jth ganglion cell, and K~(xj — x) is the linear ganglion cell kernel (x, is the center of the cell's receptive field. Here we assume translation invariance of the kernel K, which means that all ganglion cell kernels are the same function, but translated on the retina). Once adapted to bright light, this ganglion cell kernel, i n spatial frequency space, is a bandpass filter. 2
X-ceUs in cat, P-pathway cells in monkey.
539
198
Typical retinal filters at high luminosity are shown in Figure \A and C , where the experimental responses K(f) [actually the contrast sensitivity which is K(f) times the mean luminance 1 ] are plotted against stimulus frequency. The data shown in Figure I A are from De Valois et al. (1974), while the data in Figure 1C are from Kelly (1972). Now to see how the power spectrum is modified by the retina, we need only multiply the input spectrum R(f) by K(f)K" (f) since the average output spectrum is (O(f)O'(f)) = ((K(f)Z.(f))(K(f)L(f))-). We can also plot the square root of this output spectrum — the amplitude spectrum — simply by multiplying the experimentally measured kernels K(f) in Figure I A and C by the input amplitude spectrum
3
0
\Jm = ic This has been done in Figure IB and D, which shows an intriguing result. At low frequencies, the input spectrum | f | is converted into a flat spectrum at the retinal output: (0(f)0"(f)) = constant. This whitening of the input by the retina continues up to the frequency where the kernels in Figure I A and C peak. Had this whitening continued up to the system's cutoff frequency, this would have meant the ganglion cell outputs would be completely decorrelated in space. This is because a white or flat spectrum in frequency space Fourier transforms into a delta function in space, giving (0(x;)0(x;)) ~ 6 . In other words, the signals on different ganglion cell nerve fibers would be statistically independent. So it appears that the retina is attempting to decorrelate its input, at least down to the scale of the peak frequency. -2
tj
The idea that the brain is attempting to transform its sensory input to a statistically independent basis has been suggested by Goodall (1960) and Barlow (1989) (see also Barlow and Foldiak 1989), and has been discussed by many others. Barlow has emphasized that one advantage of having a statistically independent set of outputs O, is that all of their joint probabilities Pjjfc... can be obtained directly from knowledge of the relatively small set of individual probabilities P,. The values of the individual P; can also be represented by taking the output strengths O, to be proportional to their improbability, — log(P,), that is, to the amount of information in each output. This then gives a very compact representation of not only the signals, but also their probabilities. In such a statistically 3
Actually, what is plotted in Figure I A and C are the results of psychophysical contrast sensitivity measurements, rattier than of single ganglion cell responses. The singlecell results, however, are qualitatively similar, and in this short paper for conciseness we compare theory exclusively to psychophysical results (all figures). In general, we believe that the psychophysical data represent an envelope of the collection of singlecell contrast sensitivities. Then, given our assumption of translation invariance, the psychophysical envelope and the single-cell results should coincide. However, we do not exclude the possibility of a more complicated relationship between psychophysical and single-cell contrast sensitivities.
540
199
lOOi) 300
1000
; A -
300
!00
r
100
JO
-
30
10 fe-
10
1000
i.ii—
.1
to
>'
10
1
* •' too
E-
-
6
r
-
I
.1
' '""I
• I
I 10
.3
JO
100
1000 c¬
300
•- C
100
r
30
h
D 5 0 0
, 0
". •
° r • .
30 -
10
10
3
I
i
3
3
1
B-
r
3
• * -'
i
1
1
i 3
1
—'
10
1
30
1
100
1
• ,
-
I " " "I .1
.3
1
• I 3
I
10
• I • mul 30
100
Figure 1: Retinal filters ( A , C ) i n Fourier space at high mean luminosities, taken from the contrast sensitivity data of De Valois el at. (1974) ( A ) and K e l l y (1972) (C). B (D) is the data in A ( C ) multiplied by l / | f ] , w h i c h is the amplitude spectrum of natural scenes. T h i s gives the retinal ganglion cells' output amplitude spectrum. Notice the whitening of the output at low frequencies. T h e ordinate units are arbitrary.
i n d e p e n d e n t b a s i s , t h e o u t p u t s O, r e p r e s e n t "features," for e x a m p l e , i n E n g l i s h text t h e y w o u l d c o r r e s p o n d r o u g h l y to " w o r d s " ; they a r e the statistical s t r u c t u r e s that c a r r y u s e f u l i n f o r m a t i o n . F i n d i n g t h e s e f e a t u r e s effectively r e d u c e s the r e d u n d a n c y i n the o r i g i n a l s e n s o r y m e s s a g e s , l e a v i n g o n l y t h e s o - c a l l e d "textual" (not p r e d i c t a b l e ) i n f o r m a t i o n . O n e m a y t h e r e f o r e state t h i s g o a l of s t a t i s t i c a l i n d e p e n d e n c e i n i n f o r m a t i o n t h e o r y l a n g u a g e a s a t y p e of r e d u n d a n c y r e d u c t i o n . B a s e d o n the e x p e r i m e n t a l e v i d e n c e i n F i g u r e I B a n d D , o n e
might
a d v a n c e t h e h y p o t h e s i s that t h e g o a l of the r e t i n a l p r o c e s s i n g is to p r o d u c e a d e c o r r e l a t e d r e p r e s e n t a t i o n of a n i m a g e .
H o w e v e r , this cannot
b e the o n l y g o a l i n the p r e s e n c e of i n p u t n o i s e s u c h a s p h o t o n n o i s e o r b i o c h e m i c a l t r a n s d u c t i o n noise.
I n that c a s e , d e c o r r e l a t i o n a l o n e w o u l d
r.4i
200
be a very dangerous computational strategy as we now illustrate: If the retina were to whiten all the way up to the cutoff frequency or resolution limit, the kernel K(i) would be proportional to |f| up to that limit. This would imply a constant average squared response KRK' to "natural" signals L(x), which for R ~ |f|" have large spatial power at low frequencies and low power at high frequencies. But this same K ( f ) ~ |f| acting on input noise whose spatial power spectrum is approximately flat (noise is usually already decorrelated) has a very undesirable effect, since it amplifies the noise at high frequencies where noise power, unlike signal power, is not becoming small. Therefore, even if input noise were not a major problem without decorrelation, after complete decorrelation (or whitening up to cutoff) it would become a problem. Also, if both noise and signal are decorrelated at the output, it is no longer possible to distinguish them. Thus, if decorrelation is a strategy, there must be some guarantee that no significant input noise is passed through the retina to the next stage. 2
Further evidence that the retina is concerned about not passing significant amounts of input noise is found in experiments in which the mean stimulus luminance is lowered. In response to this change, the ganglion cell kernel K(f) makes a transition from bandpass to lowpass filtering. This is just the type of transition expected if the kernel is adapting to a lower signal to noise ratio, since lowpass filtering is a standard signal processing technique for smoothing away noise. Such a bandpass to lowpass transition also occurs when the temporal modulation frequency of the stimulus is increased (the retinal kernel is actually a function of both the spatial frequency f and the temporal frequency w, which has up to now been suppressed). In this case too there is an effective decrease in the spatial signal to noise ratio, so it is also evidence for noise suppression. In a previous paper (Atick and Redlich 1990) we found an information theoretic formalism that unifies redundancy reduction and noise suppression. That formalism predicts all the qualitative aspects of the experimental data. However, it is highly technical and uses parameters that do not seem to have clear physical roles. This makes it more difficult to do quantitative comparisons with experiments, since the necessary dependence of these parameters on, for example, mean luminance is not intuitive. In this paper we adopt a modular approach where noise suppression and redundancy reduction are done in separate stages. This has two advantages: first it produces parameters with more direct physical meaning, and second it gives a clearer theoretical understanding of the purpose of retinal processing. In the next section we formulate our theory mathematically making more concrete the heuristic notions of decorrelation and noise suppression. We then derive a simple theoretical retinal transfer function, and compare it to experiments.
542
201
2 Decorrelation as a Computational Strategy in Retina 2.1 Decorrelation In the Absence of Noise. In the previous section, we gave some experimental evidence leading to the hypothesis that the goal of retinal processing is to produce a representation with reduced redundancy. This implies a representation where the ganglion cell activities are as decorrelated as possible (more generally, statistically independent), given the inherent problem of input noise in the retina. In this section, we formulate this notion as a mathematical theory of the retina. We first set up the decorrelation problem ignoring noise, and later introduce the simple but important modification needed for noise suppression. The outputs { 0 ( x ; ) j of the array of ganglion cells are completely decorrelated iff ( 0 ( x ) 0 ( x , ) ) ~ fy, where the brackets denote an ensemble average over natural stimuli. In general, due to the presence of noise, the retina will not decorrelate completely. Instead the filter K will only tend to decorrelate (or decorrelate up to a given scale). For this reason it is most natural to formulate the problem in terms of a variational principle with an "energy" or cost functional, E{K}, that grades different kernels according to how well they decorrelate the output. A n y constraints on this process are easily incorporated as penalty terms in the energy functional. To find the correct energy functional for decorrelation one may use Wegner's theorem (Bodewig 1956), which states that ;
with equality if and only if the matrix (0(x,-)0(xy)) is diagonal. This means that decorrelation can be achieved by keeping det(0(x,)0(x,)) fixed and minimizing Ylii&iXj)). One reason for keeping det{0(x/)0(x,)) = delfS^RK) fixed is that this ensures a reversible transformation, since it is the same as requiring d e t ( K K ) > 0. [Here we are treating the kernel as a matrix Kg = K(x; — Xy)J Actually, there are a couple of mathematical steps that lead to a simpler energy functional. First, with the assumption of translation invariance we can minimize (O^xn)) for one ganglion cell at location x instead of l i { 0 ( x , ) ) . Again by translation invariance, this is equivalent to minimizing the explicitly invariant expression E ^ O ^ x , - ) ) — T r f K R K ) . Finally, it is moTe convenient to hold fixed l o g d e t ( K K ) rather than detfl^K). Thus r
0
r
2
7
T
4
E { K J = T r ( K R K ) - p log d e t ( K K ) T
T
(2.2)
p is a lagrange multiplier used to fix det(K K) to some value, but since we do not know this value we will subsequently treat p as a parameter penalizing small d e t ( K K ) . T
r
We should point out that the decorrelating filter K that minimizes 2.2 is not the usual Karhunen-Loeve transform which would be the Fourier transform for translationally invariant R . This K L transform gives a nonlocal, nontranslationally invariant K. 4
543
202
To find the kernel K that minimizes equation 2.2, it is best to work in frequency space, where traces such as TT{KRK ) become integrals over frequencies. Also, the second term in equation 2.2 can be converted to an integral, by first using the matrix identity l o g d e t ( K K ) = T r l o g ( K K ) . The equivalent energy functional becomes T
7
E { K } =JM
\K(()\
2
r
R(f) - p j di log |K(f)
(2.3)
which when varied with respect to K(f) gives (2.4)
2
With Field's R(f) ~ l / | f | , this gives the whitening filter K(f) ~ ^p\f\. Having arrived at the energy functional [equation 2.2 (or 2.3)1 as the one that produces decorrelation, it is now straightforward to explain its information theoretic interpretation. Minimizing the first term in equation 2.2 is equivalent (see Atick and Redlich 1990) to minimizing the sum of bit entropies £ , f i j = - £ J d O , P ( 0 ; ) l o g [ P ( 0 , ) ) , where P(Oj) is the probability density for the ith ganglion cell output O; = 0 ( x ) . The second term in equation 2.2 is the change in entropy H (including correlations, not just bit entropy) due to the retinal transformation, so requiring this term to vanish would impose the constraint that no information is lost — this is related to requiring reversibility, although it is stronger. Therefore minimizing E in equation 2.2 has the effect of reducing the ratio of bit entropy to true entropy: y j ; H f / H , which is what we mean here by redundancy. Minimizing this ratio reduces the number of bits carrying the information H; technically, it reduces all but the first order redundancy. Also, one can prove that E ; H , < H with equality only when the 0{xj) are statistically independent, so minimizing this ratio produces statistically independent outputs. (
2 . 2 Introducing the Noise. Since here we are primarily interested in testing redundancy reduction, we take a somewhat simplified approach to the problem with noise. A s discussed earlier, instead of doing a fullfledged information theoretic analysis (as in Atick and Redlich 1990), we work in a formalism where the signal is first low-pass filtered to eliminate noise. The resulting signal is then decorrelated as before. Actually, since we will be comparing with real data, we have now to be more explicit about the stages of processing that we believe precede the decorrelation stage. In Figure 2 we show a schematic of the signal processing stages that we assume take place in the retina. First, images from natural scenes pass through the optical medium of the eye and in doing so their image quality is lowered. It is well known that this effect can be taken
544
203
Optical
Whitening
MTF
K
Measured K
Figure 2: Schematic of the signal processing stages assumed to take place in the retina.
into account by multiplying the images by the optical modulation transfer function or M T F of the eye, a function of spatial frequency that is measurable in purely non-neural experiments. In fact, an exponential of the form e x p [ - ( | / | / / ) ° ] , for some scale f characteristic of the animal (in primates / ~ 22 c / d e g and a ~ 1.4) is a good approximation to the optical MTF. T h e resulting image is then transduced by the photoreceptors and is low-pass filtered to eliminate input noise. Finally, we assume that it is decorrelated. In this model, the output-input relation takes the form c
£
c
O = K-[M-{L
+ n) + n ]
(2.5)
0
where the dot denotes a convolution as defined in equation 1.2. n(x) is the input noise (such as quantum noise) while flo(X(J is some intrinsic noise that models postreceptor synaptic noise. Finally, M is the filter that takes into account both the optical M T F as well as the low-pass filtering needed to eliminate noise. A n explicit expression for M w i l l be derived below. With this model, the energy functional determining the decorrelation filter K is E{K}
= j d i \K(f)\
2
2
2
2
{M (f)[R(f) + N ] + N }
2
2
-pjdilog|K(f)| (2.6)
where J V ^ f ) = ( | « ( f ) | ) and JS^(f) = ( | n ( i ) l ) are the input and synaptic noise powers, respectively. This energy functional is the same as that in equation 2.3 but with the variance R ( f ) replaced by the output variance of O in equation 2.5. 2
0
545
204 As before, the variational equations 6E/BK = 0 are easy to solve for K. The experimentally measured filter K^p is then this variational solution, K, times the filter M :
|JWWI =
w
M(*) = 7 -TT72 {M^fHR^ + N^+N*}'
™
A n identical result can be obtained in space-rime trivially by replacing the autocorrelator R(t) and the filter M(f) by their space-time analogs R(f, w) and M(l,w), respectively, with w the temporal frequency. However, we focus here on the purely spatial problem where we have Field's (1987) measurement of the spatial autocorrelator R(f) of natural scenes: R(f) =
W2.3 Deriving the Low-Pass Filter. In our explicit expression for K^p, below, we shall use the following low-pass filter 1 1 / 2
l
R(f)
Jo
RM + N
(2.8)
e-mtkV 2
The exponential term is the optical M T F while the first term is a low-pass filter that we derive next. The reader who is not interested in the details of the derivation can skip this section without loss of continuity. It is not clear in the retina what principle dictates the choice of the low-pass filter or how much of the details of the low-pass filter influence the final result. In the absence of any strong experimental hints, of the type that imply redundancy reduction, we shall try a simple information theoretic principle to derive an M: We will insist that the filter M should be chosen such that the filtered signal O' — M • (L + n) carries as much information as possible about the idea! signal L subject to some constraint. To be more explicit, the amount of information carried by C , about L , is the mutual information I{0',L). However, as is well known (for L and n statistically independent gaussian variables, see Shannon and Weaver 1949) 1(0, L) ~ [H(O') - N o i s e Entropyj, and thus if we maximize / ( C , L ) keeping fixed the entropy fi(O') we achieve a form of noise suppression. We can now formulate this as a variational principle. To simplify the calculation we assume gaussian statistics for all the stochastic variables involved. The output-input relation including quantization units, n , takes the form O' = M • (L + n) + n . A standard calculation leads to q
¥
M^R + N^+N? I(0',L)
=
Jdftog
2
Similarly, one finds for the entropy H ( O ' ) = -Jdf logfM^R + N ) +N*]. The variational functional or energy for smoothing can then be written
546
205
as E{M) = - J ( 0 ' , L ) - J J H ( O ' ) . It is not difficult to show that the optimal noise suppressing solution 6E/6M — 0 takes the form 1/2 2
N J \r)R + N
with the parameter T, ~ N | ' in order to hold H ( O ' ) fixed with mean luminance. Actually, below we w i l l be working in the regime where the quantization units are m u c h smaller than the signal and noise powers and hence we can safely drop the - 1 term in M i since the IT; term dominates for small N . We can also ignore any overall factors in M that are independent of f. This then is the form that we exhibit in the first term in equation 2.8. 0
2
2.4 A n a l y z i n g the Solution. Let us now analyze the form of the complete solution 2.7, with M given in equation 2.8. In Figure 3 we have plotted Koxp(f) (curve a) for a typical set of parameters. We have also plotted the filter without noise R ( f ) ' / (equation 2.4) (curve b) and Mff) (equation 2.8) (curve c). There are two points to note: at low frequency the kernel K„p(f) (curve a) is identically performing decorrelation, and thus its shape in that regime is completely determined by the statistics of natural scenes: the physiological functions M and N drop out. A t high frequencies, on the other hand, the kernel coincides with the function M , and the power spectrum of natural scenes R drops out. We can also study the behavior of the kernel in (equation 2.7) as a function of mean luminosity / . If one assumes that the dominant source of noise is quantum noise, then the dependence of the noise parameter on Io is simply N = I Q W where N' is a constant independent of Jo and independent of frequency (flat spectrum). This gives an interesting result. At low frequency where K^ goes like l/VR its J dependence will be K„ ~ l/Io (recall R ~ J ,) and the system exhibits a Weber law behavior, that is, its contrast sensitivity J J C is independent of J . While in the other regime — at high frequency — where the kernel asymptotes M with N > R then J C ~ l/f w h i c h is a De Vries-Rose behavior J o K ^ ~ tj . This predicted transition from Weber to De Vries-Rose with increasing frequency is in agreement with what is generally found (see Kelly 1972, Fig. 3). -
2
0
2
2
x?
0
2
xp
0
2
P
0
/2
P
2
0
G i v e n the explicit expression in equation 2.7 and the choice of quantum noise for N we can generate a set of kernels as a function of J . The resulting family is shown for primates i n Figure 4. We need to emphasize that there are no free parameters here which depend on J . T h e only variables that needed to be fixed were the numbers f , a, p, and W and they are independent of h- Also we work in units of synaptic noise n , so the synaptic noise power N , is set to one. We have superimposed on this family the data from the experiments of Van Ness and Bouman (1967) on human psychophysical contrast sensitivity. It does not take 0
0
c
0
2
547
206
Spatial frequency,
c/deg
Figure 3: C u r v e a is the predicted retinal filter from equation 2.7 for a typical 1
2
set of parameters, while curve 6 is R ( f ) - ' ' , w h i c h is the pure whitening filter. Finally, curve c is the low-pass filter M. T h e figure shows that at low frequencies curves a a n d b coincide a n d thus the system is whitening, w h i l e at high frequencies curves a a n d c coincide a n d thus the retinal filter is determined b y the low-pass filter.
m u c h i m a g i n a t i o n to see that the a g r e e m e n t is v e r y r e a s o n a b l e e s p e c i a l l y k e e p i n g i n m i n d that this is not a fit b u t a
parameter free
prediction.
3 Discussion
O n e m a j o r a i m of this p a p e r h a s b e e n to a n s w e r the q u e s t i o n , w h a t d o e s the r e t i n a k n o w a b o u t its v i s u a l e n v i r o n m e n t ? O u r i n i t i a l a n s w e r c o m e s f r o m n o t i n g that t h e e x p e r i m e n t a l g a n g l i o n c e l l k e r n e l w h i t e n s t h e s p a t i a l p o w e r s p e c t r u m of n a t u r a l s c e n e s f o u n d i n c o m p l e t e l y
\f\~
2
indepen-
d e n t e x p e r i m e n t s b y F i e l d (1987). T h i s s h o w s that the r e t i n a l c o d e h a s b e e n o p t i m i z e d — a s s u m i n g w h i t e n i n g a s a d e s i g n p r i n c i p l e — for a n 2
e n v i r o n m e n t w i t h a |f|~
s p e c t r u m . I n o t h e r w o r d s , t h e r e t i n a k n o w s at
least o n e statistical p r o p e r t y of n a t u r a l s c e n e s : the s p a t i a l a u t o c o r r e l a t o r .
548
207
Figure 4: T h e family of solid curves are the predicted retinal filters (equation 2.7) at different IQ separated by one log units, a s s u m i n g that the dominant source 2
of input noise is q u a n t u m noise (IV ~ / o ) . N o other parameters depend on f . 0
T h e fixed parameters are f = 22 c / d e g , a c
5
1.4, p = 2.7 x 10 , N' = 1.0. T h e
data are from h u m a n p s y c h o p h y s i c a l contrast sensitivity measurements of Van N e s s a n d B o u m a n (1967).
But what is useful about whitening the input signal? One possible answer is that whitening compresses the (photoreceptor) input signal so that it can fit into a channel with a more limited dynamical range, or capacity. Such a limitation may be a physical one in the retina such as at the bipolar cell input synapses or it may be in the ganglion cell output cable, the optic nerve (see also Srinivisari et al. 1982). Another possible explanation for the whitening is Barlow's idea that a statistically independent, or redundancy reduced representation is desirable as a cortical strategy for processing sensory data. From this point of view, the retinal filter is only performing the first step in reducing redundancy, by reducing second-order statistics (correlation). With this explanation, the capacity limitation is located further back in the brain, and may be best understood as an effective capacity limit, which is due to a computational
549
208
bottleneck, for example, the attentional bottleneck of ** 40 bits/sec. Of course, since redundancy reduction usually allows compression of a signal, there is no reason both explanations for whitening — physical bottleneck in the retina or computational bottleneck in cortex — must be mutually exclusive. Also, to paraphrase Linsker (1989), the brain may create physiological capacity limitations at one stage in order to force an encoding whose true utility is in its use as part of a larger strategy, such as Barlow's redundancy reduction. There is, however, some evidence favoring the cortical redundancy reduction hypothesis: First, assuming a physiological bottleneck in the retina implies that the output code has a fixed and limited number of states available, and these are fewer than the number of states at the input. If one assumes that all of these outputs states are being used maximally at all luminosities, this produces a dependence on I that does not match experiment. One finds that such a capacity limitation constraint predicts a Weber (K ~ I ) type scaling with I at all frequencies so long as the kernel is bandpass; this is contradicted by experiments that show a significant decrease in contrast sensitivity (Derrington and Lennie 1982), for example, at peak frequency, even while there is little change in the shape of the kernel. Second, some animals show bandpass (whitening) filtering even at very low luminosities where the input signal to noise is such that no capacity limitation is likely. Third, the ganglion cell bandpass characteristic is sharpened at later stages, such as in the L G N , and in monkeys some cortical cells have receptive fields very much like those of ganglion cells (Hubel and Wiesel 1974). Finally, some animals have orientation selective cells already in their retinas. This, together with the third point, suggests that whitening (giving bandpass filtering) is likely to be a first stage in a strategy of visual processing which is continued in the cortex, and which may also explain, for example, orientation selectivity. 0
a
a
To finally decide on the true purpose of the retinal whitening of natural scenes will require more experiments. In particular, to avoid some assumptions, it would be best to experimentally measure the correlation between ganglion cell outputs (also cortical cells) for an animal in its natural environment. Because of the need to suppress noise, as shown here, we would predict some correlation for nearby ganglion cells, but a much smaller correlation length for ganglion cells than for the natural luminance signal. Also, the stimulus must be the animal's natural environment, or at least have a |f|~ spectrum, because of course any other type of input correlation will show up as output correlation. 2
Beyond such questions about the purpose or presence of decorrelation, we should stress that without considering the problem of noise one cannot fully explain the form of the experimental ganglion cell kernel. In fact, too much whitening of a signal that includes noise can be dangerous. This is an obvious point that has not always been appreciated. We find consideration of this need to suppress noise is the only other ingredient needed in order to explain an abundance of experimental data. It gives
550
209
a n e x p l a n a t i o n o f the r e l a t i v e l y l o w p e a k f r e q u e n c y of the r e t i n a l filter i n b r i g h t l i g h t . It a l s o l e a d s to the p r e d i c t i o n of a b a n d p a s s to l o w p a s s t r a n s i t i o n w i t h d e c r e a s i n g m e a n s t i m u l u s l u m i n a n c e . I n fact, o u r s o l u tions predict a n a p p r o x i m a t e l y Weber b e h a v i o r at l o w frequencies, a n d a s s u m i n g q u a n t u m n o i s e , a n a p p r o x i m a t e l y D e V r i e s - R o s e b e h a v i o r at h i g h frequencies. T h e s a m e p r o p e r t y of o u r s o l u t i o n s t h a t l e a d s to the o b s e r v e d b e h a v i o r w i t h c h a n g i n g l u m i n a n c e a l s o e x p l a i n s a n o t h e r set of e x p e r i m e n t s :
a
s i m i l a r b a n d p a s s to l o w p a s s t r a n s i t i o n is o b s e r v e d w h e n the t e m p o r a l f r e q u e n c y of t h e s t i m u l u s i s i n c r e a s e d . T h a t is, the effect of l o w e r i n g I
0
is p r e d i c t e d t o b e v e r y c l o s e to the effect of r a i s i n g t e m p o r a l frequency. A more complicated relationship between color processing a n d changes i n s t i m u l u s f r e q u e n c y i s a l s o p r e d i c t e d b y o u r theory, a s i s the c o n e to rod transition.
S o a v e r y l a r g e c l a s s of e x p e r i m e n t a l o b s e r v a t i o n s c a n
a l l b e e x p l a i n e d a s t h e c o n s e q u e n c e of a s i n g l e p r i n c i p l e . T h e y a l s o , a s m e n t i o n e d , p r o b e m o r e s p e c i f i c p r o p e r t i e s of a n a n i m a l ' s e n v i r o n m e n t , s o t h e y f u r t h e r test the d e p e n d e n c e of r e t i n a l p r o c e s s i n g o n e n v i r o n m e n t . A l l of t h e s e s p a c e — t i m e - c o l o r - l u m i n a n c e i n t e r a c t i o n s a r e e x p l o r e d i n a s e p a r a t e p a p e r ( A t i c k et a l . 1992).
Acknowledgment
W o r k s u p p o r t e d i n p a r t b y a g r a n t f r o m the S e a v e r Institute.
References
-
.
Atick, J. ] . , a n d R e d l i c h , A . N . 1990. T o w a r d s a theory of early v i s u a l processing. Neural Comp. 2, 308-320; a n d 1990. Quantitative tests of a theory of r e d n a l processing: Contrast sensitivity curves. Report no. I A S S N S - H E P - 9 0 / 5 1 . Atick, J . J., L i , Z „ a n d Redlich, A . N . 1992. Understanding retinal color coding from first principles. To appear i n Neural Comp. 1992. Barlow, H . B. 1989. U n s u p e r v i s e d learning. Neural Comp. 1, 295-311. Barlow, H . B . , a n d Foldiak, P. 1989. The Computing Neuron. Addison-Wesiey, N e w York. Bodewig, E . 1956. Matrix Calculus. N o r t h - H o i land, A m s t e r d a m . D e m n g t o n , A . M , a n d Lennie, P. 1982. T h e influence of temporal frequency and adaptation level o n receptive fieid organization of retinal ganglion cells in cat. / . Physioi. 333, 343-366. De Valois, R. L„ M o r g a n , H . , a n d Snodderly, D . M . 1974. Psychophysical studies of m o n k e y vision-Ill. spatial luminance contrast sensitivity tests of macaque a n d h u m a n observers. Vision Res. 14, 75-81. Field, D . J . 1987. Relations between the statistics of natural images a n d the response
properties of cortical cells. / . Opt. Soc. Am. A 4, 2379-2394.
Goodall, M . C . 1960. Performance of a stochastic net. Nature (London) 185, 557-558.
551
210
Hubel, D. H . , and Wiesel, T. N . 1974.
Sequence regularity and geometry of
orientation columns in the monkey striate cortex, ]. Comp. Neurol. 158, 267¬ 294. Kelly, D . H . 1972, Adaptation effects o n spatio-temporal sine-wave thresholds. Vis. Res. 12,89-101. Linsker, R, 1989.
A n application of the principle of m a x i m u m
preservation to linear systems.
information
In Advances in Neural Information Processing
Systems, Vol. 1, D . S. Touretzky, ed., pp. 186-194. M o r g a n K a u f m a n n , S a n Mateo, C A . Shannon, C . E„ a n d Weaver, W. 1949. The Mathematical Theory of Communication. T h e University of Illinois Press, U r b a n a . Srinivisan, M . V., L a u g h l i n , S. B., and Dubs, A . 1982. Predictive coding: A fresh v i e w of inhibition i n the retina. Proc. R. Soc. London Ser. B 216, 427-459. V a n Ness, F. L„ and Bouman, M . A . 1967. Spatial modulation transfer i n the h u m a n eye. /. Opt. Soc. Am. 57, 401-406.
Received 15 July 1991; accepted 3 October 1991.
552
Neural Codes
T h e first article of this section, (39J by Barlow et al, attempts to put into practice a principle of optimal encoding (often evoked earlier in the context of sensory representations). A simple idealised test example (coding of keyboard characters) is worked out in mathematical form. This thorough treatment of a tractable model allows for the discovery of unintuitive findings and subtleties that are likely to be missed at a more superficial level of discussion. Except for some allusions to alternative possibilities (in articles (6), (10) and (28)), the dominant viewpoint for neural coding, so far, has been that the information transmitted by a neuron is coded in its firing rate, averaged over some suitable span of time (left undefined). Also, the emphasis has been on neural representations of spatial, time-independent, visual scenes. Articles (40) and (41) extend the analysis of sensory representations to considerations of temporal aspects. Letter (40) explores a general mechanism for reconstruction of a time-varying stimulus from one and only one timesequence of neural spikes (this is a task that a behaving animal must be able to perform, in contrast with the neurophysiologist who may perform statistical averaging). T h e analysis brings together quantitative biological data and theoretical estimates about sensory information processing (signal to noise ratio, information flow per spike). Article (41) is a neat presentation of puzzling temporal aspects of visual processing, and it offers still another alternative to the firing-rate coding hypothesis by putting emphasis on differential spike arrival times — not only externally-driven but also intern ally-induced. T h e diversity of proposals described in this section is to be taken as a healthy reminder that the issue of neural codes is far from settled; indeed, further discussion of this issue will recur in subsequent chapters.
553
Reprinted with permission from Neural Compulation, Vol. l.pp. 412^23,1989 © 1989 Massachusetts Institute of Technology
Finding Minimum Entropy Codes H.B. Barlow T.P. Kaushal G.J, Mitchison Physiological Laboratory, Downing Street, Cambridge CB2 3EG, England To determine whether a particular sensory event is a reliable predictor of reward or punishment it is necessary to know the prior probability of that event. If the variables of a sensory representation normally occur independently of each other, then it is possible to derive the prior probability of any logical function of the variables from the prior probabilities of the individual variables, without any additional knowledge; hence such a representation enormously enlarges the scope of definable events that can be searched for reliable predictors. Finding a Minimum Entropy Code is a possible method of forming such a representation, and methods for doing this are explored i n this paper. The main results are (1) to show how to find such a code when the probabilities of the input states form a geometric progression, as is shown to be nearly true for keyboard characters in normal text; (2) to show how a Minimum Entropy Code can be approximated by repeatedly recoding pairs, triples, etc. of an original 7-bit code for keyboard characters; (3) to prove that in some cases enlarging the capacity of the output channel can lower the entropy.
1 Reasons for Minimum Entropy Coding When any combination of sensory stimuli — a sensory event — occurs the brain needs to know whether it is an expected, common, usual event, or an unexpected, rare, unusual event. It is evident that the brain works this out for itself because animais make startle or alerting reponses to unusual stimuli, and the following explanation of the need to attend to the unusual is also fairly obvious. One of the brains's most important jobs is to find predictive or causal relationships between the sensory events that impinge on it, the motor actions it takes, and the rewards and punishments these lead to. Now unexpected rewards and punishments are somewhat unusual, and for an animal with good knowledge of its environment they are presumably very unusual, so it follows that the sensory events it is seeking as new but reliable predictors are themselves somewhat or very unusual. For the purpose of learning something new the vast majority of sensory events can be ignored, but it is necessary to
554
413
know the prior probability of what is happening in order to select the small minority of events that might form reliable new predictors. Another way of seeing the importance of prior probabilities is to consider what is needed to detect "suspicious coincidences," which are the clues for detecting causal relations not only in detective stories but also in real life. Obviously the co-occurrence of two events is not in the least suspicious if they both occur frequently, so it is essential to know that the constituent events are rare before one can reach any conclusion at all about the significance of a coincidence that might point to a previously unsuspected causal factor. A sensory event is signaled by the joint activity of a good many nerve cells, and it is therefore necessary to know the prior probability, not just of the signals from individual cells, but of combinations and perhaps other logical functions of such signals. There is only one condition under which this can be done, short of having available a past record of the occurrence of every combination that occurs, and that is for the activities of each of the nerve cells to be independent of all others in the sensory environment to which the animal is accustomed. Minimum Entropy Coding aims to achieve this by measuring the entropies of individual representative variables and choosing the reversible code for which the sum is a minimum. The summed entropy seems an easily computed measure for assessing putative codes, and the aim of this article is to explore methods of putting the idea into practice. The general idea of Minimum Entropy Coding has been discussed by Watanabe (1981, 1985) and the principle is described in a previous article (Barlow 1989); the aim is to find a set of symbols to represent the messages such that, in the normal environment, the occurrence of each symbol is independent of the occurrence of any of the others. If such a set can be found it is called a factorial code, since the probabilities of the I s and Os in the code word are factors of the probability of the input state the word represents. In such a coded representation any nonindependence between two symbols signals the occurrence of a new, hitherto undiscovered, association, whereas with non independent codes one must always take into account the associative structure of the normal messages from the environment. The idea of minimum entropy coding comes from thoughts about redundancy reducing codes (Attneave 1954; Barlow 1960; Watanabe 1981; 1985) and is related to decorrelation (Barlow and Foldiak 1989); such a code separates knowledge of the environment derived from the redundancy of past messages from the information in the current inputs, and its specification stores the knowledge that is required for the detection of new associations. In the previous article the principle was explained using for an example the coding of keyboard characters in 7 binary digits, as in the familiar 7-bit A S C I I code. The advantages of this choice are its familiarity and simplicity, the fact that the characters occur nonrandomly with known frequencies (Kucera and Francis 1967; Zettersten 1978), and the ready
555
414
a v a i l a b i l i t y of s a m p l e s of E n g l i s h text w i t h n o r m a l statistical structure. A n a p p e n d i x s h o w s s o m e of the m o r e g e n e r a l m a t h e m a t i c a l properties of m i n i m u m e n t r o p y codes, w h i l e b e l o w w e d i s c u s s h o w the 7-bit c o d i n g c a n actually be d o n e , a n d w h a t its results are.
2 Finding the Codes
—
T h e r e q u i r e m e n t i s to code k e y b o a r d characters that o c c u r w i t h probabilities Aj o n to 7 b i n a r y o u t p u t s that o c c u r as n e a r l y as possible i n dependently
of e a c h other w h e n transmitting n o r m a l messages.
The
e n t r o p y calculated from the probabilities of the i n p u t states i s E(A) = l o g Aj, a n d for a c o d e b this is n o r m a l l y less than the s u m of the bit entropies e(A,b)
given by
e(A, b) = - £ p , log
pt-Y,
l o
8 Sfi
w h e r e p = 1 - q, i s the probability of the i t h bit t a k i n g the v a l u e 1 in the e n c o d e d form of o r d i n a r y text. T h e a i m is to find a reversible c o d e that m i n i m i z e s e(,4,b). W e a s s u m e that p, < q„ w h i c h i s a trivial restriction since s u b s t i t u t i n g its logical c o m p l e m e n t for an o u t p u t s i m p l y interchanges pi a n d q . ;
t
2.1 T h e Number of P o s s i b l e Codes. C o n s i d e r a list of the i n p u t states i n a n y order, a n d n u m b e r t h e m w i t h a s e v e n digit b i n a r y n u m ber 0000000, 0000001, 0000010, 0000011, . . . , 1111111. C h a n g i n g the o r d e r of i n p u t states i n the list c h a n g e s the code, a n d a n y reversible code c a n be p r o d u c e d by a list i n a p p r o p r i a t e order; hence the n u m b e r of reversible codes is the n u m b e r of p e r m u t a t i o n s of a list of 2 , that is (2 )!. B u t m a n y of the. a b o v e codes are e q u i v a l e n t f r o m the point of v i e w of r e d u n d a n c y . O n e c a n substitute the c o m p l e m e n t for e a c h output, interchanging the v a l u e s of p a n d q a n d o b v i o u s l y l e a v i n g the entropy u n c h a n g e d . Since this c a n be d o n e i n d e p e n d e n t l y for e a c h of the o u t p u t s o n e m u s t d i v i d e b y 2 . I n a d d i t i o n , one c a n p e r m u t e a l l the o u t p u t s w i t h o u t c h a n g i n g the entropy s u m so one m u s t also d i v i d e b y 7\. T h u s the total n u m b e r of n o n e q u i v a l e n t c o d e s i s ( 2 ) ! / ( 2 • 7!), still a v e r y large number. 7
7
7
7
7
2.2 Finding the Code with the M i n i m u m Sum of B i t E n t r o p i e s . W e h a v e failed to find a g e n e r a l m e t h o d , other than searching t h r o u g h a l l codes a n d c a l c u l a t i n g the s u m m e d entropy for each, w h i c h the n u m b e r of codes m a k e s i m p r a c t i c a l . W e h a v e so far tried three m e t h o d s that are d e s c r i b e d briefly below. M i n i m i z i n g the s u m of 7 quantities that obey the constraints of the bit probabilities of a reversible code h a s the flavor of a p r o b l e m suitable for a n e t w o r k solution, b u t this h a s not yet been tried. O n e a p p r o a c h is to m i n i m i z e the a v e r a g e probability of a bit b e i n g active, since g i v e n that p, < 1/2, the l o w e r it is the less its i n d i v i d u a l
556
415
Sum of bit entropies r{A,b)
Code
Redundancy f? (%)
M i n i m u m bit probability Binary sequence
4.50 4.37
3.7 0.7
ASCII
5.42
24.9
Random-1 Random-2
6.89 6.77
58.8 56.0
Random-3
6.74
55.3
Table 1: Entropies and Redundancies of Different Codes. Character entropy E(A) ~ 4.34 bits. Redundancy R = \e{A, b) E(A)]/E{A).
c o n t r i b u t i o n is to the s u m m e d entropy. T h i s c a n be done a s follows.
First
m e a s u r e the probabilities of e a c h of the i n p u t states a n d r a n k t h e m i n o r d e r of d e c r e a s i n g probability. T h e most probable is a s s i g n e d the o u t p u t w i t h a l l 0s, the next 7 are a s s i g n e d to the o u t p u t s w i t h just one bit active, the next 21 to those w i t h 2 active, the next 35 to those w i t h 3 active, and
so on.
W e call these Minimum
Bit Probability codes. A l t h o u g h this
g i v e s the l o w e s t a v e r a g e n u m b e r of active outputs, it d o e s not necessarily m i n i m i z e the s u m of bit entropies (see A p p e n d i x ) . T a b l e 1 s h o w s the v a l u e s of the a v e r a g e entropy E(A)
obtained from
the c h a r a c t e r frequencies i n a p o r t i o n of a scientific text, a n d the s u m of bit entropies for the s a m e text c o d e d in this way.
F o r a factorial code
they w o u l d be e q u a l , a n d it w i l l be seen that the s u m of the bit entropies c o m e s q u i t e close to the c h a r a c t e r entropy. A n o t h e r c o d e c a n be o b t a i n e d b y listing i n p u t state probabilities as before a n d then n u m b e r i n g t h e m i n a n o r d i n a r y b i n a r y sequence.
This
Binary Sequence Coding g i v e s a factorial code p r o v i d e d that the p r o b a b i l ities of the i n p u t states f o r m a geometric p r o g r e s s i o n (see A p p e n d i x ) . It w i l l b e s e e n from table 1 that this c o d e reduces the s u m m e d bit e n t r o p i e s to v e r y n e a r l y the s a m e as the character entropy, the r e s i d u a l r e d u n d a n c y b e i n g o n l y 0.7%. The
T h u s , it is v e r y nearly a factorial code.
r e a s o n s for this s u c c e s s are that the character probabilities form a
g e o m e t r i c p r o g r e s s i o n , a s s h o w n i n F i g u r e 1, for w h i c h b i n a r y s e q u e n c e c o d i n g y i e l d s a factorial c o d e . W e c a n c o m p a r e t h e s e c o d e s w i t h the r e g u l a r 7-bit A S C I I code, the e n t r o p i e s for w h i c h a r e a l s o s h o w n i n Table 1.
T h o u g h this w a s not
d e f i n e d w i t h a n y i d e a of m i n i m i z i n g e n t r o p y in m i n d , the s u m m e d bit e n t r o p y is i n fact a great d e a l less than 7, because the m o r e frequently u s e d letters h a v e f e w e r active bits.
2.3 S u b s e c t i o n C o d i n g . I n o r d e r to test h o w successfully the next set of r e c o d i n g p r o c e d u r e s m i n i m i z e d the entropy w e n e e d e d to start
557
416
0-.
g I
c, . 0.01-
DC
a.
£ io <
0.001 -
5
0.0001-
I
o 0.00001
—r10
I
20
I 30
40
50
60
70
I 80
90
CHARACTER RANK
Figure 1: The ranked character probabilities plot close to a straight line on these log-linear axes, showing that the ranked probabilities form a geometric progression. The text sample analyzed was a portion of a scientific text similar to this paper, and was composed of approximately 42,000 printable ASCII characters. Similar distributions are commonly found (Kucera and Francis 1967; Zettersten 1978). with a 7-bit binary code with maximum entropy, so we have generated random codes in which the binary numbers up to 1111111 were assigned at random to the input states; three examples of such codes are included in Table 1. Minimum bit probability and binary sequence codes are based on measurements of the frequencies of the 128 different input states of a 7-bit code. This is not difficult to do, but it would become impossible if one were dealing with the very large input channels that feed the brain, so we have been very interested in how well one can achieve a minimum entropy code by repeated recoding of small subsections. The codes listed in Table 2 as pairs were formed by taking 2 bits at a time, recoding on the principle of mimimum bit probability or binary sequence coding, and doing this repeatedly until no further improvement occurred. Similar codes were formed taking 3, 4, or 5 at a time for those marked, triples, quadruples, and quintuples. For the minimum bit probability code, Figure 2a graphs the summed bit entropies as a function of the number of recoding attempts, taking different numbers of bits at a time; Figure 2b does likewise for the binary sequence code. It will be seen that recoding small subsets of elements is surprisingly ineffective in produc-
558
417
Code
Subsection
M i n i m u m bit
Binary
probability
sequence
A
HA.b)
R
(%)
Table 2:
{%)
Pairs
5.76
32.7
5.76
32.7
Triples
4.61
6-2
4.75
9.4
Quadruples
4.55
4.8
4.44
2.3
Quintuples
4.51
3.9
4.37
0.7
A l l 7 bits
4.50
3.7
4.37
0.7
Entropies and Redundancies ot Subsection Codes;
Recoding of
Random-2 by Subsection Method. Entropy of Random-2 code = 6.77 bits; character entropy EiA)
= 4.34 bits; redundancy R = [e{A, b) -
E(A)\/E(A).
i n g l o w e n t r o p y c o d e s , a n d m a n y repetitions are n e e d e d to a p p r o a c h l o w values.
2.4 E x p a n s i v e C o d i n g . It is s h o w n in the a p p e n d i x that c o d i n g o n to a n e x p a n d e d c h a n n e l c a n i n s o m e cases decrease the entropy. T h i s is u n e x p e c t e d , b e c a u s e i n s u c h cases s o m e of the o u t p u t states d o not o c c u r a n d this r e p r e s e n t s a d d i t i o n a l r e d u n d a n c y .
F u r t h e r m o r e , the r e c o d i n g
t e n d s to i n t r o d u c e n e g a t i v e d e p e n d e n c i e s b e t w e e n s o m e of the o u t p u t v a r i a b l e s , a n d w h e n this h a p p e n s the code cannot be a true m i n i m u m e n t r o p y code.
N o n e t h e l e s s , w h e n w e c o m p a r e the efficiency of associ-
a t i o n f o r m a t i o n w i t h v a r i o u s c o d e d representations w e p l a n to i n c l u d e s u c h e x p a n d e d c o d e s s i n c e they h a v e some p l a u s i b i l i t y a s the type of c o d i n g that the c e r e b r a l cortex does.
3 Conclusions
T h e 7-bit c o d i n g of k e y b o a r d c h a r a c t e r s has s e r v e d to illustrate the p r o b l e m of f i n d i n g m i n i m u m e n t r o p y codes.
In this e x a m p l e there h a p p e n s
to be a g o o d a p p r o x i m a t i o n to a factorial code b e c a u s e the probabilities of the i n p u t states f o r m a n a p p r o x i m a t e geometric p r o g r e s s i o n , but it d o e s not l e a d to a c o d e i n w h i c h the i n d i v i d u a l bits h a v e a n y r e a d i l y i n t e r p r e t a b l e significance; for i n s t a n c e the bits i n the c o d e d o not corres p o n d to significant features of the characters, s u c h a s b e i n g v o w e l s or n u m b e r s (see T a b l e 3). A m i n i m u m entropy code w o u l d c o m e into its o w n w h e n the states b e i n g c o d e d h a v e a natural c o m b i n a t o r i a l character, b u t it is u n k n o w n h o w far this is true of the w o r l d of s e n s o r y d a t a .
559
418
(b)
Subsection coding u s i n g binary sequences
7_
NUMBER OF R E C O D I N G S A P P L I E D
Figure 2 : Coding subsections of the input.
These graphs show the gradual
reduction in entropy that is typically achieved by repeatedly recoding subsections of the 7-bit input.
In (a) a Minimum Bit Probability code w a s applied
to pairs, triples, etc, of bits; the group of bits { 6 ; , 6 j , 6 i - , . . . ) chosen for recoding was the one that maximized P(b, & bi k & ...) - P(b,)
- P(bj)
•
P(6 .).... (
A x
1
criterion was also tried, without improvement. Coding was repeatedly applied to pairs, triples, etc. until there was no further reduction of the s u m m e d bit entropy. The procedure in (b) was the same, except that a binary sequence code w a s applied to the pair, triple, etc. selected for recoding. The character entropy, marked on the graphs at 4.34 bits/character, is the theoretical minimum that can
be achieved by any recoding, and the better the coding, the closer and faster
the curves w i l l fall to this baseline. A s can be seen in (b), the full width (7-bit) binary sequence code achieves this goal in a single step.
560
419
Rank
0 1 2 3 4 5 6 7 S 9
10 11 12 13 14 15 16 17 18 19 20 21 22 21 24 25 26 27 26 29 30 31 32 33 34 35 36 37 38 39 40
Character 'space' e t i
0 a n •
r h 1 c u Li
f
n
b P
8
y w V (
N
fc 1 T X
q ) (
V p i
sB A
;
2
Code 6543210
0000000 0000001 0000010 0000011 0000100 0000101 0000110 0000111 0001000 0001001 0001010 D001011 0001100 0001101 0001110 0001111 0010000 0010001 0010010 0010011 0010100 0010101 0010110
0010111
0011000 0011001 0011010
0011011 0011100
0011101 0011110 0011111 0100000 0100001 0100010 0100011 0100100 0100101 0100110 0100111 0101000
Probability
Rank
0.17595 0.O9843 0.07817 0.06272 0.06214 0.05791 0.05773 0.05620 0.04375 0.03604 0.03050 0.02610 0.02610 0.02345 0.01953 0.01796 0.01768 0.01768 0.01231 0.01032 0.01025 0.00943 0.00753 0.00490 0.00253 0.00251 0.00195 0.00195 0.00181 0.00176 0.00171 0.00150 0.00148 0.00129 0.00122 0.00117 0.00117 0.00117 0.00108 0.00098 0.00096
Code 6543210
Probability
41
i.
4:
O C
0101001 0101010 0101011 0101100 0101101 0101110 0101111 0110000 0110001 0110010
0.00070 0.00066 0.00063 0.00061 0.00056 0.00054 0.00052 0.00047 0.00045 0.00042 0.00033 0.00028 0.00028 0.00028 0.00026 0.00026 0.00021 0.00019 0.00019 0.00019 0.00016 0.00014 0.00014 0.00014 0.00012 0.00012 0.00009 0.00009 0.00009 0.00007 0.00007 0.00007 0.00007 0.00007 0.00005 0.00002 0.00002 0.00002 0.00002 0.00002 0.00002
43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81
1'rob.ibililv ol each bil Bit 6 5 4 3 2 1 0
Character
Probability 0.001196 0.018945 0.110085 0.247532 0.368637 0.401721 0.439799
Table 3: Actual Binary Sequence Code
561
0
R
j
/ 1 F
'
D
M H L
1
6 Q w •
onoon
0110100 0110101 0110110 0110111 0111000 0111001 0111010
onion
<
0111100 0111101 0111110 0111111 1000000 1000001 1000010 1000011 1000100 1000101 1000110 1000111 1001000 1001001 1001010 1001011 1001100 1001101 1001110
J I 1
1010000 1010001
i 3 G
+
u 4 f X 5 8 9
> K t
% 7
loonn
420
A l t h o u g h i n this case the c o d e d o e s n o t i m p o s e a p a r t i c u l a r l y inform a t i v e classification o r c a t e g o r i z a t i o n o n the c h a r a c t e r s , it n e v e r t h e l e s s has the a d v a n t a g e s p r o m i s e d i n t h e i n t r o d u c t i o n , n a m e l y that the p r o b ability of the c h a r a c t e r s c a n be p r e d i c t e d from the probabilities of the bits that represent t h e m . T r a n s l a t i n g this to the field of s e n s o r y e v e n t s , if these w e r e represented b y v a r i a b l e s that o c c u r r e d i n d e p e n d e n t l y u n der n o r m a l c i r c u m s t a n c e s , one's ability to detect u n u s u a l o r s u s p i c i o u s c o i n c i d e n c e s w o u l d be e n o r m o u s l y e n h a n c e d .
4 Mathematical Appendix (G.J. Mitchison) S u p p o s e w e a r e g i v e n a set of states w h o s e probabilities a r e A
}
(with
£ Aj - 1), a n d a code b that a s s i g n s to e a c h c h a r a c t e r a b i n a r y string. I n the c o d i n g p r o b l e m c o n s i d e r e d i n the m a i n text the states are k e y b o a r d characters. L e t t h e ;th bit of the c o d e for the j t h state be d e n o t e d b y b,j. T h e probability of the i t h bit t a k i n g the v a l u e 1, s u m m e d o v e r a l l the states, i s denoted b y p,. F o r m a l l y
P*=£4
M.l)
a n d w e define q = 1 - p * . We s a y that b is a factorial code if the p r o b a b i l i t y t
of e a c h state A, is the p r o d u c t of the probabilities that the bits h a v e the v a l u e s i n the c o d e for A
r
Formally,
Aj = JTJ
(4.2)
where
m=n »•n s <>,j=l
t.j=0
It is a s s u m e d here that there i s a state for e a c h p o s s i b l e p o l y n o m i a l JTj, i.e., that there are 2
N
states, w h e r e N i s the n u m b e r of bits.
T h e state e n t r o p y E(A) i s d e f i n e d b y E(A) = - £ A, log Aj. T h e s u m of the bit entropies, e(A,b), P r o p o s i t i o n 1. e(A,b)
is d e f i n e d b y e(A,b) > E(A), with e(A,b)
- - EPilogp, -
Eg;log5,.
= E(A) if and only if b is
factorial. T h i s p r o p o s i t i o n is p r o v e d i n W a t a n a b e (1985). I n t u i t i v e l y , it m e a n s that the c a p a c i t y of t h e bits of t h e c o d e i s greater than that of the states they represent, except i n the case of a factorial c o d e w h e r e t h e states are represented b y a p r o d u c t of i n d e p e n d e n t bits. T h e f o l l o w i n g short proof u s e s a n a d a p t a t i o n , s u g g e s t e d b y Dr. C . J . S t C . W e b b e r ( p r i v a t e c o m m u n i c a t i o n ) , of a n a r g u m e n t i n J o n e s (1979):
562
421
Proof.
U s i n g (4.1) a n d the definition of p, f o l l o w i n g (4.2) w e c a n w r i t e :
e(A,b)~E(A)
=
-I>aogp -£
=
- £ < £ ^ > i ° 8 K - £ < £ ^ ) i ° g f t
i
9
b„"0
l,j=l
+ =
+
A
}
log
Xj
- ^ l o g f ^ / A , )
I t is e a s y to c h e c k that l o g a; < x - 1 for x > 0, w i t h e q u a l i t y o n l y for i = l . Thus: A
-T. >
I°S - £
-1) =£ $ - £
= i - 1 =o
u s i n g the fact that £ T T J = rj(p. + ft) = 1. E q u a l i t y o c c u r s if a n d o n l y if Kj = Aj for a l l j , w h i c h m e a n s that b is factorial. T h i s r e s u l t i m p l i e s that, for a n y set {A } that has a factorial c o d e , a n y c o d e b w h i c h m i n i m i z e s the e n t r o p y e C 4 , W Will also be factorial. M i n i m i z i n g e C 4 , b ) c a n therefore be r e g a r d e d as a strategy for f i n d i n g a c o d e that is "as factorial a s possible." O f c o u r s e , not e v e r y set of 2 state p r o b a b i l i t i e s h a s a factorial code. I n fact, the factorial c o d e s h a v e d i m e n s i o n N ( o n e for e a c h c h o i c e of the N g j s j a n d the sets of Aj w i t h 52 Aj = 1 h a v e d i m e n s i o n 2 - 1, so it is h i g h l y u n l i k e l y that a n a r b i t r a r y set {Aj} w i l l h a v e a factorial code. H o w e v e r , one case of interest is the following: }
N
N
Proposition
2. If A = Kx>, for i = 1 . . . 2", for some x, with K chosen so that £ Aj - 1, then A has a factorial code.
Proof.
3
2
D e f i n e p, b y Pijqi = x '~\ 3
s e q u e n c e c o d e , Hj = Q • x ,
for i = 1 to / V . T h e n , for the b i n a r y
w h e r e Y,Qx'
- 1. It f o l l o w s that Q = K a n d
Aj = TTj, so A h a s a factorial code. T h i s r e s u l t i l l u s t r a t e s the difference b e t w e e n m i n i m i z i n g the s u m of bit e n t r o p i e s eiA,b)
a n d m i n i m i z i n g £p-,
the total p r o b a b i l i t y of the
bits. T o a c h i e v e the latter one s h o u l d a s s i g n the b i n a r y s t r i n g c o n s i s t i n g entirely of zeros to the largest A^, the s t r i n g s w i t h just one "1" to the next s m a l l e s t A,s,
a n d so o n .
T h e factorial c o d e for i n p u t states w i t h
p r o b a b i l i t i e s i n a g e o m e t r i c p r o g r e s s i o n d o e s not follow this s e q u e n c e . U n t i l n o w w e h a v e a s s u m e d that there are N bits a n d 2
N
states. M o r e
g e n e r a l l y , one c o u l d try to c o d e M states o n N bits, w h e r e N > l o g
2
M,
w i t h the c o n v e n t i o n that one a s s i g n s a p r o b a b i l i t y of zero to the codew o r d s that a r e n o t u s e d . A s i m p l e e x a m p l e s h o w s that a l o w e r e n t r o p y c a n s o m e t i m e s be o b t a i n e d b y c o d i n g o n to m o r e than the m i n i m u m pos2
s i b l e n u m b e r of bits. L e t the state probabilities be a, a, Ko?, 1 - 2a — Ka , 2
w h e r e 0 < a < 1 a n d \ —2a0
0
0
1
1
0
1
1
Ka
> 0. O n e can c o n s i d e r t w o codings:
w i t h probability 1 - 2 a -
a a Ka
1
563
Ka
2
422
and
0
0
0
0
0
1
0
1 0
1 0
with probability 1 - 2 a -
Ka
1
a a Ka?
0
3
It is s t r a i g h t f o r w a r d to check that w h e n a is s m a l l e n o u g h for t e r m s i n a
a n d h i g h e r p o w e r s to be neglected, then the s e c o n d ( e x p a n s i v e ) c o d i n g g i v e s a l o w e r e n t r o p y w h e n l o g K > 3. r
O n e c a n p r o v e other r e s u l t s of this k i n d . F o r i n s t a n c e , if there a r e M states, M - 1 of w h i c h h a v e the s a m e p r o b a b i l i t y a, then the e x p a n s i v e c o d i n g o n M -1 bits that a s s i g n s a c o d e w o r d w i t h a s i n g l e 1 to e a c h state of probability a a n d the a l l - z e r o c o d e w o r d to the state w i t h p r o b a b i l i t y 1 — ( M - D a , g i v e s m i n i m u m entropy p r o v i d e d a is s m a l l e n o u g h . F o r s u p p o s e there w e r e a c o d i n g o n L bits w i t h L < M — 1, a n d let t h e bit probabilities p, be ak
if
w h e r e the fc; a r e integers.
S i n c e there m u s t be
at least o n e bit w h i c h i s set to 1 i n t w o or m o r e c o d e w o r d s it f o l l o w s that
> M.
F o r s m a l l bit probabilities w e c a n a p p r o x i m a t e t h e bit
entropy b y - £ p , logp;. T h e n the difference A e b e t w e e n the bit e n t r o p i e s for the c o d i n g o n L bits a n d that o n M — 1 bits i s a p p r o x i m a t e l y A e = — E a f c j l o g l o ^ l + t M - D n l o g a > — a l o g a - a E f c i l o g f c ; . S i n c e e a c h fc; < M 2
the m a x i m u m v a l u e of E f c ; l o g f c ; is M l o g M , s o A e > 0 p r o v i d e d that !
- l o g o > M l o g M or a<
M
(l/M) '.
References
Atrneave, F. 1954. Informational aspects of visual perception. Psychol. Rev. 61, 183-193. Barlow, H.B. 1960. T h e coding of sensory messages,
pp. 331-360.
I n Current
Problems m Animal Behaviour, W . H . Thorpe and O . L . Zangwill, eds., C a m bridge University Press, Cambridge. Barlow, H . B . 1989. Unsupervised learning. Neural Comp. 1, 295-311. Barlow, H . B . , and Foldiak, P.F. 1989. Adaptation and decorrelation in the cortex. I n The Computing Neuron, R. Durbin, C . Miall, and G.J. Mitchison, eds. Addison-Wesley, N e w York. Jones, D.S. 1979. Elementary Information Theory. Oxford University Press, London. Kucera, H , a n d Francis, W.N. 1967. Computational Analysis of Present day American English. Brown University Press, Providence, R I . Watanabe, S. 1981. Pattern recognition as a quest for m i n i m u m entropy. Pattern Recognit. 13, 381-387. Watanabe, S. 1985. Pattern Recognition: Human and Mechanical. Wiley, N e w York.
564
423
Zettersten, A , 1978.
A Word Frequency List Based on American English Press
Reportage. Universitetsforlaget,
1 Kopenhavn, Akademisk Forlag, Copen-
hagen.
Received 30 March 1989; accepted 22 May 1989.
565
Reprinted wilh permission from Science, Vol.2S2.pp. 18S4-LB57. 1991 ^ 1991 The American Association for the Advancement of Science
Reading a Neural Code WILLIAM BIALEK," FRED RTEKE,* ROB R . DE RUYTER VAN STEVENINCK,! DAVID WARLAND Traditional approaches to neural coding characterize the encoding of known stimuli in avenge neural responses. Organisms face nearly the opposite task—extracting information about a n unknown time-dependent stimulus from short segments o f a spike train. Here tbe neural code was characterized from the point o f view of the organism. Culminating i n algorithms for real-time stimulus estimation based o n a single example o f
ihe spike train. These methods were applied to an identified movement-sensitive neuron in the fly visual system. Such decoding experiments detennined the effective noise level and fault tolerance o f neuraJ computation, and the structure of the decoding algorithms suggested a simple model for real-time analog signal processing with spiking neurons.
A
LL OF AM ORGANISM'S INFORMA-
•on about the sensory world comes from real-time observation of the activity of its own neurons. Incoming sensory information is represented in sequences • f essentially identical action potentials, or "spikes." To understand real-time signal processing in biological systems, one musr first understand this representation; Does a single neuron signal only discrete stimulus "features," or can the spike train represent a continuous, time-varying input? How much information is carried by the spike train > Is the reliability of the encoded signal limited by noise at the sensory input or by noise and inefficiencies in the subsequent layers of neural processing* Is the neural code robust to errors in spike timing > Clear experimental
W. Bialek. F. Riekc, D. Warlind, I >rpinmrnr. of Phytic* ind I >f pirrmenl of Molecular ind Cell Biology, Univtf•!':.:••!; Berkeley Berkeley CA 94720. P.. R. &Ru|rtCTvw5ic*eniric< Depigment of Biophysics RjjkjurUvcrsirclT Grwungtrv ihe Nethcrlandi. 'Promt !.'••. NEC Research Iruriime, 4, Independence Ytiy. Princeton. N | 08?40 tPrcsent iddirss. Department of Audio logy, UnJversirv Hospital, P.O. Bu* i0.00l 9700 KB. Groniiigen, [he NcihcrUndj.
answers to these questions have been elusive {1, 2). We present an approach to the characterisation of rhe neural code that provides explicit and sometimes surprising answers to these questions. The first recordings from single sensory neurons demonstrated that the intensity of a static stimulus can be coded in the firing rate of a sensory neuron (3). This concept of rate coding, extended to time-dependent stimuli, provides the framework for most studies of neural coding, leading to the definition of receptive fields, temporal filter characteristics, and so on. Beyondratecoding, a variety of different statistical measures have been proposed—interval distributions, correlation functions, and so form (J 4). As with the rate itself, rhesc quantities can be seen as moments of the probability distribution TKJk'Oj that describes the likelihood of different spike trains {rj, given the stimulus s[i) (5). These moments, however, are not properties of a single spike nain; they are average properties of an ensemble of spike trains {6). Organisms rarely have the opportunity to compute these averages: To say that information is coded in firing rates is of no use to the organism unless one can
b
566
(
3855
explain how the organism could estimate these firing rates from real-time observation of the spike trains of its own neurons (7). The simplest problem of real'timc signal processing is decoding the spike train to estimate the signal waveform. If one chooses an inappropriate definition of the signal this reconstruction will fail; for example, we have srudicd auditory neurons that provide enough information for reconstruction of the envelope of the acoustic stimulus but not of the waveform itself (5). One can define the signal encoded by a particular neuron to be that signal which is reconstructed most accurately from observation of the spike train. If one can reconstruct analog signals, then one can begin to understand how spike trains could be manipulated in subsequent stages of neural circuitry to perform more complex processing of these signals. It may not be possible, however, to interpolate berween the discrete spikes ro estimate a continuous stimulus. The deeding problem is completely specified by the probability P[S(T)|{J,}] of a particular stimulus waveform J ( T ) conditional on the spike train {f,}. From this distribution. One can estimate the stimulus, for example, by rinding the function oftimethat maximizes ^{s(T)|(fJ]. Thus, one approach lo the decoding problem is to design experiments that directly measure P [ J ( T ) J { * } ] (2). An alternative approach is to model the encoding process and analytically develop decoding algorithms within the context of the model; this approach (9) indicates that there is a broad regime in which linear filtering of the spike train results in an optimal estimate of the stimulus waveform [JO). Arc such simple decoding algorithms applicable to a real neuron? The problem of reading the neural code is essentially the problem of building a (generally nonlinear) filter that operates continuously on the spike train to produce a real-time estimate of the unknown stimulus waveform (Fig. 1). If the spikes arrive at times {f,}, our estimate of the signal is (
in Eq. 1 were not significantly different, as quantified below. The filters used in the reconstruction integrated over short time intervals, so the optima] estimate of angular velocity at each instant of time was controlled by just a handful of spikes, as expected from behavioral srudies. How good are the reconstructions* The reconstructions consist of a piece that is Fig. 1. Schematic view of die decoding process The "black box filtersdie spike train input (rJ to determinisrically related to the stimulus and a random noise piece. We separated these by produce an estimate - of rhe itimulus. introducing afrequency-dependentgain such that s ,(ni) = £ ( O I } |J() is approximately Gaussian. entire visual field (JJ). Flies and other inPlotting the spectral density of the angular sects exhibit visually guided Highly during displacement noise, we found (Fig. 3) a peak chasing behavior, course corrections can oo signal-to-noise rado (SNB,} of berter than cur on time scales as short as 30 ms (14). 5T, and an SNR. of greater than one across a The maximum firing rate in H I is 100 to bandwidth of roughly 25 Hz. Using Shan200 s" , so behavioral decisions are based non's formula (16), shown in Eq. 2 , wc On just a few spikes from this neuron. Fur- converted these spectra into an estimate of the thermore, the horizontal motion detection average rate at which we gained information system consists of onlv a handful of cells, so about the stimulus by virtue of observtheflyhas no opportunity to compute aver¬ ing the spike train; the result was 64 ? 1 bits age responses (for example, firing rates). per second (baud). In our experiments, the stimulus i(i) was dm the angular velocity of a rigidly moving flinfo = — I0g2 1 + (2) random pattern. We chose J(T) from an 71
a
J
f
1 2TT
ensemble that approximated Gaussian noise with standard deviation 132 deg s J the spectrum of I [ T ) was constant up to a cutoff frequency of 1 kHz. We recorded the spike arrival times exnacellulariy from the H I neuron [2, 15). We began by trying to reconstruct J(T) with just the linear teim of the general expansion, Eq. 1 (Fig. 2). Reconstructions including higher Order terms
The second term in the expansion improved the information rate by less than S%. We have not explored conditions that might maximize this information transmission. The noise level achieved in the reconstructions is the noise against which an observer of the H I spike trains (such as the fly) must discriminate to estimate horizontal motion. Fig 2. First-Order re instruction (solid method 1 (11). The stimulus is shown dotted line for comparison, and the spike train, ii shown .> die bottom. This example is from a segment of the spike train that was not used in the filter calculations. Stimulus and reconstruction were smoothed with a 5-ms half-width ' '<•:..- •;. filter. (Inset) filters calculated from methods 1 and 2 (If J; the time scale tone) using J
is the Same as the mainfigure,the Male bar
= 10 deg I \ and the arrow marts f = 0 HI has a highly asymmetric response profile, with a much larger dynamic range for movement in the excitatory direction. To compensate for this asymmetry, we recorded spike rrJ: ii-. in response to r[i/ and ro the inverted stimulus —s(t,; these are shown as positive and negative spikes in the spike train- These two spike trains approximate the trains thai would be generated by HI cells, on opposite sides of the head during a rigid rotation of thefly(2). Our reconstruction is then -
To optimize the reconstructions one chooses thefilters{F } to minimize \ = J di\s{t) - j (f)j , where s(f) is the true stimulus and the integration is over the duration of rhe experiment (Jj).Thr stimulus s{i) is nor restricted to simple sine waves or Gaussian noise, and thus one can study the coding of complex and naturalistic signals {8, 12). In this initial experiment, however, we used relatively simple stimuli. 2
n
1
a
[
0700
8800
8900
9000 BIO0 Tlma {maj
567
j .w=2i i<'- 'i-' i"-'r)i n
F
,
:
1856
C o d i n g is often used to reduce the effects o f noise on signal transmission. Does the neural code have any such noise immuniryr For several noise sources, such as riming errors, dropped spikes, and spontaneously generated spikes, we created an ensemble of spike trains which were randomly corrupted versions o f the original data and then rreated *inhE(l/2M+o/W) ] (*) these as new data that required decoding. when: 5« if die spectral densiry of the random contrast paneto used in the eiperiment. L\$< is the We were able to recover 95% o f the original information R for the following noise widrh of rhe photoreceptor aperture, 4> is the photoreceptor spacing along the direction of levels: (i) spikes added to increase the firing movement, 7"() is the spectral density of voltage noise in deviation o f 2 ms inrroduced to each spike rhe phororccepior. The limiting noise power spectrum varies as the inverse of die number N of rime. One objection to 'spike timing as a photoreceptors. These quantities were measured coding strategy is the need for precise meain photoreceptor recordings under conditions surement o f spike arrival times. I n the case identical to tflOSC used for the H I experiments o f H I , this objection is irrelevant. T h e code { } $ ) , SO wc can make meaningful comparisons of iheory and experiment. is robust to errors of several milliseconds in spike timing and to other corruptions of the spike train.
4) I n vibratory receptors o f the bullfrog sacculus, we reconstructed the waveform o f groundbome vibration using a linear decoding algorithm, although in this case the reconstructions improved substantially (by - 1 0 to 15%) with the addition o f secondorder terms (S). W e again measured information rates close to 3 bits per spikeIt is, o f course, not known if organisms perform the sort o f reconstructions demonstrated in F i g . 2. Because linear reconstruction is possible, however, analog processing o f the encoded signals can be done in a simple way. I t is not unusual for the postsynaptic voltage response to a single presynaptic spike to have the qualitative form o f the optimal filter, with a relatively sharp positive peak followed by a slower negative tail. T h u s simple synapses could serve as decoders, W i l h this decoding done, cells could then perform analog computations using the nonlinearitics contributed by voltage-gated channels along the dendrites and cell body, much as envisioned for nonspiking cells (20). T h e results o f such analog computations could then be encoded by the spike-generating region o f the cell, and the process could then begin again. I n this view o f computation with spike trains, the combination of nonlinearities in spike generation and the filter characteristics o f the synapse results in an essentially linear transmission o f analog signals from presynaptic cell bodies to postsynaptic dendrites. T h e dramatic "all-or-none" nonlinearities of spike generation (the focus of so many models for neural computation) are then not as important as the more subde analog dynamic properties o f nonspiking regions o f the cell.
T h e absolute noise level o f the reconstructions is very low. With a bchaviorally relevant integration tune of 30 ms, one could judge the amplitude of a 20-Hz. dither to within - 0 . 1 deg, which should be compared to the photoreceptor spacing o f
It is surprising that time-dependent signals can be recovered so simply from neural spike trains. Reconstruction o f the stimulus waveform permitted quantification of the fault tolerance o f the code and allowed us to show that the fly visual system approaches optimal real-time computation These results demonstrate that die representation o f time-dependent sensory data in the nervous system is simpler than might have been expected. Correspondingly simpler models o f sensory signal processing may be appropriate.
rent voltage in one cell by the alternating current voltage in its nearest neighbor ( 5 ) . T h i s is cssenti allv the "correlation* scheme for movement detection proposed by Reichardt (f8)\ in our case this algorithm was not a minimal model but rather the optimal computational strategy. Analysis of the correlation scheme (J) led to the limiting angular displacement noise level shown in F i g . 3, where the displacement noise from the linear reconstruction approaches the limits imposed by the photoreceptor noise, at least at frequencies above 10 H z , where our theory o f the limiting noise level is valid. T h e fly visual system thus performs an optimal and nearly noiseless extraction of movement signals from the array o f photoreceptor voltages. h
Frequency (Hi) Pig. 3. Stimulus level (smooth: curve) and spectral density of displacement neat* from die reconstruction (middle C U J V I | E u bottom curve is die limit to the resolution of small displacements (valid for frequencies > J 0 Hz] set by noise in die photoreceptor array (5)
3
a
lnIa
v
71
Information about movement across the visual field is earned in the sparioremporal correlations of photoreceptor outputs, but these correlations are degraded by noise in the photoreceptors. H o w accurately can one estimate rigid morion if one optimally processes these noisy photoreceptor signals? I n our stimulus ensemble, the angular displacements were small (6fl < <];, •• for frequencies above 10 H z . I n this limit, the optimal movement estimator involves multiplying the direct cur-
Preliminary results from four studies suggest that linear decoding is not a special property o f H I under particular stimulus conditions but a more general property of sensory neurons. 1) I n H I , we performed reconstruction experiments using stimuli with different spatial characteristics. These different stimulus ensembles provided different S N B s , and these differences were reflected in the reconstructions. Despite large changes in SNR., linear reconstruction continued to work and the reconstruction filters were essentially identical up to a constant scale factor. 2) I n the mechanoreceptor cells of the cricket cereal system, the displacement wave form for motions o f the filiform hairs was reconstructed from the spike trains o f primary afferent neurons (12). F o r this system, the information transfer rates exceeded 3 0 0 baud (~-i bits per spike). 3) I n simulations on realistic models (19) for spike initiation, we reconstructed the waveform o f injected currents by linearly filtering the spike train.
568
REFERENCES AND NOTES 1. D. Pcikrl and T Bullock^ MpHHft Ka. Program BuK. 6, 221 {t968}. 2. ft. ft. de ftuyter van Sieveninck and W. Biileh, Pint. P. S « . Lordon Ser. fl 234. 379 (IMS). 3. E. D. Adnan. / . Phytlci. 61. 47
1857 possible- iigturitante of lemporsj JTUCTOtt and COfOuuiJ System (RjjkjunlverslTeil GTOningcn. GfOflnrlilctlfinngin the visual cortex [C M Gray and W. ingen. die Wet tier lands, I986J. Singer, * W Nrftf. / I M J . Al. U SA. &b. lo9B 16. C. Shannon, Btit SyP. Trth jw 17, i7V (19411) (1989)). All ttac obscrvaTions, however. involve 17 G. VVtstheiineT, frwif. Ophlhatmet. 14, S70 (1975). measurements of average properties of spike rraiiu 1H W Reichardt, m Ptiwxiptii o/$rwa^ CemwMMvri. tJtm repeated prescntaoOiu of 1 nlmuJui. Although W Rosenblith. Ed. (Wiley. Mew York, 1961}. •!.- - arc • ' of coding strategic, 19. C. Koch and 1. Segev. Eds.. Mtiheds •n ,\r, «' that djfltnrnr systems mighi use. none directly anAfrfrf^ JMIT Press, Cambridge. MA, 1969) swer, rhe question o f how rhe Qrgtnum could 20 C. Koch.T. Poggio, V. Torre, Pmr. Sail, l .' Sfl. Citfitl thji coded information from single spike U S-4_ 80, 2799 |1983.>. traiTD in m l ame 21 Wc ehank W L Brlu» M Criir, W Geniner, L S. W Bufck, in I W U r t i m O f l c . SyXfJ, SFI KrugJyak, } P MiQer, W G O-Cn. A Zee. and SlWia m Ac Sn™a ^" C-nplcnjy. E. Jen Ed. G /.wepg lor helpful diJi\iiiiL^r'L ir J 11 A liylDc. (AddnOt. Wok*. Reading. MA. 1990), vol 2. pp T> GEuer. and M. Meister for ihoughnSd com' SI3-S9Smenrs on the manuscript Preliminary work on simulaiHHU of aodel rmnm *ci done with L ft The i r a l general of such approaclies are the while, Krugj ak. (he results on the cricket cereal system rait methods, such ai reverse torrrbnon [P. Marrnarelu and V Mannarelis. Analyiu tj Phynal^ujl Syri™. JV tfrldfc .ApprD^Ji (Fknum. New York, l " - Ahhough (hoc method* nnrepeatedpresentation of identical STimuli! they nonetheless yield models thn ideally predjcT ihe bring rate in response. CO arbitrarily Chosen stimuli Reverie eorreUllon rhus suffers iht sime lnTuTat IOTU u oiher me-based approaches. r
b
L
r
7 Other author? have realized the importance ofaprflojehijift neural codingfromthe point of view of dw organism. In early wort. Tt. Fuzhugh [J. On. PtiyPet. 41.475 (19&B) ] discussesreal-rimedecuion miking At far a we knOw P.I M. Tohannesma |m G Sfcekely, E Ubca, S. Damj*novKh, Ed,., Ak> b
Pkp*t. fit 30, 103
19B1), Akadfrruai KuJ: .
Budapest] como doseil to our approach 1 f Bjckc.W.Yij™da.K MooragM.E R Lewis. W Bukt in^-iiwy PfcynJtjv' and Pcttptut*- A i m * \*p oftfw f 99 J Extopajn Hearing IVartcJwp. in preu
9. W Rulek and A Zee, J . Sat. /.v. ml (1990) 10 Lmcaj decoding trnghr wort for a enrol reason Many Knsoty neurons have a regime in whjth the fifing EiK varies linearly with stimulus amplitude TfUJ U an example of linear encoding ind linear decoding might work u well in ihis regime. However. Bialck and Zee |?) predict thai linear decoding is possible beyond the icgime of linear input-output tdidorn. The ceJ we siudied in this work has i linear input-output regime, but the stimuli lUC-J drove the cell outside dUs limit into sanitation. See, for example. H . Ecken [J. Cemp. Phynei. A 135, 29. (1980) ], whofindsnonmoncnoiuc nie-i-eloctry profile* in the range of sumuli used hrre 11. To cUuie (hat dV reconsmtcnon process could be •mplemenied in real ume. we required (hat the fiiltli be causal for cumpk, F|(T < 0| =• 0. Wecaleulired the minimum x* causalfillersin TWO ways:
inhere f(u> - J dir"" J(T). The averages (inside triangular brackflU} are over rhe ensemble of iDntuli used in rhe erpenment. ThufilterCan be ihjFEed by a delay and taysaEI ry can be imposed by setting the Shiftedfillerto zero al negative times (M j can be minimized with respect to pUltly eiuu] functions by expansion of thefilters(f„f in a complete set at hinctjonl thai vanish at negative timei. In thu mCthoet - delay lime muit be explicitly introduced dur measures the Lag between the true 1
ftunuJus and die teconstruetJOn, so I ' I ' H J )
I -T r — •
J ( J i
—
"
U nvnimiifd Theac ••<• >
methods leather enure rhat the OpflmiJ eausaj tJltrs are caieulafed L2 D.WtcfciMl, M A. IjneWfa, J P. Miller. W Bialek. i n . t u V «diVf<J(iusf a//**«f iysi™, F £^l S», I I I
(1974). 15 R. B de Ruvxer van Stevenuick. RtaS-Tt** Prrfc'r u i i H e/i jWowmen/rSmilnVe NbUflH AJouyly
569
were obtained in collaboration with M- A. LandoUa and J. P Miller, and rhe enpenmenii on the bullfrog auditory sysiem were done in collaboration with W YxmadiandE P. Lewis. Work at Berkeley wu lupponcd in pari by the NSE (hraugk a Presidential Young Investigiior Award to W.B. supplemenied by funds from Cray Hr veuxh. Sun Microsystems, and the NEC Reteirch 111 -' 11 •.. -: ind ihroiigh 4 CfldtUH FfUuwtrup m F R. DW. was supported in part by the Syuerru and Intrgraove BioLogy Training Program of the N T H IIUUJI work w» supported by the Nether lands Ogamzin'on for Purt $citnrifi< Research Rescaidi also performed al NEC Rcvtireh Irutjture by W B and E R L
2 N.
-r, 1990. accepred 10 May 1991
Reprinted with permission from Parallel Processing in Neural Systems and Computers, pp. 91-94, 1990 © 1990 Elsevier Science Publishers
SPIKE ARRIVAL TIMES : A HIGHLY EFFICIENT CODING FOR NEURAL
SCHEME
NETWORKS
Simon J . T H O R P E Instltut deB Neurosciences Departement des Neurosciences de La Vision Universlte HeiTe et Marie Curie Quai St. Bernard, Paris. France
1. Introduction - Temporal constraints o n neural coding. The
speed wilh which v i s u a l processing occurs i n the h u m a n b r a i n puts heavy
constraints on Ihe way In which Information Is processed. Indeed, we have recently argued that processing is so fast, that a great deal of visual processing m u s t be achieved using a single reed-forward pass through the v i s u a l system (Thorpe. 1988: Thorpe & Imbert. 19891. Briefly, the argument Is as follows. (1) Neurophysiological data have shown that neurons In the temporal cortex have highly selective responses {to faces for example), which start with latencies that are often In the range 100 to 140 m s (Perrett. Rolls & C a a n . 1982). (2) Anatomically, it would appear that there are at least 10 synaptic stages between the photoreceptors of ihe retina and these visually responsive neurons in the temporal lobe (I.e.. 2 In the retina. 1 in the lateral geniculate nucleus. 2 each In V I . V 2 . and V 3 . and one more In inferolemporal cortex) (see DeYoe & V a n E s s e n , 1988: Zekl & Shlpp. 1988). 13) Since the 10 synaptic layers must be crossed In 100 msec, this implies that each layer has only a n average of 10 msec processing time. (4) Given that the firing rates of cortical neurones are typically In the range 0 to lOO spikes per second, It follows that a neuron In any given layer will only generate 1 or perhaps 2 spikes before neurons in the next layer have to respond. Clearly, this puts major constraints on the way in which information is coded. For example, it is unreasonable to propose that firing rate per se c a n be used a s a n accurate code for analogue values. One possibility is that with two spikes, one c a n use the interval between them a s a code
the closer the spikes, the stronger the activation. However, the
evidence Indicates that a large amount of information c a n be transferred using only 1 spike per neuron. In this paper. I would like to suggest what I believe Is a novel way or coding Information In neural networks. It will be proposed that analog Information c a n be encoded by the relative arrival times of spikes. It will also be shown how s u c h a scheme allows very rapid resolution of the classic "Winner-takes-all" problem.
• Funded by E S P R I T Grant 3149 - Mulllaensory control of movement.
570
92
2. The spike arrival time hypothesis. it Is a well known neurophysiological fact l h a t the time taken for a neuron to reach its threshold for generating a spike depends on the strength of the stimulus - the stronger the stimulus, the faster the neuron depolarizes, a n d the sooner it generates a spike. Imagine, therefore, w h a l happens when an Image is flashed onto a simple retinal array of sensory cells. If the cells In the retina are sensitive to local Intensity, a wavefront of spikes will be generated I n the optic nerve, with the leading spikes corresponding lo the points on the retina where the Intensity Is highest. A s a result, even if each retinal cell generated one, and only one. action potential, the complete intensity distribution of the Image could be coded by the relative timing of spikes In the different fibres of the optic nerve. Obviously, the example given here is not biologically realistic
retinal ganglion cells do not simply
code the local intensity level. Rather, their activity Is more clearly related to local contrast. Nevertheless, the argument is j u s t the same
whatever the parameter coded by the input
array, the relative arrival time of spikes c a n be used to code analog values with considerable precision, a n d this, even if only one spike per fibre Is available. Of course. I n order for relative spike arrival times to be a useful code, it Is necessary for the next processing layer to be able to m a k e use of s u c h information. Is s u c h a pmcess physiologically plausible? T h e answer would appear lo be a definite yes. Poggio a n d Koch (19861 have recently proposed that synaptic veto mechanisms could exist in the retina with the property that if a spike from one input arrives before another, it could block the t r a n s m i s s i o n of the second
spike. Furthermore,
it h a s been known for some time that
neurons in the auditory system are sensitive to the relative arrival times of auditory stimuli In the two ears - s u c h information Is used In determining the spatial position of auditory s t i m u l i ( K n u d s e n et a l . 1987), Another example concerns
the processing of sensory
information I n electric fish. Heiligenberg a n d h i s collaborators have recently reported the some n e u r o n s are sensitive to electrical pulse arrival time differences in the microsecond range (Kawasaki. Rose & Heiligenberg. 1988). I n a l l the above examples, the differences in spike arrival time are due to timing differences i n the arrival times of actual p h y s i c a l stimuli, be they points of light, auditory stimuli to the left and right ears, or electrical discharges to the left and right sides of a fishes body. What Is being proposed here I s simply that the nervous system could also make use of differences in spike arrival times resulting not from temporal
differences in the
stimulus Itself but rather the delays introduced by the coding.
3. The problem of continuous Input. While It is relatively easy to see how a n Image flashed on the retina could lead to a Information-rich wavefront of spikes I n the optic nerve, it may be less obvious how the visual system could cope with continuously changing Information. However, one Important factor that may well reduce this problem are eye-movements, 11 h a s recently been reported that t r a n s m i s s i o n in the visual pathways in inhibited during eye movements, a n d that this
571
93
inhibition i s relaxed a s soon as the eye movement has term ma ted (Lai &
Frtediander.
1989). It could be that this could provide the synchronization necessary to allow relative spike arrival times to be used later i n the system. Another possibility tor providing the synchronization may
be found In the oscillatory
activity found In many parts of the nervous system. In particular, It Is possible that the 50 Hz
oscillations recently reported i n the v i s u a l cortex by
E c k h o m and
coworkers (1988) could be used
G r a y and
Singer (1989)
and
to this effect. They have reported that
neurons tend to fire during the negative going part of the local field potential oscillations. More strongly activated neurons would tend to fire earlier in the cycle, and this information could be used during processing by subsequent stages.
4. An application : The "Winner-TakeB-AU" problem. One
standard problem in connectlonist models Is the so-called "Winner-takes-all"
problem. Consider an array of units, with varying levels of activity. We most active one. One
want to select only the
commonly used method Is to introduce inhibition between a l l the
units. Since the most active unit Inhibits the others the most strongly, after a number of iterations the activity In all units except the most active will decrease until i n the end. only one unit r e m a i n s active. The problem Is that the number of Iterations c a n be quite large. Although it is possible to optimize such parameters as the amount of lateral Inhibition, it is c l e a r that at least 3
Iterations
are required
(Feldman
&
Ballard
1982;
personal
c ommu n ic at io n).
Figure L
A network using relative spike arrival time to solve the Winner-Takes-All problem.
The figure below Illustrates how.
If spike arrival times are used a s an analog code,
solving the Winner-lakes-all problem is almost trivial. The four input units (A-D) have direct connections to four output units [E-H). but they also have collateral connections to a n
572
94
Inhibitory lntemeuron (shown In black). Imagine that all four Input units generate spikes, but that the spike i n unit B is initiated a few milliseconds before the others. Unit F will respond to Its input a s usual, but because B's spike also activates the Inhibitory lntemeuron. the other three units will either fall to respond, or respond with longer latencies
. As a
result, the output pattern will correspond to a Winner-Take-All function of the input, but unlike conventional n e u r a l algorithms, the time taken c a n be as short a s a few milliseconds.
5. Perspectives. The
coding s c h e m e proposed
here would appear to provide a powerful way
Increasing the processing speed of neural networks, and may Into how
of
provide a n Important Insight
the v i s u a l s y s t e m c a n process highly complex visual Inputs with only 100-150
msec of processing time. It may
also be the case that s u c h a coding scheme could be used
in artificial neural network systems. A n analog retina In which the individual pixel elements generate Impulses with a delay that depended on local image Intensity would be relatively easy to construct. Subsequently, circuits of the type described in the preceding section could be used to "pick off" the most active Inputs a s well as
performing
other more
complicated operations.
6. Bibliography. DeYoe E.A. & Van Essen D.C. (1988) Concurrent
processing
streams In monkey
ulsual carter.
Trends in Neurosciences. 1 1 , 219. Feldman J.A. & Ballard D.H. (1982) Connection 1st models
and (heir properties.
Cognitive Science,
6, 205-254. Gray C.M. & Singer W. (1989) Stlmulus-speclilc oscillations In orientation columns of cat visual cortex. Proc. National Kawasaki M..
Academy
of Sciences.
Rose G. & Heiligenberg W.
86, 1698-1702.
(1988) Temporal
hyperacuity
In single
neurons
of
electric
fish. Nature, 336, 173. Knudsen E.I.. du Lac S. & Esterley S.D. (1987) Computational
maps in the brain. Annual Review of
Neuroscience. 1 0 . 41-66. Lai R. & Frledlander M.J. (1989) Coring of retinal signals.
transmission
by afferent
position and mouement
Science, 243, 93.
Perrett D.J., Rolls E.T. & Caan W.C. 11982) Visual neurons responstue (o faces temporal
corlex.
In Ihe monkey
Experimental Brain Research. 47, 329-342.
Poggio T. & Koch C. (1986) Synapses that compute
motion.
Scientific American. 256. 46-71.
Thorpe S.J. (1988) Traltement d'lmages chez i'homme. Techniques et Sciences informatlques. 7, 517-525. Thorpe S.J. & Imbert M. (1989) Biological constraint on connection^! models. In PfellTer R et al (ed.) "Connectionism In Perspective'. Amsterdam : North-Hoi land, p 63-92. Zelti S. & Shlpp S. (1988) The Junctional role of cortical
573
connections.
Nalure. 33B. 311-316.
Chapter 4
BRAIN AREAS, C I R C U I T S AND D Y N A M I C S
In contrast to Chapter 3 which is mainly theoretical, this chapter is a return to facts, with an attempt at pointing towards domains where the cooperation between experimental investigations and theoretical interpretations have known some measure of success. This is the chapter for which the editors had to rely most on friendly advices from competent colleagues. The proportion of recent letters is high. This entails a risk of fast obsolescence, of which we have been continuously aware. But there seemed to be no other way to give a reasonably wide account of the present state-of-the-art. Actually there is so much biological diversity here that, purposively, our introductory remarks will be deferred to the introductions of the various sections. Section 4a is mainly concerned with visual pathways in primates; but the leading article (42) addresses other senses and other animals. T h e last paper (46) is an illustration of the new mix of electrode recording data and neural network (here, multi-layer perceptron) calculations. Section 4b contains two papers that attempt to bridge the gap between psychophysics and neurophysiology, a promising line of investigations. Section 4c covers brain areas and functions other than the visual cortex: cerebellum, hippocampus, frontal lobe and basal ganglia. Section 4d deals with the representations of space in the brain, and more precisely with eye saccades. This is a privileged domain for interplay between mathematical physics and neurobiology. One paper alone makes section 4e, the last section; it defines the binding problem (how scattered features are bound together in object perception)
574
and proposes one possible biological solution through synchrony effects and oscillations.
575
4a
Sensory and Motor Pathways
The first article (42) of this section, by Konishi, discusses two sensory systems: the auditory system of the barn owl, and the electrosensory system of the electric fish. Much knowledge about computational goals (coming from behavioural observations), processing algorithms and neural data, are brought to bear in these neuroethological syntheses. A side remark about methodology is in place here: much has been said about the structural and functional research strategies in neuroscience; the study of these two systems has been an example of the latter in the case of the barn owl, and of the former in the case of the electric fish. In both cases, a computational task is separated into two components (like vertical and horizontal position detection), which are processed separately, along different pathways. The author's analysis concludes in favour of a hierarchical feedforward scheme ending with a few "grandmother cells" at the output level, and thereby against an alternative hypothesis of distributed "cell assemblies". This might seem to favour Barlow (section 2b) against Hebb (section 2a), but this interpretation would be partly misleading, because the two species studied do not have a mammalian cortex, with its conspicuous divergence (in neuron numbers) from early sensory areas to association cortices. In these nonmammalian species, there is extensive convergence of lower order neurons onto output neurons selective for behaviourally relevant stimuli. In complement to article (42), the other four articles of this section are concerned with the primate visual system. Article (43) by Livingstone and Hubel is a remarkable review, which provides many suggestive interpretations of anatomical, physiological and perceptual facts, to support the notion of separate visual pathways (the magno- and parvo-systems). This paper connects with almost all the other topics raised in this book. It will be no surprise to learn from follow-up papers that some of the theses presented in article (43), about a rather rigorous segregation of the two pathways, have led to subsequent controversy (echoed in articles (44) and (45)).
576
Article (44), by Goodale and Milner, pursues the description of the primate visual system higher-up, with an assessment of the separate functions of the parietal and temporal lobes. The distinction between visual perception (object identification, "object-centred") and visual control of action (action-relevant information, "viewer-centred") — or for short, between apprehension and prehension — is discussed with great care for biological data, and supported by neuropsychological studies on patients with brain damage and perceptual dysfunctions. The article ends with interesting speculations about the neuroanatomy of consciousness. Article (45), by Wise and Desimone, further extends the discussion toward phenomena of attention (filtering of unwanted information, enhancement of responsiveness), of spatial coordinate systems ("retina-centred", "body-centred", "environment-centred"), and of the learning of visuomotor coordination. T h e reader is advised to note the comprehensive exposition of the population coding principle (already presented in article ( 1 7 ) ) , as an effective computational scheme to achieve the smoothness and spatial resolution, needed for motor control, from the pooling of broadly tuned neurons. T h e last article (46) of this section, by Zipser and Andersen, is quite different in style because it attempts a general modelling of neurocomputation involved in coordinate transformations (here from "retina-cent red" to "head-centred"). T h i s paper has become an often quoted reference (it will be critically discussed, though, in article (65)), because it demonstrates how a simple — even if not so plausible biologically — learning algorithm, that learns from input-output examples, can lead a network toward a suitable processing algorithm (unspecified in advance), with testable consequences that are in qualitative agreement with experiment.
577
Reprinted with permission from Neural Computation. Vol. 3, pp. 1-18,1991 © 1991 Massachusetts Institute of Technology
D e c i p h e r i n g the Brain's C o d e s Masakazu K o n i s h i Division of Biology, California Institute of Technology, Pasadena, California 91125 USA The two sensory systems discussed i n this review use similar algorithms for the synthesis of the neuronal selectivity for the stimulus that releases a particular behavior, although the neural circuits, the brain sites involved, and even the species are different. T h i s stimulus selectivity emerges gradually i n a neural network organized according to parallel and hierarchical design principles. T h e parallel channels contain lower order stations with special circuits for the creation of neuronal selectivities for different features of the stimulus. Convergence of the parallel pathways brings these selectivities together at a higher order station for the eventual synthesis of the selectivity for the whole stimulus pattern. The neurons that are selective for the stimulus are at the top of the hierarchy, and they form the interface between the sensory and motor systems or between sensory systems of different modalities. T h e similarities of these two systems at the level of algorithms suggest the existence of rules of signal processing that transcend different sensory systems and species of animals.
1 Introduction Both peripheral and central sensory neurons do not respond to all stimuli or stimulus variables but to a certain modality, range, configuration, or pattern of stimuli or stimulus variables. This property of sensory neurons will be called stimulus selectivity or, simply, selectivity. Some neurons are selective for primary stimulus variables, such as frequency and wavelength, and others are selective for more complex patterns of stimulus variables, such as direction, velocity, and disparities. In some systems, lower order neurons show selectivity for simpler stimuli and higher order neurons for more complex stimuli. Such a correlation between stimulus selectivities and the anatomical levels of a sensory system suggests the possibility of finding where and how selectivities for complex stimuli are derived from the integration of selectivities for simpler stimulus variables. Of particular interest are sensory systems that contain higher order neurons selective for the same stimuli that cause specific behaviors or percepts. If these systems are amenable to analysis of successive stages of processing leading to the selectivity of these neurons, we may
578
2
understand h o w the whole system is designed to analyze behaviorally relevant stimuli. For various reasons, many complex sensory systems are not amenable to this form of analysis. However, in the auditory system of the b a m owl and the electrosensory system of the electric fish, Eigenmannia, such an analysis has been successfully carried out {HeiUgenberg 1986; Konishi et al. 1988). In this review, I shall discuss what we can learn from these examples about neural codes, neural algorithms, and network organization.
2 Behavioral A n a l y s i s The sense organs and the brain of an animal must be designed for the processing of the stimuli that are relevant for its survival and reproduction. It is, therefore, important to determine what stimulus to use in the analysis of neuronal selectivities. This section describes first the characteristics of the stimulus for sound locahzation by the bam owl and then the stimulus for the electric fish. O n hearing a sound, the owl turns its head i n the direction of the sound source (Knudsen et al. 1979). Experiments show that the owl uses interaural time differences for localization in azimuth and interaural amplitude differences for localization in elevation (Moiseff and Konishi 1981; Moiseff 1989). Of two possible sources of interaural time differences, namely, stimulus onset time and phase disparities, the owl extracts and uses the interaural phase differences of all audible frequencies contained in the signal. The vertical asymmetries of the b a m owl's ears enable the ow! to use interaural amplitude differences for localization i n elevation. A higher sound level in the right ear and left ear causes the owl to r u m its head upward and downward, respectively. The owl obtains both binaural disparities simultaneously from a single sound signal. Each location in the owl's two-dimensional auditory space is thus uniquely defined by a combination of interaural time and amplitude differences. The second example is the electric fish, Eigenmannia. This species creates electrical field potentials around its body to detect objects having conductivities different from the conductivity of the water. The electrical potential varies almost sinusoidally over time, and the fish can change its frequency. W h e n the electrical fields of two fish have only slightly dissimilar frequencies, the fish try to increase the differences in signal frequency. This behavior is called the jamming avoidance response {Heiligenberg 1986). The main problem that a fish must solve in encountering another fish is to determine whether its o w n frequency is higher or lower than that of the other fish. The pacemaker cells that drive the electrical organ cycle by cycle could, in theory, provide a copy of the efferent command for comparison with the frequency of the incoming signal. Eigenmannia does not, however, use this metbod. Instead, the fish determines the sign of frequency differences from the waveform created by the mixing
579
3
of its own and the neighbor's signals. The phase and amplitude of the waveform at one locus on the body surface are different from those at another locus, because the sources of the two electrical fields are located within the bodies of the two fish that are separated in space. The fish uses these phase and amplitude differences between many loci on its body surface to determine the sign of frequency differences (see Fig. 1 for further explanation).
3 Successive Stages of Signal Processing
_
—
Complex sensory pathways have input, intermediate, and output stages. In the "bottom-up" approach, one starts with the sense organ and proceeds to higher order stations in the ascending sequence. The study of neuronal selectivities need not start with the input stage. In the "topdown" approach, one starts with output or other higher order neurons and goes downward through intermediate stages to the sense organ. Neither approach is easy when the neural network is complex. The bottomup approach is difficult because of nonlinear properties of most neural systems. The top-down approach is difficult because the output neurons of the network may not be easily found. The output neurons of a hierarchically organized neural network occupy the anatomically highest stage of the hierarchy and project to other functionally identifiable networks such as the motor system and other sensory systems. Under favorable circumstances, the point of transition from one network to the next can be recognized by combinations of anatomical and physiological methods. Starting with higher order neurons has distinct advantages because the investigator has the defined goal of explaining the stimulus selectivity of the higher order neurons. In the owl, we were lucky to start with what has turned out to be the output neurons. The following description of the owl's auditory system similarly starts with the output neurons and explains how their stimulus selectivity is established.
3.1 T h e T o p - D o w n Approach i n the O w l . We looked for higher order auditory neurons that responded only when sound came from a restricted area in space. A cluster of such cells, which are called spacespecific neurons, occurs in the external nucleus of the inferior colliculus (Knudsen and Konishi 1978; Moiseff and Konishi 1983). This nucleus is the highest station in that part of the owl's auditory system that processes the stimulus for sound localization (cf. Fig. 3). Experiments with earphones show that these neurons are selective for a combination of interaural time and amplitude differences (Moiseff and Konishi 1981; Olsen et al. 1989). The neurons are selective for a particular spatial location, because they are tuned to the combination of interaural time and amplitude differences that results when the sound source is located at that site. We also know that a neuron's selectivity for interaural time and amplitude
580
4
Figure 1: Deterrnination of the sign of frequency differences b y the electric fish, Eigenmannia. ( A ) Electrical signals.
Eigenmannia generates nearly sinusoidal
electrical signals for navigation a n d orientation. W h e n a n i n d i v i d u a l { S i ) encounters another i n d i v i d u a l (S;), they avoid j a m m i n g each other by changing the frequency of their signals. T h e fish uses the beat waveform (Si +S2) to determ i n e whether its o w n frequency is higher or lower. (B) T h e fish uses differences in the phase a n d a m p l i t u d e of the beat w a v e f o r m between different body loci to determine the sign of frequency differences. I n this figure, the solid-line a n d dotted-line w a v e f o r m s s h o w different degrees of contamination of Si b y S2; the solid-line w a v e f o r m registered at one b o d y locus is more contaminated a n d the dotted-line w a v e f o r m registered at another locus is less contaminated. T h e small a r r o w h e a d s indicate the phase relationships between the t w o waveforms. T h e left-slanted a r r o w h e a d s indicate that the phase of the solid-line waveform is a d v a n c e d relative to that of the dotted-line w a v e f o r m . W h e n these phase relationships a n d the rise a n d fall of amplitude are considered jointly, the sign of frequency differences c a n be determined unambiguously. T h u s , the sequence, a fall i n a m p l i t u d e with a phase advance followed by a rise in amplitude w i t h a phase delay, indicates that the fish's o w n frequency is lower than that of the other fish.
581
5
differences determines, respectively, the azimuthal and elevational centers and widths of its receptive field. It is reasonable to assume that a neuron's selectivity for a complex stimulus is due both to its intrinsic morphological and biophysical properties and to the integration of information conveyed by the input channels converging on it. Thus, the next step in the top-down approach is to detennine what circuits and processes underbe the stimulus selectivity of space-specific neurons. A survey of all binaural stations below the level of the external nucleus of the inferior coUiculus showed that they could be classified into two groups, one containing neurons sensitive to interaural amplitude differences and the other containing neurons sensitive to interaural phase differences (Moiseff and Konishi 1983; Sullivan and Konishi 1984; Takahashi et al. 1984). Subsequent anatomical studies established separate pathways from the cochlear nuclei, the first auditory stations of the brain, to the inferior colliculus in the midbrain (Takahashi and Konishi 1988a,b). These findings led to the hypothesis that the owl's auditory system uses parallel pathways for separate processing of interaural phase and amplitude differences. A more direct test of this assumption came from an experiment in which the response of space-specific neurons to interaural time and amplitude differences was observed while one of the two pathways was partially inactivated by injection of a local anesthetic. The beginning stage of the "time pathway" is one of the cochlear nuclei, nucleus magnocellurais, and that of the "amplitude pathway" is the other cochlear nucleus, nucleus angularis. Partial inactivation of the nucleus magnocellularis drastically changed the response of space-specific neurons to interaural time differences without affecting their response to interaural amplitude differences. The converse was observed when the nucleus angularis was partially anesthetized. These cochlear nuclei are both anatomically and physiologically different from each other. Neurons of the nucleus magnocellularis are sensitive to stimulus phase but insensitive to variation in stimulus amplitude. By contrast, neurons of the nucleus angularis are sensitive to variation in stimulus amplitude but insensitive to stimulus phase. The phase sensitivity means that the neuron fires at or near a particular phase angle during the tonal period. This phenomenon, phase-locking, occurs at frequencies as high as 8.3 k H z in the owl. Neurons do not fire during every tonal period of such a high frequency, but whenever they fire, they phase-lock to the stimulus. W h e n the stimulus is noise, neurons phase-lock to the phase of the spectral components to which they are tuned. The next step i n this research was to determine where and how the neuronal selectivities for interaural phase and ampUtude differences are established in the two pathways. The third-order nucleus in the time pathway, nucleus laminaris, is the first station that contains neurons selective for interaural phase differences. The owl uses phase-locked spikes from the left and right ears to measure interaural phase differences from
582
6
Figure 2: Neural circuits for the detection of interaural time differences. The inset shows a model of neural circuits for the detection of interaural time differences. It uses the principles of coincidence detection and delay lines. Binaural neurons, A, B, C , D, and E , fire maximally when impulses from the two sides arrive simultaneously. Except for C, the paths for impulse transmission to each neuron are different between the two sides. These asymmetries cause interaural differences in the arrival time of impulses. A neuron fires maximally when an imposed interaural time difference compensates for the asymmetry in impulse transmission time. This array of neurons thus encodes different azimuthal locations of sound systematically. The main figure shows the neural circuits. Nucleus magnocellularis is one of the first brain stations in the owl's auditory system. Nucleus laminaris receives inputs from both the ipsilateral and contralateral magnocellular nuclei. The figure shows axon collaterals from single ipsilateral and contralateral neurons projecting into nucleus laminaris, which contain binaural neurons. For the sake of clarity, the ipsilateral and contralteral axons are shown separately, although they interdigitate in reality. These interdigitating axons serve as delay lines, and the laminaris neurons as coincidence detectors. Interaural phase differences are computed separately for each frequency band.
which it eventually derives interaural time differences. The circuits that compute interaural phase differences use the principles of delay lines and coincidence detection (Fig. 2), corresponding to a model first proposed by Jeffress (1948). Laminaris neurons are innervated by axons from both
583
7
ipsilateral and contralateral magnocellular neurons. The parts of these axons that lie within the boundaries of the nucleus laminaris act as delay lines, and laminaris neurons themselves as coincidence detectors (Carr and Konishi 1988, 1990). A binaural disparity in the arrival time (A( > of impulses at a laminaris neuron includes tbe difference in the acoustic transmission time to the two ears (ITD) and the difference in the impulse conduction time ( A t ) from the two ears to the neuron, hence A ( = ITD+At,.. Both I T D and A t vary, but the delay lines are organized such that for each neuron A f equals a particular I T D in magnitude but opposite in sign (i.e., I T D = - A e ) ; impulses from the two sides arrive simultaneously and the laminaris neuron fires maximally. Laminaris neurons are, however, not perfect coincidence detectors, because they pass monaural signals. Interestingly, an unfavorable phase difference elicits a smaller number of impulses than that triggered by either of the monaural signals. Nonlinear processes such as inhibition may thus contribute to the computation of interaural phase differences. a
c
a
c
c
c
All of these processes occur in each audible frequency band. L a m i naris neurons thus convey their selectivity for interaural pbase differences to higher order nuclei in separate frequency channels. A single laminaris neuron responds to multiple ITDs that are separated by integer multiples of the stimulus period. This phenomenon occurs because phase is a circular variable. Thus, if an interaural time difference I T D corresponds to an interaural phase difference, IPD, then all I T D + nT also correspond to I P D , where n and T are integer and the period of the stimulus tone, respectively. Laminaris neurons respond to all I T D + nT as long as they are within their physiological range. Laminaris neurons send their axons, in separate frequency channels, to two higher order nuclei, the central nucleus of the inferior colliculus and one of the lemniscal nuclei. The inputs from the nucleus laminaris endow the neurons of these nuclei with selectivity for interaural phase differences. Consequently, these neurons also respond to multiple ITDs. The neurons of the central nucleus of the inferior colliculus are, however, more sharply tuned to interaural phase differences. These neurons project to the lateral shell of the central nucleus of the inferior colbculus, and the neurons of this nucleus project, in turn, to the external nucleus of the inferior colliculus where space-specific neurons reside. Unlike the lower order neurons, space-specific neurons are broadly tuned to frequency and respond only to one I T D when a broad-band signal is used. This fact indicates that space-specific neurons receive inputs from the frequency channels that are selective for the same I T D and its phase equivalents (ITD + nT) (Wagner et al. 1988). Space-specific neurons or their immediate precursors in the lateral shell get rid of the frequency dependent variable I T D + nT. This ability of space-specific neurons to respond exclusively to one interaural time difference is due to excitatory and inhibitory interactions between the different frequency channels that they receive (Takahashi and Konishi 1986; Fujita 1989).
584
8
The last issue in the synthesis of the stimulus selectivity of spacespecific neurons concerns their ability to respond selectively to combinations of interaural time and amplitude differences. This capacity derives from the convergence of the two pathways in the lateral shell of the central nucleus of the inferior colliculus. Interaural amplitude differences are first encoded, however, in one of the lemniscal nuclei (Manley et al. 1988). Stimulation of the contralateral ear excites and that of the ipsilateral ear inhibits the neurons of this nucleus. The response of these neurons is, therefore, determined by interaural amplitude differences. These neurons are, however, not exclusively selective for interaural amplitude differences, because the contralateral ear alone can drive them. The outputs of these neurons are eventually used to produce the ability of space-specific neurons to tune to interaural amplitude differences. The convergence of the two pathways is not a simple addition but involves another nonlinear operation, which endows space-specific neurons with the ability to respond only to a combination of interaural time and amplitude differences. 3.2 T h e Bottom-Up Approach i n the Electric Fish. I n the electric fish, Heihgenberg and his associates have used the bottom-up approach to discover the neural mechanisms for the determination of the sign of frequency differences in the jamming avoidance response. It should be noted, however, that they used the results of behavioral analysis to guide their search for relevant neuronal stimulus selectivities. I shall briefly review the steps by which they discovered the output neurons. The selectivities for primary stimulus variables, phase and amplitude, are established in the sense organs themselves. The electric fish bas two kinds of electroceprive sensory cells in the skin (Scheich et al. 1973). One of them fires a single impulse at each positive zero-crossing of the nearly sinusoidal electrical signal. The fish uses these impulses to convey to the brain information about the phase angles of the signal over the entire body surface. The other type of sensory cell is sensitive to variation in the amplitude of the electrical signal. The phase and the amplitude-sensitive cells are mixed over the body surface, but they project to different layers of the first brain station. These layers constitute the starting points of separate pathways for phase and amplitude (Carr et al. 1982; Heiligenberg and Dye 1982). Each layer contains three separate maps of the electrical field variables over the body surface, and the phase and amplitude maps are in register (Shumway 1989a,b). This nucleus, the electrosensory lateral line lobe, contains neurons and their circuits that are sensitive to the rise and fall of signal amplitude (Saunders and Bastian 1984; S h u m w a y and Maler 1989). The electrosensory lateral line lobe projects to the multilayered torus semicircularis, presumably the homolog of the inferior colliculus. Like the owl, the electric fish uses special neural circuits for the computation of phase differences between different body loci. These circuits are found
585
9
i n l a m i n a 6 of the t o r u s ( C a r r et al. 1986). O f t h e t w o c l a s s e s of o u t p u t n e u r o n s of these c i r c u i t s , o n e r e s p o n d s to p h a s e a d v a n c e a n d the o t h e r to p h a s e delay. N e u r o n s s e l e c t i v e for the r i s e or fall of a m p l i t u d e o c c u r i n o t h e r l a m i n a e of t h e t o r u s . A s i n the o w l , t h e p h a s e a n d a m p l i t u d e p a t h w a y s c o n v e r g e o n e a c h other i n s p e c i f i c l a y e r s of the t o r u s . T h i s c o n v e r g e n c e g i v e s r i s e to f o u r c l a s s e s of n e u r o n s that are s e l e c t i v e for f o u r different c o m b i n a t i o n s of a m p l i t u d e a n d p h a s e , that is, a m p l i t u d e f a l l - p h a s e a d v a n c e , a m p l i t u d e r i s e - p h a s e delay, a m p b t u d e f a l l - p h a s e delay, a n d a m p U t u d e r i s e - p h a s e a d v a n c e ( H e i U g e n b e r g a n d R o s e 1986; R o s e a n d H e i l i g e n b e r g 1986). T h e first t w o a m p U t u d e - p h a s e c o m b i n a t i o n s i n d i c a t e that t h e fish's o w n freq u e n c y is l o w e r , a n d the s e c o n d t w o that the fish's o w n f r e q u e n c y is h i g h e r . T h e s e f o u r n e u r o n t y p e s a r e , h o w e v e r , not e x c l u s i v e l y
selective
for p a r t i c u l a r a m p b t u d e - p h a s e c o m b i n a t i o n s , b e c a u s e they s h o w r e s p o n s e s to o t h e r c o m b i n a t i o n s .
some
A l s o , the r e s p o n s e of t h e s e n e u r o n s
d e p e n d s o n the r e l a t i v e o r i e n t a t i o n of the fish's o w n e l e c t r i c a l field a n d that of its neighbor, b e c a u s e their r e c e p t i v e fields a r e r e s t r i c t e d to s m a l l b o d y s u r f a c e a r e a s ( H e i l i g e n b e r g 1986). T h e next stage of p r o c e s s i n g t a k e s p l a c e i n t h e n u c l e u s
electrosenso-
r i u s w h e r e s e n s o r y c h a n n e l s f r o m different b o d y s u r f a c e loci c o n v e r g e o n s i n g l e n e u r o n s ( K e l l e r a n d H e i l i g e n b e r g 1989). T h e r e s p o n s e of t h e s e n e u r o n s to a m p l i t u d e - p h a s e c o m b i n a t i o n s b e c o m e s l a r g e l y i n d e p e n d e n t of the r e l a t i v e o r i e n t a t i o n of o v e r l a p p i n g e l e c t r i c a l fields. T h e f i n a l s t a g e of p r o c e s s i n g is the p r e p a c e m a k e r n u c l e u s i n the d i e n c e p h a l o n ( R o s e et al. 1988). Its n e u r o n s u n a m b i g u o u s l y d i s c r i m i n a t e b e t w e e n the s i g n s of f r e q u e n c y differences.
4 The Output Neurons
T h e o w l ' s s p a c e - s p e c i f i c n e u r o n s are the o u t p u t n e u r o n s of the n e t w o r k i n v o l v e d i n s o u n d l o c a l i z a t i o n , b e c a u s e they o c c u p y the t o p of the h i e r a r c h y of t h e b r a i n s t e m a n d p o n t i n e a u d i t o r y n u c l e i a n d project to the o p t i c t e c t u m . T h e s e n e u r o n s a r e s e l e c t i v e for the s a m e s t i m u l u s t h a t i n d u c e s the s o u n d l o c a l i z i n g r e s p o n s e i n the o w l . T h i s s t i m u l u s s e l e c t i v i t y is a r e s u l t of a l l p a r a l l e l a n d s e r i a l c o m p u t a t i o n s that a r e c a r r i e d
out
b y l o w e r o r d e r n e u r o n s i n t h e p a t h w a y s l e a d i n g to the o u t p u t n e u r o n s . S p a c e - s p e c i f i c n e u r o n s f o r m a m a p of a u d i t o r y s p a c e i n the e x t e r n a l n u c l e u s of the i n f e r i o r c o l l i c u l u s ( K n u d s e n a n d K o n i s h i 1978).
This
map
projects to the o p t i c t e c t u m w h e r e a n a u d i t o r y - v i s u a l m a p of s p a c e is f o u n d ( K n u d s e n a n d K n u d s e n 1983).
T h i s b i m o d a l m a p a p p e a r s to be
l i n k e d to the m o t o r m a p of h e a d o r i e n t a t i o n .
E l e c t r i c a l s t i m u l a t i o n of
the o p t i c t e c t u m elicits s a c c a d i c h e a d m o v e m e n t s , w h i c h a r e s i m i l a r to t h o s e r e l e a s e d b y n a t u r a l s o u n d s t i m u l i . T h e s p a t i a l l o c u s to w h i c h the o w l o r i e n t s c o r r e s p o n d s to t h e r e c e p t i v e fields of a u d i t o r y - v i s u a l n e u r o n s l o c a t e d at the site of e l e c t r i c a l s t i m u l a t i o n ( D u L a c a n d K n u d s e n 1990;
586
10
Masino and K n u d s e n 1990). The exact mechanisms of translation from sensory codes to motor codes are, however, not yet known. In the electric fish, neurons of the prepacemaker nucleus in the diencephalon are the output neurons of the system for the determination of the sign of frequency differences, because they occupy the top of the hierarchy of nuclei involved in this behavior and project directly to the premotor nucleus that controls the electrical organ (Rose et al. 1988). The stimulus selectivity of prepacemaker neurons is a result of all parallel and serial computations that take place in lower order nuclei in the pathways leading to them. These neurons fire more when the fish's o w n frequency is higher and less when the fish's o w n frequency is lower. This response pattern is exactly what the fish shows in response to the sign of frequency differences; when its o w n frequency is higher, the fish raises it, and when its o w n frequency is lower, the fish lowers it so that the frequency difference between the two fish becomes larger. Moreover, just as the fish's response is rather independent of the relative orientation of the other fish, so is the response of prepacemaker neurons. A rise and fall of the discharge rate in these neurons, respectively, raises and lowers the frequency of firing in pacemaker neurons that trigger each discharge cycle of the electrical organ.
S S t i m u l u s Selectivities and Neural Codes Neural codes are pieces of information that neurons convey to other neurons. This section discusses first neural codes in the above sense and then the relationships between these codes and behavior. In the o w l and the electric fish, we see how the timing and rate of impulses in the input stage are directly correlated with the phase and amplitude of the stimulus, respectively. Furthermore, we know how the selectivities for phase and amplitude disparities are derived from these inputs. Thus, phase-locked and rate-variable impulses are the neural codes for phase and amplitude, respectively. However, neither impulse timing nor rate is uniquely correlated with amplitude and phase disparities. Nevertheless, convergence of neurons selective for phase and amplitude disparities gives rise to neurons selective for combinations of the two disparities, indicating that the disparity-sensitive neurons convey relevant information to other neurons. The only code for this information is the neuron's place or address. "Place-coding" is the most universal signaling method in all neural systems. It is, therefore, justifiable to equate stimulus selectivities with neural codes, and the study of stimulus selectivities in successive stages of a sensory network can show how complex stimuli are encoded. The two examples show that the neurons at the top of a hierarchically organized system represent the final result of all computations that are carried out by lower order neurons. Representation of a large network by a small number of output neurons is an interesing problem
587
12
neurons are somewhat similar to the command fibers of invertebrates. Recent reports show that a group of command fibers contributes to the control of several different motor output patterns, but the amount and nature of the contributions by different fibers vary in different patterns (Larimer 1988). Some of the oscillator circuits of invertebrates also show elements of combinatorial control of multiple output patterns by a small group of neurons (Getting 1989). Thus, in some systems, a small number of neurons represents the outputs of a network and controls relatively complex behaviors.
6 Similarities in Algorithms
In the present context, an algorithm refers to steps and procedures in signal processing. Figure 3 compares the algorithms for the processing of the signals for sound localization by the owl and for the jamming avoidance response by the electric fish. Both systems use parallel pathw a y s for the processing of different stimulus features. Signal processing within each of the pathways occurs in a hierarchical sequence of nuclei. First, the codes for the primary stimulus variables, phase and amplitude, are sorted out at an early stage and routed to appropriate pathways, then different stimulus features are detected and encoded in each pathway by special neural circuits. Further processing in higher order stations makes the neural representations of the stimulus features more accurate and less ambiguous. Finally, the codes for these features are brought together by convergence of the parallel pathways. The result of convergence is not simply the addition of the codes from the input channels but the creation of a new code. In both examples, the inputs to the output neurons carry the codes for phase and amplitude disparities, but the output neurons do not respond to either feature alone but only to a combination of the two features. The output neurons of the en fire network can be recognized in both the owl and the electric fish. These neurons occur at the top of the hierarchy of processing stages, and they encode the signals for sound localization and jamming avoidance unambiguously. The output neurons serve as the interface between the signal processing and motor system or between signal processing networks of different modalities such as the auditory and visual systems in the owl. There are thus remarkable similarities between the owl and fish algorithms (Konishi 1991). Both the auditory and electrosensory systems are thought to have evolved from the lateral line system, which the fish uses to detect disturbances in the surrounding water. This explanation does not account for specific aspects of the algorithms, such as the separation of the phase and ampUtude codes in two different nuclei in the owl and two different layers of a nucleus in the electric fish. The fact that both
589
g 3
!i H
0 . HI
1 |
H 1|
1
1]
P
LL' I
S
a
i
J2
3
Ia i
a
si* 3
- • • —
•s.
t£ -a -o 590
-
6-
11
from the point of view of network organization and coding. When a study of intermediate stages of a sensory system uncovers only neurons selective for simple stimulus features, this observation does not necessarily allow the investigator to conclude that the output of the system is encoded by a large ensemble of simple neurons. For example, Heiligenberg and associates interpreted the results of their early studies in terms of activities of a large ensemble of neurons. They found that the fish would fail to determine reliably the sign of frequency differences, if sensory inputs from a large part of its body surface were eliminated. Their models also indicated that comparisons of phase and amplitude between many pairs of body loci are necessary for the determination of the sign of frequency differences. In addition, their early studies of lower order nuclei uncovered only those neurons selective for separate lower order features of the stimulus for jamming avoidance. These observations prompted them to write papers with titles such as "The jamming avoidance response revisited: The structure of a neuronal democracy" (Heiligenberg et al. 1978) and "The neural basis of a sensory filter in the jamming avoidance system: No grandmother cells in sight" (Partridge et al. 1980). However, when they studied the diencephalon, which contains the nucleus electrosensorius and the prepacemaker nucleus, they found extensive convergence of inputs from the body surface onto single neurons. These neurons, particularly prepacemaker neurons, unambiguously discriminated the sign of frequency differences. This finding resulted in a paper entitled " 'Recognition units' at the top of a neuronal hierarchy? Prepacemaker neurons code the sign of frequency differences unambiguously" (Rose et al. 1988). Thus, one cannot predict either from behavioral analyses alone or from the study of intermediate stages of a network how its output stage encodes relevant stimuli. In both the electric fish and the owl, we see extensive convergence of lower order neurons onto the output neurons that are selective for the behaviorally relevant stimulus. The ratio of the output neurons to lower order neurons has not been determined in either system, but the volume of tissue containing the output neurons appears to be much smaller than that of lower order nuclei in both systems. The prepacemaker nucleus, being about 100 /im in diameter, is the smallest nucleus in the electrosensory system (Keller et al 1990). Moreover, this nucleus consists of two parts, one for the control of the jamming avoidance response and the other for "chirps," which occur in courtship and aggression. The number of neurons in the "chirp" area is estimated to be about 200 on each side of the brain and the jamming avoidance area contains perhaps twice as many neurons (Heiligenberg, personal communication). These numbers are small in a system in which most lower order nuclei contain thousands of neurons. Intracellular stimulation of a single "chirp" neuron can induce weak "chirps" and stimulation of many neurons with glutamate induces strong "chirps" (Kawasaki and Heiligenberg 1988; Kawasaki et at 1988). These
588
14
animals deal with sinusoidal signals may be the reason for the similar algorithms, because the primary stimulus variables are the same in both sound and electrical signals. The goals of the two systems are also similar, because both systems ultimately encode combinations of phase and amplitude disparities. The neural implementations of the algorithms are, however, different between the systems. The electric fish uses different sensory cells to encode phase and amplitude, whereas the owl uses the same auditory neurons to encode both phase and amplitude. The electric fish uses electrical synapses to transmit phase-locked spikes in all relay stations below the stage where phase differences are computed. The owl uses chemical synapses for the same purpose, although they are specialized synapses. The electric fish uses the differences in the arrival time of phase-locked spikes between somata and dendrites to detect phase disparities between different body loci, whereas the owl uses axonal delay lines. Both animals use the convergence of different input channels to eliminate ambiguity in neuronal stimulus selectivity, but the convergence occurs in different parts of the brain, the midbrain in the owl and the diencephalon in the fish. Do similar algorithms occur in other complex sensory systems? The answer to this question is difficult to obtain, because few studies of complex sensory systems have investigated successive stages of signal processing. The visual system of the macaque monkey is the only other system that has been studied well enough for the discussion of algorithms. This complex system is also organized according to parallel and hierarchical design principles (Van Essen 1985; Maunsell and Newsome 1987; Hubel and Livingstone 1987; Livingstone and Hubel 1987, 1988; De Yoe and Van Essen 1988); parvocellular and magnocellular pathways are physiologically and anatomically distinct and the way stations in each pathway within the extrastriate cortex are hierarchically organized. These network hierarchies appear to be correlated with the processing hierarchies. L o w e r order features such as stimulus orientation are encoded in the striate cortex, whereas relatively higher order features, such as velocity and geometric patterns like faces, are encoded in higher stations, the middle temporal area encoding velocity and the inferotemporal area encoding faces (Gross et at. 1972; Perret et at, 1982; Maunsell and Van Essen 1983). However, much remains to be explored before we can understand the mechanisms and functional significance of feature extraction in this system as well as we do in the electric fish and the owl.
7 Concluding Remarks
..
—
.
Neuroethology, which studies the neural bases of natural behavior, has something to offer to the students of computational and neural systems. The tenet of neuroethology states that the brain is designed to process biologically relevant stimuli and control behavior essential for the
591
15
s u r v i v a l a n d r e p r o d u c t i o n of the a n i m a l . O n l y b e h a v i o r a l o b s e r v a t i o n s and
analyses can identify biologically significant stimuli. T h e t w o sys-
t e m s d i s c u s s e d a b o v e c o u l d not h a v e b e e n a n a l y z e d a d e q u a t e l y a n d u n d e r s t o o d w i t h o u t u s e of a n d reference to s u c h s t i m u l i . T h e s e e x a m p l e s a l s o s h o w that the s t u d y of s u c c e s s i v e s t a g e s of s i g n a l p r o c e s s i n g i s ess e n t i a l f o r the u n d e r s t a n d i n g of b o t h the a l g o r i t h m a n d its n e u r a l i m p l e mentation. T h e s e e x a m p l e s a r e r e l e v a n t to c o m p u t a t i o n a l n e u r o s c i e n c e the a i m o f w h i c h i s to u n d e r s t a n d the w o r k i n g s of the b r a i n . T h i s field i s , h o w e v e r , t h e o r y - r i c h a n d d a t a - p o o r . T o a c h i e v e its g o a l , the field n e e d s b e n c h m a r k n e u r a l s y s t e m s i n w h i c h b o t h the a l g o r i t h m a n d i t s n e u r a l i m p l e m e n t a t i o n h a v e b e e n w o r k e d o u t . T h e e l e c t r i c fish a n d the o w l p r o v i d e s u c h f r a m e s of reference for t h o s e w h o e x p l o r e o r m o d e l s e n s o r y n e t w o r k s .
Acknowledgments
I t h a n k J a c k G a l l a n t a n d W a l t e r H e i l i g e n b e r g for r e a d i n g the m a n u s c r i p t . This work w a s supported by N I H Grant DC00134.
References
Carr, C . E . , and Konishi, M . 1988. A x o n a l delay lines for time measurement i n the owl's brainstem. Proc. Natl. Acad. Sci. U.S.A. 85, 8311-8315. Carr, C . E . , a n d K o n i s h i , M . 1990. A circuit for detection of interaural time differences in the brainstem of the barn owl. / . Neurosci. 10, 3227-3246. Carr, C . E . , Maler, L . , a n d Sas, E . 1982. Peripheral organization a n d central projections of the electrosensory nerves in gymnotiform fish. /. Comp. Neurol. 211, 139-153. Carr, C . E . , Maler, L . , and Taylor, B. 1986, A time-comparison circuit in the electric fish midbrain. II. Functional morphology. / . Neurosci. 6, 1372-1383. De Yoe, E . A . , a n d Van E s s e n , D . C . 1988. Concurrent processing streams i n m o n k e y visual cortex. Trends Neurosci. 11, 219-226. D u L a c , S., and K n u d s e n , E . I, 1990. N e u r a l maps of head movement vector and speed in the optic tectum of the barn o w l . / . Neurophysiol. 63, 131-146. Fujita, L 1989. T h e role of G A B A mediated inhibition in the formation of a u d i tory receptive fields, Uehara Mem. Res. Found. Rep. 2, 159-161. Getting, P. A . 1989. Emergent principles governing the operation of n e u r a l networks. Annu. Rev. Neurosci. 12,158-204. Gross, C . G „ Rocha-Miranda, C . E . , a n d Bender, D . B. 1972. V i s u a l properties of neurons i n inferotemporal cortex of the macaque.
/. Neurophysiol. 35,
96-111. Heiligenberg, W. 1986. Jamming avoidance responses.
I n Electroreception, T. H .
B u l l o c k and W. Heiligenberg, eds., pp. 613-649. Wiley, N e w York.
592
16
Heiligenberg, W „ Baker, C , a n d Matsubara, J. 1978. T h e j a m m i n g avoidance response revisited: T h e structure of a neuronal democracy. /. Comp. Physiol 1 2 7 , 267-286. Heiligenberg, W., a n d D y e , J. 1982. Labelling of electro receptive afferents i n a gymnotoid fish by intracellular injection of H R P : T h e mystery of multiple maps. /. Comp. Physiol. 1 4 8 , 287-296. Heiligenberg, W., a n d Rose, G . 1986. G a t i n g of sensory information:
Joint
computations of phase a n d amplitude data i n the m i d b r a i n of the electric fish, Eigenmannia. /. Comp. Physiol. 1 5 9 , 311-324. H u b e l , D . H . , a n d Livingstone, M . S. 1987. Segregation of form, color and stereopsis in primate area 18. /. Neurosci. 7 , 3378-3415. Jeffress, L . A . 1948. A place theory of sound localization. /. Comp. Physiol Psych. 4 1 , 35-39. K a w a s a k i , M „ a n d Heiligenberg, W. 1988. I n d i v i d u a l prepacemaker neurons can modulate the pacemaker cycle i n the gymnotiform electric fish, Eigenmannia. ]. Comp. Physiol 1 6 2 , 13-21. K a w a s a k i , M . , Maler, L . , Rose, G . J . , a n d Heiligenberg, W. 1988. Anatomical a n d functional organization of the prepacemaker nucleus i n gymnotiform electric fish: T h e accommodation of two behaviors in one nucleus. /. Comp. Neurol 2 7 6 , 113-131. Keller, C . H . , a n d Heiligenberg, W. 1989. F r o m distributed sensory processing to discrete motor representations i n the diencephalon of the electric fish,
Eigenmannia. ]. Comp. Physiol. 1 6 4 , 565-576. Keller, C . H „ Mater, L . , a n d Heiligenberg, W. 1990. Structural and functional organization of a diencephalic sensory-motor interface in the G y m n o t i f o r m fish, Eigenmannia. J. Comp. Neurol. 2 9 3 , 347-376. K e n d r i c k , K . M . , a n d B a l d w i n , B. A . 1987. C e l l s in temporal cortex of conscious sheep can respond preferentially to the sight of faces. Science 2 3 6 , 448-450, K n u d s e n , E . [., Blasdel, G . G . , a n d K o n i s h i , M , 1979. Sound localization by the b a m o w l (Tyto alba) measured w i t h the search coil technique. /. Comp.
Physiol. 1 3 3 , 1 - 1 1 . K n u d s e n , E . I . , a n d K n u d s e n , R F. 1983. Space-mapped auditory projections from the inferior colliculus to the optic tectum in the barn o w l (Tyto alba). ]. Comp. Neurol. 2 1 8 , 187-196. K n u d s e n , E . L , a n d K o n i s h i , M . 1978. A neural m a p of auditory space in the o w l . Science 2 0 0 , 795-797. K o n i s h i , M . 1991. Similar algorithms i n different sensory systems a n d animals.
Cold Spring Harbor Symp. Quant. Biol, (in press). K o n i s h i , M . , Takahashi, T, T , Wagner, H . , Sullivan, W. E . , and Carr, C . E . 1988. Neurophysiological a n d anatomical substrates of sound localization i n the o w l . I n Auditory Function, G . M . E d e l m a n , W. E . G a l l , a n d W. M . C o w a n , eds., p p . 721-745. Wiley, N e w York. Larimer, J . L . 1988. T h e c o m m a n d hypothesis: A n e w v i e w using an old example. Trends Neurosci. 1 1 , 506-510. Livingstone, M . S., a n d H u b e l , D . H . 1987. Psychophysical evidence for separate channels for the perception of form, color, movement, and depth. / . Neurosci. 7, 3416-3468.
593
17 Livingstone, M . S., a n d Hubel, D . H . 1988. Segregation of form, color, movement, a n d depth: Anatomy, physiology, a n d perception. Science 2 4 0 , 740¬ 749. Manley, G . A . , K o e p p l , C , a n d K o n i s h i , M . 1988. A neural m a p of interaural intensity difference in the brainstem of the barn o w l . / . Neurosci. 8 , 2665¬ 2676. Masino, T . , and K n u d s e n , E . I. 1990. Horizontal a n d vertical components of head movement are controlled by distinct neural circuits i n the b a m o w l . Nalure (London) 3 4 5 , 434-437. M a u n s e l l , J . H . R „ a n d N e w s o m e , W. T. 1987. V i s u a l processing in m o n k e y extrastriate cortex. Annu. Rev. Neurosci. 1 0 , 363-401. Maunsell, J. H . R., a n d Van Essen, C . D . 1983. T h e connections of the m i d d l e temporal visual area ( M T ) a n d their relationship lo a cortical hierarchy i n the macaque monkey. /. Neurosci. 3 , 2526-2586. Moiseff, A . 1989. Bi-coordinate sound localization b y the barn o w l . / . Comp. Physiol. 1 6 4 , 637-644. Moiseff, A . , a n d K o n i s h i , M . 1981. Neuronal a n d behavioral sensitivity to binaural time difference in the o w l . / . Neurosci. 1, 40-48. Moiseff, A . , a n d Konishi, M . 1983. Binaural characteristics of units i n the owl's brainstem auditory pathway; Precursors of restricted spatial receptive fields. /. Neurosci. 3 , 2553-2562. O l s e n , J . F„ K n u d s e n , E . I . , a n d Esterly, S. D . 1989. N e u r a l maps of interaural time and intensity differences i n the optic tectum of the barn o w l . /. Neurosci. 9, 2591-2605. Partridge, B. L „ Heiligenberg, W., a n d Matsubara, J. 1980. T h e neural basis of behavioral filter: N o grandmother cells in sight. / . Comp. Physiol. 1 4 5 , 153-168. Perret, D . L , Rolls, E . T., a n d C a a n , W. 1982. Visual neurons responsive to faces in the monkey temporal cortex. Exp. Brain Res. 4 2 , 319-330. Rose, G . J . , and Heiligenberg, W. 1986. N e u r a l coding of frequencies i n the midbrain of the electric fish Eigenmannia: R e a d i n g the sense ef rotation i n a n amplitude-phase plane. / , Comp. Physiol. 1 5 8 , 613-624. Rose, G . J., K a w a s a k i , M . , a n d Heiligenberg, W. 1988. 'Recognition units' at the top of a neuronal hierarchy? Prepacemaker neurons i n Eigenmannia code the sign of frequency differences unambiguously.
/ . Comp. Physiol. 1 6 2 ,
759-772. Saunders, J . , a n d Bastian, J. 1984. T h e physiology a n d morphology of two types of electrosensory neurons i n the weakly electric fish Apteronolus leptorhynchus. ]. Comp. Physiol. 1 5 4 , 199-209. Scheich, H . , Bullock, T. H . , a n d Hamstra, R. H . 1973. C o d i n g properties of two classes of efferent nerve fibers: H i g h frequency electroceptors i n the electric fish, Eigenmannia. j. Neurophysiol. 3 6 , 39-60. S h u m w a y , C . A . 1989a. Multiple electrosensory m a p s i n the m e d u l l a of weakly electric Gymnotiform fish, 1. Physiological differences. / , Neurosci. 9, 4388¬ 4399. S h u m w a y , C . A . 1989b. Multiple electrosensory m a p s i n the m e d u l l a of w e a k l y
594
18
electric Gymnotiform fish. II Anatomical differences. /. Neurosci. 9, 4400¬ 4415. Shumway, C. A., and Maler, L. 1989. GABAnergic inhibition shapes temporal and spatial response properties of pyramidal cells in the electrolateral line lobe of gymnotoid fish. /. Comp. Physiol. 164, 391-407. Sullivan, W. E., and Konishi, M 1984. Segregation of stimulus phase and intensity in the cochlear nuclei of the barn owl. /. Neurosci. 4, 1787-1799. Takahashi, T. T., Moiseff, A., and Konishi, M. 1984. Time and intensity cues are processed independently in the auditory system of the owl. /. Neurosci. 4, 1781-1786. Takahashi, T T , and Konishi, M. 1986. Selectivity for interaural time difference in the owl's midbrain. /. Neurosci. 6, 3413-3422. Takahashi, T. T, and Konishi, M. 1988a. Projections of the cochlear nuclei and nucleus laminaris to the inferior colliculus of the barn owl. /. Comp. Neurol. 274, 190-211.
Takahashi, T. T, and Konishi, M. 1988b Projections of nucleus angularis and nucleus laminaris to the lateral lemniscal nuclear complex of the barn owl. /. Comp. Neurol. 274, 212-238. Van Essen, D. C. 1985 Functional organization of primate visual cortex. In Cerebral Cortex. A. Peters and E. G. Jones, eds., pp. 259-329. Plenum, New York. Wagner, H , Takahashi, T. T, and Konishi, M. 1987. Representation of interaural time difference in the central nucleus of the barn owl's inferior colliculus. /, Neurosci. 7, 3105-3116. Received 30 August 90; accepted 20 September 1990.
595
Kispri miriJ wilh permission [com Same*, Vol. 240, pp ]<JBft The Amenciin Assoc Ialion lor Ihe Advancement t>f Science
•9
Segregation of Form, Color, Movement, and Depth: Anatomy, Physiology, and Perception MARGARET LIVINGSTONE A N D D A V I D
HUBEL
separate parts whose functions are quite distinct. In this article we
Anatomical and physiological observations in monkeys indicate that the primate visual system consists of several separate and independent subdivisions that analyze different aspects of the same retinal image: cells in cortical visual areas 1 and 2 and higher visual areas are segregated into three interdigitatuig subdivisions that differ in their selectivity for color, stereopsis, movement, and orientation. The pathways selective for form and color seem to be derived mainly from the parvoccllular geniculate subdivisions, the depth- and movement-selective components from the magnocellular. At lower levels, in the retina and in the geniculate, ecus in these two subdivisions differ in their color selectivity, contrast sensitivityj temporal properties, and spatial resolution. These major differences in the properties of cells at lower levels in each of the subdivisions led to the prediction that different visual functions, such as color, depth, movement, and form perception, should exhibit corresponding differences. Human perceptual experiments are remarkably consistent with these predictions. Moreover, perceptual experiments can be designed to ask which subdivisions of the system are responsible for particular visual abilities, such as figure/ground discrimination or perception of depth from perspective or relative movement—functions that might be difficult to deduce from single-cell response properties.
P
summarize some o f these anatomical, physiological, and humanperceptual observations.
Physiological and Anatomical Studies Occasionally people with strokes sutler surprisingly specific visual losses—for example^ loss o f color discrimination without impairment u f form perception, loss o f motion perception without loss o f color or form perception, or loss o f face recognition without loss o f the ability to recognize most other categories o f objects or Joss o f color or depth perception [1). Such selectivity seems to indicate that the visual pathway is functionally subdivided at a fairly gross level. Anaiomical and physiological studies in monkeys also support this idea of functional divergence within the visual pathway. T h e y reveal major anatomical subdivisions at the earliest peripheral stages in rhe visual system as well as segregation o f function at the highest known cortical stages, but until recently there was tittle information about corresponding subdivisions in the intermediate levels, the first and second cortical visual areas. Subdivisions at catty stages in the visaed pathway. I t has been known for a century that the nerve fibers leaving the eyes diverge to provide input both to the lateral geniculate bodies and to the superior colliculir T h e colliculus seems to be relatively more important in lower mammals than it is in primates, in which its main role is probably orientation toward targets o f interest; here we will be
EOPLE WITH NORMAL COLOR VISION WJ [X PROBABLY FIND the ten illustration in fig;. L leu clear and three-dimensional than Eh? One on the right. But it springs forth if you look at it
through a blue filter* such as a piece of colored glass or ccllophane. In the left version the gray and yellow are equally bright, or luminaiit, for rhe average person, whereas the right version has Luminance-contrast information. T h e ability to infer distance i n d three-dimensional shape from a rwo-dimcnsional image is an example o f ^ visual
fiinciinn
that can use luminance bur not color
differences. Depth from perspective and color perception are thus aspects o f vision ihat seem to be handled by entirely s c p a m c channels in our nervous system. Even though intuition suggests that our vision can plausibly be subdivided
into several components—color,
depth,
movement,
form* and texture perception—our perception of any scene usually seems well unified. Despite this apparent wholeness, studies of anatomy, physiology, and human perception arc converging toward the conclusion thai our visual system is subdivided into several The author? arc membrn of die ficultv, Depinment of rVeurabiology. Harvard Medina! School, Boston, MA 02115
Fig. 1. The same image at equdurninancc [leftf and non-equiltirruriance (right). Dcpih from perspective, spatial organizauon, and rlgurc'graund segregation arc diminished in the eqiiilununanr version. T o convince yourself chat the left version don indeed contain the same information « the other, look at ii through a piece of blue cellophane or glass. These two colors may not be close enough to your cquHiimiriance point to be effective. Changing the lighr KfORX may help.
596
741 f*9
>- The primate lateral ecnicu
retinal region and inhibited (impulse rare slowed d o w n ) by illumination o f a larger surrounding region, whereas others are the
uthe &rsr stag, u, t h ^ . j system J j ™ e retina, r*o dirnnct subdivisions, the ventral two niagnocelluW layers and the dorsal fbur parvoceUuiar layers The ™ 0 eye* project tfjfc^ Die intcrdigitiurLg faihion shown; c uidieatcs ] r s thai ire innervated by the contralateral eye^ i indicates l*ycrs with uiput from the ipsilateral e
j (
|
0
c
o
n
s
i
s
H
o
reverse, inhibited from the center and excited from die surround.
f
Because o f the antagonism between center and surround, large uniform spots produce feeble responses or none. T h i s
eenter-
sunround arrangement is found also at earlier levels, starting with the
ta
retinal bipolar cells. Clearly these cells arc wired up so as to convert the information from the photoreceptors into information about
i y C
spatial discontinuities in light patterns. T h i s should not be surprising, since we ourselves are very poor in judging overall levels o f ulunuharion, as anyone w h o tries doing photography without a light meter well knows—we are lucky if we can come within a n / srop (a factor o f 2) o f die right exposure. O n the other hand we c m detect a spot that is as litde as a few percent brighter or darker than its immediate surround. T h e magno and parvo divisions nevertheless differ physiologically in four major ways—color, acuity, speed, and contrast sensitivity { ? 10).
Fig, 3, Rcteptive fields for (Wt) rypical eolor-opponenr parvoceUuiar gcniai laic neuron, excited over a small region by red iighi and inhibited over a larger region by green light and • right rypical broadband nugnoceUular neuron, exeiied by all wavelengths in the eejiier and inhibited by ail wavelengths in its surround.
Color. A b o u t 90% o f the cells in the parvoceUuiar layers o f rhe geniculate
are strikingly sensitive to differences
in wavelength,
whereas cells in the magnoccUular layers are not. T h e three types o f cones in the primate retina have broad, overlapping spectral sensitivities and can be loosely termed red-, green-, and blue-sensitive, to indicate that their peak sensitivities are in the long-, middle-, i n d short-wavelength regions o f the spectrum. Parvo cells are wavelength selective because they combine these cone inputs so as in effect to subtract chem ( F i g . 3, left). A typical parvo ceU may, for example, receive excitatory inputs to its receptive field center from
concerned exclusively with the geniailo-cortical part o f the visual
red cones only, and inhibitory inputs ro its receptive field surround
system, w h i c h seems to be directly concerned with visual perception
from green cones only S u c h a cell will be excited by long wave-
( 2 ) — w h a t w e think o f as seeing.
lengths (reds), inhibited by short wavelengths (blues and greens),
T h e primate lateral geniculate body is a six-layered srjucrure, with
r
and be unresponsive to some intermediate wavelength (yellow).
two obviously different subdivisions: the four dorsal, small-cell
Besides such r e d - o n center, green-off surround cells, most o f the
(parvoceUuiar) layers and the t w o ventral, large-cell (magnocellular)
other possibilities also occur, most commonly red cones antagonized
layers; these t w o subdivisions differ both anatomically and physio-
by green, and blue versus the sum o f red and green (that is, yellow).
logically. I n 1 9 2 0 M i n k o w s k i (3) discovered that each eye projects
In conrrasi to the colot selectivity o f most parvo ceUs magno cells
to three o f ihe six layers in the peculiar alternating fashion shown in
(and also rhe remaining 10% o f the parvo ceUs) sum the inputs o f
F i g , 2: each haff-retina is mapped ihrcc times onto one geniculate
the tiltee cone types, so that the spectral sensitivity curves are broad,
body, twice to the parvoceUuiar layers and once to the magnocellu-
and the response to a change in illumination is o f the same type,
lar, and all sue topographic maps o f the visual field arc in precise
either o n or off, ar all wavelengths ( F i g . 3 , right) ( i i ) . T h e magno
register (4).
system is thus in effect color-blind; as in black-and-white photogra-
T h e four parvoceiluiar layers seem to be very similar, if not identical, anatomically and physiologically. B u t the magno- and parvoceUuiar divisions are profoundly different, implying a major split in the visual pathway. T h i s division is most obvious, and was
h
phy, two different colors, such as red and green, at some relative brightness will be incusonguishable. Acuity. T h e second difference between magno and parvo cells is ihe size o f their field centers. F a r both systems the average size o f the
first recognized, in the geniculate, but it does not originate there,
receptive field center increases with distance from the fovea, consist-
rhe two geniculate subdivisions receive input from two intermixed
ent with the differences in acuity between fovea! and peripheral
bur anatomically distinct types o f retinal ganglion cells: type A cells
vision. Vet at any given eccentricity, magno cells have larger
ate larger and project to the magnocellular division, and the smaller
receptive field centers than parvo cells, by a factor o f 2 or %.
type B cells project to the parvoceUuiar division (5). These two
Speed. M a g n o ceils respond faster and more rransiendy than parvo
subdivisions o f the visual pathway, which we wiU refer to as magno
cells. T h i s sensitivity to the temporal aspects o f a visual stimulus
and parvo, are distinguishable both anatomically and physiological-
suggests that the magno system may play a special role in detecting
ly. W h e t h e r this duality in the visual path arises even earlier, at the
movement. M a n y cells at higher levels in this pathway ate selective
bipolar o r horizontal cells in the retina, is not k n o w n . W e can at least
for direcuon o f movement.
be reasonably certain that the two components must both derive
Conirast.
Shapley it .d (10) found that magno cells are much
their inputs from the same rods and cones and that the marked
more sensitive than parvo cells to low-contrast stimuli. B o t h begin
differences in response properties must therefore depend o n the way
to respond when the center and surround brightnesses differ by only
the photoreceptor inputs are combined.
1 Or 2%, but with increasing contrast magno responses increase
T h o u g h they differ significantly in rheir response characteristics,
rapidly and level off at about 10 to 15% contrast, whereas parvo
the magno and parvo systems do share some basic physiological
responses increase more slowly, and saturate at far higher contrasts.
properties T h e i r receptive fields (the regions o f retina over which
These four major differences between the two subdivisions, in
their impulse activity can be influenced) are all circularly symmetri-
color, acuity, quickness, and contrast sensitivity, imply that they
cal, and about 9 0 % s h o w center-surround opponency {6, 7); some
contribute to different aspects o f vision. Exactly what aspects have
cells are excited (impulse rate speeded u p ) by iUumination o f a small
become clearer recently, with new anatomical techniques that have
597
7« nude it possible io follow these subdivisions farther into the central nervous sysiem and to correlate them with the response selectivity of cells •• later stages in each subdivision for more abstract stimulus features. Concinvaiion
of the magno and parvo iubdipiiions in pintai area 1.
The segregation of the two pathways is perpetuated in the primary usual cortex (12) (Fig. 4). Cells in the magnocellular geniculate layers project to layer 4Co, which projects in turn to layer 4B, which then projects to visual area 2 and to cortical area MT. Parvo cells project to layer 4CB, and from there the connections go to layers 2 and 3, and from there to visual area 2, The parvoceUuiar division splits to form an additional subdivision in the upper layers of visual area l The first evidence for this further subdivision came in 1978 when Wong-Riley (13) stained visual area 1 for the mitochondrial enzyme cytochrome oxidase and saw alternating regions offightand dark staining. The dark regions are round or oval in sections cut parallel to the surface; they are most prominent in the upper layers [2 and 3) but are also faintly visible in layers 5 and 6. They turned out to represent pillar-like structures about 0.2 mm in diameter, spaced 0-5 mm apart (14). We term these srrucrures blobs because of their three-dimensional shape Blobs are found only in the primary visual cortex; they occur in ail primates that have been looked at, and in the piosirnian Galagu, but have not been found in other prosimians or any lower mammals (15). Since layers 2 and I receive most of their inputs from parvorecipicnt layer 4C[3, both the blobs and the interblobs could be considered continuations of the parvo subdivision. Nevertheless rhe blobs should probably be thought of as a separate subdivision, because they have somewhat different inputs and very different response properties from the interblobs (16—IS). The visual response properties of cells in the blobs suggest that they may also receive magnoceUular input {17, i.'< : Thus by the output stage of visual area 1 the magno system remains segregated, and the parvo system seems to have split into two branches. All three subdivisions, magno-»4Ga-*4B, parvo—*4CB—»• interblob, and par^(+magno?)—*4Cp—•blob, then project to visual area 2. These anatomically defined subdivisions in the primary visual cortex differ from each other in the kinds of visual information they cany (18)* as in earlier stages. r
r
In the magno pathway, cells in layet -4B are orientation selective; that is, they respond best to lines of a particular orientation, and most of them also show selectivity for the direction of movement (IS, 19)—fbt example, a cefi preferring horizontal lines may respond when an edge is moved upward but not when it moves downward. Like magnocellular geniculate cells, cells in 4B lack color selectivity. In rhe interblobs, most, perhaps all, cells are also orientation selective. Unlike cells in layer 4B, most are not direction selective; 10 to 20% are end-slopped, responding ro short but not long line or edge stimuli. The receptivefieldsare small, and rhe optimum line thickness is similar to the optimum spot size of cells in the geniculate parvoceUuiar layers at the same eccentricity. This system may therefore be responsible for high-resolution form perception. Although anatomical evidence indicates that the incerblob system receives its major input from the color-coded parvoceUuiar geniculate layers, most of the interblob cells arc not explicitly color-coded: they show no color opponency and respond well to achromatic luminance contrast borders. Nevertheless, many of them respond to an appropriately oriented color-contrast edge regardless of the colors forming the edge or the relative brightness of the two colors. Similarly, they usually respond to lines or borders of any brightness contrast (Ught-on-dark or dark-on-Ught), even though the antecedent geniculate cells are either on-center or off-center but not both. This suggests that much of the color-coded parvoceUuiar input is pooled in such a way that color contrast can be used to identify borders but that the information about rhe colors (including black versus white) forming the border is lost (20). Blob cells are nor orientation selective but are either color or brightness selective. The blob system thus seems to cany information complementary ro rhe information carried by the interblob system. The brightness-selective (non-color-coded or broadband) blob cells have larger receptive field centers than the broadband geniculate cells but are otherwise similar—they are either excited or inhibited by small spots of light, and rhey respond less weU to large spots, indicating surround inhibition. These broadband blob cells could receive input from either the magnocellular geniculate cells or from rhe broadband parvo cells, bur rhe physiological properties of many of them would be more consistent with inputfromdie magno system (17, IS). We assume that the color-opponent blob cclis receive inputfromthe color-opponent parvoceUuiar geniculate cells,
598
743 Fig. S. Section parallel to the surface through visual areas I and 1 of a squirrel monkey, stained for cytochrome oxidase. Visual area i is on the left, the blobs appear as small round dots. In Visual area 2 the cytochrome -oxidase stain reveals a parcem of alternating thin, thick, and pale stripes.
ent with the properties of cells in the antecedent subdivisions of visual area 1 {18, 23). Cells in the thin stripes showed no orientation selectivity, and ovet half were color-coded, just as we had found in the blobs. As in the blobs, most of the color-coded cells were doubly opponent, with two antagonistic inputs to their centers, and surround antagonism for both of these center inputs. About half of the thin-stripe cells, both broadband and color opponent, exhibited an additional property not seen in the blob cells: the receptive field centers were bigget, yet optimum spot sizes were about the same. A typical cell might tespond best, say. to a 0,5° diameter spot, give no response at all to 2" or 4° spots (indicating surround antagonism), and yet respond actively to the 0.5° spot anywhere within an area about 4° in diameter. These cells can be broadband or color opponent. Several years ago Baizer, Robinson^ and Dow (24) described this kind of broadband cell, which they called "spot cells," in visual atea 2. Celts in the pale stripes are orientation selective but not direction though they differ from them in that their receptive field centers are selective. At least half of diem are end-stopped; this represents a larger and rhdr color coding is doubly opponent—they give oppo- dramatic increase in the proportion of end-stopping over what is site responses to different pans of the spectrum in the center (say, on seen in visual area I . We have argued that end-stopping, like centerto red and off to green), and both rypes of center response are surround antagonism, is an efficient way of encoding information reduced when rhe spot is made larger. about shape [23). Like cells in the interblobs, pale-stripe cells are not The blob and interblob systems thus work in entirely different and explicitly color-coded, and we expect rhat they would respond to complementary ways. Blob cells are explicitly color-coded, excited color-contrast borders at all relative brightnesses, though we have by colors in one region of the spectrum and inhibited by others, and not yet rested this. not selective for stimulus orientation. Interblob cells arc selective for In the thick stripes the great majority of cells likewise show stimulus orientation but mosdy arc not color selective, responding otientation selectivity, but are seldom end-stopped. The most to a line or edge of the correct orientation regardless of its color. The Consistent response selectivity we see in the thick stripes is for strategy of carrying orientation information in a sysrem rhat mosdy stereoscopic depth—most cells respond poorly to stimulation of pools color information and color-contrast information in a separate cither eye alone but vigorously when both eyes arc stimulated system that docs not carry orientation information is ptobablv more together, and fot most cells the tesponscs are extremely sensitive io efficient than having single cells selective for both the orientation variations in the relative hotizontal positions of the stimuli in the and color of a border. Nevertheless, as emphasized earlier, although two eyes (teunal disparity). Poggio and Fischer (25) had seen most of the interblob cells are not overtly color selective, they similar disparity-tuned cells in visual area 1 in alert monkeys, probably receive their inputs from explicitly color-coded parvoceUu- predominantly in layer 4B, the layer that pro|ccts to the thick stripes iar geniculate cells and are most likely not color-blind in rhe sense of visual area 2. In the thick stripes we find the same three basic rhat rhe cells in the magno system probably are. Most interblob cells, even though they lose the information about the colors that form a border or the sign of the contrast of the border, should respond to color-contrast borders b which the two colors arc equally bright; such borders would be invisible to magno cells. Visual area 2. The main target of visual area 1 is visual area 2 (Brodmann's area IS), which shows an equally intriguing pattern when stained for cytochrome oxidase (Fig. 5) (16, 21). Instead of small round dots, tangential sections show a pattern of stripes, much coarser than the blobs of visual area \\ these alternately dark and light stripes are several millimeters wide and run perpendicular to rhe border between visual areas 1 and 2, probably extending over rhe entire 8- to 10-mm width of visual area 2. The dark, stripes are themselves of two types, thick and thin. The regularity of this partem of thick, thin, and pale stripes varies from animal to animal and is clearer in New World monkeys than in Old World ones, at least partly because in Old World monkeys visual area 2 is buried in rhe lunate sulcus. Given three histologically defined regions in visual area 2 and rhe fact that visual area 1 has three kinds of subdivisions that project to other conical areas, it was natural to ask if they were related. And indeed, from tracer injections into the three kinds of Fig. 6. (A) Lossofdepih from parallax at equilummante. The position of the stripes in visual area 2, wc found that the blobs are reciprocally middle bar is rrndc to vary with the observer's head position. ] this case, the connected to the thin stripes, the interblob regions to the pale center bar appears to lie in front of the reference bars, except when the bars are madeequiluminant with the background. (B> Two frames of a mo^ic in smpes, and layer 4B to the thick stripes (Pig. 4) (18 22). which the movement of dots generates the sensation of a three-dimensional The next step was to record from cells in visual area 2, to learn object. The dois appear to Lie on the surface of a sphere (which you can see whether the three subdivisions carry different types of visual infor- by stereo-viewing these twoframe*).All sensation of depth is lost when the mation. Wc did indeed hnd marked differences, which were consist- dots are eqiiilumwanr with the backgroundn
r
r
599
144
Fig, 7. Computer-generated images in which shape is generated by shading. In ihe middle image (he two colon are equiluminam. and the thrte-dimcnsioiiit shape is harder to discern than in the other two images, which have luminance^ contrast.
Am
classes of celts described by Poggio and Fischer—cells selective for neat stimuli, far stimuli, ot stimuli falling on exactly corresponding retinal points. Like cells in the pale stripes, these cells show no color selectivity, moreover, we would predict that diese cells wnuld be like their magnocellular predecessors and would HOT respond to colorcontrast borders when the colors are equally bright, though wc have not yer tested responses to equiluminani color-contrasi borders eirhcr in layer 4B of visual area 1 or in the thick stripes of visual area 2. Other studies (26, 27) have not reported such a clean segregation of cells with different physiological properties in visual area 2. ot as clear a correlation of physiological subtypes with the three rypes of stripes. How clear the functional segregation is in visual area 2 remains to be resolved, bur we suspect that these differences arc due in choice of classification criteria (23). Higher visual a*tas. Meanwhile, explorations of visual areas bevond I and 2 are helping close the gap berween the functions suggested by electrophysiological studies and what clinical observations imply about die segregation of various functions in the human visual system. The response properties of cells at levels beyond visual area 2 suggest that the segregation of functions begun at the earliest levels is perpetuated at the highest levels so Mr studied. Indeed, rhe segregation seems to become more and more pronounced at each successive level, so that subdivisions thai are inrerdigitated in visual ateas 1 and 2 become segregated into entirely separate areas at still higher levels. One higher visual area in the middle temporal lobe, MT, seems to be specialized for rhe analysis of movement and stereoscopic depth (28). It rccenes input not only from laver 4B in visual area 1 (29), which is also rich in directionality and disparity selectivity, but also from the thick stripes in visual area 2 (26 30), which, as already described, contain manv cells selective for binocular disparity {23). Anothci higher visual area, visual area 4. has been reported to contain a preponderance of color-selective cells (31) but just how specialized visual area 4 is for color is still unclear since many of the cells show some selectivity for orientation. Visual area 4 receives input from the colot-codcd thin stripes in visual area 2 and possiblv from the pale stripes (26, 30, 32). The notion rhat there is a higher visual area devoted largely to the processing of color information is consistent with the clinical observation that patients wilh strokes in the posterior inferior occipital lobe (perhaps in a region homologous to visual area 4) can lose color perception without impairment of form or movement perception.
.
A%V '
functionally distinct divisions of visual association areas; the temporal-occipital region, necessary for teaming to identity objects by their appearance, and the paricto-occipital region, needed for tasks involving rhe positions of objects, a distinction they refer to as "where" versus "what. Visual area 4 preferentially projects to the temporal division and MT primarily TO parietal cortex (Jd).Thus the temporal visual areas mav represent the continuation of the parvo system, and the parietal areas the continuation of the magno pathwav. There can be little doubt that in the next few yeats wotk on the dozen or so areas north of the srriate cortex will gready enhance our understanding of vision in general. 71
H u m a n Perception
Despite many gaps, the picture beginning to emerge from the anatomical and electrophysiological studies summarized above is that the segregation begun in the eve gives rise to separate and independent parallel pathways. At early levels, where there are two major subdivisions, the cells in these two subdivisions exhibit at least Ebur basic differences—color selectivity, speed, acuity, and contrast sensitwitv. At higher stages the continuations of these pathways are selective for quite different aspects of vision (form, color, movement, and stereopsis). thus generating the counter intuitive prediction that different kinds of visual tasks should differ in their color, temporal, acuity, and contrast characteristics. To test this prediction, wc asked whether the differences seen in the geniculate can be detected in consoous human visual perception by comparing the color, temporal, spatial, and contrast sensitivities of different visual functions. Many of these questions, not surprisingly, have already been asked, and the answers arc wonderfully consistent with the anatomy and physiology. For several decades psychologists have accumulated evidence for two channels in human vision, one chromatic and the other achromatic, by showing that different tasks can have very different sensitivities to color and brightness contrast. Given what we know now about the clcctrophysiology and the anatomy ol the subdivisions of the primate visual system, we can begin to try to correlate the perceptual observations with these subdivisions (37). Though at higher cortical levels there seem to be thtee subdivisions, possibly with some mixing of magno and parvo inputs to the blob system, the most important distinction is probably between the magno system (magno—»4Cai—*4B—*MT) and the parvo-derived subdivisions Iparvo—'4C&—'interblobs—'pale There arc strong suggestions that these channels remain segregat- stripes—'- visual area 4?) and [parvo{ + magno?)—»4C0—»blobs-»rhin ed through still higher levels in the brain (33). From lesion studies stripes-'visual area 4]. In our discussion of human perception we Pohl (34) and Ungerleider and Mishkin (35) have dehhed two will, therefore, stress the distinctions between functions that seem to r
b
600
745 Gibson's corridor i l lusion.fFrom^T) " i t h p c r Hussion, copyright 1950. H o u g h t o n Mifflin] AT equi »«™n*nce die image no longer appears to recede into the distance, and the cylinders a" appear to be the same sire, as indeed they actually arc.
Finding cells in the duck stripes of visual area 2 and in MT mat are tuned to retinal disparity suggests that the magno system is also involved i n stereoscopic depth perception. Consistent with this, L u and Fender (42) found lhat subjects could not see depth in equiluminant color-contrast random-dot stereograms even though the dots making up rhe stereogram remained perfectly clear (43). This Ending has been disputed, but we found that differences in results can arise from variations in subjects' equiluminance points with eccentricity, which make it difficult ro achieve equiluminance across rhe visual field. Like movement perception, stereopsis fails for stereograms contairiing only high, but resolvable, spatial frequencies, but it is not curninished for rapidly alternating or very low contrast stereograms (37) {Table 1). Deduction
offurther
magno or parvo junctions from perceptual
tats.
Since rhe functions that electrophysiological studies had suggested should be carried by the magno system did indeed show ail four distinguishing characteristics of that system, we decided to ask whether other visual functions, ones not predicted by single-cell response properties, might also manifest some or all of these be carried exclusively by the magno system and those that seem to be properties. carried by the parvo-derived pathways. If a particular magno cell sums red and green inputs, there will be From the fact that the magno sysiem is color-blind and is faster a red : green ratio at which the red and green will be equally effective than the parvo system, we can predict rhat discrimination of color in stimulating the cell. This need not imply that every magno cell has and discrimination of brightness should have different temporal the same ratio of red to green inputs and therefore necessarily the properties. This is indeed so: in 1923 Ives (38) showed that people same equiluminance point. Nevertheless, the fact that movement can follow brightness alternations at much faster rates than pure and stereopsis fail at equiluminance implies that, for a given color alternations. observer, rhe null ratio must be very similar for the majority of his The high incidence of movement and direction selectivity in M T cells responsible fot that function. Krilget (44) found that of 33 suggests that this area may be particularly concerned with move- magnocellular geniculate cells studied in two monkeys, 75% wete ment perception. Because anatomically MT receives its major inputs unresponsive to a moving color-contrast border at a particular from layer 4B of the primary visual cortex and from the thick stripes relative brightness—a brightness ratio that was very dose to a of visual area 2, both part of the magno pathway, one would predict human observer's equiluminant point. Thus not only do individual that human movement perception should somehow reflect magno cells in the magno system seem ro be color-blind, but the properties characteristics • colot blindness, quickness, high contrast sensitivity, of stereopsis and movement perception indicate that the magno and low acuity. Perceptual experiments indicate that movement system as a whole is color-blind. (There is, however, currendy some perception does indeed have these characteristics- First, it is im- disagreement about whether the magno system is inactive at equilupaired for patterns made up of equdurninant colors: Cavanagh. minance (45).1 People with the most common forms of color Tyler and Favreau (39) found that if they generated moving red and green sincwave stripes, "the perceived velociry of equjluminous gratings is substantially slowed . . . the gratings often appear to stop even though their bars are clearly resolved . . . the morion is appreciated only because it is occasionally noticed that the bars are at some new position'' [39* p. 897; 40). Second, movement perception • • Display is impaired at high spatial frequencies, consistent with the Jowet * acuity of the magno system. Campbell and Maffei (41) viewed slowly rotating gratings and found a loss of modon perception at the highest resolvable frequencies, "At a spatial frequency of 16 and 32 cyeles/deg a strange phenomenon was experienced, rhe grating was perceived as rotating extremely slowly and most of the time it actually appeared stationary. Of course, the subject could call upon Percept ot his memory and deduce that the grating must be moving for he was non-eqiJl luminaries aware rhat some seconds before the grating had been at a particular 'clock-face position/ Even wilh this additional information that the grating musl be rotating the illusion of'stopped motion persisted" (41, p. 714). What is most surprising about the perception of both the equiluminant stripes and the very fine stripes is that even though the sensation of movement is entirely, or almost entirely, lost, rhe Percept er aqul luminance stripes themselves are still clearly visible—they are clear enough that changes in their position can be seen, even though they do not seem to be moving. Last, movement can be vividly perceived with very rapidly alternating or very low contrast images (37, 41). Thus, as summarized in Table 1, the properties of human movement percep- Flfl, fl- Linking by movement is lost at cquiliomirianee. A l l nine o f rhe ambiguous morion squares appear to move in synchrony, even though any tion are remarkably consistent with the properties of the magno one seen alone could be seen moving either horizontally or vertically. This sysiem. linking disappears at equiluminance, and rhe dots move every which way. v
TTTTTi
Ii II I i It It It S I
1
601
746 blindness, due to the lack of one of the three cone pigments, are not nearly as color-blind as the magno system appears to be. They still have two cone types to compare, and so they COD fuse only a small traction of possible color pairs and can differentiate most color pairs at all relative brightnesses. Since both motion perception and stereoscopic depth perception are lost at cqu il urn in a nee, we suspected that the ability to use relative motion as a depth cue might also be lost. Relative motion is a very powerful depth cue; when an observer moves his head back and forth or moves around in his environment, the relative motion of objects provides information about their distance. In the experiment shown in Fig. <•••*. rhe position of rhe middle bar was coupled to head movement, and the middle bar appeared ro be either behind or in front of the reference bars, depending on whether its movement was the same as, or contraiy to, the head movement. When the bars were made equiluminant with the background, all sensation of depth disappeared (37). Relative movement of different parrs of a three-dimensional object is also a powerful depth cue. Figure 6B shows rwo frames of a movie in which random dots move, some to the right and some to the left, as if they were pasted on a rotating spherical surface. The movie gives a powerful sensation of a rotating spherical surface— unless the dots are cquunjiunani with the background, and then all sensation of depth is lost (37), and the dots seem to dance aimlessly. Thus depth from motion, both from viewer parallax and from object motion, seems also to depend on luminance contrast and could well be a function of the magno system. Consistent with this idea, we could see depth from motion at very low levels of luminance contrast (37). The retinal image is of course two-dimensional, and to capture the three-dimensional relationships of objects the visual systems uses many kinds of cues besides stereopsis and relative motion—perspective, gradients of texture, shading, occlusion, and relative position in the image, VVc wondered whether the sensation of depth from any of these other cues might also exhibit magno characteristics. I i
seemed especially likely that the ability to perceive depth from shading might be carried by an achromatic system, because shading is almost by definition purely luminance-contrast information; that is, under natural bghring conditions a shaded region of an object has the same hue as the unshaded parrs, simply darker. But in biology just because something could, or seemingly even should, be done in a certain way does not mean that it will be. Nevertheless, Cavanagh and Leclerc (46) found rhat the perception of three-dimensional shape from shading indeed depends solely on luminance contrast. That is, in order to produce a sensation of depth and three dimensionality, shadows can be any hue as long as they are darker than unshaded regions of the same surface. Many artists seem to have been aware of this; for example, in some of the self-portraits of Van Gogh and Matisse the shadows on their faces are green or blue, but they still convey a normally shaped face. Black-and-white photographs of these paintings (taken with film that has approximately the same spectral sensitivity as humans) confirm that the shadows are actually darker than the unshaded parts. The converse can be seen in Fig 7; here the green shadows do not convey a sensation of depth and shape when they are the same brightness as the blue but do when they are darker (when the blue is darker, the blue parts are interpreted as shadowed). Perspective was well known to artists by the time of the Renai** santc and is a powerful indicator of depth. Converging lines or gradients of texture are automatically interpreted by the visual system as indicating increasing distance from rhe observer; thus the image in Fig, 8 (47) Looks like a cotridor receding into the distance despite the coiu^cting information from other depth cues, the absence of stereopsis or relative motion, which tells us we are Looking at a fiat surface. The perception of depth from perspective probably underlies many illusions: the rwo cylinders in Fig 8 are the same size (and they each cover the same area on your retina), but are perceived by most people as being unequal. We found that when images with strong perspective are rendered in equiluminant colors instead of black and white, the depth r
r
Tibto 1. Summary of the correlations between human psychophysical results and the physiological properties of rhe three subdivisions of rhe priirure gctuculo-eoitical visual system. A cheek indicates rhar the psychophysical results are consistent with the physiology, .n.l a. blank indicates thai such an experiment his riot been done. Magno System
Physiology Human perception Movement perception Movement detection Apparent movement Depth cues Stereopsis Fjiteroeular rivalry Parallax Depth from motion Shading Contour lines Ocdusion Perspective Linking properties Linking by movement Linking by collin canty (•Ihuory borders) Figure/ground discrimination
Color Contrast ielecseraitivity bvity no high
Parvo System Parvo —• Interblob pathway
Temporal Spatial tcsorcsolution luuon fast low
Physiology Human perception
Color Contrast Temporal Spatial selctsensiresore» ovity tiviry lutioa fubal yes low ilow high
Shape discrimination Orientation disciimination Shape discrimination
Parvo* (Magno?) -s Blob pathway
Physiology Human perception Color perception Color determination Flicker rjhotomctry
602
Color Contrast Temporal Spatial selecsensiresoresorjvfty tiviry iurion lutioa yes high slow low
747
Fkjj. 10. Linking by .u:- ft is den which edges arc part of the same object, even when oeduded by another object. At cqiitiuiruhanee this linking disappears, and it looks like a jumble of lines instead of a pile ofbloclu. After
Fig. 11. llliisory benders which disappear at equiiiaminancc. Redrawn from (52).
discrete objects, to distinguish them from each other and from the background. Barlow (49) has called these "linking features" because they are used to link or join related elements. These unking features include: common movement (objects move against a stationary background; contours moving in the same direction and velocity are likely to belong to the same object, even if they are different in orientation or not contiguous); common depth (contours at differ¬ ent distances from the observer are unlikely to belong to the same object); collinearity (if a straight or continuously curved contour is interrupted by being occluded by anothet object, it is still seen as a single contour); and common color or lightness. The results described below suggest, however, that only luminance contrast, and not color cUfibrcnces, is used to link parts together, Rarnachandran and Anstis (50) discovered a powerful example of linking by movement. If two dots on a diagonal are alternated with two other dots, in mirror-image positions, an observer sees apparent movement, which can be cither horizontal or vertical. The direction of the observed alternating movement is completely ambiguous; observers usually see one direction for a few seconds, and then flip to the other. With a display of several such ambiguous-motion squares in an array (Fig. 9) all the squares are perceived as moving in the same direction, like Rockettes, either all horizontally or all vertically (even though any one of them viewed alone is equally likely ro be perceived as moving in either direction), and when one flips its apparent direction of movement, they all flip When the dots are made equdurtiinant with the background the synchrony breaks down and they all seem to move independently (37). Linking by coUincaxity {Fig. 10) also breaks down when the lines are equiluminant with the background; the figure then just looks like a jumble of lines instead of a pile of blocks. Linking by collinearity is seen in the phenomenon of illusory contours (51,52), figures that produce a vivid perception of an edge in the absence of any real discontinuity (Fig, 11), When thesefiguresare drawn in equiluminant colors, the illusory borders disappear, even though the elements defining them (the pacmen, the spokes, rhe Lines, or the circles) remain perfectly visible. Because rhe perception of illusory borders also manifests fast temporal resolution, high contrast sensitivity, and low spatial resolution, we suspect that it too may represent a magno function. Illusory borders have been called "cognitive contours" because of the suggestion that the perception of the border is due to a high-level deduction that there must be an object occluding a partially visiblefigure(53). We suspect rhat this is not rhe case because the illusory borders disappear at equiluminance, even though the real parts of the figure are still perfectly visible. Fifty years ago the Gestalt psychologists observed that figure/ ground discrimination and the ability to organize the elements in a scene decrease at equiluminance. Equiluminantfigureshave been described as "jazzy," "unstable," "jelly-like," or "disorganized" {43, 54), Koffka (55) pointed our that luminance differences are strikingly more important than color clifferences forfigure/groundsegregation; Thus two greys which look very similar will give a perfectly stable organization if one is used for thefigureand the other for the ground, whereas a deeply saturated blue and a grey of the same luminosity which look very different indeed will ptoduce practically no such organization'' (54, p. 127). Edgar Rubin's popular demonstration of the problem offigure/grounddiscrimination is the vase/ faces (Fig. 12). At non-equiluminance the percept is bistable, so that one sees either the faces or rhe vase, but usually not both at the same time. At equiluminance the two percepts tevetse rapidly, and one can occasionally see both the vase and the faces simultaneously. The distinction between figure and ground thus gets weaket ot even disappears entirely. r
sensation is lost or grcady diminished (37). Illusions of size are likewise Josl at equ Luminance—the cylinders in Fig. 8 are then all corrccdy perceived as being the same size. As with movement and stereopsis, the most sranling aspect of this phenomenon is that even though the sensation of depth and the illusory distortions due to inappropriate scale all disappear at equiluminance, the lines defining the perspective and the individual elements in the image are nevertheless still dearly visible. This seems to us to rule out highlevel, cognitive explanations for depth from perspective and the illusions of perspective; if you see depth because you merely know that converging lines mean increasing distance, you should be able to perceive the depth from the converging lines at equLuminance. Thus at a relatively low level in the visual system some simple interactions must initiate the automatic interpretation of a twodimensional image into three-dimensional information; moreover, these operations seem to be performed only in the achromatic magno system, not in the parvo system. Why should the depth and movement functions described above all be carried by the magno system and not by the parvo system? We at Gist assumed that it was because they might all be performed best by a system with the special characteristics of the magno system. But later we wondered if these various functions might be more related than they seemed atfirst—whetherthey could all be parts of a more global function. We were struck by the similarity between the list of functions we had ascribed to rhe magno sysiem and the Gestalt psychologists' list of features used to discriminate objects from each other and from rhe background—figurc'ground [discrimination (48). Most scenes contain a huge amount of visual information, information about light intensity and color at every point on the retina and the presence and orientation of discontinuities in the light pattern. The Gestalt psychologists recogruzed rhat one important step in making sense of an image must be to correlate related pieces of visual information, that is, to decide whethet a series of Eight/dark discontinuities forms a single edge, whether adjacent edges belong to the same object, whether two pans of an occluded edge are related, and so on. They drtcrmined that several kinds of cues are Color amtraft versus color bleeding. At any poult in the visual field, used in this way and to organize the visual elements in a scene into cells in rhe blobs have receptive field centers that are two to four
603
748 Fig. 1 2 . Rubin's demonstration of i,;.-.ir:: -.T. I disciiminanon [after {48}) L l a lumiriarire-riirttrast image like : !'• you sec cither the vase Or rhe faces but not both. At equiluminance you can see both simultaneously, or ihey alternate very rapid-
I
times larger than those in the interblobs \lf!)- Since only the blobs seem to retain information about the sign of color contrast, we suspect that they are responsible for the perception of the actual colors of objects, as opposed to the ability to use color or liiminance contrast to perceive the holders of objects. This implies that color perception should have lower spatial resolution than form perception. This difference in spatial resolution may explain a phenomenon of color perception described by Chevrcul (56) in 1839 and by von Bezold (57) in 1876, the phenomenon of bleeding. The way two adjacent colors can affect each other depends on their geometrical arrangement. When two large regions of color abut, their apparent colors and lightnesses repel each other, each making the other took more like its complement, a phenomenon consistent with the center/ surround antagonism in the blob system. For example, a gray spot surrounded by red will look slightiv greenish, and the same gray surrounded by green will appear slighdy reddish; surrounding the gray bv white will make it appear dark, and surrounding it by black will make it seem lighter. This is called simultaneous contrast and can be seen in Fig. 13. Two colon can have exactly rhe opposite effect on each other if their geometrical arrangement is such that one forms a very fine pattern, such as fine stripes or dots, with the other as a background. In the lower half of Fig. 13, the mortar seems to bleed into the surrounding gray; the white mortar makes the gray look lighter, and the black mortar makes the same gray look darker. We suspect that bleeding occurs when a partem is too fine to be resolved by the low acuity color system but not too fine for the higher-resolution form system. Thus you see a pattern, but the colors do not seem to conform to the pattern. We think that the interblob system and the magno system can both define shape, and we cannot predict whether one or the other is more important in defining the borders to which the colot is assigned. Some observations, however, suggest that the magno system can influence the spreading of colot; color bleeding can be contained by illusory borders or by borders denned only by stereopsis; also, stationary patches of color can seem to move with moving Luminance-contrast stimuli 158).
Of course a pattern can be too fine to be seen by either system, as in the microscopic dots used in magazine illustrations. In this case the individual dots cannot be seen, and rhe colors simply blend. Many artists of the Impressionist period were aware of rhe way the colors in a resolvable partem can bleed; they often made dots or dabs of paint large enough to be seen, but small enough that their colors blended [59). The television industry takes advantage of these differences in spatial resolution by broadcasting the color part of the image at a lower resolution than the black and white parr, thus reducing the amount of information to be carried. Why should the visual system ire subdivided}
Electrophysiological
studies suggest that the magno system is responsible for carrying information about movement and depth. We extended our ideas about the possible functions of the magno system with perceptual studies and concluded that the magno system may have a mote global function of interpreting spatial organization. Magno func-
tions may include deciding which visual elements, such as edges and discontinuities, belong to and define individual objects in the scene, as well as determining the overall three-dimension^ organization of the scene and the positions of objects in space and movements of objects. If the magno system covers such a broad range of functions, then what is the function of the tenfold more massive parvo system? The color selectivity of the parvo system should enable us to see borders using color information alone and thus borders that might be camouflaged to the color-blind magno system. But defeating camouflage may be only a small part of what the parvo system is specialized for. Experiments with fading of low contrast images [37) indicate that rhe magno system is not capable of sustained scrutiny, since images that can be seen by only the magno system disappear after a few seconds of voluntary fixation. Thus while the magno system is sensitive primarily to moving objects and carries information about the overall organization of rhe visual world, the parvo system seems to be important for analyzing the scene in much greater and more leisurely detail. These postulated functions would be consistent with the evolutionary relation of the two systems: the magno system seems to be more primitive than the parvo system (60) and is possibly homologous to the entire visual system of nonprimate mammals. If so, it should not be surprising that the magno system is capable of what seem to be the essential functions of vision for an animal rhat uses vision to navigate in its environment, catch prey, and avoid predators. The parvo system, which is well developed only in primates, seems to have added the ability to scrutinize in much more detail rhe shape, color, and surface properties of objects, creating the possibility of assigning multiple visual artribures to a single object and correlating its parts. Indeed, if the magno system needs to use the various visual attributes of an object in order to link its parts rogether, this could preclude its being able to analyze the attributes independently. It thus seems reasonable to us that the parvo—• —•temporal lobe system might be especially suited for visual identification and association. Is the existence of separate pathways an accident of evolution or a useful design principle? Segregating the processing of different types of information into separate pathways might facilitate the interactions between cells carrying the same type of information. It might also allow each system to develop functions particularly suited to its specialization. If the parvo system did evolve after the magno system, by duplication of previously existing structures, it should not be surprising to find some redundancy in the properties of the two systems. Indeed, both seem to carry information about orienta-
m Fig. 1 3 - Simultaneous contrast versus bleeding. This phenomenon is shown for black and white, but it is also true for colors. When a spot is surrounded by another color or brightness, the apparent color of die spot rends toward the opposite, or complement, of the surround. The exact opposite hap' pens when one color forms a fine partem on the other; then the colors bleed.
604
MM
749 don, and perceptual experiments indicate that both systems can be used to determine shape. We have summarized the anatomical, physiological, and psychological evidence for segregation of function in the primate visual system. By comparing our own perceptual abilities with the electrophysiological properties of neurons in dilierenr subdivisions of the visual system, we may be able to deduce functions of particular visual areas, functions that might not have been obvious from electrophysiological observations alone. We can now go back to physiological experiments to test some of the ideas raised by the perceptual experiments. REFERENCES AND NOTES
22. 23. 24. 25.
M S LLvlngstone and D H Hubel./ «™-wn 7.337) (1987) D. H. Hubel and M. 5. Livingstone, M , p. 3378¬ ]. S.flaizer.D. L. Robinson, B. M. Dow./. Nturvpbynal. 40, 1024 (1977). G.'F. ?oggioand B. Fucher. ibid , p 1392L G. F Poggio, in l>m*ii.^;;^ o/ NtawtitMlFunoi&vG. M. Edelman, W. £. GaJ. W. M. Co«an Ed* (Wiley New York, 19B4). pp 631-632. 16. E. A DeYoe and D. C. Van EMOI, Nnurr (Lendortt 317, 58 (I985J. 27 A Burkhalter and D C Van E«cn,/. No*rwn 6, 2327 (1986) 2fi. R. Dubrwr and S. M. Zcki. Brain Rti. 35, 528(1971)J. H. R. Maunsell and D. C-
VanEuen./ N ™ ^ J H J 49, 1140 (19B3).
29. J. 5. Lund, R. D. Lund, A. E. Hendncbon. A. H. Buni, A. E Fuehl /. Camf. WW. 164, 2B7 (1975), W. fl. Spatz, Sm.>i flu 92. 450 (1975); L. G. UnnrlJodcr and M. Mishkin, /. Gwp. rJeunf IBB. 347(1979). 30. S. StUpp and S. Zcki. NtHnri {Ltmdanl 3L5. 322 (1935) 3L. S Zeki, P*IW. 284.112 (L980) 32. References. (2d) and (30) art in agreement [hat the dudcimpes; project ro frfTatwl Tfla! the thin stripes; project to vittial area 4, but only |2fl) reports, thai the pale stripa project to visual area 4. 33. Fora review, f. H. R. MaunscL,.jtfa/rmd/in/fl££Ai
1. H . Liuaurr, Arch, piychlaa Nlmthr 21, 22 (IBM); J- Bodimcr. J W 179 6 (19*7); A R Damasio, T Yamida. H. Damasio, J Corben, J. McKce. iJnrrtgy 30, IQfi* (1980); A L PearlmanJ. Birch. J. C. Meadows, Ann. Nturol. 5, 253 3*. W. Pohl, /. Comp. Phyna Ptytfai. B2, 227 (1973). (1979); D Vcrrcy.^1* OpblbalmU. (fm] B. 2*9(1868}; R. Balint,Mmaiatr. 35. LG. TJn|Cikjder and M. Mishkin, In A naiym ofVimei BiHaw, D. I Ingk.M. A Prjchmr. Wfml. 25. 5) (1909); J. Zihl, D. VonCramQn.N. Mai.flVain 106, 3L3 Goodafe, R.). W. Mansfield, Eds. (MIT Press, Cambridge, MA. 1982), pp 549¬ (19B3). Cortical loss of color perccpQon and loss of face reeognlrion HJUiliy Occur 586. •OgCthcr. but each can ccnir independently. 36. K. 5. Rockland and D. N Pandys,flmii.Ra 179. 3 (1979); R. Dwimonc. J. 2. H. Munk, CmnvlU. Praia. AngrnhriU. 3, 255(1879). Fleming, C. G. Grou. M 1B4. 41 (1980), J. tt. B. MaimteU and D C. Van 3. M. Minkowski. Anh. Nn.ml. Pryihvtir 6, 201 (1920). Essen./ Nturati. 3. 2563 (1983). *. W. E. Le Gros Clark and G- G. Penman, PIK. R. Sac. Land. Srr.BlH.29l (193*) 37. M. S. Livingstone and D. H. Hubel,/ Ntmaa. 7, 3416 (1987). 5- Brady. Prec. Km Ntd. Akad Wn. 37, 724 (1934); |. G Malpeli and F F3. 3B. H. E \*a,J. Oft S«. Am. Rn Sa 7-utr 7, 363 (1923) Baktr,/. Omp. tttrn*. 161, 569 (1975). 39. P Cavanagh, C-W. Tyler, O. E. FiVrtau./ Oct. Sw.Am. B. 893 (19B*) 5. A G . LrvenduJ. R. W. RodJ«k, B. Dieher. Sam* 213, 1139 (1981). 40 We found (37) lhal apparent movemem ai well H real movement diiappearid it 6. S. W. Kuffler./. Neuroptytil 16. 37 (1953). equiluminanee. This reult is controversial, buE we suspect that diBerenr findings may be due to difliciiltiei m achieving eaijjiJJTjCUnce across the i1>u*L Held. 7. T. N. Wmd and D. H Hubel 29, 1115 (1966)i P. H. Schiller and J. G. 41- F. W. CatnpbeJ md L Matfe, Vuian Rei. 23, 713 (19HI) M.,l;-:l: *1. 7flg (1978). 8. R. L. DeVjJoh.E. Abramov. G. H. Jacobs./. Op: Hoi Am 56, 966 (1966); R. L 42. C. Lu and D. Fl Fender, b M OpbthAlmoi. 11. 4B2 (1972) Dr Valois, D. M. Snodderry, Jr., E . W. Yuitd, N K. Hepter. Hem Finn 1, 244 43. R. L . Gregory, PmrpHm6, 113 (L977). (1977); A. M. Dernngton, J. Krausfcopf, F Lcnnle,/. fVtfJ. (I™4mJ 357, 241 44- ). KrUgti.E^ B-u*Rn 30.297(1979) (1981); E. Kaplan and R_ M Shapley, ittf. 330,125 (I982hf"*. J M <W. to. 45. Demngcon and LennJt (?) have reported that magnOCcElular neurons art less tTJ-rf B3, 1755 (1986); B. Dreher. Y. Fukada, R W. Rodieck, / Physiei. reSpOnsrvB than parvoceUuiar neurons at equiluminance, buc are not unresponsive. (London) 258,433 (l97fi);T F Hicks. B. B. Lo, T . R. Vidyasagar. ibid. 337, P H- SchuJef and G. L . CoJby [VidtoRn 23, 1631 (I9B3)| and A C HurJbert, 183 (1983); P. Gouras, ifrtf. 199, 533 (1968); J J V 204. 407 (1969); F. N. K. Logothetis, E.ft..Charles, and P. H. Schiller [Sat. AVuiwi Aft* 13. 204 dcMariincria and P. Gouru, iM. 251, 167 (1975). (1987]] have reported thai cells in ihe parvo sysEem, not die magno sysrem, become unretporuive at eouiluminanet This 4fuC cJcarly remain* IO be resolved 9. A M. Drmngion and F- Lerinic,/. P^aal. (Lmdm) 357, 219 (1984). and is discussed in (37). 10. P_ Shaplry, E. Kaplan, R. Soodak, Nolurf fl^^ 292. 543 (1981). 11. The magno Sysiem dearly combines rhe inpunfrom,die red and the green COJ5CS.46. P. Cavanigh and Y. LCCIEIC, Invul. OpbtmJmel. find St> (suppl.) 26. 252 but the conthbudon from the blue tones Ls -n small thai it Is. not clear whether ihe (19BS) magna systemreceivesany inpur at all from ihe blue eontl. Also, magno cells are 47. F- T- GibsOn Tbi Peruptian of th Ytrual Worbt. L. Camudiael, Ed. (Houghton not completely broadband. In thai rheirreceptiverield surrounds arc often Mifflin, Bolton, 1950). weighted nward the red (7). 48. E. Rubin. Stra^lmde Figum (Glydendalska, Copenhagen, 1915) 49. H. B. Barlow, Pmc. R Set. Lamtm 212, 1 (1981). 12. D.HHubelandT.rJ.WH^L/Gj^./J™^. 1*6.421 (1972)if.S Lund. 4iV 1*7, *SS (1973); and R G. Hoodie, ibid. 159, 305 (197&) 50. V. 5. Rarruchajidran and 5. A. Anms, Pcteptm 14. 135 (I985f 51. F. Schumann, Z. PyJnl. 23, 1 [1900). IB- M. Wong. Riley, personal communicaDon. 14. A- E- ! • . ^ • .• S. P. HunT,*ndJ. Y Wu. tim.n (Le*dm} 292. 605 (1981)L 52. G. Kaniui. Rf. Piitakgia 49. 7 (1955). J. C. Horcon and D. H Hubel. Und., p. 7fi2. 53. B. L- Gregory, NaBm {lun&n) 238, 51 (1972). 15. ). C. HocTOH, PhJa. Trim. R. Sx Lmbr* 304, 199 (1994); E. HcGuinnas, C 54. S Liebmann, Pr/thd Panih. 9,300(1926) Maefjonak, M. Sereno, 1. AJimin. See. Ntumti Aton. 12. 130 (1986). 55. K. Kofflu. rWipla tfGnaitPiytltctyj (Harcoun Brase. New York* L935). 16. M.S. Livingstone and D.H. Hubel. Pnt. N/ui. A urf. JH. U • SA - 79,6098 (1982). 56- M. E. Chevreul, Di la Let du CanCrailt Simtdtani Sa Qndeun (Fmoi-Levraolt. 17. C R_ MkhaeL S« tJennbei Abrtr. 13, 2 (19B7); D. Eilzpauick, K [robi, I. T. Pans, 1839) Diamond,/. tJnami. 3.673 09SS). R B. H Tootell, 5. L. Hamilton. E. Swiika. 57. W. von Beaold, Thl Ttwry afCeM, S. R. Koehlot. Tririsl (Prang, Boston, 1876). L L D f Valois, /UTOJ. Qphthabwi Visual Sri. (suppl.) 26, 8 (1985). 58. H. E. J-M. van Tuijl, AaaPythei. 39,441 (1975); K. Nafcaama and S. 5hlmojo. personal comsnunjcaoon; V. S. Ramachandran. Nttm (Lawim) 328.645 (1987). IB. M. S. Lr-uigstone and D H. Hubel/ Nn-wn 4. 2830 (1934). 19. B. Dow,/.rJ(uny*H^, 37.927 (197*). 59. M. S. Livingstone. Set Am. 258, 78 (lanuary 1988). 20. P. Oourai and |. KrUger, ittJ. *Z, 850 (1979). 60. B_ W. Guillen/, Pn$. Brain Ra. 51, 403 (1979); S. M. Shemun, Prqg. Pycinbwl. 21 R. B. H TcoleJ tl J , Satnu 220, 737 (19B3) Ptrpial. PrrdvJ 1, 233 (1985) r
r
r
h
r
T
605
Reprinted with pemussion from Trendt In Nenmscunnes. Vol.15. No.l.pp. 2Q-3S.1»M 8 1992 Elscvici Sricm* Publishers Lid (UK)
Separate visual pathways for perception and action Melvyn A. Goodale and A David Milner Melvyn A. Goodale IsAccumulating
neuropsychological, electrophysiological and behavioural evidence suggests lhat the neural PiychtAtgy, substrates of visual perception may be quite distinct University ol Westernfrom those underlying the visual control of actions. In Ontario. London,other words, Ihe set of object descriptions that permit Ontario N6A5C2. identification and recognition may be computed indeCauda, and A, David pendently of the set of descriptions thai allow an Mifntrlsatthe Dept of Psychology, observer to shape the hand appropriately to pick op an UiiiYmtyofSl object. We propose that the ventral stream of projections cortex plays Andrews, St Andrewsfrom the striate cortex lo the infratemporal KYI69JU, UK. the major role in the perceptual identification of objects, while the dorsal stream projecting from the striate cortex to the posterior parietal region mediates the required sensorimotor transformations for visually guided actions directed at such objects.
affile Deplol
In an influential article that appeared in Science in 1969. Schneider' postulated an anatomical separation between the visual coding of the location of a stimulus and the identification of that stimulus. He attributed the coding of the location to the ancient retinotectal pathway, and the identification of the stimulus to the newer geniculostriate system; this distinction represented a significant departure from earlier monolithic descriptions of visual function. However, the notion of 'localization' failed to distinguish between the many different patterns of behaviour that vary with the spatial location of visual stimuli, only some of which turn out to rely on tectal mechanisms . Nevertheless, even though Schneider's original proposal is no longer generally accepted, his distinction belween object identification and spatial localization, between 'what' and 'where', has persisted in visual neuroscience. 2-1
Two cortical visual systems In 1982, for example, Ungerleider and Mishkin concluded that 'appreciation of an object's qualities and of its spatial location depends on the processing of different kinds of visual information in the inferior temporal and posterior parietal cortex, respectively.' They marshalled evidence from a number of electrophysiological, anatomical and behavioural studies suggesting that these two areas receive independent sets of projections from the striate cortex. They distinguished between a 'ventral stream' of projections that eventually reaches the inferotemporal cortex, and a 'dorsal stream' that terminates finally in the posterior parietal region. The proposed functions of these two streams were inferred Largely from behavioural evidence derived from lesion studies. They noted that monkeys with lesions of the inferotemporal cortex were profoundly impaired in visual pattern discrimination and recognition , bot less impaired in solving 'landmark' tasks, in which the location of a visual cue determines which of two alternative locations is rewarded. Quite the opposite pattern of results was observed in monkeys with posterior parietal lesions' ,
for identifying objects, while the posterior parietal lesions interfered with neural mechanisms underlying spatial perception. Thus, within [he visual domain, they made much the same functional distinction between identification and localization as Schneider, but mapped it onto the diverging ventral and dorsal streams of output from the striate cortex. Since 1982, there has been an explosion of information aboul the anatomy and electrophysiology of cortical visual areas and. indeed, the connectional anatomy among these various areas largely confirms the existence of the two broad "streams' of projections proposed by Ungerleider and Mishkin (see Fig. 10-11
It has recently been suggested" lhat these two streams can be traced back to the two main cytological subdivisions of retinal ganglion cells: one of these two subdivisions terminates selectively in the parvoceUuiar layer, while the other terminates in the magnocellular layer of the lateral geniculate nucleus ILGN) ' . Certainly, these 'parvo' and 'magno' subdivisions remain relatively segregated at the level of the primary visual cortex (VI) and in the adjacent visual area V2. They also appear to predominate, respectively, the innervation of area V4 and the middle temporal area (MT). which in turn provide the major visual inputs to the inferotemporal and posterior parietal cortex, respectively. However, it is becoming increasingly clear that the separation between magno and parvo information in the cortex is not as distinct as initially thought. For example, there is recent evidence for a parvo input into a subset of MT neurones as well as for a large contribution from the magno pathway to V4 neurones ^ and to the 'blobs' in V I (Ref. 19). In short, it now appears that the dorsal and the ventral streams each receive inputs from both the magno and the parvo pathways. 11-
6
17
1
5
6
Two visuomotor systems: 'what' versus 'how' Our alternative perspective on modularity in the cortical visual sysiem is to place less emphasis on input distinctions (e.g. object location versus object qualities) and to take more account of output requirements . Il seems plausible from a functional standpoint that separate processing modules would nave evolved to mediate the different uses to which vision can be put. This principle is already generally accepted in relation to 'automatic' types of behaviour such as saccadic eye movements , and it is possible that it could be extended to other systems for a range of behavioural skills such as visually guided reaching and grasping, in which close coordination is required between movements of the fingers, hands, upper limbs, head and eyes. 2121
22
It is also our contention that the inputs and transformations required by these skilled visuomotor acts differ in important respects from those leading to what is generally understood as 'visual perception.' Indeed, as has been argued elsewhere, the ftinctional So, according to Ungerleider and Mishkin's 1932 modules supporting perceptual experience of the version of the model of two visual systems, the world may have evolved much more recently than inferotemporal lesions disrupted circuitry specialized those controlling actions within it . In this article, it is -9
21
606
21
proposed that this distinction ('what' versus 'how') rather than the distinction between object vision and spatial vision ('what' versus where') - captures more appropriately the functional dichotomy between the ventral and dorsal projections. Dissociation between prehension and apprehension Neuropsychological studies of patients with damage to one projection system but nol the other have also been cited in support ol the model proposed by Ungerleider and Mishkur' '. Patients with final agnosia following brain damage that includes, for example, the occipitotemporal region, are often unable to recognize or describe common objects, faces, pictures, or abstract designs, even though they can navigate through the everyday world - at least at a local level - with considerable skill-'. Conversely, patients suffering from optic ataxia following damage to the posterior parietal region are unable to reach accurately towards visual targets that they have no difficulty recognizing . Such observations certainly appear to provide support in humans for an occipitotemporal system mediating object vision but nol spatial vision, and a parietal system mediating spatial vision but not object vision. 2
25
Closer examination of the behaviour of such patients, however, leads to a different conclusion. Patients with opiic ataxia nol only have difficulty reaching in the right duection, bul also in positioning their fingers or adjusting the orientation of their hand when reaching toward an object that can be oriented at different angles . Such patients may also have Fig. 1. The 1962 version ot Ungerleider and Mishkin's* model ol two visual trouble adjusting their grasp lo reflect the size of the systems is illustrated in the small diagram ot the monkey brain inset into the object they are asked to pick up. larger bo< diagram. In the original model. V1 it shown sending a dorsal stream 25
Visualfy guided grasping was recently studied in a patient who had recovered from Balint's syndrome, in which bilateral parietal damage causes profound disorders of spatial attention, gaze and visually guided reaching . While this patient had no difficulty in recognizing line drawings of common objects, her ability lo pick up such objects remained quite impaired. For example, when she reached out for a small wooden block that varied in size from trial to trial, there was little relationship between the magnitude of the aperture between her index finger and thumb and the size of the block as the movement unfolded. Not only did she fail to show normal scaling of the grasping movement: she also made a large number of adjustments in her grasp as she closed in on the object adjustments rarely observed in normal subjects. Such studies suggest that damage lo the panetal lobe can impair the ability of patients to use inlormation about the size, shape and orientation of an object to control the hand and fingers during a grasping movement, even though this same information can still be used lo identify and describe the objects. Clearly, a 'disorder of spatial vision' fails to capture this range of visuomolor impairments. 25
ol protections to the posterior panetal corten i°G), and a ventral stream ot protections to the inferotemporal cone* iTf). The boa diagram illustrates one otthe most recent versions ot the mterconnectivity ot the visual cortical areas, showing that they can still be broadly segregated into dorsal and ventral streams. However, there is crosstalk between the different areas in the two streams, and there may be a third branch ot processing projecting into the rostral superior temporal sulcus tSTSi that is intimately connected with both the dorsal and ventral streams. (This is illustrated in both the brain and box diagrams.) Thus, the proposed segregation of input that charactenzed the dorsal and ventral streams in Ihe original model is not nearly as clear cut as
once was thought. (Modified, with permission, from Ref. 11.)
poorly on a task in which visual guidance was needed to learn the correct roule through a small ten-choice maze by moving a hand-held stylus . However, he was quile unimpaired on a locomotor maze task in which he was required lo move his whole body through space when working from a two-dimensional visual plan. Moreover, he had no difficulty in recalling a complex geometrical pattern, or in carrying out a task involving short-term spatial memory *. Such dissociations between performance on different 'spatial' tasks show that after parietal damage spatial information may still be processed quite well There are. of course, other kinds of visuospatial for some purposes, but not for others. Of course, the disorders, many of which are associated with parietal fact lhat visuospatial deficits can be fractionated in lobe damage, while others are associated with tem- humans does not exclude combinations of such poral lobe lesions °. Unfortunately, we lack detailed impairments occurring after large lesions, nor would it analyses of the possible specificity of most such exclude possible selective input disorders occurring disorders: in many, the deficit may be restricted lo after smaller deafferentation lesions close to where particular behavioural tasks. For example, a recently the dorsal stream begins. described patient wilh a panelal injury performed 23
2
27 2
607
22 •dimensions o( the objects she was about to pick up even though she appeared to be unable to 'perceive' those dimensions. A similar dissociation was seen in her responses to the orientation of stimuli. Thus, when presented with a large slot that could be placed in one of a number of different orientations, she showed great difficulty in indicating the orientation eiEher verbally or manually (i.e. by rotating her hand or a hand-held card). Nevertheless* she was as good as normal subjects at reaching ouE and placing her hand or the card into the slot, turning her hand appropriately from the very onset of the movemenC - . h
30 31
5.0 4.5 4 0 3 5 3.0 2.5 Width of obiact (cm)
5 0 4.5 4 0 3.5 3.0 £.5 Width ol objecl (cm)
5.0 4.5 4.0 3.5 3.0 2-5 Width of object (cm)
These disparate neuropsychological observations lead us to propose that the visual projection system to the human parietal cortex provides action-reievant information about the structural characteristics and orientation of objects, and not just about their position. On the other hand, projections to the temporal lobe may furnish our visual perceptual experience, and it is these that we postulate Co be severely damaged in DF.
Fig. 2. In both (A) the manual matching task and (B) the grasping task, five white plaques (each with an overall area of 25 cm on Ibe top surface, but witb dimensions ranging from 5 x 5cm to 2.5 x 10 cm) were presented, one at a time, at a viewing distance of approximately 45 cm. Diodes emitting infrared light (IREDs) were attached to the tips of tbe index finger and thumb of the right hand and were tracked with two infrared-sensitive cameras and stored on a WATSMARTcomputer (Northerrt Digital inc., Watedoo, Canada). The three-dimensional position of the IREDs and the changing distance between them were later reconstructed off line. (A) In the manual matching task, DFand two control subjects were instructed to indicale the width of each plaque over a series of randomly ordered thals by separating the index finger and thumb of their right hand. In DF, unlike the controls (CG and CJ), the aperture between the finger and thumb was not systematically related to the width of the target. DF also showed considerable trial to trial variability. (B) In contrast, when they were instructed to reach out and pick up each plaque, DF's performance was indistinguishable from that of the control subject*. The maximum aperture between the index finger and thumb, which was achieved well before contact, was systematically related to the width Dorsal and ventral systems in of the plaques in both DF and the two control subjects. In interpreting all these graphs, it is the the monkey slope of the function that is important rather than the absolute values plotted, s.nce the placement How well do electrophysiological of the IREDs and the size of the band and fingers varied somewhat from subject to subject. Bars studies of the two projection sys2
represent means ± SE. (Modified, with perniission, from Ref. 31.>
Complications also arise on the opposite side of the equation (i.e. in relation to the ventral stream), when the behaviour of patients with visual agnosia is studied in detail. The visual behaviour of one patient (DF) who developed a profound visual-form agnosia following carbon monoxide poisoning was recently studied. Although MRI revealed diffuse brain damage consistent with anoxia, most of the damage in the cortical visual areas was evident in areas 18 and 19. with area 17 apparently remaining largely intact Despite her profound inability lo recognize the size, shape and orientation of visual objects. DF showed strikingly accurate guidance of hand and finger movements directed at the very same objects ™. Thus, when she was presented with a pair of rectangular blocks of the same or different dimensions, she was unable to distinguish between them. When she was asked to indicate the width of a single block by means of her index finger and thumb, her matches bore no relationship to the dimensions of the object and showed considerable trial to trial variability (Fig. 2A). However, when she was asked simply to reach out and pick up the block, the aperture between her index finger and thumb changed systematically with the width of the object, just as in normal subjects (Fig. 2B!. In other words, DF scaled her grip to the 3
608
tems in the visual cortex of the monkey support the distinction we are making? While any correlations between human neuropsychology and monkey neurophysiology should only be made with caution, it is likely lhat humans share many features of visual processing with our primate relatives - particularly with the Old World monkeys in which most of the electrophysiology has been carried out. Furthermore, lesion studies of the two projection systems in the monkey should show parallels with the results of work done on human patients. ft was noted earlier that although there are differences in the major retinal origins of inputs to the dorsal and ventral systems in the monkey brain, there is subsequently a good deal of pooling of information. Moreover, there are convergent similarities in what is extracted within the two systems. For example, both orientation and disparity selectivity are present in neurones in both the magno and parvo systems within cortical areas VI and V2 (Ref. 15). Nevertheless, there are special features In the properties of individual neurones in the posterior parietal cortex (and in its major input areas V3A and MT) that are not found in the ventral system. The most striking feature of neurones in the posterior parietal region is not their spatial selectivity (indeed, like those of inferotemporal cells, their receptive fields are typically large), but rather the fact that their
25
responses depend greatly on the concurrent be- fore, il is unsurprising that monkeys with inferolemhaviour of the animal with respect to the stimulus. potal lesions have profound deficits in visual recogSeparate subsets of cells in the posterior parietal nition; however, as noted by Pribram' , they remain cortex have been shown to be implicated in visual highly adept at the visually demanding skill of catching fixation, pursuit and saccadic eye movements, eye- (EM! hand coordination, and visually guided reaching A further peculiarity of many visual cells in the movements . Many cells in the posterior parietal temporal cortex is that they continue to maintain their region have gaze-dependent responses; i.e. where selective responsiveness over a wide range of size, the animal is looking determines the response ampli- colour, optical and viewpoint transformations of the tude of the cell (although nol the retinal location of its object" . Such cells, far from providing Ihe momenreceptive field) . In reviewing these studies. Ander- tary udorrnation necessary for guiding action, specifisen emphasizes thai most neurones in this area cally ignore such changing details. Consistent with 'exhibit both sensory-related and movement-related this, behavioural studies have shown that by lesioning activity.' In a particularly interesting recent develop- the inferotemporal cortex (but not the posterior ment, Taira et al. * have shown thai some parietal parietal cortex), a monkey is less able to generalize its cells are sensitive to those visual qualities of an object recognition of three-dimensional shape across viewing that determine the posture of the hand and fingers conditions" ". during a grasping movement. They studied neurones selectively associated with hand movements made by the monkey in reaching and picking up solid objects. Visual and attentions! requirements for Many of these cells were selective for the visual perception and action appearance of the object that was to be manipulated, As DeYoe and Van Essen' have suggested, including its size and in several cases its orientation. 'parietal and temporal lobes could both be involved in shape analysis but associated with different compuThe posterior parietal cortex may receive such tational strategies.' For the purposes of identification, form uiforrriation from one or both of the areas V3 or learning and distal (e.g. soda!) transactions, visual V4. bolh of which project to area M T . Other visual coding often (though not always" * ) needs to be inputs pass through area MT and the adjacent medial 'object-centred'; i.e. constancies of shape, size, superior temporal (MST) area, both of which contain colour, lighiness, and location need to be maintained cells variously selective for object motion in different across different viewing conditions. The above evidirections, including rotation and motion in depth . dence from behavioural and physiological studies Thus, the posterior parietal cortex appears to receive supports the view that the ventral stream ol prothe necessary inpuls for continually updating the cessing plays an important role in the computation of monkey's knowledge of the disposition and structural such object-specific descriptions, fn contrast, action qualities of objects in its three-dimensional ego-space. upon the object requires thai the location of the object Also, many motion-sensitive cells in the posterior and its particular disposition and motion with respect parietal cortex itself appear to be well suited for the to the observer is encoded. For this purpose, coding visual monitoring of limb position during reaching of shape would need to be largely 'viewer-centred'* , behaviour ; in contrast, motion-sensitive cells in the with the egocentric coordinates of the surface of the temporal lobe have been reported not to respond Io object or its contours being computed each time the such self-produced visual motion . As for the output action occurs. We predict lhat shape-encoding cells in pathways, the posterior parietal region is strongly the dorsal system should predominantly have this linked with those pre-motor regions of the frontal property. Nevertheless, certain constancies, such as cortex direcdy implicated in ocular control . size, would be necessary for accurate scaling of grasp reaching movements of the limb , and grasping aperture, and it might therefore be expecled that the visual properties of the manipulation cells found by actions of the hand and fingers . Taira et al. in Ihe posterior parietal region would Thus, the parietal cortex is strategically placed to have this property. serve a mediating role in the visual guidance and integration of prehensile and other skilled actions (see It is often suggested that the neuronal properties of Ref. 40 for a detailed account of Ihis argument). The the posterior parietal cortex qualify it as the prime results of behavioural analyses of monkeys with mediator of visuospatial attention . Certainly, many posterior parietal damage support this further. Like cells (e.g. in area 7a) are modulated by switches of patients with optic ataxia, such animals fail to reach attention to different parts of the visual field . correctly for visual targets", and they also have (Indeed, the 'landmark' disorder that follows posterior difficulty in shaping and orienting their hands when parietal damage in monkeys may be primarily due to a attempting to retrieve food' * . Their reaching im- failure to atlend or orient rather than a failure to pairment is. therefore, one symptom of a wider localize * .) However, it is now known that attenvisuomotor disorder, and most of the deficits that tional modulation occurs in neurones in many parts of have been reported on 'maze' tasks following pos- the cortex, including area V4 and the inferotemporal terior parietal damage may also be visuomotor in region within the ventral stream ** This might nature** . explain the occurrence of landmark deficits after inferotemporal as well as posterior parietal damage . Nonetheless, neurones in the dorsal stream do not !n general terms, attention needs to be switched to show the high-resolution selectivity characteristic of neurones in the inferotemporal cortex, which are particular locations and objects whenever they are the or for strikingly sensitive lo form, pattern and colour . In targets either for intended action this and in nei^bouring temporal lobe areas, some identification *. In either case, this selection seems cells respond selectively to faces, to hands, or to the typically to be spatially based. Thus, human subjects appearance of particular actions in others". There- performing manual aiming movements have a 5
32
33
3
-
5
6
35
33
36
31
33-3a
39
39
31
50
51
2
3
9-
0-52
53-
0
76
51,55
10
5
609
24 predilection lo attend to visual stimuli that occur with- between the two systems (see Fig. 1). In addition, in the 'action space' of Ihe hand " . In this instance there are many polysensory neurons in these areas, [he attentional facilitation might be mediated by mech- so that not only visual but also cross-modal interaction anisms within the dorsal projection system; in other between these networks may be possible. This may instances il is probably mediated by the ventral provide some of the integration needed for the system. Indeed, the focus of lesions causing the essential unity and cohesion of mosl of our perceptual human attentional disorder of 'unilateral neglect' is experience and behaviour, although overall control of parietotemporal [unlike Ihe superior parietal focus for awareness may ultimately be the responsibility of optic ataxia "'), as is Ihe focus for object constancy superordinate structures in the frontal cortex '. impairments . We conclude Lhat spatial attention is Nevertheless, it is feasible to maintain the hypothesis physiologically non-unitary , and may be as much that a necessary condition for conscious visual experiassociated with the ventral system as with the dorsal. ence is thai the ventral system be activated. 5
1
6
2
57
55
A speculation about awareness The evidence from the brain-damaged patient DF described earlier suggests thai the two cortical pathways may be differentiated with respect to their access to consciousness. DF certainly appears to have no conscious perception of the orientation or dimensions of objects, although she can pick them up with remarkable adeptness. It may be that information can be processed in the dorsal system without reaching consciousness, and that this prevents interference with the perceptual constancies intrinsic to many operations within the ventral system that do result in awareness. Intrusions of viewer-centred information could disrupt the continuity of object identities across changing viewpoints and illumination conditions. If this argument is correct, then there should be occasions when normal subjects are unaware of changes in the visual array to which their motor system is expertly adjusting. An example of such a dissociation has been reported in a study on eye-hand coordination during visually guided aiming . Subjects were unable to report, even in forced-choice testing, whether or not a larget had changed position during a saccadic eye movement, although correction saccades and manual aiming movements directed at the larget showed near-perfect adjustments for the unpredictable target shift. In other words, an illusory perceptual constancy of target position was maintained in the face of large amendments in visuomotor control. In another recent example, it has been repotted that the compelling illusion of slowed motion of a moving coloured object that is experienced at equuuminance does not prevent accurate ocular pursuit under the same conditions (see Ref. to Lisberger and Movshon. cited in Ref. 59). Such observations may illustrate the independent functioning ol the ventral and dorsal systems in normal humans. 59
We do not, however, wish to claim that the division of labour we are proposing is an absolute one. In particular, the above suggestion does not imply that visual inputs are necessarily blocked from awareness during visuomotor acts, although that may be a useful option to have available. Rather, we assume that the two systems will often be simultaneously activated (with somewhal different visual information!, thereby providing visual experience during skilled action. Indeed, the two systems appear to engage in direct crosstalk; for example, the posterior parietal and inferotemporal cortex themselves interconnect and bolh in turn project to areas in the superior temporal sulcus"" There, cells that are highly form selective tie close to others that have motion specificity *, thus providing scope for cooperation 33 6 0
3
4
Concluding remarks Despite the interactions between the dorsal and ventral systems, the converging lines of evidence reviewed above indicate thai each stream uses visual information about objects and events in the world in different ways. These differences are largely a reflection of the specific transformations of input required by perception and action. Functional modularity in cortical visual systems, we believe, extends from input right through to output. Selected references 1 Schneider. C . £ (1969) Science 163. 895-902 2 Ingle. D. I 11982) In Analysis ol Visual Behavior tingle. D- J., Coodal!. M. A. and Mansfield, ft. J. W„ eds), pp 67-109, MIT Press 5 Goodale. M A. and Munson, P.. (1975) Brain des. S8. 243-255 4 Goodale. M. A. and Milner, A. L>. (1982) In Analysis ol Visual Behavior tingle, 0 1, Goodale. M A. and Mansfield, S.J W„ ids), pp 263-299. MIT Press 5 Ungerleider. t. 0. and Mishkin. M. (1932) in Analysis ol Visual Behavior (Ingle, 0. J., Goodale. M- A- and Mansfield, Ft. J. W., ed!), pp. 549-536, MiT Press 6 Cross, C. G- (1973) Prog. Physiol. Psychol 5, 77-123 7 Pohl, W (1973) / Comp. Physiol. Psychol. 82, 227-239 9 Ungerleider. L 0, and Brady, B. A. (197?) hp. Neurol. 56. 265-280 9 Milner, A. 0., Ockleford. E. M. and Dewar, W. (1977) Corte, 13, 350-360 10 Desimone, a. and Ungedeider, L G. (19B9) in Handbook ot Neuropsychology. Vol. 2 (Boiler. F and Grafman, I., eds), pp. 267-299, Elsevier 11 Soussaoud, 0 , Ungedeider, L G. and Desimone, Ft. (1990) J. Comp. Neurol. 296, 462-495 12 Morel. A. and Bullier, J. (1990) Visual Neurosci. 4. 555-578 13 Baizer, J, S , ungerleider, t. G. and Desimone. P. (1991) I. Neurosci. 11, 168-190 14 Livingstone, M, and Hiibel, D. ( 1 9 9 8 ) Science !40, 740-749 15 DeVoe. E A. and Van Essen, D C (1988) Trends Neurosci. I I , 219-226 16 Schiller, P. H. and logoltietis, N. K (1990) Trends Neurosci 13, 392-398 17 Maunsell, I. H. B„ Mealy, T. A. and De Pnest, D. D. (1990) J. Neurosci. 10. 3323-3334 18 Nealey, T. A. and Maunsell, J. H. P.. (1991) Invest. Ophthalmol. Visual So. 32 (Sjppl.) 1117 19 Ferrera. V. P., Nealey, T. A. and Maunsell. J. H ft. (1991} Jmrest. Ophthalmol, visual Sci. 32 (Suppl.) 111? 20 Goodale. M. A. (19B3) In Behavioral Approaches to Brain flesearcMftobinson, T. Ei, ed.), pp. 41-61, Ox+ord University Press 21 Goodale, M. A. (1988) in Computational Processes in Human Vision: An Interdisciplinary Perspective (Pylyshyn, Z , ed.). pp. 262-285. Ablai 22 Sparks. D. L. and May, L. E. (1990) Annu. Rev. Neurosci. 13, 309-336 23 Newcombe, F.. ftatdiff, G- and Damasro, H. (1987) Neuropsychotogia 25. 149-161 24 Farah, M. (1990) Visual Agnosia. MIT Press 25 Perenin, M-T. and Vighetto, A. (19B8) Brain 111, 643-674 26 Jakobson. t. S , Archibald, V. M , Carey. D. and Goodale. M. A. (1991) NeuropsychOlOgiaM. B03-B09 27 Milner, 8. (1965) Neuropsychologia 3. 317-338
610
15 Smith, M. L and Milner, 8. 11989) Hfuropsychologia 27. 7I-B1 29 Ettlinger. 0. (1990) Co/let 26, 319-341 30 Milner, A. 0 etal. (1991) Brain 114, 405-128 31 Goodale, M. A , Milner, A. D , Jakobson, L. S. and Carey, D P (1991) Nature 349, 154-156 32 Andersen, R. A. (1987) in Higher functions of tbe Brain. Part 2 (The Nervous System, Vol. V. Handbook of Physiology. Section 1) (Mountcastle, V. B , Plum, F. and Geiger, S. Ft., eds), pp. 483-518, Amencan physiological Association 33 Andersen, ft. A., Asanuma, C Essick, G. and Siegel, ft. M. (1990) J. Comp. Heutol. 296, 65-113 34 Taira, M., Mine, S , Georgopoulos, A. P , Murala, A. and Sakata, H. (1990) E>p. Brain Pes 83, 29-36 35 FeHemaivD.J and Van Essen, D. C. (1987)7. Neurophysiol. 57, 889-920 36 Mountcastle, V. B„ Matter, B. C , Steinmen. M. A. and Dutfy, C J. (1984) m dynamic Aspects of Neocctical function (Edelman, G. M . Gall, W. E. and Cowan. W. M., eds). pp. 159-193, Wiley 37 Penetl, 0. I., Mlstlin, A. J., Hacries, M. H. and Chilly, A. J. (1990) in Vision and Action; The Control ot Grasping (Goodale, M. A., ed.), pp. 163-180, Able. 38 Canada, C . and Goldman-Rakic, P. S. (1989) J Comp. Neurol. 287, 422-445 39 Gentilucci. IA. and Rizzolatti. G. (1990) in Vision and Action: The Control ot Grasping (Goodale. M. A., ed). pp. 147-162. Ablei 40 Milner. A. D. and Goodale, M A. Prog. Brain Pes. (in press) 41 Bates, I. A. V. and Ettlinger, G. (1960) Arch. Neurol. 3, 177-192 42 Faugier-Grimaud, 5., Frenois, C . and Stein, D. G. (1978) Neuropsychology 16, 1 5 1 - 1 6 8 J3 Hiaima, ft. and Kuypers. H. G. I. M. (1975) Brain 98. 239-260
2S
44 Perrett. D. I.. Mistlin. A. J. and Chitty, A. J. ( 1 9 8 7 ) Trends Neuiosci. 1 0 , 358-364 45 Prjbram. K. H. (1967) in Brain Function and Learning (Lindsley. D. 8. and Lumsdaine, A. A., eds), pp. 7 9 - 1 2 2 , University ol California Press 46 Perrett. O. I. etal (1991) tip. Brain Pes. 8 6 . 1 5 9 - 1 7 3 47 Humphrey, N. K and Weiskranti, L. (1969) Quart I. Sip Psychol. 2 1 . 225-23B 48 Weiskrantz. 1 and Saunders. R. C. (1984) drain 107, 1033-1072 49 Marr. D. (1982) Vision. Freeman 50 Goldberg. M. E. and Colby. C. L. (1989) in Handdoot of Neuropsychology (Vol. 2) (Boiler, F. and Gralman. J., eds). pp. 301-315. Elsevier 51 Bushnell. M C , Goldberg. M. E. and Kobinson. D. 1. (19B1) J. Neurophysiol. 46, 755-772 52 lawler. K. A. and Cowey. A. (1987) £*•. Brain Pes 65. 695-69B 53 Fischer. 8. and Booh. P.. (1981)f.p. flraiij Pes 44. 129-137 54 Moran. I. and Desimone. P.. (1985) Science 229. 7 8 2 - 7 B 0 55 Rizzolatti, G . . Gentilucri. M. and Matelli. M. (1985) in Attention and Performance XI (Posner, M. I. and Marin. O. S. M , eds), pp. 251-265, Eribaum 56 Tipper, S., Lortie, C. and Baylis, G. ) . Cup. Psychol. Human Percept. Perform, (in press) 57 Warrington, E. K. and Taylor, A. M. ( 1 9 7 3 ) Cortei 9, 152-164 58 Goodale, M. A , Pelisson, D and Prablanc, C. ( 1 9 8 6 ) Nature 320, 7*8-750 59 Sejnowski, T. I t>991) Nature 352. 669-670 60 Canada, C. and Goldman-Rakic, P. S. (19B9) J. Comp. Neurol. 287. 393-171 61 Posner, M. I. and Bothbart. M. «. (1991) in Tire Neuropsychology of Consciousness (Milner, A. D. and ftugg, M. D , eds). pp. 91-111, Academic Press
611
Acknowledgements The authors are grateful to D. Carey, t.lattobsonand 0. Penett tor their comments on a draft ol this paper.
Reprinted with permission from Science. Vol. 242. pp. 736-741.198a © 1988 The American Association for [f-c Advancenwni of Science
Behavioral Neurophysiology: Insights into Seeing and Grasping S T E V E N P . W I S E AND R O B E R T D E S I M O N E
One marvels at a batter's ability to hit a baseball traveling at 150 kilometer* pet hour or a monkey's skill in snatching a flying insect. Indeed,, the ability of many animals to reach out, grasp, and manipulate objects is a feat of biological engineering unmatched by even state-of-the-art robots. But how are the objects of our attention chosen and how are the eyes and hands directed to it? Recent progress in behavioral neurophysiology has clarified some of the brain mechanisms at work.
eyes steady. A cue, such as a small dot, comes on to indicate where attention should be directed, without making eye movements. I f an arrav o f different stimuli is then briefly presented on the screen, the subject will best remember rhe stimulus at the cued location. I n a different experiment, when a "go" stimulus appears on the screen, the subject must respond as quickly as possible by pressing a key. I f a prior cue directs the subject's attention to the go-stimulus Location, movements begin sooner than if attention is drawn to a different location. T h u s , attention serves both to control access to memory and to facilitate behavioral responses (JJ,
I
Nvurvphyuoloxy trfatiattiett Research in primate behavioral neurophysiology has shown that attention can have powerful effects on the responses o f individual neurons. I n some cases attention leads to an enhancement of neuronal responses to attended items; in others it leads to suppression o f responses to unattended items. W e discuss some effects of attention i n the visual system, but analogous effects occur in other sensory systems as well (4).
N OUfl EVERYDAY ACTIVITIES WE KEGULATU-Y EXAMIEJE OB¬ jects with our eves, hands, and minds. In. this article, we survey some recent progress in understanding how rhe central nervous sysiem directs attention io Dne among several objects, moves the eyes to an object of interest* and generates the limb movements necessary lo grasp and manipulate it
Attention In a typical visual scene a myriad of shapes, colors, and textures impinge on the retina, fust as we cannot make an eye or limb movement to all objects at once so also we cannot fully arrend to more than one or rwo objects at a time. M u c h unwanted information must be filtered out by the central nervous system. T h e first stage in this filtering and selection process is largely automatic and involves distinguishing figures from their background (figureground separation}. A red apple on a green field or a fly moving against a wall are examples of objects that "pop our" from their backgrounds on the basis of differences in color, shape* texture, distance, or motion (J). T h e neural mechanisms underlying figureground separation most likely begin with simple lateral inhibition in the rerina, which serves to reduce neuronal responses to regions o f uniform luminance. These processes continue as visual information is relayed through the thalamus to the cerebral c o n e s , where neurons are especially responsive to stimuli that differ from their backgrounds in form* color, or motion, just the features that produce pop-out phenomena perceptually [2).
T h e primary visual corte.v ( V I ) , also known as the striate cortex, appears to be the source of two major cortical pathway? (5). Each pathway relays information through several additional visual conical fields { F i g . 1}, collectively known as the extrastriatc cortex. One pathway is directed into the temporal cortex and appears to be critical for object recognition. T h e other pathway is directed into the posterior parietal cortex ( P P ) and appears to be important for visuomotor performance* visuospatial perception, and spatial attention ,
h
Even alter figure-ground separation, a processed visual scene will still be very complex, typically containing many different figures. T h u s , a second stage of object or feature selection is necessary. Unlike the mechanisms for figure-ground extraction, which operate in parallel across an entire visual scene* this second stage, attention, is thought to operate serially on one or two objects ar a time. In studies o f attention, a subject is rvpicallv instructed about where to focus attention pending the subsequent presentation of visual stimuli. F o r example, the subject looks at a screen, holding the
PP and \he sonstoi of tpatial uttiMffatt- Neurons in P P appear well suited for a role in spatial perception and attention rather than form Or object recognition^ changes in their discharge rates reflect more the location o f a stimulus and whether it is attended ro than the features of a stimulus, sueh as its shape or color. PP neurons have "receptive fields" in rhe sense that a given neuron will respond to stimuli only within a certain restricted portion o f the visual field (6). i\ stimulus appearing within the receptive held o f a P P neuron, if it triggers a behavioral response, will elicit a much larger neuronal response than if the monkey ignores it, in some cases, ignored stimuli do not cause any neuronal response at all. F o r most, perhaps all* PP cells* the nature o f the behavioral response does not matter: eye or aim movements toward the stimulus, or even generalized responses triggered by but not directed toward the stimulus, are associated with equally potent enhancements o f the neuronal response. These findings suggest that the enhancement seen in P P reflects the monkey's arrention to a specific location in space rather than any aspect o f the motor response; therefore, it may be preferable to think o f these cells as having an "attention field" rather than a receptive field.
5. P. Wiw n in the LaborarQiv of NniiOphvuOloEY. Nmnml Irmiturc of Meruit Hflllri, P.O. Bo* 139. ItaEcsvLllt, MD 2083? H. D'uunonc it in dit Labora«orv of NturornvtholoBv, NJIIOHJI [qgjcutv of Mental Hulcfl, Building 9, Room Bnhoda, MD70B92.
612
737 FJQ 1, Lateral view of macaque brain showing lot actons of some visual and motor areas. Abbrcviadoru; FEF, ttanai eyefietdiIT, mfcrolemporal cortex; SEF* Supplcmcntary eyefield.PP. posterior parietal COTM*. Ml, primary motor cortex* VJ, primary visual {or striare} eoncx; and V4, fourth visual area.
SEF,
stimuli only. By contrast, attentional effects cover a larger spatial .*£C~?~*"^r~^~^^ range in IT, where receptive fields are much larger* in some cases as yf^Tp&v^J'n^^-O^ large as the entire visual field. These results suggest at least two A T _ ^ T W ^ \ , stages offilteringunwanted information: thefirststage works over a / \ f y, W \ / \ - \ ^ ) small spatial range in V4 and the second stage works over a much | J \ \ larger range in IT. No such effects of attention have yet been found _ A in either V I or V2, suggesting that V4 is the first area along the ""^v ^S* J temporal pathway in which attention profoundly influences visual V~\ ' J information processing. Although attention appears ro filter unwanted information by Since PP lesions in humans often impair rhe ability to disengage causing suppression of V4 and IT neuronal responses to ignored anention from one location and shift it to another [7), it is tempting stimuli* other results indicate that anention can also cause enhanceto suggest lhat its neurons play a causal role in shifting attention. In ment of responses to stimuli when the amount of attention, or this view, the responses of a P P neuron reflect the mechanism of cognitive effort, devoted to an already attended stimulus is inredirecting attention to its "receptive field. One argument against creased. For example, extrastriatc neurons show different degrees of this notion is that the receptivefieldsof parietal neurons are very responsiveness depending on whether the monkev is idle, engaged large* in some cases a quadrant or more of the visualfield.However, in an easy detection task, or engaged in a difficult discrimination task as we argue later, broad spatial tuning by individual neurons need (11). Further, V4 neurons respond more vigorously and are more not lead to poor spatial resolution, if the neurons are part of an tightly tuned to the orientation and color of stimuli when a monkey appropriate neuronal networkuses the stimuli in a difficult discrimination task than when it uses the same stimuli in an easier version of the same lask [12). This Effettt of 'attention on the torikat syaem Joi obucl remgniticn. The second major visual pathway in the primate cortex, the one that is enhanced processing does not appear to be due to nonspecific necessary for object recognition* is directed into the temporal rather factors, such as arousal, because neuronal responses to irrelevant than the parietal lobe (Fig. 1}. Information in this temporal pathway stimuli presented concurrently with the attended stimuli are not is transmitted from V I , through areas V2, V3, and V4 into die enhanced. Similarly, behavioral data indicate the monkey's discrimiinferior temporal cone* {IT} (S). Unlike neurons in the parietal native abilities improve when it is engaged in a more difficult task. pathway, neurons in the temporal pathway are concerned wirh the Thus, when an animal is challenged by a difficult task, it appears to concentrate its attention, of which rwo neural correlates are enfeatures of objects* such as colot* orientation, texture, and shape. Recently, it has been shown that attention affects visual process- hanced neuronal responsiveness and sharpened selectivity. ing at the Later stages of this pathway, although not quite in the same way as in PP. Neurons in areas V4 and IT have receptivefieldsso large that many stimuli will typically fall within them. One might expect that the responses of such cells would reflect the properties of From Attention to Action all stimuli inside their receptivefields.However, it has been found After attention brings an object to the (bcus of the brain's that when a monkey restricts its arrenrion to one location within a information processing system, the next step in exploring an object V4 or IT cell's receptive field, the response of the cell is determined is usually to direct the eyes to the item of interest, a process called primarily by the stimulus at the attended location, almost as if the foveation or a gaze shift (13). Indeed, more than eye movements are receptive field "shrinks" around the attended stimulus (9). For typically performed to "look at" an attended item. Coordinated example, consider a cell that responds strongtv ro red stimuli and not movements of the eyes, head, and body bring fovcat scrutiny to bear to green when only a single stimulus appears inside its receptive an the ob|ect of interest, allowing the greatest possible amount of field. If red and a green stimulus appear simultaneously at different visual information to be gathered and promoting accurate reaching locations within the field, and the animal focuses its attention on only the red one, the cell will respond strongly. If, however, the GOJF tonuol. Several cortical and subcortical structures play animal attends to only the gteen stimulus, the cell will respond important roles in the control of eye movements, and we will not weakly or not al all to the red stimulus* even though the red srimulus attempt to review this vast literature here. But ii is clear that among is still inside the receptive field and the retinal stimulation is identical the areas participating in oculomotor control, the PP, the frontal eye to the previous condition {Fig- 2). Thus* the cell selectively process- fields (FEF), and the superior collmulus (SC) are especially impores information about the stimulus at the locus of attention, inside it* tant in higher order aspects of targei selecnon and ga?e control. Of receptive field, at the expense of information about unattended these three structures* FEF and SC appear to be critical* since stimuli. In addition to the effects of attention on stimuli at different combined lesions of those two structures severely impair the generalocations, attention to one aspect of a stimulus* such as rhe tion of saccadic eye movements. Interestingly, ablation of either orienration of a vertical red grid, also modulates neuronal responses structure alone causes minimal impairment of eye movements, in extrastriatc cortex {SO), possibly reducing the processing of except for short-latency movements, which suggesrs a high degree of unattended information even further. Together, these results may parallel function in the two structures {15). explain why wc perceive and remember the properties of a particular The enhancement of visual responses, like that seen in P P for any stimulus out of the many that may be impinging on the retina at a attended stimulus, also occurs in FEF {16) and SC (/?}, but only given moment. when the stimulus is to be a target of a saccadic eye movement and Surprisingly, unlike in P P , the attention effects observed in V4 not when it is merely the focus of attention. Neurons in these two depend on both the attended and ignored stimuli being located structures have "motorfields,"rather than strictly sensor)' receptive within a recorded neuron's receptive field, which is typically only a fields, which specify eye movements of a given amplitude and few degrees in size {9). If the animal attends to a srimulus outside a direction, independent of the initial eye position {IS. 19). cell's receptive field, the response to an ignored stimulus inside the Limb mavemmi coiiuoS. Directing attention, then the fovea, to the field is as strong as when the animal attends to it. Thus, in V4, point or feature of interest docs little to affect the environment or attention to a stimulus serves to attenuate responses to nearby achieve any goal beyond parsing and acquiring information. Influr
y
11
613
73* encing die environment requires use of the limbs or mouth for most SC with hdocaine does not eliminate the "optimal" eye movements mammals, and for primates, in particular, j* often means moving the coded by cells at rhe focus of the injection, bur rather decreases their hand to the object. Mueh of the central nervous system is devoted to velocity and increases their duration. Further, when the same region the control of movement, but we will focus here on the motor is inactivated, saccades to targets other than the optimal one axe cortex. inaccurate. If the optimal movement for die inactivated region is a Like the visual cortex., the motor cortex consists of a number of horizontal saccade to the left, and the target is 45 degrees up and to functionally distinct cortical fields (20, 21), and, like the visual rhe left, then the eye movement will miss the target by being too far cortex, and the SC, most motor areas are organized topographically. to the nght, as if some "pull" to the left is lacking. Such a result However, the details of topographic organization in the motor fields would be expected if an eye movement is caused by an averaged are not as well understood as in visual areas [22). ft is accepted, response of cells with a wide range of preferred eye movement though, that in the primary motor cortex (Ml), the hindlimb is vectors, distributed throughout much of SC Although the precise represented medially, the head and face laterally, and the forelimb in manner in which SC neuronal responses are averaged has not yet between. Much work, on the cerebral control of movement has been worked out, the averaging does not appear to be linear {29). focused on that "forelimb representation" [21), and it was there that Models positing linear summation predict that when the zone making the largest contribution to that vector is inactivated, ail eye practical behavioral neurophysiology began {23). Motor cortex cells begin discharging, on average, about 100 ms in movements would undershoot the intended target, whereas experiadvance of a limb movement and more than SO ms before the earliest mentally observed eye movements after deactivation somenmcs increases in electromyographic (EA1G) activity in the muscles that overs boor, the target. Whereas distributed population codes are efficient, they suffer will move the limb. In primates, at least some of these cells project through the pyramidal and corticospinal tracts to terminate directly from interference effects when two or more stimuli must be coded on spinal motor neurons. Physiological and anatomical studies show simultaneously, a difficulty termed the binding problem in network that they contact several different motor pools with excitatory theory {25). Combining the codes for two stimuli is no solution, synapses {24). Thus, there is a direct causal link between the because that would result in a third code, one specifying a completedischarge of at least some M l neurons and motor neuron excitation. ly different location. One possible mechanism for solving the binding problem is selective attention, which, by filleting out certain stimuli and enhancing others, could controf access to both sensory and motor networks. Yet, it may be objected that the Population C o d i n g attentionalfilteringof neuronal responses is never complete. Stimuli Little has been said up to this point about how neurons code that are unattended and rhat will not be the target of an eye highly accurate eye* head, and limb movements. One conceivable movement nevertheless may elicit at least weak neuronal discharges. mechanism would be to assign individual "command" neurons to code individual inputs or outputs, but theoretical work indicates greater precision can be achieved by neural networks in which the FIX individual elements are coarsely runed (25). As a simple illustration, I 1" assume a collection of eight neurons, each of which either responds RF totally or is silent, [f each neuron responds to a srimulus at only a single location, onlv eight different locations can be coded. However* if each neuron responds to stimuli at any of 128 locations, and the set of locations is unique for each neuron, then 256 different locations (that is, 3 bits) could theoretically be coded by the network with the use of a binary coding system. The spatial specificity of the network is increased, whereas the specificity of the individual neural elements is decreased. Although such binary coding schemes are nor actually used by the nervous system, a significant coding efficiency could similarly be achieved by broad, overlapping tuning functions or large, overlapping receptive and motor fields. In such coarse coding schemes, the neural code for a stimulus or response is not localized to a specific neuron, but rather is distributed across the population. In addition to coding efficiency, population codes have other useful properties such as resistance to the effects of local damage. Population codes appear ro be the codes actually used by the vertebrate nervous system (26). Flfl. 2. I- ii-: i of selective anention on responses of i ncmon n. V4. At the mended location (circled), rwo stimuli, sample and test, B*rt presented Papulation nam and g&ze shijii. In accord with the concept of sequentially and the monkey was rr quired to respond diETcrcndy depending coarse coding, the motorfieldsof individual FEF neurons appear to on whether they were the same or diflcrcnt. Irrelevant stimuli were presented be broadly tuned: activity al half the peak discharge frequency can simultaneously with the sample and test but it a separate location in the extend over aboul SO degrees of visual space {18). Similarly, broad receptive lick! (HF). In the initial mapping of the [<S, rhe cell responded well motor tuning curves arr observed in SC {27). At first glance, the ro red bars bm nor at all to green bats, A red bar (the effective stimulus) was breadth or coarseness of the motor tuning curves seems paradoxical: rhen placed at one location in the field and a green bar (the ineffective saccades are accurate to within a few degrees, at worst, and electrical itirniilus) at another. When the animal attended ro rhe location of the red bar the time of presentation of either the sample or lest, the cdl gave a good stimulation of SCor FEF causes eye movements similarly restricted at response (left), but when the animal attended to the Location of die green bar in amplitude and direction {IS, 28). Recent work {29) has clarified [right), the cell gave only a small response, even though the sensory this problem by showing that a broad region of SC contributes to an conditions were identical co the previous condition. Horizontal bars under eye movement, not just the focal region in which cells respond the histograms indicate the 200-ms period when the sample and test stimuli optimally before that movement. Deactivation of small regions of were on. FIX the fixation point during aft stimulus presentations. [Adapted from J . Moran and R. Do un OTIC {9)\ 1
614
739 I f large populations o f neurons code sensory stimuli and motor
averaged activity o f a population o f such cells correlates closely with
responses* w o u l d
movement direction. T o calculate such a population vector, each o f
nor even weak
responses
interfere wich rhe
population code* T h e answer seems to be that unattended
o
r
rhe cells in M l can be considered to represent a vector specifying its
untaxgeied items in the visual field can indeed cause a certain
"preferred" direction, defined as the arm-movement direction corre-
amount o f interference w i l h both recognition o f ( J ) , and eye
lated with the greatest discharge modulation. W h e n each neuronal
movements io (JO), an anended item. A c c o r d i n g to Findlay* under
vector is weighted by its activity, rhe calculated M l
certain conditions
vector corresponds well with the experimentally observed direction of limb movement,
. . , when two targets are presented simultaneously in the visual field* uteades land at a position intermediate between the two. It appear* that the saccade system docs not completely resolve the individual targets, and even voluntary effort cannot entirely overcome this tendency.
population
typically within less than I S degrees
(34).
Population activity o f M l cells also correlates better rhan single-cell activity w i l h die force generated by rhe limb {35, 36). Intcrutingly, although much M l
activity reflects maintained
postures and the production o f forces in the absence o f movement Attention, therefore* appears to ameliorate rhe binding problem,
(21, J 7 ) , for the majority o f M1 cells, movement-related changes in
but in itself is not sufficient to overcome it. A special case o f the
discharge rate arc not significantly affected by the starring position
binding problem occurs when a stimulus appears in the peripheral
of the limb
visual field, but an animal wants to continue looking at (or fixating)
population vector as calculated by Georgopoulos and his colleagues
a central target. I f the peripheral stimulus causes a neuronal response
{38) correlate with rhe direction o f limb movement
Most M l
neuronal modulation and the M l (or force
in S C and F E F , should not that activity lead ro a small saccade in its
production) rather rhan final limb position. T h i s finding is reminis-
direction* I t appears that the nervous system has evolved special
cent o f those in F E F and S C , where gaze-shift vectors are specified
circuits to circumvent this problem. I n the cat S C there is a tension
in relative terms, producing the same movement regardless o f the
between maintaining fixation on a stimulus and breaking fixation to
initial eye position
(28).
produce targeted saccades ( J I ) . Neurons representing central space, that is, near the fixation point, discharge maximally when the cat attentive I v fixates a stimulus. Neural activity is reduced when [he animal's attention is directed away from the fixation point and ceases
Coordinate Systems for Movement
altogether just before the saccade. I f these cells inhibit the neurons
Because we perceive the environment as stable while moving our
involved in saccade generation "downstream" from the S C , as they
eyes and body, many neurophysiologists have searched for a neural
ire thought ro {31), then eye movements lo unatrended stimuli
represenration o f space that remains invariant over eye, limb, or
could be prevented and the binding problem could be circumventedr Population
vtxtori and arm movem/nit.
body movement. A retina-centered coordinate sysrem does not have
Before and during arm
this property, because the retinal coordinates o f objects vary with
movements made in either two- or three-dimensional space. M l
gaze. Although a fixed reference sysrem may seem attractive, or even
neurons show a coarse tuning analogous to those in F E F and S C
necessary, ii has been shown that movements can be made without
(J2, J J ) . A s with eye movements* the coarseness o f tuning observed
reference to such a coordinate sysiem in many circumstances.
for individual neurons contrasts with the high degree o f accuracy
Indeed, as described above, neurons in the F E F have been argued ro
with which arm projection movements can be made. A rypical
code only relative movement vectors {28).
example o f an M l directional tuning curve is shown in F i g 3. T h e
Nevertheless, neurons in other structures may specify movements
cell is active before all movements to the left, with a peak activity
in a fixed coordinate sysrem. There is evidence that some neurons
modulation before and during movements berween 135 degrees and
may
2 2 5 degrees. T h e tuning o f the cell for direction is similar to the
others code limb movements in a body-centered coordinate system,
directional tuning mentioned
above for F E F neurons {IS). A l -
and still others code body movements in an envirottmenr-centered
though the riming o f each individual cell is very broad, the vector-
(allocentric) coordinate system. F o r example, a recently discovered
r
-*4d
i
"sin
icw
'
\
JsW
i
rio
code eye movements in a head-centered coordinate system,
ista
Fig, 3v Directional tuning of a cell in the primary motor cortex ( M l ) . (A) Raster displays, aligned on the beginning of movement (M), show the activity of one cell before, during, and after movements in eight directions. Vertical marks making up chr raster lines Indicate me rime of each action •ii* it 'tlx ' ii potential. The equal-amplitude movements began at the center of a circle. A visual soimuJiu presented the target of each limb movement 200 to 300 ms before movement onset. [B) The mean activity during the period from target presentation to the end of the movement is plotted against movement direction (mean ± 5 E M n = 5). |From A P, Georgopoulos « nl. {32), with permission]
615
740 conicalfield,ihe supplementary eye field (SEF), may be involved in time the system makes an average error of about 4 percent or 4 directingfixationto a particular location in space. The SEF (Fig. 1) degrees. The system generalizes over all reachable space, in accord lies immediately rostral to, or as some would argue is the rostral part with the finding rhat experience with visually guided reaching leads of, rhe supplementary motor area. Eye movements can be evoked to such generalization, whereas Jack of experience prevents it (46). with intracorticaJ stimulation of SEF and its cells discharge in relation to eye movements {39, 40). Many of the eye movements that can be evoked from the SEF converge on a given orbital position, regardless of the eye's initial position in the orbit {41). Studies of Conclusion single-unit activity in SEF also indicate that the activity is related to We have, of course, only rouched the surface of work currently head-centered spatial locations, rather than being strictly linked with taking place in thefieldof behavioral neurophysiology. But it occurs movements of a given amplitude and direction (40), as is typically to us that die stare of behavioral neurophysiology today resembles the case in FEF. Premolar cortical areas {20) may similarly provide a that of functional neuroanatomy in the early 1970s, At that time, body-centered coordinate system for goal acquisition by arm move- new techniques and approaches, many taken from cell biology, ted ments [40, 42). And, finally, cells in the hippocampus, at least in rats, to a revolution in neuroanatomical practice, one built on the are thought ro code an animal's location in an allocentric coordinate experience, thought, and strategies of the previous 20 years work system (43)with experimental fiber-tracing methods. Similarly, behavioral neuHow could a head- or body-centered neural representation of rophysiologists have had, for about two decades, a method for space be constructed* There is evidence that PP may be important in exploring neuronal activity in alert, behaving animals. We expect this function The responses of many neurons in PP refit a not only that the experience and experimental strategies developed during the position of a stimulus on the retina bur also the position of the that period, with the infusion of ideas from psychophysics, neural eye in the orbit. Zipser and Andersen [44) have developed a three- network theory, and engineering, will serve as the foundation for a layer neural network model lhat shows how spatial maps can be new era in neurophysiology. constructed out of neurons with those properties. Cells in the output layer of iheir network specify spatial location, cells in the REFERENCES AND NOTES input laver specify the position of the eye in the orbit and the retinal 1. B. lulctt- fVrturr 190, 9| < I9P7I. A. Trelsmin. SH. An,. 255. IJ4B (November I9BD). coordinates of a stimulus* and cells in the middle layer "learn" 2 | Allnun, F Miran, E MeGuinncM, Am Pi» B, 407 {19BS):, R. (through the adjustment of "synaptic" weights) to associate inputs Dcfimont ind 5. Schcin,J. N*«ropJivi^ 57, B3S {19B7f. with the appropriate outputs. Importantly, the "cells" in the middle 3. M.I. Posner ind f. A. Walker. Trtndi JVeurrmT. 10,1341987] Atrcnbem, of COUTH. play? an important rule in many perceptual and cognitive p wesses In addition » layer of the theoretical network develop response properties remarktriose cued here | « e M. I. Posner and O. 5. M. Minn. Eds., A'lintvn aid ably similar to those of PP neurons fojfcfitdrw. Sc. XI (ErirHum, Hillsdale, NJ, 1985)] J
4 S. Hochemui, A. ItzhJu, E. Gilat. 0 » n S„ 230, 65 (1981)1 J- M Milkr, I>. Sunon. H. FUngst. A. Ryan. H. Beaton. &#Kr (''72). Also see G. R MangUn and S. A. Hillyird fin Nmrc*\o\eg«4, Subureta ef Higher Cognitiw FMm1i»i, A. 6. ScheibeL. Ed. (Guilford Press, New Ycuk. m preuj for a revieu' of recent Itudiu in human cvent-relared porenoals; that have found effects •(" anennon remarkably consistent w|[h U O l K found in neuronal reto rding stud i a in aninub. 5. L G. Ungerleider and M. Mishkin, in A«ur,i•. of : ::-.:>•• D. J. Ingk.M. A. Goodaje. R. J. W Manifold. Ed*. (MIT Preu, Cambridge, 1984). pp 549-586 6 1 C Lynch. V B. Mountcuik. W. H . Tiltat, T. C T . Yin,./, ts'rt.raphyiiol 4 0 , 362 (19771; t>. L . Robin™, M. E. Goldberg, G- B. Etinron. iM- 4 1 , 910 (I978) M. C Buihndl, M E Goldberg, D L Robinson, ibid. 4fi. 755 {1981). 7. M. [. Posner "a!. NiMrt-piffrologia 25, 135 (1987). 5 R. Desimone. 5.. |. Schein. ]. MdnUI, L . G . UngEtkider. Vti. Ro 25,441 (I985 , |. H . R . Miunstl and W T Nrwsomc, Amm Ri,- NWwfl 10.363(1987) 9. I Monn and R. Desimone, Sntwr 229, 782 |1985> LO. D. J. Biaitman.flrjiNRts 307. 17 (1984); S. Hocfotem and ] K R Maunsell. Spc r W n r Ai»'. I I , 1244 119551. H. Spirzer and B J Richmond, tfrf., p. 1245; | M Fuster and } P. icrveyj N W n 2,36! <1982>. I L V. B. MountcutlE, B. C. Motlrr. M. A. Steinmen, A #• SeitokiS,/ Stunvu 7, 2239 (1987); H. Spine* ind B |. Richmond. Pttapimu 14. A13 (L9B5): B- | Richmond indT SatoJ NrMyfcjtlW. 58. 1292 IL9B7). 12 H . Spirjer. R OciimtUIC. I Moran Utrfnr 24D. iii ,]988). Improved oncUtiOon tuning k alw found in V4. bui not VI. when repermon o( J visual partem [rigcerlJ bchaviuriL response See V E. Hacnnvind P. H . Schiller Exp BUM Rti 69. 225 (1987). 13. Although tvr discuss mew processes serially, ur do nr»r mean ro implv char, in all tircunuiincu, There i* a wnal activation of brain irnjCturCt or QacdHuBnl involved in these proceucs H O Boct, Lxp Bicw\ Rti. bi. 4?6|L9B6r-; f. ^arllard, in J4M|HH pfV\utalBti\*w*, D. I IngLc, M. A Goodale, Ft ]. W. Mansfield. Ed*. (MIT Presj. Oim&ndge, \9&*t. PP 367-3S5. 15. P. H. SehiLer, S D Ttuc, J. L. Con^vav.J N n M p ^ U»(WWfi t H SchiUer, | H Sandell. I . H H- MaunseU, KW 57.1033(1987). 16. M. E. Guldbcrg and M, C. BmhneJ. .h,J 4& 773 \19S[, 17 K H WutnandC W Mohler. ,iid 39,745(19761. IB. C. I Bruce and M. E. Goldbcig, <M 53. 603 (19S5t 19 L E- Mavs and D. L 5paHu. rSrJ 43. 207 (L9B0>. 20. M. Wiesendangcr and S P. Wiw, m 77r, Fn-mJ Ub, StiOm * J fyttpiin, P, ChauiTl.A V Delgado-Esjucta, E. Hllgnn, |. Bancaud. Eds. |Riven, Wev. fort, in prcu). IL M.-C. Hepp-Revmond. in Cni^pjrarii'r Phuwr PWiji. inl. I off Jh>nVllil.iHJ. ft. D. StekLs and I Erwin, Edi. (Lis*. N«w Vort, 19S8>. pp. SOI-624. 12 ) . T . Murphv, H C. kwan, W.A MicKiV. Y C WongJ f * r t u * « l j 41,1132 (1978); I*. L Strick and I B Presron. ifctf. 4 8 . 139 (|9S2); C N Woolwv dW , HtfL PiM. Aw. ff«. hAW, M, . Di,. 30, 13fl (1932); D. R. Humphtev. ^m™i£rjfhif>rir Gjfir. N'mrPpJijuiJ ^rppJ 36. 393 | L9flL>. l 7 7
Learning Visuomotor Relations Even if eye and arm movements result from the activity of neuronal populations, ir remains unclear how these populations are programmed to produce the exact motor signals needed to acquire a sensory target- Indeed, this is quite a problem in a developing organism: as the limbs grow, they change rheir inertia! mass and the muscles become stronger. Those and a myriad of other developmental events change the relation between neural force commands and the resultant limb movements. There is evidence thai accurate reaching depends on prior visuomotor experience. Development of visuomotor control requires experience with vision and movement together, animals reared without experiencing the visual effects of their own motions cannot perform accurate reaching movements until they obtain that experience {45. 46). Held and Bauer [46) concluded that for monkeys . . . the ability to guide the hand visually with the normallv achieved requires mapping of the coordinates of motor response onto the space for vision for which prior exposure seems essential. accuracy
Likewise, neural network models of visually guided movement, such as that developed by Kuperstein [41), use accumulated experience ro adjust and maintain the spatial consistency between sensory input maps and motor output maps {48). His model posits that initially random motor signals produce movements, resulting in arm and hand positions that the system can observe. During learning, the errors in reaching a target are used to modify weightings between the sensory input maps, showing the target location in space, and the moior output maps necessary to acquire that target, something like Zipser and Andersen's model [44) of spatial mapping in FF After a few thousand trials, the weightings adjust ro rhe point thai the motor and sensory maps are spatially consistent, at which r
616
;
r
r
4
v
4
23 E. V. Evarn, EJrfmrafpMttff. Clin. M H ^ A C U 24. 83 (1968): sec also H. H . laaper, in ff«*,r,ir rV™arJ,m ihr Brain* H. H . Jasper, L. D. Proctor, R. S. Kruigton, W. C Noshiy, R . T . CoircUo, Eds. (LmJt Brown, Boston, 1959), pp. 423-434; D H HubeL; FJiyuCJ 150,91 (WS*). 24. C. G. Phillips and R. Porter, CtrtraipituI Noaai, (Aezelernic Press, New York, 1977); E, E . Fed ind P. D. Cheney, >, tfrunynyriri 44, 751 ,1980); R N. Lemon, G W H . Mantel, R. B. Muir, / fflyjirf. 381,497 (1986); Y.Shinoda, | T.Yokor^T.FLJjml.nleu-Dur. Ijtt 23, 7 (1981); D. G- Lawrence, R. Poitn, S. 1. Redman, J Camp rVrurrrJ. 232. 499 (193$ 25. G. E . Hinton, 1- L. McClrliand, D. E . Rumelhan, in Paraiiti Duinturrf Prraimnp. EipJmn'onj in ffre Mieoitrvtmn ojCegmHon, D. E . Rumdhan and J- L . McClelland. Ed*. ( M I T PICM, Cambridge, 1986), pp. 77-109 26. J. T. Mclwain, fur. Rev 10,223 (L976);_J. Nwnphyml. 38,219(1975); S. P_ Lehky and T ]. Sejnowik,, Nubut 111, 452 (]9BB). 27. C W Moh!er and R. H WurtzJ Atanphpitr. 39, 722 (1976). 28. M . E. Goldberg and C. L. Colby, in Handboek oj Nruroprfihalogy. F. Holler and ] Gnnrvin, Ed). (Elsevier, Amsterdam, in pits*}, vol 2. 29. C. Lee, W. H . Rohrrr, D. L. Sparks, iVdi-rr 332, 357 (1988). 30. 1. M. FindJay, in Spatial', Oninrrd BrWior, A. Heln and M. Jeinnerod, Eds (Spnngc^Verlag, Berlin, 1982), pp. 261-302 31. £>. P. Munoa and D. Guitfon, Sot- fVmnjn. Abilr 14, 956 (1988); Can./PJiyrioJ Pnannd^r 65, Amv {1987). 32. A. P. Gcorgopoiuos, 1. F. Kalaskz, R. CaminiU, l - T . MisseyJ. rV«nuri. 2, 1527 (1982). 33. A. B. Schwartz, ft. E- Kermer, A. P. Georgopoulos, ibid. 8, 2913 (1988). 34. A. P. Geargapoulos, R. E. KeHncr, A. B. Schwartz, jrW., p. 2928; A. P. GAtgupOuJca, A. B. Schwartz, R. E. Kcnner. Snenir 233, 1416 '1986); A. P.
ftprf,
741 Georgopoulos, R_ CamiruU, J. F. Klllfha, J. T. MiStty, Exp Bnun flu. Sup/. 7, 327 (1983). 35. D. ft. Humphrey, E . M. Schmidt W. D. Thompson, Sneme 170, 758 ( W O ) . 36. Similarly, population vetTotj as calculated From the activity of movement-relared neurons in area 5 of PP |J. F. Kalaska, R. CamiritO, A. P- Georgopoulos, Exp. Brain Rrt 51, 247 (1983)] and visual cells in area 7 of ?P ( M A Steinmetz, B. C . Motter, C. J. Dully, V. B. MounlcasrJrJ Nruram. 7, 177 (1987)1 tOrrciporid to rhe limb movement and Visual Stimulus direction*, r5Sp«o\dy. 37. 38. 39 40 41.
42. 43 44 4546. 47. 48. 49.
617
1,1, KataskaandM L. Hyde, M l fl«. 59. 197 (1955). R. E. KEttnEr, A. B. Schwartz, A. P. Georgopoulos J iVnuwml B, 2938 (1988) J. SchlagandM. SthJag.ReyJ Mtu.epky,io< 57,179(1987). S E. Mann, R. Thau, P. HI- • Exp Drain ka. 69,460(1988). A similar head-centered coordinate mapping is observed for eye and head mo-rmenn evoked from the caudal pin of the SC in cats;. See D Glutton. M. CronunElinch. A. ROIKOUI. £xp Brain Rt, 39.03 (1980); rbJ, p. 75. G RitaoLarti, in Motor Arem of the Crrrftraf Conn. C-k Syrup win m * (Wiley, New Vo*. 1987), pp. 171-181 B. L. McNaughton, C A. Runes, f. O'Keele, E. NtDBld. 102, 148 (1988) G. M. EckLm^, ,-^ri-niJ Dfln»inurri (Buic Books, New York, 1987); E. I Knudsen, / ^n-mphjiFB^. S, 3094 (1985). WerJiiflkE. E. Fen, M. I. Posner, T . R. Colbum, B. M. Smith, L . G . Ungerleider, and E. A. Murray for their comments on earlier versions of this, manuscript r
Rcprimcd wilh permission from Neltiiz, Vol. 3Jl.pp.6?9-6W.1988 © 1988 Macmiltan Moguincs Lid.
A back-propagation programmed network that simulates response properties of a subset of posterior parietal neurons David Zipser" & Richard A. Andersen' 1
Institute for Cognitive Science. University uf California. San Diego, La Jolla, California. 92093, USA T Department, of Brain and Cognitive Sciences;. Massachu-ni-. Institute al Technology. Cambridge, Massachusetts 02139, USA
Neurons in area 7a of the posterior parietal cortex of monkeys respond to both the retinal location of a visual stimulus and the position of the eyes and by combining these signals represent the spatial location of external objects. A neural network model, programmed using back-propagation learning, can decode this spatial information from area 7a neurons and accounts for their observed response properties. THIS article addresses Ihe queslion of how the brain carries out compulations such as coordinate transformations which translaic sensory inpuis io motor outputs. V i s u a l inputs are collected in ihe coordinate frame of the retina on which Ihe visual environment is imaged, b u l molor movements such as reaching are made to locations in external space. C h a n g e s in eye position will alter ihe retinal locations o f targets while (heif spatial locations remain constant. A s a result, visual inpuis m u s l be transformed from retinal coordinates lo coordinates thai specify
616
the localion of visual objects with r e s p e c l l o the body to perform accurately directed movements. Lesions lo the posterior parietal cortex in monkeys and humans produce profound spatial deficits in both motor behaviour a n d p e r c e p t i o n . H u m a n s with lesions to this area can still see but they appear to be unable lo integrate the position of iheir bodies with respect to visual inputs. T h e lesion data further suggest that it is the inferior parietal lobule, which comprises the posterior half of the posterior parietal cortex. 1-1
6M
Hf. I a, Experimental protocol Tor determining spatial gain field*, with ihe projection screen viewed from behind ihe monkey's head. These experiments were carried QUI several years before Ihe slant of ihe made)ring described here , bul some of ihe daia are being presented for ihe first time. To determine [he effect of eye position, Ibe monkey •> nl. head fixed,ftxateson a point. / at one of 9 symmetrically placed locations on ihe projection screen. The stimulus. S, is always presented al ihe same retinal location, chosen as the max imunvrespon.se zone of Ihe retinal ^receptivefield.The Stimulus consists of 1- or 6-degree, diameter spotsflashedfor 500 ms. Each measurement is repealed 3 times,fc,Peri-slimulus histograms of a typical gain field determination. The nine histograms are located in the same relative positions as tbe fixations lhat pfoduced ihem. The vertical line indicates the lime of visual stimulus onset, e, A graphic method for illustrating these data in which ihe diameter of the darkened inner circle, representing the visually evoked gain fields is cakulaled by subtracting the background activity recorded 500 ms before the stimulus onset from the lolal activity during the stimulus. The outer circle diameter, representing the total response gainfields,corresponds ihe total activity during the stimulus. The annul us diameter Corresponds to ihe background activity lhat is due to an eye-position signal alone, recorded during the 500m* before the stimulus presentation. 10
which is involved in spatial processes. Anatomical and physiological experiments in macaque monkeys indicate lhal the inferior parietal lobule conlains al leasl four separate conical fields. Area 7a contains visual and eyeposition neurons ' ; area 7b contains somatosensory and reachrelated cells '", area MST contains visual motion and smooth pursuit eye movement activity (refs 12-14; R. H H. Wurli and W. X Newsome, personal communication); and area LIP con¬ tains visual and saccade-related activity' , tt has been propased '"' * that the area most likely to perform spatial transformations is area 7a. Most of the cells in this region were found lo receive a convergence of both eye-position and retinal signals. The interaction between eye-posiiion and visual responses was non-linear, and in most cases the visual rsponse could be modelled as a gain that was a function of eye position multiplied by the response profile of the retinal receptive field. Thus the visual receptive field remained cetinolopic. but the magnitude of ihe response was modulated by eye position. This modulation can be shown to produce a tuning for ihe location of targets in head-centred coordinates that is eye-position-dependent; ihe cells will fire most for a particular location in era ototopic space, but only when Ihe eyes are at the appropriate posilions in Ihe orbits. No cells were found lhat coded larget location over all eye posilions {eye-position-in dependent coding), indicating that this information can only be contained in the pattern of activity •f a population of neurons- Here we describe a neural network model lhat shows how eye-posit ion-in dependent location can be extracted from a population of area 7a neurons. The model also reproduces the non-linear inleraclions of eye position and retinal position information seen in actual area 7a neurons, and 6 10
9
1,14
10
1
619
demonstrates response properties, such as large receptive fields, which are strikingly similar to those observed in single-unit recording studies. The neural modelling technique we use differs significanlly from mosl previous approaches, which first found an algorithm to accomplish Ihe compulation, and ihen specified neural models lo implement the algorithm. Our approach is based on the use of a neural network (raining procedure, called 'back propagation", which can programme artificial neural networks lo compute arbitrary functions (refs 19-73; S- IL Lehky and T. J. Sejnowski, personal communication)- Unlike computer programming, programming by training uses only examples of input and output. This means that it is not necessary to know in advance the algorithm that the network will use; ihe learning process will discover an appropriate algorithm. Back propagalion nelworks have internal or hidden units that are free to take on the response prop?rites to best accomplish ihe computational lask being learned, fl is the properties oT these hidden units that we find resemble those of cortical neurons. The back-propagation procedure accomplishes learning by adjusting ihe strengths of the synapses within the network. Experimental results From area 7a The experimental dala that must be accounted for by any model of area 7a were collected previously in an extensive series of sludies with awake, unanaestheiized monkeys . Here we describe new analyses of these data lhat facilitate a comparison or area la cells with the units generated by Tiaining ihe network model. Three of the major classes oT area 7a neurons are of interest here: the eye-position cells responding to eye-position only (15% or all cells sampled from area 7a). the visual cells responding to visual stimulation only (21%). and the spatially tuned cells responding to both eye position and visual stimulation 157%). Neurons in ihe first [wo classes presumably represent the eye position and retinal locaton information available to area 7a as input. The interaction of eye position and visual information found in the third class of cells produces a representation for the head-centred location of visual targets that is eye-positiondependent. 10
The experimental protocol involved recording neuronal activity exlra-cellularly from awake, unanaesthetized monkeys trained in various visuospatial tasks (see Fig. 1), The eyeposition sensitivity was tested by having the animal fixate on a small point at different eye positions, with ihe head fixed in otherwise total darkness. The eye-position cells typically showed a linear increase in activity for a range of horizontal or vertical eye positions, although some cells showed more complex eyeposition coding. An ensemble of 30 eye-position unit responses is shown in Fig. 4c. The receptive fields of the visual cells'were tested by flashing a spot stimulus at d L fie re nt locations in the visual Geld while the animal fixated on a target at a single eye position. Surfaces were fit to these data points using a gaussian interpolation. These cells typically had large receptive fields equally distributed across the visual field for the population of neurons wilh a single peak of activity. The shape of the receptive fields approximated a symmetrical gaussian with a 1/e width of 15 degrees (see legend to Fig. A). 10
The spatially tuned neurons were the Jargesl group and showed a convergence of eye posilion and retinal position information. The receptive fields are very large (often over 80 degrees in diameter] and have one or more peaks that form a smoothly changing, hilly landscape, A set of 12 retinal receptive fields from spatially tuned cells, arranged according lo peak eccentricity and complexity, is shown in Fig. 2. As mentioned above, the evoked visual response of spatially tuned neurons varies as a function of eye posilion. This effect was examined by collecting data under the condition in which •he visual stimulus always appears at the peak location in ihe reiinal receptive field, but with the animal fixating al nine
10
40
30
20
Fig. 2 The receptivefieldsof spatially tuned neurons from area 7a. . . r - r i n rows *itb the eccentricity of thefieldmaxima increasing lo •heright,and in columns wilh ihe complexity of Ihe fields increasing downwards. Receptivefieldswere sampled ai I" radially spaced points, wilh one sample laken al the centre of ihe field* and Tour samples taken on each or four circles of radius 10, 20, 30 and 40 degrees. All the • elds in row a have single peaks. Those in row b have a single Urge peak bul some complexities in thefield.The Fields in row e are rhe most complex with multiple peaks. The data have been normalized so lhat Ihe highest peak in each field is ihe same height.
10
different eye posilions . I n esc plots are referred to as spatial gainfields.Figure 1 demonstrates the experimental protocol for mapping spatial gain fields. The majority of the spatial gainfieldsare roughly planar. This planar behaviour is evident in Fig. 1 c, where the darkened inner circles are proportional to the magnitude of the visual evoked response, the outer circle diameter to the lolal activity during the flashed stimulus, and the annulus diameter Ihe background activity due to ihe eye position alone. The data for the visually evoked response in Fig. lc can be fitted by a plane lilted up for eye positions lo the left and also lilted up for downward eye positions. Analysis using linear regressions in the two dimensions of eye position indicated lhat the gainfieldsTor the visually evoked activity (represented by the dark inner circles) were planar, or had a large planar component, for 55% of rhe neurons. Interestingly, £0% of the total response gain fields (represented by the outer circles) were planar or largely planar. A useful way to further characterize the nonlinear combinations of eye and retinal information is lo compare the contributions of each lo the total response of the cells. This can be done simply by comparing the dark inner circles, which represent the visual contribution lo the response, with the while annuli, which iepresent the eye posilion contribution to ihe response. This comparison shows ihree basic lypes of gain fields. For 28% the background and evoked aclivities change in a parallel fashion (Fig. H>. e, f) In most of the cells (43%), Ihe evoked activity changes wilh eye position while ihe background activity, if any, remained conslanl (Fig, la, c, of); ihree-quarters of these cells had very low or undetectable background activity. The remaining 28% of ihe neurons showed the interesting piopeny lhat the background and evoked activities changed in differenl directions, so thai the activity of either alone was grossly non-planar, but ihe overall activity was planar (Fig 3g, h, J). r
620
The neural network model We used a three-layer network (illustrated in Fig 4) lhat was trained to map visual largels lo head-centred coordinates, given any arbitary pair of eye and retinal positions' The first, or input, layer has two sections, an array of units on which the visual stimulus is represented, and a set of units representing eye position. The second layer consists of the hidden units, which map the input to the ouiput. Each hidden unit receives input r
a
. ..d©®©9®®0 • •• ®® © 0 0 0
• • • <§)®® O® o •
•
•
•
•
Ǥ)## m s
§ • • •
O
®
®
©
@)
°
o
«
©
®
®
0
• • '®®o © » .e ® 0 O
©® ® • * ©
Fig. :< The spatial gain fields of 9 neurons (.J - il from area 7a in the formal of Fig, 1c.
Fig. 4 ... Back-propagation network used lo model area 7a. The visual input consists of 64 units wilh gaussian OUTPUT I receptive fields with l / e widths of 15 degrees. The centre of each receptive field occupies a position in an 8 by 8 array wilh 10 degree spacings. The shading represents the level of activity for a single-spot stimulus, wilh darker shading representing higher rates oT activity. The units ILIJtD 1 1 - a L O P l ] have been arrayed topographically For illustrative purposes only; this paliern is not an aspect of the model READ K ' S L O I I ) • • • • • • • • • D u D B D D D i s each hidden unil receives input from every one of • • • • • • • • the W retinal inpul units. The eye position inpul consists • 9LOPI] • • • • • • • • • • • • • • • • oT* sets of & units each with two sets coding horizontal Btxa x posilion (one for negative slope and one for positive BEHU X t ' B L O r i ) • • • • slope) and two sets coding vertical position. Shading represents the level of activity. The intercepts have teen ordered for illustrative purposes only and do not represenl information available to the hidden layer. Each eye posilion cell projects tu every unil in the hidden, layer. Two output representations were used; the flIDDEU gaussian output formal is shown on Ihe right and the UNITS monotonic formal on the left. The gaussian formal units have gaussian shaded receptive fields plotted in headcentred coordinates- They have 1/e widths of 15 degrees and arc centred on an 8 by S array in head coordinate apace with 10 degree spacings. The monolonic format units have firing rales thai are a linear function of posi• H2 O • • • • x r E T t SLOPE) tion of the stimulus in head-centred coord in alts. There are four sets of S units with two Sets of opposite slope i n i | t = :.ir.rii n pn • • • t for vertical position and two sets for horizontal position • • • • • • • • in head-cent red coordinates. Again, shading represents • D O Q B D D D • • • • • • • • M-SLO"! the degree of activity and the topographic ordering is for illustrate purposes Only. The small boxes containing ;-s W indicale the location of Ihe synapses whose weights are trained by back propagation. Each hidden unit proAITIhJL x jects lo every cell in the output layer. The output activity of the hidden and output layer units is calculated by Ihe logistic function: output = 1/(1 + e""'* J, where n e t (weighted sum of inputs) + bias. The arrow for the connections represents Ihe direction of activity propagation; error was propagated back in the opposite direction. The back-propagation procedure guarantees that the synaptic weight changes will always move the network towards lower errpr by implementing a gradient deScrnl in error in the multi-dimensional synaptic weight space. FlB-FOSXlOa b. Area 7a visual neuron receptive field with a single peak near the fovea. Receptive fields were plotted using the same method as In Fig. 3. Visual cells that had no eye-posilion,-related activity or modulation of their responses by eye position were used to model the retinal input lo the network, c, A composite of 30 area it ion units, whose firing rales are plotted as a function of horizontal or vertical eye deviation.
• ••••••• • ••••••• • •••••••
1
•••••••a
t
s
l
•••••••• •••••••• ••••••••
1
The slopes and intercepts are experimental values for eye-posilion neurons.
from every input unit a n d projects to every unit in the third output layer. T h e output o f each unit of the h i d d e n a n d output layers is computed as an S shaped (logistic) function of the synaptic strength weighted sum o f its inputs, plus a bias term. The training paradigm uses b a c k propagation learning, w h i c h consists o f choosing an input a n d desired output, applying the input lo ihe first layer of the network a n d propagating the activity it generates through ihe network to the output units. T h e actual Output is ihen subtracted from Ihe desired output lo generate an error. T h i s error is used to adjust Ihe weights of synapses on the output layer unils a n d hidden layer units in a manner prescribed by the back propagation procedure *. T r a i n i n g begins with all weights r a n d o m i z e d , resulting in large errors, and the training cycle is repealed until the error is reduced lo desired levels.
a gaussian format in w h i c h each unit had a gaussian receptive field similar lo the representation of the retinal input, but coding location in head-centred rather than retinal coordinales. T h e other (output I in F i g . 4 a ) was a monolonic format, in which the activity o f each neuron is a linear function of the Location o f ihe stimulus in head-centred coordinates. F o r the gaussian format, a 64-unit array similar to ihe retinal input array was used a n d Tor the monotonic format, a 32-unil array similar lo the eye position input array was used. T h e gaussian a n d monotonic formats were chosen because they represent the most c o m m o n types of coding formats found for brain cells. Also the monotonic formal has the interesting feature that it has ihe same representation as the eye-positron code at ihe input. T h u s , if the animal foveales the visual stimulus, the resulting eye-position signal could be used as the teacher lo indicale the correct location of the stimulus in head centred, coordinates.
1
The retinal position and eye position inputs lo the network are modelled using characteristics of the cells in the posterior parietal cortex that respond to visual stimuli only and eye position only. T h e visual input consisted of 64 gaussian-shaped receptive fields, with 1/e widths o f IS degrees a n d with each peak separated by 10degrees in an 8 by 8 array. T h e eye posilion input consisted of four sets of 8 units with single sets for positive and negative slopes for horizontal and vertical eye posilion. We used two representations of location in head-centred coordinates at the output layer. O n e (output 2 in F i g . 4 a ) was
The model network was trained using randomly selected pairs of input eye positions and retinal positions. T h e teacher signal (desired output) used to train the output units was the true spatial location in head-centred coordinates implied by ihe inputs, and was represented in either the monotonic or gaussian formal. T h e network trained quickly; after —1,000 trials, accuracies equivalent to ihe distance between retinal unil centres were reached. When training was continued, error continued to decrease, but a I a lower rate.
621
•n 10
30
20
40
Fli[, 5 Hidden unil retinal receptivefieldsgenerated by the back propagation model. These plots were generated by holding ihe eye position input 10 (he network constant and simulating visual stimulation at the same 17 retinal positions used in ihe experiments on area 7a, The hidden unil aclivilies were normalized and pJolled in the same way as Ihe experimental data shown in Fig, 3- The data shown here are from a leries of 4 training sessions using networks, wilh 25 hidden unils and ihe monatonic format output. Similar results weie obrained for the gaus^an format output. All ihefields,except forC-l0,C-2Q and C-30. arc from networks lhat have received 1.000 learning trials. The remaining three are from untrained networks* resulting only from the random synaptic weights assigned al the start of a trainingran-Very complex fields are only rarely found in trained networks. No hidden unit with a. single peak ai 10 degrees appeared in this data set and such units ace very rare in trained networks. No Spatially luned neurons wilh ccnlral receptivefieldswere found in area 7a, and no suchfieldsappeared in the trained model. Bul central receptivefieldsare found among ihe visual neurons in 7a, and this kind of unil was among those used as input LQ The model network. Agreement of model with experiment To evaluate the model, we First compare the experimental and model retinal receptive fields. The model receptive fields were categorized according to their complexity, and the eccenlricity of their activity maxima (Fig. 5), as had been done previously
3
9
fee)® ®
oo
®<s>® ® 0 O • ® i b
• • •
e
®0O h
®® ® o®m
• •• ® ® ® ®®• ® ® • •# ® « '<§><•>©'• • • ® ® #• •
mm® c
• *»
• • •
0 0 ®
Fig, 6 Hidden unil spatial gain fields generated by the model network. Fields o-f were generated using Ihe monotonic formal Output; the rest used the gaussian format output.
622
for the experimental receptive fields (Fig, 2), Comparison of Ihe top lines in Figs 2 and 5 shows that the trained models generate single-peak receptive fields resembling those observed experimentally at all eccentricities except 10 degrees. The fully trained model also produces moderately complex fields Like those found in line 2, but rarely produces receptive fields as complex as those in the bouom line of Fig. 2. This kind of highly complex field is not distinguishable from the unlrained model receptive fields shown in the bottom of Fig. 5. The comparison process contains an element of subjectivity, but it demonstrates that ihe trained model generates retinal receptive fields remarkably similar lo ihe experimentally observedfields.The gain fields generated by ihe model are shown in Fig, 6", All ihe total-response gainfields,whether generated by the monotonic or the gaussian output format, were planar in shape. This result compares wilh 80% of the experimental fields in Ihis planar class. When the model gain fields are examined in more detail, taking into accounl the non-linearilies of ihe visual responsefields,there are significant differences between the eye position and retinal output formats. For example, when trained with the monotonic output format, 67% of the visual response gain fields were planar, but when trained with the gaussian output format only 13% fall jn this class. Thesefigurescompare wilh 55% in ihis class for ihe experimental data. The irregular visual response gain fields generated by the gaussian output format are more radically irregular than those generated by ihe monolonic oulpul format. Thus it appears lhat lo account for the details of the visual response gain fields, it may be necessary to use both types of output representation. It should be pointed out, however* that whereas the visual receptive fields and the total gain fields
were virtually unaffected by changing parameters of the model, substantially alter our conclusions concerning the degree oT the visual response gain fields were very sensitive lo parameters similarity between model and experimental data. such as threshold value and output representation. The number From the physiological perspective these experiments raise of hidden units had little effect, giving similar results for simulathe possibility that the posterior parietal cortex Learns lo associtions ranging from •> to 36 hidden units. ate body posilion wilh visual posilion to localize accurately the The striking similarly between model and experimental data position of objects with respect to the body. This idea aboul the certainly supports the conjecture lhat the cortex and ihe network importance of learning is reasonable considering thai It would generated by back propagation compute in similar ways. As nol be practical for spatial representations to be hard-wired back propagation generates optimal solutions which produce because the body dimensions change during development. Furihe least error, these results also suggest lhal the brain chooses thermore, adaptation experiments show thai distortion of space optimal solutions with respect to error. The similarity of the wilh prisms leads lo rapid recali bra lion, suggesting lhat these model and lest results raises the question of what physiological representations are still plastic in adults. As the model, by mechanisms could subserve this equivalence. Presently the backdeli nit ton, does not have a topographic organization to localize propagation paradigm is structured at a level higher than in space, it shows thai the brain does not need a topographic implementation, and obviously cannot be applied literally to organization to Localize in space. The organization of the the brain, because information does not travel backwards rapidly network is not a product of Ihe spatial position of the cell bodies, through axons. That the back-propagation method appears to but rather is contained in the pattern of the weights of Ihe discover the same algorithm lhat is used by the brain in no way synaptic connections, implies that back propagation is actually used by (he brain. One Finally, there is the question of where the output units oT (he approach to understanding the physiological significance of model could exist in (he brain, One possible location would be these results is lo generate models incorporating features found areas thai receive projection from area 7a Although eye position in the brain, such as Hebbian-like learning at synapses and effects on visual responses have been described at several locareciprocal back-projection pathways for propagating error, and tions in the brain (reft 24-26; S. Funahashi, C. L Bruce. P. S. determine whether these models generate similar results. In ihis Goldman-Raklc; and R. Lai, M. J. Friedlander, personal comregard i l is interesting to note that all cortico-cortical and munication), an eye-posilion-independent coding has yet to be thalamocortical connections have reciprocal feedback pathunequivocally demonstrated. But it is also possible that the Final ways. spatial output could only exist in the behaviour of the animal. For example, the muscles innervating the eye or limb are broadly An important consideration in interpreting the results is bow tuned, and the position of theeye or limb is coded in adistributed closely does the model response actually resemble ihe cortical data? This is a complex issue because there is error in the fashion over the activity of several muscles. Thus ihe final spatial Output may not exist in any single cell in ihe brain, bul rather experimental data, and additional errors introduced by the intermight be found only in the pointing of the eye or finger accurately polation process used to produce the full-field views or the to a location in space. receptive fields. Examination of the magnitudes of these errors We lhank David Rumelhart, Francis Crick and Emilio Bizzi indicates that they could not account for Ihe various receptive for stimulating discussion during the development of this model field types, or eye-position gain fields observed experimentally and Carol Andersen for editorial assistance. D,Z. was supported and in the model. The methods of. comparison between model by granls from the Sysiem Development Foundation, the and experiment that we have used are lo some degree subjective. AFOSR and the Office of Naval Research, R.A.A. was supported Perhaps a more objective comparison procedure will eventually by the NIH, the Sloan Foundation and Whitaker Health Sciences be developed, but it is unlikely thai, given the complex Foundation. nature of Ihe dala to be compared, any such technique will r
fttunvfd
l«OciDb-rr | S | I - M H f f l d I? JWMhh? 19El
t
e m i r , v . B . Plum, F t G F I I « , S- R. * imt. b. M n u n l t l l l k . V. B . L j m J i , 1. C , Geariopeifloi. A . S l S i U , H- * A
\y
pJ (Twin J f « i l . 3 l « - » ! { L t l T l . A t i d E n r r . P. A . C B U , O. K. * S » | * l . P. M . St**'t J » , l i B - O l f t f l j j KrtM
c
tttttm t m-isir»«>.
tyjOtf,
r
r
f l
L v r r t i , l - C . W o ^ ! « n l r , V . B . T J I I U H , W . H . 4 V i n , T . C . T . J. NwpphytiDt 4$, JG7-JI? I. «. 10. 11. 12
U . S i k i n . K . S h i b u l i r l . H Ha. V. t flhtfWl. K. Ei"l B'm\* Kit U . tSS-tfi] II91D|. tt G n i d i . J. W. i A i d c i c i , P. A. tJpJ ffuJn ffu, Lin Ihe p>m). | l i A r d f n c i . H. A. t CjFijdl. I . W. in Rriimn I" Onvtamow Rfirirch Vol. U r d * W u r l i . H_ a Geldfc* r t , M.) r F l i t ^ i i r . A m u t r d i m , in iht p i o i j . IT. Aridrr»
I . OfccMfy, M - T V t W r o ' U b f j ( R t f M t " c w v ^ i . nun, J_ L^ficfc, J. C. Btt-w fl.Jur.Srt 1 . 4 4 - U 4 [ M B ) , 1. A f l d m c i , H. A i n SpH'iaf Cefl'Uf- fiMJn 0«'J and DE|*r,7jinnrnl j r d i S L i l r i . O * . ' i . t , K r i l t h r r l t V . M. 4 B * | ) u | i U l < TJrivtnily al CJiiugn P i t l > l . »n l i l t pitu). I Bock 0 EtVn-illlcr, Ft. * k r d t n r n . R. A B'O'H RfL t i n llvr pi'ltY i. A n d i - i m , n . A . la H B - j S f c l f / W j l i i l i i i l T * r f f n H fyjifMi y - H . J I B MDUM.
r
CWBIH
T
JJ. A c h i H , 0 . 4 . . Hinlon. 0. E . 4 SrpmD-'iki. T. J. CBJxirinr S r i T, N J - l t 9 i l 9 B J l . J I SEinI*I« 32D, 1 £ 0 - ] ' S 11f I t ] TJ. S n U i t . J . S r h l i | r R . rt„ f w f c . C . H . 4 JeitpK. J. Ewpt B'lt* t>r- *>, l l f l - I M (1*10).
m,
T r
M. P f t k . C K , S t N » | - H r , M- A S t h l i | , J. ramp /Vrvwl I H . f f - l l t ( 1 K D 1 It- A i o r d i , G . E n i i j l i n i . P. P. A G i l l n i i . 0 . / . Wiittol. I n " " ! I f l (Ifll). T
623
n
4b Bridges between Psychophysics and Physiology
The two articles of this section pursue the study of the primate visual system, in the search for even closer correlation between neurophysiological data and perceptual functions, and in the analysis of the neurocomputational level for some specific tasks (neural coding for motion detection in area M T , neural circuitry for contour perception in area V 2 ) . Article (47), by Newsome, Britten and Movshon, is an influential letter that has established a "paradigm", demonstrating how neuronal and perceptual data can be obtained at the same time, under well defined conditions. As a result, it is shown that detection and discrimination judgments could be based on the activity of a relatively small number of neurons. Thus, the present evidence is that these perceptual functions are not subserved by a neural code of the "population coding" type, in contrast with the motor tasks discussed in articles (17) and (45). Article (48), by Peterhans and von der Heydt, is a synthesis of the remarkable phenomenon of subjective contours. In their general introduction, the authors suggest that a key to correlate neurophysiology and psychophysics consists in classifying neural signals according to whether they show properties of perceptual constancy or not. Thus this paper is along the line of Delbriick's text (article (13)). It is also along the line of LivingstoneHubel's synthesis (43), with which it expresses some differences, though. This search for the localisation of "complex" neurons detecting subjective contours is interesting to contrast with the different strategy of article (55), looking for collective synchronous firing of a group of "simple" cells. F i nally, the debate between explanations of subjective contours in terms of stimulus-driven mechanisms at early processing levels, and feedback mechanisms implying top-down influence from higher levels, will be evoked again in section 5b.
624
Reprinted wilh permission from Nature. Vol 341. No 6237. pp 52-54, 1 9 8 9 0 1 9 B 9 Mucmillan Magazines Lid.
Neuronal correlates of a perceptual decision
il
William T. New some*t Kenneth H. Bnttten't & J. Anthony Movahont
iflui
p
ro
" Department ol Neurobiology and Behavior. Stale University ot New York, Stony BrooK. New Yorti 11794. USA t Department of Psychology and Center for Neural Science. New York University, New Von* 10OO3. USA T H E relationship between neuronal activity and psychophysical judgement has long been of interest to students of sensory processing- Previous analyses of this problem have compared the performance of human or animal observers in detection or discrimination tasks with the signals carried by individual neuron*, but hate been hampered because neuronal nod perceptual data were not obtained at Ibe same time and under the same conditions *. W e bave now measured the performance uf monkeys and o l visual cortical neurons while the animals performed a psychophysical task well matched to the properties of the neurons under study. H e r e we report that the reliability and sensitivity of most neurons on this task equalled or eKceeded that of the monkeys. W e therefore suggest that under our conditions- psychophysical judgements could be based on tbe activity of a relatively s m a l l numbet o l neurons.
c o r n M t n - T2.8%
correlation - 3.2%
correlation - 0.8%
20
1-
0
1 1
100
Spikes par trial
O u r general methods for monitoring unit activity and eye position in alert, behaving monkeys are derived from those devised by W u r t z ei o f a n d o u r p s y c h o p h y s i c a l methods were based o n those described by N e w s o m e a n d P a r e , I n brief, animals were trained to report the direction of motion of a random dot d i s p l a y in w h i c h some dots moved coherently while [he r e m a i n d e r m o v e d at r a n d o m . W e varied the strength o f the motion signal by varying the proportion o f the dots moving coherently: at 0% c o r r e l a t i o n , all Ihe motion was r a n d o m : at 100% c o r r e l a t i o n , all the motion was coherent. N e a r t h r e s h o l d , the stimulus resembled the d y n a m i c noise seen on a domestic television set tuned between stations, combined with a barely perceptible sensation of global motion. W e recorded singleneuron activity from area M T ( V 5 ) . a region of the extrastriafe visual cortex c o n c e r n e d with motion processing, w h e r e most neurons respond o p t i m a l l y to visual stimuli o f a particular direction a n d speed of m o t i o n ' . Because efficient extraction o f motion signals from this stimulus requires considerable integration over space, it seemed likely thai neurons in M T . w h i c h have relatively large receptive fields, w o u l d be particularly suited to this task. N e w s o m e a n d P a r e have recently s h o w n lhat lesions o f M T elevate perceptual ihresholds for this lask. 5
6
7
0.1
FIG. 1 Physiological and psychophysical data obtained simultaneously from a rhesus monkey, a, The responses ol a directionally selective MT neuron al three different motion correlations spanning physiological threshold. The hatched bars represent responses to motion In the neuron's preferred direction the solid Oars indicate responses to motion in the null direction (lSO*oppo5lte to the preferred,. Sixty trials were performed in each direction for each o' the three conelation levels. Response distributions for a range of correlation levels were used to compute a 'neurometric' function that characterized the neuron's sensitivity to the motion signal and cduld be compared whtn [he psychometric 'unction computed from the mor*ey's behavioural responses b. Comparison of simultaneously recorded psychometric and neurometry functions. Psychophysical performance of the monisy. O; performance of the neuron. # . Psychophysical performance at each correlation Is given by [he proportion of trials on which the monkey correctly Identified the direction of motion. Neuronal performance Is calculated from distributions ol responses like those in Fig. l a using a signal-Detection method described in the text. The physiological and psychophysical data form similar curves, but the data for the neuron lie to the left ol the data for ihe mor*ey. meaning that the neuron was somewhat more sensitive than the monkey. We fit the data with smooth functions of the form introduced to psychophysics by Quick . Threshold, defined as the correlation for which the direction of motion was identified coireclty an 8 7 % of the trials, was 6-1% for the morfrey and 4.4% tor the neuron.
6
P
W h i l e m e a s u r i n g the p s y c h o p h y s i c a l threshold, we recorded the activity o f the M T neuron for w h i c h the stimulus parameters
' PrewM xHtess
D f l M ' u n f U H Neurmioic*,. Slanrcd univsrsiTv " W n a i SHntf. Siantgra Ctfif a -
"OO
Correlation l * >
1 0
W e used a two-alternative forced-choice procedure to measure thresholds. W E p l a c e d o u r stimulus so that it j u s t covered the receptive held of the n e u r o n u n d e r s t u d y , and adjusted the speed to match that preferred by the n e u r o n . M o t i o n was presented either in the neuron's preferred direction or in the 'null' direction 1 B 0 away. O n an i n d i v i d u a l trial, the monkey was required lo h o l d fixation for 2 seconds while the motion stimulus was presented. At ihe e n d of The trial, the monkey indicated his j u d g m e n t by transferring his gaze lo one o f two s m a l l I ight-e milting diodes, c o r r e s p o n d i n g to the preferred or null direction o f motion. W e presented at least 30 trials (15 in each direction) for each o f several correlation values c h o s e n to elicit performance that varied f r o m c h a n c e to near perfection, a n d c o m p i l e d these data into p s y c h o m e t r i c functions. R e c a l l i n g that performance w o u l d be 50% correct by c h a n c e , w e defined the threshold as Ihe correlation required for the m o n k e y to j u d g e the direction of motion correctly on 82% o f the trials.
10
t.O
11
were optimized. T h e computer counted the action potentials elicited on each trial, a n d compiled distributions like those shown for a typical neuron in Fig, l o . I n these distributions, filled bars represent trials in w h i c h the motion was in the null direction, a n d cross-hatched bars indicate trials for the preferred direction. It is evident that at a correlation of O . B % the two distributions were not different, whereas at a correlation of 12.3%, where the neuron was strongly direction-selective, they barely overlapped. T o c o m p a r e these neuronal data with the p s y c h o p h y s i c a l data, we postulated lhat performance d e p e n d e d on a c o m p a r i s o n between ihe activity o f two neurons, the one u n d e r study a n d another differing only in that it preferred the
625
opposite direction of motion. Under this assumption, we could use the distributions in Fig. In torepresentthe responses of the neuron under study and its anlineuron'; we simply reversed the preferred and null directions for the anlineuron. On any individual trial, therefore, the observer would compare a response drawn from the distribution represented by the hatched bars in Fig. la with one drawn from the distribution represented by the solid bars. The direction chosen would be the preferred direction of the neuron giving the larger response. The performance of an MT neuron could then be characterized as the probability that a randomly selected response from the hatched distribution in Fig. la was larger than a randomly selected response from the solid distribution. We chose this method for analysing physiological data because il most directly related neuronal performance lo the directional discrimination task lhat the monkey was engaged in. For the data in Fig. In. ai a correlation of 0,8%, this decision rule chose the correct direction only on about half the trials {random performance), whereas al a correlation of I2 8% it performed nearly perfectly. We used a method based on signal detection theory" lo estimate this choice probability for each correlation value, and plotted the results as neuromelric functions' formally equivalent lo the psychometric functions representing the psychophysical data '*. The two functions for this example neuron are shown in Fig. i '•: filled circles represent neurometric data, open circles represent psychometric data. Evidently the two curves are very similar, with the neurometric data poinis lying slightly to the left of the psychometric data points; in ihis case ihe neuronal threshold was slightly lower than the psychophysical one. We used a likelihood-ratio statistic to test the hypothesis that the psychometric and neurometric functions were the same. For this neuron, this hypothesis could r
L
I I 111 [||— [ I I I fl III I II I 0.3 1 3 10 30 Threshold ratio Lneuron/behaviour) FIG. 2 Comparison ot psychophysical and physiological thresholds obtained tor 60 MT neurons In two rhesus moneys. The frequency histogram shows the distribution of the ratio of the physiological threshold to the psychophysical Threshold 'or all the neurons for which we ohtained data. A value of 1 reoresen is perfect correspondence between psychophysical and physiological thresholds; values <:1 indicate that the physiological threshold was lower lhan the psychophysical threshold, whereas values >1 indicate the converse. The directional preferences of Ihe 60 neurons were roughly uniformly distributed, and there was no reliable association between a neuron's direction or speed preference and its threshold relative to the perceptual threshold. I
0.1
3
not be rejected
{P>0.05).
We performed this analysis for 45 neurons recorded from one monkey, and 15 neurons from a second. Figure 2 shows a histogram of the distribution of the ratio of neurometric to psychometric thresholds for these 60 neurons. Values of this ratio of < \ represent cases where the neuron's threshold was lower than the monkey's; values > I represent cases where the monkey's performance was better than the neuron s. Intuitively, it might be expected that the behavioural threshold would be lower than any particular neuronal threshold but, in most cases, neuronal thresholds and perceptual thresholds were similar. Indeed in some cases, neuronal thresholds were substantially lower than perceptual thresholds. For 20 of the 60 neurons in our sample, the psychometric and neuromelric functions were statistically indistinguishable (P>0.05); in 13 of the 40 remaining cases, neuronal thresholds were lower than perceptual thresholds. In other words, if the monkeys were able to select and measure ihe discharge of some of ihese neurons as we did. their performance could have been beller lhan it actually was An inability to select the most informative signals can be considered as a kind of perceptual uncertainly, of ihe kind modelled by Pelli". Obliged to monitor signals from many sources Less informative lhan ihe one perfectly luned lo ihe visual target, ihe animal's perceptual performance would be degraded, because each sub-optimal source would contribute more noise than signal- Neuronal performance would then exceed psychophysical performance. Our results suggest, however, that Ihis effect is not large. Substantial uncertainty would make the psychometric function sleeper than the neurometric function", but as was the case for the example shown in Fig. lo, the slopes of these two functions are usually similar. We thus conclude that under our conditions, the monkey's perceptual decision is not greatly affected by irrelevant signals inlroduced by uncertainty. The apparent absence of uncertainty leads, however, to T
r
626
another question: iT a perceptual decision can be based with relative certainly on the discharge of ihe ruosl informative neurons, why is behavioural performance noi further enhanced by using a pooled signal derived from many such informative neurons? If enough such neurons were present, such pooling would substantially improve psychophysical performance by averaging out ihe noise that obscures weak signals. Our data show that in most cases, the neuronal and psychophysical performances are similar, indicating thai signals from many neuronal sources arc not pooled lo reduce perceptual thresholds. One way to account for the absence of either pooling or uncertainty efleetsis to suggest that the van ability in ihe responses of similarly tuned neurons is correlated. Both pooling and uncertainty act as we have stated only if different neuronal signals are perturbed by independent sources of variation. If the sources are not independent, then uncertainty does no damage and pooling provides no benefit, because different neurons are carrying similar signals. The rich network of shared connections that link MT neurons with the retina might well produce correlation among neurons with related selectivities, bur this possibility has not been studied. Our lack of information about rhe degree of shared variability makes ii impossible for us to assert rhat the neurons whose responses we have recorded are the ones that contribute io the monkey's perceptual judgements. Nonetheless, our results show rhat a reasonable account of the monkey's performance can be constructed, using a simple decision rule, from signals carried by small numbers of neurons whose selectivities are well matched to the demands of Ihe perceptual task. • U r a r v r f 10 for II. rcrpiarJ 3S Jufr 1 » 9 i X 3 A S s 7 S. 9 io LI. 13. 13.
a&ritrv. K B 4 i r . -- « K i an. 1-24 TolhufHO ( . hfcmVHVi. J At DHn'jk F HUXHflH a , 775-7S5 (19B3J Pnrtinr. A. J f H**«fn tt J J ogl Sac Am * 2 . 1101-11H [1985k fr«l!y.ft..3om]rie.C.O'^-^.l.Saii.G tP<«fli«»HLD ^ AWPMhfSrff 67.7SS-7 J20987J hU-Bml. A . r4c*WCH « ' I WuTO. R K Jteuropf >3&K 1308-13?? (19651 v
P K t » r * . t t r.s p«e E a j fwurpw t2201-2211 [ I S B I Difiner. A A Etfd, S *A Sfffin 3S. 53B-S3? (19TD Ma 5. M. J JfljSiOJ 2m, 5*9-573 H974I. M a i m * , j rt ft 1 Van Essen. D C. J r^apflyvoi *S. l U J - l H f 119*9, "mripit i o i Nftmwtysff ea, 1106-1130 n « * i & W 1 ( I I 5n"PU. A. 5»1*£)fl!Ber«n " w r y **S nyOofifysla. Quia. R F KfOftnpr* U , 65-87 ri97*|. M 0. 0. / oot Soc Am A l i S O f l - l H Z (19H5>
ACwowifHH«rrs Fffc-VUD la W T N
1.
Thi* ~w% • * » SUD&XIM D> • q n l i r™n NIH
tHUttfoe*tmK 19661
m » Sloan
ft™*i*i
Rf primed wilh permission from Trends in Nturoicttne/i. Vol IJ, No .3,rjo. 113-ll«, 1991 © 1991 Elsevier Scienc! Publishers Ud.(UK)
Subjective contours - bridging the gap between psychophysics and physiology Esther Peterhans and Rudiger von der Heydt Esther Peterhans andMuch is known aboul Ihe initial stages of visual Rudiger von derprocessing up to Ihe striate cortex, but how is visual Heydl are at the information represented and handled al subsequent Dept of Neurology,stages? Phenomena of contour, color and movement University Hospitalperception have been used lo identify functions of Ziirieh CH-8091 neurons and to reveal functional differences between Zurich Switzerland. cortical concepts related provide ditions
areas lhat application of classical receptive-field has nol suggested. These differences can be lo theoretical stages of visual processing thai stability of perception under changing conof stimulation.
Psychophysical methods have long been used to investigate physiological processes that were hidden in the brain and, when physiological data eventually became available, they often confirmed the conclusions drawn from psychophysical experiments, or indicated a correlation between the physiological and psychophysical phenomena. The measurements of cone absorption spectra, lor example, confirmed the Young-Helmholtz theory of color vision and demonstrated the physiological correlate of trichromatism. More often, however, particularly in recent decades, physiological studies have led tofindingsthat neither confirmed expectations nor provided correlates of known psychophysical concepts, such as the receptive fields of cortical neurons. Physiology became emancipated, so to speak, and was used independenUy to study vision. However, many authors have since studied visual cortical processing without reference to perception. Indeed, it might appear to be pointiess to try to rind corresponding physiological data for the vast variety of psychophysical phenomena. Yet physiology needs to be linked to perception. The complexity of the visual cortex defies an approach that is comparable to taking a clock apart in order to understand its function'.
case of color. We are used to thinking that color perception depends on the wavelength composition of light, specifically on the degree to which the three cone types are activated. However, in general, the perceived color of a surface is not determined solely by the light that reaches the eyes from lhat surface, but depends also on the light reflected from other parts of the scene; that is, we speak of "color contrast' and 'induction effects'. Cortical signals that relate to perception of color should be subject to similar effects. Thus the functional role of a signal can be identified from its co-variance with perception under critical conditions. Co-variance also implies constancy under certain conditions. For example, visual systems are expected to tend to achieve color constancy. An attribute that characterizes the reflectance properties of object surfaces and remains constant in spire of changes in illumination (which of course affects the Light that reaches the eyes from a surface) needs to be established. Psychophysics gives an indication of the degree to which human vision achieves such constancy. On the basis of this knowledge (or one's own perception], neural signals can be classified according to whether they show this degree ol constancy or not. By comparing the signals recorded in various parts of the visual system, it might be possible to pinpoint the site of this processing and eventually understand the underlying mechanism. There are many other concepts in psychophysics analogous to color that could be exploited in this manner; the most promising are those that can be related to a theory (such as the goal of constancy in the example of color). Thus, the emphasis is on understanding the physiology and building theories on physiological observations rather than looking for correlates of perceptual phenomena.
This article discusses examples of physiological studies that illustrate this idea, rather than attempting Psychophysical concepts as tools in physiology to provide a comprehensive review of studies on The approach that is suggested here is the use of visual cortical processing. First, examples in which perceptual phenomena as tools to make sense of the connection to perception is relatively loose, as in neuronal signals - . In spite of the knowledge that has studies using anesthetized animals, will be discussed. accumulated concerning cortical receptive fields, cor- This will proceed to comparisons of neuronal tical signals are often still enigmatic. This is particu- responses in alert, behaving monkeys with examples larly true for recordings from the extrastriate areas, of human perception, and end with studies in which but even responses from the striate cortex can be the link is most stringent, as in attempts to relate puialing when stimuli slightly more complex than neuronal responses to the perceptual decisions of a edges and bars are used. Researchers who have not monkey. themselves listened to these signals and tried to understand them might not grasp the difficulty of Color, movement, depth guessing the meaning of the signal that the electrode The relative constancy of color perception has been happens to pick up. The difficulties are accentuated if recordings are made under more natural conditions, compared with the responses of cells in areas VI and V4 of the monkey . A collage of various colored for example, in an awake animal. papers was illuminated with light of varied compoSince signals in the visual cortex are probably sition. It turned out that, in general, responses of related to perceptual categories, such as form, color, cells in VI change more with alterations ol illumination depth, movement and texture, the co-variance of a than do the perceived colors. For example, responses signal with these aspects of perception can help in of a cell sensitive to long-wave (red) light that was interpreting the signal. This is most apparent in the stimulated by a red paper cease when middle-wave 2 3
4,5
627
Ill
Fig. 1. The problem of defining occluding contour*. (A) Photograph. (B) Mapot the contrast edges extracted from IA}. The lines of abrupt changes tn grey value are depicted as given by a procedure known as 'Sobel operator' in the field of computer vision and by setting a threshold al 5% ol the maximum contrast. Since foreground end background can have equal gray values at Ihe border {'anomalous contour'). Ihe obfecls are delineated incompletely and spurious connections appear, tn (CJ, Ihe true contours, that is, the tines of discontinuity of depth, are shown and truncations of background structure by occlusion are indicated flight blue shading). This representation corresponds approximately lo what we perceive when looking at (A).
(green) light is added to the illumination, even though the paper might still appear red. On the other hand, responses of some cells of V4 co-vary with perception, for example, a red-sensitive V4 cell can still fire under greenish illumination as long as the paper under its receptive field appears red. Thus it seems that ihe colors, as we perceive them, are first elaborated in V4. The underlying mechanism is still unknown. Although the generalization of these findings could be questioned, this is an important experiment that certainly should be repeated in the alert animal. Objects can move in space, but there is no physical movement in images. The system infers movement from changes ol intensity. Il is not obvious how this is done, given the direction- and orientation-selective cells in the cortex. For example, when a square moves diagonally up and to the right, the cells stimulated by its edges signal either up or to the right, but none signals the true diagonal motion. But it is the diagonal motion that we perceive. It has been shown with moving gratings and plaids (two superimposed gratings of different orientations) that, in anesthetized monkeys, cells in V I signal only movements ol components, while some cells oE area MT (V5) are able to integrate these and signal the movement ol the pattern as it is perceived . 6
mobon is due to an eye movement. In experiments with alert monkeys, it has been found that many cells of area V3A (Ref. 7) and the posterior parietal area 7a (Ref 81 fulfill postulates (1) and (3). or at least show a reduction in response to eye movement-related stimulus mobon. while this was much less frequent in V I and V2 (Ref. 71. Whether the reduction in response was due to suppression of a motion signal or to its cancellation by an eye movement signal (vector subtraction) is not known. Such cells were called "realmotion cells' , although responses during tracking have not been shown. 7
Visual tracking neurons have been described in the posterior parietal cortex and in area MST*"' . However, it was argued that these play a role in the generation of tracking eye movements rather than in the coding of object movement, because MST lesions produce eye movement deficits and because the receptive field structure of these cells seems inappropriate lor discriminating object motion from self motion . On the other hand, visual tracking neurons have been shown to signal induced movement*. Note that postulates (2) and (31 do not necessarily require an exlraretinal signal since, under certain conditions, the eye movement signal could be derived visually, lor example, by assuming that the background is stationary. Neuronal mechanisms that might be appropriate for this have been demonstrated . Thus, we argue that relatively simple considerations of the rules of perception might provide a useful classification of the motion-related cortical signals. This is independent of the elucidation of the underlying mechanisms. !
13
12
Object movement can only be inferred from the retinal image if the eyes are stationary. In general, however, eye movements must be taken into account. In the extreme, when the eyes track a moving object, its image might not move at all on the retina, but the object is nevertheless perceived as moving. Other non-moving objects are then perceived as stationary, although their images move. Thus, a Stereoscopic depth depends on disparity and is population of neurons whose responses co-vary with fairly independent of the patterns conveying the movement perception should (1) signal the direction disparity (for example, we can see depth in randomof movement of a stimulus moving across the dot stereograms l. In the alert monkey, simple cells receptive field while the eyes are stationary, (2) signal in V I can signal disparity of isolated bars, but only the direction of movement of gaze during tracking complex cells can also decode random-dot when the stimulus remains fixed in the receptive field, stereograms". Depth perception seems to require an and (3) signal no movement when retinal stimulus important additional step ol processing. When an le
628
114
ft
match (within the limits set by noise of image and detector) are problematic. The photograph in Fig. IA illustrates this. Figure IB is a map of the contrast edges extracted from Fig. IA. It represents the contours only partly. Also 'mergers', such as the connections between the contours of the lamp and the window frame on the left, are found and so are deserters', parts of contour that are completely detached from the foreground object and join a background structure to constitute new features, such as the half-moon shapes to the right of the foreground lamp.
ft
o
C)
o
ft
c
B
D
Fig. 2- 'Subjective contours'. Simple configurations can produce illusory perception of contours (A). (B) end ID). The contours disappear when lines closing the gaps in the inducing elements are added (C). The subjective contours are not necessarily extrapolations of given edges or lines (D). They mayfA). IB), or may not (DI be accompanied by a brightness illusion. (Taken,
In Fig. 1C the true contours have been drawn, based on our knowledge of the photographed objects. The lamp in front is delineated completely, and terminations produced by occlusion are marked by light blue shading to indicate that these line endings with permission, from Ref. 31.) and comers are not features of an object, and that the intersected edges or lines are likely to continue object is seen at various distances, the size of its behind the occluding object. This is how the picture is retinal image changes approximately in inverse pro- to be interpreted, and how the scene should be portion to viewing distance, but the span of disparity represented in terms of the 'real' contours of the changes in inverse proportion to the square of viewing photographed objects. This is, in (act. how we see it. distance. Thus, the ratio between span of disparity We do not see the spurious features and false and size of image shrinks when objects move away combinations; Instead we see the foreground object from the observer. Nevertheless, they do not seem clearly set apart from the background by a complete to flatten. A book appears to be about as thick at a contour. Does the visual system complete and 'clean distance of 80 cm as at 40 cm. while the disparities up' the contours before we see them? are four times smaller. Our visual system seems to Some simple drawings give the impression of lines scale the disparity signals with viewing distance. Viewing distance signals have been demonstrated in or borders bridging gaps (Fig. 21. These 'illusory contours' have intrigued many investigators . In posterior parietal cortex " and dependence on stimulus disparity has been demonstrated in many celis of such figures, contour perception is 'illusory' or visual cortex, but. to our knowledge, no-one has 'subjective', but if one looks at Fig. IA or the corresponding real scene the perceived contours are looked for 'real-depth cells' yet. actually truthful. The phenomenon is clearly the Findings that show that neuronal signals do not same. We use the term 'anomalous contour' " to correspond to perception in the expected manner are denote the physical condition when a contour, defined also illuminating. In a checkerboard pattern, for by the two-dimensional projection of a threeexample, the orientation-selective cells of VI do not, dimensional scene, has vanishing contrast. Extending in general, signal the rows and columns of the checks this definition, afigureis called an 'anomalous-contour we perceive, but rather the diagonals and various figure' if, in principle, it could have been generated in oblique orientations that correspond to the two- such a way. Thus, in contrast to illusory (or subjecdimensional Fourier components of the pattern . tive] contours, anomalous contours exist indepenWhile many cells of VI are selective for spatial dently of whether they can be perceived or not. The frequency, they might not signal the fundamental need for this terminology will become clear in the frequency of low-contrast, square-wave gratings, following discussion. though this is the periodicity perceived 27-29
2
3
21
22
The vexing contours When a three-dimensional scene is projected onto a two-dimensional retina, information from near and distant objects is jumbled. If the system is to recognize something, it must detect the contours in the image: that is, the borders that separate the objects from their background. It is often assumed that contours are denned by contrast, so that the system needs only to look for lines of abrupt changes of intensity. Since cells in the primary visual cortex of cat and monkey signal orientation and position of contrast borders , it might be assumed that these cells define contours. The extraction of contrast borders is also a fundamental operation in computer vision* ' . 23
5 M
Contour definition based on contrast, however, is inherently incomplete and faulty. Even though foreground and background intensities can differ statistically most of the time, the few instances in which they
629
Contour coding in the visual cortex Are anomalous contours represented in the visual cortex? Recordings from alert monkeys have shown ^ that, while cells in area VI ignore such contours, many cells in area V2 respond to anomalous-contour figures as if the contours were given by edges or lines. These cells fire when an anomalous contour passes over their response field, providing its orientation is appropriate (Fig. 3). Their responses co-vary with perception in several ways. When intersecting lines are added to the illusory-bar figure, responses are reduced or abolished (Fig. 3B and C; compare Fig. 2B and C). and when either half of the figure is presented alone, the cell fails to respond (not shown). Responses elicited by the abutting gratings (Fig. 3D1 increase with the number of line-ends, from zero for one line-end to a maximum for ten line-ends on average, just as the perceptual strength of illusory contours depends on the number of inducing elements (compare Refs 35, 36). Tilting 31-
Hi the lines relative to the contour reduces the responses (compare Refs 37, 38) and shifts the peaks of orientation tuning, just as perceived orientation is altered by oblique lines in the Zollner and Hering illusions ". Many cells in VI are also activated by anomalouscontour figures, but these responses are related to the contrast-defined elements. For example, the lineendings of the abutting gratings (Fig. 3D) can excite end-stopped cells of VI with receptivefieldsorthogonal to the anomalous contour. Also, contrary to cells in V2 that respond to figures with gaps of several degrees of visual angle. VI cells are often sensitive to even liny gaps in bars or edges . This can be compared with the lack of response to the edges of checkerboards mentioned above and the inhibition produced by contrast-reversed segments in bars"-* found in cat area 17. Thus, while the V2 mechanisms tend to bridge gaps, VI responses specifically imply continuity of contrast borders. It can be seen, then, that V2 cells show interesting new properties that one would never have guessed from their bar and edge responses. It seems that V2 is closer to contour perception than V I . 3
53
have been interpreted similarly . Physiology also predicts that adaptation with anomalous contours in one eye should affect test contours in the other eye (interocular transfer), since only cells of V2 that are nearly always binocular are involved, whereas adaptation with ordinary lines should be only partially effective in transference, since it also involves the monocular populations of VI; this has in fact been confirmed . Within V2, anomalous-contour responses occur in the thick stripes and the interstripes.
Unil 3GD5
10
51
2
It might come as a surprise that a phenomenon that gave rise to early gestalt psychology*' and that was later regarded as cognitive should be evident in monkey cortex. But even cats see illusory contours, as shown in behavioral studies *' . Thus, 'cultural differences' do not seem to be a factor. It is surprising to find these signals at a level as low as V2 (in the cat they have even been noted in area 17; Ref. 46, but see also Ref. 47), where the cells still appear to be 'driven' by the stimulus, as is the case In V I . This suggests that we are dealing with 'low-level' processes, although 'top-down' Influences cannot easily be ruled out, since the recordings are from alert animals. 4
45
Cognitive contours? Cognitive theory assumes that perception results from the system trying to find the most likely explanation for given sensory data. In the case of tbe illusory triangle, it might consider the possibility that three sectored black discs and three angles happen to be arranged in that particular way or. alternatively, 2° 0.5S (2°) that there is a white triangle overlying three complete discs and an outline triangle - the latter is accepted as Fig. 3. The responses of a cell recorded in area V2 of the the more likely explanation. monkey visual cortex. While the monkey is fixating the One argument against the involvement of higher cross, the cell's response field (ellipse) is tin verged by a centers is the relatively short latency of the dark bar moving back and forth within a stationary iigbt anomalous-contour responses (compare Refs 49, rectangle (A), by an illusory bar tB) or by a contour by discontinuity ot tines (D). hi (CJ, the moving 50). Other arguments come from psychophysical produced notches ire closed by thin (2 min arc) lines. rVo stimulus is experiments. Prolonged inspection of anomalous con- present in (E). The responses of the cell are shown on the tours of one orientation results in subsequentiy right where each dot corresponds to an action potential, viewed contours of near-by orientations appearing and each row to a cycle of stimulus movement. The tilted, just as in the till after-effect with ordinary forward sweeps are plotted in the left halves of tbe display l i n e s . However, while adaptation with lines makes and the backward sweeps are plotted in the nght halves, other lines and anomalous contours appear tilted, with a reversed time axis Average numbers of action adaptation with anomalous contours only tilts these potentials per cycle are shown under each panel. The cell contours and hardly affects ordinary lines . This is responds to tbe bar (A) as well as to the different precisely what is predicted by the physiological data, anomalous contours (B), (D), but with the closing lines tbe since anomalous contour-sensitive cells are only a illusory-bar response is virtually abolished (C). For percepsubset of the orientation-selective cells in V2 (Refs tion, see Fig. 2B and C Orientation selectivity was found 33, 34). The rest of these remain unadapted and thus to be tbe same for ordinary (A) and anomalous contours provide unbiased orientation signals. Asymmetry (B). (0). While this cell responds weakly to anomalous other cells in V2 respond as strongly as to bars found in the training effects of orientation judgement contours, and edges or even more strongly. 43,4 5
33
5 1
52
630
H6 A
to create an illusion. As sketched above, the detection of occluding contours is a key problem of understanding images. It seems that, as a result of evolutionary adaptation, the circuitry of V2 can detect these contours at an early stage of processing by the fast, parallel action of a large number of relatively simple modules. The principle ol this process relies on Ihe fact that various image features occur in statistical correlation with occluding contours , such as the abrupt changes of (1) intensity, (2) structure (see Fig. IA). (3) binocular disparity, (4) morion and (5) the dynamic occlusion/disocchision of the background, to list only a few possibilities of what relatively simple mechanisms could cope with. A change in disparity (3) occurs because occluding contours, by definition, are lines of discontinuity of depth. Features (4) and (5) result when the observer or the objecl moves. In stationary images, only (1), (2) and (3) are available: monocularly. only (1) and (2) are available. Each of these indicators taken by itself is insufficient to reveal the contours (Fig. IB), but when combined they permit reliable detection. 59
A possible mechanism Figure 4 shows a simple model of a mechanism that combines evidence from contrast and discontinuity-ofbackground patterns (more sophisticated models including computer simulations exist "). Contrast borders stimulate an input from classical simple or complex cells in the center (grey shading), and terminations of lines and edges excite end-stopped cells with receptive fields distributed along the contour and mainly orthogonal to it (blue shading). Pairs of these cells straddling the center field are connected by gating units to ensure that activation on one side only has no effect (so that illusory contours do not spread out from single comers). These signals are added to that of the classical input. This means that a coincidence of terminations can be equivalent to contrast (compare Rei. 62). in figures like those in Fig. 2. this mechanism is wrong, hence the illusion, but when looking at normal three-dimensional scenes, it can be largely correct It also produces geometrical illusions when the intersected elements are not orthogonal to the contour, but are tilted . This is a consequence of the mainly orthogonal orientation of the end-stopped cells. Clearly, the circuits of lowlevel mechanisms must be rather simple because they have to be replicated many times over for the different locations in the visual field. Illusions are the price of simplicity; speed of parallel processing is the gain. 60
Fig, 4. A possible mechanism for contour perception. (A) .'r assumed that contour cells sum two different kinds ot signals that both originate from the same patch of retina - one tor detection of contrast borders fgrey), and another for detection of pattern discontinuity flight blue). The cell shown in IA) has a vertical orientation. The first type of input comes from simple or complex receptive fields ot the same orientation (vertical ellipse) and the second comes from a set of end-stopped receptive fields that are distributed along this orientation, but oriented perpendicular to it (honzontal ellipses with circular disks symbolizing inhibitory end-zones, only four are shown) These cells are thought to respond to comers and line-ends, but not to edges and continual lines. Pairs of end-stopped cells with distent receptive fields are connected by non-linear elements (x) which produce a signal only it bolh cells are activated simultaneously. In fTJ these signals are summed, together with tne hrst input to produce the contour signal. (B) The mechanism is activated by the anomalous contour of a triangle and it is turned off when tbe open sectors are closed by circular line segments. Two comers excite a pair of end-stopped cells: the line segments inhibit them.
but not in the thin stripes of the cytochrome-oxidase pattern *, where color-coded cells predominate". This might relate to the observation that illusory contours fade when the inducing shapes are displayed in equiluminant color contrast" *. These findings suggest that mechanisms similar to those in monkey exist in the human visual cortex. Orientation-specific adaptation and disappearance of anomalous contours at equduminance seem to imply low-level mechanisms. 4
6
The theoretical level Physiological data per se are not a satisfactory explanation ol the phenomenon of illusory contours , even though one encounters expressions such as 'physiological explanation' or 'physiological theory' in the literature. The existence of a large number of cells signaling anomalous contours (one-third ol the cells of V2, which is the second largest visual area ) rather raises the question of why the system puts so much effort into analysing these contours - it can hardly be 58
631
39
Preattentive parallel processing Can parallel processing of anomalous contours be demonstrated in a visual search task? With ordinary lines one can delect, say. a horizontal target line among vertical lines with a reaction time that is independent of the number of lines presented". The same is true when bars made up of anomalous contours of the type shown in Fig. 3D are used . A horizontal anomalous bar pops out from an array of vertical anomalous bars. However, since the anomalous contour-sensitive cells - * also respond to contrast borders, anomalous contours do not pop out when they have to compete with other contours (exclusive anomalous-contour detectors might not 61
33
3
117 exist). Thus, an illusory triangle produced by a triplet ol sectored discs in a group of other triplets of sectored discs cannot be detected by parallel search . The simplicity of Ihe proposed mechanism (Fig. 4) should not be taken as indicating that contour perception as a whole is a simple process As pointed out, one has to assume several different mechanisms acting in parallel, and the manner in which their outputs are combined is sophisticated. For example, binocular disparity not only generates contours . but also interacts with the formation of contours from pattern discontinuities : pattern discontinuity produces depth perception by itself , or by interaction with motion" and dynamic occlusion/disclosure ™ Dealing with situations of occlusion implies that there exists some representation of the hidden parts of objects as sketched in Fig. 1C ("amodal completion'* *), which can be demonstrated perceptually . Some of these demonstrations, like those of stereoscopic illusory contours , were meant to support a cognitive theory. While these have disclosed much of the complexity of contour processing, it is intriguing to investigate how much cognitive reasoning can be translated into the language of low-level mechanisms '. 65
1
30-M
67
59
96
71
66
51
Fig. 5. St often seems that contour perception involves interpretation' of sensory information in terms of a three-dimensional layout of objects. However, this msght simply reflect 'implicit knowledge' in the design ot lowlevel cortical mechanisms. Global consistency is not checked- and perception can contradict known geometry of space. (Reproduced, with peiimsiofi. from Ref
It is important to note that the mechanisms present in visual cortex have evolved to a high degree of sophistication, incorporating a great deal of information about the environment, optical imaging and statistical correlations in images. This implicit knowledge ^ makes the action of these mechanisms appear to be cognitive. It could be said that perceptual hypotheses are 'wired in'. However, the resemblance is superficial Many demonstrations show that the system does not really evaluate the hypotheses or check for consistency at a global level. Perception can be grossly inconsistent with the known peometry o! space (Fig. 5). 7
Certainly, form perception also involves processing at higher levels, and some aspects of illusory contours cannot be explained by stimulus-driven mechanisms alone , but the question is whether feedback to lower levels must be assumed. 73
28.)
whether this paradigm can be extended to more complex perceptual or cognitive functions. Anatomical segregation of function The stripy pattern of cytochrome-oxidase activity seen in V2 has been interpreted as representing three interlaced systems for color, form and depth processing . THs intriguing hypothesis is based on the distributions of neuronal selectivity in three sets of stripes for stimulus orientation and length, wavelength and binocular disparity, and on the pattern of connections with VI (Refs 55, 78, 79). It has received further support from demonstrations of selective connections with the prestriate areas V4 and MT, whose specializations are in accordance with such a scheme " . However, the segregation of the selectivities in V2 is not quite as clear as might be ex¬ pected from the proposed segregation of function * ^. Ln fact, this makes sense since perceptual qualities like contour, color, motion and depth are not characterized primarily by unique 'cues', but by their lawful dependence on certain stimulus conditions, including, for example, invariance with changes of contour configuration, illumination, eye movements and viewing distance. Selectivity might not reveal function. Selectivity for disparity, for example, can serve depth as well as form perception * (see Ref. 80 for further discussion of the multiple roles of cues'). Cells in V4 are not more selective for wavelength than are cells in VI or retinal ganglion cells * , but they show color constancy , Cells in V2 are not more selective for orientation than are cells in VI, but represent contours in a generalized sense . 55
00
63
1
Strengthening the link between physiology and perception The link with perception is indirect in all the examples discussed above, since neuronal signals and perception were compared in different subjects or even species. Perhaps the most exciting recent development is the introduction of simultaneous recording and perceptual testing This technique was first used to compare neuronal and perceptual spectral-sensitivity thresholds in the monkey . By comparing the neuronal response and perceptual decision for each single-stimulus presentation, it has been shown that variations in response of certain neurons in area MT correlate with the momentary fluctuations of movement perception at threshold and under conditions of binocular rivalry , and that myostimulation of small pools of cells in MT can alter perception dramatically in the way predicted from the motion selectivity of these cells . These results demonstrate convincingly that the perception of motion direction is related to the activity in this area and might actually depend on the responses of only small groups of cells. It will be most interesting to see 74
76
1
1
1
1
4 5
33 3 1
Thus, the approach of correlating physiology with perception might give a clearer picture of the courses of cortical processing than the distributions of selectivity. Recent findings in alert monkeys are encouraging. Anomalous-contour responses are found in interstripes and thick stripes, but not in thin stripes {thought to process color), while orientation
632
118 Acknowledgements
selectivity is common even in thin stripes and length
We think Dorothea selectivity is equally frequent in all t h r e e sets^. Weniger tot Indications of shape-from-motion processing were correct/tie the English found to be most frequent in the thick stripes, while Friednch Heitger and direction selectivity was equally rare in all three sets Lukas Rosenthaler tor of stripes'"' 87 . The question is, what is achieved at providing the the higher stages of cortical processing, rather than computed contrast how m u c h of the recepbve-field properties of the borders of Fig. 18. lower stages is preserved? and Ernst Peterbans for modeling the background lamp. Concluding remarks We also think Physiological studies that explicitly used perceptual Refer 0. Bishop and Peter Himmond for phenomena for interpreting neuronal signals have comments and been discussed. These seem to mark a promising new cnHcism ot an earlier trend in vision research. A more representative version of the account of the attempts to relate perception to minuscript. physiology can be found in Ref. 88. In addition, the Supported bf Swiss discussion of the psychophysical and theoretical studHEGrant 3 939 Si. ies on contour processing is limited, and thus biased
by the scope of this review. Contours are necessary, even if they are subjective. Selected references 1 Marr, D. (1980) Phil. Trans, fl. Soc. London Ser. B 290, 199-218 2 Baumgartner, C . (1936) in Diseases ol the Nervous System: Clinical neurobiology (Vol. 2) (Asbiny, A. K., McKann, G- M. and Mr Donald. W. I . eds). pp. 804-815. Saunders 1 von der He/at, R. (1987) Rev. Physiol. Biochem. Pharmacol. 109. 69-150 4 Zeki. 5. (1963) Neuroscience 9. 741-765 5 Zeki. S. (1983) Neuroscience 9. 767-781 6 MDVshon, J. A.. Adelson, f. H., Gizzi, M. S. and Nevrsome, w . T. (1985) Exp. emn Pes. (Suppl. 11), 117-151 7 GalleMi, C . Battaglini, P. P. and Fatten. P. ftp. Brain Pes (in press) 8 Sakata, H.. Shibutani, H.. Kawano. K. and Harrington, r. t. 11985) Vision Res. 25, 453-163 9 Sakata. H». Shibutani. H. and Kasvano. K ( 1 9 8 3 ) / Neurophysiol. 49. 1364-1380 10 Komatsu. hi and Wurtz, R. H. (1988) I Neurophysiol. 60, 580-603 11 Nensome. W. T„ Wurtz, R. H. and Komatsu. R (198B1 I. Neurophysiol 60, 604-620 12 Komatsu R and Wurtz. R. H. (198B) I Neurophysiol. 60. 621-644 13 Dursleler. M. R. and Wurtz. Ft. H. ( 1 9 8 8 U Neurophysiol. fa. 940-965 14 Allman, )., Miezm, F and MeGuinness. E. (1985) Perception 14. 105-126 15 Tanaka, K. etal ( 1 9 8 6 ) / Neurosci. 6, 134-144 16 Frost. 8. ).. Scilly. P. t. and Wong, S. C. P. (1961) Cup. Brain Res 43. 173-185 17 Orban, G. A., Gulyas. 8. and Vogels. R. (1987) J. Neurophysiol. 57. 1792-1812 I B Julesz. 8. (1971) foundations ol Cyclopean Perception. University ol Chicago Press 19 Poggio, G. F. 11964) in Dynamic Aspects ol Neocortical Function (Edelman. G. M.. Gall, W. E- and Cowan. W. M„ eds). pp- 613-635. Wiley 20 Sakata. R . Shibutani. H. and Kawano. K. (19B0) I. Neurophysiol. 43. 1654-1672 21 De Valois, K. K-, Dt Valois. R. t. and lund. E. W. (1979) J. Physiol. 291. 483-505 22 Maffei, L . Morrone. C . Pirchio, M. and Sandini, G- (1979) 1 Physiol. 296, 57-17 23 Hubel. D. R and Wiesel. T. N. (1962) J. Physiol. 160. 106-154 24 Hubel, D. H. and Wiesel. T N. (1968) I. Physiol. 195, 215-243 25 Marr. D. and Hildreth. F. (I960) Proc. R. Soc. London Ser. B 207, 167-217 26 Poggio, T„ Gamble. E. B. and Little. J. ). (1968) Science 242, 436-440 27 Schumann, F. (1900)7 Psychol. 23. 1-32 28 Kanlisa, G. (1979) Organization in Vision. Essays on Cestalt
633
Perception. Piaeger 29 Petry, S. and Meyer, 0. E, (1987) The Perception ol Illusory Contours, Springer 30 Lasvson. R. B. and Gutek. W. L (1967) Vision Pes 7. 271-297 31 von der Heydt, Ft., Peterhans. E. and Baumgartner. G. (1984) Science 224, 1260-1262 32 Peterhans, E., von del Heydt. R. and Baumgartner. G- (1986) In Visual Neumscience (Pettigrew, ). D., Sanderson. K. J. and Levick, W ft., eds), pp. 343-351. Cambndge University Press 33 von der Heydt, ft. and Peterhans. E (1989) J. Neurosci 9. 1731-1748 34 Peterhans, E. and von der Heydt. R. (1989) / Neurosci. 9. 1749-1763 35 Petry, S-. Harbeck, A.. Conway. J. and tevey. J. (1983) Percept. Psycfiopfiyi. 34, 169-174 36 Fuld. k. and O'DonneH. K. (1984) in Sensory frpenence. Adaptation, and Perception .Feslschnlt lor Ivo Kohler (Spillmann. L. and Wooten. 6. R., eds). pp. 461-469. Lawrence Erlbaum Associates 37 Spillmann. L. (1975) rn Cestalttheorie der Modernen Psychotogie (Ertel. 3., Kemmler. L. and Sladler. M., eds), pp. 210-218. Stemkopf 38 Kennedy. J. M (1976) Perception 7, 605-607 39 von der Heyrlt, R. and Peterhans, C. (1989) in Neural Mechanisms of Visual Perception (Lam, O. M. K. and Gilbert, C. D„ eds). pp. 157-170. Portfolio Publishing 40 Peterhans, E. and von der Heydt. P. (1989) in Seeing Contour and Colour (Kuhkowski. ). J.. Dickinson. C. M. and Murray, I. J.. eds), pp. 125-130, Pergamon 41 Hammond, P. and MacKay. D. M. (19B3) t. Physiol 337. 69-87 42 Hammond, P. and MacKay, D M (1985) I. Physiol 359. 315-329 43 Gregory. R L. (1972) Nature 23B. 51-52 44 Bravo, M.. Blake, R. and Morrispn. S. (1986) Vision Res 28, B61-865 45 De Weerd, P , Vandenbussche. E.. De Bruyn. 8. and Orban, G. A (1990) Behav. Brain Res. 39, 1-17 46 Redies. C., Crook, I. M. and Creutzfeldt. O . 0. (19S6) ftp. Brain Pes. 61. 469-181 47 Gregory, R. L. (1987) in The Perception of Illusory Contours (Petry. S. and Meyer. G. E . eds). pp. 61-89. Springer 46 Rock, I. and Anson, ft. (1979) Perception B. 665-681 49 Reynolds. R I (1961) Perceptron 10, 107-115 50 Cellatly. A. R. H. (I960) Perception 9, 599-602 51 Smith. A. and Over. P. (1975) Nalure 257, 561-582 52 Paradiso. M. A.. Shimojo. S. and Nakayama. K. (1989) Vision Res. 29, 1205-1213 53 vogels, R and Orban, G. A. (19B7) Vision Res 27. 453-167 54 Peterhans. 6. and von der Heydt. ft. (1989) Soc. Neurosci. ADstr. 15. 161 55 Hubel. D. H. and Livingstone. M. S. (1987) ;. Neurosci. 7, 3378-3415 56 Bngner, W. L. and Gallagher. M. B. (1974) Perceptual and Motor Skills IS. 1047-1053 57 Marr, D. (1962) Vision. A Computational Investigation into Ihe Human Representation and Processing of Visual Information, Freeman 56 Van Essen, D. C- (1985) in Cerebral Cortex (Vol 3)(Peters.A. and Jones. E. G . eds). pp. 259-329, Plenum 59 von der Heydt. P. and Peterhans. E. (1989) in Seeing Contour and Colour (Kuhkowski. I. J.. Dickinson. C. V . and Murray, L J . . eds), pp. 729-734, Pergamon 60 Grossberg, S. and Mingolia, E. (19B7) in 7ne Perception ol Illusory Contours (Petiy, S. and Meyer, G. E., eds). pp. 116-125. Springer 61 Finkel. L. H. and Edelman, G M. (1989) J Neurosci. 9. 318B-3208 62 Spillmann. L.. Fuld. K. and Neumeyer. C. (1984) Perception 13.513-520 63 Treisman, A (1988) O. ( Ftp. Psychol IA) 40. 201-237 64 Gurnsey, R. and Humphrey, G. X. (1990) Invest Ophthalmol. VisualSci. 31, 107 (AP.VO suppl.) 65 Grabowecky, M. and Treisman. A. (1989) Invest. Ophthalmol. Visual Sci. 30, 457 (ARVO suppl.) 66 Harris,). P. and Gregory, ft. L. (1973) Percepfiorr 2, 235-247 67 Coren. S (1972) Psychol Rev 79. 359-367 68 Vonas. A.. Craton, L. G- and Thompson, w . 8. (1987) Percept. Psychophys. 41. 53-59 69 Michotte. A.. Thines. G. and Crabbe. G. (1964) tes Comp¬ lements Amodaux des Structures Perceptives IStudia psycho-
119 logical, Publications Un^er5itaifff6 de Louv-ain 70 Kaplan, G. A. (1969) Percept Psychophys 4. 193-19B 71 Nakayama, K. and Shimojo. S (1989) m Neural Mechanisms of Visual Perception (Lam, O. M. K. and Gilbert, C. D.. eds), pp. 281-296. Portfolio Publishing 72 De Weert. C. M. M. and van Kruysbergen, N. A. W. H. (1987) in The Perceptron of Illusory Contours (Petry, S. and Meyer. G. E.. eds). pp. 165—170. Springer 73 Bradley, D. R.and Domais, 5 T. (1975) Nalure 257, 582-584 74 Sperling, H. G., Crawford, M. L. J. and Espinoia, S. (19781 Mod Probl. Ophthalmol. 19, 2-1B 75 Newsome. W. T. Britten. K. H.. Movshon. J. A. and Shadlen, M. (19B9) in Neural Mechanisms ol Visual Perception (Lam, D. M, K. and Gilbert. C. D . eds), pp. 171-198. Portfolio Publishing 76 Logolhetis, M K. and Schall. 1. D. (1989) In Neural Mechanisms ol Visual Perception (Lam, D. M. K. and Gilbert. C . D., eds), pp. 199-222, Portfolio Publishing 77 Ssriman, C. 0., Britten, K. H. and Newsnme, W. T. (1990)
78
r
79 80 BI B2 83 B4 B5 B6 87 8B
Invest Ophthalmol. Vision Sci. 3 1 , 238 (ARVO suppl.) Livingstone, M. 5. arid Hubel, D. H. ( 1 9 B 4 ) I Neurosci. 4, 309-356 Livingstone, M. 5. and Hubel, M. (1987) I. Neurosci. 1. 3371-3377 OeVoe. E. A. and Van Essen, O. C . (198B) Trend) Neurosci 11, 219-226 Zeki, S and Shipp. S. (19BS) Nature 335. 311-317 Shipp, 5 and Zeki. S (1989) fur. /. Neurosci 1. 333-354 Zeki, 5. and Shipp. S. (1989) fur. 1. Neurosci 1. 494-506 DeYoe, E. A. and Van Essen. D. C . (1985) Nature 317. 5 3 - 6 1 Levitt, J. B. and Movshon, 1. A. (1990) invest. OphttafrBOl. Vision Sci. 31. 89 (ARVO suppl.) Schem, S. J . . Marrocco. «. 1. and Oe Monasterio, F. M. (1982)7. Neurophysiol. 47, 193-213 Peterhans. E and von der Heydt. B ( 1 9 B 9 ) Invest. Ophthalmol. Vision Sci. 30. 299 Spillmann, L. and Werner, I. S. (1990) VituaV Perception The Neurophysiological Foundations, Academic
ERRATUM In the article Dopaminergic innervation of the cerebral cortex: unexpected differences between rodents and primates' by B. Berger. P. Gaspar and C. Vemey (January 1991, Vol. 14. pp. 21-27) an error was introduced to the colour shading of part of Fig. 1. The DA cell groups A9 and A10 shown in the section at the bottom of the Figure were given the wrong shades of blue. We reproduce the correct Figure here and apologise to the authors and readers for this error.
1
7 tern
6
A10-A?
European Neuroscience Conference A meeting on K* channels and neuronal mechanisms will be held at Chateau de Fillerval. Mouy, France from 24-28 November, 1991. A maximum of 65 attendees, in addition to 25 invited speakers, will be selected. All expenses for participants from Eastern European countries will be met by the organizers. In addition, one or more fellowships will be offered to Eastern European participants on a competitive basis to enable them to visit a Western European laboratory involved in research on the topic of the meeting. Interested applicants should contact the organizer, Dr Y Ben-Ari. INSERM U-29, Hopital de Port-Royal, 123 Bd de Port-Royal, 75674 Paris Cedex 14, France as soon as possible with a brief c.v.. publication list and area of interest (two pages maximum).
634
4c Structures and Functions of Various Brain Areas
Much emphasis has been laid, in the previous sections of this chapter, on the primate visual cortical system, for good reasons — as it should, by now, be amply clear. Nevertheless, it would be quite misleading to ignore the variety of other brain areas that subserve other important functions. The first article (49) of this section is a review written by Walle Nauta, who is one of the influential neurobiologists of this century. The question raised in his title is lucidly and cautiously answered in the conclusion. The main part of the paper will perhaps appear arid to some readers, but it is a good lesson on the experimental tools that allow one to recognise the identity and boundaries of a brain area, and to trace its afferent and efferent innervations. T h e identification of four "return loops", involving the basal ganglia, the limbic system and the neocortex, leads to suggestions concerning some motor aspects of thought: initiation of effort and timing. Article (50), by Braitenberg, attempts to formalise the structure of the cerebellum, briefly evoked earlier in article (23), and later in article (65). The basic neuroanatomical data have been known for a long time, yet the precise functional role of the cerebellum in motor control remains shrouded in mystery. T h i s paper makes attractive suggestions, underlining a capacity of the cerebellar network for monitoring accurate time delays, in relation to the general need, in large vertebrates, for counteracting inertial forces. Article (51), by McNaughton and Morris, describes the hippocampal formation in mammals at anatomical, physiological and behavioural levels. It examines the hippocampus role in memory storage, especially spatial learning in rats. A useful review of correlation matrices for associative memories, and of the synaptic plasticity accompanying long-term enhancement, brings the analysis of behavioural physiology down to the level of neural network implementation. Further analysis of the hippocampus appears in article (59).
635
Reprinted with permission from Neurology and Psychiatry: A Meeting of Minds, pp. 43-63, 1989 © 1989 Karger, Basel
Reciprocal Links of the Corpus striatum with the Cerebral Cortex and Limbic System: A Common Substrate for Movement and Thought? WalleJ. H. Nauta Department of Psychology and Brain Science, Massachusetts Institute of Technology, Cambridge, Mass., U S A
Broadly speaking, the cerebral hemisphere of mammals can be said to be composed of three major anatomical and functional realms: the neocortex, the limbic system, and the extrapyramidal system (also referred to as basal ganglia or corpus striatum). Traditionally, and not without good physiological and clinicopathological reason, the neocortex is considered to embody the highest level of sensory analysis and perceptual integration, as well as certain of the more differentiated mechanisms subserving somatic motor function. Last but not least, it is regarded as the staging ground for ideational processes. The limbic system, by contrast, is generally viewed as the main cerebral representative of the internal milieu, expressing its functions, in part at least, in the form of affects and motivation. Andfinally, the basal ganglia are generally considered to be essential components of the brain's somatic motor system. Useful as the notion of a tripartite composition has been (and still is) as a broad-scale first approach to the functional organization of the cerebral hemisphere, recent findings suggest that it may require some qualification. Among the more compelling reasons for revision is an anatomical one: observations accumulated over the past decade or so indicate that the three subdivisions are more closely interconnected than they once seemed to be. More in particular, several places have been identified in which their respective circuitries converge and thus could lead into communal or parallel effector channels. One, and apparently the most extensive of such confluences, occurs in the striatum. In recent years its remarkable pattern has begun to raise the question whether the extrapyramidal system, beside its well-documented role in skeletomotor function, could play a part in perceptual and ideational processes as well [Teuber, 1976].
636
44
In the following account the anatomical foundation of this notion will be dealt with in some detail. No comprehensive survey of extrapyramidal circuitry is intended here. For more encompassing reviews of that circuitry reference can be made to a recent Ciba Foundation symposium [Evered and O'Connor, 1984]. The striatum, however, being so much of a nodal intersection of cortical, limbic and extrapyramidal circuits, will be given somewhat more extensive coverage.
General Terminological Remarks The term, extrapyramidal system, introduced by S. A . K . Wilson in 1912, has never been satisfactorily defined anatomically. Although literally the term can be interpreted so as to denote all of the brain's suprasegmental motor mechanisms not transmitting their discharge by way of the pyramidal tract, convention over the years has made it nearly synonymous with the basal ganglia and their efferent connections. The much older term, basal ganglia, originally subsumed all of the gray masses at the base of the cerebral hemisphere, including even the thalamus, but gradually became restricted to the corpus striatum (striatum and globus pallidus), and only later came to include also the subthalamic nucleus and substantia nigra, both linked to the corpus striatum by reciprocal fiber connections.
The Striatum A survey of extrapyramidal circuitry most conveniently begins with the striatum, for it is this structure rather than the globus pallidus (or pallidum) that receives the principal extrinsic afferents converging upon the basal ganglia. In all primates and many nonprimate mammals the striatum is subdivided by a plate-like internal capsule into two main districts, the dorsomedially situated caudate nucleus and the ventrolateral putamen. In other mammalian forms, including the rat, the internal capsule passes through the striatum in the form of a brush rather than a plate; in such species the striatum lacks an overt bisection and is therefore often referred to as caudatoputamen. In either case, however, the anterior part of the striatum includes two further subdivisions that require separate mention.
637
45
Fig. 1. Semidiagrammatic representation of three limbic system-associated projections to the striatum in rat brain: the dopamine projection from the ventral tegmental area (vertical lines), the projection from the frontal and cingulate cortices (horizontal lines), and the projection from the amygdala (stipple pattern). Not included in the figure (to avoid crowding) is the projection from the hippocampal formation to the nucleus accumbens. Left blank in the more rostral section (a) is the anterior, dorsolateral striata! region receiving only sparse limbic afferents but heavily projected upon by the sensorimotor cortex (cf. fig. 2). N A = Nucleus accumbens; O T = olfactory tubercle; G P = globus pallidus. Reproduced from Kelley et al. [1982] with permission of Pergamon Press, Oxford.
One of these, the ventromedial region nearest the base of the septum, and known as the nucleus accumbens septi (fig. l a ) , was the first striatal region found to receive afferents from the limbic system, namely, fibers from the precommissural fornix [Fox, 1943; Sprague and Meyer, 1950; Swanson and Cowan, 1977]. On its ventral side the accumbens region extends into the olfactory tubercle, providing the latter with its core population of striatal cells [Heimer, 1978]. All districts of the striatum exhibit basically the same cytoarchitecture. Throughout its extent, more than 98% of its neurons are small to mediumsized, while the remaining 1-2% are large, cholinergic, multipolar cells that were long regarded as the source of the striatum's extrinsic efferents but are now believed to be intrinsic, or 'local-circuit' neurons. On account of its lack of any broad-scale regional differentiation obvious on cursory inspection of preparations stained by routine survey
638
46
techniques, the striatum has traditionally been described as a homogeneous structure. More recent studies, however, have disclosed a more covert, smaller-scale stereometric pattern of compartition that subdivides the striatum into an intricate labyrinth of small compartments ('striosomes') embedded in a more homogeneous cell matrix, and distinguished from the latter by peculiarities of histochemistry and cell-packing density [Graybiel and Ragsdale, 1983; Graybiel, 1984], The relationship of this mosaic to the intrastriatal distribution of extrinsic afferents, to circuits intrinsic to the striatum itself, and to the sites of origin of striatal efferents is currently under intensive investigation. The findings reported thus far strongly suggest that the mosaic reflects a spatial ordering of striatal afferents and efferents of various neurotransmitter and/or neuropeptide content [Graybiel, 1984]. It appears likely, moreover, that intrinsic striatal neurons may provide an orderly system of cross-links between the striosomal and the matrix compartment, and thus between various classes of afferent and efferent conduction lines.
Afferent Connections of the Striatum The striatum, the main entrance portal for neural inputs to the basal ganglia, receives its most voluminous afferent connections from the entire expanse of the cerebral cortex (including the hippocampal formation and the olfactory cortex), from the intralaminar thalamic nuclei, and from the substantia nigra (including the dopamine neurons of the ventral tegmental area). Additional substantial inputs originate from the amygdala, as well as from the serotonin neurons of the dorsal raphe nucleus of the midbrain. It is important to note that several of these afferent channels come from major limbic structures (hippocampal formation, amygdala, and cingulate cortex), whereas others arise from structures implicated in the subcortical circuitry of the limbic system: ventral tegmental area and dorsal raphe nucleus. The afferents from the cerebral cortex are distributed in the striatum in an intricate stereometric pattern that by and large preserves the topology of the cytoarchitectural subdivision of the cortical mantle but is considerably less sharply defined than the latter. Moreover, as will become evident below, the intrastriatal distribution of the projection from the neocortex as a whole must widely overlap that of projections from the limbic
639
47
1
system, especially in the caudal half of the striatum. A somewhat more detailed accounting of the afferentation of the striatum by the limbic system is necessary at this point. It is unfortunate that such an accounting can at present be given only for the rat brain. Even in that species, moreover, no complete description of the pattern of the corticostriatal projection can be given.
The Limbic Innervation of the Striatum Recent studies in the rat by the aid of the autoradiographic fibertracing method [Kelley et al., 1982] have led to the conclusion that the anterior half of the striatum (the large part rostral to the level at which the anterior commissure crosses the midplane) can be subdivided into (a) a ventral and medial region including, but far exceeding, the nucleus accumbens and olfactory tubercle, and (b) a dorsolateral region. The reason for the distinction is that the former, anteroventromedial striata! subdivision is characterized by being densely innervated by several substantial and partially overlapping afferent fiber systems, each originating either from a major component of the limbic system such as the hippocampal formation, amygdala, or cingulate cortex, or from a structure involved in the circuitry of the limbic system, such as the ventral tegmental area and the prefrontal cortex. The remaining, anterodorsolateral striatal sector, by contrast, receives only sparse inputs of limbic origin and is, instead, massively innervated by the sensorimotor cortex. The subdivision of the anterior striatum here described is illustrated by figure l a in which the limbic-afferented striatal region (for the sake of convenience to be called 'limbic striatum') is indicated by a variety of line and stipple patterns. The dorsolateral, 'nonlimbic' striatal sector heavily innervated by the sensorimotor cortex has been left open in the same drawing. At more caudal levels, as shown by figure l b , the striatal region in which multiple limbic afferents overlap retains its medial position, but at these levels no 'nonlimbic' sector can be delineated, for the reason that the massive amygdalostriatal projection (indicated by stippling) here expands over the entire
1
As here used, Ihe term, overlap, solely refers to a convergence of afferent lines upon a large volume of striatal tissue; it carries no implications regarding communal or differential termination within either Ihe matrix or striosomal compartment of the striatum.
640
fig. 2. Radioactive fiber labeling in the rat striatum following an injection of tritiated leudne-and-proline in the sensorimotor cortex. The injection site is shown in jet black in b. Al the rostral levels shown in a and bnote that the bulk of the projection involves the dorsolateral, relatively "nonlimbic' striatal sector (led blank in figure la). Farther caudally Id. e) the motocortical projection overlaps the projection from the amygdala (cf. fig. lb), gp = Globus pallidus Reproduced from Kelley et al. [1982| with permission of Pergamon Press, Oxford.
641
49
cross-section of the striatum, and thereby overlaps the caudal extent of the corticostriatal projection arising from the motor cortex. (As shown by figure 2, the projection from the motor cortex continues from its major distribution area, a-b caudally and ventrally far enough to involve the entire antero-posterior dimension of the striatum.) The evidence that the striatum - or at least its rostral half - comprises a limbic-innervated and a 'nonlimbic' district raised the question whether this dualism might not be reflected somehow in the distribution of the massive striatofugal projection to the globus pallidus. In more general terms, the question would be: Could the two striatal districts be the respective origins of two separate and distinct striatal exit lines to the pallidum? Further study produced an affirmative answer. Comparison of a large number of cases, in each of which a small injection of tritiated leucine had been placed in one or the other striatal locus and the resulting anterograde fiber labeling had been recorded autoradiographically, clearly showed that the limbic-afferented anteroventromedial striatal region indicated in figure la projects exclusively to an only recently identified ventral component of the globus pallidus, the so-called ventral pallidum (vp). The 'nonlimbic' striatal sector, by contrast, projects to a larger pallidal subdivision, here to be called dorsal pallidum (dp) that forms most of the globus pallidus as conventionally outlined in atlases of the rat brain. This subdivision of the globus pallidus into two distinct parts needs some further comment.
The Ventral Pallidum The vp was first described by Heimer and Wilson [1975] in the rat as an extension of the globus pallidus, stretching ventrally and rostrally beneath the anterior commissure. Its identification as a pallidal subdivision of the infracommissural forebrain region was originally based on cytological characteristics, as well as on the finding that it receives a massive striatal projection originating in particular from the nucleus accumbens [Wilson, 1972; Williams et al., 1977; Nauta et al., 1978]. Its pallidal nature was further documented by evidence that it also shares with the main mass of the globus pallidus a substantial projection from the subthalamic nucleus [Ricardo, 1980] as well as the histochemical characteristics of a high iron content [Switzer and Hill, 1979; Hill and Switzer, 1984] and a strong
642
50
enkephaliri-like [Haber and Nauta, 1981, 1983; Switzer et al., 1982] and glutamic acid decarboxylase-like immunoreactivity [Fallon and Ribak, 1980]. Despite these similarities, the vp must be viewed as distinct from the main mass of the globus pallidus. A principal reason for this distinction is histochemical: Whereas both the vp and the main mass of the globus pallidus are infiltrated by a dense plexus of enkephalin-positive fibers, the vp is pervaded by an additional and almost equally dense plexus of substance-P-positive fibers that extends over only a short distance into the main pallidal mass. For a proper understanding of the criteria followed in defining the vp for the purposes of the present account, a brief elaboration of this last point is necessary. It is important to note that intrapallidal enkephalin and substance P are not intrinsic to the globus pallidus proper, but are associated with the striatopallidal projection. As first observed by Fox et al. [1974] in Golgi material, striatopallidal fibers enmesh the dendrites of pallidal neurons individually in sleeve-like plexuses. The characteristic histological picture reflecting this junctional arrangement in material immunoreacted for enkephalin ['woolly fibers', Haber and Nauta, 1983] provides a lightmicroscopic criterion for determining the extent of the globus pallidus. In such material the dense plexus of enkephalin-positive 'woolly fibers' filling the main mass of the globus pallidus is seen to continue without interruption ventrally into the infracommissural region where it extends rostrally over a wide area bordering the ventral side of the striatum and bed nucleus of the stria terminalis [Switzer et al., 1982; Haber and Nauta, 1983]. Interestingly, such material also shows that this infracommissural plexus extends around the lateral border of the nucleus of the diagonal band into the deep, fibrocellular layer of the olfactory tubercle [Switzer et al., 1982; Haber and Nauta, 1983]: this is in agreement with an earlier suggestion, made by Heimer [1978] on cytoarchitectural grounds, that the tubercle is a striatopallidal complex, as well as with the description by Ribak and Fallon [1983] of large cells associated with the islands of Calleja and electron microscopically indistinguishable from pallidal neurons. Since the density of the enkephalin-positive intrapallidal fiber plexus varies little throughout its extent, it cannot serve to delineate the vp from the main, supra- and retrocommissural mass of the globus pallidus. Such a demarcation is, however, possible in sections processed for substance-PLike immunoreactivity. Such material revealsa dense plexus of substance-Ppositive 'woolly fibers' that, with minor exceptions, is coextensive with the
643
51 infracommissural enkephalin-positive plexus. But, unlike the latter, it extends over only a short distance into the main pallidal mass, then abruptly declines in density and only sparsely extends through the large remainderof the globus pallidus. The curved line along which this decline in density of the substance-P-positive 'woolly-fiber' plexus occurs suggests itself as the border between the vp and the remaining part of the globus pallidus, here for purposes of contrast to be referred to as dp [Haber and Nauta, 1981; cf. also the present fig. 3 in which the extent of the substance-P-positive plexus is indicated by a stippled pattern]. In summary, the vp is here defined as the largely infracommissural region massively pervaded by enkephalin-positive and substance-P-positive (and probably additional) striatofugal fibers terminating in the peridendritic-plexus fashion characteristic of the striatopallidal projection. One further peculiarity of the vp must be mentioned here. The region corresponds to that part of the globus pallidus which contains the densest population of large, cholinergic neurons (fig. 3). Such cells are more sparsely scattered throughout the remaining parts of the globus pallidus; collectively, they form the intrapallidal component - or subgroup C h 4 [Mesulam et al., 1983] - of the magnocellular nucleus of the basal forebrain. 2
Differential Projections from Dorsal and Ventral Pallidum The evidence that the subdivision of the striatum into a limbicafferented and a 'nonlimbic' district is mirrored in a ventral versus dorsal dualism in the intrapallidal distribution of the striatopallidal projection raises the question whether the separateness of the two striatal exit lines might not extend even beyond the globus pallidus. To investigate this question, a small deposit of tritiated leucine was placed in either the dorsal or the vp in a large number of rats; the distributions of fibers radioactively 1
In the human and other primates the magnocellular nucleus of the basal forebrain forms a large, compact group of large cholinergic cells, the basal nucleus of Meynert, located in the substantia innominata just ventral to the globus pallidus. Even in these more highly differentiated species, however, trails of cholinergic cells extend from the basal nucleus dorsally into the medullary laminae of the overlying globus pallidus. In rodents and carnivores the cholinergic cells of the magnocellular nucleus are more widely and diffusely distributed, and lack the circumscript local aggregation that enabled Meynert to recognize a basal nucleus in the human brain.
644
52
fig. 3. Drawings of two frontal sections at the level of, respectively, the anterior commissure (a) and rostral part of the retrocommissural globus pallidus fb). The extent of the dense substance-P-positive fiber plexus, coextensive with the vp of Heimerand Wilson [1975], is indicated by stippling (cf. Haber and Nauta, 1983]. Coarser black dots each represent a strongly acetylcholinesterase (AChE)-positive intrapallidal neuron; together, these neurons compose the intrapallidal subdivision, celled subgroup Ch4 by Mesulam et al, [1983], of the magnocellular nucleus of the basal forebrain. The numerous further cells of the magnocellular nucleus, distributed in the nucleus of the diagonal band and other sublenticular structures [cf. Butcher, 1983; Mesulam et a l . , 1983] have been omitted in these drawings. Note that such cells are much sparser in the dorsal (dp) than in the vp. A C " Anterior commissure; bns = bed nucleus of stria terminalis; C I " capsula interna; C P -
caudatoputamen; dp •
dorsal
pallidum; F = fornix; S M = stria medullaris. Reproduced from Haber et al. [1985] with permission from Alan R , Liss, New York.
Fig. 4. Semischematic illustration of the two partly segregated striatofugal conduction lines described in the text, a 'Limbic' (shaded) and 'nonlimbic' striatal subdivisions and their respective main telencephalic afferents. b The striatofugal line leading from the motor cortexinnervated 'nonlimbic' striatal subdivision to the dorsal pallidum (heavy arrow) which in turn projects to the subthalamic nucleus and substantia nigra, c The line leading from the limbicafferented striatal subdivision to the vp (heavy arrow) and from there not only to the subthalamic nucleus and substantia nigra but also (o the frontocingulate sensorimotor
cortex, lateral habenular and mediodorsal
and medial
thalamic nucleus, amygdala,
hypothalamus, ventral tegmental area and more caudal regions of the midbrain tegmentum, abl = Amygdala; A C = anterior commissure; C C = cerebral peduncle; cgs = central gray substance; C I ™ capsula interna; dp — dorsal pallidum; ep = en to peduncular nucleus (internal segment of globus pallidus); F C C = frontocingulate cortex; H = hippocampus; hi = lateral habenular nucleus; ht - hypothalamus; md = mediodorsal nucleus; na " nucleus accumbens; rt = reticular nucleus of thalamus; S M C - sensorimotor cortex; sn - substantia nigra; st subthalamic nucleus; vp = ventral pallidum; vta ™ ventral tegmental area. Reproduced from Haber et al. [1985] with permission from Alan R. Liss, New York.
645
646
54
labeled by each of these isotope placements were then recorded autoradiographically, carefully charted, and compared. Since a documented account of the findings has been published previously elsewhere [Haber et al., 1985], only the main conclusions will be reported here. The results of the study showed pronounced differences between the exit lines from the dorsal and vp, respectively. The findings are illustrated by figure 4 and can be summarized as follows: (1) The efferent connections of the dorsal pallidum (fig. 4b) compose the classical projection from the external pallidum segment and involve primarily the subthalamic nucleus and substantia nigra. This pallidal exit line appears to lie in the extension of the striatopallidal projection arising from the 'nonlimbic' dorsolateral striatal sector (heavy arrow in fig. 4b), the striatal region that in turn receives its main telencephalic input from the sensorimotor cortex (fig. 4a). (2) The vp projects, like the dp, to the subthalamic nucleus and substantia nigra, but it has further projections to several structures implicated in the circuitry of the limbic system, particularly the amygdala, the lateral habenular nucleus, and the mediodorsal nucleus of the thalamus (fig. 4c). It projects more sparsely to the hypothalamus, to the ventral tegmental area, and to more caudal regions of the midbrain tegmentum, including the region of the pedunculopontine nucleus. Of some special interest are those projections of the vp that ascend to the prefrontal cortex and an adjoining region of the sensorimotor cortex: There is evidence suggesting that this projection, as well as the one to the amygdala, originates largely, at least, from the cholinergic cells of the so-called magnocellular basal nucleus that lie embedded in the vp [Grove et al., 1983]. All these exit lines from the vp, however, appear to lie in the transsynaptic extension of that component of the striatopallidal projection (heavy arrow in fig. 4c) which originates from the limbic-afferented striatum, that is to say, the large striatal region densely innervated by a variety of afferents from within the circuitry of the limbic system (the shaded region in fig. 4a).
Discussion One of the principal realizations to emerge from the anatomical findings of the past few years is that the striatum, a common target for voluminous projections from the cerebral cortex and the limbic system, has
647
55
multiple exit channels available that could serve as 'return loops' to the cerebral cortex. Up until about ten years ago only one such loop was generally recognized: the extrapyramidal motor circuit leading from the striatum over, successively, the pallidum and the thalamic V A - V L complex back to areas within the motor domain of the neocortex. Naturally, the discovery of this loop (a step-wise disclosure that was not completed until 1945; cf. Nauta and Mehler, 1966] added further support to the prevalent view of the basal ganglia as a central somatic motor mechanism. Evidence of further trans-striatal loops to the cortex, not specifically targeted on the motor cortex, is of more recent date. The first such loop to be recognized turned out to lead from the striatum over the pars reticulata of the substantia nigra -rather than over the globus pallidus proper - to the ventromedial nucleus of the thalamus and from there to most of the extent of the neocortex [Cole et al., 1964; Carpenter et al., 1976; Herkenham, 1976]. A second potential return loop to the cortex was found to lead from the striatum over the globus pallidus to the reticular nucleus of the thalamus [Nauta, 1979a; Haber et al., 1985], a thalamic cell group that is unique in the sense that it lacks a projection to the cortex and instead projects to many - i f not indeed all - other thalamic nuclei [Scheibel and Scheibel, 1966]; it thus could be suspected of being able to affect the functional state of virtually the entire expanse of the cerebral cortex by modulating the latter's inputs from the thalamus. (The pallido-reticular connection is schematically indicated in fig. 4b.) 3
It must be emphasized that neither of these two striato-pallidothalamo-cortical return loops is at present known to be directed at any functionally specified cortical area in particular. On the contrary, both seem likely to involve most if not all of the neocortex. Since it appears to be unknown whether they originate throughout the striatum or from any particular striatal subdivision, nothing can be said at present concerning the functional nature of the corticostriatal input the two loops could 'feed back' to the cortex. This is not as categorically true for the two cortico-striatopallido-thalamo-cortical circuits to be discussed next. A distinction between the latter two circuits became possible only after Heimer and Wilson [1975] had recognized the vp, and Kelley et al. [1982] had noted that the anterior half of the striatum comprises a predominantly
5
On both anatomical and physiological grounds the pars reticulata of the substantia nigra can be interpreted as a caudatmost component of the globus pallidus [cf. Nauta, 1979b; DeLong and Georgopoulos, 1981).
648
56
motor cortex-innervated (or 'nonlimbic') and a convergently limbicinnervated subdivision. The evidence presented in a foregoing section of this paper indicates that the limbic versus nonlimbic dichotomy in the innervation of the striatum reflects itself in the existence of two largely segregated striatofugal conduction tines. The present evidence of this dualism is schematically illustrated by figure 4, and can be summarized as follows. A dorsolateral sector of the anterior striatum receiving only sparse afferents of limbic origin but instead heavily innervated by the sensorimotor cortex [Kelley et al., 1982] projects to an antero-dorsolateral region of the main, supra- and retrocommissural (i.e. conventionally recognized) mass of the globus pallidus; this pallidal region ('dorsal pallidum') in turn gives rise to the classical pallidofugal pathway descending to the subthalamic nucleus and substantia nigra (fig. 4b). By contrast, the heavily hmbic-innervated antero-ventromedial striatal region (shaded in fig. 4a) projects to the largely infracommissural, but partly also supra- and retrocommissural vp (fig. 3), from which in turn efferents pass not only to the subthalamic nucleus and substantia nigra but also to various limbic system-associated structures, in particular the amygdala, frontocingulate (and adjoining sensorimotor) cortex, mediodorsal thalamic nucleus, lateral habenular nucleus, hypothalamus, ventral tegmental area, and more caudal and dorsal regions of the midbrain tegmentum (fig. 4c). As thus defined, the two striato-pallidofugal lines share the subthalamic nucleus and substantia nigra as common distribution targets. If it is assumed that both these target structures are first and foremost components of the somatic motor system, the present findings suggest both as sites where central skeletomuscular mechanisms are accessible to striatopallidal output of limbic antecedents, conveyed by the line leading from the limbic afferented striatum over the vp. It is likely that further sites of such access lie farther caudally in the midbrain. A small part of the great exit pathway from the globus pallidus (the ansa lenticularis), instead of recurving dorsally into the thalamic V A - V L complex, continues on a straight course into the midbrain tegmentum where it terminates mainly in the region of the so-called pedunculopontine nucleus. On the basis of physiological findings Grillner and Shik [1973] have referred to this tegmental area as the 'mesencephalic locomotor region'. Interestingly, fibers from the region of the vp have been found distributed here as well; Swanson et al. [1984] have interpreted this finding as evidence of a limbic innervation of Grillner and Shik's somatic motor mechanism.
649
57
A third route by which the limbic system could be thought to have access to central somatic motor mechanisms could lead from the Hmbicinnervated striatum over the strongly AChE-positive cells embedded in the vp. The present anterograde evidence suggesting that at least some of these cells project to the sensorimotor cortex only confirms earlier findings in retrograde-labeling experiments [McKinney et al., 1983; Saper, 1984], Even more difficult to interpret from a functional point of view are the limbico-striato-pallidofugal lines leading to various limbic and limbicsystem-associated structures, summarized in figure 4c. From one vantage point, these connections could be viewed collectively as a differentiated 'return loop' enabling the limbic system to monitor the effect of its outputs to the striatum (and thus possibly acting as an aid in adjusting the organism's motivational set to the output efficacy of the central somaticmotor system). But other possibilities must be considered. The question arises, for example, if one significance of the much-interrupted Hmbicostriato-pallido-limbic circuit here proposed may not lie in the fact that the striatum is the site of an unusually wide variety of intricately interspersed neurotransmitters and neuromodulators [Graybiel and Ragsdale, 1983], particular combinations of which might be required no less stringently for optimal functioning of limbic and cognitive-cortical mechanisms than the same or other combinations are required for optimal somatic-motor function. The possibility of such a more general functional role of the corpus striatum suggests itself from a remarkable parallel between the 'limbic' and 'nonlimbic' trans-striatal conduction lines: both prominently include channels over which part of their impulse flow can be led back to the domain of its origin in the cerebral hemisphere. To be more explicit: the familiar 'extrapyramidal motor circuit', motor cortex - » striatum —* pallidum —»thalamic V A - V L complex —* premotor cortex - * motor cortex (fig. 5a) appears to have limbic counterparts in the sequences: limbic system ^ striatum—»vp—» amygdala, and: limbic system —> striatum —»vp —* thalamic mediodorsal nucleus —» frontal and anterior limbic cortex (fig. 5b). If it is considered likely on the basis of long-known anatomical connections that the motor function of the corpus striatum expresses itself in part at least through a major transthalamic circuit affecting the mechanisms of the motor cortex, the more recent anatomical data here reviewed make it appear no less likely that the corpus striatum, through analogous though less massive circuits involving the amygdala and prefrontal cortex, can affect a class of functions that is cognitive rather than primarily skeletomuscular.
650
58
Fig. 5. Semidiagrammalic illustration of the two cortico-striato-paIIido-thalamo-cortical circuits associated with, respectively, the motor cortex (a) and the frontocingulate cortex (b). Note that the extrapyramidal motor circuit shown in a sequentially involves the 'nonlimbic" striatum, dp as well as internal pallidal segment (ep), and thalamic V A - V L complex (va). By contrast, the analogous circuit associated with the frontocingulate cortex, shown ini>, involves the 'limbic' striatum, vp and thalamic mediodorsal nucleus (md); note also the shorter loop
651
59
General Comment The recent evidence that the corpus striatum has various transthalamic exit lines available over which it could be thought to modulate the function not only of the motor cortex, but of most or all of the remaining cortical areas as well, does not come entirely unexpected. That the basal ganglia are implicated not only in motor but also in cognitive functions had been suspected for some time from findings in neurobehavioral experiments [Teuber, 1976], as well as from the frequent concurrence of extrapyramidalmotor and mental disorder. Most unambiguously perhaps, from detailed behavioral studies evidence had been reported of impaired frontal-lobe function in Parkinson's disease [Bowen, 1976] and, more recently, certain lesions of the striatum have been found to cause aphasia with marked impairment of language comprehension [Damasio, 1985]. Exactly which aspects of cognitive function could be influenced by the basal ganglia is at present unknown. One would be inclined to search for a functional attribute that is shared in common by cognition and movement, and affected in common by basal ganglia disorder. It is of interest in this connection to recall John Hughlings Jackson's vigorous argumentation against the notion, prevalent before the discovery of the motor cortex by Fritsch and Hitzig in 1870, that the 'convolutions' (i. e. the cerebral cortex) were the 'organ of mind', and thus exclusively for 'ideas', while 'movements' were represented solely in lower parts of the brain. Jackson's views are illustrated by the following quotations from his selected writings [Taylor, 1958]: 'But of what "substance" can the organ of mind be composed, unless of processes representing movements and impressions; and how can the convolutions differ from the inferior centres, except as parts representing more intricate coordinations of impressions and movements in time and space than they do?' [Hughlings Jackson, 1370). 'There is , . . proof, although indirect, of the opinion that faint and central excitation of sensori-motor processes suffices in thought' [italics as in original; Hughlings Jackson, 1872}.
associated with the amygdala (abl). To avoid excessive complexity of the diagrams, only one of several possible cross-links between the a and b circuits has been indicated; as shown in a, the subthalamic nucleus - innervated by both dp and vp-projects back to both dp and vp, as well as to the internal pallidal segment (ep). For further labels see legend to figure 4. Reproduced from Haber et al. [1985] with permission from Liss, New York.
652
60
It is, of course, difficult to say how literally Jackson's view should be taken. Is ideation really the internal expression of the same sensori-motor mechanisms which, when activated more forcibly, would manifest themselves as movements? On the other hand, it is easy to see at least parallels between thought and movement. To begin with, both seem to require a motivational vis a tergo to be initiated and to be kept in progress. It takes an effort to change from a state of relaxed daydreaming - a sort of resting tonus perhaps - to one of goal-directed thinking, and it is something like an act of endurance to keep such a thought process going. Movement and thought share with each other a need for 'feedback' allowing for corrective adjustments. Thought often spills over into the motor sphere, in the form of unintentional changes in body posture and facial expression, pacing the floor, or thinking aloud. More similarities could be mentioned, including the evidence that the physiological substrates of both movement and thought embody a dopamine-dependent mechanism. The 'primary thought disorder' of schizophreniform psychoses may respond well to dopaminereceptor blocking agents, and the protracted use of such drugs can result in dyskinesia. From all this it could be suspected, as Hughlings Jackson did, that the central motor mechanism and the neural apparatus of ideation have elements in common and may even be in partial overlap. At present, however, any suggestion as to where the two mechanisms might intersect must remain very largely hypothetical. A main reason for this is that it is still impossible to state where in the brain the impetus to a deliberate movement arises, and by what sequence of steps it is ultimately transduced into the requisite pattern of motoneuronal activation and inhibition. The neural substrate for thought processes, needless to say, is even more obscure. It seems reasonable to assume that ideation is primarily a function of the neocortex, but even if it were exclusively a cortical phenomenon it would be continuously affected by subcortical inputs varying from the selectively targeted specific-sensory thalamocortical radiations on the one extreme to the pervasively distributed serotonergic and cholinergic innervations of the cortex by, respectively, the mesencephalic raphe nuclei and the basal nucleus of Meynert on the other extreme. The striato-pallido-thalamocortical circuits are among this diversity of afferents to the cortex. Two of the four such circuits described in the foregoing account - the ones leading over, respectively, the ventromedial and the reticular nucleus of the thalamus - are at least potentially diffuse in the sense that they appear to be distributed to much if not all of the neocortex. The two remaining circuits,
653
61
however, are more selectively targeted. One of these (fig. 5a) is the classical 'extrapyramidal motor circuit' leading over the thalamic V A - V L complex to the motor domain of the cortex, the other (fig. 5b) the more recently identified parallel loop leading over the mediodorsal thalamic nucleus to the frontal and anterior limbic cortex. Even though hypothetical, it seems reasonable to postulate that the loop involving the frontal cortex could play a role in cognitive functions that is comparable to the role of the 'extrapyramidal motor circuit' in bodily movement. In line with this hypothesis, it is tempting to suggest that movement and ideation share with each other a need for propulsion and timing, and that the basal ganglia might be for both functions a source of such dynamic properties.
References Bowen, F . P . : Behavioral alterations in patients with basal ganglia lesions; in Yahr, The basal ganglia (Raven Press, New York 1976). Butcher, L . L . ; Acetylcholinesterase histochemistry; in Bjdrklund, Hokfelt, Handbook of chemical neuroanatomy, vol. 1 (Elsevier, Amsterdam 1983). Carpenter, M . B . ; Nakano, K . ; Kim, R . : Nigrothalamic projections in the monkey demonstrated by autoradiographic technics. J . comp. Neurol. 165:401-416 (1976). Cole, M . ; Nauta, W.J.H.;Mehler, W.R.; The ascending efferent projections of the substantia nigra. Trans. A m . Neurol. Ass. 1964: 74-78. Damasio, A . R . : Language and the basal ganglia; in Evarts, Wise, Bousfield, The motor system in neurobiology (Elsevier Biomedical Press, Amsterdam 1985). DeLong, M . A . ; Georgopoulos, A . P . : Motor functions of the basal ganglia; in Brooks, Motor control, part 2, pp. 1017-1061 ( A m . Physiological Society, Bethesda 1981). Evered, D . ; O'Connor, M . (eds); Functions of the basal ganglia. Ciba Fdn Symp., No. 107 (Pitman, London 1984). Fallon, J . H . ; Ribak, C . E . : Multiple neurotransmitter studies in the islands of Callej a complex of the basal forebrain. I I I . Connections, correlations and reservations. Soc. Neurosci. Abstr. 6: 114 (1980). Fox, C . A . : The stria terminalis, longitudinal association bundle and precommissural fornix fibers in the cat. J . comp. Neurol. 79: 277-295 (1943). Fox, C . A . ; Andrade, A . N . ; L u Qui, I . J . ; Rafols, J . A . : The primate globus pallidus: a Golgi and electron microscopic study. J . Hirnforsch. 15:75-93 (1974). Graybiel, A . M . : Neurochemical^ specified subsystems in the basal ganglia; in Evered, O'Connor, Functions of the basal ganglia. Ciba Fdn Symp., No. 107, pp. 114-144 (Pitman, London 1984). Graybiel, A . M . ; Ragsdale, C . W . : Biochemical anatomy of the striatum; in Emson, Chemical neuroanatomy (Raven Press, New York 1983). Grillner, S.; Shik, M . T . ; On descending control of lumbosacral spinal cord from the 'mesencephalic locumotor region'. Acta physioi. scand. 87: 320-333 (1973).
654
62 Grove, E . A . ; Haber, S.N.; Domesick, V . B . ; Nauta, W . J . H . : Differential projections from AChE-positive and AChE-negative ventral-pallidum cells in the rat. Soc. Neurosci. Abstr. 9: 16(1983). Haber, S.N.; Groenewegen, H . J . ; Grove, E . A . ; Nauta, W . J . H . : Efferent connections of the ventral pallidum: evidence of a dual striato-pallidofugal pathway, J . comp. Neurol. 235: 322-335(1985). Haber, S.N.; Nauta, W . J . H . : Substance P, but not enkephalin, immunoreactivity distinguishes ventral from dorsal palbdum. Soc, Neurosci. Abstr. 7:916 (1981). Haber, S . N . ; Nauta, W . J . H . : Ramifications of the globus pallidus in the rat as indicated by patterns of immunohistochemistry. Neuroscience 9: 245-260 (1983). Heimer, L . : The olfactory cortex and the ventral striatum; in Livingston, Homykiewicz, Limbic mechanisms, the continuing evolution of the limbic system concept (Plenum Press, New York 1978). Heimer, L . ; Wilson, R . D . : The subcortical projections of the allocortex: Similarities in the neural associations of the hippocampus, the piriform cortex, and the neocortex; in Santini, Golgi Centennial Symp., pp. 177-193 (Raven Press, New York 1975). Herkenham, M . : The nigro-thai a mo-cortical connection mediated by the nucleus ventralis medialis thalami: evidence for a wide cortical distribution in the rat (Abstract). Anat. Rec. 184: 426 (1976). Hill, J . M . ; Switzer, R . C . : The regional distribution and cellular localization of iron in the rat brain. Neuroscience 11: 595-603 (1984). Hughlings Jackson, J . ; see Taylor (1958). Kelley, A . E . ; Domesick, V . B . ; Nauta, W . J . H . : The amygdalostriatal projection in the r a t an anatomical study by anterograde and retrograde tracing methods. Neuroscience 7. 615-630(1982). McKinney,M.;Coyle,J.T.;Hedreen, J . C . : Topographic analysis of the innervation of the rat neocortex and hippocampus by the basal forebrain cholinergic system. J . comp. Neurol. 217:103-121 (1983). Mesulam, M.-M.;Mufson, E . J . ; Wainer, B . H . ; Levey, A . I . : Central cholinergic pathways in the rat: A n overview based on an alternative nomenclature (Ch 1-Ch 6). Neuroscience 10:1185-1201 (1983). Nauta, H.J.W.; Projections of the pallidal complex: an autoradiographic study in the cat. Neuroscience •*: 1853-1873 (1979a). Nauta, H.J.W.: A proposed conceptual reorganization of the basal ganglia and telencephalon. Neuroscience 4: 1875-1881 (1979b). Nauta, W . J . H.;Mehler, W . R . : Projectionsofthelentiform nucleus in the monkey. Brain Res. 7:3-42(1966). Nauta, W. J . H . ; Smith. G . P.; F a u l l , R . L . M . ; Domesick, V . B . : Efferent connections and nigral afferents of the nucleus accumbens septi in the rat. Neuroscience 3: 385-401 (1978). Ribak, C . E . ; Fallon, J . H . : The islands of Calleja complex of rat basal forebrain. I . Light and electron microscopic observations. J . comp. Neurol. 205: 207-218 (1983). Ricardo, J . : Efferent connections of the subthalamic region in the rat. I . The subthalamic nucleus of Luys. Brain Res. 202: 257-271 (1980). Saper, C . B . : Organization of cerebral cortical afferent systems in the rat. II. Magnocellular basal nucleus. J . comp. Neurol. 222: 313-342 (1984). Scheibel, M . E . ; Scheibel, A . B . : The organization of the nucleus reticularis thalami: a Golgi study. Brain Res. / : 43-62 (1966).
655
63 Sprague, J . M . ; Meyer, M . : A n experimental study of the fornix in the rabbit. J . Anat. 84: 354-368(1950). Swanson, L . W . ; Cowan, W.M.: A n autoradiographic study of the organization of the efferent connections of the hippocampal formation in the rat. J . comp. Neurol. 172:49-84 (1977). Swanson, L . W . ; Mogenson, G . J . ; Gerfen, C . R . ; Robinson, P.: Evidence for a projection from the lateral preoptic area and substantia innominata to the 'mesencephalic locomotor region' in the rat. Brain Res. 295:161-178 (1984). Switzer, R C : Hill, J . : Globus pallidus component in the olfactory tubercle: evidence based on iron distribution. Soc. Neurosci. Abstr. 5: 79 (1979). Switzer, R . C . ; Hill, J . ; Heimer, L . : The globus pallidus and its rostroventral extension into the olfactory tubercle of the rat: a cyto- and chemoarchitectural study. Neuroscience 7: 1891-1904 (1982). Taylor, J . : Selected writings of John Hughlings Jackson, vol. I (Basic Books, New York 1958). Teuber, H . - L . : Complex functions of the basal ganglia; in Yahr, The basal ganglia (Raven Press, New York 1976). Williams, D . J . ; Crossman, A . R . ; Slater, P.: The efferent projections of the nucleus accumbens in the rat. Brain Res. 130: 217-227 (1977). Wilson, R . D . ; Efferent connections of the nucleus accumbens in the rat; Master thesis, M I T , Cambridge, Mass. (1972).
656
Reprinted with, permission from Network, Vol. 4. No. 1.1993 © 1993 IOP Publishing Ltd.
The cerebellar network: attempt at a formalization of its structure Valentino Braitenberg M a x - P I a j i c k - I n s t i t u t e for Biological C y b e r n e t i c s , S p e m a n n s t r a s s e 38, 7400 T u b i n g e n , Federal R e p u b l i c of G e r m a n y
1. Introduction Modellers of brain function often make little use of information from neuroanatomy. Sometimes they refer to neural structures in the brain only secondarily, projecting into the neuroanatomy schemes of computation which they had first developed from quite different considerations. In doing so, they pay little regard to the question whether the neuroanatomy by which they illustrate their abstract scheme is necessarily such as it is or might well be otherwise. We have seen the idea of the 'perceptron' used as an explanation of the cerebral cortex, or of the hippocampus, or of the cerebellar cortex. One and the same idea could hardly provide a cogent explanation for the structure of such widely different networks. M y tendency is to take information from neuroanatomy more seriously. The structure of different regions of the grey substance provides precise constraints on the neural interactions which take place there even if the physiological nature of these interactions is not known in full detail. I have shown in previous papers (Braitenberg and Atwood 1958, Braitenberg 1961, 1967, 1983; Braitenberg and Onesto 1962) that in the cerebellar cortex inferences from structure to function come particularly easy and I shall carry this one step farther here.
2. Symmetries T h e cerebellar cortex has translational symmetry in the latero-lateral as well as in the front-to-back direction. If there is a fibre running from a point A to a point B , a certain distance to its right (or left) there is a similar fibre running from point B to another point C , separated from B by the same distance along the same direction (barring, of course, the situation at the margin of the cortex). Similarly, the fibres running forward or backward 657
from any point of the cerebellar surface are the same everywhere. There is also mirror symmetry. For every fibre running to the right there is a similar fibre to the left. For every fibre running forward there is a similar fibre running in the opposite direction (again, with due respect to the margin of the cerebellar sheet). There is no rotational symmetry, however, except for rotations which are multiples of 180°. The great majority of fibres in the cerebellar cortex are parallel or antiparallel to either the left-to-right or the front-to-back direction. We may formalize these anatomical relations and thus obtain general statements about the interaction of signals in the cerebellar cortex (the arrows signify synaptic influences, x is the left-to-right axis, y is the back-to-front axis). Reflexivity (mirror symmetry):
%i H? Xj = Xj —* X; pi -* Vj =
Translational symmetries:
yj->yi
x; —> Xj = xi + Ax —* Xj + Ax Iff -*• Vj = Vi + Ay -* yj +
Ay
(with no exception at the midline!) It is important to note that all of these synaptic influences in the cerebellar cortex are mediated by relatively short fibres: a few millimeters in the x-dfrection, even less in the y-direction. Contrary to the cerebral cortex, interactions in the cerebellum are essentially local. 3. U n i d i m e n s i o n a l layout The fibres oriented in the two orthogonal directions in the cerebellar cortical plane are very different in their form and physiological characteristics. Eccles et al (1967) gave proof of the excitatory nature of 'parallel fibres' which form the bulk of the population of fibres oriented parallel to the x-axis, and the vast majority of all fibres in the cerebellum. Eccles also showed that the axons of stellate and basket cells which form the population of fibres parallel to the T/-axis are inhibitory. Hence the interaction of the elements are radically different in the two axes of the cerebellar cortical plane: summation and reciprocal enhancement are to be expected in the latero-lateral direction x while along the antero-posterior direction y we expect competition between elements and suppression of the weaker signal. In a functional context it would be wrong, therefore, to talk of the cerebellar cortex as of a two-dimensional computer. Rather, we may describe it as essentially one-dimensional, as an array of linear elements — bundles of parallel fibres mediating the excitatory interaction — separated from each other by inhibitory influences. The more activity spreads laterally through the
658
bundles or beams of parallel fibres, the more it will stand out on the background of the silenced neighbouring bundles. 4. M e t r i c i n v a r i a n t s a n d t i m i n g T h e variety of animals makes it possible to define the structural invariants of an organ that is shared by many species, the structural invariants of the vertebrate leg, for instance, or of the liver. We may describe the essence of an organ as the set of properties which are common to all instances of that organ in different species, and to all parts of it. If we do this for identified portions of the vertebrate brain, for the striatum, for example, or for the cerebral cortex, we discover structural invariants which refer to the layout of neurons and fibres, to the patterns formed by their intertwined processes, to the shape of individual dendritic and axonal ramification. These are specified mostly in terms of a 'topological' description, where neighbourhood matters, but distances may be grossly distorted between one part of the structure and another. For example, the layers of the cerebral cortex form a succession which is quite constant and characteristic for that part of the brain, but their relative thickness varies a great deal depending on their location or the summit of a convolution or in the depth of a furrow. Metric, rather than topological invariants, are the exception, and when they occur they point to an unusual importance of distances in the function of a piece of nerve tissue. In the cerebellum, the molecular layer preserves a remarkably constant thickness in spite of the rich folding of the cerebellar cortex. Also, with all the folding, the 'parallel fibres' remain remarkably straight and the planar dendritic trees of Purkinje cells, which are arranged at right angles to the parallel fibres, stay parallel to each other. These metric invariants led to a model of the cerebellar cortex in which measurement of time intervals between successive signals (with a resolution of about 0.1 ms) played the central role (Braitenberg and Atwood 1958). 5. T r a n s f o r m a t i o n of i n p u t by t h e p a r a l l e l fibres T h e vast majority of cerebellar fibres (several orders of magnitude more than the other elements) axe represented by the 'parallel fibres'. These relay the (mossy fibre) input from any point of the cerebellar cortex a few millimeters to either side along the z-direction, passing it on to all the other neurons which they meet on their way (mainly Purkinje cells and various kinds of stellate cells). T h e 'molecular layer' of the cerebellar cortex is essentially a thick felt of oriented fibres whose main function seems to be to smear out the input in space and in time in an anisotropic fashion. Neglecting details of physiology as either irrelevant or not sufficiently well defined, we may take both the input I(x, y, t) and the activity A(x, y t) generated in the cerebellum as densities of excitation in elements dx dy dt, t
659
and the function P of the parallel fibres as an operation on the distribution of the excitation. T h e activity A(x, y, t) generated in the cerebellar cortex by the input I(x, y, t) via the parallel fibres which conduct at a speed v can thus be expressed by a convolution of / with P 0
A = I * P
or explicitly A(x,
y, t) =
/ * I" I{x ±tl,y,t-T) Jo Jo
P ( f , r ) d£ d r .
Since the function P ( f , T ) may be simply taken as P(t,
T) = 6(1;
-
rv ) 0
(where 6 is Dirac's delta function) this reduces to A(x,
y,t)=
I I(x ± v , r , y , t - r) dr . Jo 0
We may represent also the inhibition coming from neighbouring rows (or beams, or grooves) at a distance Ay as simply opposing the activity and get
A(x,
y,
i) = /
I(x
Jo
± V T,
y, t — T)AT — A(x, y ± Ay,
0
t).
6. Tidal waves Considering only one strip of cerebellar cortex oriented in the direction of the x-axis — a 'beam of parallel fibres' in Eccles' terminology — the transformation of the input into the output can be simply described as lateral propagation through parallel fibres from every point a few millimeters to the right and to the left at a velocity VQ. Estimates for the length of parallel fibres vary between 2 and 5 mm on each side, and conduction velocities measured were about 0.5 m s in mammals and 0.3 m s in frogs. We assume a length ( = 4 mm and a velocity VQ = 0.5 m s , which gives as a maximum delay T = £maxf o = 8 ms generated by an individual parallel fibre between the input and the output (Purkinje) cell. Such delays of a few milliseconds may be involved in some neuronal information handling schemes, especially in sensory contexts, but they are of little use in motor physiology, where the cerebellum is undoubtedly involved. The episodes of motor behaviour whose coordination is severely impaired when the cerebellum is defective typically have durations of a few hundred milliseconds. The idea of the cerebellum generating the delays between - 1
- 1
- 1
MAX
m a x
v
660
the activation of an agonist muscle (e.g. biceps) and of an antagonist (e.g. triceps) starting and stopping a movement (e.g. flexion of the forearm) has its merits but cannot be related to cerebellar parallel fibres in the simple form which was favoured in the past. However, even if individual parallel fibres are too short to generate significant delays, the tissue which is formed by their staggered arrangement has a width greatly exceeding their length, especially in larger animals. From one side of the cerebellum to the other, a beam of parallel fibres in the human cerebellum may have a length of over 10 cm. With the known velocity VQ of conduction in parallel fibres, a signal travelling from one side of the cerebellar cortex to the other would take a few hundred milliseconds, a time well compatible with the organization of movement. Is there any circuitry in the cerebellum that could relay signals from one parallel fibre to the next along a 'beam', thus generating the long delays which we need? There is not, but if we shift our attention from the idea of 'generation of delay' to that of 'discovery of temporal sequences' we can still make good use of the width of the cerebellum in terms of the organization of movement. T h e idea is to consider the molecular layer of the cerebellar cortex as a whole as a (issue conducting signals at the velocity v (of conduction in parallel fibres, of which it is composed) in the latero-lateral direction. If such a conducting medium is excited by a stimulus which moves (in the appropriate direction) at the velocity of the intrinsic velocity VQ, a sort of tidal wave will be elicited in the tissue and the response will be maximal, for the moving stimulus will add continually to the excitation which already propagates through the tissue. For velocities of the input layer larger or smaller than v there will be less of a tidal wave. The dependence can be described quantitatively as in figure 1. Q
0
Let the input be a local stimulus moving along the beam of parallel fibres at the velocity v over a length A x . The amount of excitation which it sets up at each input point is constant and independent o f f , hence the total excitation produced is E ~ Ax. A is the activity elicited in the cerebellum, which we define as the density of excitation per unit length of the beam. T h i s depends of course on the spread d of the individual wave of excitation moving along a parallel fibre, and on the dispersion of the excitation in the ensemble of parallel fibres which occurs when the velocity of the stimulus v is different from the intrinsic velocity wrj. T h e dispersion is proportionate to the time it took the stimulus to travel a length A x , and to the (positive) difference between intrinsic and extrinsic velocity. The expression
(Ax/v)\v
661
- vo\+ d
AX
v
0
V
0
F i g u r e 1.
F i g u r e 2.
T h e tidaJ wave effect.
A c t i v i t y A as a function of spread d for ( a ) A r = d a n d (b) A J =
lQd.
is plotted in figure 2(a) as a function of v for Ax = d. Figure 2(6) shows the same for Ax = lOd. Evidently, the cerebellar cortex is a sensitive detector for signals moving at a certain velocity over the array of its input fibres. But is it used as such?
662
T h e question remains open, because not enough is known about the insertion of the cerebellum into the rest of the motor (or, generally, nervous) system. However, we can ask ourselves whether an important condition for 'movement detection' is met there, namely the orderly representation of sensory systems on the surface of the cerebellar cortex, which would make movement detected in the cerebellum correspond to real movement of something in the outside world. A complicated system of maps of the body on the anterior lobe of the cerebellum of the cat has been described by Oscarsson and his group in Lund (e.g. Oscarsson 1980). T h e orientation of these maps, and hence their relation to the parallel fibre system varies and so does the scale of the representation. O n the basis of these maps it is possible to determine to what velocities in the cat's body the intrinsic velocity of the cerebellum corresponds. T h i s is, for one of the maps, a velocity of a few hundred m s , perhaps the velocity of passive mechanical propagation of shock waves through the animal's body. - !
B u t other kinds of projection in different regions of the cerebellum have been described by Welker (1987). Here, the physiologist proceeding in the direction of parallel fibres through the cerebellar cortex does not find a continuously varying relation with some sensory map, but a surprising fragmented projection of various parts of the animal body. T h e two observations suggest that the detector of sequential input in the cerebellum may at one time detect real movement, at another something rather of the nature of sequences of disparate events.
7. Conclusion Inferences from geometry are likely to be safe inferences in the cerebellum, since many physiological details are known about synaptic interaction of neurons there. However, plasticity, i.e. changes in the neuronal interconnections in the cerebellum as a consequence of experience, is not well known yet although undoubtedly it plays a crucial role. Therefore it would be premature to propose functional model of the cerebellum. On the basis of the observations made in the present paper I suggest the following idea. T h e cerebellum responds to sequences of events (in the motor, somatosensory, other sensory systems, but in other parts of the brain as well) which it picks up because they happen exactly in the correct temporal order and at the right speed of succession. Such a sequence not only activates a 'beam' of parallel fibres, but in doing so furnishes, through successive activation of Purkinje cells, the sequence of signals which are needed to deal with those events. In particular, it is likely that the cerebellum thus counteracts the complicated patterns of inertial forces which are produced by any active movement of the body. It does so by exerting a series of inhibitory
663
influences on the motor system: the output neurons of the cerebellar cortex, the Purkinje cells, are inhibitory. Relaxing the tension in a muscle that pulls in the same direction as a passive mechanical force arising during a movement is more economical than counteracting the force by an antagonist. T h i s way of dealing with the inertia! corollaries of movement is well compatible with Honk's (1987) idea of the cerebellum as an adjustable pattern generator, as well as with myographic observations on sequential activation of muscles (e.g. Wadman et al 1979) during the execution of simple limb movements.
Acknowledgements Thanks are due to Hubert Preissl, Sonja Griin and Detlef Heck for discussions and advice, and to Margarete Ghasroldashti and Claudia Martin-Schubert for preparing the manuscript and the figures.
References Braitenberg V 1961 Functional interpretation of cerebellar histology Nature 190 539-640 , 1967 Is the cerebellar cortex a biological clock in the millisecond range? Progress in Brain Research vol 25 The Cerebellum
ed C A Fox and R S Snider
(Amsterdam: Elsevier) pp 334-6 , 1983 The cerebellum revisited /, Theor. Neurobiol. 2 237-41 Braitenberg V and Atwood R P 1958 Morphological observations on the cerebellar cortex / . Comp. Neurol. 109
1-33
Braitenberg V and Onesto N 1962 The cerebellar cortex as a timing organ. Discussion of an hypothesis: Proc. 1st Int. Conf. on Medical Cybernetics
(Naples:
Giannini Editore) pp 3-19 Eccles J C , Ito M and Szentagothai J 1967 The cerebellum as a neuronal machine (Berlin: Springer) Honk J C 1987 Model of the cerebellum as an array of adjustable pattern generators Cerebellum
and Neuronal
Plasticity ed M Glickstein, C Yeo and J Stein (New
York: Plenum) pp 249-60 Oscarsson 0 1980 Functional organization of olivary projection to the cerebellar anterior lobe The Inferior
Olivary Nucleus:
Anatomy
and Physiology
eds J
Courville, C de Montigny and Y Lamarre (New York: Raven) Wadman W J , Denier van der Gon J J , Geuze R H and Mol C R 1979 Control of fast goal-directed arm movements J. Human Movement Studies 5 3-17 Welker W 1987 Comparative study of cerebellar somatosensory representations: the importance of micromapping and natural stimulation Cerebellum and Neuronal Plasticity ed M Glickstein, C Yeo and J Stein (New York: Plenum)
664
Krprimeil wilh permission from fre'tdl m Hturoscuncts. Vol. 10. No. 10, pp .408-415. 1987 5> I9S7 EJseviir Science PuhlLsJicrsLW. (UK)
Hippocampal synaptic enhancement and information sto within a distributed memory system B. L. McNaughton and R. G. M . Morris 8. L McPiaugbton rs at the Department ot Psychology. University of Colorado. Boulder. CO30S09, USA. and R. C. M. Morns vat 'be laboratory for Cognitive Neuroscience. Department ol Pharmacology, University at Edinburgh, Edinburgh EH89II UK.
The
hypolhesis
lhal
Ihe physical
substrate
oj
memory
may represent Ihe activation of mechanisms that normally subserve associative memory. This phenomenon is known as long-term potentiation (LTP) or neuroscience "~ and cognitive science ' long-term enhancement (LTE)*. It has been most and has intensively invesSigated within Ihe hippocampal formabeen widely investigated in behavioural, physiological tion, a brain sSrucSure lhat plays a crucial role in and theoretical studies. Although this hypolhesis certain forms of associative memory. Physiological remains unproven, considerable evidence suggests that investigation has revealed that LTE exhibits most of a particular form of synaptic strengthening, induced the properties implicit in Hebb's original suggestion by electrical stimulation of certain CNS fibre systems, that associative memory results from a synaptic strengthening that is contingent upon the conjunction of Y NPUTS activity in pre- and postsynaptic Y INPUTS 1 1 0 0 o i Y4 elements, in this article, we outline a 1 simple neuronal model capable of 1 10 Y3 1 0 0 1 Y3 multiple memory 0 1 0 1 t Y2 0 0 1 • 1 1 Y J superimposing traces within the same matrix of 1 1 o 1 o0 Y 1 1 0 1 0 0 Y1 connections, and consider the corX D 1 0 0 0 1 0 1 1 X 0 0 1 0 0 0 1 0 1 i respondence between such models 0 0 1 1 0 0 0 1 1 D 0 0 0 0 0 0 0a and the properties of LTE in the I N 1 1 0 1 0 i i i i i a 1 1 1 1 i 1 N 1 1 context of the hippocampal circuitry P o o P 1 0 0 1 1 1 1 0 10 0 11 0 10 1 in which il occurs. Certain predicU 1 1 1 1 1 1 1 1 1 u o 1 1 1 1 1 1 1 i i tions are derived from this frameT 0 1 0 1 1 1 1 1 1 0 1 1 0 1 0 1 work concerning the behavioural 8 X4 X3 X2 XI X3 X2 X1 consequences of experimental manip*Long term ulation of LTE, and we conclude by enhancement'(LTE) n CORRECT RECALL SATURATION describing experimental evidence usedbetem that confirms these predictions and preference to the 001011 • X3 011100 : X4 more commonly used X3 • C = 322332 suggests that LTE is, in fact, Xa • C = 331213 'long-term fundamentally involved in memory. 11 331213 = 110001 potentiation'(ITPt 322332 = 100110 = Y3 because the word 3 3 'potentiation' has a BUT longpnorhistoryof To introduce Ihe concept of synapPATTERN COMPLETION specific application to tic associative memory, we begin 001011 = X3 001001 C X3 a non 'associative and by outlining the properties of a 332332 shod-laslmgincrease 001001 • 211221 correlation matrix a formalm the probability of 332332 • 110110 9 Y3 211221 = 100110 = Y3 ism lhal has been rediscovered transmitterrelease 3 several times over the last 30 2 following telarnt years. Although not a neural model stimulation' . Fig. 1. The fundamental principles of distributed associative memory can be perse, it illustrates the fundamental Potentiation can be easily illustrated by considering storage and recall of event pairs (Xl:VI, etc.) principle that many memory traces observedata very using a mechanism known as a cor/elation matrix, in this example, two sets of six can coexist as an overlaid distribularge variety ofboth binary information channels converge at a matrix C of 36 nodes. All nodes have tion o( connection strengths within central andpenpheiat an initial value of zero, but are permanently changed to a value of one whenever synapses (including a single network. the corresponding X and Y input lines have the value one simultaneously. The those-n the matrix in {A) has experienced three sets of paired events. Any member of a pair Consider a matrix C of sot horihippocampus), ana can be recalled by multiplying the matrix by the other member, and dividing by zontal channels interacting with six has been ihownio the number of ones in the multiplier (decimals are ignored). Moreover, complete vertical channels (Fig. I A ) at 36 differmechanistically recall can also be accomplished by presenting only part of the corresponding nodes. Activity on a channel is tromllE" While LIP pattern, provided that that part is unique (i.e. is not also a part of a different represented by the value one and LT£ refer lo She event)-All three patterns stored in A can be correctly recalled, in spite of the fact inactivity by zero. An input event iame phenomenon. that they Overlap each other to some degree. Although the number of patterns (e.g. X I : V I ) is a pattern of activity In*indiscriminate use such a matrix can store increases dramatically with the size ot the matrix, there is of the word on the set of input channels. The always a limit above which recall of both new and older events become 'potenbabon'has, state space of the nodes is also inaccurate. This is illustratedin (B), where the storage of a fourth pair, which can mourview. binary and 'Hebbian' (i.e. if and only itself be recalled correctly, has interfered with the recall of event 3. In general, ledtoconstderable if both inputs to a node are simulthis limit depends on the magnitudes and relative ovedap of the input vectors. contusionin the taneously in a f state, the node This saturation effect forms the basis for an experimental test ot the relation literature. undergoes an irreversible transibetween LTE and memory described in the text. Bi Ihe mammalian
synaptic
brain
resides
efficacy has been proposed J
A
in alterations
frequently 1
a a a
B
1
I
of
in both
i a
i
0
o
a
i
a
xa* c >
6
h
665
40*
l»"n from U to 1). Such a matrix • .m itore associations between pairs of binary siimulus evtnis. Recall of an inpul evenl is accomplished by multiplying the matrix C\ representing the connection strengths, by [he conespoinding paired associate, and performing an integer division (i.e. division with decimal truncation) on Iheresull. The divisor is ihe number of ones in ihe cuing pattern. This division is analogous
io suiting a variable threshold and is essential lor accurate recall when patterns thai are nol orthogonal (i.e. those sharing common active elememsl are stored in ihe malrix. Perfect recall can be achieved by this means provided nut too many different paliems have been presented I Fig. 1 A). Errors in recall will begin to occur as the matrix approaches saturation (Fig, 1B).
O Lflleral entorhmal corten (LEC) .ind perforant path. 0 Medial enlorhinal corten (ME • • and perfoianl path 0 O
Granule cells and cell field Hl[
r
a mossy cells
Q
CA3 pyramidal cells
0
' Al pyramidal tells
A
Etaski'l
multiplication, Mined here.mvotm multiplying each row clement ot Ihe matnr by tht corresponding elementofthe inpul vector. and then summing the column
cells
fig 2. The maim aspects of hippocampal connection topography are illustrated m a transverse section ( A ) and tn unfolded two-dimensional representations ( B . C \» devised by Swanson el al. The circuitry of the hippocampal formation appears to mandest several of the formal properties of the simple nets considered
666
410 Conceptual neural implementation
The learning performed by this network is hetero-
""'•atiKr, ayM How might the correlation matrix formabsm relate to associative in that the synaptic modification process in an actual nervous system. One way. first outlined by Man"*, is illustrated in Fig. 3A. This network contains six principal neurons, one inhibitory interneuron and two types of inputs. The V inputs each make a single powerful synapse with a principal neuron. This synapse is regarded as a detonator"
one pathway is governed by the activity of certain other fibres whose connection strengths are not themselves i * > i find/ m modified. One example of such learning is classical conditioning in vertebrates. Here, the Y inputs correspond to a representation of the unconditioned stimulus (US), the X inputs to the conditioned stimulus (CS), ' ^ k " ' '"" Produces intense depolarization and fixing and the output partem of the principal neurons to the of the post-synaptic neuron. Trie \ inputs each contact memory representation of the US which comes to be rfttoc&onmflie all principal neurons. These synapses have an initial evoked by activation of the CS". Although this simple latency olthe resulbnclPSPonthe strength ol zero, but become enhanced to a strength of model fails to account for several important conditionpnnopalceHs. in one according toHebb's conjunction rule. The X inputs ing phenomena (e.g. blocking), it does explain how a KW V connections with the CS comes to evoke a new response. This association ol Mfdibori. boll} multiple mtetneurons. inhibitory lntemeuron, which thus feeds forward an this internal representation of events is the denning and a wnibte number inhibitory signal proportional to the total number of characteristic of most higher forms ol learning ol spikes per active elements in the input pattern'. The principal as well, inlemevntn would neurons have a resting threshold of one. The inhibitory Heleroassociation is a special case of a more general contribute to the signal performs the division operation. The system necessity depicted in Fig. 3A has stored the same set of paired paradigm known as autoassociation. This is required in situations such as spatial learning, where it may proportionality. associates as that of Fig. IA. and these can be recalled be necessary to recall a complete representawith the same accuracy. tion of a learned input from any partial subset of the original. Unlike the heteroassociative model, there are no explicit instructing inputs. Rather, elements of a neuronal population that have been active together become coupled in such a way that X3 XI Xi subsequent activation of a few members of the original set results in completion of the whole. This is the essence of Hebb's notion of a cell assembly. As an example, consider our ability, uponenteringa familiar room in near darkness, to locate objects accurately un the basis of the few stimuli available. Autoassociation may be reabzed (Fig 3B) by replacing the Y input lo • PPINCIPH NEUPOH the network of Fig. 3A with specific 1 0 0 INHIBITOR f (OnrSlONJ HtUPOhS detonator contacts from the X •• -' 0 0 • 1 MK INPUT PaTTCHHI 1 inputs. EachXfibremakesontyone 1 0 I D • I I . - J . . . i • .. '-L--. • such contact, and each principal 1 i i HOhfUNCTION «L SIHAPSE neuron receives only one (more I 0 i X! X3 DETONATOR S V H i n i complex examples might require • INHIBITOR* conjunction of several detonators). .1 The inhibitory division operation permits each of the stored uiput patterns to he correctly recalled Fig. 3. (A) A simple 'neuronal'implementation ol the heteroassociahve correlation matnx of Pig. IA. even if the network is presented The Y input fibres have enc'unVe control ol the output cells by virtue ol their terminating in with only part of the original detonator synapses The detonators also initiate Hebbian' synaptic enhancement ol those pattern. For example, pattern X2 terminals fromthenon-speciticXinputthatwerecoactivewiththeparticular Y input. Accurate recall may be completed by presentation is enabled by the inhibitory mtemeuron which is presumed to leed lorward an inhibitory signal that of either 100000 or 001010. In divides the excitation olthe principal neunjns by the number of X fibres active in the current event In general, this is true for any subset real nervous systems, this division may be approximated by inhibitory synapses whose equilibnum that is not also a subset of one of the potentials lie near Ihe postsynaptic resbng potential, and hence shunt the synaptic current arnving at J
f
J
w
6
fixed
e l c i , a l o r
c
°.
the soma via the dendntes While a single detonator input has been used here for simplicity, the idea should not be taken too literally The same general pnnciples apply when convergence ot several or many inputs are required lo initiate postsynaptic activabon. with Ihe exception that Ihe output pattern will be some transform ot the input, rather than identical to it. (B) A simple linear autoassociative network which has the ability to complete a stored representation when presented with only part of the original This network is equivalent to making the It and Y inputs of Fig 2Athe same and might be implemented simply by having each input fibre make a small number of detonator contacts and a large number ol weak, but modihable. ones Evidence for such a conhgucation has been delected within the fascia dentata. (OA recurrent autoassociative network -,t..,h ,„~-:si»i the input to the principal cells with their own output Depending on the details of its 7mo,emen^
A variant of this network also makes use of a single nonmodifiable detonator synapse to drive each principal neuron cxter"
nan-ems active by oreiftifion of new fragment
other stored patterns. The performance of such networks improves dramatically wilh size.
reverberation. It this reverberatory activity ,s not explicitly silenced before information, this network will store event sequences and recall them Irom a of the initial event in a sequence.
667
•?
°
U t p U 1
pattern is fed back into the dendntes of the principal neurons via modifiable synapses (Fig. 3C).
411
This re-entrant organization was referred to as a 'collateral effect' by Marr . Its properties differ from the previous network in two important ways. First, the output pattern of the re-entrant network persists after the input is terminated (unless it is explicitly turned off). Hebb first drew attention to such reverberatory activity as a possible mechanism for short-term' storage of patterns. Second, for autoassociation, input to the re-entrant network must be held constant long enough during learning for the feedback to coincide with the input, and activity from the preceding cycle must be terminated before the arrival of new information. An interesting capacity to learn sequences arises when these conditions do not hold. In this case, the network associates its current input with the output of the preceding cycle. If trained with the sequence Xj, X . Xj, and then presented with the initial event Xi only, the complete sequence is recalled. 1
2
The degradation of stored information through saturation of modifiable connections is a serious problem confronted by all distributed memory systems. As the number of stored events increases, so does the probability of spurious recall. We make use of predictions based on this saturation effect in the experiments described below. There are several ways of dealing with this problem, including efficient input coding (i.e. keeping the number of elements used to code a representation to a minimum), expansion of the input pathway onto a memory network containing a larger number of units, in such a way as lo reduce the relative overlap of input patterns (e.g. Marr's codon hypothesis*) and making the memory impermanent (either by spontaneous decay or by competitive interactions). There may be particularly sound reasons for doing the latter. For example, once experience has established which elements of a particular environment are stable, more efficient representations of the stable elements might be stored elsewhere in a permanent memory while the large amounts of information stored initially can he dispensed with.
0
15
30
45
TIME (mln) Fig, 4. Data from an experimentIllustrating non-associative potentiation, and associative LTE at terminals of the lateral perforant path in the fascia dentata. Data points represent the amplitude of electrically evoked synaptic field poten tial responses {normalized} to weak, fixed intensity stimuli delivered once every three seconds. At L. a stimulus train of 100 stimuli at 200 Hz was delivered at the same stimulus intensity (data points not shown). This resulted in transient potentiation of synaptic efficacy. At H. a second stimulus train was given which was identical to the first except that, during the train, the stimulus intensity was raised to coactivate many more convergent afferents. Long-term enhancement was superimposed on the non-associative potentiation. Similar studies have revealed cooperation (i.e. association) between different pathways, such as the medial and lateral, or the crossed and uncrossed, perforant pathways to the dentate gyms, or the apical and basrlar dendritic inputs to CAl pyramids. The traces at the top of the figure illustrate the extracellulariy recorded field EPSP evoked by test stimuli before, and two hours after Induction of LTE.
25
larger . These could act as the covert detonator synapses of the model. The granule cells of the dentate gyrus project in a highly specific manner onto the pyramidal cells of C A 3 . Each granule cell makes few hut relatively powerful contacts close to the pyramidal cell bodies. The CA3 pyramidal ceils project heavily back into their own dendrites via the longimdinal association' pathway. This configuration is clearly analogous to the re-entrant network of Fig, 3C, In addition, the same perforant path fibers that project to the dentate gyrus also terminate distally on the CA3 pyramidal cell dendrites. Thus, the granule cell output to CA3 represents a transform of the perforant path input to CA3. This arrangement thus resembles a heteroassociative configuration that might, in practice, be carrying out ante-association by virtue of the detonator information being a transform of the information arriving distally.
Instantiation in hippocampal circuitry? Although the dynamics of activity within hippocampal circuitry are incompletely understood, several aspects of its anatomy and physiology correspond surprisingly well with the conceptual implementations iust described (Fig. 2). Cortical inputs projecting via the perforant path contact a considerably expanded number of cells (the granule cells of the denate gyrus) within transverse strips (lamellael along the longitudinal axis. Within this projection there are some clear analogies to the basic matrix arrangements of Fig. 3. The perforant path consists of two distinctfibresystems The medial pathwayresemblesthe Y inputs of Fig. 3A in that it terminates close to the granule cell bodies, has synapses that are initially more powerful than the lateral pathway, and terminates topographically in a We have emphasized above the importance of a restricted transverse strip. The lateral pathway has synapses that are initially weak, and terminates in a division operation for accurate recall, Inhibitory interneurons, with what appear to be the necessary charactopographically diffuse way - hence it resembles the X inputs. However, embedded in this system may also be teristics, are found throughout the hippocampus. For a structure analogous to the autoassociative system of example, the basket cells of the dentate gyrus receive Fig. 3B. Most medial path synapses generate EPSPs excitatory input from the afferents to the molecular of about 0.1 mV amplitude. However, about 3% of layer, and feed inhibition forward onto the granule ll 27-2s sirnilar configurations are found in CAl and these terminals generate EPSPs 10-20 times CA3, where the excitatory afferents terminate both on 20
2
ce
668
s
411
CONTROL LTE
patterns be presented to quiescent networks. Otherwise, interference with preceding activity will occur. The whole hippocampal formation undergoes a cycle of excitation and inhibition ol about 7Hz (the so called theta rhythm) while the animal is engaged in active movement*"' through its environment. Most hippocampal units are phase-locked to this cycle. One function of the theta rhythm maybe to clear the system before the arrival of new input information. The duration of one theta cycle is also long enough, in principle, (or several cycles of the collateral effect to generate a progressively more accurate recall of stored patterns when the input is incomplete. There are, of course, a multitude of other intrinsic and extrinsic connections to consider, and the evidence for the proposals just made is far from complete. Nevertheless, the hippocampal network seems to he organized in a manner compatible with the operation of a distributed, correlation-matrix memory 38
= Q <
UJ
• P TRIAL 1
TRIAL2
TRIALS
TfilflLJ
TRIALS
Fig. 5. The hypothesis that the LTE mechanism underlies spatial information storage within the hippocampus predicts that bilateral saturation oi this mechanism should produce a severe deficit in the acquisition of a spatial problem. This prediction is derived from the concept of matrix saturation illustrated m Fig. 'fr. The prediction was confirmed using the Barnes circle maze task, in which animals learn to escape from bright light into a dark tunnel located under one of 18 holes equally spaced around the periphery of a circular platform. Once a day. the animals were released in a random orientation at the platform centre. The error score shown here is the distance of the bole that the animal first investigated from the correct one (in number of boles). The group receiving LTE induction had a severe impairment that was specific to learning of new locations and memory for recently learned locations. Spatial memory that was well established before the treatment was left intact, implying that the stored information in the hippocampus is ultimately transferred elsewhere.
HJ
the pyramidal cells and on inhibitory interneurons' . The inhibitory interne U X O R S are far fewer in number than the principal neurons, yet have an extensive, diffuse, axonal trajectory . Physiological investigations have shown thai the equilibrium potential for the inhibitory neurotransmitter GABA is close to the resting potential of the principal neurons . Postsynaptic inhibition is thus primarily a current shunt. As a result of these (actors, inhibition has the approximate effect of dividing the excitation by a variable related to the number of active afferents. The excitability properties and spontaneous activity of basket inhibitory cells is also consistent with such a role. These cells respond to stimulation of afferent fibers more quickly, and at a lower threshold, than the principal neurons . This is necessary to insure that the division operation is already in effect before the principal cells begin to fire. Unlike the principal cells, the response of intemeurons is graded in terms of the number of spikes elicited in response to a single input event. Single cell recording from freely moving animals has shown that putative basket cells (i.e. theta cellsj show phasic activity, and convey little, if any, specific information . This is in sharp contrast to recordings from pyramidal cells, which reveal a remarkable degree of spatial selectivity . Thus, it appears that the basket cells are capable of signalling how many, but not which afferents are active.
Long-term enhancement of hippocampal connections Occurrence
LTE is a persistent increase in the efficacy of synaptic transmission which results from highfrequency stimulation of certain CNS fiber systems* '™. It may be induced by brief episodes o( electrical stimulation of the principal afferents to the hippocampus (perforant path-granule cell synapses; Fig. 4), and many, but not all of the intrinsic excitatory connections within and between the principal hippocampal subfields. Several other intrinsic and extrinsic connections have failed to exhibit LTE when tested by comparable techniques, although several of these inputs exhibit a modulatory role on LTE of the principal inputs. Persistent changes, similar to LTE in magnitude and timecourse, have been induced in electrically elicited responses in other cortical regions'" (most prominently in other areas of limbic cortex). 9
AZ
5
32
35
36
There is another possible role for inhibition. Except for systems storing event sequences, effective storage and recall would appear to require that the input
Mechanisms
of induction,
expression and decay
In most cases, a few impulses delivered at intervals approaching the axonal refractory period have been most effective in eliciting LTE . However, under conditions in which the postsynaptic neuron is artificially depolarized prior to the input, single stimuli are sufficient " " In addition to high frequency, it is clear that under normal circumstances the induction of LTE requires the convergence of a critical number of active afferents. In the dentate gyrus, both medial and lateral perforant paths can exhibit LTE. However, when activated in conjunction, the two pathways cooperate to generate considerably more LTE than either pathway alone. Indeed, at certain input strengths the association of the two inputs is required for any LTE . Similar cooperativity has been demonstrated among other excitatory fibers within the hippocampus. While the 'threshold' for LTE in fascia dentata is apparently near the postsynaptic discharge threshold, spike discharge of the postsynaptic neuron is neither necessary nor sufficient for its induction . Nevertheless, Hebb's principle is obeyed at these synapses in a statistical sense, if not in a physical one - the crucial regulatory factor is the depolarization of the dendritic tree to somewhere above the level normally required for cell discharge. One is naturally led to ask why high frequency is
669
JJ
4 1
45
r
46
46
413
normally ol such importance. There would appear to be at least two possible explanations. Firstly, most hippocampal excitatory synapses are bolh extremely weak and rather unreliable. In the dentate gyrus, for example, the average EP5P generated by a perforant path synapse is a tenth of a millivolt, and about 400 convergent fibres are needed to discharge the granule cell* . Moreover, when activated at tow frequency, a single terminal may fail to release any transmitter about 30% of the time. High frequency leads both to spatio-temporal summation, and to a dramatic increase in the probability of transmitter release, which may double or triple the effective depolarization Induced by a group of convergent afferents ". Secondly, it is possible that some of the processes regulating the development of LTE may have somewhat long activation times, and may thus require that the depolarization be extended longer than the timecourse of a single EPSP. This might be of particular use in recurrent systems where the network output must be correlated with the appropriate input. h
35 S3
5
1
The locus of control over the induction of LTE resides with the postsynaptic neuron. This was originally inferred from the cooperativity effect. Subsequently, injections of the calcium chelator EGTA. into single hippocampal neurons were found to block LTE in the impaled neuron without affecting either evoked synaptic transmission itself or LTE in the rest of the population'" . More recently, it has been demonstrated that specific blockade of the N M D A-preferring glutamate receptor in the hippocampus blocks L T E . The N M D A receptor complex is unusual because it is both chemical and voltage dependent , requiring prior depolarization of the postsynaptic cell in order for bound agonists to initiate a conductance increase to calcium ions. This appears to account for cooperativity, the input specificity of LTE, and the reduced requirement for high-frequency input to cells that are depolarized. The properties of the NMDA receptor may also account for the apparent modulatory effects of other inputs. For example, stimulation of the medial septum facilitates LTE of the perforant pathway . On the other hand, high-frequency excitation of commissural afferents to the denate gyrus appears to reduce or prevent LTE of the perforant path *. 1
4 9 5 0
5152
53
5
While considerable progress has been made in deteirnining the factors regulating the induction of LTE, the physical mechanism by which the increased synaptic efficacy is expressed is far from clear, Evidence has been marshalled for the following possibilities: formation of extra postsynaptic glutamate receptors , swelling of dendritic spines and reduction of their neck length , formation of new synaptic contacts*", and increase in the amount of transmitter released , fn none of these cases has cause been adequately dissociated from secondary effects. A few possible mechanisms have been ruled out, including: changes in the postsynaptic membrane potential *, increased probability of transmitter release such as underlies post-tetanic potentiation , reduced inhibition , and reduced threshold in the afferent Gbres themselves . Also, there are two important constraints on any possible mechanism imposed by the pre- and post-synaptic specificity of LTE. The matrix models require that only those synapses on a given postsynaptic neuron that were coactive, and only those terminals of a given afferent that were involved in a 55
565 7
53
Fig. 6 . intraventricular infusion of the drug AP5, which blocks LTE, caused a severe learning impairment for spatial information. Rats were trained to escape onto a bidden platform in a circular water-maze for 1 5 trials over 5 days. In a retention test conducted one day later, the platform was first removed. Control rats swam across the former location of the platform (train) far more often than other locations in adjacent quadrants of tbe pool to the left or right (adj/L, adjR), or in the opposite quadrant (opp). Rats given AP5 showed little spatial bias. Data based on a replication of Ref. 73 Control rats N=22, APS rats N= 14. The blockade of LTE affects learning of new locations, but not memory for old ones. 1
convergence event of sufficient magnitude, should undergo LTE. These requirements have been bom out by experimental observations . The timecourse of the change in synaptic responsiveness following high-frequency activation is complex. After induction. LTE takes approximately 30 s to develop its full magnitude . During this growth time, potentiation, a presynaptic increase in the probability of transmitter release (see footnote", p. 408), is present bul decays over about 5 min iKef. 47). After reaching its peak LTE also begins to decay. While LTE is a remarkably persistent phenomenon, whenever it has been studied in chronically prepared animals over adequate periods, it has been found to decay with a half-life of not more than several weeks Although daily repetition does extend the half-life in much the same manner as memories are stabilized by repetition , hippocampal LTE may not be a mechanism for permanent memory. This decay may reflect the necessity to avoid saturation of the hippocampal network with information as discussed above. However, it is worth noting that in the visual cortex of kittens irreversible changes in ocular dominance are controlled by a mechanism that bears a strong pharmacological and conceptual resemblance to LTE in the hippocampus . 62-64
65
r
66
h
67
The factors governing the decay of LTE have not been studied in detail. There are two possibilities, which are not necessarily exclusive. The commonly accepted view is that decay is spontaneous. The alternative is that it reflects a competitive process. While there is some evidence for heterosynaptic depression of this sort in the dentate gyrus . properties are not yet well understood. 6
i l e
1
47 5
61
60
Behavioural evidence for a role of L T E in memory. Evidence from both single unit recording and lesion studies '"in rats indicates that the hippocampus plays a maior role in spatial memory. The first indications that hippocampal LTE might underlie this type of memory came from the demonstration that the persistence of LTE was significantly correlated with performance of a spatial memory task . More recently, experiments have addressed the behavioural effects of 63
71
670
^Probability'n used herein the speck contettoftne Quanta! release eqtttbonm = nip. Changesm the parameter V have notbeen ruled outandmayi/ndedie themcreasem trantmitter release obsenedby Shis andhn cotieagires**
414
interfering with LTE. There are at least three ways in learning task in which they had to swim in a large pool of which such interference could be used to compromise opaque water to escape onto a submerged platform the function of a distributed memory system. One way located at a particular place. Over the course of five would be to saturate the network artificially, by days of training, the group receiving AP5 took signifiincreasing the efficacy of all (or most) of its synapses to cantly longer and more circuitous routes to the platnear their maximum possible strength by the electrical form than controls". Following training, the platform induction of LTE. This should result in both retrograde was removed and the animals were required to swim and anterograde amnesia. The second way would be to for 60 s without the possibility of escape. While the block enhancement pharmacologically, in a manner that controls swam directly to the former position of the platform and searched persistently in its vicinity, the left baseline transmission unaffected. This should AP5 animals showed little or no evidence of spatial produce anterograde amnesia only. Finally, natural memory (Fig. 6). In a second study, we confirmed that LTE could, in principle, be reversed. This should the same AP5 treatment was sufficient to block L T E produce only retrograde amnesia. Our studies have without affecting normal evoked synaptic responses. addressed the first two of these possibibties. As yet, Moreover, a similar experiment, involving acute the technical means for selective reversal of LTE are microinfusions of AP5, demonstrated both that the not available. spatial memory Impairment Is reversible, and that information acquired prior to the APS> treatment is not disrupted. Saturation The first prediction has been tested by examining the effects of high-frequency. LTE-inducing stimAlthough the same caveat concerning possible ulation of the perforant path upon both recall and new secondary effects of the amnesic treatment is applilearning in the circular platform spatial 'reference' cable to both sets of experiments, the demonstration memory paradigm devised by Barnes . In this test, that two radically different methods of interfering with rats learn to escape from bright light into a tunnel LTE lead to the predicted learning and memory accessible from only one point on the maze periphery, impairments shifts the burden of proof in favour of the and invisible from the maze center. hypothesis that LTE reflects a fundamental storage In one experiment, rals with chronically implanted mechanism. electrodes were Erst trained to their asymptotic performance on the task with the tunnel fixed in one location relative to the extramaze cues. Half of the rats Concluding remarks were then subjected to bilateral induction and saturaWe hope to have illustrated In this article that full tion of LTE, the remainder receiving low-frequency appreciation of the functional significance of LTE can stimulation to the same pathway. The following day the only be achieved when its properties are considered in animals were returned to the circular platform, butrelation to both the formal structure of the hippocampal were now required to find the goal tunnel at a new network in which it occurs, and to the type of location, 135° awayfromits original location. The LTE Information processed by this system. A number of group failed completely to learn the new position of the Important questions remain. For example, although goal tunnel over 5 days of training, whereas the low- there have been encouraging suggestions . it is still frequency control rats learned rapidly (Fig. 5).not known whether LTE actually occurs during the However, on the first relocation trial, both groups course of exposure to novel information. The likelihood were equally good at remembering the original tunnel ol ever detecting such changes will, of course, depend location. High-frequency stimulation had thus pro- on such parameters as how much new information is duced anterograde, but not retrograde amnesia. In a presented, how many synapses are involved in the second experiment, LTE-inducing stimulation deli- storage, and whether there is active competition for vered several minutes after the first relocation trial synaptic efficacy, as alluded to above. These questions blocked the memory for the new tunnel location, are unanswered. At present, there are no data on demonstrating a limited retrograde amnesia. Finally, in whether LTE is distributed within the population of a third experiment, rats trained on the general task afferent terminals as moderate changes in a large demands in one room, and then subjected to LTE. number of connections, or large changes at a few were impaired on the acquisition of the tunnel location terminals (as assumed in the present models). Also, when the maze was moved to a different room. the detailed routes by which information (presumably) stored in the hippocampal matrix returns to the These experiments show that driving the synaptic neocortex, and the use to which this information is put population to its maximal strength disrupts recently (e.g. consolidation), are largely unknown. Much more stored information and prevents new spatial learning", theoretical work needs to be done on the properties of as predicted by the hypothesis, but that hippocampal sparsely connected networks, and the effects of topoLTE may only be involved in the temporary storage of graphically restricted connectivity, both of which are information destined for some other, more permanent characteristics of the hippocampus. These sorts of memory system. The caveat, of course, is that high- uncertainty obviously leave a serious gap in our frequency stimulation may have other effects apart understanding of the exact nature of the role of LTE in memory. One prediction is clear. Further understandfrom the induction of LTE. ing of how the physical dynamics of this network contribute to a solution of the computational problems Blockade of learning and memory will require a considerably LTE can be blocked in rats implanted with an closer collaboration between the behavioural, cogniintraventricular cannula attached to an osmotic mini- tive, and neural sciences. pump conraining the NMDA receptor antagonist APS. Animals prepared in this way were trained in a spatial 72
71
,I5
%tissomev/hatola paradoxtbal nictitating membrane conditioning's improved by prior saturabonotlTErn OK hippocampus* However.information storageforthis type ot learning lies outsidethe hippocampus' , and blockade ot contextual condiboningbypnor saturation of LTE wvtd, perhaps, be responsibleforme taster learning in the delayed conditioning paradigm used by 9
671
415
Selected references
42 Lee. K. (1983) IVeurribrofojyo'lrieHj^ocar7rpus<SiefBrtW., 1 Tanzi, E. (1893} ftfwst* sperimental di Freniatria e Mediclna ed.), pp 365-272. Academic Press Legale delle Allenazioni Mental* Societa Italiana de Psichiatria 43 Douglas, R. M. (1977) Brain Pes. 126, 361-365 19, 419-472 44 Wigstrom, H . and GustafJson. B. (1985) Acta Physioi Scand. 2 Hebb. D. 0.(1949) The Organization of Behavior. John Wiley 123. 519-522 & Sons 45 Wigstrom, H „ Gustafsson, 8., Huang. Y. Y. and Abraham, 3 Brindley. C. 5. (1969) Proc. S. See. London Ser. B. 1S4. 173¬ W . C . (1986) A d a Physiol. Scand 126, 317-319 191 46 McNaughton, 8. L , Douglas, ft. M. and Goddard, G. V. a Matt. D. (1971) Pro!, fl Soc. tendon Ser. S 262, 23-8) (197B) Brain Bes. 157, 277-293 5 Eccles, J. C. (1977) Brain Acs 127, 327-351 47 McNaughton, B. t. (1982) I. Physiol. (London! 324. 249-262 6 Kohonen, T. (1978) Associative Memory. Springer-Vertag 48 Lynch, G. S., tarson.J., Kelso. 5., Barrinuevo.G. and Schottler. 7 Palm. CJ. 0982} rVerrraMssembiies. An Alternative Approach F. (1983) Nature 305, 719-721 to Artificial Intelligence. Springer-Verlag 49 Collingridge, G. t., Kehl, S. J. and Mctennan. H (1983) 3 Hinton. G. and Anderson, J. A. (1981) Parallel Model! ot I Pnysiol. (London! 334. 33-46 Associative Memory, Erlbaum 50 Hams. E. W.. Ganong, A. H. and Cotman, C. W. (1984) Brain 9 Feldman, I. A. (19B2) Biol. Cybem. 46, 27-39 fles 323, 132-137 I D Rumelhart, D. E. and McClelland, I. L. (1986) Parallel Distri- 51 Mayer, M . L , Westbrook, G. t. and Guthrie, P. 8. (1984) buted Processing (Vols 1 fl- 2), MIT Press Nature 309, A3 81-A! 63 11 12 13 14 15 16 17 18 19 20 21 22 23 34 25 26 27 28 29 30 31 32 33 34 35 36 37 3S 39 40 41
tittle. W. and Shaw. C . 11978) (Math, Biosci. 3 9 , 281-250 Moptield.J.J.(1982)P™.NatMcac/.Ici.tJW79, 2554-2558 Steinbuch, K. (1961) Kybemetik 1, 36-45 WiHshaw, D. J.. Buneman, O. P. and tonguet-Hlgglns, H. C. (1969) Nature 222,960-964 Marr, D. (1969) J. Physiol. {London: 202. 437-170 Andersen, P. and Loyning, ¥. (1962) Colloquium Interna, tionale de CUDS 107. 23-45 Dickinson. A-11980) Contemporary Animal Learning theory. Cambridge University Press Berger, T. W (1984) Science 224, 627-630 McCormick. D. A-. Gark, E. A., Larond. D. G. and Thompson, R. F. (198i) Proc. Natl Acid. Sci. USA 79. 2731-2742 Andersen, P., Bliss. T. V. P. and Skrede. K. (19711 &rp. Brain lies 13. 2 2 2 - 2 3 8 H|orth-Simonsen. A. and Jeune. & (1972) I Comp. Neurol 144, 215-232 Steward, O. (1976) /. Comp. Neurol. 1 6 7 . 285-314 McNaughton, B. L. (1981) Brain Pel. 1 9 9 . 1-19 Wyss, I. M. (19B1) J. Comp. Neurol. 199, 495-512 McNaughton, B. L-, Barries, C. A. and Anderson. P. (19811 J. Neurophysiol. 46, 952-966 Blackstad, T. VJ., Brink, K.. Hem. I. and leune, 8. (1970J I. Comp. Neurol. 138. 433-450 Buszaki. G. and Eldelberg, E. (1981) Brain Pes 230. 346-350 Douglas. B. M., McNaughton. B. L. and Goddard, <,. V. (1983) I. Comp. Neurol. 219, 285-294 Abraham. W. C. and Bliss, T. V. F.<1985)Inrnfies. 33:1, 303¬ 313 Frotscher, M. (1985)* Neurocytol 14, 245-259 Struble, «. G-. Desmond, N. t. and Levy, W. B. ( 1 9 7 8 ) Brain Pes. 152. 580-585 Alger, R. E. and Nicolt R. A. (1982) J. Physiol. (London) 32B. 105-123 Bland, 8. H„ Andersen, P., Ganes, T. and Sveen. O . (1980) tip. Brain. Pes. 38. 205-219 Foi, S. E. and Hanck. I. B.. Jr(1981) E»p. Brain Pes. 41, 399¬ 410 Ranek, J. B. (1973) tip. Neurol. 41, 461-555 O'Keefe. I. and Dostrovsky. I. (19711 Brain Pes. 34, 171-175 Vanderwalf, C. H. (1969) ££G Clin. Neurophysiol. 26. 407¬ 418 Fo>, S. E., Wollson, S. and Ranck. J. B. (1986) Exp. Brain Pes. 62, 495-506 tomo, T (1966) Acts Physiol Stand. 68 (Suppl. 277) Bliss, r. V. P. and Lomo. T. (1973) 1. Physiol. (London! 232, 331-356 Racine, tt. I.. Milgram. N. W. and Hafner. S. ( 1 9 8 3 ) Brain fles. 260. 217-231
52 Nowak, t-, Bregestovski, P.. Ascher. P.. Heibet. A. and Prochlantr, A 11984) Nature 307, 462-465 53 Robinson, G and Racine. R. I. (1982) Brain Hei 249,162-166 54 Douglas, P. M.. Goddard, G. V. and Rlrves, M. (1982) Brain Bes 240, 259-273 55 tynch. G 5., Halpain. S. H and Baudry. M. (1982) Brain Bes 344. 101-111 56 Fifkova. E and Van Harreveld. A. (197S) J. Neurocytol. 6, 211-230 57 tee. K„ Schottler, F„ Oliver. M. and tynch, G. (1980) I. Neumphysiol. 44, 247-258 58 Chang. F L. F. and Gieenoirgh, W. T. (1984) Brain Pes. 309, 35-46 59 Dolphin, A . t . . Ertlngton, M. L and Bliss. T. V. P. (1982) Nature 297, 496-198 60 Anderson, P., Sundberg, 5- H-, Sveen. Q . Swann. I. W. and Wigstrom. H. (1980) ,. Physiol. (London; 302, 463-482 61 Haas. H. t. and Rose. G. (1983) J. Physiol. (London!329.541¬ 552 62 tevy.W. B and Steward, O. (1979) Brain Res. 175.233-245 63 McNaughton. B. L. and Barnes. C. A. (1977)/ Comp. Neurol. 175, 439-f54 64 Andersen, P.. Sundberg, 5. H-. Sveen, O. and Wigstorm, H. 11977) Nature 266, 737 65 McNaughton. 8. t. (1983) m Neurobiology of the Hippocampus (Siefert. W., ed.), pp. 233-252. Academic Press 66 Bames. C. A. and McNaughton. B. L (1980) in Psychobiology ol Aging (Stein. D. G. ed.). pp. 253-272. Elsevier 67 Singer, W. (1985) In Synaptic Modification. Neuron Selectivity and Nervous System Organization (Levy. W. B.. Andersen. J. A. and Lehmkule, S., eds), pp. 1-38. Erlbaum 68 Abraham, W. C. and Goddard, G. V. (1983) fVaftm:305.717¬ 719 69 O'Keefe, J. and Nadel, t. (1978) The Hippocampus as a Cognitive Map. On+ord University Press 70 Morris, R. G. M. (1983) in Neurobiology of the Hippocampus (Siefert, W„ ed.). pp. 405-432. Academic Press 71 Barnes. C. A. (1979),. Comp Physiol. Psychol 93, 74-104 72 McNaughton. B. L , Bames, C A.. Rao. G., Baldwin, I. and Rasmussen. M. (1986) Neurosci. 6, 565-571 73 Morns, R. G- M.. Anderson, E.. tynch, G. S. and Baudry, M. (1986) Nature 319. 774-776 74 Sharp, P., McNaughton, B. t. and Bames, C. A. (1985) Brain Res 339. 361-365 75 Sharp, P., Bames, C A. and McNaughton, 8. t.(1987) Befiav. Neurosci. 101, 170-178 76 Magleby, K L (1979) Prof. Brain Pes 49, 175-162
672
Acktiowleigemcpts Preparation of this paper Hat supported by PHS grant number NS20331to8LMcN and a grant/mm me MRCtoPCMM. We aregratetulto C. A. Barnes, p. I Sutherland, E. 1. Green and D I WXshaHtor ttieirrommentson eaiierdraltsofLhe manuscript and to Q. Rao andS. Scott tor assistance with Hgures.
4d
Representations of Space in the Brain
When attention selects a particular object in the visual field, it usually triggers eye movements intended to bring the object into the foveal region in the retinal plane. This is a complex process requiring the detection of the object location in a suitable coordinate frame, followed by the derivation and control of the incipient eye ball rotation. Eye movements, involved in spatial information processing, are an essential part of the visual perception of primates. They have been discussed, in this context, in the first section of the present chapter, mainly in papers (44) and (45). The present section contains three articles focusing on two aspects (one perceptual and one motor) of representations of space in primate brains. Paper (52) reports on electrode recordings from parietal cortex visual neurons, responding to visual stimuli before, during and after saccadic eye movements. It addresses the old puzzle, first raised by Helmholtz, and discussed in this volume by Delbriick (paper (13)), of the apparent position constancy of perceived objects, despite the shift in their retinal position during eye movement. T h e authors find evidence for a change in the neural representation of retinal stimuli, prior to the actual occurrence of eye saccades. At least some neurons anticipate the retinal consequences of the intended eye movement. This fact ties in nicely with the idea, elaborated by Delbriick (paper (13)) that an "efference copy" of the command, issued to the muscles to implement an eye movement, is sent to the relevant cerebral region. It is this early information which would allow the perceptual system to compensate for the shifting of the retinal image. T h e precise neural mechanism remains unexplained, but the quantitative findings reported in this letter are remarkable. They demonstrate that the integration of a visual scene, from the patches provided by successive eye fixations, can be achieved in a retinotopic system of coordinates. Papers (53) and (54), both signed by Klaus Hepp, theoretical physicist
673
and neurobiologist, deal with the representation of three-dimensional eye rotations in the superior colliculus — a brain area involved in generating saccadic motion. The first of these papers is a mathematical text which will, hopefully, not turn off the reader because of a "user unfriendliness" (to quote the author himself). Even professional theoretical physicists may have difficulty following the mathematical arguments, so the ordinary reader should not feel guilty. It turns out that the theoretical analysis of paper (53), and the experimental investigations of paper (54), illuminate each other. So, with a little patience, some back and forth reading, the problem and the solution should become clearer, and the reader feel smarter. In brief, the problem is to decide between two possible ways of describing the kinematics of saccades: by retinal target vectors or by three-dimensional rotation vectors. Conclusive experimental evidence supports the notion that the motor map of saccadic movements, in the superior colliculus, is two-dimensional (i.e. retina-centred) rather than three-dimensional. Thus, the torsional component of the angular velocity must be determined downstream. These two articles provide a neat example of a successful theory-experiment interplay.
674
Reprinied wilh pennis^or, ham Science, VW.255, pp. 90-92. 1992 > 1992 The American Association forih^ Advancercicjit • 5. . i...
The Updating of the Representation of Visual Space in Parietal Cortex by Intended Eye Movements JEAN-RENE DUHAMEL, CAJIOL L , COLBY, MICHAEL E . GOLDBERG* £v«y rye movement produces a shirt io ihe visual image on the retina. The receptive field, or retinal response area, of an individual vuiual neuron moves with the eyes so that after an eye movement it coven a new portion of visual space. For some parietal neurons, the location of the receptive lit Id is shown to shift transiently before an eye movement. In addition, nearly i l l parietal neurons respond when an eye movement brings the site of a previouslyflashedstimulus into the receptive field. Parietal cortex both anticipates: the retinal consequences of eye move merits and updates the retinal coordinates of remembered stimuli to generate a continuously accurate representation of visual space.
A
S WE MOVfc OUR EVES, A STATION¬
ary object excites successive locations on the retina. Despite this constantly shifting input, we perceive a stable visual world. This perception is presumably based on an internal representation derived from both visual and nonvisuaJ information. Helmholtz proposed that rhe brain uses information about intended movement to interpret retinal displacements f 1). We show that single neurons in monkey parietal cortex use information about intended eye movements to update the representation of visual space (2), Libcnlory of SeTLMiTJTiaror Research, hi J be ml Eye lr.ir.r_o: National hubrurcf of Health, Building ID. Room IOC 101. Bclhoda, MD 20892 "To whom corrapondence should be iddreued.
The shift in the visual image on the retina produced by a saccade is determined by the size and direction of the eye movement. This predictability enables the representation of visual space in parietal cortex to be remapped in advance of the eye movement. At the single cell level, the intention to move the eyes evokes a transient shift in the retinal location at which a stimulus can excite the neuron. Our results are summarized schematically in Fig. 1, in which an observer transfers fixationfromthe mountain top to the tree. During fixation, the representation of the visual scene in parietal cortex is stable. A given neuron encodes the stimulus at a certain retinal location (the cloud). Immediately before and during the saccade, the corneal representation shifts into the coor-
675
dinates of the next intended fixation. The neuron now responds ro the stimulus at a new retinal location (the sun) and stops responding to the stimulus at the initial location (the cloud). The neuron thus anticipates the retinal consequences of the intended eye movement: the cortical representation shifts first, and then the eye catches up. After the eye movement, the representation in parietal cortex matches the rea/ferent visual input and the neuron continues to respond to the stimulus (the sun). This process constitutes a remapping of the stimulus from the coordinates of the initial fixation to those of the intended fixation. We demonstrated this remapping by studying the visualresponsivenessof neurons in the lateral intraparietal area (LIP) of alert monkeys performingfixationand saccade tasks (J). Neurons in LIP have retinocentric receptive fields and carry visual and visual memory signals (4). An example is shown in Fig. 2. When the monkey fixates, this neuron responds to the onset of a visual stimulus in its receptive field at a latency of 70 ms (Fig. 2A). Receptive field borders were defined while the monkey maintainedfixation,and, under these conditions, stimuli presented outside these borders never activated the neuron. In the saccade task, the fixation target jumps at the same time that a visual stimulus appears. The visual stimulus is positioned so that it will be in the receptive field when rhe monkey has completed the saccade. If there were no predictive remapping, the cell would be expected to begin discharging 70 ms after the eye movement brings the stimulus into
Oculomotor
IflKftd r;t
•no •* man.
f
Fig. 1. Remapping of the visual \ < \ \ i n parietal eoiru. Each panel represents the visual image at a point in tune relative to a sequence of oculomotor events. Receptive field of a parietal neuron, dashed circle; tenter of current g u c bcanon, solid circle- and coordinates of the cor ijeaj representation, cross hairs-
91 the receptive field. Instead, the cell begins to the target in the absence of the visual stim- lary to the saccade. We found that 16 of 36 discharge 150 ms earlier, that is, SO ms ulus nor when the srimulus is presented at (44%) of the neurons studied in LIP before the beginning of the saccade (Fig. the new location but the monkey does not showed evidence for predictive remapping 2B). This early response shows that the make rhe saccade. The cell also does not (*)• location of the receptivefieldshifted before discharge when the monkey shifts its attenWe also see the effects of remapping when the eye movement. Completion of the sac- tion to the saccade target without actually a saccade shifts a stimulus out of the recepcade restores the receptive field to its origi- making an eye movement (5), Only when tivefield.The truncation of the visual renal retinal location, enabling the cell to the monkey intends to make a saccade that sponse is sharper and occurs earlier than continue responding to the stimulus as reaf- will bring the stimulus into itsfixationre- would be expected from the response of the fcrent visual information becomes available- ceptive field does the cell respond. These neuron when the stimulus is simply extinRemapping is dependent on both the results indicate that neurons have access not guished. In the fixation task, the cell gave a presence of the visual stimulus and the exe- only to visual information in the fixation tonic visual response that persisted for sevcution of the saccade. The cell discharges receptive field but also to information at eral hundred milliseconds after the stimulus neither when the monkey makes a saccade to other retinal locations and to a signal corol- in the receptivefieldwas extinguished (Fig. 2A). This tonic response was truncated by an eye movement (Fig. 2C) even though the retinal effect was the same as turning off die stimulus. h
The above twofindingsshow that some parietal neurons anticipate the retinal consequences of intended eye movements. A third finding reveals that parietal neurons take eye movements into account in their processing of remembered stimuli. Nearly all LIP neurons (22 of 23, 96%} discharge when a saccade brings the site of a flashed stimulus into the receptivefield,even though the stimulus itself is no longer present. An example is shown in Fig, 3, This neuron discharged at the onset of a stimulus in the receptivefield(Fig. 3A). In the task where a saccade brings rhe stimulus into the recepSaccitJe 0*cj Inning Saccarta b a g l n n l r g tivefield,this neuron began to discharge Fig, 2- Effect uf impending .••.:.!:• on visual responsiveness. Diagrams for each - 11 = - '.i • •• only the after the saccade (Fig. 3B), thus yielding no evidence for predictive remapping. In fixation point {dai). visual stimulus (star), receptive held (dashed circle), ind saccade (arrow). Time lines below show the horizontal (H.) and vertical (V.) eye position and the beginning and ending oftheflashedstimulus task, the stimulus is stimulus (Snm.). R.aster display shows neuronal responses for 16 consecutive trials. Rasters and illuminated for 50 ms and is turned off at histogram are aligned on the event indicated by the long vertical Imc Calibration mark at left indicates least 150 ms before the saccade (Fig. 3C}. 100 spikes per second. (A) Visual response ro stimulus in receptive field infixationtask. (B) Visual response ro a stimulus thai is initially outside of the receptivefield.The visual srimulus and the newThe cell fires after the saccade has brought fixation rargcr appear simultaneously. Lett rasrer aligned on stimulus appearance,rightraster on the location of the stimulus into rhe recepbeginning of saccade. Discharge precedes saccade. (C) Response after a saccade that removes stimulus tivefield,even though the stimulus is gone. from receptive field. Rasrer aligned on beginning of saccade. Note the truncation of response relativeWe interpret this discharge as the response ro rhe response ro The disappearance of me stimulus in (A). to a visual memory trace chat has been remapped in conjunction with the eye movement. Control experiments demonstrate that the neuron did not respond to the stimulus in the absence of the eye movement, nor did it respond in relation to the saccade in the absence of the stimulus. The amplitude of the memory trace response averaged about half of that to a stimulus presented in the receptivefield.Memory trace responses could be elicited by stimulus flashes less than 50 ms in duration presented even 1 s before the saccade. This finding indicates that the visual memory trace is coded in a rebnotopic format that is updated with each eye movement. The remapping of the visual representation in cortex allows visual input from successivefixationsro be kept in register. S e c c a d o beg I n n If ^
Fig. 3. Effect of saccade on response roflashedstimulus. The task was executed in dimfight.(A) Visual response ro constant stimulus in receptive trefd. B Response after saccade that brings stimulus into receptivefield.Raster and histogram aligned on beginning of saccade. Response begins when stimulus These results suggest a mechanism for enrers the teceprivefield.(C) Response after saccade when stimulus was flashed for 50 ms. Snmuius spatial is processing of visual information. At gone before stimulated sire enters receptive field, thetimea saccade is planned, the parietal
676
92 representation of the visual world undergoes a shift analogous to the shift of ihe image on the retina. Unlike the retinal shift that follows an eye movement, the parietal shift precedes the eye movement and predicts the location of rcaffercnt visual inpul. This dynamic link between successive retinal images may contribute to the integration of visual information across eye movements and ro the construction of a continuously accurate, retinocentric representation of visual space. REFERENCES AND NOTES 1. H. .• ii Helmholtz, TnaHtr w> l.-ptel Optks (Dover, New York, 1962), vol. 3. 2. See ibe> M. & Goldberg ind C. B. Bruce, J. Ntmphyr. 64, 439 1990) M. E. Goldberg, C. L- ColbyJ.-R. DuhAnwLCWJSpn^ ntMiorAM Q u u Bur ££. 729 (1990); J. Droulea and A. Bcrtho*. in Mater O n W . CwiopU and kna, D. R. Humphrey and H . |. Freuiwl, Eos- (Wiley, Chi-
chejttr. 1990), pp. 137-161. J . We recorded from 16 neurons In die LIP of mo rhesus monkeys (jVfaaaea tnnlsOi). Mankeyi were prepared for behavioral md physiological recording uivitr terik surgery with keramirLC and sodium penrorha] or Uanuannc u anenhcrics. Training, eye poainDn recordings and nrurophysio logical recording w m done u described [B_ H. Wurtz, j . N n r i V W - 3J. " i M '•' Goldberg, LA Morwdi in o , i , . L , hfoi-afcinitf^ J. L. Barker and I F MtKervy, Eds. (Wiley. New York, 1963), vol. 3, pp. 225-248]. 7
:
4. T . C. T . Yin and V. & Mounrearot Science 197. 1381 (1977); D. U Kobinson, M, E- Goldberg. G. B. Stanton, J Hnntptyiuf 41.910 \"\'f.\ J. W. Gnadr and R. A. AnderUn, • Brain Ka. 7Q, 216 (1988);H. A-Andersen,R.M. Braeewell.&Baraih. I. W. GnidL. L Eogiui. J; N«
677
fixation ask. foveal amnrion i i maintained by rewarding the monkey foe releasing the bar at the dimming of rhe Hiaban rrimuhu. 6. A neuron wiihouT pmiica.T remapping begins ro discharge with a visxuJ LateDCy from the time the stimulus enter* therecepdvefield,dial is, U the end of the saccade. Any neuron discharging with a shorter brency shows predictive remapping. Because we canrioc iSelerrnine prcbsely when the stimulus crosses tile outer Tjoundan/ of the recepnkr li-LJ, we adopted a conservative wdidcal criieriDn. Foe each Miran, a r resr wis used ro compare mean response latencies for blocks of 16 saccade andfieadonTrials. NeurOna with a Latency rebdve to the beginning of a saccade (when the stuniiliu is mil outside the rccepirve Geld) dial was shorter Or HOE ILgnificanriy different from the latency relative to the onset of a snrnuliia in the Fixation task were considered ID show peeddedve remapping. 7. Wc are gratefUl to the naff of The Laboratory o* Sensorimotor Research far its hdp: J- Raber fol vererinary care, K. Powell and K. Maegnider i- r animal care, C. Crist and 1. Rutrherformiohining, L leruen for deerroaicj. A- Hayes lor computer systems, and ]. Serinberg for raralilanng i; 14 June 1991; accepted 10 October 1991
Reprinled wilh permission from Commun. Math. Phys,, Vol. 132, pp. 285-292, 1990 ©1990Springer-verlag
On Listing's Law Klaus Hepp Physics Department, E . T . H . , CH-8093 Zurich, Switzerland
Dedicated to Res Jost and Arthur Wightman
Abstract.
L i s t i n g ' s l a w states that v i s u a l d i r e c t i o n s o f sight are r e l a t e d to
r o t a t i o n s o f the eye s o that a l l r o t a t i o n a x e s lie i n a p l a n e . T h e g e o m e t r y o f SO (3) i n d i c a t e s s e v e r a l p l a u s i b l e a l g o r i t h m s h o w the h u m a n b r a i n c o u l d relate v i s i o n to eye m o v e m e n t s s a t i s f y i n g L i s t i n g ' s l a w , a n d suggests c r u c i a l e x p e r i m e n t s w h i c h we h a v e c a r r i e d o u t .
1. Introduction M a t h e m a t i c a l objects, l i k e the q u a n t u m m e c h a n i c a l r o t a t i o n g r o u p SU(2),
are
i n t r i n s i c a l l y b e a u t i f u l a n d " u n r e a s o n a b l y efficient" i n theories o f o u r p h y s i c a l w o r l d . B o t h R . J o s t a n d A . S . W i g h t m a n h a v e m a d e w o n d e r f u l use o f SU(2)
i n their
scientific w o r k a n d t e a c h i n g . M a t h e m a t i c i a n s often h a v e n the P l a t o n i c attitude that objects l i k e SV(2)
are ideas w h i c h exist i n d e p e n d e n t l y
of our material world.
N e u r o b i o l o g i s t s see t h e m m o r e m a t e r i a l i s t i c a l l y r e l a t e d to c o r t i c a l a c t i v i t y p a t t e r n s in the b r a i n s o f e n s e m b l e s o f m a t h e m a t i c i a n s a n d laid d o w n i n their p u b l i s h e d c o m m o n k n o w l e d g e . W h i l e the p h i l o s o p h i c a l issue [1] is u n d e c i d a b l e , a n a t u r a l scientist c a n a s k the m o r e m o d e s t q u e s t i o n , w h e t h e r the SU(2)
is r e p r e s e n t e d i n o u r
b r a i n a n d , i f so, w h e t h e r i n a g i v e n h a r d w a r e i m p l e m e n t a t i o n o f the n e r v o u s s y s t e m , a s c o a r s e l y d e s c r i b e d b y n e u r o - a n a t o m y , g e o m e t r y a l l o w s u s to m a k e e x p e r i m e n t a l l y refutable p r e d i c t i o n s a b o u t the o p e r a t i o n s o f the h u m a n n e u r a l n e t w o r k . T h e SO(3)
is i n t r i n s i c a l l y l i n k e d to o u r s e n s o r i - m o t o r p e r i p h e r y . I n the l a b y r i n t h
o f the i n n e r ear, for i n s t a n c e , r o t a t o r y m o v e m e n t s o f the h e a d i n d u c e flow p a t t e r n s in 3 a l m o s t o r t h o g o n a l s e m i c i r c u l a r c a n a l s , w h i c h at i n t e r m e d i a t e f r e q u e n c i e s are t r a n s d u c e d as h e a d a n g u l a r v e l o c i t y . T h e S 0 ( 3 ) a l s o a p p e a r s i n the c o n f i g u r a t i o n space o f o u r l i m b s a n d , i n its p u r e s t f o r m , i n the k i n e m a t i c s o f the eye. T h e p u r p o s e o f this note is to e x p l a i n s o m e s i m p l e g e o m e t r i c a l ideas a r o u n d L i s t i n g ' s l a w for eye m o v e m e n t s a n d t h e i r r e l a t i o n to f o v e a l v i s i o n a n d c e n t r a l v i s u o o c u l o m o t o r t r a n s f o r m a t i o n s i n m a n . T h e g e o m e t r y o f SU(2)
suggests n o v e l w a y s
for i n t e r p r e t i n g n e u r a l a c t i v i t y i n the b r a i n i n terms o f p a r a m e t e r s o f the e x t e r n a l
678
2 8 6
world, which we are investigating in our laboratory. A s an introduction to the neurophysiology of vision and eye movements we recommend recent reviews [2,3,4],
2 . Kinematics of the E y e In a good approximation the eye is a center-fixed sphere. We .shall only consider conjugate eye movements with parallel lines of sight of both eyes, so that the SO (3) is the configuration space of a "double eye." We choose a head-fixed right-handed orthonormal system with e[ in the frontal, e in the lateral and e in the upwards direction and an eye-fixed system ej, e^e^ withe[ along the direction of sight. Every eye position is given by the rotation R = (R ) withe,' = Ze R . Let e with | e | = 1 and 0 % Q < 7t/2 be the axis and angle of R (e, g) by the right-hand rule, where we restrict ourselves to foreword eye positions as a simple neighborhood of the natural oculomotor range. In this range eye positions R are uniquely characterized by rotation vectors [5] 2
3
IJ
K
LTI
r = tan( /2)e ,
(1)
e
and also the product R R L
2
has this representation with
Equation (2) follows from the local isomorphism between S O ( 3 ) and SU(2), given by R(e, e)*-*cos (e/2) - i sin (e/2)2>'o-(, using the multiplication rules for the Pauii matrices a . We shall always rely on this correspondence between R(T) and r. Saccades are stereotyped rapid eye movements, which move the eye from R , to R along a geodesic in S O ( 3 ) , i.e. with a fixed-axis rotation [6] :
2
R(t)
= R(e,g(t))R,
,
(3)
1
where e is the axis and g the angle of R^Ri • The angular trajectory g(t) increases strictly from g (*,) = 0 at the start t, to g(t ) = Q at the end i with bell-shaped velocity g > 0 and an almost linear increase of duration t - t with amplitude g. The rotation vectors of saccades between forward eye positions are straight lines: 2
2
2
x
L e m m a 1. Let IrJ < 1 and ||r || < 1. Then the geodesic (3) between R , and R has the 2
2
form rfO^r.+fftO^-r.) where
ff(/i) = 0, <7(/ ) = 1 and d(t)>Q
,
(4)
otherwise.
2
Proof. R(e,g) has the rotation vector r?>(—tj), which multiplied by z(t) = g(t)lg gives the rotation vector of R(e,g(t)). F o r f close to (, the latter can be composed with r,. Here the terms with rj A T cancel, and one obtains (4) with 2
T(00
+
I ' I |
2
)
^-i+itoKir+d-xtoK.r, •
679
F
« ,
287
Since for forward eye positions (5) is analytic in O ^ r ^ l , (4) follows by continuation in T. F o r an "axiomatic" derivation of Listing's law, which is a geometrization of data on visual after-images by Listing and Helmholtz [7], we take the experimental facts that during fixation the direction of sight determines uniquely the eye position (Donders law) and that the set of Donders points is invariant under saccades connecting them (saccade invariance [8,9]). F o r concreteness we take the subset V= {s: ||s|| = 1 s • ej > l / [ / 2 } as the head-fixed visual saccade range, which can be viewed by directing the eye towards it, such that s becomes the line of sight through the center of the pupil to the fovea, the highest acuity spot on the retina. 3
Lemma 2 (Listing). Let s->e be a map of the visual saccade range V onto forward eye positions e| = s , e , e 3 = e j Ae , whose graph contains with any two positions the geodesic (3) between them. Then the corresponding rotation vectors lie in a plane through the origin, which satisfy in an adapted head-fixed frame 2
2
2
r(s)- =0 , e i
r( )=0
(6)
e i
- Helmholtz [7] has tried to prove Listing's law from Donders law without assuming saccade invariance and from a "principle of easiest orientation." O u r elementary proof uses the fact that saccades as geodesies on SO(3) (hence also characterized by an extremal principle) are straight lines in the Euler-Rodrigues coordinates (1). - We shall call the saccade invariant set of Donders positions Listing's positions, the set of its rotation vectors (6) Listing's plane and the line of sight normal to it primary direction. In the monkey, the primary direction lies in good approximation in the symmetry plane of the head and in the plane of the horizontal semicircular canals [10]. This and a well-adapted geometrical arrangement of eye muscles [11] allows a very efficient generation of compensatory eye movements in response to head rotations, where the eye can move out of Listing's plane and make saccades with any axis of rotation. - Also for visually evoked saccades in Listing's plane the brain matches sensory and motor coordinates in a very efficient way: When seeing a target at the direction $2 with the eye in Listing position r the components J ^ I 2 "^( i)et are the coordinates of the target relative to the fovea, which are almost identical to the saccade vectors in (4): =
r
s
l t
Lemma 3. Let s be a direction in space in the visual range Vands the solution T of e, T = 0 and >?(r )e, =83 is 2
2
2l
2
=R(— r j s j . Then
2
r =e AS /(l+e 'S ) , 2
1
2
1
r -T =e ,\& J2 2
l
l
+
1
(7)
2
3
0(,Q )
(8)
up to cubic corrections in r, andi , which introduce relative errors of less than. 08 in a visual and oculomotor range of'25 deg. 2
Proof. We choose the head-fixed frame by taking e according to Donders law when ei is the line of sight. F o r any 3 directions of sight s , , . . . ^ the triangle with the 2
680
288
corners r(S!),...r(s ) consists of Donders positions because of saccade invariance. Hence by a triangulation of V one shows that all Donders positions satisfy r • n = 0 for some unit vector n. e, is the primary position, if e! = n , which can always be achieved by rotating the head-fixed system around e, A H with twice the angle between e, and n [5]. 3
Obviously (7) is a Listing position. Using (1) and Ji(e,g)x = (x-e)e + sin(e)eAx-cos(e)eA(eAx) ,
(9)
one shows t h a t f ? ^ ^ = s . By expanding (9) up to 0 ( g ) o n e obtains for k= 3
2
1,2,
*2 =Wr )er*(r K) 1
1
2
= 2 ( r - r ) . ( e A e ) + 0(fl ) , J
2
l
1
(10)
1
since r - e — r e = t - e - 0 . The torsional 1-component of r 2 - r , is zero by Listing's law, while the vertical and horizontal components satisfy l
1
1
l
l
k
2{
-2(4 - r?) ?, .
(11)
s
I n the visual and saccadic system the errors increase with the distance from the fovea and the saccadic amplitude. A s a measure of the relative error e in (8) we therefore take the quotient of the Euclidean distance of r — r , from r = e , A S / 2 (expressed in radians using p = 2arctan(||r||)) and the angle ^ = a r c c o s ( s ) between target direction and the direction of sight. In the periprimary range the linear approximation is very good: I f
2 1
2 1
21
2
In the last section we have solved the computational problem how to generate a saccade from a Listing position r , to a visual target in the direction s using the retinal e r r o r s ^ . The implementation of this algorithm in the brain depends on the "hard-ware" architecture of the central nervous system. In order to work out eyperimentally testable alternatives we shall briefly summarize some important findings (see [4]). 2
Lesion studies and the analysis of firing patterns of identified neuronal populations in the behaving monkey delineate the following main pathway for visually evoked saccades: VIS-tSC-RF-MN
(12)
V I S are visual and visuo-motor areas which transform images of "pixels" on the retina and select a target vector s . The understanding of visuo-spatial operations in this distributed network is stil! incomplete, but rapidly progressing, and provides an interesting window through many high-level cortical and subcortical areas. 2 1
S C , the superior colliculus, carries a neuronal motor map which, when only the direction ej of the eye is recorded, seems to be topologically related to to retina: Microstimulation at a point P on the S C motor map elicits a saccade, by which the eye is driven from r , to r = r , -l-d, where the saccade vector d appears to depend on P 2
681
289
but not on the initial eye position. In the same 2d analysis single neurons in S C in the neighborhood of P show an intense burst of activity preceding all visual saccades to d(P). Somehow idealized S C appears to have a 2d map of central pattern generators, where a network of neurons at P generates a saccade with vector i(P), either when stimulated by a retinal target at d(P) or after a small local current injection. These neurons carry an abstract motor command, in the sense that their firing generates a preprogrammed motor pattern, but the trajectory of the saccade (3) is not temporally encoded by the neural firing rate [12]. R F consists of several areas in the reticular formation, one for the generation of the horizontal and one for the vertical-torsional components of all visually- and head motion evoked 3 d saccades. Here the present 3 d analysis [13] has been made only in the linear approximation and therefore without quantifying the noncommutative aspects of 5 0 ( 3 ) . Lesions in one of the specific subareas of R F permanently abolish the generation of the corresponding saccade components. By their firing pattern the local neurons are identified as directed burst neurons: they are silent except shortly before and during saccades, where the firing r a t e / ( f ) i s i n a f a i r approximation a linear combination of the component of eye velocity JC(() and acceleration x{t) in a direction e (which as the sensitivities X and it depends on the neuron):
f(t) = Xx(t)
+
l
ix(l)
.
(13)
Hence in the linear approximation the number of spikes in the burst 2
K = J f(t)dt = X(x(t )-x(t ))*2X(r -r )-e 2
l
2
(14)
l
I
is proportional to the component of eye position change, since x(t ) = x(t ) = 0. M N are the motoneurons for the 12 eye muscles, which receive the saccadic eye velocity and acceleration input from R F . In addition, R F feeds into a reverberatory network I N T , called the position integrator, which transforms as in (14) the eye velocity-acceleration input pattern into an eye position signal, which is needed by M N for holding the eye between saccades. If one extends the mathematical analysis of Listing's law up to the first nontrivial non-abelian corrections (the best one can hope to measure with the present techniques in this highly nonlinear distributed system), (12) allows several alternative implementations: 1
2
- The motor map in S C can be either retinal, i.e. parametrized by
" z i ^ - r ^ e ^ s ^
.
(15)
or rotatory, i.e. parametrized by to^rW-O^i
+ riArj .
(16)
A retinal organization of the S C means that the visual saccades encoded by the command neurons form a 2d map without a torsional component. Stimulation at a point representing d on the retinal map with the eye at initial position r leads to r' = d+r, and the eye displacement within the oculomotor range is always independent of the initial eye position. A second stimulation at d' drives the eye
682
290
to d + d ' + r . Hence the stimulation induced eye rotation commands act commutatively and leave Listing's plane invariant. A rotatory S C map has to be 3-dimensional, since for r, and r in Listing's plane and not collinear r, A r is torsional. Therefore one should find command neurons in the S C , which fire with the highest rate when not only r - r , is close to a specific value but also r , A F . T w o consecutive S C stimulations at q and then at q' should drive the eye t o q = r a n d then t o q ° q ' ° r . Since q = q ' - q ' ° q = s 2 q A q ' , this action of SO0) is not commutative and does not preserve Listing's plane. 2
2
2
2
- Similary the velocity coding (13) of the directed burst neurons in R F can either be in terms of coordinate velocity f f(t)
= 2(Xi{t)
+
li
T(t))
,
(17)
e .
(18)
e
or in terms of angular velocity Q
f(t)=(ASi(t)+nf)(t))
In the first alternative (14) holds, while otherwise the non-abelian relation 2
Jl = 2(r + r A f ) / ( l + ||r|| )
(19)
implies for a fixed-axis rotation (3) « s s 2 A ( r - r , + r, A T ) e. In this case a crucial experiment is to investigate (for neurons in the vertical-torsional area of R F with a not purely vertical on-direction) whether a linear fit o f n relative to d *e or to q e i s better for 3d saccades in the neighborhood of Listing's plane with a fixed vertical and a variable torsional component. 2
2
The answer to these questions is important for understanding the neural operations along (12). A quaternion model has been proposed [14], in which Listing's law is implemented upstream between V I S and S C , in which the representation of visually-evoked saccades in S C is rotatory and where the firing rate of the directed burst neurons in R F is in terms of angular velocity. This introduces a number of nonlinear multiplicative feedbacks involving quaternion operations on the level of S C , R F and I N T and necessitates neurons, whose firing patterns have never been observed. There is a much simpler vector model, which is an extension of [15]: - With the eye in the Listing position r , and the target in ^ V I S determines s , which is linearly transformed by (15) into the S C address d . F o r large saccades the 0(Q ) error will be compensated by a small corrective saccade. 2 1
2 l
3
-
T h e S C pattern generator at d
2 1
generates the velocity pattern c ( r ) for the saccade
(4), which is distributed linearly to burst neurons in R F with on-directions e via synaptic strengths proportional to d -
2 1
-e.
T h e velocity output of R F is linearly transformed by I N T as in (14) into the new
eye position r to be used by M N . - I f M N needs an eye velocity signal proportional to Q ( f ) ( c . g . if the firing rate of the motoneurons is linearly related to the muscle contraction velocity and if the
683
291 moment directions of the muscles are approximately constant [16]), then this signal is generated downstream of R F by a feedback circuit, which also adapts the central eye position signal r to the static requirements of each muscle. The vector model can be extended to incorporate the pathway generating headmotion induced saccades and compensatory eye movements and feedback protecting visual saccades against perturbations. A second, direct pathway from V I S to R F via the frontal eye fields is of secondary importance in the normal monkey [17].
4. Conclusion The purpose of this note was to show how geometrical ideas together with information from functional anatomy can lead to predictive models in neurophysiology. O u r ideas about possibly observable second order effects related to a neural implementation of 5 0 ( 3 ) were the content of a paper submitted Spring 1989 to a neuroscience journal. Because of its mathematical "user unfriendliness" the article could not pass the referee, like a short Jo st-to-Wight man note [18] in the good old days, where the Physical Review Letters were unresponsive to mathematical physics and neural networks. In the meantime we have completed the crucial experiments with microstimulation of S C and the 3d analysis of neuronal firing patterns in S C and R F and have found in both structures strong evidence against the quaternion model and consistence with the vector model [17,19], In addition, we have studied Listing's law in humans for the rotatory degrees of the eye, head and arm in different reaching and grasping paradigms. In all three systems unconstrained movements in the configuration space S O ( 3 ) are by neural control restricted to 2d planes in rotation vector coordinates, and the trajectories in these planes are often closely correlated [20], reducing the dimensionality of configuration space from 9 to 2 in the spirit of Bernstein's theory of synergistic control [21], We expect that Listing's law and the geometry of SO (3) will become an important tool to understand the neurodynamics of multidimensional sensori-motor systems [22]. Acknowledgements. In conclusion I would like to (hank my laboratory colleagues V. Henn, B. Hess, J. van Opstal, D. Straumann, and T. Vilis for stimulating collaborations and R. Jost and A.S. Wightman for their generous friendship.
References
1. Changeux, J.P., Connes, A.: Matiere a pensee. Paris: Jacob 1989 2. Robinson, D.A.: Control of eye movements. In: Handbook of physiology - The nervous system. II. Brooks, V.B. (ed.). Baltimore, M D : Williams and Wilkius 1981 3. Berthoz, A., Melvill Jones, G. (eds.): Adaptive mechanisms in gaze control. Amsterdam: Elsevier 1985 4. Wurtz, R.H., Goldberg, M.E. (eds.): The neurobiology of saccadic eye movements. Amsterdam: Elsevier 1989 5. Haustein, W.: Biol. Cybern, 60, 411-420(1989)
684
292 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22.
Sternberg, S.: Lectures on differential geometry. Englewood Cliffs, N J : Prentice Hall 1964 Helmholtz, H.von: Handbuch der physiologischen Optik. Hamburg: Voss 1896 Ferman, L . , Collewijn, H . , van den Berg, A . V . : Vision Res. 27, 929-938, 939-951 (1987) Tweed, D . , Vilis, T . : A n n . N . Y . Acad. Sci. 545, 128-139 (1989) Hess, B . J . M . et a t : (In preparation) Simpson, J . I . , Rudinger, D . , Reisine, H . , Henn, V . : Neurosci. Abstr. 12, 325-326 (1986) Sparks, D . L . , Mays, L . E . : Brain Res. 190, 39-50 (1980) Vilis, T . , Hepp, K . , Schwarz, U . , Henn, V , : Exp. Brain Res. 77, 1-11 (1989) Tweed, D . , Vilis, T . : Neural Networks 3, 75-86 (1990) Robinson, D . A . , Zee, D . S . I n : Progress in oculomotor research, Fuchs, A . F . , Becker, W. (eds.). Amsterdam: Elsevier 1981 Hepp, K . , Henn, V . : Lecture Notes in Physics, vol. 284, pp, 163-177. Berlin, Heidelberg, New Y o r k : Springer 1987 van Opstal, J . , Henn, V . , Hess, B . J . M . , Straumann, D „ Hepp, K . : Neurosci. Abstr. 16 (1990) Hepp, K . , Jost, R . , Ruelle, D „ Steinmann, O . : Helv. Phys. Acta 34, 542-544 (1961) Hepp, K . , van Opstal, J . , Hess, B . J . M . , Straumann, D . , Henn, V . : Neurosci. Abstr. 16(1990) Straumann, D . , Hepp, K . , Hepp-Reymond, M . - C , Haslwanter, T . : Neurosci. Abstr. 16 (1990) Bernstein, N . : The co-ordination and regulation of movements. Oxford: Pergamon Press 1967 Schwartz, A . B . , Kettner, R . E . , Georgopoulos, A . P . : J . Neurosci. 8, 2913-2927, 2928-2937, 2938-2947 (1988)
Communicated by A . Jaffe
Received April 28, 1990
685
position (primary position). Experimental evidence shows that ail virtual rotation axes describing eye positions lie in a single plane (Listings law [12)], which is defined as Listing's plane (Fig, \ ) if the head is upright and stationary. The other description uses rhe rotation of the eye from rhe starring position of the saccade to the instantaneous position. It has been shown experimentally that in this description saccades have fijed angular velocity a*es rhat are not confined to a plane {13) and therefore require a threeTwo- Rather Than Three-Dimensional Representation dimensional (3-D) parameter space. These rwo descriptions are equivalent and are a of Saccades in Monkey Superior Colliculus consequence of the noncommutativity of rotations in 3-D space {10, 11), A . JOHN VAN OPSTAL,* KLAUS HEPP, BERNHARD I , M , HESS, DOMLNIK STRAUMANN, VOLKER H E N N We have investigated the neural implementation of Listing's law. Listing's law Saccade; arc controlled by neurons in the brainstem reticular formation that receive may be implemented upstream from the input from the superior coU.tculu.-s and cortex. Recently two quantitative models have motor SC fthe quaternion model {!})]-The been proposedforthe role of the colticuius in tbe generation of three-dimensional eye axis, when the eye moves from initial posimovements. In order to test these models, ihrcc-dimensional eye movements were tion r tofinalposition r . both in Listing's measured in the alert monkey to investigate whether the saccadic motor map of the plane (Fig. 1), is thought to be coded in the superior colliculus is two-dimensional, representing retinal target vectors, or three- motor colliculus as the rotation vector {10, dimensional, representing thru-dimensional motor error for the rotation of Che eye. I4 15): Electrical stimulation of the superior colli cuius produced tvy^dimensionaj, not rAH^ffl + *r (1) rhrct-dimensionil, eye movements. It is therefore concluded that rhe colli cular motor map is twO'dhncnsinnal. where x denotes the vector outer product {10, 11). In this model, the follicular vector H E M O N k E Y SUPERIOR COLLtCULUS compensate for most of the deficits after r^H, the direction of which is the angular velocity axis of an eye rotation, will, in fSQ is a brainstem srrucrure rhat is ablation of the SC (7). important in the generation of sacSo far, researchers have investigated the general, be tilted out of Listing's plane • [r cadic eye movements [1), Electrical stimula- function of the SC by measuring eye move- because irs torsional component tion of the SC produces saccades with short ments in two dimensions {horizontal and x r y, is nonzero whenever r and r are latencies al low stimulation thresholds. Sac- vertical). Recently, however, theoretical nonparailel rotation vectors {13), cade amplitude and direction depend pre- studies have shown that a complete descripA corollary of Eq. 1 is that electrical dominantly on the site of stimulation {2, 3). tion of the rotational kinematics of eye stimulation will, in general, yield saccades Cells in the deeper layers of the SC burst movements most be in three dimensions, that bring the eye out of Listing's plane (11) vigorouslyforsaccades that are directed into that is, it must include torsion {10, 11). in a specific: way: the torsional component of their movement Geld (the range of saccade There are rwo ways of describing the kine- rhe eye, achieved after stimulation, is detervectors for which a particular SC cell is matics of saccades. One description is a mined by activated), and the neural activity related io trajectory; eye position is described by a movement is tightly coupled to saccade on- virtual rotation from a head-fixed reference i | - feu I • '* m set {1, 4). Furthermore, the SC provides an important input to the reticular formation, where the burse generator for all rapid eye Reprinted wilh permission Irani Science, Vol. 252. pp 1313-1315. 1951 . The American Association for ihe Advancement o l Science
A
B
t
a
T
A
a
m o v e m e n t is situated (5, r>). Whereas abla-
fl
&
B
tion of the SC [7, 3) causes only minor permanent deficits in saccadic performance, small local injections of either muscimol or hdocaine {?), which inhibit neural activity, cause profound deficits for saccades direcied into the affected movcmeni field. Parallel pathways, presumably incorporating the frontal eye field (FEF} may be able lo T
Vomica? CnmprfnBi-il A. f. van OpstiL B. J M. HcU. D. Soauirunn, V. Hflnn, Neurology Dcpuuncnr. UftAemry Hospital. CH-S0°1 ZLirich. Switzerland. K. Hepp, forties rjeparnneni, EldgenDssische Technische Hochschulr. CH-9093 Clinch. SwiEfilind T D whom [QirrapOndcnte should be addittied. at the Dcpubncnl of Medical Physics 4nd Biophysics. University rfNijmeften, Gtert Grwtepkm Ncord I I , EZ Nijmtgtn, The Netherlands.
•
n:
Topglonal cflmncinHn!
Flo. 1. Lifting's lav.I •• eye positioned visually evoked eye movements. Eye positions {15) were sampled ac regular Intervals during and between saccades in die light (1575 data points) with rhe head pitched downward by 15". All eye positions lie in a well-defined plane w i t h zero torsion. Typical standard deriarion in the torsional direction tcss than 1". The plane is perpendicular to the torsional direction, thus defining the direction o f primary position {crou mack). Data from monkey Ca. (A) Frontal (horizontal-vertical) view o f Listing's plane. The center o f the oculomotor range is, for this monkey, downward from primary position. (B) Side view o f Listing's plane along rhe interaural line. (C) Top view o f Listing's plane along die vertical axis. Units for both figures are the following: a rotation vector component o f 0,1 corresponds approximately to 0 • 10° according to the formula r =
686
1314 1
where is given by Eq. 1 and r is the componcni of rhe onset position vector, r , dial is orthogonal to theroiation r,,. (i6). A second consequence of this model is lhat follicular movementfieldsare 3-D even for visually evoked saccades. One should therefore 6nd rhat the majority of cells have their movementfieldcenter outside Listing's plane. Such cells willfireoptimally for saccades having a rotational axis with a specific torsional component, and theirfiringrate should thus depend in a systematic way on saccade onset positionOn the other hand, the SC may be conceived as a two-dimensional (2-D) oculocentric motor map (2) that elicits horizontal and vertical components of saccades corresponding to the desired eye displacement {10} dxn, from Listing position r to r„: ^ A H - s ~ A (the vector model). Thus, the torsional component of the angular velocity vector, if needed by the eye-muscle system, is determined downstream of the SC. for example, in an eye position-dependent feedback loop {10). In this model, the centers of colli ail ar movementfieldsare all in Listing's plane (17), and no deviation of Listing's law is expected after saccades elicited by electrical stimulation: r = d ^ + r ,
deeper layers of the SC (110 sites in six rion Our interpretation is further supported by colliculi; current strengths between 10 and the results of single unir analysis in rheSC {17). 100 u-A) at locations where neural activity Because a parallel pathway that bypasses the SC related to movement had been recorded is known to exist, by which the FEF can send earlier, while letting the monkey look it-signals directly ro rhe reticular formation (7), around; wc thereby obtained a large range Listing's law may be implemented upstream of initial eye positions in Listing's plane, from the SC. As Listing's Jaw is preserved after Microsrimulation at a typical site of the collicular stimulation (Fig. 2), this possibility right SC of monkey Ca yielded saccade would require the bypass pathway ro provide trajectories in the horizontal-vertical and on-line torsional error. Since local inacrivabon horizontal-torsLonai plane (Fig. 2, A and B). of the SC reduces the dynamics of saccades into We examined the amount of eye torsion the affected movement field [9), which suggests after electrically evoked saccades as a func- that this parallel pathway may in fact be nortion of r (Eq. 2) (Fig. 2C). A least squares mally ineffective, rhe possibility of such last and regressionfitfor the data points (slope, accurate on-line error feedback of eye torsion is -0.0037 ± 0.001) is much closer to the unlikely. Also, a bilateral chemical uiactivation prediction (slope, 0.0) of the vector model of neuron populations in the SC, which was than that of the quaternion model. The large enough to drastically reduce the frequenresults for all stimulation sites are presented cy and amplitude of visually elicited saccades, in Fig. 2D- If the quaternion model applied did not lead to a deterioration of Luting's plane to our data, the values would be scattered for the remaining spontaneous eye movements around the Line with slope 1.0 (line Q). and did not abolish rapid phases of the vestibInstead, the data are much closer to the ule-ocular reflex in three dimensions i2Q). horizontal axis, although a weak relation These results arc different from those obemerges, especially for large-amplitude sites. tained from neurons in the rostral interstitial The results obtained for monkeys Br and Ce nucleus of the medial longitudinal fasciculus axe similar to those for monkey Ca. The (riMLF) in the midbrain, the structure that slight slopes that we found might partly be directly controls the motoneurons that generdue to nonlinearities of the eyeball-eye mus- ate vertical and torsional rapid eye movement Wc have tested the predictions of these cle system when the eye reaches the limits of components. A bilateral riMLF lesion leads to two models by measuring eye position in its motor range. the permanent loss of all rapid eye movements three dimensions in three alert rhesus monWe conclude that Listing's law is imple- with a vertical or torsional component; this keys wirh an implanted dual scleral search mented ekjwnsrream of rhe SC on the basis of result functionally Localizes the riMLF between coil {IS, IP). We electrically stimulated the experimental data with electrical micmstimula- rhe SC and the extraocular motoneurons (6). Analysis of unit activity in the riMLF reveals thai many neurons code torsional movement Rg. 2. Results of electrical u u B stimulation in deeper layers components: unilateral inactjvaaon Leads to A . of right SC Data from torsional deficits (fj), and electrical stimulation \ " monkey Ca. (A) Saccade traalways induces eye movements with a torsional jectories in the yt plane as a i component {21). result of electrical mig on Wc postulate that Listing's law for visualcrostimuFatioft at sire • | f_al542 (SO-iiA cathodal ly evoked saccades is implemented downpulses L train duration, 70 stream of the SC. Like the visual map in rhe ms; pulse width, 0,25 ms; superficial layers, the motor map in the 330 Hz). Starring positions <* deeper layers of the SC is organized in of the eye were scattered Vertical componenl Tofsion al com&oriani widely within the oculomo•culocenrric coordinates. This 2-D organitor range. Saccades were dization greatly simplifies multisensory coor, rected leftward and downdination, such as the generation of saccades ward. Mean amplitude to auditory targets {22) or the coordination equals 29.Z" (B) Same data as in (A] in side view. Sacof eye and head (23) or eye and arm {24). cades remain in Listing's plane. (C) Quantitative test of the prediction of the REFERENCES AND NOTES quaternion model (Eq. 2) E. P. H. Schiller ind F. Koerner. _/. i V n J m ^ i i i J - 34, and the vector model for 920<1971):P H Schiller and M . Srryker, ih.d. 35, collicular coding of sacP15 (1972 >; R. H. WurHUidM E. Goldberg, ihltf., cades. Data from (A) and p 575 (B). The continuous line Drlhoganal compc.njnr 1. D. A. Robinson, Virion Rn. 12, 1795 (1972). (Qj represents the predicI , A. 1. v a n O p n l J . A. M. "inGisbergen. A C Smit, tion of the quaternion model (slope Hr^H = 0.261). whereas the dotted line is thefinedregression line f j p . Brain Rti. 79, 199 (1990). through the data points {i ' = -0.0113-0,0037 r - S D offittedslope: c, = 0-001; n = 47) Note 4. D. L. Sparks and L. E. Mays, Brain Ra. 190, 39 the differences in scale for ordinate and abscissa. (D) Result of analysis, as in (C), for ail stimulation sites {\9S0), D. L. Sparks, i M 156, I (1978) in monkey Ca. data from (C)|- Predicted slope based On the quaternion model (given by Hr^H).5 M- 5 RafhoLim ind E L Kelkf. J. f ^ w ^ W Measured slopes (a) from data based on regression line analysis. Linear regression yields a = -0.0001 40, Ml (1977). K Hepp, V Henn. T VIIK. B. Cohen, In Thr + 0.102 • Hr^J (o, = 0.DO6; n • 63), which is dose to a = 0.0. Similar results were obtained forft Wniffliiflinfv of Sonffdit Eyr Mowmeixls, R. H . monkey Br: a - 0.0214 + 0:181 tft " 02; n = 31). and monkey Cc; • = -0 0121 + W u r a i n d M. E. Goldberg, Eds. (Elwvjer, Amsterdam, 19119), pp. 10S-212. 0.227 (
y
1
A
r
R
s
Dn
Mr
r
1
s
55
687
r
1315 7. ?. H. Schilk.. S D . True, | L Conway, J N^AjpkyM. 44. 1175 (1980). 8 f. E. Albany M. Mishjjn. L. E. WprbfOt*. K. H 48. 338 (1982). 9 O. Hiko^ikj and R H. Wiutz. ibid. S3. 266 (l^aSJiE^fl-amfifj 61,531 (1986); C Lee. W, H. Bohrer, D. L. Sparks, fV.in.rr 333, 357 (|9IB)10 K Hepp. C D A M I M . Afu* PAyi 132, 28S (1990). 11. TJ B.Tweed and T. Vilis, J. NruKphymi. 58, 832 (1987); Ataul Nttw-ki 3, 7S (1990). 12. H von HeLnholEz, HanJbiHh if Ph rotate to go horn primary position ro rheeurrenr eye \ - a n d p Sdk amount of iDCabon iboul n 'I his nounor, n fully equivalent TTJ rhe quaternion icpnieniadOn [11, li). We hair adopted a nght handed cnonlirdie tynem KnftOO ii the COmponenT Of r along ihe forwardpoinring * a n (right ear down morion, pcBioviJ, radial position along The lertwaid'pain ting / iiis (do^*ifd, poffinvc), and horiroopJ pcAdcci along rhe upward-polniing a- H I S (leftward, positive). As an etampfc. a poiidon 20* Co rhe left of primary pOnnon [(Xj*) = (0.0,0)] I* repfesenred by die ccofdiriaro (*.VJ] - (0.0,0.176). All rnathonaoal esorenions in EOtjr I •-•-'< -ir,! I " :i..,,I, .iiifi (JliTierXinS in die angle p, an rmn that is beyond esperimairal resohjOon.
A
m
17.
18.
r
19. 20 21.
L
16. Equation 2fallow*from Eq. 1 if one subsnmtes r
a
688
by r „ B (the stamuIaOon-iriduced rounon) and r by e ((he initial eye p- =•-•:. •.- • and calculate? the torsional component We UK the fa a thai both r^, (Fig. I) andr^a llwwn} lie in Listing's plane. The result* of our analysis are consLstrnE with twodirnensionil movement 'fieIds; K. Hepp. A I. win OpstaJ B. | . H r a * , D. Swan V. Herns, SV. jV^rcin. Abtr. 16, 1094(1990) D. A. Robinson. IEEE Tta\u. BMflied. Eng. 10, 137 (I96J); L Fennin. H Colkwijri. T C. larner,, A V van den Berg. Vitim Ru- 27, All (I9B7) B. I. M. Hen. tfbfeii & u . 30, 597 (1990) A I van OpnaL V. Ham, B . ) . M. Heu, D. Strtuirunn, K Hepp, 5 s Nrwimi. Ahf. 16.1034(1990) V. Henn. D- Seraumann. B. f. M. Has. A. J, van Opstal, Hi. Hepp, In ftstibutet did Btain Sim C M IfetqfEyr, Head jrnf Bfldy Mowraft, H- Shirruzu and Y. Shine-la, Ens. (Springer, Heidelberg, Germany, 1991). H . F . J a y a n d D . L Sp4rfcl,/rJri.rat*p^r, 57.35 (1987) D. Tsveed and T. Wis, S«. ^ftiwri'. ^Uslr. 14. 958 (1988): D. StraumanTk K- Hepp.T K u k - m a , A I van QpWJj, £-^,V hJnmuft. Snytpr. 3.163.(1990). D. Straumann. T. Haslwuitcr, M. C. Hepp-Rcymond. K. Hepp. E»p. flrain ftfc, in press. We rhank V. Furrer-lsoviirj and hi. DiirslcJa fbt mhrK i aJ assLsrance. Suppoited try the EuJOpW Striregie Programme tor rUtearch ind l^wioprnoic in lnfonnanon Technology (.^uorni 1J49 and SNF J199025239) SNF 28008.89 (B •:•:<• ind D.S.> and EMlX>-5nrrung ZUnth
21 23.
r
24. 25.
9 October 1990; JOXpfed I I March 1991
4e
Oscillations and Synchrony
T h e editors felt that article (55), by Engel et al, deserved a section by itself because it covers several purposes. It provides an excellent summary of the pros and cons in the debate between the "single neuron doctrine" of Barlow and the "cell assembly" proposal of Hebb (so, it gives adequate preparation for reading the two conflicting — or perhaps complementary — viewpoints expressed in section 5a). Its exposition of key concepts, such as the "binding problem" and the "superposition catastrophe", and of the need for coactivation of several assemblies in the same perceptual brain area (which might not be a request in motor areas, where a simpler population coding might hold) is a mode] of clarity. The early experimental findings of synchronous firing in mammalian visual cortex on a millisecond time scale, that have stimulated so many subsequent investigations, are reviewed here; it is also explained how oscillatory signals may be suited as carrier signals for a temporal code. Related considerations, concerning olfactory computations, come in article (64). T w o additional remarks. The reader is invited to compare the analysis given to perception of an interrupted bar, with those of subjective contours in articles (43) and (48) (which are closer to the attitude of the single neuron doctrine, and of segregation of separate visual pathways). The reader is also invited to ponder on the role given in this paper to lateral reciprocal connections ("lateral" meaning here "within the same perceptual layer") in contrast to the importance given in section 5b to feedback reciprocal connections (retroprojections from a higher to a lower layer).
689
Rcprinltd with pf tnusiian from Trends in Hcurolcienccs. •Jul. 15, No. 6, pp. 21B-226. 1992 O 1992 Elssviai Scimcs Publishsrs Lid. (UK)
Temporal coding in the visual cortex: new vistas on integratio in the nervous system Andreas K. Engel, Peter Konig. Andreas K. Kreiter, Thomas B. Schillen and Wolf Singer well known hy now that the visual system exhibits a high degTee of functional specialization . Neurons in most areas of the visual cortex process information only from a limited part of the visual field and respond only to a restricted range of feature constellations. Thus, the outputs of numerous cells must be integrated to create a complete representation of a particular object. Moreover, neurons detecting different attributes of an object tend to be compartneurons synchronize their responses depending on how mentalized tn a modular fashion, and it has been coherent features are m Ihe visual field. These results argued that different features such as form, colour or support Ihe hypothesis that temporal correlation of motion are analysed independendy by separate proneuronal discharges may serve to bind distributed cessing streams". Accordingly, object represenneuronal activity into unique representations. Furthertation requires integration across these different pathmore, these studies indicate that neuronal responses ways. Unfortunately, there is no evidence for conwith an oscillatory temporal structure may be particuvergence of these processing streams onto a single larly advantageous as carrier signals for such a target region that could provide the basis for a unified temporal coding mechanism. Based on these recent percept . tn the visual system, the binding of findings, it is suggested here that binding of neuronal features pertaining to individual objects appears to be activity by a temporal code may provide a solution to the a prerequisite for figure-ground segregation and problem of integration in distributed neuronal netscene segmentation, i.e. for the distinction between works. several objects present in the visualfield.Of course, similar problems arise in other sensory modalities, During the past few decades, neuroscience has been and mechanisms for binding are also required where pervaded by the idea that the relevant level for sensory events have to be linked with motor acts or describing how nervous systems work is that of the when stored information must be recombined during single cell. Guided by this assumption, which has been memory recall . addressed by Barlow as the 'single neuron doctrine', considerable progress has been made in understanding the constituents of neuronal systems at theClassical approaches to the binding problem cellular and molecular level. In contrast, our knowlIn the framework of the 'single neuron doctrine' it is edge about the integrative functions of the nervous assumed that the binding problem can be solved by system is still poorly developed. This problem is convergence of input from the primary processing particularly evident in cortical neurobiology. Although stages onto single cells with highly specific response much has been learned concerning the structural and properties - Such 'cardinal cells' are supposed to be functional properties of single neurons and their located in 'higher' integrative cortical areas that connections, crucial questions concerning integration correspond to the presumed final stages of visual of cortical activity are still unresolved - . Increasing information processing. However, a number of arguevidence suggests that many cortical functions are ments suggest that single cell representations, while based on distributed processes that occur in parallel at perhaps effective for specialized functions, cannot different sites. However, it is still enigmatic how provide a general solution to the binding problem , relationships are established between such distributed fl) This model suffers from a 'combinatorial explosion'. neuronal activities, even though this seems required Since every new feature constellation would require a to represent information about the environment or the new 'cardinal unit', far too many cells would be needed internal states of the organism and finally to achieve to cope with the complexity of the perceived world coherent perception or action. and the variability of its aspects. 12) A large number of uncommitted cells would have to be reserved for the Visual information processing may be taken as an representation of new objects. (3) The model lacks example to illustrate this need for integration, which is unequivocal experimental support. The discovery of commonly addressed as the 'binding problem'. It is
Andreas K. Engel,Although our knowledge of the cellular components of Peter Konig, Andreas the cortex is accumulating rapidly, we are still largely It. Kreiter, Thomas Signorant about how distributed neuronal activity can be Sehibn and Well integrated to contribute to unified perception and Singer are at tne behaviour. In the visual system, it is still unresolved Mai-Planck-lnsbtothow responses of feature-detecting neurons can be hit Himbrsdiung, bound inlo representations of perceptual objects. Recent DltlBlbordenstr. 46, crosscorrelation studies show thai visual cortical tW)frinkhjrt7t, FRC
1-8
3,
3
1
1
2
23
3
690
219
2-1
-60
+ 60
FJg. 1. long-range synchronization is influenced by stimulus coherence. MuHtunJt activity was recorded horn two sites in area 17 ol cat visual cortex that were separated by 7 mm. The two cell groups preferred vertical orientations. (A), (B), (C) Plots of tbe receptive fields. The coiinear arrangement ot tbe fields allowed tbe companion ol three different stimulus paradigms: a long continuous tight bar moving across botb fields {A), two independent Sight bars moving in the same direction (B). and tbe same two bars moving in opposite directions (Q. The circle represents the center of tbe visual field, and the thicti line drawn across each receptive field indicates the preferred orientation. [D) (E), (F) The respective crosscorrelograms obtained with each stimulus paradigm. Using the long Sight bar. tbe two oscillatory responses were synchronized, as indicated by the strong modulation of the crosscorrelogram with alternating peaks and troughs (Di. If the continuity ol the stimulus was interrupted, tbe synchronization became weaker IE), and it totally disappeared if the motion ol the stimuli was incoherent (Fi. This change ot the stimulus configuration affected neither the strength nor tbe oscillatory nature of tbe two responses (not shown). The graph superimposed on each of the correlograms represents a Gabor function that was fitted to the data to assess the strength of the modulation^. The number in the upper right comer indicates the 'relative modulation amplitude', a measure of correlation strength that was determined by computing the ratio of the amplitude ot the Gabor function to its offset Abbreviation: ns. not significant. 5cale bars indicate the number of spikes. r
0.15
2-A
20
so-called face-selective cells in monkeys was first considered as a verification of Bartow's proposal (reviewed in Ret 9j, However, il has now been recognized Lhat most ol these cells respond to a range of face stimuli too broad to be compatible with the notion of cardinal ceils' "'". Moreover, neurons selective for many other items of our visual world have so far not been identified. 1
These problems have solicited alternative proposals that go back to the work of Hebb . Hebb nii
suggested that visual objects should be represented by assemblies of interconnected neurons rather than by the activity of single cells. Such assemblies are assumed to consist of neurons encoding elementary features and thus, in contrast to Barlow's proposal, complex representations can in fact be formed at early processing stages. In Hebb's model, cells responding to features of a particular object are bound into an assembly by the concurrent elevation of their average firing rate Representing objects by assemblies has
691
220 cortical neurons, rather than by mere elevation of average firing rates, tn such a 'correlational model', perceptual coherence of features in a visual scene is reflected by synchronous firing of the corresponding feature-detecting neurons. Thus, neurons responding to features of the same object discharge in synchrony, whereas cells responding to different objects are assumed to fire in an uncorrelated manner. This proposal is attractive for several reasons. (1) It preserves the general advantages of assembly representations, such as robustness and parsimony. (2) Binding by synchrony makes the grouping process resistant to amplitude fluctuations. (3) Assemblies are defined in a highly dynamical manner and cells can rapidly switch between assemblies by subtly changing the temporal relationships of their firing patterns. 14) Using such a temporal code would permit the coexistence of several assemblies in the same region Fig. 2. Cartoon of tbe experiment illustrated in Fig. 7. Tbe recording situation of cortex, since the members of each assembly are was projected onto a map oi orientation columns in area 17 generated by the tagged by their synchronousfiring.Taking all of this 2-deoxygtucose technique (kindly provided by 5. towel). Shaded areas into account, it seems that binding neurons into represent bands of orientation columns responding to vertical contours. assemblies by a temporal code may provide an Electrode symbols indicate the two cell groups from which recordings were attractive solution to the binding problem. talten. The cartoon illustrates the measurement shown In Fig. 1C. F. where recordings were taken in response to tight bars moving in opposite directions. The two cell groups are presumed to be part of two large assemblies (represented by dots and circles), which are activated by the two light bars. It can be assumed that the members of each of the assemblies oscillate in synchrony, whereas no constant temporal relationship exists between them, as shown by the physiological measurement. Hole that the two assemblies are spatially separate. Using the long continuous light bar, the intervening shaded orientation bands would also be activated, thereby bndging the gap and inducing the formation ol one large assembly of synchronously oscillating groups. Scale bar is I mm. Abbreviations: HM, representation of the horizontal meridian; VM, representation of the vertical meridian. 13
several distinct advantages . (1} Representations are more resistant to the loss of individual neurons. 12) Because individual feature detectors can at different times participate in different assembles, this binding strategy economizes on the number ol cells required for object representation. However, there is a major drawback of the Hebb model, which has only recently been recognized" . It relies on the assumption that only one assembly is activated at a time in a particular region of cortex, whereas other assemblies are suppressed . While this might hold true for representations at very high levels of processing, it cannot be the case at more peripheral stages where processing of natural scenes, which always contain multiple objects on a complex background, requires the coactivation of several assemblies. This, in turn, poses a severe problem, because the concurrent activation of cells responding to different objects in the same cortical region makes it impossible to determine which cells belong to which particular assembly. As a result, false conjunctions of features become unavoidable ' . This dilemma has been addressed as the "superposition catastrophe' of the Hebb model . ,s
13
Stimuli is-induced oscillations in cat visual cortex Recent experimental studies support the proposal that synchrony may be the 'glue' that binds distributed neuronal activity into unique representations. These studies were triggered by the discovery of stimulusinduced gamma oscillations in the primary visual cortex of the anesthetized cat". Initially, these oscillations were observed in localfieldpotentials, suggesting that numerous cells dose to the recording electrode had synchronized their discharges and participated in rhythmic activity. Subsequent multiunit recordings confirmed that adjacent neurons indeed have a strong tendency to fire in synchrony and to engage in recurrent bursts at frequencies between 30 and 70 Hz . This suggested that local groups™ of coheremly firing cells could be the building blocks for corticalrepresentationsand, based on these observations, it has been proposed that spatially separate groups might be bound into assemblies by synchronization of their oscillatory burst responses . ie,ls
1819,21
Local groups of simultaneouslyfiringneurons seem to be particularly suited to establishing temporal relationships between spatially separate sites because these can influence distant targets much more effectively than individually discharging single cells. It should be emphasized that in this conceptual framework, which extends von der Malsburg's hypothesis, synchrony among distributed groups is the relevant code for binding, whereas the intrinsic temporal structure of the responses is. as such, not assumed to represent particular aspects of the visual input. This view is supported by the evidence that these oscillations exhibit a high degree of variability, being best described asfluctuationswith a broad frequency spectrum - , and do not depend on,forinstance, Binding in the temporal domain orientation or velocity of the visual stimuli . OscillatThe outcome of these considerations is that a ory responses with these characteristics have also solution for the binding problem is needed that been observed in areas 18 and 19 of cat visual preserves the advantages of the Hebb model but cortex , as well as in a visual association area provides additional degrees of freedom to label the located in the posteromedial lateral suprasylvian cells that are part of one assembly. To this end. von sulcus (PMLS) . In addition, they have been demonder Malsburg and Abeles' have suggested that strated in the visual cortex of awake cats *. assemblies might be defined by synchronousfiringof H
15
15
18
21
19
21,22
23
6
2
692
221
100
Fig. 3. Stimulus dependence of shortrange interactions. Mulburtit activity was recorded from four different orientation columns of area 17of cat visual cortex separated by 0.4 mm. The four cell groups bad overlapping receptive fields and orientation preferences of 22" (group 1), 112" (group 2), 157° (group 3) and 90° (group 4). as indicated by the thick line drawn across each receptive field in (A)-tD). Tbe figure Shows a comparison of responses to stimulation with single moving light bars of varying orientation (left) with responses to the combined presentation of two superimposed light bars (right) For each stimulus condition, tbe shading of the receptive fields indicates the responding cell groups. Stimulation with a s'ngle light bar yielded a synchronization between all cells activated by tbe respective orientation. Thus, groups 1 and 3 responded synchronously to a vertically orientated (0") light bar {A), groups 2 and 4 to a light bar at an orientation of 112" (B). and cell groups 2 and 3 to a Sight bar of intermediate orientation (C). Simultaneous presentation of two stimuli with orientations of 0" and 112", respectively, activated all tour groups (D). However, in this case the groups segregated into two distinct assemblies, depending on which stimulus was closer to tbe preferred orientation of each group. Thus, responses were synchronized between groups 1 and 3, which preferred the vertical stimuSus, and between 2 and 4. which preferred the stimulus oriented at 112". The two assemblies were desynchronized with respect to each other, and so there was no significant synchronization between groups 2 and 3. The crosscorrelograms between groups T and 2. J and 4, and 3 and 4 were also flat (not shown). Note that the segregation cannot be explained by preferential anatomical wiring of ceils with similar orientation preference because cell groups can readily be synchronized in all possible pair combinations in response to a single lightbar. The correlograms are shown superimposed with their Gabor function (see Fig. 1). The number to the upper right of each correlogram indicates tbe relative modulation amplitude. Abbreviation: ns. not significant Scale bars indicate the number of spikes. (Taken from Re'. 37.) 15
150 -60
0 Time (ms)
Altogether, the phenomenon that has been denoted 'oscillation' can be defined as recurrent synchronous bursting oi neuronal groups, with a fluctuation of the burst frequency over a broad range in the gamma band. As wiU be discussed below, the mtrinsie temporal structure of such firing patterns may offer additional advantages for the establishment of synchrony between spatially distributed neurons and, hence, oscillatory signals may be well suited as carrier signals for a temporal code.
Binding within and between visual areas Following the discovery of stimulus-induced oscillations, several predictions made by the correlational model of assembly formation have been tested in the cat. A key prediction is that spatially distributed cell groups should synchronize their responses when activated by a single coherent object. Indeed, recordings made using multiple electrodes have revealed a cross-columnar synchronization of oscillatory responses in area 17, both in anesthetized and awake
693
223
processing streams onto a single integrating cortical area. Substrate of response synchronization Thesefindingsraise the question of how response synchronization is achieved across columns and areas. The data on interhemispheric synchronization suggest that temporal correlation is established at the cortical level and is not triggered by common input from the thalamus . Since the visual afferents to the two hemispheres remain entirely segregated beyond the optic chiasm, there is no common input available and response synchronization between areas in the left and right hemisphere can only be achieved by interhemispheric connections, such as the corpus callosum. Indeed, it has been shown that if the corpus callosum is sectioned prior to recording from the two hemispheres, the response synchronization between with overlapping receptive fields.hemispheres disappears while synchronization within columns responding to vertical either hemisphere is preserved . 27
25,211
a. Assembly fofmafion by cell groups Shaded areas represent bands of orientation contours. If celt groups with ovedappmg receptive fields are stimulated by two independent objects (Fig. 3D), two spatially overlapping assemblies emerge that are desynchronized relative to each other iindicated by dots and circles). Each of these assemblies comprises synchronously firing cells, which is demonstrated directly by recording Irom four different sites in the cortex (Fig. 3D). In fact, the two bars activate most of the cells in this region, because the neurons have a limited orientation selectivity. By simply considering mean activity levels, it would only be known that a number of different feature detectors are active, but not whether they actually respond to two different objects or just a single one. However, with evaluation of the temporal relationships the whole set of acbve cells can be partitioned into two distinct assemblies. Scale bar is 1mm. Abbreviations: HM. representation of the horizontal meridian: VM. representation of the vertical meridian.
Fig.
Further evidence for response synchronization by cortico-cortical connections comes from recent experiments with strabismic cats "- . In area 17 of the visual cortex of the cat, squint is known to induce a breakdown of binocularity *. The two sets of monocular cells tend to be clustered in columns driven by the left or right eye ' ' Physiological and anatomical studies of area 17 in strabismic cats show (1) that response synchronization occurs preferentially between columns of like ocular dominance", and (2) that only columns with the same ocular dominance are linked by tangential connections Thus, a tight correlation between synchronization and topography of intracortical connections is observed. Altogether, eats " - On average, the recorded cell groups these data indicate that synchrony is achieved by synchronize with zero phase lag. even over spatial reciprocal coupling at the cortical level. These results separations of more than 7 mm when activated by have two important implications. (1] They prove Lhat coherent stimuli . reciprocal connections can establish synchrony with Another importani prediction is that response zero phase lag despitefiniteconduction delays , tn synchronization should occur between the two cer- case of the callosal connections, these delays may ebral hemispheres. Since the visual cortex of each amount to several milliseconds . Previously, it has hemisphere receives direct retinal input from only half always been assumed that synchronousfiringwithout of the visual field, interhemispheric synchronization is phase lag could only be achieved by common input . required to bind features of objects that extend into (2) These results assign a new putative function to both hemifields. The occurrence of interhemispheric reciprocal connections within and between cortical response synchronization was discovered recently by areas. Rather than contributing exclusively to recepsimultaneous recordings from area 17 of the left and tive field properties, these connections may actually right cerebral hemispheres and was found to be mediate the binding of distributed activity. similar to synchronization within area 17 with respect to strength and average phase relationship" 3
81
3
1
3
3
31
31 a 1 - 2 6
25,26
53
34
35
Finally, cell groups located in different visual areas should also be capable of synchronizing their responses. This follows if binding of the different features of an object is achieved by response synchronization and if different areas indeed represent different feature domains . Interareal synchronization has so far been observed between areas 17. 18 and 19 |Hefs 21, 22| and between areas 17 and PMLS". Since area PMLS seems to be involved in the analysis of motion whereas area 17 processes fine details of an object, this may be considered as evidence for binding across feature domains. Altogether, these findings suggest that features of visual objects can indeed be linked by synchronization of feature-detecting neurons on a millisecond timescale. By using such a temporal code, distributed activity could be bound across columns, areas and even different hemispheres without the need for 'cardinal cells' or anatomical convergence of 1-8
694
Stimulus dependence of synchronization If a temporal code is indeed used in the cortex for assembly formation and feature binding, it must be shown that correlated firing occurs in a stimulusdependent manner and, moreover, that this mechanism permits the coexistence of different assemblies. Uniform and stereotyped synchrony of neuronal responses could hardly convey any useful information about the coherence of features in visual scenes. Thus, cells in Che cortex should oscillate in synchrony only if they respond to the same object, but otherwise fire in an uncorrelated manner. As shown already by the Gestalt psychologists, criteria such as continuity, proximity, similarity and common fate are used perceptually to group parts into wholes . If temporal correlation between cortical neurons provides a basis for object representations it should reflect these coherence criteria. 36
Evidence for dependence of synchronization on such Gestalt criteria has been obtained in a recent study . It was demonstrated that cell groups in area 17 with nonoverlapping receptive fields oscillate synchronously if they are activated by a single continuous stimulus, whereas their firing is uncorrelated if two independent stimuli are presented that move in different directions (Fig. 1). Thus, when activated by a single stimulus, the cells join one assembly, whereas when they are activated by two stimuli, they couple to two different assemblies that are spatially disjunct and desynchroiuzed relative to each other (Fig. 21, This experiment shows that Gestalt criteria such as continuity of contours and coherent motion are indeed important for the establishment of synchrony. Furthermore, it provides evidence for a stimulus-dependent assembly formation by response synchronization. 25
Similar effects can be observed if the recorded cell groups have overlapping receptive fields (Fig. 3). If stimulated with a single light bar, such cells fire synchronously, even if they differ in their preferred orientation . However, if two superimposed stimuli with different orientation and direction of motion are presented the same cells segregate into two assemblies. Although these assemblies are spatially superimposed (Fig. 4). they can be recognized as distinct because their firing is uncorrelated. These data show that response synchronization allows, in principle, the distinction of several co-existirg representations in the same region of cortex. Hence, they suggest that the superposition problem may be solved by temporal coding . Moreover, these findings underline the flexibility and parsimony of temporal coding as a potential binding mechanism. If the input configuration is altered, the very same cells can recombine into quite different assemblies by changing their temporal relationships. Figure 5 illustrates how this effect can be explained in terms of a simple model .
0'
/
/ /
1
\ \
\
45"
90"
\
—
6
i
135' —
/
/
ii
27
25,
=1
Time
+1p
\ V
\
1415
59
Why should cortical responses be oscillatory? The data obtained in the cat indicate that stimulusinduced synchronization of distributed responses is usually associated with the generation of oscillatory activity. This raises the question of whether the intrinsic temporal structure of such oscillatory signals is relevant for synchronization. As discussed above, the oscillatory multiunit responses observed in the cat visual cortex can be characterized as burst sequences with a broad frequency spectrum. Thus, the distribution of spikes in such responses is certainly not random nor. on the other hand, is it strictly periodic. The limitation of thefrequencyband. i.e. the fact that the interburst intervals do not vary randomly but preferentially adopt values between 15 and 35 ms, has several potential advantages. First, cells engaged in such a rhythmic response pattern can be synchronized more easily by slowly conducting reciprocal connections than cells exhibiting entirely unstructured activity. Response synchronization between the two hemispheres provides a paradigmatic example* . In this case, synchronization without phase lag is possible because the oscillatory signals foster cooperative effects between coupled neurons that improve the synchronization from one burst to the next . It seems hard to imagine how transcallosal synchroniz7
53
-15T
0 Time
+1.5T
-1.5T
0
+1,5T
Time
Fig, 5. Stimulus dependence of synchronization in a neural network model. (A) The network consists of a chain of 1€ oscillators that are assumed to represent orientation-selective neuronal populations with ovedapping receptive fields. Thrs arrangement is similar to that of a cortical 'hypercolumn'. Each oscilfator (symbolized by a square) consists of an excitatory and an inhibitory subpopulation that are mutually coupled by delay connections. The excitatory subpoputation receives external input, which has no intrinsic temporal Structure. Tangential connections between different oscillators have a dual function. Oscillators with similar orientation preference are linked by synchronizing connections. In addition, desyncbronizing connections are implemented between oscillators that differ by 22" to 56" in their orientation preference. Both sets of connections are excitatory, and their differential effect results from the fact that synchronizing connections terminate at inhibitory subpopulatlons, whereas de Synchronizing connections provide input to the excitatory subpopulation of the target oscillators . The networtt has cyclic boundary conditions. (For further details, see Rets 33, 38) (B) In analogy to the physiological experiment illustrated in Fig. 3, the network is tested with two stimulus configurations: either a single bar (left) or two superimposed bars ihght)- IC) Distribution of the input activity along tbe chain of oscillators (ordinate), representing the two stimulus conditions shown in (Q). tO) Activity traces from the two oscillators tinted in (A). Neuronal activity is modeled as a continuous time series that is analogous to a field potential In the case of a Single stimulus (left), the two monitored oscillators are strongly synchronized and are therefore assumed to belong to the same assembly. In contrast, if the network is stimulated by two bars (right) the two oscillators do not have a fixed phase relabonship and are coupled to two different assemblies. Under these input conditions the desyncbronizing connections drive the two assemblies out of phase. Note that within each stimulus configuration, input to the two monitored oscillators is identical (C). (E) Mean normalized autocorrelograms (dashed line) and crosscorrelograms (solid line) of activity traces from the two oscillators shown in (D). The correlograms are averaged over 20 trials, each lasting for 20 cycles. The averaged correlograms confirm that two stimuli induce a segregation of the recorded oscillators into two assemblies. This change of the temporal relationship occurs without any modification of synaptic efficacy Abbreviation: T, period time of the isolated oscillator. 30
695
-125
0
+125
-80
Time (ms)
0
+80
Time (ms]
C
-80
0 Time (ms)
+80
-80
0 Time (ms)
+80
Fig. 6. Oscillatory responses >n extrastnate cortex of the awake macaque monkey. Multiumt responses to moving Sight stimuli were recorded from motion-sensitive areas in the supenor temporal sulcus while the monkey was performing a fixation task. The figure illustrates data from four different recording sites. (AJ, (B) Two examples of response epochs with an oscillatory discharge pattern. Below the spike trams the corresponding autocorrelograms are shown. The presence of multiple peaks and troughs m the autocorrelograms indicates that the burst-and-pause sequences, which we term oscillations, had a comparatively narrow frequency band. Note that spikes of different cells are grouped into bursts. [<2). (D) Autocorrelograms of oscillatory response episodes that were averaged over ten (Qand five (0) trials to the same stimulus, respectively The correlograms were computed m a 500ms window. The average oscillation frequencies were 49 Hz (Q and 33 Hz (D). Each correSogram is shown superimposed with the Gabor function {thick continuous Imei that was fitted to the correlogram to assess the significance of the modulation* (see Figs T, 3). Scale bars indicate tbe number of spikes. (Modified from Ref. AM 6
ation could be achieved with randomly arriving spikes. On the other hand, it seems advantageous that the As shown by simulation studies, a second advan- observed oscillations are not strictly periodic but tage ol osci]lalory activity is that network elements exhibit some jitter of their frequency' . (1) Cell that are not linked directly can be synchronized via groups can be de synchronized more easily if their intermediate oscillators ' . This may be import- oscillations are broad banded so that the network can ant, for instance, to establish relationships between be prevented from entering global states of synchrony different sensory modalities. Finally, coupled oscil- that would be inappropriate for information pro¬ lators can be reliably synchronized with zero phase lag cessing. (2) The number of possible representations even if the conduction delays show a broad that can coexist in the same cortical region is distribution" and are not homogeneous in the net- increased by vanable oscillation frequencies, because work - a constraint that a physiological binding spurious correlations due to aliasing effects will be mechanism certainly has to cope with. In conclusion, rare and only of very short duration. Thus, a broadthe recurrent temporal structure of oscillatory re- banded oscillatory signal appears a reasonable comsponses seems to have crucial advantages for the promise between several opposing constraints. synchronization of spatially separate cells in the cortex. Therefore, we suggest that oscillations, while not conveying stimulus-specific information per se, Evidence for temporal coding in other species may be instrumental as carrier signals for the and modalities A question of particular interest is whether formation of assemblies by a temporal code. synchronization of oscillatory responses can also be 11
11
r,llJ
3 1
696
225
observed in Lhe monkey visual system - in particular, when the animal is alert and performs a behavioural task. Recently. Kreiter and Stager" have discovered oscillatory responses, which can synchronize between spatially separate sites, in the extrastriate cortex of the awake behaving macaque (Fig. 61. As in the cat, coherent bursting of cell groups is a prominent feature of the responses. However, the oscillations seem to be more irregular in the monkey and, as a result, are even more difficult to detect with averaged autocorrelograms. Synchronized oscillatory responses have also been recorded from the striate cortex of anesthetized monkeys '. Furthermore, gamma activity can be observed in EEG recordings from the visual cortex of monkeys , providing additional evidenceforsynchronous activity of large neuronal populations in this frequency range. These findings demonstrate unequivocally that gamma oscillations with the overall characteristics discussed above are present in the primate visual system, and they suggest that temporal codes may be employed in species other than the cat. 1
13
systems, this mechanism may provide a solution to the fundamental problems of scene segmentation and figure-ground segregation. Studies of the visual cortex indicate that segmentation may be achieved by synchronization of thefiringof neurons responding to the same object, while desynchronization occurs when cells code for different objects in the visual field. A similar mechanism might integrate distributed information in the motor system and in association cortices. (3) The available evidence suggests that oscillatory activity, maintained by local groups of coherendy active cells, might be particularly well suited as a carrier signal for a temporal binding mechanism. Oscillations per se do not seem to have a representational function, but they may offer distinct advantages for the establishment of synchrony, especially between widely separated groups. While the data available so far are compatible with the hypothesis that expressing relationships by a temporal code may contribute to the integration of parallel and distributed processes, a number of serious problems remain unresolved. One is related to the 'readout' of temporal codes. Since neurons function essentially as coincidence detectors, they are ideally suited to discriminate between synchronous and asynchronous inputs. Readout of temporally coded assemblies could thus be achieved by networks at other processing stages that are capable of treating coherent and incoherent input differentially. However, it remains an intricate problem how cell assemblies in sensory areas can be linked to assemblies in other centres, especially in those involved in controlling action. Suchfinksmust be highly dynamic and flexible since a particular sensory input can trigger quite different behavioural patterns. Hence, depending on the context, sensory assemblies must be able to couple to quite different sets of cells in effector areas. However, it should be stressed lhal this problem of a flexible readout is not specific to the concept of temporal coding but is of similar significance in the framework of other binding models. The temporal coding strategy advocated here might actually have certain advantages for such a dynamic readout since, as indicated by the data discussed above, binding that is flexible and dependent on context is one of the characteristic features of this mechanism.
Synchronized gamma oscillations have also been observed outside the visual system. Synchronization in this frequency band is well known to occur in the olfactory bulb and entorhtaal cortex of various species". Remarkably, a synchronization of oscillatory activity between somatosensory and motor areas has recently been described in the awake behaving monkey . This suggests the intriguing possibility of crossmodal integration by a temporal code. Moreover, recent magnetic field recordings from human subjects have also provided evidence for synchronization phenomena in the gamma frequency band - . In addition to this growing evidence for response synchronization across spatially separate sites, oscillatory activity has now been reported in a wide variety of structures. These include somatosensory , motor^ and association cortices as well as thalamic nuclei , and also the avian optic tectum . In cases where this oscillatory activity has been observed infieldpotential or EEG recordings it can be assumed that this resulted from the synchronous activation of a large number of neurons. It needs to be emphasized, however, that the mere presence of oscillatory discharge patterns or the mere occurrence of synchronous firing are not yet sufficient evidence to prove the hypothesis of temporal coding. Rather, this requires the demonstration of feature- or action-specific synchronization, which needs to be Even if it were possible to demonstrate that assessed by simultaneous recording from spatially evaluation of temporal relationships could, in prinseparate neurons. ciple, be employed to solve the binding problem and to estabbsh flexible links between cell populations in sensory and motor areas, it remains to be shown that the nervous system actually uses temporal coding. Concluding remarks This article has briefly reviewed recent experimen- Clearly, further experiments on alert and behaving tal findings compatible with the hypothesis that animals are required to establish more direct correintegration in distributed neuronal networks occurs on lations belween the temporal relationship of simulthe basis of a temporal code. Central to the problem of taneously recorded neurons and perceptual or becortical integration is the need to bind distributed havioural performance. Ultimately, in order to prove neuronal activity into organized patterns. A possible their causal relevance it will be necessary to manipusolution to this problem that emerges from the results late these temporal relationships and to demonstrate discussed above can be summarized as follows: (1) predicted changes in behaviour. Binding of cells into functionally coherent assemblies may be achieved by a temporal code using synchronization on a millisecond timescale. A crucial advantage Selected references of this mechanism would be that it permits the 1 Barlow, H. B . (1972) Perteolion 1, S71-394 coexistence of several assemblies in the same region 2 Crick, F (1984) Plot. NatlAad- So. USA 81. 4586-1590 3 Damasio, A B (19901 Semlrt Neuiotci. 2. 287-296 of cortex. (21 In the particular case of sensory 4 Zeki, 5 (1978) Ndtiwe 274, 423-128 15
17
15
5
28
29,19
60
697
126
Acknowledgements 5 Mishkm, M„ Ungerleider, L G. and Macko, K. A. mm Trends Neurosci. 6, 414-417 We think S. towel lor 6 DeYoe. A. E. and van Essen, D. C . (19B8) Trends neurosci. helpful comments an 11. 219-226 an early draft ol the manuscript, and 7 Desimone, R. and Ungerleider. L. G. (1989) in Handbook ot Neuropsychology (Vol. 2) (Boiler, F. and Gralman, I., eds). ft. PM-Vitlsingfor pp. 267-299. Elsevier her help in prepanng B Felleman, D.J. and Van Essen, D. C. (1991) Cereb. Cortex \, the figures. 1-47 9 10 11 12 it 14
15 16 17 1B 19 20 21 22 23 24 25 26 27
28 Lllnis, R. P.. Grace, A. A. and Yarom. V. (1991) Proc. Natl Acad. Sci. USA SB. 897-901 29 Steriade, M., Curro Dossi. R.. Pare. D. and Oakson, C. £1991) Proc. Natl Acad Sci. USA SB. 4396-1400 30 Konig, P., Engel. A. K.. Lowel. 5. and Singer. W. (1990) Sot. Neurosci. Abslr. 16. 1269 31 LBwel, S. and Singer. W. 11992) Science 255, 209-212 32 Hubel, D. H. and Wiesel. T. N. (1965) J. Neurophysiol. 28, 1041-1059 Perrel. D. I., Mlstlin. A. J. and Chitty, A J. (1987) Trends 33 Konig, P. and Sdilllen, T. 8. (1991) Neural Comput 3, 155-166 Neurosci. 10, 358-364 Baylis, C. fcj Ralls. E. T. ind Leonard. C. M. (1985)flrarnRes. 34 Innocent!. G. M. (1980) Arch. Hat Slot 118. 124-188 343, 91-102 35 Ts'o. 0. Y., Gilbert, C. D. and Wiesel. f. N. (1986) t Neurosci. 6. 1160-1170 Rolls, t. T. (1991) Curr. Opin. Neurotnof. 1, 274-278 36 Rock, I. (1983) The loeK of Perception, Mil" Press Hebb, D. O. (1949) The Organiiatton of Behavior. Wiley Palm, G. 11982) Neural Assembles. Springer-Veda! 37 Engel, A. K„ Konig. P and Singer. W (1991) Proc Natl Acid. Set U S * 88. 9136-9140 von der Malsburg. C- (1981) The Correlation Theory of Brain Function (Internal Report 81-2). Max-Planck-Institute lor 38 Schillen, T. B. and Konig. P. (1991) Neural Comput 3. Biophysical Chemistry 167-179 von der Malsburg, C. (19S6) in Brain Tneoiy (Palm. G- and 39 Spoms. CV, Gaily. J. A.. Reeke, G. N. and Edelman. C . M. (1989) Proc. Natl Acad. Sci. USA 86. 7265-7269 Aertsen, A., eds). pp. 161-176. Spnnger-Verlag 40 Sompolinsky, H„ Colon*. D. and Klelnleld, 0 (1990) Proc. Abeles, M. (1982) Local Cortical Circuits. Spnnger- Verlag Nad Acad Sci. USA 87, 7200-7204 Gray. C. M. and Singer. W. (1987) Soc. Neurosci. Abstr. 13, 1449 41 Kreiter. A. K. and Singer. W (1992) fur I Neurosci. A, 369-375 Cray, C M. and Singer, W. (1989) Proc Natl Acad. So. USA 42 Livingstone. M. S. (1991) Soc. Neurosci. Abstr. 17. 176 86, 1698-1702 Cray, C . M „ Engel. A. K . Konig, P and Singer. W. (1990) 43 Freeman, w . I. and von Dijk. B. W (19B7) Brain Res 422. 267-276 fur. J. Neurosci. 2, 607-619 44 Freeman, W. J. (1975) Mass Action in the Nervous System. Edelman, C . M (19B7) Neural Darwinism Basic Books Academic Press Eckhorn, R etal. (1988) Biol. Cybern. 60. 121-130 Kruse. W . Eckhorn, R and Bauer. R. (1990) Perception 19. 45 Murthy. V N. and Feb, E. E (19911 Soc. NeurosrJ. Abstr. 17, 377 310 Engel, A. K., kreiter, A. K.. Konig, P. and Singer, W. (1991) 46 Pantev, C . ei tt (1991) Proc Natl Acad. Sci. USA 88. Proc. Natl And sci. USA 88, 6048-6052 8996-9000 d a y , C. M.. Raether, A. and Singer. W. 11989) Soc. Neurosci. 47 Ribary, U. el al (1991) Proc. Nad Acad Sci. USA 88. Abslr. 15, 79B 11037-11041 Cray, C. M.. Konig, P., Engel, A. K. and Singer. W. (1989) 48 Ahissar, E. and Vaadia, E. (1990) Proc. Natl Acad. Sci. USA Nalure 338. 334-337 87, B935-8939 Engel, A K , Konig, P., Cray, C . H . and Singer, W (1990) 49 Ghose, G. M. and Freeman, R. D. (1990) Soc. Neurosci Eur. / Neurosci 2, 588-606 Abstr. 16, 1270 Engel, A. K., Krjnig, P.. Kreiter. A. K. and Singer, W. (1991) 50 Neuenscbwander, S. and Varela. F. J. (1990) Soc. Neurosci Science 252, 1177-1179 Abstr 16. 109
698
Chapter 5
D E B A T E S AND SPECULATIONS
Our attitude in the choice of articles has significantly evolved for this last chapter. I n the spirit of "Se non e vero, e ben trovato", we admitted articles that are, each one in its own way, more incisive or even polemical, more audacious or even at times extravagant, more ambitious and far-fetched. In this manner the book makes a return to some of the grand issues introduced in Chapter 1 and draws some long-term perspectives, pointing to unresolved issues, puzzles and enigmas. In section 5a we found amusing the twin papers, one from two neurobiologists, the other from a theoretical physicist, that seem to advocate opposite scientific strategies, each one with good reasons. Section 5b focuses on the role of retroactivation, the functions of these pervasive feedback pathways in the brain. Francis Crick is a bold physicist, bold enough to be one of the founders of molecular biology, who turned to brain theory in the seventies, and leaves nobody indifferent. Rolls is a theoretically inclined neurophysiologist; Damasio is a neuropsychologist; and Mumford is a mathematician distinguished for his works in algebraic geometry, who draws here an ambitious fresco, perhaps the most ambitious since David Marr's book on Vision {Freeman, 1983). Section 5c contains three articles that offer some new proposals for biological computational strategies. Anderson and Van Essen (a leader in primate visual anatomy) deal with early vision and attention. Hopfield addresses the computations of olfactory animals. Poggio (an expert in cybernetics and artificial intelbgence) brings ideas, from applied mathematics, to bear on how the brain might solve function interpolation tasks, and trade off memory for computing power. T h i s chapter ends in section 5d with three papers of two already en-
699
countered biologists, Barlow and Hebb. They write with sober eloquence about the nature of intelligence, language, consciousness and other remaining mysteries.
700
5a Theory-Experiment Interplay
It was tempting to join within the same section two polemical articles, one against the "hegemony of the single neuron", the other "in defence of single electrode recordings". The reader, who dislikes expression of personal style in scientific communication, may feel uneasy but he will nevertheless draw benefit from a number of insights into current issues at stake, and also from several precise suggestions. Article (56), by Douglas and Martin, is not actually advocating multielectrode recordings, but asking for more regard to anatomy (in the spirit of Braitenberg, articles (23) and (50)), and in particular to cortical microcircuitry, rather than merely to macrocircuitry between areas (treated as boxes). T h e question would shift from: What are the neural selectivities? to: How do such selectivities emerge from neural connections? The article also brings useful quantitative estimates about local processing in cortical neurons, and about neural coding. Article (57), by Daniel Amit, is not an attack on multi-electrode recordings, and it is certainly not in support of the single neuron doctrine. It is actually fostering investigations of collective neural phenomena: cell assemblies, attractor states, reverberations, as substrates of the processing involved in higher brain functions. T h e apparent paradox which the author strives to elucidate — from knowledge gained in the statistical mechanics of magnetic systems — is that single electrode recordings, in the framework of delayed matching tasks (as studied by Miyashita), can reveal much more on collective processing than commonly expected.
701
R«prui<ed wilh permission from Trends in Neuroscitnces. Vol. 14. No. 7, pp. 286-293, 1 » ] S f f l l E l s e ™ Scicna Fubluhes Lid (UK)
Opening the grey box Rodney J. Douglas and Kevan A. C. Martin ftcdneyt. Douglas- The single neurone has been the guiding light lor generations able ot to use the building blocks of centre-surround and Kevan A C. neuroscientists Now there are signs Irom experimental and theoretireceptive fields, provided by the thalamic relay Martin are at Ihe neurones, to construct the simple orientation- and cal wo/ion the neocortex that we are on the threshold of a revolution MPC Anatomical in which the hegemony ot Ihe single neurone will be replaceddirection-selective receptive fields of neurones in by Neuropharmacology much more circuit-oriented concepts. We consider here why traditionlayer 4 of the primary visual cortex. In their view, the Unit Deptof role of intracortical connections was to relay the al views ot Ihe significance ol single mrottts are lading in power, Pharmacology. South and consider the problem ol deciding on the form ol a new order.simple fields to other cortical layers to form more Paris Poad. Oxford complex receptive fields. This view, together with O i l 301 UK.
Contemporary neuroscientists cut their milk teeth their discovery that neurones with similar orienon the concept that the single neurone is the tation and ocular dominance were located in the fundamental unit of organization in the nervous same vertical column, gave rise to hypothetical system. The idea of the neurone as a 'unit' came out circuits that were inevitably strongly thai am ocen trie, of the anatomical revolution that was precipitated serial and hierarchical (Fig. 1). by the Golgi studies of Ramon y Cajal . The concept The inventiveness of Hubel and Wiesel in providwas translated into physiology through the spinal ing logical microcircuits to explain their physiological cord studies of Sherrington" and Eccles', and It data had far-reaching consequences Most signifiunderpinned the spectacular successes of Mount- cantly, physiologists, not neu roan atomists, took the castle and Hubel and Wiesel in the neocortex. Bar- lead in the analysis of cortical microcircuitry. Taking low extended the concept into psychology with hrs their cue from Hubel and Wiesel, cortical physiol'neurone doctrine'. In his view the activity of single ogists systematically catalogued the response cortical neurones was the elemental basis of per- characteristics of single units in the belief that this ceptual experience. He proposed that a substantial would reveal the underiying architecture of cortical proportion of the neurones i n the neocortex were processing. For over three decades this programme 'cardinal' neurones, each coding for individual provided a rich, albeit frequently contradictory, depercepts. scription of the different properties of individual cortical neurones (see Ref. 7). It is now clear that the programme has made little impact on even such Origin of species basic questions as the circuitry underlying orientaIn the eady days of single unit physiology in the tion selectivity in simple and complex cells. visual cortex, concerns about the circuitry underlying the single unit responses were paramount. Hubel and Wlesel's view of the cortical microcir- Cortical cartography cuifry, produced 30 years ago , continues to domiThe techniques used to map the primary visual nate Their hypothetical microcircuits (Fig. 1), now cortex were soon extended to the eitrastriate solidified as textbook diagrams, were derived not cortex The realization that cortical cartography from microanatomy but from the most parsi- might provide important functional insights, monious functional interpretation of the receptive together with the development of non-alchemeic field structure of single units within the thalamus methods of tracing anatomical connections, led to a and the striate visual cortex. They were more prolific period of cortical research "' . The success successful than anyone before or since, because of the cartographers is evident in their macrocircuit they defined quite precisely the nature of the maps, which are now of si I icon-chip complexity (Fig. thalamic input to the visual cortex. Thus, they were 2). In their macrocircuits, the different cortical areas 5
1
4
5
5
6
5
8-15
1£>
702
5
287 appear as boxes, enigmatically labelled V1, MT, TEO and the like' -' . As tbe connections between the boxes came increasingly to dominate the picture, the original view of a simple serial hierarchy of visual areas with increasingly selective single units was supplanted by poorly-defined notions of parallelism, feedforward, feedback, and multi-stage integration"-' . 4
5
5
Despite an intensive search by physiologists, the boxes of the macrocircuit diagrams were not found to be filled with Barlow's cardinal neurones. However, this discovery was no more than a temporary setback, for the single neurone managed to retain its traditional doctrinal role of explaining high-order visual properties, e.g. stereopsis, motion, colour, in terms of the selective responses to visual stimuli . Although the cartogram of the extrastriatc cortex became clearer, the single unit and anatomical studies failed to extend our knowledge of the intrinsic organization of these areas: they remained grey boxes, their contents only partially specified One reason for this is that the neurone doctrine gave no encouragement to look beyond the phenomenology of single unit responses to either mechanisms or to microcircuits 1617
18
Fig. 1. Hubel anc/ Wiesel's hypothetical circuits for (A) simple and {BI complex cells . The simple receptive fields arise from tbe convergence of geniculate afferents onto a single target neurone in the cortex. Many simple cells converge to form a complex receptor field. 5
Artificial intelligentsia A further disincentive to investigate cortical microcircuits arrived from the unexpected quarter of artificial intelligence (Al). Because of the popular expectation that processing in the nervous system was analogous to formal computation, the principles of computer science were applied to the brain. Thus Marr , in his popularization of formal Al. was able to sell the notion that computation and implementation can be separated and considered entirely independently. He defined three levels of analysis for computational systems: abstract problem analysis, algorithm, and physical implementation. He argued that the most efficient strategy for solving the computational problems of neuroscience was to move downwards from the top level to the bottom. The clear implication of his top-down approach was that the biological implementation did not matter, at least at this stage in the theoretical development. In practice, Marr paid closer attention to the biology than his philosophy seemed to require.
powerful argument to the debate on neocortical processing: the same set of rather simple 'units' can do a wide range of clever things as a collective. However, the early promise of the new connectionist networks has been tempered by scepticism as to their biological relevance, particularly since little effort has been made to establish the equivalence of theoretical neural networks to the microcircuits of the neocortex". The models have been long on simulation, but short on analysis. For example, much has been made of the similarity of the cortical receptive fields to those of units in the hidden layer' of the connectionist networks. Yet for some tasks it was possible to remove the entire hidden layer with little effect on the performance of the network '. Most comparisons of network performance with biological data have been qualitative. The views of the data often seem to owe more to the writers of Other theoreticians saw the advantages of a textbooks than to the original research papers. For closer relationship with neurobiology. Prominent example, the popular image of simple cells as striped among these were the 'connectionists', who formed rectangles belies the true complexity of their the most popular movement in brain simulation response properties . Similarly, the receptive fields since the resurgence of formal Al in the 1970s (Ref. of 'face cells' of inferotemporal cortex", or the 20). The circuit they commonly used for their 'colour cells' of area V4 (Ref. 26), are far more comsimulations was a triple-layered reincarnation of plex than their common descriptions suggest. Rosenblatt's much-maligned 'perceptron' . Unlike Although textbook caricatures are surely easier the formal Al models, connectionist models had the to model than the real thing, closer attention could inestimable attraction of being 'brain-flavoured', be paid to what actually is to be modelled. even to the extent of using networks built of highly simplified neurones'. The results were intriguing and provided remarkable simulations of motor, perceptual and cognitive pracesses" '" (Fig. 3). In Getting physical Just as in analogue computation, where the focusing attention back onto the theoretical and computational problems of modelling large ensem- physical process is the computation, so in the bles of neurones, the connectionists contributed one neocortex the physical structure provides us with constraints that are so conspicuously absent from 19
2
5,7
17!a
21
-
703
activity occurring in various dendritic branches converges onto a final common output from the soma. However, the extent to which a dendritic tree is organized into local subunits that permit independent regional computations is debatable. There is no rigorous definition of a local computational subunit, but if the dendritic tree is to be composed of subunits, then the following minimum conditions must hold First, the subunits must be electroton ical I y decoupled from one another. This condition requires that the attenuation between synapses within a subunit is small, but that the attenuation between synapses in different subunits is large. Second, the subunits must be decoupled from the soma, which is the final, independent output of the neurone. Finally, inputs from different sources should map onto specific regions Fig. 3. Tbe wiring diagram of the cortical visual areas In the macaque monkey: (A) Ills 1983 of the dendritic tree, and these version" and IB) the 1990 version'*. regions should be of the same order of size as the subunits. Koch, Poggio and Torre' have shown theoretically that the dendritic tree of retinal ganglion cells current 'computational' approaches. It is precisely a might contain as many as 20-30 electrotonic belief in the indivisibility of structure and function in subunits. Although similar studies have not yet been tbe neocortex that now drives us back to confront performed on cortical neurones, analyses of morthe cortical microcircuits and to ask in what sense phologically identified cortical pyramidal neurones they can be said to transform their biology into have shown that their dendritic structure is unsuitcomputation. Should individual neurones be able for a high degree of compartmentalregarded as a collection of independent processors ization . Their dendrites are electrotonically performing separate logical operations , or as a shorter, thicker, and less branched than those of single processing unit whose output has a symbolic ganglion cells, and many synapses are made on interpretation in its own right ? Or should they be proximal dendrites ' . These factors do not favour regarded as members of a population of inter- decoupling. Moreover, there is no anatomical eviconnected neurones whose symbolic significance dence that excitatory inputs from different sources lies in their collective action ? In other words, what cluster on particular regions of the dendritic tree, as simplifying assumptions are permissible, and what they do in the hippocampus. The clearest segrefacets of cortical structure and function simply gation of inputs is seen in the GABAergic cells, cannot be factored out of the equations and must which can be extraordinarily selective in their choice therefore form an essential part of our theoretical of postsynaptic target . models? 1
9
13,35,36
39
6
37
38
50
18,JS
One of the first issues that needs to be decided upon, for both experimental and theoretical purposes, is where the essential computations are physically happening. For instance, many speculations have been forthcoming about the potential processing power of a single cortical neurone, based on the complexity of the dendritic tree (Fig. 4). Thus, the geometry of the dendritic tree of cortical neurones and the dynamic properties that arise from the interaction of synaptic, ion- and voltage sensitive conductances and second messenger systems, may require us to consider each neurone as a sophisticated microprocessor "". 31
Subunlx Oendro-dendritic junctions have rarely been seen in neocortex . Thus, the dendrites of neocortical neurones can only act as a processing tree where the 34
704
Dendritic spines, however, could permit a high degree of local processing. Spines are the sites of most of the excitatory input for pyramidal neurones, which constitute 75% of neurones In the neocortex . It has been shown theoretically that the logical AND-NOT operation can be generated by an inhibitory and excitatory input located on the same spine, i.e. excitation only passes to the parent dendrite if there is excitation and not inhibition . This spine-based device has been proposed as the basis of direction and orientation selectivity in the visual cortex . The hypothesis is attractive because such a logic device on spines would greatly increase the potential processing capabilities of single neurones, since individual neurones contain between 1000 and 10000 spines (Anderson, J. C, unpublished observations). 38
32
32,33,40
Unfortunately, recording from single spine; is not yet possible, so we cannot do the physiological experiments required to test whether this logical operation is in fact being carried out. However, we can establish whether the necessary circuitry exists, i.e. whether single spines receive an input from both inhibitory and excitatory synapses. The answer that has emerged from extremely laborious ultra structural studies ' is that dual inputs to spines are found, but on less than 10% of spines. This means that the excitatory input to most spines is transferred to the shaft of the parent dendrite unscathed by local gating. Thus, there is no structural reason for believing that cortical neurones are fractionated into large numbers of spine-based processing units. 41
OUTPUT
3
42
2
Neuronal effluence The arguments presented so far suggest that, at best, cortical neurones have only a few local computational subunits. However, each neurone • 4 cannot even encode a particular configuration of a few active subunits in its discharge rates: the discharge of the initial segment of the axon is driven only by the total intrasomatic current, which is inevitably insensitive to the particular subunits that are its source. An alternative role of subunits could be INPUT that the dendritic tree computes just a single function many times over, so that the final effect on the soma can be interpreted simply as the degree to Fig. 3, Connectionist network of three layers used to which that function is present, i.e. if local compu- produce speech from text* , to determine surface tational subunits are indeed present on the dendrites, curvature , and to decode spatial information* . Layer their role might not be to increase the compu- two is the layer ot 'hidden units'. Unlike neocortex, there tational complexity of the neurone, but rather to are no lateral connections between units within a layer. assess the reliability of a single computation by averaging over the available subunits. 1
11
4
Of course we need not appeal to dendritic morphology and dynamics in order to enhance the computational complexity of the cortical neurone. Even if we treat dendrites as simple Iransmission lines, each neurone could serve as a formidable threshold logic device. A typical spiny neurone receives on the order of 5000 excitatory synaptic inputs and probably requires about 100 active inputs to generate an output discharge '". Consequently, there is an astronomical number of combinations of input that could potentially be signalled. The bottleneck is the output of the neurone, because the information capacity of a rate-en coded output is only about 4.5 bits One recurring suggestion is that an interval code could add considerably to the capacity of the neurone to transmit information This notion, however, has inherent deficiencies of its own.
code"^ . Even if the output appears to be coded, it is difficult to see how the recipient neurones could interpret such a code in anything but the crudest terms. The standard deviation of the interspike intervals of most neurones is roughly the same order as the mean interval, and so a temporal code would be particularly unreliable at low frequencies of discharge. On the other hand, the low-pass filter inherent in the time constant of the postsynaptic membrane distorts the intervals at high frequencies. For example, the time constants of cortical neurones are 10-20 ms (Refs 18. 36) or longer, and this implies that signal components higher than about 15 Hz are attenuated . However, during responses to natural stimuli, the action potentials must arrive in any terminal bouton at rates well above 30 per second (see Ref. 7). Thus the recipient neurone will be unable to distinguish the fine structure of synaptic events during optimal stimulation.
Noise and vibrations A significant component of interspike interval variability is due to noise. Indeed, many neurones have interspike interval distributions that closely fit low-order gamma distributions, indicating that the variability in interspike intervals is fully explained by the stochastic nature of the spike generation process. On the other hand, some authors argue that a fine structure exists within the discharge of certain neurones and that it constitutes a neural
From the view of a given neurone, the arrival of action potentials from hundreds of other neurones will produce a spatio-temporal dispersion of the synaptic currents across the dendritic tree. Accordingly, the net excitatory current that eventually arrives at the initial soma segment, where the action potentials are generated, will last far longer than any single synaptic event . Individuality of inputs cannot be preserved under these conditions: the output must inevitably reflect a high degree of
18
6
18
705
19
190
discharge seen in response to natural stimuli. Coupled to all this are the profound difficulties of interpreting highly technical experiments and understanding their limitations, even if one is personally advancing the microelectrode into the brain or looking down the microscope. Taken altogether, it seems too much to ask any theorist who is developing high-level models to take even a few of the experimental facts into account. However, such a view forces us back onto the same philosophical cart as the adherents of formal Al, whose theories were largely unconstrained by brain hardware. Although the separation of theory from implementation may have been an acceptable philosophy for the revivalist meetings of formal Al in the 1970s, it is certainly an irrelevant philosophy if the theory is to be neurally based. In fact, most current models are timid. They are afraid to come to terms with the large body of data on cortical circuits that has accumulated over the past ten years and they refuse to recognize the paper thinness of their Fig. A, The complex logical possibilities contained in biological content. The usual excuse is that all that combinations of excitatory ie) and inhibitory (i) synapses wet stuff is too complicated and. anyway, experon a branched dendritic tree: a theoretical scenario imentalists don't even agree on their own data, do they? Nevertheless, one of the most pressing tasks is produced by Koch, Poggio and Torre isee Ref. 29). to reconnect theory with the experimental data at a level that provides the maximum communication. temporal and spatial summation. At every turn, The best meeting place is now the microcircuit, then, we are laced with the problem of understand- not the single neurone. ing the concerted action of many neurones. Neuronal collectives Reconnecting to reality If we are to move to any deeper understanding of So, after 30 years of single cell recording and cortical microcircuits. we need the analytical and macroscopic circuit tracing, we arrive back at the synthetic techniques of dealing with large numbers place we first began: what microcircuits? The of interconnected neurones. Our present hypotheti- anatomical investigation of cortical microcircuitry cal circuits would not even pass quality control on had been forced into the background during the the production line for the 'mind of the worm' heady days of single unit physiology, but it had not Caenorhabditis elegans' . The burgeoning field stagnated. On the contrary, a palace revolution called 'computational neuroscience' would seem the had taken place. The traditional methods of descripobvious route for progress , but as yet it has not tive Golgi studies and jigsaw puzzle anatomy were been such. For the present it seems that the greatest deposed- In their stead were powerful tools with immediate gains in understanding the collective which to analyse cortical microcircuits: methods for action of neurones will come from examination of analysing the synaptic connections of identified the basic organization of neural connections and neuronal types and characterizing their putative their approximate functional weights. It is exactly to neurotransmitters . The introduction of intracelluthis aspect that computational neuroscientists have lar recording and staining of individual neurones paid the least attention. enabled structure and function to be linked directly . The results revealed that the single unit The reasons for this neglect are many. One is the recordings had stripped the neuropil bare, leaving us difficulty in adequately assessing data coming from only with visions of balloon-like neurones trailing many different levels of experimental work. The defoliated axons. relevant mi c roan atom ical data are often presented in the form of inscrutable electron micrographs, The new structural studies showed that the reality accompanied by detailed morphometry measure- was very different. Every cortical neurone, even ments that provide little insight into how the those projecting out of the local cortical area, had physiological properties of neurones arise. Similady, a complex dendritic tree and a local network of physiologists continue to debate the presence or axon collaterals that branched profusely, forming absence of inhibition in cortical function (see Ref. synapses on hundreds or thousands of nearby 49), seemingly oblivious of the neurocytochemists' neurones" - To make matters worse, both the demonstration that one in five neurones and one thalamic afferents and the cortical pyramidal neurin six synaptic boutons contain the inhibitory ones formed only a few synapses with any target neurotransmitter CABA, and that every neurone in neurone - , again a sharp contrast to the views of the cortex receives a rich OABAergic input . The the connectivity that were forthcoming from single channels revealed by the patch-clampers' pipette unit physiology (see Ref. 471. Thus, the excitatory seem only distantly related to the action potential drive to any single neurone was provided by the 7
33
51,53
545 5
52
50
51
49
49
706
56
53
191 convergent action of hundred; of neurones The system engineer's nightmare - positive feedback appeared everywhere; over 8 5 % of the connections made by excitatory neurones were to other excitatory neurones . The traditional idea that the main role of the axon collaterals of pyramidal cells was to provide recurrent inhibition through inhibitory interneurones was dead For those lulled into a sense of security by the simplicity of the textbook cortical circuits, these new anatomical images were profoundly disturbing. 5753
Level pegging As we have crudely demonstrated here, it is now possible for human neural networks, using no more than neuroxed literature and a silicon microprocessor running at only 6 MHz, to begin to define levels at which effective syntheses can be made of the cortical circuits. The conclusion we have come to is that the anatomy, the biophysics and theory all concur: we do not now need to model the neuronal interactions on a synapse-by-synapse basis, or model sophisticated logical operations occurring over a complex dendritic tree, or decipher a digital spike code, or be concerned with fine gradations in L5 pyramid Curreni (nA| the output of individual neurones. Instead we need to consider large clusters of highly interconnected Fig. 5, Conversion ol actual cortical pyramidal neurones into simplrlied neurones, each performing faidy simple tasks and idealized neurones (A) for use in a theoretical study of the effects ol axoevolving a response over many milliseconds The axonic, or basket cell inhibition on pyramidal cells (B). The current-discharge response of individual neurones then reflects not its curves (C) ot the model cells were close to those of actual neurones" own specific interest but rather the complex behaviour of a microcircuit containing many neurones ' structure, function and theory, thereby avoiding some of the classic errors that the segregation of Such a programme seems to go against our these streams has produced in the past. For examinclinations to be reductionist By making the ple, the cortical microcircuits have traditionally been microcircuit the focus of analysis, we might appear seen as slaves of the thalamus , but this is a to ignore the details of channels, receptors, indi- powerful fantasy conjured up by the electrophysividual synapses, or the morphological distinctions ologist's imagination Anatomists find that thalamic between the component neurones that make up the axons contribute no moie than about 20% of microcircuits Our reply is that the programme we the excitatory synapses In layer A. the main layer of suggest is actually highly integrative. No level can or termination of the thalamic axons Most of the should be ignored: history makes it clear that to do remaining excitatory synapses are from intracortical so is folly The essential step is to go beyond the sources . No explanations have yet been forthcomcurrent narrow focus on the single neurone, which ing from physiologists as to why the intracortcal until now was the only thread that linked the components of excitation have been so effectively different levels of investigation For the future we excluded or relegated to the minor roles in the need a framework that allows many more links microcircuits This is particulariy strange when the inhibitory components, which form only 15% of the between the various levels cortical synapses , are often invested with enorOne framework we have in mind would be mous functional power and appear like confetti something akin to logical tree structures, where the sprinkled through the different models' details of individual components are recursively 'folded back' to a single parent stem, which then effectively represents the group of components. In Revolution and evolution this way the description can become highly simpliThe predominance of the circuits derived from fied and yet include accurate and detailed infor- single unit physiology means that our views of the mation obtained at other levels This 'minimalist' ap- microcircuits of the visual cortex are tightly bound to proach"'* (Fig. 5) contrasts with the more particular functions, such as orientation and direcelaborate style of modelling advanced by others . tionality. However, for many anatomists, it seems The minimalist approach is of course very different perverse to regard the visual cortex as an ad hoc from that of more traditional computational models, collection of specialist circuits, rather than a set of where divisions are drawn between 'simple' or 'real' basic circuits adapted to perform many different models**. tasks* Nevertheless, we should certainly not expect the present course of experimental and For the neocortex such an approach also offers theoretical work to provide us with any early the real possibility of forging tight links between 5
5
17
60
51
17
0
61
5
707
61-45
2K
Acknowledgements answers as to how neurones become orientation Our thanks to the selective, or direction selective. Although the solmany allies (not tent ution of any one of these problems would be very DrM. F. Khtzer) who impressive, given the trail of failed attempts, it is also encouraged ind not self evident that we would thereby arrive at any guided the general rules about the operation of the cortical progressive circuits. development el the All this might seem to be a rather gloomy view of paper. KACM is Ihe Henry Head Research what is universally judged to be an inordinately successful programme of research, but it implies, Fellow cf the Royal Society. paradoxically, that we have not been concerned
enough about the detail, and that we have been too committed to trying to understand the specificities of particular functions in isolation from others Part of the solution will lie in devoting more theoretical work to understanding the general properties of cortical processing, and more experimental work to the basic mechanisms and circuits that are available for these processes. In this new order, theorists will no longer gain street credibility with n euro biologists by claiming their theory is neurally inspired'. Principles of cortical operation cannot be extracted from simulated circuits that bear distant resemblances to those of the cortex. If the theory is biologically relevant, it needs to be embedded in the stuff of neocortex, period. Similarly, experimentalists will no longer be able to hide behind a tradition of informal descriptions and stick-and-ball models of cortical circuits. For the neocortex, an unconventional class of models needs to be developed - models that are neural networks, but based directly on the biology; derived from visual cortex, but not designed to solve a particular problem in visual processing. Such hybrid models would differ fundamentally from the textbook views of the neocortical microcircuits in dealing not only with the problems of large arrays of highly interconnected neurones, but in taking on board the richness of the biological data. For the first time in a century of neuroscience we can see an end to the conceptual domination of the single neurone and its replacement by the microcircuit. The integration of theory and experiment that has been the power of the natural sciences might now produce its own revolution in the neurosciences. For neocortical research in particular, the tragically Long period of independent evolution of theory and experiment should ensure that the union of these two streams produces offspring with hybrid vigour. Selected references 1 Ramon y Ca|a! S. 11954) Neurone Theory or Reticular Theory? Obfective Evidence of Ike Anatomical Unity ot Nerve Cells (translated by Uheda Purkiss. M. and Fo., C A.), Consejo Superior tie Invesligaciones Cientihcas 2 Sherrington, C. S. (1941) Man on his Nature. Cambridge University Press 3 Eccles. J. C. (1964) The Physiology ol Synapses. Springer 4 Powell. T . P. S. and Mountcastle. V. B. (1959) Bull, lohns Hopkins Help 105, 133-162 5 Hubel. D. H. and Wiesel, T . rJ. (1962) I. Physiol. 160, 106-154 6 Barlow, H . B (1972) Perception 1. 3 7 1 - 3 9 4 7 Orban. G. (1984) Neuronal Operations in the Visual Coriei. Springer B Hubel, 0. H and Wiesel, T N. ( 1 9 6 5 ) J. Neurophysiol. 28.
708
229-289 9 Martin. K. A. C. ( 1 9 8 8 ) Trends Neurosci. 11, 10 Van Essen, D- C. and Maunsell, J. H. Ft. Neurosci. 6, 370-375 11 Ungerleidei, L G- and Mishkin. M. (1962) Visual Behavior (Ingle. D. J., Goodale. M. A. Ft J. W . eds). pp. 549-5B6, MIT Press
380-367 (1963) Trends In Analysis ot and Mansfield,
12 De Voe, t. A. and Van Essen, 0. C. ( 1 9 6 8 ) Trends Neurosci. 11. 219-226 13 Zeki, S. M. (197B) Nature 274. 423-138 14 Zeki, S. and Shipp, S. (19B9) Nature 335. 311-317 15 Van Essen, D. C„ Felleman, D. 1.. De Yoe. E. A., Olavarria, I. and Knierim, J. Cold Spring Harbor Symp Quant Biol. 55 (in press) 16 Hubel, D. H. and tivingstone, M. S. ( 1 9 3 7 ) 1. Neurosci. 7, 337B-3415 17 Livingstone. M S. and Hubel, D. H . (1987) J. Neurosci 7. 3416-3468 18 Douglas, fl. J. and Martin, • A. C. (1990) in The Synaptic Organisation ot the Brain (Shepherd. G. M., ed.), pp. 389-43B. Okford University Press 19 Marr. D. (1982) Vision. W. H. Freeman 20 McClelland, J. L anil Rumelhart, D. E. 11966) Parallel Distributed Processing (Vol 21 Bradford Books 21 Minsky, M. and Papert, S (1969) Perceptrons. An Introduction to Computation Ceometry. MIT Press 22 Sejnowski, T.I. and Rosenberg. C. R. (1987) Complex Syst. 2, 145 23 Lehfcy. S fl and Se|no™ski. T. J (1990) Proc. P Soc London Sen 8 240, 251-278 24 Zipser, D. and Andersen. R. A (1988) Nature 331. 679-684 25 Gross, C . Rocha-Miranda. C. and Bender, D. (1972) / Neurophysiol. 35. 96-111 26 Zeki. S. M. (1973) Brain Hes. 5 3 , 422-427 27 fllcfimond, B. J., Optican, L. M., Podell. M. and Spitzer. H. (19B7) I Neuropkysiol. 57. 132-146 28 Fischer. B.. Boch. R. and Bach, M. (1981) Exp. Brain Res. 4 3 , 69-77 29 K0
293 (1986) Philos. T,ins. B. Soc London Ser. 8 314, 1-340 48 Sejnowski, 7. J., Koch, C. arid ChuKhland. P. S. (1988) Science 241, 121-244 49 Martin, K. A. C. (1988) tt J. t'p. HMeaW 73, 637-702 50 Gaboon, P L. A, andSomoeyi, P. (1986) f«p. Brain Pes. 61. 323-331 51 Beaulieu, C . and Sornogyi, P. (1990) Eur. I. Neuropil. 2. 296-303 52 Somogyi, P. and Freund. T. P. (1969) in Neuroanatomical Tract-Tracing Methods (Vol. 2) (Heimer. L. and Zaborszlry. L.. eds), pp. 239-264, Plenum Press 53 Somogyi. P. (1990) J. electron. Microsc. Tech. 15. 332-351 54 Gilbert C . D. and Wiesel, T. N (1979) Nature2B0. 120-125 55 Martin, K. A. C and Whittendge. D (19B4) J. Wiysiol. 353. 463-504 56 Preund. T. F.. Martin, K. A. C . Somogyi, P. and Whitleridge.
D. (1965) /. Comp Neurol. 242. 275-291 57 Kisvarday. 2. F. e l al. (1986) Itp. Brain Bes. 64, 541-552 58 Gabbott, P., Martin. K. A. C. and Whitleridge, O. (19B71 / Comp. Neural. 259. 364-381 59 Douglas. R ).. Martin. K. A C and Whittendge. D. (19B9) Neural Comp. 1. 430-48B 60 Douglas. R. ). and Martin. K. A. C. (1990) Neural Comp. 2. 283-292 61 Wehmeier. U.. Dong. O., Koch. C and Van Essen, D. (1989) in Methods in Neuronal Modelling: Irom Synapses to Networks, pp. 335-360. Bradford Books 62 Szentagothai. J. 11978) Proc. it. Soc. London Ser 3 201. 219-24B 63 Powell. T. P. S. (1981) in Brain Mechanisms and Perceptual Awareness (Pompeiana. O. and A|mone Marsan. C . eds). pp. 1-19, Raven Press
709
Reprinted with permission from Network, Vol. 3, pp. 385-391,1992 ©1992 10P Publishing Ltd.
In defence of single-electrode recordings Daniel J Arnitt INFN, Sezione di Roma and Istimto di Fisica, Universita di Roma, 'La Sapienza', Pie Aldo Moro 2, 1-00185 Roma, Italy
Received 17 March 1992
Abstract. As a physicist observing Ihe empirical struggle of neurophysiology (o penetrate the computational secrets of neo-conex, one is struck by the coexistence of two extreme positions. One has it that ihe compulations performed in neo-cortex are too complex to be approachable by single-electrode recordings. The other is continuously discovering single neurons which can perform such complex tasks as to recognize the individual faces of Ihe senior staff in the laboratory, •rawing on experience gained in ihe study of condensed matter. I would like lo reopen the discussion, pointing out lhat some of Ihe single-electrode recordings which are exposing such remarkable computational features could not possibly be of great relevance for the description of 'higher brain function'. On Ihe other hand, some singleelectrode experiments, which detect rather mundane features of performance, display remarkable dynamical features, which must surely underlie cortical function. All this leads us to the conclusion lhal, though simultaneous electrode recordings may be required for the exposure of some complex neuro-cognilive effects, the field of single-eleclrode investigation is only beginning to benefit from the treasures it has always been exposing, and which are stilt going undervalued.
1. Single-electrode recordings in a magnetic system The gate through which physics entered, several times [1-3], the field of theoretical brain study has always been through the analogy with magnetism. 1 would like to use the same analogy, to dwell not on the formal similarity but on the observational intuitions instead. 1 imagine a physicist presented with a magnetic system, i.e. a bulk piece of matter containing a very large number of interacting magnetic moments. I will assume, as physicists do quite often, that the individual moments can point in only one of two directions, up or down. There is a whole variety of uniaxial ferromagnets for which this is a a good description. O u r physicist is told that an external magnetic field of complicated and unknown spatial distribution may be applied to the sample. He is then provided with 'single-electrode' experiments, namely he is able to follow in time the orientation of individual magnetic moments in the sample. What will he see? What will he report? The sites he will be recording from can be classified into those which show significant mean preferred orientation up (or down) for the local magnetic moment, as a function of time, and those whose orientation fluctuates very rapidly, to produce t On leave from Racah Institute of Physics, Hebrew University, Jerusalem, Israel.
710
386
no significant mean orientation. He may find that a vast majority of the sites he observes have a strong preference for pointing down, which persists indefinitely in time. If other samples, placed in the same environment, show the same tendency, while there is never a corresponding tendency for a large majority of moments to persist in pointing up, the conclusion of the physicist will be that there is an external field aligning the moments down. (One may find it useful to associate such a field with a neural threshold, which makes a quiescent (down) neural state more popular than an active state.) Fluctuations are not excluded from this description. When our physicist measures the orientation of the moments, with his single electrodes, he will typically find that the moments which prefer the down directions, flip every so often up, only to remain in that state for relatively short periods. Among the magnetic sites recorded some may be found to have a significantly higher predisposition for the up orientation. The situation will be then tentatively described as: upon the background of a uniform down orienting external field, there is an up orienting field on these particular sites. If the distribution of external fields is then slightly modified, so will the distribution of the up tending sites, in the sea of down pointing moments. The physicist will then find it natural to test those sites which showed a preferred up orientation, upon removal of the external up aligning magnetic field. A further classification can then be performed: some sites which had been previously oriented up may remain so oriented, while others may revert to the majority, i.e. point down, in the absence of the up orienting external field. The latter situation will be considered, by the physicist, of great interest, for several reasons: first because the persistence of the selected up activity is clearly due to the structure of the matrix by which the magnetic moments interact. Secondly, it will be typically the case that small variations in the distribution of the up external field, i.e. in the choice of which moments it affects, will not affect the set of individual moments preserving the up orientation, following the removal of the external alignment. Moreover, significantly different external field distributions, he will find, may lead to different distribution of up tending sites, following the removal of the external field. This leads to the terminology of aUractor for each set of up oriented sites which persist after the removal of the field. The third reason for ihe physicist's interest is lhat physicists are fascinated by phase transitions, and what is observed is one. In other words, it is very unusual that a subset of the magnetic moments can remember, for essentially an indefinitely long time, the external agent (the external field, alias: stimulus) that provoked it, and then disappeared. Yet, it is quite common. This apparent contradiction requires a word of explanation. Consider first an important example, if there were no mutual influences between the individual magnetic moments, the special spatial distribution of any initial external field would be very quickly effaced, after the field is removed. The system would have been ergodic. The system would return to the state in which every single electrode would register the down orientation. Furthermore, each electrode, whether recording a down or up tending site, would typically indicate that noise is affecting the dynamics of the system, i.e. the orientation of each moment fluctuates about its mean. The physicist would expect that such noise render the system's behaviour uniform, just by the second law of thermodynamics. The probability for states of uniform distributions is much higher than for special types of distributions. Even considerations of energy are not of much help, because our physicist has found that significantly different stimuli (external fields) provoke
711
3S7
different groups o f persistent up sites, and so the noise s h o u l d be able to mix at least those, r a t h e r t h a n stick to o n e p a r t i c u l a r distribution, w h i c h r e m e m b e r s the original sin. T h i s non-ergodicity, in the presence of noise, c a n only manifest itself fully in a system with an infinite n u m b e r o f degrees o f freedom. I n that sense the p h e n o m e n o n s h o u l d be c o n s i d e r e d u n u s u a l . O n the other h a n d , o n e h a s l e a r n e d , over a century of statistical m e c h a n i c s , that a system with a few thousand e l e m e n t s w i l l be rather s i m i l a r to a n infinite system, in the sense that non-ergodicity will manifest itself, albeit not indefinitely, b u t effectively so. T h i s is the way in w h i c h physics explains to itself the presence of structure in the universe a n d a r o u n d us. T h e s y m m e t r y of the laws o f physics w o u l d have l e d us to a s s u m e that a l l s h o u l d be u n i f o r m .
Y e t , the n o n -
uniformity w e observe a r o u n d us, is the effect of non-ergodicity i n a very large system. I n this sense the p h e n o m e n o n
is quite c o m m o n .
T h e t h e r m o d y n a m i c n a t u r e of this
non-ergodicity m a k e s it also very robust a n d reproducible. R e c o r d i n g from the s a m e site, u p o n p r e s e n t a t i o n a n d r e m o v a l o f s i m i l a r external field distribution, will p r o d u c e the s a m e qualitative after-field b e h a v i o u r at t h e site. W e s h o u l d a d d that o u r physicist with the single electrode m a y e n c o u n t e r yet o n e other type of site:
it is o n e w h i c h is not aligned w h e n t h e external aligning
is o n , but b e c o m e s aligned up, w h e n it is r e m o v e d .
field
S u c h a n o b s e r v a t i o n will be
i n t e r p r e t e d by pointing to the possibility that w h e n the field w a s o n , it might have aligned sites w h i c h w o u l d n o t be aligned in its absence. have misaligned the misbehaving site.
T h e s e sites, w h e n o n might
W h e n the field is t u r n e d off, the extra sites
b e c o m e aligned down a n d the misaligned sites c a n align themselves. F i n a l l y , nothing in t h e above discussion, nor in the single-electrode e x p e r i m e n t s o f the physicist, is to indicate that groups o f sites aligned up in two different expressions of t h e non-ergodicity, a r e exclusive.
1.1. Summary of magnetic observations A m o n g sites w h i c h show e n h a n c e d tendency for up o r i e n t a t i o n , with a single electrode w e c a n distinguish four different types: 1.
O r i e n t e d while external field on a n d disoriented w h e n r e m o v e d .
2.
O r i e n t e d w h e n external field on and preserve their up o r i e n t a t i o n w h e n external field
removed.
3.
N o t o r i e n t e d while field present, but oriented after r e m o v a l .
4.
O r i e n t e d w h e n p r o v o k e d with significantly different external field distributions.
O f these four types only the last three will be significant a n d r e p r o d u c i b l e . there a r e , of course, also the sites unaffected by the external
2. Single-electrode
Then
field.
neurobiology
T h e m e t a p h o r employed in the above parable s h o u l d be quite transparent i n its terms and observations.
It m a y not be so in its interpretations.
C l e a r l y , o u r sites c a n be
c o n s i d e r e d n e u r o n s ; the up a n d down orientations c a n be identified with the e m i s s i o n o f a spike or a spike's a b s e n c e in a time slice of o r d e r 1 ms; the u n i f o r m field in the down a l i g n m e n t , is t h e threshold leading to a p r e p o n d e r a n c e of refractoriness in time; t h e external field, orienting up, is nothing b u t a n afferent net excitatory c u r r e n t ,
712
388
r e p r e s e n t i n g the stimulus; a n d the interactions between the magnetic m o m e n t s is the s y n a p t i c efficacy c o n n e c t i n g pairs of n e u r o n s . The
fluctuating
expressing
purely
down
p r e v a l e n t state is the very low activity state o f
spontaneous
p r e d o m i n a n t l y refractory.
activity.
In
this
condition,
every
neurons,
neuron
is
I m p o s i n g the stimulus w i l l typically e n h a n c e the rate o f
s p i k e e m i s s i o n of s o m e selective set o f n e u r o n s , in those c o r t i c a l regions w h i c h d e a l with the p a r t i c u l a r type of s t i m u l u s . A l s o h e r e , as in the magnetic a n a l o g u e , i n the a b s e n c e of s y n a p t i c couplings b e t w e e n the neurons, the r e m o v a l of the stimulus w i l l lead to the d i s a p p e a r a n c e o f the e n h a n c e d activity w i t h i n a few tens of milliseconds.
Figure L Four types of neuronal behaviour observed wilh a single electrode (from [ 6 ] , figure 3). (a) Aclive in presence of stimulus and persists in absence, case 1 (b) Neuron active in delay period of a different stimulus, case 4. (c) Aclive in presence of stimulus, inactive in absence, case 1. (d) Inactive in presence of stimulus and aclive in delay period, case 3.
It is quite s t r i k i n g that the w h o l e zoology of single-cell r e c o r d i n g r e p e r t o i r e listed for the m a g n e t i c exemplified
system
by instances
manifests
itself in the
primate's cortex.
in the e x p e r i m e n t s o f M i y a s h i t a et al
w e p r e s e n t a collage of four
figures
from these papers.
H e r e it will
[4-6).
In
W e note also the
figure
be 1
notable
r e p r o d u c i b i l i t y o f the selective activity of single neurons in the delay period, i.e. i n the a b s e n c e o f the s t i m u l u s .
See e.g.
figure
2 of [4], figure 1 of (5] a n d the raster
a b o v e e a c h h i s t o g r a m in figure 2 of [5]. M i y a s h i t a et al a r e not the first to note that there is this selective associative
cortex.
W e have
found
instances
N i k i [8], as far b a c k as the early seventies. the first to z o o m in on the p h e n o m e n o n , in m o n k e y s .
of s u c h
findings,
activity in
by F u s t e r [7] a n d
Y e t to my k n o w l e d g e M i y a s h i t a et al a r e as a specific clue to 'cognitive' b e h a v i o u r
H e r e 'cognitive' is used with a lower case ' C , to stand for learned tasks
w h i c h i n c l u d e the c o m p a r i s o n of one brain event to a p r e c e d i n g o n e a n d r e a c t i n g a c c o r d i n g to the results of the c o m p a r i s o n .
713
389
3. T o u c h s t o n e for m e n t a l activity
The
single-electrode
experiments
discussed
above
exposed
all four categories
of
individual c o m p o n e n t b e h a v i o u r expected in a system w h i c h has a collective o p t i o n for b r e a k i n g ergodicity—so essential for any c o m p u t a t i o n . T h i s p a r t i c u l a r aspect was not part of the e x p e r i m e n t a l p a r a d i g m motivating these studies.
I n fact, the e x c e p t i o n a l
a p p e a r a n c e of selective persistence of elevated s p i k e rates has n e v e r b e e n worthy of c o m m e n t by neurobiology.
deemed
I h a v e failed to discover the implicit m o d e l
w h i c h a l l o w e d s u c h an easy assimilation of this p h e n o m e n o n . that of specialists i n the nervous systems of invertebrates.
T h e only r e c o u r s e is
T h e y w o u l d a r g u e that
n e u r o n s that receive a stimulus afferent may change their intrinsic properties so as to individually sustain the elevated activity. T h e fact that s o m e o f the n e u r o n s w h i c h are directly affected
by the stimulus r e m a i n inactive d u r i n g the delay p e r i o d [4], is
rebutted by insisting that different n e u r o n s react differently to a n afferent c u r r e n t . N e u r o n s in invertebrates s e e m
to actually be as individualistic as that [9], a n d
the central pattern generators in the crab's s t o m a c h may indeed be c o m p o s e d d o z e n c u s t o m - m a d e neurons.
of a
T h e functioning o f s u c h a s p e c i a l tuner m a y e v e n be
a c c o u n t e d for theoretically. B u t if a s i m i l a r variability manifests itself i n a n essential way in cortex, w h e r e functional assemblies are c o m p o s e d
o f very large n u m b e r s of
neurons, theoretical progress w o u l d be rather unlikely.
T h i s should convince any
neurobiologist
that in the a b s e n c e
of firm evidence
to the contrary, the
attitude
that in cortex n e u r o n s a r e s i m p l e r , in that they are functionally very s i m i l a r , s h o u l d be upheld.
It may be useful to recall that the possibility of theoretically c a p t u r i n g
aspects of cortex, as a substrate of m i n d , underlies the motivation for r e s e a r c h in the n e u r o s c i e n c e s as a w h o l e , vertebrates and invertebrates alike. B e that as it may, I w o u l d like to argue that the persistent activity in the a b s e n c e of the stimulus, is a touchstone of mental activity. Not that I fee) qualified to define what m e n t a l activity is. B u t , a touchstone limiting either structure o r function.
is a necessary c o n d i t i o n , without thereby
T h i s type of persistent activity is the tool by
w h i c h cortex c a n activate one of the diversity of m e m o r i z e d reactions it h a d stored in a n y particular region, a n d keep it selectively active to c o m p a r e with future or past e v o k e d reactions in other regions.
It is this process of retrieval,
recognition,
re-evaluation a n d j u d g e m e n t w h i c h a p p e a r s to be of the essence, as vague as this may s o u n d . A n d this can be m a d e possible by the existence of these type of reverberations. It should be pointed out that the selective persistence in these r e v e r b e r a t i o n s is over a n d beyond the discussion o f short- a n d long-term m e m o r y . T h e latter is about the d u r a t i o n a n d m a i n t e n a n c e of learned or innate repertoires in s y n a p t i c connections. T h e former c o n c e r n s the activation of one o f the items i n s u c h a r e p e r t o i r e into its functional expression, w h i c h is the delay activity o f neurons. In
contrast,
suppose
that
an experiment
reveals
a
neuron
w h i c h reacts,
by
e n h a n c i n g its rate of spike emission, u p o n the presentation o f as c o m p l e x a stimulus as a n individual face. T h i s special n e u r o n will fall into o n e of the inclusive categories listed in section 1.1. I f the activity w e r e to decay u p o n the r e m o v a l of the face, o n e w o u l d have to c o n c l u d e that this particular r e a c t i o n could not e v e n help the m o n k e y to c o m p a r e the face with a n o t h e r face, or even with itself.
It c o u l d serve for s o m e
l e a r n e d direct reflexes, of course, but for not m u c h more. W e could not e v e n be s u r e that it is in the afferent path o f the original s t i m u l u s , since it may be activated by other n e u r o n s w h i c h directly receive the stimulus. O n the other h a n d , it may be that the e n h a n c e d activity of the face
714
recognition
390 n e u r o n persists after the r e m o v a l of the stimulus for a significantly long time.
Then
it c a n b e c o n c l u d e d that there is a fairly large n u m b e r o f a d d i t i o n a l n e u r o n s , all o f e n h a n c e d activity, w h i c h collectively m a i n t a i n the selective activity o f e a c h other. W e a r e t h e n b a c k to the r e v e r b e r a t o r ( a t t r a c t o r ) picture d e s c r i b e d above. T h e co-activists c a n be e x p o s e d o n e
by o n e
in a s y s t e m a t i c single-electrode
s c a n n i n g o f the local
region. T h e i r n u m b e r m u s t be quite large b e c a u s e a fairly large n u m b e r o f i n c o m i n g excitatory s p i k e s a r e r e q u i r e d , w i t h i n the integration time of the post-synaptic n e u r o n , to p r o v o k e a s p i k e i n the latter. T h e r e p r o d u c i b i l i t y of the attractor b e h a v i o u r ensures that o n different p r e s e n t a t i o n of the s a m e face, the s a m e n e u r o n s w i l l be a r o u s e d .
4. E m p i r i c a l The
consequences
physicist's
magnetic
experimentation. w i t h single not.
fable
has
a
profound
moral
for
n e u r o - p h y s i o logical
It implies that s o m e c e n t r a l brain facts c a n a n d s h o u l d be s t u d i e d
electrodes.
Not
R a t h e r , single e l e c t r o d e s
that s i m u l t a n e o u s
m u l t i p l e electrode
recordings
are
can reveal the existence as w e l l as the s t r u c t u r e o f
r e v e r b e r a t i o n s in different m o d u l e s of the cortex. T h e i r r e p r o d u c i b i l i t y ensures that e a c h n e u r o n w h o s e activity is e n h a n c e d selectively i n the delay p e r i o d is part o f the i n t e r n a l r e p r e s e n t a t i o n of the c o m p u t a t i o n performed in this region o n the particular stimulus.
A t this level, m u l t i - e l e c t r o d e data can s p e e d
the collection of the data
o n the variety o f n e u r o n s participating in any o n e o f the attractors. T h i s is a n o n negligible a d v a n t a g e , given that typically there w o u l d be s o m e 1 0 0 0 0 0 n e u r o n s in any given m o d u l e . T h e e s s e n t i a l features u n d e r l y i n g the above observation are: 1.
D e l a y activity is fully r e p r o d u c i b l e o n the single-electrode
2.
D e l a y activity can persist onfy d u e to specific synaptic arrangements. Once
electrode
the spatial structure o f the recordings
are
essential
for
level.
attractors ( r e v e r b e r a t i o n s ) is k n o w n , the
exposition
of
special
temporal
multispike
correlations.
5. E p i l o g u e T h i s p o l e m i c a l note must end in a paradox:
l b a condensed
matter physicist the
a b o v e c o n s i d e r a t i o n s , c o n c e r n i n g magnetic systems, w o u l d appear a banality. a physicist rarely sees 'single-electrode' p r o p e r t i e s , o r relatively c o m p l e x
measurements.
Yet,
H e typically m e a s u r e s b u l k
correlations, s u c h as given by neutron scattering.
B u t the r e p r o d u c i b l e non-ergodicity d e s c r i b e d h e r e is never q u e s t i o n e d , b e c a u s e it is c o n s i d e r e d part o f s u c h a basic p h e n o m e n o n expressing the structure a n d d y n a m i c s of his system. T h e neuro-biologist, in contrast, w h o regularly m e a s u r e s non-ergodicity,
single-electrode
takes it for g r a n t e d , as if it does not tell h i m anything about
the
s t r u c t u r e a n d the d y n a m i c s of the system he is so interested in.
References [1]
Cragg B G and Temperley H N V 1954 The organization of neurons: a cooperaiive analogy
Eleciroenceph. Clin Neurophysiol. 6 85
715
391 [2] [3J [4] [5] [6]
Little W A 1974 The esistence of persistent stales in ihe brain Math. Biosci 19 101 Hopfieid J J 1982 Neural networks and physical systems wilh emergent selective computational abilities Proc Natl Acad. Sci USA 79 2554 Miyashila Y and Chang H S 1988 Neuronal correlate of pictorial short-ierm memory in [he primate temporal conex Nature 331 68 Miyashita Y 1988 Neuronal correlate of visual associative long-term memory in the primate temporal cone* Nature 335 817 Sakai K and Miyashita Y 1991 Neural organization for the long-term memory of paired associates
Nature 3S4 152 [7] [8] [9]
Fuster J M 1973 Behavioral electro physio logy of Ihe prefrontal cortex / NeuropystoL 36 61 Niki H 1974 Prefrontal unit activity during delay alternation in the Monkey Brain Res. 68 185 Goldwash J and Marder E 1992 Ionic currents of the lateral pyloric neuron of the stomatogastric ganglion of the crab J. NeurophysioL 67 318
716
5b Roles of Retroactivation
It is a pervasive neuroanatomical fact that when brain area A sends fibers to area B , area B sends return fibers to area A . This massive phenomenon of reciprocal connections, or retrop reject ions, is not well understood. Its interpretation is thus a domain open for debates and speculations. T h e four authors of this section bring many facts of anatomy, physiology and psychology, to support proposals which are sometimes complementary between each other, sometimes at odds. From these mutual criticisms, the reader should acquire a sense that many of the issues raised in this section are indeed at the frontier of knowledge — and that sense should be used as an incentive to develop one's own judgement. T h e emphasis put on reciprocal connections between the neocortex and the thalamus (in articles ( 5 8 ) and (61)) or the hippocampus (in article ( 5 9 ) ) introduces aspects absent in section 4a, which attempted to define information pathways within the visual cortex. Moreover, in the latter section, sensory processing was essentially described as a feedforward bottom-up process, whereas in this section much interest is given to intracortical top-down processes (in articles (60) and (62)). These considerations lead to the concept of "vertical assemblies", as defined by Crick, in contrast to "horizontal assemblies", which seem to be the more commonly accepted view about cell assemblies (as exposed in many previous articles, such as articles (15) and (55)).
So much is discussed in the five articles of this section about the binding problem, the attentional searchlight, putative fast-modifiable synapses, working memory, learning and memory storage, retroactivation from convergence zones, active blackboards, flexible templates and residuals, mental images, temporal timing and buffering, that it defies a summary. T h e reader should not fail to note many references to articles or authors of other sections of this volume, that will provide him with relevant contextual reading.
717
Repnnied from/W. Mill. Acud So Vol. ia,pp 45erM5»0. 19&4
USA.
r
Function of the thalamic reticular complex: The searchlight hypothesis (conJuQcdoiu/rapId bunii/MaJiburg lynaptes/verUral cell wmblkt/synflrc ebjiru] FRANCIS C K J C K
The
Silk
bWltof. 10010 Nonh Ton-ey Pinei
Road. U loHi. CA
9irfl7
Confrriwferf 6j fttttfrfj Ore*. Ap«i 6. / W ABSTRACT 11 Is suggested thai In (he brain Ihe fnttnuil attention at searchlight, proposed by Treisman and others. Is controlled by Ihe reticular complex of the thalamus (Including the closely- related perigenlculore nucleus) and lhat ihe expression of the searchlight Is ihe production of rapid bursts of firing In a subset of thalamic neurons. It Is also suggested that the conjunctions produced by Ihe attentional searchlight are mediated by rapidly modifiable synapses—here called Malsburg synapses—and especially by rapid bursts acting on them- The activation of Marburg synapses Is envisaged as producing transient cell assemblies, including "vertical" ones lhat temporarily unite neurons at different levels En the neural hierarchy.
unity. How then does The brain put Together all of these different activities to produce a unified picture so that, for example, for any object the right color is associated with [he right shape? The Searchlight The pioneer work of Treisman and her colleagues (5-8). supported more recently by the elegant experiments of Julesz [9—11), have revealed a remarkable fact, [f only a very short space of time is available, especially in the presence of "dis[ractors " the brain is unable to make these conjunctions reliably. For example, a human subject can rapidly spoi an "S" mixed in with a randomly arranged set of green Jfs and brown I s — i t "pops out" at him. His performance is also rapid for a blue letter mixed in with the same set. However, if he is asked to detect a green T (which requires that he recognize the conjunction of a chosen color with a chosen shape), he usually takes much more time. Moreover, the lime needed increases linearly with the number of distract ors (ihe green .u and brown 7s) as if ihe mind were searching the letters in serin, as if the brain had an internal attentional searchlight lhal moved around from one visual object [0 the next, with steps as fast as 70 msec in favorable cases. In this metaphor the searchlight is nol supposed 10 light up pan of a completely dark landscape bul. like a searchlight at dusk, ii intensifies part of a scene that is already visible lo some extent. h
This paper presents a sei of speculative hypotheses concerning ihe functions of the thalamus and. in particular, the nucleusreticularisof the thalamus and the related perigeniculate nucleus. For ease of exposition 1 have drawn my examples mainly from the visual system of primal es. but I expect the ideas to apply to all mammals and also to other systems, such as The language system in man. Visual System It is now well established that in the early visual system of primates there are at least 10 distinct visual areas in the neocortex. fFor a recent summary, see Van Essen and Maunsell 1 '• j If we Include all areas whose main concern is with vision, there may be perhaps twice thai number To a good approximation, ihe early visual areas can be arranged in a branching hierarchy. Each of these areas has a crude "map" of (part of} the visual world. The first visual area (area 17. also Called the striate cortex) on one side of the head maps one-half of the visual world in rather fine detail. Its cells can respond to relatively simple visual "features," such as orientation, spatial frequency, disparity (between the two eyes), etc- This particular area is a large one so thai The connections between different parts of it are relatively local. Each part therefore responds mainly to Ihe properties of a small local part of [he visual field 1
As one proceeds lo areas higher in the hierarchy. The "mapping" becomes more diffuse. Al the same lime the neurons appear to respond to more complex Feaiures in the visual field. Different cortical areas specialize, to some extent, in different features, one responding mainly lo motion* another more to color, etc. In the higher areas a neuron hardly knows where in ihe visual field The stimulus (such as a face) is arising, while [he feature il responds to may be so complex thai individual neurons are often difficutl to characterize effectively 13. 4). Thus. ihe different areas analyze ihe visual field in different ways. This is not, however, how we appear to see the world. Our inner visual picture of the external world has a Thr publication costs of 1(115. antde were defrayed In part by page charge payment- This article H u t iherefore be hereby marked "advertisement' in accordance wnh 19 U.S.C. 4)734 solely to indicate in • fact.
718
[f ihere is indeed a searchlight mechanism in the brain, how does it work and where is it located? To approach this problem we musl study the general layout of the brain and. in particular* rhat of the neocortex and the thalamus. The essenlial facls we need at this stage are as follows. Thalamus The thalamus is often divided into two pans: the dorsal thalamus* which is Ihe main bulk of it, and the ventral thalamus. |For a general account of the thalamus, see the review by Jones (12). I For the moment when 1 speak of the thalamus I shall mean the dorsal thalamus. Almost all input to the conex. with the exception of the olfactory input, passes through the thalamus. For this reason it is sometimes called Ihe gateway lo ihe cortex. There are some exceptions—the diffuse projections from the brainstem, the projections from the claustrum, and also some projections from Ihe amygdala and basal forebrain—that need nol concern us here. This generalization is not true for project .own from ihe cortex, which do nol need to pass through the lhalamus. Nevertheless, for each projection from a region of the lhalamus ihere is a corresponding reverse projection from lhal part of ihe cortex to Ihe corresponding region of the thalamus. In some cases at least this reverse projection has more axons than the forward projection.
45B7 Most of ihe neurons m ihe lhalamus are relay cells—lhat is. ihey receive an input from outside Ihe lhalamus (for example, ihe lateral geniculate nucleus of the thalamus gets a major input from the retina) and project directly to the cortex. Their axons form type I synapses and therefore are probably excitatory. There is a minority of small neurons in the thalamus—their exact number is somewhat controversial—that appear to form type II synapses and are therefore probably inhibitory.
with most of ihe nuclei of the thalamus, the principal cells of which have few. if any. collaterals either within each nucleus or between nuclei. The nuclei of ihe lhalamus [with the exception of ihe intralaminar nuclei! keep themselves to themselves. The neurons of the reticular complex, on the other hand* appear to communicate extensively with each other. Moreover. il is characteristic of them that they fire in long bursts at a very rapid rate (2i).
While on the face of it The lhalamus appears lo be a mere relay, this seems highly unlikely. Its size and ns strategic position make it very probable thai il has some more important function. Reticular Complex Much of the rest of [he thalamus is often referred lo as the ventral thalamus. This includes the reticular complex fpan of which is often called the pengemculaie nucleus), the ventral lateral geniculate nucleus, and the zona incerta. In what follows 1 shall, for ease of exposition, use the term reticular complex to include the perigeniculate nucleus. Again, "thalamus" means ihe dorsal thalamus. Ahhough much of the following informaiion comes from the cat or the rat. there is no reason TO think that it does not also apply to the primate thalamus. The reticular complex is a ihin sheet of neurons, in most places only a few cells thick, which partly surrounds the (dorsal) thalamus (13-32) (set Fig. 1). Ali axons from the thalamus lo ihe cerebral cortex pass Through il. as do all of the reverse projections from the cortex to ihe lhalamus. The intralaminar nuclei of ihe lhalamus, which project very strongly lo the striatum, also send their axons through it. as may some of the axons from rhe globus pallidus thai project back to the lhalamus. El is believed that many of the axons lhat pass in both directions through the reticular complex give off collaterals that make excitatory synaptic contacts in it (IS, 18. 21. 29. 30). If the lhalamus is the gateway lo ibe corlex. the reticular complex might be described as the guardian of the gateway. Its exact function is unknown. Sot onl> is ns position remarkable, bul ils structure is also unusual. It consists largely (if not entirety) of neurons whose dendrites often spread rather extensively in the plane of ihe nucleus I29>. The size of these neurons is somewhat different in differeni parrs of ihe complex (31). Their axons, which project to the thalamus, give off rather extensive collaterals that ramify, sometimes for long distances, within the sheet of the reticular complex U°. » h This is in marked contrasi
RETICULAR COMPLEX
J
An even more remarkable property of reticular neurons concerns their output. Whereas all of ihe ouiput neurons of ihe thalamus make type I synapses and appear to be excitatory* many (if not all) of Ihe neurons in the reticular complex appear to be GABAergic (GABA = y-arninobutyric acid) and Thus almost certainly inhibitory (26-28) The excitation in the complex must come almost exclusively from the activity of the various axons passing through it. Both the input and the output of the complex are arranged topographically (16* 29* 30, 32). It seems likely lhat if a particular group of axons going from ihe lhalamus lo ihe cortex passes through a small region of the reticular complex, the reverse projection probably passes through or near that same region. There may well be a rough map of ihe whole cortex on the reticular complex, though how precise ihis map may be is nol known. Il should be remembered, however, that the spread of Ihe receiving dendrites of the reticular nucleus is quire large. The projection of the reticular complex lo [he thalamus is also noT random. Though any individual axon may spread fairly widely. Ihere is a very crude topography in ihe arrangemenl. The projeclion from any one part of the reticular complex probably projecis to thai pan of rhe thalamus from which it receives input as well as other neighboring parts. The exacl nalure of these various mappings would repay further study. The neurons of ihe reticular complex project to the (dorsal) lhalamus. The evidence suggesis lhal they mainly contact rhe principal (relay} cells of the thalamus 122). What effect does the reticular inpul have on ihe behavior of the cells in the dorsal lhalamus? Obviously this is a crucial question. Let us consider two oversimplified but contrasting hypotheses- The first is lhat the main effect of the reticular complex is inhibitory. This would lead to the following general picture. The traffic passing through the reticular complex will produce excitation. Ler us assume thar one patch of the complex is more excited than the rest because of special acliviry in ihe thalarno-conical pathways The effect of this will be 2-fold. That region will tend io suppress somewhat ihe other pans of the reticular complex, because of the many inhibitory collaterals, h will also suppress ihe corresponding thalamic region. These rwo effects will damp down ihe thalamus in its most active region and have ihe opposite effecT (since the inhibition from the reticular complex will be reduced there) on the remaining parts. The total effect will be lo even oui rhe activity of ihe thalamus. This is noi a very exciting conclusion. The function of rhe reticular complex would be lo act as an overall Thermostat of thalamic activity, making ihe warm pans cooler and the cool parts warmer. The second hypolhesis is just the opposite. Let us assume that the effect of ihe reticular complex on ihe dorsal thalamus is mainly excitation in some form or other. Then we see lhal. once again* an aclive patch in rhe complex will tend ro suppress many other parts of The complex. This time, however, the effect will be to heat up the warmer parts of the Thalamus and cool down the cooler parts. We shall have positive feedback rather lhan negative feedback* so thai "anention" will be focused on ihe most active thalamo-cortical re gions. How can GABAergic neurons produce some son of excitatory effect on the relay cells of the lhalamus? One possibili-
FIG. I - The main connections of Ihe reticular complex, highly diaaramaiic and not al all lo scale. Solid lines represent eiciiaiory axons. Dashed lines show GABAergic inhibitory! axon*. Arrows represent synapses.
719
088 iy is lhat ihey might synapse only onio ihe local inhibilory neurons in the lhalamus. By inhibiting these inhibilory cells they would thereby increase The effect of incoming excitation on ihe relay cells. This is certainly possible bul the anatomic evidence 422) suggests thai in the main the neurons of The reticular complex project directly to ihe thalamic relay neurons. One would expect rhat ihis would inhibit these neurons. We must therefore ask if thalamic neurons show any unusual types of behavior. Properties of Thalamic Neurons The recent work of Llinas and Jahnsen {33-35) on thalamic slices from the guinea pig confirms lhal Ihis is indeed the case- Their papers should be consulted for the detailed results, which are complicated* bur* broadly* rhey show lhal all thalamic neurons display iwo relatively distinct modes of behavior. When the cell is near its normal resting potential (say, -60 mV) il responds to an injected curreni by firing (producing axonal spikes) at a fairly modest rate, usually between 25 and 100 spikes per second The rate increases with the value of the current injected. If. on the other hand, ihe negative potential of the membrane is increased somewhat (that is. if the cell is hyperpoLarized) lo* say. -70 mV, then a neuron responds lo an injected current, after a short delay, with a spike or a short fast burst of spikes, firing briefly at rates nearer 300 spikes per second. Moreover, the after-effect of this burst is lhat. even though the injected curreni is maintained constant* the cell will not produce a further burst for a lime of the order of 80¬ 150 msec. Jahnsen and Llinas (34* 35)* by means of many eleganl controls, have shown lhal this behavior depends on a number of special ion channels* including a Ca^-dependent K' conductance. Thus* it is at least possible that the effect of the GABAergic neurons of the reticular complex on ihe thalamic relay cells is to produce a brief burst of firing in response to incoming excitations, followed by a more prolonged inhibition. Whether this is actually the effect they produce in natural circumstances remains to be seen, since il is not easy to deduce this with certainty from the results of Jahnsen and Llinas on slices.
So far I have lumped the perigeniculate nucleus U7-24| in wilh the reticular nucleus proper which adjoins ii. It seems probable that ihe lateral geniculate nucleus (which in primates projects mainly to the first visual area of the cortex) sends collaterals of its output to ihe perigeniculate nucleus, while ihe resiof the dorsal thalamus sends collaterals to the reticular nucleus proper (20). This suggests that [here may be at least iwo searchlights: one for the firsl visual area and anolher for all of the rest. Indeed* (here may be several separate searchlights. Their number will depend in part on the range and strength of the inhibitory collaterals within the reticular complex. Clearly, much more needs to be known about both the neuroanatomy and the neurophysiology of •he various pans of ihe reticular complex. Makburg Synapses We must now ask: what could rhe searchlight usefully do? Treisman's results (5-8) suggest Lhat what we wanl il to do is to form temporary "conjunctions' of neurons. One possibility is that the conjunction is expressed merely by the relevant neurons ruing simultaneously, or al least in a highly correlated manner. In artificial inielligence the problem would be solved by "creating a line" belween the units. There is no way thai the searchlight can rapidly produce new dendrites, new axons, or even new axon terminals in the brain. The only plausible way lo create a line in a short time is to strengthen an existing synapse in some way. This is the essence of the idea put forward in 1981 by von der Malsburg in a little known bul very suggestive paper." After describing the conjunction problem in general terms he proposed that a synapse could alter its synaptic weight (roughly speaking, the weight is ihe effect a presynaptic spike has on the potential at the axon hillock of the poslsynaplic cell) on a fast lime scale ("fractions of a second"). He proposed that when rhere was a strong correlation between presynaptic and postsynaptic activity, the strength of the synapse was temporarily increased—a dynamic version of Hebb's well-known rule (36)—and that wilh uncorrelated pre- and postsynaptic signals ihe strength would be temporarily decreased below its normal resting value.
Whal do we require of a searchlight? Il should be able lo sample the activity in the cortex and/or Ihe thalamus and decide ""where ihe action is." It should then be able io intensify the thalamic inpul to lhal region of the cortex, probably by making ihe active thalamic neurons in that region fire more rapidly than usual. ft must Ihen be able to turn off its beam, move to the next place demanding attention, and repeat the process. Il seems remarkable, lo say the least, thai the nature of Ihe reticular complex and ihe behavior of the thalamic neurons fit this requirement so neatly. The extensive inhibitory collaterals in the reticular complex may allow n to select a small region thai corresponds lo ihe most active pan of the ihalamo-conical maps. Its inhibitory output, by making more negative ihe membrane polenliai of the relevant thalamic neurons* could allow them to produce a very rapid, short burst and also effectively turns them off for 100 msec or so. This means that the reticular complex will no longer respond at that patch and its activity can ihus move to the next mosi active patch. We are thus Led to two plausible hypotheses:
Notice that we are not concerned here with long-term alterations in weight, as we would if we were considering learning, but very shon-term transient alterations that would occur during the act of visual perception. The idea is not. however* limired lo the visual system but is supposed to apply to all parts of the neocortex and possibly lo other parts of the brain as well. Most previous theoretical work on neural nets does not use this idea, though there are exceprions (37, 38). The usual convention is that while a net. or set of nels, is performing. the synaptic weights are kept constant. They are only allowed to alrer when learning is being srudied. Thus, von der Malsburg s idea represent a rather radical alteration to the usual assumptions. L propose lhat such (hy pot helical) synapses be called Malshurg synapses. Notice lhat in ihe cortex Ihe number of synapses exceeds the number of neurons by at least three orders of magnitude. Let us then accept for ihe moment that Malsburg synapses are al leasi plausible. We are still a long way from knowing •he exact rules for their behavior—How much can their strength be increased? What exactly determines ihis increase (or decrease) of strength? How rapidly can ihis happen? How does this temporary alteration decay?—to say nothing of the molecular mechanisms underlying such changes.
(fl The searchlight is com rolled by the reticular complex of thalamus. tit) The expression of the searchlight is the production of rapid firing in a subset of active thalamic neurons.
'von der Marburg, C. (19B1) Internal Report Bl-2 (Department of Neurobiology. Max-PlancMnslituie for Biophysical Chemistry. Goettingen* F.R.G I.
The Searchlight Hypothesis 1
the
720
4589 In spile of all of ihese uncertainties il seems nol unreasonable to assume lhal the effect of ihe searchlight is to activate Malsburg synapses. We are thus led to a third hypothesis.
tJ
a much more limited number of units (the 10 or so neurons in the central nervous system), each new combination being brought into action as ihe circumstances demand and then largely discarded. Without this device the brain would either require vastly more neurons to do the job or its ability to perceive. Think, and acl would be very severely restricted. This is the ihrust of von der Malsburg s arguments. A somewhat similar set of ideas about simultaneous firing has been put forward by Abeles. His monograph (401 should he consulted for details. He proposes the concept of "synfire chains™—sets of cells, each set firing synchronously, connected in chains, which fire sequentially. He gives a plausible numerical argument, based mainly on anatomical connectivity, which suggests lhat to establish a functioning synfire chain only a few (perhaps five or so) synapses would be available al any one neuron. Since ibis is such a small number he deduces that the individual synapses must be strengthened (if these five synapses by themselves are to fire ihe cell) perhaps by a factor of 5 or so. However, he gives no indication as to how this strengthening might be done.
(Hi) The conjunctions produced by (he searchlight are mediated by Molsburg synapses, especially by rapid bursts acting on them.
We still have to explain exactly how activated Malsburg synapses form associations of neurons. This is discussed by von der Malsburg in his paper in some detail but most readers may find his discussion hard to follow. His argument depends on the assumption that the system needs to have more than one such association active al about the same time. He describes at some length how correlations, acting on Malsburg synapses, can link cells into groups and thus form what he calls topological networks. What characterizes one such cell assembly is that the neurons in il fire "simultaneously." an idea thai goes back to Hebb (36) von der Malsburg suggests That two kinds of signal patterns can exist in a topological network: waves running through the network or groups of cells switching synchronously between an aclive and a silent state. He next discusses how a sei of cells rather than a single cell might form what he calls a "network element" Finally, after an elaborate development of ihis theme he broaches the "bandwidth problem." In simple terms, how can we avoid ihese various groups of cells inierfering with each other? Cell Assemblies The cell assembly idea is a powerful one. Since a neuron can usually be made lo fire by several different combinaiions of its inputs, the significance of its firing is necessarily ambiguous, ft is thus a reasonable deduction thai this ambiguity can be removed, al least in part, by the firing pattern of an assembly of cells. This arrangement is more economical than having many distinct neurons, each with very high specificity. This type of argument goes back to Young (39) in 1807There has been much theorelical work on what we may loosely describe as associative nets. The nets are usually considered to consist of neurons of a similar type, receiving input, in most cases, from similar sources and sending their oulpul mainly lo similar places. If we regard neurons {in. say ihe visual system) as being arranged in some sort of hierarchy, then we can usefully refer to such an assembly as a horizontal assembly. von der Malsburg s ideas, however, permit another type of assembly In his theory a cell at a higher level is associated wilh one at a lower level twe are here ignoring the direction of the connection), and these, in turn, may be associaled whh ihose at a still lower level. (By "associated wnh" [ mean lhal the cells fire approximately simultaneously.) For example, a cell at a higher level that signified Ihe general idea "face" would be temporarily associaled. by Malsburg synapses, wilh cells that signified the pans of the face, and. in urn. perhaps with their pans. Such an assembly might usefully be called a vertical assembly. Il is ihese vertical assemblies lhal have to be constructed anew for each different visual scene, or for each sentence, etc. Without them il would be a difficult job lo unite the higher level concepts with their low level details in a rather short lime. This idea is reminiscenl of ihe K lines of Minsky. T
r
1
The idea of transient vertical assemblies is a very powerful one. It solves in one blow rhe combinational problem—lhat is. how the brain can respond lo an almost unlimited number of distinct sentences, passages of music, visual scenes* etc. The solution is to use temporary combinaiions of a subset of 'Minsky, M. 41979) Artificial Intelligence Memo No. 516 (Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cant ridge, MA].
1
Abeles argument stresses ihe importance to the system of the exact lime of firing of each spike,ratherthan the average rate of firing, which is often taken to be the more rclevant variable. This exact timing is also an important aspect of von der Malsburg's ideas. These arguments can also be supported by considering ihe probable values of the passive cable constants of cortical dendrites. Very rough estimates (for example, T - 8 msec. X - x/5) suggest that inputs will nol add satisfactorily unless they amve within a few milliseconds of each other (see figure 3.18 in ref. 41). Notice ihe idea that a cell assembly consists of neurons firing simultaneously (or at least in a highly correlated manner) is a very natural one. since this means lhat the impact of their joint firing on other neurons, elsewhere in the system, will be large. The content of the cell assembly—the "meaning" of all of ihe neurons so linked together—can in ihis way be impressed on the rest of Ihe system in a manner lhat would not be possible if all of the neurons tn it fired at random times, unless they were firing very rapidly indeed. Therefore, our founh hypothesis follows, fiv) Conjunctions are expressed by cell assemblies, especially assemblies of cells in different cortical regions.
It should not be assumed thai cell assemblies can only be formed by the searchlight mechanism. Some important ones may well be Laid down, or partly laid down, genetically [e.g.. faces?) or be formed by prolonged learning (e.g.. reading lellers or words?). Il is clear lhat much funher theoretical work is needed to develop these ideas and make them more precise. Lf the members of a vertical assembly fire approximately synchronously, exactly how regular and how close together in time do these firings have lo be? Are there special pathways or devices to promote more simultaneous firing? Are dendritic spikes involved? How does one avoid confusion between different cell assemblies? Do neurons in different cell assemblies briefly inhibit each other, so that accidenlal synchrony is made more difficult? Etc. The idea that the dorsal lhalamus and the reticular complex are concerned with anention is nor novel {19. 42, 43). Whai is novel (as far as I know) is the suggesrion rhat they control and express the internal altenlional searchlight proposed by Treisman (5-8). Julesz (9. 10). Posner (44. 45). and others. For this searchlight at least two features are required. The first is rhe rapid movemeni of the searchlight from place to place while the eyes remain in one posilion. as discussed above. There is. however, another aspect. The brain musl know what il is searching for (the green Tin the example given earlier) so that ii may know when its hum is successful. In other words, the brain musl know whal to attend lo. That aspect, which may involve other conical areas
721
4590 such BI ihe frontal cortex, has nol been discussed here. The basic searchlight me cll an ism may depend on several parts of the reticular complex, but these may be influenced by topdown pathways, or by other searchlights in other parts of the reticular complex, which may be partly controlled by which ideas are receiving attention. An important function of the reticular complex may be to limit the number of subjects the lhalamus can pay attention to al any one lime. Experimental Tests These will not be discussed here in detail. It suffices to say thai many of the suggestions, such as the behavior of the dorsal lhalamus and the reticular complei. are suscepuble to fairly direct tests. The exact behavior of reticular neurons and thalamic neurons is difficult to predict wilh confidence, since Ihey contain a number of very different ion channels. Experiments on slices should therefore be complemented by experiments on animals. Obviously, most of such experiments should be done on alert, behaving animals, if possible wilh natural stimuli. An animal under an anesthetic can hardly be expected to display all aspects of alrention. Various psychophysical lests are also possible. Other aspects of these ideas, such as ihe behavior of Maisburg synapses, may be more difficult to test in the immediate future. It seems more than likely rhat dendritic spines are involved, both the spines themselves and the synapses on them 146, 41). The existence and the importance of rapid bursts of firing can also be tested. Such bursts, followed by a quiet interval, have been seen in neurons in the visual cortex of acurarized. unanes then zed and artificially respired cat when they respond to an optimal visual signal (see figure 1 in Mortrll (48)]. It is unlikely that the Iwo systems—Ihe rapid-burst sysiem and the slow-firing sysiem—will be quite as distinct as implied here. In facl. as von der Malsburg has pointed out. one would expect ihem to interact. Thus, all of these ideas, plausible though Ihey may be. must be regarded at the moment as speculative until supported by much stronger experimental evidence. Jn spite of this, ihey appear as if ihey might begin to form a useful bridge between certain parts of cognitive psychology, on the one hand and ihe-world of neuroanatomy and neurophysiology on the other.
7. Treisman. A. 4 Schmidt. H. 119821 Cogmt. PsycM. 14. 107¬ 141. B. Treisman. A. 119831 in Physical and Biological Processine uf Images, eds. Braddick. O. J. A Sleigh, A. C, (Springer. New York). PP. 116—323. 9. Julesl. EL 119801 Phtloi Tram. ft. Soc. London Ser B 190. 83-94 10. Julest. B (19811 Nature'London) 290.91-91. 11. Bergen. J R. 4 Julesl. B I19SJ) iVawrr ILandon, 303. 696¬ 698 12- Jones. E. G. (19*3) in Chemical Neuroanatomy, ed. Emson. P. C. I Raven, New Ywtk pp. 2)7-293 13. Sumitomo. I Nakamura. M dt [wama. K. 119761 £cp ,\rural. g|, 110-113. 1«. Dubin. M. w. ekClclaad. 8 G (19771/. Vmftrjnr. 40, tio•17 15. Monttro. V M.. Guillery. B. w 4 WocJsey. C. tt. (1977i Brim Brs 134. 407-421.
16. Montero. V. M. and Scott. G. L. (19811 Nairuscienci 6. 25612)77. 17. Ahlsen. G.. Lindstrflm. S 4 Sybirska. E. '1976] Brum ftei IH, 106-109. IB. Ahlltn. G. 4 Lindstram, S. (19821 Brain Res 134. 477-181 19. Ahlstn. G. 4 Lindstrflm. S. (19821 Brain Brs lit. 20. AhliCn. G.. Linditrom. S 4 Lo. F.-S. 119821 Exp. Brain Bit 46. 111-126. 21. Ahlstn. G. 4 Lindstram. S I19S3) Arm Ph\siol. Stand. 111. IB1-184 27. Ohara, P T Sefton. A- I. * Lieberman. A R 11980) Bruin Bei. IT), 503-506 23. Hilt. P T„ Seflon. A J.. BauT. L. A 4 Coiiee. L. I.lini: Lip Brain /tel. 45, 217-229. 24. Ide, L. S. (19821 J Comp. Ntura 210. 317-334 » Schlai. I. 4 Wasiak. M. 119711 Ere, Neural, il. 79-97 26. Houser, C. R.. Vaughn, J. £.. Barber. R. P. 4 Roberts. E 119801 Brain /lei MO. 3*1-154. VI. Oertel. W. H Graybiel,.A. M Mugnaim. £ Elde ft P Schmechel, D E 4 Kopin. [ J 11983)/. Netiraici- i. 13:2¬ 1332. 28 Ohara. P. T„ Lieberman. A. R.. Hum. S. P S Wu. I -Y. 11983) I. 189-211. 29 Scheibel. M. E. 4 Scheibel. A. B. 11966) Brain Bes. 1.43-42. 30 Jonts. E. G. 119751 I. Comp. Neurol. 161. 285-308 31. Scheibel. M. E. & Scheibel. A. B. (1972) Exp. Neural. 34. 316-322. 32. Minderhoud. I. M. 11971) Exp. Brain lies. 12. J35-U4. 33. Hints. R. 4 Jahnsen. H. (19821 Narun iLondonl 197. 406¬ 408. 34. Jahnsen. H. 4 Llints. tt 119841/. Physiol 149. 205-226 35. Jahnsen. H. 4 Llints. R. (19841/. Physiol J49. 227-141 36. Hebb. D. 0 11949) Ortamcalion aj Behaviur iWiley New ^oM xddtd In Proof. Recent unpublished experimental work sugYorkl. gests lhal the reticular complei may produce bunts of firing in some thalamic neurons bul merely an increase of firing race in others. 17 Little. W A 4 Shaw.G. L. (197)1 Beba. Bid 14. 115-133 38 Edelman G M 4 Reeke. G. N imii Proc. Nail. Acad. S.i USA 14, 2091-2095 This tiork originated as a result of extensive discussions wilh Dr. 39 Young. T 11802) Philat Trans. 12-48. Chnslupher Longuet-Higgins. I lhank him and many Olher col40. Aheles. M. (19B2) Local Conical Circuits: Srudies uf Brum leagues "ho h*ve commenled an the idea, in particular. Drs RichFunction (Springer. New Yorkl, Vol. 6. ard Anderson. Max Cowan. Simon LeVay. Don MacLeod. Graeme 41. Jack, I. 1. B Noble. D. £ Tsien. R W. (19151 Electric CurMitchison. Tomdsu Poggio. V 5 ftamachandran. Tcrrencc Sejrent Flow m Excitable Celts rClarendon. Oxford! nowski. and Chrisroph von der Malsburg. This work has been sup42. Yingling.C. D. 4 Skinner. J E. (19771 in Arrerriwn. Vakmn ported hi ihe 1 A Kieckheler Foundation and ihe System DevelContraction and Ei tnl-Cerebral Potentials, ed. Dcimedi. opment Foundanon. J. E. tKargcr. Basel. Switierland). pp. 70-96. 43. Skinner. J. E. 4 Yingling. CD. (1977) inArrenrir>n. Vdbnuars 1. Van Essen. D C t Maunsell. J H ft 119*31 Trends NevoContraction and Ei-tnl-Cerebrat Potentials, ed. Deimedt id. * . J70-J.75. J. E. (Karget. Basel. Switrerland). pp. 30-69. 2. Hubel. D H S Wiesel. I H. (1977) J>,or ft Sue London 44 Posner. M. I 119821 An. Ps\chal. 31. 168-179. Ser. B 198, 1-59 "5 Posner. M. L. Cohen. Y. 4 Rafll. R. D. 11982) Pliilas. Trim. I. Bruce C. Desimone. ft & Gross. C O 119811 / /VtvopAvi B. Soc. London Ser B 298. 187-19* *. kW-Jca 46. Perkel. 0 H. 119831 /. Pltyiiol. tParisl 78.695-699 4 Perrett. D I . Rolls. E. T 4Caan. W. EipflrornRn 47 Koch. C. 4 Poggio. T 11993) Proc P. Sac f III til Sc * at. 1:9-342 118. 455-477 5 Treisman. A 11977) Prrttpt. Pi\chophvi. 11. 1—LI 4B. Morrell. F. (19721 , Brain and Human Behu.uir. nil Kjrce6 Treisman. A M ft Gelade.G 119801 CO/JIUI. FiveM. U. 91mar. A. G 4 Eccles. J C (Spnnger. New York) pp :5Vm 2S9. n
722
Reprinted wilh permission from Neural Models of Plasticity. pp. 240-265, 1989 © 1989 Academic Press Inc.
Functions of Neuronal Networks in the Hippocampus and Neocortex in Memory Edmund T. Rolls 1. Functions of the Primate Hippocampus in Memory It is known that damage to certain regions of the temporal lobe in humans produces anterograde amnesia evident as a major deficit following the damage in learning to recognize new stimuli (Scoville and Milner, 1957; Milner, 1972; Squire, 1986; this volume, Chapter 12). The anterograde amnesia has been attributed to damage to the hippocampus, which is within the temporal lobe, and to its associated pathways such as the fornix (Scoville and Milner, 1957; Milner, 1972; Gaffan, 1974, 1977), but this has been questioned, and instead it has been suggested that damage to both the hippocampus and the amygdala is crucial in producing anterograde amnesia, in that combined but not separate damage to the hippocampus and amygdala produced severe difficulty with visual and tactual recognition tasks in the monkey (Mishkin, 1978, 1989,1982; Murray and Mishkin, 1984, 1985). In investigations of the particular aspects of memory for which the hippocampus may be essential, it has been shown that monkeys with damage to the hippocampo-fornical system have a learning deficit on memory tasks that require them to make associations between a stimulus—for example, a picture—and a spatial motor response such as touching one part of a screen (Gaffan, 1985; Rupniak and Gaffan, 1987), and are also impaired on memory tasks that require combinations of stimulus attributes with their locations in space to be processed together, such as memory not only for which object was shown but where it was shown (Gaffan and Saunders, 1985). Further, humans with right temporal-lobe damage are also impaired in conditional spatial response and object-place memory tasks (Petrides, 1985; Smith and Milner, 1981). In order to analyze the functions being performed by the hippo-
723
241 c a m p u s i n memory, the activity of 1510 single h i p p o c a m p a l n e u r o n s w a s recorded in rhesus monkeys learriing and performing these memory tasks k n o w n to be i m p a i r e d by damage to the h i p p o c a m p u s or fornix (Rolls et al, 1989; Miyashita et al, 1989; C a h u s a c et al, 1989). I n an object-place memory task i n w h i c h the m o n k e y had to rem e m b e r not only w h i c h object had been seen i n the previous 7-15 trials but also the position i n w h i c h it had appeared o n a video monitor, n e u rons were found that responded differentially depending o n which place on the monitor screen objects were s h o w n (Rolls etal, 1989). T h e s e n e u rons comprised 9.4% of the population recorded. It is notable that these neurons responded to particular positions in space (whereas "place" cells in the rat respond w h e n the rat is in a particular place; O ' K e e f e , 1983). I n addition, 2.4% of neurons responded more to a stimulus the first time it was shown in a particular position than the second time. These neurons thus responded to a combination of information about the stimulus being s h o w n a n d about position i n space, for only by responding to a combination of this information could the n e u r o n s r e s p o n d only w h e n a stimulus w a s s h o w n for the first time i n a certain position i n space. I n tasks i n w h i c h the m o n k e y s had to acquire associations b e t w e e n visual stimuli a n d spatial responses, 14.2% of the n e u r o n s responded to particular combinations of s t i m u l i a n d r e s p o n s e s ( M i y a s h i t a et al, 1989). F o r example, i n a task in w h i c h the monkey h a d to perform one response (touching a screen three times) w h e n one visual stimulus w a s s h o w n , but had to perform a withholding response for 3 sec to obtain r e w a r d w h e n a different stimulus w a s s h o w n (Gaffan, 1985), 9.2% of the neurons responded to one of the stimuli if it w a s l i n k e d to one of the responses i n this task. T h e same n e u r o n s typically did not respond if the same stimuli or the same responses were u s e d i n different tasks, or if other stimuli were associated w i t h the same responses i n this task. T h u s these neurons responded to a combination of a particular stimulus with a particular spatial motor response (Miyashita et al, 1988). It w a s possible to study the activity of 41 h i p p o c a m p a l n e u r o n s w h i l e the m o n k e y s learned n e w associations between v i s u a l stimuli a n d spatial responses. In some cases it w a s possible to s h o w that the activity of these neurons became modified d u r i n g this learning ( C a h u s a c et al, 1986, 1989). Interestingly, 33% of the neurons that altered their responses during this learning s h o w e d a sustained differential response, but 67% of the neurons differentiated between the stimuli only at or just before the monkey learned the task, and stopped differentiating after 5-10 more trials. T h i s is consistent with the possibility d i s c u s s e d below that the neurons that show large sustained differential responses inhibit the other neurons that s h o w transient modification, so that as a result of competition not all neurons are allocated to one s t i m u l u s - s p a t i a l response association. T h e s e results s h o w that h i p p o c a m p a l n e u r o n s i n the primate have
724
242
responses related to certain types of m e m o r y . O n e type of m e m o r y involves complex conjunctions of environmental information—for example, w h e n information about position in space (perhaps reflecting information from the parietal cortex) m u s t be m e m o r i z e d i n conjunction with w h a t that object is (perhaps reflecting information from the temporal lobe visu a l areas), so that w h e r e a particular object w a s seen i n space can be r e m e m b e r e d . T h e h i p p o c a m p u s is ideally placed anatomically for detecting such conjunctions, i n that it receives highly processed information from association areas s u c h as the parietal cortex (conveying information about position in space), the inferior temporal visual cortex (conveying a v i s u a l specification of a n object), a n d the s u p e r i o r t e m p o r a l cortex (conveying a n auditory specification of a stimulus) ( V a n H o e s e n , 1982) (see F i g . 5). T h e positions of stimuli i n space may be represented by the firing of h i p p o c a m p a l n e u r o n s as described above so that conjunctions of, for example, objects a n d their position c a n be formed. It is also suggested that one neurophysiological m e c h a n i s m by w h i c h "place" cells in the rat (see O'Keefe, 1983) may be formed is by conjunction learning of sets of simultaneously occurring stimuli in different parts of space, each set of w h i c h defines a place.
II. Computational Theory of the Hippocampus A possible theoretical basis for these results, a n d in particular h o w the h i p p o c a m p u s m a y perform the conjunction or combination learning just described, is n o w considered. A s c h e m a t i c d i a g r a m of the c o n n e c t i o n s of the h i p p o c a m p u s is s h o w n i n F i g . 1. O n e feature is that there is a sequence of stages, i n each of w h i c h there is a major set of i n p u t axons that connect via a form of matrix w i t h the output n e u r o n s of that stage. T h e type of computation that could be performed by one of these stages is considered first. T h e perforant path connections w i t h the dentate granule cells may be taken as an example. A version of this represented as a simplified matrix is s h o w n i n F i g . 2. A l t h o u g h the perforant path m a k e s one set of s y n a p s e s w i t h the output n e u r o n s i n a form of matrix, the matrix is clearly very different from a n association matrix memory, i n that i n the h i p p o c a m p a l s y s t e m there is no unconditioned stimulus that forces the output n e u r o n s to fire (see Rolls, 1987). N o r is there for each output cell a climbing fiber that acts as a teacher as i n the cerebellum (see Ito, 1984; see C h a p t e r s 9 a n d 10 i n this volume). In the h i p p o c a m p a l circuit there is apparently no teacher—that is, this appears to be an example of an u n s u p e r v i s e d learning system. T h e following describes one m o d e of operation for s u c h a network. Later, properties of the h i p p o c a m p u s that suggest that it m a y operate i n this w a y are d i s c u s s e d .
725
243
Haocortar,
Figure 1 . (a) Representation of connections within the hippocampus. Inputs reach the hippocampus through the perforant path (1), which makes synapses with the dendrites of the dentate granule cells and also with the apical dendrites of the C A 3 pyramidal cells. The dentate granule cells project via the mossy fibers (2) to the C A 3 pyramidal celts. The well-developed recurrent collateral system of the C A 3 cells is indicated. The C A 3 pyramidal cells project via the Schaffer collaterals (3) to the C A l pyramidal cells, which in turn have connections (4) to the subiculum. (b) Schematic representation of the connections of the hippocampus, showing also that the cerebral cortex (neocortex) is connected to the hippocampus via the para hippocampal gyrus and entorhinal cortex, and that the hippocampus projects back to the neocortex via the subiculum, entorhinal cortex, and para hippocampal gyrus.
726
244
stimuli
K
responses Figure 2. A matrix for competitive learning in which the input stimuli are presented along the rows of the input axons (a,), which make modifiable synapses ( s j with the dendrites of the output neurons, which form the columns (d ) of the matrix. c
C o n s i d e r a matrix m e m o r y of the form s h o w n i n F i g . 2 i n w h i c h the strengths of the s y n a p s e s between horizontal axons and the vertical dendrites are initially random (postulate 1). Because of these random initial s y n a p t i c weights, different input patterns o n the horizontal axons will tend to activate different output n e u r o n s (in this case, granule cells). T h e tendency for each pattern to select or activate different n e u r o n s can then be e n h a n c e d by providing mutual inhibition between the output n e u r o n s , to prevent too m a n y n e u r o n s r e s p o n d i n g to that stimulus (postulate 2). T h i s competitive interaction can be v i e w e d as e n h a n c i n g the differences i n the firing rates of the output cells [cf. the contrast e n h a n c e m e n t described by G r o s s b e r g (1982)]. Synaptic modification then occurs according to the rules of long-term potentiation in the hippocamp u s , n a m e l y , that s y n a p s e s between active afferent axons a n d strongly activated postsynaptic n e u r o n s increase i n strength (see M c N a u g h t o n , 1983; L e v y , 1985; K e l s o et al., 1986; W i g s t r o m et al, 1986; see C h a p t e r 14 of this volume) (postulated). T h e effect of this modification is that the next time the s a m e s r i m u l u s is repeated, the neuron responds more (because of the strengthened s y n a p s e s ) , more inhibition of other neurons occurs, a n d then further modification to produce even greater selectivity is p r o d u c e d . T h e response of the s y s t e m thus climbs over repeated iterations. O n e effect of this observed i n simulations is that a few n e u r o n s then obtain s u c h strong synaptic weights that almost any stimulus that h a s any i n p u t to that n e u r o n will succeed i n activating it. T h e solution to this is to limit the total synaptic weight that each output (postsynaptic) n e u r o n can receive. I n simulations this is performed by normalizing the s u m of the synaptic weights on each neuron to a constant (e.g., 1.0) (cf. v o n d e r M a l s b u r g , 1973; R u m e l h a r t a n d Zipser, 1986). T h i s has the effect of distributing the output n e u r o n s evenly between the different input patterns received by the network.
727
245
A simulation of the operation of such a matrix is s h o w n i n F i g . 3. It is s h o w n that the network effectively selects different output neurons to r e s p o n d to different combinations of active input horizontal lines. It thus performs a type of classification i n w h i c h different complex i n p u t patterns are encoded economically onto a few output lines. It s h o u l d be noted that this classification finds natural clusters i n the input events; orthogonalizes the classes i n that overlap i n input events c a n become coded onto output n e u r o n s with less overlap, a n d i n that m a n y active input lines may be coded onto few active output lines; a n d does not allocate n e u r o n s to events that never occur (cf. M a r r , 1970, 1971; R u m elhart and Zipser, 1986; Grossberg, 1982). It may be noted that there is no special correspondence between the input pattern a n d w h i c h output lines are selected. It is thus not useful for any associative m a p p i n g between an input a n d an output event, a n d is thus different from linear associative matrix memories (Rolls, 1987). Instead, this type of matrix finds associations or correlations between input events ( w h i c h are expressed as sets of simultaneously active horizontal input lines or axons), allocates output n e u r o n s to reflect the complex event, a n d stores the required association between the input lines onto the few output neurons activated by each complex input event. There is some evidence that i n the h i p p o c a m p u s the s y n a p s e s between inactive axons and active output n e u r o n s become weaker (see M c N a u g h t o n , 1983; L e v y , 1985). T h e effect of this i n the learning system described w o u l d be to facilitate accurate a n d rapid classification, i n that w e a k e n i n g synapses onto a postsynaptic n e u r o n from axons that are not active w h e n it is strongly activated w o u l d reduce the probability that it will respond to a stimulus that must be placed into a different class. It is also of interest that (postulate 4) it is not physiologically unreasonable that the total synaptic strength onto a postsynaptic n e u r o n is s o m e w h a t fixed (Levy a n d D e s m o n d , 1985). Another feature of hippocampal circuitry is the mossy-fiber system, w h i c h connects the granule cells of the dentate gyrus to the C A 3 pyramidal cells of the hippocampus. E a c h mossy fiber forms approximately 10 "mosses," i n w h i c h there are dendrites of p e r h a p s five different C A 3 p y r a m i d a l cells. T h u s each dentate granule cell m a y contact approximately 50 C A 3 pyramidal cells (in the mouse; see Braitenberg and Schuz, 1983). I n the rat, each mossy fiber forms approximately 14 "mosses" or contacts w i t h C A 3 cells, there are 1 x 10 dentate granule cells a n d thus 14 x 10 mosses onto 0.18 x 10 C A 3 cells ( D . A m a r a l , personal communication), a n d thus each C A 3 p y r a m i d a l cell may be contacted by approximately 78 dentate granule cells. T h i s means that (in the rat) the probability that a C A 3 cell is contacted by a given dentate granule cell is 78 synapses/10 granule cells = 0.000078. T h e s e mossy-fiber s y n a p s e s are very large, presumably because w i t h s u c h a relatively s m a l l n u m b e r on each C A 3 cell dendrite (and a m u c h smaller n u m b e r active at any one time), each s y n a p s e must be relatively strong. 6
6
6
6
728
246
INPUT:
fl
INPUT : f
CYCLE:
202
CYCLE :
202
C O M P E T I T I O N
UTTER
MATRIX
F 1 1 T Eft .
Figure 3. Simulation of learning i n a competitive matrix memory. The architecture is as s h o w n in Fig. 2, except thai there are 64 horizontal axons and 64 vertical dendrites, w h i c h form the r o w and columns, respectively, of the 64 x 64 matrix of synapses. The strength of each synapse, w h i c h was initially r a n d o m , is indicated by the darkness of each pixel. The activity of each of the 64 i n p u t axons is represented in the 64-element vector at the left of the diagram by the darkness of each pixel. The output firing of the
729
247
O n e effect that c a n be achieved by this low probability of contact of a particular dentate granule cell with a particular p y r a m i d a l cell is pattern separation. T h i s is achieved in the following w a y . C o n s i d e r a pattern of firing present over a set of dentate granule cells. The probability that any two C A 3 p y r a m i d a l cells receive s y n a p s e s from a similar subset of the dentate granule cells is very low (because of the low probability of contact of any one dentate granule cell w i t h a p y r a m i d a l cell), so that each C A 3 p y r a m i d a l cell is influenced by a very different subset of the active dentate granule cells. T h u s each pyramidal cell effectively samples a very small subset of the active granule cells, a n d it is therefore likely that each C A 3 p y r a m i d a l cell will respond differently to the others, so that in this w a y pattern separation is a c h i e v e d . [The effect is similar to codon formation described i n other contexts by Marr (1970). | W i t h modifiability of the mossy-fiber synapses, C A 3 n e u r o n s learn to r e s p o n d to just those subsets of activity that do occur i n dentate granule cells. M o r e over, because of the low probability of contact, and because of the competition between the C A 3 neurons, the patterns that occur are evenly distributed over different ensembles of C A 3 neurons. T h i s pattern separation effect c a n be seen i n F i g . 4. (It may be noted that even if competition does not operate i n this system to increase orthogonality, then the l o w probability of c o n n e c t i o n s j u s t d e s c r i b e d w o u l d n e v e r t h e l e s s m e a n that the h i p p o c a m p u s could operate to produce relatively orthogonal representations.) It is notable that i n addition to the m o s s y fiber inputs, the C A 3 pyramidal cells also receive inputs directly from perforant path fibers
vertical neurons is represented in the same way by the output vectors at the bottom of the diagram. The upper output vector is the result of multiplying the input stimulus through the matrix of synaptic weights. The vector resulting from the application of competition between the output neurons (which produces contrast enhancement between the elements or neurons of the vector) is shown below by the vector labeled "after filter." The state of the matrix is shown after 203 cycles, in each of which stimuli with 8 of 64 active axons was presented and the matrix allowed to leam as described in the text. The stimuli were presented in random sequence, and consisted of a set of vectors that overlapped in 0, 1, 2, 3, 4, 5, or 6 positions with the next vector in the set. The columns of the matrix were sorted after the learning to bring similar columns together, so that the types of neuron formed, and the pattern of synapses formed on their dendrites, can be seen easily. The dendrites with random patterns of synapses have not been allocated to any of the input stimuli. It is shown thai application of one of the input stimuli (overlap four A) produced one pattern of firing of the output neurons, and that application of input stimulus overlap four B produced a different pattern of firing of the output neurons. Thus the stimuli were correctly classified by the matrix as being different.
730
248
Cfi3
:
P . r f m i n l
pilh>,i
3
INPUT D H N TflT I
H I PPOCAMPAL SIMULATION
GYJtUS
C V C L E : 2 0 0 0
T_ Cft3
:
Hoe»
H
r i bir I.
OUTPUT JJICTOJt fllJUSflB 0UT7UT
f
INPUT; DENTHTX
HIPPOCAMPflL S I M U L f l T I ON
GrRUS
CYCLE:26B0
OUTPUT
UICTOH.
nDJusTiB
oi'TPur.
; :
c«i
Figure 4. Hippocampal stimulation. Conventions as in Fig. 3. The dentate gyrus is shown as a competition matrix at the left, receiving input stimuli from the perforant path. The vertical dendrites of the C A 3 pyramidal cells extend throughout the three submatrices shown in the middle. The middle submatrix receives the output of the dentate granule cells via the mossy fibers with potentially powerful synapses and a low contact probability, and operates as a competition matrix. Pattern separation can be seen to operate in that input vectors are converted into output vectors with many elements activated by the inputs about which the submatrix has learned,
731
249 (see Figs. 1 and 2). T h i s is not a sparse projection, i n that each p y r a m i d a l cell may receive o n the order of 2300 s u c h s y n a p s e s . [This is calculated u s i n g the evidence that of 15 m m of total dendritic length with 10,000 spines, approximately 3.5 m m (range 2.5-4.5 m m ) is i n the l a c u n o s u m m o l e c u l a r e a n d thus r e c e i v e s i n p u t s from the perforant p a t h ( D . G . Amaral, personal communication).] What w o u l d be the effect of this input together with the very sparse, but strong, s y n a p s e s from the m o s s y fibers? O n e effect is that the mossy-fiber input w o u l d cause the pattern of s y n a p s e s (considered as a vector) on each p y r a m i d a l cell to point i n a different direction i n a multidimensional space. H o w e v e r , the precise direction i n that multidimensional space could not be w e l l specified by the relatively small n u m b e r of mossy-fiber synapses onto each C A 3 pyramidal cell. H o w e v e r , once pointed to that part of space by the m o s s y fibers, a particular cell w o u l d show cooperative Hebbian learning between its activation by the m o s s y i n p u t a n d the direct perforant path input, allowing the direct input to come by learning to specify the exact direction of that cell i n multidimensional space m u c h more effectively than by the coarse mossy-fiber input alone. T h i s effect can be seen i n F i g . 4. T h e relative weighting i n this simulation was that the mossy-fiber input had a n effect on each n e u r o n that w a s five times greater than that of the direct perforant path input. T h u s it is suggested that the combination of the sparse mossy-fiber i n p u t a n d the direct perforant path i n p u t is to achieve pattern separation, and at the s a m e time to allow the response of the n e u r o n to be determined not just by the sparse mossy-fiber input, but m u c h more finely by making use i n addition of the direct perforant path input. A n additional feature of the h i p p o c a m p u s , w h i c h is developed i n the C A 3 p y r a m i d a l cells i n particular, is the presence of strong recurrent collaterals, w h i c h return from the output of the matrix to cross over the neurons of the matrix, as s h o w n i n Figs. 1 a n d 2. T h i s anatomy immediately suggests that this is a n autoassociation matrix. T h e effect of such recurrent collaterals is to m a k e that part of the matrix into a n a u toassociation (or autocorrelation) matrix. T h e autoassociation arises because the output of the matrix, expressed as the firing rate of the C A 3
and in that different output vectors are produced for even quite similar input vectors. The upper C A 3 submatrix operates as a competition matrix with a direct perforant path input. The lower CA3 submatrix operates as an autoassociation matrix (formed by the recurrent collaterals). The output of the CA3 cells (summed vertically up and down Ihe dendrite) is then used as the input (via the Schaffer collaterals) to the C A l cells, which operate as a competition matrix. The states of the matrices after 2000 presentations of the same set of stimuli used for Fig. 3 are shown. One point demonstrated is that two very similar stimuli, overlap six A in Fig. 4a, and overlap six B in Fig. 4b, produce output vectors at C A l that are relatively orthogonal to each other.
732
250 pyramidal cells, is fed back along the horizontally running axons so that the pattern of activity in this part of the matrix (the CA3 pyramidal cells) is associated with itself (see, e.g., Kohonen et al, 1981; Rolls, 1987a). It can be noted here that for this suggestion to be the case, the synapses of the recurrent collaterals would have to be modifiable, and the modification rule would require alteration of synaptic strength when both the presynaptic fiber and the postsynaptic dendrite were strongly activated. Further, the probability of contact of the neurons in the autoassociation matrix must not be very low if it is to operate usefully (see Marr, 1971). Given that the region of the CA3 cell dendrites on which the recurrent collaterals synapse is long (approximately 11.5 mm), and that the total dendritic length is approximately 15 mm and has approximately 10,000 spines (D. G. Amaral, personal communication), approximately 7700 synapses per CA3 pyramidal cell could be devoted to recurrent collaterals, which with 180,000 CA3 neurons in the rat makes the probability of contact between the CA3 neurons 0.043. This is high enough for the system to operate usefully as an autoassociation memory (see Marr, 1971). It is remarkable that the contact probability is so high, and also that the CA3 recurrent collateral axons travel so widely in all directions that they can potentially come close to almost all other CA3 neurons (D. G. Amaral, personal communication). The importance of the autoassociation performed by this part of the matrix is that it forms a recognition memory, with all the advantageous emergent properties of a matrix memory, such as completion, generalization, and graceful degradation (see Kohonen et al, 1977, 1981; Kohonen, 1984; Rolls, 1987). One property that is particularly relevant here is completion, in that if part of a stimulus (or event) occurs, then the autoassociation part of the matrix completes that event. Completion may operate particularly effectively here, because it operates after the granule-cell stage, which will reduce the proportion of neurons firing to represent an input event to a low number partly because of the low probability of contact of the granule cells with the CA3 pyramidal cells. It is under these conditions that the simple autocorrelation effect can reconstruct the whole of one pattern without interference, which would arise if too high a proportion of the input neurons was active. It is of interest that a scheme of this type, although expressed in a different way to the autoassociation matrix formulation, was proposed by Marr in 1971. Another effect of the autoassociation matrix is that patterns of activity that are not similar to those already learned by this type of recognition memory are lost, so that noisy patterns can be cleaned up by the autoassociation matrix. It is further notable that these completion and deaning-up processes benefit from several iterations (repeated cycles) of the autoassociation feedback effect. It has been suggested by B. McNaughton (personal communication, 1987) that one function of hippocampal theta activity may be to allow this autoassociation effect pro-
733
251 duced by the recurrent collaterals to cycle for several iterations (in a period of approximately 50 msec), a n d then to stop, so that the system can operate again w i t h maximal sensitivity to n e w inputs received on the m o s s y fiber a n d perforant path systems by the C A 3 cells. The C A l pyramidal cells that receive from the C A 3 cells are considered to form a further stage of competitive learning, w h i c h has the effect of further classification of signals received, p e r h a p s enabled by the pattern of connections within the h i p p o c a m p u s to form these classifications over inputs received from any part of the association neocortex. The firing of the C A l cells w o u l d thus achieve a m u c h more economical and orthogonal classification of signals than that present i n the perforant path input to the hippocampus. T h e s e signals are then returned to the association neocortex via the s u b i c u l u m , entorhinal cortex, a n d parahippocampal gyrus, as indicated i n Figs. 1 a n d 2. It is suggested below that one role that these economical (in terms of the n u m b e r of activated fibers) a n d relatively orthogonal signals play in neocortical function is to guide information storage or consolidation i n the neocortex. It may be noted that multilayer networks (such as the hippocampus) can potentially solve classes of problems that cannot be solved in principle by single-layer nets (Rumelhart a n d Zipser, 1986). T h i s is because subcategories formed i n an early stage of processing can enable a later stage to find solutions or categories that are not linearly separable i n the input information space. H a v i n g considered the computational theory of h o w the hippocampal circuitry m a y function, w e can n o w turn to a systems-level analysis, in which the inputs and outputs of the hippocampus are considered, and the function performed by the h i p p o c a m p u s i n relation to overall brain function can be formulated.
III. Systems-Level Theory of Hippocampal Function First the anatomical connections of the primate h i p p o c a m p u s w i t h the rest of the brain will be considered, i n order to provide a basis for considering h o w the computational ability of the h i p p o c a m p u s could be u s e d by the rest of the brain. T h e h i p p o c a m p u s receives inputs by two m a i n routes, the entorhinal cortex and the fimbria/fornix. T h e entorhinal cortex (area 28) provides it with extensive inputs from the neocortex (see F i g . 5). T h u s all temporal neocortical areas project to area 35, the perirhinal cortex, or to area T F - T H , in the parahippocampa! g y r u s , w h i c h i n turn projects to the entorhinal cortex ( V a n H o e s e n and P a n d y a , 1975a,b; V a n H o e s e n , 1982; A m a r a l , 1987). T h e parietal cortex (area 7) projects to area T F - T H , and thus c a n potentially influence the h i p p o c a m p u s . T h e orbitofrontal
734
252
AFFERENT CONNECTIONS
EFFERENT CONNECTIONS
Par a hippocampal
Parariipoocampal
Figure 5 . Conneclions of the primate hippocampus with (he neocortex (from Van Hoesen, 1982). A media) view of the macaque brain is shown below, and a lateral view is shown inverted above. The hippocampus receives its inputs via the para hippocampal gyrus, areas TF and TH, and the entorhinal cortex, area 28. T h e return projections to the neocortex (shown on the right) pass through the same areas. Cortical areas 19, 20, and 21 are visual association areas, 22 is auditory association cortex, 7 is parietal association cortex, and 9, 46, 12, and 13 are frontal cortical association areas.
cortex, areas 12 a n d 13, projects directly to the entorhinal cortex ( V a n H o e s e n et al., 1975). In addition, the entorhinal cortex receives inputs from the a m y g d a l a . T h e entorhinal cortex itself projects via the perforant p a t h to reach primarily the dentate granule cells of the h i p p o c a m p u s proper. T h u s by these routes the hippocampus receives information after it h a s been highly processed through the temporal, parietal, a n d frontal cortices. It m u s t thus be of great importance for h i p p o c a m p a l function i n the primate that b y its m a i n input, the perforant path, it receives information from the highest parts of the neocortex. There are also inputs to the h i p p o c a m p u s via the fimbria/fornix from the cholinergic cells of the m e d i a l s e p t u m a n d the a d j o i n i n g limb of the diagonal b a n d of Broca. T h e h i p p o c a m p u s also receives a noradrenergic i n p u t from the locus coeruleus, a n d a 5-hydroxytryptamine ( 5 - H T ) input from the m e d i a n raphe nucleus. A m a j o r output of the h i p p o c a m p u s arises from the h i p p o c a m p a l p y r a m i d a l cells, a n d projects back via the s u b i c u l u m to the entorhinal
735
253
cortex, w h i c h i n turn h a s connections back to area T F - T H , w h i c h i n turn projects back to the neocortical areas from w h i c h it receives inputs ( V a n H o e s e n , 1982) (see Figs. 5 a n d 1). T h u s the h i p p o c a m p u s c a n potentially influence the neocortical regions from w h i c h it receives inputs. T h i s is the p a t h w a y suggested as being involved in guiding m e m o r y storage i n the neocortex. A second efferent projection of the h i p p o c a m p a l s y s t e m reaches the s u b i c u l u m from the C A l p y r a m i d a l cells, a n d travels via the fimbria a n d (postcomm is sural) fornix to the anterior thalamus, a n d to the m a m m i l l a r y bodies, w h i c h i n turn project to the anterior thalamus. T h e anterior thalamus i n turn projects into the cingulate cortex, w h i c h itself has connections to the supplementary motor cortex, p r o v i d i n g a potential route for the h i p p o c a m p u s to i n f l u e n c e motor o u t p u t ( V a n H o e s e n , 1982). It is suggested that functions of the h i p p o c a m p u s in, for example, conditional spatial response learning utilize this output path to the motor system. T h e connections of the hippocampus w i t h other parts of the brain, a n d the internal connections and synaptic modifiability described above, suggest that the h i p p o c a m p u s s h o u l d be able to detect, a n d classify onto a few specifically responding neurons, specific conjunctions of complex (cortically processed) events, such as that a particular object (presumably reflecting temporal lobe visual processing) h a s a p p e a r e d i n a particular position in space (presumably reflecting parietal input). Another example might be that a particular stimulus s h o u l d be associated w i t h a particular spatial motor response. It h a s been s h o w n above that this is the type of quite specific information that comes to activate different h i p p o c a m p a l neurons w h i l e m o n k e y s are performing object-place memory a n d conditional spatial response learning tasks. I n d e e d , the neurophysiological findings described above provide evidence that supports the m o d e l of hippocampal function just described. T h e m o d e l is also supported by the evidence that d u r i n g learning of conditional spatial responses some hippocampal neurons start, but then stop, showing differential responses to the different stimuli, consistent with competitive interactions berween hippocampal neurons d u r i n g learning, so that only some h i p p o c a m p a l n e u r o n s become allocated to any one learned event or contingency (see above). T h e analyses above have s h o w n that the h i p p o c a m p u s receives from high-order areas of association cortex; is able by virtue of the large n u m b e r of synapses on its dendrites to detect conjunctions of events even w h e n these are widely separated i n information space, w i t h their origin from quite different cortical areas; allocates n e u r o n s to code efficiently for these conjunctions probably u s i n g a competitive learning mechanism; and has connections back to the neocortical areas from which it receives, as well as to subcortical structures via the fimbria/fornix system. W h a t could be achieved by this system? It appears that the longterm storage of information is not i n the h i p p o c a m p u s , at least i n h u -
736
254
mans, i n that damage to the hippocampus in h u m a n s does not necessarily result i n m a j o r retrograde a m n e s i a (Squire, 1986; this volume, chapter 12). O n the other h a n d , the h i p p o c a m p u s does appear to be necessary for the storage of certain types of information (characterized by the description declarative, or k n o w i n g that, as contrasted w i t h procedural, or k n o w i n g h o w ) . H o w could the h i p p o c a m p u s then be i n v o l v e d i n the storage of information? T h e suggestion that is m a d e o n the basis of these a n d the other findings described above is that the h i p p o c a m p u s is specialized to detect the best w a y i n w h i c h to store information, a n d then by the return paths to the neocortex directs memory storage there. Clearly the hippocampus, w i t h its large n u m b e r of s y n a p s e s o n e a c h n e u r o n a n d its potentiation type of l e a r n i n g , is able to detect w h e n there is coherence (i.e., conjunctive activation of arbitrary sets of its i n p u t fibers), a n d is able, as i n d i c a t e d b o t h theoretically a n d b y r e c o r d i n g s m a d e i n the b e h a v i n g m o n k e y , to allocate n e u r o n s to economically (i.e., with relatively few n e u r o n s active) code for each complex input event. S u c h n e u r o n s could then represent a n efficient w a y i n w h i c h to store information, i n that r e d u n d a n c y w o u l d effectively h a v e been r e m o v e d from the input signal. I n a sense, the optimal w a y i n w h i c h to build high-level feature analyzers c o u l d be d e t e r m i n e d by the h i p p o c a m p u s . It s h o u l d be noted that this theory is not inconsistent w i t h the possibility that the h i p p o c a m p u s prov i d e s a w o r k i n g m e m o r y , i n that i n the present theory the h i p p o c a m p u s sets u p a representation u s i n g H e b b i a n learning, w h i c h is useful i n determining h o w information can best be stored i n the neocortex. [The representation found by the hippocampus could provide a useful working m e m o r y (see O l t o n , 1983), a n d i n d e e d i n the object-place m e m o r y task described above the object a n d place combinations formed onto single h i p p o c a m p a l n e u r o n s w o u l d provide a useful w o r k i n g m e m o r y . It m a y be that by u n d e r s t a n d i n g the operations performed by the h i p p o c a m p u s at the neuronal network level, it can be seen how the hippocampus could contribute to several functions that are not necessarily inconsistent.] T h e question then arises of w h e r e the long-term storage occurs, a n d h o w it m a y be directed b y the h i p p o c a m p u s . N o w the h i p p o c a m p u s is reciprocally connected via the s u b i c u l u m a n d entorhinal cortex with the parahippocampal gyrus, w h i c h i n turn is reciprocally connected with m a n y high-order areas of association neocortex (see Fig. 5). It is therefore possible that the actual storage takes place i n the parahippocampal gyrus, a n d that this might be particularly appropriate for multimodal memories. However, having detected that, for example, a visual stimulus is regularly associated with an event i n another modality such as a response, it might be useful to direct the u n i m o d a l representation of that visual image, so that it is stored efficiently a n d c a n provide a useful i n p u t to the multim o d a l conjunction store. T h u s it is suggested that return p a t h w a y s (for example, via the p a r a h i p p o c a m p a l gyrus) to u n i m o d a l cortex (for ex-
737
255 ample, inferior temporal cortex, area T E ) might be u s e d to direct unimodal storage, too, by contributing to detection of the most economical w a y i n w h i c h to store representations of stimuli. T h e question of h o w the h i p p o c a m p a l output is u s e d by the neocortex will be considered next. G i v e n that the hippocampal output returns to the neocortex, a theory of backprojections in the neocortex will be needed. T h i s is developed next. By w a y of introduction to this, it m a y be noted that which particular hippocampal neurons happen to represent a complex input event is not determined by any teacher or forcing (unconditioned) stimulus. T h u s the neocortex m u s t be able to utilize the signal rather cleverly. O n e possibility is that any neocortical n e u r o n w i t h a n u m b e r of afferents active at the same time that h i p p o c a m p a l return fibers in its vicinity are active modifies its responses, so that it comes to respond better to those afferents the next time they occur. T h i s learning by the cortex w o u l d involve a Hebb-like learning m e c h a n i s m . It m a y be noted that one function served by what are thus in effect backprojections from the hippocampus is some guidance for or supervision of neocortical learning. It is a problem of u n s u p e r v i s e d learning systems that they can detect local conjunctions efficiently, but that these are not necessarily those of most use to the whole system. It is proposed that it is exactly this problem that the h i p p o c a m p u s helps to solve, by detecting useful conjunctions globally (i.e., over the whole of information space), and then directing storage locally at earlier stages of processing so that filters are built locally that provide representations of input stimuli that are useful for later processing.
IV. Theoretical Significance of Backprojections in the Neocortex The forward and backward projections that will be considered are s h o w n in F i g . 6 [for further anatomical information see Peters a n d Jones (1984)]. In primary sensory cortical areas, the m a i n extrinsic "forward" input is from the thalamus and ends in layer 4, w h e r e synapses are formed onto spiny stellate cells. T h e s e i n turn project heavily onto p y r a m i d a l cells in layers 3 a n d 2, w h i c h i n turn s e n d projections f o r w a r d to terminate strongly i n layer 4 of the next cortical layer [on small p y r a m i d a l cells in layer 4 or on the basal dendrites of the layer-2 a n d -3 (superficial) pyramidal cells]. ( A l t h o u g h the forward afferents end strongly i n layer 4, the forward afferents have some s y n a p s e s also onto the basal dendrites of the layer-2 pyramidal cells, as well as onto layer-6 p y r a m i d a l cells and inhibitory interneurons). I n p u t s reach the layer-5 (deep) p y r a m i d a l cells from the pyramidal cells i n layers 2 a n d 3 (Martin, 1984), a n d it is the deep pyramidal cells that s e n d backprojections to e n d i n layer 1 of the
738
256 Cortical
connections
Amygdal a Hippocampus
Figure 6. Schematic diagram of forward and backward projections in the neocortex. The superficial pyramidal cells (triangles) in layers 2 and 3 project forward to terminate in layer 4 of the next cortical area. The deep pyramidal cells in the next area project back (mainly) to layer 1 of the preceding cortical area, in which there are apical dendrites of pyramidal cells. The hippocampus and amygdala also are the source of backprojections that end (mainly) in layer 1. Spiny stellate cells are represented by small circles in layer 4. See text for further details.
p r e c e d i n g cortical area (see F i g . 6), w h e r e there are apical dendrites of p y r a m i d a l cells. T h e r e are few current theories about the functions subs e r v e d by the cortico-cortical backprojections, even though there are a l m o s t as m a n y b a c k p r o j e c t i n g as f o r w a r d - p r o j e c t i n g a x o n s . It is i m portant to note that i n addition to the axons a n d their terminals i n layer 1 from the s u c c e e d i n g cortical stage, there are also axons a n d terminals in layer 1 i n m a n y stages of the cortical hierarchy from the amygdala a n d (via the s u b i c u l u m , entorhinal cortex, a n d p a r a h i p p o c a m p a l cortex) from the h i p p o c a m p a l formation (see F i g . 6) ( V a n H o e s e n , 1981; T u r n e r , 1981; A m a r a l a n d Price, 1984; A m a r a l , 1986, 1987). T h e amygdala a n d h i p p o c a m p u s are stages of information processing at w h i c h the different s e n s o r y m o d a l i t i e s ( s u c h as v i s i o n , h e a r i n g , t o u c h , a n d taste for the a m y g d a l a ) are brought together, so that correlations between inputs i n different modalities c a n be detected i n these regions, but not at prior cortical processing stages i n e a c h modality, as these cortical processing stages are u n i m o d a l . A s a result of bringing together the two modalities, significant correspondences between the two modalities can be detected. O n e example might be that a particular visual stimulus is associated with the taste of food. A n o t h e r example m i g h t be that another v i s u a l stimulus is associated w i t h painful touch. T h u s at these stages of processing, but not before, the significance of, for example, v i s u a l a n d auditory stimuli can be detected a n d signalled, a n d s e n d i n g this information back to the neocortex thus c a n provide a signal that indicates to the cortex that i n -
739
257 formation s h o u l d be stored, but e v e n more than this, provides an orthogonal signal that could help the neocortex to store the information efficiently. T h e way in w h i c h backprojections could assist learning in the cortex can be considered u s i n g the architecture s h o w n in F i g . 7. T h e (forward) input stimulus occurs as a vector applied to (layer-3) cortical p y r a m i d a l cells through modifiable s y n a p s e s in the standard w a y for a competitive net. (If it is a primary cortical area, the i n p u t stimulus is at least partly relayed through spiny stellate cells, w h i c h m a y help to normalize a n d orthogonalize the input patterns in a preliminary w a y before the patterns are applied to the layer-3 p y r a m i d a l cells. If it is a n o n p r i m a r y cortical area, the cortico-cortical forward axons m a y e n d more strongly o n the basal dendrites of n e u r o n s in the superficial cortical layers.) T h e lower set of synapses on the pyramidal cells w o u l d then start by competitive learning to set u p representations o n the lower parts of these n e u r o n s , w h i c h w o u l d represent correlations in the i n p u t information space and could be said to correspond to features i n the i n p u t information space, w h e r e a feature is defined simply as the representation of a correlation in the input information space. Consider n o w the application of one of the (forward) input stimulus vectors w i t h the conjunctive application of a pattern vector via the backprojection axons w i t h terminals in layer 1. G i v e n that all the synapses in the matrix start w i t h random weights, some of the p y r a m i d a l cells will by chance be strongly influenced both by the (forward) input stim-
BACK PROJECTION INPUT STIMULUS
OUTPUT Figure 7. The archilecture used to simulate the properties of backprojections. The forward-projected ("input stimulus") and backprojected axons make Hebb modifiable synapses onto the same set of vertical dendrites, which represent cortical pyramidal cells.
740
258 u l u s a n d by the backprojecting vector. T h e s e strongly activated n e u r o n s will then compete with each other as i n a standard competitive net, to produce contrast e n h a n c e m e n t of their firing patterns. [The relatively short-range (50 u.m) excitatory operations produced by bipolar and double bouquet cells, together w i t h more w i d e s p r e a d (300-500 urn) recurrent lateral inhibition p r o d u c e d by the smooth n o n p y r a m i d a l cells and perh a p s the basket cells, may be part of the m e c h a n i s m of this competitive interaction.] Next, H e b b i a n learning takes place as in a competitive net, w i t h the addition that not o n l y are the s y n a p s e s between forward-projecting axons a n d active postsynaptic neurons modified, but also the s y n a p s e s i n l a y e r 1 b e t w e e n the b a c k w a r d p r o j e c t i n g a x o n s a n d the (same) active postsynaptic n e u r o n s are modified. T h i s functional architecture h a s the following properties. First, the backprojections, w h i c h are assumed to be relatively information-rich and orthogonal to each other as a result of the conjunctions formed by the hippocampus, amygdala, or next cortical stage, help the neurons to leam to r e s p o n d differently to (and thus to separate) input stimuli (on the forward projection lines) even w h e n the stimuli are very similar. T h i s is illustrated i n the simulation s h o w n i n F i g . 8, in w h i c h it is s h o w n that input stimuli that overlap even i n six positions out of eight can be easily learned as separate if presented conjunctively w i t h different orthogonal backprojecting "tutors." [For a similar idea o n the guidance of a competitive learning s y s t e m see R u m e l h a r t a n d Zipser (1986).] I n the neocortex, the backprojecting tutors can be of two types. O n e originates from the a m y g d a l a and h i p p o c a m p u s , a n d , by benefiting from cross-modal c o m p a r i s o n , c a n provide a n orthogonal backprojected vector. Moreover, this backprojection m a y only be activated if the m u l timodal areas detect that the visual stimulus is significant, because (for e x a m p l e ) it is associated with a pleasant taste. T h i s provides one w a y in w h i c h guidance c a n be provided for a competitive learning system as to w h a t it s h o u l d learn, so that it does not attempt to lay d o w n representations of all i n c o m i n g sensory information. A n o t h e r w a y for this i m p o r t a n t f u n c t i o n to be a c h i e v e d is by a c t i v a t i o n of n e u r o n s that "strobe" the cortex w h e n n e w or significant stimuli are s h o w n . T h e cholinergic s y s t e m originating i n the basal forebrain ( w h i c h itself receives information from the a m y g d a l a ) , a n d the noradrenergic input to layer 1 of the cortex from the locus coereleus may also contribute to this function (see Rolls, 1987; Bear a n d Singer, 1986). H o w e v e r , i n that there are relatively few n e u r o n s i n these systems, it is suggested that these projections only provide a simple "strobe," rather than carrying patternspecific i n f o r m a t i o n to g u i d e h o w i n f o r m a t i o n is c o n s o l i d a t e d ( R o l l s , 1987). T h e second type of backprojection is that from the next cortical area i n the hierarchy. T h i s operates i n the s a m e m a n n e r , and because it is a competitive s y s t e m , is able to further categorize or orthogonalize the stimuli it receives. T h i s next cortical stage then projects back these
741
259 more orthogonal representations as tutors to the preceding stage, to effectively build at the preceding stage better filters for the diagnosis of the categories being found at the next stage. A second property of this architecture is that if only the tutor is presented, then the neurons originally activated by the forward projecting input stimuli are activated. T h i s occurs because the s y n a p s e s from the backprojecting axons onto the p y r a m i d a l cells have been modified only w h e r e there w a s conjunctive forward a n d backprojected activity d u r i n g learning. T h i s thus provides a m e c h a n i s m for recall. A simulation of this is s h o w n i n F i g . 8. C o n s i d e r the situation w h e n i n the v i s u a l s y s t e m the sight of food is forward-projected onto pyramidal cells, and conjunctively there is a backprojected representation of the taste of the food. N e u r o n s that have conjunctive inputs from these two stimuli set u p representations of both, so that later if only the taste representation is backprojected, then the visual neurons originally activated by the sight of that food will be activated. I n this w a y m a n y of the low-level details of the original visual stimulus can be recalled. E v i d e n c e that d u r i n g recall relatively early cortical processing stages are activated comes from cortical blood flow studies i n h u m a n s , in w h i c h it has been found, for example, that quite early visual cortical association areas are activated d u r i n g recall of visual (but not auditory) information (Roland a n d Friberg, 1985; R o l a n d etal., 1980). A third property of this architecture is that attention could operate from higher to lower levels to selectively facilitate only certain pyramidal cells by using the backprojections. Indeed, the backprojections described could produce m a n y of the "top-down" influences that are c o m m o n in perception. A fourth property is that semantic p r i m i n g could operate by using the backprojecting neurons to provide a small activation of just those neurons in earlier stages that are appropriate for responding to that semantic category of input stimulus. A fifth property of such a return m e c h a n i s m , w h i c h on detecting a conjunction (perhaps across modalities) i m p r o v e d u n i m o d a l representations of the input stimuli, w o u l d be a form of positive feedback, w h i c h w o u l d result i n gradually i m p r o v i n g storage as a result of the reciprocal interactions, as the feedback effect produced a better representation at the preceding level to be fed forward, with this occurring repeatedly. T h i s might provide a neurophysiological a n d computational basis for any gradient of retrograde amnesia that may occur for the period just before disruption of temporal-lobe function (Squire, 1986; this volume, chapter 12). A sixth property of the backprojections is that they w o u l d assist the stability of the preceding competitive networks by providing a relatively constant guiding signal as a result of associations made at a higher stage, for example to a n unconditioned taste or somatosensory input. T h i s theory of the functions of backprojections i n the
742
neocortex
M A T R I X
BflCKPBOJICTIOH
OUTPUT
PART
UECTOR.
ADJUSTED
OUTPUT.
MATRIX
miPIOJICTIOM
ADJUSTED
PART
OUTPUT.
Figure 8. Simulation of neocortical backprojection learning matrix. Conventions as in Fig. 3. (a) During learning, both a forward input (chosen from the same set as used in Fig. 3) and a backprojected vector that was orthogonal to the other backprojected vectors were presented simultaneously. After 206 cycles with input stimulus + backprojected tutor pairs chosen in random sequence, the synapses had modified as shown. Two quite similar input
743
MATRIX •ACKPROJECTIOH
PART
TUTOR:
2 —>
r O l W A l l
PROJ
OUTPUT
PART
CYCLE:
206
I HPUT :
t
UKCTOR.
ADJUSTED
OUTPUT.
HATRIX •ACKPROJECTIOH
PART
TUTOR: CYCLE:
296
I HPUT:
id)
ADJUSTED
OUTPUT.
stimuli (overlap six A and six B) produce different outputs (b and c). The learning has been guided by the backprojected tutors presented during learning. If only the tutor originally paired with input stimulus overlap six A is presented, then recall ot the output vector normally recognized by input stimulus overlap six A occurs (d).
744
262
requires a large n u m b e r of backprojecting axons, as pattern-specific information (used
to g u i d e l e a r n i n g b y p r o v i d i n g a s e t o f m u t u a l l y o r -
t h o g o n a l g u i d a n c e s i g n a l s , o r to p r o d u c e r e c a l l ) m u s t b e p r o v i d e d b y t h e b a c k p r o j e c t i o n s . It a l s o s o l v e s t h e d e - a d d r e s s i n g p r o b l e m , for
the
h i p p o c a m p u s d o e s n o t n e e d to k n o w e x a c t l y w h e r e i n t h e c o r t e x i n f o r m a t i o n s h o u l d be stored. I n s t e a d , the b a c k p r o j e c t i o n signal s p r e a d s w i d e l y i n l a y e r 1, a n d t h e s t o r a g e s i t e i s s i m p l y o n t h o s e n e u r o n s t h a t h a p p e n to r e c e i v e s t r o n g ( a n d p r e c i s e ) f o r w a r d a c t i v i t y a s w e l l a s b a c k projected activity. T h i s s c h e m e is consistent w i t h neocortical a n a t o m y , i n t h a t it r e q u i r e s t h e s a m e p y r a m i d a l c e l l to r e c e i v e b o t h f o r w a r d a n d ( m o r e d i f f u s e ) b a c k p r o j e c t e d a c t i v i t y , w h i c h t h e a r r a n g e m e n t of p y r a m i d a l cells w i t h a p i c a l d e n d r i t e s that e x t e n d all the w a y u p into layer 1 a c h i e v e s ( s e e P e t e r s a n d J o n e s , 1984). I n d e e d , i n c o n t r a s t to the r e l a t i v e l y l o c a l i z e d t e r m i n a l d i s t r i b u t i o n s of f o r w a r d c o r t i c o - c o r t i c a l a n d t h a l a m o cortical afferents, the cortico-cortical b a c k w a r d projections that e n d in l a y e r 1 h a v e a m u c h w i d e r h o r i z o n t a l d i s t r i b u t i o n , o f u p to s e v e r a l m i l l i m e t e r s ( A m a r a l , 1986). T h e s u g g e s t i o n is t h u s that this e n a b l e s
the
b a c k w a r d p r o j e c t i n g n e u r o n s to s e a r c h o v e r a l a r g e r n u m b e r of p y r a m i d a l c e l l s i n t h e p r e c e d i n g c o r t i c a l a r e a for a c t i v i t y t h a t i s c o n j u n c t i v e w i t h their o w n . It is a l s o of interest that the t h e o r y utilizes a H e b b i a n l e a r n i n g s c h e m e t h a t p r o v i d e s for l e a r n i n g to o c c u r w h e n c o n j o i n t l y t h e r e i s forw a r d a n d b a c k p r o j e c t e d i n p u t to a p y r a m i d a l c e l l r e s u l t i n g i n s u f f i c i e n t p o s t s y n a p t i c a c t i v a t i o n to p r o v i d e f o r m o d i f i c a t i o n of s y n a p s e s t h a t h a p p e n to b e a c t i v e . T h i s p r o v i d e s t h e o p p o r t u n i t y to m a k e it c l e a r t h a t t h e t h e o r e t i c a l i d e a s i n t r o d u c e d h e r e m a k e c l e a r p r e d i c t i o n s that c a n be e m pirically tested. F o r e x a m p l e , the theory of backprojections just p r o p o s e d p r e d i c t s that the b a c k p r o j e c t i o n s i n the c e r e b r a l cortex h a v e
modifiable
s y n a p s e s o n p y r a m i d a l cells in the p r e v i o u s cortical area. If this w e r e f o u n d n o t to b e the c a s e i n e m p i r i c a l tests, t h e n the t h e o r y w o u l d
be
rejected. T h e ideas i n t r o d u c e d here also h a v e m a n y theoretical implications. O n e i s t h a t if t h e b a c k p r o j e c t i o n s a r e u s e d for r e c a l l , a s s e e m s l i k e l y a s d i s c u s s e d a b o v e (see also R o l a n d a n d F r i b e r g , 1985), t h e n this
would
p l a c e s e v e r e c o n s t r a i n t s o n t h e i r u s e for f u n c t i o n s s u c h a s e r r o r b a c k p r o p a g a t i o n . E r r o r b a c k p r o p a g a t i o n is a n interesting a n d p o w e r f u l a l gorithm in parallel distributed processing
n e t w o r k s for s e t t i n g
the
w e i g h t s i n h i d d e n units (i.e., n o d e s i n layers that are not i n p u t a n d o u t p u t l a y e r s ) to a l l o w n e t w o r k s to l e a m u s e f u l m a p p i n g s b e r w e e n i n p u t a n d o u t p u t l a y e r s ( R u m e l h a r t etal., 1 9 8 6 ) . H o w e v e r , t h e b a c k p r o j e c t i o n s i n t h e a r c h i t e c t u r e i n w h i c h t h i s a l g o r i t h m is i m p l e m e n t e d
have very
p r e c i s e f u n c t i o n s i n c o n v e y i n g e r r o r f r o m t h e o u t p u t l a y e r b a c k to t h e e a r l i e r , h i d d e n , l a y e r s . It w o u l d b e d i f f i c u l t to u s e t h e w e i g h t s ( s y n a p t i c s t r e n g t h s ) f r o m t h e b a c k p r o j e c t i n g n e u r o n s to n e u r o n s i n e a r l i e r l a y e r s b o t h to c o n v e y t h e e r r o r c o r r e c t l y a n d to h a v e t h e a p p r o p r i a t e s t r e n g t h s for recall. In conclusion, in this chapter experimental evidence on a n d the-
745
263
oretical approaches to the function of the hippocampus and of backprojections in the neocortex have been considered. Theories of how the hippocampus functions and of the functions of backprojections in the neocortex have been proposed. The theories are at the level of neuronal networks, and are based partly on evidence on the fine architecture of the networks, on the rules of synaptic modifiability incorporated, and on the systems-level connections. It is suggested that this approach will be useful in the future in linking anatomical evidence on structure to physiological evidence on modifiability, understanding the global properties of the networks, and thus understanding the role of the networks in brain function and behavior. Acknowledgments The author has worked on some of the experiments and neuronal n e t w o r k m o d e l i n g described here w i t h P. Cahusac, D . Cohen, J. Feigenbaum, R. Kesner, Y. Miyashita, and H . N i k i , and their collaboration is sincerely acknowledged. Discussions w i t h David G. Amaral of the Salk Institute, La Jolla, California, were also much appreciated. This research was supported by the Medical Research Council.
References Amaral, D . G. (1986). Amygdalohippocampal and amygdalocortical projections i n the primate brain. In "Excitatory A m i n o Acids and Epilepsy" (R, Schwarz and Y. Ben-Ari, eds.), p p . 3-17. Plenum, N e w York. Amaral, D . G. (1987). Memory: Anatomical organization of candidate brain regions. In "Handbook of Neurophysiology—The Nervous System. A m . Physiol. Soc., Washington, D.C. Amaral, D. G., and Price, J. L. (1984). Amygdalo-cortical projections i n the monkey iMacaca fascicuhris). f. Comp. Neurol. 230, 465-496. Bear, M . F . , and Singer, W. (1986), M o d u l a t i o n of visual cortical plasticity by acetylcholine and noradrenaline. Nature (London! 320, 172-176. Braitenberg, V . , and Schuz, A . (1983). Some anatomical comments on the hippocampus. In "Neurobiology of the Hippocampus" (W. Seifert, ed.), Chapter 2, pp. 21-37, Academic Press, L o n d o n . Cahusac, P. M . B., Feigenbaum, J . , Rolls, E. T., Miyashita, Y . , and N i k i , H . (1986). M o d ifications of neuronal activity d u r i n g learning i n the primate hippocampus. Neurosci. Lett. S24, S29. Cahusac, P. M . B., Miyashita, Y„ and Rolls, E. T. (1989). Responses of hippocampal neurons in the monkey related to delayed response and object-place memory tasks. Behav. Brain Res., i n press. Cahusac, P. M . B., Rolls, E. T . , Miyashita, Y., a n d N i k i , H . (1989). Modification of hippocampal neuronal responses d u r i n g the learning of a conditional response task i n the monkey. I n preparation. Gaffan, D . (1974). Recognition impaired and association intact i n the memory of monkeys after transection of the fornix. /. Comp. Physiol. Psychol. 8 6 , 1100-1109. Gaffan, D. (1977). Monkey's recognition memory for complex pictures and the effects of fornix transection. Q. ]. Exp. Psychol. 29, 505-514. Gaffan, D. (1985), Hippocampus: M e m o r y , habit and voluntary movement. Philos. Trans. R. Soc. London, Ser. B 308, 87-99.
746
264
Gaffan, D . , and Saunders, R. C . (1985). Running recognition of configural stimuli by fornix transected monkeys. Q. j. Exp. Psychol. 37B, 61-71, Grossberg, S. (1982). "Studies of Mind and Brain." Reidel, New York. Ito, M . (1984). ' T h e Cerebellum and Neural Control." Raven Press, New York. Jones, E . G . , and Peters, A . , eds. (1984). "Cerebral Cortex," Vol. 2. Plenum, New York. Kelso, S. R „ Ganong, A. H . , and Brown, T. H . (1986). Hebbian synapses in the hippocampus. Proc. Nail. Acad. Sci. U.S.A. S3, 5326-5330. Kohonen, T . (1984). "Self-Organization and Associative Memory." Springer-Verlag, Berlin and New York. Kohonen, T . , Lehtio, P., Rovamo, J . , Hyvarinen, J . , Bry, K . , and Vainio, L . (1977). A principle of neural associative memory. Neuroscience 2, 1065-1076. Kohonen, T„ Oja, E . , and Lehtio, P. (1981). Storage and processing of information in distributed associative memory systems. In "Parallel Models of Associative Memory" (G. E . Hinton and J. A. Anderson, eds.). Chapter 4, pp. 105-143. Erlbaum, Hillsdale, New Jersey. Levy, W. B. (1985). Associative changes in the synapse: LTP in the hippocampus. In "Synaptic Modification, Neuron Selectivity, and Nervous System Organization" (W, B. Levy, J. A . Anderson, and S. Lehmkuhle, eds.). Chapter 1, pp. 5-33. Erlbaum, Hillsdale, New Jersey. Levy, W. B., and Desmond, N . L . (1985). The rules of elemental synaptic plasticity. In "Synaptic Modification, Neuron Selectivity, and Nervous System Organization" (W. B. Levy, J. A . Anderson, and S, Lehmkuhle, eds.). Chapter 6, pp. 105-121. Erlbaum, Hillsdale, New Jersey. Marr, D. (1970). A theory for cerebral cortex. Proc. R. Soc. London, Ser. B 176, 161-234. Marr, D. (1971). Simple memory: A theory for archicortex. Philos. Trans. R. Soc. London, Ser. B 262, 23-81. Martin, K . A , C . (1984). Neuronal circuits in cat striate cortex, in "Cerebral Cortex" ( E . G . Jones and A . Peters, eds.). Vol. 2, Chapter 9, pp. 241-285. Plenum, New York. McNaughton, B. L . (1983). Activity dependent modulation of hippocampal synaptic efficacy: Some implications for memory processes. In "Neurobiology of the Hippocampus" (W. Seifert, ed.}. Chapter 13, pp. 233-252. Academic Press, London. Milner, B. (1972). Disorders of learning and memory after temporal lobe lesions in man. Clin. Neurosurg. 19, 421-446. Mishkin, M . (1978). Memory severely impaired by combined but not separate removal of amygdala and hippocampus. Nature (London) 273, 297-298. Mishkin, M . (1982). A memory system in the monkey, Philos. Trans. R. Soc. London, Ser. B 298, 85-95. Mishkin, M . (1989). Neural circuitry and mechanisms for visual recognition and memory. In "Neural Models of Plasticity: Theoretical and Empirical Approaches" (J. Byrne and W. O . Berry, eds.). Academic Press, New York. Miyashita, Y „ Rolls, E . T . , Cahusac, P. M . B., and Niki, H . (1989). Activity of hippocampal neurons in the monkey related to a conditional response task. /. Neurophysiol., in press. Murray, E . A . , and Mishkin, M . (1984). Severe tactual as well as visual memory deficits follow combined removal of the amygdala and hippocampus in monkeys. ]. Neurosci. 4, 2565-2580. Murray, E . A . , and Mishkin, M . (1985). Amygdalectomy impairs crossmodal association in monkeys. Science 228, 604-606. O'Keefe, J . (1983). Spatial memory within and without the hippocampal system, in "Neurobiology of the Hippocampus" (W. Seifert, ed.). Chapter 20, pp. 375-403. Academic Press, London. Olton, D. S. (1983). Memory functions and the hippocampus. In "Neurobiology of the Hippocampus" (W. Seifert, ed.), Chapter 19, pp. 335-373. Academic Press, London. Peters, A . , and Jones, E . G „ eds. (1984). "The Cerebral Cortex," Vol. 1. Plenum, New York.
747
265 Petrides, M. (1985). Deficits on conditional associative-learning tasks after frontal- and temporal-lobe lesions in man. Neuropsychologic! 23, 601-614. Roland, P. E . , and Friberg, L . (1985). Localization of cortical areas activated by thinking. /. Neurophysiol. S3, 1219-1243. Roland, P. E . , Vaemet, K . , and Lassen, N. A . (1980). Cortical activations in man during verbal report from visual memory. Neurosci. Lett., Suppl. 5, S478, Rolls, E. T . (1987). Information representation, processing and storage in the brain: analysis at the single neuron level. In "The Neural and Molecular Bases of Learning" (J.-P, Changeux and M. Konishi, eds.), pp. 503-540. Wiley, New York. Rolls, E . T . (1989). Visual information processing in the primate temporal lobe, in "Models of Visual Perception: From Natural to Artificial" (M. Imbert, ed ). Oxford Univ. Press, London and New York. Rolls, E . T . , Miyashita, Y . , Cahusac, P. M. B., Kesner, R. P., Niki, H . , Feigenbaum, ] . , and Bach, L . (1989). Hippocampal neurons in the monkey with activity related to the place in which a stimulus is shown. /. Neurosci.. in press. Rumelhart, D. E . , and Zipser, D. (1986). Feature discovery by competitive learning. In "Parallel Distributed Processing" (D. E . Rumelhart and ] . L . McClelland, eds.), Vol. 1, Chapters, pp. 151-193. MIT Press, Cambridge, Massachusetts. Rumelhart, D. E . , Hinton, G. E . , and Williams, R. J. (1986), Learning interna I representations by error propagation. In "Parallel Distributed Processing" (D. E . Rumelhart and J. L . McClelland, eds). Vol. 1, Chapters, pp. 318-362. MIT Press, Cambridge, Massachusetts. Rupniak, N. M. ] , , and Gaffan, D. (1988). Monkey hippocampus and learning about spatially directed movements. /. Neurosci. 7, 2331-2337. Scoville, W. B., and Milner, B. (1957). Loss of recent memory after bilateral hippocampal lesions. /. Neurol., Neurosurg. Psychiatry 20, 11-21. Smith, M . L . , and Milner, B. (1981). The role of the right hippocampus in the recall of spatial location. Neuropsychologic! 19, 781-793. Squire, L . (1986). Mechanisms of memory. Science 232, 1612-1619. Squire, L . (1989). Complex connections involved in memory functions, fn "Neural Models of Plasticity: Theoretical and Empirical Approaches" (J. Byrne and W. O . Berry, eds,). Academic Press, New York. Turner, B. H . (1981). The cortical sequence and terminal distribution of sensory related afferents to the amygdaloid complex of the rat and monkey. In "The Amygdaloid Complex" (Y. Ben-Ari, ed.), pp. 51-62. Elsevier, Amsterdam. Van Hoesen, G . W. (1981). The differential distribution, diversity and sprouting of cortical projections to the amygdala in the rhesus monkey, in "The Amygdaloid Complex" (Y. Ben-Ari, ed.), pp. 79-90. Elsevier, Amsterdam. Van Hoesen, G . W. (1982). The parahippocampal gyrus. New observations regarding its cortical connections in the monkey. Trends NeuroSci. 5, 345-350. Van Hoesen, G . W. and Pandya, D. N. (1975a). Some connections of the entorhinal (area 28) and perirhinal (area 35) cortices in the monkey. I . Temporal lobe afferents. Brain Res. 95, 1-24. r
Van Hoesen, G . W., and Pandya, D, N . (1975b). Some connections of the entorhinal (area 28) and perirhinal (area 35) cortices in the monkey. I I I . Efferent connections. Bruin Res. 95, 39-59, Van Hoesen, G . W., Pandya, D. N . , and Butters, N . (1975). Some connections of the entorhinal (area 28) and perirhinal (area 35) cortices in the monkey. II. Frontal lobe afferents. Brain Res. 95, 25-38. von der Malsburg, C . (1973). Self-organization of orientation-sensitive columns in the striate cortex. Kybernetik 14, 85-100. Wigstrom, H . , Gustaffson, B., Huang, Y . - Y . , and Abraham, W. C . (1986). Hippocampal long-term potentiation is induced by pairing single afferent volleys with intracellularly injected depolarizing currents, Ada Physiol. Scand. 126, 317-319.
748
Reprinted with permission from Neural Computation, Vol. l , p p . 123-132, 1989 © 1989 Massachusetts Institute of Technology
The Brain Binds Entities and Events by Multiregional Activation from Convergence Zones Antonio R. Damasio Department of Neurology, Division of Behavioral Neurology and Cognitive Neuroscience, University of Iowa College of Medicine, Iowa City, IA, USA The experience of reality, in both perception and recall, is spatially and temporally coherent and "in-register." Features are bound in entities, and entities are bound in events. The properties of these entities and events, however, are represented in many different regions of the brain that are widely separated. The degree of neural panellation is even greater when we consider that the perception of most entities and events also requires a motor interaction on the part of the perceiver (such as eye movements and hand movements) and often includes a recordable modification of the perceiver's somatic state. The question of how the brain achieves integration starting with the bits and pieces it has to work with, is the binding problem. Here we propose a new solution for this problem, at the level of neural systems that integrate functional regions of the telencephalon. 1 Introduction Data from cognitive psychology, neurophysiology, and neuroanatomy indicate unequivocally that the properties of objects and events that w e perceive through various sensory channels engage geographically separate sensory regions of the brain (Posner 1980; Van Essen and Maunsell 1983; Damasio 1985; Livingstone and Hubel 1988). The need to "bind" together the fragmentary representations of visual information has been noted by Treisman and Gelade (1980), Crick (1984), and others, but clearly the problem is a m u c h broader one and includes the need to integrate both the sensory and motor components in both perception and recall, at all scales and at all levels. This broader concept of binding is closer to that of Sejnowski (1986). T h e traditional and by n o w untenable solution to the binding problem has been that the components provided by different sensory portals end up being displayed together in so-called multimodal cortices, where the most detailed and integrated representations of reality are achieved. This intuitively reasonable view suggests that perception depends on a unidirectional process which provides a gradual refinement of signal extraction
749
1 2 4
along a cascade aimed cowards integrative cortices in anterior temporal and anterior frontal regions. Some of the most influential accounts for the neural basis of cognition in the post-war period as well as major discoveries of neurophysiology and neuroanatomy over the past two decades, have seemed compatible with this view. After all, anatomical projections do radiate from primary sensory cortices toward structures in the hippocampus and prefrontal cortices via a multi-stage sequence (Pandya and Kuypers 1969; Jones and Powell 1970; Nauta 1971; Van Hoesen 1982), and the farther away neurons are from primary sensory cortices, the larger their receptive fields become, and the less unimodal their responses are (Desimone and Ungerleider 1989). However, there are several lines of evidence on the basis of which this traditional solution can be rejected.
2 Experimental Evidence
.
Evidence from Experimental Neuroanatomy: The notion that integration of perceptual or recalled components depends on a single neural meeting ground calls for the identification of a neuroanatomical site that would receive projections from all neural regions involved in the processing of entities and events as they occur in experience. Despite considerable exploration no such region has yet been found. The anterior temporal cortices and the hippocampus do receive projections from multiple sensory areas, but not from motor regions (Van Hoesen 1982). The anterior frontal cortices, the most frequently mentioned candidates for ultimate integration, are even less suited for that role. The sensory and motor streams that reach them remain segregated in different regions (Goldman-Rakic 1988). In other words, there seems to be no structural foundation to support the intuition that temporal and spatial integration occur at a single site. Advances in experimental neuroanatomy have added a new element to neuroanatomical reasoning about this problem: at every stage of the chain of forward cortical projections, there exist prominent projections back to the originating sites. Moreover, the systems are just as rich in multi-stage, reciprocating feedback projections as they are in feedforward projections (Van Hoesen 1982; Van Essen 1985; Livingstone and Hubel 1987). The neuroanatomical networks revealed by these studies allow for both forward convergence of some parallel processing streams, and for the flow of signaling back to points of origin. In the proposal we will describe below, such networks operate as coherent phase-locked loops in which patterns of neural activity in "higher" areas can trigger, enhance, or suppress patterns of activity in "lower" areas. Evidence from Experimental Neuropsychology in Humans with Focal Cerebral Lesions: If temporal and frontal integrative cortices were the substrate for the integration of neural activity on which binding depends, the bilateral destruction of those cortices in humans should: (a) preclude the
750
125
Motor -& PremotorCortices
1
Frontal "integrative ' cortices
Figure 1: Fundamental divisions of human cerebral cortex depicted in a simplified diagram of the external and internal views of the left hemisphere. The motor and premotor cortices include cytoarchi tec tonic fields 4, 6, and 8. The early and intermediate sensory cortirces include the primary visual, auditory, and somatosensory regions (respectively fields 17, 41/42, and 3/1/2), and the surrounding association cortices (fields 18/19, 7, 39, 22, 40, 5). The temporal "integrative" cortices include fields 37, 36, 35, 20, 21, 38, and 28, i.e., neocortical as well as limbic and paralimbic areas. The frontal "integrative" cortices include fields 44, 45, 46, 9, 10, 11, 12, 13, and 25, i.e., prefrontal neocortices as well as limbic. p e r c e p t i o n o f r e a l i t y as a coherent m u l t i m o d a l experience a n d reduce experience t o disjointed, m o d a l tracks of sensory o r m o t o r processing; (b) reduce the i n t e g r a t i o n of even such m o d a l track processing; a n d (c) disable m e m o r y f o r a n y f o r m of past integrated experience a n d interfere w i t h a l l levels a n d types of memory. However, the results of bilateral des t r u c t i o n of the anterior t e m p o r a l lobes, as w e l l as bilateral destruction of p r e f r o n t a l cortices, falsify these predictions (see Fig. 1). Coherent perceptual experience is n o t altered b y bilateral damage t o the anterior t e m p o r a l regions, n o r does such damage d i s t u r b perceptual q u a l i t y (see C o r k i n 1984; Damasio et al. 1985; Damasio et al. 1987). O u r p a t i e n t Boswell is a case i n p o i n t . H i s extensive, bilateral damage i n
751
126
anterior temporal cortices and hippocampus, disables his memory for unique autobiographical events, but not his ability to perceive the world around in fully integrated fashion and to recall and recognize the entities and events that he encounters or participates in, at non-unique level. His binding ability breaks down at the level of unique events, when the integration of extremely complex combinatorial arrangement of entities is required. Bilateral lesions in prefrontal cortices, especially when restricted to the orbitofrontal sector, are also compatible with normal perception and even with normal memory for most entities and events except for those that pertain to the domain of social knowledge (Eslinger and Damasio 1985; Damasio and Tranel 1988). Finally, it is damage to certain sectors of sensory association cortices that can affect both the quality of some aspects of perception within the modality served by those cortices, and recognition and recall. Depending on precisely which region of visual cortex is affected, lesions in early visual association cortices can disrupt perception of shape, or color, or texture, or stereopsis, or spatial placement of the physical components of a stimulus (Damasio 1985; Damasio et al. 1989). A patient may lose the ability to perceive color and yet perceive shape, depth and motion normally. More importantly, damage within some sectors of modal association cortices can disturb recall and recognition of stimuli presented through that modality, even when basic perceptual processing is not compromised. For instance, patients may become unable to recognize familiar faces that they perceive flawlessly (although, intriguingly, they can discriminate familiar from unfamiliar faces at covert level; Tranel and Damasio 1985; 1988). The key point is that damage in a posterior and unimodal association cortex can disrupt recall and recognition at virtually every level of the binding chain, from the entity-categorical level to the event-unique level. It can preclude the kind of integrated experience usually attributed to the anterior cortices.
3 A N e w V i e w on the Binding Problem The evidence then indicates: (a) that substantial binding, relative to entities or parts thereof, occurs in unimodal cortices and can support recall and recognition at the level of categories; (b) that recall and recognition at category level, are generally not impaired by damage confined to anterior integrative cortices, i.e., knowledge recalled at categoric levels depends largely on posterior sensory cortices and interconnected motor cortices; (c) that recall and recognition of knowledge at the level of unique entities or events, requires both anterior and posterior sensory cortices, i.e., a more complex network is needed to map uniqueness; anterior integra-
752
127
five s t r u c t u r e s a l o n e a r e n o t sufficient to r e c o r d a n d r e c o n s t r u c t u n i q u e knowledge. T h e i m p l i c a t i o n is that the e a r l y a n d i n t e r m e d i a t e p o s t e r i o r s e n s o r y c o r t i c e s c o n t a i n f r a g m e n t a r y r e c o r d s of f e a t u r a l c o m p o n e n t s w h i c h c a n b e r e a c t i v a t e d , o n the b a s i s of a p p r o p r i a t e c o m b i n a t o r i a l a r r a n g e m e n t s (by f r a g m e n t a r y f e a t u r a l c o m p o n e n t s w e m e a n " p a r t s of entities," at a m u l t i p l i c i t y of s c a l e s , m o s t n o t a b l y at feature l e v e l , e.g., color, m o v e m e n t , t e x t u r e , s h a p e a n d p a r t s thereof). T h e y a l s o c o n t a i n r e c o r d s of the c o m b i n a t o r i a l a r r a n g e m e n t of f e a t u r e s that d e f i n e d entities ( " l o c a l " o r "entity" b i n d i n g ) , b u t d o n o t c o n t a i n r e c o r d s of the s p a t i a l a n d t e m p o r a l r e l a t i o n s h i p s a s s u m e d b y v a r i e d entities w i t h i n a n e v e n t ( " n o n - l o c a l " o r "event binding").
T h e latter r e c o r d s , the c o m p l e x c o m b i n a t o r i a l c o d e s n e e d e d
for e v e n t r e c a l l , a r e i n s c r i b e d i n a n t e r i o r c o r t i c e s . I n this p e r s p e c t i v e the p o s t e r i o r c o r t i c e s c o n t a i n t h e f r a g m e n t s w i t h w h i c h a n y e x p e r i e n c e of e n tities o r e v e n t s c a n p o t e n t i a l l y b e r e - e n a c t e d , b u t o n l y c o n t a i n the b i n d i n g m e c h a n i s m to r e - e n a c t k n o w l e d g e r e l a t i v e to entities.
Posterior cortices
r e q u i r e b i n d i n g m e c h a n i s m s i n a n t e r i o r s t r u c t u r e s i n o r d e r to g u i d e the p a t t e r n o f m u l t i r e g i o n a l a c t i v a t i o n s n e c e s s a r y to r e c o n s t i t u t e a n event. T h u s p o s t e r i o r c o r t i c e s c o n t a i n b o t h b a s i c f r a g m e n t s a n d local b i n d i n g r e c o r d s a n d a r e e s s e n t i a l for r e c r e a t i n g a n y p a s t e x p e r i e n c e .
Anterior
cortices contain non-local or event-binding records and are only crucial for r e c o n s t i t u t i o n of c o n t e x t u a l l y m o r e c o m p l e x
events.
P e r h a p s the most i m p o r t a n t distinction between this perspective a n d the t r a d i t i o n a l v i e w , is that h i g h e r - o r d e r a n t e r i o r c o r t i c e s are s e e n a s repositories
of c o m b i n a t o r i a l c o d e s for i n s c r i p t i o n s that lie
elsewhere
a n d c a n b e r e c o n s t r u c t e d e l s e w h e r e , r a t h e r t h a n b e i n g the s t o r a g e site for t h e m o r e r e f i n e d " m u l t i m o d a l " r e p r e s e n t a t i o n s of e x p e r i e n c e s .
Al-
t h o u g h anterior cortices receive m u l t i m o d a l projections w e conceptualize the r e c o r d s t h e y h a r b o r a s a m o d a l . If p a r t s of t h e r e p r e s e n t a t i o n of a n entity a r e d i s t r i b u t e d o v e r d i s t a n t r e g i o n s o f t h e b r a i n , t h e n m e c h a n i s m s m u s t b e a v a i l a b l e to b i n d together the f r a g m e n t s .
A p r o p o s a l for a n e w s o l u t i o n to t h e b i n d i n g p r o b l e m
( D a m a s i o 1989) is i l l u s t r a t e d i n f i g u r e 2 a n d p r e s e n t e d i n o u t l i n e a s follows: 1. T h e n e u r a l a c t i v i t y p r o m p t e d b y p e r c e i v i n g the v a r i o u s p h y s i c a l p r o p e r t i e s o f a n y entity, o c c u r s i n f r a g m e n t e d f a s h i o n a n d i n geog r a p h i c a l l y separate regions located i n early sensory cortices a n d in m o t o r cortices. S o - c a l l e d "integrative" cortices d o not contain s u c h fragmentary inscriptions. 2. T h e i n t e g r a t i o n of m u l t i p l e a s p e c t s of e x t e r n a l a n d i n t e r n a l r e a l i t y i n p e r c e p t u a l o r r e c a l l e d e x p e r i e n c e s , d e p e n d s o n the p h a s e - l o c k e d coa c t i v a t i o n of g e o g r a p h i c a l l y s e p a r a t e sites of n e u r a l a c t i v i t y w i t h i n the above m e n t i o n e d s e n s o r y a n d motor cortices, rather than on a t r a n s f e r a n d s p a t i a l i n t e g r a t i o n of different r e p r e s e n t a t i o n s w a r d s anterior h i g h e r - o r d e r cortices.
753
to-
C o n s c i o u s n e s s of t h o s e c o -
128
a c t i v a t i o n s d e p e n d s o n their b e i n g a t t e n d e d to, i.e., o n s i m u l t a n e o u s e n h a n c e m e n t of a p e r t i n e n t set of a c t i v i t y a g a i n s t b a c k g r o u n d activity. 3. T h e p a t t e r n s of n e u r a l a c t i v i t y that c o r r e s p o n d to d i s t i n c t p h y s i c a l p r o p e r t i e s of entities are r e c o r d e d i n the s a m e n e u r a l e n s e m b l e s
Figure 2: Simplified diagram of some aspects of the proposed neural architecture.
V, SS, and A depict early and intermediate sensory cortices in visual,
somatosensory, a n d auditory modalities. In each of those sensory sectors, separate functional regions are represented by open a n d filled dots.
Note feed-
forward projections (black lines) from those regions toward several orders of convergence zones ( C Z 1 , C Z 2 , C Z n ) , and note also feedback projections from each C Z level toward originating regions (red lines). H depicts hippocampal system, one of the structures where signals related to a large number of activity sites can converge.
Note outputs of H toward last station of feedforward
convergence zones ( C Z n ) and toward noncortical neural stations ( N O in basal forebrain, brain stem, and neurotransmitter nuclei. Feedforward and feedback pathways should not be seen as rigid channels. T h e y are conceived as facilitated lines which become active w h e n concurrent firing in early cortices or C Z s takes place. Furthermore, those pathways terminate over neuron ensembles, in distributed fashion, rather than on specific single neurons.
754
129
in w h i c h they o c c u r d u r i n g perception, but the combinatorial arr a n g e m e n t s ( b i n d i n g c o d e s ) that d e s c r i b e t h e i r p e r t i n e n t l i n k a g e s in entities a n d i n events (their spatial a n d t e m p o r a l coincidences), are stored in separate neural ensembles called convergence
zones.
4. C o n v e r g e n c e z o n e s t r i g g e r a n d s y n c h r o n i z e n e u r a l a c t i v i t y p a t t e r n s c o r r e s p o n d i n g to t o p o g r a p h i c a l l y o r g a n i z e d f r a g m e n t r e p r e s e n t a t i o n s o f p h y s i c a l s t r u c t u r e , t h a t w e r e p e r t i n e n t l y a s s o c i a t e d i n exp e r i e n c e , o n the b a s i s of s i m i l a r i t y , s p a t i a l p l a c e m e n t , t e m p o r a l seq u e n c e , o r t e m p o r a l c o i n c i d e n c e , o r c o m b i n a t i o n s thereof. T h e t r i g g e r i n g a n d s y n c h r o n i z a t i o n d e p e n d s on feedback projections from the c o n v e r g e n c e z o n e to m u l t i p l e c o r t i c a l r e g i o n s w h e r e f r a g m e n t records can be activated. 5. C o n v e r g e n c e z o n e s a r e l o c a t e d t h r o u g h o u t
the t e l e n c e p h a l o n ,
at
m u l t i p l e n e u r a l l e v e l s , i n a s s o c i a t i o n c o r t i c e s of different o r d e r s , limbic cortices a n d subcortical l i m b i c nuclei, a n d non-limbic subcortical n u c l e i s u c h as the b a s a l ganglia. 6. T h e g e o g r a p h i c l o c a t i o n o f c o n v e r g e n c e z o n e s for different
entities
v a r i e s a m o n g i n d i v i d u a l s b u t is n o t r a n d o m . It is c o n s t r a i n e d b y t h e s u b j e c t m a t t e r o f the r e c o r d e d m a t e r i a l (its d o m a i n ) , a n d b y c o n t e x t u a l c o m p l e x i t y of e v e n t s (the n u m b e r o f c o m p o n e n t
entities
t h a t i n t e r a c t i n a n e v e n t a n d t h e r e l a t i o n s they a d o p t ) , a n d b y the a n a t o m i c a l d e s i g n of t h e s y s t e m . C o n v e r g e n c e z o n e s that b i n d feat u r e s into entities a r e l o c a t e d e a r l i e r i n the p r o c e s s i n g s t r e a m s , a n d c o n v e r g e n c e z o n e s that b i n d entities i n t o p r o g r e s s i v e l y m o r e c o m p l e x e v e n t s a r e g r a d u a l l y p l a c e d m o r e a n t e r i o r l y i n the p r o c e s s i n g streams. 7. T h e r e p r e s e n t a t i o n s i n s c r i b e d i n the a b o v e a r c h i t e c t u r e , b o t h t h o s e that preserve t o p o g r a p h i c / t o p o l o g i c
r e l a t i o n s h i p s a n d t h o s e that
c o d e for c o m b i n a t o r i a l a r r a n g e m e n t s , a r e c o m m i t t e d to p o p u l a t i o n s of n e u r o n e n s e m b l e s a n d t h e i r s y n a p s e s , i n d i s t r i b u t e d f o r m . 8. T h e
c o - o c c u r r e n c e of a c t i v i t i e s i n m u l t i p l e sites that is n e c e s s a r y
for b i n d i n g c o n j u n c t i o n s , is a c h i e v e d b y r e c u r r e n t f e e d b a c k i n t e r a c tions. T h u s , w e p r o p o s e that the p r o c e s s i n g d o e s not p r o c e e d i n a s i n g l e direction but rather t h r o u g h temporally coherent phase-locking a m o n g s t m u l t i p l e r e g i o n s . A l t h o u g h the c o n v e r g e n c e z o n e s that r e a l i z e the m o r e e n c o m p a s s i n g i n t e g r a t i o n a r e p l a c e d m o r e a n t e r i o r l y , it is a c t i v i t y i n the m o r e p o s t e r i o r c o r t i c a l r e g i o n s that i s m o r e d i r e c t l y r e l a t e d to c o n s c i o u s experience. B y m e a n s of feedback, convergence zones repeatedly return processi n g to e a r l i e r c o r t i c e s w h e r e a c t i v i t y c a n p r o c e e d a g a i n t o w a r d s the s a m e or other convergence zones. Integration takes place w h e n activations occ u r w i t h i n the s a m e t i m e w i n d o w , i n e a r l i e r c o r t i c e s . T h e r e is n o n e e d
755
130
to postulate a "final" and single integration area. This model accommodates the segregation of neural processing streams that neuroanatomical and neurophysiological data continue to reveal so consistently, and is compatible with the increase in receptive fields of neurons that occurs in cerebral cortex, in the posterior-anterior direction. It accords with the proposal that fewer and fewer neurons placed anteriorly in the system are projected on by structures upstream and thus subtend a broader compass of feed-forwarding regions. Broad receptive field neurons serve as pivots for reciprocating feedback projections rather than as accumulators of the knowledge inscribed at earlier levels. They are intermediaries in a continuous process that centers on early cortices. 4 Conclusions
The problem of how the brain copes with the fragmentary representations of information is central to our understanding of brain function. It is not enough for the brain to analyze the world into its components parts: the brain must bind together those parts that make whole entities and events, both for recognition and recall. Consciousness must necessarily be based on the mechanisms that perform the binding. The hypothesis suggested here is that the binding occurs in multiple regions that are linked together through activation zones; that these regions communicate through feedback pathways to earlier stages of cortical processing where the parts are represented; and that the neural correlates of consciousness should be sought in the phase-locked signals that are used to communicate between these activation zones. Several questions are raised by this new view. For instance, what is the precise nature of the feedback signals that provide temporally coherent phase-locking among multiple regions? H o w large are the convergence zones in different parts of the brain? How are the decisions made to store an aspect of experience in a particular zone? There are several possible approaches to test the hypothesis proposed here. One approach is to develop new techniques for recording from many neurons simultaneously in communicating brain regions. Another relies on neuropsychological experiments in neurological patients with small focal lesions in key areas of putative networks dedicated to specific cognitive processes. Finally, modeling studies should illuminate the collective properties of convergence zones and provide us with the intuition we need to sharpen our questions.
Acknowledgments
Supported by N I N C D S Grant P O l NS19632.
756
131
References
.
C o r k i n , S. 1984. L a s t i n g Consequences of Bilateral M e d i a l T e m p o r a l Lobectomy: C l i n i c a l C o u r s e a n d E x p e r i m e n t a l F i n d i n g s i n H M . Seminars in Neurology 4, 249-259. C r i c k , F. 1984. Function of the T h a l a m i c Reticular Complex: T h e Searchlight H y p o t h e s i s . Proc. Natl. Acad. Sci. USA 81, 4586-4590. D a m a s i o , A . 1989. Multiregional Retroactivation: A Systems Level M o d e l for S o m e N e u r a l Substrates of Cognition. Cognition, i n press. . 1985. D i s o r d e r s of C o m p l e x V i s u a l Processing.
In: Principles of Be-
havioral Neurology, e d . M . M . M e s u l a m , C o n t e m p o r a r y Neurology Series, 259-288. Philadelphia: F A . D a v i s . D a m a s i o , A . , P. Eslinger, H . D a m a s i o , G . W . V a n Hoesen, a n d S. C o r n e l l . 1985. M u l t i m o d a l A m n e s i c S y n d r o m e F o l l o w i n g Bilateral T e m p o r a l a n d Basal Forebrain Damage. Archives of Neurology 42, 252-259. D a m a s i o , A . , H . D a m a s i o , D . Tranel, K . Welsh, a n d J . Brandt. 1987. A d d i t i o n a l N e u r a l a n d Cognitive E v i d e n c e i n Patient D R B . Society for Neuroscience 13, 1452. D a m a s i o , A . a n d D . Tranel. 1988. Domain-specific A m n e s i a for Social K n o w l edge. Society for Neuroscience 14, 1289. D a m a s i o , A . R . , H . D a m a s i o , a n d D . Tranel. 1989. Impairments of Visual Recognition as C l u e s to the Processes of Memory. In: Signal a n d Sense: L o c a l a n d G l o b a l O r d e r i n Perceptual M a p s , eds. G . E d e l m a n , E . G a l l , and M . C o w a n , Neuroscience Institute Monograph. Wiley a n d Sons. D e s i m o n e , R. a n d L , Ungerleider. 1989. N e u r a l M e c h a n i s m s of Visual Processing i n M o n k e y s .
In: H a n d b o o k of Neuropsychology,
Disorders of V i s u a l
Processing, e d . A . D a m a s i o , i n press. Eslinger, P. a n d A . D a m a s i o .
1985. Severe Disturbance of H i g h e r Cognition
after Bilateral Frontal L o b e Ablation. Neurology 35, 1731-1741. G o l d m a n - R a k i c , P S . 1988. Topography of Cognition: Parallel Distributed Netw o r k s i n Primate Association Cortex. In: A n n u a l R e v i e w of Neuroscience H , A n n u a l R e v i e w s Inc., Palo Alto, C A , 137-156. Jones, E . G . a n d T.P.S. P o w e l l . 1970. A n A n a t o m i c a l S t u d y of C o n v e r g i n g Sens o r y P a t h w a y s w i t h i n the Cerebral Cortex of the Monkey. Brain 93, 793-820. L i v i n g s t o n e , M , a n d D . H u b e l . 1988. Segregation of F o r m , Color, Movement, and Depth: Anatomy, Physiology, a n d Perception. Science 240, 740-749. . 1987. Connections between L a y e r 4B of A r e a 17 and T h i c k C y t o c h r o m e O x i d a s e Stripes of A r e a 18 in the Squirrel Monkey, journal of Neuroscience 7, 3371-3377. N a u t a , W . J . H . 1971. T h e Problem of the Frontal Lobe: A Reinterpretation. /. Psychiat. Res. 8, 167-187. P a n d y a , D . N . a n d H . G . J . M . K u y p e r s . 1969. Cortico-cortical Connections i n the Rhesus Monkey. Brain Res. 13, 13-36. Posner, M . I . 1980. O r i e n t i n g of Attention.
Quarterly journal of Experimental
Psychology 32, 3-25. S e j n o w s k i , T.J. 1986. O p e n Questions about C o m p u t a t i o n i n Cerebral Cortex. In:
757
132
Parallel Distributed Processing, eds. J.L. M c C l e l l a n d a n d D . E . Rummelhart, 372-389. Cambridge: M I T Press. Tranel, D . a n d A . Damasio. 1988. Nonconscious Face Recognition i n Patients w i t h Face Agnosia. Behavioral Brain Research 30, 235-249. —
. 1985. K n o w l e d g e without Awareness: A n A u t o n o m i c Index of Facial Recognition b y Prosopagnosics.
Science 228:21, 1453-1454.
Treisman, A . a n d G . Gelade. 1980. A Feature-integTation T h e o r y of Attention. Cognitive Neuropsychology 12, 97-136. V a n Essen, D . C . 1985. Functional CTganization of Primate V i s u a l Cortex. In: Cerebral Cortex, eds. A . Peters a n d E . G . Jones, 259-329. P l e n u m Publishing. V a n Essen, D . C . a n d J.H.R. Maunsell.
1983. Hierarchical Organization and
Functional Streams in the Visual Cortex. Trends in Neuroscience 6, 370-375. V a n Hoesen, G . W . 1982. T h e Primate Parahippocampal G y r u s : N e w Insights Regarding its Cortical Connections. Trends in Neurosciences 5, 345-350.
Received 18 November; accepted 25 November 1988.
758
Reprinlcd with permiuion from Bioh/gkol Cybernetic v ., 65. pp. r.i-u= 1991 ©1991 Springer-verlag
On the computational architecture of the neocortex I . The role of the thai a mo-cortical loop D. Mumford Mathemaiki Department. Harvard Uriiveriin. I Oxford Slreel. Cambridgi, MA 0313!. USA Rtcaivcd February 9. 1991 Abstract. This paper proposes thai each area of the cortex carries on its calculations wilh Ihe aclive partictpaiion of a nucleus in the thalamus wilh which it is reciprocally and topographically connected. Each cortical area is responsible for maintaining and updating the organism's knowledge of a specific aspect of the world, ranging from low level raw data 10 high level abstract representations, and involving interpreting stimuli and generating actions. In doing this, it wilt draw on multiple sources of expertise, learned from experience, creating multiple, often conflicting, hypotheses which are integrated by the action of the thalamic neurons and then sent back to the standard input layer of the cortex. Thus this nucleus plays the role of an 'active blackboard' on which ihe current best reconstruction of some aspect of ihe world is always displayed. Evidence for this theory is reviewed and experimental lesis are proposed. A sequel to this paper will discuss the cortico-cortical loops and propose quite different computational roles for them.
1 Inrrodncrion When one attempts to theorize about the functioning of the human brain, the task seems nearly impossible because of the extraordinary complexity of the brain. Anatomically, the brain is divided inlo hundreds, even thousands of areas and nuclei and structures, connected in a totally bewildering way, while physiologically more and more chemicals, exchanged by neurons, are being discovered which cause more and more types of effects of one neuron on another. Likewise, if you start by analyzing the computational task that the brain faces, using Ihe tools of either pattern recognition, artificial intelligence or control theory, only Ihe mosi simplified versions of these tasks seem tractable. In the selling of the real world, sensory data and motor requirements are so complex and need to be robust in the fact of so much noise, so many unforeseen disturbances, as 10 be totally beyond present techniques.
759
The idea of the present two-part paper is that, at Ihe appropriate level of analysis, Ihere are certain uniformities in Ihe structure of the brain that suggest that some simple general principles of organization must be at work, [f this is Ihe case, looking al Ihe tasks performed by ihe brain from a computational perspective, it may be possible to link the structures observed in the brain with elements in a theoretical analysis of what is needed to perform ihese lasks. The first part of this paper will pul forward a proposal for the role of the thalamus based specifically on the existence of pathways belween Ihe corlex and the thalamus which are at least roughly topographic, i.e. preserving the two-dimensional layout of the cortical sheel. and inverse lo each olher. The second part of Ihe paper will make proposals for Ihe computational significance of the reciprocal cortico-cortical pathways and its relation lo [he pyramidal cell populations in different layers of the cortex and [he fast oscillations recently observed in the cortex. The uniformity which 1 have in mind is (he uniformity of the neocortex of mammals. In essentially all species of mammal, including the very primitive opossum, the neocortex has an extremely similar structure throughout il has six layers, a small number of cell lypes. one of which. Ihe pyramidal cell, accounts for over half of all cells and a standard pattern of connectivity, locally, globally within the cortex and subcortical Iy. Phylogenetically. this structure has nol been changed in Ihe evolution of mammals (except for being simplified in some orders). The major expansion of 'association' cortex in the primate order has nol involved revision of [his basic plan. but. apparently, simply replicating il over ever larger areas. This suggests very strongly lhal this structure embodies a basic computational module so versatile that it can be booked together in ever larger configurations and still function, with ever increasing subtlety, to both analyze sensory input and organize motor actions. Even in producing the mosi remarkable achievemenl of [he brain - language - ihe areas of [he brain involved have used the identical structure. The fact lhal this structure
136
is present in much simpler animals, moreover, suggests lhal il may nol be [hat hard to understand and lhal its mode of operation may be fully revealed in quite simple tasks. On the other hand, these structures are much more specific, wilh their own characteristic architecture, lhan I hose normally studied from a theoretical perspective under the name 'neural nets'. This suggests ihe possibilily of studying the architecture embodied in ihese specific structures, looking for their computational significance. Since this paper deals wilh a proposal for applying computational ideas lo biological structures, we have tried lo present the ideas so as lo be clear bolh lo computer scientists and biologists. In order to do Ihis, il has been necessary lo include a considerable amount of basic anatomy for Ihe sake of the former and of basic computer science for the sake of the latter. I found with preliminary versions of ihe paper that when this background was left out. there were frequently misunderstandings and confusions and for this reason. I feel it is essential to develop my ideas al this length. I wanl to lhank Francis Crick, Terry Deacon, Stephen Kosslyn, Adam Mamelak. Ken Nakayama and Sieve Zucker for critical readings of various drafts of this paper lhat helped me immensely in refining and clarifying my ideas. In particular, while preparing this paper. 1 learned of the work of Erich Harth (19831, who made proposals in a similar direction for the role of the thalamus.
2 The connections of the cortex and the thalamus: a review In order to put. our proposals for the role of the Lhalamus in perspective, 1 need to lay oui Ihe basic facts about the structure of the cortex and its connections with The lhalamus Everything in this section is standard neuroanatomy, but il is included here so lhat readers with other backgrounds can follow oui ideas. The neocortex has an area of about 200.000 sq.mm. in humans, a thickness of 2-3 mm., and a neuron density around 100,000/sqmm;' Over half of these cells are so-called pyramidal cells, characterized by the fact that at least one branch of their axons projects lo distant, e.g. a centimeter or more, targets. The neocortex has a uniform structure with 6 layers, characterized by their cell populations, which [ will discuss in greater detail below. There are local variations in the thickness and prominence or the layers." but in general the same structure is there. {Reproduced in Fig. I is a composite photo of Ihe six layers in three different stains.) The r
1
There is a wide: discrepancy in minute*, ranging from 12,000/ sq.mm La lC0,0OOi M[.mm.. bur l00.Q00/sq.mm. resulting in 30 billion corneal cells. H o n io be a frequently died figure? e.g. Rocbel ei al. ,1980,. after allowing for 1E% shrinkage in each dimension, or Chcmiuk 11990]. Note lhal Ihe primary visual area is an exception wilh at least twice J S many tells per sq.mm * e.g. the claboralLun o f layer IV in primary sensory areas and ils sparsencss in ihe primary motor area
Fig. L The ft cortical layers in Golgi, Nissl and myelin stains (from Brodal 1961)
cortex as a whole includes Iwo more primitive parts, the paleocortex and ihe archicortex. with 4% of the lotal conical area in man (Blinkov and Glazer I96B, p 331). This will play essentially no role in this paper so that I usually refer lo the neocortex simply as cortex. These more primitive parts have a similar but simplified pattern with a less elaborate pattern of layers. The cortex of every mammal seems to be divided into areas, each with a specialized role. The original identification of Ihese areas was based on Liny differences of cell types and cell distributions and led to maps of conical areas due to Brodmann and oihers. There are, of course, species differences, so thai the primitive mammals have relatively few areas and primates more, bul the general map and oflen many of ils details are closely homologous for all species studied. More recently, the possibilily of tracing pathways in cortex very accurately using chemicals which move bolh forward and backward through axons has modified and frequently subdivided Brodmann's areas (e.g. ihe third visual area. Brodmann's area 19, turned oui to he composed oT more lhan one areaj. bul (he picture of a map-like division of ihe surface of the brain into independent computational modules with specific interconnections has been repeatedly confirmed. For a very recent and comprehensive review of the areas known in ihe Macaque monkey, see {Fellemann and Van Essen 1991).
l
r
All input to the cortex, except for ihe olfactory sense, comes to il via the thalamus, which sits al the top of the brain stem in two parts, one in the middle of each cerebral hemisphere. It is nol easy to expose because il is totally surrounded by the while mailer of
760
l?7
and ihe remaining neurons al] project directly to the cortex with no collaterals (with one exception: see discussion of RE thalamus below). Thus, except for ihe RE nucleus, the nuclei in Ihe lhalamus are not directly
connected to each other. Where does the thalamus get its input? Some nuclei in the thalamus are the principal route for sensory signals and 'relay' these up to the primary sensory areas of the cortex, e.g. LGN to VJ And the nucleus VLp transmits Ihe moior related signals from the cerebellum to area 4 in (he cortex. Other nuclei get more elaborated sensory, motor or emotional signals from further subcortical structures - the superior colliculus. globus pallidus. amygdala, mammilary nuclei, etc. - but, by and large, their largest input is from the cortex itself, via the reciprocal cortico-thalamic pathways described above. Roughly speaking, it seems as though each area of the mammalian cortex receives input, via the thalamus, from (hat sub-cortical structure which was per¬ forming similar cognitive functions in more primitive animals. For instance, analysis of visual input and integration of visual, auditory and tactile information is carried out in the various layers of the superior colliculus (or teclum) in primitive animals. The superior colliculus, in mammals, projects to ihe pulvinar complex in ihe thalamus and the nee lo the association areas of the occipital, parietal and temporal lobes, which carry out the same functions. Bul, in the evolution of the primate line, the visual inpul (o the cortex via the collicular-pulvinar pa(h plays a smaller and smaller role compared to (he direct pathway from the retina to the LGN to V I . For instance, the strength of (his secondary visual pathway can be assessed by considering (he degree of blindness exhibited by animals in which VI has been destroyed, so that they must rely wholly on Ihis secondary palhway. Cats are not badly impaired by such a Joss, monkeys much more so and humans lose all their sigh( except for a peculiar guessing skill known as 'blindsight'.* The outpul of (he cortex is more complex. As stated above, every part of the cortex is talking to its corresponding part of the lhalamus Tn addition, ihe principal motor outpul, ihe pyramidal tract, bypasses the thalamus and goes directly from area 4 to the spinal cord, resulting in an extremely fast command sysiem. Area 4, in other words, has two outputs, the pyramidal tract and the posterior ventral lateral nucleus, VLp, of the thalamus with which it is reciprocally connected. Another output goes from vision-related frontal areas to a subcortical structure, the superior colliculus, and appears to be a motor outpul specifically for eye movements. A Ihird group of output pathways goes from many cortical areas to the subcortical structures called r
5
H g . 2. The location of the thalamus within ihe cone* (from Lurid 1969]
afferent and efferent axons, bul it is shaped roughly like a pair of small eggs, side by side (see Fig. 2). It is composed of a set or something like fifty nuclei (not all clearly marked) Each part of the cortex is reciprocally connected in a dense, continuous fashion with some nucleus in the thalamus. Two examples wilt be repeatedly referred to in this paper. The first is the primary visual area of The cortex V I (Brodmann's area 17) which is reciprocally connected To the lateral geniculate nucleus, or LGN. in the thalamus. The second is the primary motor area, Brodmann's area 4, which is reciprocally connected to the posterior ventral lateral nucleus, or VLp, in the thalamus. Tt appears in the cases which have been closely studied that the connection is set-up by dividing up the thalamic nucleus into parallel columns (i.e. volumes extended in one dimension along some curve, but of small extent in the two dimensions perpendicular to this curve), each of which is connected to a column of cortical tissue cutting vertically across the 6 layers Geometrically, it is as if the thalamus were an elaborate 7th layer of the cortex, with a long (hence slower) neuronal loop tieing it to the cortex proper The loop is made by the thalamus sending axons up (o the corlex where they synapse mainly in layer IV or the deeper part of layer I I I , and receiving axons originating in pyramidal cells in layer VI or the deeper part of layer V of the corlex (Steriade and Llinas 1988). The thalamus has a small population of inhibitory local interneurons faboul 25%. cf. Steriade and Llinas 1988, p. 659) 1
4
s
Far instance, the outpul nucleus of the globus pallidus. the internal segment, is estimated to conlain merely ITOJJOO neurons (Shepherd 19W)
* Il seems reasonable to conjecture that all the subcorneal inputs to the thalamus play an jmporlanl role in development in providing the initial seed that starts each area of (he corte* moving towards its ultimate cognitive role, but that many ol" thcie inputs are not essenltal for the cognitive functions of Ihe cortex in an adult
1
When wc talk of ihe lhalamus, wc shall always mean the dorsal thalamus, which is ils largest part * See S K I J , 6 J in (Jones I 9 B S ) and. especially Fig 3-20
761
13B
connected to multiple cortical areas and vice versa. Whether or not a single thalamic nucleus is ever connected to several cortical areas in a specific way, i.e. with the thalamus synapsing in the middle layers IV and III (deeper part) of the cortex, does not seem to have been clearly settled. The simplest hypothesis is that the specific projections set up reciprocal maps between the whole of Ihe cortex and the specific nuclei of the thalamus which are roughly one-to-one in each direction. Alternately, each nucleus in the thalamus may communicate to one or more cortical areas via specific projections (see Graybiel and Berson 1931, for such a view).* 7
output
outpul t e
It
•l--;lln
• 11 H n n i
Limbic s r ' r -
'>.:n;-.
Fig. J , Simplified schematic of cortical connections
the basal ganglia. These seem to be concerned wilh initiating complex actions. Finally the archicortex has a special output path to some subcortical structures concerned with emotional and motivational states. The picture which I have sketched is depicted schematically in Fig 3, which may help the reader to put everything together. Before discussing the role of the lhalamus, two complications should he mentioned. As mentioned above, each cortical area is richly and continuously connected to a corresponding nucleus in the thalamus. The first complication is that some nuclei in the thalamus have no specific connections to the cortex, but only so-called diffuse or non-specific connections synapsing over large portions of the cortex (Jones 1985). These seem to play some global regulatory role. Moreover there are also diffuse pathways which go between nuclei of the thalamus which are already specifically connected to one area of the cortex, but which connect it to another area of cortex in a more 'diffuse', less point-topoint way. In addition to being diffuse, they have a different synaptic pattern: they are set up by pyramidal cells in layer V of the cortex instead of VI and are returned by thalamic neurons synapsing in layer I instead of layers III and IV (Jones 1985). For example, the primary visual area VI has its specific connection with the LGN, but it is also connected diffusely to at least one nucleus in the pulvinar area of the thalamus (the layers involved in these connections are described in Weller and Kaas 1981). r
Because of this second type of connection, it usually appears as though each nucleus in the thalamus is
762
The second complication is Lhat all pathways belween Ihe cortex and thalamus pass through a thin layer of cells on the surface of the thalamus known as the reticular complex of the thalamus, or R E thalamus. There the pathways in both directions excite RE cells, which in (um send inhibilory axons both to each other and back to the thalamus to the area of origin of ihe pathway. Although the R E neurons are inhibitory, experimental evidence (Steriade et al. 1986) shows that peaks of activity in a pari of the R E thalamus occur al the same time as peaks of activity in the corresponding nuclei of the thalamus proper. For this reason, the mechanism by which the R E lhalamus and the lhalamus proper interact is not clear (Steriade and Llinas 1938. p. 712; Crick 1984, p. 4588; Sherman and Koch 1936, p. 12). In any case, as Crick says (Crick 1984, p. 4537): "If the Lhalamus is ibe gateway lo the cortex, the reticular complex might be described as the guardian of the gateway.' 1
3 The thalamus as a window an the world What, then, is the function of Ihe thalamus? Originally it was Lhoughi to be merely a passive relayer of sensory signals to the cortex proper. The small number of
* There is one well-known exception lo this hypothesis: in small mammals, the somalo-sensory and molor areas of rhe cortex can Overlap or even coincide. Then iwo quite separate parts of the Thalamus, VP and V L , project to ihe middle layers of overlaping parts of cortex. Ii La nol known whether this is an isolated exception, or indicates a frequcni pattern ' Several problems complicate this issue. One is that when very precise tracer experiments are carried oui. it sometimes seems rhat the part of the thalamus projecting to a particular cortical area does not exactly coincide with a thalamic nucleus, and may even cross the boundary between two nuclei [Bender 1981, p. tTi% Moreover, many papers report on connections wilhoul specifying ihe Laminar pattern of cortical synapses. The real issue, it seems lo me, is nol whether the cortical areas and ihe thalamic nuclei correspond one-to-one, because further studies may suggest subdividing both areas and nuclei, but how topographic are the specific, reciprocal projections. Jn older words, one seeks corresponding parts of the cortex and thalamus belween which ihese projections give a continuous map in both directions between the cortical surface and a set of 'rods' or 'columns.' In Ihe lhalamus (see Jones Sect- 3.6.3). Il may tum out that, at leas! in primates, both the cartel and The lhalamus are divisible into pieces such lhal ihe specific projections set up a one-to-one correspondence between ihem consisting of such topographic projections, Or Ihis may fail In various ways
I3S
intemeurons in ihe thalamus. Ihe total lack of connections between nuclei within the thalamus or of any intra-thalamic axonal collaterals and the analysis of single-cell recordings all suggest thai the thalamus does little or no computation by itself. But if it is merely a relay station, a) why do even association areas of the cortex receive thalamic input and b) why does the cortex reciprocate with a massive projection of fibres back to the thalamus? Both of these are biologically very expensive and even if they were built because of some phylogenetic quirk, ihey would decrease in size through selection if they weren't essential. Stepping back from details for a minute, one can argue like this: phylogenetically. the association areas of cortex (those not closely connected lo input or output circuits) developed by apparently replicating the structures and functionality inherited from the more primitive areas out of which more primitive brains were made. 1 want to ignore the motor end of the system for the time being, and concentrate on the sensory end. The original sensory cortex clearly acted as some kind of pattern analyzer taking its input from the thalamus, via layer IV. Assuming that evolution did not modify this plan, this suggests that other corlex also analyzes in some way the dala presented to il in layer rv. If so, then the data in the corresponding nucleus of the thalamus will be lhat area of cortex's view of the world: it will carry a signal which will be sent to layer IV of cortex and analyzed as though this was the signal that some new sophisticated sense could deliver. The nuclei of the thalamus, from this view, are pseudo-sense organs with different views of ihe world, lo be analyzed by the corlex. Each area of cortex is like a homunculus which has a certain narrow view of the world, in which it tries to remember patterns, recognizing familiar ones and lumping similar ones into categories. In the 'higher' sensory areas this would mean that the thalamus sends them not the raw sensory data but a processed version of the input in which noise and irrelevant stuff have been dropped, and the interesting features are marked as such. For instance, in vision, a higher level representation mighl record a code for 'grass' in place of a whole area of intricately textured detail, producing a kind of cartoon version of the stimulus,' The psychological experiments of Bransford et al. (1972), which demonstrate lhal what we remember about a sentence is often whal we thought or assumed was there, rather than whal was really there, are consistent with the proposal that higher areas of the brain process a kind of ralional reconstruction of the world rather than the raw data. Still higher sensory areas, especially multi-modal areas integrating data from many senses, should process quite abstract data structures as their view of the world. The sort of structure I have in mind is a sort of geometrical net, with nodes corresponding to various
5
By The melaphor of cartoon, we mean a data slmclure which ia no longer a pixel-by-pixel record, oui which is simplified Lo a list ot areas, annotated by Iheir features and boundaries
763
objects or parts of objects, and with links expressing ihe geometrical relationship between them (e.g. 'to Ihe left of. 'a pari of). This sort of structure was first proposed by Minsky (1975) and was elaborated by Win-
ston (1975) and Marr (19S2) (his so-called 3-D model) and has been investigated from a 'neural net' point of
view in (Mjoloess et al. 1988). 4 Active blackboards But why, then, should there be a recurrent pathway from the cortex to the thalamus? For higher areas, this pathway seems to be the principal way to excile the corresponding nuclei in ihe thalamus. The cortical area, then, receives its primary, layer IV input, its 'view' of the world as we have argued, from a thalamic nucleus which it is exciting. Note lhat theserecurrentpathways were apparently copied from those present in the primary sensory pathways of simpler brains. Thus the visual signal goes from the retina to the lateral geniculate nucleus, or LGN. of the lhalamus, and from ihere to Ihe corlex. But the LGN VI pathway is returned by an equally massive projection VI =• LGN which is certainly not needed for supplying the visual signal, is there some computational device which not only /unctions as Input to be read by the computer but on which the computer also writesl
A key idea in early Al work on speech perception (the CMU work on HEARSAY, cf. Erman et al. I9B0) as well as the psychological model embodied in 'Pandemonium' (see Selfridge 1959 or Lindsey and Norman 1977, p. 259), is that of Ihe blackboard. When several sources of expertise, e.g. several different constraints, must be broughl to bear on a problem, it is natural to Lry to carry out the computation in parallel, in independent streams, one devoted to working out the consequences of each source of expertise. These modules must, however, coordinate their work and the simplest way lo do this is to have a common blackboard visible to each module, on which they write from lime to lime their suggestions or conclusions. Similarly, if some constraint or algorithm must be applied multiple limes, it is natural to keep the running result on a blackboard, and the computational module must simply keep checking the blackboard and make small modifications to implement local constraints, or repeal some algorithm until satisfied with the result My proposal is that the thalamus is something like a blackboard. To use the thalamus as a blackboard means that the corlex must write on as well as read from this blackboard. Thus the thalamocortical fibers convey to the cortex the current picture of those aspects of the world with which thai area of Ihe cortex is concerned, distributing this dala via their axonal arborizations locally in the corlex. The cortico-thalamic fibers convey to the lhalamus proposed additions and revisions to this picture arrived at by many compulations carried out in the cortex, which are integrated in ihe thalamus via the dendritic arbors of Ihe thalamic neurons.
140 Think of the corlex as containing multiple experts wilh deep understanding of specific patterns and constraints usually present in the world: each expert makes guesses based on its knowledge and. while many of these guesses are compatible and presumably correct, some contradict others and decisions belween them musl be made. It is these decisions which i suggest are made by a kind of voting, taking place in the summation of stimuli in ihe dendrites of the thalamic cells. The need and the techniques for such 'data fusion' in the case of vision, have been extensively studied in the recent monograph of Clark and Yuille (1990). and I am proposing that ihe thalamus implements something like the algorithms that they propose. Note that the lhalamus not only integrates ils multiple cortical inputs with each other, but also with whatever sub-cortical input, like sensory data, this nucleus receives. 10
Bul Ihere are several ways in which the blackboard metaphor may be misleading. For one thing, a blackboard in a computer or in a Professor's office is a passive structure on which you merely write for communication: I have proposed that the thalamus plays an active role in synthesizing the results of calculations by various expert paitem recognizing modules in the cortex. Another major difference is that ihe computer's and the Professor's blackboard will slore ideas indefinitely until erased. But the brain is a volatile computational structure, always reacting to new stimuli and its blackboards would get cluttered and unreadable unless they erased themselves. In other words, the current calculations of the brain must usually be completed within tens or hundreds of milliseconds or Ihey become irrelevant, and the blackboard for such work must be actively refreshed by the senses or by cortical stimulus or it will fade. Thus the thalamus does nol sustain its activity by itself, but can only project back to ihe cortex ils integration of Ihe data being sent down lo il right now. When the brain wants Lo Luck some idea away for anything from minutes to years, it would not he a good idea to use these thalamic blackboards which are continuously bombarded wilh new ideas. These memory functions are accomplished by a different route and apparently require a complex interaction with nonneocortical areas: the hippocampus and the entorhinal cortex. Because of these differences, it seems better to call the thalamus an active blackboard, i.e. a blackboard which is volatile and continually presents tbe latest ideas, synthesized from multiple cortical sources. Although thalamic activity normally disappears as soon as the neurons fire, there are indications that the thalamus has some mechanisms for maintaining traces of its activity over something like 100ms. This idea comes from an analysis of the effect of calcium channels and has been linked with the possibility that the RE lhalamus may play a major role in keeping the atten-
tion of an area of cortex focussed on some complex of ideas on a shorter time scale (Crick I9S4, cf. Sect. 7 below). Tn other words, the R E thalamus might have a role nol merely in gating but in sustaining cortical attention. The mechanisms for such an effect are still very speculative. In Ihe context of Ihe blackboard metaphor, one can differentiate the role of Ihe specific and the diffuse projections of the lhalamus on the cortex. The specific projections are the ones ['ve been talking about so far: Ihe link between each computational area of the corlex and the aclive blackboard which it reads and writes on. On Ihe other hand, the diffuse projections from nuclei wilh specific connections to one cortical area can inform other areas of the cortex working on related sensory problems of what is the current hypothesis on this blackboard. For instance, blackboards in the pulvinar complex of the thalamus containing the optical flow field (data on the movemenl of the visual stimulus across the relina, thought to be computed in visual area MT) should be visible to areas concerned with figure/ ground separation (possibly areas V2 and V4), so that motion clues can be used lo distinguishfigureand ground. This coordination might also be achieved by direct cortico-cortical pathways, so the brain seems lo have Iwo paths for coordinating different areas with overlapping concerns: through diffuse connections to Ihe same nucleus of Ihe thalamus or through corticocortical pathways. Is the lhalamus big enough to play such a role? An estimate which is often cited puts the ratio by weight or volume of Ihe thalamus lo the cortex at 2%. I am suggesting that Ihe cortex contains multiple independent 'experts' which analyze different aspects of each area's data and that their results are merely inLegraLed in the thalamus. Moreover, 1 will propose in the second part of this paper that one of the computational burdens of the corlex is thai it must translate its data into forms readable to both higher and lower areas, while the thalamus need only store the data in the coded form usable to the given area. Thus a size ratio of 50:1 doesn't seem unreasonable for such a function. lr
S HaruYs theory Ideas of the kind expressed in Seel. 4 can be found in the work of Luria (1969) who drew attention strongly to the parallel structures in the cortex and the thalamus and reciprocal connections between the Iwo which occur on every level. But he did not invoke the specific computational metaphor of a blackboard. As far as I know, the only person lo do so is Erich Harth. who developed that he called the Alopex theory of the interaction belween Ihe LGN, the visual area VI and higher visual areas. Tn his popular book "Windows on the Mind" [Harlh 1983). he expresses his iheory like this:
1 9
These calculations may also involve rhe interneurons in rhe thalamus, e.g. using microciTcuits involving 3-way dendro-deudritic synapses with Ihese intemeurons. Because these inremeurons are inhibilory. [hey also allow cortical cells to cast negative votes, i e . l a Inhibil some thalamic cells
11
e.g. 10 cubic cm. for the lhalamus 1 bilaterally > Lo 500 cubic cm. for Ihe grey matter of Ihe cortex, or r*r> roughly 2 million cells per nucleus in the lhalamus lo 100 million in an average cortical area
764
141 "Recall thai ihe part of thalamus that is concerned with vision, the LGN, preserves some of ihe character of the retina: activity is distributed over sheets of neurons which mirror the pattern of tight failing on the retina. It is possible that canicofugal messages weave simitar patterns on this inner retina, as Wolf Singer has catted it. This is suggested by the fact that ihr fibres coming back from the corlex art about as numerous as thoie going in ihe opposite directions, and have the same spatial distribution over the sheet of neurons m the relay nucleus. Also there is evidence that the returning messages are feature specific, that is, thry can enhance, select, and perhaps mimic sensory patterns. Another nit of evidence IS the finding thaf in cats, activity in the LGN IS heightened during REM steep in which t*t are supposed lo dream. Moreover, this activity was found 10 be similar m character to that evoked by real visual input.
ingly to the cortex and specifically to Area 4 which is the origin of a direct projection from cortex to motor neurons in the spinal cord. ' This connection, the pyramidal tract, is set up by giant pyramidal cells (the cells of Belz) in layer V which control the muscles, with only a single synaptic relay in the spine. Whoever has his hand on the throttle, is in control and this pathway is clearly crucial in shifting control to the cortex, away from the more primitive sub-conical motor systems, which are demoted to deal only with involuntary actions and simple requirements like balance" 1
I would Idee to suggest an extension of Singer's concept af an uftemal retina' to what I called an internal sketchpad'. The idea is that sensory patients are laid dawn m the LGN by sensory input, but simitar patterns may also be sketched there by higher tenters. The LGN IT a passible location for such a process, but certainty not ihe only place where this may occur."
I l.rili with various coworkers, has gone much further and given a precise algorithm which they propose as a model of how the conical feedback to Ihe LGN might be computed. I quote from the abstract to his Science article (Harth e( a) 1987): "The mammalian visual system has a hierarchic Structure with extensive reciprocal connections. A model is proposed Ot which the feedback pathways serve to modify afferent sensory sianuli in ways thai enhance and complete sensory input paiiems. suppress irrelevant features, and generate quasi-sensory patterns when afferent Stimulation is weak or absent. Such inversion of sensory coding and feature extraction can be achieved by optimization processes in which the scalar responses derived from high level neural analysers are used as cost functions to modify the filter properties of more peripheral sensory relays. An optimisation algorithm. Alopex which Is used in the model, is readily implemented with known neural circuitry." L
In addition, some of the axons of the pyramidal tract terminate in the brain stem, in ihe red nucleus, from which ihey project to a new section of the cerebellum, the neo-cercbellum consisting of the deep lying dentate nucleus and the lateral zones on Ihe surface of the cerebellum. The output of ihe dentate nucleus projects primarily not to the brain siem or spine bul up to the thalamus, to the nucleus VLp, and thence to the motor area. Here is our scenario of what ihese new struclures are doing. The cortex, through complex integrated actions of all lobes, 'decides* to make a cenain movement. This command winds up in the motor area, which does two things: it writes Ihe motor command on ils blackboard VLp. and it sends it off down the pyramidal tract. Apparently this message can be either imperative or tentative. The imperative mode is reserved Tor commands which brook no delay and which should be done instantly, even if awkwardly: these take about 7 ms from motor area activity lo muscle response (Evarts 1973). Most commands get caught at the red nucleus, and in the local spinal circuits and don't stan muscle action immediately. In Tact, experimental recordings show delays of around 100 ms between pyramidal IracL activity and muscle response.' 4
This sketchpad hypothesis of Harth is similar to mine but the Alopex theory goes much further in proposing a specific algorithm. I am suggesting more simply that the different parts of the cortex have many different computations to do and lhat the thalamus has an essential but relatively passive role, in integrating the reconstructions, schemes, ideas, etc. of each area and broadcasting them to this and other areas of the cortex. But Ln contrast to the Alopex theory. I am not proposing that the feedback loop between thalamus and cortex is part of any specific pattern recognition computation and, indeed. [ want to ascribe to conico-cortical Loops versions of some of the computations Hanh is interested in These ideas will be developed in the second pan of this paper. h
6 The thalamus and tbe cerebellum Now whal is the role of the thalamus for the primary motor area. Brodmann's area 4? En all higher mammals. Area 4 is an easily distinguishable architectonic area which received no direct sensory input, bul is responsible for initiating movement on the lowest, muscle-bymuscle, level. The evolution of mammals shows a clear progression in which motor control is shifted increas-
765
The cerebellum meanwhile analyzes this molor command and does what it has done in all vertebrates: it modifies and specifies in detail whal combination of forces for whal periods of lime in which muscle groups best carry oui this command Then the cerebellum writes this prescription on VLp. for the motor area to
Area 6, ihe pre-motor area, also projects directly To ihe cord via Ihe pyramidal Iracl The hierarchy of malor areas will be discussed in Che second pan of this paper il would be nice lo be able lo say not merely lhal mammals are unique Ln having a massive cortcn directly controlling their muscles viii Ihe pyramidal Iracl. bul also that mulalions creating [he pyramidal trad distinguish mammals Irom repliles. In fad. Ihe homologies between conical structures in mammals and other structures, in repliles are complex, and. by one theory, pari of the mammalian cortn evolved from rhe rephhan dorsal stnalum which don have a motor output, albeil wilh relays in ihe brainstem " An cilcnsivc v n n of papers by Sasaki aud Gemba (cf. Gemba el al. M l ) have recorded the difference in field potentials between layer I and layer VI in area 4 during the performance of "lined movemenu. They interpret lurfacc negative, depth povuve polenliali u due lo currents in the apical dendrites of superficial pyfiOinlai oeils and llK OppoiiuE potential u due io currents in deep pyramidal cells, such as the Beu cells. The fatter typically sian up aboul 100ms before the muscle ceLli tire. Single cell recordings due lo {Georgopoulos el al. 1989) sho* roughly the same delay | J
1 1
142 read. This allows the cortex to use the carefully learned muscle programs expressed in the synaptic weights of the cerebellum. In the next few milliseconds, the commands sent down Ihe pyramidal tract strengthen or are somehow modulated to say. now do it, and a polished movement is executed. Of course, more primitive pans of the oerebellum will also monitor the on-going movement and modify il by more direct paths, which supplement the cortico-spinal path. In all of this, VLp is playing the role of a low-level motor blackboard, in which projected and on-going movements are encoded in terms of specific forces lo be exerted by specific muscle groups. Although Ihe cerebellar input to VLp will be more informative in terms of whal forces will accomplish therequiredmovement most efficiently and smoothly, the cortex has available lo it more highly processed sensory data thai may lead lo modifying Ihe specifications for the movement. The blackboard VLp will integrate these needs. Luria (1969, English edition, p. 55) has a quite similar analysis of these circuits, though he doesn't use the word blackboard of course: "The principle of feedback ii applied quite differently in Ihe ociinily of that part of the cortex responsible for Ihe organization, programming and execution of oolmlaiy motor activity, for in this realm it becomes the main source of information on Ihe effects of the movements and actions performed. The physiological rote of Ihe matt/' cortex essentially consists of matching tlte "assigned program" of a motor act, formed mainly oo the basis of the analytical arid iniegratine cortical activity of the posterior divisions of the hemispheres, wilh the actual course of its performance, i.e. in detecting signals of success and signals of error tagreement or disagreement between Ihe program and the performance) and in making Ihe required corrections at Ihe right lime in the course of the actions. In view of whal has been said, it will be apparent that both the centrifugal and centripetal {responsible for feedback) chains of relays of impulses, connecting the motor cortex to Ihe subcortical formations, ate included in Ihe extrapyramidal system's of the brain, which are known to be of essential tntporlonce to the coordination of voluntary rnoventerti."
muscular commands over the next second or so, then, assuming the animal's current plan is not interrupted, there should be a correlation between neuronal activity in Area 4 at a given time and the action taken after moderate time lags, as well as the action taken immediately afterwards. The temporal buffering could be done geometrically with different strips of neurons, or it could be done by a subtler in-place coding (see Sect. 9).
7 The thalamus and attention The most widely discussed alternative suggestion about the function of the thalamus is that, in addition to relaying dala. the thalamus gates it in some way. For example, it may be used in focussing attention on some part of the stimulus, or blanking out other parts of the stimulus (e.g. retinal input during a saccade). This theory was developed at length by Crick (1984), correlating it with Treisman's experiments suggesting the mind had an internal 'spotlight of attention' thai could be moved around ihe visual field without actually moving ihe eye (Treisman 19S8), In particular, he suggests thai the position of the RE thalamus, smack in the middle of the pathway to the cortex, makes il the logical candidate for implementing a focus of attention. The exact mechanism he proposes is rather subtle, involving an unusual property of thalamic neurons, related to calcium channels which cause 'low threshold spikes' (Jahnsen and Llinas 1984), and has been disputed by others (Sherman and Koch 1986). Nonetheless, the idea lhat the R E thalamus in some way gates ihe flow of data from the thalamus to ihe cortex is very plausible, given its location and is quite compatible with the thalamus being a blackboard.
The hypolhesis thai Area 4 and VLp perform incremental calculations converging step by step lo the precise muscular act to be performed is consistent with the startling results of Georgopoulos et al. (1989) from single cell recordings in Area 4. He found nol only thai the paltera of excitation in Area 4 at the lime of an arm movement correlated closely with the direction of arm movement in eachrepetitionof his experiment, but that in the 100 ms period before arm movement, Ihe pattern of excitation in Area 4 built up in a definite sequence which can be interpreted as forming mental images of arm movements intermediate between reaching straight ahead and reaching in the direction now desired. It was exactly as if the required muscular commands were being computed in stages, starting from simpler ones for which a template was known and making incremental modifications, using the VLp as blackboard lo record the current proposed arm movement.
What seems most implausible lo me in the theory lhat gating and attention are the primary uses of the corticothalamic projection, is why the need to gate this flow of data would require such a massive projeclion, at least as big as the thalamocortical projection. Bit for bit, it would seem that transmission of data requires more bandwidth than Ihe selection of part of the data. Moreover, Ihis theory ignores the fact that the cortex needs to write most of this data on the non-sensory nuclei of the thalamus before it can read it, and the data doesn't automatically stay around, i.e. the thalamus doesn't have loops so that it can maintain a stale of excitation. Why go to all this trouble lo send data down to Ihe thalamus, as well as sending the gating signals, so that a subset of this data will echo back? This operation makes more sense if, at each stage in a calculation, writing and reading from the thalamus bolh have roles, as they would if il was serving as a blackboard, synthesizing the cortical results via the dendritic arbors of the thalamic neurons, and distributing them back lo the local cortical area via the spread of the thalamo-cortical axons.
A specific prediction lhal I would Uke to make is thai, Uke Ihe auditory areas, the low level malor blackboards musl have a certain amount of temporal buffering. If the data structure for Area 4 is tbe sequence of
Let's look at some numbers to bring ihis home. Unfortunately, quantitative neurobiology is not in vogue and most papers avoid estimating Ihe numbers of neurons and axons in the structures and pathways
766
Hi
discussed. However, for the LGN, Sherman and Koch's article in (Shepherd 1990) gives somefiguresfor the so-called A-laminae of the LGN in the cat. There are Iwo pathways between the retina and the cortex: Ihe X pathway (homologous to the P pathway in primates) concerned with shape and color, and the Y pathway (homologous to the M pathway) concerned with molion. The X pathway has 90,000 axons from the retina lo the LGN, which synapse on some 175,000 relay cells, while the Y pathway has 10,000 axons synapsing on 125,000 relay cells (allfiguresshould be considered as ±20% or so). The cortico-geniculate pathway, on the other hand, contains 4,000.000 axons synapsing on the X and Y relay cells in the A-lamina (how many on each is not known). Thus the cortical input to the LGN is aboul 40 times bigger than the relinal inpul. and 13 times bigger than the reciprocal LGN to cortex pathway. Even allowing largely for multiple synapses of the retinal axons, they estimate lhat only 10-20% of the synapses on the LGN cells arise from relinal axons, while 80-90% arise from cortical axons (Shepherd 1990, p. 264 and p. 278). I suggest that the only way lo make sense of thesefiguresis that most of the data in the LGN is calculated nol directly from the retinal input, but via one or more passes through Ihe geniculocortical loop, this data representing the visual input with considerable image processing added (see Seel. 8). The attention theory has also been proposed by Ojemann (cf. his review article, 1983). Ojemann has carried oui experiments stimulating varous thalamic nuclei in awake humans in the course of operations in which various subcortical structures are being surgically destroyed. He proposes that the thalamus is responsible for a 'specific alerting response' Specifically, he observes that suitable thalamic stimulation can i) improve verbal memory if given al the time of presentation of the item to be remembered and U) increase rate of response and number of errors if given al ihe time of recall. Moreover, such stimulation can also cause persevervation, both on the first syllable of a word being pronounced, or on an earlier response which is not correct for ihe next task. Let me point out, however, lhat these results are also compatible with the blackboard role for the thalamus: if the thalamus is a blackboard, stimulating it could have Ihe effect (a) of highlighting one of Ihe data items represented on it, resulting in better memory for Ihe item, and (b) of hindering Ihe power of corticothalamic pathways to revise and update the data in the thalamus appropriately, causing perseveration and the oiher types of enor Ojemann observes.
8 Possible tests This proposal admits some straightforward experimental tests. The most unambiguous corroboration would be to demonstrate an effect of Ihe corticothalamic projection on the LGN. If Ihe LGN serves as a blackboard, its state should be determined not merely by retinal stimulation, but by Ihe ongoing analysis of this
767
stimulation by the corresponding area of the cortex, V1 or Brodmann's Area 17. Bolh ihe LGN and VI have two classes of cells: fast-responding ones concerned wilh motion (called Y in cats and M in primates) and slower, sustained-response cells concerned with relatively static shapes (called X in cats and P in primates). One simple class of LGN neurons of Ihe second type are the black-white center-surround opponent cells, which respond in a sustained way to the difference of the amount of lighl in a roughly circular central field minus that in a roughly circular annulus around it (or vice versa). Now for these cells, VI seems concerned with using Ihis data 10 find edges and fines in the retinal image, computing their orientation, and how far they continue straight (as in the responses of so-called endstopped cells). In the real world, this is a non-trivial operation because of noise, shading, texture, etc. What I predict, therefore, is thai over some cycle of maybe 50 milliseconds, the picture held by Ihe LGN will improve, noise being removed, edges and fines being sharpened, filled in for instance where the veins on Ihe retina cross them. Part of the image is. of course, changing, and Ihe motion sysiem concerned with Ihis should be making ils own kind of improvements to the LGN responses. But for the relatively sialic parts of the relinal image, i.e. objecis which are slill when the eye is still, or objecls being tracked when the eye is in tracking mode, I would conjecture that the sustained response cells alter their rate of firing in this 50 ms window to mimic what would happen if the retinal signal were improved, much like the image processing which astronomers perform on satellite images. To test Ihis, Ihe first requirement is not to use the extremely simple stimuli typical of these experiments: bars, dots, sine-waves, etc. but more realistic abstractions of real world data: bars with white or other noise superimposed, edges with small gaps. etc. Secondly, thefiringof the LGN neurons in Ihe response period should not be averaged, bul counted separately in each 5 or 10 ms interval following ihe stimulus. For an example, see Fig. 4, where 1 conjecture lhat the response to Ihe bar with blurry spot will start out less than that to the whole bar, but build up, when conical feedback kicks in. to the same as the response to the full bar. A more speculative proposal is that the increase in the number of X-relay cells in the LGN versus the number of axons of X-ganglion cells in the retina is due to the need, for stereo fusion, of constructing shifted versions of the raw input. In other words, the cortex seeks to compare the signal from the left and right retinas. Because of the geometry of stereo vision, these often match up closely after a horizontal shift (whose size is a function of the distance to Ihe viewed surface and the vergence of the two eyes), and the proposal is that such a shift may be physically realized in the responses of some of the LGN cells during the process of fusion of the images from the two eyes. To test this, one need only present image pairs to Iwo paralyzed eyes with varying degrees of disparity, and observe the time course of response of LGN X-relay cells: Ihe possibility
L-M
(Van der Heydt and Peterhans 1989). Therefore, 1 propose lhat neurons in Pli will record boundaries between objects in a scene, no matter how they are marked (or obscured) in the raw image. Moreover Ihe areas of a scene which are part of a single object must somehow be marked as such in order that the shape of the object can be analyzed, leading to its identification. This is an operation called 'coloring' by Ullman (1984). without which the regions corresponding to individual objects cannot be dealt with as units. One plausible conjecture is lhal the responses of some neurons will be locked on to particular objects, so that, when the scene is shifted in front of the eyes, the neuron will fire so long as its receptive field overlaps (or is contained in) the visible surface of the object and will drop off as soon as the region moves away. 9 Temporal sequencing
Fig. 4. Stimulus far lest Of L G N trspanir
is that their receptive fields will sometimes shift to achieve better registration of the two images. Another area in which we may test our theory of active blackboards is by examining the responses of thalamic cells which are not principally driven by subcortical input. These cells receive most of their input from the cortex and I would propose that their responses will indicate clearly what that area of cortex is concerned with. A good example is the inferior pulvinar nucleus which is reciprocally and specifically (i e. synapsing in layer IV or deep IIIf connected lo visual area V2. A conjecture is that while VI is concerned with identifying small pieces of edges, their orientation and motion and possibly curvature. V2 is assembling this data into a global picture of the scene. More specifically, I mean tracing long edges, finding regions of coherent color and texture and deducing from this a segmentation of the scene into individual objects. In psychology, this includes what is called figure-ground separation and it involves crucially the principles of Gestalt psychology. Accomplishing this segregation properly involves integrating the data from stereo vision (because individual objects have continuous disparities without jumps) and from motion (because individual objects move coherently). Neurophysiological dala suggests that V2 does at least some of this
Some of the thalamic active blackboards deal with relatively static information and some deal with rapidly changing information, lhat is only relevant for less than 100 ms say. Thus higher level conclusions, about the geometry of your immediate surround and the objects and the people there, will change only slowly, provided nothing is in rapid motion. Bul dala about a speech signal is superceded by Ihe nexl phoneme with 50¬ 100 ms or so, * For those blackboards dealing with rapidly evolving data whose temporal pattern is essential for ils classification, such as the lower level auditory and motor blackboards, it is essential to maintain a certain amount of temporal buffering: i.e. to keep al any instant the description of the inpul over some fixed period up to the present. Such a prediction should he easy to test. There are several ways to set up such a buffer. e,g. a) write the new signal cyclically into a family of neurons, b) write the latest data always in the same place but shift earlier data along some line, or cf write the data on lop of itself with some 'in-place* coding. Another possibility is that d) the thalamic-cortical loop itself is employed i n doing this buffering, so thai aspects of the time-delayed signal are fed back hy cortico-thalamicfibresand recur once or several times in thalamic activity. In cases a), b) and dt. careful measurement of the time Lags between stimulus and neuron response in primary or secondary auditory cortex, or in the medial geniculate nucleus. MGN, of ihe thalamus should reveal such buffering, especially if multi-neuron recordings are made. In all cases, ihere should be some correlation between activity in auditory cortex at a given time and the stimulus presented some hundreds of milliseconds earlier. 1
Moreover, the Y/M-cells in the visual pathway deal with motion, and their signal is also often evolving rapidly. In fact, an obstacle to Ihe blackboard hypothesis lhat is often raised is how the LGN can be used as a blackboard when the visual signal is changing so fasi. Now the brain has evolved a special tracking mode of eye motion precisely lo keep a uniformly moving objeel nearly stationary on the retina, but this usually
" A J , Terry Dcjcon pointed oui lo me. the rate oE phoneme production is not necessarily the rale al which new data appears in Ihe CortcK. Although sipeech it described hy linguists as a sequence of phonemes, m the sound itself the phonemes overlap, i.e. the time intervals in which each phoneme affects Ihe sound overlap. If subcorneal structures extract features which encode ihe clues for all the overlapping phonemes at each instant, then new data appears; in the cortex roughly wilh each new syHabit, nol each new phoneme. This suggests new dala every lOO-lSOms 1
768
145
works for only one object at a time and motion may not be uniform, nor the object rigid. Fd like to propose that this is exactly why, in the Y/M-pathway, there is such a large increase in number of LGN cells to number of retinal cells (estimated at 12 :1, see Sect. 7). The extra LGN cells are available for temporal buffering, storing the motion history during a single fixation of the eye. Making a more precise prediction requires a specific hypothesis of how temporal buffering is done: hopefully the same mechanism is used by mammals in the visual, auditory and motor domains, but I don't want to make a guess. References Be ndcr D B 11961) Relinotopic organization of Ihe macaque pulvinar, J Neurophys 4«:672 - 693 Blinkov S M , Glazer II (1968) The human brain in figures and tables. Basic Books, New York Bransford J . Barclay JB, Franks J (1972) Sentence memory: a conStructure •.• interpretive approach. Cogn Psychol 3:193-209 Brodal A (1981) Neurological anatomy. Oxford University Press. Oxford Chemiak C (1990) The bounded brain. J Cogn Neurosci 2:56-68 Clark; J. Yuille A [1990) Data fusion for sensory information processing sysrems. Kluwer Academic Press, Amsterdam Crick F (1984) Function of the thalamic reticular complex: the searchlight hypothesis. Proc Nail Acad Sci £1:4536-4590 E m a i l i n . Hayes-Roth F , Lesser V R . Ready P. {1980) Tbe H E A R S A Y - I I speech understanding system. Comput Surv 12:211-253 Evans E V (1973) Molor cortex reflexes associated with learned movement. Science 179:501-503 Felleman D I . Van Essen D C (1991) Distributed hierarchical processing in primate cerebral cortex. Cerebral Cortex: (io he published) Gcmha H. Hashimoto S. Sasaki K (1981) Cortical Geld potentials preceding visually initiated band movements in monkeys- Exp Brain Res 42:435-441 Georgopoulos Af\ Lurito JT. Pelrides M . Schwartz A B . Masscy JT (1989) Mental rotation of the neuronal population vector. Science 241:234-236 Graybiel A M . Bcrson D M (19BI) Families of related cortical areas in the extrastriatc visual system. In: Conical sensory organization, vol 2. Humana Press, Ctifion, N J . pp 103-120 Harth E (19>&3> Windows on the mind. Quill, New York Harth E , Unnikrishnan K P . Pandya AS (I9S7I The inversion of sensory processing by feedback pathways: a model of visual cognitive functions. Science 1967:184-187 lahnscn H . Llinas R 419G4) Electrophysiological properties of guinea-
pig thalamic neurons: an in vitro study. J Physiol London 149:205-247 Jones E B 11985) The thalamus. Plenum Press. New York Lindsey P. Norman D (1977) Human informaiion processing- Academic Press, New York Luria A R (1969] Higher cognitive functions in man. 2nd edn. Moscow University Press. Moscow (English edn 1980, Basic Books, New York) Marr D (1982) Vision. Freeman, San Francisco Minsky M (1975) A framework for representing knowledge. In: Winston P (ed) The psychology of computer vision. McGrawHill. New York Mjolness E . Gindi G , Anandan P (1988) Optimization in model matching and perceptual organization. Research report YaieU/ DCS/RR-614 Ojemann G (1981) Brain organization for language from the perspective of electrical stimulation mapping. Behav Brain Sci 2:189¬ 206 Rockel A J . Hioms RW. Powell TPS (1980) The basic uniformity in structure or Ihe neocortex. Brain 103:221-244 Selfridgc O (1959) Pandemonium: a paradigm for learning. In: Symposium on the Mechanization of Thought Processes. HM Stationary Office, London Shepherd G (1990) The synaptic organization of the brain, 3rd edn. Oxford University Press, Oxford. Sherman S M . Koch C {1986) The control of rerinogenjculate transmission in the mammalian lateral geniculate nucleus. Exp Brain Res 63:1-20 Steriade M. Llinas R R (I98S) The functional states or the thalamus and the associated neuronal interplay. Physiol Rtv 68:649- 742 Sleciade M , Domich L . Oakson G (1986) Reticularis thalami neurons revisited: activity changes during shifts in states of vigilance. I Neurosci 6:68-61 Treisman A (1988) Features and objects. Q J Exp Psychol 40A: I98B Ulhnan S (1984) Visual routines. Cognition 18:97-159 Van der Heydt R. Peterhans E (1989) Conical contour mechanism* and geometrical illusions. In: Lam D M , Gilbert C D (eds) Neural mechanisms of visual perception. Gulf. Houston, Texas Weller R E , Kaas J H (1981) Cortical and subcortical connections of visual cone* in primates. In: Conical sensory organization, vol 2. Humana Press, Clifton, NJ, pp 121-156 Winston P ( l 9 7 5 ) Learning structural descriptions from examples. In: Winston (ed) The psychology of computer vision. McGraw-Hill. New York
Dr David Mumford Mathematics Department Harvard University I Oxford Street Cambridge, MA 02138 USA
769
Reprinted with permission from Biological Cybernetics, \o].66,pp. i 4 [ - 2 S l . 1992 © 1992 Springer-Verlag
O n the computational architecture of the neocortex II The role of cortico-cortical loops
D. Mumford Mathematics Department, Harvard University, I Oxford Street. Cambridge. M A 02138. LISA Received June 29, 1991/Accepted July 12. 1991
Abstract. This paper is a sequel to an earlier paper which proposed an active role for the thalamus, integrating multiple hypotheses formed in the cortex via the thalamocortical loop. In this paper, i put forward a hypothesis on the role of the reciprocal, topographic pathways belween two cortical areas, one often a 'higher' area dealing with more abstract information about the world, the other 'lower', dealing wilh more concrete data. The higher area attempts to fil its abstractions to the data it receives from lower areas b y sending back to them from its deep pyramidal cells a template reconstruction best fitting the lower level view. The lower area attempts to reconcile the reconstruction of ils view that it receives from higher areas with what il knows, sending back from its superficial pyramidal cells the features in its data which are not predicted b y the higher area. The whole calculation is done with all areas working simultaneously, but with order imposed by synchronous activity in the various lop-down, bottom-up loops. Evidence for Ihis theory is reviewed and experimental tests are proposed. A third part of this paper will deal wilh extensions of these ideas to the frontal lobe.
I Introduction The point of view of these papers and the motivation behind them was described in the introduction to the first part of Ihis paper. Summarizing, the idea is that the uniformity and highly specific layered structure of the neocortex of mammals suggests that some quite universal computational ideas are embodied by this architecture. This paper presents iwo proposals for computational mechanisms embodied in this structure. The first part dealt with a conjecture for the role of the reciprocal and largely topographic pathways con-
770
necting each area of the cortex wilh a corresponding nucleus in the thalamus. It was proposed that a very importani part of the compulation performed by the cortex made use of this loop: • that Ihe corlex learns multiple patterns that recur in sensory stimuli and in stereotyped motor outpul, • that at any given time, the cortex is attempting to analyze Ihe present situation in terms of ihese patterns and, in so doing, generates multiple hypotheses, often conflicting, • lhal all Ihese hypotheses are sent down to the lhalamus where a kind of voting takes place in the dendritic arbors of the thalamic neurons, • lhal Ihe consensus is then broadcast back to Ihe cortex as an updated view of that aspect of the world dealt with by that area of cortex. This theory was summarized by describing the role of Ihe thalamus as lhal of an 'active blackboard' bearing the data on which Ihe cortex was working. The second pari will deal wilh the reciprocal and largely topographic pathways lhat are found throughout the cortex connecting pairs of cortical areas. A detailed proposal will be made for the nature of computation performed by exchange of messages via this loop. The present paper will expand these ideas for sensory processing carried out in the posterior half of the cortex and a third part of Ihe paper will apply [hem lo the computations underlying planning and action carried out in the frontal lobe. I will propose specific tesls for some of Ihese ideas. As in Ihe earlier paper, we have included a good deal of background both on neuroanatomy and on computer science in order to make the ideas as clear as possible to readers from various specialties. Finally many people have had ideas similar in various ways with ideas presented below: ihe ones I know of are the 'Adaptive resonance theory' of Carpenter and Grossberg (1987), Ihe 'HyperBF' theory of Poggio and collaborators (Poggio 1990), the 'counlercunenl' processing theory of Deacon (1988) and recent theories of Rolls (1990) on the 'back-projections' in the brain.
341
2 Pyramidal neurons and cortico-cortical pathways
the cortex strikingly from other bodies such as the olfactory bulb and Ihe cerebellum, in which intemeurons substantially outnumber the cells with long axons. Some crude numerical estimates may be useful: using the estimates cited in the first part, each hemisphere of ihe human cerebal cor [ex may contain roughly 10 billion neurons. Then an average area would have aboul 100 million neurons, with say 60 million pyramidal cells projecting lo some other cortical area If this area is connected lo 30 others, each pathway comes out as containing the order of 2 million fibres, the same order of magnitude as the optic nerve. In strong contrast, the cerebellum has an order of magnitude more neurons than ihe cortex, and mosi of these are its principal intemeurons, Ihe granular cells, whose number is estimated al JfW billion in man! The number of Purkinje cells, ils outpul cells, is only 15 million or .03% of the total (Ito 1984). The pathway between Ihe cerebellum and the rcsl of Ihe brain is also an order of magnitude bigger than Ihe cortical pathways: in man it is estimaled lo conlain 20 million axons (Brodel 1981, P- 297).
I begin by reviewing some neuroanatomies! facts about cortical pathways. As described above in the first part of this paper, each hemisphere of the primate cortex seems to be divided into something of the order of a hundred areas each with a specialized role. There are, of course, species differences', but the general map and often many of ils details are roughly homologous for most species. Tracing pathways supplements Ihe map of cortical areas with a diagram of their interconnections. These interconnections turn out to be relatively sparse, in the sense that of the 10.000 possible one-way pathways that could exist between 100 areas in each hemisphere, perhaps only the order of magnitude of 2000 exist (Felleman and Van Essen 1991). This could well be the result of limitations of space inside the cerebral hemispheres, which can only contain so much white matter, but it obviously has computational significance. A very important fact, central to the theory in this paper, is that all or almost all (there are some ambiguous cases) interconnections so far discovered are reciprocal: if area
The fact lhat the majority of conical cells have inier-area projections, as opposed to exclusively iniraarea projections, seems already Io bear an important computational message: it means that almost nothing goes on internally in one area without this activity being transmitted to at least one other area. The classical view
A projects lo area B. then B projects lo A.
What types of cells set up these pathways? There are two main types of neuron in the neocortex: pyramidal cells and intemeurons. Pyramidal cells are large, excitatory, with a pyramid shaped cell body, spiny dendntes and a long myelinated axon (myelin is nature's way of insulating axons to ensure stronger, faster long-distance signals) that projects to another area of the brain (another cortical area or subcortically), usually with branches projecting locally'. These are the neurons which create these cortico-cortical pathways. Intemeurons are small with only local projections, spineless dendrites and are usually inhibitory, An intermediate class consists of the spiny stellate cells populating layer IV which generally project only locally but resemble pyramidal cells in being excitatory and having spines: they are roughly pyramidal cells without a long axonal projection, and are sometimes called small pyramidal cells. The percentage of neurons in human cortex which are pyramidal has been variously estimated as 60%80%, although there seems to be some doubt about counting accurately the intemeurons which have smaller cell bodies' This distinguishes the structure of
The major difference ii that the number of areas increases with increasing brain size. For instance, the frontal lobe in humans is much larger than in other primates and seems to contain many more areas The existence of extensive local collaterals is a major difference between the output, pyramidal cells of the cortex and the output cells of the lhalamus. It allows the cortex to tarty on local calculations indefinitely without further stimulation, whereas Ihe lhalamus cannot do this Various references can be found in the discussion in DeFelipe and Jones (1988), pp. 590-599 The counts in Winfield el al. (19B0) seem representative and are confirmed by immunochemical determination of the percentage of G A B A cells. They find on average S B * pyramidal. SM large stellate and 28% smaller intemeurons (presumably inhibitory) in tat and rat cortex 1
on the significance of ihe different areas of ihe brain was that ii was similar to the modular decomposition of a computer program inlo subroutines. In computer programs, each module performs a specific task and the various modules pass input and output back and forth by means of messages (or via globally accessible data structures, like blackboards). The analogy suggests that each area in the brain has a specific capability, e.g. looking up words in a lexicon, sequencing motor acts, etc. and that the inter-area pathways exchange requests and answers. But this analogy would only make sense if the number of intra-area locally projecting neurons were an order of magnitude larger than the number of inter-area globally projecting neurons. Since Ihis isn't the case, a different paradigm must be sought. This is the main idea of this paper, which I will develop in stages. In essence. I want to propose that the bulk of the computational work of the cortex is nol carried out by one area at a time, but by information going back and forth over reciprocal pathways connecting pairs of areas: in doing Ihis. each such pair of areas is trying lo reconcile their constructs by some kind of
relaxation algorithm. Before developing this idea further, we need some more anatomical facts.
1
3 Higher versus loner areas For a long time, there have been attempts, using the above menlioned modular view of the brain, to give each of Ihe different cortical areas a particular functional significance and to describe the nature of the information represented by neuronal activity in each
1
771
143
cortical area. From such assignments, we can describe the pathways between the areas in terms of passing data from an area with one sort of concern to another. A persistent theme is to distinguish lower cortical areas, with direct sensory or motor connections from higher ones which are associating information from lower areas, so that information moves first from lower, more sensory areas to higher, more cognitive association areas and secondly from these association areas back down to lower motor areas. There are several ways of establishing such functional correlations: firstly, the distance of an area from the nearest area with direct sensory or motor connections (the primary sensory and motor areas) is one indicator of how high-level it is. This is confirmed by comparative neuroanatomy, in that lower mammals have almost all their cortex taken up by the primary motor and sensory areas*, while an increasing amount of secondary tissue appears in mammals with greater intelligence. Secondly, direct stimulation of the cortex of humans, first employed in operations for intractable epilepsy by Penfield, resulted in the patient's experiencing a variety of thoughts, ranging from very concrete sensations or motor reactions to quile elaborate memories or abslracl ideas. Thirdly, the loss of functions in strokes can be correlated to Ihe cortical area destroyed by the slroke. Fourthly, single cell recordings in animals, especially primates, enable one to correlate the firing of a particular neuron to the presence of various stimuli, or the performance of various tasks, and these show a clear gradient from elementary sensory or motor responses, to elaborate complex responses (e.g. the presence of a monkey's face in Ihe field of view). These four techniques give a fairly consistent, though imprecise, idea of which areas were 'higher' and which 'lower' than others and roughly what sort of data was being dealt with. Traditionally, one way in which this data has been pui together is via a division of the cortex into primary sensory and motor areas, secondary sensory and motor areas and tertiary 'association' areas. A much more precise way of ordering cortical areas, which agrees with and extends ihe above higher/lower ordering was found by analyzing ihe connections of the areas in terms of the layers of origin and Ihe layers of termination of each pathway. To describe this, I need to first sketch the cell populations of the six layers. The pyramidal cells occur in two populations: the deep pyramidal cells in layers V and VI and the superficial ones in layers II and III. Layer IV in the middle is occupied mainly by the spiny stellate cells and. as we have seen, is the principal inpul layer for sensory data and olher thalamic projections driving cortical calculation (although nole that Ihe celts in layer IV, by virtue of their arborization, will already perform some transformation on the input data). Layer I. called by Cajal the plexiform layer, has extremely few cell bodies of any kind, but is the zone for a rich set of connections
between a second type of axonal input, the intemeurons and the apical dendrites of the pyramidals. The inhibitory intemeurons occur in all layers except I, and themselves break up into a dozen types or so with differing geometry and distributions. Tn terms of layers of origin and termination, there seem to be three types of long distance cortex to cortex connections between areas, all set up by pyramidal cells. In this division, I am quoting the results in the exhaustive survey paper Felleman and Van Essen (1991). To see a particular example in detail, the projections to and from VI are shown in detail in Perkel et al. (1986) and Van Essen et al. (1986). The survey paper deals primarily with ihe posterior, sensory-oriented areas of the cortex, and I will restrict the discussion at first to these areas. The first of these types of pathway originates in deep pyramidal cells, usually in layer V, and terminates heavily in layers I and VI, avoiding layer IV completely. The second of these originates in superficial pyramidal cells and terminates primarily in layer IV. The third of these originates in superficial pyramidal cells, but instead terminates outside layer IV, mostly in layers I and VI. There are a few reports suggesting further types of connections, but these, if present, don't seem lo be widespread. What makes this division into types impressive is that, whenever two areas A and B are reciprocally connected, and the previously discussed evidence shows clearly that if area A is 'higher', B is 'lower', in terms of their function, then: 1. The ascending pathways from d to .4 is set up by superficial pyramidals in B terminating in layer IV of A. Note lhal this is consistent with layer IV being the standard input layer at each stage of the stream of data all the way from the senses themselves Lo the highest cognitive areas. 2. The descending pathway from A to B always includes deep pyramidal cells in layer V of A terminating mainly in layers I and VI orB. If A is 'much higher' (in some loose sense, see (Felleman and Van Essen 1991)) lhan B . this is the only projection from A lo B. Note that the projections from A lo the thalamus are also set up by deep pyramidals (chiefiy in layer VI), so we have a consistent picture of deep pyramidals projecting to lower structures, either in the cortex or sub-cortical. This is also consistent wilh the idea that A delivers a different kind of inpul to B from its standard input. I'll call Ihese the standard descending paths. 3. The descending pathway from A to B may also include superficial pyramidal cells of A terminating chiefly in layers I and VI of B. This occurs if the ordering belween two areas is nol so clear, A is only slightly higher than B in Felleman and Van Essen's sense. Note that again terminations in layer IV are avoided by the 'higher' to 'lower' projections. I'll call Ihese the extra descending pathways. A summary of this pattern is shown in Fig. 1. It would be nice if we could extend this picture unequivocally lo the frontal lobe. While Ihere have
1
This is ignoring rhe limibic areas, dealing wilh emotion, social behavior and memory, which are also large in lower mammals
772
244
Fig, I, Conico-corneal pathways by layer
been fewer studies of ihe laminar connections to and within the frontal lobe, present evidence seems lo favor the idea thai this same laminar pattern is present (Deacon in preparation; Primrose and Strick 19S5). Felleman and Van Essen (1991), however, qualify this conclusion wilh the remark lhal "The patterns illustrated in the literature are difficult to interpret unambiguously .. .". (cf. section entitled ' Hierarchical relationships in other areas", subsection ''Somatosensory and motor cortex ). The data suggests a picture in which {a) [he primary motor area. Area 4, is lowest, and the other frontal areas starting with the premotor and supplementary motor areas gel higher and higher, in a complex pattern, while {b, the layers ol~ the connections conform to the three types (J), (ii) and (JJJ) above. Another important caveat is that when two areas A and B are reciprocally connected, one needn't be higher, the other lower, in any clear way In this case, one can imagine that all types of connection are possible, and the neuroanotomy doesn't reveal anything directly about the functional nature of the pathway. Can we make some sort of hypothesis aboul the functional role of the ascending and descending pathways? In the rest of this paper 1 will try lo the analyze the role ol~ these pathways in the sensory half of the brain. Ihe occipital, parietal and temporal lobes, and leave to part III an extension of our theory to the motor half of the brain, the frontal lobe. Now the ascending pathways have never seemed to be problematic, because information must obviously flow from the senses up to cognitive areas. This ascending stream of information is referred to as 'bottom-up' processing. However, there is a general realization of the importance of "top-down' processing too. involving the active use of high-level knowledge to help disambiguate lowlevel perceptions (as in Ihe ability to discern the dalmation dog in Gregory's famous picture, see Fig. 2). This is what I want to analyse first. 11
773
4 I riding pathways carry Templates Let us slep back and make some elementary observations aboul whal descending pathways must do. Without any preconceptions about the computational nalure of Ihe cortex, one would expect that activity in lower sensory areas of Ihe brain is directly correlated with elementary properties of the sensory input, while activity in higher areas is correlated wilh the presence or absence of some more subtle properties of the sensory input, e.g. the presence of a face. This is clearly bom out by single cell recordings for instance (cf. £>esimone's survey paper (1991) for a history and description of the so-called Tace' cells in inferior temporal cortex). One might say that the higher area is speaking a more sophisticated, more abstract language. In lhat case, when information is passed from a higher to a lower level of the brain, it must be translated from The abstract language of Ihe higher area to the concrete terms employed by the lower area. If some particular pattern of bils in a higher area happens to mean face', there is no point sending this encoded pattern down to a lower area which knows nothing aboul faces. You have to translate face' into a signal in the terms used by the lower area. e.g. a pattern of bits signifying the appropriate concrete configuration of lines, shapes and colors. Such a translation is whal in psychology would be called a menial image, a reconstruction of a detailed sensory signal that instantiates an abslract class of signals. In the language of pattern recognition, it is what is called a template. Early work in pattern recognition centered around the idea of recognizing classes of signals by having a specific template (i.e. a standard example of a signal in each class), which could be matched, feature by feature, second by second, or pixel by pixel, against the signal lo be classified. Our proposal is that the axons of ihe deep pyramidal cells in the descending pathways store templates in the weights of their synapses in the lower area. The single bit represented by a pulse on the axon of such a cell musl stimulate, via the weights on ils synapses in ihe lower area, a low level template-like response repre-
243 senting Ihe translation of the information in that bit in the high level representation scheme into more concrete information in the low level representation scheme. One must not oversimplify here: a simplistic form of our hypothesis would be that specific deep pyramidal neurons, or small sets of them, were responsible for each template in the lower area. For example, one might suppose that several dozen deep pyramidals in inferior temporal cortex constructed an eye template, in the sense that the strength of their synapses in lower order visual areas created excitation equivalent to a retinotopic eyelike stimulus. This would be an elegant hypothesis, but il looks totally unbiologicaL. Much more likely, it seems, is that the compulation is distributed, that thousands of neurons are simultaneously carrying eye, nose, mouth, face, etc. templates. This would explain why recordings from such a large percentage of inferior temporal neurons show responses to faces, and lhat the ability to recognize faces is robust in the face of small local damage to corlex: alt ihe usual arguments in favor of distributed representations in neural nets. Some evidence for this hypothesis comes from the experimental fact thai the axonal arbors of descending pathways are, on the whole, more extensive than those of ascending pathways: we would expect this if the descending pathways were recreating Ihe pattern of excitation characteristic of some higher level construct, because such higher level constructs embody common extended patterns of excitation in Ihe lower area. Very carefully drawn illustrations of the arborizations of typical V2 => VI axons can be found in (Rockland and Virga 1989). Her pictures suggest nol only that a rather intricate excitation pattern is created by the top-down activity of such a neuron, but even that this pattern may reproduce Ihe effect of extended lines, which would usually be part of any higher level visual pattern. This is seen in the fact that in some of her pictures the axon sprouts synapses al regular intervals, interspaced with synapse-free zones, as it extends in a specific direction in layer I; because of the known division of VI inlo orientalion-speciric columns, this could well be the structure needed to stimulate successive parts of an extended line wilh afixedorientation. Recall lhat Ihere are also exlra descending paths formed by superficial pyramidal neurons, but only in case of areas which are not loo far apart, one not being loo much higher lhan the other. Our hypolhesis is that the standard descending pathways which are always present carry templates, especially because the further apart two areas are, the most different witl be their 'languages', hence the greater the need for template-like translations from one language to another. After I extend my hypolhesis to the ascending pathways, I will come back and make some speculation on the role of these extra descending pathways.
5 Templates must be flexible Early pattern recognition work using templates was never very successful, however. The difficulty was how
774
to account for the range of variation in the objects being recognized: two eyes are never the same, and one must be able to recognize as eyes all the variations which normally occur, including eyes in people never seen before, eyes in strangely lit faces, cartoon eyes, etc. The problem of allowing for normal variations arises already when trying to classify some object on the basis of a few measurements: a classic example in the statistical literature was that of discriminating three species of Iris from the length and width of its petals. One must model the allowable variations of length and width within each species or, better, the full probability distribution of the measured features for each class before making an informed decision on the species from these Iwo features. The problem gets harder for ID signals such as speech: an essential adjustment is called limewarping, in which templates for the various phonemes are scaled to allow for the speaker's rate of speech. In 2D signals such as vision, the problem is much more difficult. Letters can be varied in many non-linear ways while still being readable, faces dislort wilh differing expressions and shadows change the appearance of even simple industrial parts on an assembly belt. Other types of variation are not usually continuous but correspond to the object belonging to one of several subcategories: e.g. faces with and without glasses, a person being male or female, a screwdriver being plain or Phillips, etc. What alt this suggests is lhal you need a flexible template: a template wilh built-in variability embodied in a sel of parameters, whose values can be chosen so that the template will nearly match Ihe example of the class in the signal being analyzed. In vision, early work in Ihis direction is due to Fischler and Etschlager (1973), and a recent version can be found in Yuille (1991).
The parameters in a flexible template are not imagined to vary arbitrarily, but to have some restrictions placed on them which form an essential part of the template: 1. Ihey generally have an allowable range, individually or jointly (e.g. two parameters may have to lie in some subset of ihe plane). 2. there may be a prior joinl probability distribution, i. one may store a set of useful examples - e.g. the parameter values for a prototype instance and some key borderline cases, showing the worst instances you've encountered. How do these parameters and their allowable range fit into our neural theory? What I want to propose is lhat when a high level area has neurons which fire in the presence of eyes, then the full pattern offiringin this area will encode a sel of parameters for eyes. The value of these parameters will determine some of the synaptic input on an assemblage of deep pyramidal cells, and thus it will modulate pari of the signal being sent on the descending pathway. Therefore, instead of having one signal on this pathway that says 'eye' and produces a fixed template response in the lower area, there will be a family of varying signals, with spikes of varying rales and phases, representing eyes with different values of
244 the eye parameters. Each such signal will stimulate the lower area differently, producing Ihe same effect as a flexible template with built-in parameters. Further, it is logical lo suppose that Ihe strengths of Ihe synapses of other neurons on the dendrites of the deep pyramidals in Ihe higher area are the place where Ihe limits of this allowable variation is stored These limits may be learned by gradual modification of ihese synaptic strengths, presumably by the presentation of multiple examples and by some mechanism which stores nol jusi their mean but their variance in some form. An interesting proposal for a specific way of representing and learning Ihe variances of natural categories, as well as predicting the parameters for best fits, is being developed hy Poggio and collaborators (Poggio and Girosi 1990, Poggio 1990). They hypothesize that both probability distributions for membership in a category and the values of associated parameters, as functions of a vector of features, may be approximated by a family of functions, called 'Hyper basis functions', of [he form: M
= £ c.G(|x-t,|i,).
R
—
B—3
Expected symbol Best match. with smuceje by template Fig. 3. Problem* in rccognilion by template
Unexpected symbol
does one judge whether or not lo accept thefitof Ihe template and decide lhal the signal does contain a valid instance or the class of objects in question? An early idea was lo decide that the letter B' was present on a page when part of Ihe writing was more like a 'B' than any other letter. This procedure can go wrong in two ways. The page might have an unusual characler here, not matching any English characler, say the 'there exists' sign 3 of mathematics, and you shouldn't have accepted 'B' just because the symbol 3 was closer lo a 'B' than any other English character. Or the paper might have an 'R' partially obscured by a smudge making the whole shape a bit closer to anB' than an •R' say (see Fig. 3). The point is that identification is only complete when you have analyzed all ways in which the signal differs from the template (after putting in optima] values for ils parameters). These differences are whal i call Ihe residuals. Many things may be happening. L
Here xis the vector of features, t, are the exemplars from which the function has been learned. C is a function like a multi-dimensional Gaussian. Ihe subscript W on the norm is a weighting of Ihe individual features (e.g. an inverse of a covariance matrix) and r, arc learned weights. They propose neural mechanisms for implementing Ihe calculation of such fs as well as learning Ihe weights. Developing algorithms of this kind for learning variances or some other measure of natural variability of exemplars and for clustering similar exemplars seems to me to be a central problem for neural nel architectures. Fodor and Pylyshin (1988) have raised the question of how neural nets can express composite concepts and can rapidly build new composite concepts which have never been entertained before. Our notion of flexible templates seems to incorporate a limited form of composilionality in a natural way When a template is active. Ihe values of its parameters naturally associate to thai concept a sel of qualifying properties, much like a noun phrase may be formed by a principle noun and a sel of adjectives and clauses. When several templates are active, their parameters don'l get confused: the two scenes "Black dog and while cat" and "While dog and black cat" correspond to Iwo different slates of menial activity. Such linking does not allow us to arbitrarily combine Two different concepts, but only to form combinations when one already occurs as a dimension or variability of the other. In a more linguistic context, there could be a template for the action "hit", whose parameters included a description of the object hitting and for the object being hit.
ri Residuals Flexible templates were a major improvemenl on templates but flexible templates also have problems. How
775
1. The residual may be so large that ihe template is plain wrong (e.g. you were trying lofilthe eye template to a mouth), and this particular identification should be rejected. 2. II may be lhal a definite part of [he template is missing in [he signal (e.g. an objeel in a scene is partially occluded by something In fronl of it), so you should accept Ihe identification provided lhal Ihe missing parts can be explained. 3. Il may be lhal Ihe signal contains something extraneous in addition to ihe template (e.g. while the sentence "Everything's fine al home" is being uttered, a child's scream breaks in), and again ihe identification should be accepted provided lhal the extraneous pan can be explained. 4. Even when ihe correlation belween the signal and Ihe template is overwhelmingly large, so the identification is clearly correct, there are many situations when the residual contains very useful information about the world (see example in Sect. 8 below). 5. Finally. Ihe residua! may be 'noise' or 'clutter': unidentifiable, seemingly random stuff and one should then stop with Ihe identification and not burden (he resl of Ihe algorithm further with its analysis or storage. The moral here is lhal an animal should not rest until it has 'explained' the full sel of signals coming to il from the world, as far as its past experience allows, and must also be able to recognize when the signal indicates - because of variations beyond ihe normal limits - something never encountered before Ideas in Ihis direclion have been pul forward by many people in
247 different contexts, for example, in computer vision by PavUdis (19S8). The idea of residuals is closely related to concepts in The theory of robust statistics (Huber 19Sl) In robust statistics, one considers the problem of estimating the mean of a distribution from a sample, in the case where either the distribution itself has large tails or the sample is somehow corrupted. In both cases, the sample is Likely To contain a few large outliers, which will cause large changes in the sample mean. JIuber's solution is to explicitly identify the outliers and use the thinned sample to estimate more robustly Ihe distribution's mean, Thus if 60% of a model or template Jits a signal very well, one should explicitly mark the remaining 40% as outliers, and measure the goodness of fit of the remaining 60%. If This fit is good enough, it is usually stronger evidence than 90% of the template fitting the signal cnjdely. r
7 Ascending pathways carry residuals The last part of the proposal deals with the role of the superficial pyramidal cells. As before, we will only consider sensory areas in this section. Let us assume lhat a lower area B is interconnected with a higher area A. We always get two sets of connections and sometimes a third (with the single arrow): Higher Area A Lower Area B deep pyramidal ceils ^> synapses in layers I and VI synapses in layer IV superficial pyramidal cells superficial pyramidal cells -c synapses in layers I and IV
strange place with no expectations or are totally surprised by something* then Ihe algorithm starts with a clean slate in area A. Then B just sends ils whole picture of the world to A which excites some possible higher level objects. At each stage. A writes on its blackboard its besT guess in its language (objects and their parameters) aboul the identity of the higher level objects found in ZTs picture. How do the extra descending paths from the higher area A to lower area Bfitinto our theory? Various even higher areas ( are all predicting what A sees and this explains all but some residual of A's picture. A can tell Ihe higher areas C, about these unexplained features, to try io find a top-down explanation of them, and it can tell lower areas such as B about them. If the superficial pyramidals express the residual part of the world picture of higher area A. then the extra descending paths would carry such a message to B. Their effect could be to modify the world picture of area B, weakening the evidence on which these conclusions of A were based, pushing the lower area B to seek alternate parses of ils data and to explain away the residual on a bottom-up basis. Note that this is different from sharing with B Ihe reconstruction of the world which area A is entering: such sharing of conclusions can be accomplished through the lhalamus, on which these conclusions are written. My description is only ihe beginning of an algorithm for processing sensory inpul like a visual signal. But I have convinced myself of its plausibility by analyzing particular complex scenes of the world and seeking a 'rational reconsiruction of the process that the brain might follow in finding the correct semantic high-level interpretation, Such an approach has been followed by Cavanagh (1991), who analyzed recognition of faces in extreme lighting conditions producing dark shadows and confusing contours. His conclusion is that an algorithm very similar to our (emplate/residuat loop is the most likely possibilily. What is mosT striking to someone who has experimented wilh small algorithms in computer vision - which operate without human prompting - is that any system of [his type working on real visual input could be siable, could reliably integrate multiple small clues and find ihe one combination of hypotheses which explains the whole image. Nonetheless. I am proposing (hat a large number of independent loops* each looking for its pet structure in a lower level blackboard, working simultaneously on low and high levels, can in real world situations converge rapidly to the correct solution, without huge oscillations and without creating fanciful high level images unconnected to reality. 1
Our proposal is That Ihe loop with double arrows embodies an iterative algorithm that attempts to identify a specific higher level object in the lower level data. More specifically, the deep pyramidals of A send a signal to B containing the template for each predicted object O. Area B compares these templates to its blackboard, which gives ils own present reconstruction of the world from its vantage point and computes a residual, a description of lhat pari of the world which isn't expected or predicted. Its superficial pyramidals then send back to A this residual, a description of what doesn't Jit A'i prediction. The weights on its synapses in the higher area translate this residual into the higher level language. Then area A modifies the parameters in the flexible template to try lo improve ihe fit and sends this back lo B and it may also hypothesize the presence of further objects in if's world. After a few turns, either a good fit is found and ihe residual is acceptably small or ihe hypothesis is rejected and area A turns to other previously suppressed hypotheses. In the ultimate stable state, the deep pyramidals would send a signal lhat prefectly predicts what each lower area is sensing, up lo expected levels of noise, and ihe superficial pyramidals wouldn*i fire at all*. At the other extreme, if you wake up in a y
B Comparison with other top-dowri/bottom-up theories The proposed sketched above has many similarities both to the 'adaptive resonance theory of Carpenter and Grossberg (I9B7X the 'counter-current processing model' of Deacon (1988), Poggio's Hyper basis function theory (1990) and Roll's theory of backprojections in cortex (1990). 1
In some sense, [his is [he stale lhal ihe corlex is slriving I D achieve: perfect prediction of ihe world, like ihe oriental Nirvana, as Tal-Slng Lee suggested to me. when nothing surprises you and new slimu.fi cause Ihe merest ripple in your consciousness i
776
249
Fig. 4a-i The "Adaptive Resonance Theory" of Carpenler-Grossberg (Carpenter and Gros&berg 1987)
Thus Carpenter and Grossberg's theory is summarized in Fig. 4. and il works like ihis: F, and F are mo cortical areas projecting to each olher. A pattern of activity X in F, (shown by the symbolic pattern in (a)) evokes a signal 5 on the bottom-up pathway lo F . S stimulates a pattern of activity T among all the stored categories inlo which X might be classified, and. by a winner-lake-all algorithm, the best fitting category l is selected. The pattern Y in F evokes a signal U on the top-down pathway, which stimulates a pattern of activity ViaF,, the template or 'learned expectation'. fand combine to form X'\ either X' is close to X, in which case the network stabilizes and classifies X as being an instance of Y, or else mismatch occurs. Panel (b) in the figure shows the latter, and, in this case, it results in an 'arousal bunt' from module A which inhibits Y in a long-lasting way (panel (c)). Now the same pattern of activity Ton F no longer selects Ybul the second best matching category 1", which is in turn compared with X, etc. We see that ART posits a recursive calculation in a top-down/bottom-up loop which is very similar to ours. The first major difference, however, is that in ART, the templates store some kind of mean or median representative of each learned category, and make no attempt to explicitly encode the variation within a category as in the work of Poggio el al. (1990). In Poggio's analysis, a suitable set of exemplars for each category are stored, and used to generate a smooth function which approximates the probability that a new feature vector input should be interpreted as a member of the same cate2
2
2
2
777
gory. As explained above, I feel that Ihis is essential to any successful recognition algorithm. Moreover, along with storing variances, the degree of mismatch should not be merely a number, whose size determines whether or nol to seek a new category, bul a signal representing what does not match. This is the second major difference, and leads to our idea of residuals. Making explicit such residuals allows the higher area to seek complex explanations of [he input in which several templates are superimposed. The following vastly simplified illustration may explain why I feel storing variances and describing residuals is essential in real-life situations. Suppose two numerical features x and y are computed from an olfactory stimulus, and suppose the world contains [wo animals A and B. Suppose that the smell of A excites x and y roughly equally, but that ihe smell of B excites x but not y. In a noisy world, we should never say the B's smell doesn't ever excite y, but ralher something like: in the presence of B alone, [he value of y is almost always at mosi 1 /20th Lhal of x. Finally, suppose A is dangerous while B is not. Then suppose the input has values x = 5. y = 1. The template for A is x = y = any positive value (the smell may be strong or weak depending on [he proximity of A), and the template for B is x = any position value, y =0. Clearly, we gel a much better correlation of the input with the template for S, wilh a suitable parameter put in. Bul, knowing the variation expected in the smell of B, we see that there is non-trivial residual. The besl fit by B alone might be x = about 5. y - 0.25, with a residual x - unknown, y = 0.75. The residual can be fitted wilh the smell of I. and we recognize the presence of danger (see Fig. 5). Note lhat to carry out this procedure, we need both to explicitly encode the variability offl'ssmell and to separate the relatively small unexplained part of the input from the dominant part explained by the first template. Situations of this type occur more often than not in the analysis of real visual data for instance. Deacon has also proposed a theory of cortical processing based on top-down/botlom-up loops, that he calls 'counter-current' processing. Like ours, his theory is motivated by the laminar assymetry between ascending and descending cortico-cortical pathways. He proposes thai a form of relaxalion belween the information in two mulally connected areas takes place, the higher areas sending down foci of attemion. expectation and associated imagery, while lower areas send up perceptual details and recognized patterns. He does not assign as precise a computational role, however, to the two streams as I do, bul develops an interesting metaphor of two fluids moving through adjacent tubes in opposite directions, where some quantity like heal diffuses from one 10 another at all points of contact of the lubes. Finally. Rolls has discussed the bottom-up/ top-down loops in corlex in connection with memory and Ihe learning of categories and has a primitive neural net simulation of this loop. He stresses the importance of separating two stimuli which are close in one sensory modality, but which are learned to be very different from experience in other modalities. Both
249 A, F a c t u a l Database odor 2
odor I
S m a l l at dangerous animal A
B)
Sensory Input
C]
Conc1Ion&
- best. [ I t -1111 a n i m a l &
odor I
Smell o f f r i e n d l y animal B
^residual
I n d i c a t i n g animal A
Odor I
atio'l
via--2 Fig. S A - C .
1
?
j
Odor 2
The importance of variances and residuals
Deacon and Rolls analyze the entorhinal/hippocampal complex at the top end of the cortical area hierarchy and its role in the formarion of memories, which [ am excluding from this paper.
9 Managing the top-down/bottom-up loop If the top-down phase and bottom-up phase of a pattern recognition algorithm are to work effectively together, it would seem necessary to coordinate them. In sequential, von Neumann architecture terms, one could think of the loop consisting of a) one 'cycle' of computation in the lower area, 6) passing the dala up to the higher area, c) one 'cycle' Ihere and finally d\ passing the data back down. Then ihe brain would operate by a relaxation algorithm, in which ihe loop is repeated until il stabilizes. The brain being, by nature, highly parallel, and there being not just one pair of lower/higher areas but many, it is more reasonable to imagine the lower and higher area working at once, and then exchanging their data. Very recent experiments by Cray and Singer (19891 have discovered thai strong local oscillations with a 20-30 ms period (corresponding frequency 35-50 Hz) accompany periods of intensive computation in at least some areas of ca! cortex (VI and V2). The oscillation may be detected in Ihe mean local electrical held, or in single cell recordings from a large number of individual cells. Freeman (cf. the review Freeman and Skarda 19E5) has found similar, somewhat faster, oscillations in
778
rabbil olfactory bulb caused by alternating bursts in pyramidal cells and intemeurons. but the bulb, like the cerebellum, differs from corlex in having an order of magnitude more intemeurons than output neurons, suggesting quite different computational principles. It seems logical to propose that these oscillations are caused by or synchronize calculations in the ascending/descending pathway loop. (The same suggestion has been entertained by Hubel and Livingstone - oral communication.) It may be that iteration in this loop would be unstable if the top-down and bottom-up phases occurred asynchronously. This would predict that oscillations like those found in VI by Singer will be found in every area of Ihe cortex and that successive waves of top-down signals and bottom-up signals occur al specific phases of the local oscillation. Bursts of this oscillation will coincide with active local computations using this loop. This makes a very specific prediction: that if simultaneous recordings are made from deep pyramidals in a higher sensory area and superficial pyramidals in a lower sensory area which project to each others 'columns', i.e. to near each other, then bursts in the two populations will be phase locked with each other, and with the local mean electrical field. Allowance must be made for the fact that signals are by no means simultaneousfiringsof all pyramidals of each class: The information is precisely in which ones arefiringand probably in timing differences of their individual spikes too. But suitably averaged (as in the recordings thai demonstrate the oscillation to begin with), I would expect to see this synchronization of remote neurons. There are other parts of our theory in which synchronization is needed. The time buffering in auditory and motor corlex presumably needs some kind of pace maker. But Ihis would be much slower, e.g. I 10 10 Hz. The time scale of our top-down, bottom-up loop must be much faster or the brain would never gel anything done. The experimental finding of 35-S0Hz seems about right: in half the period, 10-15 ms, the local iniracolumnar circuils of the corlex should have time to do non-trivial calculations, and there is time for half a dozen iterations of the loop before the books close on interpreting a stimulus. An intriguing possibility is lhal the claustrum plays some role in modulating the 'top-down', 'bottom-up' calculation between various cortical areas. The claustrum is a relatively small subcortical nucleus lhat is located like a seventh layer of Ihe cortex just beneath a certain cortical area, the insula, but separated from il by a thin layer of white matter, the extreme capsule. Il is connected to almost the whole neocortex, but not topographically'. This is a major exception to the pattern for other connections and means that if two cortical areas A and B projects to parts A' and B' in the claustrum, lhan A' and B' often overlap. In fact. Pearson et al. (1982) have made the following generalization on the basis of extensive primate studies: • A' and B' overlap if and only if the cortical areas A and B are directly connected by cortico-cortical pathways.
250 It should also be noted that Ihe clauslrum is an evolulionarily conservation Ibrm, being present and similarly connected in the most primitive mammals. These facts makes it look likely that the claustrum is connected to the operation of these reciprocal pathways in some essential way. Now the clauslrum seems to have too few neurons to play a role in the substance of the calculation taking place in the loop, but it is ideally situated to modulate the relaxation algorithm between the areas in some way, e.g. initiating and terminating it or in some way maintaining its stability (see also Crick and Koch (1990) for a related proposal).
10 Mental images Another enticing speculation is to consider the action of the brain in a purely introspective state. In the course of reflecting about some problem, we can block out the actual stimulus being received by our senses, or we can close our eyes. At that point, all the neural machinery for sensory processing is available for thought. I want to conjecture that the process of thinking things through often involves writing in a purely top-down mode on Ihe active blackboards of low level areas, and using Ihe various reciprocal pathways to better understand a situation or problem which is not physically in front of us. This can be done by Ihe deep pyramidals cells which will evoke a template in the activity of the lower area, and thus write this template on its blackboard. I want io propose lhat this is exactly whal we do when we form a mental image of some object. This sysiem of thinking can be applied, e.g. to work out tricky things about the three-dimensional geometry (can we carry a piano up the apartment stairs), or lo work out more abstract problems using amorphous objects as tokens for parts of some situation. The mental rotation experiments of Shepard and collaborators (Shepard and Cooper 1982), suggest that analog, continuous, real-lime rotation is often performed on mental images. A natural interpretation of their results in the present context is lhal this step-bystep transformation is carried out by ihe top-down, bottom-up loop between cortical areas. In many ways it is analogous to the relaxation algorithms using the same loop by which parameters in a template are ileratively adjusted to achieve a betterfitwith a stimulus. In Ihis case, however, a mental image which is a rotated version of one stimulus is ileratively adjusted to achieve a belter fit wilh another stimulus. Moreover, it is also known that the brain is working intensely during dreams without any sensory inpul and that the thalamus is quite aclive. The visual images present during dreams would seem to be stimuli evoked purely by lop-down pathways. During dreaming sleep, the brain also receives diffuse cholinergic stimulation from brain stem nuclei via so-called 'PGO waves' (Mamelak and Hobson 1988). Bul the vivid, often realistic though bizarre, images of dreams would
779
seem to require something like our template generating deep pyramidal neurons. I hypothesize that the same mechanism that gives rise to mental images when awake drives the formation of dream images. Their bizarreness may result from the evocalion of multiple top-down images simultaneously for some as yet unknown cognitive/emotional function.
II Possible tests A much debated property of VI neurons is that of 'end-stopping'. Neurons with this property fire in the presence of bars or edges, moving or still, with a fixed oriemaiion and a fixed location provided they are not too long. That is. the stimulus must be contained in a certain receptive field and il mustn't continue outside this field. Zucker and collaborators (Dobbins et al. 1987) have hypothesized lhal this is due to the neuron computing Ihe curvature of the bar or edge, and that it would fire strongly if the bar or edge continued bul turned wilh approximately a specific curvature. A radically different hypolhesis is that the neuron does fire briefly to a longer line, but that as soon as lop-down signals from V2 incorporate this long line into a global segmentation of the scene, the line is accounted for and the firing slops. In other words, ilsfiringindicates that the line is unexpected, and not pan of a coherent global pattern. A short line never fits into such a pattern and firing continues: it remains a residual. This theory would predict that superficial end-stopped cells, i.e. those in layer II and III. would be responding to residuals. This predicts that their end-stopping would not be absolute: they would have a transitory response to longer edges, which would be inhibited as soon as Ihe VI => V2 => VI loop kicks in (say 20 ms). It would further predict that deep end-stopped cells respond more consistently as in Zucker's theory to some properly of the stimulus. More generally, a plausible prediction of the Iheory is that many of the responses of superficial pyramidal cells should be transitory. The idea is that when everything being sensed is predicted or explained by the high levels' models of the state of the world, there are no more residuals to send upstream. In a calm state of meditaiion, for instance, their overall activity would diminish substantially. Another conjecture would be similar to the classic experiment of DeValois et al. (1979) in which the retinotopy of VI was revealed by fixing a visual stimulus on ihe retina, injecting the animal with radioactive glucose taken up in metabolism, killing the animal after a short period and examining the pattern of radioactivity present post mortem in VI. I would propose stimulating strongly a deep pyramidal cell in an area like V2 or V4, connected to VI. while again marking cell activity via a radioactive metabolite. If V4 is concerned with shape recognition, template-like shapes should appear in VI. Precisely because V4 is nol finely retinotopic, the pattern of aclivily in VI would be extended and not precisely localized.
251
References
Pavlidis T (1938) Image analysis. Ann Rev Comput Sci 3:121¬
Brodal A (1981) Neurological analomy. Oxford University Press, Oxford Carpenter G. Grossberg S (1987) A massively parallel architecture for a self-organizing neural pailetn recognition machine. Comp Vision Graphiu Image Proc 37;54-115 Cavanagh P {199]) Whai's kip in top-down processing. In: Gorei A (ed.) Reprtscntaiions of vision. Camb. University Press, Cambridge, pp 295-304 Click F> Knock C (1990) Towards a neurobiological theory of consciousness. Semin Neurosci {in press] Deacon T (1988) Holism and associationism in neurophyschology: •-" anatomical synthesis. In: Perecman E (ed) Integrating theory and practice in clinical neuropsychology. Erlbaum, Hillsdale NJ Deacon T (in preparation) Laminar organization or frontal corticocortical connections in the monkey brain DeFeJips J, Jones E (1988) Cajal on [he cerebral cortex Oxford University Press, Oxford Desimone R (1991) Face selective cells in (he temporal cortex o f monkeys. J Cogn Neurosci 3:1-8 DeValois KK, DeValois R, Yund E W (1979) Responses o f slriate cortex cells to grating and checkerboard patterns. J Physiol (London) 391:453-505 Dobbins A, Tucker S, Cynader M (1987) Endslopping In the visual cone*: a neural s-ubsiratc for calculating curvatuTc. Nalure 329 96-103 Felleman D I , Van Essen D C (1991 > Distributed hierarchical processing in primate cerebral cortex. Cerebral Cortex 1:1-47 Fischler M, Elschlagtr R A (1973? The representation and maiching or pictorial siructurcs. I E E E Trans Comp 22:67-92 Fodor J , Pylyshjn X (1968) Conneclionism and cognitive architecture. Cognition 28:3-71 Freeman W, Skarda C A (1935) Spatial E E G patterns, non-linear dynamics and perception. Brain Res Rev 10:147-175 Gray C . Singer W (1989) Stimulus-specific neuronal oscillations in orientation columns of cat visual cortex. Proc Natl Acad Sci
Pearson R C A . Brodal P. Caller K C , Powell T P S (1932) The organization or ihe connections between Ihe cortex and the claustrum in Ihe monkey. Brain Res 234:435-441 Perkel DJ. Bullier J Kennedy H (1966) Topography of the ancient connectivity of area L7 in ihe Macaque monkey. J Comp Neurol 253:374-402 Poggio T (I990J A theory of how the brain might work In: The Brain. Proc Cold Spring Harbor Symp 55 Poggio T, Girosi F (1990) A theory of networks for learning. Science 247:978-982 Primrose D. Slrick P (1985) The organization of interconnections between the premotor areas of the primate frontal lobe and the arm area of the primary motor cortex. Soc Neurosci (abslr) 111274 Rock 1 (1984) Perception. Sci. Am. Hooks, New York Rockland K R , Virga A (1939) Terminal arbors of individual 'feedback' aions projecting fl™ area V2 lo V I in ihe macaque monkey J Comp Neurol 285:54-72 Rolls E T (1990) The reprcsenlation of information in Ihe temporal lobe visual cortical arteas of macaques, (n: Eckmiller R (ed) Advanced neural computers, Elsevier, New York Amsterdam pp 69-78 Shepard R, Cooper L A {1982) Menial images and iheir transformations. M I T Press, Lancaster Van Essen D C , Newsomc WT, Maunsell JHR, Bixby J L (1986) The projections from striate cortex to areas V2 and V3 in Ihe Macaque monkey. J Comp Neuiol 244:451-480 Winfield DA, Gaticr K C , Powell TPS (1990) An electron microscopic sludy of the types and proportions of neurons in Ihe cortex of the motor and visual areas of ihe cat and rat. Brain 103:245¬
146
r
258 Yuille A (1991) Deformable templates for (ace recognition. J Cogn Neurosci 3:59-70
B6:I69E-I7Q2
Hubcr P (1981) Robust statistics. Wiley, New York Ho M (1934) The cerebellum a i w « r o i Raven Press, New York Marnelak A, Hobson JA (1983) Dream biianeness as ibe cognitive correlate of altered neuronal behavior in R E M sleep. J Cogn Neurosci 1:201-222
Dr. D. Mumford Mathematics Department. Harvard University I Oxford Street Cambridge, MA 02138 USA
n C U l
780
Computational Strategies
T h e reader who has come this far had ample opportunity to realise that this is a field of enormous diversity. Not only are there various disciplines involved and different research methods come to play, but the multitude of organisms, brain areas and functional goals offer a variety of anatomical structures, physiological schemes and processing mechanisms. Theoretical modelling in this domain, aimed at discovering some underlying principles, requires imagination and a certain degree of boldness. The three articles in the present section fall in this line. A sentence at the end of article (65), by Poggio, defines aptly their speculative style: "It is very likely that, although plausible, the ideas are wrong — unlike the work on which they rest". Article (63), by Anderson and Van Essen, proposes a revision in the traditional notion of static receptive fields of sensory neurons. A number of visual tasks, i.e. stereopsis, motion analysis, scaling, directed attention, require a dynamical modification of the retinal inputs used by a neuron. We have seen before that there is evidence for such changes in higher areas of the visual system: article (45) reports on the "shrinking" of an initially very broad receptive field of neurons in area V 4 , following attention cues in behaving monkeys; article (52) demonstrates a remapping of the visual space in the parietal cortex, caused by intended eye movements. The authors of article (63) suggest that such processes occur early along the visual pathways already in the striate cortex (area V I ) . They propose a highly engineered circuitry ("shifter circuits"), using shunting inhibition to shift the flow of information along alternative neural trajectories. The analysis, based on neuroanatomical and psychophysical data, complements the accounts of the primate visual system in articles (26), (56) and in sections 4a and 4b. Notice the predictions of dynamic receptive fields, and of top-down mechanisms, in analogy with some predictions of article (62), but note the quantitative discrepancies in the neuron counts reported for cat and monkey early visual areas. Although there is at present no convincing evidence to support the ideas evoked in paper (63), it serves to highlight a new and simple compu-
781
tational algorithm, which provides a natural solution to a variety of routing problems in vision and, possibly, in other sensory systems. Article (64), by Hopfield, starts with a reminder that some animals solve remote sensing problems with olfaction, instead of with vision as primates do. It thus deals with problem of spatial orientation by processing odor information of the environment. T h e physical delivery of this information (variable air flow) requires analysis of time fluctuations of the intensity of the different odors that make up the composite odor signal. There is behavioral evidence that such analysis does actually take place. The author proposes a computational scheme, implementable on a neural network, to perform this task. A n essential aspect of this scheme is a fast modification of synaptic connections during the recognition process, a theme in common with article (58). Notice also another proposal for the role of oscillations, as queries done in multiplex. The last article in this section, (65) by Poggio, is even more ambitious than the other two in terms of the breadth and generality of its goals. It proposes a computational strategy for the recognition of objects, features and situations in a continuous multi-dimensional parameter space (in a manner announced and motivated in paper (62)). The author suggests that the brain can be partly explained in terms of "approximation modules" of discrete instances (specific orientations in space, points of a trajectory, positions on a curved surface, etc.), which can be viewed as extensions of "look-up tables", and of "grandmother cells". These basic modules are used to generalise from the learned situations to new ones. The generalisation is here accomplished by population coding (cf. papers (17) and (45)). The article ends with the idea that the emergence of intelligence is related to a trade-off between memory and computation, an echo of von Neumann's considerations in paper (10). The reader is invited to note some analogies and differences with papers (32),(46) and (50).
782
Reprinted from Free. Nail Acad. Sri US*. Vet 14, pp 6297-6301, 1987
Shifter circuits: A computational strategy for dynamic aspects of visual processing Istereopsis/aitention/motion analysis/vision/striate cortex) C. H . A N D E R S O N * * A N D D . C . V A N ESSEN**
• Di-id SimofT Restoreh Cenlci. Rrimeion. Commanicaled
by Jnhn J. Hopfield,
NJO8M0;
ind iDivision of Biolojy, 216-76, Crii'ornis InsMuie llf TnjlMln. Piudenn. CA 9UU
May II, 1987 (received ffir refit*
ABSTRACT We propose • genera] strategy for dynamic control of information flow between arrays of neurons at different levels of Ihe visual pathway, suiting in the lateral geniculate nucleus and the geniculorecipient layers of cortical area VI, This strategy can be used for resolving computational problems arising in the domains of stereopsis, directed visual attention, and the perception of moving images. In each of these situations, some means of dynamically controlling how retinal outputs map onto higher-level targets is desirable—in order to achieve binocular fusion, to allow shifts of the focus of attention, and to prevent blurring of moving images. The proposed solution involves what we term "shifter circuits." which allow for dynamic shifts in the relative alignment of input and output arrays without loss of local spatial relationships. The shifts are produced in increments along a succession of relay stages that are linked by diverging excitatory inputs. The direction of shift is controlled al each stage by inhibitory neurons lhat selectively suppress appropriate sets of ascending inputs. The shifter hypothesis is consistent wilh available anatomical and physiological evidence on the organizalion of the primate visual pathway, and it offers a sensible explanation for a variety of otherwise puzzling facts, such as tbe plethora of cells in the geniculorecipienl layers of VI.
Information-processing sysEekis. whether biological or electronic, should be designed kittefliciem and timely routing of the signals used for conpotation. In a standard digital computer. Ihe routing of inftVrnaiion to and from the central processor is inherently diftftmic. With each computational cycle, ihe central processor selects its inpul data from avast array of memory addresses amd'likewise sends Ihe output to any desired address Are there analogous switching processes that regulate infcnlmation How in the nervous system? That is. can ihe inputs uaesjby a neuron for its compulations be dynamically swiiched. or are they rigidly fixed? The possibility of dynamic switching processes in the visual system has been suggested in relation to several functionally distinct aspects of perception, including directed visual attention, stereopsis. and the compensation for motion blur (1-61. In this report, we will show how a specific type of information-routing strategy, implemented by what we call a "shifter circuit," may provide a common mechanism underlying each of these seemingly disparate perceptual processes. The shifter circuit offers a genera] means of linking an array of lower-level neurons |such as retinal ganglion cells (RGCs)] lo a higher-level processor in a manner thai allows for dynamic shifts in the relative alignment of the two levels without the loss of local spatial relationships. We first consider the case of stereopsis. as it allows for the clearest
formulation of both the computational problem and our proposed solution. The Registration Problem in Slereopsis. in stereopsis. information aboul binocular disparities of images in the left and right eyes is used to make inferences aboul the depth of objects in the visual field. This requires having the iwo eyes properly converged, so that the Iwo retinal images are in register. However, this mechanical alignment process, mediated by vergence eye movements, is imperfect; the misalignment is several minutes of arc under optimal conditions and can be an order of magnitude larger under more realistic conditions where head movements are allowed Oi. Nonetheless, images are readily fused binocularly and arc perceived to be stable in depth, and relative disparities of only a few seconds of arc are sufficient for stereoscopic depth discriminalion (8). Thus, specialized neural mechanisms musl exist to provide for binocular fusion, perceptual stability, and stereo discrimination in Ihe face of binocular vergence fluctuations. Poggio and his colleagues (4, 8, 91 have demonstrated ihai cells in cortical area VI (striate, conexl of alert monkeys show disparity tuning thai is considerably sharper lhan the measured variability in binocular alignment; they accordingly have suggested thai a dynamic neural alignment process occurs early in Ihe primate visual pathway. An efficient way to compensate for misregistration of the eyes would be to shift the monocular image representations phono the first stage of binocular integration. An outline of this strategy is illustrated schematically for a one-dimensional situation in Fig. 1 The processing sequence begins with iwo sets of RGCs, one from each eye. Each ROC layer projects by way of a shifter circuit to an intermediate layer, designated ihe registration stage, where the inputs are still monocular bul have been adjusted for optimal binocular alignment. The shifting process should respect several major constraints. lilEach cell in the registration stage is. at a given moment, excited by only a single RCC. (ill Local ordering of inputs is preserved, so lhal the inpuis from each eye activate a contiguous sel of cells at the registration stage, (iii) The relative alignment between the two image representations is under the control of a visually driven dynamic shifting process, which in this case has compensated precisely for the misalignment of ihe illustrated luminance patterns in the two eyes. Once registration is achieved. Ihe iwo monocular inpul arrays projeci lo Ihe binocular iniegration stage, where disparity-iuned neurons can be generated by way of a stereo algorithm thai need not be specified here. Thus. Ihe introduction of a dynamic shifting process does not solve the well-known "correspondence problem" in stereopsis (B). bul ii offers a major computational simplification by greatly
The publication COSTS of this article were defrayed in pan by page charge payment. This article musl Iherefure be hereby marked odvertliemenl" in accordance wilh IS U.5.C tl)M solely to indicate this Facl. H
March 17. I9S7)
Abbreviations: RGC. retinal (an|lion cell. LGN, laleral geniculate nucleus. 'Present address: Jet Propulsion Laboratory, Mailstop 198-330.4800 Oik Grove Drive, Pasadena. CA 91109. •To whom reprint requests should be addressed.
783
6298 BINOCULAR REGISTRATION INDEX
LEFT
EVE
RIGHT
EYE
FIG. 1. Schematic diagram of how a shining process could provide binocular registration at the conical level (VI) despite misregistration of luminance patterns for the Iwo eyes. Note that the sharp luminance peak, which activates noncorresponding RGCs (hatched circles), maps onto corresponding cells at the regisusjion stage. R. right: L . left. reducing the range over which false correspondences need to be tested. A Basic Shifter Circuit. An elementary shifter circuit capable of producing dynamic realignments is illustrated in Fig. 2. For simplicity, it is shown only for one dimension and for a small, monocular input array. At the core of the circuit is a hierarchy of cell layers (four in A: three in B) connected by ascending, symmetrically diverging excitatory connections. Switching of information flow is mediated by a set of laterally directed inhibitory pathways that selectively suppress particular sets of inputs. In the ascending component of the pathway (Fig. 2A) each cell in level 1 connects to two target cells in level 2: one a half-step to the left and the other ahalf-step to the right. Each cell in level 2 in turn connects to two target cells in level 3. each one a full step to the right or left. Finally, between levels 3 and 4 each cell bifurcates and contacts cells two full steps to the right or left. As a result, any one cell in level 1 has potential access to eight different cells in level 4. Information flow through each level is regulated by inhibitory switches that selectively suppress one of the two inputs
Fio. 2. A simple shifter circuit. M) Ascending components for a four-level circuit with eight cells al the bottom. Cells at each level bifurcate and coniacl a pair of large! cells at the next level. (01 A complete shitier circuit for a three-level circuit starting with four cells. Specific dendritic innervation patterns are shown for both ascending inputs and inhibitory neurons involved in shift control. Heavy lines in A and B represent an activity pattern involving successive shifts to theright,left, and (in At again lo Ihe right.
784
to each cell (Fig. 2fl). The scheme illustrated here presumes that individual dendrites operate as discrete functional units, with each dendrite receiving excitatory input from a single lower-order cell and an inhibitory connection from a single inhibitory neuron. The inhibitory input is presumed to act as a spatially restricted shunt that can "veto" excitatory inputs arising more distally on the same dendrite (10). Coordinated shifts occur at each level because of the geometry of the ascending connections and of the inhibitory cell connections; At each level, the left branch of the ascending axon contacts the right dendrite of a higher-order cell, while the right branch contacts the left dendrite of a different higher-order cell. At all levels, [he inhibitory neurons follow a simple rule: each one either contacts exclusivelyright-sidedendrites or exclusively left-side dendrites. Thus, depending on which Inhibitory neurons are active (thick lines and filled cell bodies) and which are inactive (dashed lines and open cell bodies), information at each stage will be directed to the left or to the right, but it will automatically be kept in register for the whole array. Linkages involving noncontiguous cells are excluded as Jong as one and only one of the inhibilory neurons is active out of each pairShifter circuits can be readily extended to two dimensions. One straightforward way to accomplish this is to have four connections from one cell to its targets at the next level, which suffices to ensure complete coverage of al! possible shifts. The number of stages needed is no different for two dimensions than for one—-namely, log? k for a shifting range of it cells. Descending Control. In the circuit of Fig. IB, the degree of shifting at any given moment is specified by the pattern of activity in the inhibitory neurons at each Level. For stereopsis, changes in the direction and magnitude of the shift for each eye could be provided by a visually driven feedback system that involves some measure of the degree of alignment of the two monocular activity patterns over a restricted part of the visual Held (represented by the "binocular registiation index" in Fig. 1). This could account for psychophysical evidence that the depth at which an object is perceived can be modulated dramatically by the disparity cues provided by nearby features in the visual field (11). The binocular registration index might simply be based on the relative balance of activity among neurons tuned for "near" and "tar" disparities (9). with binocular alignment driven in one direction if the activity of near cells exceeded that of far cells and in the opposite direction if the balance of activity were reversed. Alternatively, the binocular registration index might be based upon the pooled activity of "tuned excitatory" cells (9) lhat respond optimally to objects on or near the
6299 perceptual horopter. In this case, the shifters could operate by keeping tbe binocular registration index maximized for a given set of visual inputs. Simple versions of either type of control mechanisms should nol be difficult to implement. The circuitry needed to attain and maintain binocular registration in realistic operating circumstances is likely to be rather complex, though, since the sysiem must be capable of adjusting rapidly lo a dynamic visual environment, must perform smoothly over a range of spatial scales and over the entire visual field, and may also be linked to the motor system For vergence eye movements. Neuroanalomical Substrates. For reasons of computational simplicity and efficiency, shifting related to stereopsis should take place before binocular convergence and before extensive feature analysis has taken place. This clearly points to the lateral geniculate nucleus (LGN1 and layers 4C and 4A of area VI, because cells at these levels are monocularly driven and have smalt, cenier-surround receptive fields that reportedly differ little from those of RGCs (12.13). Fig. 3 illustrates schematically some of the major routes of information flow into and out of these layers as determined from connectivity studies in the macaque monkey (12.15,16). There is a basic segregation into magnocellular and parvoceUuiar streams, which originate as separate cell populations at the RGC level and continue as largely independent pathways through the LGN and VI. The magnocellular stream includes The P-alpha class of ganglion cells, the magnocellular layers of the LGN. and layers 4Ca and 4B of VI. Layer 4B is the first stage at which cells are binocularly driven and show a high incidence of selectivity for binocular disparity, orientation, and direction (9.12,13). The parvoceUuiar stream includes the P-beta ganglion ceils, the parvoceUuiar LGN layers, and four stages in VI; lower 4Cb. upper 4Cb. layer 4A, and finally the supragranular layers 12 and 3). where binocularity and other integrative properties first appear. The larger number of relays associaled with the parvoceUuiar stream may be related to the smaller size of receptive fields (171. which would necessitate more stages of shifting to achieve a given absolute displacement. Layer 6 of VI provides massive feedback to all of the relay slagesin the LGN and in the layer 4 complex 116. IK). At each of these stages there are also numerous y-aminobutyric acidcontaining, presumably inhibitory intemeurons (IE, 19). Thus, the essential elements needed for assembling a shifter circuit like lhat in Fig. 2 exist in an appropriate general configuration. The absolute numbers of cells are also more than adequate, as indicated in Fig. 3 by the number of cells
* w
14 24
2-3
T $
T t
h ft
4CC-biijip«r K""
•p
Motion Compensation, The world is perceived as stable and unblurred despite the fact that images on the retina are in constant motion, even during periods of careful fixation (6, 7). We suggest thai shifter circuits may play a key role in preventing motion blur. The proposed strategy is to introduce a compensatory cortical shift whose velocity is equal but opposite to the locally measured retinal velocity field. As illustrated in Fig. 4. this would transform a coherently moving retinal image (shown al discrete times. E| and 1?) inlo a physically stabilized image representation at the level identified as the "stabilization stage." analogous to the registration stage for stereopsis. As with stereopsis. it makes sense to carry out this stabilization prior to detailed form analysis, again implicating the layer 4 complex of VI as the primary site for neural implementation. The maximal range of motion-compensating shifts is indicated schematically in Fig. 4 by the dashed lines between the ganglion cells and the stabilization stage. The patlem on the right indicales the number of cells at the stabilization stage accessible to a particular ganglion cell (the divergence range). Equivalent ly, the convergence range is represented on Ihe left as the number of ganglion cells having access to a
6
T i
pjmia LGN \r I
L A Y E R
al each level, in millions of neurons per hemisphere (14). Indeed, shifter circuits provide a sensible explanation for what has been a puzzling overabundance of cells in layers previously regarded as simple relay stages. A biological shifter circuit might have a rather different configuration lhan the simple binary configuration of excitatory inpuis and outputs shown in Fig. 2, while nonetheless conforming to the same basic design principles. Irrespective of the detailed connectivity pattern, the dynamic range over which shifts can occur is constrained by the aggregate horizontal spread of interconnections, summed over successive stages between input and binocular registration stages. To compensate for binocular misalignments of ±0.2°, the approximate range suggested by eye-movement measurements (4, 7), relative shifts of ±3 mm across the cortical surface (±1.5 mm per eye) would be needed in the foveal region, where the cortical magnification factor is 15 mm/ degree (20). For the magnocellular system, The extent of individual geniculocortical afferent fibers is abouT 1 mm, while the intrinsic conical connections spread about 1 mm in each direction in layer 4Ca and even further (±4-5 mm) in layer 4B (15. 16). Thus, the magnocellular registration stage might be represented in either layer 4Ca or layer 4B. The parvoceUuiar system is more restricted in ils spread within layers 4Cb and 4A (±0 S mm), whereas there is a divergence of ±1 mm in the projections to layer 3 (15, 16). Unless the parvoceUuiar system operates over a narrower dynamic range, its registration stage would apparently be constrained to lie in layer 3 rather than layer 4A.
T PflRUD RGC
POfiUDCELLULOH STREAM
- - I nflEMD L 6 H
inncMD use
| 0.
srneiLimTiDrf STRGE
1
OOOOOOOOOOC-OOC4-RGC
e. i
MOGHDCELLULfiR STREAM
F I G . 3. Connectivity patterns in the parvoceUuiar and magnocellular processing streams of Ihe visual pathway in the macaque monkey. Solid lines indicate major ascending pathways; dashed lines indicate Feedback. Not shown are identified ascending connections Uiai skip one or more levels, such as that From the L G N directly lo layer IA. Numbers next lo each box represent millions of cells per hemisphere (data from ref. 14). Terminology for their sublayers has beenrevisedto match that of ref. 13.
785
i.unj NONCE POTTFRWS
A RET[NHL
IHSGE nOTION
F I G . A- A stiaTegy for moiion. compensation. A veJocily field measure me n[ feeds into the shift control mechanism, .hereby cornpen sating for relinal image motion and p r i c i n g a stationary activity pillcrn ai Ihe conical subi.iuiion stage.
6300 patticular conical neuron at the top of the cascade. When the limit of the shifter is reached, the circuit should be quickly reset. Such a sawtooth shifting process would translate smooth, continuous retinal image motion into a succession of stationary conical images that jump in saltatory fashion across the cortical surface. The trade-off between space and time implied by this scheme could account for spatiotemporal interpolation processes reponed psychophysically (21). Exposure times of at least 100 msec are needed to suppress motion blur (6). This might signify a lower bound on the duration of the "frames' associaled with (he shifting process. At 3°/sec, the speed at which Vernier hyperacuity begins to decline (22). the shifting process would need to operate over a conical distance of 4-5 mm per 100 msec in the foveal representation. This is within the range of the horizontal spread of connections already noted for the magnocellular stream. Scaling and Blurring. At any given eccentricity, we are able to analyze patterns over a wide range of spatial scales. One efficient strategy for obtaining this range is to create a "pyramid" consisting of multiple spatial representations, each of which is coarser and more blurred than its predecessor (231. The shifter circuit of Fig. 2 can be generalized to provide such a multiresolution representation of the visual world. For example, silencing of all inhibitory inputs at any one level would automatically lead to a 2-fold spatial blurring of the information that reaches each successive stage. This could be done in adynamic fashion (by relaxing the constraint lhat one and only one inhibitory neuron be active at each level) or in hard-wired fashion (by setting up separate shifter circuits for each spatial scale). The latter scheme would have the attractive feature that information over a range of scales would be simultaneously available for subsequent form, lexture. and motion analysis. Directed Visual Attention. Al a given moment we are capable of concentrating our attention on a small portion of the visual field, known as Ihe focus of attention (1,24). Within this region, intricate details of the visual world can be closely scrutinized, while information streaming in from the rest of the visual world is attended lo a lesser degree, if at all. The focus of attention can be shifted rapidly (and without eye movements) to any part of the visual field, and if can also be scaled to different size ranges (24). Whal types of neural circuitry could mediate shifts in the focus of attention? Despite our lack of knowledge about the location, number, and internal organization of the high-level centers responsible for attentive scrutiny, the problem can nonetheless be analyzed profitably as a general question of how to route information from any restricted portion of the visual field lo an arbitrary high-level center without inordinate loss of detailed spatial relationships. This is similar in several respects to the registration problems already discussed in the stereo and motion domains, suggesting that shifter circuits might again prove useful. However, the attentional system needs the capacity for shifts corresponding to very large physical displacements, with inputs tunneled into the highest level from anywhere on Ihe retina but arising from only a small region at a given time. To achieve this goal efficiently, we propose an attentional shifting process that is divided into two major phases, as illustrated schematically and for one dimension in Fig. 5. En Ihe first phase, the objective is Io bring information from the desired attentional focus, whose retinal location is arbitrary, into alignment with a set of discrete blocks, or modules, that are situated at some intermediate processing stage (Fig. 5A). This alignment can be achieved by a series of "microshifts." similar to those postulated for stereo and motion (••.' Fig. 2). which map the attentional focus onto Ihe nearest cortical module. In the second phase, the objective is to link the appropriate module to a higher center, of which 1
786
HTTENTIONflL FOCUS
B
tlODULfS Ftc. 5. Ml A strategy for shifting the (ocus of attention. A sequence of microshifts {cf. Fig. 21 aligns the attentional focus wilh the nearest "altenlional module"; a series of macroshifrs direcis inputs from thai module to the appropriate higher center. {B) Details of rhe macros hi ft log circuitry, which allows a higher module lo select inputs from either of a pair of lower modulesthere are two shown at the top of the pyramid. This can be achieved by a series of "macroshifts." in which each module at a higher stage dynamically switches between a pair of modules at the next lower stage, using circuitry of the type illustrated in Fig. 5B. In this scheme, each cell in the upper module receives an excitatory input onto its left dendrite from the corresponding cell in the left module and an excitatory input onto itsrightdendrite from the corresponding cell in the right module. As with the microshifts. switching is controlled by a pair of inhibitory neurons, one that shunts all of the left dendrites of cells in the upper module and another that shunts all of the right dendrites in the module. The number of modules at each stage decreases by a factor of 2 (4 for an analogous two-dimensional scheme). This might continue until it reached a single "attention center" at the highest level. Alternatively, there might be divergence near the top. so lhal there is more than one center at the highest processing level (e.g.. two in Fig. 5,4). The focus of attention has been estimated to cover 3-10 degrees- when centered at parafoveal eccentricities (3-5 degrees) in humans 125). An equivalent size in the macaque monkey would conespond to 7000-15.000 RGCs from each eye, which is roughly 1% of the total ganglion cell population (17). Based on known cortical magnification factors (20, 26), an attentional focus of Ihis size would map onto a region 5-6 mm in diameter for VI. much greater than the dimensions of ocular dominance stripes (12). but onto a slightly smaller region in area V2, comparable to a full cycle of the stripe pattern revealed by cytochrome oxidase histochemistry (27). Thus, the compartroental organization of V2 might conceivably be linked lo the proposed modularity of the attentionshifting system. Physiological Tests for Shifter Circuits. The shifter hypolhesis leads to a set of specific predictions about the dynamics of receptive field locations for cells at particular stages of the visual pathway. The most novel proposition is that cells in 1
6301 layers 4C and 4A of V I , and possibly also in the LGN, should undergo outright receptive-field shifts when driven by stimulus patterns that provide an appropriate reference frame based on stereo, molion. or attentional cues. The range over which shifts occur should increase al successive stages, beyond layer 4C. the shifts should be many times the size of RGC receptive fields. Previous studies (28, 29) involving precise mapping of VI receptive fields in anesthetized, patalyzed monkeys indicate that receptive fields are, under some circumstances, stable to within a few minutes of arc. On the other hand, there are also hints that dynamic receptive-field shifts may indeed occur (29, 35). We do not consider the existing evidence to be compelling, either for or against the shifter hypothesis. In particular, we note that anesthesia might seriously disrupt cortical feedback control circuitry, thereby reducing or eliminating any dynamic shifting and possibly also increasing receptive-field sizes, by way of the blurring process described in an earlier section. A more decisive test would be to record simultaneously from cells at different levels of the putative shifter circuit in alert animals, so lhat alow-level cell could serve as a reference for eye position; the experiment should also use a visual stimulation paradigm that would modulate the shifting process over its full range and in a controlled fash ion. If shifter circuits are indeed present in the early visual pathway, traditional notions of neuronal receptive-fieId boundaries as static entities must be substantially modified. Receptive fields are typically plotted using a simple stimulus moving on a blank background. In the absence of a welldefined reference frame, Ihe mapping stimulus itself might tend to drive the shifting mechanism and thereby induce the receptive field to move dynamically along with the stimulus. If so. tbe classical receptive field would be considerably larger than the actual receptive-field size at any given instant (see Fig. 4). ln extrastriate cortex, evidence for dynamic modulation of receptive-fie Id size by attentional cues has been presented for area V4 (30) and for inferotemporal cortex (31) in monkeys. These findings are compatible with several possible mechanisms for selective attention (2. 3) and thus do not on their own constitute strong evidence for the shifter hypothesis. Also, we have nol dealt with the full complexity of attentive mechanisms, such as Ihe ability to shift attention along nonspatial dimensions (e.g.. for color). Thus, the shifter hypothesis is arguably more speculative and less fully developed for attention than for slereopsis and motion. Nonetheless, it emphasizes thai a relatively simple strategy may apply to a variety of dynamic routing problems in vision, Shifter circuits might also be relevant to other systems and processes, such as the rapid adjustment of visual and auditory maps in ihe primate superior colliculus (32) and Ihe much slower modulation of sensory maps in the somatosensory cortex after peripheral nerve lesions (33). More generally, the use of shunting inhibition to modulate a restricted subset of the connections onto a given neuron offers a powerful strategy that can be exploited for tasks of computation per se (10, 34). as well as for the routing strategies discussed in the present report.
787
We thank E r i c MJolsness and many other colleagues for valuable discussions, ideas, and suggestions. T h i s work was supported in part by contract N 00014-85 K-0O68 from the Office of Naval Research to D.C.V.E. 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29 30. 31. 33
Julesz. B- (1984) In Dynamic Aspects of Neacorlicuf Function (Wiley. New Vork). pp. 5B5-612. Crick. F . (1984) Proc. Natl. Acad Sci. USA SI, 45B6-4590 Koch. C . & Ullman, S , (1985) Hum. Neuroblol. A, 219-227. Moller, B . & Poggio, G . (1984) £ i p . Brain Res. 54, 304-314. Barlow, H . B . (1981) /Vac. fl. Soc. London Ser. B 211, 1-36 Burr, D C , & Ross, J . (19S6) Trends NeuroSci. 9, 304-307. Steinman. R M , Cushman. W. B & Marlins, A . J . (19B2) Hum. Nrurobiol. 1, 97-109. Poggio. C . & Poggio, T . (1984) Anna. Rev. Neurosci 1, 379¬ 412. Poggio. G . & Talbot, W. K . (1981) J. Physiol. 315, 469-492. K o c h . C Poggio. T . & Torre, V . (1983) Proc. Natl. Acad. Sci. USA BO, 2799-2802. Mitchison. G . J . & M c K e e . S , P. (1985) Nature iLondan)i\S, 402-104. Hubel, D. H. & Wiesel. T . N. (1977) Proc. R. Soc. London Ser. B 198, 1-59 Blasdel, G. G . & Fitzpatrick. D. J . (1984) J. Neurosci. 4, 880-895. O ' K u s k y . j Si Colonnier. M. (1982) J. Comp. Neurol. 210, 278-290. Fitzpatrick. D. S . . Lund. J . S. & Blasdel, G . G . (1985) J Neurosci. 5, 3329-3349. Blasdel, G . G . . Lund, J . S . & Fitzpatrick. D. (1985) J. Neurosci. 5, 3350-3369. Perry. N. H . & Cowey. A . (1985) Vision Res. 25. 1795-1810. Sherman. S . M. & K o c h , C . (1986) E x p . Brain Res 63, 1-20 Kendrickson, A . E „ Hum. S . P. & W u , J . - Y . (1981) Nalure (London) 192, 605-607. Van E s s e n . D. C . Ncwsome. W T . & Maunsell, j , H . R (1984) Vision Res. 24, 429-448. Barlow. H . B. (1979) Nature (London) 279, 189-190. Wcslhcimcr, G . & McKee, S . P. (1975) J. Opt. Soc. Am. 65, 847-850. Rosenfeld. A - 11984) Mulliresolution Image Processing and Analysis (Springer. New York). Bergen, J R. & Julesz. B . (1983) Nature ILondon) 303. 696¬ 698 Sagi, D. SL Julesz, B . (1985) Nature (London) 321, 693-695. Gattass. R.. Gross. C . G . & Sandcll. J . H . (1981) J. Crimp. Neurol. 201, 519-539. TooTell, R B. H . , Silverman. M. S - , DeValois, R G . A J a cobs. G . H . (1983) Science 220, 737-739. DeValois, R. L . Albrecht. D. G . & Thorell. L . G . (1982) Vision Res. 22, 545-559, Parker, A. & Hawken. M (1985) J. Opt. Soc. Am. 2, 1101¬ 1114. Moran. J . & Desimone, R. (1985) Science 229, 782-784, Richmond, B . J . . Wurtz. R. H , & Sato. T - (1983) J. Neurophysiol. SO, 1415-1432. Jay. M. F . & Sparks. D. L . (1984) Nulure (London) 309, 345-347.
33
Kaas, J . H . . Merzenich, M. M. & Killackey, H . P. (1983) Anna. Rev. Neurosci. 6, 325-350,
34.
Anderson, C . H . & Abrahams, E . (1987) Proceeding; of Ihe First IEEE International Conference on Neural Networks, in press.
35.
Motter. B. C . & Poggio, G , F. (1982) Soc. Neurosci. 707.
Abstr. 8,
Reprinted from Proc. tJatL Acad. Sci. USA. Vol. SB. pp. 6462^466, 1991
Olfactory computation and object perception (neural network/learnlng/sensory systems? J.
J.
HOPFIELD
Divisions of Chemistry and Biology, California Instilute Q [ j:.. - • ulogv, Pisadcna, CA 91325 Coturibnttd
by J. J. Hopfield, April tt,
1991
ABSTRACT Animals (hat are primarily dependent on olfaction must obtain a description of the spatial location and, the individual odor quality of environment!!] odor sources through olfaction alone. The variable nature of turbulent air flow makes such a remote sensing problem solvable if the animal can make use of (he Information conveyed by the fluctuation with lime of the mixture of odor sources. Behavioral evidence suggests that such analysis takes place. An adaptive network can solve the essential problem, Isolating the quality and Intensity of the components within a mixture or several individual unknown odor sources. The network structure Is an idealization of olfactory bulb circuitry. The dynamics of syn» apse change is essential lo the compulation. The synaptic variables themselves contain Information needed by bigher processing centers. The use of (he same axons to convey intensity information and quality information requires limecoding of information. Covariation defines an individual odor source (object), and this may have a parallel In vision.
Humans rely chiefly on vision for their description of the world around them. As a result, most of the olfactory psychophysics and electrophysiology literature is chiefly concerned with the question of identifying or analyzing a single odor presented to the nose. Such studies ignore deeper questions of the function of olfaction in highly olfactory animals, which is to define and locate individual odor sources in a complex environment. This computational problem is also posed by vision. The visual system must transform detailed retinal images into a much smaller amount of significant information describing physical objects and their locations. In a natural environment, vision allows us to parse our environment into objects even when the objects are unfamiliar and many objects are simultaneously present. Consider an animal that has only an olfaclory sense. To what extent can it solve the remote sensing problem, understanding the nature and Location of distant objects, particularly when the objects are not familiar? In the visual sysiem, analysis of the computational problem provided by the data and the task led to an understanding of what the "early" visual system musl compute (1). I describe here a similar problem for ihe early processing in the olfactory system. A description of a natural olfactory environment illustrates the one important problem the earliest processing of an olfactory sysiem must solve if the system is lo be able lo define and localize unknown odor sources in a complex odor environment. Behavioral evidence suggests that animals can solve this problem. This problem has a mathematical solution, which can be implemented by an elementary learning network. Dynamic changes of the synaptic strengths are essential lo this solution. Part of the information needed by higher processing areas is explicitly contained in ihe connection strengihs themselves. The last two sections present
simulations of these ideas and discuss their relation to olfaclory physiology and psychophysics. The Olfactory Environment and Task In most olfactory environments, the simple diffusion of odorant molecules is a negligible means of dispersing odorants. Odors of distant objects are brought to the nose by wind. Odorant molecules leaving the object follow the path of the air packet to which they are added. This packet already contains odorant molecules from other objects further upwind. The packet will move with the local wind, mixing slowly with other nearby packets containing odors from other objects, due to microlurbulence in the air. Local winds fluctuate markedly in bolh magnitude and direction. As a result, the odor plume has a complex spatial struclure (2) and is increasingly mixed with odors from other parts of the environment as time increases. Thus, the stimulus ai the nose due to distant objects contains mixtures of odors from many sources, whose relative contributions are constantly changing. Physical studies with a single odor source and detector in a natural environment verify these ideas (3). (Because velocities and time scales of turbulence are very differeni for water and air, aquatic environments are not necessarily similar to atmospheric environments.) Highly olfactory animals need to be able lo understand ihe location and odor properties of the various sources or "odor objects" in their space in order to hunt for food or flee from danger When a single odor source is present, the problem is simple. The perceived odor quality (i.e., the relative strengths of different components of an odor] is a fixed property of ihis sole source. The strength of the odor will fluctuate with wind direction. For an object that is near, the odor will be strong only for a narrow range of wind direct ions* and when the wind shifts from thai direction the odor will quickly grow weak. An object that is further away has an odor plume that is more contorted and on average is also broader and weaker. Its odor strength will correlate less well with the local wind direction. The relative time scale of fluctuation also contains information about distance. Thus, when a single odor source is present approximately upwind, information aboul the direction and even the approximate distance of the object is available to a stationary olfactory animal. The essential computation is an analysis of the flucluations of i h : odor intensity with time and the relation of these to fluctuations in Ihe local wind direction. The problem of one odor object is trivia] compared to the real problem of many unknown objects. Except when sniffing an object at the nose, an animal is always rn an environment of mixed and changing odor patterns. If each odor object were to stimulate a different set of receptor cells in the sensory epithelium, then the problem of separating objects would be simple. However, physiological studies of the vertebrate olfactory system generally indicate broadly tuned receptor cells and the excitation of a large number of cells by a given odor, so odors of different objects are believed to be discriminated on the basis of the patterns of excitation, as
The publicalion coslsof [his article were defrayed in part by page charge pay mem. This article rami therefore he hereby marked -odveniscrntni" in accordance wiirt 18 U . S . C . 61734 solely la indicate This fact.
788
H63 recently reviewed by Kauer (4). Similar conclusions are reached through behavioral studies (5. 6). Broad tuning makes the problem of separating unknown objects in natural environments appear very difficult. Suppose lhal it is possible to analyze the fluctuations of the instantaneous stimulation pattern in such a way lhat the contributions of different odor sources to the mixture can be separated. By nest analyzing the fluctuations of each odor object intensity in conjunction with fluctuations in the local wind direction, the animal will generate an excellent representation of the location and isolated odor of each odor source in its vicinity. This paper investigates the hypothesis that the function of the earliest pan of olfactory processing in highly olfactory animals is to accomplish Ihis first separation task—namely, to analyze the fluctuations of multisource environments into individual odors.
problem in the most general case, in the usual case they do contain enough information. A model of the olfactory world and ils processing can be described as follows. There are N different types of olfaclory processing cells receiving inpuis from sensory cells. Any particular odor k can be described by a vector S whose components S , describe the strength with which that odor excites the olfaclory processing cells of type n when odor k is present at a standard concentration. When odor k alone is present, the input to a cell of [ype n is then given by ir
k
; „ « - a,(r)Ji,
n-1.2
N
[1]
where oi(r) describes ihe time-dependent concentration of odor k at the sensory epilhelium. The details of olfactory receptors and connections from sensory cells to the processing cells are implicitty contained in the matrix 5 „. The function a U) does not depend on n. A single odor source will, by definition, have all sources of odorant molecules co located in space, so all [he components of the odor will have the same lime dependence a (l) (or if it is extended in space, ihey will have the same odor quality at each poini). t
Behavioral Evidence for Fluctuation Analysis
k
Many animals are able to search for a known odor such as a favorite food. In concept, [he animal computes a projeclion of the ambient odor input against the known template and thus measures the intensity of the target odor present- If the environment has a substantial odor, whose excitation pattern strongly overlaps the pattern of the search template (but with different relative amplitudes), a simple template match to look for a weak target odor becomes impossible. The environment itself will always appear to have an appreciable component of the target odor. A fixed environment could be removed by subtractivc adaptation, bul realistic environments reflect constantly changing strengths and mixtures of odors from a variety of sources. That an animal with a broadly tuned set of sensory cells can locate a weak known odor source indicates that an analysis of fluctuations is being carried out to provide information not available from elementary processing.
k
When multiple odor objects are present, the timedependent input to the sensory cells will be the sum of E q . I over K different odors k. I. m . . . . The nervous sysiem measures the various /„(/}. An ideal olfactory sysiem would be able lo determine both the vectors S, and the intensity lime dependences a,|r) for all the odor objects sampled by the fluctuating wind pattern. To what extent is il possible to determine Ihese quantities from the measurements available? Consider Ihe case of two odors and two processing cells. The scale of the odor vectors is arbitrary, as is the scale of Ihe functions . j i n . Each odor vector Sj and Sj has two components, one of which can in each case be picked as "1" to define the arbitrary scale. The two odor vectors can therefore be taken to be L
Experiments in Umax maximus have demonstrated the importance of fluctuations for odor learning. Hopfield and Gelperin (7) carried out learning experiments on mixtures of [wo food odors A and B. Slugs were aversively conditioned while on a mesh 1 cm above a piece of filler paper painled with a solution containing a 1:1 mixture of odoranls A and B . When later tesled, the animals were found to be aversively conditioned to the 1:1 mixture A B . but they had the same nonaversive behavior as control animals to the individual odors A and B. These experiments indicate that the mixed odor object A B is perceived as an odorant distinct from A or B . When different animals were trained with the same protocol, except that the initial conditioning was done wilh alternating siripes of A and B 0.9 cm apart, ihe slugs exhibited aversive behavior to A alone, B alone, and the mixture A B .
Odor 1 Ihus excites processing cell 2 with strength a relaiive to ils excitation of cell 1. The components a and b will be assumed to be positive, although ihis is nol essential. From Eq.l W ) - fliUf * a z W
W
• "ll'to * oiM-
[31
At any single lime r. [he measurement of lit • yields two equations in four unknowns—namely, a U ) . Qiii). a, and b. They cannot be determined from two equations. Adding more sampling times does not help, for each new sampling time t introduces two more equaiions and two more unknowns, ttitf) and OJ(I'). The problem is unsolvable in the general case. L
The distance from the odor source to the slugs was always greater than Ihe separation between the siripes. The difFusional mixing time for stripe sources of this separation is "1 sec. Thus, the odor available to the animals must always be somewhat mixed and fluctuating due to thermally induced air flows within the closed chamber. Since conditioning wilh odor stripes results in aversive response to the individual odors A and B (as well as to the mixture AB), Ihe existence of the individual odors A and B was presumably deduced from odor fluctuations. The crucial difference between the two experiments was the presence of fluctuations in the relative amount of A and B presented by the stripes but not by the 1:1 A B mixture, which allowed the animals to come to differeni conclusions about [he odor objects present.
There are. fortunately, a variety of circumstances under which the functions u lr) are Less lhan completely general and which lead lo an ability to solve for all the desired information. The simplest of these is the case in which the fluctua. lions with time of one odor are nol correlated with the fluctuations of ihe other odor. Such a description could be appropriate when the turbulence and wind fluctuations bringing one odor to the nose are independent of those lhat bring another odor via a rather differeni path. Under these circumstances, useful expectation values over these fluctuations can be computed by time averaging, denoted by ( ). (Neurobiology can evaluate such expectation values through learning and adaptation procedures.) The first moments obey k
Do Fluctuations Contain Enough Information? The following analysis shows that while fluctuations do not contain enough information lo solve the source separation
(/|) = (ai> + Wfli)
789
W - otot) + .
[41
(AM
Herault and Jutten (21) for decorrelating signals in an engineering context. The equations of motion of the cell potentials u„ are
The quantities on Ihe left are measurable lime averages, while ibe two expectation values and the two parameters on tbe right are unknown. Similarly, time averages can be taken of higher moments, as
dujdt
2
0
B
[5J
o
These three equations contain two new parameters, (a)) and {al). Third momenls yield (/!> - (a\) + 36 + IbMM)
+ °M)
a!
(I\ x /,) - a{a]) * (2uo + l](a,)(fl!)
dT /dt= ni
+ (26 + ni. )W(oi> + '' (fli> 2
I
2
+ (l + 2 l( )(o|l+-6(fli) flD
fll
tfj) - a (ai) + 3 ( i)(fli) + 3a(aj) + (aj). fl
5
n k
t
+ / {f), n
[7]
+fu xfu [& n
t
+ *lfrt--yMl
[81
The are a high-pass filtered version of the potentials u . [Il is assumed lhat the synapse change mechanism can filter out ihe average potential by simple chemical means, just as the chemosensory sysiem of Escherichia coti is sensitive to concentration changes, not absolute concentration (9, 10). Alternatively, the average of the input variables could be removed by the sensory cells themselves.] The synapse change rule is of the Hebbian type, involving only variables of the pre- and postsynaptic neurons. The idea behind this structure of network and learning rule can be seen from examining the case of two odors and two neurons, and the simplification T is small. Suppose the synaptic matrix is such lhat the first column of ( E / T + T) is the vector Si, and the second column is the vector Sj. ( E is the identity matrix.) Il directly follows from Eq. 7 that w,1fj " aid) and ivj(r) = o (r), and all other u (f) will vanish. If a learning rule could be made lo converge to this T, the information about the relative components of a particular odor will occur as a column in the T matrix, while the information about the fluctualions in the intensity of the individual odor objects will be faithfully conveyed by the ••':'.•}. (The choice of which neuron will represent the strength of which odor is made by the operation of the Learning rule; in simple cases, it selects the neuron most strongly driven by a particular odorj Thus, the network has a structure that allows it to solve the problem if the Learning rule can cause the T matrix to take on the desired form. The learning rule of Eq. 8 has been chosen, consistent with a linear analysis of the stability of the desired solution for T for the positive asymmetry expected for The fluctuations. This linear analysis guarantees that if T is initially roughly correcl. the Learning rule will lead to the correct matrix forTMany other forms of modification rules have this property, including ihe simpler case y - i : Without the term in C, the learning rule is ihe same as that used by Barlow and Foldiak Cll) and Foldiak (12) to decouple two mixed Gaussian communication channels by minimizing their covariance. (In the present case, merely minimizing covariance cannot generate the desired solution, since minimizing the covariance results in a symmetric matrix for T. while the desired result is not symmetric-) A non-Gaussian mixed channel problem has been examined by Herault and Jutten (21). The olfactory problem is simpler lhan their case in that the third moment of the distribution will be nonzero. Olfaction is made complicated by the faci that the synapse strengths have important meaning, which needs to be available to higher processing centers. The approach is related in spirit io ihe work of von der Malsburg and Bienenstock (13), who have emphasized the role of fast synaptic modification as a computational element in biology, and to novelty filters (14). Further mathemaiical analysis suggested that the Learning rule should be adequate lo generate the desired T even without initial p
(/, x l\i m afoj) + (2a + fl'h)(fli>(fl }
3
- Zr n
The resting potential of the cell In the absence of external or synaptic inpul has for convenience been chosen as zero. With the sign convention of Eq. 7, all '/'., will be positive, Eq, 7 describes the evolution of the cell potentials due to the sensory input and Ihe present values of the synaptic connection strengths. However, the synapses also change with lime. Eq. 8 describes Ihe learning rule for the inhibitory synapses, where 5, y. and e are constants determining Ehe learning speed
«1> - « + 2fl{a,>(oj) + (aj) (J, x /,) = „ { „ ] ) + (1 + i.)( ,)( ,) + b>a\).
= -ujr
[S]
a
These four equations contain two new parameters {a\) and (o!>. Eqs. 4-6 overde tannine all unknowns, including parameters a and b. Their knowledge allows the computation of the individual odor fluctuations i and ..••n :• from |(r). The analysis works in The same fashion for a larger number of neurons and odors as long as N a K. The odor fluctuations have been assumed to be asymmetric about the mean because of the large physical difference between the causes of maximal upward and downward odor fluctuations. A steady wind in the wrong direction produces essentially no odor from some particular source, while the largest odors from that source occur when the wind is roughly in ihe correct direction and there is a chance favorable fluctuation of ihe microturbulence. A positive third moment aboul the mean is therefore to be expecled as the distribution has longer Tails in the positive direction. With one reasonable supposition—namely, the statistical independence of the fluctuating contributions of two separate objects to the composite odor—the hopeless problem of identifying ihe contributions of unknown fluctuating odors becomes solvable. Alternative suppositions, such as an assumed parametrized functional form for the fluctuations or differences in the time siruciure of the fluctuations from the two sources, could also serve as the basis for separating the two odors. Approximate algorithms involving usual-case suppositions are common ia vision where, for example, the perception of a three-dimensional surface from monocular intensity information (shape-from-shading) is based on the assumption of a single localized illumination source (B), b
Solving the Problem by Adaptation In a Simple Network A set of interacting processing neurons is described with individual neurons denoted by a subscript. The dendrites of each neuron n receive direct or indirect time-varying input tjj) from sensory cells. The processing cells will be assumed to be compact {small compared with ihe electroionic length), so a cell potential u„U) is the same throughout ils dendritic arborization. The processing cells make inhibilory dendrodendritic synapses with each other. '>"„.. denotes the strength of the inhibilory synapse from neuron k to neuron n. This circuit description can be viewed as an idealization of the structure of the olfactory bulb (see Discussion). Il is also an instantiation of the kind of network equations described by >
790
3
t
information concerning the desired solution, so a simulation was carried o u i to examine this question.
the same time, the relative components o f the individual odors (ihe odor quality) have been imprinted in the matrix T.
Simulations
Discussion
A computer simulation o f the model (Eos. 7 and 8) has been carried out for a network o f six processing neurons. T w o odors were used, each having fluctuations of random duration and random amplitude above a base line, with an asymmetry between upward and downward fluctuations. The model intensities a ( i ) and a ( ( ) are shown at the lop left o f Fig. 1. The net odor was composed according lo Eq. 1. with odor A components chosen as ( 4 . 7 . 5. 2. 8. and 10) and odor B components were (7. 3. 10. K. 4. and 1). The connection matrix was initially set to zero. The bottom six traces on the left in Fig. 1 show the potentials « „ ( ' ) o f the six processing neurons in the presence o f the odor mixture. Initially, the fluctuations o f both odors appear in each channel. After synaptic learning is turned on (for clarity, the time before I ~ 20 was run without synaptic modification) the outpul patlem in the network changes. T changes so lhat eventually only channels 3 and 6 remain with any ouiput at long limes. As shown, ihe outpul o f channel 3 then follows the strength o f odor B. while the output o f channel 6 follows A . A l I - 200 Ihe synaptic matrix 7"„ (with diagonal entries zero) has become
I have shown that an adaptive neural network with changing synaptic connections can solve the essential problem in olfaclory object perception—namely, using an analysis of fluctuations to deduce the individual odors in a mixed odor environment. An olfactory sysiem that can solve this problem is capable of discerning the nature and location o f odor objects in a complicated and ever changing environment, which is esseniial for the olfactory abilities displayed by highly olfaclory land animals. This change in the pattern of synapses due to the fluctuations can be thought of as a process of network adaptation to the present environment.
A
8
The convenlional connectionist viewpoint considers the synaptic connections as an algorithm thai processes information. Although the connections may be formed through learning-induced adaptation, their usual importance is the transformation performed hy the connections on The signals flowing through them. The actual connections or architecture that performs the computation are not viewed as being of primary interest.
k
0.00
—
0.02 0.04 0.02 0.02 0.03
0.69 0.30
0.01
—
0.03 0.01 0.04
o .oo 0.00
0.79 0.40 0.11
0.02 0.02
—
0.01
—
0.01 0.01 0.01 0.00 —
0 il]
The way in which this network uses changing connections leads to an entirely different way of thinking about Ihe connections (see also ref. 13). There are two separate parts of the desired information, one having to do with discerning the varying strength o f an individual odor object, and the other having lo do with discerning the odor quality. The adaptive network produces an activity signal representing the separate strengths o f the multiple odors. The knowledge o f the components o f these odors becomes embedded (after learning) in the strengths of the connections. Thus, the connection strengths are not merely algorithm, ihey are also essential data for understanding the environment. A complete system must have an ability to query the connection strengths in order to perceive (after adaptation) the odor quality o f a particular source. This can be done by a procedure that queries a particular cell about the strengths o f its synapses.
0.33 0.66 0.53 0.13 0.76 —
The entries in column 3 above closely approximate the 0.7, 0. 3, — i 0.8, 0.4, 0.1 pattern of the strengths of components 1, 2. 4, 5, and 6 of odor B relative to the strength o f component 3 of odor B. Column 6 is similarly close to0.4.0.7. 0.5, 0.2, 0.8, — , ihe relative strengths o f components 1, 2, 3, 4. and 5 o f odor A relative to 6. The network has developed a connection matrix that solves the problem and that makes two o f the neurons into cells, each of which responds only to the fluctuations o f a corresponding single odor intensity. A t
T
*s
Tsi
V U
* ^ w .
.. .
:« . .. TIME
. TIME
791
Pic, 1, A simulation for sin dimensions ot odor und iwo independent source s. The in [entities Di" the two sources are given in Ihe lop two traces on the left. Synapse change begins al r ^ 20 sec. The time scale of the average odor fluctuation is, 3 sec; r 0.01 set. The bottom six traces on the left are the dendritic potentials Pi of the processing neurons. After MOO sec oi learning-induced network adaptation, the two that still have sizable output reflect the network view of the fluctuating intensities cA the two odor objects present. During this lime period, the synapses also change into a pattern that reflects the components of the individual odors. The strengths, of iht synapses of neuron 3, which hat captured the response lo odor B. are displayed on the right. The relative strengths of the actual odor components of this odor object are shown by the horizontal bars. The implicit strength of component 3 is unity. The synapse learning algorithm had y = 1 for mathematical simplicity and was augmented by an exponential forgetting, which occurs when the presynaptic neuron is silent.
6466 After network adaptation, a particular set of N-l synapses lake on values that describe the relative strength of the components of a particular odor. These synapses are all associated with the particular cell that now responds to Ihe strength of thai particular odor. (While Ihe simple network generates a temporary "grandmother cell" representation of odor intensity, the concept described can be embedded in a distributed representation more appropriate to neurobiology.) Thus, tofindout the odor quality of a particular odor, the system needs to send a signal lo the dendrite of the principal neuron that is reporting out the strength of that odor. The signals back from other processing cells will then correspond lo ihe relative strengths of that odor. Only one odor quality can be queried at once, for the same set of axons is being used to report the quality for each particular odor. This querying process must be made in a way that distinguishes activity generated by the query from the ongoing processing of odors. One possible answer is to multiplex a second signal on The same sel of output axons. Since Ihe intensiTies only vary slowly, information about Them can be carried on Ihe axons as a low frequency or "DC" response, while a high-frequency signal can be used to query the synaptic strengths and to carry That information on Ihe same axons. It should be noted lhat the DC information carried by the outgoing axons changes greatly during adaptation. Before adaptation, it represents the total odor quality, while after adaptation it represenls the information aboul the lime fluctuation of the intensily of various odor objects. If a computation of the son described is to be carried out by a nervous system, iI musl be performed before (here are major nonlinear transformations of input signals. In the mammalian olfacTOry system, the glomeruli [each of which is innervated by 1000 sensory cells (15)] may provide reliable pooled information from sensory cells. Eq. 1 potentially describes Ihe response of a glometulus to odors. The nexi stage of processing would then be used for a decomposition compuTaTion. while signals are sTill reasonably linear. In The mammalian system, the secondary mitral cells of the olfactory bulb are each innervated by a single glomerulus and could correspond to the processing cells of Ihe model. (It is nol clear whether mitral and Tufted secondary neurons should have fundamentally different computational roles; for example, one could be involved in adaptive processing, white ihe other could be used to convey odor quality o r the sensory stimulus ilself.) The model makes use of inhibilory interactions belween Ihe mitral cells. Real mitral cells are excitatory, bul the dominant synapse type in The olfactory bulb is a reciprocal dendro-dendrilic connection between a mitral cell and an inhibitory axonless granular eel] (15). This local circuit produces an effectively inhibitory dendro-dendritic connection belween mitral cells. Dendro-dendritic computation is preferable To using action potentials, since linear processing is desired. The anatomical elements essential for the bulb io carry out the anticipated computation exist. (The inhibilory periglomerular cells could play a similar role in dendro-dendritic processing.) These general features seen in the vertebrate olfactoty system (151 are also present in highly olfactory teiTestrial mollusks (16¬ 18). The one essential additional property is lhat the efficacy of the dendro-dendritic connect ions be appropriately modified during behavior, either by direct change (e.g., Hebbian learning! or indirectly via descending pathways. Synaptic modifications in The olfactory bulb would be Ihe basis of The shortterm memory of Ihe present environment of several odor objects. Long-term memory of odors, and The faciliTy for Ihe comparison of present and dislant past odors, would lie in higher areas. Within The present model, the adaptation of the network is lo odor quality and is not dependent (presuming linearity) on Ihe slrenglh o f the odors. When only one odor is present, most of the neurons become silent and are capable of
792
responding io new odors bul nol lo (he fluctuations in intensity of the adapted odor. Recent experiments (22) suggest the modifiability of synapses in the accessory olfactory bulb, closely related in both anatomy and function, on a time scale of several minutes. The separation of the individual odors by such a network is based on the assumption of independent fluctuations of the contributions of differeni odor objecis. This will nol be universally correct, and when Ihis supposition is violated, an "olfactory illusion" should occur, [n the case of Ihe Umax A-B stripe experiment, fluctuations can take place involving diluting A with pure air. or diluting B. or exchanging A forB. or mixed AB for A, etc. With iwo such nearby sources, some correlation of fluctuations will be inevitable. A variety of effects can be seen in simulations involving partial correlation, depending on the similarity of the two objects and Ihe model of the correlation. These have ranged from a failure to construct objects al all to the percepiion of a phantom odor object lying between A and B. The A-B stripe experiment suggests the presence of an olfactory illusion in Umax. In the presence of fluctuating olfactory signals, correlations define what is a single odor object, and the dynamically changing synapses manage to capture information about individual objects. Visual processing could have a similar way of using correlation for understanding moving objects, and the oscillatory behavior due to signals from differeni parts of a moving stimulus (19) might reflecl a mechanism to probe synaptic strengths set up by correlated motions. Such processing would require very rapid synapse changes (20) in the visual system. The author thanks A. Gelperin, J. F. Hopfield, and D. W. Tank for many significant discussions. This research was supported in pan by the Ronald and Maxine Linde Fund. 1. Marr. D. (1982) Vision (Freeman, San Franciscol. 2. Wright, R. H. (1964) The Science of Smell (Basic Books. New York), pp. 16-24. 3. Murlis, J S Jones, C. O. (1981) Physiol. Entomol. t, 71-86 4 Kaucr. J. S. (1991) Trends Neurosci. 14, 79-85 5. Derby. C. D., Oirardol, M.-N., Daniel, P. C. 4 Fine-Levy, J. B. (1989) in Percepiion of Complex Odors und Tastes, eds. Laing, D. C. Cain. W. 5.. McBride. R. L. St Acht, B W (Academic. New York), p. 80. 6. Neuhaus, M. (19S7) Z. Vet. Physiol 39, 624-633. 7. Hopfield. J. F. 4 Gelperin. A. 11989) Behav. Neurosci 103. 329-333. 8. Horn. B. K. P. (197J) in The Psychology of Computer Vision. ed. Winston. P. H. (Mcgraw-Hill. New York), pp. 115-155. 9. Macnab. R 4 Koshland. D. E.. Jr. (1972) Proc. Null. Acad. Sci. USA 69. 2509-251!. 10. Brown, D. A. 4 Berg. H. C. (19741 Pro,: Nail. Acad. Sci. USA 71, 1388-139!. 11 Barlow. H. B 4 Foldiak. P. (19891 in The Computing Neuron. eds. Miall. C, Durbin. R. M. & Milcheson, G. J. (Addison. Wesley. New Yorkl. pp. J4-72. 12. FSIdiak. P. (1989) Proc. IEEE/INNS Int. Joint Conf. Neural Net. I. 401. 13. von der Malsburg. C. 4 Bienenstock, E. (1985) in Disordered Systems and Biological Orguniutlion. eds. Bienenstock. E.. Fogelman Soulie, F. St Weisbuch. G. (Springer. Berlin). !47251. 14. Kohonen. T. (1988) Self Organisation and Associative Memory (Springer. Berlin), pp. 112-118. 15. Shepherd. G. M. (1979) The Synaptic Organization of Bruin (Oxford Univ. Press. New York), pp. 152-174. 16. Chase. R. (1986) Chem. Senses 11, 411-426. 17 Gelperin. A. (1974) Prat. Willi. Acad. Sci. USA 71, 966-910 18 Sahley, C. L.. Rudy. I W 4 Gelperin. A. (19S1) J. Camp Physiol. 144, 1-!.
19. 20 21. 22.
Gray. C. 4 Singer, W. (1989) Nalure (London) 33%, 334-331. Crick, F. (1982) Trends Neurosci. 5, 44-66. Hcraull. J. & Jutten, C. (1986| AIP Conf. Proc. 151, 206-211. Brennan, P.. Kaba, H. St Keverene, E. B. (19901 Science ISO, 1223-132*.
Reprinted with permission from Tht Brain. pp. S99-9I0. IWO © 1990 Cold Spring Harbot Laboratory Press
A Theory of How
the Brain Might Work T. Poccio
IM.S.T., Massachusetts
Pooo. 38100 Trtnto. Italy; Thinking Machines Co.. Artificial Intelligence Laboratory. Cambridge. 02142; Center far Biological Information Processing. Massachusetts Institute of Technology. Cambridge. Massachusetts 02139
I wish to propose a quite speculative new version of [he grandmother cell theory to explain how the brain, or parts of it, may work, ln particular. 1 discuss how the visual system may learn to recognize three-dimensional objects. The model would apply directly to the cortical cells involved in visual face recognition. 1 also outline the relationship of our theory to existing models of the cerebellum and of motor control. Specific biophysical mechanisms can be readily suggested as part of a basic type of neural circuitry that can learn to approximate multidimensional inpul^output mappings from sets of examples and that is expected to be replicated in differeni regions of the brain and across modalities The main points of (he theory are: 1. The brain uses modules for multivariate function approximation as basic components of several of ils information processing subsystems. 2. These modules are realized as HyperBF networks (Poggio and Girosi 1990a.b) 3. HyperBF networks can be implemented in terms of biologically plausible mechanisms and circuitry. The theory predicts a specific type of population coding lhal represents an extension of schemes such as look-up tables. 1 conclude with some speculations about the trade-off between memory and compulation and the evolution of intelligence.
[.
THE
GRANDMOTHER NEURON
THEORY
A classic theme in the neurophysiological literature, at least since the work of Hubel and Wiesel (1%2), is the idea of information processing in the brain as leading lo "grandmother" neurons responding selectively to the precise combination of visual features lhat are associated with one's grandmother. Even when not explicitly slated, Ihis notion seems to capture how many neuroscienlisls believe that the brain works. The grandmother neuron theory is of course nol restricted to vision and applies as well to other sensory modalities and even lo motor control under the form of cells corresponding lo elemental movements. Why is this idea so attractive? The idea is attractive because o( its simplicity: It replaces possibly complex information processing wilh Ihe superficially simpler task of accessing a memory. The problem of recognition and motor
793
control would be solved by simply accessing look-up tables containing appropriate descriptions of objects and of motor actions. The human brain can probably exploit a vast amount of memory with ils IO ' or so synapses, making attractive any scheme that replaces computation with memory. In the case of vision, Ihe apparent simplicity of this solution hides the difficult problems of an appropriate representation of an object and of how to extract i( from complex images. Even assuming that these problems of representalion, feature extraction, and segmentation could be solved by other mechanisms, a fundamental difficulty seems to be intrinsic to the grandmother cell idea. The difficulty consists of the combinatorial explosion in the number of cells that any scheme of the look-up table type would reasonably require for either vision or motor control. In the case of three-dimensional object recognition, for instance, there should be for each object as many entries in (he look-up table as there arc two-dimensional views of the object, in principle an infinite number! 1
The difficulty of a combinatorial explosion lies at the heart of theories of intelligence that attempt to replace information processing wilh look-up tables of precomputed results. In this paper, wc suggest a scheme lhat avoids the combinatorial problem, while retaining the attractive features of the look-up table. The basic idea is to use only a few entries and interpolate or approximate among them. A mathematical theory based on this idea leads to a powerful scheme of learning from examples lhat is equivalent lo a parallel network of simple processing elements. The scheme has an intriguingly simple implementation in terms of plausible biophysical mechanisms. We discuss in particular the case of three-dimensional object recognition but propose that the scheme is possibly used by the brain for several different information-processing tasks. Many information-processing problems can be represented as the composition of one or more multivariate functions that map an input signal inlo an outpul signal in a smooth way These modules could be synthesized from a sufficient set of input 'output pairs—the examples—hy the scheme described here Because ol the power and general applicability of this mechanism, we speculate that a part of the machinery of the brain, including perhaps some of [he cortical circuitry that is somewhal similar across [he differeni modalities, may be dedicated to Ihe lask of function approximation.
900 D. HOW TO SYNTHESIZE THROUGH LEARNING THE BASIC APPROXIMATION MODULE: REGULARIZATION NETWORKS This section describes a technique for synthesizing the approximation modules discussed above through learning from examples. Ifirstexplain how to rephrase the problem of learning from examples as a problem of approximating a multivariate function. The material in this section is from Poggio and Girosi (1989, 1990a,b), where more details can be found. To illustrate the connection, let us draw an analogy between learning an input/output mapping and a standard approximation problem, Iwo-dimensional surface reconstruction from sparse data points. Learning simply means collecting Ihe examples, i.e., the input coordinates i , , y,, and the corresponding output values at those locations, the heights of the surface d,. Generalization means estimating d al locations x, y where Ihere are no examples, i.e., no data. This requires interpolating or, more generally, approximating the surface (i.e.. the function) between the data points (interpolation is Ihe limit of approximation when there is no noise in Ihe data). In [his sense, learning is a problem of hypersurface reconstruction (Omohundro 1987; Poggio el al. 1988, 1989). From Ihis poinL of view, learning a smooth mapping from examples is clearly ill-posed, in the sense that (he information in the data is not sufficient to reconstruct uniquely the mapping at places where data are not available. In addition, the data are usually noisy, A priori assumptions about the mapping are needed to make the problem well-posed. One of the simplest assumptions is that the mapping is smooth: Small changes in the inputs cause a small change in the output. Techniques that exploit smoothness constraints in order lo transform an ill-posed problem into a wellposed one are well known under the term of regularization theory and have interesting Bayesian interpretations (Tikhinov and Arsenin 1977; Poggio et al. 1985; Bertero et al. 1988). We have recently shown lhat the solution lo the approximation problem given by regularization theory can be expressed in terms of a class of multilayer networks lhat we call regularization networks or Hyper Basis Functions (HyperBFs) (see Fig. 1). Our main result (Poggio and Girosi 1989) is that the regularization approach is equivalent Lo an expansion oi the solution in Terms of a certain class of functions: jV
fTx)=2c,G(x;f.) + pW
(1)
where G(K) is one such function and ihe coefficients c. satisfy a linear system of equations lhat depend on the jV "examples." i.e., the data to be approximated. The term p{x) is a polynomial that depends on the smoothness assumptions. In many cases, it is convenient to include up to the constant and linear terms. Under relatively broad assumptions, Ihe Green's function G is radial and therefore the approximating function becomes
794
a
b
Figure 1- (a) The basic learning module thai (we conjecture) is used by the brain for a number of tasks. The module learns to approximate a multivariate function from a set of examples (i.e.* a set of input/output pairs), (b) A HyperBF network equivalent to a module for approximating a scalar function of three variables from sparse and noisy data. The data, a siel of points where ihe value of the function is known, can be considered as examples to be used during learning. The hidden units evaluate the function G(x; t„) and a fixed, nonlinear, invertible function may be present after the summation. The units are in general fewer than the number of examples. The parameters thai are determined during learning are the coefficients c„, the centers t„ and the norm-weights W. In the radial case G = G ( | | K - t j j , ) and the hidden units simply compute the radial basis functions G at the "centers" t„. The radial basis functions may be regarded as matching the input vectors against the "templates" or "prototypes" that correspond to the centers (consider, for instance, a radial Gaussian around its center, which is a point in the A-dimensional space of inputs). There may be also connections computing the polynomial term of Fig. lb: Constant and linear terms (the dotted lines in n) may be expected in most cases. h
!
ixx)-fcM\\"-a )+pM i-t
(2)
which is a sum of radial functions, each with its center £, on a distinct data point and of constant and linear terms (from the polynomial, when restricted to be of degree one). The number of radial functions, and corresponding centers, is the same as the number of examples. Our derivation shows lhat the type of basis functions depends on the specific a priori assumption of smoothness. Depending on il, we obtain the Gaussian G(r) e , Ihe well-known "thin plale spline" G(r) - r' lnr, and other specific functions, radial and not. As observed by Broomhead and Lowe (1988) in Ihe radial ease, a superposition of functions like Equation 1 is equivalent to a network of the type shown in Figure lb. The interpretation of Equation 2 is simple: in the twodimensional case, for instance, the surface is approximated by the superposition of, say, several two-dimensional Gaussian distributions, each centered on one of the dala points. The network associated with Equation 2 can be made more genera! in lerms of Ihe following extension
901 (3) where Ihe parameters t„, which we call "centers," and the coefficients e are unknown, and arc in general many fewer than the dala points (nfiW). The norm is a weighted norm m
(4) where W is an unknown square matrix and the superscript T indicates the transpose. In the simple case of diagonal W, the diagonal elements » assign a specific weight to each input coordinate, determining in fact the units of measure and Ihe importance of each fealure (the matrix W is especially important in cases in which the input features arc of a different type and their relative importance is unknown). Equation 3 can be implemented by the network of Figure 1, Notice that a sigmoid function at the output may sometimes be useful without increasing the complexity of the system (see Poggio and Girosi 1989). Notice also that there could be more than one set of Green's functions, for instance, a set of multiquadrics and a set of Gaussians. each with its own W Notice that two or more sets of Gaussians, each with a diagonal W, are equivalent to sets of Gaussians with their own cs.
Learning Iterative methods can be used to find the optimal values of the various sets of parameters, the c„, the w „ and the t , that minimize an error functional on the set of examples. Steepest descent is the standard approach that requires calculations of derivatives. An even simpler method that does not require calculation of derivatives (suggested and found surprisingly efficient in preliminary work by B. Caprile and F. Girosi, pers. comm.) is to look for random changes (controlled in appropriate ways) in rhe parameter values that reduce the error. We define rhe error functional—also called energy—as a
ffir]-«.,.w-2vA,r* with A, y , - / * ( * ) = J \ - 2 s
* -1
The interpretation of the network of Figure I is as follows. After learning, the centers of the basis functions arc similar to prototypes, since they are points in the multidimensional input space Each unit computes a (weighted) distance of the inputs from its center, that is, a measure of their similarity, and applies to it the radial function. In the case of the Gaussian, a unit will have maximum activity when the new input exaclly matches its center. The output of the network is the linear superposition of the activities of all the basis functions in the network, plus direct, weighted connections from Ihe inputs (the linear terms of p|x]) and from a constant input (the constant term). Notice that in the limit case of the basis functions approximating delta functions, the system becomes equivalent to a look-up table. During learning, the weights c are found by minimizing a measure of [he error between the network's prediction and each of the examples, At the same lime, the centers of the radial functions and the weights in Ihe norm are also updated during learning. Moving the centers is equivalent to modifying Ihe corresponding prototypes and corresponds to task-dependent clustering. Finding the optimal weights W for the norm is equivalent to transforming appropriately, for instance scaling, the input coordinates and corresponds to task-dependcm dimensionality reduction. Regularization networks, of which HyperBFs are the most general and powerful version, represent a general framework for learning smooth mappings that rigorously connects approximation theory, generalized splines, and regularization with feedforward multilayer networks. They also contain as special cases the radial basis functions (RBF) technique (Micchelli 1986: Powell 1987; Broomhcad and Lowe 1988) and several wellknown algorithms, especially in the pattern recognition literature.
HL A PROPOSAL FOR A BIOLOGICAL IMPLEMENTATION In this section, we point out some remarkable properties of Gaussian HyperBF, which may have implications for neurobiology. Factorizable Radial Basis Functions
a = 1, -w
Interpretation of Ihe Network
c.c(||x. - i j i i )
In the first method, the values of c.. I., and W that minimize H[f] are regarded as the coordinates of the stable fixed point of the following dynamical system:
W=
where u is a parameter. The derivatives are rather complex (see Poggio and Girosi 1990a; and Notes section). The second meLhod is simpler: Random changes in the parameters are made and accepted if H[f\ decreases. Occasionally, changes that increase H\f\ may also be accepted (similarly to the Metropolis algorithm).
The synthesis of (weighted] RBFs in high dimensions may be easier if they are factorizable. It is easily seen that the only RBF which is factorizable is Ihe Gaussian
*H\n
795
902 of unils representing with their activity the position of features). Gaussians in higher dimensions can then be synthesized as products of one- and two-dimensional receptive fields. This scheme has three additional interesting features:
(with diagonal W). A multidimensional Gaussian function can be represented as the product of lower dimensional Gaussians. For instance, a two-dimensional Gaussian radial function centered in I can be written as | l
|
C(||x - i l & J - - " - ' i * ' = e
-<--:>'"°.>
e
e
-"-;i''"-',
(5) with or, = llw, and cr -\w , where w, and w, are the elements ol the matrix W assumed, in this section, to be diagonal. This dimensionality factorization is especially attractive from the physiological point of view, since it is difficult to imagine how neurons could compute G(||x-lJ| ). The scheme of Figure 2, on the other hand, is physiologically plausible. Gaussian radial functions in one, rwo, and possibly three dimensions can be implemented as receptive fields by weighted connections from the sensor arrays (or some retinotopic array l
1
!
1. The multidimensional radial functions are synthesized directly by appropriately weighted connections from the sensor arrays, without any need of an explicit computation of the norm and the exponential. 2. Two-dimensional Gaussians operating on the sensor array or on a retinotopic array of features extracted by some preprocessing transduce the implicit position of features in the array into a number (the activity of the unit). 3. Two-dimensional Gaussians acting on a retinotopic map can be regarded each as representing one twodimensional "feature," i.e., a component of the input vector, whereas each center represents the "template," resulting from the conjunction of those lower-dimensional features. Notice lhat in this analogy the RBF is the AND of several features and could also include the negation of certain features, thai is the AND NOT of them. W weights the importance of the different features.
Biophysical Mechanisms The network. The multiplication operation required by the previous interpretation of Gaussian GRBFs to perform the "conjunction" of Gaussian receptive fields is not too implausible from a biophysical point of view. It could be performed by several biophysical mechanisms (see Koch and Poggio 1987). Here we mention three mechanisms: 1. Inhibition of ihe silent type and related circuitry {see Poggio and Tone 1978; Torre and Poggio 1978) 2. The AND-like mechanism of NMDA receptors 3 A logarithmic transformation, followed by summation, followed by exponentiation. The logarithmic and exponential characteristic could be implemented in appropriate ranges by the sigmoid-like pre-topostsynaptic voltage transduction of many synapses.
Figure 2. A three-dimensional radial Gaussian implemented by multiplying two-dimensional Gaussian and one-dimension¬ al Gaussian receptivefields.The latter two functions are synthesized directly by appropriately weighted connections from Ihe sensor arrays, as neural receptive fields are usually thought to arise. Notice that they transduce the implicit position ol stimuli in the sensor array into a number (the activity of the unil). They thus serve the dual purpose of providing the required "number" representation from the activity of the sensor anay and of computing a Gaussian function. Twodimensional Gaussians acting on a retinotopic map can be regarded as representing two-dimensional "features," whereas the tadial basis function represents the "template" resulting from the conjunction of those lower-dimensional features.
796
If the first or the second mechanism is used, the product of Figure 3 can be performed directly on the dendritic tree of the neuron representing the corresponding radial function (alternatively, each dendritic tree may perform pairwise produces only, in which case a logarithmic number of cells would be required). The scheme also requires a certain amount of memory per basis unit, in order to store the center vector. In the case of Gaussian receptive fields used to synthesize Gaussian RBFs. the center vector is effectively stored in the position of the two-dimensional (or one-dimensional) receptive fields and in their connections to Ihe product unit(s). This is plausible physiologically. The linear terms (the direct connections from the inputs to the output in Fig. l)can be realized directly as
903
0. 1
0, 1 Figure 3. {a) The HyperBF network proposed for [he recognition of a three-dimensional ohjec[ from any of irs perspective views (Poggio and Edelman 1990). The network attempts to map any view (as defined in the text) into a standard view, arbitrarily chosen. The norm of the difference berween the output vector f and the standard view s is threstiolded to yield a 0,1 answer. The 2N inputs accommodate the input vector v representing an arbitrary view. Each of the K radial basis functions is initially centered on one of a subset of the M views used to synthesize the sysiem (K<M). During training, each of the M inputs in the training set is associated with the desired output, i.e.. Ihe standard view s. (b) A completely equivalent interpretation of a for the special case of Gaussian radial basis functions. Gaussian functions can be synthesized by multiplying the outputs of two-dimensional Gaussian receptive fields, that "look" at the retinotopic map of the object point features. The solid circles in the image plane represent the two-dimensional Gaussians associated with thefirstradial basis function, which represents thefirstview of the object. The dotted circles represent the two-dimensional receptivefieldsrhat synthesize the Gaussian radial function associated with another view. The two-dimensional Gaussian receptivefieldstransduce posilions of features, represented implicitly as aclivily in a retinotopic arTay, and their product "computes" the radial function without the need of calculating norms and exponentials explicitly. (Reprinted, with permission, from Poggio and Girosi 1990b.)
inputs to the output neuron that summates linearly its synaptic inputs (an output nonlinearity is allowed and will not change the basic form of the model, see Poggio and Girosi 1989). They may also be realized through intermediate linear units. Mechanisms for learning. Do the update schemes have a physiologically plausible implementation? Consider first the steepest descent methods, which require
797
derivatives. Equation 6 or a somewhat similar, quasiHebbian scheme is not too unlikely and may require only a small amount of neural circuitry. Equation 7 seems more difficult to implement for a network of real neurons. Methods such as the random descent method, which do not require calculation of derivatives, are biologically much more plausible and seem to perform very well in preliminary experiments. In the Gaussian case, with
90-4 basis functions synthesized through the product of Gaussian receptive fields, moving Ihe centers means establishing or erasing connections to the product unit. A similar argumenl can be made also about ihe learning of the matrix W. Notice that in the diagonal Gaussian case, the parameters lo be changed are exactly the a of the Gaussians. i.e., the spread of the associated receptive fields. Notice also that Ihe a for all centers on one particular dimension is the same, suggesting that the learning of ni may involve ihe modificalion of ihe scale factor in the inpul arrays rather than a change in the dendritic spread of The postsynaptic neurons. In all these schemes, the real problem consists in how to provide the "teacher" input (but see Fig. 5).
X....X, i_i
I
X , ... ui—i
1
I
I
t L
J
X. i-J—i—
l
IV. VISUAL RECOGNITION OF THREEDIMENSIONAL OBJECTS AND FACE-SENSITIVE NEURONS We have recently suggested and demonstrated how to use a HyperBF network lo learn lo recognize a three-dimensional object. This section reviews very briefly this work (Poggio and Edelman 1990) and then suggests that the brain may use a similar strategy. Face-sensitive neurons are discussed as a specific instance. HyperBF Networks for Recognizing Three-dimensional Objects A three-dimensional object gives rise to an infinite variety of two-dimensional images or views, because of the infinite number of possible poses relative to the viewer, and because of arbitrarily different illumination conditions. Is it possible to synthesize a module that can recognize an object from any viewpoint, after it learns its three-dimensional structure from a small sel of perspective views? We have recently shown (Poggio and Edelman 1990) that the HyperBF scheme may provide a solution to the problem provided that relatively stable and uniquely identifiable features (lhal we will call "labeled" features) can be extracted from the image. In our scheme, a view is represented as a 2N vector j„ y,, y x„. y„ of Ihe coordinates on The image plane of JV labeled and visible feature points on the object. We assume that a view of an object is a vector of this type (instead of posilion in Ihe image of feature points, we have also used angles between corners and length of segments or both), in general augmented by components that represent other properties of the object not necessarily related lo its geometric shape, such as color or lexture. We also assume lhal Ihe function lhat maps the views into 0, 1 (0 if Ihe view is of another object. 1 if [he view is of tbe coned object) can be approximated by a smoolh function (if this were false, one could approximate the mapping from the view lo a "standard" view and then apply a radial function lo Ihe result, see Poggio and Edelman 1990). t
The network used for this task is shown in Figure 3
798
y f igure 4. A hierarchical scheme in which HyperBF modules are inputs to another HyperBF module. As an example, a scheme of this type may be used for three-dimensional objeel recognition in the general case of spurious and missing features. Instead of encoding all n features, one encodes only subsets of dimensions d, where d < n. The input to each of the first row of modules is a different set of features of the object; Ihe output is a value belween 0.1 that indicates the degree ol certainly that the input is Ihe sough! object. The last module is a decision module that integrates the various inputs. Notice that all modules could be synthesized by learning through independent sets of examplessee also Fig. 4). In the simplest version (fixed centers), the centers correspond to some of ihe examples, i.e., some views of the object. Updating the centers is equivalent to modifying the corresponding "prototypical views," Updating the weights of the matrix W corresponds to changing the relative importance of the various features lhal define the views of an object. This is important in the case in which ihese features are of a completely different type: A large w indicates a larger weight in the feature in the measure of similarity and is equivalent to a small a in the Gaussian function. Features with a small role have a very large cr: Their exact position or value does not matter much. An interesting conclusion of this work consists of the small number of views required to recognize an object from the infinite number of possible views. The results clearly show that the scheme avoids the main problem of look-up table schemes, Ibe explosion in (he number of entries. Furthermore, (he performance of ihe HyperBF recognition scheme resembles human performance in a related task. As discussed in Poggio and Edelman (1990), the number of training views necessary lo achieve an acceptable recognition rate on novel views, 80-100 for the full viewing sphere, is broadly compatible with the finding that people have trouble recognizing a novel wire-frame object previously seen from one viewpoint if it is rotated away from lhal viewpoint by about 30° (it takes 72 30' x 30° patches to cover the viewing sphere). Recently, H. Buelthoff and S. Edelman (in prep.) have obtained interesting psychophysical results that support [his model for human recognition of a certain class of [hree-dimensional objecls against other possible models. In general, ihe experimental results fit closely ihe prediction of theories of the two-dimensional interpolation variety and appear to contradict Iheories that involve (hree-dimensional models.
905 Face-sensitive Neurons The HyperBF recognition scheme we have outlined has suggestive similarities with some of the data aboul visual neurons responding lo faces obtained by Perrett and co-workers recording from the temporal association cortex (see Perretl el al. 1987 and references therein; Poggio and Edelman 1990). Let us consider tbe network of Figure 3 as the skeleton for a model of the circuitry involved in the recognition of faces. One expects different modules, one for each differeni object of the type of the network of Figure 3. One also expects hierarchical organizations: For instance, a network of the HyperBF type may be used to recognize certain types of eyes and then may serve as input to another network involved in recognizing a certain class of faces, which may be itself one of the inputs lo a network for a specific face. Different types of cells may then be expected. The overall output of a network for a specific face may be identified with the behavioral responses associated with recognition and may or may not coincide with an individual neuron. There should be cells or parts of cells corresponding to the centers, i.e.. to the prototypes used by Ihe networks. The response of these neurons should be a Gaussian function of the distance •f the input to the template. These units would be somewhat simitar to "grandmother" filters with a graded response, rather than binary detectors, each representing a prototype. They would be synthesized as the conjunction of, for instance, two-dimensional Gaussian receptive fields looking at a retinotopic map of features. During learning, the weights of the various prototypes in the network output are modified lo find the optimal values lhat minimize the overall error. The prototypes themselves are slowly changed tofindoptimal prototypes for the task. The weights of the different input features are also modified to perform taskdependent dimensionality reduction.
Some of these expectations are consistent with the experimental findings of Perrel et al. (1987). Some of ihe neurons described have several of the properties expected from the units of a HyperBF network with a center, i.e., a prototype that corresponds to a view of a
Figure 5. (a| A sketch of the neurons of Ihe cerebellum and their connections. In our conjecture, Ihese would be the basic elements of a HyperBF network: The mossy fibers are the inpuis, the granule cells correspond to The various centers and basis functions G(x, x,), the Purkinje cells correspond to the output units that summate the weighted activities of the basis units, whereas Ihe climbing fibers cany rhe "teacher" signal y,. The strength of the synapses between Ihe parallelfibersand the Purkinje cells would correspond to Ihe c„. (bj The corresponding HyperBF network has Iwo basis functions corresponding to ihe Iwo granule cells in a and two output summation units corresponding to the two Purkinje cells in a.
799
specific face Some of the main dala (from Perretet al. 1987 and references therein) follow. 1. The majority of cells responsive to faces are sensitive to Ihe general characteristics of Ihe face, and they are somewhat invariant to its exact position and attitude. 2. Presenting parts of the face in isolation revealed that some of the cells responded to different subsets of features: Some cells are more sensitive to parts of the face such as eyes or mouth. 3. There are cells selective for a particular view of the head. Some cells were maximally sensitive to Ihe front view of a face, and their response fell off as the head was rotated into Ihe profile view, and others were sensitive to the profile view with no response to the front view of the face. 4. There are cells that arc specific to the views of one individual. II seems lhal for each known person there would be a sel of "face recognition units." Our model applies most directly to these neurons.
V. THEORIES OF THE CEREBELLUM AND OF MOTOR CONTROL Cerebellum Models of Marr and Albus The cerebellum is a part of the brain lhat is important in the coordination of complex muscle movements. The neural organization of ihe cerebellum is highly regular and well known (see Fig. 5). Marr (1969) and Albus (1971) modeled the cerebellum as a look-up table. The critical part of [heir theories is the assumption thai the synapses belween the parallel fibers and the Purkinje cells are modified as a function of the Purkinje cell activity and the climbing fibers input. I suggest (see Fig. 5) thai the cerebellum is a HyperBF network or set of networks (one for each Purkinje cell). Instead of a simple look-up table, the cerebellum would be a function approximation
modute [in a sense, "an approx-
imating look-up lable"). In our conjecture, baskei and Golgi cells would have different roles from Ihe roles assumed in Ihe Marr-Albus theory. In particular. Ihe
906 Golgi cells, which receive inputs from the parallel fibers and whose axons synapse on Ihe granule cells/mossy fibers clusters, may be used lo change the norm weights W. Key assumptions include: {1} granule cells correspond to basis units (there may be as many as 200.000 granule cells per Purkinje cell) representing as many "examples"; (2) Purkinje cells are the outputs of the network; (3) climbing fibers are responsible for modifying synapses from granule cells to the Purkinje cell.
Theories or Motor Control There are at least two aspects of motor control in which HyperBF modules could be used: (1) to compute smooth, time-dependent trajectories (for instance arm trajectories) given sparse points such as initial, final, and intermediate positions; (2) to associate to each position in the trajectory the appropriate field of muscle forces. These two problems may be solved by two modules thai can be used in series, thefirstone providing the inpul to the second one (see Fig. 6a,b). I first consider the problem of computing appropriate smooth trajectories from sparse points in space-time. An interesting question is: Are HyperBFs a plausible implementation for Flash and Hogan's minimum jerk principle for the coordination of arm movements? Flash and Hogan (1985) found experimental evidence that arm trajectories minimize jerk, i.e., C= ||J: 1 3 3 || S + (j* 11* where x' is the ihird temporal derivative of jr. This suggests a regularization principle with a stabilizer corresponding to additive quintic splines. HyperBF could implement it using basis units recruited for the specific motion (as many as there are constrained points) wilh Gaussian-like or spline-like time-dependent activities (boundaiy conditions may have to be taken into account). The weights would be learned during training, As Morasso and Mussa-lvaldi (1982) implied, approximation schemes of this type amount to composition of elemental movements. It is interesting lo observe lhat jerk is automatically minimized by the linear superposition of the appropriate elemental movements, i.e., the appropriate Green's functions. Thus, a scheme of the Morasso-Mussa-lvaldi type can be made to be perfectly equivalent to the Flash-Hogan minimization principle. The fact that the minimum jerk principle can be implemented directly by a HyperBF network is attractive from the point of view of a biological implementation, since biologically implausible direct minimization procedures are not required anymore. The minimization is implicit in the form of the elemental movements: weighted superposition of the elemental movements seems a much easier operation to implement in the motor system than explicit minimization.
t b)
x
I
31
The second problem requires a neural circuit lhat associates an equilibrium position to an appropriate activation. Bizzi (see, e.g., E. Bizzi et al., in prep.) suggests that a group of spinal cord intemeurons specify the limb's final position and configuration
800
F(x)=grad
U(x)
Figure 6, Two problems in motor control: (a) determining the trajectory x(r) from a small set of points (f„ x.) on the desired trajectory and {b) computing the field of muscle forces for each of the points on the trajectory. The figure suggests that two different HyperBF modules may be used to perform both tasks. In a, a HyperBF module approximates the trajectory from the sparse points by superimposing Gaussian disEribu' (ions wilh the appropriate weights in such a way as to satisfy some minimum-jerk-like principle. In b, a module of the HyperBF type has been synthesized during development and continuously adapted to generate the appropriate field of forces for each equilibrium position x. It is similar to an approximating look-up table. A behavior of the look-up table type was suggested by Bizzi because of very recent experimental data (see E. Bizzi et al. in prep.). F
through a field of muscle forces that have the appropriate equilibrium point. il Bizzi el al. (in prep.) propose Lhal the spinal cord contains aspects of motor behavior reminiscent of a look-up table. Their findings extend several results in the area of oculomotor research, where investigators have described neural structures whose activation brings the eyes or the head to a unique position. I suggest that the required Look-up table behavior may be implemented through a HyperBF module thai requires the storage of only a few equilibrium positions (or correspondingly, a few conservative-Like fields, i.e., appropriate activation coefficients for the motoneurons) and can interpolate between them (see Fig. 6). Notice that the synthesis of a conservative field
907 of muscle force could he achieved through Ihe superposition (with arbitrary weights, over Ihe index a) by the motor system of appropriate elementary molor fields of Ihe form (see F. Mussa-Ivaldi and S. Giszler. in prep.)
*(x.«) =
m*.)
with r = ||x - x || and C is a radial basis function such as the Gaussian. n
a
VI. SUMMARV: A PROPOSAL FOR HOW THE BRAIN WORKS The theory proposed in this paper consists of three main points: 1. Il assumes that the brain may use modules approximate multivariate synthesized from sparse
functions examples
lhal
and lhal can be as basic compo-
nents for several information-processing tasks. 2. It proposes that these modules are realized in terms of HyperBF networks, of which a rigorous theory is now available. 3. It shows how HyperBF networks can be implemented in terms of plausible biophysical mechanisms. The theory is in a sense a modern version of the grandmother neurons idea, made computationally plausible by eliminating the combinatorial explosion in the number of required cells, which was the main problem in the old idea. The proposal that much information processing in the brain is performed through modules that are similar to enhanced
look-up
tables
is attractive for many
reasons. It also promises to bring closer apparently orthogonal views, such as the immediate perception ol Gibson and the representational theory of Marr, since almost iconic "snapshots" of the world may allow the synthesis of computational mechanisms completely equivalent to vision algorithms such as, say, structurefrom-motiun. The idea seems to change significantly the computational perspective on several vision tasks. As a simple example, consider the different specific tasks of hyperacuity, invented by the psych ophysicists The theory developed here would suggest that an appropriate module for the task, somewhat similar to a new "routine," may be synthesized by learning in the brain. Notice that the theory makes two independent claims: The first is that the brain can be explained in part in terms of approximation modules, the second is that these modules are of the HyperBF type. The second claim implies that the modules are an extension of look-up tables. Notice that there are schemes other than HyperBF that could be used to extend look-up tables. Notice also that multilayer Perceptrons. typically used in conjunction with back-propagation, can also be considered as approximation schemes, albeit slill without a convincing mathematical foundation. Unlike HyperBF networks. Ihey cannot be interpreted as di-
801
rect extensions of look-up tables (they are more similar to an extension of multidimensional Fourier series). The theory suggests that population coding (broadly tuned neurons combined linearly) is a consequence of extending a look-up table scheme (corresponding to interval coding) lo yield interpolation (or more precisely approximation, since the examples may be noisy), that is, generalization. The theory suggests some possibly interesting ideas about ihe evolution of intelligence. It also makes a number of predictions for physiology and psychophysics. More work is needed to specify sufficiently the derails and some of the basic assumptions of the theory in order to make it useful to biologists. The next sections deal with these last three points. Evolution of Intelligence: From Memory to Compulation There is a duality between computation and memory. Given infinite resources, the two points of view are equivalent: For instance. I could play chess by precomputing winning moves for every possible state of the chessboard. More to the point, notice thai basic logical operations can be defined in terms of truth tables and that all Boolean predicates can be represented in disjunctive normal form, i.e., as a look-up table. Given that the brain probably has a prodigious amount of memory and given that one can build powerful approximating Look-up tables using techniques such as HyperBF, is it possible that part of intelligence may be built from a set of souped-up look-up tables? One advantage of this point of view is to make it perhaps easier to understand how intelligence may have evolved from simple associative reflexes, ln more than one sense (biophysical and computational), HyperBF-like networks are a natural and rather straightforward development of very simple systems of a few neurons showing basic learning phenomena such as classic conditioning.
Predictions and Remarks General
Predictions
1. Computation, as generalization from examples, emerges from the superposition of receptive fields in a multidimensional input space. 2. Computation is performed by Gaussian receptive fields and their combination (through some approximation to multiplication), rather than by threshold functions. 3. The theory predicts the existence of low-dimensional feature-like cells and multidimensional Gaussianlike receptive fields, somewhat similar to templatelike cells, a fact that could be tested experimentally on cortical cells. a. The HyperBF scheme is a general-purpose circuitry, used in the brain lo synthesize module that can be regarded as approximating look-up tables. If this
908 point of view is correct, we expect the same basic kind of neural machinery Io be replicaled in different parts of the brain across different modalities (in particular in different cortical areas). 5. The "programming style" used by the brain in solving specific perceptual and motor problems is to synthesize appropriate architectures from modules of the type shown in Figure 1 (a very simple architecture built from Ihe basic module of Fig. 1 is shown in Fig. 4).
Face Neurons 1. Some of the face cells correspond to basis functions with centers in a high-dimensional input space and are somewhal similar to prototypes or coarse "grandmother cells." 2. They could be synthesized as the conjunctions of features with Gaussian-like distance from the prototype. 3. Face cells are not detectors; often several may be aclive simultaneously. The output of Ihe network is a combination of several prototypes. 4. From our preliminary experiments (Poggio and Edelman 1990). ihe number of basis cells that are required per object is about 40-80 for Ihe full viewing sphere, bul much less (10-20) for each aspect (e.g.. frontal views). 1 conjecture that a similar estimate holds for faces. 5. Inpul to Ihe face cells are features such as eye positions, moulh posilion, and hair color. 6. Eye features cells may be themselves the output of HyperBF networks specialized for eyes.
Future The proposal of this paper is just a rough sketch of a theory. Many details (some of Ihem critical) need to be filled in. Some basic questions remain: For instance, how reasonable is Ihe idea of supervised learning schemes? To say it in a differeni and perhaps more construclive way, whal are the systems that can be synthesized from building blocks that are just function approximation modules? What types of tasks can be solved by systems of that type? On the biological side of the theory, the obvious next task is lo develop detailed proposals for the circuitries underlying face recognition and motor control (including the circuitry of the cerebellum) rhat take into account up-to-date physiological and anatomical data.
Notes to Section I 1. Segmentation of an image in parts that are likely lo correspond to separate objects is probably Ihe mosi difficult problem in vision. Remember thai already in the Perceptron book (Minsky and Papert 1969) recognition-in-context was shown lo be significantly harder than recognilion of isolated patterns. We assume here that this problem has been "solved," al leasl to a reasonable extent. 2. The same basic machinery in the brain may be used for synthesizing many different, "small" learning modules, as components of many different systems. This is very different from suggesting a single giant network that learns everything. Notes to Section II The relevant derivatives for optimization methods that need them are for the e
Cerebellum 1. The cerebellum is a set of approximation modules for learning to perform motor skills (bolh movements and posture), 2. Its neurons are elements of a HyperBF network: The mossy fibers are Ihe inputs, the granule cells correspond to the basis functions G(x. x ), the Purkinje cells correspond to Ihe output unils that summate the weighted activities of the basis units, whereas the climbing fibers carry the "teacher" signal y,, 3. The strength of the modifiable synapses between the parallel fibers and [he Purkinje cells corresponds to Ihe c„. 4. Golgi cells may be involved in modifying during learning Ihe center positions l and the norm weights W.
™-2XA,G(|k-«J|' )
(6)
w
for the centers t
f
T
=4c„ | A,G<<||« - t„||' )W W(x, - I.) :
w
V) and for W " -4W | c. £ 4,G'(||x - t.\[i)Q f
M
(8)
p
Motor Control
where Q, „ = (jr., - t„) is a dyadic product and G' is the first derivative of G {for details, see Poggio and Girosi 1990a).
The qualitative expectation is lofindcells and circuits corresponding to Ihe two slages shown in Figure 6. Spinal cord neurons, according to very recent dala by E. Bizzi et al. (in prep.), specify the limb's final position and configuration.
I. There are many nonradial functions derived from our regularization formulation, such as tensor product splines, lhal are factorizable.
r
802
Notes to Section III
909 2. 1 have assumed here that all centers have the same W. It is possible to have sets of different Green's functions, each set with its own W (see Poggio and Girosi 1990a). 3. It is natural to imagine hierarchical architectures based on the HyperBF scheme: A multidimensional Gaussian "template" unit may be a "feature" input for another radial function (again because of the factorization property of the Gaussian). Of course, a whole HyperBF network may be one of the inputs to another HyperBF network. 4. I conjecture that Equation 8 could be approximated by a Hebbian-like rule for the elements of the diagonal W such as
-,(l
+ l)-w,0)~
% c„y(i,(i)-(ij,)y ( ) t
(
m where j is the output of the upper layer of Figure la, i.e.. y = Wx and y is
recognizing different parts of an object, [t is equivalent to a set of networks each with a diagonal W with some zero entries in the diagonal, instead of one network with W with nonzero diagonal elements. 5. Not all features may be always labeled correctly. In general, one expects a significant "correspondence" problem. Possibly the easiest solution is to generate all reasonable sequence of labels for a given input vector and simply try them out on the network. This is, of course, equivalent to trying in parallel the given input on many networks each with a differeni labeling of its inputs. 6. An obvious use of these learning/approximation modules based on the HyperBF technique is based on a hierarchical composition of GRBF modules, in which the outputs of lower-level modules assigned to detect object parts and their relative disposition in space are combined to allow recognition of complex structured objects. Figure 4 is an example of Ihis architecture.
Notes to Section
y = A,G'(||x,-tJ| ) w
and i labels the tth example. Such a Hebbian rule requires back-connections from later stages in the network to the upper layer—where W is updated— in order to broadcast quantities such as the error of the overall network relative to the nh example and the derivative of G' of the activation units, 5. The mechanisms and especially the connections needed to implement the learning equations or some equivalent scheme are an open question, in terms of biological plausibility. More work is needed. Nates to Section
V
(10)
TV
1. The HyperBF scheme addresses only one part of the problem of shape-based object recognition, the variability of object appearance due to changing viewpoint. The key issue of how to detect and identify image features thai are stable for different illuminations and viewpoints is outside the scope of the network. 2. Notice that the HyperBF approach to recognition does nol require as inputs the x, y coordinates of image features: Other parameters of appropriate features can also be used. 3. In a similar vein, notice that the HyperBF network can provide, with the same centers (bul different c), other parameters of the object, such as its pose, instead of simply a yes, no recognition signal. 4. Recognition of noisy and partially occluded objects, using realistic feature identification schemes, requires an extension of the scheme. A natural extension of the scheme is based on the use of multiple lower-dimensional centers, corresponding lo different subsets of detected features, instead of one INdimensional center for each view in the example set. This corresponds to a set of networks capable of
803
Zipser and Andersen (1988) have presented intriguing simulations suggesting that a back-propagation network trained Io solve the problem of converting visual stimuli in retinal coordinates to head-centered coordinates generates receptive fields similar to the ones experimentally found in cortical area 7 of the monkey. We conjecture that Andersen's data may be better accounted for by a HyperBF network. For simplicity, let us consider the one-dimensional version of the problem Zipser and Andersen propose is solved by neurons in area 7. The position of a spot of lighl on the retina is given as r\ [he eye position relative to the head is also known as e. The problem is to compute the position of (he spot of light relative to the head, i.e., h = r + e. Stated in these terms, the problem is computationally trivial, and its solution simply requires the addition of Ihe two inputs ' and e. The situation is, however, more complicated due to Ihe actual representation in which r and e are given. In Ihe equalion, r and e are represented as numbers. Zipser and Andersen assume, in accordance with physiology, a different representation: They assume that the position r of a spot of light is coded by (he presence or absence of activity of one or more cells in a retinotopic array. From this point of view. Ihe goal of the computation carried out by the network is to change representation from array representation to number
representation.
The simplest solution 10 the problem of changing from an atray representation io a number representation is the following. Assume that only one cell in the array/(xj is excited al any given position, i.e., fix) = B(r-x). Simplifying somewhat the situation assumed by Zipser and Andersen, but not altering it in any significant way, lei us assume that e is represented directly as a number or afiringrate. The problem then is to convert the array representation f(x)
-Sfr
— x) for
the relinal position into a number (or a firing
rate)
910 Broomhead, DS and D. Lowe. 1988. Multivariate functional interpolation and adaptive networks. Complex Syst 2: 321. Flash. T. and N. Hogan. 1985. The coordination ot arm movements: An experiment confirmed mathematical model. J. Neurosci 5; 1688. Hubel. D.H. and T.N. Wiesel. 1962. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. J. Physiol. 160:106. Koch. C and T. Poggio. 1987. Biophysics of computational systems: Neurons, synapses, and membranes. In Synaptic [unction (ed G.M Edelman et al.). p. 637. Wiley, New York. Man, D. 1969. A theory of cerebellar cortex. J. Physiol. 20J: 437. Micchelli, CA. 1986. Interpolation of scattered data: Distance matrices and conditionally positive definite func-
representation. Consider a linear unit lhal summates linearly all inputs wilh the "receptive field" wfxj. The output / is given by I = J w(x)f(x)dx. For f ! x ) • 5(i r ) , the choice w{x) = x yields I = r. Thus, a simple solution to our problem of converting an array representation into a number representation only needs receptive fields that increase linearly with eccentricity (notice that v/fxf = ax may also be acceptable; simply a monotonic dependence on x may be a sufficient approximation). If a Gaussian HyperBF network with a polynomial term of degree one is used to approximate the relation of the equation from a set of input/output examples, some of the basis functions will be linear units such as the ones described above, and some will be the product of two-dimensional Gaussians representing the visual receptive fields and two-dimensional Gaussians representing the eye position. These latter cells would probably account for the multiplicative property of the area 7 cells found by Andersen. We conjecture that other features of the cells could be replicated in a HyperBF simulation.
tions. Conslr. Approx. 2:11.
Minsky. M.L. and S. Papert. 1969. Perceplrons. MIT Press, Cambridge, Massachusetts. Morasso, P. and F.A. Mussa-lvaldi. 1982. Trajectory formation and handwriting: A computational model. Biol.
ACKNOWLEDGMENTS The ideas of this paper about the biological implications of new function approximation techniques can be found, to a good extent, in Poggio and Girosi (1989). They depend critically on (he work done together with Federico Girosi, Shimon Edelman, and Bruno Caprile on the theory and the applications of regularization networks, It is very likely that, although plausible, the ideas are wrong—unlike the work on which they rest. Anya Hurlbert, Sandro Mussa-Ivaldi, and Robert Thau read the manuscript and suggested several good ways to improve il, which I managed to implement only in part. This paper describes research done in part within the Artificial Intelligence (A.I) Laboratory and the Center for Biological Information Processing in the Department of Brain and Cognitive Sciences. Support for this research is provided by a grant from the ONR Cognitive and Neural Sciences Division, and by the NATO Scientific Affairs Division (0403/87). Support for the A.l. Laboratory's artificial intelligence research is provided by the Advanced Research Projects Agency of Ihe Department of Defense under Army contract DACA-76-85-C-0010 and in part under ONR contract N-00014-85-K-0124, T.P. is supported by the Uncasand Ellen Whitaker chair.
Cybern. 4S: 131.
Omohundto. S. 1987. Efficient algorithms with neural network behaviour. Complex Sysl, 1:273. Perrett. D.L, A.J. Misdin. and A.J. Chitty. 1987. Visual neurones responsive to faces. Trends Neurosci. 10:358. Poggio, T. and S. Edelman. 1990. A network that learns tn recognize 3D objects. Nature 343:263. Poggio, T. and F. Girosi. 19B9. A theory of networks for approximalion and learning. In A.I. Memo No. 1140. Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge. . 1990a. Extension of a theory of networks for approximation and learning: Dimensionality reduction and clustering, ln A.l. Memo No. 1167. Artificial Intelligence Laboratory. Massachusetts Institute of Technology. Cambridge. . 1990b. Theory of networks for learning. Science UT. 978. Poggio. T. and V. Tone. 1978. A theory of synaptic interactions. In Theoretical approaches in neurobiology (ed. W.E,
Reichardt and T. Poggio), p. 28. MIT Press, Cambridge. Massachusetts. Poggio. T , V. Tone, and C. Koch. 1985. Computational vision and regularization theory. Nature 317:314. Poggio, T. and Ihe staff. 1988. MIT progress in understanding images. In Proceedings image understanding
workshop.
Cambridge. Massachusetts, April 1988- Morgan Kaufmann, San Mateo, California. Poggio. T. and Ihe staff. 1989. MIT progress in understanding images. In Proceedings image understanding
workshop,
Palo Alio, California. May 1989, p. 56. Morgan Kaufmann, San Mateo, California.' Powell. M.J.D. 1987. Radial basis functions for multivariable interpolation: A review. In Algorithms for approximation (ed. J.C. Mason and M.G. Cox) Clarendon Press, Oxford. Tikhonov, A.N. and V.Y. Arsenin 1977. Solutions of ill-posed problems, Winston. Washington, D.C. Torre, V. and T. Poggio. 1987. An application: A synaptic mechanism possibly underlying motion detection. In Theoretical approaches in neurobiology (ed- W.E. Reich-
REFERENCES Albus. J.S. 1971. A theory of cerebellar functions Math. Biosci. 10:25. Benero, M., T. Poggio, and V. Tone. 1988. Ill-posed problems in early vision. Proc. IEEE 76: 869.
804
ardt and T. Poggio), p. 39. MIT Press, Cambridge, Massachusetts. Zipser. D. and R.A. Andersen. 1988. A back-propagation programmed network That simulates response properties of a subset of posterior parietal ncuroos. Nature 331:679.
5d Language and Consciousness
W h a t is intelligence? According to article (66) by Barlow, it is an ability to guess right and an ability to discover unexpected orderliness. Thus it has two aspects: processing and learning. This article, and the following one (67), are elegant reviews of the author's ideas, some of them (concerning information-theoretic analyses, redundancy reduction, versatile learning, encoding, social theory of mind) already introduced in previous papers (1), (21), (22) and (39). Among the new considerations are the distinction between the representational and the social role of language, the demonstration that biological roles are not introspectively obvious, and the identification of autism as failure in consciousness. Barlow's discussion of "raw consciousness" (conscious sensations) deserves perhaps some comment. In various articles reprinted in this volume, sensation has been defined in opposition to perception (e.g. in papers (13), (26), (48)); sensation is then meant as "primary sense impressions" (to quote Delbriick in paper (13)) without the properties of perceptual constancy, which come as the result of higher perceptual processing. Insofar as "conscious sensation" differs from "conscious perception", it suggests that consciousness can scan neural activity at either of the two levels. The large span between "raw consciousness" and "higher level consciousness" (involved in the higher functions of thought, reasoning, judgement) may explain why the seat of consciousness is placed by some experts in association cortices (paper (40)), and by others in sensory cortices (paper (60)). I n article (68), published in 1954, Hebb expresses a distrust for introspection, shared by Barlow. His discussion of divided attention, and of streams of thought, announces some of the arguments developed by Weiskrantz in paper (27), and points toward the links between attention, working memory and consciousness. The importance of the nature of the interaction of the human brain with his environment is demonstrated in experiments of sensory isolation. T h e report of the after-talk discussion will provide the reader with a sense of historical depth, that we thought appropriate at the end of this volume.
805
Reprinted with permission from Matters of Intelligence, pp. 185-195,1987 © 1987 Macmillan Magazines Ltd.
Chapter 6 Intelligence, guesswork, language
Horace B , B a r l o w
6.1
Introduction 1
Sixty years ago, E . G . B o r i n g pointed out the sorry fact t h a t , for those who believed in tests, intelligence was simply what the tests tested for. T h i s circular definition could have been escaped by using an objective, external criterion for validating t h e m , but i n spite of m u c h discussion about the nature of intelligence by the pioneers of mental testing, no agreement was ever reached, and no such criteria were available.
F o r a long time, the
testers did not worry about Boring's stultifying d i c t u m , but a reassessment of the problem is now timely for two reasons.
F i r s t , a glance at current
literature on the sociology, psychology and biology of intelligence reveals an unhappy confusion that must partly originate from disagreement about its n a t u r e . Second, theoretical understanding of the information processing ' E , G . Boring, New Republic, 24, 35-37, 1923.
806
186
o p e r a t i o n s that m u s t underlie intelligence has increased to the point where it influences how w e question the n a t u r e of intelligence. Instead of asking "what is i t ? " , we ask, "what does it do for us?" C o m p u t e r s are our servants, b u t they now c l a i m to be intelligent; even if they are not successful, their c l a i m s m a y help us to clarify our own m i n d s on the subject.
6.2
D i s t u r b e d intelligence
A t t i t u d e s towards intelligence vary a great deal among the testers themselves: some seem to be completely satisfied with current tests a n d apparently regard t h e m as valuable a n d precise scientific measuring instruments; w h e r e a s others believe that the n a t u r e of intelligence is such that it is better to avoid altogether s p e a k i n g of its m e a s u r e m e n t .
1
I n this view, q u a n t i t a -
tive tests are s i m p l y a n aid to forming a j u d g m e n t , a n d a phrase such as ' s u b n o r m a l intelligence' means j u s t about what the m a n in the street takes it to m e a n , a n d no more. A revealing exchange about the role of psychological evidence in a court c a s e
3
brings out this clash between the desire to
e x a l t the s t a t u s of intelligence tests a n d the view that test results should be considered on t h e s a m e level as other evidence. T h e court preferred the latter v i e w , as will m a n y others. A t h i r d a p p r o a c h among the testers is that of the cognitive theorists. Hunt* has reviewed some of their results, a n d it is clear that this group has b r o k e n a w a y from t h e view that intelligence is "what the tests test for", a n d is t r y i n g to fractionate it into better defined sub-skills, although it is fair to s a y t h a t no simple alternative definition has emerged. T h e divergence of o p i n i o n s becomes even wider if one looks at what sociologists and biologists are w r i t i n g on the s u b j e c t , a n d a brief discussion of three recent books will b r i n g this out. 5
D . L . E c k b e r g , in Intelligence and Race
presents a vigorous a n d sus-
t a i n e d attack on the whole intelligence testing movement, based primarily on t h e c i r c u l a r n a t u r e of the v a l i d a t i n g procedure pointed out above. I t is one of several such books, a n d although his arguments are not always 2
A , Heim, The Appraisal of Intelligence, Methuen, London, 1954.
3
0 . T u n s t a l l , G . Gudjonsson and H . J . Eysenck, Bull. Br. psyckol. Soc, 3 5 , 324-331;
333, 1982; A . Heim, Bull. Br. psyckol. Soc, 3 5 , 323-333; 335, 1983. * E . Hunt, Science, 2 1 9 , 141-146, 1983. E
D . L . Eckberg, published by Pracept, New York, 1979.
807
187
convincing, they certainly u n d e r m i n e one's confidence in m a n y of the conclusions that have been d r a w n from intelligence tests. W h a t is p a r t i c u l a r l y disconcerting is t h a t these arguments are not a n s w e r e d in the next book, a collection of essays edited by H . J . E y s e n c k .
6
It was probably felt that they h a d been adequately answered elsewhere,
7
A Model for Intelligence,
but to a n outsider, it is strange that E c k b e r g is nowhere even mentioned. T h e t h i r d book, Brain
and Intelligence
view by E . M . M a c p h a i l , in h u m a n s .
8
in Vertebrates,
is an a d m i r a b l e re-
which deals very little w i t h intelligence testing
A l l the s a m e , it is surprising that he too omits mention of
E c k b e r g , and t h a t , of the first 100 authors' names in his index, not a single one appears in the index of E y s e n c k ' s book. A l l these a u t h o r s c l a i m to be w r i t i n g about intelligence, and it is understandable that the subject is in confusion if authors ignore one another's criticisms and scarcely overlap in the work they cite.
T h e underlying problem is surely the absence of an
agreed definition of their topic.
6.3
Genes, biophysics or language?
T o reinforce the need for reassessment, these books contain new hypotheses, OT perhaps they are reaffirmations of old ideas, that are interesting but totally unrelated to one another. E c k b e r g is u n c o n v i n c e d by the evidence for a strongly inherited general factor in intelligence, whereas all E y s e n c k ' s contributors accept this w i t h o u t hesitation.
T h i s old controversy will be
ignored for the moment because it seems most unlikely that intelligence, almost alone among the characteristics of living things, should be determined solely by environmental forces, although we s h a l l see later that education and language may indeed make intelligence u n u s u a l i n this r e g a r d . T h e new theme in E y s e n c k ' s book is that the general factor in intelligence may be based on simple biophysical properties of a n individual's b r a i n . T h e reader must refer to the book for details, but the results reported by J e n s e n , and others reported by B r a n d a n d Deary, show correlations up to about 0.6 between measures of central latency for various tasks a n d classical measures of intelligence. Is intelligence after all simply a m a t t e r of m e n t a l speed? G
H . J . Eysenck, Springer-Verlag, Berlin, 1982.
' F o r example, in A . R . Jensen, B i s ; tn Mental 8
E . M . Macphail, Clarendon, Oxford, 1982.
808
letting,
Methuen, London, I960.
188
Two other chapters in Eysenck's book, by A. E . and D. E . Hendrickson, look at various measures of cortical evoked potentials, and they too find high correlations with results of classical intelligence tests. It is extremely interesting that relatively simple physiological processes should be advanced as the factors underlying general intelligence and its variability between individuals, but surety what we are talking about is a very special kind of skill having something to do with the way information is handled. Even if the physiological mechanisms underlying its variability were as simple as those proposed here, this would not answer the question of what intelligence does for us. Macphail's conclusions are equally startling. He is completely unconvinced by the classical view that primates and man owe their intellectual pre-eminence to expansion of the neocortex, and especially the association areas. Instead, his review led him to two hypotheses, which he states as follows: "first, that theTe are no differences, either quantitative or qualitative, among the mechanisms of intelligence of non-human vertebrates; second, that man's intellect is distinguished from that of non-human vertebrates by his possession of the capacity for language." This is as far as one could wish from the Hendricksens' premature attempt to explain intelligence and its variations in terms of the biophysics of conduction along fibers and across synapses, and it clearly harks back to a more conservative view of the origin of mankind's pre-eminence. The three divergent suggestions—that intelligence is not inherited (Eckberg), that its variations depend upon simple biophysical factors (Eysenck), and that it rests on the capacity for language (Macphail)—form convenient points for discussion, and they bring out once more the importance of finding an answer to the question that has been unanswered for so long that nobody likes to ask it anymore.
6.4
W h a t is intelligence?
Nobody believes that intelligence is a simple skill that can be described in a few words, but it is nonetheless reasonable to hope for a simple definition of
809
189
the task that intelligence performs. A fresh attempt to do so is justified by the fact that two relevant theoretical subjects, namely statistical decision theory, and information theory, have developed since the lengthy discussions of the topic by the pioneers of mental testing. Both suggest succinct definitions of intelligence that complement each other, and they accord well with common sense. The first is "the ability to guess right", the second, "the ability to discover unexpected orderliness". They complement each other because discovering order is an important way of improving one's guesswork, and they accord with common sense because intelligence manifests itself chiefly by leading to the right instead of the wrong answer. Note also that the tasks pointed to are not simple ones; intelligence has not been insulted by finding simple definitions of what it does. Now, jumping to the right conclusion, or guessing right, might be thought to imply incautious rashness, so one needs to be reminded that all decisions are fallible simply because the evidence is never complete. Good guesswork requires efficient use of all the available information, and this is the problem that statistical decision theory illuminates by enabling variations of rashness and caution to be distinguished from variations in the amount of the available evidence that is used.
6.5
T h e efficiency of guesswork
Consider first making a simple decision about whether a sensory stimulus is present or not. The decisions involved in the exercise of intelligence of course involve more complex data, but the essence of the problem is clearer in the simple case. A judgment of threshold is now generally stated as a statistical problem. It is supposed that the signal, which is the stimulus the subject is trying to detect, is inevitably contaminated by noise, which is either received with the stimulus or added to it in the sense organ or brain. The subject's task is then to decide whether the message he has just received belongs to the population of messages signal + noise, in which case he responds positively, or to the population noise alone, in which case he says "no". Two important developments have come from treating thresholds in this way. It enables one first to understand the effect of the subject's threshold criterion, and second, to measure the absolute efficiency of the mechanisms performing the judgment.
810
190
If a subject has a low criterion, he will respond to weak signals, but will often say "yes" when there was no stimulus, and he was giving a false positive response to a message from the noise alone population. Such a subject may appear to be more sensitive than one who has a high criterion, and very rarely gives false alarms, but the techniques of signal detection theory allow one to measure sensory performance in ways that are not influenced by variations of criterion. The importance of this for intelligence is fairly obvious. One who bubbles over with new ideas that are often erroneous differs much from one who cautiously waits until he is certain before revealing the answer, but they do not necessarily differ in intelligence. Thus the first result of statistical decision theory is to show conceptually how the quality of decision making can be separated from the other variable element, rashness or caution, that is also involved in guesswork. 9
The second result is even more important, for it shows us how to express this quality on an absolute scale. The essentia! point is that there is a limit to how well a statistical judgment can be performed. If the physical signals are known, the best possible performance can be calculated, and a hypothetical ideal observer, who uses all the available information, achieves this performance. A real observer performs worse than this to an extent that would be explained if he only used a proportion of the sample of evidence available to him. This measure of efficiency was proposed by F i s h e r for comparing the merits of statistical tests, was used by Rose" to compare the performance of the eye and television on an absolute scale, and was first applied to psychological tests by Hecht et a l . Both showed that human subjects receiving an average of 100 quanta give responses suggesting that they make use of only about 6 of them. The same approach can be applied to more complicated perceptual tasks. It can, for instance, be shown that a subject offered 100 dots in approximately symmetrical positions makes judgments of symmetry indicating that he only makes use of 25 of them. What happens to the 94 unused quanta, or the 75 unused 10
1 2
13
° S e e , for example, D. M, Green and J . A. Swets, Signal Detection Theory and Psychophysics, Wiley, New York, 1966.
"•'R. A . Fisher, Statistical Methods for Researck Workers, Oliver and Boyd, Edinburgh, 1925. l l
A . Rose, Proc- Inst. Radio Engr., 30, 293-300, 1942.
1 !
S . Hecht, S. Shlaer and M . H. Pitenne, J. Gen. Physiol., 2 5 , 293-300, 1942.
1 3
H . B . Barlow and B . C - Reeves, Vision Res., 19, 783-793, 1979.
811
191
dots, are obviously interesting physiological questions, but the i m p o r t a n t point here is that the figures for efficiency, 6% and 25% a r e h a r d and absolute in a sense that is new to psychology. A vast effort has gone into the s t a n d a r d i z a t i o n and validation of intelligence tests, but they r e m a i n c r u d e measures because they have no externally defined scale on w h i c h the results c a n be expressed. T h e one used is derived from the spread of the test scores obtained on a p o p u l a t i o n . It is as if lengths were defined, not by the average size of a h u m a n foot (which would be bad enough in this day and age), b u t by the s t a n d a r d deviation of these lengths as determined by a somewhat variable i n s t r u m e n t . W h a t the use of statistical efficiencies offers is a measure more nearly comparable with those used by physiologists of the last century s t u d y i n g the balances for energy, nitrogen and so forth; it was this work that
finally
rejected
v i t a l i s m and put the study of body metabolism on a scientific basis. P e r h a p s we c a n do the same for perception and cognition,
but first
we must choose more appropriate tasks than threshold j u d g m e n t s .
The
trouble is that keen eyesight, wider knowledge, or a better memory can improve guesswork as m u c h as intelligence c a n , so we need to understand theoretically a task more relevant to intelligence than a simple sensory threshold.
6.6
Efficiency of learning
T h e obvious ones to consider are learning tasks, such as M a c p h a i l used to compare the intelligence of a n i m a l s . T h e s e require the nervous s y s t e m to detect that there is a statistical association between two sensory s t i m u l i in the case of conditioning, or between a sensory stimulus and the initiation of a movement in the case of i n s t r u m e n t a l learning. T h e r e are limits to what a statistician c a n tell you about the presence or absence of an association, j u s t as there are limits to w h a t he c a n tell you about a message belonging to the signal + noise category or noise alone category.
T h i s means that one could in principle calculate w h a t a n ideal
associator could do in a learning situation. T h e analogue of the observer w i t h a low criterion would be an associator that learned after a very s m a l l number of t r i a l s , and j u s t as the low criterion observer is liable to give frequent false a l a r m s or erroneous detections, so the low criterion associator
812
192
w o u l d learn rapidly, but would be liable to salivate or peck in response to accidentally associated s t i m u l i . J u s t as a low threshold is not a proof of efficient sensory detection, so the rapid formation of associations is not by itself a proof of efficient learning. L e a r n i n g an association is obviously a s t a t i s t i c a l p r o b l e m , a n d the efficiency w i t h w h i c h the evidence is utilized s h o u l d surely be m e a s u r e d . T h e a r g u m e n t so far has shown that good guesswork is an ingredient of intelligence, that the ability to use incomplete evidence c a n be measured on an absolute scale, and that this scale could in principle be applied to learning tasks of the type that M a c p h a i l used in his comparative study of intelligence.
T h u s , one should be able to answer unambiguously
the
question whether a fish c a n learn as efficiently as a h u m a n , but be w a r n e d t h a t , on the scale so far proposed, it might well be able to do so, at least on tasks chosen to favor the fish! Intelligence m u s t , however, imply more than efficient learning or good guesswork in specific c i r c u m s t a n c e s . A n intelligent person will also guess right in a wide variety of c i r c u m s t a n c e s , including completely new ones. In other w o r d s , intelligence implies versatile and creative guessing, as well as correct and efficient guessing.
T h i s is easily s a i d , but the associative task
required for versatile guessing is horribly difficult and complicated.
6.7
T h e combinatorial jungle
If one s a y s that certain events are associated, one means that they occur together more often t h a n would be expected of their occurrence.
from the overall frequency
T h e basic problem is that the n u m b e r of possible
associations rises nearly as the square of the n u m b e r of possible events for paired associations, nearly as the cube for triplets, a n d so on. showed more t h a n a century a g o ,
14
A s Boole
the concepts we habitually deal with
c o r r e s p o n d to logical functions that are more than simple conjunctions of two or three events, and the possible number of these logical functions rises much more rapidly. T h i s means that the task of looking for associations among the enormous number of signals that are used in a vertebrate b r a i n is t h a t of finding a path through a c o m b i n a t o r i a l jungle of quite unimaginable i 4
G . Boole, An Investigation of the Laws of Thought, 1854, reprinted by Dover, New
York.
813
193
complexity. As soon as one glimpses this jungle, one realizes that finding useful paths through it must be what intelligence is all about: intelligence is not just a matter of detecting this association or that association, but requires knowledge of the associative structure of a large body of information, and this is a formidable task. Ideas from information theory and the study of language may give more insight on this problem than the pioneers of mental testing could achieve.
6.8
Redundancy
reduction 15
Consider first an idea from information theory. If messages are correlated with each other, redundancy is present, and in principle, it is possible to find a code that will remove it. Such a code would be adapted to the structure of associations among the messages, so its specific form would store information about this structure. Furthermore, if such a code were found and used, messages would become simpler, both because fewer symbols would be required to carry the same information, and because some of the associational structure originally present would have been removed. This could be a way of beginning to clear a path through the combinatorial jungle, and it is interesting to note that some items of intelligence tests seem aimed at detecting the ability to devise such redundancy reducing codes, for instance, "Give the next member of the series 1, 5, 10, 16, 23, — " This requires identifying the principle upon which the sequence was generated, and once this has been done, the numbers (except the first two) become wholly redundant. Such a question may only determine whether an individual has been well enough trained to be able to recognize certain number-patterns, whereas the postulated redundancy reduction must mainly occur at precognitive levels in the combinatorial jungle. One can, however, imagine tests in which a computer introduces regularity into an otherwise random display, and the subject's task is to identify it. By these means, one might obtain absolute measures of the mental capacity to detect i a
F . Attneave, Psychol. Rev., 6 1 , 183-193, 1954; H. B . Barlow, in Sensory Communi-
cation, W . Rosenblith, ed., M I T Press, Cambridge, 217-234, 1961.
814
194
o r d e r l i n e s s , a n d this is c e r t a i n l y one task performed by intelligence.
16
W i t h this thought in m i n d , it is a m u s i n g to speculate on the possible future of the electronic games m a c h i n e s in a n amusement arcade.
First,
it seems that they m a y t a p a n d exercise the very skills, w h i c h it is here c l a i m e d , lie at the root of intelligence.
B u t w h a t is even more exciting is
the possibility that they could a w a r d scores for their players that would c o n s t i t u t e absolute m e a s u r e s showing how well they h a d utilized the inform a t i o n that w a s available to t h e m . T h i s would, of course, be the optimal feedback, a n d if intelligence is t r a i n a b l e , the truants in the arcade m a y soon outshine the conscientious students w h o r e m a i n in school.
6.9
T w o roles of language
B u t there is more to intelligence t h a n guessing right at a games machine, so let u s consider M a c p h a i l ' s interesting suggestion that it is m a n k i n d ' s c a pacity for language, a n d that capacity alone, that m a r k s his brain out from those of the r e m a i n d e r of vertebrates. L a n g u a g e is often defined solely by its c o m m u n i c a t i v e role—it enables one i n d i v i d u a l to convey his thoughts to a n o t h e r , a n d thereby influence his behavior. B u t it has another role, that of organizing the representation of information entirely w i t h i n one individual's m i n d .
1 7
T h i s representational role is unfamiliar to many, b u t it is not
necessarily a n u n i m p o r t a n t secondary aspect of language; quite to the cont r a r y , it could well be the case t h a t the representational role is p r i m a r y in a developmental a n d evolutionary sense. O n this view, m a n k i n d ' s intellect u a l b r e a k t h r o u g h w o u l d have been achieved in two stages: the evolution, first of the representational scheme t h a t underlies language, a n d only after t h a t , t h e development of the ability to communicate with other h u m a n s by messages based upon this scheme. If M a c p h a i l is right, one must a s k w h i c h of the two roles of language is m o s t i m p o r t a n t in giving m a n k i n d its superior intelligence; if it were the representational role, a n d if the a r g u m e n t of this article is correct, then the representational scheme u n d e r l y i n g language must be w h a t enables us 1 B
H . A . Fatmi and R . M . Younge, Nature, 2 2 8 , 97, 1970; H . B . Barlow, Nature, 2 2 8 ,
J008, 1970. i 7
J . A . Fodor, The Language of Thought, Cromwell, New York, 1975; B . Meltier, 6th
Joint Conf. A. I., 1127-1130, 1979; A . Sloman, Proc. Aelib/BCS Conf., Aslib, Lond, 1979.
815
195
to find useful paths through the combinatorial j u n g l e . T h i s is an a t t r a c tive thought, and points to an aspect of language that has received little attention.
18
B u t the two roles are not, of course, independent;
because
of the communicative role, an individual's representational scheme is not j u s t the product of his inherited genes and a passive environment a c t i n g through his senses, but also results from the active intervention of the language community to which he belongs. It is perhaps the realization of this c r u c i a l role that underlies some people's desire to m i n i m i z e the i m p o r t a n c e of the inherited component. A l t h o u g h this may be misguided, one c a n see that intelligence does indeed differ from other biological characteristics in the manner of its dependence on genes and culture.
6.10
Conclusions
We saw at the start that intelligence is not written or thought about i n a coherent way; it means different things to different people, and even a m o n g the community of intelligence testers, there are important disagreements. A new look at the problem suggests a simple role for intelligence. It is the capacity to guess right by discovering new order. T h i s not only agrees w i t h commonsense and explains why intelligence is so important for the s u r v i v a l of both m a n and beast, but also enables us to apply recent knowledge to the problem. Statistical decision theory points to statistical efficiency as an absolute measure for that part of intelligence w h i c h is concerned w i t h d r a w i n g reliable conclusions from incomplete evidence, and information theory points to the importance of redundancy reduction in h a n d l i n g the overload of information from our sense organs. B u t this is only one p a r t of the problem: intelligence is concerned with the associative s t r u c t u r e of the messages in our brains, and this is where language may play an i m p o r t a n t role. Decision and information theory have helped us to u n d e r s t a n d w h a t intelligence does: the next problem is to find out how the task is achieved.
[Horace
B. Barlow,
Physiology,
, e
Kenneth
M.D.,
F. R. S.,
is a member
Craik Laboratory,
H . B . Barlow, Perception, 3, 123-134, 1974.
816
Cambridge,
of the Department England
CB2
3EG.\
of
Reprinted with permission from Mindwaves, pp-361-374, 1987 ©1987 Blackwell Publishers
The Biological Role of Consciousness Horace Barlow Horace Barlow lias hem actively involved tpith the physiology and psychology ufperception for forty years. He mas guided an this path by E. D. Adrian, who showed where ncuropliysmhgical research was leading, and William RusliUm, who showed lhat research might be enjoyable as well as informative. He has worked on the frog's retina (discovering lateral inhibition and 'jlydcteciors'J, the rabbit retina (directional selectivity), cat retina (responses to single photons) and cat cortex (disparity selectivity), lie has also been interested in the physical and biological factors lhat limit sensation and perception, such as photon fluctuations, the dynamic range of nerve fibres as signal channels, and their large though limited ability la interconnect with each oilier. Horace Barlow is now a Royal Society Research Professor al the Physiological Laboratoiy in the University of Cambridge. Whal is [lie selective advantage conferred on the human race, or any other species, by consciousness? Is it just an epiplienomcnon of our brain mechanisms — the whine of the neural gears, the clicking nl the neural circuitry? O r can one identify a more important role for it in the survival and future uf our species? Like many others, I feel usuuitivcly certain that consciousness is of paramount importance, and it therefore becomes an intritniing task to search for and try to define this rule. T h e answer I shall put forward is thai consciousness links the individual to the community within which he lives, a link lhat is crucial to all thai is human. Normal perception, learning and memory are thought to be involved - nothing paranormal - but the individual's need for consciousness and its impossibility without social experience means that eonsciousncss itself is what forces an individual to interact with the community. This interaction occurs in both directions, for one's consciousness is much concerned with other individuals, while past experience tt-iili other individuals has much influence on ihe content ol each individual's consciousness. This idea places consciousness in the border regions belween the individual and the community; this is quite different from the border between dualism and identity theories thai Jcflrcy Gray lias been concerned w i l h , but the two ideas are complementary, not Conflicting, and if it is correct that consciousness haunts ihe intersection of these iwo boundaries we shall have made good progress in delimiting ihe types ol 1,2
William Shakespeare (1564-1616). His acute perception of the human condition reveals a superlative consciousness. 817
362 nervous activity wilh which it is concerned. Uut lei me start by explaining the converging paths which have brought me to this viewpoint; in a sense these are digressions, bul I hope they will move you gradually towards my way oflooking at the problem.
C o n s c i o u s n e s s and Single Units I started thinking about consciousness because of my interest in the role of single nerve cells in sensation and perception. Briefly, one can find neurons in sensory centres and pathways that show discriminative powers not unlike those of an intact human being. Y o u find cells in the rabbit retina that respond when things move up but not down, and we can tell if things arc moving up or down (see figure 24.1). Likewise you find cells that discriminate between different velocities of movement, and between stereoscopic disparities, and colours, and the various categories of cutaneous sensation. M o r e recently cells have been found in the monkey cortex that respond to human faces,'' and appear to be able to discriminate between the faces of particular individuals. S u c h cells have much of the discriminative capacity of our own sensations, and when you come across these facts you are bound to say to yourself, 'Well, I know that when I discriminate between red and green this is accompanied by a conscious sensation. When this single nerve cell shows the same sort of 1,4
7
!
background 10 ctt/m
..
spot 60 cd/nv*
Figure 24.I
wMfii
. ... i
L
I —-—
I2
Example of patient discrimination
. .,
position 5" S 10° A in visual field
3
at lite cellular level, Thisgtm-'lian
retina gave tt vigorous hum of impulses (Umal
muni)
mhcu a s/mi aflkhl
iltmitglt the appropriate pari of the visual field, very few when ii mtn'ctl downwards Otherdircciitms
gave intermediate
indicating
the ring tifO's. tlierernrds impulses,
thai a briefrcsptmse
occurs til tn (+) and ojf(-);
record).
of the figure by the
tu responses occurred
Then a spot mas moved through ihe receptive field in the direction of the arrows,
at ihe liases of the arrows obtained. Here the leriiettt lines are actum pnieiilialt
}%*).*
818
outside and
nr
which arc the messages signailetl la lite hrttin. wilh the numbers ol impulses Ohm
imdteated. The larvcr trace in each record shows ike itnrvcmcnl oflhcspol ortdl.evid;
upwards
(iiglast
responses {records al the side), 'llir receptive fieldmas 'first mapped
with a siuiitmaiy sprit turned OH and off, ami Ihis region is represenietl in tlietenirc ± rfmtatt,
cell in the mmed
2 lo 79)
u/'lieltl (j'rtim Harlow,
Hill
363 discriminative capacity, am I to suppose that it is actually having some kind of rudimentary sensory experience?' Although there may be some people who think that a primitive form of sensation occurs in a single cell, lhat doesn't make sense lo me, and I cannot accept it, I think this is a problem of levels of analysis: if one cannot (ind the explanation for consciousness at the level of single cells one has the choice of looking at a lower level or at a higher level. T h e lower, more analytic, level docs noi seem promising to me, lor transmitter substances and receptor molecules arc surely even further from consciousness than single neurons. G o i n g one step in the other direction, to nerve nets and groups of neurons, docs not seem to introduce any very interesting new possibilities, but the picture changes dramatically if you consider whole brains, for then types of behaviour occur that result from one hrain interacting with another. T h i s seems to me the level at which one must look for a better understanding of consciousness, and especially of its biological role.
C o n s c i o u s T h o u g h t and Social U n d e r s t a n d i n g The second digression is to compare my posilion with that of Nick Humphrey. Even though we had never previously talked to each other on the subject, we found that wc had rather similar views when wc gave talks at a conference in Cambridge about seven years ago," '' for wc agreed that consciousness is closely connected with the social behaviour of animals; however, I think 1 have carried this initial insiphi to its logical conclusion and shall try to clarify my viewpoint by comparing it with Nick's betterknown position. 1
111
H e points out that knowledge of the working of one's own mind can be very important in social behaviour: one understands the motivation and causes of the behaviour of others by analog)' with one's introspective knowledge of the motivation and causes of one's own behaviour. While fully agreeing that the survival value ot consciousness derives from its influence on social problems and their solutions, I think he has much too much confidence in the validity of the knowledge gained by introspection about the working of one's own mind. I do not think it gives accurate knowledge of the causes and motivation even of one's own behaviour, let alone lhat of others. Introspection only leads one to think one understands the reasons for one's actions, and the limitations ol this partial knowledge are crucial. Furthermore, his view does not emphasize the influence of social experience, particularly early c\periencc, on the nature of an individual's consciousness. T h i s experience may control the quality, depth and validity of an individual's introspection, thus making consciousness a joint product of the individual and his social environment. O f course this is to some extern simply an echo o f F r e u d , but once it is admitted thai introspection is fallible, one must look at the errors it leads to, as well as the advantages if confers, when considering its survival value. S o m e may find this cynical, but there are other sources ol knowledge than introspection, and in this past-Freudian, post-Marxist age it seems to me a much sounder viewpoint. M y mistrust of introspection is also the basis of my lasi digression.
819
364 B i o l o g i c a l R o l e s a r c not I ntros pec lively O b v i o u s T h e third and most important pathway to my viewpoint arises from a difficulty in the direct introspective attack on the question. A few analogies will show how deceptive such an approach can be. First consider pain: introspection tells us that pain is unpleasant and that pain-producing objects and situations are to be avoided, but it docs not tell us that pain preserves our lives and limbs. T h e fact that this is indeed the case becomes abundantly clear when one encounters unfortunate individuals who lack a sense o f pain, for they suffer repeated and potentially dangerous minor injuries from bums or abrasions that others reflexly avoid. Furthermore when they have received a minor injury, they lack the automatic reactions that rest and protect the injured part, allowing it to recover. T h e important point to realize is that direct introspection about pain does not lead one to appreciate its value and it is instructive to see how to circumvent this difficulty. Instead of contemplating pain directly, ask the question 'What is the principal action associated with the sensation of pain?' I think one would answer, cither by introspection or by observing others, that it leads to withdrawal from painful objects or situations, and avoidance of these in the future. F r o m there it is of course a small step to understand the survival value of a system of nerve fibres that can only cause unpleasantness, and to see the benefits derived from them. Now take another example - falling in love. I'oetry books are full of introspections on this subject, and as before they somehow miss the point wc are interested in. O n the other hand, asking what principal action is associated with falling in love makes it obvious that the propagation of the species and future survival of the human race are at stake. T h e poets arc unintentionally instructive on the devious fallibility of introspection, for they avoid emphasizing the fertility and motherliness of the person concerned and dwell on very different qualities.
EXPERIENCE Pain Love
Redness
INTROSPECTIVE M E S S A G E SURVIVAL VALUE Unpleasant and to be avoided Desire lor lifelong attachment, feelings of unbounded admiration, etc. Attribute of a physical object
Minimizes injuries Propagation ol the human species Ability to communicate about this attribute
Introspection on our experiences does not directly lell us their survival value (more on redness at pp. 369-72)
I think this brief consideration of pain and love has shown how hopelessly misleading it can be to try and draw conclusions about the
820
365 biological <¬ . T h a i do T altogcit^ -.hi- ... '
G
1
m
C r
'""
1
tthcn
Sa
value [,f ^' t v " - introspection. ' it h r i n , .
c
L*"'
C C t l U :
a n t l
r
a s u
c C ,
v c
" bi ' experience by direct introspection, ^ should ignore one's introspections uic misleadine _._.g aspects aspects are are pointed pointed out out wc wc may may be
1 0
s a
c s
m
' P
a
l T n f
I
o
n
c
i p s this docs make sense.' But the survival
" P ^ n c c is certainly not the tirst thing revealed to
11
s
t o
'"ok at the principal associated actions which
< R l r s
T h i s could ^ e i r and nrtfc, other Lpeople. It brings about hin yoursclfand i*
P
or thirst, and I sha I etun ,o , n * 6 < &" ' ha I return to he scnsa.U.n of redness later, but it is time to «, leave ... these digress,"ns . consciousness itself, applying the lessons Ic arnt in these prt.-hm.nary sk.rmishcs with d ove P U r S U l : ( l
a
f u r l h c r
b
, a l k i n
a b o u
h
u
n
n d
p
a
i
n
a n
Nj. longer mourn for me when 1 dead rhan you shall hear the Surly ! | n bell f-rtve warning lo the world that I an, Hed (•rprn tliis vile world, with vilest worms tn dwell: Nay. vf you read this Kite, remember not he hand thai writ it; fur 1 love you so, I hat I in your sweet thoughts would be forept, If Ihinkmjj on me then should make you wue. 0. if. I say, you look upon this verse When I perhaps compounded am wilh clay, Do not so iiiueh as my poor name rehearse; Hut let your love even wilh my life dec;iy; a
SU
m
C
Lest the wise world should look into your moan, And mock you with me after I am gone. Conscious and Unconscious Neural Activities !! our consciousness is limited, what is ii limited lo? Upon what types or classes of neural activity can we obiain an introspective viewpoint? At first one is inclined to think lhat, without consciousness, all mental life would cease, but it is certainly nol irue that all neurally mediated behaviour would cease. Consciousness is therefore associated with the working of only part of the brain, and we want to find what part this is. W c have seen that one path to the answer may be to look at the principal actions lhat arc associated with it, and also ol course at those that can occur without it. It would also be illuminating il one could find pathologies of consciousness analogous to ihe absence of the sense ol pain, or colour blindness, for then one might be able to point to the direct consequences of its absence or faulty functioning. I.ct us start hy trying to decide what parts of the working of the brain arc accessible to consciousness and what parts are not. 1 think evervhodv will accept that there are many things one's brain does for out, rhnt are not associated with consciousness. F o r instance, 1 walk towards my from door or my car, and I find my hand in my pocket getting out the key. T h a t part of my brain is actually rather stupid because it usually gets out the wrong key, but one must give it credit for doing a quite complicated action without any conscious thought passing through one's mind. I am sure everyone can think of many equivalent neural computations that their brain performs without their awareness: picking footfalls on a mountain path, many aspects ol driving a car or returning a
821
t-ifftre 2-1.2 .Shiikispture.
Srmittl
mi.
71
by
366 shot at tennis, brushing one's teeth or putting on one's clothes, fur example. Maybe these are consciously initiated, and maybe when first performed one has to pay conscious attention, bul in normal lile one's brain does many complex operations for one without the breath of consciousness. What then is the general characteristic of those brain operations that do accompany consciousness? T h e answer I would give is lhat they are concerned with social interactions, and especially with the preparation and execution of communications with other individuals. O f course this mainly means communication with other humans, though I wouldn't want to deny consciousness to the person who is talking to his dog. Somebody might object that all forms of neural activity must be brought about in the same way, by nerve impulses, synaptic potentials and so on, so that it is inadmissible to suppose that some forms of activity arc accompanied by consciousness, others not. A n analogy may make clear why this objection is invalid. Consider speech and sound: again it might he said that both are transmitted by the vibration of'air molecules, hence one cannot attribute an important property to the one and not the other. But speech is not j/tsi the vibration of air molecules; speech sounds form only a very small subset of all vibrations of air molecules, and when talking about speech it is lhat sub-set we are interested in, not the general question of how sound is transmitted to the listener, ln the same way, it is quite reasonable to sav lhat the nervous activity associaled wilh consciousness is only a small pan of all ihe activity of the brain, namely ihe part which is concerned wilh communications with other individuals; that is the important aspect of conscious activity, not the fact that it is mediated in the same way as most other operations of the brain. Now, of course it is true that you can have mutual communication between two individuals without consciousness. I f we recognize thai unconscious communication is possible, wc are not forced lo say that bees, for example, are conscious even though they certainly communicate with one another very freely and effectively. Hut one can also have consciousness without concurrent communication with another individual, and this needs more careful consideration. If I'm shut in a room by myself 1 don't become unconscious, so that the rclaiionship between consciousness and communication isn't simple and immediate; consciousness continues when communication is cut off. But I would claim that whenever one is conscious, even in one's deepest introspections, one is in a sense addressing some other individual. Arc others aware of an audience when they arc thinking to themselves? I could often name the individual or individuals to whom my thoughts are addressed, but 1 don't know how universal this is. Kvcn when the audience cannot be specified, I believe it is reasonable to insist on the importance, for all conscious thought, of some internal model of one or more other human minds. As Kenneth Craik argued, our brains are adept at modelling the environment," and the characteristics of other individuals must be among the most important candidates for such modelling. I f you have doubts about the importance of human models in introspection, ask yourself what language your thoughts are couched in, and from whom you
822
367 learnt thai language. F o r me at least il makes sense to suppose that the dawn o f an infant's consciousness comes with its early communication with its parents, and that ever afterwards the image or model of the communicatee is a partner in conscious experience. J he importance of interaction and communication for one's consciousness will probably be challenged, so my first task is to make ihe idea acceptable to common sense. 1 shall try to do this in two ways; first, see whether these ideas fit in with common usage; and second, find whether they arc plausible when we introspect about them.
C o m m o n Usage of'Conscious' I have chosen three rather different examples to show that communication is an important part o f consciousness, as this term is commonly applied. I f someone says 'he's unconscious', to test this you would immediately try tn establish reciprocal communication with the individual concerned, and i f vou failed you would be likely to agree that he was unconscious. T h i s would he so even if, for example, the person was sleepwalking and showed many indications of being in a normal state. In such a case you might go up and tap him on the shoulder, and if you didn't get sensible responses you might say, 'Well, he appeared at first to be conscious, but he is not responding to me and his consciousness must at least be impaired.' Another example indicates what seems to be a very different usage ot the word 'conscious'. I n a train coming back from L o n d o n once, I heard two football fans talking about a third person whom they both knew; they were saying, 'he's not even conscious,' It turned out that what they meant was that this third party didn't support the same team; so it seems that they were imputing unconsciousness simply because they couldn't c o m municate on this one issue.
Conscious (kpn/.is), a. [f. L . consci-us knowing something with others, knowing in oneself, privy to, conscious + - O U S . L . consci-us f. con- together + JCIknowing, as in scire to know: cf. nescius unknowing, prascius. foreknowing. There is no such word in F . , which uses cortscien! in some of the senses (as did also Bacon); but It. has conscio privy, accessary, guilty, from 16th c ] \l. Knowing, or sharing the knowledge of anything, together wilh another; privy to anything with another. Obs. (Withquot. 1651,cf. L . alicui alicujus
retconscius.]
I IIOKBKS l.evmii- I . vii. i I NfhL-re IWJI. »r m m men, Vnow n'vnt and Ihe u n u hKl, ihev are said ii.heCunHioiJs'.iril arte tnanuiher. I6W S O U T H
mi.
tWZJI 1. 394 N.'lhibR i» I" I * L'mKtffcJ ffim Ihe kilhei self- Tu be afriendand mh»-k»nw:Unjsaie leims e^uivaleni.
2. fig. Attributed to inanimate things as privy to, sharing in, or witnesses of human actions or secrets. Chiefly poei. LThe t-jrhfti ntotdti use-ihe wwl being ace <jf those rlJiruled by Ben JuhnMW. Frfjueni in the Laiin p o r a »iih 1667, t l Ovid \iu..ruir, nun cnnseii uibesiVk
823
Figure 24.3 The Oxford English Dictionary's itefiniliim of conscious agrees lhal shared knowledge, implying commonicalion, is an importani aspect of consciousness (reproduced courtesy of Oxford
English Dictionary, Oxford University Press).
368
Now let's just take a third example. ! made a conscious decision this morning to miss my dentist appointment. And what 1 mean by that is that 1 didn't accidentally miss it; 1 went through a stage when I could have told you, or the dentist, that I was going to miss it. So what the word 'conscious' adds to the meaning of the above sentence is the fact that I was in a position to communicate my decision to another person. In all these three examples, which arc really quite diverse, it is the ahiliry to communicate that is the crucial test of consciousness, so 1 wonder if anyone can think of a good example of the proper use of 'conscious' which would disprove this statement: the person who is said to be conscious is in a position to communicate, and the person who is said to he unconscious is not. In common usage, communication or the ability to communicate appears to be a crucially important aspect of consciousness. Introspective Acceptability Docs this attempt to delimit the types of neural activity that arc involved in consciousness seem plausible introspcctivcly? As I've mentioned, there arc pitfalls here, but no idea about consciousness is likely to survive unless it can at least be reconciled with our introspections on the subject. Many people will point out that although one's conscious thoughts arc largely composed of words and language, they arc not exclusively composed of them. I'm sure musicians can consciously think in terms of musical phrases and harmonics, but music is after all a form of communication, so that objection is answered. I'm also told that if a champion skier is to have any chance of winning he must, just before he starts a race, rehearse in his mind exactly how he will take every gate. Well, at first those don't seem to he communications that pass through his mind, but I wonder if the skier is not actually rehearsing messages from (or to) his trainer, rather than rehearsing the motor actions he will perform. In any case these two examples arc unusual, and the prevalence of language in consciousness will probably be conceded. Even with my more elaborate conscious thoughts I can often make myself aware of an audience whom I am addressing; this is important because I would have different thoughts if the audience were different. Perhaps many of you would agree that, when you are thinking in a particular mode, this is sometimes because you have in your mind a particular person or audience. If you accept this, then you have admitted that consciousness is not a property of a single brain, but also involves the representations in that brain of others. This is particularly important for those introspections about one's beliefs and actions that, as Nicholas Humphrey points out, help one to understand other people, for it means these others are having a reciprocal action on one's own thoughts. Thus conscious introspection is not just a simple aid to social behaviour, but is itself moulded and formed by social experience. 10
1 find that this comparatively small change in how I think of consciousness makes an important difference, and now that it has been incorporated in my thoughts, it is not only acceptable but also attractive to realize that consciousness is not all my own brain's work, but is partly caused by those with whom 1 communicate.
824
369 A u t i s m - Lite Without Consciousness? Can one point to pathological conditions of consciousness, analogous to the lack of a sensory system subserving pain, or to congenital colour blindness? It was suggested before that childhood autism, whose pathognomic symptom is the failure to develop normal social relationships, might represent such a condition,' and recent experimental and theoretical studies add some plausibility to this. Uaron-Cnhcn, Leslie and Frith found that 16 out of 20 autistic children failed a test that required them to undeistand lhal another being could hold a false helief because it had incomplete information*; a group of normal children of mental ages matched to the autistic group passed the lesi without difficulty, as did a group with Down's syndrome similarly matched. Thus the autistic children were unable lo do a simple version of precisely the task that Humphrey said consciousness facilitated - understanding the causes of another individual's actions and beliefs. As I have emphasized, our own introspective knowledge is incomplete, but aulislic children appear to lack even our limited conceptualization of mental states and their origins. 13
12
One wonders i f some impairment of the mechanism and range of consciousness mighi nol be the cause of ill-dctined 'psychopathic personalities', habitual criminals, and others whose principal problem lies in their relationship wilh other individuals and :-;ocnlv I his ma\ liliount lo no more than saying 'they do not think like us', but il may also draw attention to ihe aspect that is different, namely the capacity to introspect on one's own mental state and recognize the existence of similar mental states in others. It is also interesting to speculate aboul an individual whose introspection could extend far be vend the limits restricting; those ol the majority Such an individual would have access to superior understanding of the causes of his own and also of his friends' beliefs and actions: he or she would make a good playwright or novelist, but would we regard such an individual as unusually understanding and sympathetic, or as cold, calculating and unsocial? Cireatcr introspective power, like any other improved mental faculty, could clearly be advantageous to the individual, but it might be disruptive to tbe group and species. How much then docs social morality depend, not on ihe validity of our introspections, but on each individual's introspection having limitations and errors similar to those of oihers in ihe community?
Raw Consciousness Consciousness of course means many things. So far I have talked mainly about the aspect that decides one's future actions in the lighl of introspection on one's own mental state and that of others. Because so little is known of the underlying physiology and psychology of these processes, there is little temptation to think about them in over-simple, over-mechanistic terms; bul ihis is not true of supposedly more elementary forms of consciousness such as 'raw' redness, which some would
825
370
%c&aea Wtrmtil lamaik "ml 'nnils, lints eii'i'ig liffiiipcrl'ir bHiiniffdge "I tht-.t-aitift ii'iple's I'iht'fy n W iit'liiim 'Afar,' fcvttta PiannUhrery).
hold to be just our brains' direct response to a stimulus in the external world. T h e rather complete physiological and psychological knowledge we possess of such peripheral sensory processes seems at first to support this notion, for it might be thought to imply that there is no room for communication and interaction in generating the experience. I do not think this view is correct because current knowledge docs not explain how sensory experiences acquire their labels, and for a number of other reasons. First, we often do want to communicate about sensations like redness, and I claim this is why they form part of the content of our consciousness. We want to be able to say, 'That strawberry is red enough to pick and cat', or ' T h e red sunset forecasts good weather'; no one can deny this need lo communicate about our immediate sensations. Furthermore the influence of language, and hence necessarily of social experience, upon the raw sensations themselves is greater than one normally admits. Consider a colour-blind individual who completely lacks one of the three types ol cone photoreceptor possessed by those with normal colour vision (lhat is,
826
371 a 'dichromat', instead of a 'trichromat' like most of us). T h e r e are irrefutable reasons for believing thai such an individual's experience of colour must be very different from lhal of ihe rest of us, yel mosi of them would understand what 1 have said about colour in this essay, and could carry on a normal conversation about colours without a slip. Quite a number of them are actually unaware of their abnormality until it is revealed to them by testing: surely such a person's knowledge of colour must have been derived from sources that are certainly not accessible to us when we introspect on our sensations. It is interesting to speculate about a eolour-hlind individual who, as is the case with many of them, has a normal mother and a colour-blind father; on my views he might have a thoroughly schizophrenic consciousness of colour, since his early verbal knowledge of it would come from a mother who might seem to him 'funny about reds and greens' and a father whose experiences matched his own much belter, but who would have suffered the same problem as the son in matching these experiences to the spoken words of the majority.
The Recipe for Redness 1 Form an optical image 2 Measure Ihe light in long, medium and shorl wavebands in each pan ol it. 3 Compare amounts locally in each waveband. 4 Make comparisons over larger tegions ol Ihe image. 5 Classily the results and make an array ol possible results. These are the possible unlabelled colour sensations, and ihe curreni perceptual compulations allot each pari ol Ihe image to a posilion in Ihis array. 6 Form an array of colour words by listening Icr adjectives lor surfaces of objects, etc. 7 Observe how [hey are used and attempt lo match them to posilions in array no. 5. 6 As you succeed, attach labels ol array no 6 to Ihe appropriate positions in array no. 5. 9 Forget you ever did nos. 6, 7 and 8 bul retain labels in no. 5. 10 Use Ihese labels to describe your sensations, e.g. 'thai apple is red'. Redness is never raw: il is the result ol a complicated sequence of operations. (NB: the physiological mechanisms ol steps 1-3 are understood quantitatively; steps A and 5 are beginning to be known; steps 6-10 are almost guesswork, but something ol this sorl must happen ) T h e s e facts are less surprising when one realizes that the sensation of redness is the end result of an elaborate sequence of neural operations (see box T h e 'Recipe for Redness'). T h i s is what Edwin L a n d has been
827
372
telling us recently, and a long time ago Hermann von llelmhnltx. Was saying that perceptions were the products of unconscious induction. • It makes a lot of difference where you think the sensation of redness occurs in the sequence of events from physical stimulus to the final consequence of that stimulus, namely the utterance 'it's red', or the ability to communicate or make use of that fact in some other way. II you think only a few easy steps intervene between the physical stimulus and the sensation, whereas many steps intervene berween the sensation and the communication, then you may have a problem with my view of consciousness. But if you think (like Helmholtz and Land) that many logic-like processes go on before even the simplest sensations occur, then there is plenty of room for social experience to interject its effects, and the point of view that I am presenting becomes much more acceptable. If the sensation is much nearer the utterance than the physical stimulus, it becomes reasonable to say that the sensation of redness is merely preparing you to communicate the fact that something is red; this is another case where introspection is misleading, for redness is a carefully cooked product and is never as raw as it seems. H
1
Figure 24-5 Hchnlmliz
Hermann (1821-9-1)
van wtm is as
fami/us jar fits ciintribulinm sertsury physiology aatt iKawphysiulngr aehicveaienls
as far his in physics.
in
Having defended the view that consciousness involves communication, even in the case of raw sensations, I want to make it clear that I do not think this communicative role explains it. There would probably be something mysterious about the subjective sensation of redness that would remain after one had a complete account of the underlying mechanisms, even if this account included a description of the mechanisms relating it to social interaction and communication. This 'something' seems likely to lie beyond the range of current scientific approaches, but linking it to the extraordinary complexities of social interaction at least relieves one of the urge to attempt explanations in over-simple physiological terms.
Implications What arc the implications of the view that consciousness is an aspect of communicating, interacting, brains? Well, the first idea one must discard is that consciousness is simply a property of a brain by itself. This is where 1 started, for it was because I had accepted the usual view that consciousness is just an attribute of the brain, like mass or colour, that I was puzzled when I realized that there was probably no remnant of this attribute in the single cells I was recording from. Consciousness becomes a bit less bizarre and paradoxical if you accept that it has to do with communication and interaction between brains, and not with a single brain by itself or any part of it by itself. To illustrate the benefit of this change, take the problem of giving the reason why brains can be conscious but stones cannot. If consciousness was a simple attribute of either by itself, you would have to say the difference was the result of some physical difference between the two, and this would start you on a lengthy, but ultimately false, trail. Once you accept that consciousness is a matter of communication and interaction, the problem vanishes, for these interactions have a quite different degree
828
373
Of complexity in stones and brains. We may have landed
829
374 that the powers and limitations of consciousness arc important in this respect. Second, this insight certainly does not explain the aspect of consciousness that is mysterious and apparently beyond the range of the reductionist scientific approach; by relating it to the enormously complex area of our social interactions it may, however, point us in the direction that will enable us to understand it better.
References 1 Gray, Jeffrey A . (1971) T h e mind-brain identity theory as a scientific hypothesis. PhilosophicalQuarterly, 21: 247-54. 2 Gray, Jeffrey A . (1986) T h e mind-brain identity theory as a scientific hypothesis: a second look (this volume). 3 Barlow, H . B . (1972) Single units and sensation: a neuron doctrine for perceptual psychology? Perception, 1; 3 7 1 - 9 4 . 4 Barlow, H . B . (1985) T h e role o f single neurons in the psychology of perception. { T h e Barden Lecture of the Experimental Psychology Society) Quart. J. exp. Psychol. 3 7 : 1 2 1 - 4 5 . 5 Barlow, H . B . , Hill, R . M . and Levick, W . R. (1964) Retinal ganglion cells responding selectively to direction and speed of image motion in the rabbit. J. Physkl. (London), 173: 377-407. 6 Perren, D . 1., Rolls, E . T . and Caan, W . (1982) Visual neurons responsive to faces in the monkey temporal cortex. Exp. Brain Res. 47: 329—42. 7 Perren, D . I . , Smith, P. A . J . , Potter, D . D . , Mistlin, A . J . , Head, A . S., Milner, A. D . and Jeeves, M . A . (1985) Neurons responsive to faces in the temporal cortex: studies of functional organisation, sensitivity to identity and relation to perception. Human Neurobiologf, 3:197-208. 8 Humphrey, N . K . (1980) Nature's psychologists. In B . D.Josephson and V . S. Ramachandran (eds), Consciousness and ike Physical World, eh. 4, pp 57-80. Oxford: Pergamon. 9 Barlow, H . B . (1980) Nature's joke, l n ibid., eh. 5, pp. 81-94. 10 Humphrey, N . K . (1984) Consciousness Regained. Oxford: Oxford Univ. Press, 11 Craik, K . J . W . (1943) 77ie Nature of Explanation. Cambridge: Cambridge Univ. Press. 12 Baron-Cohen, S . , Leslie, A . M . and F r i t h , Uta (1985) Does the autistic child have a 'theory of mind'? Cognition, 21:37—46. 13 Leslie, A. M . (1987) Pretense and representation: the origins of 'theory of mind'. Psychological Review (in press). 14 L a n d , E . H a n d M c C a n n , J . J . (1971) Lightness and Retinex theory. Journal of the Optical Society of America, 6 1 : 1 - 1 1 . 15 Helmholtz, H . von (1896) Handhuch der Physiologisdten Opik, Zweite Auflag. Hamburg und Leipzig: Voss.
830
Reprinted with permission from Brain Mechanisms and Consciousness. pp. 402-417. 1954 © 1954 Blackwell Scientific Publications Ltd.
THE
PROBLEM
OF CONSCIOUSNESS
AND
INTROSPECTION By D.
O.
HEBB
It has not been easy to see what, exactly, should be my task in this symposium. I do not think it worthwhile here to elaborate on my own speculations on neurophysiology (Hebb, 1949), at least not in detail. Their purpose was to clarify and guide certain lines of experiment that are still going on. Instead, I have chosen to try to define the problem of consciousness and see what limits it must have for the scientist (as contrasted with the theologian); including particularly the question of'introspection', using the term in a general sense. The capacity of the human being to know (and report) what is going on inside himself has seemed to some writers to put his behaviour beyond the reach of scientific explanation, and we must ask whether this is so. Communication between disciplines is always defective. Most psychologists, like me, find it hard to keep up with both psychology and physiology. The rapid advance of neurophysiology these days makes it especially difficult. But also, too many physiologists seem unaware that psychological knowledge equally relevant to the problems we are considering — is not still mired in doubts and difficulties of the runeteenth century. A n eminent neurologist recently complained that the psychologists had made aphasia so complicated that no one could understand it. A plain man, he thought, might brush all the hair-splitting aside and treat the matter in a common-sense way. Unfortunately for the resulting discussion, it is not psychologists but God who complicated aphasia: the complexities are real, and must be taken into account if the problem is to be solved. The problem of consciousness is equally complex and no more likely to yield to a simple common-sense analysis; yet it is exactly in this area that the appeal to common sense has always seemed most cogent.
831
403 THE
COMMON-SENSE
VIEW
OF
CONSCIOUSNESS
The word conscious is used of course in several ways. One is simply to designate the responsiveness of the normal waking animal. In this sense 'conscious' and 'unconscious' are descriptively clear, as 'awake' and 'asleep' are clear. O u r concern is with another usage that refers to something inside which is thought of as a causal agent. For common sense, consciousness is a primary fact of existence, part of the initial data from which thought begins. It may even be regarded as the primary fact or datum. Eccles (1953) has put this clearly, quoting several writers to the effect that one knows the inner world of consciousness directly, the outer world only by inference from the inner data. The events of this inner world largely determine what one does; it follows therefore that a physiological explanation of human behaviour cannot be a complete one, for the redness of the morning sky, the feeling of sorrow, are by every common-sense criterion totally different in nature from any patterning of electrochemical activity in nerve cells. Thus speaks common sense: but is it, perhaps, also non-sense and inconsistency; Are these things obviously truer Every scientist knows that observation is fallible, that preconceptions can blind an observer. Should we give more weight to common sense in psychology than we do in physics; It is a matter of simple observation that the sun moves from east to west while the earth stands still. Common sense will tell anyone that matter and energy are wholly different things. A great part of the success of the physical sciences has resulted from assumptions that are as preposterous, to common sense, as Newton's first law of motion (though this may no longer seem preposterous, now that we have absorbed it into everyday thinking and 'see' friction wherever we see motion). As Ryle (1949) has shown very forcefully, the common-sense view of consciousness leads directly to absurdities. The only evidence for it, besides, is the demonstrably unreliable evidence of introspection. For some time, psychologists have had to abandon introspection as a crucial argument. The test of any theory is in what a subject does. Though the question of consciousness and
832
40
+
introspection is still an uneasy one, it has become quite clear that introspective evidence is a dangerous tool to handle carelessly, and that theory must rest m a i n l y o n a foundation o f observable behaviour. Others, less familiar w i t h the painful steps b y w h i c h this clarifying position was reached, still cite c o m m o n conviction to oppose w h a t they call 'behaviourism'. W h a t this amounts to is (a) a refusal to use the scientific m e t h o d (especially the scientific willingness to try out 'improbable' assumptions) and then (b) c o n c l u d i n g that the p r o b l e m o f man's behaviour is not scientifically soluble. I do not assert that it is soluble: w e have yet a long w a y to go before being sure that the solution w i l l be achieved; but there is no possible basis for asserting that a problem is insoluble because w e have not solved it yet. I have recently tried to s h o w (Hebb, 1951) that the psychologist w h o avoids physiological conceptions merely succeeds i n avoiding m o d e m ones, and is likely to have his tmnlcing dominated b y older ideas, vintage o f 1890. B u t the physiologist w h o analyses behaviour without k n o w i n g something o f modern psychology is in rather worse case. He is dominated b y the psychology and philosophy o f 1890 at the latest: and this i n turn is largely the physiology o f 1850, o f H e l m h o l t z and Johannes Miiller, w i t h their v e r y limited knowledge o f the nervous system. T h e coinage o f ideas wears smooth, their minting is hard to trace, and the common-sense idea that seems to us today to be obvious and inescapable is sometimes the daring speculation o f a hundred (or t w o hundred) years ago. W e must not take for fact, nor allow theory to be dominated by, the crude hypotheses o f eighteenthcentury tliinkers, brilliant as they were. I propose to y o u accordingly that the existence o f something called consciousness is a venerable hypothesis: not a datum, not directly observable, but an inference f r o m other facts. I propose that y o u r conviction that y o u are aware o f your awareness (and aware o f y o u r awareness o f y o u r awareness;) m a y be illusion, and must not be made the basis for analysis o f brain function. I grant at once that y o u are aware (or conscious) o f y o u r environment — in fact, I grant that I a m , t o o - j u s t as I grant that the w o r l d revolves o n its axis, or that something k n o w n as an atom exists. T h e hypothesis that there is something w e call consciousness is a
833
405 good one (what the something is may be another matter). All this is not simply hair-splitting, but affects research in a practical way; for now, i f we accept this, not only do century-old contradictions disappear but we regain freedom to modify this hypothesis of ours and see what the experimental consequences are. Furthermore, we can return to the proper limitations of the scientific method. Any attempt to 'explain how nervous impulses can be translated into a mental experience' is, in the words of the preacher, an attempt to unscrew the inscrutable and to foresee the ways of the Lord. The phenomenal world (of brightness, colour, sound, sweetness, pain) is the starting-point of thought, its ultimate 'reality'. As mature theorizing individuals we can, in thought, choose other starting-points (e.g. a universe of electrons, permeable membranes, nerve impulses) and see how far we get; but when we try to have two starting-points we ask for trouble. E x planation is a process of reducing one set of ideas to another; it is evident that such a series of reductions must have an end point. W e cannot reduce A to B , and at the same time B to A : we cannot make the world of experience the ultimate reality and then try to explain it as a set of nerve impulses, simply because this is logical nonsense. It is intellectually respectable to assume the existence of the inner world as primary, and explain the physical world in terms of it (Berkeley's procedure); or, scientifically more productive, to assume as working hypothesis that all behaviour is detennined by physiological processes; but not to have both simultaneously. Eventually the physiological hypothesis may fail, but we cannot know that yet. And may I add that no one need believe a hypothesis, and may if he likes accept it only as a means o f disproving it; CONSCIOUSNESS
AS K N O W N
F R O M BEHAVIOUR
Assume then that consciousness is something within the head which determines behaviour, and which can be studied through that behaviour. One quite respectable but aged hypothesis is that consciousness is a function of another entity, the soul; we need not quarrel with this, but shall consider the alternate hypothesis that consciousness is simply a function of the brain. T o do so, we must
834
406 ask w h a t the behaviour is i n w h i c h consciousness is manifested, and w h a t k i n d o f neural processes w o u l d produce it. W e are really asking here w h a t the distmguishing marks o f 'higher' behaviour are. It m a y help y o u to follow this part o f m y discussion i f I summarize its gist, first, b y saying that they are found i n general i n the behaviour that has obliged psychologists to conclude that the higher mammals are certainly capable o f ideation; and secondly, i n certain primate behaviour w h i c h indicates, I believe, an increasing flexibility o r freedom o f m a n i p u lation o f ideas, culminating i n h u m a n language. N o w to support this i n somewhat greater detail. T h e least disputed criterion o f consciousness is i n certain verbal behaviour (introspective report) but this is not essential; for w e have no real doubt that a deaf-mute is conscious, even w i t h o u t sign language. W h a t are the other criteria! Responsiveness is essential, but not sufficient; w e need not e n d o w amoeba, or the tail end o f a spinal dog, w i t h consciousness. I f w e put amoeba at one end o f a scale, w a k i n g n o r m a l m a n at the other, it w i l l be helpful to see w h a t trends i n development lead up to the typical h u m a n behaviour. N o one I think can deny c h i m panzees something like consciousness i n the h u m a n sense; that is, except for language, their behaviour contains the same evidence b y w h i c h w e identify consciousness i n m a n . It is not m y purpose to ask h o w far d o w n the scale this identification is possible, but o n l y to note that behaviour is not unique to m a n and that its essentials m a y be better recognized w h e n it can be observed i n different species. I hope I do not shock biological scientists b y saying that one feature o f the phylogenetic development is an increasing evidence o f w h a t is k n o w n i n some circles as free w i l l ; i n m y student days also referred to i n the H a r v a r d L a w , w h i c h asserts that any w e l l trained experimental animal, o n controlled stimulation, w i l l do as he d a m n e d w e l l pleases. A m o r e scholarly formulation is that the higher a n i m a l is less stimulus-bound (Goldstein, 194.0). B r a i n action is less fully controlled b y afferent input, behaviour therefore less fully predictable from the situation i n w h i c h the animal is put. A greater role o f ideational activity is recognizable in the animal's ability to 'hold' a variety o f stimulations for some time before acting o n them (Hebb and B i n d r a , 1952), and i n the
835
407 phenomenon of purposive behaviour. There is more autonomous activity in the higher brain, and more selectivity as to which afferent activity will be integrated with the 'stream of thought', the dorninant, ongoing activity in control of behaviour. Traditionally, we say that the subject is 'interested' in this part of the environment, not interested in that; in these terms, the higher animal has a wider variety o f interests and the interest of the moment plays a greater part in behaviour, which means a greater unpredictability as to what stimuli will be responded to and as to the form of response. This is not mdeterminism, but does tell us much about the kind of mechanism we are dealing with. Also, the phenomenon of purposive behaviour, and the expectancy of the future which it entails, does not imply teleology but a capacity for ideation. When B has followed A repeatedly in the past, e.g., the animal on seeing A 'imagines' B (the idea of B arising by some process o f learning or association). The behavioural evidence makes it clear that theory must provide somehow for foresight. Consciousness in the higher organism constantly involves ideas about the future, integrating these with ideas or memory o f the immediate past. The mark of higher behaviour is not only that it is an adjustment to more of the complexities of present stimulation, but also that the temporal span covered by incidental memory of the past, and anticipation of the future, is greater. What then of the mechanism that makes all this possibles There are two main problems: (i) about the nature of idea, image or concept — the link in the chain of thought — and (2) the nature of the linking, the temporal organization of ideas. 1. The first of these will not be discussed in detail. M y earlier discussion tried to deal with it by the conception o f the 'cell assembly', without attempting to be very specific about the locus of these processes. Is is now clearer than ever that the question of locus must be dealt with, especially in view of all the subsequent physiological studies that are the theme of this symposium. Penfield's (1952) work on the temporal lobe, and a number of other studies reviewed by Milner (Milner, 1953), further emphasize the problem of localization of function, but there is not time to enter on this here. 2. The second problem is that of the temporal organization o f
836
408 ideas. This emerged as the central issue in Humphrey's (1951) important review and analysis of the classical psychological studies. It is significant that Lashley (1951), discussing different evidence and from a strictly behavioural point of view, arrived at what amounts to exactly the same problem, o f the serial ordering o f thought or behaviour. Let me represent the issue schematically, as in Fig. t, regarding the train of thought first as a single series of discrete events (top line). W e can then represent the classical O
O
O
O
O
O
O
O
O
A A A A* '" A
5
/nsrtSrtSA\
S A S
A\ 5 r\S FIG.
5
n
r\S (A
t
Schematizing thought. Regarding the process as a series of discrete events in time, as in the top line, the motor theory of thought appears in the second line, each central event being determined by sensory input alone. A modification is presented in the bottom line, in which the centra! event is determined jointly by sensory and central facilitations. S, sensory; M, motor; C, central.
stimulus-response or motor theory of thought, which Lashley has so effectively criticized, as in the middle line: each 'idea' is fully determined by afferent input, and simply amounts to through transmission (as at the right). A significant change is made in the bottom line (if each of the central events in the diagram is a 'cell assembly', this embodies the main feature o f the theory (Hebb, 1949) published elsewhere). The central event is now determined by two influences, one sensory, one central. The central influence is a facilitation from the preceding central activity. A t each step, therefore, a selection is possible. In Fig. 2, broken circles represent potential activities, facilitated from one source only; closed circles, ones which actually occur because o f facilitation from both sources. This also opens up a further possibility, shown at the right: a strong central facilitation might
837
4°9
by itself arouse a second series, in parallel w i t h the sensorily guided one. T h i s w o u l d represent pure ideation. I do not o f course try to show the real c o m p l e x i t y o f such processes, but perhaps it is w o r t h w h i l e to present F i g . 3 also, as representing a little more adequately the sort o f thing that is implied: a constant sensory influx and a c o m p l e x interaction o f central facilitations, w i t h m a n y activities i n parallel w h i c h yet are m o r e or less unified. A t stage A , e.g., the organism m i g h t be thought o f as having a single purpose or idea; w h i l e B and C m i g h t represent the entertaining o f t w o ideas, or thinking o f t w o eventualities; and so forth.
5
A
5
§
m s
5
A
FIG. 2
Selectivity in thought. Broken circles represent central actions which might occur i f facilitated both sensorily and centrally, closed circles ones that do occur. At the right, strong centra] facilitation arouses a second series in parallel.
SPEECH AND
INTROSPECTIVE REPORT
A n d n o w let us see h o w this approach can be applied to the problem o f speech and introspective report ( i n the general sense that a subject is able to tell y o u w h a t he is thinking about). T h i s certainly must be dealt w i t h ; otherwise, i f the ability is left u n accounted for, it w o u l d be bound to leave some suspicion that the 'inner w o r l d ' (Ryle, 1949) exists after a l l , and support doubts about the w h o l e behavioural approach. T h e preceding section suggested that there m a y be t w o or more series o f ideas n i n n i n g i n parallel. W e m a y l o o k first at some comparative data suggesting that this possibility is particularly relevant to the development o f speech. T h o m p s o n and I (Hebb and T h o m p s o n , 1954) nave tried to s h o w that speech arises as the end product o f a phylogenetic increase i n the ability to entertain independent ideas or trains o f thought at the same time. A n earlier stage o f development is achieving an independence o f ideation
838
411 from the environment in which the animal finds himself; subsequently, at a higher level, independence o f ideas one from another so rhat they may be freely combined in different ways. In the course of analysis we defined three levels of communication. The lowest (a) is reflexive, or stimulus-bound, exemplified in most or all insect interactions or in the emotional cries of higher animals. A higher level (b) is purposive communication, involving ideation but to a limited extent: as when a dog 'asks' to go out, or a chimpanzee 'begs' with his hands for an object out of reach. The highest level (c) is syntactic behaviour, whether verbal or manual: true language, not found in any animal but man. The essential feature o f syntactic behaviour is that it shows the free combination and re-combination o f two or more representative or symbolic gestures or sounds, purposefully. The parrot of course is able to produce human words, and possibly on occasion uses these speech sounds purposefully; but neither parrot nor specially-trained chimpanzee (Hayes and Hayes, 1951) is capable o f the behaviour of the two-year-old child who can use a vocabulary o f four words to produce the separate propositions 'I thirsty', 'Mommy thirsty' and 'I not thirsty', and control his environment with them. Too much emphasis has been given in the past to the special problem of vocal language. The chimpanzee and lower animals are just as incapable of sign language as o f spoken. The real problem of speech is not vocalization, but the capacity for mampulating ideas or symbolic acts. The behavioural evidence of phylogenetic development in this respect is fascinating — it appears, e.g., in the chimpanzee's capacity for deceitful attack, or in his empathic identification with another — but I shall just say here that though the great apes are definitely not capable of language they may still be close to a liminal level of intellect that would make language possible. The comparative evidence, as well as much of the literature on aphasia, clearly makes thought prior to speech, not conditional on it. N o w let me return to schematizing, to see what we can do in principle with the verbal behaviour that informs us about otherwise unobserved events within the organism. T o emphasize that this is schematic, let Us talk about a 'thinking machine' such as an electronic brain. Suppose that we cannot inspect its internals
840
412 while it is working, but still must be able to find out what activities occur in a given operation. That is, could we design the machine so as to report on its own activity: one that will 'introspect' i First, it is routine that we can make a machine that when stimulated (buttons pressed, tapes fed in) will produce a given kind o f behaviour. Secondly, it would be routine to build a machine that could on request repeat a given behaviour without the original stimulation. Thus there is no special problem, in principle, about the capacity of a man to respond adequately when you say to him Repeat what you just did — especially since it will often be found that unless he is prepared in advance for this request he will often fail. He can however be 'set' to hold the original stimulation for a repeat performance, as a machine could be. Memory will account for this, and this is not the real problem of introspection. Thirdly, there is no difficulty about having a machine with more than one kind of output, so that when it is set to multiplying or integrating, with an answer to appear on a tape, it could also have a loudspeaker system that would announce Now I am multiplying while multiplication goes on. O r it could announce I have just multiplied, after completion of the task. Here again, therefore, there is no special problem about the human being's ability to make such verbal report about other activities. It is important to realize that I am not suggesting that there is no problem about speech, or about memory and tliinking. There is, o f course. What I am asking is whether 'subjective report' need be treated as a special case, a difficulty of a different order from other intelligent behaviour. So far, there is no difficulty in this sense. But introspective report goes further. C a n we find a way of asking a machine how it does a task; Its answer need not be complete (or perfect) to be an analogue of human introspection, for all a man really tells us is whether one of his activities involves processes related to some other activity which can be named. That is, when the subject says he uses visual imagery in mental arithmetic, what he really tells us is that his thought involves processes that characteristically occur in seeing things. Thus wc could ask a machine how it integrates, e.g., as follows:
841
413 We want to know perhaps whether this involves the processes that occur in addition; so we set the machine to integrating, and as soon as it is done we feed in an addition problem at a subliminal level of stimulation. I f the tubes of the 'add' system (3, in Fig. 4 ) are still warmed up, its circuits still showing some reverberation or the switches that constituted the circuits still closed, a low-level add stimulus will produce an answer that we would not get i f this system had not been active irnrnediately before. Similarly with multiplication or subtraction. It is clear that if system 3 (Fig. 4 ) is highly damped and does not have after-activity, or reverts at once to its pre-activation state,
Integration Response: Addition.
Add FIG.
4
'Introspection' in a calculating machine. See text.
this attempt at scaiining would not work. But just such failures occur in human introspection. The 'imageless thought' problem of classical psychology was due to the discovery that sometliing occurs in thought that the most careful introspection cannot detect (Humphrey, 1951)- It is clear too that the answers we get will be completely a function of the questions we ask: the machine will not report on activities it is not asked about. But this also fits. One of the most striking features of the history of psychology is the difference between introspective reports at different stages of theoretical development. The introspections o f James M i l l about 1830 bear no faintest resemblance to those of William James 60 years later, but they do bear a very nice relationship to the fact that Mill had only the crudest sort of conception of sensory function to work with. The a priori theoretical proposal of John Stewart Mill, his son, that there might be a 'mental chemistry' among sensations and ideas, fusing the elements of thought into qualitatively different, simple compounds instead of BR
842
414 aggregations, appears to have permanently modified subsequent introspective reports. It seems that introspectors have always been able either to find what they were looking for or not to find it, but in general did not make observations unrelated to their theoretical conceptions of the human machine. The scanning works for the questions (explicit or implicit) that are put to the organism. The scanning of course need not be after the event, but can also be arranged in advance. W i t h our integrating machine (Fig. 4) we feed in a low-level add stimulation, then follow this up with the stimulus to integration. I f our machine is able to hold the add stimulus (as a man could), and i f integration does require the activity o f the add system, the second stimulus may summate with the effects of the first. The rnachine will produce an answer to the addition problem at the same time as, or instead of, the integration, showing that the processes of integration are related to those of addition. Put this in human terms: we tell the subject we want to find out whether he has coloured imagery, which sets him tluhking about colours (or induces a low-level activity in the 'assemblies' involved in seeing colours); then we tell him to imagine a familiar scene. I f the imagination does involve the colour systems, the two stimulations may summate, and the subject say 'red' or 'green'. A l l this is worded in terms of overt stimulation and overt response; but there is no difficulty in principle about supposing that the stimuli may arise in the thought process ('I wonder i f I have coloured i m a g e r y ? . . . let me recall what I had for breakfast, and see whether I recall it visually, in colour . . .') nor about supposing that the answer may remain in the form of thought ('ah, no colours'), an activity which is subliminal for producing overt response directly but which could be demonstrated by the method of summation. I do not think it profitable to elaborate such mechanical schematizing further. The crudities of some of the statements above are bound to be embarrassing both to speaker and listener, they are so far removed from the richness and subtlety of the thought process; though we must employ them at present for communication, because we have no better terms or conceptions in which to convey the sort of thing I
843
415
have been talking about, there is no object in continuing to elaborate them beyond the point which shows how, in principle, one may see the phenomenon of introspection included within the purview of a physiological theory of thought and consciousness. I N T E L L I G E N C E AND E N V I R O N M E N T A L S T I M U L A T I O N
After these general and more or less philosophical reflections, let me conclude by briefly describing some of the results of an experiment, now being carried out at McGill by W . Heron, W . H . Bexton and T . H . Scott, which seems very relevant to the subject-matter of this symposium. It appears to show that normal functioning by the human brain is much more dependent on a varied sensory input than we have suspected in the past. Even though sensory connections are intact, and the sense organ being stimulated at a level of intensity vvithin the normal range, i f this stimulation is monotonous instead of being normally varied there is soon evidence of intellectual deterioration. The subjects are male college students, paid to stay in a quiet cubicle on a comfortable bed 24 hours a day. They wear goggles which prevent pattern vision but admit Hght, and cardboard cuffs extending beyond the fingertips which permit free joint movement but little actual perception. A small speaker system provides communication with the experimenter. W e have been able to induce our subjects to endure the boredom produced by these conditions for periods of two to six days. Tests of problemsolving are given before entry into the cubicle, during their stay in it, and afterwards. Control subjects are given the same tests at the same intervals. The results show impairment in problem-solving while in the cubicle, in orally-presented problems such as mental arithmetic and anagram-type problems. Subjectively, also there are reports of inability to concentrate. O n leaving the cubicle, there is apt to be some defect of sensori-motor control, and a mild state of confusion; and significant impairment is demonstrated in the Kohs-Block and Wechsler-Digit-Symbol tests of intelligence, as well as in speed of writing. The most staking result, however, is the occurrence of hallucinatory activity, chiefly visual but also auditory and somesthetic
844
4i6 in a few subjects. The phenomena are quite like what is described in mescal intoxication, as well as what D r . Grey Walter has produced by exposure to nicker. Also, there are rare cases of hallucination in aged persons without psychotic tendency, which as with our subjects does not depend on special stimulation (our method is more like a lack of stimulation) or chemical agents. W i t h our first subjects we did not ask about such phenomena, and we do not know how high the frequency was. The last fourteen subjects were asked to report any 'visual imagery' they observed, and our report is based on these. It appears that the activity has a rather regular course of development from simple to complex. The first symptom is that the visual field, when the eyes are closed, changes from a dark to a light colour; next there are reports of dots of light, lines, or simple geometrical patterns. All 14 subjects reported such imagery, which was a new experience to them. The next step, reported by 11 subjects, is seeing something like wallpaper patterns. Then come isolated objects, without background, reported by 7 out of 14, and finally integrated scenes usually containing dreamlike distortions, reported by 3 of the 14. In genera], the subjects were surprised by these phenomena, and then were amused or interested, waiting for what they would see next. Later, some subjects found them irritating, and complained that their vividness interfered with sleep. There was some, but not much, control over content; by 'trying', the subject might see certain objects suggested by the experimenter, but not always as he intended: thus one subject, trying to visualize a pen, saw an ink blot; or, trying to visualize a shoe, saw a ski boot. The imagery usually disappeared when the subject was doing a complex task such as multiplying three-place numbers in lus head, but not if he did physical exercises or talked to the experimenter. In addition, there have been reports of hallucinations involving other senses. One subject could hear the people in his visual hallucinations talking, and another repeatedly heard the playing o f a music-box. There were also four subjects who reported kinesthetic and somesthetic phenomena. One reported seeing a miniature rocket ship discharging pellets that kept on striking his arm, and one reported reaching out to touch a doorknob he saw
845
417 before rum and feeling an electric shock. The other two subjects reported a phenomenon which they found difficult to describe, but said that it was as i f there were 'two of me' — two bodies side by side in the cubicle, and in one case these two bodies overlapped, partly occupying the same space. In addition there were reports of feelings of 'otherness' and bodily 'strangeness' in which it was hard to know exacdy what the subject meant. One other subject said 'my mind seemed to be a ball of cotton wool floating above my body' and another reported that his head felt detached from his body. There are f a m i l i a r phenomena in certain cases of migraine, as described recendy by Lippman, and earlier by Lewis Carroll in Alice in Wonderland. As Lippman points out, Lewis Carroll was a sufferer from migraine. These results seem to us to bring our understanding of normal brain function close to the developing knowledge of the reticular system and the arousal reaction. W e must, it seems, recognize two distinct functions of a sensory event: it may arouse or guide a specific response; but it has also a non-specific function in maintaining the normal waking organization of brain function. The cortex and D r . Penfield's 'centrencephalic system' are not uke a calculating machine operated by an electric motor, which can lie idle, without input, for indefinire periods and then respond immediately and efficiently; instead, it must be kept warmed up and working by a constandy varied input during the waking period at least, i f it is to function effectively. The evidence from emotional behaviour, on the other hand, suggests strongly that the sensory input must not be too varied — that is, must not be too unfamiliar, or present too great a problem for the intracranial machinery — or emotional breakdown may occur (Hebb, 1949). Thus the higher animal continually behaves in such a way as to seek an optimal degree of disturbing stimulation. This is a long story, the evidence is complex and undoubtedly may be interpreted in other ways, but we feel that there is begirming to appear a convergence between the physiological and psychological evidence in this general area that constitutes a long step forward in our understanding of man and opens wide new areas for research, psychological and physiological.
846
INDEX
Addressing (in dynamic processing) 6, 59, 63 Algorithmic complexity 6, 30, 31, 32, 33, 34 Algorithmic information content 32, 33, 35, 37 Amnesia 27, 51, 59, 68 Analog, discrete 4, 6, 10, 18, 21, 28, 32, 34, 36, 37, 40, 41, 51 Anesthesia 15 Arithmetic vs. logic 10, 28, 65 Artificial intelligence 1, 4, 9, 19, 26, 30, 32, 37, 41, 56, 61, 62, 65 Assembly (of cells) 15, 17, 23, 51, 55, 57, 58, 64, 68 Associative memory 6, 9, 22, 23, 28, 29, 32, 51, 58, 59, 65, 66 Attention 14, 22, 26, 36, 44, 45, 52, 58, 59, 60, 61, 63, 68 Attractor networks 6, 28, 29, 32, 57 Audition 2, 14, 20, 40, 41, 42, 59, 61, 62 Autism 67 Back-propagation algorithm 1, 29, 32, 46, 59, 65 Basal ganglia 49, 58, 60, 61 Bat 14 Bayesian theory 11, 12, 18, 22, 33, 34, 65 Bee 67 Binding problem 43, 45, 55, 58, 59, 60 Bird 42, 68 Blindsight 27, 61 Bottom-up, top-down 4, 26, 36, 42, 43, 58, 59, 60, 61, 62 Cat 14, 21, 36, 37, 45, 48, 50, 55, 58, 61, 62, 68 Categorisation 13, 14 Cerebellum 4, 22, 23, 26, 50, 61, 62, 65 Circuits (neural) 42, 48, 49, 50, 51, 56, 61, 62, 63, 64, 65 Claustrum 58, 62 Clocks 28, 62 Color vision 13, 14, 21, 26, 38, 43, 48, 67 Command neurons 42, 45 Complexity 3, 5, 6, 7, 8, 9, 26, 33, 34, 36, 37, 48, 66, 67 Compositionality 62, 68 Computer 4, 9, 10, 14, 21, 24, 26, 28, 30, 66, 68 Consciousness 1, 2, 7, 13, 14, 21, 26, 27, 44, 60, 67, 68 Corpus callosum 13, 23, 55 Cortex 7, 14, 17, 21, 22, 23, 24, 26, 27, 36, 37, 38, 43, 44, 45, 46, 47, 48, 49, 51, 52, 55, 56, 58, 59, 60, 61, 62, 65, 68 847
Development (ontogenesis) Distributed, localised Dynamical systems Dyslexia
7, 9, 23, 31, 33, 36, 45
14, 21, 26, 33, 45, 46, 49, 51, 55, 62, 64 6, 28, 29, 32, 34, 64, 65
33
E E G (electroencephalography) Efferent copy
21, 26, 55, 66, 68
13, 42
Effort (mental)
19, 45, 49
Eidetic images
15
Electrode recordings
4, 6, 17, 21, 26, 37, 40, 43, 44, 45, 46, 47, 48, 51, 52,
53, 54, 56, 57, 58, 59, 60, 61, 62 Emotion
2,7,27,49,61,67,68
Encoding
21, 22, 28, 32, 35, 36, 37, 39, 40, 41, 61, 62, 66
Entropy information entropy functional entropy
5, 18, 21, 22, 37, 38, 39 32, 35
entropy of a learning machine maximal entropy relative entropy
35, 37
thermodynamic entropy Esthetics
5, 30, 34
2, 21, 27
Evolution Eye
29, 33
11, 32, 33
6, 7, 12, 13, 23, 26, 28, 30, 33, 43, 61, 62, 65, 66, 67, 68
7, 14, 21, 23, 27, 37, 38, 46
E y e movements
13, 15, 41, 44, 45, 48, 52, 53, 54, 61, 68
Fast-modifiable synapses Fault tolerance
58, 64
6, 10, 21, 28, 40, 45, 55
Feature detectors
9, 14, 21, 22, 23, 26, 36, 37, 38, 40, 42, 43, 45, 55,
58, 60, 65 Feedback, feedforward
22, 23, 29, 35, 36, 41, 48, 49, 51, 53, 56, 58, 59,
60, 61, 65 Firing rate (impulse frequency)
21, 28, 36, 37, 40, 41, 42, 43, 45, 46, 47,
48, 51, 52, 53, 54, 55, 56, 57, 58, 65, 67 Fish Fly Frog
36, 41 23, 26, 40 21, 27, 36, 40
Frontal lobe
44, 45, 49, 53, 59, 60, 61, 62
Fusion of sensory modalities Generalisation
15, 32, 33, 34, 35, 36, 65, 66
Gestalt psychology Guinea pig Hallucination
14, 42, 44, 54, 55, 59, 60, 61
6, 13, 19, 26, 43, 55, 61
23, 58 15, 68
Habituation, sensitisation
21, 22
848
Hebb learning rule Hippocampus
6, 22, 28, 29, 36, 51, 58, 59, 64
49, 50, 59, 60
Homunculus
13, 14, 27, 37, 61
Ill-posed problems Illusion
26, 32, 64, 65
22, 26, 48, 64, 68
Image (mental) Immunology
14, 15, 16, 17, 27, 61, 62, 68
25
Inductive inference
11, 12, 13, 15, 19, 22, 27, 32, 36, 48, 67, 68
Inferior colliculus Information
flow
42 18, 20, 21, 26, 37, 38, 40, 51, 56, 62, 63
Information storage
4, 22, 23, 28, 29, 32, 51, 59
Information theory
18, 19, 20, 21, 22, 26, 28, 32, 33, 34, 36, 37, 38, 40, 66
Information (mutual) Innate vs. learnt Intelligence
20, 22, 36, 37, 38, 40
13, 14, 21, 25, 26, 66
5, 27, 28, 30, 35, 65, 66
Interpolation (of function) Invertebrates Language
32, 65
7, 23, 27, 57
8, 10, 13, 14, 18, 19, 20, 21, 22, 26, 33, 58, 61, 62, 66, 67, 68
Lateral geniculate nucleus
43, 44, 58, 61, 63
Lateral is at ion (of brain hemispheres) Learning (supervised, unsupervised)
13, 17 13, 18, 20, 21, 22, 23, 25, 26, 27, 28,
29, 32, 33, 34, 36, 39, 44, 45, 46, 51, 57, 59, 62, 64, 65, 66 Levels (of brain study) Limbic system
49, 59, 50, 61
Linear, nonlinear Logic
1, 3, 4, 8, 21, 26, 36, 56, 67, 68
6, 26, 28, 34, 36, 37, 38, 40, 42, 45, 46, 48, 53, 64, 65
6, 12, 21, 22, 56, 65, 66
Logical depth
10, 35
L T P (long-term potentiation) Mammals
51, 59
7, 21, 22, 24, 37, 45, 49, 51, 58, 61, 62, 68
Map (topographic, nontopographic)
4, 14, 24, 31, 36, 42, 43, 45, 46, 50,
51, 53, 54, 58, 61, 62, 63 Meaning
14, 18, 58
Memory
6, 15, 18, 20, 23, 26, 27, 28, 32, 45, 51, 52, 59, 60, 65, 67, 68
Memory capacity
20, 28, 29, 51, 57
Memory (long-term) Memory (short-term) Metastability Mind
20, 26, 51, 57, 59, 60, 64 20, 26, 44, 51, 57, 58, 59, 64
5, 28, 30, 34
1,2,21,27,67
Monkey
14, 17, 21, 26, 27, 36, 38, 43, 44, 45, 46, 47, 48, 52, 54, 55, 57,
59, 63 Motor
field
Motricity
14, 45, 53, 54, 65 14, 17, 21, 42, 46, 49, 50, 53, 54, 55, 59, 60, 61, 62, 63, 65, 67
849
Mouse, rat
14, 23, 37, 49, 51, 58
Multiplicative inputs (dendritic processing) Neural code
23, 56, 63, 65
4, 6, 10, 21, 28, 36, 37, 38, 40, 41, 42, 45, 46, 53, 54, 55,
56, 61, 64 Neuroanatomy
14, 22, 33, 42, 43, 44, 45, 46, 49, 50, 53, 56, 58, 59, 60,
61, 62, 63 Neuron (models of a single) Neuropsychology Noise
6, 23, 24, 28, 32, 40, 41, 51, 56, 65
13, 14, 27, 33, 43, 44, 45, 46, 49, 59, 60, 61, 62
6, 10, 12, 20, 21, 26, 28, 30, 32, 34, 36, 37, 38, 40, 47, 56, 57, 61, 66
Novelty detection Olfaction
22, 25, 39, 59, 64
23, 55, 62, 64
Optimisation
6, 24, 26, 29, 30, 31, 32, 33, 36, 37, 38, 61, 65
Oscillations
42, 51, 55, 62, 64
Pain, pleasure
7, 14, 15, 59, 67
Pandemonium
9, 22, 61
Parallel, serial
4, 21, 26, 29, 30, 31, 33, 36, 42, 45, 48, 55, 56, 58, 61,
62, 65, 68 Parietal lobe
43, 44, 45, 46, 48, 52, 59, 61, 62
Pattern recognition Perception
26, 28, 36, 44, 51, 57, 59, 60, 61, 62, 65
13, 14, 15, 18, 21, 22, 27, 36, 42, 43, 44, 45, 46, 47, 48, 55,
60, 61, 63, 64, 66, 67 Perceptrons
9, 29, 32, 34, 45, 46, 50, 56, 59, 65
Perceptual constancy Phantom limb Philosophy
13, 15, 21, 26, 44, 45, 46, 48
15
4, 11, 13, 14, 21, 22, 25, 26, 53, 67, 68
Population coding
17, 45, 46, 65
Possibles
7, 8, 18
Primates
38, 43, 45, 58, 59, 61, 62, 66, 68
Probability
5, 7, 11, 12, 18, 21, 22, 23, 39
Propensity (for generalisation) Psychophysics
Quantum mechanics
5, 6
Rabbit
21, 67
Reason
12, 13, 21, 27
Receptive
32, 35
16, 21, 22, 37, 38, 47, 48, 63, 65
field
14, 15, 21, 26, 32, 36, 37, 40, 43, 44, 45, 46, 47, 48, 52,
56, 60, 61, 63, 65, 67 Redundancy
19, 21, 36, 37, 38, 59, 66
Replica method
29, 34
850
Representation 4, 21, 22, 32, 37, 38, 39, 42, 45, 46, 52, 53, 54, 55, 59, 60, 61, 62, 64, 65, 66 Retina 13, 19, 21, 31, 36, 37, 38, 41, 43, 46, 52, 53, 54, 61, 67 Retroprojection 49, 55, 58, 59, 60, 61, 62 Reverberation 15, 51, 53, 57 Robot 1, 5, 26 Science 3, 5, 7, 11, 12, 13, 14, 19, 26 Selection vs. instruction 7, 21, 25, 30 Sensation 2, 13, 14, 15, 21, 26, 48, 67 Sensitive periods 21, 22, 23, 26 Sensorimotor coordination 13, 15, 42, 44, 45, 46, 49, 52, 53, 54, 55, 59, 60 Simulated annealing 4, 30, 32, 34 Size scaling 28, 30, 31, 32, 34, 35 Sleep 15, 59, 61, 62, 67, 68 Social behaviour 1, 21, 23, 27, 60, 66, 67, 68 Somatic senses 14, 15, 20, 21, 27, 40, 42, 55, 59 Sparse coding 21, 22, 28, 29, 51, 59 Spike timing codes 40, 41, 42, 55, 56, 58, 62 Spin glass 28, 30, 34 Spinal cord 61, 65 Statistical analysis 28, 36, 37, 38, 62 Statistical efficiency 36, 37, 47, 66 Statistical inference 11, 12, 32, 33, 36, 40, 66 Statistical mechanics 5, 29, 30, 34, 37, 57 Stereopsis 1, 4, 14, 21, 26, 43, 48, 55, 61, 63 Stochastic vs. deterministic 6, 10, 12, 21, 26, 28, 30, 32, 34, 37, 56 Subjective contours 13, 26, 43, 48 Superior colliculus 4, 14, 43, 45, 53, 54, 61, 63 Symmetry breaking 3, 6, 29, 34, 36, 57 Synaptic plasticity 6, 22, 25, 28, 36, 46, 50, 51, 58, 59, 64 Synchrony 10, 40, 41, 42, 50, 55, 58, 60, 62, 64 Synfire chains 58 Taste 20, 59 Temporal lobe 21, 27, 41, 43, 44, 45, 59, 60, 61, 62, 65, 68 Temporal sequences 15, 23, 26, 28, 38, 49, 50, 51, 60, 61, 63, 68 Texture vision 19, 26, 37, 43 Thalamus 21, 49, 56, 58, 61 Thermodynamics 3, 5, 26, 29, 30, 34, 37, 57 Thought 10, 12, 21, 49, 62, 66, 67, 68 Tinkering 7, 26, 43 Travelling salesman problem 6, 30, 31, 32 Turing machine 6, 26, 32, 33, 35, 37 851
Vertebrates Vision
7, 22, 23, 45, 50, 51, 66
13, 14, 19, 20, 21, 23, 24, 26, 27, 37, 38, 40, 41, 42 43
44, 45, 46, 47, 48, 52, 53, 55, 56, 59, 61, 62, 63, 64, 65, 66 Wavelet
37
852
