file:///C|/New%20Text%20Document%20(2).txt
Table of Contents Preface Acknowledgments Addresses of the PDP Research Group
I THE PDP PERSPECTIVE 1 The Appeal of Parallel Distributed Processing J.L. McClelland, D.E. Rumelhart, and G.E. Hinton
2 A General Framework for Parallel Distributed Processing D.E. Rumelhart, G.E. Hinton, and J.L. McClelland
3 Distributed Representations G.E. Hinton, J.L. McClelland, and D.E. Rumelhart
4 PDP Models and General Issues in Cognitive Science D.E. Rumelhart and J.L. McClelland
II BASIC MECHANISMS 5 Feature Discovery by Competitive Learning D.E. Rumelhart and D. Zipser
6 Information Processing in Dynamical Systems: Foundations of Harmony Theory P. Smolensky
file:///C|/New%20Text%20Document%20(2).txt (1 of 2)9/25/2008 5:03:17 PM
file:///C|/New%20Text%20Document%20(2).txt
7 Learning and Relearning in Holtzmann Machines G.E. Hinton and T.J. Sejnowski
8 Learning Internal Representations by Error Propagation D.E. Rumelhart, G.E. Hinton, and R.J. Williams
III FORMAL ANALYSIS 9 An Introduction to Linear Algebra in Parallel Distributed Processing M.I. Jordan
10 The Logic of Activation Functions R.J. Williams
11 An Analysis of the Delta Rule and the Learning of Statistical Associations G.O. Stone
12 Resource Requirements of Standard and Programmable Nets J.L. McClelland
13 P3: A Parallel Network Simulating System D. Zipser and D.E. Rabin
References Index file:///C|/New%20Text%20Document%20(2).txt (2 of 2)9/25/2008 5:03:17 PM
Preface
One of the great joys of science lies in the moment of shared discovery. One person's half -baked suggestion resonates in the mind of another and suddenly takes on a definite shape. An insightful critique of one way of thinking about a problem leads to another, better understanding. An incomprehensible simulation result suddenly makes sense as two people try to understand it together. This book grew out of many such moments. The seeds of the book were sown in our joint work on the interactive activation model of word perception. Since then , each of us has worked with the other and with other collaborators. The results of these collaborations are reported in several of the chapters of this book. The book also contains many chapters by other colleagues whose explorations have become intertwined with ours. Each chapter has its own by-line , but each also reflects the influences of other members of the group. We hope the result reflects some of the benefits of parallel distributed processing! The idea of parallel distributed processing- the notion that intelli gence emerges from the interactions of large numbers of simple processing units - has come and gone before. The idea began to seem more and more attractive to us as the contrast between our convictions about basic characteristics of human perception, memory , language, and thought and the accepted formal tools for capturing mental processes became more apparent. Symbol-processing machines, for all their Tur ing equivalence, had failed to provide useful frameworks for capturing
x
PREFACE
the simple insights about the interactive nature of processing that had lead to such models as the HEARSAY model of speech understanding. More generally, they had failed to provide a framework for representing knowledge in a way that allowed it to be accessedby content and effectively combined with other knowledge to produce useful automatic syntheses that would allow intelligence to be productive . And they made no contact with the real strengths and weaknesses of the hardware in the brain. A Cray computer can perform on the order of 100 million double-precision multiplications in a second, but it does not exhibit natural intelligence. How then are we to understand the capabilities of human thought , given the time constants and noisiness inherent in neural systems? It seemed obvious that to get any processing done in real time , the slow, noisy hardware in the brain would have to do massively parallel processing. As our interest in parallel mechanisms developed, we began to study the work of others who shared our convictions and to build on their work . Particularly important in this regard was Hinton and J. A . Anderson's ( 1981) Parallel Models of Associative Memory. Indeed, we see our book as a descendant of their book on two accounts. First , the material presented here represents further developments on the work presented in Hinton and Anderson 's book. Second, we owe a particular intellectual debt to both Hinton and Anderson. Our interest in distrib uted, associative memories goes back to interactions with Jim Ander son, beginning as early as 1968. Our interest in these topics began in earnest, however, during the period when we were developing the interactive activation model of word perception, in 1979, shortly after Geoffrey Hinton began a postdoctoral fellowship at UCSD. Geoffrey 's crisp explanations showed us the potential power and generality of models created from connections among simple processing units , and fit together nicely with our own developing conviction that various aspectsof perception, language processing, and motor control were best thought of in terms of massively parallel processing (see McClelland , 1979, and Rumelhart , 1977, for our earliest steps in this direction ) . The project culminating in this book formally began in December, 1981 when the two of us and Geoffrey Hinton decided to work together exploring the implications of network models and to write a book outlining our conclusions. We expected the project to take about six months . We began in January 1982 by bringing a number of our colleagues together to form a discussion group on these topics. During the first six months we met twice weekly and laid the foundation for most of the work presented in these volumes. Our first order of business was to develop a name for the class of models we were investigating. It seemed to us that the phrase parallel distributedprocessing(POP
PREFACE
. XI
for short ) best captured what we had in mind . It emphasized the paral lel nature of the processing , the use of distributed representations and distributed control , and the fact that these were general processing systems , not merely memories we were studying , as the phrase associative memory suggests . Thus the POP research group was born . Hinton and McClelland
left
McClelland
after
to MIT
and later
ever , has continued The group
the
first
six
months - Hinton
to
to CMU . The POP research
CMU
and
group , how -
regular meetings at UCSO up to the present time .
has varied
from
five
or six of us at times
to as many
as 15
or more at other times , and there is now a parallel group of about 15 or so psychologists and computer scientists at CMU . Shortly after leaving UCSO in 1982, Hinton began working with Terrence Sejnowski on the Boltzmann machine (Chapter 7) and decided to dl"'JP from the role of organizer of the project to a contributor , so he could spend more time working on the implications of the Boltzmann machine . Thus , the primary responsibility for putting the book together fell to the two of us. At first we expected to complete the book within a year after we began our work . Soon , however , it became clear
that
there
was
much
work
to
be
done
and
many
directions
to
explore . Thus , our work continued and expanded as we and our col leagues followed the implications of the POP approach in many dif ferent
ways .
A good deal has happened since we began this project . Though much of the initial groundwork was laid in early 1982, most of the material described
in these
volumes
did
not
take
its present
form
until
much
later .
The work has been interdisciplinary and represents what we consider a true cognitive science approach . Although the two of us have been trained as cognitive psychologists , the POP group as a whole includes people from a wide range of backgrounds . It includes people trained in physics , mathematics , neuroscience , molecular biology , and computer sciences , as well as in psychology . We also envision an interdisciplinary audience for our book . We are cognitive psychologists and we hope , primarily , to present POP models to the community of cognitive psychologists as alternatives to the models that have dominated cogni tive psychology for the past decade or so. We also , however , see our selves as studying architectures for computation and methods for artifi cial intelligence . Therefore , we hope that this book will be seen as relevant to researchers in computer science and artificial intelligence . Also , the POP approach provides a set of tools for developing models of the neurophysiological basis of human information processing , and so we hope portions of these books will seem relevant to neuroscien tists as well .
.. XII PREFACE
ORGANIZATIONOF THE BOOK Our
book
consists
of six
parts , three
in each of the
two
volumes
.
The overall structure is indicated in the accompanying table . Part I pro vides an overview . Chapter 1 presents the motivation for the approach and describes much of the early work that lead to the developments reported in later sections . Chapter 2 describes the POP framework in more formal terms . Chapter 3 focuses on the idea of distributed representation , and Chapter 4 provides a detailed discussion of several general issues that the POP approach has raised and explains how these issues are addressed in the various later chapters of the book . The remaining parts of the book present different facets of our explorations in parallel distributed processing . The chapters in Part II address central theoretical problems in the development of models of parallel distributed processing , focusing for the most part on fundamen tal problems in learning . The chapters in Part III describe various mathematical and computational tools that have been important in the development and analysis of POP models . Part IV considers
A CONDENSED
TABLE VOLUME
I . THE
POP
PERSPECTIVE
II . BASIC
OF CONTENTS I
MECHANISMS
III . FORMAL
ANALYSES
1. The Appeal of POP 2. A Framework for PDP
5. Competitive Learning 6 . Harmony Theory
9 . Linear Algebra 10. Activation Functions
3 . Distributed
7 . Boltzmann
11 . The
Representations
4. General Issues
Machines
8. Learning by
Delta
Rule
12. Resource Requirements
Error Propagation
13. Parallel Network Simulator
VOLUME
IV . PSYCHOLOGICAL PROCESSES 14 . Schemata
and POP
15. Speech Perception
II
V . BIOLOGICAL
-VI . CONCLUSION
MECHANISMS 20 . Anatomy
and
Physiology
16. Model of Reading
21. Computation in
17. Learning and Memory 18. Morphology Acquisition
the Brain 22 . Neural and
19. Sentence Processing
Conceptual Levels 23. Place Recognition 24. Neural Plasticity 25 . Amnesia
26 . Reflections
Future Directions
...
PREFACEXIII applications and implications of PDP models to various aspects of human cognition , including perception , memory , language , and higher level thought processes . Part V considers the relation between parallel distributed processing models and the brain , reviews relevant aspects of the anatomy and physiology , and describes several models that apply POP models to aspects of the neurophysiology and neuropsychology of information processing , learning , and memory . Part VI contains two short pieces : a reflection on PDP models by Don Norman and a brief discussion of our thoughts about promising future directions .
How to read this book? It i~ too long to read straight through . Nor is it designed to be read this way . Chapter 1 is a good entry point for readers unfamiliar with the POP approach , but beyond that the various parts of the book may be approached in various orders , as one might explore the different parts of a complex object or machine . The vari ous facets of the POP approach are interrelated , and each part informs the others ~ but there are few strict sequential dependencies . Though we have tried to cross -reference ideas that come up in several places , we hope that most chapters can be understood without reference to the rest of the book . Where dependencies exist they are noted in the intro ductory sections at the beginning of each part of the book . This book charts the explorations we and our colleagues have made in the microstructure of cognition . There is a lot of terrain left to be explored . We hope this book serves as a guide that helps others join us in these ongoing explorations .
December1985
JamesL. McClelland PI1TSBURGH , PENNSYLVANIA David E. Rumelhart LA JOLLA , CALIFORNIA
Acknowledgments
As we have already said, nearly all the ideas in this book were born out of interactions, and one of our most important acknowledgments is to the environment that made these interactions possible. The Institute for Cognitive Science at UCSD and the members of the Institute have made up the core of this environment . Don Norman , our colleague and friend , the Founder and Director of the Institute , deserves special credit for making ICS an exciting and stimulating place, for encouraging our explorations in parallel distrib uted processing, and for his central role in arranging much of the finan cial support this book has benefited from (of which more below) . The atmosphere depends as well on the faculty , visiting scholars, and graduate students in and around ICS. The members of the PDP Research Group itself , of course, have played the most central role in helping to shape the ideas found in this book. All those who contributed to the actual contents of the book are listed on the cover page; they have all contributed , as well, in many other ways. Several other participants in the group who do not have actual contributions to the book also deserve mention . Most prominent among these are Mike Mozer and Yves Chauvin , two graduate students in the Cognitive Science Lab, and Gary Cottrell , a recent addition to the group from the University of Rochester. Several other members of the intellectual community in and around ICS have played very important roles in helping us to shape our thoughts. These include Liz Bates, Michael Cole, Steve Draper, Don
. XVI ACKNOWLEDGMENTS Gentner , Ed Hutchins , Jim Hollan , Jean Mandler , George Mandler , Jeff Miller , Guy van Orden , and many others , including the participants in Cognitive Science 200 . There are also several colleagues at other universities who have helped us in our explorations . Indeed , the annual connectionist workshops (the first of which resulted in the Hinton and Anderson book ) have been important opportunities to share our ideas and get feedback
on them
from
others
in the field , and to learn
from
the con -
tributions of others . Jim Anderson , Dana Ballard , Jerry Feldman , Geoff Hinton and Terry Sejnowski all had a hand in organizing dif ferent ones of these meetings ; and we have learned a great deal from discussions with them and other participants , particularly Andy Barto , Scott Fahlman , Christof von der Malsburg , John Hopfield , Dave Touretzky , and more recently Mark Fanty and Gene Charniak . McClelland ' s discussions at MIT (particularly with Jerry Fodor and Molly Potter ) helped in the clarification of several aspects of our think ing , and various colleagues at and around CMU - particularly John Anderson , Mark Derthick , Dave Klahr , Brian MacWhinney , and Jeff Sokolov - have contributed a great deal through discussions over the last year and a half or so, as we have worked toward the completion of the book . Others
one or both
of us have
interacted
with
a great deal
include Bill Brewer , Neal Cohen , Al Collins , Billy Salter , Ed Smith , and Walter Schneider . All of these people have contributed more or less directly to the development of the ideas presented in this book . An overlapping group of colleagues deserves credit for helping us improve the book itself . Jim Anderson , Andy Barto , Larry Barsalou , Chris Reisbeck , Walter Schneider , and Mark Seidenberg all read several chapters of the book and sent useful comments and suggestions . Many other people read and commented on individual chapters , and we are
sincerely grateful for their careful contributions , which we acknowledge in the appropriate chapters . This project owes a tremendous amount to the help of the excellent staff of the Institute for Cognitive Science . Kathy Farrelly , in particu lar , has played an enormous role in all aspects of the production of the
book; her cheerful , thoughtful , and very careful assistance made the production of the book run much more smoothly than we have had any right to hope and allowed us to keep working on the content of some of
the chapters even as the final production was rolling forward on other sections. Eileen Conway's assistance with graphics and formatting has also been invaluable and we are very grateful to her as well . Mark Wal len kept the computers running , served as chief programming consul tant and debugger par excellence , and tamed troff , the phototypesetter . Without him we would never have gotten all the formatting to come out right . Karol Lightner worked very hard toward the end of the
.. ACKNOWLEDGMENTSXVII
project on final proofing and indexing , and Sondra Buffett , as the Administrative Director of ICS , held everything together and kept everything running throughout the entire four years of the project . Our project has been supported by a number of different agencies and foundations . Primary support came from the System Development Foundation and the Office of Naval Research . The System Develop ment Foundation has provided direct support for the PDP research group through a grant to Norman and Rumelhart , and has also sup ported several of the individual members of the group (Crick , Hinton , Sejnowski , and Zipser ) . ONR contracts that have contributed support include
NOOO14 - 79 - C -O323 , NR
667 - 437 ~ NOOO14 - 85 - K -0450 , NR
667 -
548 ~ and NOOO14 - 82 -C - O374 , NR 667 -483 .
The people behind both SDF and ONR deserve acknowledgment too . The entire PDP enterprise owes a particular debt of gratitude to Charlie Smith , formerly of SDF , who appreciated the appeal of parallel distrib uted processing very early on , understood our need for computing resources , and helped provide the entire POP research group with the funds and encouragement needed to complete such project . Henry Halff , formerly of ONR , was also an early source of support , encouragement , and direction . Charlie Smith has been succeeded by Carl York , and Henry Halff has been succeeded by Susan Chipman , Michael Si:tafto , and Harold Hawkins . We are grateful to all of these people for their commitment to the completion of this book and to the ongoing development of the ideas . Several
other
sources
have
contributed
to the support
of individual
members of the group . These include the National Institute of Mental Health , through a Career Development Award - PHS-MH -00385 - to McClelland and post -doctoral fellowships to Paul Smolensky and Paul Munro
under
Grant
PHS - MH - 14268 to the Center
for Human
Informa
-
tion Processing at UCSD . Smolensky received support in the form of a fellowship from the Alfred P. Sloan Foundation , and some of McClelland ' s work was supported by a grant from the National Science Foundation (BNS - 79-24062 ) . These and other sources of support for specific individuals or projects are acknowledged in the appropriate chapters . Finally , we would like to thank our wives , Heidi and Marilyn . Their understanding , encouragement , and support throughout the four years of this project helped to make the process of bringing this book to life much more rewarding than it might have been . I
JLM/DER
Addresses of the PDP Research Group
ChisatoAsanuma
SalkInstitute P.O. Box85800 SanDiego, CA 92138
FrancisH. C. Crick
SalkInstitute P.o . Box85800 SanDiego, CA 92138
Jeffrey L. Elman
Department of Linguistics Universityof California , SanDiego La Jolla, CA 92093
GeoffreyE. Hinton
Michael I. Jordan
Department
of
Carnegie
- Mellon
Pittsburgh
,
PA
15213
of
Information
Alan H. Kawamoto
of ,
Pittsburgh
Massachusetts
MA
Department Carnegie
Computer Science
University
01003
of - Mellon ,
Science
University
Department
Amherst
Computer
PA
Psychology University 15213
and
xx
ADDRESSES OFTHEPDP RESE)LRCH GROUP JamesL. McClelland
Departmentof Psychology Carnegie-Mellon University Pittsburgh, PA 15213
Paul W. Munro
Departmentof Information Science University of Pittsburgh Pittsburgh, PA 15260
Donald A . Norman
Institute for Cognitive Science
University of California , SanDiego LaJolla,CA 92093 DanielE. Rabin
Intellicorp 1975El Camino Real West Mountain View, CA 94040
David E. Rumelhart
Institute for Cognitive Science University of California, SanDiego La Jolla, CA 92093
TerrenceJ. Sejnowski
Departmentof Biophysics JohnsHopkins University Baltimore, MD 21218
Paul Smolensky
Department
of
University
of
Boulder
GregoryO. Stone
,
Center
CO
for
Adaptive
,
MA
Institute
02215
for
UniverSi
David Zipser
Mathematics
University
Boston
La
Systems
of
Boston
Science
80309
Department
RonaldJ. Williams
Computer
Colorado
~ y
Jolla
,
CA
Cognitive
of
California
Science
,
San
Diego
92093
Insitutefor CognitiveScience Universityof California , SanDiego La Jolla, CA 92093
CHAPTER 1
The Appeal of Parallel Distributed Processing
J. L. McCLELLAND , D. E. RUMELHART , andG. E. HINTON
What makes people smarter than machines? They certainly are not quicker or more precise. Yet people are far better at perceiving objects in natural scenes and noting their relations, at understanding language and retrieving contextually appropriate information from memory, at making plans and carrying out contextually appropriate actions, and at a wide range of other natural cognitive tasks. People are also far better at learning to do . . these things more accurately and fluently through processlng experIence. What is the basis for these differences? One answer, perhaps the classic one we might expect from artificial intelligence , is "software." If we only had the right computer program, the argument goes, we might be able. to capture the fluidity and adaptability of human information processlng. Certainly this answer is partially correct. There have been great breakthroughs in our understanding of cognition as a result of the development of expressive high-level computer languagesand powerful algorithms . No doubt there will be more such breakthroughs in the future . However, we do not think that software is the whole story. In our view, people are smarter than today's computers because the brain employs a basic computational architecture that is more suited to deal with a central aspect of the natural information processing tasks that people are so good at. In this chapter, we will show through examples that these tasks generally require the simultaneous consideration of many pieces of information or constraints. Each constraint may be imperfectly specified and ambiguous, yet each can playa potentially
4
THE PDP PERSPECTIVE
decisive role in determining the outcome of processing. After examining these points, we will introduce a computational framework for modeling cognitive processesthat seems well suited to exploiting these constaints and that seems closer than other frameworks to the style of computation as it might be done by the brain. We will review several early examples of models developed in this framework , and we will show that the mechanisms these models employ can give rise to powerful emergent properties that begin to suggest attractive alternatives to traditional accounts of various aspects of cognition . We will also show that models of this class provide a basis for understanding how learning can occur spontaneously, as a by-product of processing activity .
Multiple Simultaneous Constraints Reaching and grasping. Hundreds of times each day we reach for things. We nearly never think about these acts of reaching. And yet, each time , a large number of different considerations appear to jointly determine exactly how we will reach for the object. The position of the object, our .posture at the time , what else we may also be holding , the size, shape, and anticipated weight of the object, any obstacles that may be in the way- all of these factors jointly determine the exact method we will use for reaching and grasping. Consider the situation shown in Fi,gure 1. Figure 1A shows Jay McClelland 's hand, in typing position at his terminal . Figure 18 indi cates the position his hand assumed in reaching for a small knob on the desk beside the terminal . We will let him describe what happened in the first person: On the desk next to my terminal are several objects- a chipped coffee mug, the end of a computer cable, a knob from a clock radio. I decide to pick the knob up. At first I hesitate, because it doesn't seem possible. Then I just reach for it , and find myself grasping the knob in what would normally be considered a very awkward position - but it solves all of the constraints. I ' m not sure what all the details of the movement were, so I let myself try it a few times more. I observe that my right hand is carried up off the keyboard, bent at the elbow, until my forearm is at about a 300 angle to the desk top and parallel to the side of the terminal . The palm is facing downward through most of this . Then , my arm extends and lowers down more or less parallel to the edge of the desk and parallel to the side of the terminal and, as it drops, it turns about 900 so that the
1. TIlE APPEAL OF PDP
5
A
FIGURE 1. A: An everyday situation in which it is necessary to take into account a large number of constraints to grasp a desired object. In this case the target object is the small knob to the left of the cup. B: The posture the arm arrives at in meeting these constraints.
6
THEPOP PERSPECTIVE palm is facing the cup and the thumb and index finger are below. The turning motion occurs just in time , as my hand drops, to avoid hitting the coffee cup. My index finger and thumb close in on the knob and grasp it , with my hand completely upside down.
Though the details of what happened here might be quibbled with , the broad outlines are apparent. The shape of the knob and its position on the table~ the starting position of the hand on the keyboard; the positions of the terminal , the cup, and the knob~ and the constraints imposed by the structure of the arm and the musculature used to control it - all these things conspired to lead to a solution which exactly suits the problem . If any of these constraints had not been included, the movement would have failed. The hand would have hit the cup or the terminal - or it would have missed the knob . The mutual influence of syntax and semantics. Multiple constraints operate just as strongly in language processing as they do in reaching and grasping. Rumelhart ( 1977) has documented many of these multi ple constraints. Rather than catalog them here, we will use a few examples from language to illustrate the fact that the constraints tend to be reciprocal: The example shows that they do not run only from syntax to semantics- they also run the other way. It is clear, of course, that syntax constrains the assignment of meaning. Without the syntactic rules of English to guide us, we cannot correctly understand who has done what to whom in the following sentence: The boy the man chased kissed the girl . But consider these examples (Rumelhart , 1977; Schank, 1973) : I saw the grand canyon flying- to New York .
I sawthe sheepgrazingin the field. Our knowledge of syntactic rules alone does not tell us what grammatical role is played by the prepositional phrases in these two cases. In the first , IIflying to New York " is taken as describing the context in which the speaker saw the Grand Canyon - while he was flying to New York . In the second, "grazing in the field II could syntactically describe an analogous situation , in which the speaker is grazing in the field , but this possibility does not typically become available on first reading. Instead we assign IIgrazing in the field II as a modifier of the sheep (roughly , 'Iwho were grazing in the field II) . The syntactic structure of each of
1. THEAPPEALOFPDP
7
these sentences , then, is determined in part by the semanticrelations that the constituents of the sentence might plausibly bear to one another. Thus, the influencesappearto run both ways, from the syntax to the semanticsand from the semanticsto the syntax. In these examples, we see how syntactic considerationsinfluence semantic ones and how semantic ones influence syntactic ones. We cannotsaythat one kind of constraintis primary. Mutual constraintsoperate, not only betweensyntacticand semantic processing , but also within each of these domains as well. Here we consideran examplefrom syntacticprocessing , namely, the assignment of wordsto syntacticcategories . Considerthe sentences : like the joke. like the drive. like to joke. like to drive. In this case it looks as though the words the and to serve to determine whether the following word will be read as a noun or a verb. This , of course, is a very strong constraint in English and can serve to force a verb interpretation of a word that is not ordinarily used this way:
I like to mud.
~
~
~
~
On the other hand, if the information specifyingwhether the function word precedingthe final word is to or the is ambiguous, then the typical readingof the word that follows it will determine which way the function word is heard. This was shown in an experiment by Isenberg, Walker, Ryder, and Schweikert(1980) . They presentedsoundshalfway betweento (actuallyItAI ) and the (actuallyIdAI ) and found that words like joke, which we tend to think of first as nouns, madesubjectshear the marginal stimuli as the, while words like drive, which we tend to think of first as verbs, made subjectshear the marginal stimuli as to. Generally, then, it would appearthat each word can help constrainthe syntacticrole, and even the identity, of every other word. Simultaneous mutual constraints in word recognition . Just as the syntactic role of one word can influence the role assigned to another in analyzing sentences, so the identity of one letter can influence the identity assigned to another in reading. A famous example of this , from Selfridge, is shown in Figure 2. Along with this is a second example in which none of the letters, considered separately, can be identified unambiguously, but in which the possibilities that the visual
8 THE PDP PERSPECTIVE
I
FIGURE 2. Some ambiguousdisplays. The first one is from Selfridge, 1955. The secondline showsthat three ambiguouscharacterscan eachconstrainthe identity of the others. The third, fourth, and fifth lines show that thesecharactersare indeedambiguous in that they assumeother identities in other contexts. (The ink-blot techniqueof makinglettersambiguousis due to Lindsayand Norman, 1972).
information leaves open for each so constrain the possible identities of the others that we are capable of identifying all of them . At first glance, the situation here must seem paradoxical: The identity of each letter is constrained by the identities of each of the others. But since in general we cannot know the identities of any of the letters
1. THEAPPEALOFPOP
until
we have
established
the identities
9
of the others , how can we get
the process started ?
The resolution of the paradox , of course , is simple . One of the dif ferent possible letters in each position fits together with the others . It appears then that our perceptual system is capable of exploring all these possibilities without committing itself to one until all of the constraints are taken
into account .
Understanding through the interplay of multiple sources of knowledge. It is clear that we know a good deal about a large number of different standard situations . Several theorists have suggested that we store this knowledge in terms of structures called variously : scripts (Schank , 1976) , frames (Minsky , 1975) , or schemata (Norman & Bobrow , 1976 ; Rumelhart , 1975) . Such knowledge structures are assumed to be the basis of comprehension . A great deal of progress has been made within
the context
of this view .
However , it is important to bear in mind that most everyday situa tions cannot be rigidly assigned to just a single script . They generally involve
an interplay
between
a number
of different
sources
of informa
-
tion . Consider , for example , a child ' s birthday party at a restaurant . We know things about birthday parties , and we know things about restaurants , but we would not want to assume that we have explicit knowledge (at least , not in advance of our first restaurant birthday party ) about the conjunction of the two . Yet we can imagine what such a party might be like . The fact that the party was being held in a restaurant would modify certain aspects of our expectations for birthday parties (we would not expect a game of Pin -the - Tail -on -the -Donkey , for example ) , while the fact that the event was a birthday party would inform our expectations for what would be ordered and who would pay the bill .
Representations like scripts , frames , and schemata are useful struc tures for encoding knowledge , although we believe they only approxi mate the underlying structure of knowledge representation that emerges from the class of models we consider in this book , as explained in Chapter 14. Our main point here is that any theory that tries to account for human knowledge using script -like knowledge structures will
have to allow
them
to interact
with
each other
to capture
the gen -
erative capacity of human understanding
in novel situations . Achieving
such
greatest
interactions
has been
one
of
the
difficulties
with implementing models that really think generatively frame -like representations .
associated
using script - or
10
THEPOPPERSPECTIVE
PARALLEL DISTRIBUTEDPROCESSING In the examples we have considered, a number of different pieces of information must be kept in mind at once. Each plays a part, constraining others and being constrained by them . What kinds of mechanisms seem well suited to these task demands? Intuitively , these tasks seem to require mechanisms in which each aspect of the informa tion in the situation can act on other aspects, simultaneously influenc ing other aspects and being influenced by them . To articulate these intuitions , we and others have turned to a class of models we call Parallel Distributed Processing (POP) models. These models assume that information processing takes place through the interactions of a large number of simple processing elements called units , each sending excitatory and inhibitory signals to other units . In some cases, the units stand for possible hypotheses about such things as the letters in a particular display or the syntactic roles of the words in a particular sentence. In these cases, the activations stand roughly for the strengths associated with the different possible hypotheses, and the interconnections among the units stand for the constraints the system knows to exist between the hypotheses. In other cases, the units stand for possible goals and actions, such as the goal of typing a particular letter , or the action of moving the left index finger , and the connections relate goals to subgoals, subgoals to actions, and actions to muscle movements. In still other cases, units stand not for particular hypotheses or goals, but for aspects of these things. Thus a hypothesis about the identity of a word, for example, is itself distributed in the activations of a large number of units.
POPModels: CognitiveScienceor Neuroscience ? One reason for the appeal of POP models is their obvious "physiological" flavor : They seem so much more closely tied to the physiology of the brain than are other kinds of information -processing models. The brain consists of a large number of highly interconnected elements (Figure 3) which apparently send very simple excitatory and inhibitory messages to each other and update their excitations on the basis of these simple messages. The properties of the units in many of the POP models we will be exploring were inspired by basic properties of the neural hardware. In a later section of this book, we will examine in some detail the relation between POP models and the brain.
12 THE POP PERSPECTIVE The Microstructure of Cognition The process of human cognition , examined on a time scale of seconds and minutes , has a distinctly sequential character to it . Ideas come, seem promising , and then are rejected; leads in the solution to a problem are taken up, then abandoned and replaced with new ideas. Though the process may not be discrete, it has a decidedly sequential character, with transitions from state-to-state occurring, say, two or three times a second. Clearly, any useful description of the overall organization of this sequential flow of thought will necessarily describe a sequence of states. But what is the internal structure of each of the states in the sequence, and how do they come about? Serious attempts to model even the simplest macrosteps of cognition - say, recognition of single words- require vast numbers of microsteps if they are implemented sequentially. As Feldman and Ballard ( 1982) have pointed out , the biological hardware is just too sluggish for sequential models of the microstructure to provide a plausible account, at least of the microstructure of human thought . And the time limitation only gets worse, not better, when sequential mechanisms try to take large numbers of constraints into account. Each additional constraint requires more time in a sequential machine, and, if the constraints are imprecise, the constraints can lead to a computational explosion. Yet people get faster, not slower, when they are able to exploit additional constraints. Parallel distributed processing models offer alternatives to serial models of the microstructure of cognition . They do not deny that there is a macrostructure, just as the study of subatomic particles does not deny the existence of interactions between atoms. What PDP models do is describe the internal structure of the larger units , just as subatomic physics describes the internal structure of the atoms that form the constituents of larger units of chemical structure . We shall show as we proceed through this book that the analysis of the microstructure of cognition has important implications for most of the central issues in cognitive science. In general, from the PDP point of view, the objects referred to in macrostructural models of cognitive processing are seen as approximate descriptions of emergent properties of the microstructure . Sometimes these approximate descriptions may be sufficiently accurate to capture a process or mechanism well enough~ but many times, we will argue, they fail to provide sufficiently elegant or tractable accounts that capture the very flexibility and openendedness of cognition that their inventors had originally intended to capture. We hope that our analysis of PDP models will show how an
1. THE APPEAL OFPDP 13 examination of the microstructure of cognition can lead us closer to an adequate of the real extent of human processing and learn. .description . Ing capacities. The development of POP models is still in its infancy . Thus far the models which have been proposed capture simplified versions of the kinds of phenomena we have been describing rather than the full elaboration that these phenomena display in real settings. But we think there have been enough steps forward in recent years to warrant a concerted effort at describing where the approach has gotten and where it is going now, and to point out some directions for the future . The first section of the book represents an introductory course in parallel distributed processing. The rest of this chapter attempts to describe in informal terms a number of the models which have been proposed in previous work and to show that the approach is indeed a fruitful one. It also contains a brief description of the major sources of the inspiration we have obtained from the work of other researchers. This chapter is followed , in Chapter 2, by a description of the quantitative framework within which these models can be described and examined. Chapter 3 explicates one of the central concepts of the book: distributed representation . The final chapter in this section, Chapter 4, returns to the question of demonstrating the appeal of parallel distributed processing models and gives an overview of our explorations in the microstructure of cognition as they are laid out in the remainder of this book.
EXAMPLES OFPDPMODELS In what follows , we review a number of recent applications of POP models to problems in motor control , perception, memory, and language. In many cases, as we shall see, parallel distributed processing mechanisms are used to provide natural accounts of the exploitation of multiple , simultaneous, and often mutual constraints. We will also see that these same mechanisms exhibit emergent properties which lead to novel interpretations of phenomena which have traditionally been inter preted in other ways.
Motor Control
Having. started with an example to . . of how multiple .constraints appear . operate In motor programming , It seems appropriate to mention two
14 THE POP PERSPECTIVE models in this domain . These models have not developed far enough to capture the full details of obstacle avoidance and multiple constraints on reaching and grasping , but there have been applications to two prob lems with
some of these characteristics
.
Finger movements in skilled typing. One might imagine, at first glance , that typists carry out keystrokes successively , first programming
one stroke and then , when it is completed, programming the next . However , this is not the case. For skilled typists, the fingers are continually anticipating upcoming keystrokes . Consider the word vacuum. In this word , the v, a, and c are all typed with the left hand , leaving the right hand nothing to do until it is time to type the first u. However , a
high speed film of a good typist shows that the right hand moves up to anticipate the typing of the u , even as the left hand is just beginning to type the v. By the time the c is typed the right index finger is in posi tion over the u and ready to strike it . When two successive key strokes are to be typed with the fingers of the same hand , concurrent preparation to type both can result in similar or conflicting instructions to the fingers and / or the hand . Consider , in this light , the difference between the sequence ev and the sequence ere The first sequence requires the typist to move up from home row to type the e and to move down from the home row to type the v, while the second sequence , both the e and the f are above the home row .
in
The hands take very different positions in these two cases. In the first case, the hand as a whole stays fairly stationary over the home row . The middle finger moves up to type the e, and the index finger moves
down
to type
the
v.
In the second
case , the hand
as a whole
moves up , bringing the middle finger over the e and the index finger over the f . Thus , we can see that several letters can simultaneously influence the positioning of the fingers and the hands . From the point of view of optimizing the efficiency of the typing motion
, these
different
patterns
seem very
sensible . In the first
case ,
the hand as a whole is maintained in a good compromise position to allow the typist to strike both letters reasonably efficiently by extending the fingers up or down . In the second case, the need to extend the fingers is reduced by moving the whole hand up , putting it in a near optimal position to strike either key . Rumelhart and Norman ( 1982) have simulated these effects using POP mechanisms . Figure 4 illustrates aspects of the model as they are illustrated in typing the word very. In brief , Rumelhart and Norman assumed
that the decision
to type a word
caused activation
of a unit
for
that word . That unit , in turn , activated units corresponding to each of the letters in the word . The unit for the first letter to be typed was made to inhibit the units for the second and following letters , the unit
1. THE APPEAL OFPOP 15 for the second to inhibit the third and following letters, and so on. As a result of the interplay of activation and inhibition among these units , the unit for the first letter was at first the most strongly active, and the units for the other letters were partially activated. Each letter unit exerts influences on the hand and finger involved in typing the letter . The v unit , for example, tends to cause the index finger to move down and to cause the whole hand to move down with it . The e unit , on the other hand, tends to cause the middle finger on the left hand to move up and to cause the whole hand to move up also. The r unit also causes the left index finger to move up and the left hand to move up with it . The extent of the influences of each letter on the hand and finger it directs depends on the extent of the activation of the letter ." Therefore , at first , in typing the word very, the v exerts the greatest control .
upward ...tw8rd +
Response System
w W "' X "' " ow ~ zoo - a -: ~ ~ -z ...
w
W
~ 5 ~ -l oQz ~ z ~ -
~
(I:
upward
THUMB
inward PALM
PALM
RESPONSE LI (- 1 , + 0 .5)
- l
LM (+ 1 , - 0 .3 )
;ftWid + OUIw ... downward
SYSTEM LI (+ 1, - 0 .3 )
RI (+ 1, + 1.3 )
.target .. current
finger finger
position position
Keypress Schemata Word Schema FIGURE 4. The interaction of activationsin typing the word very. The very unit is activatedfrom outside the model. It in turn activatesthe units for each of the component letters. Each letter unit specifiesthe target finger positions, specifiedin a keyboardcoordinatesystem. Land R standfor the left and right hands, and I and M for the index and middle fingers. The letter units receiveinformation about the current finger position from the responsesystem. Each letter unit inhibits the activation of all letter units that follow it in the word: inhibitory connectionsare indicatedby the lines with solid dots at their terminations. (From "Simulatinga Skilled Typist: A Study of Skilled Motor Pe~formance" by D. E. Rumelhartand D. A. Norman, 1982, CognitiveScience , 6, p. 12. Copyright1982byAblex Publishing . Reprintedby permission .)
16 THE POP PERSPECTIVE Becausethe e and r are simultaneously pulling the hand up, though , the v is typed primarily by moving the index finger , and there is little movement on the whole hand. Once a finger is within a certain striking distance of the key to be typed, the actual pressing movement is triggered, and the keypress occurs. The keypress itself causes a strong inhibitory signal to be sent to the unit for the letter just typed, thereby removing this unit from the picture and allowing the unit for the next letter in the word to become the most strongly activated. This mechanism provides a simple way for all of the letters to jointly determine the successive configurations the hand will enter into in the process of typing a word. This model has shown considerable success predicting the time between successive keystrokes as a function of the different keys involved . Given a little noise in the activation process, it can also account for some of the different kinds of errors that have been observed in transcription typing . The typing model represents an illustration of the fact that serial behavior - a succession of key strokes- is not necessarily the result of an inherently serial processing mechanism. In this model, the sequential structure of typing emerges from the interaction of the excitatory and inhibitory influences among the processing units . Reaching for an object without falling over. Similar mechanisms can be used to model the process of reaching for an object without losing one' s balance while standing, as Hinton ( 1984) has shown. He considered a simple version of this task using a two-dimensional " person" with a foot , a lower leg, an upper leg, a trunk , an upper arm, and a lower arm. Each of these limbs is joined to the next at a joint which has a single degree of rotational freedom . The task posed to this person is to reach a target placed somewhere in front of it , without taking any steps and without falling down. This is a simplified version of the situation in which a real person has to reach out in front for an object placed somewhere in the plane that vertically bisects the body. The task is not as simple as it looks, since if we just swing an arm out in front of ourselves, it may shift our center of gravity so far forward that we will lose our balance. The problem, then, is to find a set of joint angles that simultaneously solves the two constraints on the task. First , the tip of the forearm must touch the object. Second, to keep from falling down, the person must keep its center of gravity over the foot . To do this , Hinton assigneda single processor to each joint . On each computational cycle, each processor received information about how far the tip of the hand was from the target and where the center of gravity was with respect to the foot . Using these two pieces of information , each joint adjusted its angle so as to approach the goals of maintaining
I. THE APPEAL OFPDP 17 balance and bringing the tip closer to the target. After a number of iterations , the stick-person settled on postures that satisfied the goal of reaching the target and the goal of maintaining the center of gravity over the "feet." Though the simulation was able to perform the task, eventually satisfying both goals at once, it had a number of inadequacies stemming from the fact that each joint processor attempted to achieve a solution in ignorance of what the other joints were attempting to do. This problem was overcome by using additional processors responsible for setting combinations of joint angles. Thus, a processor for flexion and extension of the leg would adjust the knee, hip , and ankle joints synergistically, while a processor for flexion and extension of the arm would adjust the shoulder and elbow together. With the addition of processors of this form , the number of iterations required to reach a solution was greatly reduced, and the form of the approach to the solution looked very natural. The sequence of configurations attained in one processing run is shown in Figure 5. Explicit attempts to program a robot to cope with the problem of maintaining balance as it reaches for a desired target have revealed the difficulty of deriving explicitly the right combinations of actions for each possible starting state and goal state. This simple model illustrates that we may be wrong to seek such an explicit solution . We see here that a solution to the problem can emerge from the action of a number of simple processors each attempting to honor the constraints independently.
+
+
+
+
FIGURE 5. A sequenceof configurationsassumedby the stick "person " performingthe reachingtask describedin the text, from Hinton (1984). The small circle representsthe center of gravity of the whole stick-figure, and the cross representsthe goal to be reached . The configurationis shownon every seconditeration.
18
THEPOPPERSPECTIVE
Perception Stereoscopic
vision
cessing
was
Marr
and
tion
(
depth
in
of
a
Julesz
'
-
of
,
dot
-
dots
in
region
sidered
alone
alone
is
6
pattern
page
far
the
among
front
left
region
of
or
as
behind
.
and
a
the
other
Yet
the
a
-
pattern
two
of
views
con
surfaces
,
when
to
with
depending
of
the
the
let
these
right
eye
shifted
on
-
,
one
other
,
,
patterns
different
the
surface
pattern
with
two
of
of
eye
shown
the
of
either
surfaces
the
the
entirely
presence
those
to
each
consists
one
two
horizontally
of
in
indicate
the
that
shifted
Each
information
in
to
of
as
except
is
-
)
challenges
consists
such
patterns
images
-
6
.
other
.
percep
Figure
,
region
the
direction
of
.
FIGURE
left
can
sees
in
the
see
-
by
the
;
stereogram
the
pattern
no
explain
1971
interesting
A
of
of
is
,
stereogram
the
projected
observer
.
of
those
there
relations
patterns
shift
one
retinal
that
hovering
the
so
depth
dot
an
,
simple
copy
of
two
.
pro
proposed
mechanism
on
a
distributed
to
Julesz
present
exact
of
to
dots
In
rest
corresponding
(
percepti
an
parallel
perception
proposed
stereograms
.
the
random
theory
stereograms
processing
dot
is
a
to
using
depth
Their
dot
depth
pattern
respect
.
-
patterns
one
)
model
stereoscopic
distributed
of
here
1976
random
mechanisms
random
early
of
random
simple
s
One
model
Poggio
of
terms
.
the
in
the
eye
Some
wall
)
- dot
in
right
and
.
Random
dots
the
.
the
right
Cyclopean
Perception
Copyright
1971
pattern
to
be
,
Bell
p
the
able
right
to
the
.
21
Telephone
,
The
two
the
patterns
left
by
B
eye
,
achieve
Julesz
Laboratories
are
such
the
by
into
,
,
,
Inc
the
area
line
Chicago
.
appears
of
sight
.
permission
the
(
e
. g
Foundations
of
by
the
above
point
From
University
to
to
hover
distant
(
the
respect
projects
to
a
that
with
pattern
to
:
Reprinted
except
over
left
converging
the
1971
identical
shifted
that
shifted
this
figure
.
are
pattern
stereoscopically
interposing
by
of
viewed
may
then
.
region
When
readers
and
stereograms
central
Chicago
a
of
Press
. )
. ,
.
1. THE APPEAL OFPDP 19 What kind of a mechanismmight we proposeto account for these facts? Marr and Poggio(1976) beganby explicitly representingthe two views in two arrays, as human observersmight in two different retinal images. They noted that correspondingblack dots at different perceived distancesfrom the observer will be offset from each other by gifferent amounts in the two views. The job of the model is to determine which points correspond . This task is, of course, made difficult by the fact that there will be a very large number of spurious correspondences of individual dots. The goal of the mechanism , then, is to find those correspondences that representreal correspondences in depth and suppressthosethat representspuriouscorrespondences . To carry out this task, Marr and Poggioassigneda processingunit to each possible,conjunction of a point in one image and a point in the other. Sincethe eyesare offset horizontally, the possible conjunctions occur at various offsets or disparitiesalong the horizontal dimension. Thus, for each point in one eye, there was a set of processingunits with one unit assignedto the conjunctionof that point and the point at eachhorizontal offset from it in the other eye. Each processingunit receivedactivation wheneverboth of the points the unit stood for containeddots. So far, then, units for both real and spurious correspondenceswould be equally activated. To allow the mechanismto find the right correspondences , they pointed out two generalprinciplesabout the visual world: (a) Each point in each view generallycorrespondsto one and only one point in the other view, and (b) neighboringpoints in spacetend to be at nearly the samedepth and therefore at about the samedisparity in the two images. While there are discontinuities at the edges of things, over most of a twodimensionalview of the world there will be continuity. These principlesare calledthe uniqueness and continuityconstraints, respectively . Marr and Poggio incorporatedthese principles into the interconnections betweenthe processingunits. The uniquenessconstraintwascaptured by inhibitory connectionsamongthe units that stand for alternative correspondencesof the same dot. The continuity principle was capturedby excitatoryconnectionsamongthe units that standfor similar offsets of adjacentdots. These additional connectionsallow the Marr and Poggio model to "solve" stereogramslike the one shown in the figure. At first, when a pair of patternsis presented , the units for all possiblecorrespondences of a dot in one eye with a dot in the other will be equally excited. However, the excitatory connectionscause the units for the correct conjunctionsto receivemore excitationthan units for spuriousconjunctions, and the inhibitory connectionsallow the units for the correct conjunctionsto turn off the units for the spuriousconnections. Thus,
20 THEPOP PERSPECTIVE the model tends to settle down into a stable state in which only the correct correspondence of each dot remains active. There are a number of reasons why Marr and Poggio ( 1979) modified this model (see Marr , 1982, for a discussion) , but the basic mechanisms of mutual excitation between units that are mutually consistent and mutual inhibition between units that are mutually incompatible provide a natural mechanism for settling on the right conjunctions of points and rejecting spurious ones. The model also illustrates how general principles or rules such as the uniqueness and continuity princi ples may be embodied in the connections between processing units , and how behavior in accordance with these principles can emerge from the interactions determined by the pattern of these interconnections. Perceptual completion of familiar patterns . Perception, of course, is influenced by familiarity . It is a well-known fact that we often misperceive unfamiliar objects as more familiar ones and that we can get by with less time or with lower-quality information in perceiving familiar items than we need for perceiving unfamiliar items. Not only does familiarity help us determine what the higher-level structures are when the lower-level information is ambiguous~ it also allows us to fill in missing lower-level information within familiar higher-order patterns. The well-known phonemic restoration effect is a case in point . In this phenomenon, perceivers hear sounds that have been cut out of words as if they had actually been present. For example, Warren (1970) presented /egi# /ature to subjects, with a click in the location marked by the # . Not only did subjects correctly identify the word legislature~ they also heard the missing / s/ just as though it had been presented. They had great difficulty localizing the click , which they tended to hear as a disembodied sound. Similar phenomena have been observed in visual perception of words since the work of Pillsbury ( 1897) . Two of us have proposed a model describing the role of familiarity in perception based on excitatory and inhibitory interactions among units standing for various hypotheses about the input at different levels of abstraction (McClelland & Rumelhart , 1981~ Rumelhart & McClelland , 1982) . The model has been applied in detail to the role of familiarity in the perception of letters in visually presented words, and has proved to provide a very close account of the results of a large number of experiments. The model assumes that there are units that act as detectors for the visual features which distinguish letters, with one set of units assigned to detect the features in each of the different letter -positions in the word. For four -letter words, then , there are four such sets of detectors. There are also four sets of detectors for the letters themselves and a set of detectors for the words.
I. THE APPEAL OFPOP 21 In the model, each unit has an activation value, corresponding roughly to the strength of the hypothesis that what that uni t stands for is present in the perceptual input . The model honors the following important relations which hold between these "hypotheses" or activations: First , to the extent that two hypotheses are mutually consistent, they should support each other . Thus, units that are mutually consistent, in the way that the letter T in the first position is consistent with the word TAKE, tend to excite each other . Second, to the extent that two hypotheses are mutually inconsistent, they should weaken each other . Actually , we can distinguish two kinds of inconsistency: The first kind might be called between-level inconsistency. For example, the hypothesis that a word begins with a T is inconsistent with the hypothesis that the word is MO VE. The second might be called mutual exclusion. For example, the hypothesis that a word begins with T excludes the hypothesis that it begins with R since a word can only begin with one letter . Both kinds of inconsistencies operate in the word perception model to reduce the activations of units. Thus, the letter units in each position compete with all other letter units in the same position , and the word units compete with each other . This type of inhibitory interaction is often called competitiveinhibition. In addition , there are inhibitory interactions between incompatible units on different levels. This type of inhibitory interaction is simply called between-level inhibition. The set of excitatory and inhibitory interactions between units can be diagrammed by drawing excitatory and inhibitory links between them . The whole picture is too complex to draw, so we illustrate only with a fragment: Some of the interactions between some of the units in this model are illustrated in Figure 7. Let us consider what happens in a system like this when a familiar stimulus is presented under degraded conditions . For example, consider the display shown in Figure 8. This display consists of the letters W, 0 , and R , completely visible , and enough of a fourth letter to rule out all letters other than Rand K . Before onset of the display, the activations of the units are set at or below o. When the display is presented, detectors for the features present in each position become active (i.e., their activations grow above 0) . At this point , they begin to excite and inhibit the corresponding detectors for letters. In the first three positions, W, 0 , and R are unambiguously activated, so we will focus our attention on the fourth position where Rand K are both equally consistent with the active features. Here, the activations of the detectors for Rand K start out growing together, as the feature detectors below them become activated. As these detectors become active, they and the active letter detectors for W, 0 , and R in the other positions start to activate detectors for words which have these letters in
I. THE APPEAL OFPOP 23 Word Level
Letter Level
FIGURE 8. A possible display which might be presented to the interactive activation model of word recognition , and the resulting activations of selected letter and word units. The letter units are for the letters indicated in the fourth position of a four -letter display.
K detector allows it to dominate the pattern of activation , suppressing the R detector completely. This example illustrates how POP models can allow knowledge about what letters go together to form words to work together with natural constraints on the task (i .e., that there should only be one letter in one place at one time ) , to produce perceptual completion in a simple and di rect way. Completion of novel patterns . However, the perceptual intelligence of human perceivers far exceeds the ability to recognize familiar pat-
terns and fill in missing portions . We also show facilitation in the
24 THE PDP PERSPECTIVE perception of letters in unfamiliar letter strings which are word -like but not themselves actually familiar . One way of accounting for such performances is to imagine that the perceiver possesses, in addition to detectors for familiar words , sets of detectors for regular subword units such as familiar letter clusters , or that they use abstract rules , specifying which classes of letters can go with
which
others
in different
contexts
. It turns
out , however
, that the
model we have already described needs no such additional structure to produce perceptual facilitation for word -like letter strings ~ to this extent it acts as if it " knows " the orthographic structure of English . We illus trate this feature of the model with the example shown in Figure 9, where the nonword rEAD is shown in degraded form so that the second letter is incompletely visible . Given the information about this letter , considered alone , either E or F would be possible in the second position . Yet our model will tend to complete this letter as an E . The reason for this behavior
of words are partially or F, A , and D , but example ) and others READ , for example activated
as a result
is that , when
YEAD is shown , a number
activated . There is no word consistent with Y, E there are words which match YEA- ( YEAR , for which match _ EAD (BEAD , DEAD , HEAD , and ) . These and other near misses are partially of
the
pattern
of
activation
at the
letter
level .
While they compete with each other , none of these words gets strongly enough activated to completely suppress all the others . Instead , these units act as a group to reinforce particularly the letters E and A . There are no close partial matches which include the letter F in the second position , so this letter receives no feedback support . As a result , E comes to dominate , and eventually suppress , the F in the second position . The fact that the word perception model exhibits perceptual facilita tion to pronounceable nonwords as well as words illustrates once again how behavior in accordance with general principles or rules can emerge from the interactions of simple processing elements . Of course , the behavior of the word perception model does not implement exactly any of the systems of orthographic rules that have been proposed by linguists (Chomsky & Halle , 1968; Venesky , 1970) or psychologists (Spoehr & Smith , 1975) . In this regard , it only approximates such rule -based descriptions of perceptual processing . However , rule systems such as Chomsky and Halle ' s or Venesky ' s appear to be only approximately honored in human performance as well (Smith & Baker , 1976) . Indeed , some of the discrepancies between human performance data and rule systems occur in exactly the ways that we would predict from the word perception model ( Rumelhart & McClelland , 1982) . This illustrates the possibility that POP models may provide more accurate accounts of the details of human performance than models
1. THEAPPEALOFPOP
25
Word Level
Letter Level
E
~ 1: W~ ~J
C& F L Time
Jl 1: wI~ ~
FIGURE 9. An example of a nonword display that might be presented to the interactive activation model of word recognition and the response of selected units at the letter and word levels. The letter units illustrated are detectors for letters in the second input position.
human competence - at least in
~
based on a set of rules representing some domains.
Retrieving Information
From Memory
Content addressabi/ity. One very prominent feature of human memory is that it is content addressable. It seems fairly clear that we
26
THEPOPPERSPECTIVE
can access.information in. memory . based on nearly any attribute of the representation we are trYing to retrieve . Of course, some cues are much better than others. An attribute which is shared by a very large number of things we know about is not a very effective retrieval cue, since it does not accurately pick out a particular memory representation. But, several such cues, in conjunction , can do the job . Thus, if we ask a friend who goes out with several women, "Who was that woman I saw you with ?" , he may not know which one we mean- but if we specify something else about her- say the color of her hair , what she was wearing (in so far as he remembers this at all) , where we saw him with her - he will likely be able to hit upon the right one. It is, of course, possible to implement some kind of content addressability of memory on a standard computer in a variety of dif ferent ways. One way is to search sequentially, examining each memory in the system to find the memory or the set of memories which has the particular content specified in the cue. An alternative , somewhat more efficient , scheme involves some form of indexing keeping a list , for every content a memory might have, of which memories have that content . Such an indexing scheme can be made to work with error-free probes, but it will break down if there is an error in the specification of the retrieval cue. There are possible ways of recovering from such errors, but they lead to the kind of combinatorial explosions which plague this kind of computer implementation . But suppose that we imagine that each memory is represented by a unit which has mutually excitatory interactions with units standing for each of its properties. Then , whenever any property of the memory became active, the memory would tend to be activated, and whenever the memory was activated, all of its contents would tend to become activated. Such a scheme would automatically produce content addressability for us. Though it would not be immune to errors, it would not be devastated by an error in the probe if the remaining properties specified the correct memory . As described thus far , whenever a property that is a part of a number of different memories is activated, it will tend to activate all of the memories it is in . To keep these other activities from swamping the "correct" memory unit , we simply need to add initial inhibitory connections among the memory units . An additional desirable feature would be mutually inhibitory interactions among mutually incompatible property units. For example, a person cannot both be single and married at the same time , so the units for different marital states would be mutually inhibitory .
1. THE APPEAL OFPOP 27
CI ).CI ).CI ).CI }.CI }.CI }.CI }.CI }.CI }.CI }.CI }.CI } .~~~~~~N ============ ~~~~~
McClelland ( 1981) developed a simulation model that illustrates how a system with these properties would act as a content addressable memory. The model is obviously oversimplified , but it illustrates many of the characteristics of the more complex models that will be considered in later chapters. Consider the information represented in Figure 10, which lists a number of people we might meet if we went to live in an unsavory neighborhood, and some of their hypothetical characteristics. A subset The Jets and The Sharks
Name
Gang
Age
Edu
Mar
Occupation
40 ' s
Sing .
Pusher
30 ' s
Mar
20 ' s
Burglar Bookie
30 ' s
Sing . Sing . Sing .
20 ' s
Div .
20 ' s
Mar .
40 ' s
.
.
Bookie Bookie Burglar Pusher
20 ' s
Mar
30 ' s
Sing .
Burglar Bookie Burglar
20 ' s
Mar
20 ' s
Div .
20 ' s
Sing Sing Sing Sing
. . . .
Mar
.
20 ' s 20 ' s 30 ' s
.
Sing . Sing . Mar
.
Mar
.
Mar
.
Sing . Mar
.
Div . Mar
.
Sing . Div .
Burglar Bookie Pusher Pusher Pusher
Pusher Bookie Pusher Burglar Bookie Bookie Burglar Burglar Burglar Pusher Bookie Pusher
FIGURE 10. Characteristics of a number of individualsbelongingto two gangs, the Jets and the Sharks. (From " Retrieving General and Specific Knowledge From Stored Knowledgeof SpecificS ' by J. L. McClelland, 1981, Proceedings of the ThirdAnnualConferenceof the CognitiveScienceSociety , Berkeley, CA. Copyright 1981by J. L. McClelland. Reprintedby permission .)
28 THE POP PERSPECTIVE of the units needed to represent this information is shown in Figure 11. In this network , there is an " instance unit " for each of the characters described in Figure 10, and that unit is linked by mutually excitatory connections to all of the units for the fellow 's properties. Note that we have included property units for the names of the characters, as well as units for their other properties. Now, suppose we wish to retrieve the properties of a particular indi vidual , say Lance. And suppose that we know Lance's name. Then we can probe the network by activating Lance's name unit , and we can see what pattern of activation arises as a result . Assuming that we know of no one else named Lance, we can expect the Lance name unit to be hooked up only to the instance unit for Lance. This will in turn activate the property units for Lance, thereby creating the pattern of
FIGURE 11. Someof the units and interconnectionsneededto representthe individuals shownin Figure 10. The units connectedwith double-headedarrowsare mutuallyexcitatory. All the units within the samecloud are mutually inhibitory. (From " Retrieving General and Specific ..KnowledgeFrom Stored Knowledge of Specifics " by J. L. McClelland, 1981, Proceedings of the Third Annual Conference of the CognitiveScience Society , Berkeley, CA. Copyright1981by J. L. McClelland. Reprintedby permission .)
1. THEAPPEAL OFPOP 29 activation
corresponding
representation described Of
of
.
properties
and
it
the
instance
and
fill
unit
Graceful
ticular
that
item
contains
of
some
other
particular this
model
probe
not
.
This
also
requires
graceful
we
do
in
the
not
. cess1ng
many
and
error
nature
this
fits
the
-
but
not
the
corresponding content
they
make
of
- recovery
probe
point
to
probe
closer of
varies
, errors to
a in
in the
the
wrong
or partial
work
to as the
activation
probe
mechanism
node which
sufficient
nodes
incomplete
probe
one
the of
name
scheme
retrieval
A
, as long
of the
the
' s handling
be
make
degree
to node
the
than
other
set
a par -
sufficient
node .
still
any
kind
the
instance
activate cue
will
does
exact
of the
instance
it
, though
is
the
strongly
than
vary
probes
- it is a natural
that
it is capable
of
. of
the
behavior
of
is that other
we
have
which
.
view
models
this in
Sharks
as the
examined from
will been
have
the
deserve
One
reason
as a stepping
stone
of
model , this
using
this
book
, for
like
model
interactions
occurred
examining
.
models
parts
simple
model
allows
model
other
this
already emerge
have
and
space
, such
occur
It probably we
Jets
present
to say about
we
situations
the
the
, that
more
models
of
than
them
properties
the
is
In
to retrieve
which
be a poorer
strongly
of the
of the
special
assignment of
clearly
general
unless
, have
. un1 ts .
Default in
will
consideration
other
who .
of this
as we
activate
other
information
In
representations
useful
's
know units
properties
features will most
model
development
the
some
of
item will
of
go into
do , however
Lance
, Ken , who
in an attempt
any
kind
aspects
distributed of
.
degradation detailed
set
be fatal
of the
These more
Any
" more
node
no
by - product
memory
information
as a function
will
memory
have other
of
20s
happens
than
misleading
instance
and
desirable what
features
This
answer
item
we
, given
some
individual
the
the
strongly
best .
" right
introduction to
.
misleading
the
of
a particular
misleading
no
activate
a
as well .
to probe
more
it matches
what
do you
Shark
is a single
retrieved
a name with
, say " Who
considering
' s name
contains
just
retrieve
the
have
here .
start
system
A few
from
characterize
which that
.
visible
individual
to
we
than
us stop
might
the
properties
degradation
uniquely
wish
we
,
happen let
by activating
there
effect
we activate these two properties , we will activate . Ken , and this in turn will activate his name unit ,
for
we use
In
will
may
asking
other
are
of features
for
we
that
.
moment
case ,
So , when
in his
model
More
20s ?"
out
.
Lance
the
this
in his
turns
description
of
In
, effectively
a Shark
.
for
, sometimes
information
case
Lance
so far , but
course
to
We
some
exhibits
of
to the , there
more .
the
reader
will
pro -
that
be other
30
THEPOPPERSPECTIVE
activations occurring which may influence the pattern of activation which is retrieved . So, in the case where we retrieved the properties of Lance, those properties, once they become active, can begin to activate the units for other individuals with those same properties. The memory unit for Lance will be in competition with these units and will tend to keep their activation down, but to the extent that they do become active, they will tend to activate their own properties and therefore fill them in . In this way, the model can fill in properties of individuals based on what it knows about other , similar instances. To illustrate how this might work we have simulated the case in which we do not know that Lance is a Burglar as opposed to a Bookie or a Pusher. It turns out that there are a group of individuals in the set who are very similar to Lance in many respects. When Lance's properties become activated, these other units become partially activated, and they start activating their properties. Since they all share the same "occupation," they work together to fill in that property for Lance. Of course, there is no reason why this should necessarily be the right answer, but generally speaking, the more similar two things are in respects that we know about, the more likely they are to be similar in respects that we do not , and the model implements this heuristic . Spontaneous generalization . The model we have been describing has another valuable property as well - it tends to retrieve what is common to those memories which match a retrieval cue which is too general to capture anyone memory . Thus, for example, we could probe the system by activating the unit corresponding to membership in the Jets. This unit will partially activate all the instances of the Jets, thereby causing each to send activations to its properties. In this way the model can retrieve the typical values that the members of the Jets have on each dimension - even though there is no one Jet that has these typical values. In the example, 9 of 15 Jets are single, 9 of 15 are in their 20s, and 9 of 15 have only a Junior High School education~ when we probe by activating the Jet unit , all three of these properties dominate. The Jets are evenly divided between the three occupations, so each of these units becomes partially activated. Each has a different name, so that each name unit is very weakly activated, nearly cancelling each other out . In the example just given of spontaneous generalization, it would not be unreasonable to suppose that someone might have explicitly stored a generalization about the members of a gang. The account just given would be an alternative to " explicit storage" of the generalization. It has two advantages, though , over such an account. First , it does not require any special generalization formation mechanism. Second, it can provide us with generalizations on unanticipated lines, on demand.
1. THE APPEAL OFPDP 31 Thus, if we want to know , for example, what people in their 20s with a junior high school education are like , we can probe the model by activating these two units . Since all such people are Jets and Burglars, these two units are strongly activated by the model in this case; two of them are divorced and two are married , so both of these units are partially activated. 1 The sort of model we are considering, then , is considerably more than a content addressable memory . In addition , it performs default assignment, and it can spontaneously retrieve a general concept of the individuals that match any specifiable probe. These properties must be explicitly implemented as complicated computational extensions of other models of knowledge retrieval , but in POP models they are natural by-products of the retrieval process itself .
REPRESENT ATION AND LEARNING IN PDP MODELS In the Jets and Sharks model, we can speak of the model's active representationat a particular time , and associate this with the pattern of activation over the units in the system. We can also ask: What is the stored knowledge that gives rise to that pattern of activation ? In considering this question, we see immediately an important difference between PDP models and other models of cognitive processes. In most models, knowledge is stored as a static copy of a pattern. Retrieval amounts to finding the pattern in long-term memory and copying it into a buffer or working memory . There is no real difference between the stored representation in long-term memory and the active representation in working memory . In POP models, though , this is not the case. In these models, the patterns themselves are not stored. Rather, what is stored is the connectionstrengths between units that allow these patterns to be re-created. In the Jets and Sharks model, there is an instance unit assigned to each individual , but that unit does not contain a copy of the representation of that individual . Instead, it is simply the case that the connections between it and the other units in the system are such that activation of the unit will cause the pattern for the individual to be reinstated on the property units .
I In enced in
this
activated
this
and
by
partially
all
case , there , giving
other
cases
activated is Married
a
, there
, near Jet
is a tendency neighbors
AI , who
a slight
edge
is
a over
, which Married Divorced
for
the do
pattern
not
Burglar in
quite .
the
The
of
activation
match unit
simulation
to
the for .
probe AI
gets
be
innu
-
.
Thus
,
slightly
32 THE PERSPECTIVE POP This
difference
between
PDP
models
and conventional
models
has
enormous implications , both for processing and for learning. We have already seen some of the implications for processing . The representa tion of the knowledge is set up in such a way that the knowledge necessarily influences the course of processing . Using knowledge in process ing is no longer a matter of finding the relevant information in memory and bringing it to bear ~ it is part and parcel of the processing itself . For learning , the implications are equally profound . For if the knowledge is the strengths of the connections , learning must be a matter of finding the right connection strengths so that the right pat terns of activation will be produced under the right circumstances . This is an extremely important property of this class of models , for it opens up the possibility that an information processing mechanism could learn , as a result of tuning its connections , to capture the interdependencies between activations that it is exposed to in the course of processing . In recent years , there has been Quite a lot of interest in learning in cognitive science . Computational approaches to learning fall predom inantly into what might be called the " explicit rule formulation " tradi tion , as represented by the work of Winston ( 1975) , the suggestions of Chomsky , and the ACT . model of J. R . Anderson ( 1983) . All of this work shares the assumption that the goal of learning is to formulate explicit rules (propositions , productions , etc .) which capture powerful generalizations in a succinct way . Fairly powerful mechanisms , usually
with considerable innate knowledge about a domain, and/ or some starting set of primitive propositional representations , then formulate hypothetical general rules , e.g., by comparing particular cases and for mulating explicit generalizations . The approach that we take in developing POP models is completely different . First , we do not assume that the goal of learning is the for mulation of explicit rules . Rather , we assume it is the acquisition of connection strengths which allow a network of simple units to act as though it knew the rules . Second , we do not attribute powerful compu tational capabilities to the learning mechanism . Rather , we assume very simple connection strength modulation mechanisms which adjust the strength of connections between units based on information locally available
at the connection
.
These issues will be addressed at length in later sections of this book . For now , our purpose is to give a simple , illustrative example of the connection strength modulation process , and how it can produce net works which exhibit some interesting behavior .
Local vs. distributed representation . Before we turn to an explicit consideration of this issue , we raise a basic question about
I. THEAPPEALOFPOP
33
representation. Once we have achieved the insight that the knowledge is stored in the strengths of the interconnections between units , a question arises. Is there any reason to assign one unit to each pattern that we wish to learn? Another possibility - one that we explore extensively in this book - is the possibility that the knowledge about any individual pattern is not stored in the connections of a special unit reserved for that pattern, but is distributed over the connections among a large number of processing units . On this view, the Jets and Sharks model represents a special case in which separate units are reserved for each instance. Models in which connection information is explicitly thought of as distributed have been proposed by a number of investigators. The units in these collections may themselves correspond to conceptual primitives , or they may have no particular meaning as individuals . In either case, the focus shifts to patterns of activation over these units and to mechanisms whose explicit purpose is to learn the right connection strengths to allow the right patterns of activation to become activated under the right circumstances. In the rest of this section, we will give a simple example of a POP model in which the knowledge is distributed . We will first explain how the model would work , given pre-existing connections, and we will then describe how it could come to acquire the right connection strengths through a very simple learning mechanism. A number of models which have taken this distributed approach have been discussed in this book's predecessor, Hinton and J. A . Anderson 's ( 1981) Parallel Models of Associative Memory. We will consider a simple version of a common type of distributed model, a pattern associator. Pattern associators are models in which a pattern of activation over one set of units can cause a pattern of activation over another set of units without any intervening units to stand for either pattern as a whole. Pattern associators would , for example, be capable of associating a pattern of activation on one set of units corresponding to the appearanceof an object with a pattern on another set corresponding to the aroma of the object, so that , when an object is presented visually, causing its visual pattern to become active, the model produces the pattern corresponding to its aroma. How a pattern associator works. For purposes of illustration , we present a very simple pattern associator in Figure 12. In this model, there are four units in each of two pools. The first pool, the A units , will be the pool in which patterns corresponding to the sight of various objects might be represented. The second pool, the B units , will be the pool in which the pattern corresponding to the aroma will be represented. We can pretend that alternative patterns of activation on
34 THEPOP PERSPECTIVE From Vision
A Units
B Units FIGURE 12. A simple pattern associator. The example assumes that patterns of activation in the A units can be produced by the visual system and patterns in the B units can be produced by the olfactory system. The synaptic connections allow the outputs of the A units to influence the activations of the B units. The synaptic weights linking the A units to the B units were selected so as to allow the pattern of activation shown on the A units to reproduce the pattern of activation shown on the B units without the need for any olfactory input .
the A units are produced upon viewing a rose or a grilled steak, and alternative patterns on the B units are produced upon sniffing the same objects. Figure 13 shows two pairs of patterns, as well as sets of inter connections necessaryto allow the A member of each pair to reproduce the B member. The details of the behavior of the individual units vary among dif ferent versions of pattern associators. For present purposes, we' ll assume that the units can take on positive or negative activation values, with 0 representing a kind of neutral intermediate value. The strengths of the interconnections between the units can be positive or negative real numbers. The effect of an A unit on a B unit is determined by multiplying the activation of the A unit times the strength of its synaptic connection with the B unit . For example, if the connection from a particular A unit to a particular B unit has a positive sign, when the A unit is
1. THEAPPEALOFPOP
+1
-1
-1
-1
+1
+1
-1
35
+1
- .25 +.25 +.25 - .25
- 1
-1
- .25 +.25 +.25 - .25
- 1
+1
+.25 - .25 - .25 +.25
+ 1
+1
+.25 - .25 - .25 +.25
+1
-1
FIGURE 13. Two simpleassociators representedas matrices. The weightsin the first two matricesallow the A patternshownabovethe matrix to producethe B patternshownto the right of it. Note that the weightsin the first matrix are the sameas thoseshownin the diagramin Figure 12.
excited (activation greater than 0) , it will excite the B unit . For this example, we'll simply assume that the activation of each unit is set to the sum of the excitatory and inhibitory effects operating on it . This is one of the simplest possible cases. Suppose, now, that we have created on the A units the pattern corresponding to the first visual pattern shown in Figure 13, the rose. How should we arrange the strengths of the interconnections between the A units and the B units to reproduce the pattern corresponding to the aroma of a rose? We simply need to arrange for each A unit to tend to excite each B unit which has a positive activation in the aroma pattern and to inhibit each B unit which has a negative activation in the aroma pattern. It turns out that this goal is achieved by setting the strength of the connection between a given A unit and a given B unit to a value proportional to the product of the activation of the two units . In Figure 12, the weights on the connections were chosen to allow the A pattern illustrated there to produce the illustrated B pattern according to this principle . The actual strengths of the connections were set to :::t: .25, rather than :t: 1, so that the A pattern will produce the right magnitude , as well as the right sign, for the activations of the units in the B pattern. The same connections are reproduced in matrix form in Figure 13A. Pattern associators like the one in Figure 12 have a number of nice properties. One is that they do not require a perfect copy of the input to produce the correct output , though its strength will be weaker in this case. For example, suppose that the associator shown in Figure 12 were presented with an A pattern of ( 1,- 1,0, 1) . This is the A pattern shown in the figure , with the activation of one of its elements set to O. The B pattern produced in response will have the activations of all of the B units in the right direction ; however, they will be somewhat weaker than they would be, had the complete A pattern been shown. Similar
36
THEPOP PERSPECTIVE
effects are producedif an element of the pattern is distorted- or if the model is damaged , either by removing whole units, or random sets of connections, etc. Thus, their pattern retrieval performance of the model degrades gracefully both under degraded input and under damage . How a pattern associator learns. So far, we have seen how we as model builders can constructthe right set of weightsto allow one pattern to causeanother. The interestingthing, though, is that we do not need to build these interconnectionstrengthsin by hand. Instead, the pattern associatorcan teach itself the right set of interconnections through experienceprocessingthe patterns in conjunction with each other. A number of different rules for adjustingconnectionstrengthshave beenproposed . One of the first- and definitely the bestknown- is due to D. O. Hebb (1949) . Hebb's actual proposalwas not sufficiently quantitativeto build into an explicit model. However, a number of different variantscan trace their ancestryback to Hebb. Perhapsthe simplest version is: When unit A and unit B are simultaneouslyexcited, increase the strengthof the connectionbetweenthem. A natural extension of this rule to cover the positive and negative activationvaluesallowedin our exampleis: Adjust the strengthof the connectionbetweenunits A and B in proportionto the productof their simultaneousactivation. In this formulation, if the product is positive, the changemakes the connectionmore excitatory, and if the product is negative, the change makesthe connectionmore inhibitory. For simplicity of reference, we will call this the Hebb rule, although it is not exactly Hebb's original formulation. With this simple learning rule, we could train a "blank copy" of the pattern associatorshown in Figure 12 to producethe 8 pattern for rose when the A pattern is shown, simply by presentingthe A and B patterns togetherand modulatingthe connectionstrengthsaccordingto the Hebb rule. The size of the change made on every trial would, of course, be a parameter . We generallyassumethat the changesmadeon each instanceare rather small, and that connectionstrengthsbuild up gradually. The valuesshownin Figure 13A, then, would be acquiredas a result of a number of experienceswith the A and B pattern pair.
1. THE APPEAL OFPDP 37 It is very important to note that the information needed to use the Hebb rule to determine the value each connection should have is locally available at the connection. All a given connection needs to consider is the activation of the units on both sides of it . Thus, it would be possible to actually implement such a connection modulation scheme locally, in each connection, without requiring any programmer to reach into each connection and set it to just the right value. It turns out that the Hebb rule as stated here has some serious limi tations, and, to our knowledge, no theorists continue to use it in this simple form . More sophisticated connection modulation .schemeshave been proposed by other workers~ most important among these are the delta rule , discussed extensively in Chapters 8 and 11; the competitive learning rule, discussed in Chapter 5~ and the rules for learning in stochastic parallel models, described in Chapters 6 and 7. All of these learning rules have the property that they adjust the strengths of connections between units on the basis of information that can be assumed to be locally available to the unit . Learning , then , in all of these cases, amounts to a very simple process that can be implemented locally at each connection without the need for any overall supervision. Thus, models which incorporate these learning rules train themselves to have the right interconnections in the course of processing the members of an ensemble of patterns. Learning multiple patterns in the same set of interconnections. Up to now, we have considered how we might teach our pattern associator to associate the visual pattern for one object with a pattern for the aroma of the same object. Obviously , different patterns of interconnections between the A and B units are appropriate for causing the visual pattern for a different object to give rise to the pattern for its aroma. The same principles apply, however, and if we presented our patterri associator with the A and B patterns for steak, it would learn the right set of interconnections for that case instead (these are shown in Figure 13B) . In fact, it turns out that we can actually teach the same pattern associator a number of different associations. The matrix representing the set of interconnections that would be learned if we taught the same pattern associator both the rose association and the steak association is shown in Figure 14. The reader can verify this by adding the two matrices for the individual patterns together. The reader can also verify that this set of connections will allow the rose A pattern to produce the rose B pattern, and the steak A pattern to produce the steak B pattern: when either input pattern is presented, the correct corresponding output is produced. The examples used here have the property that the two different visual patterns are completely uncorrelated with each other . This being
-+
-+ -
38 THE POP PERSPECTIVE
+
+
+
+
+
-
-
+
-
-
-
-
+
+ +
++
- -
-++
FIGURE 14. The weightsin the third matrix allow either A patternshownin Figure 13 to recreatethe corresponding8 pattern. Eachweight in this caseis equalto the sum of the weightfor the A patternand the weightfor the B pattern, as illustrated.
the case, the rose pattern produces no effect when the interconnections for the steak have been established, and the steak pattern produces no effect when the interconnections for the rose association are in effect . For this reason, it is possible to add together the pattern of intercon nections for the rose association and the pattern for the steak association , and still be able to associate the sight of the steak with the smell of a steak and the sight of a rose with the smell of a rose. The two sets of interconnections do not interact at all. One of the limitations of the Hebbian learning rule is that it can learn the connection strengths appropriate to an entire ensemble of patterns only when all the patterns are completely uncorrelated. This restriction does not , however, apply to pattern associators which use more sophisticated learning schemes.
_ I ! III
111 _ 11 1'.
-
.
~
I
~
-
~
.
Attractive properties of pattern ass0cia tor models. Pattern associator models have the property that uncorrelated patterns do not interact with each other , but more similar ones do. Thus, to the extent that a new pattern of activation on the A units is similar to one of the old ones, it will tend to have similar effects. Furthermore , if we assume that learning the interconnections occurs in small increments, similar patterns will essentially reinforce the strengths of the links they share in common with other patterns. Thus, if we present the same pair of patterns over and over, but each time we add a little random noise to each element of each member of the pair, the system will automatically learn to associate the central tendency of the two patterns and will learn to ignore the noise. What will be stored will be an average of the similar patterns with the slight variations removed. On the other hand, when we present the system with completely uncorrelated patterns, they will not interact with each other in this way. Thus, the same pool of units can extract the central tendency of each of a number of pairs of unrelated patterns. This aspect of distributed models is exploited extensively in Chapters 17 and 25 on distributed memory and amnesia.
1. THE APPEAL OFPOP 39 Extracting the structure of an ensemble of patterns . The fact that similar patterns tend to produce similar effects allows distributed models to exhibit a kind of spontaneous generalization, extending behavior appropriate for one pattern to other similar patterns. This property is shared by other PDP models, such as the word perception model and the Jets and Sharks model described above~ the main differ ence here is in the existence of simple, local, learning mechanisms that can allow the acquisition of the connection strengths needed to produce these generalizations through experience with members of the ensemble of patterns. Distributed models have another interesting property as well: If there are regularities in the correspondences between pairs of patterns, the model will naturally extract these regularities. This property allows distributed models to acquire patterns of interconnections that lead them to behave in ways we ordinarily take as evidence for the use of linguistic rules. A detailed example of such a model is described in Chapter 18. Here, we describe the model very briefly . The model is a mechanism that learns how to construct the past tenses of words from their root forms through repeated presentations of examples of root forms paired with the corresponding past-tense form . The model consists of two pools of units . In one pool, patterns of activation representing the phonological structure of the root form of the verb can be represented, and, in the other , patterns representing the phonological structure of the past tense can be represented. The goal of the model is simply to learn the right connection strengths between the root units and the past-tense units , so that whenever the root form of a verb is presented the model will construct the corresponding past-tense form . The model is trained by presenting the root form of the verb as a pattern of activation over the root units , and then using a simple, local, learning rule to adjust the connection strengths so that this root form will tend to produce the correct pattern of activation over the past-tense units . The model is tested by simply presenting the root form as a pattern of activation over the root units and examining the pattern of activation produced over the past-tense units . The model is trained initially with a small number of verbs children learn early in the acquisition process. At this point in learning, it can only produce appropriate outputs for inputs that it has explicitly been shown. But as it learns more and more verbs, it exhibits two interesting behaviors. First , it produces the standard ed past tense when tested with pseudo-verbs or verbs it has never seen. Second, it " overregularizes" the past tense of irregular words it previously completed correctly . Often , the model will blend the irregular past tense of the word with the regular ed ending, and produce errors like CAMED as the past of
40
THEPOPPERSPECTIVE
COME . These phenomena mirror those observed in the early phases of acquisition of control over past tenses in young children . The generativity of the child ' s responses - the creation of regular past tenses of new verbs and the overregularization of the irregular verbs - has been taken
as strong
evidence
that the child
has induced
the
rule which states that the regular correspondence for the past tense in English is to add a final ed (Berko , 1958) . On the evidence of its per formance , then , the model can be said to have acquired the rule . How ever , no special rule -induction mechanism is used , and no special language -acquisition device is required . The model learns to behave in accordance with the rule , not by explicitly noting that most words take ed in the past tense in English and storing this rule away explicitly , but simply by building up a set of connections in a pattern associator through a long series of simple learning experiences . The same mechanisms of parallel distributed processing and connection modi fica -. tion
which
are used in a number
of domains
serve , in this case , to pro -
duce implicit knowledge tantamount to a linguistic rule . The model also provides a fairly detailed account of a number of the specific aspects of the error patterns children make in learning the rule . In this sense, it provides a richer and more detailed description of the acquisi tion process than any that falls out naturally from the assumption that the child is building up a repertoire of explicit but inaccessible rules . There is a lot more to be said about distributed models of learning , about their strengths and their weaknesses , than we have space for in this preliminary consideration . For now we hope mainly to have suggested that they provide dramatically different accounts of learning and acquisition than are offered by traditional models of these processes . We saw in earlier sections of this chapter that performance in accor dance with rules can emerge from the interactions of simple , intercon nected units . Now we can see how the aquisition of performance that conforms to linguistic rules can emerge from a simple , local , connec tion strength modulation process . We have seen what the properties of POP models are in informal terms , and we have seen how these properties operate to make the models do many of the kinds of things that they do . The business of the next chapter is to layout these properties more formally , and to introduce some formal tools for their description and analysis . Before we turn to this , however , we wish to describe some of the major sources of inspiration for the POP approach .
1. THEAPPEAL OFPOP 41
ORIGINSOF PARALLEL DISTRIBUTEDPROCESSING The ideas behind the PDP approach have a history that stretches back indefinitely . In this section, we mention briefly some of the people who have thought in these terms, particularly those whose work has had an impact on our own thinking . This section should not been seen as an authoritative review of the history , but only as a description of our own sources of inspiration . Some of the earliest roots of the POP approach can be found in the work of the unique neurologists, Jackson ( 1869/ 1958) and Luria ( 1966) . Jackson was a forceful and persuasive critic of the simplistic localizationist doctrines of late nineteenth century neurology, and he argued convincingly for distributed , multilevel conceptions of processing systems. Luria , the Russian psychologist and neurologist, put for ward the notion of the dynamicfunctional system. On this view, every behavioral or cognitive process resulted from the coordination of a large number of different components, each roughly localized in different regions of the brain, but all working together in dynamic interaction . Neither Hughlings-Jackson nor Luria is noted for the clarity of his views, but we have seen in their ideas a rough characterization of the kind of parallel distributed processing system we envision . Two other contributors to the deep background of POP were Hebb ( 1949) and Lashley ( 1950) . We already have noted Hebb's contribu tion of the H,ebb rule of synaptic modification ~ he also introduced the concept of cell assemblies- a concrete example of a limited form of distributed processing- and discussed the idea of reverberation of activation within neural networks. Hebb's ideas were cast more in the form of speculations about neural functioning than in the form of concrete processing models, but his thinking captures some of the flavor of parallel distributed processing mechanisms. Lashley's contribution was to insist upon the idea of distributed representation. Lashley may have been too ' radical and too vague, and his doctrine of equipotentiality of broad regions of cortex clearly overstated the case. Yet many of his insights into the difficulties of storing the "engram" locally in the brain are telling , and he seemed to capture quite precisely the essenceof distributed representation in insisting that "there are no special cells reserved for special memories" (Lashley, 1950, p. 500) . In the 1950s, there were two major figures whose ideas have contri buted to the development of our approach. One was Rosenblatt ( 1959, 1962) and the other was Selfridge ( 1955) . In his Principles of Neurodynamics ( 1962) , Rosenblatt articulated clearly the promise of a neurally inspired approach to computation , and he developed the perceptron convergenceprocedure, an important advance over the Hebb rule for
42 THE POP PERSPECTIVE changing synaptic connections . Rosenblatt ' s work was very controver sial at the time , and the specific models he proposed were not up to all the hopes he had for them . But his vision of the human information
processing system as a dynamic, interactive , self-organizing system lies at the core of the POP approach. Selfridge's contribution was his insistence on the importance of interactive processing , and the develop ment of Pandemonium , an explicitly computational example of a dynamic , interactive mechanism applied to computational problems in perception . In the late 60s and early 70s, serial processing and the von Neumann computer dominated both psychology and artificial intelligence , but there were a number of researchers who proposed neural mechanisms which capture much of the flavor of PDP models . Among these fig ures , the
most
influential
in
our
work
have
been
J . A . Anderson
,
Grossberg , and Longuet -Higgins . Grossberg ' s mathematical analysis of the properties of neural networks led him to many insights we have only come to appreciate through extensive experience with computer simulation , and he deserves credit for seeing the relevance of neurally inspired mechanisms in many areas of perception and memory well before the field was ready for these kinds of ideas (Grossberg , 1978) . Grossberg ( 1976) was also one of the first to analyze some of the properties of the competitive learning mechanism explored in Chapter 5. Anderson ' s work differs from Grossberg ' s in insisting upon distributed representation , and in showing the relevance of neurally inspired models for theories of concept learning (Anderson , 1973, 1977) ; the work in Chapters 17 and 25 on distributed memory and amnesia owes a great deal to Anderson ' s inspiration . Anderson ' s work also played a crucial role in the formulation of the cascade model ( McClelland , 1979) , a step away from serial processing down the road to PDP . Longuet -Higgins and his group at Edinburgh were also pursu ing distributed memory models during the same period , and David
Willshaw , a member of the Edinburgh group, provided some very elegant mathematical analyses of the properties of various distributed representation schemes (Willshaw , 1981) . His insights provide one of the sources of the idea of coarse coding described at length in Chapter 3. Many of the contributions of Anderson , Willshaw , and others distributed modelers may be found in Hinton and Anderson ( 1981) . Others who have made important contributions to learning in POP models include Amari ( 1977a) , Bienenstock , Cooper , and Munro ( 1982) , Fukushima ( 1975) , Kohonen ( 1977 , 1984) , and von der Malsburg ( 1973) . Toward the middle of the 1970s, the idea of parallel processing began to have something of a renaissance in computational circles . We have already mentioned the Marr and Poggio ( 1976) model of stereoscopic
1. THEAPPEALOFPOP
43
depth perception . Another model from this period , the HEARSA Y model of speech understanding , played a prominent role in the development of our thinking . Unfortunately , HEARSAY ' s computa tional architecture was too demanding for the available computational resources , and so the model was not a computational success. But its basically parallel , interactive character inspired the interactive model of reading (Rumelhart , 1977) , and the interactive activation model of
word recognition
(McClelland & Rumelhart , 1981~ Rumelhart &
McClelland , 1982) . The ideas represented in the interactive activation model had other precursors as well . Morton ' s logogen model (Morton , 1969) was one of the first models to capture concretely the principle of interaction of dif ferent sources of information , and Marslen -Wilson (e.g., Marslen Wilson & Welsh , 1978) provided important empirical demonstrations of interaction between different levels of language processing . Levin ' s ( 1976) Proteus model demonstrated the virtues of activation competition mechanisms , and Glushko ( 1979) helped us see how con spiracies of partial activations could account for certain aspects of apparently rule -guided behavior . Our work also owes a great deal to a number of colleagues who have been working on related ideas in recent years . Many of these col leagues appear as authors or coauthors of chapters in this book . But there are others as well . Several of these people have been very influential in the development of the ideas in this book . Feldman and Ballard ( 1982) laid out many of the computational principles of the POP approach (under the name of connectionism ) , and stressed the bio logical implausibility of most of the prevailing computational models in artificial intelligence . Hofstadter ( 1979, 1985) deserves credit for stressing the existence of a subcognitive - what we call microstructural - level , and pointing out how important it can be to delve into the microstructure to gain insight . A sand dune , he has said , is not a grain of sand . Others have contributed crucial technical insights . Sutton and Barto ( 1981) provided an insightful analysis of the connection
modification
scheme
we call the delta rule and illustrated
the
power of the rule to account for some of the subtler properties of classical conditioning . And Hopfield ' s ( 1982) contribution of the idea that network models can be seen as seeking minima in energy landscapes played a prominent role in the development of the Boltzmann machine (Chapter 7) , and in the crystallization of the ideas presented in Chapters 7 and 14 on harmony theory and schemata . The power of parallel distributed processing is becoming more and more apparent , and many others have recently joined in the exploration of the capabilities of these mechanisms . We hope this book represents
44
THEPOPPERSPECTIVE
the nature of the enterprisewe are all involved in, and that it doesjustice to the potentialof the POP approach .
ACKNOWLEDGMENTS This researchwas supported by Contract NOOO14 -79-C-O323, NR 667-437 with the Personneland Training ResearchProgramsof the Office of Naval Research , by grants from the System Development Foundation, and By a NIMH CareerDevelopmentAward (MHOO385 ) to the first author.
CHAPTER 2 A General Framework for Parallel Distributed Processing
D. E. RUMELHART , G. E. HINTON , andJ. L. McCLELLAND
In Chapter 1 and throughout this book, we describe a large number of models, each different in detail- each a variation on the parallel distributed processing (POP) idea. These various models, and indeed many in the literature , clearly have many features in common , but they are just as clearly distinct models. How can we characterize the general model of which these specific models are instances? In this chapter we propose a framework sufficiently general so that all of the various models discussed in the book and many models in the literature are special cases. We will proceed by first sketching the general framework and then by showing properties of certain specific realizations of the general model. 1
The GeneralFramework
It our
1 class are
is
useful
to
models
We of papers
are
and
,
models with
of
begin then
course .
Kohonen similarly
,
with
an
describe
not
the
first
( 1977
,
general
analysis the
to 1984
aims
attempt ) ,
.
of
various
Amari
the
various
specific
a
general
( 1977a
components
of
assumptions
we
characterization ) , and
Feldman
of and
this
Ballard
can
general ( 1982
)
46
THEPOP PERSPECTIVE
make about these components . There are eight major aspectsof a paralleldistributedprocessingmodel: . . . . . .
. .
A set of processingunits A state of activation An outputfunction for each uni t A pattern of connectivityamong units A propagation rule for propagating patterns of activities through the network of connectivities An activation rule for combining the inputs impinging on a unit with the current state of that unit to produce a new level of activation for the unit . A learning rule whereby patterns of connectivity are modified by . experIence An environmentwithin which the system must operate
Figure 1 illustrates the basic aspects of these systems. There is a set of processing units generally indicated by circles in our diagrams; at each point in time , each unit Ui has an activation value, denoted in the diagram as 0; (t ) ; this activation value is passedthrough a function I ; to produce an output value i (t ) . This output value can be seen as passing through a set of unidirectional connections (indicated by lines or arrows in our diagrams) to other units in the system. There is associated with each connection a real number , usually called the weight or strength of the connection designated Wi} which determines the amount of effect that the first unit has on the second. All of the inputs must then be combined by some operator (usually addition) - and the combined inputs to a unit , along with its current activation value, determine , via a function F, its new activation value. The figure shows illus trative examples of the function I and F. Finally , these systems are viewed as being plastic in the sense that the pattern of interconnections is not fixed for all time ; rather, the weights can undergo modification as a function of experience. In this way the system can evolve. What a unit represents can change with experience, and the system can come to perform in substantially different ways. In the following sections we develop an explicit notation for each of these components and describe some of the alternate assumptions that have been made concerning each such component.
A set of processingunits. Any parallelactivation model beginswith a set of processingunits. Specifyingthe set of processingunits and what they represent is typically the first stage of specifying a POP model. In somemodelstheseunits may representparticularconceptual objectssuch as features, letters, words, or concepts; in others they are
2. A FRAMEWORK FORPOP
47
Gt (I)
1
/ } (o} ) 0 .m
0
+ m
Thr ~ hoId Output Function
M a,0 -m MI ,=10 (I)ljOj (l) SigmoidActivation Function
FIGURE 1. The basiccomponentsof a paralleldistributedprocessingsystem. simply abstract elements over which meaningful patterns can be defined. When we speak of a distributed representation, we mean one in which the units represent small, feature-like entities. In this case it is the pattern as a whole that is the meaningful level of analysis. This should be contrasted to a one-unit- one-concept representational system in which single units represent entire concepts or other large meaningful entities . We let N be the number of units. We can order the units arbitrarily and designate the ith unit U; . All of the processing of a POP model is carried out by these units . There is no executive or other overseer. There are only relatively simple units, each doing it own relatively simple job . A unit 's job is simply to receive input from its neighbors and, as a function of the inputs it receives, to compute an output value which it sends to its neighbors. The system is inherently parallel in that many units can carry out their computations at the same time .
48
THEPOPPERSPECTIVE
Within
any
types
of
system
units
sources
:
to
input
which
.
den
units
tem
we
modeling
state
of
of
by
tor
a
of
tion
over
the
unit
.
unit
or
.
small
If
set
may
such
as
,
are
mean
models
,
denoted
As
slightly
,
0
we
, +
.
I
}
,
or
shall
,
a
see
.
different
small
each
, +
for
}
)
in
this
and
is
of
book
]
,
.
they
of
be
It
is
{
leads
part
the
of
implications
the
taken
it
is
to
inactive
values
{
.
-
l
to
I
,
2
to
,
3
,
4
a
,
5
l
the
,
6
,
}
set
7
model
program
of
of
, +
values
restricted
as
-
Sometimes
usually
that
the
,
max
discrete
such
assumptions
.
cases
values
.
is
a
and
activation
to
may
of
and
nortbinary
values
any
other
binary
mean
restricted
-
or
or
In
When
1
to
continu
continuous
.
where
-
activation
minimum
are
times
set
determine
1
Other
these
to
1
taken
are
.
example
finite
characteristics
represented
assumptions
Thus
to
0
,
evolu
be
values
often
-
unbounded
are
value
0
activa
.
the
be
-
-
the
units
may
some
[
0
of
of
binary
vec
activa
representing
as
set
a
pattern
The
is
units
activation
need
the
pattern
may
,
.
system
they
most
values
-
the
about
interval
and
t
system
the
between
values
{
is
take
an
they
activation
It
-
primarily
of
time
the
models
we
is
the
values
may
the
active
simply
involved
I
the
is
.
the
,
as
,
)
Activation
they
values
unit
.
value
example
t
,
This
element
at
-
sys
.
units
.
hid
the
representing
Each
in
some
number
to
the
other
,
discrete
restricted
that
often
-
for
to
,
over
continuous
real
)
assumptions
in
any
(
within
of
t
units
activity
on
are
real
on
t
what
different
take
Thus
(
processing
discrete
.
any
restricted
they
are
take
imum
see
set
.
Qj
of
they
values
on
may
If
a
the
captures
pattern
to
they
of
take
they
are
a
of
designated
to
of
allowed
.
one
that
make
is
discrete
bounded
is
useful
,
,
units
The
systems
time
the
simply
.
outside
the
of
or
modeling
are
to
at
numbers
of
models
a
In
is
to
system
processing
units
time
Different
real
t
"
in
out
systems
are
either
system
signals
outputs
visible
,
the
of
time
of
It
through
values
are
at
set
time
,
N
set
"
be
processing
send
we
and
from
may
motoric
addition
of
activation
Uj
the
any
ous
of
the
the
ones
three
inputs
inputs
units
the
not
In
state
vector
for
tion
tion
the
over
stands
at
.
of
activation
are
characterize
receive
These
of
inputs
They
activation
representation
specified
only
.
.
affect
to
to
units
output
directly
whose
useful
parts
The
external
those
are
The
either
is
Input
study
other
.
it
.
under
embedded
systems
are
hidden
system
may
other
,
and
from
is
They
modeling
,
inputs
model
influence
are
output
the
or
the
system
{
,
external
sensory
(
we
input
these
,
8
,
9
}
.
with
research
various
.
Output of the units. Units interact. They do so by transmitting signals to their neighbors. The strengthof their signals, and therefore the degree to which they affect their neighbors, is determined by their degreeof activation. Associatedwith each unit, u;, there is an output function, Ii (ai (1)) , which mapsthe current state of activation Oi(1) to
2. A FRAMEWORK FORPOP 49
an output signal 0; (t ) (i.e., 0; (t ) ~ I ; (a; (t ) . In vector notation , we represent the current set of output values by a vector, 0 (I ) . In some of our models the output level is exactly equal to the activation level of the unit . In this casef is the identity function f (x )~ x . More often , however, I is some sort of threshold function so that a unit has no affect on another unit unless its activation exceeds a certain value. Sometimes the function f is assumed to be a stochastic function in which the output of the unit depends in a probabilistic fashion on its activation values. The pattern o/ connectivity. Units are connected to one another. It is this pattern of connectivity that constitutes what the system knows and determines how it will respond to any arbitrary input . Specifying the processing system and the knowledge encoded therein is, in a parallel distributed processing model, a matter of specifying this pattern of connectivity among the processing units . In many cases, we assume that each unit provides an additive contri bution to the input of the units to which it is connected. In such cases, the total input to the unit is simply the weighted sum of the separate inputs from each of the individual units . That is, the inputs from all of the incoming units are simply multiplied by a weight and summed to get the overall input to that unit . In this case, the total pattern of connectivity can be represented by merely specifying the weights for each of the connections in the system. A positive weight represents an excitatory input and a negative weight represents an inhibitory input . As mentioned in the previous chapter, it is often convenient to represent such a pattern of connectivity by a weight matrix W in which the entry W;j represents the strength and sense of the connection from unit Uj to unit u; . The weight Wij is a positive number if unit Uj excites unit Ui; it is a negative number if unit Uj inhibits unit Ui~ and it is 0 if unit Uj has no direct connection to unit Ui. The absolute value of Wij specifies the strength of the connection. Figure 2 illustrates the relationship between the connectivity and the weight matrix . In the general case, however, we require rather more complex patterns of connectivity . A given unit may receive inputs of different kinds whose effects are separately summated. For example, in the previous paragraph we assumed that the excitatory and inhibitory connections simply summed algebraically with positive weights for excitation and negative weights for inhibition . Sometimes, more complex inhibition / excitation combination rules are required. In such casesit is convenient to have separateconnectivity matrices for each kind of connection. Thus, we can represent the pattern of connectivity by a set of connectivity matrices, Wi , one for each type of connection. It is common, for example, to have two types of connections in a model: an
-
-
-
-
-
-
-
-
-
-
-
-
-
-
.
-
-
-
-
-
-
-
-
-
-
.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
,
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
uO'lelUaSaJda ~ X!J\BV-.j
-
-
-
_
-
_
-
_
.
.
.
50 THE POP PERSPECTIVE
2.
A FRAMEWORK FORPOP
51
inhibitory connection and an excitatory connection. When the models assume simple addition of inhibition and excitation they do not constitute different typesof connections in our present sense. They only constitute distinct types when they combine through some more complex rules. The pattern of connectivity is very important . It is this pattern which determines what each unit represents. As we shall see below, many of the issues concerning whether top-down or bottom-up processing systems are correct descriptions or whether a system is hierarchical and if so how many levels it haS, etc., are all issues of the nature of the connectivity matrix . One important issue that may determine both how much information can be stored and how much serial processing the network must perform is the fan -in and fan -out of a unit . The fan-in is the number of elements that either excite or inhibit a given unit . The fanout of a unit is the number of units affected directly by a unit . Note , in some cases we need more general patterns of connectivity . Specifying such a pattern in the general case is complex and will be addressed in a later section of this chapter. The rule of propagation . We also need a rule which takes the output vector, 0 (t ) , representing the output values of the units and combines it with the connectivity matrices to produce a net input for each type of input into the unit . We let netj} be the net input of type ; to unit u} . Whenever only one type of connectivity is involved we suppress the first subscript and use net} to mean the net input into unit u} . In vector notation we can write netj (t ) to represent the net input vector for inputs of type i. The propagation rule is generally straightforward . For example, if we have two types of connections, inhibitory and excitatory , the net excitatory input is usually the weighted sum of the excitatory inputs to the unit . This is given by the vector product nete = W eO(t ) . Similarly , the net inhibitory effect can be written as netj = Wjo (t ) . When more complex patterns of connectivity are involved , more complex rules of propagation are required. We treat this in the final section 01:- the chapter. Activation rule . We also need a rule whereby the net inputs of each type impinging on a particular unit are combined with one another and with the current state of the unit to produce a new state of activation . We need a function , F , which takes a (1) and the vectors net j for each different type of connection and produces a new state of activation . In the simplest cases, when F is the identity function and when all connections are of the same type, we can write a (1+ 1) = Wo (1) = net (1) . Sometimes F is a threshold function so that the net input must exceed some value before contributing to the new state of activation . Often ,
52 THEPDPPERSPECTIVE the new state of activation dependson the old one as well as the current input. In general, however, we have a (t+ 1) ~ F (8 (t ) ,net (t ) 1,net (t )2,...) ~ the function F itself is what we call the activation function is assumed to be deterministic . Thus , threshold is involved it may be that aj (t ) = 1 if the some threshold value and equals 0 otherwise . assumed
that
F
is stochastic .
Sometimes
activations
rule . Usually , the for example , if a total input exceeds Other times it is are assumed
to
decay slowly with time so that even with no external input the activa tion of a unit will simply decay and not go directly to zero . Whenever a; (t ) is assumed to take on continuous values it is common to assume
that F is a kind of sigmoid function . In this case, an individual unit can saturate
and reach a minimum
or maximum
value of activation
Perhaps the most common class of activations functions linear
activation
function
.
In this
case the activation
.
is the quasi-
function
, F , is a
nondecreasing function of a single type of input . In short ,
Qj(/+ 1) = F (netj(I )) = F (I . wjjoj). j It is sometimesuseful to add the constraint that F be a differentiable function. We refer to differentiablequasi-linear activationfunctions as semilinearfunctions (seeChapter8) . ModifYing patterns of connectivity as a jilnction of experience. Changing the processing or knowledge structure in a parallel distributed processing model involves modifying the patterns of interconnectivity . In principle this can involve three kinds of modifications :
1. The developmentof new connections. 2. The lossof existingconnections. 3. The modification of the strengthsof connectionsthat already exist. Very little work has been done on (1) and (2) above. To a first order of approximation , however, (1) and (2) can be considered a special case of (3) . Whenever we change the strength of connection away from zero to some positive or negative value, it has the same effect as growing a new connection. Whenever we change the strength of a connection to zero, that has the same effect as losing an existing connection . Thus, in this section we will concentrate on rules whereby strengthsof connections are modified through experience.
2. AFRAMEWORK FOR PDP 53 Virtually all learning rules for models of this type can be considered a variant of the Hebbian learning rule suggested by Hebb in his classic book Organization of Behavior ( 1949) . Hebb 's basic idea is this : If a unit , Ui, receives a input from another unit , u} ; then , if both are highly
active, the weight, Wi} , from Uj to Ui should be strengthened . This idea has been extended and modified so that it can be more generally stated as
4 W;j ~ g (a; (1) ,1; (1 h (OJ(1) ,W;j ) , where ti (t ) is a kind of teaching input to Ui. Simply stated , this equa tion says that the change in the connection from Uj to Ui is given by the
product of a function , gO , of the activation of Ui and its teaching input ti
and another function , hO , of the output value of Uj and the con -
nection strength Wij. In the simplest versions of Hebbian learning there is no teacher and the functions g and h are simply proportional first
arguments
to their
. Thus we have
AW I)" = '. nQ . O) ' , "
~
where 11 is the constant of proportionality representing the learning rate . Another common variation is a rule in which h (OJ(t ) ,W;j ) = OJ(t ) and g (a; (t ) ,t; (t = '7) (t; (t )- a; (1 . This is often called the Widrow Hoff rule (Sutton & Barto , 1981) . However , we call it the delta rule
because the amount of learning is proportional to the difference (or delta ) between the actual activation achieved and the target activation provided by a teacher . (The delta rule is discussed at length in Chapters 8 and 11.) In this case we have ~ W;j ~ 'r) (t; (t )- a; (t
Oj (t ) .
This is a generalization of the perceptron learning rule for which the famous perception convergence theorem has been proved . Still another variation
has
6 Wij ~ 7) Q; (I ) (OJ(/ )- Wij) . This is a rule employed by Grossberg ( 1976) and a simple variant of which has been employed in Chapter 5. There are many variations on this generalized rule , and we will describe some of them in more detail when we discuss various specific models below .
Representation of the environment. It is crucial in the development of any model model
to have a clear
model
of the environment
is to exist . In POP models , we represent
in which
the environment
this as a
time -varying stochastic function over the space of input patterns. That
54 THEPDPPERSPECTIVE is, we imagine that at any point in time , there is some probability that any of the possible set of input patterns is impinging on the input units . This probability function may in general depend on the history of inputs to the system as well as outputs of the system. In practice, most PDP models involve a much simpler characterization of the environ ment . Typically , the environment is characterized by a stable probability distribution over the set of possible input patterns independent of past inputs and past responses of the system. In this case, we can imagine listing the set of possible inputs to the system and numbering them from 1 to M . The environment is then characterized by a set of probabilities , Pi for i = 1, . . . , M . Since each input pattern can be considered a vector, it is sometimes useful to characterize those patterns with nonzero probabilities as constituting orthogonal or linearly independent sets of vectors. 2 Certain PDP models are restricted in the kinds of patterns they are able to learn: some being able to learn to respond correctly only if the input vectors form an orthogonal set; others if they form a linearly independent set of vectors; and still others are able to learn to respond to essentially arbitrary patterns of inputs .
CLASSES OFPDPMODELS There are many paradigms and classesof POP models that have been developed. In this section we describe some general classesof assumptions and paradigms. In the following section we describe some specific PDP models and show their relationships to the general framework outlined here.
Paradigmsof Learning Although most learning rules have roughly the form indicated above, we can categorize the learning situation into two distinct sorts. These are: .
Associative learning, in which we learn to produce a particular pattern of activation on one set of units whenever another particular pattern occurs on another set of units . In general, such a learning scheme must allow an arbitrary pattern on one set of
2 See Chapter 9 for explication
of these terms .
2. AFRAMEWORK FOR POP 55 units to produce another arbitrary pattern on another set of units. .
Regularity discovery, in which units learn to respond to " interesting " patterns in their input . In general, such a scheme should be able to form the basis for the development of feature detectors and therefore the basis for knowledge representation in a PD P system.
In certain cases these two modes of learning blend into one another, but it is valuable to see the different goals of the two kinds of learning. Associative learning is employed whenever we are concerned with storing patterns so that they can be re-evoked in the future . These rules are primarily concerned with storing the relationships among subpatterns. Regularity detectors are concerned with the meaning of a single units response. These kinds of rules are used when feature discoveryis the essential task at hand. The associative learning case generally can be broken down into two subcases- pattern association and auto-association. A pattern association paradigm is one in which the goal is to build up an association between patterns defined over one subset of the units and other patterns defined over a second subset of units . The goal is to find a set of connections so that whenever a particular pattern reappears on the first set of units , the associated pattern will appear on the second set. In this case, there is usually a teaching input to the second set of units during training indi cating the desired pattern association. An auto-association paradigm is one in which an input pattern is associated with itself . The goal here is pattern completion . Whenever a portion of the input pattern is presented, the remainder of the pattern is to be filled in or completed. This is similar to simple pattern association, except that the input pattern plays both the role of the teaching input and of the pattern to be associated. It can be seen that simple pattern association is a special case of auto-association. Figure 3 illustrates the two kinds of 'learning paradigms. Figure 3A shows the basic structure of the pattern association situation . There are two distinct groups of units - a set of input units and a set of output units . Each input unit connects with each output unit and each output unit receives an input from each input unit . During training , patterns are presented to both the input and output units . The weights connecting the input to the output units are modi fied during this period. During a test, patterns are presented to the input units and the response on the output units is measured. Figure 3B shows the connectivity matrix for the pattern associator. The only modifiable connections are from the input units to the output units . All other connections are fixed at zero. Figure 3C shows the ha.'\ic
2. AFRAMEWORK FOR PDP 57 potentially a modifiable connection from every unit to every other unit . In the case of pattern association, however, the units are broken into two subpatterns, one representing the input pattern and another representing the teaching input . The only modifiable connections are those from the input units to the output units receiving the teaching input . In other cases of associative learning the teaching input may be more or less indirect . The problem of dealing with indirect feedback is difficult , but central to the development of more sophisticated models of learning. Barto and Sutton ( 1981) have begun a nice analysis of such learning situations. In the case of regularity detectors, a teaching input is not explicitly provided~ instead, the teaching function is determined by the unit itself . The form of the internal teaching function and the nature of its input patterns determine what features the unit will learn to respond to. This is sometimes called unsupervised learning. Each different kind of unsupervised learning procedure has its own evaluation function . The particular evaluation procedures are mentioned when we treat these models. The three unsupervised learning models discussed in this book are addressedin Chapters 5, 6, and 7.
HierarchicalOrganizationsof PDP Networks It has become commonplace in cognitive science to describe such processes as top-down, bottom-up, and interactive to consist of many stages of processing, etc. It is useful to see how these concepts can be represented in terms of the patterns of connectivity in the PDP framework . It is also useful to get some feeling for the processing consequences of these various assumptions.
Bottom-Up Processing The fundamental characteristic of a bottom -up system is that units at level i may not affect the activity of units at levels lower than i . To see how this maps onto the current formulation , it is useful to partition the coalitions of units into a set of discrete categories corresponding to the levels their inputs come from . There are assumed to be no coalitions with inputs from more than one level . Assume that there are Lj units at level i in the system. We then order the units such that those in level L 1 are numbered UI, . . . , ULI' those in level L 2 are numbered UL1+ 1, . . . , ULI+L2' etc. Then , the constraint that the system be a pure
58 THE POP PERSPEcrIVE bottom -up system is equivalent to the constraint that the connectivity matrix , W , has zero entries for Wij in which Uj is the member of a level no higher than U; . This amounts to the requirement that the upper right -hand region of W contains zero entries. Table 1 shows this constraint graphically. The table shows an example of a three-level system with four units at each level.3 This leads to a 12x 12 connectivity matrix and an a vector of length 12. The matrix can be divided up into 9 regions. The upper-left region represents interactions among Levell units . The entries in the left -middle region of the matrix represents the effects of Level 1 units on Level 2 units . The lower-left region represents the effects of Levell uni ts on Level 3 uni ts. Often bottom -up models do not allow units at level i effect units at level i + 2. Thus, in the diagram we have left that region empty representing no effect of Level 1 on Level 3. It is typical in a bottom -up system to assume as well that the lowest level units (Levell ) are input units and that the highest level units (Level 3) are output units . That is, the lowest level of the system is the only one to receive direct inputs from outside of this module and only the highest level units affect other units outside of this module .
TABLEt Levell Input Units ul u2 u3 u4
Levell
Units level 2 Units
Level 3 Units
ul u2 u3 u4
within Levell effects
uS u6 u7 u8
Levell affecting Level 2
Level 2 Hidden Units uS u6 u7 u8
within level 2 errects
u9
Leve .l 2
ul0
affecting
u 11
Level 3
Output Units u9 ulO ull u12
Level
3
within
Level 3 effects
u12
3 In general , of course , we would expect many levels and many units at each level .
2. A FRAMEWORK FORPOP
59
Top-Down Processing The generalization to a hierarchical top-down system should be clear enough. Let us order the units into levels just as before. A top-down model then requires that the lower-left regions of the weight matrix be empty - that is, no lower level unit affects a higher level unit . Table 2 illustrates a simple example of a top-down processing system. Note, in this case, we have to assume a top-down input or " message" that is propagated down the system from higher to lower levels as well as any data input that might be coming directly into Levell units .
Interactive Models Interactive models are simply models in which there can be both top-down and bottom -up connections. Again the generalization is straightforward . In the general interactive model, any of the cells of the weight matrix could be nonzero. The more restricted models in which information flows both ways, but in which information only flows between adjacent levels, assume only that the regions of the matrix more than one region away from the main diagonal are zero. Table 3 illustrates a simple three-level interactive model with both topdown and bottom -up input . Most of the models that actually have been suggestedcount as interactive models in this sense.
TABLE 2 Levell Input Units ul u2 u3 u4 ul
within
Levell
u2
Levell
Units
u3
effects
Level 2 Hidden Units uS u6 u7 u8 Level
Level 3 Output Units u9 ulO ull u12
2
affecting Levell
u4 uS
Level 2 Units
within
u6
Level 2
u7
effects
Level
3
affecting Level
2
u8 u9
Level 3 Units
within
ulO
Level
uII
effects
u12
3
60 THE POP PERSPECTIVE TABLE3
ul
Levell
Uni1s
u2 u3
Levell Input Units ul u2 u3 u4
Level 2 Hidden Units u5 u6 u7 u8
within level 1 effects
Level 2 affecting Levell
Levell affecting Level 2
within Level 2 effects
Level 3 Output Units u9 uIO uII u12
u4 u5
Level 2 Units
u6 u7
Level
3
affecting Level
2
u8
u9 ulO Units ull u12
Level 3
Level 2 affecting Level 3
within level 3 errects
It is sometimes supposed that a " single level " system with no hierarchical structure in which any unit can communicate with any other unit is somehow less powerful than these multilevel hierarchical systems. The present analysis shows that , on the contrary, the existenceof levelsamounts to a restriction, in general, of free communication among all units . Such nonhierarchical systems actually form a superset of the kinds of layered systems discussed above. There is, however, something ~to the view that having multiple levels can increase the power of certain systems. In particular, a "one-step" system consisting of only input and output units and no communication between them in which there is no opportunity for feedback or for hidden units is less powerful than systems with hidden units and with feedback. Since, in general, hierarchical systems involve many hidden units , some intralevel communication , and some feedback among levels, they are more powerful than systems not involving such hidden units . However, a system with an equal number of hidden units , but one not characterizable as hierarchical by the communication patterns is, in general, of more potential computational power. We address the issue of hidden units and "single-step" versus " multiple -step" systems in our discussion of specific models below.
2. AFRAMEWORK FOR POP 61 Synchronous Versus Asynchronous Update Even given all of the components of the POP models we have described so far , there is still another important issue to be resolved in the development of specific models; that is the timing of the application of the activation rule . In some models, there is a kind of central timing pulse and after each such clock tick a new value is determined simul taneously for all units . This is a synchronousupdate procedure. It is usually viewed as a discrete, difference approximation to an underlying continuous , differential equation in which all units are continuously updated. In some models, however, units are updated asynchronously and at random. The usual assumption is that at each point in time each unit has a fixed probability of evaluating and applying its activation rule and updating its activation value. This later method has certain theoretical advantages and was developed by Hopfield (1982) and has been employed in Chapters 6, 7, and 14. The major advantage is that since the units are independently being updated, if we look at a short enough time interval , only one unit is updating at a time . Among other things, this system can help the stability of the network by keeping it out of oscillations that are more readily entered into with synchronous update procedures.
SPECIFICVERSIONSOF THE GENERALPARALLEL ACTIVATION MODEL
In the following sections we will show how specification of the particular functions involved produces various kinds of these models. There have been many authors who have contributed to the field and whose work might as well have been discussed. We discuss only a representative sample of this work .
Simple Linear Models Perhapsthe simplest model of this classis the simple linear model. In the simple linear model, activation valuesare real numberswithout restriction. They can be either positive or negative and are not bounded. The output function, f (at) , in the linear model is just equal to the activationlevel at. Typically, linear modelsconsistof two setsof units: a set of input units and a set of output units. (As discussed
62 THE POP PERSPECTIVE below , there is no need for hidden units since all computation possible with a multiple -step linear system can be done with a single -step linear system .) In general , any unit in the input layer may connect to any unit in the output layer . All connections in a linear model are of the same type . Thus , only a single connectivity matrix is required . The matrix consists of a set of positive , negative , and zero values , for excitatory values , inhibitory values , and zero connections , respectively . The new value of activation of each unit is simply given by the weighted sums of the inputs . For the simple linear model with connectivity matrix W we have
a (t + l ) = Wa (t ) . In general , it can be number of limitations . be computed from two single step . This follows
shown that a linear model such as this has a In particular , it can be shown that nothing can or more steps that cannot be computed by a because the above equation implies
a (t ) = W ' a (0) . We can see this by proceeding step by step . Clearly , a (2) = Wa ( 1) = W ( Wa (0
= W 28 (0) .
It should be clear that similar arguments lead to a (t ) = W ' a (0) . From this , it follows that for every linear model with connectivity matrix W that can attain a particular state in t steps , there is another linear model with connectivity matrix W t that can reach the same state in one step . This means , among other things , that there can never be any computational advantage in a linear model of multiple -step systems , nor can there ever be any advantage for allowing feedback . The pattern association paradigm is the typical learning situation for a linear model . There is a set of input units and a set of output units . In general , each input unit may be connected to any output unit . Since this
is a linear
network
, there
is no feedback
in the
system
nor
are
there hidden units between the inputs and outputs . There are two sources of input in the system . There are the input patterns that estab lish a pattern of activation on the input units , and there are the teach ing units that establish a pattern of activation on the output units . Any of several learning rules could be employed with a linear network such as this , but the most common are the simple Hebbian rule and the d~lta rule . The linear model with the simple Hebbian rule is called the simple linear associator (cf . Anderson , 1970; Kohonen , 1977, 1984) . In this case, the increment
in weight
Wij is given by 4 wij = T) aj ti '
In
matrix notation , this means that .6W ~ TITaT . The system is then tested by presenting
an input
pattern
without
a teaching input
and
2. A FRAMEWORK FORPOP
63
seeing how close the pattern generated on the output layer matches the original teaching input . It can be shown that if the input patterns are orthogonal ,4 there will be no interference and the system will perfectly produce the relevant associated patterns exactly on the output layer. If they are not orthogonal , however, there will be interference among the input patterns. It is possible to make a modification in the learning rule and allow a much larger set of possible associations. In particular , it is possible to build up correct associations among patterns whenever the set of input patterns are linearly independent. To achieve this , an error correcting rule must be employed. The delta rule is most commonly employed. In this case, the rule becomes .6W;j = T) (t;- a; )aj . What is learned is essentially the difference between the desired response and that actually attained at unit u; due to the input . Although it may take many presentations of the input pattern set, if the patterns are linearly independent the system will eventually be able to produce the desired outputs. Kohonen ( 1977, 1984) has provided an important analysis of this and related learning rules. The examples described above were for the case of the pattern associator. Essentially the same results hold for the auto-associator version of the linear model. In this case, the input patterns and the teaching patterns are the same, and the input layer and the output layer are also the same. The tests of the system involve presenting a portion of the input pattern and having the system attempt to reconstruct the missing parts.
Linear Threshold Units The weaknessesof purely linear systemscan be overcomethrough the addition of nonlinearities. Perhapsthe simplest of the nonlinear system consists of a network of linear threshold units. The linear threshold unit is a binary unit whose activation takes on the values {O,l } . The activation value of unit Uj is 1 if the weighted sum of its inputs is greaterthan some threshold(Jj and is 0 otherwise. The connectivity matrix for a network of such units, as in the linear system, is a matrix consistingof positive and negativenumbers. The output function, /, is the identity function so that the output of a unit is equal to its activationvalue.
4 See Chapter 9 for a discussion of orthogonality , linear independence , etc.
64
THEPOP PERSPECTIVE
It is useful to see some of the kinds of functions that can be computed with linear threshold units that cannot be computed with simple linear models. The classic such function is the exclusiveor (XOR ) illus trated in Figure 4. The idea is to have a system which responds { I } if it receives a {O,l } or a { I ,O} and responds to} otherwise. The figure shows a network capable of this pattern. In this case we require two
x 0 R Netwo rk
Output
Unit
+1
+ 1
Internal Units Thresholds= .01
+1
Input Units
- 1
+1
Input Output 00 11} 0 01 10J
1
FIGURE4. A networkof linearthresholdunitscapable of responding correctlyon the XORproblem .
2. A FRAMEWORK FORPOP
65
layers of units . Each unit has a zero threshold and respondsjust in case its input is greater than zero. The weights are :t: 1. Since the set of stimulus patterns is not linearly independent, this is a discrimination that can never be made by a simple linear model and cannot be done in a single step by any network of linear threshold units . Although multilayered systems of linear threshold units are very powerful and, in fact, are capable of computing any boolean function , there is no generally known learning algorithm for this general case (see Chapter 8) . There is, however, a well-understood learning algorithm for the special case of the perceptron. A perceptron is essentially a single-layer network of linear threshold units without feedback. The learning situation here is exactly the same as that for the linear model. An input pattern is presented along with a teaching input . The perceptroD learning rule is precisely of the same form as the delta rule for error correcting in the linear model, namely, 4. Wjj = 7) (tj- aj )aj . Since the teaching input and the activation values are only 0 or 1, the rule reduces to the statements that: 1. Weights are only changed on a given input line when that line is turned on (i.e., Qj = 1) . 2. If the system is correct on unit i (i .e., t; = a;) , make no change
on any of the input weights. 3. If the unit j responds0 when it should be 1, increaseweights on all active lines by amount11. 4. If the unit j responds1 when it should be 0, decreaseweights on all active lines by amount1} . There is a theorem , the perceptron convergence theorem , that guarantees that if the set of patterns are learnable by a perceptron, this learning procedure will find a set of weights which allow it to respond correctly to all input patterns. Unfortunately , even though multilayer linear threshold networks are potentially much more powerful than the linear associator, the perceptron for which a learning result exists can learn no patterns not learnable by the linear associator. It was the limi tations on what perceptrons could possibly learn that led to Minsky and Papert's (1969) pessimistic evaluation of the perceptron. Unfortunately that evaluation has incorrectly tainted more interesting and powerful networks of linear threshold and other nonlinear units . We have now developed a version of the delta rule - the generalized delta rule - which is capable of learning arbitrary mappings. It does not work for linear threshold units , but does work for the class of semilinear activation
66
THEPOPPERSPECTIVE
functions full
(i .e., differentiable
discussion
. As we shall
activation functions ) . See Chapter 8 for a see in the course
of this book , the limita -
tions of the one -step perceptron in no way apply to the more complex networks
.
Brain State in a Box The brain state in a box model was developed by J. A . Anderson ( 1977) . This model too is a close relative of the simple linear associator .
There
is , however
,
a maximum
and
minimum
activation
value
associated with each unit . Typically , units take on activation values in the interval [- 1, 1] . The brain state in a box (BSB) models are organ ized so that any unit can, in general , be connected to any other unit .
The auto-associator illustrated in Figure 3 is the typical learning paradigm for BSB. Note that with this pattern of interconnections the system feeds back on itself and thus the activation can recycle through the system in a positive feedback loop . The positive feedback is especially evident in J. A . Anderson and Mozer ' s ( 1981) version . Their activation rule is gi ven by
aj (t+ l ) = aj (t )+ Ew ;jaj (t ) if aj is less than 1 and greater than - 1. Otherwise , if the quantity greater
than
1, aj = 1 and if it is less than - 1, aj = - 1. That
is
is , the
activation state at time t + 1 is given by the sum of the state at time t and the activation propagated through the connectivity matrix provided that total is in the interval [- 1, 1] . Otherwise it simply takes on the maximum or minimum value . This formulation will lead the system to a state in which all of the units are at either a maximum or minimum
value . It is possible to understand why this is called a brain state in a box model by considering a geometric representation of the system . Figure ' 5 illustrates the " activation space" of a simple BSB system con sisting of three units . Each point in the box corresponds to a particular value
of activation
on each of the three
units .
In this case we have a
three -dimensional space in which the first coordinate corresponds to the activation value of the first unit , the second coordinate corresponds to the
activation
value
of
the
second
unit , and
the
third
coordinate
corresponds to the activation value of the third unit . Thus , each point in the space corresponds to a possible state of the system . The feature that each unit is limited to the region [- 1, 1] means that all points must lie somewhere within the box whose vertices are given by the points (- 1,- 1,- 1) , (- 1,- 1,+ 1) , (- 1,+ 1,- 1) , (- 1,+ 1,+ 1) , (+ 1,- 1,- 1) , (+ 1,- 1,+ 1) , (+ 1,+ 1,- 1) , and (+ 1,+ 1,+ 1) . Moreover , since the
2. AFRAMEWORK FOR POP 67
( - ,+ ,+ )
( + ,+ ,+ )
11 I I ( - ,- ,
1(- .+,- ) ~
-
(+,+,- )
.~t'J /
Activation of Unit 2
~
~~ /
0'
.~-(:t / .~ ~ ~
I
~ / / ( - ,- ,- ) ,
,
~
Activation of Unit 1 FIGURE 5. The statespacefor a three-unit versionof a BSBmodel. Eachdimensionof the box representsthe activationvalue of one unit. Eachunit is boundedin activation between[- 1,1] . The curving arrow in the box representsthe sequenceof statesthe system moved through. It beganat the blackspot near the middle of the box and, as processingproceeded , movedto the (- ,+ ,+ ) corner of the box. BSBsystemsalwaysend up in one or anotherof the corners. The particularcorner dependson the start stateof the network, the input to the system, and the patternof connectionsamongthe units.
systeminvolves positive feedback, it is eventuallyforced to occupyone of these vertices. Thus, the state of the system is constrainedto lie within the box and eventually, as processingcontinues, is pushedto one of the vertices. Of course, the samegeometricanalogycarriesover to higher dimensionalsystems. If there are N units, the state of the systemcan be characterizedas a point within this N-dimensionalhypercubeand eventuallythe systemendsup in one of the 2N comersof the hypercube.
68 THEPOPPERSPECfIVE Learning in the BSB system involves auto-association. In different applications two different learning rules have been applied. J. A . Anderson and Mozer (1981) applied the simplest rule . They simply allowed the system to settle down and then employed the simple Hebbian learning rule . That is, 4 W;j = 11a;aj ' The error correction rule has also been applied to the BSB model. In this case we use the input as the teaching input as well as the source of activation to the system. The learning rule thus becomes 4 W;j = "1(I;- a; )aj where t; is the input to .unit i and where Q; and Qj are the activation values of the system after it has stabilized in one of the corners of the hypercube.
Thermod lynamic Models Other more recent developments are the thermodynamic models. Two examples of such models are presented in the book. One, harmony theory, was developed by Paul Smolensky and is described in detail in Chapter 6. The other , the Boltzmann machine, was developed by Hinton and Sejnowski and is described in Chapter 7. Here we describe the basic ,idea behind these models and show how they relate to the general class of models under discussion. To begin, the thermo dynamic models employ binary units which take on the values {O, I } . The units are divided into two categories: the visible units corresponding to our input and output units and the hidden units . In general, any unit may connect to any other unit . However, there is a constraint that the connections must be symmetric . That is, the Wlj = Wji . In these models, there is no distinction between the output of the unit and its activation value. The activation values are, however, a stochastic func tion of the inputs. That is,
1 p(0;(/)=I)=-I+e-(Ij,W ;jOj +"l;-8;)/T
where 11; is the input from outside of system into unit i , 9; is the threshold for the unit , and T is a parameter, called temperature, which determines the slope of the probability function . Figure 6 shows how the probabilities vary with various values of T . It should be noted that as T approaches zero, the individual units become more and more like linear threshold units . In general, if the unit exceeds threshold by a great enough margin it will always attain value 1. If it is far enough below threshold , it always takes on value O. Whenever the unit is above threshold, the probability that it will turn on is greater than 1/ 2.
2. A FRAMEWORK FORPOP
Temperature
=
I
0.0 I I 1__.J--t ----I ~ - 25 - 20 - 15 - 10
- 5
Net
0 5 Input
10
15
20
FIGURE 6. Probabilityof attainingvalue 1 as a function of the distanceof the input of the unit from threshold. The function is plotted for severalvaluesof T.
~. 0 ~. 0
lilll -
. -
m. 0 0. ...
BUWL;1! WJ: 1
N. 0
II
Whenever it is below threshold , the probability that it will turn off is greater than 1/ 2. The temperature simply determines the range of uncertainty as to whether it will turn on or off . This particular configuration of assumptions allows a formal analogy between these models and thermodynamics and allows the proof of theorems concerning its performance as a function of the temperature of the system. This is not the place to discuss these theorems in detail, suffice it to say that this system, like the BSB system, can be viewed as attaining states on the corners of a hypercube. There is a global measure of the degree to which each state of the system is consistent with its input . The system moves into those states that are maximally consistent with the input and with the internal constraints represented by the weights. It can be shown that as the temperature approaches0, the probability that the system attains the maximally consistent state approaches 1. These results are discussed in some detail in Chapters 6 and 7. There is a learning scheme associated with the Boltzmann machine which is somewhat more complex than the others. In this case, the learning events are divided into two phases. During one phase, a set of patterns is randomly presented to the visible units and the system is allowed to respond to each in turn . During this phase of learning, the system is environmentally driven ; a simple Hebbian rule is assumed to
70 THE POP PERSPECTIVE apply
so
and
1
ever
that
d
this unit
i
second
of
and of
.
even
anti
is
the
no
actual
the
.
mined
only
ment
stimuli
we
It
on
after
the
of
the
visible
present turned
on
learned
the
with
. These
are
.
,
of
free
- running
connections its
actually
a
portion
it
will
Quite of
complete
probability the
a
powerful units
due
learning are
turned portion
units that
Chapter
the
environ
remaining
those
in
the
input
subpattern
addressed
the
deter
performance
the
that
given
again
the
phase
reflect
to
its
is
is
correctly
out
sim
by
phase
due
subtract
is
,
To
performance
then
if
.
a
intuition
and
the
are
respond
phase
The
driven
patterns
issues
this
..,., ajaj
interconnections
the
stimulus
to
environmentally
This
has
continue
During -
equal
inputs
of
at
.
=
the
an
no
0 -
During for
which
will
.
Wjj
.
the
and
that
units
~
when
during
look
shown
system
in
presented
of
1)
respond
in
it
values
pattern
set
alone be
to
,
on amount
occurs
state
stochastic
,
an
change
allowed
during
structure
can
is
are
the
take by
- running
employed
should
structure .
system
is
is
activations
no
free
internal
internal
internal
scheme
the
since
incremented
otherwise
performance
the
,
plus
,
both
The
by
environment
,
performance by
environment
on
system
rule
that
, is
so - called
Since
determined
to
a
Note
weight
learning
- Hebbian
roughly
the are
in
though
ple
1) Q ; a j .
j
time
presented
=
that
phase
period
W ;j
says
had
been
had
been
7 .
Grossberg
Stephen of
Grossberg
this
class
important be
some
the
' s
allowed
to
maximum cations
, a
only
if
argues the
the
take
on
value
.
activation the
level output
is
than
of the
inhibitory rule , but
that
.
don
Grossberg they
a its
typically
unit
.
threshold
.
have
discussed
' t
simply
sum
have
,
presented the
but
relates of
units
minimum
and
,
appear number
-
unit
Grossberg function
rule in
a
' s appli
S - shaped
far
are
another
activation
a form
's
affect
thus
it
Grossberg
or
's
we
how
Moreover
sigmoid
Grossberg
has
of will
instead
summary
a
many
many
will
Grossberg
unit
a
show
clearest ) .
models
contains We
between in
given
be
.
and
the
is ,
must
others
work
value
above
the
inputs
his
to
and here
( 1980
function
function
value
of
activation
so
contributors
complex
Perhaps
Grossberg
output
function
its
rules
.
major is
review
aspects
real
the
work
cannot
in
any The
threshold
activation
activation
central
of
His
we
appears
that
and
one
.
framework
work
complex
tory the
of
activation
more
been
years
which
general
Grossberg
of
the
details
descri to
has
over
is
that
rather excita
separately of
possible
in
2. AFRAMEWORK FOR POP 71
OJ
where
( / +
A
tion
is
of
of
another Grossberg and
Chapter
and
. t1W
.. '1
Grossberg cases
of
but
. ( O '1
in
that
each
unit
unit
in
,
unit
like
the ( by
yield
a j
.
" net
This
learning is
unit
itself
a .
the
and
number
The
one
rules
of
learning
analyzed
in
beyond
a
the
.
in
number
of
scope
Thus
BSB the input net
,
,
of
the
the
.
"
to
the is
the
the
layers
layer
that
and
Let combined
netj
=
and I ,
with
all
the
is values
into
;
be previous
in
the
addition with a
, the
kind for
each
summed wija
In
-
that
units .
model
coming )
are exci
below
inconsistent
minimum
information
.
and to
are
units with
unit
activation
strengths unit
which
above
and
interactive
connects
that
below
value The
in layer
with
maximum The
was
.
any
threshold
activation
negatively
interactive
then
a
on
a
activation the
which
take
was
pattern in
connects
own
units
could
threshold
inconsistent
interconnection
input
if
Rumelhart
had
Units
of
in
its
with
model
0
element
and
and )
function
connectivity
are in
system
output
above
elements
units
.
difference
an
unit
words
was
that all
McClelland ( 1982
The
was
that
all
of
and
the
a
such
below
feedback ,
the
unit
schemes to
similar
output
that
and inhibits
weighted
] .
to
with
question
positive
the
the
inhibitory
which
over
similar
applications
activation
with
above
, max
involves
layers
consistent
layers
letters
the
connections
are
the in
networks
McClelland
,
equal the
model in
tatory
below
that
learning is
model
[ min
if
organized
and
these
features
was
threshold
activation
these
,
and
range
and
the
-
Band
inhibited
from
( I )
excita
than
field
come
in
activation
such
threshold
be
assumes
different
Rumelhart
the
function
of
;j
Model
visual
value
degree
) nel
.
and
represented
can
C
.
however
this of
interactive )
( / ) +
magnitude
inhibitory inputs
( OJ
W .. ) '1 .
Activation
The
-
maximal in
unit
recurrent
,
( I )
by
review
Interactive
( 1981
the
the
several
discussion
nelej
smaller
learning
applied
a
( /
represents
system
most
' nQ ."
oj
generally
the
ven
-
much
of
studied
gi
( B
excitatory
studied
has ,
present
unit
level
=
B is
kind
the
is
+
Grossberg
a
studied
5
,
)
amount
and
has has
A
C
O .
has
he
( I -
rate and
from
embedded
years
( I )
maximal
come
from
to
,
value
inputs
OJ
decay
unit the
resting
rule
=
the
the
represents
is
I )
of each
unit
is
algebraically the
net
input activation
to
72 THE POP PERSPECfIVE value
to
produce
activation
aj
where
@
words
,
rule
:
(/ +
1)
is
or
input
into ,
specific
aspects
such
and
is
formulation
input
0
.
In
other
activation
pushes
the
value toward
magnitude
similar
the
model goals
word
was
were
to
of
that
excitation
the the
net
employed
and
.
book
.
we
has
been
might
how ,
by
inhibition
,
aimed
have
are
for
of
no
work
( See
-
for
specific
model we
on
plausible
.
pro
network
the
giving
learned
a
account
particular
much at
been
model could
was
the
see
a
we
there
where
shall
as
show Thus
explain
As
designed
to
perception to
network
and
learning
accounts
especially
of
Chapters
Feldman
Ballard
and
connectionist which
they
call
10 , 10 ] .
9 ) .
a
somewhat
what
they
(t +
the
Feldman units
1)
to and
,
available
discussed
aj
.
.
In
+
f : Jnetj
reaches
Ballard
also
below
within rules
(t )
value
the
our
take
on
kind
of
a
number a
.
this
current examined
of .
the
activation
other
Their
call
values in
the
,
range
function
which
integer
values
discrete
rule
maximum
a
or
is
unit
types
simplest rule
is
( 1981 learning
each
unit
type
given
by
value
)
has
. the
is
In
to
our
class
, we
sim
.
sigma
- pi
learning
machinery
practice
is
inhibition
considered
more
same
it
self
similar also
offers
of
by
unit
framework are
minimum
implemented
cofljunctive
Feldman to
value
threshold
of
they
activation
(t ) .
Decay
have
framework
any
number
case
its .
approach
actually
is
small
a
continuous
activation
P - unit
=
can
proposed
that
described general
learning
have
activation
pinned
a
proposed have
function
different the
have
units which
on
They
call
aj
ply
take
)
The
output
to
Oi ~
Once
( 1982 .
potential
Their
allowed
(O~
Ballard
modeling
with
In
rule
no old
that
on
following
otherwise nelj >
the
factor
the
6 .)
Feldman
[-
this
by
depending
activation
our
from this
a
a
min aj ( I
given
given )
to
.
of
in
how
activation is
minus
activation
came
reported
is
in
according
net :
I
the
( or
proposed
assumed
of
This
and
learning
+
value .
interactive system
)
value
value
combined
The
8
rate
plus
except
cessing
5
,
unit
algebraically
of
( / ) ( 1-
maximum
the
Grossberg
activation
activation
decayed
minimum
aj
decay
new
properly
new
=
the
the
the
.
than
however have
, already
is the
2. AFRAMEWORK FOR POP 73
SIGMA -PI UNITS Before completing our section on a general framework , it should be mentioned that we have sometimes found it useful to postulate units that are more complex than those described up to this point in this chapter. In our descriptions thus far, we have assumed a simple additive unit in which the net input to the unit is given by I . W;j a; . This is certainly the most common form in most of our models. Sometimes, however, we want multiplicative connections in which the output values of two (or possibly more) units are multiplied before entering into the sum. Such a multiplicative connection allows one unit to gate another. Thus, if one unit of a multiplicative pair is zero, the other member of the pair can have no effect , no matter how strong its output . On the other hand, if one unit of a pair has value 1, the output of the other is passed unchanged to the receiving unit . Figure 7 illustrates several such connections. In this case, the input to unit A is the weighted sum of the products of units Band C and units D and E. The pairs, BC and DE are called conjuncts. In this case we have conjuncts of size 2. In general, of course, the conjuncts could be of any size. We have no applications, however, which have required conjuncts larger than size 2. In general, then , we assume that the net input to a unit is given by the weighted sum of the products of a set of individual inputs. That is, the net input to a unit is given by I . w;;II ~ a; 1a;2 . . . a;k where i indexes the conjuncts impinging on unit j and Uj1, Uj2 , . . . , Ujk are the k units in the conjunct. We call units such as these sigma-pi units. In addition to their use as gates, sigma-pi units can be used to convert the output level of a unit into a signal that acts like a weight connecting two units . Thus, assume we have the pattern of connections illustrated in the figure . Assume further that the weights on those connections are alII . In this case, we can use the output levels of units B and D to , in effect , set the weights from C to A and E to A respectively . Since, in general, it is the weights among the units that determine the behavior of the network , sigma-pi units allow for a dynamically programmable network in which the activation value of some units determine what another network can do. In addition to its general usefulness in these cases, one might ask whether we might not sometime need still more complex patterns of interconnections. Interestingly , as described in Chapter 10, we will never be forced to develop any more complex interconnection type, since sigma-pi units are sufficient to mimic any function monotonic of its inputs.
74
THEPOPPERSPECTIVE
SigmaPi Units
A
net A = IWiA11 ' Coni i
Con 1 - as . aC
Con2= aD ' aE
FIGURE 7. Two conjunctiveinputs to unit A from the conjunct8 and C and D and E. The input to unit A is the sum of the productof the outputsof units DC and DE.
CONCLUSION We have provided a very general mathematical and conceptual framework within which we develop our models. This framework provides a language for expressing PDP models, and, though there is a lot of freedom within it , it is at least as constrained as most computational formalisms , such as production systems or high-level languagessuch as Lisp. We must take note of the fact, however, that the framework does not specify all of the constraints we have imposed on ourselves in our model building efforts . For example, virtually any computing device, serial or parallel, can be described in the framework we have described here.
2. A FRAMEWORK FORPOP
75
There is a further set of considerations which has guided our particular formulations . These further considerations arise from two sources: our beliefs about the nature of the hardware available for carrying out mental processesin the brain and our beliefs about the essential character of these mental processesthemselves. We discuss below the additional constraints on our model building which arise from these two beliefs. . First , the operations in our models can be characterized as "neurally inspired." We wish to replace the "computer metaphor" as a model of mind with the " brain metaphor" as model of mind . This leads us to a number of considerations which further inform and constrain our model building efforts . Perhaps the most crucial of these is time . Neurons are remarkably slow relative to components in modern computers. Neurons operate in the time scale of milliseconds whereas computer components operate in the time scale of nanoseconds- a factor of 106 faster. This means that human processes that take on the order of a second or less can involve only a hundred or so time steps. Since most of the processeswe have studied- perception, memory retrieval , speech processing, sentence comprehension, and the like - take about a second or so, it makes sense to impose what Feldman ( 1985) calls the " 100step program" constraint. That is, we seek explanations for these mental phenomena which do not require more than about a hundred elementary sequential operations. Given that the processes we seek to characterize are often quite complex and may involve consideration of large numbers of simultaneous constraints, our algorithms must involve considerable parallelism. Thus, although a serial computer could be created out of the kinds of components represented by our units , such an implementation would surely violate the tOO-step program constraint for any but the simplest processes. A second co~sideration differentiates our models from those inspired by the computer metaphor: that is, the constraint that all the knowledge is in the connections . From conventional programmable computers we are used to thinking of knowledge as being stored in the state of certain units in the system. In our systems we assume that only very short term storage can occur in the states of units ; long term storage takes place in the connections among units . Indeed, it is the connections- or perhaps the rules for forming them through experience- which primarily differentiate one model from another. This is a profound difference between our approach and other more conventional approaches, for it means that almost all knowledge is implicit in the structure of the device that carries out the task rather than explicit in the states of units themselves. Knowledge is not directly accessible to interpretation by some separate processor, but it is built into the processor itself and directly determines the course of
76 THE pOP PERSPECTIVE processing . It is acquiredthrough tuning of connectionsas these are used in processing , rather than formulated and stored as declarative facts. In addition to thesetwo neurally inspired working assumptions , there are a number of other constraintsthat derive rather directly from our understandingof the nature of neural information processing . These assumptionsare discussedmore fully in Chapter4. The second class of constraintsarises from our beliefs about the nature of human information processingconsideredat a more abstract, computationallevel of analysis. We see the kinds of phenomenawe have been studying as productsof a kind of constraintsatisfactionprocedure in which a very large number of constraintsact simultaneously to producethe behavior. Thus, we see most behavior not as the product of a single, separatecomponentof the cognitivesystem, but as the product of largeset of interactingcomponents, eachmutually constraining the others and contributing in its own way to the globally observable behaviorof the system. It is very difficult to use serial algorithms to implement such a conception, but very natural to use highly parallel ones. These problems can often be characterizedas best match or optimizationproblems. As Minsky and Papert (1969) have pointed out, it is very difficult to solve best match problemsserially. However, this is precisely the kind of problem that is readily implemented using highly parallel algorithms of the kind we consider in this book. See Kanerva (1984) for a discussionof the best match problem and its solution with parallelprocessingsystems. To summarize, the POP framework consists not only of a formal language , but a perspectiveon our models. Other Qualitativeand Quantitative considerationsarising from our understandingof brain process ing and of human behavior combine with the formal system to form what might be viewed as an aestheticfor our model building enterprises. The remainderof our book is largelya study of this aestheticin practice.
ACKNOWLEDGMENTS This research was supported by Contract NOOO14- 79-C-O323, NR 667 -437 with the Personnel and Training Research Programs of the Office of Naval Research , by grants from the System Development Foundation , and by a NIMH Career Development Award (MHOO385 ) to the second
author .
CHAPTER 3
Distributed
Representations
G. E. HINTON , J. L. McCLELLAND , andD. E. RUMELHART
Given a network of simple computing elements and some entities to be represented, the most straightforward scheme is to use one comput ing element for each entity . This is called a local representation. It is easy to understand and easy to implement because the structure of the physical network mirrors the structure of the knowledge it contains. The naturalness and simplicity of this relationship between the knowledge and the hardware that implements it have led many people to simply assume that local representations are the best way to use parallel hardware. There are, of course, a wide variety of more compli cated implementations in which there is no one-to-one correspondence between concepts and hardware units , but these implementations are only worth considering if they lead to increased efficiency or to interesting emergent properties that cannot be conveniently achieved using local representations. This chapter describes one type of representation that is less familiar and harder to think about than local representations. Each entity is represented by a pattern of activity distributed over many computing elements, and each computing element is involved in representing many different entities. The strength of this more complicated kind of representation does not lie in its notational convenience or its ease of implementation in a conventional computer , but rather in the efficiency with which it makes use of the processing abilities of networks of simple, neuron-like computing elements.
78
THEPOPPERSPECTIVE
Every
representational
uted
scheme
representations
very
are
naturally
Other
from
properties
trarv
the
,
best
, .
their
of
for
bad
points
desirable
as
store
to
achieve
distributed
.
-
arise
representations
a
.
large
As
.
set
we
those
arbi
see
is
match
of
shall
representations
weaknesses
Distrib
properties
activity
temporarily
harder
and
and
of
to
much
good
Some
patterns
ability
strengths
first
section
distributed
of
lems
can
.
A
final
avoided
by
be
-
,
the
of
better
advocates
processing
the
degree
the
of
on
.
distributed
an
human
wrong
to
tional
view
been
useful
in
fruitful
abstract
to
representations
gent
properties
when
in
working
in
,
situation
.
mented
in
sonable
So
using
or
the
.
For
simultaneous
the
.
in
are
,
situation
We
the
schemata
characterizations
shall
chapter
a
large
being
of
mechanisms
14
)
constructs
.
opera
-
degree
it
representa
we
will
see
only
on
-
rules
that
distributed
rely
for
with
fitting
the
There
of
- level
provide
which
primitive
consistent
partially
to
of
( Chapter
- level
higher
are
of
-
automatic
properties
in
applied
-
unrea
as
emergent
number
imple
not
these
captured
properties
schemata
higher
rule
the
current
are
,
implement
representations
rule
is
memory
of
easily
these
other
models
to
-
automatic
the
it
-
distributed
,
fits
appropriate
b
primitives
,
,
way
Some
not
of
each
examine
on
and
easy
distributed
,
abstract
an
no
.
application
current
relevant
is
computers
example
as
best
- addressable
of
there
representations
malisms
content
Distri
emer
example
that
It
more
:
taken
representations
selection
though
rule
have
.
proviso
memory
more
distributed
like
conventional
the
that
abilities
even
distributed
the
assumes
brain
,
,
in
one
of
-
that
unexpected
be
For
be
representa
these
- addressable
selection
would
intelligence
one
.
content
status
It
to
artificial
therefore
of
their
.
and
formalism
for
the
treat
generalization
operations
and
can
abstract
good
if
the
to
tions
more
are
about
implementing
with
of
favor
systems
powerful
the
.
alternative
of
but
some
as
in
production
way
,
to
properties
a
representations
generalization
rise
These
an
-
often
focusing
clear
and
one
networks
give
.
as
parallel
be
prob
such
processing
psychology
them
of
are
object
to
as
cognitive
classes
which
,
information
or
efficiency
arguments
important
of
distributed
sequential
detailed
networks
view
schemes
uted
is
human
semantic
certain
structured
representations
like
found
more
it
the
issues
the
a
the
,
of
distributed
schemes
of
virtues
why
representations
examining
theory
for
and
aspects
Before
overall
ones
structure
different
the
clearly
difficult
distributed
representations
within
local
of
considers
shows
some
constituent
effort
/ aimers
than
some
section
and
discusses
of
Disc
second
,
section
representation
stresses
The
representations
representations
to
chapter
.
distributed
is
this
representations
of
mate
its
.
.
The
that
use
evidence
which
mind
has
exception
the
are
psychological
to
tions
like
associations
~
no
is
-
clearly
approxi
distributed
-
3. DISTRIBUTED REPR ~ ENT ATIONS 79 representations. Thus, the contribution that an analysis of distributed representations can make to these higher-level formalisms is to legiti mize certain powerful , primitive operations which would otherwise appear to be an appeal to magic~ to enrich our repertoire of primitive operations beyond those which can conveniently be captured in many higher-level formalisms ; and to suggest that these higher-level formal isms may only capture the coarse features of the computational capabilities of the underlying processing mechanisms. Another common source of confusion is the idea that distributed representations are somehow in conflict with the extensive evidence for localization of function in the brain (Luria , 1973) . A system that uses distributed representations still requires many different modul(;s for representing conipletely different kinds of thing at the same time . The distributed representations occur within these localized modules. For example, different modules would be devoted to things as different as mental images and sentence structures, but two different mental images would correspond to alternative patterns of activity in the same module . The representations advocated here are local at a global scale but global at a local scale.
VIRTUESOF DISTRIBUTEDREPRESENTATIONS This section considers three important features of distributed representations: (a) their essentially constructive character; (b) their ability to generalize automatically to novel situations; and (c) their tunability to changing environments . Several of these virtues are shared by certain local models, such as the interactive activation model of word perception, or McClelland 's ( 1981) model of generalization and retrieval described in Chapter 1.
Memory as Inference People have a very flexible way of accessing their memories: They can recall items from partial descriptions of their contents (Norman & Bobrow, 1979) . Moreover , they can do this even if some parts of the partial description are wrong. Many people, for example, can rapidly retrieve the item that satisfies the following partial description: It is an actor, it is intelligent , it is a politician . This kind of content-addressable memory is very useful and it is very hard to implement on a conventional computer because computers store each item at a particular
80 THE POP PERSPECTIVE address , and to retrieve an item they must know its address . If all the combinations of descriptors that will be used for access are free of errors and are known in advance , it is possible to use a method called hash coding that Quickly yields the address of an item when given part of its content .- In general , however , content -addressable memory requires a massive search for the item that best fits the partial descrip tion . The central computational problem in memory is how to make this search efficient . When the cues can contain errors , this is very dif ficult because the failure to fit one of the cues cannot be used as a filter for Quickly eliminating inappropriate ans .'ers . Distributed representations provide an efficient way of using parallel hardware to implement best -fit searches . The basic idea is fairly sim ple , though it is Quite unlike a conventional computer memory . Dif ferent items correspond to different patterns of activity over the very same group of hardware units . 1\ partial description is presented in the form of a partial activity pattern , activating some of the hardware units . 1 Interactions between the units then allow the set of active units to influence others of the units , thereby completing the pattern , and generating the item that best fits the description . A new item is "stored " by modifying the interactions between the hardware units so as to create a new stable pattern of activity . The main difference from a conventional computer memory is that patterns which are not active do not exist anywhere . They can be re-created because the connection strengths between units have been changed appropriately , but each con nection strength is involved in storing many patterns , so it is impossible to point to a particular place where the memory for a particular item is stored . Many people are surprised when they understand that the connec tions between a set of simple processing units are capable of supporting a large number of different patterns . Illustrations of this aspect of dis tributed models are provided in a number of papers in the literature (e.g., Anderson , 1977~ Hinton , 1981a) ~ this property is illustrated in the model of memory and amnesia described in Chapters 17 and 25. One way of thinking about distributed memories is in terms of a very large set of plausible inference rules . Each active unit represents a "microfeature " of an item , and the connection strengths stand for plau sible " microinferences " between microfeatures . Any particular pattern 1 This is easy descriptionis simply a set of features, but it is much more . if the partial difficult if the partialdescriptionmentionsrelationshipsto other objects. If, for example, the systemis askedto retrieveJohn's father, it must representJohn, but if John and his father are representedby mutually exclusivepatternsof activity in the very samegroup of units., it is hard to seehow this can be done without preventingthe representationof John"s father. A distributedsolution to this problemis describedin the text.
3. DISTRIBUTED REPRESENT AnONS 81 of activity of the units will satisfy some of the microinferences and violate others. A stable pattern of activity is one that violates the plausible microinferences less than any of the neighboring patterns. A new stable pattern can be created by changing the inference rules so that the new pattern violates them less than its neighbors. This view of memory makes it clear that there is no sharp distinction between genuine memory and plausible reconstruction . A genuine memory is a pattern that is stable becausethe inference rules were modified when it occurred before. A "confabulation " is a pattern that is stable becauseof the way the inference rules have been modified to store several dif ferent previous patterns. So far as the subject is concerned, this may be indistinguishable from the real thing . The blurring of the distinction between veridical recall and confabulation or plausible reconstruction seems to be characteristic of human memory (Bartlett , 1932; Neisser, 1981) . The reconstructive nature of human memory is surprising only because it conflicts with the standard metaphors we use. We tend to think that a memory system should work by storing literal copies of items and then retrieving the stored copy, as in a filing cabinet or a typical computer database. Such systems are not naturally reconstructive. If we view memory as a process that constructs a pattern of activity which represents the most plausible item that is consistent with the given cues, we need some guarantee that it will converge on the representation of the item that best fits the description, though it might be tolerable to sometimes get a good but not optimal fit . It is easy to imagine this happening, but it is harder to make it actually work . One recent approach to this problem is to use statistical mechanics to analyze the behavior of groups of interacting stochastic units . The analysis guarantees that the better an item fits the description, the more likely it is to be produced as the solution . This approach is described in Chapter 7, and a related approach is described in Chapter 6. An alternative approach, using units with continuous activations (Hopfield , 1984) is described in Chapter 14.
Similarity and Generalization When a new item is stored, the modifications in the connection strengths must not wipe out existing items. This can be achieved by modifying a very large number of weights very slightly . If the modifi cations are all in the direction that helps the pattern that is being stored, there will be a conspiracy effect : The total help for the intended pattern will be the sum of all the small separate modifications .
PERSPECTIVE 82 THEPOP
For
unrelated
patterns , however
, there
will
be very
little
transfer
of
effect because some of the modifications will help and some will hinder . Instead of all the small modifications conspiring together , they will mainly cancel out . This kind of statistical reasoning underpins most
distributed
memory
models , but there
are many
variations
of the
basic idea (See Hinton & Anderson , 1981 , for several examples ) . It is possible to prevent interference altogether by using orthogonal patterns of activity for the various items to be stored (a rudimentary example of such a case is given in Chapter 1) . However , this elim inates one of the most interesting properties of distributed representa tions : They automatically give rise to generalizations . If the task is simply to remember accurately a set of unrelated items , the generaliza -
tion effects are harmful and are called interference . But generalization is normally a helpful phenomenon . It allows us to deal effectively with situations that are similar but not identical to previously experienced .
.
situations
.
People are good at generalizing newly acquired knowledge . If you learn a new fact about an object , your expectations about other similar objects tend to change . If , for example , you learn that chimpanzees like onions you will probably raise your estimate of the probability that gorillas like onions . In a network that uses distributed representations , this kind of generalization is automatic . The new knowledge about chimpanzees is incorporated by modifying some of the connection strengths so as to alter the causal effects of the distributed pattern of activity that represents chimpanzees . 2 The modifications automatically change the causal effects of all similar activity patterns . So if the representation of gorillas is a similar activity pattern over the same set of units , its causal effects will be changed in a similar way . The very simplest distributed scheme would represent the concept of onion and the concept of chimpanzee by alternative activity patterns over
the
very
same
set of units .
It would
then
be hard
to represent
chimps and onions at the same time . This problem can be solved by using separate modules for each possible role of an item within a larger structure . Chimps , for example , are the " agent " of the liking and so a pattern representing chimps occupies the " agent " module and the pat tern representing onions occupies the " patient " module (see Figure 1) . 2 The internal structure of this pattern may also change . There is always a choice between changing the weights on the outgoing connections and changing the pattern itself so that different outgoing connections become relevant . Changes in the pattern itself
alter its similarity to other patterns and thereby alter how generalization will occur in the future . It is generally much harder to figure out how to change the pattern that represents
an item than it is to figure out how to change the outgoing connections so that a particular pattern
will
have
the desired
effects
on another
part
of the network
.
3. DISTRIBUTED REPR ~ ENTAnONS
83
Each module can have alternative patterns for all the various items, so this scheme does not involve local representations of items. What is localized is the role. If you subsequently learn that gibbons and orangutans do not like onions your estimate of the probability that gorillas like onions will fall , though it may still remain higher than it was initially . Obviously , the combination of facts suggests that liking onions is a peculiar quirk of chimpanzees. A system that uses distributed representations will automatically arrive at this conclusion, provided that the alternative patterns that represent the various apes are related to one another in a particular way that is somewhat more specific than just being similar to one another: There needs to be a part of each complete pattern that is identical for all the various apes. In other words, the group of units used for the distributed representations must be divided into two
RELATIONSHIP
PATIENT
AGENT
I I I
FIGURE 1. In this simplified schemethere are two different modules, one of which representsthe agentand the other the patient. To incorporatethe fact that chimpanzees like onions, the patternfor chimpanzees in one module must be associatedwith the pattern for onions in the other module. Relationshipsother than "liking" can be implementedby havinga third group of units whosepatternof activity representsthe relationship. This pattern must then "gate" the interactionsbetween the agent and patient groups. Hinton (1981a ) describesone wayof doing this gatingby usinga fourth group of units.
84
THEPDPPERSPECTIVE
subgroups, and all the various apes must be represented by the same pattern in the first su~group, but by different patterns in the second subgroup. The pattern of activity over the first subgroup represents the type of the item , and the pattern over the second subgroup represents additional microfeatures that discriminate each instance of the type from the other instances. Note that any subset of the microfeatures can be considered to define a type. One subset might be common to all apes, and a different (but overlapping) subset might be common to all pets. This allows an item to be an instance of many different types simultaneously. When the system learns a new fact about chimpanzees, it usually has no way of knowing whether the fact is true of all apes or is just a property of chimpanzees. The obvious strategy is therefore to modify the strengths of the connections emanating from all the active units , so that the new knowledge will be partly a property of apes in general and partly a property of whatever features distinguish chimps from other apes. If it is subsequently learned that other apes do not like onions, correcting modifications will be made so that the information about onions is no longer associatedwith the subpattern that is common to all apes. The knowledge about onions will then be restricted to the subpattern that distinguishes chimps from other apes. If it had turned out that gibbons and orangutans also liked onions, the modifications in the weights emanating from the subpattern representing apes would have reinforced one another, and the knowledge would have become associated with the subpattern shared by all apes rather than with the patterns that distinguish one ape from another. A very simple version of this theory of generalization has been implemented in a computer simulation (Hinton , 1981a) . Several applications that make use of this property can be found in Part IV of this book. There is an obvious generalization of the idea that the representation of an item is composed of two parts, one that represents the type and another that represents the way in which this particular instance differs from others of the same type. Almost all types are themselves instances of more general types, and this can be implemented by divid ing the pattern that represents the type into two subpatterns, one for the more general type of which this type is an instance, and the other for the features that discriminate this particular type from others instances of the same general type. Thus the relation between a type and an instance can be implemented by the relationship between a set of units and a larger set that includes .it . Notice that the more general the type, the smaller the set of units used to encode it . As the number of terms in an intensional description gets smaller, the corresponding extensionalset gets larger.
3. DISTRIBUTED REPRFSENT ATIONS
8S
In traditional semantic networks that use local representations , gen eralization is not a direct consequence of the representation . Given that chimpanzees like onions , the obvious way of incorporating the new knowledge is by changing the strengths of connections belonging to the chimpanzee unit . But this does not automatically change connections that belong to the gorilla unit . So extra processes must be invoked to implement generalization in a localist scheme . One commonly used method is to allow activation to spread from a local unit to other units that represent similar concepts (Collins & Loftus , 1975; Quillian , 1968) . Then when one concept unit is activated , it will partially activate its neighbors and so any knowledge stored in the connections emanating from these neighbors will be partially effective . There are many variations of this basic idea (Fahlman , 1979; Levin , 1976; McClelland , 1981) . It is hard to make
a clean distinction
between
systems
that use local
representations plus spreading activation and systems that use distrib uted representations . In both cases the result of activating a concept is that
many
different
hardware
units
are active . The
distinction
almost
completely disappears in some models such as McClelland ' s ( 1981) generalization model , where the properties of a concept are represented by a pattern of activation over feature units and where this pattern of activation is determined by the interactions of a potentially very large number
of units
for
instances
of the concept . The
main
difference
is
that in one case there is a particular individual hardware unit that acts as a " handle " which makes it easy to attach purely conventional proper ties like the .name of the concept and easier for the theorist who con structed the network to know what the individual parts of the network stand for .
If we construct our networks by hand -specifying the connections between the units in the network , a local representation scheme has some apparent advantages . First , it is easier to think ' one understands the behavior of a network if one has put in all the " knowledge " - all the connections
- oneself . But
if
it
is the
entire , distributed
pattern
of
interacting influences among the units in the network that is doing the work , this understanding can often be illusory . Second , it seems intui tively obvious that it is harder to attach an arbitrary name to a distrib uted pattern than it is to attach it to a single unit . What is intuitively harder , however
, may not be more
efficient
. We will
see that one can
actually implement aribitrary associations with fewer units using distrib uted representations . Before we turn to such considerations , however , we examine a different advantage of distributed representations : They make it possible to create new concepts without allocating new hardware
.
86
THEPOPPERSPECTIVE
CreatingNew Concepts Any plausible scheme for representing knowledge must be capable of learning novel concepts that could not be anticipated at the time the network was initially wired up. A scheme that uses local representations must first make a discrete decision about when to form a new concept, and then it must find a spare hardware unit that has suitable connections for implementing the concept involved . Finding such a unit may be difficult if we assume that, after a period of early development, new knowledge is incorporated by changing the strengths of the existing connections rather than by growing new ones. If each unit only has connections to a small fraction of the others, there will probably not be any units that are connected to just the right other ones to implement a new concept. For example, in a collection of a million units each connected at random to ten thousand others, the chance of there being any unit that is connected to a particular set of 6 others is only one in a million . In an attempt to rescue local representations from this problem , several clever schemes have been proposed that use two classes of units . The units that correspond to concepts are not directly connected to one another. Instead, the connections are implemented by indirect pathways through several layers of intermediate units (Fahlman, 1980; Feldman, 1982) . This scheme works because the number of potential pathways through the intermediate layers far exceeds the total number of physical connections. If there are k layers of units , each of which has a fan-out of n connections to randomly selected units in the follow ing layer, there are n k potential pathways. There is almost certain to be a pathway connecting any two concept-units , and so the intermediate units along this pathway can be dedicated to connecting those two concept-units . However, these schemes end up having to dedicate several intermediate units to each effective connection, and once the dedication has occurred, all but one of the actual connections emanating from each intermediate unit are wasted. The use of several inter mediate units to create a single effective connection may be appropriate in switching networks containing elements that have units with relatively small fan-out , but it seems to be an inefficient way of using the hardware of the brain. The problems of finding a unit to stand for a new concept and wiring it up appropriately do not arise if we use distributed representations. All we need to do is modify the interactions between units so as to create a new stable pattern of activity . If this is done by modifying a large number of connections very slightly , the creation of a new pattern need not disrupt the existing representations. The difficult problem is
3. DISTRIBUTED REPRESENT ATIONS
87
to choose an appropriate pattern for the new concept. The effects of the new representation on representations in other parts of the system will be determined by the units that are active, and so it is important to use a collection of active units that have roughly the correct effects. Fine-tuning of the effects of the new pattern can be achieved by slightly altering the effects of the active units it contains, but it would be unwise to choose a random pattern for a new concept because major changes would then be needed in the weights, and this would disrupt other knowledge. Ideally, the distributed representation that is chosen for a new concept should be the one that requires the least modification of weights to make the new pattern stable and to make it have the required effects on other representations. Naturally , it is not necessary to create a new stable pattern all in one step. It is possible for the pattern to emerge as a result of modifications on many separate occasions. This alleviates an awkward problem that arises with local representations: The system must make a discrete allor-none decision about when to create a new concept. If we view concepts as stable patterns, they are much less discrete in character. It is possible, for example, to differentiate one stable pattern into two closely related but different variants by modifying some of the weights slightly . Unless we are allowed to clone the hardware units (and all their connections) , this kind of gradual, conceptual differentiation is much harder to achieve with local representations. One of the central problems in the development of the theory of distributed representation is the problem of specifying the exact procedures by which distributed representations are to be learned. All such procedures involve connection strength modulation , following "learning rules" of the type outlined in Chapter 2. Not all the problems have been solved, but significant progress is being made on these problems. (See the chapters in Part II .)
DISTRIBUTED REPRESENTATIONS THAT WORKEFFICIENTLY In this section, we consider some of the technical details about the implementation of distributed representations. First , we point out that certain distributed representation schemes can fail to provide a suffi cient basis for differentiating different concepts, and we point out what is required to avoid this limitation . Then , we describe a way of using distributed representations to get the most information possible out of a simple network of connected units . The central result is a surprising one: If you want to encode features accurately using as few units as
88 THE POP PERSPECTIVE possible , it pays to use units that are very coarsely tuned , so that each feature activates many different units and each unit is activated by many different features . A specific feature is then encoded by a pattern of activity in many units rather than by a single active unit , so coarse coding is a form of distributed representation . To keep the analysis simple , we shall assume that the units have only two values , on and off .3 We shall also ignore the dynamics of the system because the question of interest , for the time being , is how many units it takes to encode features with a given accuracy . We start by considering the kind of feature that can be completely specified by giv ing a type (e.g., line -segment , corner , dot ) and the values of some continuous parameters that distinguish it from other features of the same type (e.g., position , orientation , size .) For each type of feature there is a space of possible instances . Each continuous parameter defines a dimension of the feature space, and each particular feature corresponds to a point in the space. For features like dots in a plane , the space of possible features is two -dimensional . For features like stopped , oriented edge-segments in three -dimensional space, the feature space is six -dimensional . We shall start by considering two dimensional feature spaces and then generalize to higher dimensionali ties . Suppose that we wish to represent the position of a single dot in a plane , and we wish to achieve high accuracy without using too many units . We define the accuracy of an encoding scheme to be the number of different encodings that are generated as the dot is moved a standard distance through the space. One encoding scheme would be to divide the units into an X group and a Y group , and dedicate each unit to encoding a particular X or Y interval as shown in Figure . 2. A given dot would then be encoded by activity in two units , one from each group , and the accuracy would be proportional to the number of units used . Unfortunately , there are two problems with this . First , if two dots have to be encoded at the same time , the method breaks down . The two dots will activate two units in each group , and there will be no way of telling , from the active units , whether the dots were at (x 1, y 1) and (x2 , y2 ) or at (x 1, y2 ) and (x2 , y 1) . This is called the binding prob lem . It arises because the representation does not specify what goes with what .
3 Similar argumentsapply with multivalued activity levels, but it' is important not to allow activity levelsto havearbitrary precisionbecausethis makesit possibleto represent an infinite amount of information in a singleactivity level. Units that transmita discrete impulsewith a probability that varies as a function of their activationseemto approximatethe kind of precisionthat is possiblein neuralcircuitry (seeChapters20 and 21).
3. DISTRIBUTED REPR ~ ENT ATIONS89
.
Ygroup
X group
0
1
.
0 Y group
0 1
.
0 0 0
00100010
X group FIGURE 2. A: A simpleway of usingtwo groupsof binary units to encodethe position of a point in a two-dimensionalspace . The active units in the X and Y groupsrepresent the x- and y-coordinates . B: When two points must be encodedat the sametime, it is impossibleto tell which x-coordinategoeswith which y-coordinate.
The second problem arises even if we allow only one point to be represented at a time . Suppose we want certain representations to be associated with an overt response, but not others: We want (x 1, y 1) and (x2 , y2 ) to be associated with a response, but not (xl , y2 ) or (x 2, y 1) . We cannot implement this association using standard weighted connections to response units from units standing for the values on the two dimensions separately. For the unit for x 1 and the unit for x2 would both have to activate the response, and the unit for
90
THEPOPPERSPECTIVE
y 1 and the unit for y 2 would both have to activate the response. There would be no way of preventing the response from being activated when the unit for x 1 and the unit for y 2 were both activated. This is another aspect of the binding problem since, again, the representation fails to specify what must go with what. In a conventional computer it is easy to solve the binding problem . We simply create two records in the computer memory . Each record contains a pair of coordinates that go together as coordinates of one dot, and the binding information is encoded by the fact that the two coordinate values are sitting in the same record (which usually means they are sitting in neighboring memory locations) . In parallel networks it is much harder to solve the binding problem .
Conjunctive Encoding One approach is to set aside , in advance , one unit for each possible combination of X and Y values . This amounts to covering the plane with a large number of small , nonoverlapping zones and dedicating a unit to each zone . A dot is then represented by activity in a single unit so this is a local representation . The use of one unit for each discrimin able feature solves the binding problem by having units which stand for the conjunction of values on each of two dimensions . In general , to permit an arbitrary association between particular combinations of features and some output or other pattern of activation , some conjunc tive representation may be required . However , this kind of local encoding is very expensive . It is much less efficient than the previous scheme because the accuracy of pin pointing a point in the plane is only proportional to the square root of the number of units . In general , for a k -dimensional feature space, the local encoding yields an accuracy proportional to the kth root of the number of units . Achieving high accuracy without running into the binding problem is thus very expensive . The use of one unit for each discriminable feature may be a reason able encoding if a very large number of features are presented on each occasion , so that a large fraction of the units are active . However , it is a very inefficient encoding if only a very small fraction of the possible features are presented at once . The average amount of information conveyed by the state of a binary unit is 1 bit if the unit is active half the time , and it is much less if the unit is only rarely active .4 It would 4 The amount of information conveyedby a unit that hasa probabilityof p of beingon is - plogp - (1 - p ) 10g(1 - p ).
91 3. DISTRIBUTED REPRESENTATIONS therefore be more efficient to use an encoding in which a larger fraction of the units were active at any moment . This can be done if we abandon the idea that each discriminable feature is represented by activity in a single unit .
Coarse Coding Suppose we divide the space into larger, overlapping zones and assign a unit to each zone. For simplicity , we will assume that the zones are circular , that their centers have a uniform random distribution throughout the space, and that all the zones used by a given encoding scheme have the same radius. The question of interest is how accurately a feature is encoded as a function of the radius of the zones. If we have a given number of units at our disposal is it better to use large zones so that each feature point falls in many zones, or is it better to use small zones so that each feature is represented by activity in fewer but more finely tuned units ? The accuracy is proportional to the number of different encodings that are generated as we move a feature point along a straight line from one side of the space to the other . Every time the line crosses the boundary of a zone, the encoding of the feature point changes because the activity of the unit corresponding to that zone changes. So the number of discriminable features along the line is just twice the number of zones that the line penetrates. 5 The line penetrates every zone whose center lies within one radius of the line (see Figure 3) . This number is proportional to the radius of the zones, r , and it is also proportional to their number , n . Hence the accuracy, a , is related to the number of zones and to their radius as follows : QCXnr. In general, for a k -dimensional space, the number of zones whose centers lie within one radius of a line through the space is proportional to the volume of a k -dimensional hypercylinder of radius , . This volume is equal to the length of the cylinder (which is fixed ) times its (k - 1) -dimensional cross-sectional area which is proportional to , k - I.
S Problems arise if you enter and leave a zone without crossing other zone borders in between becauseyou revert to the same encoding as before, but this effect is negligible if the zones are dense enough for there to be many zones containing each point in the space.
92 THE POP PERSPECTIVE
.
FIGURE 3. The
number of zone boundaries that are cut by the line is proportional to
the numberof lone centerswithin one-lone radiusof the line.
Hence, the accuracyis given by a a: nrk - 1.
So, for example, doubling the radius of the zones increases by a factor of 32, the linear accuracy with which a six-dimensional feature like a stopped oriented three-dimensional edge is represented. The intuitive idea that larger zones lead to sloppier representations is entirely wrong because distributed representations hold information much more effi ciently than local ones. Even though each active unit is less specific in its meaning, the combination of active units is far more specific. Notice also that with coarse coding the accuracy is proportional to the number of units , which is much better than being proportional to the kth root of the number . Units that respond to complex features in retinotopic maps in visual cortex often have fairly large receptive fields . This is often interpreted as the first step on the way to a translation invariant representation. However, it may be that the function of the large fields is not to achieve, translation invariance but to pinpoint accurately where the feature is! Limitations on coarse coding. So far , only the advantagesof coarse coding have been mentioned , and its problematic aspects have been ignored. There are a number of limitations that cause the coarse coding strategy to break down when the " receptive fields " become too
3. DISTRIBUTED REPRFSENTATIONS 93 large . One obvious limitation occurs when the fields become compar able in size to the whole space. This limitation is generally of little interest because other , more severe , problems arise before the recep ti ve fields become this large . Coarse coding is only effective when the features that must be represented are relatively sparse. If many feature points are crowded together , each receptive field will contain many features and the activity pattern in the coarse-coded units will not discriminate between many alternative combinations of feature points . (If the units are allowed to have integer activity levels that reflect the number of feature points fal ling within their fields , a few nearby points can be tolerated , but not many .) Thus there is a resolution / accuracy trade -off . Coarse coding can give high accuracy for the parameters of features provided that features are widely spaced so that high resolution is not also required . As a rough rule of thumb , the diameter of the receptive fields should be of the same order as the spacing between simultaneously present feature points .6 The fact that coarse coding only works if the features are sparse should be unsurprising given that its advantage over a local encoding is that it uses the information capacity of the units more efficiently by making each unit active more often . If the features are so dense that the units would be active for about half the time using a local encoding , coarse coding can only make things worse . A second major limitation on the use of coarse coding stems from the fact that the representation of a feature must be used to affect other representations . There is no point using coarse coding if the features have to be recoded as activity in finely tuned units before they can have the appropriate effects on other representations . If we assume that the effect of a distributed representation is the sum of the effects of the individual active units that constitute the representation , there is a strong limitation on the circumstances under which coarse coding can be used effectively . Nearby features will be encoded by similar sets of active units , and so they will inevitably tend to have similar effects . Broadly speaking , coarse coding is only useful if the required effect of a feature is the average of the required effects of its neighbors . At a fine enough scale this is nearly always true for spatial tasks . The scale at which it breaks down determines an upper limit on the size of the receptive fields .
6 It is interesting that many of the geometric visual illusions illustrate interactions between features at a distance much greater than the uncertainty in the subjects ' knowledgeof the positionof a feature. This is just what would be expectedif coarsecoding is beingusedto representcomplexfeaturesaccurately .
94
THEPOPPERSPECTIVE
Another limitation is that whenever coarse-coded representations interact, there is a tendency for the coarseness to increase. To counteract this tendency, it is probably necessary to have lateral inhibition operating within each representation. This issue requires further research. Extension to noncontinuous spaces . The principle underlying coarse coding can be generalized to noncontinuous spaces by thinking of a set of items as the equivalent of a receptive field . A local representation uses one unit for each possible item . A distributed representation uses a unit for a set of items , and it implicitly encodes a particular item as the intersection of the sets that correspond to the active units . In the domain of spatial features there is generally a very strong regularity : Sets of features with similar parameter values need to have similar effects on other representations . Coarse because it allows this regularity to be expressed
coding is efficient in the connection
strengths . In other domains , the regularities are different , but the effi ciency arguments are the same : It is better to devote a unit to a set of items than to a single item , provided that the set is chosen in such a way that membership in the set implies something about membership in other sets . This implication can then be captured as a connection strength . Ideally , a set should be chosen so that membership of this set has strong implications for memberships of other sets that are also encoded by individual units . We
illustrate
these
points
with
a very
simple
example .
Consider
a
microlanguage consisting of the three - letter words of English made up of w or I , followed by i or e, followed by g or r . The strings wig and leg are words , but weg , lig , and all strings ending in r are not . Suppose we wanted to use a distributed representation scheme as a basis for representing the words , and we wanted to be able to use the distributed pattern as a basis for deciding whether the string is a word or a non word . For simplicity we will have a single " decision " unit . The prob lem is to find connections from the units representing the word to the decision unit such that it fires whenever a word is present but does not fire when no word is present . 7
7 Note that the problem remains the same if the decision unit is replaced by a set of units and the task of the network is to produce a different pattern for the word and nonword decisions. For when we examine each unit , it either takes the same or a different value in the two patterns~ in the caseswhere the value is the same, there is no problem , but neither do such units differentiate the two patterns. When the values are different , the unit behavesjust like the single decision unit discussed in the text .
3. DISTRIBUTED REPRESENTATIONS9S
Figure 4 shows three representation schemes: a distributed scheme that does not work , a distributed scheme that does work , and a local scheme. In the first scheme, each letter / position combination is represented by a different unit . Since there are only five letter / position possibilities, only five units have connections to the output unit . Each word and nonword produces a different and unique pattern over these five units , but the connections from the five units to the decision unit cannot be set in such a way as to make the decision unit fire whenever one of the words is present and fail to fire whenever one of the nonwords is present. The reason for the problem is simply that the connections between the letter / position units and the decision units can only capture the degree to which each letter indicates whether the string is a word or not . The g tends to indicate that a word is present, whereas the r indi 'cates that the item is not a word; but each of the other letters, taken individually , has absolutely no predictive ability in this case. Whether a letter string is a word or not cannot be determined conclusively from the individual letters it contains; it is necessary to consider also what combinations of letters it contains. Thus, we need a representation that captures what combinations of letters are present in a way that is sufficient for the purposes of the network . One could capture this by using local representations and assigning one node to each word, as in the third panel of Figure 4. However , it is important to see that one need not go all the way to local representations to solve the
+
0
0
0
FIGURE 4. Three networksappliedto the problem of determiningwhich of the strings that can be made from w or I, followed by i or e, followed by g or , form words. Numbers on the connectionsrepresentconnection strengths~ numbers on the units representthe units' thresholds. A unit will take on an activationequal to 1 if its input exceedsit threshold~otherwise, its activationis O.
96 THEPOP PERSPECTIVE problem facing our network. Conjunctive distributed representations will suffice. The schemeillustrated in the secondpanel of the figure providesa conjunctive distributed representation . In this scheme, there are units for pairs of letters which, in this limited vocabulary, happento capture the combinationsthat are essentialfor determining whether a string of letters is a word or not. These are, of course, the pairs wi and Ie. Theseconjunctive units, together with direct input to the decisionunit from the g unit, are sufficient to construct a network which correctly classifiesall strings consistingof a w or an I, followed by an i or an e, followed by a g or r. This exampleillustrates that conjunctivecoding is often necessaryif distributed representationsare to be used to solve problemsthat might easily be posedto networks. This samepoint could be illustrated with many other examples- the exclusiveor problem is the classicexample (Minsky & Papert, 1969) . Other examplesof problemsrequiring some sort of conjunctive encodingcan be found in Hinton (1981a ) and in Chapters7 and 8. An applicationof conjunctivecodingto a psychologi cal model is found in Chapter18. Someproblems(mostly very simple ones) can be solvedwithout any conjunctive encodingat all, and others will require conjunctsof more than two units at a time. In general, it is hard to specifyin advancejust what "order" of conjunctionswill be required. Instead, it is better to searchfor a learning schemethat can find representationsthat are adequate. The mechanismsproposedin Chapters7 and 8 representtwo stepstowardthis goal.
Implementing an Arbitrary Mapping Between Two Domains The attentive reader will have noticed that a local representation can always be made to work in the example we have just considered. However, we have already discussed several reasons why distributed representations are preferable. One reason is that they can make more efficient use of parallel hardware than local representations. This section shows how a distributed representation in one group of units can cause an appropriate distributed representation in another group of units . We consider the problem of implementing an arbitrary pairing between representations in the two groups, and we take as an example an extension of the previous one: the association between the visual form of a word and its meaning. The reason for considering an arbitrary mapping is that this is the case in which local representations seem most helpful . If distributed representations are better in this
3. DISTRIBUTED REPRESENT ATIONS
97
case, then they are certainly better in cases where there are underlying regularities that can be captured by regularities in the patterns of activa tion on the units in one group and the units in another . A discussion of the benefit distributed representations can provide in such cases can be found in Chapter 18. If we restrict ourselves to monomorphemic words , the mapping from strings of graphemes onto meanings appears to be arbitrary in the sense that knowing what some strings of graphemes mean does not help one predict what a new ,string means .8 This arbitrariness in the mapping from graphemes to meanings is what gives plausibility to models that have explicit word units . It is obvious that arbitrary mappings can be implemented if there are such units . A grapheme string activates exactly one word unit , and this activates whatever meaning we wish to associate with it (see Figure SA) . The semantics of similar grapheme strings can then be completely independent because they are mediated by separate word units . There is none of the automatic generalization that is characteristic of distributed representations . Intuitively , it is not at all obvious that arbitrary mappings can be implemented in a system where the intermediate layer of units encodes the word as a distributed pattern of activity instead of as activity in a single local unit . The distributed alternative appears to have a serious drawback . The effect of a pattern of activity on other representations is the combined result of the individual effects of the active units in the pattern . So similar patterns tend to have similar effects . It appears that we are not free to make a given pattern have whatever effect we wish on the meaning representations without thereby altering the effects that other patterns have . This kind of interaction appears to make it diffi cult to implement arbitrary mappings from distributed representations of words onto meaning representations . We shall now show that these intuitions are wrong and that distributed representations of words can work perfectly well and may even be more efficient than single word units . Figure 58 shows a three -layered system in which grapheme / position units feed into word-set units which , in turn , feed into semantic or sememe units . Models of this type , and closely related variants , have been analyzed by Willshaw ( 1981) , V . Dobson (personal communica tion , 1984) , and by David Zipser (personal communication , 1981) ; some further relevant analyses are discussed in Chapter 12. For simpli 8 Even for monomorphemicwords there may be particularfragmentsthat have associ ated meaning. For example, words starting with sn usually mean somethingunpleasant to do with the lips or nose (sneer , snarl, snigger ) , and words with long vowelsare more likely to stand for large, slow things than words with short vowels (GeorgeLakoff, personalcommunication ) . Much of LewisCarroll' s poetry relieson sucheffects.
98
THEPOPPERSPECTIVE
A
5/ 3
B
FIGURE 5. A: A three-layer network. The bottom layer containsunits that represent particulargraphemesin particularpositionswithin the word. The middle layer contains units that recognizecompletewords, and the top layer contains units that represent semanticfeaturesof the meaningof the word. This network useslocal representations of wordsin the middle layer. B: The top and bottom layersare the sameas in (A), but the middle layer uses a more distributed representation . Each unit in this layer can be activatedby the graphemicrepresentationof anyone of a whole set of words. The unit then providesinput to every semanticfeature that occursin the meaningof any of the words that activateit. Only those word sets containingthe word cat are shown in this example. Notice that the only semanticfeatureswhich receiveinput from all theseword setsare the semanticfeaturesof cat.
city, we shall assumethat eachunit is either active or inactive and that there is no feedbackor cross-connections. These assumptionscan be relaxedwithout substantiallyaffecting the argument. A word-set unit is activatedwheneverthe pattern of the grapheme/position units codesa word in a particularset. The set could be all the four-letter wordsstarting with HE, for example, or all the words containingat leasttwo T's. All that is required is that it is possibleto decidewhether a word is in
3. DISTRIBUTED REPR ~ ENTAnONS
99
the set by applying a simple test to the activated grapheme/ position units . So, for example, the set of all words meaning "nice" is not allowed as a word set. There is an implicit assumption that word meanings can be represented as sets of sememes. This is a contentious issue. There appears to be a gulf between the componential view in which a meaning is a set of features and the structuralist view in which the meaning of a word can only be defined in terms of its relationships to other meanings. Later in this chapter we consider one way of integrating these two views by allowing articulated representations to be built out of a number of different sets of active features. Returning to Figure 58, the question is whether it is possible to implement an arbitrary set of associations between grapheme/ position vectors and sememe vectors when the word-set units are each activated by more than one word. It will be sufficient to consider just one of the many possible specific models. Let us assume that an active word-set unit provides positive input to all the sememe units that occur in the meaning of any word in the word set. Let us also assume that each sememe unit has a variable threshold that is dynamically adjusted to be just slightly less than- the number of active word-set units . Only sememe units that are receiving input from every active word-set unit will then become active. All the sememes of the correct word will be activated because each of these sememes will occur in the meaning of one of the words in the active word sets. However, additional sememes may also be activated because, just by chance, they may receive input from every active word-set unit . For a sememe to receive less input than its threshold , there must be at least one active word set that does not contain any word which has the sememe as part of its meaning. For each active word set the probability , i , of this happening is i = (1 - p ) (w- 1) where p is the proportion of words that contain the sememe and w is the number of words in the word set of the word-set unit . The reason for the term w - 1 is that the sememe is already assumed not to be part of the meaning of the correct word, so there are only w - 1 remaining words that could have it in their meaning. Assume that when a word is coded at the graphemic level it activates u units at the word-set level. Each sememe that is not part of the word's meaning has a probability i of failing to receive input from each word-set unit . The probability , f , that all of these word-set units will provide input to it is therefore
100
THEPOPPERSPECTIVE
f = (1- i )U = [1- (1- p)(w- }>]U By inspection, this probability of a " false-positive " sememe reduces to zero when w is 1. Table 1 shows the value of f for various combinations of values of p , u , and w . Notice that if p is very small, f can remain negligible even if w is quite large. This means that distributed representations in which each word-set unit participates in the representation of many words do not lead to errors if the semantic features are relatively sparse in the sense that each word meaning contains only a small fraction of the total set of sememes. So the word-set units can be fairly nonspecific provided the sememe units are fairly specific (not shared by too many different word meanings) . Some of the entries in the table make it clear that for some values ofp , there can be a negligible chance of error even though the number of word-set units is considerably less than the number of words (the ratio of words to word-set units is w/ u ) . The example described above makes many simplifying assumptions. For example, each word-set unit is assumed to be connected to every relevant sememe unit . If any of these connections were missing, we could not afford to give the sememe units a threshold equal to the number of active word-set units . To allow for missing connections we could lower the threshold. This would increase the false-positive error rate, but the effect may be quite small and can be compensated by adding word-set units to increase the specificity of the word-level representations (Willshaw , 1981) . Alternatively , we could make each word-set unit veto the sememes that do not occur in any of its words. This scheme is robust against missing connections because the absence of one veto can be tolerated if there are other vetos (V . Dobson, personal communication , 1984) . There are two more simplifying assumptions both of which lead to an underestimate of the effectiveness of distributed representations for the arbitrary mapping task. First , the calculations assume that there is no fine -tuning procedure for incrementing some weights and decrementing others to improve performance in the cases where the most frequent errors occur. Second, the calculations ignore cross-connections among the sememes. If each word meaning is a familiar stable pattern of sememes, there will be a strong "clean-up" effect which tends to suppress erroneous sememes as soon as the pattern of activation at the sememe level is sufficiently close to the familiar pattern for a particular word meaning. Interactions among the sememes also provide an explanation for the ability of a single grapheme string (e.g., bank) to elicit two quite different meanings. The bottom-up effect of the activated
3. DISTRIBUTED REPRESENTATIONS101
TABLEt u
p
f
u
5
5
.2
0 .071
5
5
10
.2
0 .49
5
5 5 5
f
u
w
p
5
5 10 20 40 80
.01 .01 .01 .01 .01
f
w
p
5 10 20 40 80
.1 .1 .1 .1 .1
0.0048 0.086 0.48 0.92 1.0 0.0074 0.23 0.85 1.0 1.0
10 10 10 10 10
10 20 40
.01 .01 .01 80 .01 160 .01
2.3x10-11 2.5x10-8 1.3x10-5 0.0024 0.10
0 .52
40 40 40 40 40
40 80 160 320 640
2.7x1<:t""20 3.5x1<:t""11 0.00012 0.19 0.94
5
20
.2
0 .93
5
40
.2
1.0
5
80
.2
1.0
10 10 10 10 10
10 20 40 80 160
.2 .2 .2 .2 .2
0.24 0.86 1.0 1.0 1.0
10 10 10 10 10
.1 .1 .1 80 .1 160 .1
40 40 40 40 40
40 80 160 320 640
.2 .2 .2 .2 .2
0.99 1.0 1.0 1.0 1.0
40 40 40 40 40
40 80 160 320 640
100 100 100 100
100 200 400 800
.2 .2 .2 .2
1.0 1.0 1.0 1.0
10 20 40
0 .99 1.0 1.0 1.0
10 100 .1 10 200 .1 100 400 .1 100 800 .1
0.99 1.0 1.0 1.0
5 5 5 5
100 100 100 200 100 400 100 800
.01 .01 .01 .01
9.5x10- 8 4.8x 10- 6 0.00016 0.0036 0.049
9.0x 10 --21 4.8x 10- 7 0.16 0.97
~
~
I
The probability, f, of a false-positivesememeas a function of the numberof activewordset units per word, u, the numberof wordsin eachword-set, w, and the probability, p, of a sememebeingpart of a word meaning.
11 _
1
~
~
~
word-set units helps both sets of sememes, but as soon as top-down factors give an advantage to one meaning, the sememes in the other meaning will be suppressed by competitive interactions at the sememe level (Kawamoto & Anderson, 1984) . A simulation . As soon as there are cross-connections among the sememe units and fine -tuning of individual weights to avoid frequent errors, the relatively straightforward probabilistic analysis given above breaks down. To give the cross-connections time to clean up the output, it is necessary to use an iterative procedure instead of the simple "straight-through " processing in which each layer completely determines the states of all the units in the subsequent layer in a single, synchronous step. Systems containing cross-connections, feedback, and asynchronous processing elements are probably more realistic, but they are generally very hard to analyze. However, we are now beginning to discover that there are subclasses of these more complex systems that behave in tractable ways. One example of this subclass is described in
102 THE POP PERSPECTIVE more detail in Chapter 7. It uses processing elements that are inherently stochastic . Surprisingly , the use of stochastic elements makes these networks better at performing searches , better at learning , and easier to analyze . A simple network of this kind can be used to illustrate some of the claims about the ability to " clean up " the output by using interactions among sememe units and the ability to avoid errors by fine -tuning the appropriate weights . The network contains 30 grapheme units , 20 word - set units , and 30 sememe
units . There
are no direct
connections
between grapheme and sememe units , but each word -set unit is con -
nected to all the grapheme and sememe units . The grapheme units are divided
into
three
sets of ten , and each three -letter
word
has one active
unit in each group of ten (units can only have activity levels of 1 or 0) . The " meaning " of a word is chosen at random by selecting each sememe unit to be active with a probability of 0.2. The network shown in Figure 6 has learned to associated 20 different grapheme strings with their chosen meanings . Each word -set unit is involved in the represen tation
of many
words , and each word
involves
many word - set units .
The details of the learning procedure used to create this network and the search procedure which is used to settle on a set of active sememes
when given the graphemic input are described in Chapter 7. Here we simply summarize the main results of the simulation . After a long period of learning , the network was able to produce the
correct pattern of sememes 99.9% of the time when given a graphemic input . Removal of anyone of the word -set units after the learning typ ically caused a slight rise in the error rate for several different words rather than t~ complete loss of one word . Similar effects have been observed in other distributed models (Wood , 1978) . In our simula tions , some of the erroneous responses were quite interesting . In 10,000 tests with a missing word -set unit there were 140 cases in which the model failed to recover the right sememe pattern . Some of these consisted of one or two missing or extra sememes , but 83 of the errors
were exactly the pattern of sememes of some other word. This is a result of the cooperative interactions among the sememe units . If the input coming from the word -set units is noisy or underspecified as it may be when
units
are knocked
out , the clean - up effect
may settle
on a
similar but incorrect meaning . This effect is reminiscent of a phenomenon called deep dyslexia which occurs with certain kinds of brain damage in adults . When shown a word and asked to read it , the subject will sometimes say a different word with a very similar meaning . The incorrect word sometimes has a very different sound and spelling . For example , when shown the word PEACH , the subject might say APRICOT . (See Coltheart , Patterson , & Marshall , 1980 , for more information about acquired dyslexia .)
104 THE POP PERSPECTIVE it
has
been
every
damaged
connection
performance
from
retrained
and
original
rate
rapid
there
very
by
when
from
.
more the
retraining
sememe
pairings
have
that
were the
selected
is not
representations the
.
words
tends unit
All
that
to
each
recovery
of
tri buted
representations
STRUCTURED
In
this .
section
involved
weight
behave items
of
representations
the
two
some
of
the
that
-
pair words
subset
added
used
as a qualitative
extensions that
of
.
have
unclear
how
of
noise
a separate
can
view
spon -
signature
of dis -
scheme directions
to deal
with
the been
structure .
that these
The can
of
of
because
some
particularly
parts
be
taken
be in
the
in of
to in
section
extending
these
a distri
give
-
some
distributed .
9 The error rate was 99 .3% rather than 99 .9% in this example because the network forced to respond faster , so the cooperative output .
of
structure
attuned
considerations
-
field
proponents
captured
of this
-
represen
from
importance
is to
representa
distributed
insights
two
important
PROCESSES
distributed
idea
major
Perhaps not
of
the
the
concerning processes
representations is often
the
of distributed
the
way , so one
AND
illustrate
with
and
-
grapheme all
use
so
. A scheme in this
is the
is substantially
recovery
, and
set
are omit
other
in encoding
retraining
every
consider
representational
indication
are
during
"
that
.)
because
of the
This shows
exhibit
words
the
" spontaneous is a result
its
.
intelligence
distributed
buted
The again
unrehearsed
we
representations
issues , it
to them
.
resulting that
words
relation
not
extensions
is consistent
artificial
these
REPRESENTATIONS
These
tations
for
intrinsic .
The
of the
though
from
taneous
rate
then
than
strengths
discussion
if a few
was
which
strengths
further
occurs
weights
would
set .
to the
64 .3 % correct
connection
, even
shown
word
of
noise
reduced faster
argument
connection 7 for
adding
This network
was
proceeds
shown
the
are
be removed
for
tions
no
a set
error
.
, much
a near - perfect of
The
randomly
network
performance
effect
.
retraining
relearning
a geometrical
to
by
unit
64 .3 % . 9 The
its
( See Chapter
as the
- set
by
sets
surprising
reduced
ings
to
about
noise
other
damaged
word
rapid
special
adding from
even
correct
predicted
is something
was a
very
learning
performance
An
network
involved
99 .3 %
was
different
same
The
exhibited
of
is generated
ted
it
recovery
that
.
that
was
effects had less time to settle on the optimal
3. DISTRIBUTED REPRESENTATIONS 105 RepresentingConstituent Structure Any system that attempts to implement the kinds of conceptual structures that people use has to be capable of representing two rather different kinds of hierarchy. The first is the " IS-A" hierarchy that relates types to instances of those types. The second is the part/ whole hierarchy that relates items to the constituent items that they -are composed of . The most important characteristics of the IS-A hierarchy are that known properties of the types must be " inherited " by the instances, and properties that are found to apply to all instances of a type must normally be attributed to the type. Earlier in this chapter we saw how the IS- A hierarchy can be implemented by making the distributed representation of an instance include, as a subpart, the distributed representation for the type. This representational trick automatically yields the most important characteristics of the IS- A hierarchy, but the trick can only be used for one kind of hierarchy. If we use the part/ whole relationship between patterns of activity to represent the type/ instance relationship between items, it appears that we cannot also use it to represent the part/ whole relationship between items. We cannot make the representation of the whole be the sum of the representations of its parts. The Question of how to represent the relationship between an item and the constituent items of which it is composed has been a major stumbling block for theories that postulate distributed representations. In the rival , localist scheme, a whole is a node that is linked by labeled arcs to the nodes for its parts. But the central tenet of the distributed scheme is that different items correspond to alternative patterns of activity in the same set of units , so it seems as if a whole and its parts cannot both be represented at the same time . Hinton ( 1981a) described one way out of this dilemma . It relies on the fact that wholes are not simply the sums of their parts. They are composed of parts that play particular roles within the whole structure . A shape, for example, is composed of smaller shapes that have a particular size, orientation , and position relative to the whole. Each constituent shape has its own spatial role, and the whole shape is composed of a set of shape/ role pairs.tO Similarly , a proposition is composed of objects that occupy particular semantic roles in the whole propositional 10 Relationships representing can view
the
of an object mented
between
shape / role various as the
by positive
pairs
different fillers
parts
important
it allows
locations
of these
interactions
are
is that
slots .
between
within
as
well .
different
pairs
an object
Knowledge the various
One
as slots
of a whole slot - fillers
advantage
to support and shape .
each
of
explicitly
other . One
the shapes can then
of parts be imple -
PERSPECTIVE 106 THEPOP
structure . This suggestsa way of implementing the relationship between wholes and parts: The identity of each part should first be combined with its role to produce a single pattern that represents the combination of the identity and the role, and then the distributed representation for the whole should consist of the sum of the distributed representations for these identity / role combinations (plus some additional " emergent" features) . This proposal differs from the simple idea that the representation of the whole is the sum of the representations of its parts because the subpattems used to represent identity / role combinations are quite different from the patterns used to represent the identities alone. They do not , for example, contain these patterns as parts. Naturally , there must be an access path between the representation of an item as a whole in its own right and the representation of that same item playing a particular role within a larger structure . It must be possible, for example, to generate the identity / role representation from two separate, explicit , distributed patterns one of which represents the identity and the other of which represents the role. It must also be possible to go the other way and generate the explicit representations of the identity and role from the single combined representation of the identity / role combination (see Figure 7) . The use of patterns that represent identity / role combinations allows the part/ whole hierarchy to be represented in the same way as the type/ instance hierarchy. We may view the whole as simply a particular instance of a number of more general types, each of which can be defined as the type that has a particular kind of part playing a particular role (e.g., men with wooden legs) .
SequentialSymbol Processing If constituent structure is implemented in the way described above, there is a serious issue about how many structures can be active at any one time . The obvious way to allocate the hardware is to use a group of units for each possible role within a structure and to make the pattern of activity in this group represent the identity of the constituent that is currently playing that role. This implies that only one structure can be represented at a time , unless we are willing to postulate multiple copies of the entire arrangement. One way of doing this , using units with programmable rather than fixed connections, is described in Chapter 16. However , even this technique runs into difficulties if more than a few modules must be " programmed" at once. However, people do seem to suffer from strong constraints on the number of structures of the same general type that they can process at once. The
3. DISTRIBUTED REPRFSENT AnONS
sequentiality
that they exhibit at this high level of description is initially
surprising given the massively parallel architecture of the brain, but it
I
becomes much easier to understand if we abandon our localist predelic tions in favor of the distributed alternative which uses the parallelism to give each active representation a very rich internal structure that allows the right kinds of generalization and content -addressability . There may be some truth to the notion that people are sequential symbol processors if each " symbolic representation " is identified with a
108 THEPOP PERSPECTIVE successive state of a large interactive further
discussion
network .
See Chapter
14 for
of these issues .
One central tenet of the sequential symbol processing approach (Newell , 1980) is the ability to focus on any part of a structure and to expand that into a whole that is just as rich in content as the original whole of which it was a part . The recursive ability to expand parts of a structure for indefinitely many levels and the inverse ability to package up whole
structures
into
a reduced
form
that
allows
them
to be used
as
constituents of larger structures is the essence of symbol processing . It allows a system to build structures out of things that refer to other whole structures without requiring that these other structures be represented in all their cumbersome detail . In conventional computer implementations , this ability is achieved by using pointers . These are very convenient , but they depend on the use of addresses . In a parallel network , we need something that is func tionally equivalent to arbitrary pointers in order to implement symbol processing . This is exactly what is provided by sub patterns that stand for identity / role combinations . They allow the full identity of the part to be accessed from a representation of the whole and a representation of the role that the system wishes to focus on , and they also allow explicit representations of an identity and a role to be combined into a
less cumbersome representation, so that several identity / role combinations can be represented simultaneously tation of a larger structure .
in order to form the represen -
SUMMARY
Given a parallel network , items can be represented by activity in a single , local unit or by a pattern of activity in a large set of units with each unit encoding a microfeature of the item . Distributed representa -
tions are efficient whenever there are underlying regularities which can be captured by interactions among microfeatures . By piece of knowledge as a large set of interactions , it achieve useful properties like content -addressable automatic generalization , and new items can be created to create new connections
at the hardware
encoding each is possible to memory and without having
level . In the domain
of con -
tinuously varying spatial features it is relatively easy to provide a mathematical analysis of the advantages and drawbacks of using distri buted representions . Distributed representations seem to be unsuitable for implementing purely arbitrary mappings because there is no underlying structure and so generalization only causes unwanted interference . However , even
3. DISTRIBUTED REPRFSENT ATIONS
109
for this task, distributed representations can be made fairly efficient and they exhibit some psychologically interesting effects when damaged. There are several difficult problems that must be solved before distributed representations can be used effectively . One is to decide on the pattern of activity that is to be used for representing an item . The similarities between the chosen pattern and other existing patterns will determine the kinds of generalization and interference that occur. The search for good patterns to use is equivalent to the search for the underlying regularites of the domain. This learning problem is addressedin the chapters of Part II . Another hard problem is to clarify the relationship between distrib uted representations and techniques used in artificial intelligence like schemas, or hierarchical structural descriptions. Existing artificial intel ligence programs have great difficulty in rapidly finding the schema that best fits the current situation . Parallel networks offer the potential of rapidly applying a lot of knowledge to this best-fit search, but this potential will only be realized when there is a good way of implement ing schemas in parallel networks. A discussion of how this might be done can be found in Chapter 14.
ACKNO WLEDGMENTS This chapter is based on a technical report by the first author , whose work is supported by a grant from the System Development Foundation . We thank Jim Anderson, Dave Ackley , Dana Ballard, Francis Crick , Scott Fahlman, Jerry Feldman, Christopher Longuet-Higgins, Don Norman , Terry Sejnowski, and Tim Shallice for helpful discussions.
CHAPTER 4 PDP Modelsand General Issues in Cognitive Science
D. E. RUMELHARTandJ. L. MCCLELLAND
We are naturally optimistic about parallel distributed processing as a valuable framework for creating cognitive models. This does not mean, however, that there are no tough problems to be solved. Indeed, we have spent much of our effort convincing ourselves that POP models could form a reasonable basis for modeling cognitive processesin general. In this chapter we shall address some of the objections that we and others have raised to the work and sketch our answers to these objections. However, we should like to say at the outset that we do not believe that any such general considerations as those discussed here will , in the end, bear much weight. The real proof is in the pudding. If PDP models are a valuable way to proceed, their usefulness will be proved in the added insights they bring to the particular substantive areas in which they are applied. The models we describe in later chapters are largely intended to constitute the beginnings of such a proof . Many of the questions and issues raised below are addressed by material described in detail in other chapters in the book. For this reason, much of our present discussion is in the form of pointers to the relevant discussions. In this sense, this chapter serves not only as a discussion of our approach but as an overview of the issues and topics ~hat are addressedin the chapters that follow .
4.GENERAL ISSUES 111
SOMEOBJECTIONS TOTHEPDPAPPROACH POP Models Are Too Weak The one-layer perceptron. The most commonly heard objection to POP models is a variant of the claim that POP models cannot perform any interesting computations. One variant goes like this : "These POP models sound a lot like perceptrons to me. Didn 't Minsky and Papert show that perceptron-like models couldn 't do anything interesting ?" This comment represents a misunderstanding of what Minsky and Papert (1969) have actually shown. A brief sketch of the context in which Minsky and Papert wrote will help clarify the situation . (See Chapter 5 for a somewhat fuller account of this history .) In the late 1950s and early 1960s there was a great deal of effort in the development of self-organizing networks and similar POP-like computational devices. The best known of these was the perceptron developed by Frank Rosenblatt (see, for example, Rosenblatt, 1962) . Rosenblatt was very enthusiastic about the perceptron and hopeful that it could serve as the basis both of artificial intelligence and the modeling of the brain. Minsky and Papert, who favored a serial symbolprocessing approach to artificial intelligence , undertook a very careful mathematical analysis of the perceptron in their 1969 book entitled , simply , Perceptrons . The perceptron Minsky and Papert analyzed most closely is illustrated in Figure 1. Such machines consist of what is generally called a retina, an array of binary inputs sometimes taken to be arranged in a twodimensional spatial layout; a set of predicates, a set of binary threshold units with fixed connections to a subset of units in the retina such that each predicate computes some local function over the subset of units to which it is connected; and one or more decision units, with modifiable connections to the predicates. This machine has only one layer of modifiable connections; for this reason we will call it a one-layer perceptron . Minsky and Papert set out to show which functions can and cannot be computed by this class of machines. They demonstrated, in particular, that such perceptrons are unable to calculate such mathematical functions as parity (whether an odd or even number of points are on in the retina) or the topological function of connectedness (whether all points that are on are connected to all other points that are on either directly or via other points that are also on) without making use of absurdly large numbers of predicates. The analysis is extremely elegant and demonstrates the importance of a mathematical approach to analyzing computational systems.
PERSPECTIVE 112 THEPOP
FIGURE 1. The one-layer perceptronanalyzedby Minsky and Papert. (From Perceptrons by M. L. Minsky and S. Papert, 1969, Cambridge , MA: MIT Press. Copyright 1969by MIT Press. Reprintedby permission .)
Minsky and Papert's analysis of the limitations of the one-layer perceptron, coupled with some of the early successesof the symbolic processing approach in artificial intelligence , was enough to suggest to a large number of workers in the field that there was no future in perceptron-like computational devices for artificial intelligence and cognitive psychology. The problem is that although Minsky and Papert were perfectly correct in their analysis, the results apply only to these simple one-layer perceptrons and not to the larger class of perceptronlike models. In particular (as Minsky and Papert actually conceded) , it can be shown that a multilayered perceptron system, including several layers of predicates between the retina and the decision stage, can compute functions such as parity, using reasonable numbers of units each computing a very local predicate. (See Chapters 5 and 8 for examples of multilayer networks that compute parity) . Similarly , it is not diffi cult to develop networks capable of solving the connectedness or inside/ outside problem . Hinton and Sejnowski have analyzed a version of such a network (see Chapter 7) . Essentially, then , although Minsky and Papert were exactly correct in their analysis of the one-layer perceptron, the theorems don't apply to systems which are even a little more complex. In particular, it doesn' t apply to multilayer systems nor to systems that allow feedback loops. Minsky and Papert argued that there would not be much value to multilayer perceptrons. First , they argued that these systems are suffi ciently unrestricted as to be vacuous. They pointed out , for example, that a universal computer could be built out of linear threshold units .
4.GENERAL ISSUES 113 Therefore , restricting consideration of machines made out of linear threshold units is no restriction at all on what can be computed. We don't , of course, believe that the class of models sketched in Chapter 2 is a small or restrictive class. (Nor , for that matter, are the languages of symbol processing systems especially restrictive .) The real issue, we believe, is that different algorithms are appropriate to dif ferent architectural designs. We are investigating an architecture in which cooperative computation and parallelism is natural. Serial symbolic systems such as those favored by Minsky and Papert have a natural domain of algorithms that differs from those in POP models. Not everything can be done in one step without feedback or layering (both of which suggest a kind of "seriality " ) . We have been led to consider models that have both of these features. The real point is that we seek algorithms that are as parallel as possible. We believe that such algorithms are going to be closer in form to the algorithms which could be employed by the hardware of the brain and that the kind of parallelism we employ allows the exploitation of multiple information sources and cooperative computation in a natural way. A further argument advanced by Minsky and Papert against perceptron-like models with hidden units is that there was no indication how such multilayer networks were to be trained. One of the appealing features of the one-layer perceptron is the existence of a powerful learning procedure, the perceptron convergence procedure of Rosenblatt . In Minsky and Papert's day, there was no such powerful learning procedure for the more complex multilayer systems. This is no longer true . Chapters 5, 6, 7, and 8 all provide schemes for learning in systems with hidden units . Indeed, Chapter 8 provides a direct generalization of the perceptron learning procedure which can be applied to arbitrary networks with multiple layers and feedback among layers. This procedure can, in principle , learn arbitrary functions including , of course, parity and connectedness. The problem of stimulus equivalence. A second problem with early POP models- and one that is not necessarily completely overcome by multilayer systems- is the problem of invariance or stimulus equivalence. An A is an A is an A , no matter where on the retina it appears or how large it is or how it is oriented ~ and people can, in general, recognize patterns rather well despite various transformations . It has always seemed elegant and natural to imagine that an A , no matter where it is presented, is normalized and then processed for recognition using stored knowledge of the appearanceof the letter (Marr , 1982~ Neisser, 1967) . In conventional computer programs this seems to be a rather straightforward matter requiring , first , normalization of the input , and,
PERSPECTIVE 114 THEPDP
second , analysis of the normalized input . But in early PDP models it was never clear just how normalization could be made to work . Indeed , one of the main
criticisms
of perceptrons
- one that is often
leveled
at
more recent PDP models , too - is that they appear to provide no mechanism of attention , no way of focusing the machine on the analysis of a part of a larger whole and then switching to another part or back to the consideration
of the whole .
While it is certainly true that certain PDP models lack explicit atten tional
mechanisms
, it is far
from
true
that
PDP
mechanisms
are in
principle incapable of exhibiting attentional phenomena . Likewise , while it is true that certain PDP models do not come to grips with the stimulus equivalence problem , it far from true that they are incapable of doing this in principle . To prove these points , we will describe a method for solving the stimulus equivalence problem that was described by Hinton ( 1981b) . The idea is sketched in Figure 2. Essentially , it involves two sets of feature detectors . One (at the bottom of the figure ) consists of retinocentric feature detectors and the other (above the retinocentric units ) consists of canonical feature detectors . Higher order units that recognize canonical patterns (in this example , letters ) sit above the canonical feature detectors and can have mutually excitatory connections to these feature detectors , just as in the interac tive activation model of word recognition . What Hinton described was a method for mapping retinocentric feature patterns into