Cognitive Systems Monographs Volume 1 Editors: Rüdiger Dillmann · Yoshihiko Nakamura · Stefan Schaal · David Vernon
Paolo Arena and Luca Patanè (Eds.)
Spatial Temporal Patterns for Action-Oriented Perception in Roving Robots
ABC
Rüdiger Dillmann, University of Karlsruhe, Faculty of Informatics, Institute of Anthropomatics, Robotics Lab., Kaiserstr. 12, 76128 Karlsruhe, Germany Yoshihiko Nakamura, Tokyo University Fac. Engineering, Dept. Mechano-Informatics, 7-3-1 Hongo, Bukyo-ku Tokyo, 113-8656, Japan Stefan Schaal, University of Southern California, Department Computer Science, Computational Learning & Motor Control Lab., Los Angeles, CA 90089-2905, USA David Vernon, Khalifa University Department of Computer Engineering, PO Box 573, Sharjah, United Arab Emirates
Editors Dr. Paolo Arena
Dr. Luca Patanè
Università di Catania Dipto. Ingegneria Elettrica Elettronica e dei Sistemi Viale Andrea Doria, 6 95125 Catania Italy E-Mail:
[email protected]
Università di Catania Dipto. Ingegneria Elettrica Elettronica e dei Sistemi Viale Andrea Doria, 6 95125 Catania Italy E-Mail:
[email protected]
ISBN 978-3-540-88463-7
e-ISBN 978-3-540-88464-4
DOI 10.1007/978-3-88464-4 Cognitive Systems Monographs
ISSN 1867-4925
Library of Congress Control Number: 2008936216 c 2009
Springer-Verlag Berlin Heidelberg
This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable for prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typeset & Cover Design: Scientific Publishing Services Pvt. Ltd., Chennai, India. Printed in acid-free paper 543210 springer.com
Preface
The basic principles guiding sensing, perception and action in bio systems seem to rely on highly organised spatial-temporal dynamics. In fact, all biological senses, (visual, hearing, tactile, etc.) process signals coming from different parts distributed in space and also show a complex time evolution. As an example, mammalian retina performs a parallel representation of the visual world embodied into layers, each of which represents a particular detail of the scene. These results clearly state that visual perception starts at the level of the retina, and is not related uniquely to the higher brain centres. Although vision remains the most useful sense guiding usual actions, the other senses, first of all hearing but also touch, become essential particularly in cluttered conditions, where visual percepts are somehow obscured by environment conditions. Efficient use of hearing can be learnt from acoustic perception in animals/insects, like crickets, that use this ancient sense more than all the others, to perform a vital function, like mating. Also motion in living systems is derived from highly organised neural networks driving limbs and other parts of the body, in response to sensory inputs. A huge number of muscular cells are controlled in real time and in an adaptive fashion. Biological locomotion generation and control patterns were demonstrated to be efficiently modelled by using the paradigm of the Central Pattern Generator (CPG) as well as the decentralised control approach (Walknet). In locomotion, different and differently specialised neuronal assemblies behave in a self-organised fashion in such a way that, as a consequence of certain stimuli, particular patterns of neural activity arise, which are adeguately sent to peripheral fibres to generate rhythmic activities of leg motion and control. Recent results have also shown that nonlinear and complex dynamics in cellular circuits and systems can efficiently model both sensing and locomotion. After analysing and being involved in such topics, researchers from various scientific fields and from different European Laboratories came up to the idea to glue their effort to try to go up from sensing and locomotion modelling, to perception. The present volume collects the most significant results of the European research Project called “SPARK”, whose aim was to develop completely new sensing-perceiving-moving artefacts inspired by the basic principles of living systems and based on the concept of “self-organization”. The project activities were carried out by scientists from six institutions: the Institute for Biological Cybernetics of the University of Bielefeld (Germany), the Institute of Perception, Action and
VI
Preface
Behaviour, University of Edinburgh (UK), the Instituto Pluridisciplinar, Universidad Complutense de Madrid (Spain), the Companies ANALOGIC s.l. from Budapest (Hungary) and Innovaciones Microelectronicas s.l., ANAFOCUS, from Seville (Spain). The Consortium was Coordinated by the Dipartimento di Ingegneria Elettrica Elettronica e dei Sistemi of the Unversity of Catania (Italy). The main scientific objective of the project was to consider the concept of sensorymotor integration and control tightly based upon the paradigm of complex spatialtemporal dynamics leading to the formalisation of novel bio-inspired control strategies. As far as perception is concerned, current perceptual schemes are often based on information derived from visual routines. Since real world images are quite complex to be processed for perceptual needs with traditional approaches, more computationally feasible algorithms are required to extract the desired features from the scene in real time, to efficiently proceed with the subsequently action. Traditional full representational approaches have to be replaced by more parsimonious and flexible strategies. The focus of the project was to answer in a novel approach, from a methodological point of view, to the key question in action-oriented perception on how can perception provide necessary information for motor behaviour. In our spatial-temporal approach, perception is considered as the result of a dynamic pattern forming process, in which a particular pattern will evolve in a spatial-temporal structure, from the information deriving from sensors. This pattern will concisely represent the environment information. Recent results in neurobiology and psychology have shown that this is based on internal representations that combine aspects of sensory input and motor output in a holistic, unified way. These representations can be modelled by recurrent neural networks added to spatial-temporal nonlinear patterns. The pattern of internal representation of the environment will directly influence the particular associated motor behaviour. Patterns of cellular activity are quite common in nature. For example, Turing-like patterns are usually employed in mathematical biology to represent the dynamics of morphogenesis, the process underneath the geometrical growth of organisms from initial gastrulation to adult shape (more is to be said about this below). Plateau potentials (patterns) of neural activity are common in lower invertebrates and in molluscs, to select particular locomotion patterns from given sensory information. These results give the possibility to formalise a new paradigm for active perception based on principles borrowed from ecological psychology, synergetic and dynamical systems approaches, following Haken’s and Kelso’s ones to perception and action. The project aimed also at introducing a new architecture for action-oriented perception, trying to emulate living beings, where some actions are inherited, like escape or feeding actions, while some others have to be learnt from the environment, subjected to the capability of the senses. The active collaboration among the partners resulted in the introduction of a new general model for actionoriented perception. The target biological architecture taken into account was the insect brain. Insects were selected because, though often regarded as relatively simple reactive agents, nevertheless are capable of interesting complex perception for action, for example a range of impressive navigation tasks that require memory and multi-sensory integration. Their sensory systems contain a multiplicity of specialised receptors that perform intricate filtering processes even at the periphery, both in anatomical arrangement and in processes of thresholding, spatial and temporal filtering, adaptation etc.
Preface
VII
Many multimodal interactions between these sensory systems are observed, for example: between visual, olfactory and mechanosensory wind detectors in following odor trails; between auditory and visual reflexes; and in multimodal cues for flight stabilisation. In summary, insects are not so complex to prevent careful modelling, and not so simple for not gaining very interesting and useful insights for new classes of perceptual machines. Starting from biological studies on the insect nervous system, a scheme for an insect brain architecture was introduced and, consequently, a new artificial perceptual model was conceived. This is a hierarchical structure, which takes inspiration from previous work on environmentally mediated perception, but enhances the existing models in several aspects, reflecting each individual partner’s expertise. Parallel precognitive behaviours, acting as hardwired systems, cooperate to provide suitable actions to drive the robot from the very beginning of its navigation. Higher structures provide an ever increasing level of adaptation, including plasticity and learning. In particular a proto-cognitive correlation layer is devoted to detect time varying causal correlations among the behaviours so as to learn to anticipate one behaviour through another. A further layer provides “Representations” of the environment. These are conceived as emergent dynamical flows in complex nonlinear dynamical cellular systems, under the form of Turing Patterns. These provide an abstract and concise representation of the environment as an emerging pattern, whose codified version is used to learn, in an unsupervised way, the suitable modulation of the proto-cognitive behaviours of the lower levels. Further memory models save successful sequences of modulated behaviours for future exploitation. Such a new and complex model needs to be assessed starting from the single elements. Moreover, each of the single blocks has to be studied from the neurobiological, mathematical and engineering points of view. This book provides a thorough analysis within the whole model, through the investigation within the single components: basic pre cognitive behaviours (including locomotion), proto-cognitive behaviours, the correlation block, the representation layer. All these constitute the various blocks of a perceptual architecture that will confidentially lead to a general architecture for action oriented perception in roving robots. Although the main focus is methodological, some efforts were performed to provide realistic test cases and, as a consequences, the project delivered some robot prototypes able to implement the perceptual routines. The test beds to carry out the experiments were a legged and a wheeled robot prototype, where the perception task will be focalised to solve the locomotion/navigation problem in a cluttered environment, fusing together the different kinds of sensing capabilities to carry out specific tasks. The robots will integrate low level responses from sensors: they will learn to create an iconic, abstract and simplified world representation under the form of a dynamic pattern and will generate, as a result, a proper action under the form of a motor pattern. The action selection will result not on the basis of an “if-then” process, but as a consequence of a spatial-temporal self-organisation, which will be represented through a pattern of activity. The book is organised in three parts. Part I Principles of insect perception reports a thorough survey of the state of the art in perception. Here perception is considered in the sense of Perception for action, which, from a high level, aims at describing how sensory signals are transformed into actions, that, affecting the environment, are transformed back into sensory events. The traditional approach seems to pose in primary relevance
VIII
Preface
the existence of a representation stage, within the perceptual system, at the aim to build a kind of internal model useful for producing an action. This model would be able also to predict the effect of all (or of part of) sensory events, so as to react also to traces of previous experienced situations. The first chapter analyzes where and when the notion of representation is needed, i.e. which kind of behaviours really requires internal models. It results that most of action-relevant perceptual behaviours would not need, in principle, any internal model. Moreover, in most cases, a complete description of the environment is not needed, but only that part relevant for the action to be executed. This has also several links with the direct view of Gibsonian perception and the associated notion of affordance. A different point of view considers perception as a transformation, in the sense that the agent transforms the sensory signals to give rise to motor output. According to this view, there is no need for an internal model. This approach well matches with the known fact that biological sensory systems perform much more than a mere sensing. They rather process signals in a sometimes very complex way. Under this prospective, perception is active. Vision can not be considered as a “passive visual scene recording”, but rather a large amount of special purposes algorithms; haptics is an active handling used also to obtain object qualities. The same consideration could be also done with respect to the motor stage: body configuration and motor systems are tightly connected to the perceptual stage, and indeed they contribute to attain high perceptual capabilities. Once again the concept of direct perception by Gibson is a paradigm in this sense. This approach can be extended, hypothesizing parallel transformations for multiple sensory-perceptual-motor loops, each one and all together closed through the environment. This closely resembles Rodney Brook’s approach. The existence of this kind of parallel sensory motor pathways was found in the insect brain architecture, that, in these decades, is starting to be progressively unraveled. Therefore this approach can be taken into account for the design of the basis of an insect brain inspired architecture. This structure should be constituted of merely sensory motor capabilities that represent basic, reflex-driven behaviours. Examples of these are locomotion and the related sensory systems, like mechanosensors, environmental sensors and antennae, olfactory, visual and sound system. Building upon this, the possibility to scale-up to intelligent behaviours could be faced with referring to the possibility to exploit and further transform information from the parallel sensory motor loops, leading to the emergence of context-dependent new solutions. This could include the possibility, for example, to predict one sensory motor loop by using another one, decide about termination or maintenance of a given behaviour in front of certain environment contingencies, create motivation-driven complex behaviours, and some forms of non elemental learning. Once again, works on insects can answer to these questions: current knowledge on insect behavioural and physiological neuroscience can help to derive models able to show also the emergence of complex behaviours. That is why most of the biologically relevant part of the book will be devoted to insects, and details of the nervous system basic physiology will be analyzed. An insect brain inspired control architecture has been derived. This is basically a block diagram, in which some blocks are known with a certain detail, some others with less accuracy, some others remaining practically unknown. The latter are exactly
Preface
IX
those involved in higher perceptual competencies: here are involved nonlinear dynamics, plasticity and learning, in a way whose details are largely unknown, leaving room to some working hypotheses. In this way traditional dynamical systems cannot help us very much. The new tools deriving from complex and self-organising systems, chaotic dynamics and bio-inspired learning strategies can be exploited. The hard problem of how to co-ordinate multiple parallel sensory motor loops can find a plausible solution. Part II Cognitive Models develops upon what is presented in the previous part. Here high cognitive capabilities are discussed, they require internal representations about the agent, contextualized within its environment and subject to its current motivation. Starting from an introduction on behaviour-based and knowledge-based approaches, here a bottom-up approach for cognitive control is introduced, referring in particular on how to reach cognitive behaviours grounding on reactive systems. This original approach is based on the introduction of the so-called Situation models. These develop upon the network structure for locomotion (Walknet) introduced in the first part of the book. According to this approach, cognition is defined as the capability of planning ahead; this cannot be separated from low-level behaviours, and tightly related to the presence of a “body that perceives” and the peculiar environment situation. Since the approach is mostly based on the theory of special Recurrent Neural Networks (RNN), mathematical details are also reported to formally assess the network capabilities in learning and representation of dynamic situations. Formal details include also other topics that, even if they are not yet discovered in living beings, nevertheless help in dealing with real sensors, in particular in situations where noise occurs at a considerable level. For this aim a probabilistic approach is introduced for modeling the sensor and the motor layer. Additional to bacteria chemotaxis, a short term memory is introduced into the sensory motor loop, leading to the concept of memotaxis. This strategy was experimentally tested in a real robot, demonstrating a great improvement in really adverse situations. The second part of the book reports other approaches to model reactive and precognitive behaviours, among which the so-called weak-chaos control strategy, inspired by W. Freeman theories based on experiments on the rabbit olfactory bulb. Perception is modeled as a process of spatial temporal patterns emerging as stable limit cycles from a chaotic attractor, representing the whole reservoir of all perceptual information, controlled through the sensory stage. The particular emerging cycle acts as a fingerprint of the environment situation, recorded through sensors. A meaning is then assigned through the association of a successful action learned by using a simple reward-based method. A simple application to navigation control shows the suitability of the approach to learn general action maps in real time on board of a robot prototype. Another efficient approach for building a cognitive architecture is represented by a correlation layer. This was realised using a network of spiking neurons, endowed with a simple spike-based learning algorithm. The approach allows to train a high level sensor using information from a lower level one. Examples are reported where avoidance based on distance sensors is learned through contact sensors, or where a visual object is recognised as a target, based on lower level, simpler sensor information. This efficient real time approach, is very useful in many cases: for example, the network could be continuously trained to cope with a serious problem like sensor degradation. Another efficient solution to the homing problem is also given exploiting the capabilities of the RNNs
X
Preface
introduced in the previous chapters. The nest position constitutes the equilibrium of the network, that is able to relax to this position starting from any initial condition, represented by the current pose of the robot searching for home. An excellent degree of robustness against noise and partially obscured landmarks increase the suitability of the strategy. An additional reliable quality is added to the approach: it consists in selecting reliable landmarks among many landmark candidates, and filtering out unreliable (i.e. moving) ones. The last chapter of Part II presents a high level approach to robot perception, based on the theory of complex dynamics. Nonlinear partial differential equations, discretised on a space lattice, are used as a complex system where steady state solutions emerge from initial conditions. The latter are signals coming from the sensor stage. The emerging solutions are so associated to a particular environment portrait. Being such solutions equilibrium points in a dynamical system, they are associated to a particular basin of attraction, (i.e. set of all the environment conditions leading to the emergence of that solution). A suitable learning at the sensor level, using ad hoc designed neurons, contributes to shape the geometry of each basin of attraction. In this way, at the end of the learning stage, all the environment conditions that require the same action are associated to the same basin of attraction, leading to the emergence of a solution associated (through a reward based learning at the efferent stage) to that action. The emerging solutions have been shown to be formally equivalent to the so-called Turing Patterns, the well-known morphogenetical pattern described by Alan Turing in 1952, which arise in spatial-temporal dynamical systems, the so-called reaction-diffusion systems. The efficiency of the Turing pattern approach to perception was demonstrated through experiments on roving robots. Based on these results and on all the other models for the parallel sensory motor loops (sensory systems, locomotion networks, correlation layer and memory), a complex control architecture for action oriented perception, including cognitive capabilities is reported at the end of Chapter 7. This is called SPARK architecture. Here the parallel sensory motor pathways looping through the environment and acting at the lowest level as pre and proto-cognitive behaviours, are coordinated by higher layers of increasing complexity whose capabilities are incrementally learned, guided by a motivation, describing the robot mission. The main higher layers are a correlation layer for anticipation and a representation layer, where emerging patterns are used to modulate basic behaviours. This architecture enables a robot to exploit the capabilities of basic behaviours, during the initial stages of learning phase, to perform basic and safe actions. The higher layers incrementally allow the robot to reach more complex behaviours that better satisfy the robot motivation. Turing patterns, at the end of the learning phase, embed a coded representation of the environment with associated a suitable modulation of the basic behaviour for best matching the robot mission. Under this point of view, within this layer, a representation of the environment is incrementally formed. The architecture introduced is in full accordance with the arguments treated in Part I of the book. In fact the abstract (pattern based) model built is strictly dependent to the robot mission, and so not a general one. Moreover, while an internal model is not needed for the realisation of the basic (reflex based) behaviours, a more complex, context dependent representation of the environment is needed to gain cognitive capabilities. These are incrementally built upon the basic behaviours and help the robot to solve the given mission in an even
Preface
XI
increasing efficiency. Also in this case the suitability of the approach is testified by a number of experimental results where the added value given by the representation layer is confirmed with the capability of the robot to escape from being trapped into local minima, unavoidable without an intelligent, context dependent, modulation of the basic behaviours. During the thorough study that led first to the introduction of the insect brain model and then to the SPARK cognitive architecture, it was observed that a number of perception relevant details of the insect brain are still lacking, in particular referring to the role played by the Mushroom Bodies (MB) and the Central Complex (CX). Progressively discovering the architecture and the functional role of these neural centers will be precious to progressively refine the SPARK architecture. This is a topic of actual deep research that is expected to play an impact on the capability of future cognitive robotic architectures. The last part of the book Part III Software/Hardware cognitive architecture and experiments is related to the practical but not less important part of the project activities. In particular the software and hardware architecture where the cognitive model was implemented are reported. These include the new visual sensor/processor used for complex visual routines implementation. Both the architectures and the visual perceptual algorithms are reported in details. The SPARK board, the custom hardware board based on a powerful FPGA plus additional memory and a huge number of IO pins, is reported and described in details. Finally, the main actors of the project are described. These are the various robotic platforms that were used to implement the various tiles that constitute the complex cognitive model. A number of experimental results are finally reported, whose related multimedia material can be found in the SPARK web page: www.spark.diees.unict.it
March 2008
Paolo Arena Luca Patan´e
Acknowledgements
Editors would like to thank all the partners that contributed with their expertise to the success of the project, with a professional and proactive collaboration. A sincere gratitude is expressed to the project reviewers, Prof. Gurvinder S. Virk and Prof. Ronald Tetzlaff, for their steering comments that greatly contributed to the research activity during the 36 months of research. For the great importance that he gave to the project, a special thank is addressed to the EC Project Officer, Dr. Hans-George Stork, for his continuous, constructive comments and encouragement, that was essential in our research project. We think that without such a support from the EC, we would never been able to work together in such a way, enjoying while hard working. A special thank is addressed to Prof. L. Fortuna, Dean of the Engineering Faculty of the University of Catania, who, through his continuous support, encouraged the formulation and submission of the SPARK Project proposal, supervising and stimulating with new ideas all the steps of the research activity during the project development.
Contents
Part I: Systems 1
2
Perception for Action in Insects . . . . . . . . . . . . . . . . . . . . . . . . . . . . B. Webb, J. Wessnitzer 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 The Traditional View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Perception as Transformation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Closing the Loop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4.1 Active Perception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4.2 Dynamical Systems Theory and Perception . . . . . . . . . . . . 1.4.3 Dynamics and Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4.4 Further Bio-inspired Architectures for Perception-Action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Predictive Loops . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.6 Perception for Action in Insects . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.7 Basic Physiology and the Central Nervous System . . . . . . . . . . . . 1.8 Higher Brain Centres in Insects . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.8.1 The Mushroom Bodies (Corpora Pedunculata) . . . . . . . . . 1.8.2 The Central Complex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.9 Towards ‘Insect Brain’ Control Architectures . . . . . . . . . . . . . . . . . 1.10 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Principles of Insect Locomotion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . H. Cruse, V. D¨ urr, M. Schilling, J. Schmitz 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Biological Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3 3 3 6 8 8 10 11 14 16 17 19 21 21 27 30 33 35 43 43 44
XVI
Contents
2.3 Sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.1 Mechanosensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.2 Environmental Sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Leg Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.1 Swing Movement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.2 Stance Movement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 Coordination of Different Legs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 Insect Antennae as Models for Active Tactile Sensors in Legged Locomotion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.7 Central Oscillators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8 Actuators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.9 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Low Level Approaches to Cognitive Control . . . . . . . . . . . . . . . . B. Webb, J. Wessnitzer, H. Rosano, M. Szenher, M. Zampoglou, T. Haferlach, P. Russo 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Sensory Systems and Simple Behaviours . . . . . . . . . . . . . . . . . . . . . 3.2.1 Mechanosensory Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 Olfactory Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.3 Visual Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.4 Audition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.5 Audition and Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Navigation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 Path Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.2 Visual Homing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.3 Robot Implementation and Results . . . . . . . . . . . . . . . . . . . 3.4 Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.1 Neural Model and STDP . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.2 Non-elemental Associations . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.3 Associating Auditory and Visual Cues . . . . . . . . . . . . . . . . 3.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
48 49 51 52 52 57 65 75 78 83 85 86 97
97 98 98 100 102 115 123 129 129 137 143 156 157 158 163 166 167
Part II: Cognitive Models 4
A Bottom-Up Approach for Cognitive Control . . . . . . . . . . . . . . H. Cruse, V. D¨ urr, M. Schilling, J. Schmitz 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Behavior-Based Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 A Bottom-Up Approach for Cognitive Control . . . . . . . . . . . . . . . 4.4 Representation by Situation Models . . . . . . . . . . . . . . . . . . . . . . . . 4.4.1 Basic Principles of Brain Function . . . . . . . . . . . . . . . . . . . .
179 180 181 185 188 190
Contents
4.4.2 Recurrent Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.3 Memory Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.4 Recurrent Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.5 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.6 Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Towards Cognition, an Extension of Walknet . . . . . . . . . . . . . . . . . 4.5.1 The Reactive and Adaptive Layer . . . . . . . . . . . . . . . . . . . . 4.5.2 Cognitive Level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
6
Mathematical Approach to Sensory Motor Control and Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M.G. Velarde, V.A. Makarov, N.P. Castellanos, Y.L. Song, D. Lombardo 5.1 Theory of Recurrent Neural Networks Used to Form Situation Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.1 RNNs as a Part of a General Memory Structure . . . . . . . . 5.1.2 Input Compensation (IC) Units and RNNs . . . . . . . . . . . . 5.1.3 Learning Static Situations . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.4 Dynamic Situations: Convergence of the Network Training Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.5 Dynamic Situations: Response of Trained IC-Unit Networks to a Novel External Stimulus . . . . . . . . . . . . . . . . 5.1.6 IC-Networks with Nonlinear Recurrent Coupling . . . . . . . 5.1.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Probabilistic Target Searching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.2 The Robot Probabilistic Sensory - Motor Layers . . . . . . . . 5.2.3 Obstacles, Path Complexity and the Robot IQ Test . . . . . 5.2.4 First Neuron: Memory Skill . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.5 Second Neuron: Action Planning . . . . . . . . . . . . . . . . . . . . . 5.2.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Memotaxis Versus Chemotaxis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.2 Robot Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.3 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
XVII
192 192 195 199 203 207 208 209 215 216
219
219 219 220 223 229 236 242 246 249 249 250 253 254 257 259 260 260 261 265 266
From Low to High Level Approach to Cognitive Control . . . . 269 P. Arena, S. De Fiore, M. Frasca, D. Lombardo, L. Patan´e 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269 6.2 Weak Chaos Control for the Generation of Reflexive Behaviours . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
XVIII
7
Contents
6.2.1 The Chaotic Multiscroll System . . . . . . . . . . . . . . . . . . . . . . 6.2.2 Control of the Multiscroll System . . . . . . . . . . . . . . . . . . . . . 6.2.3 Multiscroll Control for Robot Navigation Control . . . . . . . 6.2.4 Robot Navigation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.5 Simulation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Learning Anticipation in Spiking Networks . . . . . . . . . . . . . . . . . . 6.3.1 The Spiking Network Model . . . . . . . . . . . . . . . . . . . . . . . . . 6.3.2 Robot Simulation and Controller Structure . . . . . . . . . . . . 6.3.3 Spiking Network for Obstacle Avoidance . . . . . . . . . . . . . . 6.3.4 Spiking Network for Target Approaching . . . . . . . . . . . . . . 6.3.5 Navigation with Visual Cues . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Application to Landmark Navigation . . . . . . . . . . . . . . . . . . . . . . . . 6.4.1 The Spiking Network for Landmark Identification . . . . . . 6.4.2 The Recurrent Neural Network for Landmark Navigation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.3 Simulation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
272 272 275 276 278 279 281 284 286 289 293 295 297
Complex Systems and Perception . . . . . . . . . . . . . . . . . . . . . . . . . . . P. Arena, D. Lombardo, L. Patan´e 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Reaction-Diffusion Cellular Nonlinear Networks and Perceptual States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 The Representation Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.1 The Preprocessing Block . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.2 The Perception Block . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.3 The Action Selection Network and the DRF Block . . . . . . 7.3.4 Unsupervised Learning in the Preprocessing Block . . . . . . 7.3.5 The Memory Block . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4 Strategy Implementation and Results . . . . . . . . . . . . . . . . . . . . . . . 7.5 SPARK Cognitive Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6 Behaviour Modulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6.1 Basic Behaviors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6.2 Representation Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.7 Behaviour Modulation: Simulation Results . . . . . . . . . . . . . . . . . . . 7.7.1 Simulation Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.7.2 Learning Phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.7.3 Testing Phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Appendix I - CNNs and Turing patterns . . . . . . . . . . . . . . . . . . . . . . . . . Appendix II - From Motor Maps to the Action Selection Network . . .
309
298 301 305 306
309 311 312 313 313 320 321 323 325 330 333 333 334 334 334 336 336 337 338 340 344
Contents
XIX
Part III: Software/Hardware Cognitive Architecture and Experiments 8
New Visual Sensors and Processors . . . . . . . . . . . . . . . . . . . . . . . . . 351 L. Alba, R. Dom´ınguez Castro, F. Jim´enez-Garrido, S. Espejo, S. Morillas, J. List´ an, C. Utrera, A. Garc´ıa, Ma.D. Pardo, R. Romay, ´ Rodr´ıguez-V´ C. Mendoza, A. Jim´enez, A. azquez 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351 8.2 The Eye-RIS Vision System Concept . . . . . . . . . . . . . . . . . . . . . . . . 353 8.3 The Retina-Like Front-End: From ACE Chips to Q-Eye . . . . . . . 356 8.4 The Q-Eye Chip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 358 8.5 Eye-RIS v1.1 Description (ACE16K Based) . . . . . . . . . . . . . . . . . . 362 8.5.1 Interrupts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364 8.6 Eye-RIS v1.2 Description (Q-Eye Based) . . . . . . . . . . . . . . . . . . . . 364 8.6.1 Digital Input/Output Ports . . . . . . . . . . . . . . . . . . . . . . . . . . 366 8.7 NIOS II Processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367 8.7.1 NIOS II Processor Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368 8.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368
9
Visual Algorithms for Cognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . ´ Zar´ A. andy, Cs. Rekeczky 9.1 Global Displacement Calculation . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2 Foreground-Background Separation Based Segmentation . . . . . . . 9.2.1 Temporal Foreground-Background Separation . . . . . . . . . . 9.2.2 Spatial-Temporal Foreground-Background Separation . . . 9.3 Active Contour Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4 Multi-target Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10 SPARK Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . L. Alba, P. Arena, S. De Fiore, L. Patan´e 10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2 Multi-sensory Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2.1 Spark Main Board . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2.2 Analog Sensory Board . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3 Sensory System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
371 371 374 375 376 377 380 383 383 385 385 386 387 388 391 396 397
11 Robotic Platforms and Experiments . . . . . . . . . . . . . . . . . . . . . . . . 399 P. Arena, S. De Fiore, D. Lombardo, L. Patan´e 11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 399 11.2 Robotic Test Beds: Roving Robots . . . . . . . . . . . . . . . . . . . . . . . . . . 400
XX
Contents
11.2.1 Rover I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2.2 Rover II . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 Robotic Test Beds: Legged Robots . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3.1 MiniHex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3.2 Gregor III . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4 Experiments and Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4.1 Visual Homing and Hearing Targeting . . . . . . . . . . . . . . . . 11.4.2 Reflex-Based Locomotion Control with Sensory Fusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4.3 Visual Perception and Target Following . . . . . . . . . . . . . . . 11.4.4 Reflex-Based Navigation Based on WCC . . . . . . . . . . . . . . 11.4.5 Learning Anticipation via Spiking Networks . . . . . . . . . . . 11.4.6 Landmark Navigation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4.7 Turing Pattern Approach to Perception . . . . . . . . . . . . . . . 11.4.8 Representation Layer for Behaviour Modulation . . . . . . . . 11.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
400 402 402 402 404 405 405 408 409 411 413 415 416 420 422 422
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423 Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425
List of Contributors
Paolo Arena, Sebastiano De Fiore, Mattia Frasca, Davide Lombardo, Luca Patan´e Department of Electrical, Electronic and System Engineering, University of Catania, I-95125 Catania, Italy
[email protected],
[email protected],
[email protected],
[email protected],
[email protected] Barbara Webb, Jan Wessnitzer, Matt Szenher, Markos Zampoglou, Thomas Haferlach, Hugo Rosano Institute of Perception, Action and Behaviour, University of Edinburgh
[email protected],
[email protected] Holk Cruse, Malte Schilling, Josef Schmitz University of Bielefeld, Department of Biological Cybernetics and Theoretical Biology, P.O. Box 100131, D-33501 Bielefeld, Germany
[email protected],
[email protected],
[email protected]
Volker D¨ urr University of Cologne, Institute of Zoology, Weyertal 119, D-50931 K¨ oln, Germany
[email protected] Manuel G. Velarde, Valeri A. Makarov, Nazareth P. Castellanos, Yong-Li Song Instituto Pluridisciplinar, Universidad Complutense de Madrid, Paseo Juan XXII 1, 28040 Madrid
[email protected] L. Alba, R. Dom´ınguez Castro, F. Jim´enez-Garrido, S. Espejo, S. Morillas, J. List´ an, C. Utrera, R. Romay, C. Mendoza, A. ´ Rodr´ıguez-V´azquez Jim´enez, A. AnaFocus (Innovaciones Microelectr´ onicas S.L.) Av. Isaac Newton 4, Pabell´ on de Italia, Planta 7, PT Isla de la Cartuja, 41092, Sevilla
[email protected] ,
[email protected] AKos Zar´andy, Csaba Rekeczky AnaLogic Computers Kft., Vahot u. 6, H-1119 Budapest, Hungary
[email protected],
[email protected]
Part I
Principles of Insect Perception
1 Perception for Action in Insects B. Webb and J. Wessnitzer Institute of Perception, Action and Behaviour, University of Edinburgh
[email protected],
[email protected]
Abstract. We review the concept of ‘perception for action’, contrasting the traditional view of perception as internal representation with the idea of transformation in a closed loop system. This introduces recent approaches using active perception, dynamical systems theory, actionbased agent architectures and consideration of the role of predictive loops. We then apply these ideas to insect behaviour and neurophysiology, with particular attention to higher brain centres. We propose an insect brain control architecture for robotics.
1.1 Introduction The idea of ‘perception for action’ has received wide and varied discussion. Our aim in this chapter is to present a general guide to this discussion, highlighting certain issues, and to then examine in more depth what can be learnt about these issues from insect behaviour and neuroethology. It is probably worth specifying at the outset that we do not intend to address issues of perceptual awareness, qualia or consciousness. We are interested in perception from the point of view of explaining how animals do it and how robots could use it. This in itself is one meaning of perception for action, i.e. we are interested in perception in terms of how it can be used (to control action) rather than as an end in itself, as it is commonly discussed within psychology, for example. We can think of the general problem of perception, cognition, and action of an agent, a robot or animal in terms of a loop through the system and the environment (Fig. 1.1) where A is a ‘transfer function describing how sensory events are transformed into actions, and E a ‘transfer function’ describing how the motor output of the system, and any external disturbance, is transformed by the environment into new sensory inputs.
1.2 The Traditional View The traditional view in Artificial Intelligence and Cognitive Science focuses on the agent A and decomposes the ‘transfer function’ into perceiving → thinking → acting. The purpose of perception, thus conceived, is to deliver or update an internal model of the environment E (or some part of E) that can be manipulated to determine actions. Or, in different words, the perceptual process produces a internal state that represents E, and some process on that state will produce the appropriate action. The former description P. Arena and L. Patan`e (Eds.): Spatial Temporal Patterns, COSMOS 1, pp. 3–42. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com
4
B. Webb and J. Wessnitzer
Sensors
A
Motors
E Disturbances
Fig. 1.1. General issue of perception for action: A is a transfer function (via the agent) from sensors to motors, and E a transfer function (via the environment) from motor output, plus external disturbances, to sensors
tends to imply that the internal model is symbolic and can be manipulated by formal inference rules; the latter is more neutral with respect to the nature of the state and the processing. However both imply that there is necessarily some central stage, between sensing and acting, where the (action-relevant) contents of the environment are represented. An issue that thus arises is whether and how the environment needs to be ‘re’presented within the agent. It would seem in many cases that a process on the sensory input itself could suffice to produce the appropriate action. The idea of an internal model was introduced by Craik [28] explicitly for the purpose of explaining how intelligent agents are able to act in ways that depend on more than the current sensory input, e.g. to make a plan to obtain something it currently cannot sense (Fig. 1.2). In this sense, a defining function for the model or internal representation is that it can be used ‘off-line’, so, a simple activation trace of previous sensory stimuli is not sufficient to count as an internal model of the environment. However, as several authors (e.g. [52, 86]) have noted, this distinctive and interesting sense in which there is a claim that brains ‘represent’ is not always clearly understood, because of the common use of ‘representation’ to refer to any internal state that causally co-varies with an external state, such as the activity of a brain neuron that is proportional to velocity of visual motion, or a non-linear transform of the voltage level in a robot sensor. This latter ‘causal’ sense of representation has different explanatory force to the intentional sense of representation, in which something (the internal model) stands in for something else (the currently absent sensory stimuli). A resulting problem is blurring of the important issue of determining which behaviours actually require internal models. It could be said that the motivation of Newell and Simon [129] in introducing their ‘symbol system hypothesis’ was to distinguish ‘cognitive’ (or intelligent) behaviours that seem to require internal models, from mere ‘perception-action’ control1. Following this view, one (but not the only) function of perception is to generate an internal model, for those tasks that require it. However, in 1
Current criticisms of this approach often elide the distinction by re-defining ‘cognition’ and ‘intelligence’ to include the most basic forms of ‘perception-action’ tasks, many of which can, of course, be accomplished without internal models (e.g. [6]).
Perception for Action in Insects
A
B s(t)
m=f(s(t))
m
C
s(t)
m=f(s(t),s(t−1),...)
m
s(t)
m=f(s(t),s(t−1), ...,i(t),i(t−1),...)
m
D s(t) m=f(s(t),i(t))
m
i(t)
E
5
s’(t) m’=f(s’(t),i(t)) i(t)
i(t) F s(t)
s’(t) m=f(s(t),s(t−1), m ...,s’(t),s’(t−1),...) ...,i(t),i(t−1),...) i(t)
Fig. 1.2. A illustrates a purely reactive system, i.e. the motor output is determined entirely by the current sensory input; applying the same input always produces the same output. B illustrates a system with memory, such that the output depends on the history of sensory input (e.g. the system might show adaption). C illustrates a system in which there are some intrinsic variables (e.g. motivational factors) whose current value, jointly with current sensory input, will determine the output. In D the system in C is dependent on the history of both sensory and intrinsic variables. So far, we do not need to introduce a representational account. In E, the system is operating ‘offline’ using a stand-in s’(t) for the missing sensory input s(t) to produce an imagined output m’. In F, the internal representation, the real sensory input, and the intrinsic variables are combined to determine the output.
practice, this view became distorted in two ways: first, by treating generation of an internal model as the only function of perception, indeed as a definition of what perceptual systems are doing [111]; and second, by often studying this problem in isolation from any particular task, hence assuming a general purpose and veridical model is required, i.e., the aim of perception was understood to be reconstruction of the external cause of the sensory input. In the field of computer vision, for example, a textbook definition of the perception problem [160] is solving the inverse function from the input stimulus to the environmental situation that produced it. An implicit widespread acceptance of this view is found in much of the neurophysiology of perception, with the approach to the whole problem described in terms of decoding [152]. In reaction to this view, it is often pointed out that perceptual processing cannot be well understood in isolation from the task. This is one interpretation of what is meant by the phrase ‘perception for action’. Although often presented as a ‘radical’ insight, in fact
6
B. Webb and J. Wessnitzer
much traditional cognitive science does start from the definition of what kind of model is required for a particular task, and then looks to see how the perceptual system can deliver it. Or, in other words, how the perceptual system can extract task-appropriate information from the environment. For example, the same textbook (Russell and Norvig) lists several computer vision applications and notes: “None of these applications requires extraction of complete descriptions of the environment”. This task-based view can encompass both ‘high-level’ tasks, such as deliberative planning (where most detail of the sensory input may be irrelevant) and ‘low-level’ tasks, like immediate avoidance of a looming object (where again most details of the sensory input may be irrelevant). This action-relevant view of perception is often associated with the notion of ‘affordances’. Gibson defined affordances as “what [the environment] offers the animal, what it provides or furnishes, either for good or ill”. Affordances are thus the opportunities for action in a given situation, and Gibson proposed that these are what we perceive, rather than physical properties of objects and environments [45]. The term is now so widely used that its precise meaning is not clear. [Existing models of affordances tend to be for very limited action set and basically becomes idea of using success of some action as a means of classifying sensory input, see e.g. [26, 27]. An argument that is sometimes presented, usually by those who consider high-level tasks to be important, is that because sensory input has to be used in many tasks, perception does need to deliver a ‘general purpose’ model of the world, hence reconstruction is the right way to understand the purpose e.g. [198]. However this seems to miss the point, as it just puts the issue of how to get the task relevant information back one step it now has to be extracted from the internal world model instead of from the world. It is a valid point that we, as cognitive agents, do seem to be (sometimes) able to re-utilise internally stored sensory information for novel tasks, but the extent to which we can do so, and the extent to which any other animal can do so, is far from clear; and thus it seems unjustified to take this as a general characterisation of the basic functioning of perceptual systems.
1.3 Perception as Transformation A different criticism of the traditional approach is that to view perception as only, or even mostly, a problem of reconstructing the environment is to miss the essential point that the actual purpose of perception is to transform the sensory signal. The agent does not want to reconstruct the cause of the signal, but rather it wants to use the signal to produce some motor output (irrespective of whether this is a direct reaction or some distant plan). It is possible, but far from given, that building an internal model will be necessary to produce the right output. Given that action or task-relevant transformation is the purpose, we can argue that perceptual systems are constructed so as to carry out this transformation - even, that the ‘output’ of perception is action, as there is no other clear end-point of the perception process. A simple example is provided by the results of training or evolving neural networks to control robots in tasks like obstacle avoidance or discrimination of targets - the resulting networks can successfully produce appropriate actions but resist any decomposition of the network’s operation as construction and
Perception for Action in Insects
7
manipulation of an internal model, or even as ‘extracting information’ by which to choose an action [132, 82]. One aspect of perception that this viewpoint helps to emphasise is that much of the processing can occur at a very peripheral stage. For example, the physical layout of the sensors may already represent task specific demands, a principle discussed by Wehner [193] as ‘matched filtering’ in biological systems (see examples in chapter 3). Similarly, as the motor interface determines the end-point of perception, its design can be critical to understanding the process: for example it may simplify any co-ordinate transformation required from the sensors to the actuators, or eliminate some control problems (an impressive illustration of this is provided by ‘passive walkers’ [23]). Interestingly, there are several lines of evidence that suggest close involvement of the motor system in our ability to perceive [81], for example the suggestion that motor production of language is involved in recognition of phonemes [104, 105] or the ’mirror neuron’ system in which observing and performing an action appear to activate the same neurons [153]. This is also related to Gibson’s more radical views of ‘direct’ perception — that from the sensory array, invariants are extracted predicting the function or nature (i.e. the affordance) of an object. Gibson argues these invariants are extracted without any complex cognitive processing but rather that perceptual systems ‘resonate’ to the invariant structure, are ‘tuned’ to it. Invariants occur because sensory information is constrained by the way it arises during interaction with the environment, for example, the flow-fields that occur during locomotion, which can be used directly for control [163]. Tuning to such higher-order invariants in the perceptual array can help explain a wide variety of adaptive responses [19], such as catching a ball. Gibson was never specific about what actual mechanisms might be required for ‘resonance’ but the notion is suggestive of both ‘matched filtering’ and of dynamical systems theory (discussed below). As perception is used in multiple tasks and for multiple purposes, it does not make sense to think of it as a single transformation, but as multiple parallel transformations. This echoes Brooks’ [15] well known arguments for horizontal rather than vertical decomposition of complex behavioural capabilities. As usefully summarised in the critique by Kirsh [89] this view is that: 1. Behaviour can be partitioned into a set of task-oriented activities, each with their own sensing and control requirements. 2. There is a partial ordering among these, so a system can be constructed in layers from simpler to more complex activities. 3. Information is available in the world so internal models are unnecessary. 4. Smart sensors can be used to sample the relevant subset of the world. 5. A remaining hard problem is how to co-ordinate the activities. An interesting extension of this issue to higher level cognition is found in the idea that - insofar as thinking involves the construction of models for off-line reasoning those models are effectively simulations of lower-level sensorimotor capabilities. Thus, it is argued, we consistently use body-dependent analogies in even our most abstract thought [95]. This usefully suggests a route for ‘scaling up’ from basic sensorimotor competence to intelligent behaviour ([51, 29] - see section 2.5). There is currently much
8
B. Webb and J. Wessnitzer
interest in examining how language use might emerge via categorisation of sensorymotor contingencies [173]. ’Cognition is not a phenomenon which can successfully be studied while marginalising the roles of body, world and action’ [19].
1.4 Closing the Loop All the above is still only about the sensors → A → motors part of the system. A further, interrelated critique of the traditional approach is its tendency to ignore the closed-loop nature of the perceptual process [144]. For many tasks this is a sufficiently tight loop that the performance of the behaviour cannot be properly understood unless it is regarded as a whole. The paradigm case is a simple feedback system where (again) no internal model manipulation is required. But more generally, almost all real cognitive tasks depend in various ways on this loop. Examples from a high-level perspective are things like actively seeking information, and using things in the world to help thinking e.g. the use of spatial arrangements of objects to aid reasoning [90]. 1.4.1
Active Perception
Active vision is a very successful example of the application of this principle. Ballard [4] discusses in particular the advantages of a visual system that has gaze control, in allowing it to search, use specific movements to simplify processing, and to use exocentric co-ordinate frames. This can lead to some very efficient algorithms. Ballard comments “it may be that the visuo-motor system is best thought of as a very large amount of distinct special-purpose algorithms where the results of a computation can only be interpreted if the behavioural state is known”. A popular example (again originating with Gibson) is optic flow as a cue to self motion. The heading of a moving observer can be estimated by localising the ‘focus of expansion’ (FOE). The FOE is the motion in the optic array expanding out of a singular point along the direction of travel. However, optic flow needs to be distinguished from retinal flow. Eye or head movements induce visual motion on the retina. Retinal flow is thus composed of translational (body movement) and rotational (eye or head movement) elements. In [96], a large body of research from psychophysical studies analysing retinal flow is reviewed, suggesting the visual system combines a multitude of sensory signals for retinal flow analysis including: • efference copies of motor commands, • proprioceptive signals, • visual depth cues. Knowledge of depth perception is thought to be useful for separating translational and rotational flow. The motion of distant points could be used for estimating rotation whereas nearby points could be used for obtaining translational information. On the other hand, motion parallax is a strong cue for depth. Note that this process involves several loops, including internal ones (efference copies), and also several modalities. There is also evidence of ‘direct’ tuning to flow fields, for example in pigeons [205]. Van den Berg and Beintema [10, 8] describe a model combining a template tuned to pure observer rotation, with another sensitive to the derivative of the first template to rotation.
Perception for Action in Insects
9
Auditory perception also involves action, for example in the disambiguation of forward-back sound directions by head movements. Gaver [43] argued for ‘ecological acoustics’ i.e. the importance of interpreting auditory perception as the perception of events, rather than sounds. Rosenblum [158] investigated the auditory equivalent of ‘time-to-contact’ judgements [100, 190] which can be derived from simple cues without independent calculations of distance and speed. Rosenblum et al. [159] looked at the auditory affordance of ‘reachability’ of objects and found subject’s distance judgements in this natural task were much better than expected from previous studies of static auditory distance perception. Tactile perception is also a system that strongly depends on action, as the study of ‘haptics’ has shown. Far more information about object properties - such as weight, texture, consistency, temperature - are obtained from active handling than passive touch, and recognition of the function of objects is greatly improved if they can be manipulated [99]. Turvey [184] has suggested a dynamical systems approach to haptics, looking particularly at how mechanoreceptor information produced when wielding an object can act as invariants to provide size and shape information. Metta and Fitzpatrick [121] have examined how visual learning of categories in a robot can be enhanced by active manual manipulations, such as pushing objects to see how they react. O’Regan and Noe [139] propose that interactions are what constitute perception and that sensorimotor contingencies are what give perception its experiential character — for example, that vision is a mode of exploration of the world that is mediated by knowledge, on the part of the perceiver, of what [they] call sensorimotor contingencies. A sensorimotor contingency is a law or set of laws that describes the relation between self-actions and resulting changes in sensory input. According to this view, perception is an ongoing activity of exploration rather than abstraction of perceptual experience into a final interpreted percept. This view has been implemented in an approach which analyses the laws linking motor outputs and sensory inputs with the aim of discovering information about the structure of the world [145], when no information about the devices making up the body is known a priori. Sensorimotor contingencies contain properties related to the structure of the physical world and an organism’s body which is situated and embodied in the environment. The algorithm aims at discovering and subdividing these properties into parameterisable subgroups. Philipona et al. have shown that the notion of space can thus be generated with a sensorimotor rather than a purely sensory approach. Another sense in which ‘active perception’ is sometimes used is to emphasise that the neural systems involved in perception are not just passive receivers, but actively interact with the input. This is often characterised as ‘top-down’ processes that shape perception by previous knowledge and expectations. Within psychology it is sometimes described as a process of ‘hypothesis testing’ or inference [47, 154] in which different interpretations of the sensory input are (unconsciously) considered for coherence and fit to the data. More recent versions of this idea use Bayesian approaches to describe perception as a process of statistical (rather than logical) inference e.g. [88]. An alternative view of this issue, less dependent on notions of symbolic reasoning, is that of perception as a dynamic system with intrinsic activity that is perturbed by the input. Interestingly, this is a possible means of reconciling constructivist and Gibsonian views.
10
1.4.2
B. Webb and J. Wessnitzer
Dynamical Systems Theory and Perception
The dynamical systems view [6, 147, 7, 171] of cognition sees closing the sensorimotor loop as a paramount concern, and emphasises that this introduces real time considerations into the system. This is more fundamental than saying that the process must operate under time pressure; rather it stresses that the sensing and acting are both continuous processes so their interaction via A and E cannot (in general) be treated as a sequence of computational steps around the loop. This “Radical embodiment” [19] view suggests a different approach to traditional analyses of behaviour, drawing on dynamical system concepts such as state spaces, stability, attractors, limit cycles and chaos. As proposed in [147], the dynamical hypothesis is the claim that: cognitive agents are dynamical systems, and can be scientifically understood as such. This is presented as an alternative to the “symbol system hypothesis” of Newell and Simon, although it has rather different force, as ‘being dynamic’ is not a property that distinguishes cognitive from other systems. The point, rather, is that an computational/symbol-processing account may not suffice to explain what is really going on. There has been a rapid increase in attempts to use non-linear dynamics in the analysis of psychological data [53, 62] including EEG analysis, psychophysics [48], movement control [87, 150], language [130], memory [21], organisations [53], and development [182]. Guastello [53] suggests that several older theoretical perspectives in psychology can be interpreted in terms of dynamics, including Gestalt psychology (which focused on emergent properties in perception), and Piagetian development stages, which he describes as a selforganising process involving approach-avoidance dynamics. Gibsonian approaches to perception also fall fairly naturally within this approach. The main methodological developments in this area are the use of non-linear approaches to time-series analyses, rather than the more typical static group comparisons used in psychological statistics. Heath [2000] provides an overview of these methods, which essentially aim to recover a transfer function in the form of a polynomial, e.g. using non-linear system identification methods or gradient descent approaches. These methods have been more widely applied in analysis of neural data (reviewed in [91, 1]) but can also be used for behavioural data, e.g. reaction times. Heath notes that there is still a problem of interpreting the transfer function once it has been derived, as it may not be easy to determine the underlying mechanisms that produce it. One possibility is to investigate different models of psychological processing by comparing the transfer functions they produce with those obtained from data sets. A common emphasis in these approaches is the issue of identifying deterministic chaotic processes that might underly apparent random noise. This may be investigated by graphical methods, such as phase plots, recurrence plots, Poincar´e plots, to identify attractors, limit cycles and so on. Glazier et al. [46] describe the use of these methods within sports biomechanics, for example. Heath also describes the use of quantitative indices, i.e. attractor dimensionality and Lyapunov components. Though interesting, it is difficult to apply these methods to complex psychological processing, and as yet their use in psychology is not widespread. Kelso [87] describes behaviour as a self-organising pattern, emerging from interaction of subsystems. He argues that this is not merely a high level description of what could be an underlying representational mechanism, but rather points to a different
Perception for Action in Insects
11
mechanism to the usual representational account, one that could be directly implemented in brain dynamics. It suggests that perception, learning, action and memory arise as metastable spatial-temporal activity patterns, produced by cooperative interactions among neural assemblies. For example, reorganisation of topographic maps in motor cortex (after amputation) is related to the temporally correlated activity among neurons. Neural assemblies may be thus constituted and destroyed dynamically, depending on changes of sensory input, thus producing transitions between patterns of activity. The overall behaviour can thus be described by (relatively simple) differential equations; the potential function describes an attractor landscape, that alters with (small number of) control parameters. To date, this has been most successively applied to simple rhythmic motor tasks, such as gait transitions. As an example of a more cognitive task, van Rooij et al. [155] provide a dynamical system theory account of imagined action (a judgement of affordance) to account for hysteresis, enhanced contrast and critical boundaries in decision making. Based on Tuller’s [183] account of speech categorisation, they use the equation: V (x) = kx − 12 x2 + 14 x4 in which varying k produces a function with one or two minima, and the decision trajectory is taken to be a random walk on this landscape; with k dependent on the stimulus and on previous decisions. This work is characteristic of the approach, which is to find a relatively simple non-linear function to characterise behavioural trajectories and then compare the predictions of this model to actual behaviour. If successful, the collective variable V (x) is “assumed to be a non-reducible description of the task specific system”. Very similar approaches can be found in work on collective behaviours of groups of animals e.g. [30] in which a simple differential equation describing arrival/departure dynamics can be shown to predict self-organising aggregation behaviour. McFarland and Boesser [117] present a highly simplified dynamical account of the interaction of basic motivations in animals (e.g. hunger vs. thirst) to account for behaviour choice. A dynamical description of some problem domain does not necessarily account for the underlying mechanisms, i.e. neural processes, but it may well be of use to understand the problem domain itself. This criticism of the dynamical approach is that if high-level causal interactions (between system components, agent-environment, etc.) are described, explanation for underlying neural processes are omitted. The underlying mechanism may then be engineered in an appropriate fashion (evolutionary methods are one popular approach). Eliasmith [34] sees the natural role of dynamical systems theory as one of describing higher-level and temporal behaviours of connectionist networks and suggests that dynamical models and connectionist approaches should be merged in an attempt to achieve more complete explanations. 1.4.3
Dynamics and Networks
An obvious complement to the above methods of analysing psychological data using non-linear dynamics is the development of models of neural dynamics, in particular in the range of network architectures that use recurrence, in contrast to the standard feedforward architectures which do not usually close the realtime loop. There is a great deal of work in this area; the following will highlight some examples, with particular focus on those that have been used in perception for action applications, and without
12
B. Webb and J. Wessnitzer
attempting cover the many theoretical studies of dynamic network behaviours that have been undertaken in the last two decades. Hopfield nets [77], one of the best known recurrent net architectures, use full interconnectivity of the units. Used in pattern recognition tasks, a partial input activating the net will result in it settling in an attractor state that represents the completed input, i.e. acting as an associative memory (where Hebbian learning is used to store the memories). Many variant architectures for associative memory have been constructed, including bidirectional associative nets [92, 137] which have two interconnected layers; Boltzmann machine (statistical rather than deterministic change of weights to avoid settling into local minima). Li and Dayan [103] discuss the potential advantages of using non-symmetric connections between units (in particular distinguishing inhibitory and excitatory units) which is more biologically plausible than the standard symmetric connectivity. They argue this has a particular advantage for selective amplification of particular signals, through the dynamics of delayed inhibitory feedback. Another well known architecture was introduced by Elman [36] using recurrent connections to an extra hidden layer and using backpropagation methods. This can be generalised to the idea of multiple copy layers that record previous instants in time, and be trained using back-propagation through time, or real time recurrent learning. These architectures have been used extensively for sequence learning, particularly in language. Hochreiter and Schmidhuber [69] have introduced an augmented version that attempts to overcome some limitations of this architecture, i.e. difficulty of learning over long intervals for sequence learning. Hoshino [78, 79] uses reciprocally connected networks for features and objects to investigate mechanisms of recognition and attention. The system operates in a ‘randomly itinerate state’ with recognition expressed as a dynamic phase transition to a basin attractor that has been learnt through Hebbian processes. The attractor dynamics can be sufficient for attention focus without explicit top-down signals. The same method can be used as model of multimodal interaction. The method is only tested on simple simulated input, but tries to reproduce psychophysical characteristics of crossmodal facilitation. Tani and Ito [181] propose a simple architecture in which the lower level is a recurrent neural net that receives inputs and generates motor commands, and encodes multiple sensory-motor reflexes or primitives. External ‘control neurons’ connect to all the neurons in the lower level, and their modulation shifts the network from one primitive to another. These control neurons are then embedded in a higher level recurrent network that generates appropriate motor sequences [140]. These levels are (loosely) compared to spinal cord, brainstem and cortex. Grossberg and Carpenter [50, 18, 17] have presented a series of models based on ‘Adaptive Resonance Theory’ (ART - essentially bidirectional nets) and used them to model various aspects of perceptual psychology. They argue that this system can “can self-organize, self-stabilize, and self-scale a sensory recognition code in response to an arbitrary, possibly infinite, list of binary input patterns” and provides a complementary mechanism to error based learning. The basic idea is that when top-down and bottom-up signal patterns match, positive feedback leads to amplification and prolongation of the activation sufficient to allow adaptive learning. Mismatch activates an orienting system
Perception for Action in Insects
13
that resets the top-down expectation, to either find a match, or recruit new cells to learn the new information. Another view is interest in synchrony as a neural processing mechanism, in particular that large scale synchrony may enable global influences on local processing. Engel et al. [37] review some of the evidence for this view: “We discuss recent evidence that predictions about forthcoming events might be implemented in the temporal structure of neural activity patterns. Specifically we focus on spatiotemporal patterns of ongoing activity that translate the functional architecture of the system and the prestimulus history into dynamic states of anticipation”. One proposal, advanced particularly by Freeman [170, 42] is that the critical mechanism for dynamic neural computation is ‘control of chaos’. Based particularly on multielectrode studies of activity patterns in the olfactory bulb, he proposed that it showed a base state of chaotic activity that bifurcates, due to sensory input, to a selected limit cycle attractor. This behaviour can be intepreted as the emergence of an attractor that ”enslaves” the cortical system by acting as an ”order parameter” [55] or the ”emergence of order out of chaos” [151]. Analysing electroencephalographic signals (EEG) from extracranial and intracranial measurements, the focus is on spatial patterns of amplitude modulation of EEG waves in the gamma spectrum (20-80 Hz). These patterns can switch rapidly (due to the chaotic base state) and reflect not simply the stimuli features, but relate to attention, expectancy and learning factors controlled by the limbic system. Freeman argues that as adaptive self-organising systems: “Nervous system dynamics is a self-organized process constrained by the requirement that the system anticipate and incorporate the immediate consequences of its own output within the larger constraints of regulating its well-being and the long-term optimisation of its chances for survival. This is subsumed in J. J. Gibson’s [1979] theory of “affordances”. He thus takes this to be the mechanism by which brains create ‘meaning’ from sensory information. Korn and Faure [91] review the current status of experimental evidence for chaotic brain processes. Some modelling approaches to ‘control of chaos’ also reviewed in [62]. Harter and Kozma [60] have adopted this approach for a robot navigation task. They propose a ‘KIV’ architecture based on the limbic system, consisting of four subsystems: what (sensory/perception); where (orientation/memory); why (goals/drives/value systems); and how (motor/actions). Each involves a similar form of aperiodic dynamics, e.g. to form cognitive maps in the where system, or to control goal-oriented motor output. Eliasmith and Anderson [35] present a systematic view of how different attractor networks can be constructed from spiking elements, and how the dynamics can be controlled. This is a very promising approach that has been applied to several biological control problems, such as integrating multiple sensory sources to obtain an estimate of velocity for path integration [25]. Another interesting development in this area is the idea of combining a recurrent/ chaotic/non-linear network that responds to input with a linear readout system that can learn to classify the patterns. Two versions of this architecture were developed independently as Echo-state networks [85] and Liquid state machines [109]. In the latter, a recurrent microcircuit of spiking neurons acts as an unbiased analog memory. The
14
B. Webb and J. Wessnitzer
resulting ‘liquid state’ x(t) is transformed by a readout map into some target output f (x(t)). Only the readout synapses need to be trained. 1.4.4
Further Bio-inspired Architectures for Perception-Action
In the previous section we discussed several architectures that have been proposed as general accounts of how biological systems might obtain perception for action. Here we summarise a number of alternative approaches that also draw strongly on the ideas of action-oriented perception and closed loop control, and claim to be based (more or less closely) on biology. Brooks’ subsumption architecture was briefly characterised above. This was an extremely influential architecture in robotics, proposed as an alternative to the (then) standard paradigm of a central, task-independent representation [14]. Brooks argued that building more complex behaviours by gradually adding layers of sensorimotor loops was also more biologically justified (as following evolution) although in general a close connection to any particular biological system was not claimed. In the earlier roving robots, biology inspired the tasks rather than the implemented control structures; in more recent work on the COG project [16], more specific attempts to model particular brain systems have been attempted, for example in hand-eye co-ordination and visual attention mechanisms. Emerging quite directly from Brooks’ approach and associated ‘animat’ research has been further investigation of the problem of behaviour co-ordination, which in the subsumption architecture is basically a pre-wired priority ordering between the layers. Typically this is formulated as the ‘action selection’ problem [185], in which the current goals of the system, combined with the current sensory situation, are used to determine which behaviour to activate. Maes [110] provided an early example of a method for flexible action selection and sequencing that did not require explicit planning, but rather was “an emergent property of activation inhibition dynamics amongst the different actions the agent can take”. Different actions are linked as successors, predecessors, or conflictors in a non-hierarchical network, and activation from external situation variables or internal goal variables spreads through the network, with the first action to reach threshold being executed. Though appealing, this approach has not been very successful in application to real robot control. Another class of ‘action selection’ mechanisms uses fusion of behaviours rather than arbitration between behaviours. A well known method, of which there are now several variants, was introduced by Rosenblatt and Payton [157]. Behaviours are represented at a ’fine-grained’ level that corresponds fairly directly to motor commands. All these elements evaluate their relevance to the current situation and goals, and produce a weighted vote for which action to execute. This allows the system to find compromises between behaviours, rather than one behaviour always dominating another. Tyrrell [185] suggested an extension of this architecture, with a hierarchy from simple motor commands, to higher level actions, which he demonstrated in simulation to produce higher competence than Rosenblatt’s original scheme or Maes approach. Another well-known approach to ‘action fusion’ is the ‘potential field method’, used, for example, by [2] in his ‘AuRA’ architecture. Each behaviour produces a vector expressing its desired motion, and these outputs are combined using vector summation and normalisation.
Perception for Action in Insects
15
More recently Gurney, Prescott and Redgrave [149] have proposed an action selection model closely based on the neural architecture of the mammalian basal ganglia. One population of cells encodes the salience of possible actions, due to sensory or other inputs, and the output structures gate the candidate actions by a reduction of inhibition of the winning channels. They note several properties of this brain structure that will contribute to its efficacy as a switching mechanism, e.g. local inhibition that enhances the most active channels, dopamine modulation of selectivity, and a feedback signal that scales the output relative to the number of active channels. They have demonstrated the use of this mechanism for switching between five different behaviours in a robot. Meyer et al. [122] have combined this model with rat-inspired navigation, and also use the dopamine signal to perform actor-critic reinforcement learning of the saliencies, in their ‘Psikharpax’ project to build an artificial rat. A rather different approach to the problem is presented by Sch¨oner, who uses dynamical systems methods [164]. Behaviours are described by differential equations relating a particular behavioural variable (e.g. speed or heading) to its time derivative via a function that describes the task constraints. Typically these describe attractors or repellors in the state space, which are combined additively (possibly with different weights); and the final behaviour is determined by the evolution in time of the solution. An interesting principle used in this method is that the relative domination of different behaviours is determined by differences in the timescales of their dynamics. A more radical approach is suggested by Seth [169], who points out that it may be mistaken to assume a one-toone association of externally observable behaviours - which are a joint product of the agent, environment and observer - with mechanisms within the agent. He demonstrates that a Braitenberg style vehicle with several direct sensorimotor loops can be evolved to produce many features of apparent ‘action selection’ including suitable balance of persistence and switching, prioritisation, opportunism, etc. Arbib has presented an influential view of ‘action oriented perception’ in the form of a network of competitive or co-operative schemas. ‘Schema’ is not well defined, but essentially means a routine, process or control system that has an action-specific function, such as recognising prey or avoiding a predator. Schemas perform neural rather than symbollic computation, and in Arbib’s work they are closely based on biological brain circuits. They can operate in parallel - “there is no one place in the brain where an integrated representation of space plays the sole executive role in linking perception of the current environment to action”; and new schemas arise by modulating existing schemas. This approach has been used in simulation and robot models of frog attraction, avoidance and detour behaviours. Arbib also distinguishes ‘instinctive’ and ‘reflective’ behaviours, suggesting only the latter requires explicit representation, and that there may be an important shift in brain architecture between the special purpose machinery that supports the former and the general purpose machinery (e.g. cortex) that supports the latter. Another interesting architecture has been proposed by Verschure [186]. A reactive control layer implements pre-wired reflexes. Simple sensory events (unconditioned stimuli, US) activate internal state (IS) neurons, which activate motor neurons to perform an unconditioned response (UR). There is a winner-take-all mechanism at the motor stage that selects between actions, with pre-wired interactions between internal
16
B. Webb and J. Wessnitzer
state neurons to resolve conflicts. An adaptive layer is then added, such that the internal state adds the effects of the US and more complex sensory events (conditioned stimuli, CS). The latter inputs have an adaptable weight matrix, that changes according to the difference between the actual and the predicted state of the CS. As a result, the system learns to react to CS. A third ‘contextual control’ layer maintains a short-term memory of CS events and their associated actions, and stores the sequence in long-term memory when a goal is achieved. Then matching of the stored sensory events to current sensory events enables LTM to take over behavioural control if a segment match exceeds a threshold. An important result of this extra control layer is that it changes the behavioural feedback experienced by the system (by repeating successful behaviours) and this indirectly leads to improvement of the perceptual learning of the CS stimuli.
1.5 Predictive Loops So far we have described two kinds of ‘loopiness’ in perception-action systems: the loop through the world, and internal, recurrent connections. It is worth noting here that, in a simple feedback loop system, from the system’s point of view there is no distinction between A and E, because the system cannot see its motor output other than by any effect it has on its sensors. We can simply say the pattern of neural activity at one point in the loop (e.g. the sensors) gets transformed via A and E into new activity (von Foerster, 1985, cited in [146]). If there is a disturbance, the system has no way of distinguishing whether that disturbance occurred in A or E. To provide more sophisticated behaviour than feedback requires additional loops. Loops of particular interest, as they hint towards more cognitive levels of behaviour, are those that allow the agent to predict future sensory input. In some sense, even the principal of matched filtering involves prediction, in that the sensors are adapted (through evolution) to respond directly to task relevant stimuli from the environment. Similarly, a simple memory trace can be thought of as a minimalist prediction of what the next input should be if nothing changes in the world. However, more interesting are the predictions that one sensory input can make about another; and the predicted sensory consequences of actions. In principle, as discussed below, either of these forms of prediction could be hard-wired; but they also correspond to two forms of learning, i.e. CS-US associations, and reinforcement learning. If instead of considering the system as a single loop, but instead (as discussed above) considering multiple parallel loops from sensing, via agent, motors, and environment back to sensing, it is rarely the case that the various sensory inputs are independent. Another way to look at this is that other sensors provide the ‘context’ for any particular sensor. There is ample evidence that the activity of one sensory channel can affect other channels, e.g. by gating, over-riding, or modulating the response. This can happen at many levels in the processing, e.g. at the sensory periphery, centrally, or at the motor output. A simple predictive example is insect antennae: contact of an obstacle by the antennae is likely to be followed by contact with the foot. Woergoetter and Porr [200] suggest that by implementing a principle of disturbance minimisation, one feedback loop (the reflexive lifting of a foot in collision) can be subsumed by another loop, which causes
Perception for Action in Insects
17
the action to occur in advance (i.e. raising the foot in response to the antennae contact). Their learning scheme has the advantage over standard reinforcement learning approaches that the system does not require a ‘actor-critic’ distinction, or prior definition of what is ‘reward’ and ‘punishment’ but simply produces whatever associations will minimise the disturbances within the feedback loops. A complementary form of prediction is provided by the notion of ‘efference copy’ or in control theoretic terms a forward model. In this case, there is an additional internal loop, from the motors to the sensors, which emulates the transfer function E. Such a loop can be used in several ways to make control more sophisticated: • for internal feedback since sensory feedback control may be too slow for rapid movements, • priming the sensory system, anticipate and inhibit the sensory consequences of movements, • provide an estimate feedback and compare to actual sensory feedback and thereby providing appropriate reinforcement for motor learning, and • state estimation combining the model’s prediction with sensory feedback, thereby allowing the evaluations of sensorimotor counterfactuals. This concept is playing an increasing role in neuroscientific explanations of motor control, context dependent action, and cognition [203, 202]. Several authors have proposed internal modelling as a unifying framework for understanding cognition, e.g. [20, 68, 51, 29]. It provides a re-grounding for the idea of internal models discussed above, as elaborated by Grush [52]. He suggests that during overt sensorimotor interaction with the environment, internal neural circuits receive efference copies and run in parallel, learning to predict the expected sensory feedback. These neural models may then be used off-line to estimate or simulate the outcomes of various possible actions. Although much of the interest in such systems is associated with this possible ‘off-line’ use, for example as an explanation of mental imagery (e.g. [197], [93]), it should be noted that the functions discussed above occur in ongoing behaviour, and would seem to be needed for competent control even in ‘minimally cognitive’ systems such as insects or more-than-reactive robots.
1.6 Perception for Action in Insects So far we have provided a theoretically oriented review of the issue of perception for action. The aim has been to show that the notion goes well beyond the idea that perceptual processes deliver action-relevant internal representations of the external environment. Rather, it emphasises that perception is a process of transformation, within closed loops. In particular: • sensory systems have evolved, developed or adapted to respond directly to task relevant stimuli; • there are multiple parallel connections between sensors and motors, rather than a central location where ‘perception’ occurs; • that interaction, i.e. a closed loop of acting and sensing, is critical and changes the nature of the problem to be solved;
18
B. Webb and J. Wessnitzer
• dynamical systems theory has been advanced as an alternative explanatory framework for behaviour to input-output computation; • that recurrent neural architectures have inherent dynamics that are shaped by the sensorimotor context - perception is not a feed-forward process; • prediction between sensory systems and from motor output to expected sensory input are critical mechanisms for obtaining more complex behaviours (rather than action selection). Although many of these ideas have received wide discussion, to date this is almost exclusively in the context of human/vertebrate behaviour and neuroscience. However, if the principals hold, they should be equally discoverable in invertebrate systems. An insect’s sensorimotor system is a product of evolution, development, learning and adaptation which work on different timescales to improve the animal’s behavioural performance. Thus, sensorimotor systems are well adapted to the particular ecological niche of animals. Insects, compared to vertebrates, are relatively simple yet highly successful in their ecological niches displaying complex behaviours supported by sophisticated sensorimotor systems. Insects exhibit rich and interesting behavioural repertoires yet have a central nervous system (CNS) and genome of a size that can be studied thoroughly and systematically. General principles of perception for action may therefore be potentially graspable in these systems before they can be understood in vertebrates. Many insects are capable of impressive orientation, homing and navigation tasks. These abilities are important for a large behavioural repertoire, e.g. the search for food, mates, nests, or reproduction sites. The need for navigation varies from species to species and many different behaviours have been observed. Some may orient themselves to specific stimuli, such as food odours or the auditory call of a potential mate. However, stimuli associated with food or reproduction are not necessarily of immediate importance to homing or generally spatial navigation which take advantage of more accurately identifiable landmarks or reference points, i.e. celestial cues including the sun or sky polarisation patterns. Many species of bees and wasps have been observed to fly in ever expanding circles around their new hive or nest identifying objects and their relation with the hive or nest opening. A particularly impressive and interesting example of navigation of the digger wasp Ammophila is described in [3]. Atkins notes: . . . Ammophila females make similar reconnoitering flights after completing a nest. However, these wasps disguise the nest opening before they leave and return with their prey on foot, often over a considerable distance. This remarkable behaviour of a digger wasp returning to her nest site with a caterpillar prey is reported in more detail in [38]: The Ammophila proceeded straight down between two rows of peas with her caterpillar slung beneath her. When she reached the end of the garden, about twenty feet away, she made a right angle and followed a plow furrow for another five feet. Then she ascended the far side of the furrow and entered a patch of weeds where, with scarcely a hesitation, she dropped her caterpillar and began to dig.
Perception for Action in Insects
19
Such impressive behaviour is of obvious significance to cognitive scientists and neurobiologists. How does the wasp find her way back to nest after long hunting and foraging excursions? Does she relate landmarks and the relative distances between them? A wide range of insects navigate using path integration. On leaving some significant starting point, an animal updates an accumulator constantly adjusting an estimate of distance and direction from the origin, a homing vector. The capability for path integration is mostly studied in the desert ant Cataglyphis, e.g. desert ant navigation [194] and ant odometry in the third dimension [201]. These insects are renowned for returning to their nests after long and tortuous foraging excursions. This behavioural feature is explained by path integration, or dead reckoning, particularly important in unfamiliar terrain. However, the accuracy of path integration decreases with distance travelled due to cumulative errors [13]. In familiar surroundings, navigation by using landmarks may override path integration. Path integration then becomes a back-up strategy that is used when navigation by landmarks fails [22]. Landmark navigation can increase the accuracy of navigation by breaking excursions into segments defined by landmarks. Pattern recognition in insects, as discussed in [64], could mean that sensory recognition of landmarks is a process of linking visual memories to certain motor or behavioural patterns. Could it be a process of retino-topic, image-based or feature-based, matching as proposed in [207] or is there evidence that these animals build cognitive maps [9]? The aim of the following sections is to survey and give an overview of insect nervous systems, and to consider what general common features or principles emerge, following Weiner’s guideline that ‘generality must be discovered, it cannot simply be declared’ [195].
1.7 Basic Physiology and the Central Nervous System An insect’s external anatomy can be divided into the head, the thorax, the abdomen and appendages. Regarding the internal anatomy, only the central nervous system is of concern in this report. Like many arthropods, insects have a nervous system with a dorsal brain and a ventral nerve cord consisting of segmental ganglia extending through the thorax to the abdomen. Ganglia are large aggregates of neurons connected by commissures. The insect brain consists of two head ganglia: the supraesophageal and the subesophageal ganglia. The supraesophageal ganglion consists of three major parts: the proto-, the deuto- and the tritocerebrum. Each part controls a certain spectrum of the insect’s activities. The protocerebrum is largely associated with vision; innervating the compound eyes and the ocelli (simple eyes only capable of perceiving light, darkness and movement, i.e. changes in light intensity). The protocerebrum also hosts higher brain centres, such as the mushroom bodies and the central complex. The deutocerebrum processes information collected by the insect’s antennae. It consists of two distinct neuropils, the antennal lobe (AL) and the antennal mechanosensory and motor centre (AMMC) [75]. The tritocerebrum is said to innervate the visceral system which controls the internal organs within the abdominal and thoracic cavities. The subesophageal ganglion innervates the insect’s mouthparts and salivary glands. In the thorax, pairs of thoracic ganglia control wing and leg movements, usually one for each pair of legs. In the abdomen, abdominal ganglia innervate a large number of sensory receptors located
20
B. Webb and J. Wessnitzer Protocerebrum
Supraesophageal ganglion Deutocerebrum Tritocerebrum Subesophageal ganglion Thoracic ganglia
Commissures
Abdominal ganglia
Fig. 1.3. The nervous system of a generalised insect. The brain (supraesophageal ganglion) consists of proto-, deuto- and tritocerebra and is posteriorly connected to the subesophageal ganglion which in turn connects to the ventral nerve cord linking the thoracic and abdominal ganglia. From [196] (Fig. 1).
on the insect’s back end, such as cerci, genitalia, etc. Insect nervous systems exhibit decentralisation: many overt behaviours (e.g. feeding, locomotion, mating,...) are, to varying extents, controlled by ganglia instead of the brain. The brain may stimulate or inhibit activities in ganglia. Fig. 1.3 illustrates the nervous system of a generalised insect. Insects have hundreds of thousands of receptor cells, varying substantially in numbers both within and across various sensory modalities. An animal’s main sensory receptor cells include photoreceptors, mechanoreceptors and chemoreceptors. Photoreceptors to consider here include compound eyes and occelli, found in a variety of insect species. Compound eyes are made up of many facets, called ommatidia. Smell (olfaction) and taste (gustation) rely on chemoreceptors detecting molecules. Mechanoreceptors provide tactile information and proprioceptive cues to the organism, and modified mechanoreceptors are responsible for the transduction of auditory signals. Each of these modalities will be further discussed in later chapters; the following outlines some important common properties. Within sensory modalities receptor cells may be divided into subtypes. Such subtypes are tuned to different qualities, more narrow bands of the stimulus spectrum. For example, olfactory perception combines information received from receptor cells
Perception for Action in Insects
21
differentially sensitive to specific airborne molecules. Similarly, visual perception is integrating information from receptor cells differentially sensitive to specific wavelengths of light. In the honeybee Apis mellifera, the majority of receptor cells (ultraviolet, blue, green receptors) are said to have secondary sensitivities at certain wavelength regions where the other receptor types absorb maximally [119]. This multiplicity of receptor cells may have several reasons as discussed in [31]: (a) extend the range and resolution of sampling by increasing surface area; (b) extend the range of sensors that are discriminating subsets of stimulus qualities, intensities or temporal dynamics; (c) increase sensitivity and accuracy of resolution through response summation; (d) increase robustness of system despite damage to individual sensors; (e) compensate for nonfunctioning developmental stages of sensors; (f) enable the formation of specialised central processing centres with different behavioural functions. Receptor cells are flexible; neuromodulation can, for example, influence a receptor cell’s sensitivity [11] and consequently this flexibility in sensitivity throughout a nervous system can have implications on an organism’s behaviour [12]. In insects, there are profound differences in neural organisation associated with both sex [175] and social experience [66]. For a good introductory review, see [24]. Receptor cells have evolved into intricate filters focusing on particular information right at the periphery of the sensory system (e.g., [94, 172]). In this way, signal processing is often done mechanically through specific setups of receptor cells and an introductory account on these ‘matched filters’ can be found in [193]. A receptor cell transduces the stimulus energy into a change in its membrane potential. These signals encode the detection of specific stimuli and their intensities through adaptation and amplification; elementary signal transduction modules exhibit computational primitives, including: threshold operations, low- and high-pass filtering, saturation, flip-flop behaviour, etc. At the circuit level, common properties, such as divergence and convergence, feedback, receptive fields forming sensory maps, can be observed; signal transductions serve specific purposes, filtering input in space and time in order to extract functionally important stimulus features. Insect visual systems exemplify this ability to filter out task-specific information from the environment.
1.8 Higher Brain Centres in Insects The insect’s protocerebrum contains many complex and little understood neuropils (anatomically distinct dense networks of axons and dendrites). Two architecturally distinctive neuropils, the mushroom bodies and the central complex, have been widely investigated for their apparent role in controlling more complex behaviours. The next sections will review their anatomy and neuroarchitecture, and their possible behavioural functions. 1.8.1
The Mushroom Bodies (Corpora Pedunculata)
1.8.1.1 Anatomy and Connectivity The mushroom bodies are a pair of large and distinctively (mushroom-)shaped neuropils in the insect brain. In honeybees, they together contain around 340,000 neurons (the
22
B. Webb and J. Wessnitzer
bee brain contains approximately 960,000 neurons occupying 1 mm3 [120]). Though smaller in many other insects, they still comprise a significant proportion of the total brain neurons. The mushroom bodies in most insects have a similar and characteristic neuroarchitecture: namely, a tightly-packed, parallel organisation of thousands of neurons, called Kenyon cells. The mushroom bodies are further subdivided into several distinct regions: the calyces, the pendunculus, and the lobes. The dendrites (inputs) of the Kenyon cells have extensive branches in the calyces2 , and the axons (outputs) of the Kenyon cells run through the pendunculus before bifurcating, diverging and extending to form the typical α - and β -lobes3 . Synaptic interconnections between Kenyon cell axons have been reported [65]. The structure is thus summarised by [165]: • ‘the input and the output of activity due to extrinsic fibres [i.e., neurons with cell bodies external to the mushroom bodies] takes place in separate neuropil areas (calyces and lobes)’. • ‘The Corpora pedunculata represent a multi-channel system of intrinsic fibres [i.e., neurons with cell bodies internal to the mushroom bodies] with respect to the afferent [input] and efferent [output] pathways of extrinsic fibres’. Note also that there is considerable divergence (1:50) from a small number of extrinsic input neurons onto the large number of Kenyon cells, and considerable convergence (100:1) from the Kenyon cells onto extrinsic output neurons4. In most insect species, the mushroom bodies receive significant olfactory input. Interneurons relay odourant information from the olfactory receptors, via the antennal lobe in the deutocerebrum, to the mushroom body calyces. In the cockroach and locust, tactile and gustatory input have also been reported. Some Hymenoptera, e.g., bees and wasps, have substantial connections from the optic lobes on both sides of the brain to the calyces of both mushroom bodies [49]. The absence of direct visual input in other species does not necessarily mean visual information is not integrated. Rather, it may be preprocessed in other areas of the protocerebrum before feeding into the mushroom bodies [107]. Neurons in the calyx of the cockroach have been reported to respond to antennal movement; these signals may be proprioceptive in origin [126]. There is also evidence of possible input from circadian clock neurons in Drosophila [67]. Sensory afferents exhibit ordered connections with the calyces of the mushroom bodies. For example, terminals of input neurons onto the calyces of the cockroach and of the honeybee exhibit function-specific distribution patterns [131] and the calyces show a modality-specific topography [74]. At least some of this ordering is preserved by the Kenyon cells. Mushroom bodies in honeybees are subdivided into several sensory compartments [127] and modular structures in the mushroom bodies of the cockroach are reported in [124]. A recent Drosophila study has shown that Kenyon cells are subdivided regularly regarding their gene expression, suggesting extensive parallel compartmentalisation of function [156]. 2 3 4
The calyces may be a later evolutionary addition of a specific input structure to a more primitive mushroom body architecture [177]. There are exceptions to this description of the typical Kenyon cell e.g., not all Kenyon cells bifurcate [39]. These ratios are estimates based on data from [98].
Perception for Action in Insects
23
The activities of extrinsic neurons in the output regions of the mushroom bodies can be classified as (i) sensory, (ii) movement-related and (iii) sensorimotor [135]. A large majority exhibit responses to multiple sensory stimuli and therefore confirm the suggestion that the mushroom bodies participate in sensory integration. For example, in the cricket a single neuron was reported to respond to auditory, visual and wind stimuli [162]. These neurons also exhibit changes in activity and sensitivity levels which can be related to specific stimuli and stimuli combinations. Electrical recordings of single neuron and field potential activities also indicate that processing of sensory stimuli can last from seconds to sometimes minutes after stimulation [65]. This is likely to involve recurrent feedback: it has been shown that extrinsic neurons provide direct recurrent feedback to the calyces [206] and indirect feedback via other areas [127]. However, despite some examples of motor-related activity in extrinsic output neurons (see below), there is little evidence of direct connections to descending neurons. The majority of output pathways target a variety of other protocerebral regions [84, 101, 102]. A good account of the output connections of the mushroom bodies to other parts of the brain can be found in [177]. 1.8.1.2 The Mushroom Bodies Are Secondary Pathways Evidence suggests that mushroom bodies do not form the only sensorimotor pathway for any modality. Most sensory areas in the brain have direct connections to premotor areas, and thus to descending neurons. The same sensory areas also supply afferents to the mushroom bodies (either directly or indirectly) as part of a secondary pathway as shown in Fig. 1.4 [126]. For example, in most insects, the olfactory system consists of two parallel pathways as depicted in Fig. 1.5. The main olfactory pathway is the connection between the antennal lobe and the superior lateral protocerebrum (medial and outer cerebral tract). The mushroom bodies are part of a secondary pathway (inner cerebral tract). Convergence in the superior lateral protocerebrum of antennal lobe outputs and mushroom body outputs provides a locus for comparison of information processed in these neuropils [102]. Similarly to the olfactory pathway, a secondary pathway via the mushroom bodies has been reported for the conditioning of the proboscis extension reflex in the bee [57]. A direct connection from the antennal lobe to the motor neurons of the mouthparts has a secondary parallel pathway branching off via the mushroom bodies. Genetic and developmental impairment of mushroom bodies confirm the hypothesis of parallel secondary pathways. Impaired Drosophila can perceive, but cannot remember, odours and their overall behaviour was described as ‘normal’ [65]. 1.8.1.3 Functional Roles of the Mushroom Bodies The size, distinctive architecture, range of inputs and outputs, and parallel pathway arrangement raise obvious questions about the functional role of the mushroom bodies in insect behaviour. We will here discuss the main current hypotheses. A role in pattern recognition The parallel channels of intrinsic Kenyon cells perform specific processing functions on the mushroom body input. Heisenberg [63] speculated that the large divergence of sensory afferents onto the calyces of the mushroom bodies may form matrices where each element (Kenyon cell) could be specific for unique relationships of excitations in the primary sensory channels feeding the mushroom bodies, indicating a particular
24
B. Webb and J. Wessnitzer Mushroom bodies Calyces
Protocerebrum Visual
A
Lobes
E
Olfactory Contact Chemosensory Premotor
Motor
Tactile
Fig. 1.4. Evidence suggests that mushroom bodies do not form the only sensorimotor pathway for any modality. Sensory areas in the brain have direct connections to premotor areas, and thus to descending neurons in motor areas. The same sensory areas also supply afferents to the mushroom bodies (either directly or indirectly) as part of a secondary pathway. This schematic overview of the connections is drawn after [126]. From [196] (Fig. 3).
stimulus situation. The dendrites of extrinsic (output) neurons, invading the lobes, vary in size and arborisation patterns. Thus each imposes different characteristic filter parameters on signal transmission from the Kenyon cells, depending on how many and which of the Kenyon cells it interacts with [49]. In support of this idea, some extrinsic output neurons have been shown to respond with changes in sensitivity and activity levels to a certain modality only when also presented with another [162]. The topographical relationships between efferent dendrites extending across Kenyon cell axons and how these axons represent afferent projections in the calyces is discussed in [178]. Kenyon cells may also act as delay lines [165, 127] which could provide a mechanism for recognising temporal patterns in the input. Schuermann suggests “the peculiar form of the Corpora pedunculata due to the pedunculi and lobes is . . . a special arrangement of neurons for transposition of synaptic distances into sequential activation or inhibition of nerve fibres”. Some extrinsic neurons have been reported to increase or decrease their activity depending on some preceding stimulus [162]. Note that pattern recognition also has a role in learning as further discussed below - sparse coding of sensory inputs is said to “make neurons more selective to specific patterns of input, hence making it possible for higher areas to learn structure in the data” [138]. More generally, Ito et al. [84] caution that the learning and memory deficiencies observed after disrupting or ablating the mushroom bodies could indicate that
Perception for Action in Insects
25
Mushroom bodies
Multimodal afferents Calyces
inner antennocerebral tract
Lobes
Multimodal afferents
Sensory stimuli put into context
Antennal lobe
medial and outer antennocerebral tract
Superior lateral protocerebrum Integration and discrimination
Premotor areas
Fig. 1.5. Hypothesised integration stages of odour perception with other sensory modalities (redrawn from [102]). Odours are represented by spatiotemporal patterns in the glomeruli of the antennal lobe. The inner antennocerebral tract projects to the calyces and continues into the superior lateral protocerebrum whereas the medial and outer antennocerebral tracts project directly into the superior lateral protocerebrum. In the mushroom bodies, odours are placed into a multimodal context and efferents relay this information into the superior lateral protocerebrum. There, antennal lobe and mushroom body outputs converge and it is speculated the relevance of odours is assessed with respect to the multimodal context, integrated and discriminated before feeding into premotor areas.
their critical role is in preprocessing of signals (such as detecting complex spatiotemporal patterns) for subsequent learning, rather than that they are themselves the central site in which learning and memory take place. A specific example of how the mushroom bodies perform sophisticated recognition is given by studies of the olfactory pathway in locusts [97, 143, 142]. Different odours produce distinctive spatio-temporal patterns in the antennal lobe. Laurent and Perez-Orive hypothesise the Kenyon cells help disentangle the spatio-temporal codes by
26
B. Webb and J. Wessnitzer
operating as coincidence detectors selective for correlations in the input spike trains. Kenyon cells receive direct excitatory input from antennal lobe projection neurons, but also indirect inhibitory inputs from the same neurons via lateral horn interneurons, arriving shortly after the excitation. Thus the integration time for the Kenyon cells is limited to short time windows [41], making them highly sensitive to precise temporal correlations. The suggestion is that while antennal lobe processing supports basic olfactory learning, more subtle odour discrimination tasks require the mushroom bodies [58]. Evidence from bees supports this interpretation. Bees with disrupted mushroom bodies were capable of distinguishing dissimilar odourants, but could not distinguish similar ones [174]. A role in integrating sensory and motor signals Although most discussion of ‘integration’ in the mushroom bodies focuses on the combination of sensory inputs, it should be recalled that they also receive recurrent feedback, and that extrinsic neurons show motor-related responses. In the cockroach Periplaneta americana, some mushroom body extrinsic neurons are reported to exhibit activities for 100-2000ms preceding the onset of locomotion [136]. This is much earlier than activity seen in descending neurons which usually precedes movement by only 10-200ms. The timing also precludes the response being due to feedback from proprioceptive sensors during movement. Okada et al. [135] report extrinsic neuron responses which are selective to the directions of turning behaviours. These researchers propose that the mushroom bodies participate in the integration of sensory stimuli and motor signals, possibly playing a role in initiating and maintaining motor action, or patterning motor output [125]. Supporting evidence is that the mushroom bodies play an important role in the termination of active walking phases in Drosophila [112]. However, ablations of the mushroom bodies did not prevent spontaneous locomotory behaviour, hence it is thought that an indirect pathway involving the mushroom bodies converges with more direct pathways for hierarchical integration and modulation of behaviour, perhaps including the central complex, discussed below. Furthermore, extrinsic mushroom body neurons of cockroaches have been reported to discriminate self-administered antennal grooming from externally imposed antennal stimulation [125]. A fascinating possibility is that the mushroom bodies are more generally involved in integrating efferent copies with reafferent signals, thus enabling insects to learn to discriminate self-stimulation from environmental stimulation [192]. Neural substrates for learning It is widely acknowledged that the mushroom bodies play some important role in associative learning and memory, with evidence from a variety of insect species, insect behaviours, and experimental manipulation methods. For example, mushroom body ablation has been reported to impair short-term and long-term memory of courtship conditioning in Drosophila [116]; genetic mutants with structural α -lobe defects have shown deficiencies in long-term memory [141]. The mushroom bodies of the cockroach Periplaneta americana have been linked to visual place memory [126]. There is a positive correlation between the volume of the mushroom bodies and sophistication of social behaviour in Hymenoptera (c.f., [161]), and mushroom body volume increases are linked to behavioural development [134], e.g., learning the location of the hive when
Perception for Action in Insects
27
commencing foraging [40]. However olfactory learning has been the main focus of investigations to date. Kenyon cells show striking structural plasticity, shedding their fibres and growing new ones at various occasions in an insect’s lifetime [63]. In [44], evidence suggesting the mushroom bodies in Drosophila are the site of a memory trace is reviewed and evaluated. Gerber et al. [44] elaborate on criteria for a memory trace (presence of neural plasticity in the mushroom bodies, the sufficiency and necessity of this plasticity for memory, and whether memory is abolished when input or output to the mushroom bodies is blocked during testing) and conclude that localising the olfactory associative memory trace to the Kenyon cells of the mushroom bodies is a reasonable working hypothesis. Indeed, the Kenyon cells of the Drosophila mushroom bodies have been identified as a major site of expression for a number of “learning” genes (for a review, see [33]). According to Dubnau et al. [33], Hebbian processes underlying olfactory associative learning may reside in Kenyon cell dendrites. However, synaptic plasticity underlying olfactory memory has also been located in the synapses connecting Kenyon cells to extrinsic neurons [168]. Based on the mushroom bodies’ functional anatomy, particular odours are assumed to be represented by specific subsets of Kenyon cells. For any odourant to become a predictor of a given reinforcing event (e.g., sucrose reward), the output synapses of particular sets of Kenyon cells should be modified such that extrinsic mushroom body neurons could then mediate a conditioned response (e.g., approach or avoidance). In [115], the response of one particular extrinsic mushroom body neuron Pe1 was studied extensively in the context of non-associative and associative learning. It was shown to change response in various conditioning paradigms, and hypothesised to be an element for short-term acquisition of an associative olfactory memory in the honeybee. In the honeybee, the role of appetitive reinforcement has been shown to have a neural substrate, an identified neuron called VUMmx1 located in the subesophageal ganglion [56, 57, 120, 118] which arborises into the antennal lobe, the mushroom bodies and the lateral horn. Stimulation of this neuron has the same effect as presentation of a sucrose reward for classical conditioning of the proboscis extension reflex [56, 57]. In the fruitfly, the paired dorsal medial (DPM) neurons arborise throughout the mushroom body lobes. In genetic studies these neurons have been shown to play a modulatory role (said to provide negative reinforcement) in olfactory learning and memory formation [188, 189]. 1.8.2
The Central Complex
While the exact function of the mushroom body is not certain, it is clear that it has major roles in spatiotemporal sensory processing and learning. By contrast, the role of the central complex remains rather more elusive, but seems to be largely concerned with (pre-)motor processing, control of locomotion, and possibly path integration. 1.8.2.1 Anatomy and Connections The central complex has a midline-spanning position in the insect brain and a highly regular neuroarchitecture. The common structure of the central complex among arthropod species is reviewed and discussed in [108]. It is situated between the two brain
28
B. Webb and J. Wessnitzer
hemispheres and consists of a group of four distinct but interconnected neuropils: the protocerebral bridge, the upper division of the central body (in Drosophila also termed the fan-shaped body), the lower division of the central body (in Drosophila also termed the ellipsoid body), and the noduli. These neuropils are connected via columnar interneurons. The neuroarchitecture is defined by 16 columns with 8 in each brain half, and a characteristic connection pattern between the two brain hemispheres, as shown in Fig. 1.6, which has been found in all species studied so far [74]. The central complex is connected to many other protocerebral regions (but only very few connections to the mushroom bodies have been reported c.f., [84, 114]). The protocerebral bridge is considered the main input region [59] with connections from visual regions particularly dominant in locusts and honeybees. Recent research identified memory traces for visual features in the central body of Drosophila [106]. Innervations by ocellar interneurons have been reported in a number of insect species (c.f., [70]) and also from the polarisation vision pathway [74]. Although significant visual input has been shown in a number of insect species, blind insects also have pronounced central bodies which suggests that vision is not the only sensory modality processed by the central complex. Evidence suggests that mechanosensory input is provided to the ellipsoid body from the ventral nerve cord via the lateral accessory lobes in the locust Schistocerca gregaria [72]. Detailed anatomical studies of the central complex suggest the anatomy of the central body is well-suited to overlay inputs (received via the protocerebral bridge) from both brain hemispheres [59, 108]. Insects that perform sophisticated asymmetric leg movements (e.g., weaving, comb-building) appear to have a large or elaborate central complex [176]. The lateral accessory lobes, as shown in Fig. 1.6, are major output targets of the central complex and interact with the central nerve cord through ascending and descending neurons; there is also feedback from the lateral accessory lobes to the central body (or fan-shaped body) [72]. Neurons from both lateral accessory lobes were also reported to be reciprocally connected in the silk moth Bombyx mori [123]. 1.8.2.2 Functional Roles of the Central Complex Almost two decades ago, it was proposed that the central complex with its precise left-right fiber geometry functions as a well-balanced system of inhibitory and excitatory interactions between the two brain halves [70]. Behavioural and genetic studies of Drosophila have associated the central complex with functions related to higher locomotor control; including initiation and modulation of behaviour, and controlling the persistence of behaviours. An excellent review can be found in [179]. It has been proposed that the neurophysiological mechanisms responsible for generating and controlling such highly organised locomotor activity imply a ‘higher decision centre’ organising behavioural activity [113]. Producing behavioural activity patterns Neural activity in the central complex has been linked to the expression of behaviour. Some neurons in the central complex of Drosophila have been reported to represent thoracic motor patterns [71]. Activity changes in the connections between the lateral accessory lobes and the central body were linked to initiation and termination of flight
Perception for Action in Insects
29
in Schistocerca gregaria [72]. Distinct activity patterns (measured by staining methods) for different behavioural situations were reported in [5], not only for flight and walking, but also for different visual stimuli. Bausenwein et al. reported that activity labeling did not reflect the intensity of sensory stimulation but that it reflected distinct activity patterns for different (visually controlled) behavioural situations, and suggest these can therefore be thought of as central representations of behavioural activity. Locomotor coordination and regulation Central complex mutants exhibit a range of locomotion defects. Mutant fruitflies with a disrupted protocerebral bridge were unable to fly and showed abnormal walking and turning behaviours. The protocerebral bridge, and connections from the protocerebral bridge to other regions of the central complex, were reported necessary for maintaining but not for initiating locomotor activity [114]. The protocerebral bridge is involved in regulating and optimising walking speed by controlling step length [179]. In Drosophila, the central complex has generally been found to up-regulate walking activity whereas the mushroom bodies have been shown to down-regulate or suppress locomotor behaviour [112]. Orientation The overlay of signals in this mirror-symmetrical neuropil has been shown to play an important role in goal-directed motion. Mutant flies, with partially interrupted fanshaped and ellipsoid bodies along the midline of the brain, were unable to compensate for existing body side asymmetries whereas flies with an intact central complex could deal with such asymmetries [179]. These mutant flies were unable to approach targets in a straight line. Furthermore, flies with intact central complex continue to move towards a target despite the target becoming temporarily invisible and/or a second target distracting the flies [180]. However, some central complex mutations (ellipsoid body and/or fan-shaped body) cause flies to quickly loose their bearings. The inability of mutants to maintain a bearing towards a target also suggests a role of the central complex for resolving conflicts between sensorimotor pathways competing for behavioural expression. A centre for path integration The central complex has recently been pronounced a promising candidate as a centre for path integration. Vitzthum et al. [187] have identified neurons in the central complex of the locust Schistocerca gregaria sensitive to polarised light - polarised skylight patterns are an important stimulus providing compass information [194] for many insects. Additional visual inputs to the central complex, given its apparent importance in controlling locomotory behaviour (e.g., estimating distance or rotation from optical flow) strengthens the argument for the role of the central complex in path integration [73]. Some inputs to the protocerebral bridge are said to originate from the accessory medulla, the circadian pacemakers in the brains of cockroaches and flies [76, 67]. Orientation using celestial cues requires adjustment for their positional changes over the day; thus these circadian inputs may serve time compensation [74]. Note that such a role for path integration is indeed quite consistent with the idea that the central complex is a higher locomotion control centre.
30
B. Webb and J. Wessnitzer Sensory input
Protocerebral bridge
Central body (upper)
Central body (lower)
Lateral accessory lobes
Descending and ascending neurons (to and from the thoracic ganglia)
Fig. 1.6. Schematic diagram showing the neuroarchitecture of the central complex (redrawn after [73]). The anatomy of the central complex with regular columnar overlay between sides is wellsuited to coordinate inputs from both brain halves. From [196] (Fig. 4).
1.9 Towards ‘Insect Brain’ Control Architectures Based on the preceding information about insect nervous systems, we here propose a general ‘insect brain’ control architecture for obtaining adaptive behaviour in robots. Our basic motivation for making this proposal is that insects work - they have a range of competences that far exceeds current robot capabilities. It seems sensible to take a close look at insect designs when devising solutions to problems in robotics. This is not a new suggestion, but to date the focus has been on specific sensorimotor systems, rather than the overall organisation, or integration of behaviours. This has led to the largely mistaken view that insect control mechanisms can be simply described (sometimes simply dismissed) as a collection of reactive behaviours; and that we must look to mammalian nervous systems, or other engineering approaches such as hybrid control, if we want to ‘scale up’ to complex robot behaviour. Like many mistaken views, it has an element of truth, in that perceptual systems of insects did not evolve to build an internal model of the environment for general action, but rather evolved for particular tasks relevant to the animal and its particular ecological niche. Thus, as discussed in section 2.3, there are many parallel domain-specific sensorimotor pathways. These pathways form the basis of our proposed architecture, and there are already a number of successfully implemented robot systems based on such pathways that we can draw upon. Behavioural decomposition, and distributed control are features of the nervous systems of insects (and also that of vertebrates, c.f., [148]) that resemble the behaviour-based approach to robot control. Behaviour-based systems (in particular Brooks’ subsumption architecture) aim at achieving higher levels of
Perception for Action in Insects
31
converging tracts diverging tract(s)
Visual
Tactile
Chemosensory
Mushroom bodies Premotor integration and discrimination Efferent copy
Central complex
Ganglia
Proprioceptive
Fig. 1.7. An outline of the proposed model is drawn schematically. Reflexive (domain-specific) sensorimotor pathways form the basis of this architecture. For illustration purposes, three direct pathways are shown for different modalities. Sensorimotor pathways for other modalities could be added (or, for a robot architecture, substituted). Indirect secondary pathways (mushroom bodies) achieve context generalisation, associative learning, and modulate reflexive sensorimotor pathways. Additional central coordination (central complex) for integration and persistence of behaviours is necessary and may also be used for path integration. Recurrent connections may serve short-term memory (in both mushroom bodies and the central complex). Efferent copies and forward models will be used to modulate sensory processing according to expected reafference. From [196] (Fig. 5).
competence by adding behavioural modules incrementally. It has been noted that these architectures often fail to scale well as layers are added. In fact, looking at the insect brain, rather than finding ever more layers of ‘behavioural modules’, there appears instead to be two qualitatively different ‘association areas’ (the mushroom bodies and the
32
B. Webb and J. Wessnitzer
central complex) that act to modulate the direct sensorimotor loops in several important ways. We thus propose that the next element of our architecture will be an indirect secondary pathway, which forms a parallel route for sensory inflow. This can be used to place information from various sensory modalities into context, to improve on reflexive behaviours by learning to adapt and anticipate reflex-causing stimuli, or learning to substitute one stimulus for another in guiding action. There is good evidence that the mushroom bodies play such a modulatory role in the insect brain and thus their neural architecture provides a model for designing an associative memory capable of multimodal sensory integration and modulating underlying reflexes in a heterarchical, context-dependant manner. Mathematical descriptions of the ionic currents and the resulting computational properties of Kenyon cells (the intrinsic mushroom body neurons) of the honeybee have already been developed [83, 204]. The mapping of sensory neurons onto Kenyon cells shows high divergence suggesting a form of sparse coding, quite possibly serving the recognition of unique relationships in primary sensory channels or domain-specific sensorimotor loops (as discussed in section 1.8.1.3). Recently, system models of the mushroom bodies for odour classification based on this characteristic neuroarchitecture have been proposed [80, 133]. However, this is not simply a feedforward system but involves multiple recurrent connections that are likely to be essential to its function, perhaps providing a mechanism for short-term memory (and resonating sensory events). Associative memory and learning also benefits from value systems, which in insects appear to be implemented in a small number of neurons with many connections throughout the brain. These signal salient sensory events (such as appetitive reward) and are involved in the plasticity of the mushroom bodies. These value systems report activity in reflex-loops, helping to establish useful CS-US relationships. The third element of our proposed architecture is inspired by the central complex, which forms another parallel processing stream superimposed on more direct sensorimotor loops. The central complex has a remarkable neuroarchitecture and many functions of the central complex in coordinating locomotory behaviour are well described. Central integration of sensory and motor signals is needed to co-ordinate behaviours, overlay signals from the two brain hemispheres, and to maintain or switch between behaviours. The central complex may provide inspiration for solving problems of conflict resolution between behavioural modules competing for control of a limited set of effectors. In vertebrate neuroscience, the basal ganglia has been associated with this problem of ‘action selection’ [54] - the central complex might have a similar function in insects. The central complex may also be involved in path integration, which requires the association of several modalities, in particular polarised light or other compass sensing, and visual or proprioceptive measures of distance. Some computational models of neural networks capable of path integration [61, 199] share features with the central complex, in particular rows of neural elements with specific columnar and global connection patterns [128]. The final and critical loop of our architecture is provided by efference copies of motor commands, which can in principle provide predictions of expected sensory events. This is vital contextual information for distinguishing external disturbances
Perception for Action in Insects
33
from reafference. Such mechanisms play a role in priming sensory systems, to anticipate and inhibit the sensory consequences of movements. In this way, an estimate of feedback can be compared to actual sensory feedback and thereby provide appropriate reinforcement for learning. State estimation can combine the model’s prediction with sensory feedback, thereby allowing the evaluations of sensorimotor counterfactuals. Neural correlates of reward predictors exhibit different reward-related responses depending on whether rewards are expected [120]. Evidence suggests that similar rewardlike reference signals also exist in mammalian brains [166, 167]. It is a very interesting question to determine the extent to which accurate prediction - which implies the use of forward models [192] - actually occurs in insect systems, and whether this might be an additional function implemented in the mushroom bodies or other specific neural centres.
1.10 Conclusion Insects solve difficult behavioural tasks with miniature brains. We have reviewed the current state of neurobiological research on insect nervous systems to identify essential elements of their control architecture. Although much remains uncertain and speculative, nevertheless some interesting general features emerge. In particular, parallel direct sensorimotor loops (or ‘behaviours’) are supplemented by specific brain areas that serve integrative functions relating to context-dependence, learning, and smooth coordination. Moreover these areas have distinctive neural architectures, preserved across all insect species, that suggest particular ways in which these functions might be implemented. Attempting to copy these systems should be productive for robot research; and should also contribute to extending biological understanding of the organisation of behaviour in insects [191, 32].
References 1. Abarbanel, H., Rabinovich, M.: Neurodynamics: nonlinear dynamics and neurobiology. Current Opinion in Neurobiology 11, 423–430 (2001) 2. Arkin, R.: Integrating behavioral, perceptual, and world knowledge in reactive navigation. Robotics and Autonomous Systems 6, 105–122 (1990) 3. Atkins, M.D.: Introduction to insect behavior. Macmillan Publishing Co., Inc., New York (1980) 4. Ballard, D.: Animate vision. Artificial Intelligence 48, 57–86 (1991) 5. Bausenwein, B., Mueller, N., Heisenberg, M.: Behavior-dependent activity labeling in the central complex of Drosophila during controlled visual stimulation. Journal of Comparative Neurology 340, 255–268 (1994) 6. Beer, R.: A dynamical systems perspective on agent-environment interaction. Artificial Intelligence, 173–215 (1995) 7. Beer, R.: Dynamical approaches to cognitive science. Trends in Cognitive Science 4(3), 91–99 (2000) 8. Beintema, J., van den Berg, A.: Heading detection using motion templates and eye velocity gain fields. Vision Research 38, 2155–2179 (1998) 9. Bennett, A.: Do animals have cognitive maps? Journal of Experimental Biology 199, 219– 224 (1996)
34
B. Webb and J. Wessnitzer
10. van den Berg, A., Beintema, J.: Motion templates with eye velocity gain fields for transformation of retinal to head centric flow. NeuroReport 8, 835–840 (1997) 11. Birmingham, J.: Increasing sensor flexibility through neuromodulation. Biological Bulletin 200, 206–210 (2001) 12. Birmingham, J., Tauck, D.: Neuromodulation in invertebrate sensory systems: from biophysics to behavior. Journal of Experimental Biology 206, 3541–3546 (2003) 13. Bisch-Knaden, S., Wehner, R.: Local vectors in desert ants: context-dependent landmark learning during outbound and homebound runs. Journal of Comparative Physiology 189, 181–187 (2003) 14. Brooks, R.: Intelligence without reason. In: Proceedings of IJCAI 1991 (1991) 15. Brooks, R.: Intelligence without representation. Artificial Intelligence 47, 139–159 (1991) 16. Brooks, R.: From earwigs to humans. Robotics and Autonomous Systems 20(2-4), 291–304 (1997) 17. Carpenter, G., Grossberg, S.: Adaptive Resonance Theory, pp. 87–90. MIT Press, Cambridge (2003) 18. Carpenter, G.A., Grossberg, S.: The art of adaptive pattern recognition by a self-organizing neural network. Computer 21(3), 77–88 (1988) 19. Clark, A.: Embodiment: from fish to fantasy. Trends in Cognitive Sciences 3(9), 345–351 (1999) 20. Clark, A., Grush, R.: Towards a cognitive robotics. Adaptive Behavior 7, 5–16 (1999) 21. Clayton, K., Frey, B.: Inter- and intra-trial dynamics in memory and choice. In: Nonlinear Dynamics in Human Behavior. World Scientific, Singapore (1996) 22. Collett, T., Collett, M.: Path integration in insects. Current Opinion in Neurobiology 10, 757–762 (2000) 23. Collins, S., Ruina, A., Tedrake, R., Wisse, M.: Efficient bipedal robots based on passivedynamic walkers. Science 307, 1082–1085 (2005) 24. Comer, C., Robertson, R.: Identified nerve cells and insect behavior. Progress in Neurobiology 63, 409–439 (2001) 25. Conklin, J., Eliasmith, C.: A controlled attractor network model of path integration in the rat. Journal of Computational Neuroscience 18(2), 183–203 (2005) 26. Cos, I., Hayes, G.: Behaviour control using a functional and emotional model. In: Proceedings of the 7th Conference on the Simulation of Adaptive Behaviour. The MIT Press, Edinburgh (2002) 27. Cos-Aguilera, I., Hayes, G., Canamero, L.: Using a SOFM to learn object affordances. In: Proceedings of the 5th Workshop on Physical Agents (WAF 2004), Girona, Catalonia, Spain (2004) 28. Craik, K.: The nature of explanation. Cambridge University Press, Cambridge (1943) 29. Cruse, H.: The evolution of cognition - a hypothesis. Cognitive Science 27, 135–155 (2003) 30. Deneubourg, J.L., Lioni, A., Detrain, C.: Dynamics of aggregation and emergence of cooperation. Biol. Bull. 202(3), 262–267 (2002) 31. Derby, C., Steullet, P.: Why do animals have so many receptors? The role of multiple chemosensors in animal perception. Biological Bulletin 200, 211–215 (2001) 32. Dolcomyn, F.: Insect walking and robotics. Annual Review of Entomology 49, 51–70 (2004) 33. Dubnau, J., Tully, T.: Gene discovery in Drosophila: new insights for learning and memory. Annual Review of Neuroscience 21, 407–444 (1998) 34. Eliasmith, C.: Computation and dynamical models of mind. Minds and machines 7, 531– 541 (1997) 35. Eliasmith, C., Anderson, C.: Neural engineering - computation, representation, and dynamics in neurobiological systems. The MIT Press, Cambridge (2003)
Perception for Action in Insects
35
36. Elman, J.: Finding structure in time. Cognitive Science 14(2), 179–211 (1990), http://www.isrl.uiuc.edu/∼amag/langev/paper/ elman90findingStructure.html 37. Engel, A., Fries, P., Singer, W.: Dynamic predictions: oscillations and synchrony in topdown processing. Nature 2, 704–716 (2001) 38. Evans, H.E.: Wasp farm. Anchor Natural History Books. Anchor Press/ Doubleday and Company, New York (1973) 39. Fahrbach, S.: Structure of the mushroom bodies of the insect brain. Annual Review of Entomology 51, 209–232 (2006) 40. Fahrbach, S., Giray, T., Farris, S., Robinson, G.: Expansion of the neuropil of the mushroom bodies in male honeybees is coincident with initiation of flight. Neuroscience Letters 236, 135–138 (1997) 41. Farivar, S.: Cytoarchitecture of the locust olfactory system. Ph.D. thesis, California Institute of Technology (2005) 42. Freeman, W.: A neurobiological theory of meaning in perception. I. Information and meaning in nonconvergent and nonlocal brain dynamics. International Journal of Bifurcation and Chaos 13(9) (2003) 43. Gaver, W.: What in the world do we hear? An ecological approach to auditory source perception. Ecological Psychology 5, 1–29 (1993) 44. Gerber, B., Tanimoto, H., Heisenberg, M.: An engram found? Evaluating the evidence from fruit flies. Current Opinion in Neurobiology 14, 737–744 (2004) 45. Gibson, J.: The ecological approach to visual perception. Houghton Mifflin, Boston (1979) 46. Glazier, P., Davids, K., Bartlett, R.: Dynamical systems theory: a relevant framework for performance-oriented sports biomechanics research. Sportscience 7 (2003) 47. Gregory, R.: Perceptions as hypotheses. Philosophical Transactions of the Royal Society of London, B 290, 181–197 (1980) 48. Gregson, R.: n-Dimensional non-linear psychophysics. Erlbaum, Mahwah (1992) 49. Gronenberg, W.: Subdivisions of hymenopteran mushroom body calyces by their afferent supply. Journal of Comparative Neurology 436, 474–489 (2001) 50. Grossberg, S.: Neural networks and natural intelligence. MIT Press, Cambridge (1988) 51. Grush, R.: The emulation theory of representation: motor control, imagery, and perception. Behavioral and Brain Sciences (in press, 2003) 52. Grush, R.: In defense of some ‘cartesian’ assumptions concerning the brain and its operations. Biology and Philosophy 18, 53–93 (2003) 53. Guastello, S.: Nonlinear dynamics in psychology. Discrete dynamics in nature and society 6, 11–29 (2001) 54. Gurney, K., Humphries, M., Wood, R., Prescott, T., Redgrave, P.: Testing computational hypotheses of brain systems function: a case study with the basal ganglia. Network: Computation in Neural Systems 15, 263–290 (2004) 55. Haken, H.: Synergetics: an introduction. Springer, Berlin (1983) 56. Hammer, M.: An identified neuron mediates the unconditioned stimulus in associative olfactory learning in honeybees. Nature 366, 59–63 (1993) 57. Hammer, M.: The neural basis of associative reward learning in honeybees. Trends in Neuroscience 20(6), 245–252 (1997) 58. Hammer, M., Menzel, R.: Multiple sites of associative odor learning as revealed by local brain microinjections of octopamine in honeybees. Learning and Memory 5, 146–156 (1998) 59. Hanesch, U., Fischbach, K.F., Heisenberg, M.: Neuronal architecture of the central complex in Drosophila melanogaster. Cell Tissue Research 257, 343–366 (1989)
36
B. Webb and J. Wessnitzer
60. Harter, D., Kozma, R.: Navigation and cognitive map formation using aperiodic neurodynamics. In: From Animals to Animats 8: The Eighth International Conference on the Simulation of Adaptive Behavior (SAB 2004), pp. 450–455. MIT Press, Cambridge (2004) 61. Hartmann, G., Wehner, R.: The ant’s path integration system: a neural architecture. Biological Cybernetics 73, 483–497 (1995) 62. Heath, R.: Nonlinear dynamics: techniques and applications in psychology. Lawrence Erlbaum Associates, Mahwah (2000) 63. Heisenberg, M.: Central brain function in insects: genetic studies on the mushroom bodies and central complex in Drosophila. In: Neural Basis of Behavioural Adaptations. Fortschritte der Zoologie, vol. 39, pp. 61–79. Gustav Fischer Verlag, Stuttgart (1994) 64. Heisenberg, M.: Pattern recognition in insects. Current Opinion in Neurobiology 5, 475– 481 (1995) 65. Heisenberg, M.: What do the mushroom bodies do for the insect brain? an introduction. Learning and Memory 5, 1–10 (1998) 66. Heisenberg, M., Heusipp, M., Wanke, C.: Structural plasticity in the Drosophila brain. Journal of Neuroscience 15, 1951–1960 (1995) 67. Helfrich-Foerster, C.: Neurobiology of the fruit fly’s circadian clock. Genes, Brain and Behavior 4, 65–76 (2005) 68. Hesslow, G.: Conscious thought as simulation of behaviour and perception. Trends in Cognitive Sciences 6(6), 242–247 (2002) 69. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Computation 9, 2451– 2471 (1997) 70. Homberg, U.: Structure and function of the central complex in insects. In: Arthropod brain: its evolution, development, structure and functions, pp. 347–367. Wiley, NY (1987) 71. Homberg, U.: The central complex in the brain of the locust: anatomical and physiological characterisation. In: Elsner, N., Roth, G. (eds.) Brain-Perception-Cognition. Thieme, Stuttgart (1990) 72. Homberg, U.: Flight-correlated activity changes in neurons of the lateral accessory lobes in the brain of the locust Schistocerca gregaria. Journal of Comparative Physiology A 175, 597–610 (1994) 73. Homberg, U.: In the search of the sky compass in the insect brain. Naturwissenschaften 91, 199–208 (2004) 74. Homberg, U.: Multisensory processing in the insect brain. In: Methods in Insect Sensory Neuroscience. CRC Press, Boca Raton (2005) 75. Homberg, U., Christensen, T., Hildebrand, J.: Structure and function of the deutocerebrum in insects. Annual Review of Entomology 34, 477–501 (1989) 76. Homberg, U., Reischig, T., Stengl, M.: Neural organisation of the circadian system of the cockroach Leucophaea maderae. Chronobiol. Int. 20, 577–591 (2003) 77. Hopfield, J.: Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences of the USA 79, 2554–2558 (1982) 78. Hoshino, O.: Dynamic interaction of attractors across multiple cortical networks as a neural basis for intersensory facilitation. Connection Science 14, 345–375 (2002) 79. Hoshino, O.: Coherent interaction of dynamical attractors for object-based selective attention. Biological Cybernetics 89, 107–118 (2003) 80. Huerta, R., Nowotny, T., Garcia-Sanchez, M., Abarbanel, H., Rabinovich, M.: Learning classification in the olfactory system of insects. Neural Computation 16, 1601–1640 (2004) 81. Hurley, S.: Perception and action: alternative views. Synthese 129, 3–40 (2001) 82. Husbands, P., Harvey, I., Cliff, D., Miller, G.: The use of genetic algorithms for the development of sensorimotor control systems. In: Proceedings of the PerAc 1994 Conference. IEEE Computer Society Press, Los Alamitos (1994)
Perception for Action in Insects
37
83. Ikeno, H., Usui, S.: Basic computational properties of Kenyon cell in the mushroom body of honeybee. Neurocomputing 32, 167–172 (2000) 84. Ito, K., Suzuki, K., Estes, P., Ramaswami, M., Yamamoto, D., Strausfeld, N.: The organisation of extrinsic neurons and their implications in the functional roles of the mushroom bodies in Drosophila melanogaster meigen. Learning and Memory 5, 52–77 (1998) 85. Jaeger, H.: Adaptive nonlinear system identification with echo state networks. In: Proc. NIPS 2002 (2002) 86. Keijzer, F.: Representation in dynamical and embodied cognition. Cognitive Systems Research 3, 275–288 (2002) 87. Kelso, J.: Dynamic patterns: the self-organisation of brain and behaviour. MIT Press, Cambridge (1995) 88. Kersten, D., Yuille, A.: Bayesian models of object perception. Current opinions in neurobiology 13, 1–9 (2003) 89. Kirsh, D.: Today the earwig, tomorrow man? Artificial Intelligence 47, 161–184 (1991) 90. Kirsh, D.: The intelligent use of space. Artificial Intelligence 73, 31–68 (1995) 91. Korn, H., Faure, P.: Is there chaos in the brain? II. Experimental evidence and related models. Comptes Rendus Biologies 326, 787–840 (2003) 92. Kosko, B.: Adaptive bidirectional associative memories. Applied Optics 26, 4947–4960 (1987) 93. Kosslyn, S., Ganis, G., Thompson, W.: Neural foundations of imagery. Nature Reviews 2, 635–642 (2001) 94. Labhart, T., Meyer, E.: Detectors for polarized skylight in insects: a survey of ommatidial specialisations in the dorsal rim area of the compound eye. Microscopy Research and Technique 47, 368–379 (1999) 95. Lakoff, G., Johnson, M.: Metaphors We Live By. University of Chicago Press, Chicago (1980) 96. Lappe, M., Bremmer, F., van den Berg, A.: Perception of self-motion from visual flow. Trends in Cognitive Sciences 3(9), 329–336 (1999) 97. Laurent, G.: Olfactory processing: maps, time and codes. Current Opinion in Neurobiology 7, 547–553 (1997) 98. Laurent, G.: Olfactory network dynamics and the coding of multidimensional signals. Nature Reviews Neuroscience 3(11), 884–895 (2002) 99. Lederman, S., Klatzky, R.: Haptic aspects of motor control. In: Jeannerod, M. (ed.) Handbook of Neuropsychology. Action and Cognition, vol. 11. Elsevier Science Publishers, Amsterdam (1996) 100. Lee, D.: A theory of visual control of braking based on information about time-to-collision. Perception 5, 437–459 (1976) 101. Li, Y., Strausfeld, N.: Morphology and sensory modality of mushroom body extrinsic neurons in the brain of the cockroach. Journal of Comparative Neurology 387, 631–650 (1997) 102. Li, Y., Strausfeld, N.: Multimodal efferent and recurrent neurons in the medial lobes of cockroach mushroom bodies. Journal of Comparative Neurology 409, 647–663 (1999) 103. Li, Z., Dayan, P.: Computational differences between asymmetrical and symmetrical networks. Network 10, 59–77 (1999) 104. Liberman, A., Cooper, F., Shankweller, D., Studdert, M.: Perception of the speech code. Psychological Review 74, 431–461 (1967) 105. Liberman, A., Mattingly, I.: The motor theory of speech perception revised. Cognition 21, 1–36 (1985) 106. Liu, G., Seiler, H., Wen, A., Zars, T., Ito, K., Wolf, R., Heisenberg, M., Liu, L.: Distinct memory traces for two visual features in the Drosophila brain. Nature 439, 551–556 (2006) 107. Liu, L., Wolf, R., Ernst, R., Heisenberg, M.: Context generalisation in Drosophila visual learning requires the mushroom bodies. Nature 400, 753–756 (1999)
38
B. Webb and J. Wessnitzer
108. Loesel, R., Naessel, D., Strausfeld, N.: Common design in a unique midline neuropil in the brains of arthropods. Arthropod Structure and Development 31, 77–91 (2002) 109. Maass, W., Natschlaeger, T., Markram, H.: Real-time computing without stable states: a new framework for neural computation based on perturbations. Neural Computation 14, 2531–2560 (2002) 110. Maes, P.: Learning behaviour networks from experience. In: Proceedings of the First European Conference on Artificial Life. MIT Press, Cambridge (1992) 111. Marr, D.: Vision: a computational investigation into the human representation and processing of visual information. Freeman Publishers, New York (1982) 112. Martin, J., Ernst, R., Heisenberg, M.: Mushroom bodies suppress locomotor activity in Drosophila melanogaster. Learning and Memory 5, 179–191 (1998) 113. Martin, J.R., Ernst, R., Heisenberg, M.: Temporal pattern of locomotor activity in Drosophila melanogaster. Journal of Comparative Physiology A 184, 73–84 (1999) 114. Martin, J.R., Raabe, T., Heisenberg, M.: Central complex substructures are required for the maintenance of locomotor activity in Drosophila melanogaster. Journal of Comparative Physiology A 185, 277–288 (1999) 115. Mauelshagen, J.: Neural correlates of olfactory learning paradigms in an identified neuron in the honeybee brain. Journal of Neurophysiology 69(2), 609–625 (1993) 116. McBride, S., Giuliani, G., Chol, C., Krause, P., Correale, D., Watson, K., Baker, G., Siwicki, K.: Mushroom body ablation impairs short-term memory and long-term memory of courtship conditioning in Drosophila melanogaster. Neuron 24, 967–977 (1999) 117. McFarland, D., B¨osser, T.: Intelligent Behavior in Animals and Robots. MIT Press, Cambridge (1993) 118. Menzel, R.: Searching for the memory trace in a mini-brain, the honeybee. Learning and Memory 8, 53–62 (2001) 119. Menzel, R., Blakers, M.: Colour receptors in the bee eye - morphology and spectral sensitivity. Journal of Comparative Physiology A: Sensory, Neural, and Behavioural Physiology 108(1), 11–33 (1976) 120. Menzel, R., Giurfa, M.: Cognitive architecture of a mini-brain: the honeybee. Trends in Cognitive Sciences 5(2), 62–71 (2001) 121. Metta, G., Fitzpatrick, P.: Better vision through manipulation. Adaptive Behavior 11, 109– 128 (2003) 122. Meyer, J.A., Guillot, A., Girard, B., Khamassi, M., Pirim, P., Berthoz, A.: The psikharpax project: Towards building an artificial rat. Robotics and Autonomous Systems 50, 211–223 (2005) 123. Mishima, T., Kanzaki, R.: Physiological and morphological characterisation of olfactory descending interneurons of the male silkworm moth, bombyx mori. Journal of Comparative Physiology A 184, 143–160 (1999) 124. Mizunami, M., Iwasaki, M., Nishikawa, M., Okada, R.: Modular structures in the mushroom body of the cockroach. Neuroscience Letters 229, 153–156 (1997) 125. Mizunami, M., Okada, R., Li, Y., Strausfeld, N.: Mushroom bodies of the cockroach: activity and identities of neurons recorded in freely moving animals. Journal of Comparative Neurology 402, 501–519 (1998) 126. Mizunami, M., Weibrecht, J., Strausfeld, N.: Mushroom bodies of the cockroach: their participation in place memory. Journal of Comparative Neurology 402, 520–537 (1998) 127. Mobbs, P.: The brain and the honeybee Apis mellifera. I. The connections and spatial organization of the mushroom bodies. Philosophical Transactions of the Royal Society B 298, 309–354 (1982) 128. Mueller, M., Homberg, U., Kuehn, A.: Neuroarchitecture of the lower division of the central body in the brain of the locust (Schistocerca gregaria). Cell Tissue Research 288, 159–176 (1997)
Perception for Action in Insects
39
129. Newell, A., Simon, H.: Computer science as empirical enquiry: symbols and search. Communications of the Association for Computer Machinery 19, 113–126 (1976) 130. Nicolis, S., Tsuda, I.: On the parallel between Zipf’s law and 1/f processes in chaotic systems possessing coexisting attractors: a possible mechanism for language formation in the cerebral cortex. Progress of Theoretical Physics 82, 254–274 (1989) 131. Nishikawa, M., Nishino, H., Mizunami, M., Yokohari, F.: Function-specific distribution patterns of axon terminals of input neurons in the calyces of the mushroom body of the cockroach, periplaneta americana. Neuroscience Letters 245, 33–36 (1998) 132. Nolfi, S., Floreano, D.: Evolutionary Robotics. The Biology, Intelligence, and Technology of Self-organizing Machines. MIT Press, Cambridge (2000) 133. Nowotny, T., Huerta, R., Abarbanel, H., Rabinovich, M.: Self-organization in the olfactory system: one shot odor recognition in insects. Biological Cybernetics 93, 436–446 (2005) 134. O’Donnell, S., Donlan, N., Jones, T.: Mushroom body structural change is associated with division of labor in eusocial wasp workers (Polybia aequatorialis, Hymenoptera: Vespidae). Neuroscience Letters 356, 159–162 (2004) 135. Okada, R., Ikeda, J., Mizunami, M.: Sensory responses and movement-related activities in extrinsic neurons of the cockroach mushroom bodies. Journal of Comparative Physiology A 185, 115–129 (1999) 136. Okada, R., Sakura, M., Mizunami, M.: Distribution of dendrites of descending neurons and its implications for the basic organisation of the cockroach brain. Journal of Comparative Neurology 458, 158–174 (2003) 137. Okajima, K., Tanaka, S., Fujiwara, S.: A heteroassociative memory network with feedback connection. In: Caudill, M., Butler, C. (eds.) Proc. IEEE First International Conference on Neural Networks, pp. 711–718. IEEE, Los Alamitos (1987) 138. Olshausen, B., Field, D.: Sparse coding of sensory inputs. Current Opinion in Neurobiology 14, 481–487 (2004) 139. O’Regan, J.K., Noe, A.: A sensorimotor account of vision and visual consciousness. Behavioral and Brain Sciences 24, 939–1031 (2001) 140. Paine, R.W., Tani, J.: Motor primitive and sequence self-organization in a hierarchical recurrent neural network. Neural Networks 17, 1291–1309 (2004) 141. Pascual, A., Preat, T.: Localization of long-term memory within the Drosophila mushroom body. Science 294, 1115–1117 (2001) 142. Perez-Orive, J., Bazhenov, M., Laurent, G.: Intrinsic and circuit properties favor coincidence detection for decoding oscillatory input. Journal of Neuroscience 24, 6037–6047 (2004) 143. Perez-Orive, J., Mazor, O., Turner, G., Cassenaer, S., Wilson, R., Laurent, G.: Oscillations and sparsening of odor representations in the mushroom bodies. Science 297, 359–365 (2002) 144. Pfeifer, R., Scheier, C.: Understanding intelligence. The MIT Press, Cambridge (1999) 145. Philipona, D., O’Regan, J., Nadal, J.P., Coenen, O.: Perception of the structure of the physical world using unknown multimodal sensors and effectors. In: Advances in Neural Information Processing Systems (2004) 146. Porr, B., Woergoetter, F.: Inside embodiment - what means embodiment to radical constructivists? Kybernetes 34, 105–117 (2005) 147. Port, R., van Gelder, T. (eds.): Mind as motion: explorations in the dynamics of cognition. A Bradford Book. The MIT Press, Cambridge (1995) 148. Prescott, T., Redgrave, P., Gurney, K.: Layered control architectures in robots and vertebrates. Adaptive Behavior 7, 99–127 (1999) 149. Prescott, T.J., Gurney, K., Montes-Gonzalez, F., Humphries, M., Redgrave, P.: The robot basal ganglia: Action selection by an embedded model of the basal ganglia, pp. 349–356. Plenum Press, New York (2002)
40
B. Webb and J. Wessnitzer
150. Pressing, J.: Referential dynamics of cognition and action. Psychological Review 106, 714– 747 (1999) 151. Prigogine, I.: From being to becoming: time and complexity in the physical sciences. Freeman, NY (1980) 152. Rieke, F., Warland, D., de Ruyter van Steveninck, R., Bialek, W.: Spikes: exploring the neural code. MIT Press, Cambridge (1997) 153. Rizzolatti, G., Craighero, L.: The mirror neuron system. Annual Review of Neuroscience 27, 169–192 (2004) 154. Rock, I.: In defense of unconscious inference. Wiley, New York (1997) 155. van Rooij, I., Bongers, R., Haselager, W.: A non-representational approach to imagined action. Cognitive Science 26, 345–375 (2002) 156. Rosay, P., Armstrong, D., Wang, Z., Kaiser, K.: Synchronized neural activity in the Drosophila memory centres and its modulation by amnesiac. Neuron 30, 759–770 (2001) 157. Rosenblatt, J.K., Payton, D.W.: A fine-grained alternative to the subsumption architecture for mobile robot control. In: Proc. of the IEEE Int. Conf. on Neural Networks, vol. 2, pp. 317–324. IEEE Press, Washington (1989) 158. Rosenblum, L.: Acoustical information for controlled collisions. In: Schick, A. (ed.) Contributions to Psychological Acoustics. Bibliotheks- und Informationssystem der Carl von Ossietzky Universitaet Oldenburg, Oldenburg (1993) 159. Rosenblum, L., Wuestefeld, A., Anderson, K.: Auditory reachability: an affordance approach to the perception of sound source distance. Ecological Psychology 8, 1–24 (1996) 160. Russell, S., Norvig, P.: Artificial Intelligence: A Modern Approach. Prentice Hall, Englewood Cliffs (1995) 161. Rybak, J., Menzel, R.: Anatomy of the mushroom bodies in the honey bee brain: the neuronal connections of the alpha-lobe. Journal of Comparative Neurology 334, 444–465 (1993) 162. Schildberger, K.: Multimodal interneurons in the cricket brain: properties of identified extrinsic mushroom body cells. Journal of Comparative Physiology A 154, 71–79 (1984) 163. Schoener, G., Dijkstra, T., Jeka, J.: Action-perception patterns emerge from coupling and adaptation. Ecological Psychology 10, 323–346 (1998) 164. Sch¨oner, G., Dose, M., Engels, C.: Dynamics of behaviour: theory and applications for autonomous robot architectures. Robotics and Autonomous Systems 16, 213–245 (1995) 165. Schuermann, F.W.: Bemerkungen zur Funktion der Corpora pedunculata im Gehirn der Insekten aus morphologischer Sicht. Experimental Brain Research 19, 406–432 (1974) 166. Schultz, W., Dayan, P., Montague, P.: A neural substrate of prediction and reward. Science 275, 1593–1599 (1997) 167. Schultz, W., Dickinson, A.: Neuronal coding of prediction errors. Annual Review of Neuroscience 23, 473–500 (2000) 168. Schwaerzel, M., Monastirioti, M., Scholz, H., Friggi-Grelin, F., Birman, S., Heisenberg, M.: Dopamine and octopamine differentiate between aversive and appetitive olfactory memories in Drosophila. Journal of Neuroscience 23(33), 10495–10502 (2003) 169. Seth, A.: Evolving action selection and selective attention without actions. In: Pfeiffer, R. (ed.) From Animals to Animats 5, Proc. of 5th Intl. Conf. on Simulation of Adaptive Behavior. MIT Press/Bradford Books (1998) 170. Skarda, C., Freeman, W.: How brains make chaos in order to make sense of the world. Behavioral and Brain Sciences 10, 161–173 (1987) 171. Smithers, T.: On behaviour as dissipative structures in agent-environment interaction systems. In: Ritter, H., Cruse, H., Dean, J. (eds.) Prerational Intelligence: Adaptive Behavior and Intelligent Systems Without Symbols and Logic, vol. 2, pp. 243–257. Kluwer Academic Pulishers, Dordrecht (2000)
Perception for Action in Insects
41
172. Stange, G., Stowe, S., Chahl, J., Massaro, A.: Anisotropic imaging in the dragonfly median ocellus: a matched filter for horizon detection. Journal of Comparative Physiology A 188, 455–467 (2002) 173. Steels, L.: Synthesising the origins of language and meaning using co-evolution, selforganisation and level formation. In: Evolution of Human Language. Edinburgh Univ. Press (1996) 174. Stopfer, M., Bhagavan, S., Smith, B., Laurent, G.: Impaired odour discrimination on desynchronization of odour-encoding neural assemblies. Nature 390, 70–74 (1997) 175. Strausfeld, N.: Structural organization of male-specific visual neurons in Calliphorid optic lobes. Journal of Comparative Physiology A 169, 379–393 (1991) 176. Strausfeld, N.: A brain region in insects that supervises walking. Progress in Brain Research 123, 273–284 (1999) 177. Strausfeld, N., Hansen, L., Li, Y., Gomez, R., Ito, K.: Evolution, discovery, and interpretations of arthropod mushroom bodies. Learning and Memory 5, 11–37 (1998) 178. Strausfeld, N., Li, Y.: Representation of the calyces in the medial and vertical lobes of cockroach mushroom bodies. Journal of Comparative Neurology 409, 626–646 (1999) 179. Strauss, R.: The central complex and the genetic dissection of locomotor behaviour. Current Opinion in Neurobiology 12, 633–638 (2002) 180. Strauss, R., Pichler, J.: Persistence of orientation toward a temporarily invisible landmark in Drosophila melanogaster. Journal of Comparative Physiology A 182, 411–423 (1998) 181. Tani, J., Ito, M.: Self-organization of behavioral primitives as multiple attractor dynamics: A robot experiment. IEEE Trans. on Systems, Man, and Cybernetics Part A: Systems and Humans 33, 481–488 (2003) 182. Thelen, E., Smith, L.: A dynamic systems approach to the development of cognition and action. MIT Press, Cambridge (1994) 183. Tuller, B., Case, P., Ding, M., Kelso, J.: The nonlinear dynamics of speech categorization. Journal of Experimental Psychology: Human Perception and Performance 20, 3–11 (1994) 184. Turvey, M.T.: Dynamic touch. American Psychologist 5, 1134–1152 (1996) 185. Tyrrell, T.: Computational mechanisms for action selection. Ph.D. thesis, Department of Artificial Intelligence, University of Edinburgh (1993) 186. Verschure, P., Voegtlin, T., Douglas, R.: Environmentally mediated synergy between perception and behaviour in mobile robots. Nature 425, 620–624 (2003) 187. Vitzthum, H., Mueller, M., Homberg, U.: Neurons of the central complex of the locust Schistocerca gregaria are sensitive to polarised light. The Journal of Neuroscience 22(3), 1114–1125 (2002) 188. Waddell, S., Armstrong, D., Kitamoto, T., Kaiser, K., Quinn, W.: The amnesiac gene product is expressed in two neurons in the Drosophila brain that are critical for memory. Cell 103, 805–813 (2000) 189. Waddell, S., Quinn, W.: Flies, genes and learning. Annual Review of Neuroscience 24, 1283–1309 (2001) 190. Wann, J.: Anticipating arrival: Is the tau margin a specious theory? Journal of Experimental Psychology: Human Perception and Performance 22, 1031–1048 (1996) 191. Webb, B.: Can robots make good models of biological behaviour? Behavioral and Brain Sciences 24, 1033–1050 (2001) 192. Webb, B.: Neural mechanisms for prediction: do insects have forward models? Trends in neuroscience 27(5), 278–282 (2004) 193. Wehner, R.: ‘Matched filters’ - neural models of the external world. Journal of Comparative Physiology 161, 511–531 (1987) 194. Wehner, R.: Desert ant navigation: how miniature brains solve complex tasks. Journal of Comparative Physiology A 189, 579–588 (2003)
42
B. Webb and J. Wessnitzer
195. Weiner, J.: On the practice of ecology. Journal of Ecology 83, 153–158 (1995) 196. Wessnitzer, J., Webb, B.: Multimodal sensory integration in insects - towards insect brain control architectures. Bioinspiration and Biomimetics 1, 63–75 (2006) 197. Wexler, M., Kosslyn, S., Berthoz, A.: Motor processes in mental rotation. Cognition 68, 77–94 (1998) 198. Wilson, M.: Six views of embodied cognition. Psychological Bulletin and Review 9, 625– 636 (2002) 199. Wittmann, T., Schwegler, H.: Path integration - a network model. Biological Cybernetics 73, 569–575 (1995) 200. Woergoetter, F., Porr, B.: Temporal sequence learning, prediction and control - a review of different models and their relation to biological mechanisms. Neural Computation 17, 245–319 (2005) 201. Wohlgemuth, S., Ronacher, B., Wehner, R.: Ant odometry in the third dimension. Nature 411, 795–798 (2001) 202. Wolpert, D., Ghahramani, Z.: Computational principles of movement neuroscience. Nature Neuroscience 3, 1212–1217 (2000) 203. Wolpert, D., Ghahramani, Z., Jordan, M.: An internal model for sensorimotor integration. Science 269, 1880–1882 (1995) 204. Wuestenberg, D., Boytcheva, M., Gruenewald, B., Byrne, J., Menzel, R., Baxter, D.: Current- and voltage-clamp recordings and computer simulations of Kenyon cells in the honeybee. Journal of Neurophysiology 92, 2589–2603 (2004) 205. Wyley, D., Bischof, W., Frost, B.: Common reference frame for neural coding of translational and rotational optic flow. Nature 392, 278–282 (1998) 206. Yusuyama, K., Meinertzhagen, I., Schuermann, F.W.: Synaptic organization of the mushroom body calyx in Drosophila melanogaster. Journal of Comparative Neurology 445, 211– 226 (2002) 207. Zeil, J., Hofmann, M., Chahl, J.: Catchment areas of panoramic snapshots in outdoor scenes. Journal of the Optical Society of America A 20, 450–469 (2003)
2 Principles of Insect Locomotion H. Cruse1 , V. D¨urr2 , M. Schilling1, and J. Schmitz1 1
2
University of Bielefeld, Department of Biological Cybernetics and Theoretical Biology, P.O. Box 100131, D-33501 Bielefeld, Germany {holk.cruse,malte.schilling,josef.schmitz}@.uni-bielefeld.de University of Cologne, Institute of Zoology, Weyertal 119, D-50931 K¨oln, Germany
[email protected]
Abstract. Walking animals can deal with large range of difficult terrain and can use their legs for other purposes as sensing or object manipulation. This is possible although the underlying control system is based on neurons which are considered to be quite sloppy and slow computational elements. Important aspects of this control system are error tolerance and the capability of selforganization. This chapter concentrates on insect walking behaviour. Apart from some references to relevant morphology it addresses behavioural investigations which are paralleled by software simulations to allow a better understanding of the underlying principles. Furthermore, hints to neurophysiology and to hardware simulations are given. Characteristic properties of the control system are its decentralized architecture that relies heavily on internal feedback as well as on sensory feedback, and that exploits the physics of the body.
2.1 Introduction Locomotion, i.e. changing places, is the basis of all actions. Terrestrial locomotion in artificial systems is usually based on wheels or caterpillar tracks. However, both methods require a more or less flat terrain which, at least in the case of wheels, has to be construed as an artifact in the form of roads or trails. Animals, even “simple” insects, survive in any terrain and do not require artificial conditions. They walk over stony or sandy ground, climb in branches, surmount obstacles of arbitrary form and height as well as deep gaps that may be wider than their body length. Many animals can also use their legs for swimming. Furthermore, legs are used for complex manipulations like feeding or body cleaning or transportation of objects. In insects and crustaceans, leg movement is controlled by a comparatively small brain that uses computational elements, the neurons, which are, compared to modern computers, slower and less exact by many orders of magnitude. On the other hand, the control system must be extremely reliable and robust and must work successively also under adverse circumstances, e.g., loss of legs and with low energy consumption. Thus, the underlying control strategies are of high interest not only for the biologist doing basic research, but also for the engineer when it comes to construct artificial walking systems. Conversely, constructing walking systems using software simulations or building robots is also of interest for the biologist: like software simulation, such a “hardware simulation” is a critical tool to formulate and test hypotheses that cannot directly be investigated using the biological system only. P. Arena and L. Patan`e (Eds.): Spatial Temporal Patterns, COSMOS 1, pp. 43–96. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com
44
H. Cruse et al.
At first sight, walking seems to be a not very interesting behaviour, because it appears to be fairly automatic. We do not have to think consciously about moving the joints when walking. Nevertheless, walking in a natural environment requires considerable “motor intelligence” and can be regarded as a paradigm for control of behaviour in general. First of all, walking, as almost all behaviour, has to deal with redundancy. In most biological systems for motor control, particularly those concerned with walking, the number of degrees of freedom is normally larger than that necessary to perform the task. This requires the system to select among different alternatives according to some, often context-dependent, optimization criteria. As a consequence the system usually has to have some autonomy. Autonomy — as understood here — does not simply mean, as it often does in robotics, energy autonomy. Rather, we have the literal meaning in mind, i.e., to be not dependent on commands given by an external system, such as an operator. Thus, an autonomous system follows self-contained rules and makes its own decisions. Therefore, the experimenter does not have direct control of the important inputs to the motor system. Furthermore, such systems must adapt to complex, often unpredictable environments. Sensorimotor connections give rise to a “loop through the real world”, and as a result, the properties of the environment affect the behaviour of the system. Despite these experimental and theoretical difficulties, the complexity makes the study of motor mechanisms especially challenging. This is because they illustrate to a high degree the task of integrating influences from the environment, mediated through peripheral sensory systems, with central processes reflecting the state and needs of the organism. In a walking insect, at least 18 joints (i.e., three per leg) have to be controlled. Because the environment may change drastically from one step to the next, and even the geometrical properties of the body may change, the control of walking is anything but a trivial task.
2.2 Biological Systems Whereas insects and crustaceans usually use six or more legs for walking, there are other animals which are confined to four legs (most mammals) or two legs (e.g., birds, some primates), but tetrapod and bipedal locomotion can also be observed in insects, e.g., mantids [172] and cockroaches [100, 97, 28], for a review see [96]. Systems using lesser legs require a more complex controller because holding the body upright against gravity is more difficult when using a small number of legs. This does not only refer to static stability, but includes the computationally more difficult problem of dynamical stability. Another disadvantage is that loss of a leg can less easily be compensated for when the system is endowed with only a few legs. Although many animals have body appendages that can be used for walking, only a few species are investigated in sufficient detail. In crustaceans, this is the crayfish and the lobster (with essentially four pairs of walking legs, i.e. eight legs). In the six-legged insects, this is the stick insect [14, 108, 47, 17], the cockroach [75, 199, 195, 127, 152] and, to a minor extent, the locust [33]. Some investigations of ants, e.g., [210], and, more recently, of Drosophila [193], are also of interest. In four-legged animals this is the dog and, mainly, the cat. Studies concentrating on neurophysiology but not so much on behaviour have also been done with mice and rats and with newt. The two-legged
Principles of Insect Locomotion
45
animal studied is, of course, man. However, most studies concentrate on standing, on control of upright posture or on handicapped people, e.g., paraplegic patients. The lower level control problems, i.e. control and properties of individual muscles in the context of behaviour, have, however, most intensively studied in man [124] and in cat [156]. Recently the overwhelming importance of muscular properties to the control problem in general has also been shown for insects (see Actuators, Section 2.8). In the remainder, this chapter, like the previous one, mainly concentrates on studies of insects, focusing on topics relating to locomotion. Insects have been studied intensively, not only because they are simpler to experiment with, but also because they are assumed to be construed in a simpler way and therefore hopefully easier to understand than vertebrates. The body of each insect, as is the case for all arthropods, is highly segmented and essentially consists of a head, a thorax and an abdomen. Each of these parts consists of several primary segments, some of those are fused however. This segmentation is reflected by the morphology of the central nervous system (CNS), a fact that eases the neurophysiological investigations of the walking behaviour of these animals. Basically, the whole central nervous system consists of an anteriorly and dorsally located brain (head ganglia) and a double chain of ventral segmental ganglia. The latter are interconnected within the segments by commissures and between the segments by connectives. The organization of the CNS of the stick insect is depicted in Fig. 2.1. Due to the rich endowment with sensory systems most insects posses a large brain located in the head capsule, in which the olfactory and mechanosensitive antennal and the visual inputs are processed. Situated ventrally in the head capsule is the subesophageal ganglion which serves the mouthparts. The thorax contains three sections, the prothorax, the mesothorax and the metathorax, each equipped with one pair of legs. Each thorax segment is served by its own ganglion; the left and right hemiganglion is fused to one thoracic ganglion. However, the bilateral partitioning is preserved internally as was revealed by neuroanatomical studies of several insect species (stick insect: [136], cockroach: [109], locust: [197, 161]). Most important for the investigation of the walking system of insects is the fact that each hemiganglion contains all necessary neurons for processing sensory information from the adjacent leg and all inter- and motoneurons for the production of a cyclic motor output to the muscles of the adjacent leg. Each abdominal segment also contains its own ganglion, except for the first few which might be fused with the metathoracic ganglion and the terminal ganglion which is again a fused ganglion from several segments. The main morphological characteristic of arthropods, specifically insects, is their exoskeleton. Therefore the muscles are arranged inside the tube-like skeleton (for details on muscular properties see Section 2.8: Actuators). The geometry of the leg is shown in Fig. 2.2. It usually contains five segments, coxa, trochanter, femur, tibia and tarsus. In the stick insect, as in many other insects, trochanter and femur form one rigid segment whereas, e.g., in the cockroach this joint is moveable to some extent. The coxa-trochanter (β -) and femur-tibia (γ -) joints are simple hinge joints with one degree of freedom corresponding to elevation and extension of the tarsus, respectively. The thorax-coxa (α -) joint, which connects the leg to the body,
46
H. Cruse et al.
Fig. 2.1. Sketches of the body (a) and the organization of the CNS (b) of an insect (a) adopted from [7]
is more complex, but most of its movement is in a rostrocaudal direction around a fixed axis described by the Euler angles φ and ψ . The additional degree of freedom, allowing changes in the alignment of this axis, is little used in normal walking, so the leg can be considered as a manipulator with three degrees of freedom for movement in three dimensions. Insects Use Feet with Adhesive Structures: The foot, or tarsus, is a morphological structure that is extremely important for walking in insects, but has only recently been studied in more detail [106, 95, 167]. The tarsus consists of a chain of up to five segments connected by passive joints (Fig. 2.3 a). The joints are flexed when a tendon driven by the retractor unguis muscle is shortened. The antagonistic force is produced by elastic properties of the tarsal joints. Each tarsal segment is equipped with different adhesive structures. In the case of the stick insect there are, from proximal to distal, four Euplantulae, one Arolium and two claws (Fig. 2.3 b-e). The surface of the Euplantulae is covered with small hair-like structures of about 5 µ m length, whereas the Arolium has a smooth surface that is covered with some fluid. Only the Arolium, but not the Euplantulae, is able to adhere to a glass surface. When the Arolium is covered with paint and the animal stands on six legs on a glass plate, about 40 times the body weight can
Principles of Insect Locomotion
47
Fig. 2.2. a) Schematic plot of insect leg morphology and approximate position of some mechanoreceptors, front view. b) joint between thorax and coxa, front view. c), d) two close ups showing the dorsal hair plate situated at the dorsal basis of the coxa (view from anterior). Forward movement of the coxa would move the hairs further below the membrane of the thoracic-coxal joint.
be held when the animal is loaded parallel to the surface of the substrate, whereas only a force of about body weight can be produced when the animal is loaded in a direction perpendicular to the surface. Exploiting these structures enables the insect to adopt a walking strategy different to that of most robots. Usually it is assumed that a walking system should adopt a statically stable position in every moment of time (disregarding dynamic cases). Static stability is then defined in such a way that the vertical projection of the centre of gravity must fall inside the support polygon spanned by the legs on the ground. This is a particular problem for insects because the centre of gravity is often situated near the base of the hind legs. Therefore, lifting a hind leg bears the danger of reaching an instable position. Kindermann [135] could show that this is actually the case in about 90% of the hind leg steps when a stick insect walks on a horizontal glass plate, and that the duration of the “instable” situation is longer than the time the animal would need to fall and touch the ground with the body. In a control experiment, he could show that tarsal contact with front or middle legs is indeed necessary to avoid falling: When the animal has to walk from the glass plate onto a surface covered with fine sand, the animal indeed falls over after the last hind leg lifted off the glass plate. This may be due to the fact that the hind legs are still loaded because the front legs cannot take over the load by pulling the body
48
H. Cruse et al.
Fig. 2.3. Tarsus morphology of Carausius morosus. a) side view of the tarsus. The tendon of the retractor muscle is schematically indicated by dotted lines. b) view from below, left side: claws and Arolium, right: 4 pairs of Euplantulae. c) claws and Arolium, d) Euplantulae, e) surface of Euplantulae.
to the sandy ground. Interestingly, in many cases the animal decreased walking speed before the hind leg was lifted, or stopped forward walking completely. This indicates the capability to predict the occurrence of an instable situation.
2.3 Sensors Two types of sensors will be treated separately: a) sensors that monitor the state of the body (for example joint angles, joint velocity, force, e.g., developed by a muscle against direction of gravity), for short proprioceptors or “body sensors”, and (b) sensors that provide information of the situation outside the body, for example to detect obstacles, for short exteroceptors or “environmental sensors”. Biological sensors usually differ from technical solutions insofar as the former in general give mixed information. Usually position and velocity and/or acceleration are transmitted by one channel, but are separated in technical systems. On the other hand, in biology, a signal representing one value, e.g., a joint angle often is represented not by one single channel, but by a number of parallel channels each of which represents only a given, e.g., angular, range (“range fractionation”, [5, 147] often represented by a topological arrangement of the dendrites [209, 150]. Because technical sensors appear “cleaner” in a physical sense, it is no problem to mix them in order to copy biological
Principles of Insect Locomotion
49
solutions. However, except for software simulations, we are not aware of specific robot applications of these principles. Related to this, the multitude of biological sensors is usually not dealt with, although this appears to be an important basis for the property of self-regulation and self-adjustment of biological systems as has been shown by recent simulation studies, e.g., Linder [143]. 2.3.1
Mechanosensors
Let us begin with the “body sensors”. Important sensors used by biological systems monitor joint position, joint velocity and joint acceleration [113]. It is typical for biological systems that there is redundant information: there are parallel arrangements of the same type of sensors as well as of different types that monitor essentially the same variable (e.g., joint position). In biology there are also contact sensors and load sensors. Load sensors are much less investigated in biology. However, recent investigations both in insects and in mammals concentrate on the functional role of load/force sensors (that measure stress in the exoskeleton or in the muscular tendons) and their applications to the control of leg movement. Negative feedback to control muscle force and positive feedback used to maintain and stabilize stance movement have been reported [87]. Interestingly, most insects do not have special gravity receptors as they are known from mammals and many other animals. Instead, walking insects exploit the different activations of feedback loops that control leg joints or other body appendages [201, 8]. Depending on the direction of gravity, different joint actuators (i.e. muscles) require different activation to hold the body in appropriate position. These differences can be used to gain information on the direction of the gravity vector. Why do other animals use specific gravity receptors? Six-legged insects have not only at least three legs on the ground to form a support tripod, but in addition have different adhesive structures at their feet (see above) that increase stability dramatically, allowing them to walk on vertical walls and even pending from a horizontal ceiling. We would speculate that the indirect method mentioned above to determine the direction of the gravity vector that is used by many insects is sufficient for these cases, but is too imprecise and too slow for swimming or flying animals or for walking animals with only four or two legs where reaction to changes in gravity direction has to be immediate. Therefore, direct gravity sensors may be applied for in latter cases. Mechanosensors found in insect legs can functionally be separated into position sensors and load sensors. Position sensors monitor joint angles, but as biological sensors have mixed properties, they may also transmit information on velocity and/or acceleration of joint movements. Load sensors are distributed over the exoskeleton (cuticle) of the leg [163] and monitor strain within the cuticle, functionally very much like technical strain gauges. 2.3.1.1 Position/Movement Sensors Morphologically, joint position and movement sensors come in different forms. The most obvious ones are the hair plates, groups of sensory hairs that are situated at joints in such a way that during movement of the joint the individual sensory hairs are bent by the soft joint membrane. The farther the joint is moved, the more each individual hair is bent and the larger is the number of hairs being bent. Such hair plates can be found
50
H. Cruse et al.
at the thoracic-coxal (α -) joint and at the coxa-trochanter (β -) joint (Fig. 2.2). Another important type of position sensors is the chordotonal organ. This is formed by sensory cells situated within partly elastic, partly rigid structures that span the joint. Movement of the joint leads to elongation or relaxation of the elastic part and of the sensitive structures of the sensory cells. An intensively investigated example is the femoral chordotonal organ [14] that monitors the femur-tibia (γ -) joint. Chordotonal organs can also be found at the thorax-coxa (α -) joint with in part complex mechanical structures [121]. Other internal and functionally quite similar sense organs are the strand receptor organs [27]. Furthermore, there are multipolar sensory cells situated near the joint membranes [12]. Chordotonal organs and hair plates have been intensively studied and are known to contribute to leg movement via negative (resistance reflexes) and positive (assistance reflexes) feedback. Leg contact with the substrate could be measured by exploiting signals from position sensors or load sensors or by individual large sensory hairs that are distributed over the cuticle (some can be seen in Figs. 2.2). 2.3.1.2 Load Sensors Two basic types of receptors detect forces in insect legs: campaniform sensilla are sense organs that monitor forces via strains that occur in the cuticle [163]. Multipolar receptors found in direct association with muscle tendons (apodemes) can signal tensions developed by some leg muscles and potentially encode external loads [110, 12, 148]. However, little is known about the effects of apodeme receptors in posture and locomotion [12] whereas some studies are available on the role of campaniform sensilla. Campaniform sensilla monitor forces as strains in the exoskeleton and can encode force increases and decreases. A campaniform sensillum consists of a sensory neuron whose dendrite inserts into a cap at the surface of the cuticle (Fig. 2.4). Forces applied to the exoskeleton, or developed by contractions of leg muscles, generate strains that produce mechanical distortion of the cap and discharge of the sensillum. In the legs, individual receptors are directionally sensitive and responses are correlated with the orientation of the cuticular cap. The discharges of sensilla depend upon their location and the vectoral direction of the imposed or self generated forces [206, 73, 37, 129]. In contrast to muscle apodeme or tendon organs, the coupling of campaniform sensilla to muscle tension is indirect and the distribution of strain depends upon joint position [170]. Campaniform sensilla are found in groups whose arrangement is similar, but not identical in different insects [163, 112, 122], (Fig. 2.4). The largest number of groups is found on the trochanter (Groups 1-4 in cockroaches, Groups 1-3 in stick insects; [112, 77]). Another group (fCS) is also found on the proximal femur, adjacent to the trochanterofemoral joint [13, 179], which is functionally a fused joint in stick insects but mobile in cockroaches. Smaller aggregations of sensilla are found on the tibia and individual or pairs of receptors occur on the tarsal segments [134, 160]. In the tibial group of sensilla of cockroaches, the receptor caps have two mutually perpendicular orientations (proximal and distal subgroups; [189, 207]. Forces applied in directions and magnitudes that mimicked loading in upright posture showed that proximal sensilla respond to force increases and distal sensilla discharge when forces decline [168, 170]. Discharges during releases from applied forces are evident in studies of campaniform sensilla in
Principles of Insect Locomotion
51
Fig. 2.4. Anatomy of campaniform sensilla fields in insects. (A) A campaniform sensillum consists of a sensory neuron whose dendrite terminates in a small cuticular cap in the exoskeleton and whose axon projects to the central nervous system. The caps show consistent orientations within a group, as in this scanning electron micrograph of posterior trochanteral sensilla of a stick insect. (B) Campaniform sensilla are found in groups at discrete locations in the leg (photo of cockroach hindleg). Most groups are found on the trochanter and a smaller number of receptors are found on the femur and tibia. (C, D) In stick insects, a large group is located on the proximal femur (fCS), opposite the posterior trochanteral sensilla (pCS). (taken from [204])
other insects [77, 151] and regularly occur in crustacean mechanoreceptors (cuticular stress detectors [146]), but mechanisms underlying these responses are presently unknown. However, these studies showed that the system has active signals both for loading and unloading of the leg. In summary, campaniform sensilla are sense organs that are specialized to encode forces in the legs and can provide the system with detailed data about the direction, rate and magnitude of loads [169]. Interestingly, the grouped campaniform sensilla appear near the basis of the legs [10, 179, 116] where the strain is largest due to the long lever arm. This is different from most technical solutions where force sensors usually are mounted next to the tip of the leg. As for the position sensors, negative and positive feedback effects have been described (e.g., [6, 14, 44, 155, 156, 176, 177, 208]). Schmitz and Stein [182] described the convergence of the reflex effects of load and position sensors on the leg motoneuron pools and Stein and Schmitz [191] reported that already at the level of the primary afferents load and position signals are preprocessed by presynaptic inhibition. 2.3.2
Environmental Sensors
Seen from the viewpoint of a human or a cat, visual sensors appear to be the most important ones that deliver information concerning the actual environmental situation. Although many insects have highly developed eyes, recent findings have shown that their antennae (feelers) are much more important for the investigation of the nearby environment [24, 84, 200, 171], for a review see [190]. It has been argued that using actively moved antennae provide a faster and more robust solution to detect external objects. Extraction of (in particular 3D) information from vision sensors is notoriously slow and prone to all kinds of problems. In contrast, exploration of mechanical contact with the antennae (which, in evolutionary terms, can be considered as specialized
52
H. Cruse et al.
legs) gives fast and reliable information with much less computational load (for more information see Section 2.6: Insect antennae). Other environmental sensors (acoustic, olfactory), of course, exist, but appear not of direct relevance for control of leg movement and will therefore not be considered here. However, they certainly play a crucial role in course control and orientation/navigation.
2.4 Leg Controller Control of walking has to deal with two basic problems, the control of the movement of the individual leg and the spatio-temporal coordination of the different legs. We will first address the former question. The step cycle of the walking leg can be divided into two functional states, swing and stance. These two states differ strongly with respect to their control requirements, particularly because one concerns an open kinematic chain with no or hardly any mechanical coupling with another leg (swing), whereas the other concerns parallel closed kinematic chains, the movement of which is governed my mutual mechanical coupling (stance). Since the control requirements change drastically with a transition between the two states, the transition events, i.e. touch-down and lift-off of the leg, have been in the focus of many behavioural and physiological studies. The anterior (rostral) transition point, the one at touch-down of the leg, is where the transition from swing to stance occurs in a forward walking animal. It is called the anterior extreme position (AEP). The posterior (caudal) transition point where the leg lifts off the ground, is called the posterior extreme position (PEP). The PEP is where the stance-swing transition occurs. AEP and PEP depend on the walking situation, which becomes particularly evident if an animal changes the direction of heading or the orientation relative to gravity. Cruse [41] investigated walking on a horizontal plane, on a small horizontal path, walking up a vertical path and walking while hanging from a horizontal beam. Dean [65] simulated upward and downward walking by running the animal on a treadwheel which was decelerated or accelerated by a torque motor. Both authors showed that the AEP and PEP are dependent on walking conditions. In the following, different aspects of the task to control a walking leg will be considered separately as are for example, swing movement, stance movement or height control. For each aspect a specific controller module will be described that can be used to simulate this property. In a later step these elements will be put together to form a complete system (termed Walknet, Fig. 2.13) appropriate for the simulation of sixlegged walking. 2.4.1
Swing Movement
Not only the location of AEP and PEP, but also the form of the swing trajectory can vary. Cruse and Bartling [49] analysed the swing trajectory in different walking situations as walking freely on a horizontal plane, walking tethered on a treadwheel or on a slippery surface (Fig. 2.5). Application of inverse kinematics allows to compute the time courses of the different joint angles (see [49], Fig. 2.6). How can such a swing movement be controlled? To answer this question, neurophysiological studies have to be performed (see below). In addition, the behaviour of the animal can be observed first, and then
Principles of Insect Locomotion
53
Fig. 2.5. Measured trajectories of swing movements of front, middle, and hind leg. Upper panel: side view (x-z plane), lower panel: top view (x-y plane). Direction of movement is from left to right.
quantitative hypotheses concerning the underlying control structure can be formulated on the basis of simulations. In robots, swing movements are traditionally controlled such as to follow a predefined trajectory. In contrast, biological solutions are assumed to require high flexibility, even during fast swing movements (e.g., retargeted reaching movements, [85]). Many biologically inspired simulations and some robot controllers are therefore based on artificial neural networks. This is not only because they are conceptually nearer to biological systems compared to any other solution, but also because they allow for different methods of off-line and on-line learning, and for error tolerance. Generally, the task of finding a network that produces a swing movement seems to be easier than finding a network to control the stance movement because a leg in swing is mechanically uncoupled from the environment and, due to its small mass, essentially uncoupled from the movement of the other legs. As mentioned above, the leg is considered to contain three hinge joints, the thorax-coxa (α -) joint, the coxatrochanter (β -) and femur-tibia (γ -) joint. Thus, the control network must have at least three output channels: one for each leg joint. Earlier results [43, 71] have shown that the end position of the swing movement of middle and hind legs is determined by the actual position of the front and middle leg, respectively. In other words, the swinging leg uses the position of its anterior neighbour as a target. Principally, this requires the computation of direct and inverse kinematics, but Dean [62] could show that a simple, three-layered feedforward network could solve the task sufficiently well. This “target net” receives as input the three actual angles α , β and γ of the anterior leg and provides as output the desired angles αt , βt and γt of the targeting leg (Fig. 2.6). In other words, this net directly associates desired final joint angles at touch down depending on the current joint angles of a rostral leg, such that the tarsi of the two legs are at the same position [62]. There is no explicit calculation of either tarsus position. Physiological recordings from local and intersegmental interneurons [31] support the hypothesis that a similar approximate algorithm is implemented in the nervous system of the stick insect.
54
H. Cruse et al.
Fig. 2.6. Modelling mechanically uncoupled leg movements with swing-net. a) Swing-net is an ANN consisting of three neuroids, the output of which sets angular velocity of the TC- (dα /dt), CT- (dβ /dt) and FT-joint (dγ /dt). Each neuroid receives weighted input (solid circles) from external sensors that signal the current joint angles (α , β , γ ) and obstacle contact (r1 to r4). A constant input models internal resting activation (1). Further inputs are the target angles (αt , βt , γt ) from the target net. The control loop is closed via the leg itself, as movement changes the posture and, thus, the sensory input. The target net consists of six neuroids that receive sensory input signalling the current joint angles of the next anterior leg (α1 , β1 , γ1 ). b) Example trajectories of swing-net, simulating normal protraction (dotted line) and an avoidance reflex (dashed line) upon obstacle contact (arrow). The reflex trajectory matches experimental results, i.e. average tarsus positions at different times (solid circles with error bars; ± S.D.) after mechanical disturbance of a swing movement (data from [178]).
Using this hypothetical network for the determination of the endpoint of the swing movement, another net is required for the control of the actual swing movement. A simple, one-layered feed-forward net (Fig. 2.6, swing-net) with three output units and six input units can produce movements that closely resemble the swing movements observed in walking stick insects [49]. The inputs correspond to three angles defining the current leg configuration and three target angles defining the configuration desired at the end of the swing. In the simulation, the three outputs, are set as the angular velocities of the joints, dα /dt, dβ /dt, and dγ /dt. During simulation, they are formally integrated (not shown in Fig. 2.6) to obtain the joint angles. In the animal, the movement of the leg itself can be considered as an integration of the muscle activities. Average frequency of action potentials during a motoneuron burst often show a linear relation to the mean joint velocity, e.g., during running [199] and searching [195]. The actual angles resulting from the integration and any external disturbance are measured and fed back into the net. Through optimization, the network can be simplified to only 8 (front and middle leg) or 9 (hind leg) non-zero weights (for details see [50]). Despite its simplicity, the net not only reproduces the trained trajectories, it is able to generalize over a considerable range of untrained situations, demonstrating a further advantage of the network approach. Moreover, the swing-net is remarkably tolerant with respect to external disturbances. The learned trajectories create a kind of attractor to which the disturbed trajectory returns. This compensation for disturbances occurs
Principles of Insect Locomotion
55
Fig. 2.7. Vector field representing the movement of the tarsus of a left front leg produced by the swing-net. Upper panel: Projection of a parasagittal section, i.e., the x-z plane shown in the right figure (y = 12mm). Lower panel: Projection of a horizontal section, i.e., the x-y plane slightly below the leg insertion (z= -3mm). Left is posterior, right is anterior. The average posterior extreme position (start of swing movement) and of the average anterior extreme position (end of swing movement) are shown by an open square and by a closed square, respectively.
because the system does not compute explicit trajectories, but simply exploits the physical properties of the world. The properties of this swing-net can be described by the 3D vector field in which the vectors show the movement produced by the swing-net at each tarsus position in the workspace of the leg. Fig. 2.7 shows the planar projections of one parasagittal section (a) and one horizontal section (b) through the work space. The complete velocity vector fields are similar to the force vector fields shown by [22], the major difference being that force vectors would be expected to be related to joint acceleration rather than velocity. In a detailed study using genetic algorithms Linder [143] could show that, for the middle leg, 7 non-zero weights are sufficient. Concentration on six of these weights allows to interpret the swing-net as to consists of 3 negative feedback controllers, one for each joint, whereby the controller of the thorax-coxa joint (α -joint) and of the coxatrochanter joint (β -joint) are coupled via the 7th weight. All 18 weights showed a small variability with the exception of this 7th weight. It is responsible for varying the maximum height of the swing movement and might therefore be subject to a learning procedure that changes this weight depending on the ruggedness of the substrate. Levator reflex: The ability of swing-net to compensate for external disturbances permits a simple extension in order to simulate an avoidance behaviour observed in insects.
56
H. Cruse et al.
When a leg strikes an obstacle during its swing, it initially attempts to avoid it by retracting and elevating briefly and then renewing its forward swing from this new position [70, 88]. In the augmented swing-net, an additional input similar to a tactile or force sensor signals such mechanical disturbances (Fig. 2.6 a, r1). This unit is connected by fixed weights to the three motor units in such a way as to produce the brief retraction and elevation seen in the avoidance reflex. Quite interestingly, recent findings by Ebeling and D¨urr [88] show that the avoidance reflex during swing movements of walking stick insects is independendent of the kinematics of the swing movement. That is, it is context-insensitive, even if swing direction differs considerably, for example during curve walking. Moreover, although the typical reflex action consists of combined rectraction, levation and flexion, these default actions per joint were shown to occur independently of each other.
Fig. 2.8. Leg searching-movements of a stick insect walking on rough terrain. The experimental situation was such that a stick insect reached the edge of a bridge (front leg) or crossed a gap between two bridges (middle and hind leg). In case of lacking foothold at the end of protraction, the leg performed characteristic searching-movements that look different for each kind of leg. Average trajectories of the tibia-tarsus-joint (heel) are plotted in a body-fixed coordinate frame, centred on the α -joint of the respective leg. Only the horizontal component of the movement is shown (xy-projection). (Fig. 1 from [86])
The described swing-net is able to produce swing movements as can be found in free walking animals in a very simple way. This neural network has been extended to account for a wider range of movements. Swing-net 2 [185] introduced a velocity factor which enforces steady movements by keeping the velocity constant. This leads to more natural movements regarding the dynamics of the movement and especially allows for searching movements, as can be found in the insect: An important adaptation of a swing movement can be observed in the stick insect when it does not find ground at the normal end of the swing movement. When the animal then steps into the hole, the leg performs a kind of search movements that consist of more or less regularly oscillating
Principles of Insect Locomotion
57
movements (locust: [157]; cockroach: [74, 195]; stick insect: [131, 18, 80, 23]). In stick insects of the species Carausius morosus, searching movements were shown to be stereotypic in that the tarsus movements always follow characteristic trajectories [80]. Fig. 2.8 shows how these trajectories differ between front, middle and hind legs. In front legs, tarsus loops are superimposed by retraction of the thorax-coxa joint. In middle legs, this retraction component is missing, in hind legs it is reversed into a continued protraction. Thus, each leg appears to search progressively towards the body centre. Careful analysis of the leg kinematics revealed no sign of switching from swing movement to a searching movement. In fact, D¨urr [80] showed that swing-net is able to model a swing movement with a terminal set of searching loops without the need of an additional controller (as it had been previously suggested, e.g., by Espenschied et al. [90]). Rather, searching movements may be viewed as unterminated swing movements with the oscillatory component reflecting the damping properties of the network controller. 2.4.2
Stance Movement
In insecta, leg movement during stance can be divided into two subtasks, movement in vertical direction (z-axis, see Fig. 2.7) and movement in the horizontal plane (x-y plane, see Fig. 2.7). The latter concerns forward and lateral movements, the former refers to control of body height. At first, we want to look at the movements in vertical direction: Measurement of correlation between body height and vertical force in open loop experiments and in closed loop experiments indicate that body height is controlled by a proportional negative feedback system [58]. Dynamical studies in walking animals [58] have shown that the system reacts very fast (time constant smaller than 20 ms), but it is open whether this reaction is due to passive viscoelastic properties or whether there are active elements contributing. Standing animals (stick insect: [133]; cockroach: [153]) have also been investigated considering both the complete animal and individual legs [56]. In these experiments the complete animal, i.e. with all legs contributing to height control are regarded. Simulations of measurements of animals walking over obstacles of different shapes support the idea that each leg represents an independent height controller [42]. This could directly be shown for standing animals [56]. In addition, the angle of the thorax-coxa joint shows a proportional (= spring-like) change that depends on the form of the substrate [42]. In the complete controller structure for a walking animal, Walknet, height control is performed by using a simple feedforward network (height net) that determines the required coxa-trochanter angle (β angle) individually in each leg depending on the actual joint values of the leg. Considering leg movement in the x-y plane, stance trajectories can be described by a straight line when the animal walks tethered on a treadwheel, which is a mechanical necessity. This is different for free walking animals (Fig. 2.9), where stance trajectories are more or less curved when plotted in a body fixed coordinate system [135]. To some extent, this corresponds to the body showing lateral oscillations even in straight walking. These oscillations result from the combined forces developed by the different legs that simultaneously perform a stance movement. These forces depend very much on the actual walking situation. Fig. 2.10 shows schematic time courses of the ground reaction
58
H. Cruse et al.
Fig. 2.9. Comparison of walking stick insects with Walknet: straight and curve walking. a) Stance trajectories of all six feet during forward walking in an unrestrained walking stick insect, Carausius morosus. Walking direction is toward the right. Solid circles mark the locations of the CTjoints. Trajectories are curved and vary from step to step. b) In comparison, stance trajectories of the Walknet simulation (top row, left column) are shorter and less variable, yet curvature and the nearly connecting trajectories of middle and hind legs are in very good agreement with the stick insect data. Left and right columns show the impact of walking speed on stance trajectories. Middle and bottom rows depict the impact of altered yaw turning commands (yawre f in Fig. 2.11). Turning direction is leftward, i.e. to the top of the diagrams. Trajectories are more variable than during straight walking and appear more curved.
Principles of Insect Locomotion
59
Fig. 2.10. Schematic presentation of ground reaction forces measured in different walking situations. a) walking on a horizontal plane, b) walking on a horizontal path (width 30 mm), c) walking up a vertical path d) walking when hanging from a horizontal beam. x-axis: positive forces propel the body in forward direction, y-axis: positive forces point away from the body, z-axis: positive forces decrease the clearance, i.e. the distance between body and substrate.
forces for front, middle, and hind legs in different walking situations: walking on a horizontal plane, walking on a horizontal path where the tarsi grasp the margin of the substrate from the side, walking up a vertical path, and walking when hanging from a horizontal beam [41]. It is immediately obvious that the forces developed by the legs are quite different in different situations, which means that it is difficult to attribute a given function to a given
60
H. Cruse et al.
leg. Leg function rather appears to depend on the actual situation. For example, when walking on the horizontal plane, front legs seem to act as feelers and do not provide significant forces, middle legs seem to function like passive struts, and hind legs appear to provide thrust. Quite in contrast, when walking up a vertical path, front legs develop strong forces in contrast to the hind legs. This appears also to be the case when walking on a heavy treadwheel [45]. It is not clear to what extent these different behaviours simply result from the changed geometrical situations with unchanged controller output (e.g., height control) and to what extent the controller adapts its output signals. Full, Blickhan and coworkers measured the ground reaction forces developed by walking cockroaches. They considered the total force produced by the complete animal [100, 101] and forces separately produced by the single legs. The latter results were qualitatively similar to those found for the stick insect walking on a horizontal plane (Fig. 2.10 a), apart from the fact that in stick insects and in this specific situation front legs produce much less force compared to cockroaches. Ting et al. [194] used the single leg forces to compute the position of the center of pressure which was shown to perform periodic movements around the center of mass. The type of the controller that governs motor output during stance is still disputed. Apart from position PD control, many different control principles like velocity control [68], force control, positive force feedback [165, 164] and positive velocity feedback [51] have been proposed. An attractively simple solution is discussed by Full and Blickhan (see [128], and the above references) and is investigated by theoretical methods [175]. These results show that an extension of the spring-loaded inverted pendulum model [25], which can explain the mechanics of legged locomotion in a sagittal plane, can also be applied to insects using sprawled leg posture. This lateral leg-spring model describes the stability of the walking animal in the horizontal plane. It could be shown that the passive properties of the model are sufficient to cope with disturbances as for example a lateral push to the body without the need of any active control system. Other results indicate that at least in slow walking sensory feedback plays an important role. In the stick insect, Bartling and Schmitz [6] have measured the ground reaction forces when the animal walks along a horizontal path. In the critical experiment, the platform on which the leg stepped was moved by a short distance while the reaction forces were still being measured. The results are not in accord with the idea of positive feedback, but indicate that the disturbance is counteracted by a negative feedback system. However, this counteraction can only be observed during the dynamical part of the disturbance. The detailed response appeared to depend on the compliance of the substrate. To investigate this in more detail, Cruse et al. [53] looked at the reaction to disturbances in standing animals. The results were interpreted in such a way that the joints are under integral position control when the substrate is very soft, but show properties of a proportional controller for stiffer substrate. If the substrate is stiff enough, the feedback system shows properties of a D- controller. As an aside, in these experiments a reflex has been found that had also been observed in other animals (see [205], cockroach, locust). When a joint is stretched too far, a short step is performed, the function of which appears to be to move the joint away from its mechanical limits.
Principles of Insect Locomotion
61
Force distribution problem: A completely different solution concerning the control of stance is the above mentioned positive velocity feedback. This idea is supported by results of B¨assler [11, 16] and Schmitz et al. [177] on stick insects as well as by results of Dicaprio and Clarac [78] and Sillar et al. [188] on crustaceans, who described the reversal of a reflex from a resistance reflex to an assistance reflex (positive feedback) when the animal changes from a passive to an active state. This idea can be applied to the complete animal: positive feedback could have the interesting advantage that coordination between the different joints of one leg and even of neighboring legs is not necessary on the neuronal level. Instead the mechanical coupling already given would suffice. Kinematic simulation studies using the principle of positive velocity feedback were successful [52]. Moreover, the same approach was successfully applied to a real planar, two joint manipulator in a crank-turning task [183] and was also proofed feasible in a dynamics simulation of a walking 3D-test leg [184]. This so called sLPVF controller switches from positive feedback to compliant motion depending on whether the joint produces positive or negative mechanical power, respectively. This approach therefore allows for an extreme simplification of the control architecture. This solution can also cope with sudden changes in the morphology (e.g., loss of part of a leg). All the mentioned approaches look at the joints on a local level, without requiring knowledge about the state of another leg and therefore circumventing the main difficulty and problem controlling the stance movement: Actually, the task of controlling the stance movements of all the legs on the ground poses several major problems. To avoid interaction forces between the legs, it is not enough to simply specify a movement for each leg on its own: the mechanical coupling through the substrate means that efficient locomotion requires coordinated movement of all the joints of all the legs in contact with the substrate. Thus, a total of 18 joints must be controlled simultaneously when all legs of an insect are on the ground. However, the number and combination of mechanically coupled joints varies from one moment to the next, depending on which legs are lifted. The control of rotational joints is a nonlinear task, particularly when the rotational axes of the joints are not orthogonal, as is often the case for insect legs and for the basal leg joint in particular. A further complication occurs when the animal negotiates a curve, which requires the different legs to move at different speeds. In walking machines, these problems are often solved using traditional, though computationally costly, methods, which consider the ground reaction forces of all legs in stance and seek to optimize some additional criteria, such as minimizing the tension or compression exerted by the legs on the substrate. Due to the nature of the mechanical interactions and inherent in the search for a globally optimal control strategy, such algorithms require a single, central controller; they do not lend themselves to distributed processing. This makes real-time control difficult, even in the still simple case of walking on a rigid substrate. For a slow and inexact biological network this calculation poses an even larger problem. Further complexities arise in more complex, natural walking situations, making a solution difficult even with high computational power. These occur, for example, when an animal or a machine walks on a slippery surface or on a compliant substrate, such as the leaves and twigs encountered by stick insects. Any flexibility in the suspension of the joints further raises the number of the degrees of freedom that must be
62
H. Cruse et al.
considered, increasing the complexity of the computation. Further problems for an exact, analytical solution occurs when the length of leg segments changes during growth or their shape changes through injury. In such cases, knowledge of the geometrical situation is incomplete, making an explicit calculation difficult, if not impossible. Despite the evident complexity of these tasks, they are mastered even by insects with their “simple” nervous systems. Hence, there has to be a solution that is fast enough that on-line computation is possible even for slow neuronal systems. How can this be done? Several authors [29] have pointed out that some relevant parameters do not need to be explicitly calculated by the nervous system because they are already available in the interaction with the environment. This means that, instead of an abstract calculation, the system can directly exploit the dynamics of the interaction and thereby avoid a slow, computationally exact algorithm. To solve the particular problem at hand, we propose to replace a central controller with distributed control in the form of local positive displacement feedback [50] as introduced above. Compared to earlier versions [51], this change permits the stance-net to be radically simplified. The positive displacement feedback occurs at the level of single joints: the position signal of each joint is fed back to control the motor output of the same joint (Fig. 2.11, stance-net). How does this system work? As a ’thought experiment’ consider an initially stationary system in which one joint begins to move actively. Then, because of the mechanical connections, all other joints begin to move passively, but in exactly the proper way to maintain the mechanical integrity of the system. Thus, the movement direction and speed of each joint does not have to be computed explicitly, because this information is already provided by the physics. In each joint, the angular change can be registered by appropriate sense organs (Sect. 2.3), and this value can then be used as a command to move this joint in the following time step by the same amount. This corresponds to positive displacement feedback that transforms this passive movement into an active movement. This command does not provide a mathematically exact solution. It rather provides an approximation that is the better, the smaller the chosen time step is. Of course, if a single, random joint is selected for the active movement, the direction in which the body moves probably will not be straight forward, but will vary depending upon the joint chosen. Such irregular movements are avoided because (i) at the start all legs initiate forward, propulsive forces (as is the case in insects [57]) and (ii) a global supervisory system (see below) controls walking direction. There are, however, several problems to be solved. The first is that positive displacement feedback, using the raw position signal may lead to an exponential increase in movement speed, as opposed to the nearly constant walking speed that is usually desired. This problem can be solved by introducing a kind of band-pass filter into the feedback loop. The effect is to make the feedback proportional to the angular velocity of joint movement, not the angular position. In the simulation, this is done by feeding back a signal proportional to the angular change over the preceding time interval. The second problem is that using positive displacement feedback for all three leg joints leads to unpredictable changes in body height. Body height of the stick insect is controlled by a distributed system in which each leg acts like an independent, proportional controller with nonlinear characteristics (walking animal: [42, 58], standing animal: [56]. However, maintaining a given height via negative feedback appears at odds with
Principles of Insect Locomotion
63
Fig. 2.11. Modelling joint coordination during stance, using positive displacement feedback. The stance-net consists of two output units that set the angular velocity of the TC- (dα /dt) and FTjoint (dγ /dt) during stance. As the CT-joint mainly affects height of the body relative to its feet, it is controlled by a separate ANN (height net in Fig. 2.13 b). The stance-net receives sensory input from proprioreceptors signalling angular velocities, i.e. closing a positive displacement feedback loop. Also, it receives central commands that determine the velocity of forward translation (vre f *vmod ) and yaw rotation (yawre f ). Both of these signals are subtracted from a corresponding sensory input signal, thus closing negative feedback loops. A ‘walking on’ signal sets a minimum retraction velocity (via a threshold operator ‘max’) to stabilise retraction against disturbances of the positive feedback signal dα /dt. In addition, one of the coordination influences can be modelled on this level: Coordination rule 5 can be partially modelled by adding a co-activation signal to the sensory input. Passive movements imposed on the TC- or FT-joint are sensed by the stancenet and transformed into an active movement, implementing a local assistance reflex.
the proposed local positive feedback for forward movement. How can both functions be fulfilled at the same time? To solve this problem we assume that during walking positive displacement feedback is provided for the α joints and the γ joints (Fig. 2.11, stance-net), but not for the β joints. The β -joint is the major determinant of the animal’s clearance, i. e., the body height. A third problem inherent in using positive displacement feedback is the following. Let us assume that a stationary insect is pulled backward by gravity or by a brief tug from an experimenter. With positive feedback control as described, the insect should then continue to walk backwards even after the initial pull ends. This has never been observed. Therefore, we assume that a supervisory system exists that is not only responsible for switching on and off the entire walking system, but also specifies walking direction (normally forward for the insect). This influence is represented by applying a small, positive input value (Fig. 2.11, “walking on”) that replaces the sensory signal if it is larger than the latter (the box “max” in Fig. 2.11, stance net).
64
H. Cruse et al.
To test whether this stance controller is able to contribute in a sensible way to walking, a simulation of the complete walker has to be performed in the future - in contrast to swing-net, that can be tested independently. Before this can be done, two additional questions have to be considered, the control of changes between the states of swing and stance, and the spatio-temporal coordination of the quasi-rhythmic movement of (neighbouring) legs, However, already here it can be stated that the principle of positive velocity feedback to couple different legs without neuronal connection is very much related to the coupling of different individuals (e.g., ants) to carry a common load without requiring a central command. Successful application of this principle would mean that the control of legs in stance could be reached by an extremely decentralized control architecture as there are only independent joint controllers that are only coupled by the mechanics of the legs. Coordination of different legs is however necessary at a higher level as will be shown below (see Coordination of different legs, Sect. 2.5). We have now discussed the control of a leg during swing and during stance, but not yet addressed the important question of how the system can switch between these two states. The targeting behaviour clearly indicates that information concerning the point where the anterior extreme position, i.e. the point where the leg may switch from swing to stance is provided by the anterior leg (or possibly the antenna in the case of the front leg). However, as this information may not be reliable, and is not used in all cases additional information may be required. An extreme case occurs in curve walking, where stance direction of front legs is rotated by some 70◦ or more [83]. Although the distance between front leg lift-off position (PEP) and middle touch-down position (AEP) is a lot larger than during straight walking, the scatter of the AEP remains very small. Ebeling and D¨urr [88] have interpreted this as an indication of the targeting mechanism being still at work, thus causing the small scatter, but with a target location that is counter-rotated with respect to the front leg stance direction. The transition from swing to stance state is usually assumed to being simply triggered by a ground contact sensor and/or a load sensor. This means, as soon as the leg is under load, it switches to stance mode [201, 76, 2]. This has been shown by Wendler [201] who replaced the tibia and part of the femur by an artificial prosthesis. The animal continued to perform appropriate swing and stance movements. Neurophysiological results supporting this view [2] will be described in detail below. However, there might be a problem because as mentioned in the case of avoidance reflexes loading the leg leads to a levator reflex, for example, but not to a change from swing to stance. Experimental results with stick insects have shown that there is also an internal, “motivational” state that is used to interpret sensory input during swing: the same physical stimulus is interpreted as an obstacle or as ground, depending of the level of “swing motivation”. How is a “swing motivation” produced? Experimental results showed that it is not simply the position of the leg nor time elapsed since the beginning of the swing. Experimental results [52] suggest that it is the distance between front and middle leg, which determines the ,,motivation“ for the middle leg to accept a stimulus as ground contact. A neuronal circuit that is able to describe these effects is given by Cruse et al. [52]. What is actually known concerning the switch from stance to swing? This switch happens as soon as a given position (posterior extreme position, PEP) is reached. This has been shown for insects, crustaceans, and cat, and is used in most robots. However,
Principles of Insect Locomotion
65
Fig. 2.12. Selector-net. This net determines whether swing-net or stance-net has access to the motor output. Input GC: ground contact, PEP: threshold determining the posterior extreme position.
also the loading conditions of the leg play an important role (see Chapter 8). A leg under load is prevented from beginning a swing [155, 13]. Furthermore, a sudden deloading can elicit a swing movement, provided that a given position has been reached [46]. How could these sensory inputs — leg under load — position of the leg, influence the state of the leg? As only limited knowledge is available concerning the underlying neuronal system [17], a simple circuit has been proposed based on theoretical considerations [55]. This “Selector-net” contains two inputs, ground contact, or load, and leg position (Fig. 2.12). Both inputs influence, with opposite signs, two units that represent the swing mode or the stance mode, respectively. In the simple version described here these units take only Boolean values (0 or 1). Each of these units receives positive feedback on itself that stabilizes the decision made, swing or stance, to some extent. The output of these units control whether swing-net or stance-net determines the actual motor output. Fig. 2.13 b shows how these modules, which, apart from the PEPnet, have been described above, are connected. In terms used for neuronal networks, this arrangement could be called a system containing two expert nets, swing-net and stance-net, and a gating net, the selector-net. In a later expansion (“Analogous Selector”, see Schilling et al. [174]) the activation of these two units take analogue values and might also be termed to represent swing motivation or stance motivation, respectively.
2.5 Coordination of Different Legs How could these leg controllers that essentially consist of a swing-net, a stance-net, and a selector-net be coupled? A possible solution for the coordination of those legs that actually are performing a stance movement has been described above. Another important task is to coordinate the changes from stance to swing, because this transition dramatically influences the stability of the walking system. Traditional solutions apply – in the case of a hexapod – a fixed
66
H. Cruse et al.
Fig. 2.13. Implementation of coordination rules in Walknet. a) Movements of each leg are controlled by separate single-leg controllers (boxes labelled as in Fig. 2.17). Each leg receives information from its ipsilateral and contralateral neighbours (dotted arrows), affecting the likelihood of a stance-swing transition. Moreover, global commands from a higher control level (dashed arrows) model descending information from the brain. b) Each single-leg controller (e.g., R2) consists of several ANN modules. The swing-net and stance-net generate leg movements during protraction and retraction, respectively. Their outputs set the joint angular velocities of the three leg joints (TC, CT and FT, equivalent to α , β and γ ). They are gated by the selector net in a mutually exclusive manner. The state of the selector-net depends on sensory input from the own leg (half-circles inside the grey area indicate local sensory information) and the summed effects of the PEP net, coding the distance to the normal PEP, and weighted information corresponding to coordination rules 1 to 3 (dotted arrows). Thus, rules 1 to 3 affect stance-swing transition. Rule 4 acts via sensory information from the next anterior leg (dotted half-circles) that is transformed into a target posture by the target net (see also Fig. 2.6 a). Thus, rule 4 affects the swingstance transition. The height net controls the body clearance and affects only the CT-joint during stance. Except for the input to the target net, all sensory inputs are local. The stance-net receives three global commands (dashed arrows), controlling body velocity (v) and yaw (y) and forward walking (w).
alternating tripod gait: front and hind leg of one side move together with the middle leg of the other side. This has often been termed the typical insect gait (Fig. 2.19). It should be mentioned here that the term tripod gait is used in two ways differing in detail: According to the strictest definition, in each moment of time there are three legs lifted and three legs on the ground. This means that swing duration and stance duration are the same for a given walking speed, but depend on walking speed. In contrast, in adult stick insects, for example, swing duration is always the same, independent on walking speed. Therefore, Graham [107] defined tripod in the way that three legs swing together. Thus, in the stick insect tripod there are periods where all six legs are on the ground. Many
Principles of Insect Locomotion
67
insects show an intermediate situation. Swing duration depends on walking speed, but is shorter than stance duration. Therefore, also in these cases we have periods where all six legs are on the ground. Different to any of these cases, slow walking insects use the so called tetrapod gait (Fig. 2.19): at least four legs are at the ground at any moment of time [107, 108]. The terms tripod gait and tetrapod gait may be misleading. As has been shown by Graham [107], there are no separate categories of gaits. Rather there is a continuum depending on walking speed. The higher the speed (and probably the smaller the load), the more the tetrapod gait resembles a tripod gait. There is another problem with the terms tripod and tetrapod gait: Both terms imply that the cyclic movement of the legs can be described by fixed phase relations and therefore fixed phase shifts have been used in the control of walking robots. However, in walking animals such fixed gait patterns can only be observed in continuously undisturbed walking situations. In very slow walking — think of a grazing horse, for example — no fixed phase values can be observed. A closer inspection has shown that gait controllers in animals are better described as free gait controllers. Nevertheless, such a controller produces an apparently regular gate when the walking speed is not too slow and when no disturbances occur. The basic results supporting this view are known since long. v. Holst has shown for the control of fish fins, for legs of walking dogs [114] and insects [115] that each leg or fin comprises an oscillator having its own stepping period and that these oscillations could be coupled more or less strongly. This has later been shown for legs of stick insects [201] and lobsters [36]. Strong coupling leads to fixed phase relations, termed absolute coordination by v. Holst, whereas weak coupling still shows a preferred phase value, but other phases values might also occur (“relative coordination”). How are these legs coupled? The underlying control system has been studied in detail for insects, crayfish and cat. There are different local rules that are active between directly neighboring legs. Three such rules have been describe for the crayfish, five similar, but different in detail, for the stick insect (see Fig. 2.2; [47]. For the cat, four mechanisms have been described [60]). Fig. 2.14 shows the effect of a brief interruption of stance movement of one leg in the crayfish. The leg resumes normal coordination by shortening or prolonging the swing movement and/or the stance movements of some of the neighbouring legs. To illustrate the evaluation procedure, two responses to disturbances of legs 3 and 4 are shown. The situations in which prolongation of a swing occurs are presented in Fig. 2.16 in a
Fig. 2.14. Two plots showing the behaviour of crayfish leg movement after a brief disturbance (prolongation of stance of leg 4). Depending on the timing of the disturbance, shortening and prolongation (a) and shortening (b) of steps of neighbouring legs can be observed. Legs are numbered from 2 to 5. Upward lines show swing movements. The abscissa denotes time.
68
H. Cruse et al.
form similar to a phase-response curve. The sketches below the abscissa symbolize the rhythmic movement of the two legs in a normally coordinated walk (solid lines). The values on the abscissa are given as absolute values rather than as relative phase, which is usual in a phase-response curve. The ordinate does not show the absolute duration, but rather the difference relative to the swing duration of a normal step. Thus, the zero value corresponds to an unchanged swing.
Fig. 2.15. Two examples of phase response curves from data for legs 3 and 4. Reference leg is leg 4 (a) or leg 3 (b). Ordinate shows change in duration of swing (see bold lines in the lower inset figures). Abscissa shows beginning of measured swing relative to reference point (t0).
Leg Coordination in Crayfish: a Simple Case: How can a system be constructed to produce a stable spatiotemporal pattern and, at the same time, to tolerate disturbances? To investigate the properties of such a system, its reaction to different disturbances such as brief interruptions of the stance movement of one leg have been observed. As the situation in crayfish appears to be simpler than in the stick insect, we will first concentrate on leg coordination in crayfish. In Fig. 2.15 a, leg 4 is chosen as reference leg with the reference point (t0 ) being the end of swing movement. As depicted in Fig. 2.15 a, the changes in swing duration are very large when the moment where the swing of leg 3 would have ended had it not been influenced occurs just after the AEP of the backward leg. The prolongation of the swing duration decreases with decreasing lag between the start of swing and the reference point. Had the slope of the points shown in Fig. 2.15 a reached -1.0, then the prolongation would have completely compensated for the wrong phase value. Two such deviating steps are schematically shown below the abscissa (dashed lines). Fig. 2.15 b depicts results in which the duration of the swing is either shortened or remained almost the same. In this figure, leg 3 is chosen as reference leg with the reference point (t0 ) being the end of stance movement The duration of the swing seems to be prolonged to some extent when the backward leg starts its protraction “too early”. The duration of the swing is clearly shortened when it is started “too late” relative to a normal step (see dashed lines in the schema below the abscissa). The remainder of the compensation is accomplished by changing the duration of the following stance
Principles of Insect Locomotion
69
Fig. 2.16. Coordination between ipsilateral legs in crayfish. a) rostrally directed influence prolongs swing movement, b) caudally directed influence shortens swing movement. The left part of each figure shows body and range of movement of two neighbouring legs (upward is anterior). The right part indicates the movement of the reference leg; posterior leg in (a), anterior leg in (b). The behaviour of the other leg is schematically shown by plotting several traces with different phase shifts. The wedges and arrows indicate the coordinating influences.
movement as indicated by the schematic below the abscissa. The collection of a large data set led to the interpretation that two coordinating rules are active in the crayfish. One is directed rostrally. It prolongs the swing of a leg as long as its posterior neighbour performs a stance (see Fig. 2.15 a). The other is directed caudally. When the anterior leg is close to the end of its stance or at the beginning of its swing phase, this second influence has the effect of ending the swing of the posterior leg and beginning the stance of this leg (see Fig. 2.15 b). This is summarized in Fig. 2.15. A third influence acting between each contralateral pair of legs will not be considered here (see [47]). Rules for Leg Coordination between Legs in Stick Insects For the stick insect, six different coupling rules have been found in behavioural experiments. These are summarized in Fig. 2.17 a. One influence (6 in Fig. 2.17 a) serves to correct errors in leg placement; another (5) distributes the propulsive force among the legs. These will not be considered in the following section. Only the remaining four are used in the present model. The beginning of a swing movement (PEP) is shifted by three rules arising from ipsilateral legs (Fig. 2.17 b): (1) a forward directed inhibition during the swing movement prevents lift-off of the next anterior leg, (2) a forward directed excitation soon after the begin of active retraction supports lift-off of the next anterior leg, and (3) a rearward directed influence depending upon the position of the next rostral leg increases the chance of lift-off with increasingly posterior position. Influences (2) and (3) are also active between contralateral legs. The end of the swing movement (AEP) in the animal is modulated by a single, caudally directed influence (4) depending on the position of the next rostral leg. This rule is responsible for the targeting behaviour - the placement of the tarsus at the end of a swing close to the tarsus of the adjacent rostral leg. Influence (4) affects the touch down location set by the swing-net, as has been described above. Influences 1, 2 and 3 are implemented as incoming scalar values that are simply summed and shift the threshold characteristic (PEP). The characteristic determines how much a standard stance movement is shortened or prolonged. This is illustrated by
70
H. Cruse et al.
Fig. 2.17. a) Summary of the coordination rules operating between the legs of a stick insect [47]. The leg controllers are labelled R and L for right and left legs and numbered from 1 to 3 for front, middle, and hind legs. The different mechanisms (1 to 6) are explained in the text. b) – d) Illustration of the effects of rules 1, 2 and 3, respectively.
the summation element in the selector-net of Fig. 2.13 b. The strength of the different coordination influences depends on the current behavioural context [83, 81]. The interleg influences are mediated in two parallel ways. The first pathway comprises the direct neural connections between the step pattern generators. The second pathway arises from the mechanical coupling among the legs. That is, the activity of one step pattern generator influences the movements and loading of all legs and therefore influences the activity of their step pattern generators via sensory pathways. This combination of mechanisms adds redundancy and robustness to the control system of the stick insect [69]. In a modeling study using Walknet, Kindermann [135] studied the complete sixlegged walking system by investigating its behaviour in different, specific situations. To permit the system to control straight walking and to negotiate curves, he introduced a supervisory system that, in a simple way, simulates the optomotor mechanisms for course stabilisation that are well-known from insects and have also been applied in robotics. This system can also compensate for unbalanced coupling factors or other inequalities between right and left legs. This supervisory system uses information on the rate of yaw (“yawsens”, Fig. 2.11 b, stance net), such as visual movement detectors might provide. It is based on negative feedback of the deviation between the desired
Principles of Insect Locomotion
71
Fig. 2.18. Simulated walk by the basic six-legged system with negative feedback applied to all six β joints and positive displacement feedback to all α and γ joints as shown in Fig. 18b. Movement direction is from left to right (arrow). Leg positions are illustrated only during stance and only for every second time interval in the simulation. Each leg makes about five steps. Upper part: top view, lower part: side view. (a) Straight walking (θyaw re f = 0). (b) curved walking (θyaw re f = 0).
turning rate and the actual change in heading over the last time step. The error signal controls additional inputs to the α joints of the front and hind legs that have magnitudes proportional to the deviation and opposite signs for the right and left sides. In earlier versions, this bias was given to the front legs only. A much better behaviour can be found when the bias is also given to the hind legs. With this addition and yawre f set to zero, the system moves straight (Fig. 2.18 a) with small, side-to-side oscillations in heading such as can be observed in walking insects [135]. To simulate curve walking (Fig. 2.18 b), a small positive or negative bias is added to the reference value that determines curvature and turning direction. However, as mentioned above, this simulation did not apply adaptive coupling strengths, in contrast to what has been found in real animals. Finally, we have to address the question of how walking speed is determined when such a positive displacement feedback controller is used. Again, a central value is assumed which represents the desired walking speed vre f . This is compared with the actual speed, which could be measured by visual inputs or by monitoring leg movement. This error signal is subject to a nonlinear transformation and then multiplied with the signals providing the positive feedback for all α and γ joints of all six legs (Fig. 2.11 b, stancenet). Therefore, another way to describe the function of the stance net is the following: the complete system is regarded as a negative feedback system controlling walking speed vre f . Within this system, the positive feedback signals provide the gain factors for the individual joints. Like the earlier version [51] the presented model shows a proper coordination of the legs for walks at different speeds on a horizontal plane. Steps of
72
H. Cruse et al.
ipsilateral legs, i.e., legs on the same side of the body, are organized in triplets forming “metachronal waves”, which proceed from back to front, whereas contralateral legs, i.e., legs on opposite sides, on each segment step approximately in alternation. With increasing walking speed, the typical change in coordination from the tetrapod to a tripod-like gait is found. For slow and medium velocities the walking pattern corresponds to the tetrapod gait with four or more legs on the ground at any time and diagonal pairs of legs stepping approximately together; for higher velocities the gait approaches the tripod pattern with front and rear legs on each side stepping together with the contralateral middle leg. This is possible because swing duration is constant [108]. Simulation results are shown in a “bar code” plot (bars indicate stance, gaps indicate swing) in Fig. 2.19. Fig. 2.20 shows an example of a tetrapod coordinating pattern. With higher speed stance duration shortens. This gradually changes the gait from a tetrapod to a tripod coordination. For a set velocity, the coordination pattern is very stable. For example, when the movement of one leg is interrupted briefly during the power stroke, the normal coordination is resumed immediately at the end of the perturbation. Furthermore, the model can cope with obstacles higher than the normal body clearance. It continues walking when a leg has been injured, such that, for example, half of the tibia is removed Unexpectedly, the following interesting behaviour was observed in the simulation. A massive perturbation, for example by clamping the tarsi of three legs to the ground, can make the system fall (Fig. 2.21). Although this can lead to extremely disordered arrangements of the six legs, the system was always able to stand up and resume proper walking without any help (for details see [135]). This means that the simple solution proposed here also eliminates the need for a special supervisory system to rearrange leg positions after such an emergency. Further improvements of Walknet have been introduced by Roggendorf [173], who tested the limits of the system when walking over obstacles at different walking speed and included a comparison with other solutions. Recently, Linder [142] tested the ability of hexapods to self organize leg controllers and their coordination rules from scratch while applying a biologically inspired, simple fitness function, but using a sufficiently complex environment the system has to cope with. In this investigation Linder could show that pure mechanical coupling of the legs is already sufficient to produce a considerably well coordinated gait. Apart from these coordination rules that are directly derived from biology, there are other interesting solutions that have been invented by engineers and are only partly based on biological information. The robot Hamlet [91] uses some kind of proximity function that determines the lifting of a leg. Another quite simple, but effective method has been proposed by Porta and Celaya [162]. According to this rule, a leg does start a swing if it is nearer to its physical extreme position than its neighbouring legs, provided no neighbour is swinging. This leads to a good walking behaviour as long as the walker uses a high speed. To also cope with slow walking speeds, Roggendorf added the following expansions: A leg can only swing, if its position is behind a given threshold. If the leg is behind the threshold, but the PC condition is not fulfilled, the leg votes for a smaller walking speed (depending on its distance to its physical extreme position). The leg with the slowest vote determines the overall walking speed. With this expansion Roggendorf [173] showed that the behaviour is even better, i.e., more stable when walking over irregular ground than when the stick insect rules 1-4 are applied.
Principles of Insect Locomotion
73
Fig. 2.19. Coordination rules 1 to 4 are sufficient to generate typical insect gaits. Based on a model proposed by Dean [63, 64, 66, 67] a precursor model of Walknet [149] incorporated the leg coordination rules of Fig. 2.17. Like all subsequent versions, this model showed velocitydependent gait changes that are characteristic for insect walking. a) If the reference walking velocity is high, the system generates a tripod gait with alternating tripods of L2-R1-R3 (dashed lines 1) and L1-L3-R2 (dashed lines 2). The black bars mark the stance periods of each leg, i.e. when a given leg is on the ground. b) If the reference walking velocity is low, the system generates a tetrapod gait in which always at least four legs are on the ground. The tetrapod gait is characterised by a back-to-front wave of stance movements on the right (dashed lines 1) and left side (dashed lines 2). Identical footfall patterns can occur repeatedly in a tetrapod gait too (dashed lines 3 in a), but they typically drift from one step to the next. Back-to-front waves can also be found in a tripod gait (dashed lines 3 in a), but stance periods of adjacent legs overlap less than in the tetrapod gait.
Curve walking is a special problem that has been investigated in most detail in stick insects [125, 126, 83, 82]. Implementation of the results into Walknet is still in progress, but it should be mentioned that the principle of positive velocity feedback control addressed earlier has proven to be a good basis for the control of curve walking and at the same time avoids unnecessary forces across the body. Another way to avoid such forces is to apply force feedback which, however, is a complex task. But even the search for common results of the many studies on turning and curve walking is a complex task. One reason for this may be that different studies have anal-
74
H. Cruse et al.
Fig. 2.20. Leg coordination using a tetrapod gait. Leg movement is shown such that an upstroke describes a swing, and a downstroke describes a stance movement. For one leg, the left middle leg, a second trace (bold lines) illustrates the effect of the coordinating influences. This trace shows the actual PEP threshold value of this leg (Fig. 2.13 b). The transition from stance to swing is elicited as the leg reaches this position. The legs are labelled R and L for right and left and numbered from 1 to 3 for front, middle, and hind legs, [52, 135].
Fig. 2.21. Righting behaviour. (a) By clamping the tarsi to the ground (arrowheads), the system is made to fall, leading to disordered arrangement of the legs (b). Nevertheless, the system stands up without help and resumes proper walking (c). The large arrow indicates walking direction. Upper panels: top view; lower panels: side view.
ysed turning in very different behavioural contexts, including the optomotor response to visual large-field motion [211, 126, 59, 83], visual tracking of small objects [139], spontaneous turns [127] or particular behaviours such as mating [93], defence behaviour [39] or turning on a beam [94]. Moreover, the locomotor state (static vs. dynamic locomotion) and movement constraints (tethered vs. free walking, natural vs. un-natural inertia) varied among these studies. Whereas ants and flies have been reported not to change the tripod gait during walking [210, 193], stick insects do change leg coordination considerably [82]. Perhaps the one common aspect of all studies on turning is that the stance direction changes considerably and differently in all legs. In stick insects it is clear that a turn is initiated by the strong effort of the front legs, while the kinematic changes of other legs lag the change in front legs [83]. This strongly suggests that the changes observed during curve walking are not occurring at the same
Principles of Insect Locomotion
75
rate but are orchestrated in a way that separates primary active events from secondary events that may be due to either passive or active effects.
2.6 Insect Antennae as Models for Active Tactile Sensors in Legged Locomotion For the coordination of leg movements during hexapod walking, sensory feedback accounts for important adaptive properties. It is beneficial to the walking system as a whole, if the legs exchange information about their current state, e.g., via the coordination rules in Fig. 2.17. Also, if a leg hits an object during protraction, it would be most efficient to ’tell’ the neighbouring leg about the obstacle. During walking, sensory information is gathered by the walking legs. In evolutionary terms, insects have more legs than the six they use for walking. For a number of reasons, biologists consider all articulated appendages of the insect body, with the exception of the wings, to be transformed, specialised legs. These include all mouthparts (mandibles, maxillae and labium) and the antennae, or feelers. The close relation between legs and antennae can be demonstrated in stick insects that, in certain experimental circumstances, will regenerate a leg to replace an amputated antenna [61]. The idea is that, during evolution, legs of the head region proved to be not necessary for maintaining posture or propulsion and, therefore, were free to take on other functions, such as chewing or contact sensing. The process of adopting these new functions was paralleled by gradual changes in morphology and movement pattern. Thus, the nonwalking legs adapted to the specific circumstances that the species encountered. Today, all insects carry a pair of antennae that are densely covered with various kinds of sensory hairs, including mechanoreceptors for tactile sensing, but also chemosensory and thermosensory hairs (the only exception is the very primitive order Protura, the members of which have neither eyes nor antennae, but fairly long front legs).Two main types of insect antennae can be distinguished: the segmented antenna of the Entognatha and the flagellar antenna of the Ectognatha [123] (for illustrations see, e.g., Seifert [186], Fig. 166 and 167). All higher insects, including virtually all commonly known species, carry flagellar antennae that consist of three functional segments. This type of antenna has only two active joints, one moving the scape relative to the head and another moving the pedicel relative to the scape. The long flagellum, that carries most of the sensory hairs, can be moved and bent passively only. Like walking legs, antennae are also equipped with proprioceptors such as hair plates, stretch receptors and chordotonal organs. The most intensely studied of these is Johnston’s organ, a chordotonal organ specialised to sense vibration signals. In honey bees [79], mosquitos [104] and fruit flies [145, 105], Johnston’s organ functions as an ear that allows hearing of other insect’s wingbeat. For a recent review of the neurobiology of insect antennae, see Staudacher et al. [190]. Depending on the phylogenetic heritage and ecological requirements, the morphology of the stick insects and crickets have rather long antennae and actively move them during locomotion (Fig. 2.23 A). Because active movement of a long appendage raises the probability to make contact with surrounding obstacles, this behaviour is likely to improve obstacle detection. Accordingly, assuming that information from the antennae
76
H. Cruse et al.
Fig. 2.22. Various kinds of flagellar antennae. Short, evenly thick flagellar segments, e.g., as in cockroaches, are characteristic of the setiform antenna (a). The filiform antenna, e.g., in carabid beetles, are more slender and longer (b). The moniliform flagellum of termites and many basic forms of insects contains rounded segments (c). In the serrate antenna, e.g., in Elateridae, the segments are toothed (d). Each tooth can be strongly elongated and may protrude in a single or in two directions, as in the one- and two-sided pectinate antennae of sphingid moths and some Tipulidae (e, f). The flagellum of the clavate antenna is distally swollen, forming a club shape (g, h) as in some beetles. The typical geniculate antenna is found in hymenopterans (i). The leafshaped lamellate antenna is found in scarabaeid beetles (k). “needle-shaped” antennae with an unsegmented flagellum are found in cicadas (l) (Fig.167 from [186]).
descends to the thoracic and leg joints, active antennal movements may improve locomotion efficiency on rough terrain. In the stick insect species Carausius morosus, rhythmic movement of the antennae are tightly coupled to the stepping rhythm of the walking legs [84]. From the movement trajectories of legs and antennae, it is possible to predict the probability of an obstacle to be touched by an antenna before one of the legs gets there. This is a necessary condition for exploiting tactile information to adapt on-going leg movements. Not only is the predicted curve confirmed experimentally, it also corresponds nicely to the climbing performance of Walknet: The probability for the antennae to detect an obstacle increases at a height range where the local leg reflexes and coordination rules of Walknet can no longer guarantee climbing-success. Thus, the movement pattern of the antennae is suitable to reliably detect obstacles that require a change of strategy. Meanwhile, behavioural experiments on a number of species confirmed many walking insects use antennal tactile information to incline their body axis in order to adapt to detected obstacles (stick insect: [86] and Fig. 2.23 C; cockroach: [171]). Furthermore, at least in stick insects the typical antennal movement pattern during walking changes as soon as the animal steps into a gap and engages in searching-movements [80] or if the animal turns [83]. In this situation, the antennae support the searching effort of the front legs in the anterior region above the area searched by the legs. Also, antennal function in tactile localisation of objects is revealed by fast re-targeting reactions of a protracting front leg [85], Fig. 2.23 D). Experiments on other insects and crustaceans have
Principles of Insect Locomotion
77
Fig. 2.23. Active exploration and use of antennae in stick insect locomotor behaviour. a) Antennal movements of the stick insect Carausius morosus during walking are rhythmical and of fairly regular appearance. Trajectories of antennal tips are drawn to a sphere that is centred on the head and oriented as shown by the inset. The trajectory of the left antenna (right sphere) is drawn as a mirror image, to match the orientation of the right antenna. b) Abduction phases of the antennae (first and fourth row of black bars) are coupled to the pattern of swing movements of the ipsilateral walking legs. Diagonal dotted lines indicate the back-to-front sequence (a metachronal wave) of leg swing movements and antennal abduction. LA and RA: left and right antenna, respectively; L1-L3: left front, middle and hind leg; R1-R3: right legs. c) The tactile sense is exploited for efficient locomotion. Head trajectories of stick insects walking towards a rectangular obstacle (hatched area, side view) show how the insect raises to climb the edge (walking direction: left to right). Stick insects with intact antennae (left) detect the obstacle earlier (triangles = mean position at first contact) and climb the obstacle with more clearance (circles = position when reaching twice the average walking height) than antennectomised animals (right). Sighted (top) and blind (bottom) animals behave the same. d) Tactile antennal cues can cause rapid re-targeting of an on-going front leg swing movement. Side view (left) and top view (right) of a stick insect walking sequence towards a vertical pole. Three stick figures show body axis, right front leg and tarsus position (circles) at times of lift-off, first antennal contact with the pole (open circle) and leg contact with the pole (solid line: tarsus trajectory; dotted line: trajectory of the antennal tip). A normal swing movement would have been continued as indicated by the dashed arrow. Instead, no more than 60 ms after antennal contact, the swing movement is re-directed to grasp the pole. (Fig. 18 from [190]; showing data from [84] - (A, B); [86] - (C) and [85] - (D))
78
H. Cruse et al.
demonstrated the tactile use of antennae in wall-following [35] and obstacle localisation and climbing [203, 159, 200]. In general, antennae are used during locomotion for orientation relative to wind direction [19, 144, 26] and gravity [9, 118], for detection of potential dangers [192, 38, 103] for active exploration of the space ahead [84, 119, 154, 159] and for tracking of objects [117]. A recent review, Staudacher et al. [190] show that the range of behaviours in which antennal tactile information is important is remarkable. All of these studies suggest that active tactile sensors may also be useful for application on walking machines, for the following four reasons. First, their control is rather simple but effective. Second, due to the fixed morphology, little information processing is required to retrieve 3D obstacle location, in particular when compared to visual input. This reduces computational cost. Third, the gathered information is always relevant to leg movement control, because only the working-range of the legs is sampled. Fourth, they reliably work irrespective of light and other environmental conditions. Although several walking machines make use of passive tactile sensors for obstacle detection [28], active tactile sensing has not yet been implemented with nearly the same effort as visual systems, in spite of the fact that tactile sensors might be sufficient or even advantageous in many applications. To date promising mechanical engineering studies on active insensitive probes [196, 130, 198] suggest that robust sensing solutions may be possible even with few sensors on the probe. A new tactile distance-sensing technique based on a single vibration detector has been proposed by [140]. So far, however, only few studies have tested tactile sensors on moving robots. A preliminary attempt has been made by [4], who mounted moveable antennae on an underwater ambulatory robot that is inspired by an eight-legged lobster. The antennae of this robo-lobster can be positioned at one out of four postures, but active movement is not used in the sensing process. More recently, [141] applied robot feelers to switch between different alternative locomotion behaviours. The cockroach wall-following behaviour [35] has been mimicked by the legged robot ’Sprawlette’ [40], which uses passive antennae, equipped with a set of strain gauges, to control the distance to the wall. From an engineering point of view, the drawback of active antennae mainly concerns the low sampling density, particularly when compared to vision. However, if the main purpose of this sensor is to guide a leg, it can be argued that the movement pattern must only be adapted to the stepping pattern of the ipsilateral front leg and/or to characteristic features of the environment. Non-orthogonal joint axes narrow the action range of the antenna because the torus-shaped workspace of a two-jointed manipulator narrows with decreasing angle between the axes [138]. Given that both joints sinusoidally move through their entire action range, appropriate choice of axis orientation significantly decreases movement effort for optimal obstacle detection, while increasing positioning and sampling accuracy at the same time. Thus, useful technical application of active tactile sensors will be invariably linked to the appropriate choice of kinematics and an efficient movement strategy.
2.7 Central Oscillators As has been mentioned in Sect. 2.4, the network shown in Fig. 2.13 produces rhythmic motor output without having any central oscillators. The reason for this is the
Principles of Insect Locomotion
79
interaction with the environment resulting from embodiment and situatedness. Thus, central oscillators are not required to control rhythmic movements of walking legs. But even if a central oscillator is used to control the basic rhythmic movement of a leg, the problem of coordinating the different joints of one leg during walking would remain. This is not a trivial task because, depending on the specific walking situation, there is considerable variation between the time course of the basic rhythm of swing and stance and the activation of the different joint muscles (for a brief review see [48]). On the other hand, there are biological experiments that indicate that neural oscillators do exist. Detailed studies have shown that separate oscillators can be found for each leg joint (e.g., [34], reviewed by [17]). This induces the question of what might be the biological sense of such oscillators. Based on a Brown half-center oscillator, Cruse [48] proposed a simple network that explains several findings observed in insects walking in different situations (see below).
Fig. 2.24. a) Circuit for the control of an antagonistic structure. As an example, one retractor (stance) muscle and one protractor (swing) muscle are used. LPF: low-pass filter, HPF: highpass filter. w represents a negative weight, and therefore an inhibitory connection between both antagonistic parts. Nonlinear characteristics represent rectifiers. b) Simulation of the behaviour resulting from sensory input of varying strength.
This network also can explain results that by some authors have been interpreted as to indicate the existence of a central oscillator that, as such, drives the rhythmic motor output. This network (Fig. 2.24 a) describes the sensory-motor connection of each joint in a more detailed way than do the swing-net and stance-net; sensory input and motor output are not organized as one channel as shown in Fig. 2.13, but as two antagonistic channels that drive the two antagonistic muscles of one joint. Furthermore, both parallel antagonistic channels are connected via high-pass filtered mutual inhibition (for details see [48]). This circuit is assumed to exist for each joint (for possible connections between joints of one leg, see Fig. 2.13). What are the properties of such an antagonistic structure? As an example, in Fig. 2.24 b retractor and protractor muscles move the thoracic-coxal joint (α -joint in Fig. 2.2). After central activation is switched on (Fig. 2.24 b, arrow), sensory input drives rhythmic motor output like that for swing-net and stance-net in Fig. 2.13. In the simulation, clear transitions between activations of agonist and antagonist are found even when the input values are very small (Fig. 2.24
80
H. Cruse et al.
b, third stance period or sixth swing period) or when the input shows soft transitions as in the case of a sinusoidal wave (not shown). These sharp transitions result from the inhibitory connections that form a kind of winner-take-all system. Interestingly, the strength of initial activation of the agonist depends on the strength of activation of the antagonist in the preceding half-cycle. A small retractor activity (Fig. 2.24 b, symbol *) leads to a small rebound effect in the following protractor activation, and a large retractor activity leads to a large rebound excitation in the following protractor burst (Fig. 2.24 b, symbol **). This rebound effect may be a simple way to explain a number of findings described in the literature. Pearson [155] showed that, when cockroaches have to drag a load, excitation of retractor muscles is increased. Furthermore, protractor excitation increases during swing and, probably as a kinematic consequence of increased protraction velocity, swing duration decreases. Similarly, in crayfish, increasing load during stance leads to decreased swing duration [54]. It is easy to imagine that loading the animal excites stance muscles via direct sensory feedback mechanisms (positive and negative feedback has been discussed, see also Section 2.4.2: Stance movement), whereas swing muscles cannot receive such direct input because the legs are lifted off the ground during swing. However, increased excitation during swing could be explained by the rebound effect discussed here, which excites swing muscles more when their antagonists showed a higher excitation during the preceding stance. Schmitz et al. [181, 180] found that the average velocity of swing movement increases when stick insects walk uphill and decreases when they walk downhill. Again, this phenomenon could be qualitatively explained by the assumption proposed here. Walking uphill requires a higher excitation of stance muscles compared to walking on a horizontal plane. This should lead to a rebound effect on swing muscles. The effect was most obvious in hind legs, which indeed produce most downward and rearward directed forces during uphill walking [41]. Walking downhill requires small propulsive forces during stance if not forces acting against walking direction. Therefore, the rebound effect should be small or even negative, which should lead to a decrease of swing muscle excitation as has been described by Schmitz and coworkers [181, 180]. Furthermore, different shapes of swing trajectories that depend on the form of the substrate can be explained by this account [185]. In conclusion, the rebound effect, which results from the high-pass filtered mutual inhibition (Fig. 2.24), does not produce a continuous oscillation as such, but only influences the next half-cycle. The network may therefore be regarded as a ‘soft oscillator’. This system is advantageous because its “predictive” property is based on actual, local knowledge and therefore avoids the possible bad prediction of a central oscillatory system having a strict inherent rhythm. According to this idea, the quasi-rhythmic motor output observed in walking is not based on a fixed internal “world model” in the form of a traditional central oscillator, but is based on “reality”, i.e., on direct sensory information while the oscillator properties do not influence the rhythm itself, but the strength of the motor output. It should be mentioned here that oscillations have been found in the artificial situation where sensory feedback has been cut off [34]. In agreement with the model simulation, these oscillations are slower than the normal walking rhythm.
Principles of Insect Locomotion
81
How are all the joints of the six legs coordinated during walking? For the swing movement this question could, at least in part, be answered by using the “half-cycle” principle. It can serve to simplify the adaptation of the swing movement to differently shaped substrate and different load situations. During stance the question can be solved by the mixture of positive and negative feedback as explained in Sect. 2.4.2. However, other questions are still open. For example, there exist only hypotheses concerning the control of the switch from levator activation to depressor activation during the second part of swing [158]. This concerns the question of whether, in addition to these local, joint-based oscillators, a superordinate oscillator is necessary (or at least helpful) to control the rhythmic change between swing and stance (recall that movement in the joints can be quite independent from these overall states). Although positive and negative properties of central pattern generators have been discussed with respect to the behaviour in unpredictable situations [48], we do not argue that central oscillators are inappropriate for the control of rhythmic motor output in general. Apart from predictable situations, central systems may also be successfully applied when the mechanical device has few degrees of freedom, for example only one joint (possibly a wing or a fin) or a simple planar two joint leg, because in these cases the critical problem mentioned above, namely the coordination of the joints with the swing/stance rhythm, can easily be solved. One of these approaches will be briefly reviewed here. B¨uschges, Schmitz and co-workers investigated which sensory signals from the leg were able to entrain each of the three joint oscillators [34] such that coordinated transitions between the two step phases – swing and stance – were possible. Most powerful in inducing phase transitions are the load sensors of the leg. Signals from specific subgroups of the trochanterofemoral campaniform sensilla play a significant role in timing the activity of motoneurons supplying the individual leg joints through effects upon central pattern generating networks [2, 204, 3]. Sensory signals from the femoral campaniform sensilla (fCS) are able to induce transitions in activity between tibial motoneuron pools. A posterior or upward directed increase in load on the femur initiates flexor tibiae activity and terminates firing in the extensor tibiae [1]. Furthermore, signals from the fCS are able to reset the rhythm of reciprocal bursting in extensor and flexor tibiae motoneurons in the otherwise deafferented mesothoracic segment. There is, however, a considerable amount of variability in the resetting effect. Additional experiments using the semi-intact single middle leg walking preparation [92] indicated that this stimulus contributes to maintaining flexor tibiae motoneurons activity during the stance phase [1]. Other experiments have shown that sensory signals from the trochanteral campaniform sensilla (trCS) affect the timing of leg motoneuron activity at the more proximal thoraco-coxal joint. An increase of strain on the leg signaled by the trCS applied during bursts of the protractor coxae terminates the protractor activity, in an otherwise deafferented leg. This stimulus also initiates retractor coxae activity, resulting in a resetting of the rhythmic activity of both motoneuron pools [2]. Similar resetting effects were found by Pearson in bursting of cockroach leg motoneurons to pressure on the trochanter [155]. Furthermore, in stick insects, ablation of the trCS in the middle leg in a semi-intact preparation produces a massive deterioration of the intra-leg coordination
82
H. Cruse et al.
during walking. Two conclusions can be drawn from these results: (i) sensory signals from the trochanteral campaniform sensilla affect the timing of the central pattern generator governing thoraco-coxal motoneurons; (ii) signals from the trochanteral receptors play a primary role in the coordinating activities of thoraco-coxal motoneurons with the movements of the distal leg joints [2]. These findings indicate that when the locomotor system is active load signals from the leg are utilized to support the initiation of specific phases of motoneuronal activity in the locomotor cycle, i.e. load signals assist in the generation of flexor tibiae and retractor coxae motoneuron activity during stance.
Fig. 2.25. This diagram summarizes all known sensory influences on timing in intra- and interjoint coordination during walking in the single middle leg. In the figure the individual joint networks are depicted by two neurons showing mutual interaction. Motoneurons are depicted by squares. Filled symbols denote elements/neurons being active, open symbols denote inactive elements/neurons. Sensory influences on the central networks are either excitatory (+) or inhibitory (-). For further details see text. (Figure adopted from [2])
Fig. 2.25 depicts a hypothetical reflex chain that was able to generate appropriate switching between the two states stance and swing in a stick insect middle leg. The basis of this system is that all 3 leg joints have their own central-rhythm generating networks which are not centrally coupled [34] but being coordinated by sensory signals. The step cycle is produced by 4 finite state machines. The description of the sequence of transitions triggers from the 1st to the 4th as shown in the figure, first row (1) from the left to last row on the right (4). 1. during the swing phase in the single-middle-leg preparation, the levator trochanteris (Lev) and extensor (Ext) motoneurons [2, 3] are active as well as the protractor coxae (Pro) motoneurons. Extension of the FT-joint is sensed by the femoral chordotonal organ (fCO), which excites the depressor trochanteris (Dep) and inhibits the Lev motoneurons causing a depression at the CT-joint. This leads the tarsus of the leg to touch the ground [111, 32]. 2. the tarsus touching on the ground causes an increase of the load on the leg, which is sensed by the trochanteral and femoral campaniform sensilla (trCS and fCS). trCS signals excite the retractor coxae (Ret) and inhibit the protractor coxae (Pro) motoneurons. If the leg was not restrained, this would cause it to move backward at the TC-joint. The fCS would excite the flexor tibiae (Flx) and inhibit the extensor tibiae (Ext) motoneurons, which would cause the leg to pull on the ground [1, 2].
Principles of Insect Locomotion
83
3. during the stance the flexion movement of the FT-joint, sensed by the fCO, would reinforce its own movement by assisting flexor activity and inhibiting the extensor (positive feedback, first part of the active reaction) [15, 16, 177, 137]. 4. at a particular position of the FT-joint, the flexor activity (flx) terminates and the extensor (ext) starts to fire again (second part of the active reaction). Flexion of the FT-joint, signaled by the fCO would activate Lev and inactivate Dep [32, 111]. Elevating the leg would decrease the load on the leg, which would be sensed by the trCS and fCS [1, 2]. This time the fCS would assist ongoing Ext activity; signals from the trCS would activate the Pro and inactivate the Ret during the execution of the swing phase [46, 2]. It has to be mentioned that this model describes timing of transition points within the step cycle only, but does not model the magnitude control of muscle activities nor trajectory generation. Also coordinating influences from other legs are not considered. Moreover, activity phases of the different joint muscles are appropriate for front legs or sideways walking middle legs only. However, as already implemented in Walknet, the stance-swing generator of the single leg is taken as a sensory triggered reflex chain, where the swing-stance transition is triggered by ground contact and stance-swing transition is triggered by reaching the PEP. Central pattern generators may be sensible in other situations, apart from generating walking movements. In emergency situations, a central system may be used to replace the “sensory-driven oscillator”. As explained earlier, the system shown in Fig. 2.24 can directly be used to act as a central oscillator, if the value of the central excitation is chosen to be high enough. A dramatic case of such an emergency could be the loss of one or several sensors due to an injury. A less dramatic case, but for biological systems probably equally important, occurs when fast rhythms are to be produced as is the case in a fast walking cockroach. Fast, here, is meant to be relative to the time delays resulting from the slow neuronal processing: if sensory feedback is too slow, it may not be able to contribute to the production of the rhythmic output. Although, as has been argued above, such a central system might be inaccurate in the case of external disturbances, it may be better to use this approximate information than exact information that comes too late. (Note that this argument may not be relevant for an artificial electronic system, because here transmission of signals is fast enough.) However, as an alternative to the use of central oscillators as active devices to control motor output, they may be used in a more passive way, that is for predictive purposes. One way is to change sensory thresholds in a given time window [72]. Moreover, central oscillators may be used on a longer time scale to detect long term deviations (e.g., in case of sensory drift) by providing expectation values that could be compared with the sensory input. If a long term deviation is detected, this information can be used to readjust the system for example via error backpropagation mechanisms [132].
2.8 Actuators In robots, actuators are typically implemented either by electric motors or, if high forces are required, by pneumatic or hydraulic devices. Biological systems use muscles. At
84
H. Cruse et al.
first sight, using muscles seems to be no good solution for an engineer. Muscles show highly nonlinear force-length and force-velocity characteristics (e.g., Hill equation. For further details see Biewener [21]. Illustrations of the characteristics are given there in Fig. 2.3 and 2.4.), including hysteresis effects and nonlinear damping properties. A combination of active parallel elastic elements and passive series elasticity further complicate the understanding. To separate active and passive properties, most neuromechanical models [202] first transform the neural activity profile into a variable called muscle activation (activation dynamics). This variable is independent of joint geometry and determines the time course of isometric contractions, i.e. contractions without associated changes in muscle length. Muscle activation can be viewed as a scaling factor within the range of 0 and 1. It scales the characteristics of active muscle properties, in particular the active force-length curve (see Biewener [21], Fig. 2.3) and the active force-velocity curve (see Biewener [21], Fig. 2.4). Together with passive force-length curve, which is due to strain of a nonlinear spring, and passive damping properties of various tissues (e.g., joint membrane cuticle of arthropod joints, Fig. 2.26), the scaled active muscle properties determine the course of isotonic contractions. A final important aspect of muscular actuators is that muscles can only actively operate in one direction (pull). Two antagonistic muscles are required to move one joint, allowing for variable joint stiffness owing to co-contraction of antagonists. However, as has recently been demonstrated by simulation, muscle properties imply essential advantages on three levels. 1. Advantage: If unexpected disturbances occur, these need not be dealt with by application of traditional feedback loops. However, a muscle alone can already act as shock absorber when the limb unexpectedly hits a rigid obstacle. This is valuable for the mechanical maintenance of the walking system itself, but may also be helpful if the outside object is an object to be manipulated or even more if it is a human being. It is often argued that this property could also be gained when using “soft” feedback controllers. But here is a critical difference: postural reflexes owing to sensory information always require some time to be activated, whereas the mechanical properties of the muscle apply immediately. To describe this effect, the term “preflex” has been established [30]. On the other hand, stiffness of the joint can easily be increased by coactivation of both antagonistic muscles. 2. Advantage: Systems driven by muscular mechanics show unexpected properties on the level of the whole system. It could be shown in experiments and in simulation studies that a fast walking cockroach when pushed sideways is stabilized against this massive disturbance without any need of sensory feedback [98, 175, 187, 99]. In fact, the response to sensory input would have come much too late. During walking the body is “pending” in an elastic mechanical system built up by the active and passive muscles which stabilizes body position in an inverted-pendulum-like fashion. Even when fast enough sensors were available, the same effect could only be reached by a considerable computational load. The explanation of properties of dynamical mass-spring system has already been proposed by Raibert’s famous monopod hopping machines. However, only recently it could be shown that these effects could be exploited for the control of four-legged and
Principles of Insect Locomotion
85
Fig. 2.26. The knee joint of an insect illustrates that the two segments of arthropod joints are connected by a membrane of soft, elastic cuticle (arthrodial membranes: dotted regions). The muscles, in this case the extensor and flexor of the tibia, connect to long tendons made of stiff cuticle, the so-called apodemes. These apodemes connect to the arthrodial membranes, not to the stiff cuticle of the other segment. Thus, there is an elastic material between the muscle and the segment to be moved. (Fig. 122 from [186]).
six-legged walking systems [89]. It should be mentioned that the possibility to store potential energy could also be exploited in order to decrease the total amount of energy consumption. For application, there is of course the problem as to how muscle-like properties could be obtained in artificial systems. Apart from very basic research investigating different chemical substances that are to the best of our knowledge not yet ready for application, there are three lines of solution to this problem. Hu et al. [120] investigate an actuator that consists of a serial connection between an elastic spring and an electric motor where the motor is used to control the elasticity (compliance) of the spring. A similar arrangement of springs and motors is used in the robot TarryIIb [184]. A very different approach is to use “muscle wires” as actuators. These are wires made from shape memory alloys (SMA) that contract when heated, usually by electric current, developing high forces. Similar to muscles they show some elasticity. As they can only develop force during contraction, antagonistic arrangement is necessary. An important advantage is their extremely small weight, in particular when compared with electric motors plus the necessary gears, the disadvantages being the high amount of electric energy necessary to activate them and the long time necessary to cool them down after activation. Advanced prototypes are built by Ayers [4] for underwater robots. Advantage III: A third line of interest is the application of McKibben muscles and related devices. These are elastic tubes that contract when filled with high pressured air. Their behaviour is even more similar to that of muscles. Further advantages are their small weight. Disadvantage is that high pressured air is necessary and a high number of valves have to be controlled and many tubes are necessary to guide the high pressured air to the muscles. Prototypes of walking machines driven by pneumatic actuators have been built by Dillmann, Berns and co-workers [20] and by Quinn et al. [166].
2.9 Conclusion Control of walking means that a large number of degrees of freedom (e.g., at least 18 joints in a six-legged insect, or 17 DoFs in the case of the biped robot Johnnie [102] are
86
H. Cruse et al.
to be controlled simultaneously. Traditional engineering solutions have applied a centralized, hierarchically structured control architecture. This structure is usually somewhat less centralized when applied to multilegged systems as the cyclic movement of each leg is controlled by a separate pattern generator which is coupled to that of the other legs. Sensors are used only to a minor extent (e.g., to detect ground contact). Many biological investigations have shown that sense organs provide critical information that is exploited by the biological control systems. This appears to be particularly necessary when walking in very cluttered and unpredictable environment. Many of these experimental findings have been summarized by a simulation termed Walknet, which is based on artificial neural nets. In Walknet, computation is extremely simplified by taking into account the loop through the world instead of using explicit computation, a property often characterized by the terms embodiment and situatedness. As mentioned, there are also “internal states” which means that a given sensory input can be responded to in different ways depending on the actual form of that internal or “motivational” state. These states described as swing state or stance state are determined by competitive modules which operate on a longer time scale than lower level modules and provide these lower level modules with some kind of limited protection against sensory input. As these states themselves are sensory driven, the whole system may still be termed a reactive system, however in a broader sense than usually used. The six individual leg controllers are connected neuronally via local coordination rules and mechanically via the substrate. The resulting “free gait” controller allows a high degree of adaptability to difficult terrain as are obstacles or large gaps and to negotiate tight turns. Taken together, there is no central control unit. Instead, local modules responsible for the control of “minibehaviours” cooperate in a self-organizing fashion.
References 1. Akay, T., B¨assler, U., Gerharz, P., B¨uschges, A.: The role of sensory signals from the insect coxa-trochanteral joint in contolling motor activity of the femur-tibia joint. J. Neurophysiol. 85, 594–604 (2001) 2. Akay, T., Haehn, S., Schmitz, J., B¨uschges, A.: Signals from load sensors underlie interjoint coordination during stepping movements of the stick insect leg. J. Neurophysiol. 92, 42–51 (2004) 3. Akay, T., Ludwar, B., G¨oritz, M., Schmitz, J., B¨uschges, A.: Segment specificity of load signal processing depends on walking direction in the stick insect leg muscle control system. Journal of Neuroscience 27, 3285–3294 (2007) 4. Ayers, J.: A conservative biomimetic control architecture for autonomous underwater robots. In: Ayers, J., Davis, J.L., Rudolph, A. (eds.) Neurotechnology for biomimetic robots. MIT Press, Cambridge (2002) 5. Baldi, P., Heiligenberg, W.: How sensory maps could enhance resolution through ordered arrangements of broadly tuned receivers. Biol. Cybern. 59, 313–318 (1988) 6. Bartling, C., Schmitz, J.: Reaction to disturbances of a walking leg during stance. J. Exp. Biol. 203, 1211–1233 (2000) 7. B¨assler, U.: Das Stabheuschreckenpraktikum. Franckh, Stuttgart (1965) 8. B¨assler, U.: Proprioreceptoren am Subcoxal- und Femur-Tibia-Gelenk der Stabheuschrecke und ihre Rolle bei der Wahrnehmung der Schwerkraftrichtung. Kybernetik 2, 168–193 (1965)
Principles of Insect Locomotion
87
9. B¨assler, U.: Zur Bedeutung der Antennen f¨ur die Wahrnehmung der Schwerkraftrichtung bei der Stabheuschrecke Carausius morosus. Kybernetik 9, 31–34 (1971) 10. B¨assler, U.: Zur Beeinflussung der Bewegungsweise eines Beines von Carausius morosus durch Amputation anderer Beine. Kybernetik 10, 110–114 (1972) 11. B¨assler, U.: Reversal of a reflex to a single motoneuron in the stick insect Carausius morosus. Biol. Cybern. 24, 47–49 (1976) 12. B¨assler, U.: Sense organs in the femur of the stick insect and their relevance to the control of position of the femur-tibia-joint. J. Comp. Physiol. 121, 99–113 (1977a) 13. B¨assler, U.: Sensory control of leg movement in the stick insect Carausius morosus. Biol. Cybern. 25, 61–72 (1977b) 14. B¨assler, U.: Neural basis of elementary behavior in stick insects. Springer, Heidelberg (1983) 15. B¨assler, U.: Afferent control of walking movements in the stick insect Cuniculina impigra. i. decerebrated animals on a treadband. J. Comp. Physiol. A 158, 345–349 (1986) 16. B¨assler, U.: Functional principles of pattern generation for walking movements of stick insect forelegs: The role of the femoral chordotonal organ afferences. J. Exp. Biol. 136, 125–147 (1988) 17. B¨assler, U., B¨uschges, A.: Pattern generation for stick insect walking movements - multisensory control of a locomotor program. Brain Res. Rev. 27, 65–88 (1998) 18. B¨assler, U., Rohrbacher, J., Karg, G., Breutel, G.: Interruption of searching movements of partly restrained front legs of stick insects, a model situation for the start of the stance phase? Biol. Cybern. 65, 507–514 (1991) 19. Bell, W.J., Kramer, E.: Search and anemotactic orientation of cockroaches. J. Insect Physiol. 25, 631–640 (1979) 20. Berns, K., Albiez, J., Kepplin, V., Hillenbrand, C.: Airbug-insect like machine actuated by fluidic muscle. In: Berns, K., Dillmann, R. (eds.) Proc. 4th Int.Conf.Climbing and Walking Robots, CLAWAR 2001, pp. 237–244. Professional Engineering Publishers, London (2001) 21. Biewener, A.A.: Animal locomotion. Oxford University Press, Oxford (2003) 22. Bizzi, E., Giszter, S.F., Loeb, E., Mussa-Ivaldi, F.A., Saltiel, P.: Modular organization of motor behavior in the frog’s spinal cord. Trends Neurosci. 18, 442–446 (1995) 23. Bl¨asing, B.: Adaptive locomotion in a complex environment: simulation of stick insect gap crossing behaviour. From animals to animats 8, 173–182 (2004) 24. Bl¨asing, B., Cruse, H.: Mechanisms of stick insect locomotion in a gap crossing paradigm. J. Comp. Physiol. A 190, 173–183 (2004) 25. Blickhan, R., Full, R.J.: Similarity in multilegged locomotion: Bouncing like a monopode. J. Comp. Physiol. A 173, 509–517 (1993) 26. B¨ohm, H., Heinzel, H.G., Scharstein, H., Wendler, G.: The course-control system of beetles walking in an air-current field. J. Comp. Physiol. A 169, 671–683 (1991) 27. Br¨aunig, P., Hustert, R., Pfl¨uger, H.J.: Distribution and specific central projections of mechanoreceptors in the thorax and proximal leg joints of locusts. i. morphology, location and innervation of internal proprioceptors of pro-and metathorax and their central projections. Cell Tissue Res. 216, 57–77 (1981) 28. Brooks, R.A.: A robot that walks: Emergent behaviors from a carefully evolved network. Neural computat. 1, 253–262 (1989) 29. Brooks, R.A.: Intelligence without reason. In: Proceedings of the 12th International Joint Conference on Artificial Intelligence (IJCAI 1991), Sydney, pp. 569–595 (1991) 30. Brown, I.E., Loeb, G.E.: A reductionist approach to creating and using neuromusculoskeletal movement. In: Winters, M.J., Crago, E.P. (eds.) Biomechanics and neural control of movement, pp. 148–163. Springer, Heidelberg (2000)
88
H. Cruse et al.
31. Brunn, D.E., Dean, J.: Intersegmental and local interneurons in the metathorax of the stick insect Carausius morosus that monitor middle leg position. J. Neurophysiol. 72, 1208–1219 (1994) 32. Bucher, D., Akay, T., Dicaprio, R.A., B¨uschges, A.: Interjoint coordination in the stick insect leg-control system: The role of positional signaling. J. Neurophysiol. 89, 1245–1255 (2003) 33. Burrows, M.: The Neurobiology of an Insect Brain. Oxford University Press, Oxford (1996) 34. B¨uschges, A., Schmitz, J., B¨assler, U.: Rhythmic patterns in the thoracic nerve cord of the stick insect induced by pilocarpine. J. Exp. Biol. 198, 435–456 (1995) 35. Camhi, J.M., Johnson, E.N.: High-frequency steering maneuvers mediated by tactile cues: Antennal wall-following in the cockroach. J. Exp. Biol. 202, 631–643 (1999) 36. Chasserat, C., Clarac, F.: Interlimb coordinating factors during driven walking in crustacea. A comparative study of absolute and relative coordination. J. Comp. Physiol. 139, 293–306 (1980) 37. Cocatre-Zilgien, J.H., Delcomyn, F.: Modeling stress and strain in an insect leg for simulation of campaniform sensilla responsses to external forces. Biol. Cybern. 81, 149–160 (1999) 38. Comer, C.M., Parks, L., Halvorsen, M.B., Breese-Terteling, A.: The antennal system and cockroach evasive behavior ii. Stimulus identification and localization are separable antennal functions. J. Comp. Physiol. A 189, 97–103 (2003) 39. Copp, N., Jamon, M.: Kinematics of rotation in place during defense turning in the crayfish Procambarus clarkii. J. Exp. Biol. 204, 471–486 (2001) 40. Cowan, N.J., Ma, E.J., Cutkosky, M., Full, R.J.: A biologically inspired passive antenna for steering control of a running robot. In: Dario, P., Chatila, R. (eds.) Robotics Research. The Eleventh International Symposium. Springer, Wien (2003) 41. Cruse, H.: On the function of the legs in the free walking stick insect Carausius morosus. J. Comp. Physiol. 112, 235–262 (1976a) 42. Cruse, H.: The control of body position in the stick insect (Carausius morosus), when walking over uneven surfaces. Biol. Cybern. 24, 25–33 (1976b) 43. Cruse, H.: The control of the anterior extreme position of the hindleg of a walking insect, Carausius morosus. Physiol. Entomol. 4, 121–124 (1979) 44. Cruse, H.: The influence of load, position and velocity on the control of leg movement of a walking insect. In: Gewecke, M., Wendler, G. (eds.) Insect Locomotion, pp. 19–26. Parey, Hamburg (1985a) 45. Cruse, H.: Which parameters control the leg movement of a walking insect? i. velocity control during the stance phase. J. Exp. Biol. 116, 343–355 (1985b) 46. Cruse, H.: Which parameters control the leg movement of a walking insect? ii. the start of the swing phase. J. Exp. Biol. 116, 357–362 (1985c) 47. Cruse, H.: What mechanisms coordinate leg movement in walking arthropods? Trends Neurosci. 13, 15–21 (1990) 48. Cruse, H.: The functional sense of central oscillations in walking. Biol. Cybern. 86, 271– 280 (2002) 49. Cruse, H., Bartling, C.: Movement of joint angles in the legs of a walking insect, Carausius morosus. J. Insect Physiol. 41, 761–771 (1995) 50. Cruse, H., Bartling, C., Dean, J., Kindermann, T., Schmitz, J., Schumm, M., Wagner, H.: Coordination in a six-legged walking system. simple solutions to complex problems by exploitation of physical properties. In: Maes, P., et al. (eds.) From Animals to Animats, vol. 4, pp. 84–93. The MIT Press/Bradford Books, Cambridge (1996) 51. Cruse, H., Bartling, C., Dreifert, M., Schmitz, J., Brunn, D.E., Dean, J., Kindermann, T.: Walking: a complex behaviour controlled by simple networks. Adapt. Behav. 3, 385–418 (1995)
Principles of Insect Locomotion
89
52. Cruse, H., Kindermann, T., Schumm, M., Dean, J., Schmitz, J.: Walknet - a biologically inspired network to control six-legged walking. Neural Networks 11, 1435–1447 (1998) 53. Cruse, H., K¨uhn, S., Park, S., Schmitz, J.: Adaptive control for insect leg position: controller properties depend on substrate compliance. J. Comp. Physiol. A 190, 983–991 (2004) 54. Cruse, H., M¨uller, U.: A new method measuring leg position of walking crustaceans shows that motor output during return stroke depends upon load. J. Exp. Biol. 110, 319–322 (1984) 55. Cruse, H., M¨uller-Wilm, U., Dean, J.: Artificial neural nets for controlling a 6-legged walking system. In: Meyer, J.A., Roitblat, H., Wilson, S. (eds.) From animals to animats, pp. 52–60. MIT Press, Cambridge (1993a) 56. Cruse, H., Riemenschneider, D., Stammer, W.: Control of body position of a stick insect standing on uneven surfaces. Biol. Cybern. 61, 71–77 (1989) 57. Cruse, H., Saxler, G.: Oscillations of force in the standing legs of a walking insect (Carausius morosus). Biol. Cybern. 36, 159–163 (1980) 58. Cruse, H., Schmitz, J., Braun, U., Schweins, A.: Control of body height in a stick insect walking on a treadwheel. J. Exp. Biol. 181, 141–155 (1993b) 59. Cruse, H., Silva Saavedra, M.G.: Curve walking in crayfish. J. Exp. Biol. 199, 1477–1482 (1996) 60. Cruse, H., Warnecke, H.: Coordination of the legs of a slow-walking cat. Exp. Brain Res. 89, 147–156 (1992) 61. Cu´enot, L.: Regeneration de pattes a´ la place d’antennes sectionn´ees chez un phasme. Comptes Rendus Acad. Sci. Paris 172, 949–952 (1921) 62. Dean, J.: Coding proprioceptive information to control movement to a target: simulation with a simple neural network. Biol. Cybern. 63, 115–120 (1990) 63. Dean, J.: A model of leg coordination in the stick insect, Carausius morosus. i. a geometrical consideration of contralateral and ipsilateral coordination mechanisms between two adjacent legs. Biol. Cybern. 64, 393–402 (1991a) 64. Dean, J.: A model of leg coordination in the stick insect, Carausius morosus. ii. description of the kinematic model and simulation of normal step patterns. Biol. Cybern. 64, 403–411 (1991b) 65. Dean, J.: Effect of load on leg movement and step coordination of the stick insect Carausius morosus. J. Exp. Biol. 159, 449–471 (1991c) 66. Dean, J.: A model of leg coordination in the stick insect, Carausius morosus. iii. responses to perturbations of normal coordination. Biol. Cybern. 66, 335–343 (1992a) 67. Dean, J.: A model of leg coordination in the stick insect, Carausius morosus. iv. comparisons of different forms of coordinating mechanisms. Biol. Cybern. 66, 345–355 (1992b) 68. Dean, J., Cruse, H.: Evidence for the control of velocity as well as position in leg protraction and retraction by the stick insect. In: Heuer, H., Fromm, C. (eds.) Generation and modulation of action patterns, vol. 15, pp. 263–274. Springer, Heidelberg (1986) 69. Dean, J., Cruse, H.: Motor pattern generation. In: Arbib, M. (ed.) Handbook for Brain Theory and Neural Network, pp. 696–701. Bradford Book/MIT Press, Cambridge (2003) 70. Dean, J., Wendler, G.: Stick insects walking on a wheel: Perturbations induced by obstruction of leg protraction. J. Comp. Physiol. 148, 195–207 (1982) 71. Dean, J., Wendler, G.: Stick insect locomotion on a walking wheel: interleg coordination of leg position. J. Exp. Biol. 103, 75–94 (1983) 72. Degtyarenko, A.M., Simon, E.S., Norden-Krichmar, T., Burke, R.E.: Modulation of oligosynaptic cutaneous and muscle afferent reflex pathways during fictive locomotion and scratching in the cat. J. Neurophysiol. 79, 447–463 (1998) 73. Delcomyn, F.: Insect locomotion on land. In: Herreid, C.F., Fourtner, C.R. (eds.) Locomotion and Energetics in Arthropods, pp. 103–125. Plenum, New York (1981) 74. Delcomyn, F.: Acitvity and structure of movement-signalling (corollary discharge) interneurons in a cockroach. J. Comp. Physiol. 150, 185–193 (1983)
90
H. Cruse et al.
75. Delcomyn, F.: Motor activity during searching and walking movemets of cockroach legs. J. Exp. Biol. 133, 111–120 (1987) 76. Delcomyn, F.: Walking in the american coackroach: the timing of motor activity in the legs during straight walking. Biol. Cybern. 60, 373–384 (1989) 77. Delcomyn, F.: Activity and directional sensitivity of leg campaniform sensilla in a stick insect. J. Comp. Physiol. A 168, 113–119 (1991) 78. Dicaprio, R.A., Clarac, F.: Reversal of a walking leg reflex elicited by a muscle receptor. J. Exp. Biol. 90, 197–203 (1981) 79. Dreller, C., Kirchner, W.H.: Hearing in honeybees: localization of the auditory sense organ. J. Comp. Physiol. A 173, 275–279 (1993) 80. D¨urr, V.: Stereotypic leg searching-movements in the stick insect: Kinematic analysis, behavioural context and simulation. J. Exp. Biol. 204, 1589–1604 (2001) 81. D¨urr, V.: Context-dependent changes in strength and efficacy of leg coordination mechanisms. J. Exp. Biol. 208, 2253–2267 (2005) 82. D¨urr, V.: Context-dependent changes in strength and efficacy of leg coordination mechanisms. J. Exp. Biol. 208, 2253–2267 (2005a) 83. D¨urr, V., Ebeling, W.: The behavioural transition from straight to curve walking: kinetics of leg movement parameters and the initiation of turning. J. Exp. Biol. 208, 2237–2252 (2005) 84. D¨urr, V., K¨onig, Y., Kittmann, R.: The antennal motor system of the stick insect Carausius morosus: anatomy and antennal movement pattern during walking. J. Comp. Physiol. A 187, 131–144 (2001) 85. D¨urr, V., Krause, A.: The stick insect antenna as a biological paragon for an actively moved tactile probe for obstacle detection. In: Berns, K., Dillmann, R. (eds.) Climbing and Walking Robots - From Biology to Industrial Applications. Proc. 4th Int. Conf. Climbing and Walking Robots (CLAWAR 2001), Karlsruhe, pp. 87–96. Professional Engineering Publishing, Bury St. Edmunds (2001) 86. D¨urr, V., Krause, A., Schmitz, J., Cruse, H.: Neuroethological concepts and their transfer to walking machines. Int. J. Robotics Res. 22, 151–167 (2003) 87. Duysens, J., Clarac, F., Cruse, H.: Load-regulating mechanisms in gait and posture: Comparative aspects. Physiol. Rev. 80, 83–133 (2000) 88. Ebeling, W., D¨urr, V.: Perturbation of leg protraction causes context-dependent modulation of inter-leg coordination, but not of avoidance reflexes. Journal of Experimental Biology 209, 2199–2214 (2006) 89. Ekeberg, O., Bl¨umel, M., B¨uschges, A.: Dynamic simulation of insect walking. Arthropod Structure & Development 33, 287–300 (2004) 90. Espenschied, K.S., Quinn, R.D., Beer, R.D., Chiel, H.J.: Biologically based distributed control and local reflexes improve rough terrain locomotion in a hexapod robot. Robot. Autonom. Sys. 18, 59–64 (1996) 91. Fielding, M.R., Dunlop, R.: Exponential fields to establish inter-leg influences for omnidirectional hexapod gait. In: Berns, K., Dillmann, R. (eds.) Proc. 4th Int. Conf. Climbing and Walking Robots (CLAWAR 2001), pp. 587–594. Professional Engineering Publishers, London (2001) 92. Fischer, H., Schmidt, J., Haas, R., B¨uschges, A.: Pattern generation for walking and searching movements of a stick insect leg. i. coordination of motor activity. J. Neurophysiol. 85, 341–353 (2001) 93. Franklin, R., Bell, W.J., Jander, R.: Rotational locomotion by the cockroach Blattella germanica. J. Insect Physiol. 27, 249–255 (1981) 94. Frantsevich, L., Cruse, H.: Leg coordination during turning on an extremely narrow substrate in a bug, Mesocerus marginatus (heteroptera, coreidae). J. Insect Physiol. 51, 1092– 1104 (2005)
Principles of Insect Locomotion
91
95. Frazier, S.F., Larsen, G.S., Neff, D., Quimby, L., Carney, M., Dicaprio, R.A., Zill, S.N.: Elasticity and movements of the cockroach tarsus in walking. J. Comp. Physiol. A 185, 157–172 (1999) 96. Full, R.J.: Integration of individual leg dynamics with whole body movement in arthropod locomotion. In: Beer, R., Ritzmann, R.E., McKenna, T. (eds.) Biological Neural Networks in Invertebrate Neuroethology and Robotics, pp. 3–20. Academic Press, San Diego (1993) 97. Full, R.J., Blickhan, R.: Locomotion wnergetics of the ghost crab. J. Exp. Biol. 130, 155– 175 (1987) 98. Full, R.J., Koditschek, D.E.: Templates and anchors: neuromechanical hypotheses of legged locomotion on land. J. Exp. Biol. 202, 3325–3332 (1999) 99. Full, R.J., Kubow, T.M., Schmitt, J., Holmes, P., Koditscheck, D.E.: Quantifying dynamic stability and maneuverbility in legged locomotion. Integ. and Comp. Biol. 42, 149–157 (2002) 100. Full, R.J., Tu, M.S.: Mechanics of six-legged runners. J. Exp. Biol. 148, 129–146 (1990) 101. Full, R.J., Tu, M.S.: Mechanics of a rapid running insect: two-, four- and six-legged locomotion. J. Exp. Biol. 156, 215–231 (1991) 102. Gienger, M., L¨offler, K., Pfeiffer, F.: A biped robot that jogs. In: Proc. IEEE Int. Conf. Robotics Automation, vol. 4, pp. 3334–3339 (2000) 103. Gnatzy, W., Heußlein, R.: Digger wasp against crickets. i. receptors involved in the antipredator strategies of the prey. Naturwiss. 73, 212–215 (1986) 104. G¨opfert, M.C., Briegel, H., Robert, D.: Mosquito hearing: sound-induced antennal vibrations in male and female Aedes aegypti. J. Exp. Biol. 202, 2727–2738 (1999) 105. G¨opfert, M.C., Robert, D.: The mechanical basis of Drosophila audition. J. Exp. Biol. 205, 1199–1208 (2002) 106. Gorb, S.N., Jiao, Y., Scherge, M.: Ultrastructural architecture and mechanical properties of attachment pads in Tettigonia viridissima (orthoptera, tettigoniidae). J. Comp. Physiol. A 186, 821–831 (2000) 107. Graham, D.: A behavioural analysis of the temporal organisation of walking movements in the 1st instar and adult stick insect (Carausius morosus). J. Comp. Physiol. 81, 23–52 (1972) 108. Graham, D.: Pattern and control of walking in insects. Adv. Insect Physiol. 81, 31–140 (1985) 109. Gregory, G.: Neuroanatomy of the mesothoracic ganglion of the cockroach periplaneta americana (l.). ii. median neuron cell body groups. Phil. Trans. R. Soc. Lond. B 306, 191– 218 (1984) 110. Guthrie, D.M.: Multipolar stretch receptors and the insect leg reflex. J. Insect Physiol. 13, 1637–1644 (1967) 111. Hess, D., B¨uschges, A.: Role of propriosceptive signals from an insect femur-tibia joint in patterning motoneuronal activity of an adjacent leg joint. J. Neurophysiol. 81, 1856–1865 (1999) 112. Hofmann, T., B¨assler, U.: Anatomy and physiology of trochanteral campaniform sensilla in the stick insect, cuniculina impigra. Physiol. Entomol. 7, 413–426 (1982) 113. Hofmann, T., Koch, U.T., B¨assler, U.: Physiology of the femoral chordotonal organ in the stick insect, Cuniculina impigra. J. Exp. Biol. 114, 207–223 (1985) 114. Holst, E.v.: Die relative koordination als ph¨anomen und als methode zentralnerv¨oser funktionsanalyse. Erg. Physiol. 42, 228–306 (1939) ¨ 115. Holst, E.v.: Uber relative koordination bei arthropoden. Pfl¨ugers Arch. 246, 847–865 (1943) 116. H¨oltje, M., Hustert, R.: Rapid mechano-sensory pathways code leg impact and elicit very rapid reflexes in insects. J. Exp. Biol. 206, 2715–2724 (2003) 117. Honegger, H.W.: A preliminary note on a new optomotor response in crickets: Antennal tracking of moving targets. J. Comp. Physiol. A 142, 419–421 (1981)
92
H. Cruse et al.
118. Horn, E., Bischof, H.J.: Gravity reception in crickets: the influence of cercal and antennal afferents on the head position. J. Comp. Physiol. A 150, 93–98 (1983) 119. Horseman, B.G., Gebhardt, M.J., Honegger, H.W.: Involvement of the suboesophageal and thoracic ganglia in the control of antennal movements in crickets. J. Comp. Physiol. A 181, 195–204 (1997) 120. Hu, J., Pratt, J., Chew, C., Herr, H., Pratt, G.: Adaptive virtual model control of a bipedal walking robot. In: Proc. IEEE Int. Symp. Intel. Sys., pp. 245–251 (1998) 121. Hustert, R.: Proprioceptor responses and convergence of proprioceptive influence on motorneurones in the mesothoracic thoraco-coxal joint of locusts. J. Comp. Physiol. 150, 77– 86 (1983) 122. Hustert, R., Pfl¨uger, H.J., Br¨aunig, P.: Distribution and specific central projections in the thorax and proximal leg joints of locusts. iii. the external mechanoreceptors: The campaniform sensille. Cell Tissue Res. 216, 97–111 (1981) 123. Imms, A.D.: On the antennal musculature in insects and other arthropods. Quart. J. Microsc. Sci. 81, 273–320 (1939) 124. Inman, V.T., Ralston, H.J., Todd, F.: Human walking. Williams & Wilkins, Baltimore (1981) 125. Jander, J.P.: Untersuchungen zum mechanismus und zur zentralnerv¨osen steuerung des kurvenlaufs bei stabheuschrecken (Carausius morosus). Ph.D. thesis, University of K¨oln, Germany (1982) 126. Jander, J.P.: Mechanical stability in stick insects when walking straight and around curves. In: Gewecke, M., Wendler, G. (eds.) Insect Locomotion, pp. 33–42. Paul Parey, Hamburg (1985) 127. Jindrich, D.L., Full, R.J.: Many-legged maneuverability: Dynamics of turning in hexapods. J. Exp. Biol. 202, 1603–1623 (1999) 128. Jindrich, D.L., Full, R.J.: Dynamic stabilization of rapid hexapedal locomotion. J. Exp. Biol. 205, 2803–2823 (2002) 129. Kaliyamoorthy, S., Zill, S.N., Quinn, R.D., Ritzmann, R.: Finite element analysis of strains in the Blaberus cockroach leg segment while climbing. In: Intelligent Robots and Systems IEEE/RSJ Proceedings, vol. 2, pp. 833–838 (2001) 130. Kaneko, M., Kanayma, N., Tsuji, T.: Active antenna for contact sensing. IEEE Trans. Robot. Autom. 14, 278–291 (1998) 131. Karg, G., Breutel, G., B¨assler, U.: Sensory influences on the coordination of two leg joints during searching movements of stick insects. Biol. Cybern. 64, 329–335 (1991) 132. Kawato, M., Gomi, H.: The cerebellum and vor/okr learning-models. Trends Neurosci. 15, 445–453 (1992) 133. Kemmerling, S., Varju, D.: Regulation of the body-substrate-distance in the stick insect: Step responses and modelling the control system. Biol. Cybern. 44, 59–66 (1982) 134. Kendall, M.D.: The anatomy of the tarsi of Schistocerca gregoria forsk˚al. Z. Zellforsch. 109, 112–137 (1970) 135. Kindermann, T.: Behavior and adaptability of a six-legged walking system with highly distributed control. Adapt. Behav. 9, 16–41 (2002) 136. Kittmann, R., Dean, J., Schmitz, J.: An atlas of the thoracic ganglia of the stick insect, carausius morosus. Phil. Trans. R. Soc. Lond. B 331, 101–121 (1991) 137. Knop, G., Denzer, L., Buschges, A.: A central pattern-generating network contributes to “reflex-reversal”-like leg motoneuron activity in the locust. J. Neurophysiol. 86, 3065–3068 (2001) 138. Krause, A.F., D¨urr, V.: Tactile efficiency of insect antennae with two hinge joints. Biol. Cybern. 91, 168–181 (2004) 139. Land, M.F.: Stepping movements made by jumping spiders during turns mediated by the lateral eyes. J. Exp. Biol. 57, 15–40 (1972)
Principles of Insect Locomotion
93
140. Lange, O., Reimann, B., Saenz, J., D¨urr, V., Elkmann, N.: Insectoid obstacle detection based on an active tactile approach. In: Witte, H. (ed.) Proc. Int. Symp. Adapt. Motion Anim. Mach. (2005) 141. Lewinger, W.A., Harley, C.M., Ritzmann, R.E., Branicky, M.S., Quinn, R.D.: Insect-like antennal sensing for climbing and tunneling behavior in a biologically-inspired mobile robot. In: Procceedings of the IEEE International Conference on Robotics and Automation (ICRA 2005), Barcelona, April 18-22 (2005) 142. Linder, C.: Self-organization in a simple task of motor control based on spatial encoding. Adaptive Behavior (2005) 143. Linder, C.R.: Self organisation in a simple task of motor control. In: Hallam, B., et al. (eds.) From animals to animats, vol. 7, pp. 185–194. MIT Press, Cambridge (2002) 144. Linsenmair, K.E.: Die windorientierung laufender insekten. Fortschr. Zool. 21, 59–79 (1973) 145. Manning, A.: Antennae and sexual receptivity in Drosophila melanogaster females. Science 158, 136–137 (1967) 146. Marchand, A.R., Leibrock, C.S., Auriac, M.C., Barnes, W.J.P., Clarac, F.: Morphology, physiology and in vivo activity of cuticular stress detector afferents in crayfish. J. Comp. Physiol. A. 176, 409–424 (1995) 147. Matheson, T.: Range fractionation in the locust metathoracic femoral chordotonal organ. J. Comp. Physiol. A 170, 509–520 (1992) 148. Matheson, T., Field, L.H.: An elaborate tension receptor system highlights sensory complexity in the hind leg of the locust. J. Exp. Biol. 198, 1673–1689 (1995) 149. M¨uller-Wilm, U., Dean, J., Cruse, H., Weidemann, H.J., Eltze, J., Pfeiffer, F.: Kinematic model of stick insect as an example of a 6-legged walking system. Adapt. Behav. 1, 155– 169 (1992) 150. Newland, P.L.: Morphology and somatotopic organisation of the central projections of afferents from tactile hairs on the hind leg of the locust. J. Comp. Neurol. 312, 493–508 (1991) 151. Newland, P.L., Emptage, N.J.: The central connections and actions during walking of tibial campaniform sensilla in the locust. J. Comp. Physiol. A 178, 749–762 (1996) 152. Noah, J.A., Quimby, L., Frazier, S.F., Zill, S.N.: Force detection in cockroach walking reconsidered: discharges of proximal tibial campaniform sensilla when body load is altered. J. Comp. Physiol. A 187, 769–784 (2001) 153. Noah, J.A., Quimby, L., Frazier, S.F., Zill, S.N.: Sensing the effect of body load in legs: responses of tibial campaniform sensilla to forces applied to the thorax in freely standing cockroaches. J. Comp. Physiol. A 190, 201–215 (2004) 154. Okada, J., Toh, Y.: The role of antennal hair plates in object-guided tactile orientation of the cockroach (Periplaneta americana). J. Comp. Physiol. A (2000) 155. Pearson, K.G.: Central programming and reflex control of walking in the cockroach. J. Exp. Biol. 56, 173–193 (1972) 156. Pearson, K.G.: Common principles of motor control in vertebrates and invertebrates. Ann. Rev. Neurosci. 16, 265–297 (1993) 157. Pearson, K.G., Franklin, R.: Characteristics of leg movements and patterns of coordination in locusts walking on rough terrain. Int. J. Robotics Res. 3, 101–112 (1984) 158. Pearson, K.G., Iles, F.J.: Nervous mechanisms underlying intersegmental co-ordination of leg movements during walking in the cockroach. J. Exp. Biol. 58, 725–744 (1973) 159. Pelletier, Y., McLeod, C.D.: Obstacle perception by insect antennae during terrestrial locomotion. Physiol. Entomol. 19, 360–362 (1994) 160. Petryszak, A., Fudalewicz-Niemczyk, W.: External proprioceptors on the legs of insects of higher orders. Acta Biologica Cracoviensia 36, 13–22 (1994)
94
H. Cruse et al.
161. Pfl¨uger, H.J., Braeunig, P., Hustert, R.: The organization of mechanosensory neuropiles in locust thoracic ganglia. Phil. Trans. R. Soc. Lond. B 321, 1–26 (1988) 162. Porta, J.M., Celaya, E.: Efficient gait generation using reinforcement learning. In: Berns, K., Dillmann, R. (eds.) Proc. 4th Int. Conf. Climbing and Walking Robots (CLAWAR 2001), pp. 411–418. Professional Engineering Publishing, London (2001) 163. Pringle, J.W.S.: Proprioception in insects. ii. the action of the campaniform sensilla on the legs. J. Exp. Biol. 15, 114–131 (1938) 164. Prochazka, A., Gillard, D., Bennett, D.J.: Implications of positive feedback in the control of movement. J. Neurophysiol. 77, 3237–3251 (1997a) 165. Prochazka, A., Gillard, D., Bennett, D.J.: Positive force feedback control of muscles. J. Neurophysiol. 77, 3226–3236 (1997b) 166. Quinn, R.D., Nelson, G.M., Bachmann, R.J., Ritzmann, R.E.: Toward mission capable legged robots through biological inspiration. Autonomous Robots 11, 215–220 (2001) 167. Radnikow, G., B¨assler, U.: Function of a muscle whose apodeme travels through a joint moved by other muscles: why the retractor unguis muscle in stick insects is tripartite and has no antagonist. J. Exp. Biol. 157, 87–99 (1991) 168. Ridgel, A.L., Frazier, S.F., Dicaprio, R.A., Zill, S.N.: Active signaling of leg loading and unloading in the cockroach. J. Neurophysiol. 81, 1432–1437 (1999) 169. Ridgel, A.L., Frazier, S.F., Dicaprio, R.A., Zill, S.N.: Encoding of forces by cockroach tibial campaniform sensilla: implications in dynamic control of posture and locomotion. J. Comp. Physiol. A 186, 359–374 (2000) 170. Ridgel, A.L., Frazier, S.F., Zill, S.N.: Dynamic responses of tibial campaniform sensilla studied by substrate displacement in freely moving cockroaches. J. Comp. Physiol. A 187, 405–420 (2001) 171. Ritzmann, R.E., Pollack, A.J., Archinal, J., Ridgel, A.L., Quinn, R.D.: Descending control of body attitude in the cockroach blaberus discoidalis and its role in incline climbing. J. Comp. Physiol. A 191, 253–264 (2005) 172. Roeder, K.D.: The control of tonus and locomotor activity in the praying mantis (Mantis religiosa l). J. exp. Zool. 76, 353–374 (1937) 173. Roggendorf, T.: Comparing different controllers for the coordination of a six-legged walker. Biol. Cybern. 92, 261–274 (2005) 174. Schilling, M., Cruse, H., Arena, P.: Hexapod walking: an expansion to walknet dealing with leg amputations and force oscillations. Biological Cybernetics 96(3), 323–340 (2007) 175. Schmitt, J., Garcia, M., Razo, R.C., Holmes, P., Full, R.J.: Dynamics and stability of legged locomotion in the horizontal plane: a test case using insects. Biol. Cybern. 86, 343–353 (2002) 176. Schmitz, J.: Load-compensating reactions in the proximal leg joints of stick insects during standing and walking. J. Exp. Biol. 183, 15–33 (1993) 177. Schmitz, J., Bartling, C., Brunn, D.E., Cruse, H., Dean, J., Kindermann, T., Schumm, M., Wagner, H.: Adaptive properties of “hard-wired” neuronal systems. Verh. Dt. Zool. Ges. 88, 165–179 (1995) 178. Schmitz, J., Dean, J., Kindermann, T., Schumm, M., Cruse, H.: A biologically inspired controller for hexapod walking: simple solutions by exploiting physical properties. Biol. Bull. 200, 195–200 (2001) 179. Schmitz, J., Dean, J., Kittmann, R.: Central projections of leg sense organs in Carausius morosus (insecta, phasmida). Zoomorphol. 111, 19–33 (1991) 180. Schmitz, J., Kamp, A., Kindermann, T., Cruse, H.: Adaptations to increased load in a control system governing movements of biological and artificial walking machines. In: Blickhan, R., Nachtigall, W. (eds.) BIONA reports 13: Motor System (2000)
Principles of Insect Locomotion
95
181. Schmitz, J., Schumann, K., Kamp, A.v.: Mechanisms for self-adaptation of posture and movement to increased load. Abstract Viewer/Itinerary Planner. Society for Neuroscience, Washington (CD-ROM, 2000, Program No. 368.7) (2000) 182. Schmitz, J., Stein, W.: Convergence of load and movement information onto leg motoneurons in insects. J. Neurobiol. 42, 424–436 (2000) 183. Schneider, A., Cruse, H., Schmitz, J.: A biologically inspired active compliant joint using local positive velocity feedback. lpvf. IEEE Trans. Systems Man Cybern. Part B: Cybernetics 35, 1120–1130 (2005a) 184. Schneider, A., Cruse, H., Schmitz, J.: Switched local positive velocity feedback controllers: Local generation of retraction forces and inter-joint coordination during walking. In: Witte, H. (ed.) 3rd International Symposium on Adaptive Motion in Animals and Machines (AMAM 2005), Ilmenau, September 25-30 (2005b) 185. Schumm, M., Cruse, H.: Control of swing movement: influences of differently shaped substrate. J. Comp. Physiol. A 192 (2006) 186. Seifert, G.: Entomologisches Praktikum, 2nd edn. Thieme, Stuttgart (1975) 187. Seipel, J.E., Holmes, P., Full, R.J.: Dynamics and stability of insect locomotion: a hexapedal model for horizontal plane motions. Biol. Cybern. 91, 76–90 (2004) 188. Sillar, K.T., Skorupski, P., Elson, R.C., Bush, B.M.H.: Two identified afferent neurones entrain a central locomotor rhythm generator. Nature 323, 440–443 (1986) 189. Spinola, S.M., Chapman, K.M.: Proprioceptive indentation of the campaniform sensilla of cockroach legs. J. Comp. Physiol. 96, 257–272 (1975) 190. Staudacher, E., Gebhardt, M.J., D¨urr, V.: Antennal movements and mechanoreception: neurobiology of active tactile sensors. Adv. Insect Physiol. 32, 49–205 (2005) 191. Stein, W., Schmitz, J.: Multimodal convergence of presynaptic afferent inhibition in insect proprioceptors. J. Neurophysiol. 82, 512–514 (1999) 192. Stierle, I.E., Getman, M., Comer, C.M.: Multisensory control of escape in the cockroach Periplaneta americana i. Initial evidence from patterns of wind-evoked behavior. J. Comp. Physiol. A 174, 1–11 (1994) 193. Strauss, R., Heisenberg, M.: Coordination of legs during straight walking and turning in Drosophila melanogaster. J. Comp. Physiol. A 167, 403–412 (1990) 194. Ting, L.H., Blickhan, R., Full, R.J.: Dynamic and static stability in hexapedal runners. J. Exp. Biol. 197, 251–269 (1994) 195. Tryba, A.K., Ritzmann, R.E.: Multi-joint coordination during walking and foothold searching in the Blaberus cockroach i. Kinematics and electromyograms. J. Neurophysiol. 83, 3323–3336 (2000) 196. Tsujimura, T., Yabuta, T.: A tactile sensing method employing force/torque information through insensitive probes. In: Proc. IEEE Int. Conf. Robotics Automation 1992, pp. 1315– 1320 (1992) 197. Tyrer, N., Gregory, G.: A guide to the neuroanatomy of locust suboesophageal and thoracic ganglia. Phil. Trans. R. Soc. Lond. B 297, 91–123 (1982) 198. Ueno, N., Svinin, M.M., Kaneko, M.: Dynamic contact sensing by flexible beam. IEEEASME Trans. Mechatronics 3, 254–264 (1998) 199. Watson, J.T., Ritzmann, R.E.: Leg kinematics and muscle activity during treadmill running in the cockroach, Blaberus discoidalis: I. slow running. J. Comp. Physiol. A 182, 11–22 (1998) 200. Watson, J.T., Ritzmann, R.E., Zill, S.N., Pollack, A.J.: Control of obstacle climbing in the cockroach, Blaberus discoidalis. i. kinematics. J. Comp. Physiol. A 188, 39–53 (2002) 201. Wendler, G.: Laufen und stehen der stabheuschrecke Carausius morosus: Sinnesborstenfelder in den beingelenken von regelkreisen. Z. vergl. Physiol. 48, 198–250 (1964) 202. Zajac, F.E.: Muscle and tendon: properties, models, scaling, and application to biomechanics and motor control. Critical Reviews in Biomed. Eng. 17, 359–411 (1989)
96
H. Cruse et al.
203. Zeil, J., Sandeman, R., Sandeman, D.C.: Tactile localisation: the function of active antennal movements in the crayfish Cherax destructor. J. Comp. Physiol. A 157, 607–617 (1985) 204. Zill, S., Schmitz, J., B¨uschges, A.: Load sensing and control of posture and locomotion. Arthropod Structure & Development 33, 273–286 (2004) 205. Zill, S.N.: Load compensatory reactions in insects: Swaying and stepping strategies in posture and locomotion. In: Beer, R.D., Ritzmann, R.E., McKenna, T. (eds.) Biological Neural Networks in Insect Neuroethology and Robotics, pp. 43–68. Academic Press, New York (1993) 206. Zill, S.N., Moran, D.T.: The exoskeleton and insect proprioception. iii. activity of tibial campaniform sensilla during walking in the american cockroach, periplaneta americana. J. Exp. Biol. 94, 57–75 (1981a) 207. Zill, S.N., Moran, D.T.: The exoskeleton and insect propriocetion. i. responses of tibial campaniform sensilla to external and muscle-generated forces in the american cockroach periplaneta americana. J. Exp. Biol. 91, 1–24 (1981b) 208. Zill, S.N., Moran, D.T., Varela, F.G.: The exoskeleton and insect proprioception. ii. reflex effects of tibial campaniform sensilla in the american cockroach, periplaneta americana. J. Exp. Biol. 94, 43–55 (1981) 209. Zill, S.N., Underwood, M.A., Rowley, J.C., Moran, D.T.: A somatotopic organization of groups of afferents in insect peripheral nerves. Brain Res. 198, 253–269 (1980) 210. Zollikofer, C.P.E.: Stepping patterns in ants i. influence of speed and curvature. J. Exp. Biol. 192, 95–106 (1994) 211. Zolotov, V., Frantsevich, L., Falk, E.M.: Kinematik der phototaktischen drehung bei der honigbiene Apis mellifera. J. Comp. Physiol. A 97, 339–353 (1975)
3 Low Level Approaches to Cognitive Control B. Webb1 , J. Wessnitzer1 , H. Rosano1 , M. Szenher1 , M. Zampoglou1, T. Haferlach1, and P. Russo2 1 2
Institute for Perception, Action and Behaviour, University of Edinburgh
[email protected],
[email protected] Department of Electrical, Electronic and System Engineering, University of Catania, I-95125 Catania, Italy
[email protected]
Abstract. This chapter describes progress towards the building of a biologically inspired cognitive artifact. It first discusses the implementation of individual perception-action links based on insect capabilities; then how these may be combined in more complex, multimodal control. We describe several neural network implementions of insect based methods of navigation. We present the preliminary results of modelling associative learning capabilities based on the insect mushroom bodies.
3.1 Introduction In Chapter 1 we addressed the issue of active perception from psychological and biological perspectives. We suggested it is most productive to consider it as task-specific transformation of sensory input into action. This not only implies action-oriented processing (such as matched filtering and extraction of affordances) but also involves closed loops in which actions continually change the input. ‘Proto’-cognitive behaviours involve interactions between loops, learnt substitution of one loop for another, internal loops, and predictive loops: internal representations may be introduced at this level in the form of forward models. Higher cognitive capabilities such as planning may require more complex forms of internal representation and memory. We observed that nearly all these issues are relevant to insect behaviours, such as their ability to combine different sensorimotor loops smoothly, to learn, to navigate, to control a complex multi-degree of freedom body, and so on. Insects thus constitute a useful model for understanding many basic mechanisms in the context of perception for action. We proposed a general architecture for achieving cognition in robots, based on the insect brain. The main features are the use of parallel sensorimotor transformation loops that control individual behaviours, co-ordinated and modulated by higher level loops that involve multimodal inputs, learning and prediction. We noted in particular the neural architecture of the mushroom bodies as an interesting insect biological system to explore for learning capabilities. This chapter provides more biological background and describes the development of models for pre-cognitive and proto-cognitive capabilities, derived directly from P. Arena and L. Patan`e (Eds.): Spatial Temporal Patterns, COSMOS 1, pp. 97–175. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com
98
B. Webb et al.
insect neurobiology and implemented on robots. There is a hierarchy of behaviours to be discussed: • Basic pre-wired behaviours e.g. sound localisation, visual reflexes; these may still involve specialised filtering, classification, active perception, some adaptability, and solving significant motor control problems. • Multimodal and protocognitive behaviours e.g. navigation behaviours such as homing and path integration that must smoothly combine multiple sensors and require memory, but do not need a cognitive map. • Associative learning e.g. of signals predicting other stimuli, enabling anticipatory behaviour, substitution of one sensory control for another (e.g. visual guidance towards a sound source), and cancellation of expected re-afference by association with efference copy.
3.2 Sensory Systems and Simple Behaviours 3.2.1
Mechanosensory Systems
Mechanoreceptors are distributed over the whole body of most insects and depending on the position may serve a variety of sensory functions: proprioception, equilibrium, gravity, perception of air currents, audition, etc. Two major mechanosensory appendages, the antennae and the cerci, are briefly described in this section, with particular attention to their role in behaviour and possible multimodal interactions. 3.2.1.1 Antennae All insects have specialised sensors on their heads in the form of antennae. The antennae of the cockroach Periplaneta americana is highly segmented, consisting of 150+ segments where each segment is covered with hard cuticle. The antennae are flexible at the intersegmental folds and highly sensitive to touch, as well as carrying many chemical receptors. Muscle control allows the cockroach to sweep its antennae (sometimes extending up to 1.3 times the body length [10]) independently. Antennae thus are active tactile sensors and their potential use in robot locomotion is discussed in [20], who observe a tight coupling between the stick insect’s walking rhythm and the rhythmic movement of the antennae. It is argued that long antennae allow insects to actively search for obstacles in their path. The active movement increases the probability of obstacle and olfaction detection. Tactile learning in the honeybee has been reported in [25] showing that honeybees could through antennal exploration learn and discriminate physical characteristics of surfaces. Antennal tracking responses to horizontally moving objects in the visual field of the cricket were reported in [58]. Tactile stimulation of the antennae can cause escape responses in cockroaches and crickets, and their role in steering behaviour and wall-following is discussed in [10]. Receptors from the antennae innervate the deutocerebrum, which consists of two distinct neuropils, the antennal lobe and the antennal mechanosensory motor centre [57], and links olfactory and mechanosensory inputs, integrating cues from these modalities [181]. Descending neural pathways from the antennal lobe to thoracic motor centres
Low Level Approaches to Cognitive Control
99
play an important role in escape responses [7]. The antennal wall following and escape system of the cockroach Periplaneta americana is modelled on a robot in [12, 13]. In [178], the integrated roles of visual and mechanosensory cues during antennal escape responses are investigated. 3.2.1.2 Cerci Escape responses can be elicited by many different sensory modalities, visual or mechanosensory. In some groups of insects, the abdomen bears a pair of terminal appendages called the cerci, carrying hair cells sensitive to air flow, particularly the airfront of a pouncing predator. The circuitry known to control this wind-mediated escape behaviour lies in the thoracic ganglia, abdominal ganglia, and abdominal nerve cord. Although it has been shown decapitated cockroaches are still able to exhibit escape responses, descending inputs from the brain or subesophageal ganglion play a role in modulating thoracic circuitry [130]. As might be expected for an escape response, the cricket’s response is in the order of 60-100ms to detect, calculate, and execute avoidance behaviour. This wind-mediated escape has been investigated and modelled on a robot in [12]. In the cricket, hundreds of tiny filiform hairs populate each cercus. These filiform hairs are mechanoreceptors that are highly sensitive to the direction and the dynamics of air currents. Each filiform hair is tuned to wind directions in particular direction and frequency spectra, causing firing in the associated sensory neuron. The axons associated with each receptor neuron enter the Terminal Abdominal Ganglion (TAG), more specifically a neuropil called the cercal glomerulus. Here they form a continuous map of stimulus direction [69], i.e. they arborise in specific locations according to their directional tuning properties. Computation of stimulus direction by TAG interneurons can be explained by examining the anatomical overlap between their dendritic trees and the map formed by the sensory afferents [68, 104, 69]. The activity of the cerci sensory system is not only determined by the excitation of receptor cells but also by central mechanisms controlling information transmission depending on behavioural contexts [81], e.g. inhibition during flight. Interneurons project their axons from the cerci towards motor neurons in the thoracic ganglia. In the cricket, four giant interneurons (GIs) effectively monitor wind stimuli from the quadrants of the animal’s surroundings, each being receptive over roughly 90 degrees. There are other GIs, but also non-giant interneurons, responsive to more narrow ranges of the wind velocity spectrum. The concepts of sensory maps and minimalist representations are well illustrated by the cerci system. The sensory field of the insect is represented as a map of wind directions within the cercal glomerulus, i.e. a functional map rather than a topographic projection. The afferents are structurally reorganised according to stimulus/response characteristics independently of the relative receptor locations on the periphery. Unlike the retinotopic organisation in visual systems, the loss of a localised region of receptors on the cerci would not lead to a blind spot in the representation of the stimulus space. Interestingly, Kanou et al. [72] have shown that crickets can adapt the directionality of their cercal escape response after damage only if they have experienced reaffferent input during free movement.
100
3.2.2
B. Webb et al.
Olfactory Systems
Odours play a very important role in insect behaviour, e.g. courtship behaviour and tracking sex attractant pheromones, and the search for food. Many insects are capable of following odour gradients despite turbulences in air flow which cause odour concentrations to fluctuate. A behavioural strategy called anemotaxis helps animals to navigate towards odour sources by moving upwind, using hair sensors to detect the wind direction, and visual information to stabilise their course [2]. Odour concentrations will be highly dispersed in the air and the task of olfaction is the perception, identification and discrimination of odours. What makes this sensory modality different from others is the multi-dimensionality of odour space. Odours are composed of a mix of odourants resulting in virtually infinite combinations of odours and yet insects can learn and recognise specific odours and combinations thereof with great accuracy [37], at low concentrations. The task of olfaction is one of (i) classifying similar odours as the same, (ii) discriminating distinct odours from one another, and (iii) classifying odours carrying the same meaning (e.g. reward) [62]. The challenge is to understand the computational rules or algorithms used by the brain to encode, store and retrieve these complex and multidimensional stimuli. Interestingly, it appears similar solutions to these problems in olfaction may have been adopted by different species and phyla, including vertebrates [54, 147]. The problem of constructing neural representations for odours is one of mapping a multidimensional odour space onto a two-dimensional sensory surface and thence the antennal lobe [82, 83]. Odourant receptor neurons (ORN) form the sensory surface but their layout is neither receptotopic nor chemotopic. Different ORNs are tuned to different odourants, but the tuning is fairly broad resulting in combinatorial activation patterns of several types of ORN. The ORN axons project into the antennal lobe (AL) of the deutocerebrum. From there interneurons connect to higher brain centres in other parts of the brain, particularly the lateral protocerebral lobe and the mushroom bodies. The antennal lobe is further divided into areas processing very specific and particular olfactory inputs, such as sex pheromones, plant and food odours, or CO2 . Evidence of sex-specific neuroanatomical differences in antenna and the AL for pheromone detection again illustrate the concept of matched filters [1]. The way odours are mapped in the brain is through convergence of all ORN axons expressing the same OR gene onto the same (sometimes same few) glomerulus [36]. In [35, 116], odour representation is thought of as spatio-temporal patterns in the antennal lobe which are distributed and combinatorial, and are not revealed simply by looking at mean firing rates [153]. The AL consists of two main types of neurons, excitatory projection neurons (PNs) and inhibitory local neurons (LNs). LNs are thought to synchronise groups of PNs by periodic inhibitory inputs. Odour-evoked synchronisation of subpopulations of neurons in the antennal lobe leads to oscillations in the field potential [97, 95] and these oscillations appear to be required for fine odour discrimination. Synchronisation of projection neurons may help to stabilise the odour response but to what extent the temporal patterns themselves code for odour properties, reflect binding of the activated neuron population, optimisation of the response, or dynamic stimuli, remains to be seen [83, 93].
Low Level Approaches to Cognitive Control
101
ORN
LN and PN
AL
KC
MB
(a) ORN axons
PN
LN Glomerulus
LN PN KC
AL
MB
Legend Dendrite
(b) Cell body
Axon
Fig. 3.1. Schematic representation of the olfactory processing pathway. (a) In the locust, ∼50000 ORN axons converge into glomeruli where ∼830 projection neurons (PN) and ∼300 local neuron (LN) of the antennal lobe (AL) generate activation patterns feeding the ∼50000 Kenyon cells (KC) of the mushroom bodies (MB). The mushroom bodies are thought of as higher olfaction brain centres. (b) Schematic view of the interactions between local and projection neurons in the antennal lobe.
Although we do not model olfactory sensing in the present work, we note that this system illustrates many interesting general principles of sensory pathways. e.g. feedforward and feed-back inhibition, divergence and convergence, which are important for understanding neural coding [65]. The olfactory system can be considered a system for pattern classification operating in noisy, high-dimensional and rapidly-changing environments. The network dynamics progressively converge (over repeated odour samplings) by relating information across glomeruli and PNs on more easily classifiable representations [145]. The main features as summarised in [120] are: • normalisation and dynamic range compression of input, • selective amplification of foreground odourants against complex chemical backgrounds, • compression and storage of information that defines classes of behaviourally relevant odours,
102
B. Webb et al.
• construction of a transmission signal indicating the presence or absence of an identified odourant, • enhancement of signal-to-noise ratio, • modification of the selectivity to form new classes by associative learning. Various attempts at modelling the complex spatio-temporal dynamics have been made (e.g. [120, 121, 96, 122]) and this sensory modality is also increasingly used to robotics (e.g. [73, 64, 44]). 3.2.3
Visual Systems
3.2.3.1 Background Many important behaviours in an insect’s repertoire are dependent on vision, particularly the detection of visual motion [139]; including the optomotor response, the landing response, orientation towards small, fast moving objects against the background [22], distance measurement from parallax, including peering behaviour (moving the head from side to side), the centering response, and visual regulation of flight speed. The optomotor response of flies estimates self-rotation based on visual motion information and generates a compensatory torque response to maintain flight stability. The centering response is a behaviour whereby bees flying through a narrow gap tend to fly through its centre [139], by balancing the optic flow on each side. Perception of visual cues also plays an important role in navigation strategies [38]. Path integration is simplified by adjusting flight speed in order to keep optic flow constant [15]. Not only do insects use optic flow to measure distance [26] but pursuit behaviour provides compelling evidence that these animals are tuned to pick up other small moving objects in the environment. There is even evidence of predatory insects concealing their own movements whilst stalking other insects, moving in such ways as to appear stationary to their prey [140]. A fly’s compound eyes are made up of facets, or ommatidia. An ommatidium contains a facet lens and 8 photoreceptors arranged in characteristic pattern; 2 at the centre with one on top of the other, and the other six placed around those. The number of ommatidia varies with species. The spatial resolution of an insect’s compound eyes not only depends on the number of ommatidia but also on the angular differences between ommatidia themselves and between the photoreceptive elements in each ommatidium [89, 91]. The optic lobes are a large pair of structures on either side of the brain adjacent to the ommatidia which make up the retina as shown in Fig. 3.2. Each optic lobe consists of four distinct parts: the lamina, the outer and inner medulla, and the lobula. Visual images received by the facet eyes are retinotopically projected towards these underlying neuropils, i.e. the neighbourhood relationships are maintained. Photoreceptors in the retina signal temporal deviations from the ambient light level and feed these to the next layer of cells, the lamina. Lamina cells are thought to emphasise temporal change. Due to their small size, the medulla neuropils have not been investigated with respect to their response characteristics [16] but indirect evidence suggests that local detection of motion (between adjacent ommatidia) is performed there [6]. Franceschini et al. showed that stimulating just two photoreceptors in turn produces a detectable signal in subsequent neurons [28]. This first processing stage has been described computationally by the Hassenstein-Reichardt model [43], although it has been
Low Level Approaches to Cognitive Control
103
Optic lobe Retina Lobula Central brain
Lamina Lobula plate Medulla
Fig. 3.2. Schematic cross-section through a fly’s head illustrating the main components of the visual system. Adapted from [6].
suggested that what is known of the neuronal structure connecting the photoreceptors to the lobula plate tangential cells (LPTC) does not equate to this model [53, 51, 52]. Higgins and co-workers have attempted to derive the functional organisation of a subset of identified retinotopic neurons feeding into the lobula plate. Their proposed model exhibits output properties that are consistent with current anatomical and physiological observations of real neurons, and also very close to the properties of the original Hassenstein-Reichardt model. The lobula plate, a posterior part of the lobula, hosts tangential cells (LPTCs - Lobula Plate Tangential Cells). These LPTCs are complex motion-sensitive neurons. At this point in the optic lobe, the retinotopic structure is abandoned and information converges onto these tangential cells. The LPTCs pool with their large dendritic trees the outputs of the many retinotopically organised local motion elements. Some may be more responsive to translational, vertical or horizontal motion, or indeed highly complex visual flow-fields [84]. LPTCs have been subdivided into various subclasses according to their response characteristics [6]: (1) preferred orientation (i.e. whether they respond to vertical or horizontal image motion), (2) prevalent electrical response mode, i.e. spiking or non-spiking, (3) projection area (e.g. to the contralateral brain hemisphere or to the ipsilateral side), and (4) spatial integration properties (i.e. whether response increases as moving visual patterns grow, or whether they respond more to small moving patterns) [22]. Synaptic interactions between these integrating neurons increase optic flow specificity [59]. These LPTCs act as matched filters for optic flow patterns detecting body rotations [85] but also for detecting smaller objects against the background [78]. Models of the fly’s visual system can be found, for example in [74, 75, 76, 6]. Some LPTCs have been linked to specific visually-guided behaviours [22]. Flight trajectories of flies tend to consist of straight flight sequences with abrupt changes
104
B. Webb et al.
in heading called saccades. Expansion of optic flow triggers saccades away from approaching visual features [152]. Tammero and Dickinson reported that flies exhibit fixed-amplitude saccades of ± 90 degrees and that the response is away from the optic lobe experiencing the greater amount of visual motion. Saccadic movements are important for a variety of reasons [90]. First, resolution is lost if rotation blurs the image. Second, moving objects are easier to see if the retinal image of the background is stationary. Third, it becomes easier to obtain heading and distance from flow-fields resulting from pure translation (Fig. 3.3). Rotation without translation
elevation
elevation
Translation without rotation
Rotation with translational element
Translation with rotational element
elevation
azimuth
elevation
azimuth
azimuth
azimuth
Fig. 3.3. Flow field patterns on an animal’s retina resulting from (a) pure translation, (b) pure rotation and (c,d) from combinations of both.
As well as the main visual pathway, a number of other specialised visual sensory systems are found in insects. The fruitfly Drosophila has been shown to have specialised ommatidia for detecting polarised light, situated at the dorsal rim area of the compound eye [86], as do other insects including bees, crickets and locusts. Ocelli, an insect’s simple eyes, can only detect changes in light intensity over their large visual fields for controlling pitch and roll deviation [107] sometimes adapted for horizon detection [142]. This links back to the ideas of affordances and matched filters discussed in chapter 1. Complex sensory stimuli are transformed by very specific setups of sensory pathways, tuned to specific visual features. Many visual behaviours involve integration with other sensory systems. Gaze control in the blowfly Calliphora is known to process several mechanosensory and visual cues
Low Level Approaches to Cognitive Control
105
to keep balance during flight and to align its eyes with respect to its surroundings [49]. Frye and co-workers report on the importance visuo-olfactory sensory integration for odour source localisation [33, 31]. In another paper, Frye and Dickinson report on how the motor output reflects the linear superposition of visual and olfactory input [32]. There can also be substantial flexibility in visual ‘reflexes’. Evidence for operant loops in flies allowing flexibility to use and vary different motor outputs for visual steering behaviour can be seen in [174, 175]. Drosophila can learn to control flight course, contextualise sensory information in order to avoid punishment or receive reward. 3.2.3.2 Integrating Visual Control into Walking Controllers Many of the visual behaviours described above require a turning response from the animal. While visual behaviour has been more extensively studied in flying insects, we are particularly interested in application to walking robots based on six-legged insects. Assuming the visual system has supplied a correction or target direction for the animal to pursue, executing that response involves solving a complex 18 degree-of-freedom problem to control each of the three joints on six legs. It is not clear to what extent the precise trajectories have to be calculated by each leg and how each thoracic segment (front, middle and hind legs) might interpret turning signals generated locally or from the brain. Modulation of leg coordination and leg trajectories during turns has previously been analysed using the insect’s response to a continuous visual flow [21]. In that experiment, insects were fixed on top of a movable ball in the middle of a rotating visual scene. It was suggested [21] that the front legs respond first to the visual stimulus, and this might trigger the response of the other legs. Here we report on the results of experiments and models that explore the body trajectory and the different roles of legs on different thoracic segments when the freely walking insect spontaneously turns towards a visual target. A complete account of these results is provided in [127]. Stick insects are known to be attracted to bush-shaped objects and to be strongly stimulated by vertical edges [71]. Adult stick insects (Carausius morosus) were placed in an arena (67cm by 177cm) with white walls (50 cm tall) around it to eliminate external visual stimuli. The visual target consisted of a black bar, 4.5 cm wide and 60 cm tall. The target was placed within the insect’s visual field in a different direction to its current heading, and no more than 30cm away. The animal would reliably respond by turning to walk in this direction (Fig. 3.4). Just before the insect reached it, the target was quickly removed vertically and then placed in a different position, no more than 30 cm away, inducing another turn. This could be repeated around 10 times before the insect changed its attention to the walls or ceased walking. A second experiment used the same paradigm but with the front tarsi of the stick insect temporarily blocked with dried paint. This prevents gripping of the substrate, minimising the directional forces of front leg movements, thus revealing more clearly the contributions of the middle and rear legs for turning. A total of 24 turns for the intact insect and 19 turns for the insect with tarsi blocked were combined for the analysis. The turns were normalised to start in direction zero, turn to a target angle of ‘one’, and to take the same time to complete. The average time to complete a turn was 60 frames (2.4 seconds). The variables studied are shown in Fig. 3.10: the body angle θB at time t; and the direction of movement of the prothorax
106
B. Webb et al.
θP and metathorax θM at time t, calculated with respect to a point in time n frames ahead of the current position, typically 10 frames ahead. This lag removed noise and intrinsic oscillations of the stick insect, but still tended to smooth fast changes in angle, particularly affecting those changing more abruptly, like θP .
Insect size Walking direction Prothorax
Target
Mesothorax Metathorax
Prothorax Mesothorax Metathorax
Insect size Starting position
Starting position
Fig. 3.4. Representative example of the paths stick insects follow when attracted to a black vertical bar. A zoomed section when turning is shown on the right, this represents a typical turn. Here, the prothorax direction suddenly changes almost 90 degrees. From [127] (Fig. 3).
Observed turning behaviour in the stick insect Fig. 3.5 shows, in the upper three plots, the direction followed by the body, prothorax and metathorax during a turn. For comparison, the means are plotted on one graph on the lower left. Despite the smoothing, it can still be clearly seen that at the beginning of the turn, the prothorax θP changes direction within just a few time steps, and very early during the turn is pointing towards the target. Prothorax leg direction, relative to the target, during stance is also plotted in Fig. 3.5. It can be seen that particularly the inner front leg pulls towards the target. The metathorax, on the other hand, follows a smoother transition, similar to that of the body itself. The point of rotation for the body was calculated at each moment during the sequence, for rotations larger than 1 degree. Fig. 3.6 shows a normalized graph of rotation positions. Most rotations accumulated between the mesothorax and the metathorax, laterally displaced from the body towards the position that legs contact the ground. These data suggest that the specific movements of the stick insect’s legs during turns results in the prothoracic segment following mostly straight lines, pointing most of the time towards the target, whereas the mesothorax and metathorax tend to follow curves, with a rotation point between the inner legs of the metathorax and mesothorax. This can also be seen in Fig. 3.4 and Fig. 3.15. Fig. 3.7 shows two typical turns done by the stick insect when the front tarsi are blocked, and hence can make minimal contribution to the turn. Fig. 3.8 shows analogous results to those in Fig. 3.5, except for the middle bottom figure, which instead of showing front leg trajectories shows body directions for smooth turns only. For most sharp turns (bottom left) the body pointed to the target when the prothorax was still
Low Level Approaches to Cognitive Control Body angle
0.5
20 frame 40
0
60
0
Upper plots superposition
20 frame 40
0
Front Leg Direction
20 frame 40
60
Pro− & Meta− thorax speed
1.4 1.2 1
0.6
0.6
0.5
0.4
0.4
0.2
0.2
20 frame 40
60
0
1
0.8
0.8
0
0
60
1.6
1
0
1
0.5
speed/initial speed
normalised absolute angle
1.5
1
0.5
norm absolute leg direction
0
Metathorax Direction
1.5 normalised absolute angle
1
0
Prothorax Direction
1.5 normalised absolute angle
normalised absolute angle
1.5
107
0
20 frame 40
60
0
0
20 frame 40
60
Fig. 3.5. Ethological results. Bottom left figure shows with a solid line the mean direction of the Prothorax θP ; Metathorax θM with a dotted line; and the body θB with a dashed line. Top left shows progress of θB in more detail, standard deviation is shown by the error bars. The top middle is that of the prothorax θP and the top right is the metathorax direction θM . The bottom middle shows the direction front legs follow relative to the initial heading, i.e., θB + θL ; both legs are shown, inner front leg (◦) and outer (∗). The bottom right shows changes in speed for the prothorax (◦) and metathorax (∗) relative to their velocity before the turn. From [127] (Fig. 5). MOL MIL HOL HIL
2
1
0
−1 1
0
−1
−2
−3
−4
Fig. 3.6. A cumulative plot of the point of rotation during turns. The three black asterisks (∗) represent the three thoracic segments the prothorax is encircled. Turns are to the right. From [127] (Fig. 6).
108
B. Webb et al.
Walking direction
Metathorax Prothorax
Insect size
Fig. 3.7. Targeting with front leg tarsi blocked. From [127] (Fig. 8).
moving laterally; therefore the prothorax ends at a slightly larger angle compared to that of the body. For smooth turns the prothorax was just slightly deviated from the body and the metathorax direction as shown in the middle bottom graph. In contrast with results shown for the intact insect the prothorax does not point consistently towards the target. The rotation shown in Fig. 3.9 is the average of all turns. In sharp turns the inner hind leg is often arrested in position, or even moved in reverse. The rotation point in these situations is moved almost on top of the insect body. Analysis of turning behaviour The individual leg trajectories for achieving the body trajectories described above vary considerably on different thoracic segments. Moreover, individual leg speeds on either side of the turn vary greatly. The problem we address in the following section is how this complex pattern of leg behaviour might be achieved with a simple control model. Typically, turning in six-legged robots is controlled by simple techniques, similar to wheeled robots, that cause a difference in speed on either side, which determines a point of rotation. If w is the distance between wheels, vo is the outer speed to the turn and vi is the inner speed, the point of rotation is given by R = vo w/(vo − vi ). The closer this point of rotation is to the center of the robot, the sharper the turn. For legged robots there are alternative options to control this difference in speed, for instance, increasing frequency between step phases on one side. However, we are motivated to produce trajectories like those seen in the insect, and describing this motion in terms of rotations does not fit naturally. Rather, we focused on describing the trajectory of the prothorax because it represents the most salient feature of stick insect turns: the prothorax tends to follow straight line trajectories, i.e. θ˙η = 0 for the prothorax. This can be described by:
θη = arctan (tan (φ )/η ) θ˙B = |vP | sin (φ )/R
(3.1)
where η is the relative distance to the point of rotation R along the body, being 1 for the prothorax and 0 for the point of rotation. The left hand side of Fig. 3.10 shows these
Low Level Approaches to Cognitive Control Body angle
normalised absolute angle
1
Upper plots superposition
1
0.5
0
0
0
60
0
1.5
20 frame 40
60
0
Smooth turns superposition
20 frame 40
60
Pro− & Meta− thorax speed 1
0.8
1
0.6
0.5
20 frame 40
0
60
speed/initial speed
1.5
20 frame 40
normalised absolute angle
0
1
0.5
0.5
0
Metathorax Direction
1.5
1.5
1
0.5
normalised absolute angle
Prothorax Direction
2 normalised absolute angle
normalised absolute angle
1.5
109
0.4 0.2
0
0
20 frame 40
60
0
0
20 frame 40
60
Fig. 3.8. Ethological results for the insect when front tarsi is blocked. Bottom left and bottom middle figures show with a solid line the mean direction of the Prothorax θP ; Metathorax θM with a dotted line; and the body θB with a dashed line. The bottom middle shows results only for smooth trajectories (n=4). Top left shows progress of θB in more detail, deviation standard is shown by the arrows. The top middle is that of the prothorax θP and the top right is the metathorax direction θM . The bottom right shows changes in speed for the prothorax (◦) and metathorax (∗) relative to their velocity before the turn. From [127] (Fig. 7). MOL MIL HOL HIL
2
1
0
−1 1
0
−1
−2
−3
−4
Fig. 3.9. A cumulative plot of the point of rotation during turns. The three black asterisks (∗) represent the three thoracic segments the prothorax is encircled. Turns are to the right. From [127] (Fig. 9).
variables with respect to the target. For the prothorax η = 1, therefore, the direction is always that of the target given its current position, θP = θη + θB = φ + θB = θT . Transforming this model of body motion into leg trajectories v, is fairly straightforward: if L is the distance from the coxa to the tarsus, then it follows that
110
B. Webb et al. θB
φ
θT
θνP
η =1
Target
vP = Bη=1
Length
B η=(r, θ) η =0
∆θB
Fig. 3.10. Left: θT is the angle to the target, θB is the body angle and φ is the relative angle to the target. Right: The prothorax is translated by Bη =1 and rotated ∆ θB around η = 0. Bη is the translation followed by other points along the body. From [127] (Fig. 11).
v(θ˙B , θη ) = [−Ly, Lx]θ˙B − θη . However, this equation for v implies that all legs in a kinematic model need to calculate at each point how much the body needs to rotate, and the rear legs need to know their relative position to the point of rotation. Furthermore, the rear leg direction will depend not only on φ , but on arctan(tan (φ )/η ). Alternatively, implemented in a dynamic model, the equations in (3.1) could be executed independently. The prothorax could follow trajectories according to φ = θη (η = 1), while the back of the body could account for the rotation θ˙B . Moreover, at least some of the rotation could be a passive result of the trajectory of the prothorax. This further simplifies calculations for leg trajectories in the prothorax, because if they no longer need to compute θ˙B , the equation v for front leg trajectories becomes simply v = −φ as seen in Fig. 3.10.
body rotation
MOL MIL HOL HIL
body
velocity
−∆rΗ +∆rΗ
r Lmeta l meta
IARmeta
Fig. 3.11. Left: Ideally, if the body were to rotate always around the hind inner leg, the MIL and the HOL would have specific orthogonal trajectories to follow. Right: Graphical representation of leg’s active role. Hind legs average speed rH is changed by ∆ rH resulting in a distance to the axis of rotation of IARH . From [127] (Fig. 12).
Low Level Approaches to Cognitive Control
111
However, purely passive rotation of the rear segments is not sufficient to account for the sharp rotations seen in the insect behaviour, particularly when the front tarsi have been blocked. Normally, when the body rotation is sharp, the inner hind leg is almost arrested near the AEP, and the body rotates around this point. The left hand side of Fig. 3.11 shows how each leg should move to contribute to this rotation: specifically, the middle hind leg should move sideways, and the outer hind leg should move forward to back. Control could consist of both middle legs producing the same lateral force, while increasing the speed of the outer hind leg and decreasing the speed of the inner hind leg by the amount ∆α . The right hand side of Fig. 3.11 shows how this affects the rotation point. This is easily translated into joint control by the observation that the alpha joint is particularly related to back and forward movement of the leg, and the gamma joint is related to lateral movement of legs (see Fig. 3.12). By introducing differential activation of the alpha joints in the rear legs, their relative speed can be easily controlled. Similarly, introducing a bias to the gamma joint in the middle legs will produce the required lateral movements. Model A robot simulation was programmed using ODE1 libraries. Fig. 3.12 shows the stick insect based robot and its leg geometry. The angles controlled are α for rostral caudal movements; β for moving the leg up and down; and γ for movements towards and away from the body. It is assumed that the insect is able to sense directly the angular error between a visual target and its current heading (body direction). This can be derived from the retinal position of the target in the eye on the relevant side of the animal (with an appropriate adjustment if the head is not held at the same angle as the body). No explicit visual processing was implemented on the robot but just a single sensor, positioned at the centre front, which detects the target angle and sends this information to each thoracic segment. The robot’s control of leg movements is completely decentralised, and interleg coordination is based on the WalkNet model [17] (see chapter 2). Transitions between leg phases (stance and swing) result from simple excitatory and inhibitory rules between adjacent legs. We used rules 1-3 and 5b from [17]: Rule 1, a leg swinging inhibits transition to swing in the anterior leg; Rule 2, a leg starting stance excites the anterior leg; Rule 3, a leg excites the caudal leg proportional to its distance travelled. Rules 2 and 3 are also implemented between contralateral legs. Rule 5b prolongs stance according to the load that leg is supporting. Additionally, we excited a leg when load was low. Individual leg trajectories are determined by combining a velocity controller, calculating the required joint velocities for a desired speed and direction of the leg during stance, with feedforward control as proposed in [134]. For the velocity controller, equa tion 3.3 gives joint velocities, A˙ = [α˙ β˙ γ˙] , where T (α , β , γ ) is the tarsus position, J is the Jacobian matrix and v is the desired velocity of the tarsus2 . A˙ = [α˙ β˙ γ˙] = [J(T(α , β , γ ))]−1 v(θL , speed).
1 2
(3.2)
Open Dynamic Engine. http://ode.org/ In [126] it is shown that this mathematical solution can be emulated using a neural network.
112
B. Webb et al.
Fig. 3.12. The stick insect based robot created using ODE libraries is shown on the left. The right hand side shows the leg geometry. From [127] (Fig. 1).
The height of the body (to compensate for gravity) was controlled locally as in [18] and the speed of all legs was kept constant for all experiments. Therefore the only free parameter was the direction of the leg, θL in the ‘x–y’ plane. For the front legs, θL was always that of the direction of the target, i.e. θL = θφ . For the middle and hind legs this was always set to θL = 0, i.e. without the front leg influence they would always walk forward (but see below). The three leg segments always stay in the same plane, as shown in Fig. 3.12, which allows us to solve equation (3.3) without further parameters. A˙ = [α˙ β˙ γ˙] = [J(T(α , β , γ ))]−1 v(θL , speed).
(3.3)
The feedforward controller adjusted the joint velocity proportionally to the position error caused by external forces. The final angular joint velocity for each leg was thus a linear combination between the angular velocity calculated in equation (3.3) and the velocity error caused by external forces (in general, the effects of pulling by other legs). Therefore, each leg was capable of following a given direction, but at the same time it tended to follow external forces. How much it followed external forces was controlled by a subordination parameter s(sα , sβ , sγ ) for each thoracic segment. For additional turning control, two more parameters were introduced based on results from the animal, described above. These further adjusted the velocities of the middle and rear legs so as to increase the rotation. The activity of gamma joints in the mesothorax was increased or decreased proportionally to the target angle β˙ = β˙ ± kγ θφ . The middle inner gamma joint was excited, whereas the middle outer gamma joint was inhibited, producing lateral forces from these legs. Similarly, alpha joints in the metathorax were exited or inhibited proportionally to the target angle α˙ = α˙ ± kα θφ . The hind inner alpha joint was inhibited and the hind outer alpha joint was excited. As a result, the metathorax rotates towards the target whilst the mesothorax moves sideways pivoting around the metathorax.
Low Level Approaches to Cognitive Control
113
Observed turning behaviour in the robot model The robot model was made to turn at angles from 20 to 90 degrees by increments of 10 and due to the symmetry of the system all turns were made to the same side. Because gait coordination is probabilistic, and every time a different pattern was found, three runs were taken for each angle, for a total of 24 turns. Runs were stopped once the body angle was within 5 degrees of the target and the metathorax was aligned with the prothorax in the same direction. Results from the simulation were analysed using the same approach as for the insect. Body angle
0.5
20 frame 40
0
60
0
20 frame 40
Upper plots superposition
0
Front Leg Direction 1.4 1.2
20 frame 40
60
Pro− & Meta− thorax speed
1
0.6
0.6
0.5
0.4
0.4
0.2
0.2
20 frame 40
60
0
1
0.8
0.8
0
0
60
1.6
1
0
1
0.5
speed/initial speed
normalised absolute angle
1.5
0.5
norm absolute leg direction
0
1
Metathorax Direction
1.5 normalised absolute angle
1
0
Prothorax Direction
1.5 normalised absolute angle
normalised absolute angle
1.5
0
20 frame 40
60
0
0
20 frame 40
60
Fig. 3.13. Simulation results: See figure 3.5 for details. From [127] (figure 23).
MOL MIL HOL HIL
2
1
0
−1 1
0
−1
−2
−3
−4
Fig. 3.14. Simulation results. A cumulative plot of the point of rotation during turns. The three black asterisks (∗) represent the three thoracic segments the prothorax is encircled. Turns are to the right. From [127] (Fig. 24).
114
B. Webb et al.
(a) Just Prothorax
(b) Thoracic Differentiation
(c) Insect
Fig. 3.15. Comparison between simulation results and an insect. Leg trajectories relative to the body show only the first third of the turn in all cases. Dark regions indicate stance and light is swing. From [127] (Fig. 3.25).
In [127] we described the results from exploring the effects of parameter values which lead us to chose the following final set of parameters for the model: kα = 1.25, γ γ kγ = 1.5, smeso = 0.3, smeta = 0.2, sαmeso = 0.3, sαmeta = 0.5. Testing the model under normal conditions, i.e. with the front tarsi unblocked, we also introduce partial influence of the front legs by the rear legs through non-zero subordination parameters, sαpro = γ 0.25 and s pro = 0.15 Fig. 3.13 shows the simulation now responds quickly to the target stimuli and holds the direction of the prothorax more steady (top, centre). The behaviour of the metathorax is more similar to the body (bottom, left) indicating that sideways movement is reduced. Overshooting the target angle sooner means that angular speed is now increased thanks to the metathorax and mesothorax active role. The prothorax direction, particularly the initial response, is smoother because front legs now respond to external forces. The front legs point more directly towards the target than is seen in the insect, although like the insect, the outer leg has larger values than the inner leg (bottom, centre). The metathorax speed is reduced (bottom, right) although still not as much as in the insect. Furthermore, the rotation point is moved to fall at the same lateral distance as the rear legs; this is shown in Fig. 3.14. In Fig. 3.15 typical leg trajectories are shown for the simulation and the insect. Comparison to alternative models An approach that has been proposed before is to control turning by introducing biases on every alpha joint in the body and having all legs following every external force. For large curvatures, the trajectories produced are similar to the insect, but this method fails to reproduce tight turns [79]. A typical trajectory for this reduced model and the average rotation point for a set of experiments is shown in Fig. 3.16. Because the rotation point is not close to the body, and most importantly towards the rear, trajectories are large, smoother and usually failed to hit the target. Some models like Walknet incorporate other variables for the control of turning, for instance, those related with positioning the AEP and PEP. However, the turning trajectories these produce follow the pattern shown in Fig. 3.16. Similar results were found if no feedforward control was used,
Low Level Approaches to Cognitive Control
350
MOL MIL HOL HIL
2
300
1
250
115
200
0
150 100
−1
50 −400
−300
−200
−100
0
1
0
−1
−2
−3
−4
Fig. 3.16. Left, typical trajectory is shown for targeting at 60 degrees when no inverse Jacobian is used and turns are only induced by introducing biases in the alpha joint. Right, average axis of rotation (AAR) for a set of turning shows that the AAR do not get very close to the body. From [127] (Fig. 17).
and turning was done by directly changing speed of equation 3.3 on either side, or introducing biases to alpha joints as before. Again, the simulation always produced smooth trajectories and the rotation point was usually away from the body and at the level of the mesothorax. We conclude from this that models that do not differentiate the control of the different thoracic segments are unable to produce insect-like turning behaviour. 3.2.4
Audition
3.2.4.1 Background Insects use auditory responses for a wide range of behaviours, from avoidance of bat ultrasound to the location of parasite hosts. One of the most conspicuous and best studied behaviours is acoustic communication in orthoptera, including grasshoppers and crickets, in which song signals are used to identify and locate mates. A historical account on the search for neural centres of cricket and grasshopper song can be found in [24]. Substantial experimental evaluation of the behaviour of female crickets to different sound sources has been carried out, usually employing one of three methods: free walking in an arena (e.g. [146]); a fixed body but legs or abdomen able to move and thus indicate attempted steering movements (e.g. [141]); and walking freely on a sphere in a ‘Kramer’ treadmill that compensates for the movement (e.g. [133] [164]). Predator avoidance, the most common auditory behaviour in insects, usually involves sensors that are highly sensitive to broad range of ultrasound, with direct ‘command neuron’ connections to avoidance responses such as sharp steering, stalling of flight, etc. [60]. For mate finding or parasitation more accurate directionality is required, as the target has to be located, and the insect must also recognise the specific sound. For example, cricket males produce calling songs of a characteristic frequency and temporal pattern and female crickets (and parasite flies) can find males using this cue alone. The song is produced by the male closing its wings, which rubs a file on one against a scraper on the other, with the sound being amplified by a resonant area on the wing. The carrier
116
B. Webb et al.
frequency of the song corresponds to the rate at which the teeth of the comb pass over the plectrum, and is around 4-5 kHz for most cricket species. The temporal pattern consists of regularly repeated bursts of sound (syllables) corresponding to each closing of the wings, followed by a silent interval as the wings open again. These temporal patterns vary for different cricket species, and thus provide a basis by which females might recognise conspecifics. Males also respond to the song of other males, leading to either spacing out of individuals within an area, or aggressive interactions [92]. In some orthoptera more elaborate communication occurs, in which a male and female ‘duet’, responding to one another’s song. Insect auditory systems have evolved independently many times. The majority are modified chordotonal (mechanical) proprioceptors, with the addition of a tympanum (thinned cuticle) and tracheal sac [61, 177]. In some animals (e.g. Drosophila, [39]) the antennae act as auditory receivers. In different species, ears appear in very different anatomical positions, e.g. forelegs (crickets), head (parasitoid flies, some beetles), or different positions on the body (mantids, grasshopper). The transduction process is still unclear and a comparative study of current evidence is presented in [176]. Typically, the ear contains a small number of sensory neurons (1-100), the axons of which form an auditory nerve that innervates the nearest ganglia. For some animals, the sensory neurons show differential sensitivity to different frequencies, i.e. frequency tuning. This seems to be for broad separation of cues (e.g. to distinguish bat ultrasound, which triggers avoidance, from calling song which triggers approach) - as yet there is no evidence that more complex frequency processing (comparable to the vertebrate cochlea) occurs, and there is only approximate tonotopic projection into the ganglia [50], although there is some evidence of inhibitory sharpening of tuning of interneurons [148]. The sensory neurons also encode intensity of the signal. For stationary signals, the firing rate appears to correspond to the sum of signal energy in the frequency domain [40] but due to adaptation, the response to temporally varying stimuli is more complex. There is also some evidence of range fractionation in intensity coding [99]. Localisation of sound is usually provided by peripheral anatomical structures and enhanced by neural processing. For example, the cricket’s ears are a pair of tympani, on each front foreleg, which are connected to each other and to a pair of spiracles on either side of the front part of the body by a set of tracheal tubes (Fig. 3.17). Because the cricket is small relative to the wavelength and distance of the sound it is trying to localise, there is little difference in the external amplitude of sound at the left and right tympani. However, sound also reaches the internal surface of the tympani from the other auditory ports after delay and filtering in the tracheal tubes. The vibrations of the tympani are thus determined by the combination of filtered delayed and direct sounds [103]. Depending on the frequency of the signal and the direction of the sound, the phase of the delayed sounds will be shifted (relative to the direct sound) differentially for the two tympani, so that the amplitude of the summed signals will differ, even though the amplitude of the direct signals is similar. For a fixed frequency, the resulting amplitude difference indicates the direction of the sound. This mechanism is a very effective means for detecting sound direction when the physical and processing capacities of the animal can support neither sound-shadowing nor phase-locked neural comparison, the two main cues for sound localisation in vertebrate systems. However, because
Low Level Approaches to Cognitive Control sound source
117
external sound pressure internal sound pressure
left tympanum
right tympanum
tracheal tube
Fig. 3.17. The cricket’s ears are a pair of tympani, on each front foreleg, which are connected to each other and to a pair of spiracles on either side of the front part of the body by a set of tracheal tubes. Thus the tympani act as pressure difference receivers and have a directional response.
the delays are fixed, it functions effectively only around a particular frequency. This could potentially contribute to the carrier frequency selectivity found in female cricket behaviour. Similar inherent tuning to detect the direction of specific sound frequencies is found in the very different anatomical structure, consisting of eardrums linked by a flexible bridge, found in the parasite fly that responds to the same male cricket songs [124]. In crickets and grasshoppers there is contralateral inhibition between auditory interneurons in the ganglia (e.g. in the cricket via the large omega neurons (“ON1”) [172]) enhancing the difference between the two sides and possibly also acting as a form of gain control [149]. One pair of identified ascending interneurons (“AN1”) in the cricket’s prothoracic ganglion appear to be critical for phonotaxis [132]. AN1 respond best to sound at the calling song carrier frequency, and clearly encode the pattern of the song in their spiking response. Hyperpolarising one of the pair leads to a change in walking direction. There are a number of other auditory interneurons in the prothoracic ganglion but their functional role in phonotaxis has not been so clearly characterised - some are known to be involved in ultrasound escape behaviour [118]. The ascending neurons project to the protocerebrum. The most comprehensive study of the role of cricket brain neurons in phonotaxis is provided by [131], who suggests a possible filtering circuit for syllable rate recognition by the female. He identifies two main classes of auditory responsive cells: BNC1 which appears to get direct input from
118
B. Webb et al.
AN1; and BNC2 which get input via BNC1. These neurons vary in their response to the pattern of sound. BNC1d appears to require a minimum syllable duration near to the typical calling song before it reaches threshold, which makes it a lowpass filter for the syllable rate, assuming a constant duty cycle. BNC2b appears to spike around once per syllable, which makes it highpass i.e. as the syllable rate decreases the firing rate will also decrease. BNC2a shows a bandpass filtering effect, responding at somewhat less than a spike per syllable for normal rates but producing fewer spikes for slow rates or fast rates. Schildberger argues that the response of BNC2a reflects an “AND”ing of the output of BNC2b and BNC1d, to produce a neural recognition signal for the appropriate sound pattern. A schematic illustration of the critical neural connections is given in Fig. 3.18.
BNC2
BNC2
a
BNC1 d
Left auditory nerve
b
BNC1
BNC2
BNC2 b
a
BNC1
BNC1 d
Right
AN1
AN1
auditory nerve
ON1
ON1
Fig. 3.18. Schematic illustration of some critical identified neural connections in the auditory system of the cricket. Auditory interneurons in the prothoracic ganglion (AN1 and ON1) receive input from auditory receptors and connect to brain neurons (BN). The connections to motor neurons are not yet known.
Staudacher and Schildberger [144, 143] have described properties of some descending neurons, many of which show a response to sound. The response to calling songs is typically ’gated’ by whether or not the animal is walking. One of these neurons has a firing rate that correlates with the angular velocity of the animal, and another seems to be necessary and sufficient for the onset of walking. However the evidence is not sufficient to determine with any clarity the output circuitry for phonotaxis. 3.2.4.2 Extensions to a Robotic Model of Phonotaxis Our previous robot models of sound localisation in crickets combine an analog circuit that reproduces the pressure-difference receiver characteristics of the cricket ear with a computational model of a small circuit of spiking neurons capable of filtering for
Low Level Approaches to Cognitive Control
119
the correct sound patterns and determining the correct turning response to reach the sound [123]. A redesign of the sensor to produce a more compact analog circuit was carried out (Fig. 3.19). The input to the circuit is via two microphones separated by distance equivalent to a quarter of the wavelength of the carrier frequency (4.7kHz) of the stimulus, i.e. 18mm. The input from each microphone is delayed by a period equivalent to a quarter of the phase of the carrier frequency, i.e. 53 microseconds, and subtracted from the input to the other microphone. This mimics the transmission of sound through the cricket’s trachea and the combination at the tympani as shown in Fig. 3.17.
(a) Functional description of ears circuit
(b) Ears circuit and microphones
Fig. 3.19. Schematic and picture of the robot ears circuit
Recent results from cricket biology have led to several significant revisions of the neural model used on the robot. First, it appears that the turning behaviour is not the output of pattern filtering mechanisms but rather involves a fast, ‘reflex’ pathway in which every burst of sound produces a corresponding deviation in the course of the cricket within 50ms (n.b. this is within one step cycle) [45]. However, this reflex is modulated by pattern filtering mechanisms, on a slower time scale: the reflex is usually suppressed, but is ‘released’ over several seconds when the correct sound pattern is detected, and continues to be active for several seconds after the correct sound pattern ends [119]. Note this fits the general architectural scheme we have discussed for the insect brain in chapter 1, i.e. a direct reflex loop modulated by higher processing. Some neurophysiological support for the existence of this fast loop has recently been published [63]. We have implemented a revised neural circuit that incorporates this fast reflex and its disinhibition by a slower loop [128] (Fig. 3.20). We suggest that there is a relatively direct, or ‘fast’ connection from the thoracic AN1 neurons to the motor control of turning. This connection is modulated by disinhibition via the brain neurons BN1 and BN2 which filter for the temporal pattern in the sound in the same way as before. Some interesting issues have been raised from initial tests on this model. It is difficult to obtain similar dynamics for the onset and offset of the response by varying the synaptic parameters within plausible ranges, suggesting that the modulation mechanism may require a different mechanism such as neuromodulator release. It is not clear from the data so far
120
B. Webb et al.
MOTOR LEFT
MOTOR RIGHT
BNC2
GATE
BNC1
FAST
Left auditory nerve
BNC2
GATE
BNC1
FAST
Right
AN1
AN1
auditory nerve
ON1
ON1
Fig. 3.20. An alternative hypothesis for cricket phonotaxis, in which fast connections from the auditory interneurons (AN1) drive motor responses, but only if recognition of the sound by the brain pathway (BNC1 and BNC2) has released inhibition of the response by the gating mechanism
whether there is likely to be bilateral recognition, as in the current model, or whether the input from the two sides is combined in a single recognition process. I.e. a number of predictions or issues for further experimentation on the cricket have been raised. 3.2.4.3 Resonant Filtering As described above, male crickets and bushcrickets produce stereotypic and species specific songs, and female response depends on recognition of the temporal pattern of song pulses. However, the mechanism by which this recognition occurs is still a matter of debate [50]. Behavioural experiments on the bushcricket Tettigonia cantans [9] compared three alternative explanations for the female’s band-pass preference for the pulse rate in male song: a circuit involving separate high-pass and low-pass filters; autocorrelation; and resonance. A stronger response to songs at integer fractions of the preferred frequency, and to other non-natural songs that reinforce that rhythm, suggested that resonance is the best explanation. Resonance in single neurons can occur as a result of intrinsic dynamics of membrane currents, and there is much interest in the significance of this property of neural processing. Izhikevich [66] proposed a simple model for a ‘resonate-and-fire’ neuron and showed this has many interesting properties, including the capability of acting as a bandpass filter. The activity of the neuron is described by a complex variable z whose time evolution is determined by the following differential equation: z˙ = I + (b + 2π iω )z
(3.4)
Low Level Approaches to Cognitive Control
121
where I is the input current, ω is the resonant frequency, and b the rate of attraction to rest. The neuron ‘fires’ when the imaginary part (considered as membrane potential) exceeds a threshold athresh . This is simply a damped oscillator, i.e. an input pulse will cause the membrane potential to start oscillating at the resonant frequency, with a gradually decreasing amplitude. The neuron fires if the peak of the oscillation exceeds a threshold — typically this requires two or more input pulses to occur either simultaneously, or at a relative interval close to the resonant frequency, so that the second input occurs at the peak of the previously induced oscillation. In the original formulation by Izhikevich, firing is followed by a non-linear reset to a new value, but this was excluded in our model to simplify analysis, as it was not found to significantly affect the results. We implemented Izhikevich’s resonant neuron model, as described above, using the parameters b = −30, ω = 25 (the preferred pulse frequency for T. cantans), and athresh = 0.12, simulated at a time-step of 1 millisecond. The input to the neuron, I, was modulated to represent the different song patterns tested on the bushcricket. In the interest of simplicity we neglected all details of input signal shape, transduction processes, synaptic dynamics, and noise and simply used square pulses, that is, the input was either I = 0 (for gaps between sound pulses) or I ∈ [8, 9, 10, 11, 12] representing variation in sound amplitude of the pulse. The response measured is simply the number of times the neuron fires (crosses the threshold) during 1 second of song pattern. The results presented below show this response averaged across the different input amplitudes, and scaled so that the size of the maximum response corresponds to the maximum response seen in the behavioural data. In Fig. 3.21 it can be seen that this simple model closely reproduces the bush cricket behaviour.
Phonotaxis score/Spike response
Phonotaxis score/Spike response
1
0.8
0.6
0.4
0.2
0
0.8
0.6
0.4
0.2
0
8
10 12.5
18.2
25
33
Pulse rate (Hz)
50
67
10
15
20 25
50
100
Pulse rate (Hz)
Fig. 3.21. Phonotaxis response of bushcricket (dark circles) replotted from [9] compared to firing rate of resonate-and-fire neural model (light circles). Left, as an 18ms pulse is played at different rates (for 67Hz the pulse duration was 7ms). Bushcricket data is replotted from [9]. Right, with different rates but a constant 50% duty cycle. Bushcricket data is replotted from [117]. Figures from [162] (Fig. 1 and 4).
122
B. Webb et al.
We can explain the effects of varying pulse and pause duration by considering the behaviour of the system as shown in the phase plane portraits in Fig. 3.22. With a constant input, the behaviour is described by a logarithmic spiral, with a period of 2πω , winding towards a fixed point z f located at Re(z f ) = I b2 +bω 2 , Im(z f ) = I b2 +ωω 2 (marked as a square for I = 0 and as an asterisk for I = 10 in Fig. 5). For square wave input, the system switches between spiralling one fixed point and another. In order to pass above the threshold and produce spikes the system needs to maximise the radius of the spiral in the phase plane, which for a linear system corresponds to maximising its (phase) velocity z˙. This means the switch between spirals needs to be timed so as to amplify or at least sustain z˙. Geometrically, as the input is real, its effect is horizontal, and hence the switch will amplify z˙ most if it occurs when Re(z) ≈ Re(z∗ ) and Im(z) < Im(z∗ ) for the positive change, and Re(z) ≈ Re(z∗ ) and Im(z) > Im(z∗ ) for the negative change. In other words it should act in the appropriate direction at the top or bottom of the spiral, as shown by the arrows in Fig. 3.22a. Thus certain timing constraints of the input have to hold: (i) the dynamical system needs to evolve for (approximately) an integer number of complete cycles between onsets (ii) the dynamical system needs to make (approximately) half a cycle (or half a cycle after an integer number of cycles) during each constant part of the input. This translates into constraints on frequency and duty cycle, e.g. both conditions are satisfied by songs at around 25Hz or 12.5Hz with a pulse
a
b
0.1
Im(z)
Im(z)
0.05
−0.1 −0.1
Re(z)
0.1
−0.05
−0.05
Re(z)
0.05
Fig. 3.22. Phase plane portraits of the system dynamics. The square marks the zero point for I = 0 and the asterisk the zero point for I = 10. The system switches between anti-clockwise logarithmic spirals about these two points, and spikes occur when it crosses the threshold (dashed line) Im(z) = 0.12. The radius of activity, and hence likelihood of spiking, increases if the changes in input (marke d by diamonds for positive change, circles for negative change) occur a t times that increase or sustain the phase velocity z˙, as seen on the left for a square wave at 25Hz with a 50% duty cycle. Otherwise they tend to act against the phase velocity, reducing the spiral radius, as seen on the right for a square wave at 12.5Hz with a 50% duty cycle (note different scale). This accounts for the dependence of the results on both frequency and duty-cycle of the input pattern. From [162] (Fig. 5).
Low Level Approaches to Cognitive Control
123
duration around 20ms (as seen in Fig. 3.22a) but the second condition is not satistified by songs at 12.5Hz with a pulse duration around 40ms (as seen in Fig. 3.22b). The results show that that neural resonance in a single neuron is a possible mechanism for temporal pattern recognition in T. cantans. It could also play a role in pattern recognition in other species of Orthoptera, or indeed in other animals. It shows that resonant properties of neurons could have a direct role in perception. At the same time, detailed modelling of membrane currents is not necessary to reproduce the effects: a basic model of damped resonance is sufficient in this case. It is possible that the resonance could be a property of a neural circuit rather than an individual neuron, although there is some evidence that an identified neuron in the auditory system of the cricket Teleogryllus oceanicus does show a resonant response [50]. It is interesting to note that the same model system could be driven by a constant tonic input to produce spikes at the pulse rate of the song, and thus could also serve as a pace-maker for the production of the song pattern by the male cricket. 3.2.5
Audition and Vision
In section 3.2.3 one visual reflex described was the optotomotor response: rotation of the entire visual field is usually the result of self-rotation, so unintentional self-rotation can be corrected by turning in the opposite direction in response to visual rotation signals. This reflex compensates for external disturbances and inaccuracies of the musclemotor response, to maintain a straight trajectory. Like most insects, crickets exhibit this reflex. But if the cricket is at the same time performing sound localisation, as described in section 3.2.4, then the acoustic and visual mechanisms have conflicting aims. The auditory response tries to align the trajectory towards the sound source, while the visual response tries to correct for any change in the trajectory, thus counteracting the alignment attempted by the auditory system. Different solutions have been proposed to solve this sensory conflict [123, 159, 160] including: adding the outputs of the visual and auditory systems with different gains; having the auditory system output inhibit the visual system output; having the auditory system output inhibit the visual system input; subtracting the auditory system output from the visual system input; and having the auditory system output control behaviour via the visual control system. A comparison among these methods, as discussed in [160], showed that simple suppression of one sensory system by the other was a reasonably effective mechanism. However, a more suitable and flexible control method is to predict the amplitude and time-evolution of the visual stimulation that arises when an auditory response is made, and thus cancel its specific effects. It was first suggested by von Holst and Mittelstadt in 1950 that biological systems may use an internal efferent copy of motor command signals to modulate sensory processing for predicted reafference. A similar idea was simultaneously proposed by Sperry, i.e. that sensory areas receive a corollary discharge corresponding to the expected feedback. Although this concept is often referred to in biology, there is surprisingly little direct neurophysiological evidence of connections from motor to sensory systems that could support this function. Moreover, biologists often
124
B. Webb et al.
Fig. 3.23. The FeedForward compensation model
overlook the problem that the system must somehow calculate the expected feedback in control theory terms, it must implement a forward model to be able to compensate the expected disturbance [158]. The control strategy explored in this section consists of a dynamic system able to predict the external visual influences due to the acoustically driven motor command. The prediction is used to inhibit the optomotor system, so that it is smoothly combined with phonotaxis. Moreover, the system is realised as part of a network of spiking neurons, demonstrating that it is a plausible solution for the cricket, and is shown to be effective in controlling a mobile robot. 3.2.5.1 A Multisensory Control System The multisensory integration is based on feedforward compensation, a well-known method in industrial applications of control theory. It is a simple method to compensate for disturbances that are a priori known and accessible. In such cases, the correct identification of the plant allows an exact compensation of the noise. Here we consider the optomotor reflex as the plant and the phonotaxis system as the source of noise: acoustic sensing produces a series of actions that will change the motor output planned by the optomotor reflex. Hence it can be compensated by using a dynamic feedforward model of the plant to predict the visual sensory (reafferent) signal corresponding to any action caused by the phonotaxis behaviour (Fig. 3.23). Each pair of phonotactic motor signals and the corresponding visual stimulus form an output-input couple, utilized to learn the required dynamics. This controller is embodied in a neural architecture based on the simple integrate and fire neuron models described by [80]. Synaptic effects are modelled as changes in conductance, and include facilitation and depression mechanisms to allow temporal filtering. Such techniques allow the construction of small networks with useful capabilities, such as selectivity to particular patterns, or copying of external dynamics. From these building blocks, a circuit was built to resemble the cricket neural system. It can be divided into 4 different subsystems (Fig. 3.24): A: The optomotor reflex senses the left/right virtual movement of the world and consequently drives the motor controller neurons to compensate the visual stimuli. B: The acoustic sensing is based on cricket physiology and is described in detail in [123].
Low Level Approaches to Cognitive Control
125
Fig. 3.24. Schema of the cricket bioinspired neural network model. The whole network is constituted by several subsystems: the optomotor reflex (substem A), the acoustic sensing block (subsystem B), the phonotactic model (subsystem C), the feedforward compensator (subsystem D), the motor controllers and an internal source of noise.
C: The phonotactic reflex has two different pathways: the reactive one (AN1→Fast) and the recognition one (AN1→BN1→BN7→Gate→Fast) [128]. D: The feedforward compensator has been designed to take the outgoing signal from the phonotactic system and to compensate the corresponding reafferent optomotor signal. An internal source of noise is used to test the correct working of the optomotor reflex. The motor controller units are biased with random spikes, to represent disturbances that could occur during walking in crickets , due to asymmetries in the motor system or to external factors such as uneven terrain. The feedforward compensator (i.e. subsystem C in Fig. 3.24) integrates the responses of the phonotactic system and the optomotor reflex. It takes the driving commands from the phonotactic system and tries to predict the reafferent visual signal. The driving commands also stimulate the motor controllers and consequently the motor actuators. A sound recognized, e.g. from the left, produces an inhibition on the left motor controller and an excitation on the right one, making the robot steer to left towards the sound source. There is a significant delay (a few hundred milliseconds) before the optomotor
126
B. Webb et al.
reflex will sense this left turn. Without any compensation, the optomotor system would react with a correction to the right side, driven by the opto-clockwise (OC) neuron, annihilating phonotaxis. The feedforward compensator (forward-OC neuron and related synapses) will avoid this incorrect response. The Fast-left neuron will stimulate the forward-OC neuron with the same number of spikes used to drive the motor controller. If the parameters of the forward-OC neuron and its input and output synapses are tuned to predict the reafferent signal, this will counterbalance the OC neuron stimulation, holding the neuron on its resting membrane potential. Following the guidelines of classical identification, we assumed the feedforward model had a fixed dynamic structure, for which a number of parameters had to be optimally identified. The structure was fixed by the neuron/synapse structure already present into the complete model of Fig. 3.24. A subset of the parameters could be a priori fixed due to some preliminary considerations. For example, appropriate synaptic delays could be directly estimated from a correlation analysis between the efferent signal (cause) and the reafferent signal (effect). There remained 6 neuron parameters and 6 parameters for each of the two synaptic connections to be optimised. In order to do so, data were gathered from experiments in which the roving Koala robot was allowed to control steering using only phonotaxis, i.e. with the synaptic connection of the optomotor system to the motors disabled. During these experiments the reafferent visual signals were recorded. This was used to tune the parameters first by hand, and then by use of a genetic algorithm, with the aim of providing appropriate inhibition to keep the opto neuron to its resting potential during phonotactic turns, thus filtering out the phonotactic-to-motor ‘noise’. Further details of this process are given in [129]. Experimental setup The results obtained by using the feedforward compensation should allow concurrent use of both phonotactic and optomotor reflexes. The visual noise introduced by the phonotactic behavior will be efficiently filtered out without suppressing the optomotor system, which is ready to react to other external environment disturbance. To test this, we use a Koala robot, equipped with auditory and visual sensors based on insect sensory systems as described in [159]. A tether connects the robot to a PC running the neural simulation program and recording data. The auditory stimulus is a simulated cricket song. We carry out trials with the robot starting in one of three positions: from near the center of the room, facing the speaker from about 180 cm and from half-way down each side wall, facing the opposite wall, about 160 cm from the speaker. For each position ten trials are recorded. The robot is stopped when it is about to hit the speaker or else one of the lab walls; a successful trial is counted when the center of the robot is within 30 cm of the speaker at the point it is stopped. The tracks are recorded using shaft encoders, which are sufficiently accurate for dead reckoning over the short paths considered in these experiments. For each recorded track, a ’directness’ parameter is calculated [135] which combines the average heading of the robot with the time taken to complete the trial. A route at maximum speed straight towards the source would have a directness value of 1. Results for Phonotaxis only: Fig. 3.25 (left) shows the behaviour of the robot driven by phonotaxis only. It performs successfully,reaching the target every time, with
Low Level Approaches to Cognitive Control
127
directness = 0.38. The phonotaxis behaviour when a random noise is introduced into the motor output is shown in Fig. 3.25 (right). The trajectories are more winding, but phonotaxis still guides the robot towards the loudspeaker, despite the noise, so the auditory behaviour is robust. The directness is reduced (directness = 0.31). Adding the optomotor reflex without feedforward compensation: Fig. 3.26 (left) shows the trials carried out when the optomotor reflex is also enabled and a potential conflict with phonotaxis can occur. The robot does not manage to reach the target every time. The optomotor reflex allows the phonotaxis to work correctly when the angle towards the sound source small, in which case it helps to keep the robot stabilised in the correct direction, hence the directness is slightly higher overall (= 0.40). Fig. 3.26 (right) shows the trials carried out when both the random noise and the optomotor reflex are enabled, but the forward model is not active. The trajectories show the conflict between the two sensory systems. Although both the phonotaxis and optomotor systems are working against the noise, the optomotor reflex also “corrects” phonotactic turns. The directness is very low (directness = 0.22). FeedForward compensator allows integration of Phonotaxis and Optomotor Reflex: The forward model behaviour is shown in Fig. 3.27 (left). Here the feedforward controller compensates the reafferent signal, correctly predicting optical signals induced by phonotaxis. The optomotor reflex no longer tries to “correct” these turns, leaving the phonotaxis to reach the target. Directness is improved (directness = 0.44) showing that the optomotor reflex is still active in stabilising the trajectory, but now without interfering with the phonotaxis.In the final experiment the multisensory integration capabilities of the forward model are verified when a random noise is introduced in the motor system (Fig. 3.27 (right)) The prediction of the phonotactic turns is efficient: in fact the attempts look like the previous experiment when no disturbance was considered. The optomotor reflex is able to compensate turns due to random noise, without conflicting with turns caused by phonotaxis. It means that phonotaxis is able to accomplish its
Fig. 3.25. Results with phonotaxis control only. The robot starts from three different positions. For each position ten trials are recorded . Left, the robot every time reaches the target. Right, the robot can also reach the targe with noise added, but directness is reduced.
128
B. Webb et al.
Fig. 3.26. Phonotaxis and optomotor are in additive mode. The robot sometimes is not able to reach the target, particularly when noise is added.
Fig. 3.27. Forward model correctly integrates phonotaxis and optomotor systems. The robot every time reaches the target.
purpose turning the robot towards the sound source. The value of directness is lower than the experiments without noise (directness = 0.34), but improved in respect to the experiments with noise in which the phonotaxis and optomotor behaviours were not integrated by the forward model. The implemented feedforward controller showed the capability to correctly predict and filter out phonotactic induced optical noise, leaving the optomotor system ready to react to unknown disturbances, as observed in experiments with crickets. Because the feedforward compensator could be implemented in a realistic spiking neural network, it is a plausible mechanism to explain real cricket behaviour, and shows how biological systems might implement, and tune, this form of control. The result obtained in this paper also opens the way to the hardware design and implementation of the whole neural-controller system for the integration in a complete autonomous machine.
Low Level Approaches to Cognitive Control
129
3.3 Navigation 3.3.1
Path Integration
3.3.1.1 Background Path integration, a term coined by Mittelstaedt [105], is a navigational strategy that explains the ability of some animals to return home on a direct path after a long and tortuous foraging excursion. Even in environments completely lacking distinct landmarks, or in the dark, an animal can use sensing of distance and direction travelled to integrate its velocity vector over time and thus estimate where it is in relation to its starting position. Path integration is a strategy used by a wide range of animals, including rats [3, 77], dogs [136], humans [106], spiders [110] and desert ants [166]. The importance of path integration to the field of robotics as a navigation strategy is best exemplified by NASA’s Mars Rover of the Pathfinder mission [100]. The homing behaviour in the desert ant (Cataglyphis fortis) has been intensively studied and there is a rich corpus of experimental data available (e.g., [173, 4, 14]). To perform path integration, ants use the polarised light from the sun to derive their current compass heading [165]. Sahabot [88], developed at the University of Zurich by, uses polarisation sensors modelled on those of the ant, but the underlying path integration system relies on the equations proposed in [105] rather than a neural implementation. Ronacher and Wehner [125] showed that even after removing all allothetic (external) cues desert ants can accurately estimate distance traveled, and recent evidence suggests they actually ‘count’ their steps to gauge distance travelled [170]. Ants thus rely on allothetic and idiothetic (internal) inputs to determine heading and distances travelled respectively. Several neural network models for path integration in ants have been developed. Among the best known are the models by Hartmann and Wehner [42] and Wittmann and Schwegler [171]. There are also several neural models of path integration in rats, usually designed as input to place cell systems based on the hippocampus (e.g., [8]). More recently, Vickerstaff and Di Paolo [156] used an evolutionary approach and found a compact network that implements the bi-component model suggested by [105]. However this result depended on the input to the network taking the form of a sinusoidal compass sensor response function. When using more biologically plausible sensor response functions, Vickerstaff and Di Paolo report a failure in evolving successful homing behaviour. We also take an evolutionary approach, but focus on how a population of neurons can encode (lower-dimensional) sensory information for path integration. 3.3.1.2 Evolving a Neural Network for Path Integration The agent used in the genetic algorithm is simply modelled as having a position (x, y) and heading θ on an unbounded two-dimensional plane. The sensors available to the agent approximate the response of interneurons, studied in the cricket [87], that receive input from a specialised region of the eye (also found in ants and bees) adapted to sense the polarised light patterns in the sky. We define a set of three ‘direction cells’ with preferred directions evenly distributed around 360◦. The activation of each cell is
130
B. Webb et al.
1 0.8 0.6
response of direction cell
0.4 0.2 0 −0.2 −0.4 −0.6 −0.8 −1 0
50
100
150 200 heading angle
250
300
350
Fig. 3.28. Response of three direction sensors with preferred directions of 60◦ , 180◦ and 300◦ . From [41] (Fig. 1).
calculated by performing the dot product of the preferred direction of the cell (h p ) and the actual heading (ha ) of the agent: cos(ha ) cos(h p ) · + [ξ ∼ N(0, σ )] (3.5) firing rate = sin(h p ) sin(ha ) where ξ is a Gaussian noise term with mean 0 and standard deviation σ . The ensemble of direction cells thus encodes the current heading of the agent as a population code as shown in Fig. 3.28. Each agent has two turn effectors, for turning left or right. The input defines how sharp the turn will be. Currently the agent is maximally allowed a 20 degree turn per iteration. The agent is also driven forward at a constant speed. Sensors and effectors are connected by a neural network. Several standard neuron representations were explored, but the results below use two kinds: a standard sigmoid neuron: 1 oi = (3.6) −aI (1 + e i+bs ) where the output oi depends on the summed input Ii , a parameter a and bias bs and the input to a cell is calculated by summing up the output of all connected cells multiplied by the connection weights: n
Ii =
∑ w ji o j
(3.7)
j=1
where Ii is the input to cell i, w ji is the connection weight from cell j to cell i and o j is the output of cell j; and a ‘memory’ neuron for which the cell potential c of the neuron is updated as follows: dc = Ii (3.8) dt
Low Level Approaches to Cognitive Control
131
and the output of the neuron applies the sigmoid function to the cell potential: oi =
1 (1 + e−c+bc )
(3.9)
A biological equivalent of such a neuron that integrates its input over time was described in the rat [23]. A genetically inspired neural encoding for the network structure and parameters was adopted based on [34], which allows a variable number of neurons and connections encoded as a list of integers. A neuron’s parameters and excitatory or inhibitory connections are encoded between a start marker and end marker, as shown in Fig. 3.29. The genome size was limited to 500 parameters, which gave an approximate upper bound of 50 neurons. Localised tournament selection was used to determine the individuals from each generation chosen for reproduction with mutation and crossover. Fitness was evaluated during simulated journeys in which the agent is moved through two randomly generated way points, and must then home according to the output of its turn effectors as activated by its network. Each agent is evaluated until a maximum number of time steps m is reached, and at each cycle the distance to home distt is measured. The fitness combines
1.2
1.4
Fig. 3.29. Marker-Based Genetic Encoding: a chromosome consisting of integers is interpreted as a neural network. A start marker is any number that modulo k results in a remainder of 1 and an end marker is any number that modulo k results in a remainder of 2. k is typically chosen somewhere between 5 and 15. From [41] (Fig. 2).
132
B. Webb et al.
the inverse of the squared distances with penalties for spiralling (excessive difference of the agent’s heading θ from the nest heading ω ) and network complexity as follows: fitness = (1 + kn)cn (1 + kc )cc
1
m 2 0 distt |θ
− ω | dt
(3.10)
where kn is a neuron penalty constant, kc is a connection penalty constant, cn is the neuron count and cc is the connection count. Initial trials found no acceptable solutions even when letting the genetic algorithm run for a very long time. The genetic algorithm would reach local maxima and show hardly any improvement after that. This could be the result of the genetic encoding not allowing much change to the network topology without a momentary decline of fitness. In order to tackle the problem, the search space was reduced by predefining certain topological aspects of the neural network structures. The following constraints were set on the network topology: • • • •
each direction cell had to be connected to at least one memory neuron, each direction cell had to be connected to at least one sigmoid neuron, sigmoid neurons had to be connected to turn effectors, a maximum of two turn effectors, one left and one right were allowed.
3.3.1.3 Simulated and Robot Results A fit solution that was consistently evolved is depicted in Fig. 3.30. A typical simulation run is shown in Fig. 3.31. It is reasonably easy to understand how the network works, by analysing its activity in various situations, as shown in Fig. 3.32. Each direction cell
Fig. 3.30. Evolved network using direction (D) cells. Each direction cell has one memory (M) CTRNN cell and two sigmoid (S) neurons associated with it. For illustration purposes, the turn motors (represented by the left and right arrows) are duplicated and the recurrent connection of the memory neurons onto themselves are omitted. From [41] (Fig. 3).
Low Level Approaches to Cognitive Control
Nest
133
Outbound
Homebound
Fig. 3.31. Example simulation run. From [41] (Fig. 4).
is directly connected via an excitatory connection to a memory neuron. As a result, by integrating its input, the memory neuron records the distance traveled in the direction signalled by that direction cell. The population of memory neurons therefore encodes the homing vector in a very natural way, similar to the way the population of direction cells encodes the current heading direction. Each direction cell and each memory cell is also connected to two sigmoid neurons, which in turn connect to the turning effectors. These thus become active only if both the direction cell and at least one of the memory neurons are active. During homing, the memory neurons thus act like repelling forces, turning the agent away from the direction that it travelled on its outbound path. We tested the network on a Koala robot to validate that the evolved network provides a plausible real world solution. The Koala (shown in Fig. 3.33) is equipped with 16 infra-red (IR) proximity sensors, which are utilised to detect obstacles, and a 22 MHz Motorola 68331 processor with 1 Mb of RAM . Robot motion is controlled by two DC motors configured in a differential drive setup. The current heading of the robot is determined by monitoring the changes in wheel encoder readings produced during robot motion. The derived heading is then used to activate the corresponding direction cells appropriately. Fig. 3.34 shows that the robot correctly locates the home position when subjected to zig-zagged patterns of various length and direction. Additional paths were defined to test the global vector derivation ability of the network. Fig. 3.35 shows an outbound path measuring 7m in length, with the final beacon is positioned only 1m from the home position. The robot is again shown to reliably locate the home position. The sequences of 3 left and 3 right turns reduce accumulative error caused by the wheel encoders which results in an error of position of 14cm. A simple obstacle avoidance capability was added to the robot using the Koala IR sensors. The algorithm forced the robot to turn away from an obstacle overriding the motor commands issued by the path integration network. The network however still receives input from the heading and distance sensors throughout the avoidance manoeuvre, and therefore updates the global vector appropriately. Thus when the robot encounters an obstacle, it first navigates around the obstacle before returning to its
134
B. Webb et al.
(a)
(b)
(c) Fig. 3.32. Examples of network activations for different situations. Bold lines indicate strong activations. (a) The agent is heading northeast and the top-right direction cell is strongly active. The sigmoid neuron connected to the left turn motor is the only neuron to receive strong inputs from both a memory neuron and the direction cell, thus causing the agent to turn left towards the nest. (b) The agent is heading northwest. This time, the top-left direction cell as shown in the left part of the figure is active and, in conjunction with the active memory neuron, it causes the agent to turn right. (c) When the agent is pointing towards the nest, the activations of both turn effectors cancel each other out, and the agent moves straight ahead. From [41] (Fig. 5).
Low Level Approaches to Cognitive Control
135
Table 3.1. Evolved neuron parameters Parameter Evolved value a 0.667 bs −4.372 bc −1.164 τ 135518 Table 3.2. Evolved connection weights Weights Excitatory value Inhibitory value wS-left 3.974 −3.974 wS-right 3.976 −3.976 wPOL-S 0.719 wPOL-M 0.012 wM-S 3.962 wM-M 10−4 -
Fig. 3.33. The Koala robot used in the experiments. From [41] (Fig. 8).
homing behaviour. The ability of the robot to correctly home following an encounter with an obstacle on the homeward path was tested as shown in Fig. 3.36. The robot skirts the obstacle before following the altered global vector to the home position. The second obstacle avoidance test places two staggered obstacles in the homeward path of the robot. Fig. 3.36 shows the robot to correctly avoid both obstacles before following to the corrected global vector to the home location. These results show that the network does update the homing vector as it is turning towards the home position after
136
B. Webb et al. 400
400
300
300
200
200
100
100
0
0 −300
−200
−100 A
0
100
400
400
300
300
200
200
100
100
0
0 0
100
200 C
300
400
−300
−200
−100 B
0
100
0
100
200 D
300
400
Fig. 3.34. Robot path integration over a zig zag path of different lengths: (A,C) 4m and (B,D) 6m. The errors were (A) 27cm, (B) 60cm, (C) 35cm, and (D) 42cm. From [41] (Fig. 10).
200
150
100
cm
50
0
−50
−100
−150
−200 −200
−150
−100
−50
0 cm
50
100
150
200
Fig. 3.35. Convoluted path. The error for this run was 14cm. From [41] (Fig. 11).
avoiding obstacles. The system accumulates error due to multiple heading adjustments when avoiding obstacles and is at present not highly accurate for large or multiple detours but could be improved with more accurate compass sensors.
Low Level Approaches to Cognitive Control 400
400
300
300
200
200
100
100
0 −200
−100
0 A
100
200
0 −200
−100
0
100
200
137
300
B
300
300
250
250
200
200
150
150
100
100
50
50
0
0 0
100
200 C
300
0
100
200
300
D
Fig. 3.36. Robot path integration and obstacle avoidance experiments. The obstacles were only placed into the arena after the agent reached the last waypoint and starting its homeward journey. The errors were (A) 32cm, (B) 92cm, (C) 105cm, and (D) 108cm. From [41] (Fig. 12).
The overall biological plausibility of the evolved network is an open question. Investigations into the polarisation vision pathways of insects have suggested that the central complex, a neuroarchitecturally distinct neuropil in the insect brain, may be involved in functions of compass orientation and path integration (see [56] for a review). The central complex has a regular neuroarchitecture, which has been said to bear some similarities with models of path integration (c.f., [111]). It hosts a maplike representation of e-vector orientations [46], and has connections with motor centres in the thoracic ganglia. In some insects it receives, besides polarisation-sensitive input, connections from circadian clock neurons, possibly involved in time compensation by adjusting compass bearings in relation to solar azimuth with time of day [56]. Given these developments in insect neuroscience, a future goal is to map more directly our models for path integration to the neuroarchitecture of the central complex. 3.3.2
Visual Homing
3.3.2.1 Background Many insects have been shown to be capable of navigation using visual homing. For example, we have demonstrated that crickets can learn to relocate an unmarked cool
138
B. Webb et al.
spot on a heated arena floor (Fig. 3.37) using the surrounding visual scene [168], and will search in the corresponding location if the visual surroundings are rotated (Fig. 3.38). This capability allows desert ants [166] and wasps [154], among other insects, to relocate a hidden nest entrance from anywhere in the immediate surroundings. The problem of visual homing can be abstractly stated as follows. An agent, equipped with a monocular camera is located at a point C. The goal is for the agent to move from C to a home position S, using a home image taken from location S and the current visual information from the camera (Fig. 3.39) to determine its movement. Visual homing is a short-range navigation method which can lead an agent to a position with accuracy, provided the majority of the scene visible from the home position is also visible from the current position. It is generally assumed that the agent has an omnidirectional image of the environment [29], which is true for most insects, and for robots can be obtained by using a spherical, parabolic or conical mirror located above a camera pointing upwards. Methods proposed for visual homing methods can be divided into two groups: feature-based methods, where specific features are extracted from the images and matched between the current and the home image; and image-based methods, where the image pixel values are used directly. Feature-based methods include the snapshot model [11, 163] and the Average Landmark Vector method [88, 109]. The most popular image-based method is Image Warping [30]. This works under the assumption that all landmarks are situated at equal distances from the home location. Using a greyscale one-dimensional horizon image taken from the current location, the model calculates all possible deformations of the image line, for all possible rotations and movement vectors. The agent then moves according to the movement vector that produces the image most closely resembling the home image. This approach been shown to be effective and robust, but at the substantial computational cost of exhaustive search to find the best match. A simpler image-based method was proposed by Zeil [180], who studied the properties of outdoor scenes for the purpose of image homing, using a panoramic camera in the natural habitat of a wasp that is known to locate its nest by visual homing. They captured a home position image, and similar images from a grid of positions around the nest. Each image was compared to the home image using the root mean square difference function: M N 2 ∑ ∑ (IC (i, j) − IS (i, j)) i=1 j=1 RMS = (3.11) M∗N where IC (i, j) is the intensity function for pixel (i, j) in the current image, IS (i, j) is the intensity function for pixel (i, j) in the home image, and M ∗ N are the dimensions of the images. The 3-dimensional function resulting from the (x, y) grid of current image positions pairs and the corresponding RMS values is called the RMS difference surface. Zeil et al. found that the difference surface they measured was unimodal i.e. movement towards the home position resulted in gradual reduction of the difference, reaching a global minimum in the target location, with no local minima whatsoever. The size of the area for which this held true (the ‘catchment area’) was up to 3 square meters. They suggested it is possible to take advantage of the unimodal property of the catchment
Low Level Approaches to Cognitive Control
139
50cm
Camera Computer (tracking software) Canopy
92cm Temperature gradient Cricket path Arena (30cm)
Cool spot (6cm)
Hot plate Hot water
Pump
Cold water
(a)
Rotation trial
Training trial
Original Cool spot
Random
Fictive
(b)
(c) Fig. 3.37. Experimental setup used to test homing in crickets. (b) For the rotation trials, the wall of the arena is rotated by 180 degrees, changing the position of the visual cues, and creating a fictive target location relative to those cues. (c) This natural scene wallpaper is wrapped around the inside of the arena wall.
140
B. Webb et al.
Time to reach position (seconds)
Artificial cues
No cues
Natural scene
Control
300
300
300
300
250
250
250
250
200
200
200
200
150
150
150
150
100
100
100
100
50
50
50
50
0
O
F
R
0
O
F
R
0
O
F
R
0
O
F
R
Location in arena, O=original cool spot, F= fictive, R=random
Fig. 3.38. Crickets search preferentially in the ‘fictive’ position of the cool spot as indicated by rotated cues or scenery
Fig. 3.39. The homing task is to move the agent from C in the direction of S. In the simplest case, visual information is the relative location of the projection landmark L; this allows the home direction to be calculated from simple geometry. From [179] (Fig. 1).
area to perform visual homing. The agent can move towards the home location by descending towards the point that gives the minimum RMS difference. Homing can thus be regarded as a function minimisation problem. Function minimization by gradient descent (also known as steepest descent) in general works by determining the negative of the gradient of the function at the current position and moving in this direction until a line-minimum is reached [167]. However, the only information available to an insect or robot is the current RMS difference and the sequence of the inputs so far. This means that it cannot easily calculate the
Low Level Approaches to Cognitive Control
141
gradient of the function at a given position. The strategy proposed by Zeil et al. [180], called ‘RunDown’ was for the agent to move in a random direction for a fixed distance, and then determine if the RMS difference has decreased, in which case it moves forward again in the same direction, otherwise, it turns 90 degrees before moving forward again. 3.3.2.2 Evolving a Gradient Descent Algorithm for Visual Homing In an attempt to find a more efficient way to take advantage of the properties of the RMS difference surface for the purpose of visual homing, we turned towards a biologically inspired approach for function optimisation. Organisms such as the nematode Caenorhabditis elegans perform chemotaxis using a single sensor to detect only the current concentration of the chemical substance [27], and thus solve an analogous problem to a robot trying to home using only the current RMS difference. In [27], it is demonstrated that the turning behaviour of C. elegans during chemotaxis can be modelled by: dθ = ωbias + z0C(t) + z1 (C(t) − C(t − 1)) dt
(3.12)
where ddtθ is the turning rate (or the angle turned in one time step), Obias , z0 and z1 are constant parameters, C(t) the input at time t, and C(t) −C(t − 1) an estimate of the temporal gradient of the input. In most forms of gradient descent or ascent, a decreasing step size is desirable, so that the agent can approach the goal rapidly and then slow down to locate it with the desired accuracy. This can be implemented by making the forward movement at each time step proportional to C(t), i.e. C(t) multiplied by some constant U. To use this Taxis algorithm it is necessary to find appropriate parameter values of Obias , z0 , z1 and U. We used an evolutionary strategy to determine these parameters, using a simulated agent homing on RMS difference surfaces calculated from previously collected images in a robot lab environment (available at http://www.ti.unibielefeld.de/html/research/avardy/index.html), courtesy of Andrew Vardy (Memorial University, St. John’s). These sets were omnidirectional images of the same 3 x 5.1m area (an office room), three sets with the same pictures captured under different illumination levels (‘Original’, ‘Twilight’, ‘Night’), and another set where a number of chairs had been added to the room (‘Chairs’). The images corresponded to a real world grid with 30 cm distances between every two consecutive images. To obtain robust controllers, we created a variety of surfaces that used the home image from one data set compared with current images from another data set, as listed in table 3.3. The fundamental property of the global minimum appearing at the home position held true for all surfaces (Fig. 3.40). Unimodality was observed in all combinations except those that used a home image from the ‘Chairs’ dataset, when some local minima were observed. The details of the evolutionary process used to optimise the four parameters (Obias z0 , z1 ,U) of the Taxis algorithm are given in [179]. Briefly, individuals were encoded by four real numbers. The selection method used was Stochastic Universal Sampling, and genomes were mutated (changing each parameter by a small random value drawn from a uniform distribution) with a probability of 0.6, and underwent crossover with the same probability. The population size was 200. The fitness was evaluated by running each individual for a maximum of 500 time steps on six different surfaces. A run was considered successful when the agent’s distance from the goal was less than a threshold
142
B. Webb et al.
Fig. 3.40. Four characteristic root mean square (RMS) difference surfaces. A) RMS difference between a home image from the Original set and every image in the Original set. B) RMS difference between the same home image and the Chairs set. C) RMS difference between a home image from the Chairs set and the Chairs set. D) RMS difference between a home image from Chairs and the Original set. Each unit in the axes corresponds to 0.3 m. From [179] (Fig. 3).
D (=0.1 m). The fitness score was 500−number of steps to reach home, with failure to reach home within 500 steps scored as 1. To reduce dependence of the evolved solutions on details of the simulated environment, we applied Jakobi’s ’radical envelope of noise’ methodology [70], adding Gaussian noise to the RMS value, the agent’s turning angle and the displacement at each time step. 3.3.2.3 Results of Evolving the Simulated Agent The GA was able to come up with an efficient controller in about 250 generations. The controller parameters from three different runs of the GA were: Controller 1 2 3
ωbias z0 z1 U (in 0.3m) 0.6490 0.9485 44.6288 0.6744 1.3821 -0.7951 37.0456 0.6492 103250 -0.7987 36.1076 0.9403
It is possible to directly interpret these parameters, following the discussion of Ferree and Lockery (1999). In all three controllers the bias Obias and the first-order term z1
Low Level Approaches to Cognitive Control
143
had the same sign. This meant that, for a certain value range of C(t) −C(t − 1) < 0, the bias and the first-order term cancel each other out, resulting in a small or zero turning rate when the gradient is decreasing. As the values change to greater or smaller than that range, the agent is forced to turn in steeper angles. This causes a behaviour called klinotaxis, described as “a change in turning rate in response to the spatial gradient of a stimulus field” [27]. The zero-order term z0 changes the turning rate proportionally to the input itself, producing klinokinesis: “A change in turning rate in response to the scalar value of a stimulus field” [27]. The sign of the zero-order term can be either positive or negative, because the input is always positive, and, since we are dealing with angles, reducing an angle towards 0 is the same as increasing it towards 360◦. As the agent moves towards the home position, klinotaxis becomes more and more important, while the role of klinokinesis is reduced to simply adjusting the effects of the first-order term. Note that C(t) − C(t − 1) is generally much smaller than C(t) itself, hence z1 is correspondingly larger than z0 . In order to evaluate the homing abilities of the controllers, we performed 1000 runs of each controller on each of the nine surfaces, including the six on which the controller was evolved, and the three not used in evolution. The RunDown algorithm was also tested on the same surfaces for comparison. The controllers were evaluated with respect to five different measures: the homing success rate; the mean number of steps; the total distance travelled; the total angle turned; and the homing precision achieved. A sample run of each controller appears in Fig. 3.41, and the homing success rate for each controller, for 1000 runs, appears in table 3.3. The common factor for the surfaces where performance is below 100% is that they use as the current image set, the ’Chairs’ dataset. This implies that the true difficulty for homing comes when local optima appear. The RunDown algorithm is more seriously affected in these conditions, because its fixed turn size can lead it to become trapped, moving around a local minimum in square patterns. We found no direct link between the surfaces used to train the controllers, and the number of steps needed to home; RunDown tended to travel further than the other controllers per homing run; but turn less (as might be expected). Further details and discussion of the measured results for each parameter are given in [179]. 3.3.3
Robot Implementation and Results
A Koala robot (Fig. 3.42) was used for real world comparison of the Taxis, RunDown and Image Warping algorithms . Images were captured using a Creative Labs Webcam pointing downward at a parabolic mirror manufactured by Kugler. The camera was supported on a rig consisting of three narrow pillars, which were visible in the image but did not appear to significantly affect any of the algorithms. The webcam connected via a USB port to a Dell Inspiron 7500 Laptop with an Intel Pentium III processor, running Mandrake Linux 8.2. All image processing, calculations and generation of movement commands were programmed in C on the Laptop. Motor commands were sent via a serial link to the built-in robot microprocessor for execution. The robot proved very consistent and accurate in its turning angle and movement distance, well within the noise parameters used in the simulation. It proved necessary to re-run the parameter optimisation procedure for the Taxis algorithm on the real robot. We explored a fast method for obtaining an estimate of
144
B. Webb et al.
Table 3.3. Success rates for the evolved controllers on different surfaces created by combining home and current images from different sets. Those combinations marked * were used in the evolution process, the others used only for evaluation. Home image Original Original Original Original Night Twilight Chairs Chairs Night
Current image Controller 1 Controller 2 Controller 3 Original* 100% 100% 100% Twilight 100% 100% 100% 100% 100% 100% Night* 90.6% 94.7% 84.9% Chairs* Original* 100% 100% 100% 100% 100% 100% Night 100% 100% 100% Original* Chairs* 95.4% 96.9% 83.3% 91.8% 95.7% 89.8% Chairs
Overall RunDown 100% 100% 100% 100% 100% 100% 90.07% 78.6% 100% 100% 100% 100% 100% 100% 91.87% 78.2% 92.43% 77.4%
Fig. 3.41. A sample run with each controller. Upper left: controller 1. Upper right: controller 2. Lower left: controller 3. Lower right: RunDown. From [179] (Fig. 5).
the surface properties that could be used for GA optimisation, which does not require capturing a complete image grid. The robot captured a home image, and then captured a line of RMS differences by moving outwards in one direction and capturing images at equal distances. A surface was then created by rotating this line of values through 360 degrees. By capturing the RMS difference line for each of the four directions and evolving a controller that could home on all four corresponding (artificial) surfaces, the controller was optimised for the properties (steepness, value range) of the given area.
Low Level Approaches to Cognitive Control
145
Fig. 3.42. The robot used to test homing. From [179] (Fig. 4).
RunDown was used as for the simulation tests, and the Image Warping algorithm was implemented as follows: Given rotation ψ , home vector angle α and the ratio ρ of the agent’s distance from the home position d to the home position’s distance from the landmarks R, i.e. ρ = Rd , a pixel located at angle θ in the current horizon image will be displaced by: ρ sin(θ − α ) −ψ δ = arctan (3.13) 1 − ρ cos(θ − α ) Searching over all α , ψ and ρ , the best match to the home image is found, and the corresponding home vector used to move the robot for a fixed distance. The termination of homing is signalled when the home vector changes by > 170◦, which indicates the robot has just passed over the home position. The only free parameter is the step size, which was set to 0.25 metres. One problem encountered on the robot was that we could no longer assume that all images have been captured from the same orientation. To determine the RMS for a particular image, it was calculated as the minimum RMS for all possible rotations (horizontal pixel shifts of the image), which should occur when the orientations are aligned. For the difference surfaces observed by the robot, as before, the global minimum was at the home position, and there was a gradual increase in the RMS difference as the robot moves away from home. However, the RMS surface was not as steep as in the
146
B. Webb et al.
Vardy dataset. The main reason appeared to be noise introduced by using pixel shifting to internally rotate the image, which assumes the camera is perfectly orthogonal to the panoramic mirror, and the position of the mirror centre in the panoramic image (i.e. before unfolding) is known with accuracy. Our camera rig was less accurate in this respect than those used by Zeil et al. and Vardy. The performance of the three algorithms was compared, using the same measures as in the simulation. The first tests used a well-lit, unchanging environment, and consisted of two sets of ten runs for each algorithm: ten starting from different positions 1 metre away from the home position, and another ten runs starting from 2 metres away. Three different home positions were used across these trials. Algorithm Taxis RunDown Warping Success 90% 80% 100% # of steps 20.7 13.8 6.8 At one meter Distance traveled 4.2m 2.5m 1.8m Angle turned 1316◦ 360◦ 316◦ Homing precision 0.43 0.18 0.29
Algorithm Taxis RunDown Warping Success 40% 90% 60% # of steps 22.3 23.6 16.3 At two meters Distance traveled 6.0m 4.5m 3.80m Angle turned 1559◦ 580◦ 999◦ Homing precision 0.42 0.25 0.27 Unlike the simulation, the Taxis algorithm performs worse than the RunDown algorithm, particularly for the longer distance where there is a low success rate (40%), but also at the shorter distance where the number of steps, distance travelled, angle turned and homing precision are all worse for the Taxis algorithm. We believe this is probably due to greater sensitivity of this algorithm to the noise introduced by image rotation. The Image Warping algorithm performed better than both the Taxis algorithm and RunDown over shorter distances. However, note that this result is for a static environment. Our next test examined homing by each algorithm when gradual changes were made to the environment with respect to the stored home image. Algorithm Taxis RunDown Warping Success 80% 90% 0% # of steps 22.3 13.3 With changing environment Distance traveled 5.1m 1.6m Angle turned 1670◦ 209◦ Homing precision 0.22 0.19 Obviously, sufficient change in the environment will render the home image position unrecognisable, and homing will fail for any algorithm. On the other hand, all
Low Level Approaches to Cognitive Control
147
Fig. 3.43. The moved landmarks experiment A: The unfolded home image. B: An unfolded image taken from the home position after changes in the scene. From [179] (Fig. 15).
Fig. 3.44. The moved chairs scene. H: The home position. M: The local minimum. T: The perceived home position for image warping. From [179] (Fig. 16).
148
B. Webb et al.
algorithms should demonstrate a level of tolerance to minor changes in the environment. In the example of the changes shown in Fig. 3.43, it was observed that the Image Warping algorithm was unable to home. In fact, its perceived home position had been relocated from position H in Fig. 3.44 where the robot lies, to the ‘trap’ position marked with T. Thus, the Image Warping algorithm was rendered completely useless. For the RMS-based algorithms (Taxis and RunDown) the most significant problem was the appearance of a local minimum at the position marked with M in Fig. 44. But the local minimum was not as deep as the global minimum, which remained on the home position. An explanation for this is that the RMS surface is calculated using visual information not only from the horizon line, but instead from the entire scene, including floor and ceiling patterns. This means that, even when the landmarks happen to move in such a way as to misguide Image Warping, the RMS surface needs much more radical changes in the scene to lose the global minimum. Consequently, in these tests, both these algorithms were still able to home. 3.3.3.1 Building More Robust Difference Surfaces Zeil et al. [180] identified a significant problem with difference surface homing: when IS and IC are captured in different illumination conditions, the chances of homing successfully decrease, often dramatically. We see an example of this in Fig. 3.45. To produce difference surfaces which are more robust to changes to illumination there are two obvious remedies: • Transform reference and current image intensities to minimise the effects of dynamic illumination (using e.g. histogram equalisation). • Use image similarity measures other than the root-mean-square. While these approaches are not mutually exclusive, we chose in this work to follow the second approach only. Along with dynamic illumination, we are also interested on the effect of the movement of imaged objects between captures of IS and IC . We first established the success/failure rate of the standard RMS approach, using the database of images described previously [155]. Landmark layout and/or illumination conditions differ from set to set as described in Table 3.4. We created a number of difference surface sets,defined by the source of snapshot and current images. For example, one set of difference surfaces used the “Original” images for both IS and IC ; another set took IS from “Winlit” and all IC from “Original”, thus simulating dynamic illumination. The complete list of data set pairings is given in Table 3.5. We used the pairings along the diagonal of the table to create difference surfaces reflecting static environments. The off-diagonal pairings involving the “Winlit” and “Doorlit” sets simulate a laboratory environment in which illumination is non-constant. The off-diagonal pairings involving the “Arboreal” set simulate an environment in which there is change in the location of relatively unobtrusive landmark. The pairings (Chairs1,Chairs2) and (Chairs2,Chairs1) yield difference surfaces reflecting the movement of more prominent objects in the environment. Every difference surface set consists of nineteen surfaces corresponding to nineteen different home snapshot locations uniformly distributed around the experimental area. To rate the success of homing we use the criterion defined in [155]: the return ratio RR of successful homing runs to the total number of homing
Low Level Approaches to Cognitive Control
149
450
400
350
300
y(cm)
0.35 0.3 0.25
250
200
RMS
0.2
150
0.15 0.1
100
0.05 0 500 400
50
300 250
300 200
200
150 100
100 y(cm)
50
50 0
0
x(cm)
100
ls
(a)
150 x(cm)
200
250
(b)
Fig. 3.45. (a) A difference surface formed using the RMS image similarity measure defined by Equation 3.14. The reference image was captured in a different illumination condition than all current images IC . Notice that a local difference surface minimum coincides with the reference location but other local optima have appeared. (b) Here we illustrate a number of homing runs using the difference surface in (a). Each homing run starts at a grid point on the laboratory floor. The simulated agent moves so as to optimise the difference surface using a gradient descent algorithm. Successful homing runs (i.e. those ending within 30cm of the reference location) are shown in blue and homing failures are shown in red. There are a significant number of homing failures.
runs. Homing runs are initiated from every non-snapshot location in the experimental ¯ is the mean return ratio for all nineteen difference area. The average return ratio (RR) surfaces in a difference surface set. We define the RMS image similarity between grayscale images IS and IC as
1 N (3.14) RMS(IS , IC ) = ∑ (IC (i) − IS(i))2 N i=1 The gradient of the difference surface at each position x, y is here calculated using Matlab’s gradient command which uses the following two-sided differencing equation: ∇ f (x, y) ≈
f (x+h,y)− f (x−h,y) 2h f (x,y+h)− f (x,y−h) 2h
(3.15)
where h is the separation between grid points, in our case 30cm. Gradients at non-grid points in the experimental area are estimated by linear interpolation. The agent then homes by moving 30cm opposite the direction of the gradient. The agent continues a homing run until one of the following stopping criteria is satisfied: the number of gradient calculations exceeds 400; or the agent detects that its last twenty homing steps
150
B. Webb et al. Table 3.4. Description of each of the image data sets used in this work
Original All overhead lights are on. No obstructive objects have been placed on the floor of the experimental area. Winlit Only the bank of lights near the curtained window (upper half of the image) are switched on. Those near the door are off. Doorlit Only the bank of lights near the closed door are switched on. Those above the curtained window are off. Arboreal All overhead lights are on. A plant has been placed in the centre of the experimental area. Chairs1 All overhead lights are on. Three office chairs have been placed along the walls of the laboratory, out of the experimental area. Chairs2 All overhead lights are on. The three chairs in “Chairs1” have been moved to the experimental area.
Table 3.5. Source of snapshot and current images for the various difference surfaces used in this work. A indicates that the pairing was used to create a set of difference surfaces. Current image data set
Snapshot set
Original Winlit Doorlit Arboreal Chairs 1 Chairs 2 Original
-
-
Winlit
-
-
-
Doorlit
-
-
Arboreal
-
-
-
-
Chairs 1
-
-
-
-
Chairs 2
-
-
-
-
Table 3.6. Average return ratios for homing experiments carried out on RMS difference surfaces in static and dynamic environments. The standard deviation of the average return ratio for each data set pairing is given in brackets. Current image data set Original
Winlit
Doorlit
Arboreal
Snapshot set
Original 0.977 (0.066) 0.584 (0.398) 0.654 (0.412) 0.933 (0.083) Winlit
0.427 (0.164) 0.949 (0.058) 0.020 (0.059)
Doorlit 0.536 (0.210) 0.037 (0.110) 0.967 (0.051)
-
Chairs 1
Chairs 2
-
-
-
-
-
-
-
Arboreal 0.956 (0.070)
-
-
0.924 (0.075)
-
-
Chairs 1
-
-
-
-
0.975 (0.048) 0.598 (0.117)
Chairs 2
-
-
-
-
0.953 (0.065) 0.578 (0.106)
cluster around a particular location. A homing run is deemed successful if it ends within 30cm of the snapshot location. The results of this experiment are given in Table 3.6. When IS and IC are taken from the same data set, difference surface homing works quite well in almost all cases. Difference surface homing is much less successful when illumination conditions change between capture of IS and IC . Results are mixed when landmarks change positions
Low Level Approaches to Cognitive Control
151
between capture of IS and IC . There seems in fact to be a general diminution of average return ratio when IC is drawn from “Chairs2” – even when the snapshot image is also drawn from “Chairs2”. We speculate that the presence of the large objects in the experimental area, rather than movement of imaged objects between capture of snapshot and current images, causes difference surface homing to perform less well. We shall try to justify this speculation below. Analysis of RMS We would like to know why homing on RMS difference surfaces is affected by visual dynamism as described above. As given in Equation 3.14, RMS is difficult to analyse. We therefore break the equation into several terms as follows: MSD(IS , IC ) = =
1 N ∑ (IS (i) − IC (i))2 N i=1
(3.16)
1 N 2 N 1 N 2 2 [I (i)] + [I (i)] − ∑ S ∑ C ∑ IS (i)IC (i) N i=1 N i=1 N i=1
(3.17)
Note first that the square root in Equation 3.14 has been removed from Equation 3.16, transforming the RMS into an expression of mean squared differences (MSD); we have done this because the square root plays no significant role in the behaviour of RMS and slightly muddies our mathematical analysis. Equation 3.17 contains three terms. These terms can be transformed into forms more amenable to analysis. Several standard textbooks on statistics (see e.g. [150]) tell us that 1 N ∑ [IS (i)]2 ≈ Var(IS ) + (I¯S)2 N i=1
(3.18)
1 N ∑ [IC (i)]2 ≈ Var(IC ) + (I¯C )2 N i=1
(3.19)
1 N ∑ IS (i)IC (i) ≈ Cov(IS , IC ) + I¯SI¯C N i=1
(3.20)
where Var(IS ) is the variance of the intensities in IS , I¯S is the mean intensity in IS ; Var(IC ) and I¯C are defined similarly. Cov(IS , IC ) is the pixelwise covariance between IS and IC . Since the number of pixels in IS and IC is large, the difference between the left and right hand sides of each of the three equations above is exceedingly small in practice. We substitute the right hand sides of Equation 3.18, Equation 3.19 and Equation 3.20 into Equation 3.17 and perform some algebraic manipulation. The MSD is transformed into MSD(IS , IC ) ≈ Var(IS ) + Var(IC ) − 2Cov(IS, IC ) + (I¯S − I¯C )2 (3.21)
152
B. Webb et al.
450
400
350
y(cm)
300
250
200
150
100
50
50
100
150 x(cm)
200
250
Fig. 3.46. Homing paths starting from all non-goal positions for a difference surface with snapshot location at x=90cm, y=30cm. The snapshot image was drawn from the “Original” set and all current images were drawn from the “Winlit” set. No homing runs reached the goal location.
From Equation 3.21 it becomes clear that in moving our homing agent so as to minimise the RMS between IC and IS , the agent is actually simultaneously • seeking high covariance between IS and IC (i.e. minimising −2Cov(IS, IC )); • seeking low variance current images (i.e. minimising Var(IC )); and • seeking equality of the mean intensities of IS and IC (i.e. minimising (I¯S − I¯C )2 ). The second and third items above can cause homing errors. We see the equalisation of mean intensities playing a deleterious role when for example the snapshot image is taken from the “Original” data set and all current images are drawn from the “Winlit” data set. Fig. 3.46 shows the homing paths (starting from all non-goal positions) for this data set pairing when the goal location was set at x = 90cm, y = 30cm. None of the runs manages to reach the goal position, even those which begin quite close to the goal. Instead the agent is directed towards the window, which is the part of the arena in the “Winlit” set whose average light intensity is most similar to that of the snapshot, taken from the “Original” set. Equation 3.21 also implies that the homing agent will be attracted to areas whose corresponding images have relatively low variance compared with images of nearby areas. Large nearby objects (such as chairs) in the image tend to decrease the variance, so this could account for the observed problems in homing with the “Chairs2” data set. It seems clear that assessing the similarity between IS and IC with covariance rather than RMS is a more sensible approach. Exploring the Covariance Image Similarity Measure We carried out the same set of experiments on homing as described above using the RMS, except that here we used covariance to measure the similarity between IS and IC where covariance (COV ) is defined as:
Low Level Approaches to Cognitive Control
153
N
COV (IS , IC ) = ∑ (IS (i) · IC (i)) − I¯S · I¯C
(3.22)
i=1
IS , IC , I¯S and I¯C are defined as above.The results are given in Table 3.7, and the statistical significance of the difference between these data and the results using RMS shown in Table 3.6, using McNemar’s test [138], are given in Table 3.8. Taken together, Tables 3.6, 3.7 and 3.8 tell us that the COV is sometimes a better image similarity measure than RMS in static conditions – in particular those conditions with non-uniform overhead lighting. COV always outperforms RMS when illumination conditions change between captures of IS and IC . We note, though, that COV results are quite poor in the face of relatively extreme illumination change, when IS is drawn from “Doorlit” and IC is drawn from “Winlit” or vice-versa. COV sometimes outperforms RMS when objects move between capture of IS and IC – namely, the (“Arboreal”,”Original”) and (“Chairs2”, “Chairs1”) data set pairings. The average return ratios for the two similarity measures are statistically indistinguishable when large objects are placed within the experimental area during capture of current images (i.e. when current images are drawn from the “Chairs2” or “Arboreal” sets). This last point is somewhat surprising. Given our above analysis of RMS, we expected Table 3.7. Average return ratios for homing experiments carried out on COV difference surfaces in static and dynamic environments. The standard deviation of the average return ratio for each data set pairing is given in brackets. Current image data set Original
Winlit
Doorlit
Arboreal
Chairs 1
Snapshot set
Original 0.974 (0.061) 0.729 (0.227) 0.805 (0.321) 0.939 (0.084) Winlit
0.915 (0.211) 0.998 (0.007) 0.065 (0.201)
-
Doorlit 0.668 (0.283) 0.076 (0.218) 0.994 (0.015)
Chairs 2
-
-
-
-
-
-
-
Arboreal 0.961 (0.069)
-
-
0.930 (0.083)
-
-
Chairs 1
-
-
-
-
0.978 (0.043) 0.600 (0.097)
Chairs 2
-
-
-
-
0.965 (0.069) 0.582 (0.124)
Table 3.8. This table indicates whether the average return ratio given in Table 3.7 (COV results) is significantly different than the average return ratio given in Table 3.6 (RMS results) for a given data set pairing. A ’Y’ indicates a statistically significant difference for a particular data set pairing; an ’N’ indicates that there is not enough experimental evidence to reject the hypothesis that the average return ratios are equal. McNemar’s test was used with a 5% level of significance. See text for details. Current image data set
Snapshot set
Original Winlit Doorlit Arboreal Chairs 1 Chairs 2 Original
N
Y
Y
N
-
-
Winlit
Y
Y
Y
-
-
-
Doorlit
Y
Y
Y
-
-
Arboreal
Y
-
-
N
-
-
Chairs 1
-
-
-
-
N
N
Chairs 2
-
-
-
-
Y
N
154
B. Webb et al. goal distance=0 cm 250
IS pixel intensity
200
150
100
50
0
0
50
100
150 I pixel intensity
200
250
300
C
Fig. 3.47. This figure depicts a scatterplot in which we plotted the intensity of each pixel in IC against the intensity of the corresponding pixel in IS . Both images were captured at the same location (x=60cm, y=270cm) but IS was taken from the “Winlit” data set and IC from the “Original” set.
difference surface homing with the COV measure to be more successful than homing with RMS in environments with large objects in the experimental arena. We are not as yet certain why this improvement fails to occur. Exploring the Mutual Information Image Similarity Measure The covariance is only a trustworthy measure of the similarity between IS given IC when there is a linear relationship between pixel intensities in IS and IC . Such a linear relationship between IS and IC does exist in static conditions [151]. There ceases to be a linear relationship between pixel intensities in IS and IC when the two images are drawn from different data sets (i.e. in dynamic conditions). However, judging from the relationship between IS and IC in Fig. 3.47, IS is quite predictable given IC . For example, if we are told that a pixel in IC has value 200, then we can predict with high probability that the corresponding pixel in IS has an intensity close to either 60 or 175. This predictability can be measured with mutual image information. Visual homing is quite similar to the problem of image registration. An image registration algorithm attempts to find the function which best transforms one image of an object or scene into a second image of the same scene or object. Hill et al. [55] give a comprehensive review of image registration algorithms used in medical imaging applications. The paper demonstrates that image registration solutions are quite similar to many visual homing algorithms. Early registration work tried to find landmarks in the images to be aligned and used the change in pose of these landmarks to infer the overall image transformation. More recent work in image registration has attempted to align entire images, eschewing landmark selection and correspondence issues. Similar to our work, these image registration algorithms search for the image transformation which maximises the similarity between one image and a second transformed image of the same scene. A focus of this work has been on the similarity measure used to compare images. The use of mutual information in image registration was first reported by Viola and Wells [157] and apparently independently discovered by Maes et al. [98].
Low Level Approaches to Cognitive Control
155
Mutual image information (MI) can be defined as MI(IS , IC ) = H(IS ) − H(IS |IC )
(3.23)
where H(IS ) is the entropy of IS and H(IS |IC ) is the conditional entropy of IS given IC . This definition of mutual information is adapted from the one given in [55]. Entropy and conditional entropy are themselves defined as follows: G−1
H(IS ) = − ∑ pS (a)log2 (pS (a))
(3.24)
a=0
G−1 G−1
H(IS |IC ) = − ∑
∑ pSC (a, b)log2 (pS|C (a|b)
(3.25)
a=0 b=0
In Equation 3.24 pS (a) is the probability that a pixel will have intensity a (0 ≤ a < G) in image IS . In this work, pS (a) is calculated from the normalised intensity histogram of IS . Image entropy is highest when all possible pixel values are equally likely (i.e. the pixel intensity histogram has a uniform distribution) and lowest (zero) when one pixel value is certain and the others never occur. The joint probability pSC (a, b) in Equation 3.25 is the probability that a given pixel in IS has intensity a and the same pixel in IC has value b; pSC (a, b) is calculated from the normalised joint intensity histogram of IS and IC . Finally, the conditional probability pS|C (a|b) is the probability that a pixel will have intensity a in IS given that the corresponding pixel in IC has intensity b. It is clear from Equation 3.23 that maximising the mutual information between IS and IC involves minimising the conditional entropy H(IS |IC ); H(IS ) is constant while homing. Analysis of Equation 3.25 tells us that H(IS |IC ) is minimal (zero) if knowing that a pixel in IC has intensity b allows us to predict with probability 1 that the corresponding pixel in IS has intensity a for all a and b. Conditional entropy will be much higher if intensity values in IC are poor predictors of corresponding pixel intensities in IS . Hence, mutual image information is a measure of how predictable IS is given IC . We use the following equation to compute MI in our experiments. This form is equivalent to Equation 3.23 ([55]) but is slightly less computationally intensive. G−1 G−1
MI(IS , IC ) =
pSC (a, b)
∑ ∑ pSC (a, b)log2( pS (a)pC (b) )
(3.26)
a=0 b=0
We repeated the experiments using MI as the image similarity measure rather than RMS or COV . The results of these experiments are given in Table 3.9. Table 3.10 makes clear that the average return ratios of COV and MI are statistically different for all but one data set pairing, when IS is taken from “Winlit” and all current images are taken from “Original.”. The comparison between covariance and mutual information is somewhat ambiguous: mutual information does dramatically better than covariance in most cases where illumination changes between IC and IS . Mutual information does slightly less well than covariance in static environments and in environments in which the agent passes near large, monochromatic objects. We note that RMS (or something very similar) is quite often used to measure the difference between images in other image-based navigation schemes (e.g. the image warping
156
B. Webb et al.
Table 3.9. Average return ratios for homing experiments carried out on MI difference surfaces in static and dynamic environments. The standard deviation of the average return ratio for each data set pairing is given in brackets. Current image data set Original
Winlit
Doorlit
Arboreal
Chairs 1
Snapshot set
Original 0.905 (0.097) 0.602 (0.238) 0.892(0.208) 0.865 (0.098) Winlit
0.914 (0.115) 0.979 (0.072) 0.307 (0.305)
-
Doorlit 0.790 (0.119) 0.274 (0.297) 0.987 (0.023)
Chairs 2
-
-
-
-
-
-
-
Arboreal 0.892 (0.097)
-
-
0.811 (0.133)
-
-
Chairs 1
-
-
-
-
0.832 (0.135) 0.557 (0.126)
Chairs 2
-
-
-
-
0.807 (0.148) 0.498 (0.152)
Table 3.10. This table indicates whether the average return ratio given in Table 3.9 (MI results) is significantly different than the average return ratio given in Table 3.7 (COV results) for a given data set pairing. A ’Y’ indicates a statistically significant difference for a particular data set pairing; an ’N’ indicates that there is not enough experimental evidence to reject the hypothesis that the average return ratios are equal. McNemar’s test was used with a 5% level of significance. See text for details. Current image data set
Snapshot set
Original Winlit Doorlit Arboreal Chairs 1 Chairs 2 Original
Y
Y
Y
Y
-
-
Winlit
N
Y
Y
-
-
-
Doorlit
Y
Y
Y
-
-
Arboreal
Y
-
-
Y
-
-
Chairs 1
-
-
-
-
Y
Y
Chairs 2
-
-
-
-
Y
Y
algorithm of Franz et al. [30] and image-based Monte Carlo localisation [101]). As in our work, these algorithms compare a current image with one or more images captured previously. Lighting and landmark locations might well have changed in the interim. We have demonstrated that mutual information is robust to this dynamism and so could provide a useful image similarity measure in image-based robot navigation in general.
3.4 Learning In chapter 1 we discussed evidence of the learning capabilities of insects, and the important role of the mushroom body (MB) a prominent region of multimodal integration in the insect brain. In this final section we describe the development of a computational model of learning in the MB, and and test the model’s performance for non-elemental and crossmodal associative learning tasks. We employ a realistic spiking neuron model and spike time dependent plasticity, and learning performance is investigated in closed-loop conditions. We show that the distinctive neuroarchitecture (divergence onto MB neurons and convergence from MB neurons, with an otherwise non-specific connectivity) is sufficient for solving non-elemental and crossmodal learning tasks and thus modulating underlying reflexes in context-dependent, heterarchical manner.
Low Level Approaches to Cognitive Control
157
The neural architecture for the agent is based on the insect brain [169], in particular, on evidence that the MB is involved in modulating more basic, reflexive behaviours ([114],[102]) and thus acts as a neural substrate for associations underlying contextspecific and non-elemental forms of learning. However, the goal of the implemented model design was not to imitate physiological mechanisms involved in MB-mediated learning as closely as possible, but rather to find an abstract description of the underlying principles, able to reproduce associative learning in closed-loop conditions. Yet, we aim to use realistic models of the biological components as more realistic models can be quantitatively and qualitatively different from more abstract connectionist approaches. Detailed discussion of insect brain architecture is provided in chapter 1. The main idea is that of parallel pathways, with sensory inputs forming direct reflex loops, but also feeding into secondary routes that are used to place information from various sensory modalities or other domain-specific sensorimotor loops into context. The system can thus improve on reflexive behaviours by learning to adapt and anticipate reflex-causing stimuli. This adaptation process is assumed to occur in the MB, which form such a parallel pathway for sensory inputs in the insect brain. 3.4.1
Neural Model and STDP
We chose the neuron model proposed by Izhikevich [67] since it exhibits biologically plausible dynamics, similar to Hodgkin-Huxley-type neurons, but is computationally less expensive and thus, suitable for large-scale simulation: C
dv = k(v − vr )(v − vt ) − u + I + [ξ ∼ N(0, σ )] dt
(3.27)
du = a(b(v − vr ) − u), (3.28) dt where v is the membrane potential and u is the recovery current. a = 0.3, b = −0.2, c = −65, d = 8, and k = 2 are model parameters. C = 100 (pF) is the capacitance, vr = −60mV is the resting potential, and vt = −40mV is the instantaneous threshold potential. ξ is a Gaussian noise term with standard deviation σ . The variables v and u are reset if v ≥ +35mV: v←c . (3.29) u ← u+d Synaptic inputs are modelled by: I(t + ∆ t) = gS(t)(vrev − v(t)),
(3.30)
where vrev is the reversal potential of the synapse (vrev = 0mV for excitatory and vrev = −90mV for inhibitory synapses) and g is the maximal synaptic conductance. S(t) is the amount of neurotransmitter active at the synapse at time t and is updated as follows: −∆ t S(t)e τsyn + δ , if presynaptic spike S(t + ∆ t) = , (3.31) −∆ t S(t)e τsyn , otherwise
158
B. Webb et al.
where δ = 0.5 is the amount of neurotransmitter released when a presynaptic spike occurred and τsyn is the synaptic timescale. The simulation timestep ∆ t is set to 0.25ms. Synapses are modified using Spike Time-Dependent Plasticity (STDP) which has been observed in biological neural systems (e.g., [19]). In STDP, synaptic change depends on the relative timing of pre- and post-synaptic action potentials. Synaptic conductances are adapted as follows: ⎧ t −t ⎨ A e preτ+post − gmax , if t − t < 0 + pre post r ∆g = , (3.32) −(tpre −tpost ) ⎩ τ− , if tpre − tpost ≥ 0 A− e where tpre and tpost are the spiking times of the pre- and postsynaptic neuron respectively. A+ , A− , τ+ ms, and τ− ms are parameters. We modified the STDP rule proposed 3 by [137] by adding an additional term − gmax r if tpre −tpost < 0 where r = 10 is a parameter. This means that if postsynaptic spikes are not matched with presynaptic ones, the synaptic conductance between them is decreased by this term. If this modification rule of synaptic conductances g pushes the values out of the allowed range 0 ≤ g ≤ gmax , g is set to the appropriate limiting value (gmax ). A ‘forgetting’ factor is introduced in the form of a slow decay of g: −∆ t
g(t + ∆ t) = g(t)e τdecay
(3.33)
where τdecay = 105 . 3.4.2
Non-elemental Associations
Thus far, computational models of MB function have been restricted to classification of sensory inputs in open-loop conditions ([113],[112]). Here we develop a MB model that modulates reflexive sensorimotor loops through non-elemental associative learning [37], that is, forms of learning that go beyond simple associations between two stimuli (classical conditioning) or between a stimulus and a response (instrumental conditioning). In non-elemental learning tasks, the stimuli are ambiguously associated with reward or punishment; each stimulus is followed as often by appetitive (+) as aversive (–) reinforcement so that learning requires the context of the stimulus to be taken into account. In negative patterning, the agent has to learn to approach (appetitive action) the single stimuli A and B but retreat (aversive action) from the compound AB. In biconditional discrimination, the agent has to learn to respond appetitively to the compounds AB and CD but aversively to the compounds AC and BD. In feature neutral discrimination, the agent has to learn to respond appetively to B and AC but aversively to C and the compound AB. In our simulation experiments, we take ‘reinforcement’ and ‘punishment’ to be sensory cues causing different reflex responses (appetitive or aversive); in successful learning, these responses become associated with the appropriate conditioned stimuli. We propose a minimalist architecture able to modulate reflex behaviours in closedloop conditions (where the system’s output influences the system’s inputs) for nonelemental learning tasks. In this section, we show that the general neuroarchitecture
Low Level Approaches to Cognitive Control
159
Table 3.11. Stimuli-reward combinations in the non-elemental learning tasks Non-elemental learning task Negative patterning Biconditional discrimination Feature neutral discrimination
Stimuli-reward combinations A+ B+ AB– AB+ CD+ AC– BD– AC+ C– AB– B+
AB+
AC+
A+
CD+
B+
B+
AC-
AB-
AB-
BD-
C-
(a)
(b)
(c)
Fig. 3.48. The wallpapers used for the non-elemental learning tasks: (a) negative patterning, (b) biconditional discrimination, and (c) feature neutral discrimination. The agent’s field of view is represented by the 4-by-4 grid. As the field of view is gradually moved to the left, the visual patterns predict what the agent will experience (+ or –) when it reaches the left edge. Refer to text for further explanations.
of the MB (fan-out and fan-in) is sufficient for explaining the above forms of nonelemental learning. Our experimental set-up was inspired by conditioning paradigms for visual pattern avoidance in flies, in which the animal in a flight simulator learns an appropriate yaw response to a particular visual pattern which is associated with an unpleasant heat beam. In our simulation, the agent has a limited field of view (45 degrees) on a ‘wallpaper’ (of 360 degrees total width) that displays different patterns. In the absence of any action by the agent, the field of view is moved gradually to the left, at 1.5 degrees per millisecond. If it reaches the left edge the agent is “punished” - this generates a reflex action, which moves the field of view back 180 degrees to the right. Before it reaches the edge it will encounter a visual pattern, which can thus be used to predict that the edge will be encountered. The aim is to learn to associate the reflex action with the visual pattern and execute it before encountering the edge, thus avoiding punishment. This anticipatory or conditioned reflex, if executed, will move the field of view 21 degrees to the right. In the non-elemental learning tasks, there are two reflexes, X0-V0 and X1-V1 (these could be called ‘appetitive’ and ‘aversive’, but in fact they have the same effective result of turning the agent back to 180 degrees). There are two corresponding modes for the simulator, i.e. when the field of view reaches the left edge, the agent experiences either X0 or X1, and will execute the corresponding reflex, V0 or V1. Which experience will occur is predicted by the visual pattern, according to the schemes illustrated in Fig. 3.48; for example, in negative patterning, the patterns A and B predict X0(+), and the pattern AB predicts X1(–). The agent must learn to execute the correct reflex (V0 or V1) when it sees a particular visual pattern, which will move it away from the edge. If it executes the wrong reflex, then it is instead moved further towards the edge (i.e. 21 degrees to the left).
160
B. Webb et al. Secondary pathway for sensory inflow via Mushroom Body LHI=16 Excitatory Inhibitory
KC=120
PN=16
EN=2
Reflex pathways
Fig. 3.49. The implemented MB network receives sensory cues from the visual field via projection neurons (PN), which make direct excitatory connections, and indirect inhibitory connections (via the lateral horn interneurons (LHI)) to the Kenyon cells (KC). The MB output converges on a small number of extrinsic neurons (EN), which are also excited by the underlying direct reflex pathways, and can activate these pathways. Learning occurs between the KC and EN, allowing anticipation of the reflex responses due to associations with particular visual patterns.
The field of view contains 45-by-45 pixels, which are mapped onto a 4-by-4 set of sensory neurons (see network description below). White areas on the wallpaper excite these neurons, thus stimulus A will excite the first row of neurons, B the second row and so on. A typical simulation run lasts 50 seconds, during which the wallpapers are switched every 0.5 seconds. The simulation timestep is 0.25ms. The mushroom bodies in insects have a characteristic neuroarchitecture: namely a tightly-packed, parallel organisation of thousands of neurons, the Kenyon cells. The mushroom bodies are further subdivided into several distinct regions: the calyces (input), the pendunculus, and the lobes (output). The dendrites (inputs) of the Kenyon cells have extensive branches in the calyces, and the axons (outputs) of the Kenyon cells run through the pendunculus before extending to form the lobes. Synaptic interconnections between Kenyon cell axons have been reported [47]. Note that there is considerable divergence (1:50) from a small number of sensory projection neurons (PN) onto the large number of Kenyon cells (KC), and considerable convergence (100:1) from the Kenyon cells onto extrinsic output neurons (EN) (these ratios are estimates based on data from [94]). KC receive direct excitatory input from PN neurons, but also indirect inhibitory inputs from the same neurons via lateral horn interneurons (LHI), arriving shortly after the excitation. These connections are illustrated in Fig. 3.49. It is hypothesised that the MB help disentangle spatio-temporal input patterns by operating as coincidence detectors selective to particular correlations in the input spike trains [115]. The mapping of sensory neurons onto MB neurons shows high divergence which can serve for the recognition of unique relationships in primary sensory channels.
Low Level Approaches to Cognitive Control
161
In our model, nonlinear transformation, separating the activity patterns in the PN layer into sparse activity patterns in the KC layer, is implemented by a randomly determined connectivity matrix between these layers. EN linearly classify the KC activity patterns. Plasticity of KC-EN synapses is achieved with a spike timing dependent plasticity rule. The EN output mediates conditioned responses by activating the appropriate reflex responses. The inhibition from the LHI, quickly following excitation from the PN, limit integration time for the KC to short time windows, making them highly sensitive to precise temporal correlations. 3.4.2.1 Network Geometry The network geometry as shown in Fig. 3.49 retains proportional dimensions to the MB system in insects but is smaller in size. A strategy based entirely on random connectivity and self-organisation through local learning and competition is explored. Each neuron pair X-X is connected with probability pX,X . The system implements non-specific connectivity with the exception of full inhibitory connectivity between EN (c.f., [37]). We describe the various network layers, their parameters, and their roles below. Learning occurs only through modulation of the KC-EN connections. We report in Sect. 3.4.2.2 the effects on learning performance of changing connectivity between the LHI and KC layers, and varying the size of the KC layer. PN layer. This layer receives sensory input. The layer consists of 16 neurons (the agent’s 45-by-45-pixels FoV is divided into a 4-by-4 grid). The input to a single PN neuron is calculated as follows: IPN =
sum of pixel values number of pixels
255
.
(3.34)
The neurotransmitter released at each timestep is calculated as follows: S(t + ∆ t) = S(t) + IPN × δ .
(3.35)
Black areas have a pixel value of 0 whereas white areas have pixel values of 255, thus only white areas in the FoV excite the network. KC layer. The KC layer consists of 16C2 = 120 neurons. Each KC will act as a coincidence detector and receive inputs from a small number of PNs (pPN,KC = 0.1). The synaptic timescale τPN,KC is set to 2 ms. This parameter needs to be small in order to make the KC neurons very sensitive to the relative timing of incoming input from the PN layer. This setup allows the KC neurons to act as coincidence detectors. The synaptic strength of PN-KC synapses needed to be carefully adjusted (gPN,KC are initialised uniformly at random in [20,30]). In addition to this we add an uniformly distributed jitter to the synaptic strengths. We implemented excitatory and inhibitory KC-KC connections (pKC,KC = 0.1, τKC,KC = 5ms) with equal probability (gKC,KC are initialised uniformly at random in [5,10]). LHI layer. Feed-forward inhibition by lateral horn interneurons (LHI) dampens KC activity in the MB. Thus, the integration time for the KC neurons is limited to short
162
B. Webb et al.
(a)
(b)
Fig. 3.50. (a) Agent behaviour in response to first (0ms), second (2000ms) and final (18000ms) presentations of the same wallpaper during a 20s simulation run. At the start it reaches the edge position (lower dotted line) and performs a reflex turn. In the next presentation it responds to the visual stimulus (between the upper dotted lines) but the response is sometimes incorrect. By the final presentation it reliably responds to the visual stimulus and thus successfully avoids the edge. (b) Boxplots of performance for different learning tasks (1) negative patterning, (2) biconditional discrimination, and (3) feature neutral discrimination. The agent encounters a median of 5 punishments before successfully using the visual patterns to anticipate and avoid it. The simulation runs lasted 50s.
time windows, making them highly sensitive to precise temporal correlations. This was implemented through 16 LHIs receiving their inputs from the PN layer and inhibiting activity in the KC layer (pPN,LHI = 0.2, τPN,LHI = 5ms, gPN,LHI are initialised uniformaly at random in [20,30], pLHI,KC = 0.1, τLHI,KC = 5ms, gLHI,KC are initialised uniformly at random in [20,30]). EN layer. Every KC-EN pair is connected (pKC,EN = 1 and τKC,EN = 5ms). However, the synaptic conductance gKC,EN for all synapses is initialised to 0, and is subsequently modified by STDP as described below. The ENs also receive excitatory input from the underlying reflex pathways, thus the learning reflects the coincidence of activity in these pathways and particular patterns of KC activity. STDP parameter Value A− 1 A+ 2 τ− 5ms τ+ 50ms gmax 30 r 103 3.4.2.2 Non-elemental Discrimination Performance The system was able to learn each of the non-elemental associations shown in Fig. 3.48. As the system learns to respond to the visual patterns, the reflex responses to encountering the edge are executed less often and the MB drives the agent’s behaviour (as shown in Fig. 3.50(a)). The performance index used in this paper is simply the number of times
Low Level Approaches to Cognitive Control
163
35 25 30
25 Performance Index
Performance Index
20
15
10
20
15
10 5
5
0
0.1
0.2
0.3
0.4
0.5
PLHI,KC
(a)
0
10
40
70 100 Size of KC layer
130
160
(b)
Fig. 3.51. Boxplots of (a) performance with varying probability of connectivity pLHI,KC between the LHI-KC layers. Performance with varying KC layer size (10,40,70,100,130, and 160 neurons) for a probability of connectivity pLHI,KC = 0.0
the reflexes are executed. As shown in Fig. 3.50(a), the naive system will repond with one reflex action during one presentation of one wallpaper. In the biconditional discrimination setup, for example, there are Nw = 4 wallpapers which are interchanged every tc = 0.5s during tT = 50s. Thus, tT /(Nw × tc ) = 25 activations would mean that a run was unsuccessful. Fig. 3.50(b) shows boxplots of the number of times reflex pathways were active over 30 simulation runs (each lasting 50 seconds) for each of the three conditioning paradigms. All simulation runs for negative patterning were successful and only one simulation run for biconditional and feature neutral discrimination each was unsuccessful. The agent learnt after a median of 5 activations of the reflex pathways which reflex to use, in response to which visual patterns, to successfully avoid the edge. Fig. 3.51(a) shows the learning performance with varying probability of connectivity pLHI,KC . As the connectivity between LHI-KC neurons increases (and with it the inhibition from this layer), the learning performance becomes maximal at pLHI,KC = 0.1. As the inhibition increases further, the performance drops off. With increasing inhibition by the LHI neurons, the activity in the KC layer becomes sparser. Fig. 3.51(b) shows the network performance with varying KC-layer size with probability of connectivity pLHI,KC = 0.0. The performance tends to improve with increasing KC layer size. 3.4.3
Associating Auditory and Visual Cues
The MB model is tested as part of an insect brain inspired architecture within a closed loop behavioural task replicating in simulation an experiment carried out on bushcrickets. We show the system can successfully associate visual to auditory cues, so as to maintain a steady heading towards an intermittent sound source. Male bushcrickets are able to maintain a straight course to a female, by coupling visual cues to an acoustically detected direction [48]. Stabilising effects of visual information on course maintenance are found in other insects (c.f., [5]), but in this case it was also shown that optical cues could stand in for (temporarily absent) auditory signals. In particular, the animal could quickly learn to walk at an arbitrary angle to a visual
164
B. Webb et al. Phonotaxis
Visual input
PN
KC
EN
Left
Right
Excitatory Inhibitory
Motor output
Fig. 3.52. The implemented MB network receives sensory cues from the visual field via projection neurons (PN), which make direct excitatory connections to the Kenyon cells (KC). The MB output converges on a small number of extrinsic neurons (EN), which are also excited by the underlying direct reflex pathways, and can activate these pathways. Learning occurs between the KC and EN.
landmark, corresponding to the sound direction. In the absence of sound it would follow the displacement of the landmark with an appropriate change in walking direction. In a comparable task, the MB of the cockroach have been shown to play a role in place memory relating distant visual cues to an invisible target [108]. Here we show how the neural architecture of the MB can account for such capabilities, using a biologically plausible neural representation and learning rule. 3.4.3.1 Neural Architecture In the current experiment, the reflex pathway represents a response to sound (see Fig. 3.52). Each spike of the output (left or right) neurons of the phonotaxis (sound localising) circuit (based on [161]) cause the agent to turn by 1 degree in the direction of the sound source and also excite the extrinsic neurons (EN). The visual position of the landmark is mapped onto projection neurons (PN) that activate the Kenyon cells (KC) that form the mushroom body. These converge on the extrinsic neurons. During conditioning, the sound is on and the agent moves towards it. After conditioning, the agent should have associated the required movements with a particular landmark direction, and thus be able to control its course using only visual cues. PN layer. This layer consists of 72 neurons giving a visual resolution of 360 72 = 5 degrees. We assumed preprocessing of visual information and that receptive fields are not overlapping. Each PN encodes a particular relative angle towards the landmark. The neurotransmitter release (at a PN when the landmark is in view at its particular relative angle) is calculated as follows: S(t + ∆ t) = S(t) + δ .
Low Level Approaches to Cognitive Control (a)
(b)
.....
..... 71
72
PN 1 2 3
4
PN 1
(c)
165
PN 1
PN 2
PN 2
PN 3
PN 3
KC 1
KC 1
KC 2
KC 2
KC 3
KC 3
EN 1
EN 1
EN 2
EN 2
LEFT
LEFT
RIGHT
RIGHT
5
Fig. 3.53. Schematic drawing of neural activity. As the agent zig-zags towards the sound source (a), turning is controlled by the output (left, right) of the phonotaxis network (b) activating the extrinsic neurons (EN). During these movements, the visual stimulus activates several different projection neurons (PN) which activate their respective Kenyon cells (KC). The system learns to associate KC and EN firing, so that after conditioning (c) the visual stimulus alone is able to control turning.
KC layer. The KC layer consists of 72 neurons. The topographical organisation of the PN layer is maintained, i.e. each KC only receives input from one PN. The synaptic strength of PN-KC synapses (gPN,KC ) were set at random in the interval [20,30]. EN layer. The EN layer contains 2 neurons and each EN is connected to all KC, i.e., every KC-EN pair is connected (τKC,EN = 5ms). Learning occurs only through modulation of the KC-EN connections. However, the synaptic conductance gKC,EN for all synapses is initialised to 0, and is subsequently modified as described below. The EN neurons also receive excitatory input from the underlying reflex pathways, thus the learning reflects the coincidence of activity in these pathways and the activity in the KC layer. STDP parameter Value A− -20 A+ 20 τ− 5ms τ+ 10ms gmax 50 r 103 3.4.3.2 Auditory Visual Association Performance The model was tested in two scenarios. In one, the agent walks on a treadmill so that it can only change its orientation - this is directly comparable to the original behavioural experiments on the bushcricket. In the other, the agent moves in an arena towards the sound source, which is more like the natural interaction of the insect with the environmental cues. In each case the conditioning trials (with constant sound) lasted a total of 120s, with the agent starting at a random heading. The conditioning trials were repeated and reset as follows. In the ‘treadmill’ scenario the heading was randomly reset every 10s, whereas in the ‘arena’ scenario if it arrived within a small radius of the sound
166
B. Webb et al.
180 160
200
heading angle (in degrees)
heading angle (in degrees)
140 120 100 80 60 40
100
50
20
conditiong angle θ=90 θ=135
conditioning angle θ=45
0 −20 −20
150
θ=45 0
0
20
40 60 80 100 120 140 landmark displacement (in degrees)
160
180
−20
0
20
40 60 80 100 120 140 landmark displacement (in degrees)
160
180
Fig. 3.54. Mean heading angle plotted versus landmark displacement. (a) The agent can turn but does not move forward, simulating an insect on a treadmill. (b) The agent can move through the environment. In each case the agent maintains the appropriate relative heading as the landmark is displaced.
source it was returned to its starting position (with a random heading) and repeated its approach. In the ‘arena’ scenario, coupling visual cues to phonotactic behaviour has a stabilising effect on course maintenance. As a measure we used the Vector Length (VL) defined as the quotient of the distance from start position to the position of the source and the actual path length. The mean VL was 0.59 (s.d.=0.1) before conditioning and 0.818 (s.d.=0.182) after conditioning. More importantly, the landmark could now be used to stand in for the sound. This was tested by looking at the mean angle that the agent moved with sound off when the landmark was displaced. As shown in Fig. 3.54, the agent’s direction is appropriate to the conditioning angle and follows the landmark displacement consistently, for both the ‘treadmill’ scenario (Fig 3.54a, closely comparable to the biological data seen in Fig. 9 in [48]) and the more naturalistic ‘arena’ scenario (figure 3.54b). The system has associated each possible visual position of the landmark with the movement required to re-orient so that the landmark falls into the visual position it occupied during walking towards the sound.
3.5 Conclusion Animal behaviour is a continuous closed loop, with sensory events transformed by the agent into motor actions, and these actions transformed by the environment into new sensory events. Neural learning mechanisms should thus be evaluated in a closed loop context if they are to be considered biologically relevant. A fundamental role of learning for behaving animals is associating the reflex response to one cue with another cue that can refine, predict or substitute for the original cue. We have replicated this capability in several different scenarios, using a plausible model of insect brain circuitry. We are now working towards evaluating this model by incorporating the more realistic models of the sensory inputs (e.g. audition) and motor outputs (e.g. six-legged walking) that were described earlier in this chapter. We aim to demonstrate the same learning mech-
Low Level Approaches to Cognitive Control
167
anisms operating on real robots, and assess their suitability for efficient hardware implemention, e.g. using FPGAs. The same neural architecture will also be developed for application to place memory (associating landmarks to a desired location) a behaviour known to be dependent on the Mushroom Body in insects [108].
References 1. Amrein, H.: Pheromone perception and behavior in Drosophila. Current Opinion in Neurobiology 14, 435–442 (2004) 2. Arbas, E., Willis, M., Kanzaki, R.: Organization of goal-oriented locomotion: pheromonemodulated flight behavior of moths. In: Beer, R., Ritzmann, R., McKenna, I. (eds.) Biological neural networks in invertebrate neuroethology and robotics. Academic Press, Cambridge (1993) 3. Benhamou, S.: Path integration by swimming rats. Animal Behavior 54, 321–327 (1997) 4. Bisch-Knaden, S., Wehner, R.: Local vectors in desert ants: context-dependent landmark learning during outbound and homebound runs. Journal of Comparative Physiology 189, 181–187 (2003) 5. B¨ohm, H., Schildberger, K., Huber, F.: Visual and acoustic course control in the cricket Gryllus-bimaculatus. Journal of Experimental Biology 159, 235–248 (1991) 6. Borst, A., Haag, J.: Neural networks in the cockpit of the fly. Journal of Comparative Physiology A 188, 419–437 (2002) 7. Burdohan, J., Comer, C.: Cellular organisation of an antennal mechanosensory pathway in the cockroach, Periplaneta americana. Journal of Neuroscience 16, 5830–5843 (1996) 8. Burgess, N., Donnett, J., O’Keefe, J.: Using a mobile robot to test a model of the rat hippocampus. Connection Science 10, 291–300 (1998) 9. Bush, S., Schul, J.: Pulse-rate recognition in an insect: evidence of a role for oscillatory neurons. Journal of Comparative Physiology 192, 113–121 (2006) 10. Camhi, J., Johnson, E.: High-frequency steering maneuvres mediated by tactile cues: antennal wall-following in the cockroach. Journal of Experimental Biology 202, 631–643 (1999) 11. Cartwright, B., Collett, T.: Landmark learning in bees. Journal of Comparative Physiology A 151, 521–543 (1983) 12. Chapman, T.: Morphological and neural modelling of the orthopteran escape response. Ph.D. thesis, University of Stirling (2001) 13. Chapman, T., Webb, B.: A model of antennal wall-following and escape in the cockroach. Journal of Comparative Physiology A 192, 949–969 (2006) 14. Collett, M., Collett, T., Srinivasan, M.: Insect navigation: Measuring travel distance across ground and through air. Current Biology 16, R887–R890 (2006) 15. Collett, T., Collett, M.: Path integration in insects. Current Opinion in Neurobiology 10, 757–762 (2000) 16. Comer, C., Robertson, R.: Identified nerve cells and insect behavior. Progress in Neurobiology 63, 409–439 (2001) 17. Cruse, H., Kindermann, T., Schumm, M., Dean, J., Schmitz, J.: Walknet-a biologically inspired network to control six-legged walking. Neural networks 11, 1435–1447 (1998) 18. Cruse, H., Schmitz, J., Braun, U., Schweins, A.: Control of body height in a stick insect walking on a treadwheel. The Journal of Experimental Biology 181, 141–155 (1993) 19. Dan, Y., Poo, M.: Spike timing dependent plasticity of neural circuits. Neuron 44, 23–30 (2004) 20. Duerr, V., Krause, A., Schmitz, J., Cruse, H.: Neuroethological concepts and their transfer to walking machines. International Journal of Robotics Research 22, 151–167 (2003)
168
B. Webb et al.
21. D¨urr, V., Ebeling, W.: The behavioural transition from straight to curve walking: kinetics of leg movement parameters and the initiation of turning. The Journal of Experimental Biology 208, 2237–2252 (2005) 22. Egelhaaf, M., Borst, A.: A look into the cockpit of the fly: visual orientation, algorithms, and identified neurons. Journal of Neuroscience 13(11), 4563–4574 (1993) 23. Egorov, A., Hamam, B., Fransen, E., Hasselmo, M., Alonso, A.: Graded persistent activity in entorhinal cortex neurons. Nature 14, 133–134 (2002) 24. Elsner, N.: The search for neural centers of cricket and grasshopper song. In: Neural Basis of Behavioural Adaptations, Fortschritte der Zoologie, vol. 39, pp. 167–193. Gustav Fischer Verlag, Stuttgart (1994) 25. Erber, J., Kierzek, S., Sander, E., Grandy, K.: Tactile learning in the honeybee. Journal of Comparative Physiology A 183, 737–744 (1998) 26. Esch, H., Zhang, S., Srinivasan, M., Tautz, J.: Honeybee dances communicate distances measured by optic flow. Nature 411, 581–583 (2001) 27. Ferree, T., Lockery, S.: Computational rules for chemotaxis in the nematode c. elegans. Journal of Computational Neuroscience 6(3), 263–277 (1999) 28. Franceschini, N., Riehle, A., Nestour, A.L.: Directionally selective motion detection by insect neurons. In: Stavenga, D., Hardie, R. (eds.) Facets of Vision, pp. 360–390. Springer, Berlin (1989) 29. Franz, M., Mallot, H.: Biomimetic robot navigation. Robotics and Autonomous Systems 30, 133–153 (2000) 30. Franz, M., Schoelkopf, B., Mallot, H., Buelthoff, H.: Where did i take that snapshot? scenebased homing by image matching. Biological Cybernetics 79, 191–202 (1998) 31. Frye, M., Dickinson, M.: Closing the loop between neurobiology and flight behavior in Drosophila. Current Opinion in Neurobiology 14, 729–736 (2004) 32. Frye, M., Dickinson, M.: Motor output reflects the linear superposition of visual and olfactory inputs in Drosophila. Journal of Experimental Biology 207, 123–131 (2004) 33. Frye, M., Tarsitano, M., Dickinson, M.: Odor localization requires visual feedback during free flight in Drosophila melanogaster. Journal of Experimental Biology 206, 843–855 (2003) 34. Fullmer, B., Miikkulainen, R.: Using marker-based genetic encoding of neural networks to evolve finite-state behaviour. In: Toward a Practice of Autonomous Systems. Proceedings of the First European Conference on Artificial Life. MIT Press, Cambridge (1992) 35. Galizia, C.G., Menzel, R.: The role of glomeruli in the neural representation of odours: results from optical recording studies. Journal of Insect Physiology 47, 115–130 (2001) 36. Gao, Q., Yuan, B., Chess, A.: Convergent projections of Drosophila olfactory neurons to specific glomeruli in the antennal lobe. Nature Neuroscience 3(8), 780–785 (2000) 37. Giurfa, M.: Cognitive neuroethology: dissecting non-elemental learning in a honeybee brain. Current Opinion in Neuroethology 13, 726–735 (2003) 38. Giurfa, M., Menzel, R.: Insect visual perception: complex abilities of simple nervous systems. Current Opinion in Neurobiology 7, 505–513 (1997) 39. Goepfert, M., Robert, D.: The mechanical basis of Drosophila audition. Journal of Experimental Biology 205, 1199–1208 (2002) 40. Gollisch, T., Schutze, H., Benda, J., Herz, A.: Energy integration describes sound-intensity coding in an insect auditory system. Journal of Neuroscience 22(23), 10434–10448 (2002) 41. Haferlach, T., Wessnitzer, J., Mangan, M., Webb, B.: Evolving a neural model of insect path integration. Adaptive Behavior 15(3), 273–287 (2007) 42. Hartmann, G., Wehner, R.: The ant’s path integration system: a neural architecture. Biological Cybernetics 73, 483–497 (1995)
Low Level Approaches to Cognitive Control
169
43. Hassenstein, B., Reichardt, W.: Systemtheoretische Analyse der Zeit-, Reihenfolgen- und Vorzeichenauswertung bei der Bewegungsperzeption des Ruesselskaefers Chlorophanus. Zeitschrift der Naturforschung 11b, 513–524 (1956) 44. Hayes, A., Martinoli, A., Goodman, R.: Distributed odor source localization. IEEE Sensors Journal 2(3), 260–271 (2002) 45. Hedwig, B., Poulet, J.F.A.: Mechanisms underlying phonotactic steering in the cricket Gryllus bima culatus revealed with a fast trackball system. J. Exp. Biol. 208(5), 915–927 (2005) 46. Heinze, S., Homberg, U.: Maplike representation of celestial e-vector orientations in the brain of an insect. Science 315, 995–997 (2007) 47. Heisenberg, M.: What do the mushroom bodies do for the insect brain? an introduction. Learning and Memory 5, 1–10 (1998) 48. von Helversen, D., Wendler, G.: Coupling of visual to auditory cues during phonotactic approach in the phaneropterine bushcricket Poecilimon affinis. Journal of Comparative Physiology A 186, 729–736 (2000) 49. Hengstenberg, R.: Gaze control in the bowfly Calliphora: a multisensory, two-stage integration process. The Neurosciences 3, 19–29 (1991) 50. Hennig, R., Franz, A., Stumpner, A.: Processing of auditory information in insects. Microscopy Research and Technique 63(6), 351–374 (2004) 51. Higgins, C.: Nondirectional motion underlie insect behavioral dependence on image speed. Biological Cybernetics 91, 326–332 (2004) 52. Higgins, C., Douglass, J., Strausfeld, N.: The computational basis of an identified neuronal circuit for elementary motion detection in dipterous insects. Visual Neuroscience 21, 567– 586 (2004) 53. Higgins, C., Pant, V.: An elaborated model of fly small-target tracking. Biological Cybernetics 91, 417–428 (2004) 54. Hildebrand, J., Shepherd, G.: Mechanisms of olfactory discrimination: converging evidence for common principles across phyla. Annual Review of Neuroscience 20, 595–631 (1997) 55. Hill, D.L.G., Batchelor, P.G., Holden, M., Hawkes, D.J.: Medical image registration. Physics in Medicine and Biology 46, 1–45 (2001) 56. Homberg, U.: In the search of the sky compass in the insect brain. Naturwissenschaften 91, 199–208 (2004) 57. Homberg, U., Christensen, T., Hildebrand, J.: Structure and function of the deutocerebrum in insects. Annual Review of Entomology 34, 477–501 (1989) 58. Honegger, H.W.: A preliminary note on a new optomotor response in crickets: antennal tracking of moving targets. Journal of Comparative Physiology A 142(3), 419–421 (1981) 59. Horstmann, W., Egelhaaf, M., Warzecha, A.K.: Synaptic interactions increase optic flow specificity. European Journal of Neuroscience 12, 2157–2165 (2000) 60. Hoy, R., Nolen, T., Brodfuehrer, P.: The neuroethology of acoustic startle and escape in flying insects. Journal of Experimental Biology 146, 287–306 (1989) 61. Hoy, R., Robert, D.: Tympanal hearing in insects. Annual Review of Entomology 41, 433– 450 (1996) 62. Huerta, R., Nowotny, T., Garcia-Sanchez, M., Abarbanel, H., Rabinovich, M.: Learning classification in the olfactory system of insects. Neural Computation 16, 1601–1640 (2004) 63. Imaizumi, K., Pollack, G.: Central projections of auditory receptor neurons of crickets. Journal of Comparative Neurology 493, 439–447 (2005) 64. Ishida, H., Nakamoto, T., Moriizumi, T., Kikas, T., Janata, J.: Plume-tracking robots: a new application of chemical sensors. Biological Bulletin 200, 222–226 (2001) 65. Iwama, A., Shibuya, T.: Physiology and morphology of olfactory neurons associating with the protocerebral lobe of the honeybee brain. Journal of Insect Physiology 44, 1191–1204 (1998)
170
B. Webb et al.
66. Izhikevich, E.: Resonate-and-fire neurons. Neural Networks 14, 883–894 (2001) 67. Izhikevich, E.: Dynamical systems in neuroscience: the geometry of excitability and bursting. The MIT Press, Cambridge (in press, 2007) 68. Jacobs, G., Miller, J., Murphey, R.: Integrative mechanisms controlling directional sensitivity of an identified sensory interneuron. Journal of Neuroscience 6(8), 2298–2311 (1986) 69. Jacobs, G., Theunissen, F.: Extraction of sensory parameters from a neural map by primary sensory interneurons. Journal of Neuroscience 20(8), 2934–2943 (2000) 70. Jakobi, N.: Evolutionary robotics and the radical envelope-of-noise hypothesis. Adaptive Behavior 6(2), 325–368 (1997), http://dx.doi.org 71. Jander, R., Volk-Heinrichs, I.: Das strauch-spezifische visuelle perceptor-system der Stabheuschrecke (Carausius Morosus). Zeitschrift vor physiology 70, 425–447 (1970) 72. Kanou, M., Teshima, N., Nagami, T.: Rearing conditions required for behavioral compensation after unilateral cercal ablation in the cricket Gryllus bimaculatus. Zoological Science 19(4), 403–409 (2002) 73. Kazadi, S., Goodman, R., Tsikata, D., Green, D., Lin, H.: An autonomous water vapor plume tracking robot using passive resistive polymer sensors. Autonomous Robots 9, 175– 188 (2000) 74. Kern, R., Lutterklas, M., Egelhaaf, M.: Neuronal representation of optic flow experienced by unilaterally blinded flies on their mean walking trajectories. Journal of Comparative Physiology A 186, 467–479 (2000) 75. Kern, R., Lutterklas, M., Petereit, C., Lindemann, J., Egelhaaf, M.: Neuronal processing of behaviourally generated optic flow: experiments and model simulations. Network: computation in neural systems 12, 351–369 (2001) 76. Kern, R., Petereit, C., Egelhaaf, M.: Neural processing of naturalistic optic flow. Journal of Neuroscience 21, 139–144 (2001) 77. Kimchi, T., Etienne, A., Terkel, J.: A subterranean mammal uses the magnetic compass for path integration. Proceedings of the National Academy of Sciences USA 101, 1105–1109 (2004) 78. Kimmerle, B., Egelhaaf, M.: Performance of fly visual interneurons during object fixation. Journal of Neuroscience 20(16), 6256–6266 (2000) 79. Kindermann, T.: Behavior and adaptability of a six-legged walking system with highly distributed control. Adaptive Behavior 9, 16–41 (2001) 80. Koch, C.: Biophysics of Computation. Oxford University Press, Oxford (1999) 81. Kohstall-Schnell, D., Gras, H.: Activity of giant interneurones and other wind-sensitive elements of the terminal ganglion in the walking cricket. Journal of Experimental Biology 193, 157–181 (1994) 82. Korsching, S.: Odor maps in the brain: spatial aspects of odor representation in sensory surface and olfactory bulb. Cellular and Molecular Life Sciences 58, 520–530 (2001) 83. Korsching, S.: Olfactory maps and odor images. Current Opinion in Neurobiology 12, 387– 392 (2002) 84. Krapp, H., Hengstenberg, B., Hengstenberg, R.: Dendritic structure and receptive-field organization of optic flow processing interneurons in the fly. Journal of Neurophysiology 79(4), 1902–1917 (1998) 85. Krapp, H., Hengstenberg, R., Egelhaaf, M.: Binocular contributions to optic flow processing in the fly visual system. Journal of Neurophysiology 85(2), 724–734 (2001) 86. Labhart, T., Meyer, E.: Detectors for polarized skylight in insects: a survey of ommatidial specialisations in the dorsal rim area of the compound eye. Microscopy Research and Technique 47, 368–379 (1999) 87. Labhart, T., Meyer, E.: Neural mechanisms in insect navigation: polarization compass and odometer. Current Opinion in Neurobiology 12, 707–714 (2002)
Low Level Approaches to Cognitive Control
171
88. Lambrinos, D., Moeller, R., Labhart, T., Pfeifer, R., Wehner, R.: A mobile robot employing insect strategies for navigation. Robotics and Autonomous Systems 30, 39–64 (2000) 89. Land, M.: Visual acuity in insects. Annual Review of Entomology 42, 147–177 (1997) 90. Land, M.: Motion and vision: why animals move their eyes. Journal of Comparative Physiology A 185, 341–352 (1999) 91. Larsson, M., Svensson, G.: Methods in insect sensory ecology. In: Methods in Insect Sensory Neuroscience. CRC Press, Boca Raton (2005) 92. Latimer, W.: Acoustic competition in bush crickets. Ecological Entomology 6, 35–45 (1981) 93. Laurent, G.: Dendritic processing in invertebrates: a link to function. In: Dendrites, pp. 290–309. Oxford University Press, Oxford (1999) 94. Laurent, G.: Olfactory network dynamics and the coding of multidimensional signals. Nature Reviews Neuroscience 3(11), 884–895 (2002) 95. Laurent, G., MacLeod, K., Stopfer, M., Wehr, M.: Spatiotemporal structure of olfactory inputs to the mushroom bodies. Learning and Memory 5, 124–132 (1998) 96. Laurent, G., Stopfer, M., Friedrich, R., Rabinovich, M., Volkovskii, A., Abarbanel, H.: Odor encoding as an active, dynamical process: experiments, computation, and theory. Annual Review of Neuroscience 24, 263–297 (2001) 97. Laurent, G., Wehr, M., Davidowitz, H.: Temporal representations of odors in an olfactory network. Journal of Neuroscience 16(12), 3837–3847 (1996) 98. Maes, F., Collignon, A., Vandermeulen, D., Marchal, G., Suetens, P.: Multimodality image registration by maximization of mutual information. IEEE Transactions on Medical Imaging 16(2), 187–198 (1997) 99. Mason, A., Faure, P.: The physiology of insect auditory afferents. Microscopy Research and Technique 63(6), 338–350 (2004) 100. Matthies, L.: Mars microrover navigation: Performance evaluation and enhancement. Autonomous Robots 2(4), 291–311 (1995) 101. Menegatti, E., Zoccarato, M., Pagello, E., Ishiguro, H.: Image-based monte-carlo localisation with omnidirectional images. Robotics and Autonomous Systems 48(1), 17–30 (2004) 102. Menzel, R., Giurfa, M.: Cognitive architecture of a mini-brain: the honeybee. Trends in Cognitive Sciences 5(2), 62–71 (2001) 103. Michelsen, A.: Directional heading in crickets and other small animals. In: Neural Basis of Behavioural Adaptations, Fortschritte der Zoologie, vol. 39, pp. 195–207. Gustav Fischer Verlag, Stuttgart (1994) 104. Miller, J., Jacobs, G., Theunissen, F.: Representation of sensory information in the cricket cercal sensory system. Response properties of the primary interneurons. Journal of Neurophysiology 66(5), 1680–1689 (1991) 105. Mittelstaedt, H., Mittelstaedt, M.L.: Mechanismen der Orientierung ohne richtende Außenreize. Fortschr. Zool. 21, 46–58 (1973) 106. Mittelstaedt, M., Mittelstaedt, H.: Idiothetic navigation in humans: estimation of path length. Experimental Brain Research 139, 318–332 (2001) 107. Mizunami, M.: Functional diversity of neural organisation in insect ocellar systems. Vision Research 35, 443–452 (1995) 108. Mizunami, M., Weibrecht, J., Strausfeld, N.: Mushroom bodies of the cockroach: their participation in place memory. Journal of Comparative Neurology 402, 520–537 (1998) 109. Moeller, R., Lambrinos, D., Roggendorf, T., Pfeifer, R., Wehner, R.: Insect strategies of visual homing in mobile robots. In: Biorobotics - methods and applications, pp. 37–66. AAAI Press / The MIT Press (2001) 110. Moller, P., Goerner, P.: Homing by path integration in the spider Agalena labyrinthica Clerck. Journal of Comparative Physiology A 174, 221–229 (1994)
172
B. Webb et al.
111. Mueller, M., Homberg, U., Kuehn, A.: Neuroarchitecture of the lower division of the central body in the brain of the locust (Schistocerca gregaria). Cell Tissue Research 288, 159–176 (1997) 112. Nowotny, T., Huerta, R., Abarbanel, H., Rabinovich, M.: Self-organization in the olfactory system: one shot odor recognition in insects. Biological Cybernetics 93, 436–446 (2005) 113. Nowotny, T., Rabinovich, M., Huerta, R., Abarbanel, H.: Decoding temporal information through slow lateral excitation in the olfactory system of insects. Journal of Computational Neuroscience 15, 271–281 (2003) 114. Okada, R., Sakura, M., Mizunami, M.: Distribution of dendrites of descending neurons and its implications for the basic organisation of the cockroach brain. Journal of Comparative Neurology 458, 158–174 (2003) 115. Perez-Orive, J., Bazhenov, M., Laurent, G.: Intrinsic and circuit properties favor coincidence detection for decoding oscillatory input. Journal of Neuroscience 24, 6037–6047 (2004) 116. Perez-Orive, J., Mazor, O., Turner, G., Cassenaer, S., Wilson, R., Laurent, G.: Oscillations and sparsening of odor representations in the mushroom bodies. Science 297, 359–365 (2002) 117. Plewka, R.: Zur erkennung zeitlicher gesangsstrukturen bei laubheuschrecken: Eine vergleichende untersuchung der arten tettigonia cantans und leptophyes laticauda. Ph.D. thesis, University of Frankfurt (1993) 118. Pollack, G.: Neural processing of acoustic signals. In: Hoy, R., Popper, A., Fay, R. (eds.) Comparative Hearing: Insects, pp. 139–196. Springer, Berlin (1998) 119. Poulet, J., Hedwig, B.: Auditory orientation in crickets: pattern recognition controls reactive steering. PNAS 102, 15665–15669 (2005) 120. Quenet, B., Dreyfus, G., Masson, C.: From complex signal to adapted behavior: a theoretical approach of the honeybee olfactory brain. In: Burdet, G., Combe, P., Parodi, O. (eds.) Series in Mathematical Biology and Medicine, vol. 7, pp. 104–126. World Scientific, Singapore (1999) 121. Quenet, B., Horn, D., Dreyfus, G., Dubois, R.: Temporal coding in an olfactory oscillatory model. Neurocomputing 38(40), 831–836 (2001) 122. Rabinovich, M., Volkovskii, A., Lecanda, P., Huerta, R., Abarbanel, H., Laurent, G.: Dynamical encoding by networks of competing neuron groups: winnerless competition. Physical Review Letters 87, 68,102 (2001) 123. Reeve, R., Webb, B.: New neural circuits for robot phonotaxis. Philosophical Transactions of the Royal Society London A 361, 2245–2266 (2003) 124. Robert, D., Goepfert, M.: Novel schemes for hearing and orientation in insects. Current Opinion in Neurobiology 12, 715–720 (2002) 125. Ronacher, B., Wehner, R.: Desert ants, Cataglyphis fortis, use self-induced optic flow to measure distances travelled. Journal of Comparative Physiology A 177, 21–27 (1995) 126. Rosano, H.: Decentralised compliant control for hexapod robots: A stick insect based walking model. Ph.D. thesis, School of Informatics, University of Edinburgh (2007) 127. Rosano, H., Webb, B.: A dynamic model of thoracic differentiation for the control of turning in the stick insect. Biological Cybernetics 97(3), 229–246 (2007) 128. Russo, P.: Sistemi neurali biologici e controllo predittivo per l’integrazione acustico-Visiva nel grillo. Master’s thesis, Faculty of Computer Science and Engineering, University of Catania (2005) 129. Russo, P., Webb, B., Reeve, R., Arena, P., Patane, L.A.: Cricket-inspired neural network for feedforward compensation and multisensory integration. In: IEEE Conference on Decision and Control and European Control Conference (2005) 130. Schaefer, P., Ritzmann, R.: Descending influences on escape behavior and motor pattern in the cockroach. Journal of Neurobiology 49, 9–28 (2001)
Low Level Approaches to Cognitive Control
173
131. Schildberger, K.: Multimodal interneurons in the cricket brain: properties of identified extrinsic mushroom body cells. Journal of Comparative Physiology A 154, 71–79 (1984) 132. Schildberger, K., Milde, J., Horner, M.: The function of auditory neurons in cricket phonotaxis. II. Modulation of auditory responses during locomotion. Journal of Comparative Physiology A 163, 633–640 (1988) 133. Schmitz, B., Scharstein, H., Wendler, G.: Phonotaxis in Gryllus campestris l. I Mechanism of acoustic orientation in intact female cricket. Journal of Comparative Physiology A 148, 431–444 (1982) 134. Schmitz, J., Dean, J., Kindermann, T., Schumm, M., Cruse, H.: A biologically inspired controller for hexapod walking: Simple solutions by exploiting physical properties. The biological bulletin 200, 195–200 (2001) 135. Schul, J.: Song recognition by temporal cues in a group of closely related bushcricket species (genus Tettigonia). Journal of Comparative Physiology A 183, 401–410 (1998) 136. Seguinot, V., Cattet, J., Benhamou, S.: Path integration in dogs. Animal Behaviour 55, 787– 797 (1998) 137. Song, S., Miller, K., Abbott, L.: Competitive Hebbian learning through spike-timingdependent synaptic plasticity. Nature Neuroscience 3(9), 919–926 (2000) 138. Sprent, P., Smeeton, N.: Applied Nonparametric Statistical Methods, pp. 133–135. Chapman and Hall, Boca Raton (2007) 139. Srinivasan, M., Poteser, M., Kral, K.: Motion detection in insect orientation and navigation. Vision Research 39, 2749–2766 (1999) 140. Srinivasan, M., Zhang, S.: Visual motor computations in insects. Annual Review of Neuroscience 27, 679–696 (2004) 141. Stabel, J., Wendler, G., Scharstein, H.: Cricket phonotaxis: localization depends on recognition of the calling song pattern. Journal of Comparative Physiology A 165, 165–177 (1989) 142. Stange, G., Stowe, S., Chahl, J., Massaro, A.: Anisotropic imaging in the dragonfly median ocellus: a matched filter for horizon detection. Journal of Comparative Physiology A 188, 455–467 (2002) 143. Staudacher, E.: Sensory responses of descending brain neurons in the walking cricket, Gryllus bimaculatus. Journal of comparative physiology A 187, 1–17 (2001) 144. Staudacher, E., Schildberger, K.: Gating of sensory responses of descending brain neurones during walking in crickets. Journal of Experimental Biology 201, 559–572 (1998) 145. Stopfer, M., Laurent, G.: Short-term memory in olfactory network dynamics. Nature 402, 664–668 (1999) 146. Stout, J., McGhee, R.: Attractiveness of the male acheta-domestica calling song to females 2. the relative importance of syllable period, intensity, and chirp rate. Journal of comparative physiology A 164(2), 277–287 (1988) 147. Strausfeld, N., Hildebrand, J.: Olfactory systems: common design, uncommon origins? Current Opinion in Neurobiology 9, 634–639 (1999) 148. Stumpner, A.: Picrotoxin eliminates frequency selectivity of an auditory interneuron in a bushcricket. Journal of Neurophysiology 79, 2408–2415 (1998) 149. Stumpner, A., van Helversen, D.: Evolution and function of auditory systems in insects. Naturwissenschaften 88(4), 159–170 (2001) 150. Svenshnikov, A.: Problems in Probability Theory, Mathematical Statistics and Theory of Random Functions, p. 85. W.B. Saunders Company, Philadelphia (1968) 151. Szenher, M.: Visual homing in dynamic indoor environments. Ph.D. thesis, School of Informatics, University of Edinburgh (2008) 152. Tammero, L., Dickinson, M.: The influence of visual landscape on the free flight behavior of the fruitfly Drosophila melanogaster. The Journal of Experimental Biology 205, 327–343 (2002)
174
B. Webb et al.
153. Theunissen, F.: From synchrony to sparseness. Trends in Neurosciences 26, 61–64 (2003) 154. Tinbergen, N., Kruyt, W.: On the orientation of the digger wasp, philanthus triangulum fabr, III. Selective learning of landmarks. In: Tinbergen, N. (ed.) The Animal and Its World. Harvard University Press (1938) 155. Vardy, A., M¨oller, R.: Biologically plausible visual homing methods based on optical flow techniques. Connection Science 17(1-2), 47–89 (2005) 156. Vickerstaff, R., Paolo, E.D.: Evolving neural models of path integration. Journal of Experimental Biology 208, 3349–3366 (2005) 157. Viola, P., Wells, W.M.: Alignment by maximization of mutual information. International Journal of Computer Vision 24(2), 137–154 (1995) 158. Webb, B.: Neural mechanisms for prediction: do insects have forward models? Trends in neuroscience 27(5), 278–282 (2004) 159. Webb, B., Harrison, R.: Integrating sensorimotor systems in a robot model of cricket behavior. In: Sensor Fusion and Decentralised Control in Robotic Systems III. SPIE, Boston, November 6-8 (2000) 160. Webb, B., Reeve, R.: Reafferent or redundant: How should a robot cricket use an optomotor reflex? Adaptive Behaviour 11(3), 137–158 (2003) 161. Webb, B., Scutt, T.: A simple latency dependent spiking neuron model of cricket phonotaxis. Biological Cybernetics 82(3), 247–269 (2000) 162. Webb, B., Wessnitzer, J., Bush, S., Schul, J., Buchli, J., Ijspeert, A.: Resonant neurons and bushcricket behaviour. Journal of Comparative Physiology A 193, 285–288 (2007) 163. Weber, K., Venkatesh, S., Srinivasan, M.: An insect-based approach to robotic homing. In: Jain, A., Venkatash, S., Lovell, B. (eds.) Fourteenth International Conference on Pattern Recognition, pp. 297–299. IEEE, Los Alimitos (1998) 164. Weber, T., Thorson, J.: Auditory behaviour in the cricket. II. Interaction of direction of tracking with perceive temporal pattern in split-song paradigms. Journal of Comparative Physiology A 163, 13–22 (1988) 165. Wehner, R.: The ants celestial compass system: spectral and polarization channels. In: Lehrer, M. (ed.) Orientation and Communication in Arthropods, pp. 145–285. Birkhauser, Basel (1998) 166. Wehner, R.: Desert ant navigation: how miniature brains solve complex tasks. Journal of Comparative Physiology A 189, 579–588 (2003) 167. Weisstein, E.: Method of steepest descent. From Mathworld - A Wolfram Web Resource (2006), http://mathworld.wolfram.com/MethodofSteepestDescent.html 168. Wessnitzer, J., Mangan, M., Webb, B.: Place memory in crickets. Proceedings of the Royal Society of London B (2008) 169. Wessnitzer, J., Webb, B.: Multimodal sensory integration in insects - towards insect brain control architectures. Bioinspiration and Biomimetics 1, 63–75 (2006) 170. Wittlinger, M., Wehner, R., Wolf, H.: The ant odometer: stepping on stilts and stumps. Science 312, 1965–1967 (2006) 171. Wittmann, T., Schwegler, H.: Path integration - a network model. Biological Cybernetics 73, 569–575 (1995) 172. Wohlers, D., Huber, F.: Processing of sound signals by six types of neurons in the prothoracic ganglion of the cricket Gryllus campestris l. Journal of Comparative Physiology 146, 161–173 (1981) 173. Wohlgemuth, S., Ronacher, B., Wehner, R.: Ant odometry in the third dimension. Nature 411, 795–798 (2001) 174. Wolf, R., Heisenberg, M.: Basic organisation of operant behaviour as revealed in Drosophila flight orientation. Journal of Comparative Physiology A 169, 699–705 (1991)
Low Level Approaches to Cognitive Control
175
175. Wolf, R., Voss, A., Hein, S., Heisenberg, M.: Can a fly ride a bicycle? Philosophical Transactions of the Royal Society B 337, 261–269 (1992) 176. Yack, J.: The structure and function of auditory chordotonal organs in insects. Microscopy Research and Technique 63(6), 315–337 (2004) 177. Yager, D.: Structure, development, and evolution of insect auditory systems. Microscopy Research and Technique 47(6), 380–400 (1999) 178. Ye, S., Leung, V., Khan, A., Baba, Y., Comer, C.: The antennal system and cockroach evasive behaviour I Roles for visual and mechanosensory cues in the response. Journal of Comparative Physiology A 189, 89–96 (2003) 179. Zampoglou, M., Szenher, M., Webb, B.: Adaptation of controllers for image-based homing. Adaptive Behavior 14(4), 381–399 (2006), http://dx.doi.org/ 180. Zeil, J., Hofmann, M., Chahl, J.: Catchment areas of panoramic snapshots in outdoor scenes. Journal of the Optical Society of America A 20, 450–469 (2003) 181. Zeiner, R., Tichy, H.: Combined effects of olfactory and mechanical inputs in antennal lobe neurons of the cockroach. Journal of Comparative Physiology A 182, 467–473 (1998)
Part II
Cognitive Models
4 A Bottom-Up Approach for Cognitive Control H. Cruse1 , V. D¨urr2 , M. Schilling1, and J. Schmitz1 1
2
University of Bielefeld, Department of Biological Cybernetics and Theoretical Biology, P.O. Box 100131, D-33501 Bielefeld, Germany {holk.cruse,malte.schilling,josef.schmitz}@.uni-bielefeld.de University of Cologne, Institute of Zoology, Weyertal 119, D-50931 K¨oln, Germany
[email protected]
Abstract. Cognitive skills including the ability to plan ahead require internal representations of the subject itself and its environment. The perspective on internal representations and assumptions on the importance of internal representations have changed over the last years: While traditional Artificial Intelligence tried to built intelligent systems relying solely on internal representations and working on a Knowledge level, behaviourist approaches tended to reject internal representations. Both approaches produced considerable results in different but complementing fields (examples are higher level tasks as mathematical proofs, or simple tasks for robots respectively). We believe that the most promising approach for more general tasks should connect both domains: For higher level and cognitive tasks internal representations are necessary or helpful. But these representations are not disconnected from the body and the lower-levels. The higher level is grounded in the lower levels and the robot is situated in an environment. While many projects today deal with the requirements of embodiment and situatedness, we focus on an approach like that of Verschure [55, 57, 58, 56]: The control system is built from the bottom up, growing towards higher levels and more complex tasks. The primitives of the different levels are constituted through the lower level and, as a main aspect, rely on neural networks. While Verschure only addressed the most relevant lower-level models in his work, we are constructing an architecture for cognitive control. The main idea for the cognitive control is the idea of mental simulation: Mental simulation sees planning as ”probehandeln”: This notion of trying a movement by mentally enacting it without performing the action physically relies strongly on the notion of an internal model. As a first step an internal model of the own body is constructed and used which later-on may be expanded to models of the environment. This body model is fully functional: it is constrained in the same way as the body itself. It can move and can be used in the same way as the body. Therefore, hypothetical movements can be tested on their consequences. For this purpose it must be possible to decouple the body itself from the action controlling modules to use the original controllers for control of the internal body model. Our approach shall address in particular (1) the structure and the construction of the mental models, (2) the process of using learnt behaviours to control the body or to modulate these behaviours in controlling the body or the internal model, (3) the decoupling of the body from the control structures to invoke the simulation, (4) the invention of new situation models and the decision when to construct new ones. An essential aspect of the approach proposed here is, apart from the idea that memory is composed of many individual situation models consisting of RNNs, that random selection of the connections is used, in this way introducing some kind of Darwinian aspect (see Edelmans Neural Darwinism). P. Arena and L. Patan`e (Eds.): Spatial Temporal Patterns, COSMOS 1, pp. 179–218. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com
180
H. Cruse et al.
4.1 Introduction It has often, and convincingly, been stated that traditional AI is not generally suited to solve problems of cognition [5, 4, 17, 34]. These critiques include the idea that embodiment is an important concept and should therefore be taken into account. Further, they suggest that basic sensorimotor systems should be studied first, hoping that this bottom-up approach may lead to the understanding of higher cognitive functions. In the preceding chapters a number of relevant examples based on investigations of insect behaviour and on insect neurophysiology have been given. Specific attention was given to the properties of the insect brain, in particular with respect to the possible function of the mushroom bodies and related structures. This system has been considered to be responsible for different functions but the main point of interest there is that the mushroom bodies have been shown to be an important locus for learning. In this chapter, we propose a memory structure that uses simple analogue neurons and concentrate on the question of how many different situations could be stored. This structure may be compared with that of mushroom bodies, but is thought to represent a more general concept on how memories, in particular procedural memories, could be organized. After describing in general the behaviour-based approach put forward during the last two decades as a reaction to the more traditional AI approach, a way is sketched as to how the behaviour-based concept may be combined with higher, cognitive procedures. Our concept is strongly influenced by Verschure [55, 57, 58, 56] who proposed a layered structure. As the behaviour-based approach strongly relies on embodiment, we use as an example a behaviour that cannot be performed by a body with simple geometry, as for example a two-wheeled robot. Rather we use a hexapod walker with 18 degrees of freedom, i.e., a system with many extra degrees of freedom. This is challenging because such a body allows for quite complex behaviours to be performed. We have studied insect locomotion behaviour when negotiating curves. In addition, the specific question of how to cope with control of walking after losing a leg has been studied [43]. Furthermore, it has recently been shown [3] that a reactive controller can even solve complex tasks, as for example climbing over very large gaps. However, the question of how continuation of studies of reactive systems could lead to the understanding of cognitive systems is still an open problem. Therefore, classical approaches still are relevant, at least insofar as they have specified the problems, and as they have investigated to what extent earlier proposals have reached limitations. Inspired by these approaches, we will investigate how a reactive controller for hexapod walking could be improved by an internal model of the own body. This model might improve reactive behaviour, but could also be used for cognitive control in the sense of planning ahead. Studying insect behaviour and neurophysiology has provided some important insights, but most current models for cognitive systems are based on psychological and neurophysiological investigations in primates (including imaging techniques in humans). Generally the interpretations remain on a qualitative, conceptual level. Quantitative models that attempt to find a functional interpretation either suffer from the problem of dealing with a large number of neuronal units with a corresponding huge number of unspecified degrees of freedom [1], or use models not based on a neuronal structure, in this way simplifying the task [48]. If a strict neuronal architecture is used,
A Bottom-Up Approach for Cognitive Control
181
the studies concentrate on small isolated tasks, and the resulting models are therefore not suited to control an autonomous system. Nevertheless, as a crucial constraint for our architecture we would apply neuronal based structures. Therefore, after the general concept concerning the proposed memory architecture has been explained, we will introduce several types of small RNNs that can be applied as elements for procedural memories and that show the ability to learn by using simple learning rules. As a long term goal, this should lead to a complete system that has the ability to store new information, to find some order in this memory, and of course to retrieve specific information. In the following section we will briefly recall the arguments which led to the introduction of the behaviour-based approach. Following this approach we are going to propose an architecture in section 4.4 and 4.5 that could allow a six-legged walker to learn tasks of classical and operant conditioning, to search for solutions to new problems and, as the main goal, to be able to plan ahead. To this end we will propose RNN-based “engrams” embedded in this memory structure. Finally, we will discuss initial ideas for how abstract rules provided by an external teacher can be learnt and later be used to improve the performance of the corresponding behaviour.
4.2 Behavior-Based Approaches The approach of traditional Artificial Intelligence (AI) to explain intelligent behaviour relies mainly on the notion of a knowledge level [33]: Intelligence is considered as a high level process which relies on using knowledge, which is encoded in a symbolic representation. The process can be described as a manipulation of symbols. The origins of this perspective on intelligence can be traced back to traditional philosophy of mind and the view of dualism: mind and body are distinct from each other (as stated by Descartes). The development of the computer had another great impact on the young research field of AI: the metaphor of the mind as a computer and cognition and intelligence as a form of computation or calculation guided research over years. This view was supported as it showed to be successful in various tasks, which were supposed to need intelligence. Focussing only on an abstract symbolic level and on rules operating on this level appeared to be suitable on high-level tasks (as chess and mathematics), but it failed for “simpler” tasks: the recognition of speech and objects in pictures, adaptive control of movements, etc. Tasks including interactions with a not completely organised environment became hard and intractable for a pure symbolic approach. It became clear that the process of interacting with the environment can not be simplified to a sequence of successive stages, as assumed in traditional AI like in [40]: • Perceive the world through sensors. • Reasoning on an abstract level. • Action on the world through actuators. These three stages were considered as separate modules, which execute some form of transformation. This transformation is directed towards the next stage and there are no interconnections or recurrencies between the modules.
182
H. Cruse et al.
Research on perception (and today on action of humans in relation to thinking) revealed that in humans perceiving and acting on the world is not a serial process but a highly intertwined process involving lots of recurrent connections and parallel processing within the different stages and also between them. In addition, the representation is not only involved in reasoning, but strongly influences the way of perceiving and acting. Therefore, the approach of intelligence as symbol manipulation was questioned. Different problems had been encountered and although some of them could be neglected, they helped progressing research towards today’s concepts of embodiment and situatedness. Searle’s Chinese Room Argument has stimulated discussions on these issues — although today many reject the argument or, more specific, reject its premises. The problem of symbol grounding [21] and the question of how a higher level representation and its constructs can derive meaning has become an important issue (and from our point of view the most important one) in building a representation. Other problems as the frame problem [29], the frame of reference problem [6] and the problem of situatedness [53] were identified. Verschure [56] analysed the different problems and showed that they originate from the same causation: He integrated the different problems and identified as a main cause for them the problem of a prioris ([56], page 2): In short this problem is created by the critical dependance of a model of a cognitive process on the a priori specification of a world model. As a result such [in the sense of symbolic Artificial Intelligence] an approach runs the risk of specifying a system which is not grounded in the world in which it finds itself, is prone to mix up the domain ontologies involved, relies on a representational granularity which induces a search, and ignores some pertinent elements of system-environment interaction. The problem of a prioris describes the lack of pure symbolic representations to include meaning because of the disconnectedness of model and modelled objects. The model as a surrogate for the object (event, concept, . . . ) is described a priori — it is determined in advance which properties shall be modelled and included in the system and in which way. Early work in behaviour-based robotics therefore neglected the notion of an internal representation and built systems which were controlled by simple reactive systems. Nonetheless, these were able to solve tasks which were supposed to require intelligence. The concept of intelligence itself was questioned by these experiments and changed the understanding of the term. It was now assumed that intelligent behaviour can arise out of simple processes which are interacting with each other. This view has been stressed by the introduction and the notion of “prerational intelligence”. Intelligence could be thought in this way as an emergent property of the system, which can only be ascribed from the observer’s point of view when concentrating on the complex overall behaviour. Brooks’ [5] systems shaped the field itself and later research in cognitive science in general. Brooks showed that in many cases internal representations are not needed and the world can be used as its own best model [5, 4]. In fact, by using the world as its own model one circumvents the problem of a prioris, namely deciding what to represent and how to update the representation.
A Bottom-Up Approach for Cognitive Control
183
Nonetheless, we think it is not helpful to reject internal representations in general, but to incorporate internal representations: not as a starting point for an autonomous system, but as an extension in the development of a system. Steels has argued in this way ([49], page 2389): These findings suggest that representation-making might be a crucial bootstrapping device for higher mental function. At some point external representation-making becomes internalised to form the basis of thinking, in the sense of inner dialogs or mental imaging. So internal representations do not come before external representations but follow or co-evolve with them. For systems which also use an internal representation it is crucial how the latter is related to lower-level functions, as well as acting and sensing: Glenberg stated that internal representations have developed in the service of action and perception [20] and are therefore highly intertwined with the latter. Verschure introduced the research program of synthetic epistemology [56]: Before dealing with representations and intelligent operations on them we should focus on how knowledge is acquired and explored. This approach of building up knowledge representations without restricting their structure in advance was addressed by Verschure in a series of experiments on developing a bottomup control structure for robots and simulations [55, 57, 58, 56]. The internal model is not disconnected from the environment, the body, actions and sensors. It must be embodied and structures in the model must be grounded in the world. As a starting point, Cruse considered a model of the own body ([8], p. 138): The most important part of the world, [. . . ] is the own body. This leads to the speculation that a basis of such a world model might be formed by a model of the own body. The grounding of a model in the outside is realised through integrating the different activated neuronal traces in sensing and acting in an action (Glenberg calls this process meshing [20]). This builds up an internal representation of actions by relating sensations and movements (on different levels of abstraction which, only on higher levels, are accessible through language or for reasoning). Objects are represented by the actions they offer to the observer. They are interpreted from the perspective of possible actions of our body. Gibson called this affordance [18]. Concerning the grounding problem, Steels stated ([50], p. 172): Grounded robots that engage in communication using external representations not only need a physical body and low-level behaviours but also a conceptual world model which must be anchored firmly and dynamically by the robot in the environment through its sensori-motor apparatus. We argued that there is not a simple sweeping theoretical principle to turn a system that uses conceptual world models into a grounded system. Instead many processes must be carefully integrated. Or as Gallese [16] has formulated it in describing the grounding of a model of the own body: [embodied simulation] applies not only to actions or emotions, where the motor or viscero-motor components may predominate, but also to sensations like
184
H. Cruse et al.
vision and touch. It is mental because it has content. It is embodied not only because it is neurally realized, but also because it uses a pre-existing bodymodel in the brain realized by the sensorymotor system, and therefore involves a non-propositional form of selfrepresentation. We think that building an intelligent and cognitive system needs some form of internal representation for planning ahead (in the sense of an internal mental simulation). But the perspective on internal representations has changed. Internal representations can not be viewed as to be disconnected from the external world. Instead, the basic elements of the internal representation must be grounded in the environment and must have meaning. The higher-level representation must be connected in some form to the external environment it is representing and which is providing a meaning for the representation. Current results from neurobiological research —for example using imaging techniques to observe brain activities— suggest that higher level problem solving, thinking and planning ahead is related to problems of motor control: When planning a sequence of movements the same neuronal clusters were found active which were active when actually performing the movements and which therefore were thought of as being responsible for motor control. This lead to the idea of planning and understanding as a mental simulation (or Probehandeln as termed by Freud [13, 14]): we understand situations by constructing corresponding internal representations. These internal representations are dynamic representations which allow us to predict consequences of actions and to predict how a situation will develop. They are not disconnected from the environment — they are grounded in the units on the lower neural level which are used in perceiving them or acting on them. For planning these representations can be used to test the effects of actions in a mental simulation. Based on the observed effects of the simulated action one can decide to enact the action or to discard it and search for an action which is better suited with respect to the current goal. Mental simulation relies on grounded internal models. The first internal model to start with must describe the body of the system itself, including its properties and functionalities. This model must be grounded and embodied: It evolves in a bottom-up manner from lower-level functions, involving both, the sensory and the motor system. The system itself should be situated, therefore models of the environment have to be included and have to be grounded in the environment. In our approach extending the existing reactive controller, Walknet, we will start with a model of the own body which will be used for planning ahead through mental simulations. In a next step models of the environment shall be integrated — this leads to many open questions regarding these models: • • • •
learning of a model, updating existing models, retrieval of a model, creating a new model, . . .
Our approach to understand and rebuild cognitive behaviour is guided by the goal to build an architecture up to a cognitive level for planning ahead by using internal models, or even up to a language level.
A Bottom-Up Approach for Cognitive Control
185
Thus the results gained are posing the basis for a much wider research activity that will set-up first steps by defining the lower levels from which these higher levels can develop.
4.3 A Bottom-Up Approach for Cognitive Control While some researchers refused to use internal representations, others tried to combine the two approaches and therefore to combine the advantages of both and to learn from both fields. Verschure [57, 56], as mentioned, examined problems arising in the traditional approach to AI and summarized them as the problem of a prioris: the symbol systems are disconnected from the environment and the form of representation (what is represented and how) of the environment is determined in advance. Verschures’ work has influenced our own approach both, on a theoretical level, as well as own the level of implementation, as he uses a behavior-based approach realised in simulations and on a robot. Therefore, we want to briefly summarise his work —graphically depicted by the two lower black boxes in Fig. 4.1 (lower level, medium level)— and discuss connections to our own work, as well as differences. Verschure has proposed the program of synthetic epistemology: the focus is shifted away from the question of how to use knowledge. The questions of how a system acquires knowledge and how it retains this knowledge have to be addressed first. Verschure assumed that behaviour requires variables which mediate between perception and action and that an explanation of behaviour presupposes an understanding of these variables. He focussed on examining these variables — how they develop, how they are structured and how they are learnt. Verschure initiated a sequence of experiments in simulation and in real implementation on a robot — he argues that both forms of investigations are needed. Simulations allow analysis of the development of the neural control structure and evolution of this structure. On the other hand real world tests on a robot are needed to show the abilities of the robot. His aim was to combine the advantages of neural approaches —their ability to learn— with the advantages of a knowledge based system because the latter is easily understandable for the observer and allows to assign goals and states to different variables. The work of Verschure [55, 57, 58, 56] has inspired our approach on the architecture of the different levels, and gives some additional insights into the influences of learning at different stages in the control architecture (one of the key points of Verschure’s work is that even learning only at a perceptual “stage” leads to adaptation of behaviour). In our own approach the lower level and the connections to the medium level (see below) are at first not learnt but considered as given in the form of innate structures. The details of these structures as implemented in our model have been derived from biological experiments. Verschure’s research complements our own approach with respect to learning on the lowest level. As the structure of the two approaches is comparable, conclusions can be transferred. Therefore we want to briefly present Verschure’s approach and want to emphasise the main characteristics of the DAC robot series by adopting Sloman’s perspective on robotic systems.
186
H. Cruse et al.
Sloman [47] has developed a framework for examining architectures for robots — the CogAff framework. This framework is inspired by work from Artificial Intelligence but is supposed to be used for considering connectionist approaches and hybrid models. While work in AI has mostly concentrated on specialized forms of representation serving one specific purpose, Sloman encourages a broader approach for representations and in particular an approach which makes it possible to incorporate the different forms of representations. Therefore he developed the CogAff framework for describing agent architectures and their underlying representations. This provides a unified perspective on different approaches. It permits connecting different approaches and defines a space of possible architectures. In this framework Sloman talks about three stages: Perceive, reason and act, which have their origins in traditional AI and make it therefore easy to describe AI systems in terms of the CogAff framework. But while the traditional approach to AI used these stages as separated and ordered stages, for Sloman these stages are just an abstraction: there can be recurrent connections, concurrent processing in the different stages, the stages should not be regarded as confined modules but as intertwined processes. In addition to this loose division into stages he introduces 3 layers of abstraction: The lowest layer is a pure reactive layer. The medium layer introduces deliberation: planning ahead, representations, generating hypotheses and selecting one of the possible hypotheses (he calls this “what if” mechanisms). The highest layer includes reflective processes — meaning meta-management of the information processing. In Fig. 4.1 the architecture of Verschure’s approach is depicted — from the perspective of the layers of Sloman’s CogAff framework. The CogAff framework is illustrated by a 3 x 3 grid, on top of which Verschures’ as well as our concepts are explained. The DAC series of robots built by Verschure comprise complete learning systems, based on predefined reflex structures on a lower-level of a minimal behavioural competence (avoidance and attraction). On an adaptive level sensory cues from different layers can be connected to the responses and the internal state and construct sensory representations which are grounded in the lowest level. On a contextual control level these are used to build up complex representations, which represent sequences over time of motor events in association to sensory inputs. Verschure’s model is implemented in neural networks at the lowest level only. On the higher level other mechanisms are used in addition. His main ideas concentrate on the organisation and the self-formation of the different layers and the learning of behaviours. Our demand of a biologically plausible architecture makes it necessary to consider a different neural substrate for the implementation. On the lower levels our approach is very similar to Verschure’s approach: We start with a decentralized, reactive controller for hexapod walking. On the lowest level it consists of motor primitives, and simple reactive behaviours. Each leg is controlled locally by one of these controllers (see Fig. 4.1, bottom). These motor primitives are activated by elements of a medium level where it is decided which action should take place. So, basically the low-level motor primitives represent situation models, including the representations of the state in which a behaviour becomes active (stimuli) and the representation of the motor commands (response). On a medium level these motor primitives can now be activated and modulated. The lower level activations represent a state of the system whereas on the medium level
A Bottom-Up Approach for Cognitive Control
Perception
Central Processing
187
Action
MetaManagement
re? ective processes "what if" mechanisms
Reactive mechanisms
Deliberative reasoning
Internal Model
= high-level Invention of new plans = sequencing and modulation of motor primitives simulate
modulate
= medium-level
Behavior selection: local peripheral PG & Coordination In? uences sensors
control
Motor Primitives: • Stance-Net • Swing-Net
= low-level
Fig. 4.1. Illustration of the relation between different approaches, in particular the concepts of Sloman, of Verschure and of our own. The 3 x 3 grid (in gray) depicts Slomans conceptual approach, the black boxes (lower and medium level) indicate the overlay of the approach of Sloman and of Verschure. Our approach builds on top of Verschures approach by introducing a high-level structure consisting of internal models, in particular a model of the own body.
these activations of areas involved in perception and action are connected to establish a behaviour: they form prototypes (in Figure 4.1 the medium level). These selector modules are in their simplest form reactive patterns, but can be more complex sequences or situations. While the basic structure of the lower levels is comparable to Verschure’s approach, there are some essential differences which seem to be important for providing meaning to a system. Verschure represents a set of arbitrary sensor values and motor commands in his system. In our system sensors and actuators are not only represented on this lower level as variables. The main difference of our system compared to Verschure’s system is that we introduce an internal representation. This internal representation is grounded in the lower level primitives and variables. There is a direct connection between these layers. Meaning of a behaviour on the higher level emerges from the meaning of the primitives of the lower level which form and ground the behaviour. The cognitive level constitutes this higher layer. Cognition in the sense of being able to plan ahead [30] relies on an internal representation [49] which is grounded in embodied experiences [17]. So, this higher level is not distracted from the lowest level or a distinct separate symbolic layer. Instead, its primitives are embodied in the lower levels and the primitives used to construct a plan are exactly the same as in executing the plan. The mental model of the own body is depicted in the figure in the upper part. In fact, the internal model is not part of the control architecture, but is used by the control architecture to simulate possible sequences of actions. The simulation corresponds to
188
H. Cruse et al.
Slomans [47] “what if”-mechanism in the process of building plans, while the invention of new sequences comprises some kind of meta-management and a reflexive process. The main idea of this higher level is the idea of mental simulation (Gallese and Lakoff [17], page 458): Imagination is mental simulation, carried out by the same functional clusters used in acting and perceiving. Mental simulation sees planning as ”probehandeln”: This notion of testing a movement by mentally enacting without doing the action in reality relies strongly on the notion of an internal model. As a first step an internal model of the own body is constructed and used which later-on may be expanded to models of the environment. This body model is fully functional: it has to be constrained in the same way as the body itself and it can move and be used in the same way as the body. The body model can be thought of as a small internal puppet you can play around with to test hypothetical movements for their consequences. For this purpose it must be possible to decouple the body itself from the action controlling modules to use the original controllers for control of the internal representations [11]: Actions are not carried out directly, but instead trigger simulations of what they would do in the imagined situation. This ability to simulate or imagine situations is a core component of human intelligence . . .
4.4 Representation by Situation Models In this section we are going to propose our bottom-up approach for a cognitive system in more detail. It is thought as an extension of the reactive controller Walknet (see chapter 2) on the basis of which the notion of thinking as mental simulation shall be elaborated. In addition, this section will raise issues on the process of learning. The overall model will pose a set of requirements on the neuronal structure of the memory. The aim for the control architecture is to grow up from locomotion to cognitive capabilities. The system should be able to solve simple reactive tasks as well as complex tasks, including tasks which require cognitive capabilities as for example setting up plans. The involved perception processes are goal-oriented and the architecture should be able to recognise and learn situation models by self-observation. In general, an architecture for an autonomous robot (in the sense of an agent which acts following its own rules and its own desires) has to fulfil different requirements. For an architecture which even shows cognitive abilities these different aspects grow up from simple behaviours to higher mental functions. From our point of view cognition is the ability to plan ahead. This mental faculty is not detached from low level skills but it is directly linked and grounded in lower level behaviours and cannot be separated from the body (as the subject in perceiving and acting on the environment) and the current situation (including the previous actions and developments of the situation). These two topics —embodiment and situatedness [5]— have changed the research on Artificial Intelligence over the last years and are now widely accepted and a main focus in current research.
A Bottom-Up Approach for Cognitive Control
189
It is our aim to build up an architecture from the bottom up to such embodied and situated cognitive abilities. The architecture is inspired by biological insights and therefore should be implemented through a biologically plausible system in the form of neural networks. The different levels from which higher cognitive capabilities should emerge (see Fig. 4.1): • low-level = motor primitives and reactive behaviours. There are many different behaviours on this level. These can be triggered through sensory cues directly or the primitive modules are modulated through the higher levels. • medium level = combining and selecting the motor primitives. On a medium level the motor primitives are modulated to form actions (as series of goal-directed movements). The sequences could be learnt or could be generated following some general rules (like in our case the selection of the swing and stance mode for a single leg). The modulation is governed through the sensory inputs and additional higher sensory cues could be learnt (like for vision). • Higher level (→ cognitive level) = planning ahead. The cognitive level relies on internal representations and uses these representations to plan ahead. In our view the starting point for cognitive abilities and the first representation acquired is a model of the own body. So the agent should at first be equipped with a model of its own body which should then be expanded by a model of its surrounding. These models can then be used to construct a plan, when an unsuitable or unknown situation is encountered: In this situation the motor primitives from the lower level are modulated and combined through the medium level. They control the internal model of the body detached from the body itself. The consequences of the actions and of mutated actions can be observed in the model and can be then used for controlling the body. Using a sufficiently detailed body model, including some relevant aspects of the actual environmental (and internal) situations, different solutions to a given problem can be searched for by simulated trial-and-error (“2nd order embodiment”) [42]. To test and demonstrate the diverse abilities of the robot a set of tasks with increasing difficulty was proposed which should address and test the capabilities of the robot on each of the levels: 1. Control of swing and stance phase of the robot and the coordination of the legs in walking. 2. Stable walking in disturbed situations, steering control and curve walking. 3. Climbing a step. 4. An exploration task in which it is necessary that the robot learns to integrate its different sensory pathways. 5. Learning of conditioned reflexes for walking in cluttered environment and instrumental learning, e.g., for walking in a cluttered environment or for adapting to the loss of a leg. 6. A cognitive task: the body configuration problem — the robot is in an unsuitable configuration (with respect to its surrounding and its joint limits). The robot cannot move forward without tumbling down. Therefore the robot is forced to plan his
190
H. Cruse et al.
movements ahead and to mentally try/simulate different actions (e.g., one foot is stepping backwards) — before applying a successful action. The first three tasks can be solved by the Walknet architecture described in the second chapter of this book. A simple reactive system is therefore sufficient to explain these behaviours. Learning the integration of sensory pathways in an exploration task and forming of representations out of the sensory inputs will be addressed in the following chapters. Learning of situation models as such has been addressed above. In the following, we are not going to elaborate on the application of this learning procedure to the complete system or on the before mentioned other tasks. Instead, we are concentrating on the last task mentioned in the list. In this case, the solution requires knowledge about the own body, some knowledge about the interaction with the environment and knowledge about possible movements. This task is a cognitive task, meaning, it requires planning ahead. 4.4.1
Basic Principles of Brain Function
After having briefly sketched earlier concepts and related it to our ideas, in the remainder we will describe in more detail our approach which, as a main characteristic, is strictly based on the application of a neuronal architecture. Generally, attempts to simulate cognitive systems (which provides a way of understanding such systems) are usually based on structures of human-like brains with the, often implicit, assumption that cognition is only possible for a brain with highly complex structures similar to that of mammalian or avian brains. However, invertebrate animals e.g., insects or, even more obvious, cephalopods show astonishing complex behaviours that may be attributed to cognitive capabilities (see chapter 4.3). Among these tasks are “delay match to sample” and “delayed non-match to sample” tasks. Honey bees, for example, can solve “delayed matching to sample” tasks which allowed Giurfa et al. [19] to show that honey bees are able to learn the concept of “symmetry”. These authors could further show that honey bees are able to learn the concept of “difference” or “oddity” which implied these insects to be able to cope with “delayed non-matching to sample” tasks. Drosophila is assumed to be able to construct a dynamic representation of an optical pattern that has disappeared behind an occluder and expects it to appear again at the other side of that occluder [52]. Learning from observing conspecifics has been shown in octopus [23], switching context in ant navigation [59], weighting the saliency of parameters in Drosophila [54]. This means that also much smaller brains and presumably quite different architectures are capable of “higher”, cognitive functions. Studying and understanding such minimal solutions is presumably simpler than studying complex mammalian brains. It can help to understand basic principles and may therefore finally also help to understand the functioning of complex brains. Following this approach, we, in the words of R. Beer [2] study minimal cognition, i.e., will study small networks showing properties that might be called cognitive. This means that we will concentrate on networks with one unit representing one item. This might be considered as being too “localistic”, because destruction of one such unit would destroy the memory for this item which is in
A Bottom-Up Approach for Cognitive Control
191
contrast to biological, at least vertebrate brains [36]. However, we decided to live with this drawback in order to be able to meet another goal, namely to develop a system that hopefully will be able to cope with higher cognitive functions. Quiroga et al.