SERIES EDITORS
STEPHEN G. WAXMAN Bridget Marie Flaherty Professor of Neurology Neurobiology, and Pharmacology; Director, Center for Neuroscience & Regeneration/Neurorehabilitation Research Yale University School of Medicine New Haven, Connecticut USA
DONALD G. STEIN Asa G. Candler Professor Department of Emergency Medicine Emory University Atlanta, Georgia USA
DICK F. SWAAB Professor of Neurobiology Medical Faculty, University of Amsterdam; Leader Research team Neuropsychiatric Disorders Netherlands Institute for Neuroscience Amsterdam The Netherlands
HOWARD L. FIELDS Professor of Neurology Endowed Chair in Pharmacology of Addiction Director, Wheeler Center for the Neurobiology of Addiction University of California San Francisco, California USA
Elsevier Radarweg 29, PO Box 211, 1000 AE Amsterdam, The Netherlands Linacre House, Jordan Hill, Oxford OX2 8DP, UK 360 Park Avenue South, New York, NY 10010-1710 First edition 2011 Copyright Ó 2011 Elsevier B.V. All rights reserved No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means electronic, mechanical, photocopying, recording or otherwise without the prior written permission of the publisher Permissions may be sought directly from Elsevier’s Science & Technology Rights Department in Oxford, UK: phone (þ44) (0) 1865 843830; fax (þ44) (0) 1865 853333; email:
[email protected]. Alternatively you can submit your request online by visiting the Elsevier web site at http://elsevier.com/locate/permissions, and selecting Obtaining permission to use Elsevier material Notice No responsibility is assumed by the publisher for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions or ideas contained in the material herein. Because of rapid advances in the medical sciences, in particular, independent verification of diagnoses and drug dosages should be made Library of Congress Cataloging-in-Publication Data A catalog record for this book is available from the Library of Congress British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library ISBN: 978-0-444-53752-2 ISSN: 0079-6123
For information on all Elsevier publications visit our website at elsevierdirect.com
Printed and bound in Great Britain 11 12 13 14
10 9 8
7
6 5
4
3
2 1
List of Contributors A.J. Bastian, Department of Motor Learning Lab, Kennedy Krieger Institute, and Neuroscience Department of Johns Hopkins School of Medicine, Baltimore, Maryland, USA M. Casadio, Department of Physiology, Northwestern University, and Sensory Motor Performance Program, Rehabilitation Institute of Chicago, Chicago, Illinois, USA F. Champoux, Centre de Recherche Interdisciplinaire en Réadaptation du Montréal Métropolitain, Institut Raymond-Dewar, Montréal, Québec, Canada O. Collignon, Département de Psychologie, Centre de Recherche en Neuropsychologie et Cognition (CERNEC), and Centre de Recherche CHU Sainte-Justine, Université de Montréal, Montréal, Québec, Canada E.K. Cressman, School of Human Kinetics, University of Ottawa, Ottawa, Ontario, Canada Z.C. Danziger, Department of Biomedical Engineering, Northwestern University, and Sensory Motor Performance Program, Rehabilitation Institute of Chicago, Chicago, Illinois, USA M. Darainy, Department of Psychology, McGill University, Montréal, Québec, Canada, and Shahed University, Tehran, Iran E. de Villers-Sidani, Department of Neurology and Neurosurgery, Montreal Neurological Institute, McGill University, Montréal, Québec, Canada R.T. Dydew, Centre for Vision Research, York University, Toronto, Ontario, Canada J. Frasnelli, Département de Psychologie, Centre de Recherche en Neuropsychologie et Cognition, Université de Montréal, Montréal, Québec, Canada L.R. Harris, Centre for Vision Research, York University, Toronto, Ontario, Canada D.Y.P. Henriques, Center for Vision Research, and School of Kinesiology and Health Science, York University, Toronto, Ontario, Canada S. Hutchins, BRAMS Laboratory and Department of Psychology, Université de Montréal, Montréal, Québec, Canada J.N. Ingram, Computational and Biological Learning Lab, Department of Engineering, University of Cambridge, Cambridge, United Kingdom H. Jenkin, Centre for Vision Research, York University, Toronto, Ontario, Canada M. Jenkin, Centre for Vision Research, York University, Toronto, Ontario, Canada A. Kral, Department of Experimental Otology, Institute of Audioneurotechnology, Medical University Hannover, Hannover, Germany S. Lacey, Department of Neurology, Emory University, Atlanta, Georgia, USA F. Lepore, Département de Psychologie, Centre de Recherche en Neuropsychologie et Cognition (CERNEC), Université de Montréal, Montréal, Québec, Canada S.G. Lomber, Department of Physiology and Pharmacology, and Department of Psychology, Centre for Brain and Mind, The University of Western Ontario, London, Ontario, Canada w
Deceased
v
vi
L. Malone, Department of Motor Learning Lab, Kennedy Krieger Institute, and Biomedical Engineering Department of Johns Hopkins School of Medicine, Baltimore, Maryland, USA A.A.G. Mattar, Department of Psychology, McGill University, Montréal, Québec, Canada M. Alex Meredith, Department of Anatomy and Neurobiology, School of Medicine, Virginia Commonwealth University, Richmond, Virginia, USA M.M. Merzenich, W.M. Keck Center for Integrative Neuroscience, Coleman Laboratory, Department of Otolaryngology, University of California, San Francisco, and Brain Plasticity Institute, San Francisco, California, USA K.M. Mosier, Department of Radiology, Section of Neuroradiology and Imaging Sciences, Indiana University, Indianapolis, Indiana, USA F.A. Mussa-Ivaldi, Department of Physiology, Department of Physical Medicine and Rehabilitation, Department of Biomedical Engineering, Northwestern University, and Sensory Motor Performance Program, Rehabilitation Institute of Chicago, Chicago, Illinois, USA S.M. Nasir, The Roxelyn and Richard Pepper Department of Communication Sciences and Disorders, Northwestern University, Evanston, Illinois, USA E. Nava, Department of Biological Psychology and Neuropsychology, University of Hamburg, Hamburg, Germany D.J. Ostry, Department of Psychology, McGill University, Montréal, Québec, Canada, and Haskins Laboratories, New Haven, Connecticut, USA E.F. Pace-Schott, Department of Psychology and Neuroscience, and Neuroscience and Behavior Program, University of Massachusetts, Amherst, USA I. Peretz, BRAMS Laboratory and Department of Psychology, Université de Montréal, Montréal, Québec, Canada B. Röder, Department of Biological Psychology and Neuropsychology, University of Hamburg, Hamburg, Germany B.A. Rowland, Department of Neurobiology and Anatomy, Wake Forest School of Medicine, WinstonSalem, North Carolina, USA P.N. Sabes, Department of Physiology, Keck Center for Integrative Neuroscience, University of California, San Francisco, California, USA K. Sathian, Department of Neurology, Department of Rehabilitation Medicine, Department of Psychology, Emory University, Atlanta, and Rehabilitation R&D Center of Excellence, Atlanta VAMC, Decatur, Georgia, USA R.A. Scheidt, Department of Biomedical Engineering, Marquette University, Milwaukee, Wisconsin, USA R.M.C. Spencer, Department of Psychology and Neuroscience, and Neuroscience and Behavior Program, University of Massachusetts, Amherst, USA B.E. Stein, Department of Neurobiology and Anatomy, Wake Forest School of Medicine, WinstonSalem, North Carolina, USA G. Torres-Oviedo, Department of Motor Learning Lab, Kennedy Krieger Institute, and Neuroscience Department of Johns Hopkins School of Medicine, Baltimore, Maryland, USA E. Vasudevan, Department of Motor Learning Lab, Kennedy Krieger Institute, and Neuroscience Department of Johns Hopkins School of Medicine, Baltimore, Maryland, USA P. Voss, Centre de Recherche en Neuropsychologie et Cognition (CERNEC), and International Laboratory for Brain, Music and Sound Research, Université de Montréal, Montréal, Québec, Canada D.M. Wolpert, Computational and Biological Learning Lab, Department of Engineering, University of Cambridge, Cambridge, United Kingdom
Preface Basic neuroscience research over the past several decades has defined many of the central neuronal mechanisms underlying the functioning of the various sensory and motor systems. These systems work together in a cooperative fashion: perception is required to plan action, while movements often serve to acquire desired sensory inputs (e.g., eye movements to capture a visual target of interest). Performance levels reach their apogee in examples ranging from high-performance athletes to concert-level pianists, with both requiring highly refined sensorimotor abilities. There is now considerable interest in how the brain adapts its functioning in health and disease. Extensive progress has been made in understanding how use and disuse influence motor and sensory performance, and the underlying mechanisms of neuronal plasticity responsible for adaptive changes. Increasingly, basic scientists working in these fields are being challenged to translate this scientific knowledge into applications that provide new and innovative methods to restore lost function in humans following injury or disease (e.g., amputation, myopathies, neuropathies, spinal cord lesions or stroke). In recent years, scientists have risen to this challenge, collaborating with engineers and clinicians to help develop novel technologies. These have progressed to the point where devices such as cochlear implants are now commonly used in clinical practice, while applications such as neuroprosthetic devices controlled by the brain are rapidly becoming a realistic possibility for restoring lost motor function. This two-volume set of books is the result of a symposium, inspired by these new initiatives, that was held at the Université de Montréal on May 10–11, 2010 (see http://www.grsnc.umontreal.ca/32s/). It was organized by the Groupe de Recherche sur le Système Nerveux Central (GRSNC) as one of a series of annual international symposia held on a different topic each year. The symposium included presentations by world-renowned experts working on the neuronal mechanisms that play critical roles in learning new motor and sensory skills in both health and disease. The objective was to provide an overview of the various neural mechanisms that contribute to learning new motor and sensory skills as well as adapting to changed circumstances including the use of devices and implants to substitute for lost sensory or motor abilities (neural prosthetics). The symposium emphasized the importance of basic science research as the foundation for innovative technological developments that can help restore function and improve the quality of life for disabled individuals. It equally emphasized how such new technologies can contribute to our basic scientific understanding of the neural mechanisms of sensorimotor control and adaptation. Many of the participants of that meeting have contributed chapters to this book, including symposium speakers and poster presenters. In addition, we invited a number of other well-known experts who could not participate in the conference itself to submit chapters. This two-volume collection of over 30 chapters can only cover a fraction of the topics and extensive range of work that pertains to adapting our motor and sensory systems to changed conditions and to the development of technologies that substitute for lost abilities. However, it does address a range of motor functions and sensory modalities; considers adaptive changes at both behavioral and neurophysiological levels; and presents perspectives on basic research, clinical approaches, and technological innovations. The result is a collection that includes chapters broadly separated into five key themes: (1) mechanisms to enhance vii
viii
motor performance, (2) mechanisms to enhance sensory perception, (3) multisensory interactions to enhance action and perception, (4) assistive technologies to enhance sensorimotor performance, and (5) neurorehabilitation. The current volume (Volume I) focuses on the basic mechanisms underlying performance changes (themes 1–3). Volume II complements this first volume by focussing on the translation of scientific knowledge into technological applications and clinical strategies that can help restore lost function and improve quality of life following injury or disease (themes 4 and 5). The conference and this book would not have been possible without the generous support of the GRSNC, the Fonds de la Recherche en Santé de Québec (FRSQ), the Canadian Institutes of Health Research (CIHR), the Institute of Neuroscience, Mental Health and Addiction (CIHR), the Institute of Musculoskeletal Health and Arthritis (CIHR), and the Faculty of Arts and Sciences and Faculty of Medicine of the Université de Montréal. We gratefully acknowledge these sponsors as well as our contributing authors who dedicated their time and effort to present their perspectives on the neural mechanisms and technological advances that enhance our performance for action and perception. Andrea M. Green C. Elaine Chapman John F. Kalaska Franco Lepore
A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 191 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.
CHAPTER 1
Naturalistic approaches to sensorimotor control James N. Ingram* and Daniel M. Wolpert Computational and Biological Learning Lab, Department of Engineering, University of Cambridge, Cambridge, United Kingdom
Abstract: Human sensorimotor control has been predominantly studied using fixed tasks performed under laboratory conditions. This approach has greatly advanced our understanding of the mechanisms that integrate sensory information and generate motor commands during voluntary movement. However, experimental tasks necessarily restrict the range of behaviors that are studied. Moreover, the processes studied in the laboratory may not be the same processes that subjects call upon during their everyday lives. Naturalistic approaches thus provide an important adjunct to traditional laboratory-based studies. For example, wearable self-contained tracking systems can allow subjects to be monitored outside the laboratory, where they engage spontaneously in natural everyday behavior. Similarly, advances in virtual reality technology allow laboratory-based tasks to be made more naturalistic. Here, we review naturalistic approaches, including perspectives from psychology and visual neuroscience, as well as studies and technological advances in the field of sensorimotor control. Keywords: human sensorimotor control; natural tasks; natural behavior; movement statistics; movement kinematics; object manipulation; tool use.
consider grasping an object such as a coffee cup. In order to reach for the cup, sensory information regarding its three-dimensional location, represented initially by its two-dimensional position on the retina, must be transformed into a motor command that moves the hand from its current location to the location of the cup (Shadmehr and Wise, 2005; Snyder, 2000; Soechting and Flanders, 1992). Similarly, in order to grasp the cup, sensory information regarding its three-dimensional shape must be transformed into a motor command that configures the digits to accommodate the cup
Introduction Sensorimotor control can be regarded as a series of transformations between sensory inputs and motor commands (Craig, 1989; Fogassi and Luppino, 2005; Pouget and Snyder, 2000; Rizzolatti et al., 1998; Shadmehr and Wise, 2005; Snyder, 2000; Soechting and Flanders, 1992). For example, *Corresponding author. Tel.: þ44-1223-748-514; Fax: þ44-1223-332-662 E-mail:
[email protected] DOI: 10.1016/B978-0-444-53752-2.00016-3
3
4
(Castiello, 2005; Castiello and Begliomini, 2008; Santello and Soechting, 1998). Finally, once the cup is grasped, sensory information regarding the dynamics of the cup (such as its mass) must be used to rapidly engage the transformations that will mediate control of the arm–cup combination (Atkeson and Hollerbach, 1985; Bock, 1990, 1993; Lacquaniti et al., 1982). In many cases, the study of sensorimotor control endeavors to understand these transformations, how they are acquired and represented in the brain, how they adapt to new tasks, and how they generalize to novel task variations. When learning a new motor skill, for example, existing sensorimotor transformations may be adapted and new transformations may be learned (Haruno et al., 2001; Miall, 2002; Wolpert et al., 2001; Wolpert and Kawato, 1998). The study of motor learning can thus reveal important details about the underlying transformations (Ghahramani and Wolpert, 1997; Shadmehr, 2004). As such, many laboratory-based studies which examine sensorimotor control use adaptation paradigms in which subjects reach toward visual targets in the presence of perturbations which induce movement errors. In the case of dynamic (force) perturbations, the subject grasps the handle of a robotic manipulandum which can apply forces to the arm (see, e.g., Caithness et al., 2004; Gandolfo et al., 1996; Howard et al., 2008, 2010; Malfait et al., 2002; Shadmehr and Brashers-Krug, 1997; Shadmehr and Mussa-Ivaldi, 1994; Tcheang et al., 2007; Tong et al., 2002). Typically, the forces depend on the kinematics of the movement, such as its velocity, and cause the arm to deviate from the target. Over the course of many trials, the subject adapts to the perturbation and the deviation of the hand reduces. In the case of kinematic perturbations, the position of the subject's hand is measured and, typically, displayed as a cursor on a screen. Subjects reach toward visual targets with the cursor. A transformation (such as a rotation) can be applied to the cursor which perturbs it relative to the veridical position of
the hand (see, e.g., Ghahramani and Wolpert, 1997; Ghahramani et al., 1996; Howard et al., 2010; Kagerer et al., 1997; Krakauer et al., 1999, 2000, 2005). Once again, over the course of many trials, the subject adapts to the perturbation and the deviation of the cursor reduces. These laboratory-based perturbation studies have greatly advanced our understanding of sensorimotor control. However, because they predominantly focus on reaching movements during a limited number of perturbations, they do not capture the full range of everyday human behavior. Here, we present more naturalistic approaches. We begin by reviewing perspectives from psychology and go on to describe a naturalistic approach which has been successful in the study of the visual system. We then review studies which examine human behavior in naturalistic settings, focusing on relevant advances in technology and studies which record movement kinematics during natural everyday tasks. Because object manipulation emerges as an important component of naturalistic behavior in these studies, we finish with a review of object manipulation and tool use. Specifically, we present results from various experimental paradigms including a recent naturalistic approach in which a novel robotic manipulandum (the WristBOT) is used to simulate objects with familiar dynamics.
Naturalistic perspectives from animal psychology The animal psychologist Nicholas Humphrey published a seminal paper in 1976 in which he speculated about the function of intelligence in primates (Humphrey, 1976). The paper begins with a conundrum: how to reconcile the remarkable cognitive abilities that many primates demonstrate in laboratory-based experiments with the apparent simplicity of their natural lives, where food is abundant (literally growing on trees), predators are few, and the only demands are to “eat, sleep, and play.” He asked “What—if it exists—is the natural equivalent of the laboratory
5
test of intelligence?” He reasoned that if an animal could be shown to have a particular cognitive skill in the laboratory, that skill should have some natural application in the wild. He argued that the process of natural selection would not tolerate “needless extravagance” and that “We do not expect to find that animals possess abilities which far exceed the calls that natural living makes upon them.” The same argument could be applied to laboratory-based studies of human sensorimotor control. For example, if we observe that subjects can adapt to a particular perturbation during a controlled laboratory-based task, what does that tell us about the sensorimotor processes that humans regularly call upon during their everyday lives? In Humphrey's case, he answered the question by carefully observing the natural behavior of primates. In the endeavor to understand the human sensorimotor system, the natural behavior of our subjects may also be an important source of information. Whereas Humphrey encourages us to explore the natural everyday expression of the skills and processes we observe during laboratory-based tasks, other animal psychologists would argue that we should question the ecological relevance of the tasks themselves. For example, a particular primate species may fail on a laboratory-based task that is designed to characterize a specific cognitive ability (Povinelli, 2000; Povinelli and Bering, 2002; Tomasello and Call, 1997). In this case, the conclusion would be that the cognitive repertoire of the species does not include the ability in question. However, if the task is reformulated in terms of the natural everyday situations in which the animal finds itself (foraging for food, competing with conspecifics, etc.), successful performance can be unmasked (Flombaum and Santos, 2005; Hare et al., 2000, 2001). This issue of ecological relevance may also apply to the performance of human subjects during the laboratory-based tasks that are designed to study sensorimotor control (Bock and Hagemann, 2010). For example, despite our intuition that humans can successfully learn and
recall a variety of different motor skills and interact with a variety of different objects, experiments have shown that concurrent adaptation to distinct sensorimotor tasks can be difficult to achieve in the laboratory (Bock et al., 2001; Brashers-Krug et al., 1996; Goedert and Willingham, 2002; Karniel and Mussa-Ivaldi, 2002; Krakauer et al., 1999, 2005; Miall et al., 2004; Shadmehr and Brashers-Krug, 1997; Wigmore et al., 2002). However, in natural everyday life, different motor skills are often associated with distinct behavioral contexts. It is thus not surprising that when experiments are made more naturalistic by including distinct contextual cues, subjects can learn and appropriately recall laboratory-based tasks that would otherwise interfere (Howard et al., 2008, 2010; Lee and Schweighofer, 2009; Nozaki and Scott, 2009; Nozaki et al., 2006).
Naturalistic perspectives from human cognitive ethology The importance of a naturalistic approach is also advocated by proponents of human cognitive ethology (Kingstone et al., 2008). Ethology is the study of animal (and human) behavior in natural settings (Eibl-Eibesfeldt, 1989; McFarland, 1999). The emphasis is on the adaptive and ecological significance of behavior, how it develops during the lifetime of the individual, and how it has evolved during the history of the species. It can be contrasted with the approaches of experimental psychology, which focus on laboratorybased tasks rather than natural behavior and largely ignore questions of ecological relevance and evolution (Kingstone et al., 2008). In human cognitive ethology, studies of natural real-world behavior are regarded as an important adjunct to experimental laboratory-based approaches, with some going so far as to argue that they are a necessary prerequisite (Kingstone et al., 2008). An example of this approach is given by Kingstone and colleagues and consists of a pair of studies that examine vehicle steering behavior.
6
In the first study, the natural steering behavior of subjects was measured outside the laboratory in a real-world driving task (Land and Lee, 1994). In the second study, a laboratory-based driving simulator was then used to test a specific hypothesis regarding the sources of information that drivers use for steering (Land and Horwood, 1995). The experimental hypothesis was constrained by the real-world behavior of subjects, as measured in the first study, and the simulated roads were modeled on the real-world roads from which the natural dataset was collected. Kingstone and colleagues argue that all experiments examining human cognition should begin with a characterization of the natural manifestations of the processes involved. They warn against the implicit assumption that a process identified during a controlled laboratory-based task is the same process that is naturally engaged by subjects in the real world (see also Bock and Hagemann, 2010). Learning a novel dynamic perturbation in the laboratory, for example, may be nothing like learning to use a new tool in our home workshop. If we are interesting in the sensorimotor control of object manipulation, asking subjects to grasp the handles of robots that generate novel force fields may provide only partial answers. Ideally, we should also study the natural tool-using behavior of our subjects outside the laboratory, and inside the laboratory, we should ask them to grasp a robotic manipulandum that looks and behaves like a real tool.
A naturalistic approach to the visual system The receptive fields (RFs) of neurons in the visual system have been traditionally defined using simple artificial stimuli (for recent reviews, see Fitzpatrick, 2000; Reinagel, 2001; Ringach, 2004). For example, the circular center-surround RFs of retinal ganglion cells were originally defined using small spots of light (Hartline, 1938; Kuffler, 1953). The same method later revealed similar RFs in the lateral geniculate nucleus
(Hubel, 1960; Hubel and Wiesel, 1961). In contrast, bars of light were found to elicit the largest response from neurons in primary visual cortex (V1) (Hubel and Wiesel, 1959). This finding was pivotal because it provided the first evidence for a transformation of RFs from one visual processing area to the next (Tompa and Sáry, 2010; Wurtz, 2009). A hierarchical view of visual processing emerged, in which the RFs at each level were constructed from simpler units in the preceding level (Carpenter, 2000; Gross, 2002; Gross et al., 1972; Hubel and Wiesel, 1965; Konorski, 1967; Perrett et al., 1987; Tompa and Sáry, 2010). Within this framework, using artificial stimuli to map the RFs at all stages of the visual hierarchy was regarded as essential in the effort to understand vision (Hubel and Wiesel, 1965; Tanaka, 1996; Tompa and Sáry, 2010). However, beyond their role as abstract feature detectors contributing progressively to visual perception, there was little discussion as to why RFs had particular properties (Balasubramanian and Sterling, 2009). In contrast to traditional approaches based on artificial stimuli, the concept of efficient coding from information theory allows the properties of visual RFs to be explained in terms of natural visual stimuli (Barlow, 1961; Simoncelli, 2003; Simoncelli and Olshausen, 2001). Specifically, natural images are redundant due to correlations across both space and time (Simoncelli and Olshausen, 2001; van Hateren, 1992), and efficient coding assumes that the early stages of visual processing aim to reduce this redundancy (Barlow, 1961; van Hateren, 1992). Within such a naturalistic framework, the statistical structure of natural visual images becomes central to understanding RF properties. For example, retinal processing can be regarded as an attempt to maximize the information about the visual image that is transmitted to the brain by the optic nerve (Geisler, 2008; Laughlin, 1987). Consistent with this, center-surround RFs in the retina appear to exploit spatial correlations that exist in natural images (Balasubramanian and Sterling, 2009;
7
Srinivasan et al., 1982). Moreover, the RFs of both simple (Olshausen and Field, 1996) and complex (Földiak, 1991; Kording et al., 2004) cells in V1 appear to be based on an efficient neural representation of natural images. For example, simple cell RFs self-organize spontaneously under a learning algorithm that is optimized to find a sparse code for natural scenes (Olshausen and Field, 1996). Similarly, many features of complex cell RFs self-organize under a learning algorithm that is optimized to find stable responses to natural scenes (Kording et al., 2004). Thus, whereas traditional approaches to the visual system have used artificial stimuli to simply map the structure of visual RFs, a naturalist approach based on natural visual stimuli allows the RFs to be predicted from first principles (Simoncelli and Olshausen, 2001).
Naturalistic approaches to human behavior As reviewed in the previous section, an analysis of the natural inputs to the visual system (natural images) has been highly productive in the study of visual processing. Approaches that record the natural outputs of the sensorimotor system (everyday human behavior) may be similarly informative. Depending on the study, the data collected may include the occurrence of particular behaviors, the kinematics of movements, physical interactions with objects, social interactions with people, or the location of the subject. We briefly review studies and technologies associated with collecting behavioral data from subjects in their natural environment and then review in more detail the studies that specifically record movement kinematics during natural everyday tasks. Studies of human behavior in naturalistic settings have traditionally relied on observation or indirect measures. Examples of the use of observation include a study of human travel behavior which required subjects to keep a 6week travel diary (Schlich and Axhausen, 2003) and a study of the everyday use of the hand which
required an observer to keep a diary of the actions performed by subjects during the observation period (Kilbreath and Heard, 2005). In the case of indirect measures, examples include the use of e-mail logs to examine the statistics of discrete human behaviors (Barabasi, 2005), the use of dollar bill dispersal patterns to examine the statistics of human travel (Brockmann et al., 2006), and monitoring the usage of the computer mouse to examine the statistics of human movement (Slijper et al., 2009). Recently, mobile phones have become an important tool for collecting data relevant to everyday human behavior (Eagle and Pentland, 2006, 2009). For example, large datasets of human travel patterns can be obtained from mobile phones (Anderson and Muller, 2006; González et al., 2008). In addition, mobile phones include an increasing variety of sensors, such as accelerometers, which can be used to collect data unobtrusively from naturally behaving subjects (Ganti et al., 2010; Hynes et al., 2009). This information can be used, for example, to distinguish between different everyday activities (Ganti et al., 2010). Mobile phones can also interface with small wireless sensors worn elsewhere on the body. For example, Nokia has developed a combined three-axis accelerometer and gyroscope motion sensor the size of a wristwatch which can be worn on segments of the body (Györbíró et al., 2009). This combination of accelerometers and gyroscopes has been shown to overcome the problems associated with using accelerometers alone (Luinge and Veltink, 2005; Takeda et al., 2010). The Nokia motion sensors can stream data to the subject's mobile phone via bluetooth, providing kinematic data simultaneously from multiple body segments. In a recent study, this data was used to distinguish between different everyday activities (Györbíró et al., 2009). In general, mobile phone companies are interested in determining the user's behavioral state so that the phone can respond appropriately in different contexts (Anderson and Muller, 2006; Bokharouss et al., 2007; Devlic et al., 2009;
8
Ganti et al., 2010; Györbíró et al., 2009). However, the application of these technologies to naturalistic studies of human behavior is clear (Eagle and Pentland, 2006, 2009). The interaction of subjects with objects in the environment is also an important source of information regarding naturalistic behavior (Beetz et al., 2008; Philipose et al., 2004; Tenorth et al., 2009). For example, by attaching small inexpensive radio-frequency-identification (RFID) tags to the objects in the subject's environment, a instrumented glove can be used to record the different objects used by the subject as they go about their daily routine (Beetz et al., 2008; Philipose et al., 2004). This information can be used to distinguish between different everyday tasks, and can also distinguish different stages within each task (Philipose et al., 2004). A disadvantage of the use of RFID technology is that every object must be physically tagged. An alternative method to track a subject's interactions with objects uses a head-mounted camera and image processing software to extract hand posture and the shape of the grasped object (Beetz et al., 2008).
Naturalistic studies of movement kinematics The kinematics of a subject's movements provide an important source of information for studying sensorimotor control. However, most commercially available motion tracking systems are designed for use inside the laboratory (Kitagawa and Windor, 2008; Mündermann et al., 2006). The majority of studies which examine human movement kinematics are thus performed under laboratory conditions (for recent reviews, see Schmidt and Lee, 2005; Shadmehr and Wise, 2005). In contrast, naturalistic studies of spontaneously behaving humans require mobile, wearable systems which minimally restrict the movements of the subject. As discussed in the previous section, small wireless sensors which can stream kinematic data from multiple
segments of the body to a data logger (such as a mobile phone) may provide one solution (Györbíró et al., 2009; Lee et al., 2010). However, these technologies are not yet widely available. To date, naturalistic studies of movement kinematics have thus used commercial motion tracking systems which have been modified to make them wearable by subjects. These studies, which have examined movements of the eyes, hands, and arms, are reviewed in the following sections.
Eye movements during natural tasks Eye movements are the most frequent kind of movement that humans make, more frequent even than heartbeats (Carpenter, 2000). The oculomotor system has many features which make it an ideal model system for the study of sensorimotor control (Carpenter, 2000; Munoz, 2002; Sparks, 2002). Eye movements are relatively easy to measure (Wade and Tatler, 2005) and the neural circuitry which underlies them is well understood (Munoz, 2002; Sparks, 2002). Moreover, eye movements are intimately associated with the performance of many motor tasks (Ballard et al., 1992; Johansson et al., 2001; Land, 2009; Land and Hayhoe, 2001; Land and Tatler, 2009). They also provide a convenient behavioral marker for cognitive processes including attention (e.g., Corbetta et al., 1998) and decision making (e.g., Gold and Shadlen, 2000; Schall, 2000). It is not surprising, therefore, that a number of studies have examined eye movements during natural everyday tasks (for recent reviews, see Hayhoe and Ballard, 2005; Land, 2006, 2009; Land and Tatler, 2009). The purpose of eye movements (saccades and smooth pursuit, for a recent review, see Krauzlis, 2005) is to move the small high-acuity spotlight of foveal vision to fixate a particular object or location in the visual scene (Land, 1999; Land and Tatler, 2009; Munoz, 2002). As such, tracking the position of the eyes during a task provides a record of what
9
visual information subjects are using and when they obtain it (Ballard et al., 1992; Johansson et al., 2001; Land and Hayhoe, 2001; Land and Tatler, 2009). The importance of this information during the execution of natural everyday tasks has long been recognized (reviewed in Land and Tatler, 2009; Wade and Tatler, 2005). However, early tracking systems required that the head be fixed which limited recordings to sedentary tasks performed within the laboratory (reviewed in Land and Tatler, 2009). These tasks included reading (Buswell, 1920), typing (Butsch, 1932), viewing pictures (Buswell, 1935; Yarbus, 1967), and playing the piano (Weaver, 1943).
(a) Scene camera
Eye camera
More recently, light-weight head-free eye trackers have become available (Wade and Tatler, 2005) allowing the development of wearable, selfcontained systems (Fig. 1a; Hayhoe and Ballard, 2005; Land and Tatler, 2009; Pelz and Canosa, 2001). Typically, these systems include a camera which records the visual scene as viewed by the subject along with a cursor or cross-hair which indicates the point of fixation within the scene. Studies of eye movements during natural tasks have thus moved outside the laboratory where mobile, unrestricted subjects can engage in a wider range of behaviors (Hayhoe and Ballard, 2005; Land, 2006, 2009; Land and Tatler, 2009).
(b)
Scene monitor Recording backpack (c)
Fig. 1. Eye movements during natural tasks. Panel (a) is modified from Hayhoe and Ballard (2005). Copyright (2005), with permission from Elsevier. Panels (b) and (c) are modified from Land and Hayhoe (2001). Copyright (2001), with permission from Elsevier. (a) An example of a wearable eye-tracking system which consists of an eye camera and scene camera which are mounted on light-weight eyewear. A backpack contains the recording hardware. (b) Fixations of a typical subject while making a cup of tea. Notice the large number of fixations on objects relevant to the task (such as the electric kettle) whereas taskirrelevant objects (such as the stove) are ignored. (c) Fixations of a typical subject while making a sandwich.
10
Such behaviors include everyday activities such as driving a car (Land and Lee, 1994; Land and Tatler, 2001), tea making (Fig. 1b; Land and Hayhoe, 2001; Land et al., 1999), sandwich making (Fig. 1c; Hayhoe et al., 2003; Land and Hayhoe, 2001), and hand washing (Pelz and Canosa, 2001). Ball games such as table tennis (Land and Furneaux, 1997), cricket (Land and McLeod, 2000), catch (Hayhoe et al., 2005), and squash (Land, 2009) have also been studied. Two important findings arose from the early studies of natural eye movements during sedentary tasks. First, the pattern of eye movements is dramatically influenced by the specific requirements of the task (Yarbus, 1967), and second, eye movements usually lead movements of the arm and hand by about one second (reviewed in Land and Tatler, 2009). Contemporary laboratory-based studies have confirmed these findings by using tasks specifically designed to capture the essential features of naturalistic behavior (Ballard et al., 1992; Johansson et al., 2001; Pelz et al., 2001). One of the first studies to use this method found that, rather than relying on detailed visual memory, subjects make eye movements to gather information immediately before it is required in the task (Ballard et al., 1992). Subsequently, using the same task, it was found that subjects will even delay movements of the hand until the eye is available (Pelz et al., 2001). Contemporary studies of eye movements during natural everyday tasks have reported similar findings. For example, when subjects make a pot of tea (Land and Hayhoe, 2001; Land et al., 1999), objects are usually fixated immediately before being used in the task, with irrelevant objects being largely ignored (Fig. 1b). A similar pattern is seen during sandwich making (Hayhoe et al., 2003; Land and Hayhoe, 2001) and hand washing (Pelz and Canosa, 2001). The influence of task requirements on eye movements is particularly striking. When subjects passively view natural scenes, they selectively fixate some areas over others based on the “bottom-up” salience
of features in the scene. For example, visual attention is attracted by regions with high spatial frequencies, high edge densities or high contrast (for reviews see Henderson, 2003; Henderson and Hollingworth, 1999). In contrast, when specific tasks are imposed, the pattern of eye movements is driven by the “top-town” requirements of the task (see reviews in Ballard et al., 1992; Land and Tatler, 2009; Land, 2006). For example, while subjects are waiting for the go-signal to begin a particular task, they fixate irrelevant objects with the same frequency as the objects that are relevant to the task (Hayhoe et al., 2003). The number of irrelevant object fixations falls dramatically once the task begins. Before studies of naturalistic eye movements, it had been assumed that subjects used visual information obtained by the eyes to construct a detailed model of the visual world which could be consulted as required during task execution (Ballard et al., 1992). The study of eye movements during natural everyday tasks outside the laboratory and during laboratory-based tasks designed to be naturalistic has shown that rather than rely on memory, subjects use their eyes to obtain information immediately before it is required in the task.
Hand and arm movements during natural tasks The naturalistic studies of eye movements reviewed in the previous section have been made possible by the development of wearable, selfcontained eye-tracking systems (Hayhoe and Ballard, 2005; Land and Tatler, 2009; Pelz and Canosa, 2001). Two recent studies from our group have used wearable, self-contained systems to record hand (Ingram et al., 2008) and arm (Howard et al., 2009a) movements during natural everyday behavior. However, in contrast to studies of eye movements, which have invariably imposed specific tasks on the subject, we allowed our subjects to engage spontaneously in natural everyday behavior.
11
The statistics of natural hand movements Although the 15 joints of the hand can potentially implement 20 degrees of freedom (Jones, 1997; Stockwell, 1981), laboratory-based studies suggest that the effective dimensionality of hand movements is much less (reviewed in Jones and Lederman, 2006). For example, the ability of subjects to move the digits independently is limited (Hager-Ross and Schieber, 2000; Reilly and Hammond, 2000) due to both mechanical (Lang and Schieber, 2004; von Schroeder and Botte, 1993) and neuromuscular (Kilbreath and Gandevia, 1994; Lemon, 1997; Reilly and Schieber, 2003) factors. Moreover, the sensorimotor system is thought to employ synergies which reduce the dimensionality and thereby simplify the control problem (Mason et al., 2001; Santello et al., 1998, 2002; Schieber and Santello, 2004; Tresch et al., 2006). However, these conclusions are based on results from laboratory-based tasks which potentially constrain the variety of hand movements observed. To address this issue using a naturalistic approach, we obtained datasets of spontaneous everyday movements from the right hand of subjects who wore a self-contained motion tracking system (Ingram et al., 2008). The system consisted of an instrumented cloth glove (the commercially available CyberGlove from CyberGlove Systems) and a backpack which contained the data acquisition hardware (Fig. 2a). Subjects were fitted with the system and instructed to go about their normal daily routine. A total of 17 h of data was collected, which consisted of 19 joint angles of the digits sampled at 84 Hz. To estimate the dimensionality of natural hand movements in the dataset, we performed a principal component analysis (PCA) on joint angular velocity (Fig. 2b and c). Consistent with the reduced dimensionality discussed above, the first 10 PCs collectively explained almost all (94%) of the variance (Fig. 2c). Moreover, the first two PCs accounted for more than half of the variance (60%) and were well conserved across subjects.
The first PC explained 40% of the variance and reflected a coordinated flexion (closing) and extension (opening) of the four fingers. The second PC explained an additional 20% of the variance and also involved flexion and extension of the four fingers. Figure 2b shows how these first two PCs combine to produce a large range of hand postures. An important question arising from the current study is whether there are differences between the statistics of hand movements made during laboratory-based tasks and those made during everyday life. Previous studies have performed PCA on angular position data collected during a reach-tograsp task (Santello et al., 1998, 2002). In these previous studies, the first two PCs involved flexion and extension of the fingers and accounted for 74% of the variance. When the same analysis was repeated on our dataset, the first two PCs also involved finger flexion/extension and accounted for 70% of the variance. This similarity with previous laboratory-based studies suggests that reach-to-grasp movements and object manipulation form an important component of the natural everyday tasks performed by the hand. Consistent with this, 60% of the natural use of the hands involves grasping and manipulating objects (Kilbreath and Heard, 2005). Many previous laboratory-based studies have examined the independence of digit movements, showing that the thumb and index finger are moved relatively independently, whereas the middle and ring fingers tend to move together with the other digits (Hager-Ross and Schieber, 2000; Kilbreath and Gandevia, 1994). We quantified digit independence in our natural dataset by determining the degree to which the movements of each digit (the angular velocities of the associated joints) could be linearly predicted from the movements (angular velocities) of the remaining four digits. This measure was expressed as the percentage of unexplained variance (Fig. 2d) and was largest for the thumb, followed by the index finger, then the little and middle fingers, and was smallest for the ring
12 (b)
(a) Recording backpack
PC 2
Instrumented LED glove indicator
PC 1 (d)
(e)
90 80 70 60 50 40
80 60 40 20 0
30 2
4
6
8 10 12 14 16 18
(f) 100
% Variance explained
100 % Variance unexplained
% Variance explained
100
100 % Variance explained
(c)
80 60 40 20 0
T
I
M
R
L
80 60 40 20 0
T
I
M
R
L
T
I
M
R
L
PC number
Fig. 2. The statistics of natural hand movements. Panels (b) through (f) are reprinted from Ingram et al. (2008). Used with permission. (a) The wearable motion tracking system consisted of an instrument cloth glove (the CyberGlove from CyberGlove Systems) which measured 19 joint angles of the digits. A backpack contained the recording hardware. Subjects were told to go about their normal daily routine and return when the LED indicator stopped flashing. (b) The first two principal components (PC) explained 60% of the variance in joint angular velocity and combine to produce a range of hand postures. (c) The percent variance explained by increasing numbers of principal components. The first 10 PCs accounted for 94% of the variance in joint angular velocity. (d) The percent variance in angular velocity which remained unexplained for each digit after a linear reconstruction which was based on data from the other four digits (T ¼ Thumb, I ¼ Index, M ¼ Middle, R ¼ Ring, L ¼ Little). (e) The percent variance in angular velocity which was explained by a linear reconstruction which paired the thumb individually with the other digits. The gray 100% bar indicated self-pairing. (f) The percent variance explained for digit pairs involving the little finger, plotted as in (e).
finger. Interestingly, this pattern of digit independence was correlated with results from several previous studies, including the number of cortical sites encoding movement of each digit (Penfield and Broldrey, 1937) and a laboratory-based measure of the ability of subjects to move each digit individually (Hager-Ross and Schieber, 2000).
We also quantified coupling between pairs of digits, applying the linear reconstruction method separately to each digit paired separately with the other four digits. This measure was expressed as the percent variance that was explained. Results for the thumb (Fig. 2e) show that its movements are very difficult to predict based on
13
movements of the fingers. Results for the fingers show that the best linear reconstructions (highest coupling) are based on the movements of the immediately neighboring fingers, decreasing progressively with increasing distance (Fig. 2f shows this pattern for the little finger). Thus, whereas the thumb moves independently of the fingers, the movements of a given finger are more or less related to neighboring fingers based on the topological distance between them. These results from a naturalistic study of hand movements have generally supported those obtained from previous laboratory-based studies. However, because previous studies have employed a limited number of experimental tasks, it is important to verify their conclusions in the natural everyday behavior of subjects. Specifically, we have verified the pattern of digit independence in the everyday use of the hand and shown that many aspects of natural hand movements have been well characterized by laboratory-based studies in which subjects reach to grasp objects.
The statistics of natural arm movements Many laboratory-based studies have examined the ability of subjects to make bimanual movements with particular phase relations (Kelso, 1984, 1995; Li et al., 2005; Mechsner et al., 2001; Schmidt et al., 1993; Swinnen et al., 1998, 2002). Results indicate that not all phase relations are equally easy to perform. At a low frequency of movement, both symmetric movements (phase difference between the two arms of 0 ) and antisymmetric movements (phase difference of 180 ) are easy to perform, whereas movements with intermediate phase relations are more difficult. At higher frequencies, only symmetric movements can be performed easily and all other phase relations tend to transition to the symmetric mode (Tuller and Kelso, 1989; Wimmers et al., 1992). This “symmetry bias” has been extensively studied in laboratory-based
experiments and there has been much debate regarding its significance and underlying substrate (e.g., Mechsner et al., 2001; Treffner and Turvey, 1996). Its relevance to the everyday behavior of subjects, however, is not clear. To address this issue using a naturalistic approach, we obtained datasets of spontaneous everyday arm movements of subjects who wore a self-contained motion tracking system (Howard et al., 2009a). We hypothesized that the symmetry bias would be reflected in the natural statistics of everyday tasks. Electromagnetic sensors (the commercially available Liberty system from Polhemus) were attached to the left and right arms and the data acquisition hardware was contained in a backpack (Fig. 3a). Subjects were fitted with the system and instructed to go about their normal routine. A total of 31 h of data was collected, which consisted of the position and orientation of the sensors on the upper and lower segments of the left and right arms sampled at 120 Hz. We analyzed the phase relations between flexion/extension movements of the right and left elbow, calculating the natural incidence of different phase relations for a range of movement frequencies (Fig. 3b and c). At low movement frequencies, the distribution of phase incidence was bimodal with peaks for both symmetric and antisymmetric movements (see also Fig. 3d). At higher movement frequencies, phase incidence became unimodal and was dominated by symmetric movements. The progression of phase incidence from a bimodal to a unimodal distribution as movement frequency increases can be seen in Fig. 3b. These results provide an important adjunct to laboratory-based studies because they show that the symmetry bias is expressed in the natural everyday movements of subjects. The coordinate system in which the symmetry bias is expressed is an important issue which has been examined in laboratory-based studies (Mechsner et al., 2001; Swinnen et al., 1998). If the symmetry bias is expressed only in joint-based (intrinsic) coordinates (Fig. 3c), it may be a property of sensorimotor control or the
(a)
(b)
3.2–6.4
0.11 0.10
Transmitter
0.09 0.08 1.2–2.4 0.07
Relative incidence
Frequency bands (Hz)
Recording backpack
0.06 0.05
SR1 SR2
0.04
0.45–0.9 0
90
180
270
360
Phase relation (⬚)
(c)
(d)
(g)
qL
25 Lab task error (⬚)
qR
Relative incidence
Elbow angles (intrinsic coordinates) 0.10 0.08 0.06 0.04 0
90
180
270
20 15 10 5 0
360
Phase relation (⬚)
(e)
Lab task error (deg)
Relative incidence
0.08
0.14
(h) Wrist position (extrinsic coordinates)
cL
0.05
Natural incidence
(f)
cR
Low frequency
0.10 0.08 0.06 0.04 0
90
180
270
Phase relation (⬚)
360
120
High frequency
100 80 60 40 20 0.05
0.08
0.14
Natural incidence
Fig. 3. The statistics of natural arm movements. Panels (b) through (h) are reprinted from Howard et al. (2009a). Used with permission. (a) The wearable motion tracking system consisted of small electromagnetic sensors and a transmitter (the Liberty system from Polhemus). A backpack contained the recording hardware. The sensors were attached to the upper and lower segments of the left and right arms as shown (SR1 ¼ right upper, SR2 ¼ right lower; left arm and sensors not shown). (b) Distributions of relative phases between left and right elbow joint angles. Note bimodal distribution at low movement frequencies consisting of both symmetric (0 /360 ) and antisymmetric (180 ) phases and unimodal distribution at higher frequencies consisting of symmetric phase only. (c) Elbow angles represent an intrinsic coordinate system for representing movements of the arms. Symmetric movements are shown by homogenous left/right pairings of arrow heads (left filled with right filled or left open with right open). Antisymmetric movements are shown by heterogeneous left/right pairings of arrow heads (left filled with right open or right open with left filled). (d) Relative incidence of different phase relations at low frequencies for natural movements represented in intrinsic coordinates (as shown in (c)). (e) Wrist positions in Cartesian coordinates represent an extrinsic coordinate system for representing movements of the arms. Symmetric and antisymmetric movements are shown as in (c). (f) Relative incidence of different phase relations at low frequencies for natural movements represented in extrinsic coordinates (as shown in (e)). (g) Error during the low frequency laboratory-based tracking task for different phase relations plotted against log of the natural incidence of those phase relations. (h) Error during the high-frequency laboratory-based tracking task for different phase relations plotted against log of the natural incidence of those phase relations.
15
musculoskeletal system. However, if the symmetry bias is expressed in external (extrinsic) coordinates (Fig. 3e), it may be a property of the naturalistic tasks which humans regularly perform. For example, bimanual object manipulation is frequent during everyday life (Kilbreath and Heard, 2005) and imposes particular constraints on movements expressed in extrinsic coordinates (Howard et al., 2009a). Specifically, moving the hands together (or apart) to bimanually grasp (or release) an object requires antisymmetric movements, whereas transporting an object once it is grasped requires symmetric movements. If the constraints of bimanual object manipulation are important, then the symmetry bias should be more pronounced for movements expressed in extrinsic coordinates (relevant to the object). To examine this issue, we compared the phase incidence of natural movements defined in intrinsic space (elbow joint angle; Fig. 3c and d) with those defined in extrinsic space (the Cartesian position of sensors on the wrist; Fig. 3e and f). The distribution of phase incidence was bimodal in both cases. However, the incidence of 180 phase was much higher for the movements defined in extrinsic space, occurring as frequently in this case as symmetric movements. This suggests that natural everyday tasks are biased toward both symmetric and antisymmetric movements of the hands in extrinsic space, consistent with the constraints of bimanual object manipulation. An interesting question concerns the relationship between the level of performance on a particular task and the frequency with which that task is performed. It is well known that training improves performance, but with diminishing returns as the length of training increases (Newell and Rosenbloom, 1981). Specifically, relative performance is often related to the log of the number of training trials. This logarithmic dependence applies to a wide range of cognitive tasks including multiplication, visual search, sequence learning, rule learning, and mental rotation (Heathcote et al., 2000). We examined this issue by comparing the incidence of movement phases in
the natural movement dataset with performance on a laboratory-based bimanual tracking task. Subjects tracked two targets (one with each hand) which moved sinusoidally with various phase relations during a low- and high-frequency condition. The performance error on the task was negatively correlated with the log of the phase incidence of natural movements at both low (Fig. 3g) and high (Fig. 3h) frequencies. This demonstrates that the logarithmic training law holds between the natural incidence of everyday movements and performance on a laboratorybased task.
Naturalistic approaches to object manipulation In previous sections, object manipulation emerged as a key feature of naturalistic human behavior. For example, during everyday life, humans spend over half their time (60%) grasping and manipulating objects (Kilbreath and Heard, 2005). Not surprisingly, the statistics of natural hand (Ingram et al., 2008) and arm (Howard et al., 2009a) movements are also consistent with grasping and manipulating objects. Moreover, eye movements during natural tasks are dominated by interactions with objects (Land and Tatler, 2009). Studies which examine object manipulation should thus form an important component of naturalistic approaches to human sensorimotor control.
An ethology of human object manipulation The ability to manipulate objects and use them as tools constitutes a central theme in the study of human biology. For example, the sensorimotor development of human infants is divided into stages which are characterized by an increasing repertoire of object manipulation and tool-using skills (Case, 1985; Parker and Gibson, 1977; Piaget, 1954). Infants begin with simple prehension and manipulation of objects between
16
4–8 months and finally progress to the insightful use of objects as tools by 12–18 months. This first evidence for tool use is regarded as a milestone in human development (Case, 1985; Piaget, 1954). Similarly, the first evidence for tool use in the archeological record (2.5 million years ago) is regarded as a milestone in human evolution (Ambrose, 2001; Parker, 1974). The long evolutionary history of tool use by humans and their ancestors is thought to have influenced the dexterity of the hand (Marzke, 1992; Napier, 1980; Tocheri et al., 2008; Wilson, 1998) and the size and complexity of the brain (Ambrose, 2001; Wilson, 1998). Indeed, the oldest stone tools, although simple, required significant sensorimotor skill to use and manufacture (Pelegrin, 2005; Roche et al., 1999; Schick et al., 1999; Stout and Semaw, 2006; Toth et al., 1993). Object manipulation and tool use is also an important diagnostic feature for comparative studies of animal behavior, especially those comparing the sensorimotor and cognitive skills of humans with other primates (Parker and Gibson, 1977; Torigoe, 1985; Vauclair, 1982, 1984; Vauclair and Bard, 1983). It is known, for example, that a number of animals regularly use and even manufacture tools in their natural environments (Anderson, 2002; Brosnan, 2009; Goodall, 1963, 1968). However, the human ability and propensity for tool use far exceeds that observed in other animals (Boesch and Boesch, 1993; Povinelli, 2000; Schick et al., 1999; Toth et al., 1993; Vauclair, 1984; Visalberghi, 1993). Object manipulation is mediated by a number of interacting processes in the brain including visual object recognition (Wallis and Bulthoff, 1999), retrieval of semantic and functional information about the object (Johnson-Frey, 2004), encoding object shape for effective grasping (Castiello, 2005; Castiello and Begliomini, 2008; Santello and Soechting, 1998), and incorporating the object into the somatosensory representation of the body (Cardinali et al., 2009; Maravita and Iriki, 2004). Object manipulation also represents a challenge for sensorimotor control because
grasping an object can dramatically change the dynamics of the arm (Atkeson and Hollerbach, 1985; Bock, 1990; Lacquaniti et al., 1982). Thus, to continue moving skillfully after grasping an object, the motor commands must adapt to the particular dynamics of the object (Atkeson and Hollerbach, 1985; Bock, 1990, 1993; Johansson, 1998; Lacquaniti et al., 1982). This process is thought to be mediated by internal models of object dynamics (Flanagan et al., 2006; Wolpert and Flanagan, 2001), and a great deal of research has been devoted to understanding how internal models are acquired and represented and how they contribute to skillful object manipulation. This research has employed three main experimental approaches which are reviewed in the following sections. The first approach involves tasks in which subjects manipulate real physical objects. The remaining two approaches involve the use of robotic manipulanda to simulate virtual objects. As described below, the use of virtual objects removes the constraints associated with physical objects because the dynamics and visual feedback are under computer control.
Physical objects with familiar dynamics Laboratory-based experiments in which subjects interact with physical objects that have familiar dynamics allow the representations and skills associated with everyday object manipulation to be examined. As reviewed previously, the ability to perform skilled movements while grasping an object requires the rapid adaptation of the motor commands that control the arm to account for the dynamics associated with the grasped object. The efficacy of this process can be observed in the first movement subjects make immediately after grasping a heavy object. If the mass of the object is known, the kinematics of the first movement made with the object are essentially identical to previous movements made without it (Atkeson and Hollerbach, 1985; Lacquaniti et al., 1982). If the mass is unknown, subjects adapt rapidly
17
before the first movement is finished (Bock, 1990, 1993). Rapid adaptation is also observed when subjects grasp an object in order to lift it (reviewed in Johansson, 1998). In this case, subjects adapt both the forces applied by the digits to grasp the object (the grip force) and the forces applied by the arm to lift it. When lifting an object that is heavier or lighter than expected, for example, subjects adapt their grip force to the actual mass within just a few trials (Flanagan and Beltzner, 2000; Gordon et al., 1993; Johansson and Westling, 1988; Nowak et al., 2007). Subjects also use visual and haptic cues about the size of the object to estimate the grip force applied during lifting (Gordon et al., 1991a,b,c). For familiar everyday objects, subjects can generate appropriate forces on the very first trial (Gordon et al., 1993). Rapid adaptation is also observed when subjects lift a visually symmetric object which has an asymmetrically offset center of mass (Fu et al., 2010; Salimi et al., 2000; Zhang et al., 2010). In this case, subjects predictively generate a compensatory torque at the digits to prevent the object from tilting, a response which develops within the first few trials (Fu et al., 2010). This ability of subjects to rapidly adapt when grasping an object suggests that the sensorimotor system represents the dynamics of objects. Further evidence that subjects have knowledge of object dynamics comes from experiments which examine the perceptual abilities referred to as dynamic touch. Dynamic touch is the ability to perceive the properties of an object based on the forces and torques experienced during manipulation (Gibson, 1966; Turvey, 1996). In a typical experiment, subjects are required to perceive a particular object property after manipulating it behind a screen which occludes vision (reviewed in Turvey, 1996). For example, subjects can use dynamic touch to perceive both the length of a cylindrical rod (Solomon and Turvey, 1988) and the position along the rod at which they grasp it (Pagano et al., 1994). If the rod has a right-angle segment attached to its distal end (to make an elongated “L” shape), subjects can perceive the
orientation of the end segment (Pagano and Turvey, 1992; Turvey et al., 1992). These abilities suggest that subjects extract information from the relationship between the movements they make with an object (the kinematics) and the associated forces and torques. By combining information from dynamic touch with visual information, the perception of object properties can be made more precise (Ernst and Banks, 2002). Thus, both dynamic touch and vision are likely to contribute during naturalistic object manipulation.
Simulated objects with unfamiliar dynamics The range of experimental manipulations available during tasks that use physical objects is limited. The dynamics are constrained to rigid body physics and the precise control of visual feedback is difficult. An extensively used approach which addresses these limitations uses robot manipulanda to simulate novel dynamics combined with display systems to present computer-controlled visual feedback (see reviews in Howard et al., 2009b; Wolpert and Flanagan, 2010). In these experiments, the subject is seated and grasps the handle of a robotic manipulandum which can apply state-dependent forces to the hand. In many of these experiments, the forces depend on the velocity of the hand and are rotated to be perpendicular to the direction of movement (Caithness et al., 2004; Gandolfo et al., 1996; Howard et al., 2008, 2010; Malfait et al., 2002; Shadmehr and Brashers-Krug, 1997; Shadmehr and Mussa-Ivaldi, 1994; Tcheang et al., 2007; Tong et al., 2002). Visual targets are presented using the display system and subjects make reaching movements to the targets from a central starting position. In the initial “null” condition, the motors of the robot are turned off. In this case, subjects have no difficulty reaching the targets and make movements which are approximately straight lines. When the force field is turned on, movement paths are initially perturbed in the direction of the field. Over many trials, the
18
movements progressively return to their original kinematic form as subjects adapt to the perturbing dynamics. This progressive adaptation can be shown to be associated with the acquisition of an internal model of the dynamics. If the force field is unexpectedly turned off, for example, movement paths are perturbed in the opposite direction. This is because subjects generate the forces they expect, based on their acquired internal model of the perturbing dynamics. Dynamic perturbation studies have provided detailed information about the processes of sensorimotor adaptation and the associated representations of dynamics. However, the applicability of the results to everyday object manipulation is not clear (Lackner and DiZio, 2005). In some respects, the learned dynamics appear to be associated with an internal model of a grasped object (Cothros et al., 2006, 2009). In other respects, the learned dynamics appear to be associated with an internal model of the arm (Karniel and Mussa-Ivaldi, 2002; Malfait et al., 2002; Shadmehr and Mussa-Ivaldi, 1994). Moreover, the majority of studies have examined adaptation to novel dynamics, which occurs over tens or hundreds of trials. In contrast, as reviewed in the previous section, humans adapt to the familiar dynamics of objects they encounter during everyday life within just a few trials. In addition, the robotic devices used in most studies generate only translational forces that depend only on the translational kinematics of the hand. In contrast, naturalistic objects generate both translational forces and rotational torques that depend on the translational and rotational kinematics of the object (as well as its orientation in external space). In the next section, an approach which addresses these issues is presented.
Simulated objects with familiar dynamics Robot manipulanda can be used to simulate objects with familiar dynamics (see review in Wolpert and Flanagan, 2010), thereby combining
aspects from the two approaches reviewed above. This allows the processes associated with naturalistic object manipulation to be examined, without the constraints imposed by the physics of realworld objects. However, only a relatively small number of studies have used this approach. For example, the coordination of grip force has been examined during bimanual manipulation of a simulated object. In this case, the dynamics could be coupled or uncoupled between the left and right hands, allowing the effect of object linkage to be examined (White et al., 2008; Witney and Wolpert, 2003; Witney et al., 2000). When the dynamics were coupled, the object behaved like a single object that was grasped between the two hands (see also Howard et al., 2008). Grip force modulation has also been examined using a simulated object which is grasped between the thumb and index finger (Mawase and Karniel, 2010). In this case, the study replicated the object lifting task used in the many grip force studies reviewed above, but with the greater potential for experimental control offered by a simulated environment. Recently, we have taken a different approach by developing a novel planar robotic manipulandum (the WristBOT; Fig. 4a) which includes rotational torque control at the vertical handle (Howard et al., 2009b). Combined with a virtual reality display system, this allows us to simulate the dynamics and visual feedback of an object which can be rotated and translated in the horizontal plane (Howard et al., 2009b; Ingram et al., 2010). The object resembles a small hammer (Fig. 4b), and consists of a mass on the end of a rigid rod. Subjects manipulate the object by grasping the handle at the base of the rod (Fig. 4b). Rotating the object generates both a torque and a force. The torque depends on the angular acceleration of the object. The force can be derived from two orthogonal components. The first and major component (the tangential force) is due to the tangential acceleration of the mass and is always perpendicular to the rod. The second and minor component (the
19 (a)
Drive system cables
(b)
m
r Elbow pulley
a t F Grasp point
Handle pulley (c)
(d) 180 Respond with direction
Rotate object (5 s)
Response angle (⬚)
0° 90
0
−90 −180 −180
0 90 −90 Object orientation (⬚)
180
Fig. 4. The WristBOT robotic manipulandum, simulated object, and haptic discrimination task. Panel (a) is reprinted from Ingram et al. (2010). Copyright (2010), with permission from Elsevier. Panels (b) through (d) are reprinted from Howard et al. (2009b). Copyright (2009), with permission from Elsevier. (a) The WristBOT is a modified version of the vBOT planar two-dimensional robotic manipulandum. It includes an additional degree of freedom allowing torque control around the vertical handle. Cables and pulleys (only two of which are shown) implement the transmission system between the handle and the drive system at the rear of the manipulandum (not shown). (b) The dynamics of the virtual object were simulated as a point mass (mass m) on the end of a rigid rod (length r) of zero mass. Subjects grasped the object at the base of the rod. When rotated clockwise (as shown), the object generated a counter-clockwise torque (t) due to the angular acceleration (a) of the object. The object also generated a force (F) due to the circular motion of the mass. At the peak angular acceleration, the force was perpendicular to the rod, as shown. Importantly, the orientation of the force changes with the orientation of the object. (c) The haptic discrimination task required subjects to rotate the object for 5 s and then make a movement toward the perceived direction of the mass. The object was presented at a different orientation on every trial. Visual feedback was withheld. (d) Response angle (circular mean and circular standard error) across subjects plotted against actual orientation of the object. Solid line shows circular linear fit to subject responses and dashed line shows perfect performance.
centripetal force) is due to the circular velocity of the mass and acts along the rod toward the center of rotation. Simulations demonstrated that the
peak force acts in a direction that is close to perpendicular to the rod. Thus, as subjects rotate the object, the force experienced at the handle
20
will perturb the hand in a direction that depends on the orientation of the object. In the following sections, we review two recent studies which have used this simulated object.
Haptic discrimination task The direction of the forces associated with rotating an object provides a potential source of information regarding its orientation (or rather, the orientation of its center of mass). Previous studies of haptic perception have used physical objects and have suggested that subjects use torque to determine the orientation of the principal axis of the object (Pagano and Turvey, 1992; Turvey, 1996; Turvey et al., 1992). Specifically, the smallest torque is associated with rotating the object around its principal axis. We used a simulated haptic discrimination task (Fig. 4c) to determine if subjects can also use force direction to perceive object orientation (Howard et al., 2009b). In the case of our simulated object, force direction was the only source of information because torque is independent of orientation when rotating around a fixed axis. Subjects first rotated the simulated object back and forth for 5 s in the absence of visual feedback and then indicated the orientation of the object by making a movement toward the perceived location of the center of mass. Results showed that subjects could accurately perceive the orientation of the object based on its simulated dynamics (Fig. 4d). This suggests that the forces associated with rotating an object are an important source of information regarding object orientation.
Object manipulation task To examine the representation of dynamics associated with familiar everyday objects, we developed a manipulation task that required subjects to rotate the simulated object while keeping its handle stationary (Ingram et al.,
2010). The visual orientation and dynamics of the object could be varied from trial to trial (Fig. 5a). To successfully perform the task, subjects had to generate a torque to rotate the object as well as a force to keep the handle stationary. As described above, the direction of the force depends on the orientation of the object (see Fig. 4b). In the first experiment, the object was presented at different visual orientations (see inset of Fig. 5a). Subjects experienced the torque as they rotated the object, but not the forces. Instead, the manipulandum simulated a stiff spring which clamped the handle in place. This allowed us to measure the anticipatory forces produced by subjects in the absence of the forces normally produced by the object. Results showed that subjects produce anticipatory forces in directions that were appropriate for the visual orientation of the object (Fig. 5b). That is, subjects produce forces that are directed to oppose the forces they expect the object to produce. Importantly, subjects do this before they have experienced the full dynamics of the object, providing evidence that they have a preexisting representation of the dynamics that can be recalled based on visual information. In subsequent experiments, we examined the structure of this representation, how it adapted when exposed to the dynamics of a particular object, and how it was modulated by the visual orientation of the object. In a second experiment, we examined the time course of adaptation (Fig. 5c). Subjects first experienced the object with the forces normally generated by its dynamics turned off. After they had adapted to this zero-force object (preexposure phase in Fig. 5c), the forces were unexpectedly turned on. Although this caused large deviations of the handle on the first few trials, these errors rapidly decreased over subsequent trials as subjects adapted the magnitude of their forces to stabilize the object (exposure phase in Fig. 5c). After many trials of exposure to the normal dynamics of the object, the forces associated with rotating the object were again turned off (postexposure phase in
21 (a)
(b)
Mirror
267 Peak force angle (⬚)
0° −45°
y
−90°
x
−135° 180°
222 177 132 87 −180
Peak displacement (cm)
Exposure (full dynamics)
Pre-exposure (zero force)
1.2
0
(d)
Post-exposure (zero force)
0.8 Peak displacement (cm)
(c)
−135 −90 −45 Object orientation (⬚)
1.0 0.8 0.6 0.4
Training Transfer
0.6
0.4
0.2 1
48 49
100
272
Trials
320
−180
−90 −45 Object orientation (⬚)
0
Fig. 5. The representation of familiar object dynamics. Panel (a) (modified) and panels (b) and (d) (reprinted) are from Ingram et al. (2010). Copyright (2010), with permission from Elsevier. (a) Top view of subject showing visual feedback of the object projected over the hand. The mirror prevents subject from seeing either their hand or the manipulandum. Dashed line shows subject's midline. Inset shows the object presented at different visual orientations. (b) The angle of the peak force produced by subjects as they rotate the object (circular mean and circular standard error) plotted against the visual orientation of the object. The dashed line shows perfect performance. (c) Peak displacement of the handle of the object plotted against trial number. Peak displacement increases when the forces associated with rotating the object are unexpectedly turned on (exposure), decreasing rapidly over the next few trials to an asymptotic level. Peak displacement increases again when the forces are unexpectedly turned off (postexposure), decreasing rapidly to preexposure levels. (d) Peak displacement plotted against the orientation of the object. Subjects experience the full dynamics of the object at the training orientation (square) and are presented with a small number of probe trials at transfer orientations (circles) with the forces turned off. Peak displacement is a measure of the forces subjects produce as they rotate the object. The largest forces (displacements) are produced at the training orientation and decrease progressively as the orientation of the object increases relative to the training orientation. Solid line shows the mean of a Gaussian fit individually to each subject (mean standard deviation of Gaussian fit ¼ 34 ).
Fig. 5c). This initially caused large deviations of the handle, due to the large forces that subjects had learned to produce during the exposure phase. Once again, these errors rapidly decreased over subsequent trials as subjects adapted the magnitude of their forces to be appropriate for
the zero-force object. Importantly, these results show that the rapid adaptation characteristic of manipulating everyday objects can also occur when subjects manipulate simulated objects, provided the dynamics are familiar (see also Witney et al., 2000).
22
In a third experiment, we presented subjects with objects of three different masses to examine how this experience would influence the magnitude of the forces they produced. As expected, subjects adapted the force magnitude according to the mass of the object. Similar results have been obtained for grip force when subjects lift objects of varying mass (Flanagan and Beltzner, 2000; Gordon et al., 1993; Johansson and Westling, 1988; Nowak et al., 2007). The adaptation of force magnitude was further examined in a fourth experiment which examined generalization. Studies of generalization can reveal important details of how dynamics are represented (Shadmehr, 2004). Subjects experienced the object at a single training orientation after which force magnitude was examined at five visual orientations, including four novel orientations where the object had not been experienced. We observed a Gaussian pattern of generalization, with the largest forces produced at the training orientation, decreasing progressively as the orientation increased relative to the training orientation (Fig. 5d). Results from this experiment are consistent with multiple local representations of object dynamics because a single general representation would predict perfect generalization. In summary, using a novel robotic manipulation to simulate a familiar naturalistic object, we have shown that subjects have a preexisting representation of the associated dynamics. Subjects can recall this representation based on vision of the object and can use it for haptic perception when visual information is not available. During manipulation, adaptation of the representation to a particular object is rapid, consistent with many previous studies in which subjects manipulate physical objects. Adaptation is also context specific, being locally confined to the orientation at which the object is experienced. These results suggest that the ability to skillfully manipulate everyday objects is mediated by multiple rapidly adapting representations which capture the local dynamics associated with specific object contexts.
Conclusion The methods of sensorimotor neuroscience have traditionally involved the use of artificial laboratory-based tasks to examine the mechanisms that underlie voluntary movement. In the case of visual neuroscience, the adoption of more naturalistic approaches has involved a shift from artificial stimuli created in the laboratory to natural images taken from the real world. Similarly, the adoption of more naturalistic approaches in sensorimotor neuroscience will require a shift from artificial laboratory-based tasks to natural tasks that are representative of the everyday behavior of subjects. Fortunately, continuing advances in motion tracking, virtual reality and even mobile phone technology are making this shift ever more tractable. In the case of visual neuroscience, naturalistic approaches have required new analytical methods from information theory, statistics, and engineering and have led to new theories of sensory processing. Similarly, naturalistic approaches to human sensorimotor control will almost certainly require new analytical techniques, especially with regard to large datasets of natural behavior and movement kinematics. However, we expect that these efforts will be productive.
Acknowledgments We thank Randy Flanagan, Ian Howard, and Konrad Körding for their collaboration on various projects reviewed herein. This work was supported by the Wellcome Trust. References Ambrose, S. H. (2001). Paleolithic technology and human evolution. Science, 291, 1748–1753. Anderson, J. R. (2002). Gone fishing: Tool use in animals. Biologist (London), 49, 15–18. Anderson, I., & Muller, H. (2006). Practical context awareness for GSM cell phones. In 2006 10th IEEE international symposium on wearable computers, Montreux, Switzerland: IEEE, pp. 126–127.
23 Atkeson, C., & Hollerbach, J. (1985). Kinematic features of unrestrained vertical arm movements. The Journal of Neuroscience, 5, 2318–2330. Balasubramanian, V., & Sterling, P. (2009). Receptive fields and functional architecture in the retina. The Journal of Physiology, 587, 2753–2767. Ballard, D. H., Hayhoe, M. M., Li, F., Whitehead, S. D., Frisby, J. P., Taylor, J. G., et al. (1992). Hand-eye coordination during sequential tasks [and discussion]. Philosophical Transactions of the Royal Society B: Biological Sciences, 337, 331–339. Barabasi, A. (2005). The origin of bursts and heavy tails in human dynamics. Nature, 435, 207–211. Barlow, H. B. (1961). Possible principles underlying the transformation of sensory messages. In W. Rosenblith (Ed.), Sensory Communication (pp. 217–234). Cambridge, MA: M.I.T. Press. Beetz, M., Stulp, F., Radig, B., Bandouch, J., Blodow, N., Dolha, M., et al. (2008). The assistive kitchen—A demonstration scenario for cognitive technical systems. In 2008 17th IEEE international symposium on robot and human interactive communication, (Vols. 1 and 2, pp. 1–8). New York: IEEE. Bock, O. (1990). Load compensation in human goal-directed arm movements. Behavioural Brain Research, 41, 167–177. Bock, O. (1993). Early stages of load compensation in human aimed arm movements. Behavioural Brain Research, 55, 61–68. Bock, O., & Hagemann, A. (2010). An experimental paradigm to compare motor performance under laboratory and under everyday-like conditions. Journal of Neuroscience Methods, 193, 24–28. Bock, O., Schneider, S., & Bloomberg, J. (2001). Conditions for interference versus facilitation during sequential sensorimotor adaptation. Experimental Brain Research, 138, 359–365. Boesch, C., & Boesch, H. (1993). Diversity of tool use and tool making in wild chimpanzees. In A. Berthelet & J. Chavaillon (Eds.), The use of tools by human and nonhuman primates (pp. 158–174). Oxford: Oxford University Press. Bokharouss, I., Wobcke, W., Chan, Y. W., Limaru, A., & Wong, A. (2007). A location-aware mobile call handling assistant. In 21st international conference on advanced networking and applications workshops/symposia, Vol. 2, proceedings, Los Alamitos: IEEE Computer Society, pp. 282–289. Brashers-Krug, T., Shadmehr, R., & Bizzi, E. (1996). Consolidation in human motor memory. Nature, 382, 252–255. Brockmann, D., Hufnagel, L., & Geisel, T. (2006). The scaling laws of human travel. Nature, 439, 462–465. Brosnan, S. F. (2009). Animal behavior: The right tool for the job. Current Biology, 19, R124–R125.
Buswell, G. T. (1920). An experimental study of the eye-voice span in reading. Chicago: Chicago University Press. Buswell, G. T. (1935). How people look at pictures: A study of the psychology of perception in art. Chicago: Chicago University Press. Butsch, R. I. C. (1932). Eye movements and the eye-hand span in typewriting. Journal of Educational Psychology, 23, 104–121. Caithness, G., Osu, R., Bays, P., Chase, H., Klassen, J., Kawato, M., et al. (2004). Failure to consolidate the consolidation theory of learning for sensorimotor adaptation tasks. The Journal of Neuroscience, 24, 8662–8671. Cardinali, L., Frassinetti, F., Brozzoli, C., Urquizar, C., Roy, A. C., & Farne, A. (2009). Tool-use induces morphological updating of the body schema. Current Biology, 19, R478–R479. Carpenter, R. H. (2000). The neural control of looking. Current Biology, 10, R291–R293. Case, R. (1985). Intellectual development—Birth to adulthood. Orlando, FL: Academic Press Inc. Castiello, U. (2005). The neuroscience of grasping. Nature Reviews. Neuroscience, 6, 726–736. Castiello, U., & Begliomini, C. (2008). The cortical control of visually guided grasping. The Neuroscientist, 14, 157–170. Corbetta, M., Akbudak, E., Conturo, T., Snyder, A., Ollinger, J., Drury, H., et al. (1998). A common network of functional areas for attention and eye movements. Neuron, 21, 761–773. Cothros, N., Wong, J. D., & Gribble, P. L. (2006). Are there distinct neural representations of object and limb dynamics? Experimental Brain Research, 173, 689–697. Cothros, N., Wong, J., & Gribble, P. L. (2009). Visual cues signaling object grasp reduce interference in motor learning. Journal of Neurophysiology, 102, 2112–2120. Craig, J. J. (1989). Introduction to robotics—Mechanics and control (2nd ed.). Reading, MA: Addison-Wesley Publishing Company. Devlic, A., Reichle, R., Wagner, M., Pinheiro, M. K., Vanromplay, Y., Berbers, Y., et al. (2009). Context inference of users’ social relationships and distributed policy management. In: 2009 IEEE international conference on pervasive computing and communications, New York: IEEE. Eagle, N., & Pentland, A. (2006). Reality mining: Sensing complex social systems. Personal and Ubiquitous Computing, 10, 255–268. Eagle, N., & Pentland, A. S. (2009). Eigenbehaviors: Identifying structure in routine. Behavioral Ecology and Sociobiology, 63, 1057–1066. Eibl-Eibesfeldt, I. (1989). Human ethology. Piscataway, NY: Aldine Transaction. Ernst, M. O., & Banks, M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 415, 429–433.
24 Fitzpatrick, D. (2000). Seeing beyond the receptive field in primary visual cortex. Current Opinion in Neurobiology, 10, 438–443. Flanagan, J. R., & Beltzner, M. A. (2000). Independence of perceptual and sensorimotor predictions in the size-weight illusion. Nature Neuroscience, 3, 737–741. Flanagan, J. R., Bowman, M. C., & Johansson, R. S. (2006). Control strategies in object manipulation tasks. Current Opinion in Neurobiology, 16, 650–659. Flombaum, J. I., & Santos, L. R. (2005). Rhesus monkeys attribute perceptions to others. Current Biology, 15, 447–452. Fogassi, L., & Luppino, G. (2005). Motor functions of the parietal lobe. Current Opinion in Neurobiology, 15, 626–631. Földiak, P. (1991). Learning invariance from transformation sequences. Neural Computation, 3, 194–200. Fu, Q., Zhang, W., & Santello, M. (2010). Anticipatory planning and control of grasp positions and forces for dexterous two-digit manipulation. The Journal of Neuroscience, 30, 9117–9126. Gandolfo, F., Mussa-Ivaldi, F. A., & Bizzi, E. (1996). Motor learning by field approximation. Proceedings of the National Academy of Sciences of the United States of America, 93, 3843–3846. Ganti, R. K., Srinivasan, S., & Gacic, A. (2010). Multisensor fusion in smartphones for lifestyle monitoring. Proceedings of the 2010 International Conference on Body Sensor Networks (pp. 36–43). Washington, DC: IEEE Computer Society. Geisler, W. S. (2008). Visual perception and the statistical properties of natural scenes. Annual Review of Psychology, 59, 167–192. Ghahramani, Z., & Wolpert, D. M. (1997). Modular decomposition in visuomotor learning. Nature, 386, 392–395. Ghahramani, Z., Wolpert, D. M., & Jordan, M. I. (1996). Generalization to local remappings of the visuomotor coordinate transformation. The Journal of Neuroscience, 16, 7085–7096. Gibson, J. J. (1966). The senses considered as perceptual systems. Boston, MA: Houghton Mifflin. Goedert, K. M., & Willingham, D. B. (2002). Patterns of interference in sequence learning and prism adaptation inconsistent with the consolidation hypothesis. Learning and Memory, 9, 279–292. Gold, J. I., & Shadlen, M. N. (2000). Representation of a perceptual decision in developing oculomotor commands. Nature, 404, 390–394. González, M., Hidalgo, C., & Barabási, A. (2008). Understanding individual human mobility patterns. Nature, 453, 779–782. Goodall, J. (1963). Feeding behaviour of wild chimpanzees— A preliminary report. Symposium of the Zoological Society of London, 10, 9–48.
Goodall, J. (1968). The behaviour of free-living chimpanzees in the Gombe Stream Reserve. Animal Behaviour Monographs, 1, 161–311. Gordon, A. M., Forssberg, H., Johansson, R. S., & Westling, G. (1991a). The integration of haptically acquired size information in the programming of precision grip. Experimental Brain Research, 83, 483–488. Gordon, A. M., Forssberg, H., Johansson, R. S., & Westling, G. (1991b). Integration of sensory information during the programming of precision grip: Comments on the contributions of size cues. Experimental Brain Research, 85, 226–229. Gordon, A. M., Forssberg, H., Johansson, R. S., & Westling, G. (1991c). Visual size cues in the programming of manipulative forces during precision grip. Experimental Brain Research, 83, 477–482. Gordon, A. M., Westling, G., Cole, K. J., & Johansson, R. S. (1993). Memory representations underlying motor commands used during manipulation of common and novel objects. Journal of Neurophysiology, 69, 1789–1796. Gross, C. G. (2002). Genealogy of the “Grandmother Cell” The Neuroscientist, 8, 512–518. Gross, C. G., Rochamir, C. E., & Bender, D. B. (1972). Visual properties of neurons in inferotemporal cortex of macaque. Journal of Neurophysiology, 35, 96–111. Győrbíró, N., Fábián, Á., & Hományi, G. (2009). An activity recognition system for mobile phones. Mobile Networks and Applications, 14, 82–91. Hager-Ross, C., & Schieber, M. H. (2000). Quantifying the independence of human finger movements: Comparisons of digits, hands, and movement frequencies. The Journal of Neuroscience, 20, 8542–8550. Hare, B., Call, J., Agnetta, B., & Tomasello, M. (2000). Chimpanzees know what conspecifics do and do not see. Animal Behaviour, 59, 771–785. Hare, B., Call, J., & Tomasello, M. (2001). Do chimpanzees know what conspecifics know? Animal Behaviour, 61, 139–151. Hartline, H. (1938). The response of single optic nerve fibers of the vertebrate eye to illumination of the retina. The American Journal of Physiology, 121, 400–415. Haruno, M., Wolpert, D. M., & Kawato, M. (2001). Mosaic model for sensorimotor learning and control. Neural Computation, 13, 2201–2220. Hayhoe, M., & Ballard, D. (2005). Eye movements in natural behavior. Trends in Cognitive Sciences (Regular Edition), 9, 188–194. Hayhoe, M., Mennie, N., Sullivan, B., & Gorgos, K. (2005). The role of internal models and prediction in catching balls. Proceedings of the American Association for Artificial Intelligence, Fall.
25 Hayhoe, M. M., Shrivastava, A., Mruczek, R., & Pelz, J. B. (2003). Visual memory and motor planning in a natural task. Journal of Vision, 3, 49–63. Heathcote, A., Brown, S., & Mewhort, D. J. (2000). The power law repealed: The case for an exponential law of practice. Psychonomic Bulletin and Review, 7, 185–207. Henderson, J. (2003). Human gaze control during real-world scene perception. Trends in Cognitive Sciences, 7, 498–504. Henderson, J., & Hollingworth, A. (1999). High-level scene perception. Annual Review of Psychology, 50, 243–271. Howard, I. S., Ingram, J. N., Kording, K. P., & Wolpert, D. M. (2009a). Statistics of natural movements are reflected in motor errors. Journal of Neurophysiology, 102, 1902–1910. Howard, I. S., Ingram, J. N., & Wolpert, D. M. (2009b). A modular planar robotic manipulandum with end-point torque control. Journal of Neuroscience Methods, 188, 199–211. Howard, I. S., Ingram, J. N., & Wolpert, D. M. (2008). Composition and decomposition in bimanual dynamic learning. The Journal of Neuroscience, 28, 10531–10540. Howard, I. S., Ingram, J. N., & Wolpert, D. M. (2010). Context-dependent partitioning of motor learning in bimanual movements. Journal of Neurophysiology, 104, 2082–2091. Hubel, D. (1960). Single unit activity in lateral geniculate body and optic tract of unrestrained cats. The Journal of Physiology, 150, 91–104. Hubel, D., & Wiesel, T. (1959). Receptive fields of single neurones in the cat's striate cortex. The Journal of Physiology, 148, 574–591. Hubel, D., & Wiesel, T. (1961). Integrative action in the cat's lateral geniculate body. The Journal of Physiology, 155, 385–398. Hubel, D., & Wiesel, T. (1965). Receptive fields and functional architecture in two nonstriate visual areas (18 and 19) of the cat. Journal of Neurophysiology, 28, 229–289. Humphrey, N. (1976). The social function of intellect. In P. P. G. Bateson & R. A. Hinde (Eds.), Growing points in ethology (pp. 1–8). Cambridge: Cambridge University Press. Hynes, M., Wang, H., & Kilmartin, L. (2009). Off-the-shelf mobile handset environments for deploying accelerometer based gait and activity analysis algorithms. In Conference Proceedings—IEEE Engineering in Medicine and Biology Society 2009, 5187–5190. Ingram, J. N., Howard, I. S., Flanagan, J. R., & Wolpert, D. M. (2010). Multiple grasp-specific representations of tool dynamics mediate skillful manipulation. Current Biology, 20, 618–623. Ingram, J. N., Kording, K. P., Howard, I. S., & Wolpert, D. M. (2008). The statistics of natural hand movements. Experimental Brain Research, 188, 223–236. Johansson, R. S. (1998). Sensory input and control of grip. Novartis Foundation Symposium, 218, 45–59 discussion 59–63.
Johansson, R. S., & Westling, G. (1988). Coordinated isometric muscle commands adequately and erroneously programmed for the weight during lifting task with precision grip. Experimental Brain Research, 71, 59–71. Johansson, R., Westling, G., Backstrom, A., & Flanagan, J. (2001). Eye-hand coordination in object manipulation. The Journal of Neuroscience, 21, 6917–6932. Johnson-Frey, S. H. (2004). The neural bases of complex tool use in humans. Trends in Cognitive Sciences, 8, 71–78. Jones, L. A. (1997). Dextrous hands: Human, prosthetic, and robotic. Presence, 6, 29–56. Jones, L. A., & Lederman, S. J. (2006). Human hand function. Oxford: Oxford University Press. Kagerer, F. A., Contreras-Vidal, J. L., & Stelmach, G. E. (1997). Adaptation to gradual as compared with sudden visuo-motor distortions. Experimental Brain Research, 115, 557–561. Karniel, A., & Mussa-Ivaldi, F. A. (2002). Does the motor control system use multiple models and context switching to cope with a variable environment? Experimental Brain Research, 143, 520–524. Kelso, J. A. S. (1984). Phase transitions and critical behaviour in human interlimb coordination. The American Journal of Physiology, 240, 1000–1004. Kelso, J. A. (1995). Dynamic patterns: The self-organization of brain and behavior. Cambridge, MA: The MIT Press. Kilbreath, S. L., & Gandevia, S. C. (1994). Limited independent flexion of the thumb and fingers in human subjects. Journal of Physiology, 479, 487–497. Kilbreath, S., & Heard, R. (2005). Frequency of hand use in healthy older persons. The Australian Journal of Physiotherapy, 51, 119–122. Kingstone, A., Smilek, D., & Eastwood, J. D. (2008). Cognitive ethology: A new approach for studying human cognition. British Journal of Psychology, 99, 317–340. Kitagawa, M., & Windor, B. (2008). MoCap for artists. Amsterdam: Focal Press. Konorski, J. (1967). Integrative activity of the brain: An interdisciplinary approach. Chicago, IL: University of Chicago Press. Kording, K., Kayser, C., Einhauser, W., & Konig, P. (2004). How are complex cell properties adapted to the statistics of natural stimuli? Journal of Neurophysiology, 91, 206–212. Krakauer, J. W., Ghez, C., & Ghilardi, M. F. (2005). Adaptation to visuomotor transformations: Consolidation, interference, and forgetting. The Journal of Neuroscience, 25, 473–478. Krakauer, J. W., Ghilardi, M. F., & Ghez, C. (1999). Independent learning of internal models for kinematic and dynamic control of reaching. Nature Neuroscience, 2, 1026–1031. Krakauer, J. W., Pine, Z. M., Ghilardi, M. F., & Ghez, C. (2000). Learning of visuomotor transformations for vectorial
26 planning of reaching trajectories. The Journal of Neuroscience, 20, 8916–8924. Krauzlis, R. J. (2005). The control of voluntary eye movements: New perspectives. The Neuroscientist, 11, 124–137. Kuffler, S. (1953). Discharge patterns and functional organization of mammalian retina. Journal of Neurophysiology, 16, 37–68. Lackner, J. R., & DiZio, P. (2005). Motor control and learning in altered dynamic environments. Current Opinion in Neurobiology, 15, 653–659. Lacquaniti, F., Soechting, J., & Terzuolo, C. (1982). Some factors pertinent to the organization and control of arm movements. Brain Research, 252, 394–397. Land, M. (1999). Motion and vision: Why animals move their eyes. Journal of Comparative Physiology A: Neuroethology, Sensory, Neural, and Behavioral Physiology, 185, 341–352. Land, M. F. (2006). Eye movements and the control of actions in everyday life. Progress in Retinal and Eye Research, 25, 296–324. Land, M. F. (2009). Vision, eye movements, and natural behavior. Visual Neuroscience, 26, 51–62. Land, M. F., & Furneaux, S. (1997). The knowledge base of the oculomotor system. Philosophical Transactions of the Royal Society of London Series B: Biological Sciences, 352, 1231–1239. Land, M., & Hayhoe, M. (2001). In what ways do eye movements contribute to everyday activities? Vision Research, 41, 3559–3565. Land, M., & Horwood, J. (1995). Which parts of the road guide steering? Nature, 377, 339–340. Land, M., & Lee, D. (1994). Where we look when we steer. Nature, 369, 742–744. Land, M., & McLeod, P. (2000). From eye movements to actions: How batsmen hit the ball. Nature Neuroscience, 3, 1340–1345. Land, M., Mennie, N., & Rusted, J. (1999). The roles of vision and eye movements in the control of activities of daily living. Perception, 28, 1311–1328. Land, M., & Tatler, B. (2001). Steering with the head: The visual strategy of a racing driver. Current Biology, 11, 1215–1220. Land, M., & Tatler, B. (2009). Looking and acting—Vision and eye movements in natural behaviour. Oxford, England: Oxford University Press. Lang, C. E., & Schieber, M. H. (2004). Human finger independence: Limitations due to passive mechanical coupling versus active neuromuscular control. Journal of Neurophysiology, 92, 2802–2810. Laughlin, S. (1987). Form and function in retinal processing. Trends in Neurosciences, 10, 478–483. Lee, G. X., Low, K. S., & Taher, T. (2010). Unrestrained measurement of arm motion based on a wearable wireless sensor network. In IEEE transactions on instrumentation and measurement, 59(5), 1309–1317.
Lee, J. Y., & Schweighofer, N. (2009). Dual adaptation supports a parallel architecture of motor memory. The Journal of Neuroscience, 29, 10396–10404. Lemon, R. N. (1997). Mechanisms of cortical control of hand function. The Neuroscientist, 3, 389–398. Li, Y., Levin, O., Forner-Cordero, A., & Swinnen, S. P. (2005). Interactions between interlimb and intralimb coordination during the performance of bimanual multijoint movements. Experimental Brain Research, 163, 515–526. Luinge, H., & Veltink, P. (2005). Measuring orientation of human body segments using miniature gyroscopes and accelerometers. Medical and Biological Engineering and Computing, 43, 273–282. Malfait, N., Shiller, D. M., & Ostry, D. J. (2002). Transfer of motor learning across arm configurations. The Journal of Neuroscience, 22, 9656–9660. Maravita, A., & Iriki, A. (2004). Tools for the body (schema). Trends in Cognitive Sciences, 8, 79–86. Marzke, M. W. (1992). Evolutionary development of the human thumb. Hand Clinics, 8, 1–8. Mason, C. R., Gomez, J. E., & Ebner, T. J. (2001). Hand synergies during reach-to-grasp. Journal of Neurophysiology, 86, 2896–2910. Mawase, F., & Karniel, A. (2010). Evidence for predictive control in lifting series of virtual objects. Experimental Brain Research, 203, 447–452. McFarland, D. (1999). Animal behaviour: Psychobiology, ethology and evolution (3rd ed.). Harlow, England: Pearson Education Limited. Mechsner, F., Kerzel, D., Knoblich, G., & Prinz, W. (2001). Perceptual basis of bimanual coordination. Nature, 414, 69–73. Miall, C. (2002). Modular motor learning. Trends in Cognitive Sciences, 6, 1–3. Miall, R. C., Jenkinson, N., & Kulkarni, K. (2004). Adaptation to rotated visual feedback: A re-examination of motor interference. Experimental Brain Research, 154, 201–210. Mündermann, L., Corazza, S., & Andriacchi, T. (2006). The evolution of methods for the capture of human movement leading to markerless motion capture for biomechanical applications. Journal of Neuroengineering and Rehabilitation, 3, 1–11. Munoz, D. P. (2002). Commentary: Saccadic eye movements: Overview of neural circuitry. Progress in Brain Research, 140, 89–96. Napier, N. (1980). Hands. New York: Pantheon Books. Newell, A., & Rosenbloom, P. S. (1981). Mechanisms of skill acquisition and the law of practice. In J. R. Anderson (Ed.), Cognitive skills and their acquisition (pp. 1–55). Hillsdale, NJ: Erlbaum. Nowak, D. A., Koupan, C., & Hermsdorfer, J. (2007). Formation and decay of sensorimotor and associative memory in object lifting. European Journal of Applied Physiology, 100, 719–726.
27 Nozaki, D., Kurtzer, I., & Scott, S. H. (2006). Limited transfer of learning between unimanual and bimanual skills within the same limb. Nature Neuroscience, 9, 1364–1366. Nozaki, D., & Scott, S. H. (2009). Multi-compartment model can explain partial transfer of learning within the same limb between unimanual and bimanual reaching. Experimental Brain Research, 194, 451–463. Olshausen, B., & Field, D. (1996). Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381, 607–609. Pagano, C. C., Kinsella-Shaw, J. M., Cassidy, P. E., & Turvey, M. T. (1994). Role of the inertia tensor in haptically perceiving where an object is grasped. Journal of Experimental Psychology. Human Perception and Performance, 20, 276–285. Pagano, C. C., & Turvey, M. T. (1992). Eigenvectors of the inertia tensor and perceiving the orientation of a hand-held object by dynamic touch. Perception and Psychophysics, 52, 617–624. Parker, C. (1974). The antecedents of man the manipulator. Journal of Human Evolution, 3, 493–500. Parker, S. T., & Gibson, K. R. (1977). Object manipulation, tool use and sensorimotor intelligence as feeding adaptations in cebus monkeys and great apes. Journal of Human Evolution, 6, 623–641. Pelegrin, J. (2005). Remarks about archaelogical techniques and methods of knapping: Elements of a cognitive approach to stone knapping. In V. Roux & B. Bril (Eds.), Stone knapping: The necessary conditions for a uniquely human behaviour (pp. 23–34). Cambridge: McDonald Institute for Archaelogical Research. Pelz, J., & Canosa, R. (2001). Oculomotor behavior and perceptual strategies in complex tasks. Vision Research, 41, 3587–3596. Pelz, J., Hayhoe, M., & Loeber, R. (2001). The coordination of eye, head, and hand movements in a natural task. Experimental Brain Research, 139, 266–277. Penfield, W., & Broldrey, E. (1937). Somatic motor and sensory representation in the cerebral cortex of man as studied by electrical stimulation. Brain, 60, 389–443. Perrett, D., Mistlin, A., & Chitty, A. (1987). Visual neurones responsive to faces. Trends in Neurosciences, 10, 358–364. Philipose, M., Fishkin, K., Perkowitz, M., Patterson, D., Fox, D., Kautz, H., & Hahnel, D. (2004). Inferring activities from interactions with objects. In IEEE pervasive computing (pp. 50–57). New York, NY: IEEE Communications Society. Piaget, J. (1954). Construction of reality in the child. New York: Bellantine Books. Pouget, A., & Snyder, L. (2000). Computational approaches to sensorimotor transformations. Nature Neuroscience, 3, 1192–1198. Povinelli, D. (2000). Folk physics for apes. Oxford: Oxford University Press.
Povinelli, D., & Bering, J. (2002). The mentality of apes revisited. Current Directions in Psychological Science, 11, 115–119. Reilly, K. T., & Hammond, G. R. (2000). Independence of force production by digits of the human hand. Neuroscience Letters, 290, 53–56. Reilly, K. T., & Schieber, M. H. (2003). Incomplete functional subdivision of the human multitendoned finger muscle flexor digitorum profundus: An electromyographic study. Journal of Neurophysiology, 90, 2560–2570. Reinagel, P. (2001). How do visual neurons respond in the real world? Current Opinion in Neurobiology, 11, 437–442. Ringach, D. (2004). Mapping receptive fields in primary visual cortex. Journal of Physiology (London), 558, 717–728. Rizzolatti, G., Luppino, G., & Matelli, M. (1998). The organization of the cortical motor system: New concepts. Electroencephalography and Clinical Neurophysiology, 106, 283–296. Roche, H., Delagnes, A., Brugal, J., Feibel, C., Kibunjia, M., Mourre, V., et al. (1999). Early hominid stone tool production and technical skill 2.34 Myr ago in west Turkana, Kenya. Nature, 399, 57–60. Salimi, I., Hollender, I., Frazier, W., & Gordon, A. M. (2000). Specificity of internal representations underlying grasping. Journal of Neurophysiology, 84, 2390–2397. Santello, M., Flanders, M., & Soechting, J. F. (1998). Postural hand synergies for tool use. The Journal of Neuroscience, 18, 10105–10115. Santello, M., Flanders, M., & Soechting, J. F. (2002). Patterns of hand motion during grasping and the influence of sensory guidance. The Journal of Neuroscience: The Official Journal of the Society for Neuroscience, 22, 1426–1435. Santello, M., & Soechting, J. F. (1998). Gradual molding of the hand to object contours. Journal of Neurophysiology, 79, 1307–1320. Schall, J. D. (2000). From sensory evidence to a motor command. Current Biology, 10, R404–R406. Schick, K., Toth, N., & Garufi, G. (1999). Continuing investigations into the stone tool-making and tool-using capabilities of a Bonobo (Pan paniscus). Journal of Archaeological Science, 26, 821–832. Schieber, M. H., & Santello, M. (2004). Hand function: Peripheral and central constraints on performance. Journal of Applied Physiology, 96, 2293–2300. Schlich, R., & Axhausen, K. (2003). Habitual travel behaviour: Evidence from a six-week travel diary. Transportation, 30, 13–36. Schmidt, R. C., & Lee, T. D. (2005). Motor control and learning—A behavioral emphasis (4th ed.). Champaign, IL: Human Kinetics. Schmidt, R. C., Shaw, B. K., & Turvey, M. T. (1993). Coupling dynamics in interlimb coordination. Journal of Experimental Psychology. Human Perception and Performance, 19, 397–415.
28 Shadmehr, R. (2004). Generalization as a behavioral window to the neural mechanisms of learning internal models. Human Movement Science, 23, 543–568. Shadmehr, R., & Brashers-Krug, T. (1997). Functional stages in the formation of human long-term motor memory. The Journal of Neuroscience, 17, 409–419. Shadmehr, R., & Mussa-Ivaldi, F. A. (1994). Adaptive representation of dynamics during learning of a motor task. The Journal of Neuroscience, 14, 3208–3224. Shadmehr, R., & Wise, S. P. (2005). The computational neurobiology of reaching and pointing: A foundation for motor learning. Cambridge, MA: The MIT Press. Simoncelli, E. P. (2003). Vision and the statistics of the visual environment. Current Opinion in Neurobiology, 13, 144–149. Simoncelli, E., & Olshausen, B. (2001). Natural image statistics and neural representation. Annual Review of Neuroscience, 24, 1193–1216. Slijper, H., Richter, J., Over, E., Smeets, J., & Frens, M. (2009). Statistics predict kinematics of hand movements during everyday activity. Journal of Motor Behavior, 41, 3–9. Snyder, L. H. (2000). Coordinate transformations for eye and arm movements in the brain. Current Opinion in Neurobiology, 10, 747–754. Soechting, J., & Flanders, M. (1992). Moving in three-dimensional space: Frames of reference, vectors, and coordinate systems. Annual Review of Neuroscience, 15, 167–191. Solomon, H. Y., & Turvey, M. T. (1988). Haptically perceiving the distances reachable with hand-held objects. Journal of Experimental Psychology. Human Perception and Performance, 14, 404–427. Sparks, D. L. (2002). The brainstem control of saccadic eye movements. Nature Reviews. Neuroscience, 3, 952–964. Srinivasan, M. V., Laughlin, S. B., & Dubs, A. (1982). Predictive coding: A fresh view of inhibition in the retina. Proceedings of the Royal Society B: Biological Sciences, 216, 427–459. Stockwell, R. A. (1981). G. J. Romanes (Ed.), Cunningham's textbook of anatomy. Oxford: Oxford University Press, (pp. 211–264). Stout, D., & Semaw, S. (2006). Knapping skill of the earliest stone tool-makers: Insights from the study of modern human novices. In N. Toth & K. Schick (Eds.), The Oldowan: Case studies into the earliest stone age. Gosport, IN: Stone Age Institute Press, (pp. 307–320). Swinnen, S. P., Dounskaia, N., & Duysens, J. (2002). Patterns of bimanual interference reveal movement encoding within a radial egocentric reference frame. Journal of Cognitive Neuroscience, 14, 463–471. Swinnen, S. P., Jardin, K., Verschueren, S., Meulenbroek, R., Franz, L., Dounskaia, N., et al. (1998). Exploring interlimb constraints during bimanual graphic performance: Effects of muscle grouping and direction. Behavioural Brain Research, 90, 79–87.
Takeda, R., Tadano, S., Natorigawa, A., Todoh, M., & Yoshinari, S. (2010). Gait posture estimation using wearable acceleration and gyro sensors. Journal of Biomechanics, 42, 2486–2494. Tanaka, K. (1996). Inferotemporal cortex and object vision. Annual Review of Neuroscience, 19, 109–139. Tcheang, L., Bays, P. M., Ingram, J. N., & Wolpert, D. M. (2007). Simultaneous bimanual dynamics are learned without interference. Experimental Brain Research, 183, 17–25. Tenorth, M., Bandouch, J., & Beetz, M. (2009). The TUM kitchen data set of everyday manipulation activities for motion tracking and action recognition. In: Workshop on Tracking Humans for the Evaluation of their Motion in Image Sequences (ICCV). Tocheri, M. W., Orr, C. M., Jacofsky, M. C., & Marzke, M. W. (2008). The evolutionary history of the hominin hand since the last common ancestor of Pan and Homo. Journal of Anatomy, 212, 544–562. Tomasello, M., & Call, J. (1997). Primate cognition. Oxford: Oxford University Press. Tompa, T., & Sáry, G. (2010). A review on the inferior temporal cortex of the macaque. Brain Research Reviews, 62, 165–182. Tong, C., Wolpert, D. M., & Flanagan, J. R. (2002). Kinematics and dynamics are not represented independently in motor working memory: Evidence from an interference study. The Journal of Neuroscience, 22, 1108–1113. Torigoe, T. (1985). Comparison of object manipulation among 74 species of non-human primates. Primates, 26, 182–194. Toth, N., Schick, K., Savage-Rumbaugh, E. S., Sevcik, R. A., & Rumbaugh, D. M. (1993). Pan the tool-maker: Investigations into the stone tool-making and tool-using capabilities of a bonobo (Pan paniscus). Journal of Archaeological Science, 20, 81–91. Treffner, P. J., & Turvey, M. T. (1996). Symmetry, broken symmetry, and handedness in bimanual coordination dynamics. Experimental Brain Research, 107, 463–478. Tresch, M. C., Cheung, V. C. K., & d'Avella, A. (2006). Matrix factorization algorithms for the identification of muscle synergies: Evaluation on simulated and experimental data sets. Journal of Neurophysiology, 95, 2199–2212. Tuller, B., & Kelso, J. A. (1989). Environmentally-specified patterns of movement coordination in normal and splitbrain subjects. Experimental Brain Research, 75, 306–316. Turvey, M. T. (1996). Dynamic touch. The American Psychologist, 51, 1134–1152. Turvey, M. T., Burton, G., Pagano, C. C., Solomon, H. Y., & Runeson, S. (1992). Role of the inertia tensor in perceiving object orientation by dynamic touch. Journal of Experimental Psychology. Human Perception and Performance, 18, 714–727. van Hateren, J. H. (1992). Real and optimal neural images in early vision. Nature, 360, 68–70.
29 Vauclair, J. (1982). Sensorimotor intelligence in human and non-human primates. Journal of Human Evolution, 11, 257–264. Vauclair, J. (1984). Phylogenetic approach to object manipulation in human and ape infants. Human Development, 27, 321–328. Vauclair, J., & Bard, K. (1983). Development of manipulations with objects in ape and human infants. Journal of Human Evolution, 12, 631–645. Visalberghi, E. (1993). Capuchin monkeys: A window into tool use in apes and humans. In K. R. Gibson & T. Ingold (Eds.), Tools, language and cognition in human evolution (pp. 138–150). Cambridge: Cambridge University Press. von Schroeder, H. P., & Botte, M. J. (1993). The functional significance of the long extensors and juncturae tendinum in finger extension. The Journal of Hand Surgery, 18A, 641–647. Wade, N. J., & Tatler, B. (2005). The moving tablet of the eye: The origins of modern eye movement research. Oxford: Oxford University Press. Wallis, G., & Bulthoff, H. (1999). Learning to recognize objects. Trends in Cognitive Sciences, 3, 22–31. Weaver, H. E. (1943). A study of visual processes in reading differently constructed musical selections. Psychological Monographs, 55, 1–30. White, O., Dowling, N., Bracewell, R. M., & Diedrichsen, J. (2008). Hand interactions in rapid grip force adjustments are independent of object dynamics. Journal of Neurophysiology, 100, 2738–2745. Wigmore, V., Tong, C., & Flanagan, J. R. (2002). Visuomotor rotations of varying size and direction compete for a single internal model in motor working memory. Journal of
Experimental Psychology. Human Perception and Performance, 28, 447–457. Wilson, F. R. (1998). The hand—How its use shapes the brain, language, and human culture. New York: Pantheon Books. Wimmers, R. H., Beek, P. J., & Vanwieringen, P. C. W. (1992). Phase-transitions in rhythmic tracking movements—A case of unilateral coupling. Human Movement Science, 11, 217–226. Witney, A. G., Goodbody, S. J., & Wolpert, D. M. (2000). Learning and decay of prediction in object manipulation. Journal of Neurophysiology, 84, 334–343. Witney, A. G., & Wolpert, D. M. (2003). Spatial representation of predictive motor learning. Journal of Neurophysiology, 89, 1837–1843. Wolpert, D. M., & Flanagan, J. R. (2001). Motor prediction. Current Biology, 11, R729–R732. Wolpert, D. M., & Flanagan, J. R. (2010). Q&A: Robotics as a tool to understand the brain. BMC Biology, 8, 92. Wolpert, D. M., Ghahramani, Z., & Flanagan, J. R. (2001). Perspectives and problems in motor learning. Trends in Cognitive Sciences, 5, 487–494. Wolpert, D., & Kawato, M. (1998). Multiple paired forward and inverse models for motor control. Neural Networks, 11, 1317–1329. Wurtz, R. H. (2009). Recounting the impact of Hubel and Wiesel. The Journal of Physiology, 587, 2817–2823. Yarbus, A. (1967). Eye movements and vision. New York: Plenum Press. Zhang, W., Gordon, A. M., Fu, Q., & Santello, M. (2010). Manipulation after object rotation reveals independent sensorimotor memory representations of digit positions and forces. Journal of Neurophysiology, 103, 2953–2964.
A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 191 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.
CHAPTER 2
Sensory change following motor learning Andrew A. G. Mattar{, Sazzad M. Nasirk, Mohammad Darainy{,{ and David J. Ostry{,},* {
k
Department of Psychology, McGill University, Montréal, Québec, Canada { Shahed University, Tehran, Iran } Haskins Laboratories, New Haven, Connecticut, USA The Roxelyn and Richard Pepper Department of Communication Sciences and Disorders, Northwestern University, Evanston, Illinois, USA
Abstract: Here we describe two studies linking perceptual change with motor learning. In the first, we document persistent changes in somatosensory perception that occur following force field learning. Subjects learned to control a robotic device that applied forces to the hand during arm movements. This led to a change in the sensed position of the limb that lasted at least 24 h. Control experiments revealed that the sensory change depended on motor learning. In the second study, we describe changes in the perception of speech sounds that occur following speech motor learning. Subjects adapted control of speech movements to compensate for loads applied to the jaw by a robot. Perception of speech sounds was measured before and after motor learning. Adapted subjects showed a consistent shift in perception. In contrast, no consistent shift was seen in control subjects and subjects that did not adapt to the load. These studies suggest that motor learning changes both sensory and motor function. Keywords: motor learning; sensory plasticity; arm movements; proprioception; speech motor control; auditory perception.
the human motor system and, likewise, to skill acquisition in the adult nervous system. Here, we summarize two studies in which we have examined the hypothesis that motor learning, which is associated with plastic changes to motor areas of the brain, leads to changes in sensory perception. We have investigated motor learning in the context of reaching movements and in speech motor control. We have examined the
Introduction To what extent is plasticity in motor and sensory systems linked? Neuroplasticity in sensory and motor systems is central to the development of *Corresponding author. Tel.: þ1-514-398-6111; Fax: þ1-514-398-4896 E-mail:
[email protected] DOI: 10.1016/B978-0-444-53752-2.00015-1
31
32
extent to which motor learning modifies somatosensory perception and the perception of speech. Our findings suggest that plasticity in motor systems does not occur in isolation, but it results in changes to sensory systems as well. Our studies examine sensorimotor learning in both an arm movement task and a speech task. It is known that there are functional connections linking brain areas involved in the sensory and motor components of these tasks. There are known ipsilateral corticocortical projections linking somatosensory cortex with motor areas of the brain (Darian-Smith et al., 1993; Jones et al., 1978). Activity in somatosensory cortex varies systematically with movement (AgeraniotiBélanger and Chapman, 1992; Chapman and Ageranioti-Bélanger, 1991; Cohen et al., 1994; Prud'homme and Kalaska, 1994; Prud'homme et al., 1994; Soso and Fetz, 1980), and the sensory signals arising from movement can result in changes to somatosensory receptive fields (Jenkins et al., 1990; Recanzone et al., 1992a,b; Xerri et al., 1999). Likewise, auditory processing recruits activity in motor areas of the brain (Chen et al., 2008; Pulvermüller et al., 2006), and auditory and somatosensory inputs converge within auditory cortex (Foxe et al., 2002; Fu et al., 2003; Kayser et al., 2005; Shore and Zhou, 2006). In addition, there are a number of pieces of evidence suggesting perceptual changes related to somatosensory input, movement, and learning. These include proprioceptive changes following visuomotor adaptation in reaching movements and in manual tracking (Cressman and Henriques, 2009, 2010; Cressman et al. 2010; Malfait et al., 2008; Simani et al., 2007; van Beers et al., 2002) and visual and proprioceptive changes following force field learning (Brown et al., 2007; Haith et al., 2008). They also include changes to auditory perception that are caused by somatosensory input (Ito et al., 2009; Jousmäki and Hari, 1998; Murray et al., 2005; Schürmann et al., 2004). These studies thus suggest that via the links between motor, somatosensory, and auditory areas of the brain, an effect of motor learning on perception may be likely.
Below, we describe a study involving human arm movement that tests the idea that sensory function is modified by motor learning. Specifically, we show that learning to correct for forces that are applied to the limb by a robot results in durable changes to the sensed position of the limb. We report a second study in which we test the hypothesis that speech motor learning, and in particular the somatosensory inputs associated with learning, affect the classification of speech sounds. In both studies, we observe perceptual changes following learning. These findings suggest that motor learning affects not only the motor system but also involves changes to sensory areas of the brain.
The effect of motor learning on somatosensory perception of the upper limb Subjects made movements to a target in a standard force field learning procedure. In this task, subjects make reaching movements to a visual target while holding the handle of a robotic device that is programmed to apply forces to the subject's hand (Fig. 1a). Studies employing this technique have been used to document learning and plasticity in motor systems (Gribble and Scott, 2002; Shadmehr and Holcomb, 1997; Shadmehr and Mussa-Ivaldi, 1994). Figure 1b shows the experimental sequence. We interleaved blocks of trials in which we estimated the sensed position of the limb (shown in gray) with blocks of force field learning trials. We tested sensory perception twice before and once after force field learning. We also tested for the persistence of changes in sensory perception after the effects of motor learning were eliminated using washout trials. We obtained estimates of the sensed position of the limb using an iterative procedure known as PEST (parameter estimation by sequential testing; Taylor and Creelman, 1967). The PEST procedure was done in the absence of vision. On each movement in the testing sequence, the limb was displaced laterally using a force channel (Fig. 1c). At
33 (b)
0 −10 −20 −30 −40 −50
Aftereffects Perceptual test
Force field training Perceptual test
Null field
Perceptual test
50 40 30 20 10
Perceptual test
Movement curvature (mm)
(a)
Right Left 1
50
100
150
1
50
100
1
150
50
Trial number (c)
Final PEST Final PEST trial trial
(d) 0
200
PEST trial Before FF training
Hand position in Y (mm)
10 160 120 80 40
20 30 40 50
1 2 3 4 5 6 7 8 9 PEST trial
0 −40
−20
0
0
20
Hand position in X (mm)
40
60 −40 −20
0
10
30
Lateral displacement (mm)
Fig. 1. Force field learning and the perception of limb position. (a) Subjects held the handle of a robotic device, when making movements to targets and during perceptual testing. The robot was capable of applying forces to the hand. Targets were presented on a horizontal screen that occluded vision of the hand, arm, and robot. (b) Subjects learn to compensate for velocitydependent mechanical loads that displace the limb to the right or the left. Perceptual tests (gray bars) of the sensed limb position are interleaved with force field training. Average movement curvature ( SE) is shown throughout training. (c) An iterative procedure known as PEST estimates the perceptual boundary between left and right. A computer-generated force channel laterally displaced the limb, and subjects are required to indicate whether the limb has been deflected to the right. Individual PEST runs starting from left and right, respectively, are shown. The sequence is indicated by the shading of the PEST trials beginning at the right. (d) A sequence of six PEST runs (starting from the top) with the horizontal axis showing the lateral position of the hand and the PEST trial number on the vertical. The shaded sequence of trials shown at the top is the same as is shown on the right side of (c). PEST runs alternately start from the right and the left and end on a similar estimate of the perceptual boundary. Note that the horizontal axis highlights lateral hand positions between 0 and 10 mm.
the end of each movement the subject gave a “yes” or “no” response indicating whether the limb had been deflected to the right. Over the course of several trials, the magnitude of the deflection was modified based on the subject's responses in order
to determine the perceptual boundary between left and right. Figure 1b shows a sequence of PEST trials for a representative subject, prior to force field learning. The left panel shows a PEST sequence that began with a leftward deflection;
34
the right panel shows a sequence for the same subject beginning from the right. Figure 1d shows a sequence of six PEST runs. Each run converges on a stable estimate of the perceptual boundary between left and right. In the motor learning phase of the experiment, subjects made movements in a clockwise or counterclockwise force field applied by a robot arm (Fig. 1a), whose actions were to push the hand to the right or to the left. Performance over the course of training was quantified by computing the maximum perpendicular distance (PD) from a straight line joining movement start and end. Figure 1b shows movement curvature (PD values), averaged over subjects, for each phase of the experiment. Under null conditions, subjects move straight to the target. Upon the introduction of the force field, movements are deflected laterally but over the course of training they straighten to near null field levels. The reduction in curvature from the initial 10 movements to the final 10 movements was reliable for both force field directions. Curvature on initial aftereffect movements is opposite to the curvature on initial force field movements reflecting the adjustment to motor commands needed to produce straight movements in the presence of load. Curvature at the end of the washout trials differs from initial null field trials; movements remain curved in a direction opposite to that of the applied force. On a per-subject basis, we quantified perceptual performance by fitting a logistic function to the set of lateral limb positions and the associated binary responses that were obtained over successive PEST runs. For example, the sequence of PEST trials shown in Fig. 1d would lead to a single psychometric function relating limb position to the perceptual response. For visualization purposes, Fig. 2a shows binned response probabilities, averaged across subjects, and psychometric functions fit to the means for the rightward and leftward force fields. Separate curves are shown for estimates obtained before and after learning. The psychometric curve, and hence the perceptual boundary between left and right
shifts in a direction opposite to the applied load. If the force field acts to the right (Fig. 2a, right panel), the probability of responding that the hand was pushed to the right increases following training. This means that following force field learning, the subject feels as if the hand is located farther to the right. Figure 2b shows the position of the perceptual boundary in each of the four test sessions. The perceptual boundary was computed as the 50% point on the psychometric curve. For each subject separately, we computed the shift in the perceptual boundary as a difference between the final null condition estimate and the estimate following force field training. We computed the persistence of the shift as the difference between the final null condition estimate and the estimate following aftereffect trials. The shifts are shown in Fig. 2c. It can be seen that immediately after force field training there was a shift in the sensed position of the limb that was reliably different than zero. The shift decreased following washout but remained different than zero. The magnitude of the shift was no different for both force field directions. Thus, the sensed position of the limb changes following force field learning, and the shift persists even after the kinematic effects of learning have been washed out. In a control study, we examined the persistence of the perceptual change. Subjects were tested in a procedure that was identical to the main experiment, but it included an additional perceptual test 24 h following learning. The results are shown in Fig. 2c. It can be seen that the force field led to a reliable shift in the perceptual boundary that was no different across the three estimates. Thus, periods of force field learning lasting 10 min result in shifts in the perceptual boundary that persist for at least 24 h. We conducted a second control experiment to determine the extent to which the observed perceptual changes are tied to motor learning. We used methods that were identical to those in the main experiment, except that the force field learning phase was replaced with a task that did
35 (a)
Leftward field
Rightward field
100 Probability of “right” response (%)
Before learning After learning
Before learning After learning
80 60 40 20 0
−20 −10 Left
0
10
−20 −10 Left
20 Right
0
10
20 Right
Position (mm) (c) 8
3
7
Change in perceptual boundary (mm)
Perceptual boundary (mm)
(b)
6 5 4 3 2 1 e1
B
e2
FF AE er er Aft Aft Perceptual test
lin
lin
e as
B
e as
Left Right
2 1 0 −1 −2 er
Aft
FF
h AE 24 er er t f Aft A Perceptual test Left Right Passive control 24 h left
Fig. 2. The perceptual boundary shifts in a direction opposite to the applied force following motor learning. (a) Binned response probabilities averaged over subjects (SE) before (light gray) and after (black or dark gray) learning. Fitted psychometric functions reflect the perceptual classification for each force field direction. (b) Mean perceptual boundary between left and right (SE) for baseline estimates (baseline 1 and baseline 2), estimates following force field learning (after FF), and estimates following aftereffect trials (after AE). The sensed position of the limb changes following learning, and the change persists following aftereffect trials. (c) The direction of the perceptual shift depends on the force field (left vs. right). The perceptual shift persists for at least 24 h (24 h left). A perceptual shift is not observed when the robot passively moves the hand through the same sequence of positions and velocities as in the left condition such that subjects do not experience motor learning (passive control).
not involve motor learning. In the null field and aftereffect phases of the experiment, subjects moved actively. The force field learning phase was replaced with a passive task in which subjects
held the robot handle as it reproduced the movements of subjects in the leftward force field condition of the main experiment. Under positionservo control, the robot produced this series of
36
movements and the subject's arm was moved along the mean trajectory for each movement in the training sequence. Thus, subjects experienced a series of movements with the same kinematics as those in main experiment, but importantly they did not experience motor learning. The upper panel of Fig. 3 shows the mean movement curvature (PD) for subjects tested in the passive control experiment and for subjects tested in the original experiment. The lower panel shows the average difference between PD in the passive control condition and PD in the original leftward force field. The lower panel of Fig. 3 shows that in the null phase, movement kinematics were well matched when subjects in both conditions made active movements. In the force field phase of the experiment, the near-zero values indicate that subjects in the passive control experiment experienced kinematics that closely
matched the mean trajectory in the original experiment. The nonzero values at the start of the aftereffect phase indicate that in the main experiment, training in the force field resulted in aftereffects and hence motor learning that was greater than following training in the passive control experiment. Figure 2c shows measures of perceptual change for subjects trained in the original experiment, as well as for subjects trained in the passive control. Perceptual shifts depended on whether or not subjects experienced motor learning. As described above, subjects in the original experiment who learned the leftward force field showed perceptual shifts that were reliably different than zero both immediately after learning and after washout trials. In contrast, subjects tested in the passive control experiment showed shifts that did not differ from zero at either time point.
Null field
Force field training
Aftereffects
20 10 0
20
Left Passive control Left minus passive control
Perceptual test
Difference in movement curvature (mm)
−30
Perceptual test
−20
Perceptual test
−10
Perceptual test
Movement Curvature (mm)
30
10 0 −10 1
50
100
150
1
50
100
150
1
50
Trial number Fig. 3. The perceptual shift depends on motor learning. In a control experiment, subjects experience the same trajectories as individuals that display motor learning. Subjects move actively in the null and aftereffect phases of the study. In the force field training phase, the robot moves the arm to replicate the average movement path of subjects that learned the leftward force field. The top panel shows mean movement curvature ( SE) for subjects in the original leftward condition (black) and the passive control condition (light gray). The bottom panel gives the difference between active and passive movements (dark gray). Movement aftereffects are not observed in the passive condition (light gray) indicating there is no motor learning.
37
In the perceptual tests, the subject had to identify whether an auditory stimulus chosen at random from a synthesized eight step spectral continuum sounded more like the word head or had (Fig. 4). A psychometric function was fitted to the data and gave the probability of identifying the word as had. We focused on whether motor learning led to changes to perceptual performance. Sensorimotor learning was evaluated using a composite measure of movement curvature. Curvature was assessed on a per-subject basis, in null condition trials, at the start and at the end of learning. Statistically reliable adaptation was observed in 17 of the 23 subjects. This is typical of studies of speech motor learning in which about a third of all subjects fail to adapt (Nasir
The effect of speech motor learning on the perception of speech sounds In order to evaluate the idea that speech motor learning affects auditory perception, we trained healthy adults in a force field learning task (Lackner and Dizio, 1994; Shadmehr and MussaIvaldi, 1994) in which a robotic device applied a mechanical load to the jaw as subjects repeated aloud test utterances that were chosen randomly from a set of four possibilities (bad, had, mad, sad; Fig. 4). The mechanical load was velocitydependent and acted to displace the jaw in a protrusion direction, altering somatosensory but not auditory feedback. Perception of speech sounds was assessed before and after force field training. (a)
Protocol
(b)
Force field training/word repetition
Perceptual test
Perceptual test
Time
Frequency (Hz)
(c) 3000 2000 1000 0
Head
Had Auditory continuum
Fig. 4. Experimental set-up, protocol, and auditory test stimuli for the speech experiment. (a) A velocity-dependent load was delivered to the jaw by a robotic device. (b) Subjects completed an auditory identification task before and after motor learning. Control subjects repeated the same set of utterances but were not attached to the robot. (c) During perceptual testing, subjects indicated whether a given auditory test stimulus sounded more like head or had.
38
and Ostry, 2006, 2008; Purcell and Munhall, 2006; Tremblay et al., 2003). Figure 5a shows a representative sagittal plane view of jaw trajectories during speech for a subject that adapted to the load. Movements are straight in the absence of load; the jaw is displaced in a protrusion direction when the load is first applied; curvature decreases with training. Figure 5b shows movement curvature measures for the same subject, for individual trials, over the course of the entire experiment. (a)
As shown in Fig. 5a, movement curvature was low in the null condition, increased with the introduction of load and then progressively decreased with training. The auditory psychometric function for this subject shifted to the right following training (Fig. 5c). This indicates that words sounded more like head after learning. Figure 6a shows perceptual psychometric functions for adapted subjects before and after force field training. A rightward shift following (b) 2.5
0
2
−10 Curvature
Elevation (mm)
Null condition
−20
1
0
−10
0
10
0
Protrusion (mm) (c)
1.5
0.5
−30
Identification probability
Training Null condition
End training Initial training
1
50
100
150 200 Trials
250
300
350
Posttraining Pretraining
0.8 0.6 0.4 0.2 0 1 2 Head
3
4
5
6
7
8 Had
Auditory continuum Fig. 5. Speech motor learning and changes in speech perception. (a) Sagittal view of jaw movement paths for a representative subject who adapted to the load. Movements were straight in the absence of load (light gray). The jaw was deflected in the protrusion direction when the load was introduced (black). Curvature decreased with training (dark gray). (b) Scatter plot showing movement curvature over the course of training for the same subject as in (a). The vertical axis shows movement curvature; the horizontal axis gives trial number. Curvature is low on null trials (light gray) increases when the load is introduced and decreases over the course of training (black). (c) The psychometric function depicting identification probability for had before (light gray) and after (black) training. A perceptual shift toward head was observed following learning.
39 (a)
(b)
1
1
Pretraining
0.8
Identification probability
Identification probability
Posttraining
0.6 0.4 0.2 0
1 Head
2
3
4
5
6
7
0.8 0.6 0.4 0.2 0
8 Had
1 2 Head
Auditory continuum
(c)
(d)
Adapted subjects Nonadapted and control subjects
Perceptual shift (%)
Probability
Perceptual shift (%)
6
7
8 Had
20
0.3 0.2 0.1 0
5
Auditory continuum
4
0
4
(e)
0.5 0.4
8
3
−20
−10 0 10 Perceptual shift (%)
20
10 0 −10 0
10
20 30 40 50 Adaptation (%)
60
Fig. 6. Perception of speech sounds changes following speech motor learning. (a) The average psychometric functions for adapted subjects reveal a perceptual shift to the right following training (light gray: pretraining, black: posttraining). (b) There is no perceptual shift for nonadapted and control subjects. (c) The perceptual shift for adapted subjects (black) was reliably greater than the shift observed in nonadapted and control subjects (light gray), which was not different than zero. (d) Histograms showing the perceptual change for adapted (black) and nonadapted/control subjects (light gray). (e) The perceptual shift was correlated with adaptation. Subjects that showed greater adaptation also had greater perceptual shifts.
training is evident. A measure of probability, that was used to assess perceptual change, was obtained by summing each subject's response probabilities for individual stimulus items and dividing the total by a baseline measure that was obtained before learning. The change in identification probability from before to after training was used to gauge the perceptual shift. In 15 of the 17 subjects that adapted to the force field, we found a rightward shift in the psychometric function following training. This rightward
perceptual shift means that after force field learning the auditory stimuli are more likely to be classified as head. In effect, the perceptual space assigned to head increased with motor learning. The remaining six subjects who failed to adapt did not show any consistent pattern in their perceptual shifts. We evaluated the possibility that the perceptual shift might be due to factors other than motor learning by testing a group of control subjects who completed the entire experiment without
40
force field training. This control study included the entire sequence of several hundred speech movements. For control subjects, the perceptual shift, computed in the same manner as for the experimental subjects, was not different than zero (Fig. 6c). Moreover, we found that perceptual shifts obtained for the nonadapted subjects in the main experiment did not differ from the shifts obtained from control subjects. Figure 6b shows the psychometric functions averaged over nonadapted and control subjects combined, before and after word repetition (or force field training for the nonadapted subjects). No difference can be seen in the psychometric functions of the subjects that did not experience motor learning. Statistical tests were conducted on the perceptual probability scores before and after training. The analysis compared the scores of adapted subjects with those of control subjects and nonadapted subjects combined. The test thus compared the perceptual performance of subjects that successfully learned the motor task with those that did not. For adapted subjects, we found that identification scores were significantly different after training than before. For subjects that did not show motor learning, the difference in the two perceptual tests was nonsignificant. Thus, speech motor learning in a force field environment modifies perception of speech sounds. Word repetition alone cannot explain the observed perceptual effects. In order to characterize further the pattern of perceptual shifts, we obtained histograms giving the distribution of shifts for both the adapted and the combined nonadapted and control groups (Fig. 6d). The histogram for the adapted group is to the right of the histogram for the nonadapted subjects. We also examined the possibility that subjects that showed greater learning would also show a greater perceptual shift. We calculated an index of learning for each adapted subject by computing the reduction in curvature over the course of training divided by the curvature due to the introduction of load. A value of 1.0 indicates complete adaptation. Computed in this
fashion, adaptation ranged from 0.05 to 0.55 and when averaged across subjects and test words, it was 0.29 0.03 (mean SE). Figure 6e shows the relationship between the amount of adaptation and the associated perceptual shift. We found that adapted subjects showed a small, but significant, correlation of 0.53 between the extent of adaptation and the measured perceptual shift. We assessed the possibility that there are changes in auditory input over the course of force field training that might contribute to motor learning and also to the observed perceptual shift. Acoustical effects related to the application of load and learning were evaluated by computing the first and second formant frequencies of the vowel /æ/ immediately following the initial consonant in each of the test utterances. A statistical analysis found no reliable differences in either formant frequency over the course of the experiment. This suggests that there were no changes in auditory input over the course of adaptation.
Discussion In the limb movement study, we showed that motor learning results in changes in the sensed position of the limb. The passive control experiment reveals that changes in somatosensory perception depend on motor learning. The perceptual change is robust, in that it persists for periods lasting at least 24 h. In the absence of movement, sensory experience results in a selective expansion of the specific regions of somatosensory cortex that are associated with the sensory exposure, and it also results in changes in the size of sensory receptive fields that reflect the characteristics of the adaptation (Recanzone et al., 1992a,b). Changes to receptive field size in somatosensory cortex are observed when sensory training is combined with motor tasks that require precise contact with a rotating disk (Jenkins et al., 1990) or finger and forearm movements to remove food from a narrow well (Xerri et al., 1999). In these latter cases,
41
it is not clear whether it is the sensory experience, the motor experience, or both factors in combination that leads to changes in the sensory system. This issue is clarified by the findings summarized here. Changes in sensory perception depend on active movement and learning. Control subjects who experienced the same movements but did not experience motor learning showed no perceptual change. This points to a central role of motor learning in somatosensory plasticity. The idea that sensory perception depends on both sensory and motor systems has been proposed by other researchers (Feldman, 2009; Haith et al., 2008). One possibility is that the central contribution to position sense involves motor commands that are adjusted by adaptation (see Feldman, 2009, for a recent review of central and afferent contributions to position sense). In effect, sensory signals from receptors are measured in a motoric reference frame that can be modified by learning. Another possibility is that the learning recalibrates both sensory and motor processes. Haith et al. propose that changes in performance that are observed in the context of learning depend on changes to both motor and sensory function that are driven by error (Haith et al., 2008). In a second study, we found that the perceptual classification of speech sounds was modified by speech motor learning. There was a systematic change such that following learning, speech sounds on a continuum ranging from head to had were more frequently classified as head. Moreover, the perceptual shift varied with learning; the perceptual change was greater in subjects that showed greater adaptation during learning. The perceptual shift was not observed in subjects who failed to adapt to the forces applied by the robot, nor was it observed in control subjects who repeated the same words but did not undergo force field learning. This suggests a link between motor learning and the perceptual change. The findings thus indicate that speech learning modifies not only the motor system but also the perception of speech sounds.
The sensory basis of the auditory perceptual effect was somatosensory in nature. Force field training modified the motion path of the jaw and hence somatosensory feedback, but it did not affect the acoustical patterns of speech at any point during training. Hence, there was no change in auditory information that might result in perceptual modification. Thus the sensory basis of both the motor learning and the perceptual recalibration is presumably somatosensory but not auditory. This conclusion is supported by the observation that adaptation to mechanical load occurs when subjects perform the speech production task silently, indicating that it is not dependent upon explicit acoustical feedback (Tremblay et al., 2003). It is also supported by the finding that profoundly deaf adults who are tested with their assistive hearing devices turned off can still adapt to mechanical loads applied during speech (Nasir and Ostry, 2008). The perceptual shift we observed is in the same direction as in previous studies of perceptual adaptation (Cooper and Lauritsen, 1974; Cooper et al., 1976). Cooper and colleagues observed that after listening to repetitions of a particular consonant–vowel stimulus, the probability that subjects would report hearing this same stimulus in subsequent perceptual testing was reduced. The effect reported here is similar to that observed by Cooper, but there are important differences suggesting the effects are different in origin. We found no perceptual shift in nonadapted subjects who repeatedly said or heard a given test stimulus. Moreover, control subjects also repeated and listened to the same set of utterances but did not show a reliable perceptual change. Both of these facts are consistent with the idea that motor learning, but not repeated experience with the speech stimuli, is the source of the perceptual change. Influences of somatosensory input on auditory perception have been documented previously. There is somatosensory input to the cochlear nucleus, and there are known bidirectional interactions between auditory and somatosensory
42
cortex (Foxe et al., 2002; Fu et al., 2003; Jousmäki and Hari, 1998; Kayser et al., 2005; Murray et al., 2005; Schürmann et al., 2006; Shore and Zhou, 2006). In addition, there are reports that somatosensory inputs affect auditory perceptual function in cases involving speech (Gillmeister and Eimer, 2007; Ito et al., 2009; Schürmann et al., 2004). The present example of somatosensory–auditory interaction is intriguing because subjects receive somatosensory input when producing speech but not when perceiving speech sounds produced by others. Indeed, the involvement of somatosensory information in the perceptual processing of speech would be consistent with the idea that speech perception is mediated by the mechanisms of speech production (Hickok and Poeppel, 2000; Libermann and Mattingly, 1985). This view is supported by other studies demonstrating that electromyographic responses evoked by transcranial magnetic stimulation (TMS) to primary motor cortex are facilitated by watching speech movements and listening to speech sounds (Fadiga et al., 2002; Watkins et al., 2003), and that speech perception is affected by repetitive TMS to premotor cortex (Meister et al., 2007). However, the perceptual effects described here may well occur differently, resulting from the direct effects of somatosensory input on auditory cortex (Hackett et al., 2007). In summary, in both of the studies described above, we have found that motor learning leads to changes in perceptual function. In both cases, the perceptual change was grounded in motor learning; sensory experience on its own was not sufficient for changes in perception. These findings suggest that plasticity in sensory and motor systems is linked, and that changes in each system may not occur in isolation. References Ageranioti-Bélanger, S. A., & Chapman, C. E. (1992). Discharge properties of neurones in the hand area of primary somatosensory cortex in monkeys in relation to the
performance of an active tactile discrimination task. II. Area 2 as compared to areas 3b and 1. Experimental Brain Research, 91, 207–228. Brown, L. E., Wilson, E. T., Goodale, M. A., & Gribble, P. L. (2007). Motor force field learning influences visual processing of target motion. The Journal of Neuroscience, 27, 9975–9983. Chapman, C. E., & Ageranioti-Bélanger, S. A. (1991). Discharge properties of neurones in the hand area of primary somatosensory cortex in monkeys in relation to the performance of an active tactile discrimination task. I. Areas 3b and 1. Experimental Brain Research, 87, 319–339. Chen, J. L., Penhune, V. B., & Zatorre, R. J. (2008). Listening to musical rhythms recruits motor regions of the brain. Cerebral Cortex, 18, 2844–2854. Cohen, D. A., Prud'homme, M. J., & Kalaska, J. F. (1994). Tactile activity in primate primary somatosensory cortex during active arm movements: Correlation with receptive field properties. Journal of Neurophysiology, 71, 161–172. Cooper, W. E., & Lauritsen, M. R. (1974). Feature processing in the perception and production of speech. Nature, 252, 121–123. Cooper, W. E., Billings, D., & Cole, R. A. (1976). Articulatory effects on speech perception: A second report. Journal of Phonetics, 4, 219–232. Cressman, E. K., & Henriques, D. Y. (2009). Sensory recalibration of hand position following visuomotor adaptation. Journal of Neurophysiology, 102, 3505–3518. Cressman, E. K., & Henriques, D. Y. (2010). Reach adaptation and proprioceptive recalibration following exposure to misaligned sensory input. Journal of Neurophysiology, 103, 1888–1895. Cressman, E. K., Salomonczyk, D., & Henriques, D. Y. (2010). Visuomotor adaptation and proprioceptive recalibration in older adults. Experimental Brain Research, 205, 533–544. Darian-Smith, C., Darian-Smith, I., Burman, K., & Ratcliffe, N. (1993). Ipsilateral cortical projections to areas 3a, 3b, and 4 in the macaque monkey. The Journal of Comparative Neurology, 335, 200–213. Fadiga, L., Craighero, L., Buccino, G., & Rizzolati, G. (2002). Speech listening specifically modulates the excitability of tongue muscles: A TMS study. The European Journal of Neuroscience, 15, 399–402. Feldman, A. G. (2009). New insights into action–perception coupling. Experimental Brain Research, 194, 39–58. Foxe, J. J., Wylie, G. R., Martinez, A., Schroeder, C. E., Javitt, D. C., Guilfoyle, D., et al. (2002). Auditory-somatosensory multisensory processing in auditory association cortex: An fMRI study. Journal of Neurophysiology, 88, 540–543. Fu, K. M., Johnston, T. A., Shah, A. S., Arnold, L., Smiley, J., Hackett, T. A., et al. (2003). Auditory cortical neurons respond to somatosensory stimulation. The Journal of Neuroscience, 23, 7510–7515.
43 Gillmeister, H., & Eimer, M. (2007). Tactile enhancement of auditory detection and perceived loudness. Brain Research, 1160, 58–68. Gribble, P. L., & Scott, S. H. (2002). Overlap of internal models in motor cortex for mechanical loads during reaching. Nature, 417, 938–941. Hackett, T. A., Smiley, J. F., Ulbert, I., Karmos, G., Lakatos, P., de la Mothe, L. A., et al. (2007). Sources of somatosensory input to the caudal belt areas of auditory cortex. Perception, 36, 1419–1430. Haith, A., Jackson, C., Miall, R., & Vijayakumar, S. (2008). Unifying the sensory and motor components of sensorimotor adaptation. Advances in Neural Information Processing Systems, 21, 593–600. Hickok, G., & Poeppel, D. (2000). Toward functional neuroanatomy of speech perception. Trends in Cognitive Science, 4, 131–138. Ito, T., Tiede, M., & Ostry, D. J. (2009). Somatosensory function in speech perception. Proceedings of the National Academy of Sciences of the United States of America, 106, 1245–1248. Jenkins, W. M., Merzenich, M. M., Ochs, M. T., Allard, T., & Guíc-Robles, E. (1990). Functional reorganization of primary somatosensory cortex in adult owl monkeys after behaviorally controlled tactile stimulation. Journal of Neurophysiology, 63, 82–104. Jones, E. G., Coulter, J. D., & Hendry, S. H. (1978). Intracortical connectivity of architectonic fields in the somatic sensory, motor and parietal cortex of monkeys. The Journal of Comparative Neurology, 181, 291–347. Jousmäki, V., & Hari, R. (1998). Parchment-skin illusion: Sound-biased touch. Current Biology, 8, RC190. Kayser, C., Petkov, C. I., Augath, M., & Logothetis, N. K. (2005). Integration of touch and sound in auditory cortex. Neuron, 48, 373–384. Lackner, J. R., & Dizio, P. (1994). Rapid adaptation to coriolis force perturbations of arm trajectory. Journal of Neurophysiology, 72, 299–313. Libermann, A. M., & Mattingly, I. G. (1985). The motor theory of speech perception revised. Cognition, 21, 1–36. Malfait, N., Henriques, D. Y., & Gribble, P. L. (2008). Shape distortion produced by isolated mismatch between vision and proprioception. Journal of Neurophysiology, 99, 231–243. Meister, I. G., Wilson, S. M., Deblieck, C., Wu, A. D., & Iacoboni, M. (2007). The essential role of premotor cortex in speech perception. Current Biology, 17, 1692–1696. Murray, M. M., Molholm, S., Michel, C. M., Heslenfeld, D. J., Ritter, W., Javitt, D. C., et al. (2005). Grabbing your ear: Rapid auditory-somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignment. Cerebral Cortex, 15, 963–974.
Nasir, S. M., & Ostry, D. J. (2006). Somatosensory precision in speech production. Current Biology, 16, 1918–1923. Nasir, S. M., & Ostry, D. J. (2008). Speech motor learning in profoundly deaf adults. Nature Neuroscience, 11, 1217–1222. Prud'homme, M. J., & Kalaska, J. F. (1994). Proprioceptive activity in primate primary somatosensory cortex during active arm reaching movements. Journal of Neurophysiology, 72, 2280–2301. Prud'homme, M. J., Cohen, D. A., & Kalaska, J. F. (1994). Tactile activity in primate primary somatosensory cortex during active arm movements: Cytoarchitectonic distribution. Journal of Neurophysiology, 71, 173–181. Pulvermüller, F., Huss, M., Kherif, F., Moscoso del Prado Martin, F., Hauk, O., & Shtyrov, Y. (2006). Motor cortex maps articulatory features of speech sounds. Proceedings of the National Academy of Sciences of the United States of America, 103, 7865–7870. Purcell, D. W., & Munhall, K. G. (2006). Adaptive control of vowel formant frequency: Evidence from real-time formant manipulation. The Journal of the Acoustical Society of America, 119, 2288–2297. Recanzone, G. H., Merzenich, M. M., Jenkins, W. M., Grajski, K. A., & Dinse, H. R. (1992a). Topographic reorganization of the hand representation in cortical area 3b owl monkeys trained in a frequency-discrimination task. Journal of Neurophysiology, 67, 1031–1056. Recanzone, G. H., Merzenich, M. M., & Jenkins, W. M. (1992b). Frequency discrimination training engaging a restricted skin surface results in an emergence of a cutaneous response zone in cortical area 3a. Journal of Neurophysiology, 67, 1057–1070. Schürmann, M., Caetano, G., Jousmäki, V., & Hari, R. (2004). Hands help hearing: Facilitatory audiotactile interaction at low sound-intensity levels. The Journal of the Acoustical Society of America, 115, 830–832. Shadmehr, R., & Holcomb, H. H. (1997). Neural correlates of motor memory consolidation. Science, 277, 821–825. Shadmehr, R., & Mussa-Ivaldi, F. A. (1994). Adaptive representation of dynamics during learning of a motor task. The Journal of Neuroscience, 14, 3208–3224. Shore, S. E., & Zhou, J. (2006). Somatosensory influence on the cochlear nucleus and beyond. Hearing Research, 216–217, 90–99. Simani, M. C., McGuire, L. M., & Sabes, P. N. (2007). Visualshift adaptation is composed of separable sensory and taskdependent effects. Journal of Neurophysiology, 98, 2827–2841. Soso, M. J., & Fetz, E. E. (1980). Responses of identified cells in postcentral cortex of awake monkeys during comparable active and passive joint movements. Journal of Neurophysiology, 43, 1090–1110.
44 Taylor, M. M., & Creelman, C. D. (1967). PEST: Efficient estimates on probability functions. The Journal of the Acoustical Society of America, 41, 782–787. Tremblay, S., Shiller, D. M., & Ostry, D. J. (2003). Somatosensory basis of speech production. Nature, 423, 866–869. van Beers, R. J., Wolpert, D. M., & Haggard, P. (2002). When feeling is more important than seeing in sensorimotor adaptation. Current Biology, 12, 834–837.
Watkins, K. E., Strafella, A. P., & Paus, T. (2003). Seeing and hearing speech excites the motor system involved in speech production. Neuropsychologia, 41, 989–994. Xerri, C., Merzenich, M. M., Jenkins, W., & Santucci, S. (1999). Representational plasticity in cortical area 3b paralleling tactual-motor skill acquisition in adult monkeys. Cerebral Cortex, 9, 264–276.
A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 191 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.
CHAPTER 3
Sensory motor remapping of space in human–machine interfaces Ferdinando A. Mussa-Ivaldi{,{,},},*, Maura Casadio{,}, Zachary C. Danziger},}, Kristine M. Mosierk and Robert A. Scheidt# {
{
Department of Physiology, Northwestern University, Chicago, Illinois, USA Department of Physical Medicine and Rehabilitation, Northwestern University, Chicago, Illinois, USA } Department of Biomedical Engineering, Northwestern University, Chicago, Illinois, USA } Sensory Motor Performance Program, Rehabilitation Institute of Chicago, Chicago, Illinois, USA k Department of Radiology, Section of Neuroradiology and Imaging Sciences, Indiana University, Indianapolis, Indiana, USA # Department of Biomedical Engineering, Marquette University, Milwaukee, Wisconsin, USA
Abstract: Studies of adaptation to patterns of deterministic forces have revealed the ability of the motor control system to form and use predictive representations of the environment. These studies have also pointed out that adaptation to novel dynamics is aimed at preserving the trajectories of a controlled endpoint, either the hand of a subject or a transported object. We review some of these experiments and present more recent studies aimed at understanding how the motor system forms representations of the physical space in which actions take place. An extensive line of investigations in visual information processing has dealt with the issue of how the Euclidean properties of space are recovered from visual signals that do not appear to possess these properties. The same question is addressed here in the context of motor behavior and motor learning by observing how people remap hand gestures and body motions that control the state of an external device. We present some theoretical considerations and experimental evidence about the ability of the nervous system to create novel patterns of coordination that are consistent with the representation of extrapersonal space. We also discuss the perspective of endowing human–machine interfaces with learning algorithms that, combined with human learning, may facilitate the control of powered wheelchairs and other assistive devices. Keywords: motor learning; space; dimensionality reduction; human-machine interface; braincomputer interface.
*Corresponding author. Tel.: þ1-312-238-1230; Fax: 1-312-238-2208 E-mail:
[email protected] DOI: 10.1016/B978-0-444-53752-2.00014-X
45
46
Introduction
Motor learning
Human–machine interfaces (HMIs) come in several different forms. Sensory interfaces transform sounds into cochlear stimuli (Loeb, 1990), images into phosphenene-inducing stimuli to the visual cortex (Zrenner, 2002), or into electrical stimuli to the tongue (Bach-y-Rita, 1999). Various attempts, old and recent, have aimed at the artificial generation of proprioceptive sensation by stimulating the somatosensory cortex (Houweling and Brecht, 2007; Libet et al., 1964; Romo et al., 2000). Motor interfaces may transform electromyographic (EMG) signals into commands for a prosthetic limb (Kuiken et al., 2009), electroencephalogram (EEG) signals into characters on a computer screen, multiunit recordings from cortical areas into a moving cursor (Wolpaw and McFarland, 2004), or upper body movements into commands for a wheelchair (Casadio et al., 2010). Sensory and motor interfaces both implement novel transformations between the external physical world and internal neural representations. In a sensory interface, neural representations result in perceptions. In a motor interface, the neural representations reflect movement goals, plans, and commands. In a motor HMI, the problem of forming a functional map between neural signals and external environment is similar to remapping problems studied in earlier works, focused on the adaptation to force fields (Lackner and Dizio, 1994; Shadmehr and Mussa-Ivaldi, 1994) and dynamical loads. There, the environment imposed a transformation upon the relationship between the state of motion of the arm and forces experienced at the hand. The neural representation that formed through learning was an image in the brain of this new external relation in the environment. This image allows the brain to recover a desired movement of the hand by counteracting the disturbing force. Here, we take a step toward a more fundamental understanding of how space, “ordinary” space, is remapped through motor learning.
Recently, a simple and powerful idea has changed our view of motor learning. Motor learning is not only a process in which one improves performance in a particular act. Rather, it is a process through which the brain acquires knowledge about the environment. However, this is not the ordinary kind of knowledge (explicit knowledge) such as when we learn an equation or a historical fact. It is implicit knowledge that may not reach our consciousness, and yet it informs and influences our behaviors, especially those expressed in the presence of a novel situation. The current focus of most motor learning studies is on “generalization”; that is, on how experience determines behavior beyond what one has been exposed to. The mathematical framework for the concept of generalization comes from statistical theory (Poggio and Smale, 2003), where data points and some a priori knowledge determine the value of a function at new locations. If the new location is within the domain of the data, we have the problem of interpolation, whose solutions are generally more reliable than those of extrapolation problems, that is, when the predictions are made outside the domain of the data. In the early 1980s, Morasso (1981) and Soechting and Lacquaniti (1981) independently made the deceivingly simple observation that when we reach to a target, our hands tend to move along quasi-rectilinear pathways, following bell-shaped speed profiles. This simplicity or “regularity” of movement is evident only when one considers motion of the hand: In contrast, the shoulder and elbow joints engage in coordinated patterns of rotations that may or may not include reversals in the sign of angular velocities depending on the direction of movement. These observations gave rise to an intense debate between two views. One view suggested that the brain deliberately plans the shape of hand trajectories and coordinates muscle activities and joint motions accordingly (Flash and Hogan, 1985; Morasso, 1981). The opposing view suggested that the shape of the
47
observed kinematics is a side effect of dynamic optimization (Uno et al., 1989), such as the minimization of the rate of change of torque. By considering how the brain learns to perform reaching movements in the presence of perturbing forces (Lackner and Dizio, 1994; Shadmehr and Mussa-Ivaldi, 1994), studies of motor adaptation to force fields provided a means to address, if not completely resolve, this debate. Such studies have two key features in common. First, perturbing forces were not applied randomly but instead followed some strict deterministic rule. This rule established a force field wherein the amount and direction of the external force depended upon the state of motion of the hand (i.e., its position and velocity). The second important element is that subjects were typically instructed to move their hand to some target locations but were not instructed on what path the hand should have followed. If the trajectory followed by the hand to reach a target were the side effect of a process that seeks to optimize a dynamic quantity such as the muscle force or the change in joint torque rate, then moving against a force field would lead to systematically different trajectories than if hand path kinematics were deliberately planned. Contrary to the dynamic optimization prediction, many force-field adaptation experiments have shown that after an initial disturbance to the trajectory, the hand returns to its original straight motion (Fig. 1). Moreover, if the field is suddenly removed, an aftereffect is transiently observed demonstrating that at least a portion of the response is a preplanned (feedforward) compensatory response. Importantly, Dingwell et al. (2002, 2004) observed similar adaptations when subjects controlled the movement of a virtual mass connected to the hand via a simulated spring. In this case, adaptation led to rectilinear motions of the virtual mass and more complex movements of the hand. These findings demonstrate that the trajectory of the controlled “endpoint”—whether the hand or a hand-held object—is not a side effect of some dynamic optimization. Instead, endpoint
trajectories reflect explicit kinematic goals. As we discuss next, these goals reflect the geometrical properties of the space in which we move. What is “ordinary space”? We form an intuitive understanding of the environment in which we move through our sensory and motor experiences. But what does it mean to have knowledge of something as fundamental as space itself? Scientists and engineers have developed general mathematical notions of space. They refer to “signal space” or “configuration space.” These are all generalizations of the more ordinary concept of space. If we have three signals, for example, the surface EMG activities measured over three muscles, we can form a three-dimensional (3D) Cartesian space with three axes, each representing the magnitude of EMG activity measured over one muscle. Together, the measured EMG signals map onto a single point moving in time along a trajectory through this 3D space. While this mapping provides us with an intuitive data visualization technique, signal spaces are not typically equivalent to the physical space around us, the so-called ordinary space. In particular, ordinary space has a special property not shared by all signal spaces. In the ordinary space, the rules of Euclidean geometry and, among these Pythagoras’ theorem, support a rigorous and meaningful definition of both the minimum distance between two points (the definition of vector length) and the angle between two such vectors. Although we can draw a line joining two points in the EMG space described above, the distance between EMG points will carry little meaning. Moreover, what it means to “rotate” EMG signals by a given angle in this space is even less clear.1
1
Sometimes we carry out operations on signal spaces, like principal component analysis (PCA), which imply a notion of distance and angle. But in such cases, angles and distances are mere artifacts carrying no clear geometrical meaning.
48
F=
(a)
−10.1 −11.2 . r −11.2 11.1
(b)
Aftereffects
(c)
(d)
5 cm 5 cm
Fig. 1. Adaptation of arm movements to an external force field. Top-left: Experimental apparatus. The subject holds the handle of a two-joint robot manipulandum. Targets are presented on a computer monitor, together with a cursor representing the position of the hand. Top-middle: unperturbed trajectories, observed at the beginning of the experiment, with the motors turned off. Topright: velocity-dependent force field. The perturbing force is a linear function of the instantaneous hand velocity. In this case, the transfer matrix has a negative (stable) and a positive (unstable) eigenvalue. The force pattern in the space of hand velocity is shown under the equation. At the center (zero velocity) the force is zero. Bottom-left panels (A–D): evolution of hand trajectories in four successive epochs, while the subject practiced moving against the force field. The trajectories are averaged over repeated trials. The gray shadow is the standard deviation. In the final set, the trajectories are similar to those executed before the perturbation was turned on. Bottom-right: Aftereffects observed when the field was unexpectedly turned off at the end of training (modified from Shadmehr and Mussa-Ivaldi, 1994).
Euclidean properties of ordinary space The ordinary space within which we move is Euclidean (a special kind of inner product space). The defining feature of a Euclidean space is that
basic operations performed on vectors in one region of space (e.g., addition, multiplication by a scalar) yield identical results in all other regions of space. That is, Euclidean space is flat, not curved like Riemannian spaces: if a stick
49
measures 1 m in one region of Euclidean space, then it measures 1 m in all other regions of space. Although length and distance can be calculated in many ways, there is only one distance measure— the “Euclidean norm”—that satisfies Pythagoras’ theorem (a necessary condition for the norm to arise from the application of an inner product). The Euclidean norm is the distance measure we obtain by adding the squares of the projections of the line joining the two points over orthogonal axes. So, if we represent a point A in an N-dimensional space as a vector a ¼ [a1, a2, . . ., aN]T and a point B as a vector b ¼ [b1, b2, . . ., bN]T, then the Euclidean distance between a and b is qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi distða; bÞ ¼ ða bÞT ða bÞ qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ¼ ða1 b1 Þ2 þ ða2 b2 Þ2 þ þ ðaN bN Þ2 ð1Þ We are familiar with this distance in 2D and 3D space. But the definition of Euclidean distance is readily extended to N dimensions. The crucial feature of this metric, and this metric only, is that distances are conserved when the points in space are subject to any transformation of the Euclidean group, including rotations, reflections, and translations. The invariance by translations of the origin is immediately seen. Rotations and reflections are represented by orthogonal matrices that satisfy the condition RT R ¼ I
ð2Þ
(i.e., the inverse of an orthogonal matrix is its transpose). For example, if we rotate a line segment by R, the new distance in Euclidean space is equal to the old distance, since qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ½Rða bÞT Rða bÞ ¼ ða bÞT RT Rða bÞ qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ¼ ða bÞT ða bÞ ð3Þ
In summary, in the ordinary Euclidean space: 1. Distances between points obey Pythagoras’ theorem and are calculated by a sum of squares. 2. Distances (and therefore the size of objects) do not change with translations, rotations, and reflections. Or, stated otherwise, vector direction and magnitude are mutually independent entities.
Intrinsic geometry of sensorimotor signals in the central nervous system Sensory and motor signals in the nervous system appear to be endowed with neither of the above two properties with respect to the space within which we move. For example, the EMG activities giving rise to movement of our hand would generally change if we execute another movement in the same direction and with the same amplitude starting from a new location. Likewise, the firing rates of limb proprioceptors undoubtedly change if we make a movement with the same amplitude from the same starting location, but now oriented in a different direction. Nevertheless, we easily move our hand any desired distance along any desired direction from any starting point inside the reachable workspace. It therefore seems safe to conclude that our brains are competent to understand and represent the Euclidean properties of space and that our motor systems are able to organize coordination according to these properties. From this perspective, the observation of rectilinear and smooth hand trajectories has a simple interpretation. Straight segments are natural geometrical primitives of Euclidean spaces: they are geodesics (i.e., paths of minimum length). The essential hypothesis, then, is that the brain constructs and preserves patterns of coordination that are consistent with the geometrical features of the environment in which it operates.
50
Encoding the metric properties of Euclidean space Early studies of adaptation of reaching movements to force fields demonstrated the stability of planned kinematics in the face of dynamical perturbations (Shadmehr and Mussa-Ivaldi, 1994), suggesting that the brain develops an internal representation of the dynamics of the limb and its environment, which it uses to plan upcoming movements. The observation that subjects preferentially generate straight-line endpoint motions (Dingwell et al., 2002, 2004) further suggests that the nervous system also develops an internal representation of the environment within which movement occurs. Both representations are necessary to support the kind of learning involved in the operation of HMIs: Different HMIs require their users to learn the geometrical transformation from a set of internal signals endowed with specific metric properties (EEGs, multiunit activities, residual body motions, etc.) into control variables that drive a physical system with potentially significant dynamics (the orientation of a robotic arm, the position of a cursor, the speed and direction of a wheelchair, etc.). We next describe experiments that sought to test whether the brain constructs and preserves patterns of coordination consistent with the geometrical features of the environment using a noninvasive experimental approach with immediate relevance to the application of adaptive control in HMIs. Mosier et al. (2005) and colleagues (Liu and Scheidt, 2008; Liu et al., 2011) studied how subjects learn to remap hand gestures for controlling the motion of a cursor on a computer screen. In their experiments, subjects wore a data glove and sat in front of a computer monitor. A linear transformation A mapped 19 sensor signals from the data glove into two coordinates of a cursor on a computer screen: ax;1 ax;2 . . . ax;19 x0 x P¼ ¼ ay;1 ay;2 . . . ay;19 y0 y ð4Þ ½ h1
h2
...
h19
1 T ¼ AH
Subjects were required to smoothly transition between hand gestures so as to reach a set of targets on the monitor. This task had some relevant features, namely: 1. It was an unusual task. It was practically impossible that a subject had previous exposure to the transformation from hand gestures to cursor positions. 2. The hand and the cursor were physically uncoupled. Vision was therefore the only source of feedback information about the movement of the cursor available to the subjects. 3. There was a dimensional imbalance between the degrees of freedom of the controlled cursor (2) and the degrees of freedom of the hand gestures measured by the data glove (19). 4. Most importantly, there was a mismatch between the metric properties of the space in which the cursor moves and the space of the hand gestures. Specifically, the computer monitor defines a 2D Euclidean space with a well-defined concept of distance between points, whereas there is no clear metric structure for hand gestures. These features are shared by brain–machine interfaces that map neural signals into the screen coordinates of a computer cursor or the 3D position of a robotic arm. The hand-shaping task provides a simple noninvasive paradigm wherein one can understand and address the computational and learning challenges of brain–machine interfaces.
Learning an inverse geometrical model of space A linear mapping A from data-glove “control” signals to the two coordinates of the cursor creates a natural partition of the glove-signal space into two complementary subspaces. One is the 2D (x, y) task-space within which the cursor moves, HT ¼ AþAH [where Aþ ¼ AT(A AT) 1 is the Moore–Penrose (MP) pseudoinverse of A]. The second is its 17D null-space, HN ¼ (I19 AþA)H (where I19 is the 19D identity
51
matrix), which is everywhere orthogonal to the task-space (Fig. 2). Note that both task- and null-spaces are embedded in 19 dimensions. Given a point on the screen, the null-space of that point contains all glove-signal configurations that project onto that point under the mapping A (i.e., the null-space of a cursor position is the inverse image of that position under the handto-cursor linear map). Consider a hand gesture that generates a glove-signal vector B and suppose that this vector maps onto cursor position P. Because of the mismatch in dimensionality between the data-glove signal and cursor vectors (often referred to as “redundancy of control”), one can reach a new position Q in an infinite number of ways. In Fig. 2, the glove-signal space is depicted as a simplified 3D space. In this case, the null-space at
q is a line (because 3 signal dimensions 2 monitor dimensions ¼ 1 null-space dimension). Thus, one can reach Q with any configuration (C, D, E, F, etc.) on this line. However, the configuration C is special because it lies within the taskspace including B and thus, the movement BC is the movement with the smallest Euclidean norm (in the glove-signal space). In this simplified representation, the hand-to-cursor linear map partitions the signal space into a family of parallel planes orthogonal at each point to the corresponding null-space. While visualizing this in more than three dimensions is impossible, the geometrical representation remains generally correct and insightful. Consider now the problem facing the subjects in the experiments of Mosier et al. (2005). Subjects were presented with a target on the
(a)
(b)
LP
h3
h3
LQ
P
F E
B
P Q
D
B C h2
h2
h1
h1 A+ = AT (A·AT)−1 Definitions
T(A) = A+·A N(A) = IK −T(A)
Projections
Null-space component = hN = N(A)·h Task-space component = hT = T(A)·h h = hN + hT hNT ·h = 0
Fig. 2. Geometrical representation. (a) The “hand space,” H, is represented in reduced dimension as a 3D space. The matrix, A, establishes a linear map from three glove signals to a 2D computer monitor. T(A) and N(A) are the task-space and the nullspace of A. The line, LP, contains all the points in H that map onto the same point P on the screen. This line is the “null-space” of A at P. A continuous family of parallel planes, all perpendicular to the null-space and each representing the screen space, fills the entire signal space. (b) The starting hand configuration, B, lies on a particular plane in H and maps to the cursor position, P. All the dotted lines in H leading from B to LQ produce the line shown on the monitor. The “null-space component” of a movement guiding the cursor from P to Q is its projection along LQ. The “task-space component” is the projection on the plane containing BC. Bottom: The mathematical derivation of the null-space and task-space components generated by the transformation matrix A (from Mussa-Ivaldi and Danziger, 2009).
52
screen and were required to shape their hand so that the cursor could reach the target as quickly and accurately as possible. A number of investigators have proposed that in natural movements, the brain exploits kinematic redundancy for achieving its goal with the highest possible precision in task-relevant dimensions. Redundancy would allow disregarding performance variability in degrees of freedom that do not affect performance in task-space. This is a venerable theory, first published by Bernstein (1967) and more recently formalized as the “uncontrolled manifold” theory (Latash et al., 2001, 2002; Scholz and Schoner, 1999) and as “optimal feedback control” (Todorov and Jordan, 2002). These different formulations share the prediction that the motor system will transfers motor variability (or motor noise) to glove-signal degrees of freedom that do not affect the goal, so that performance variability at the goal—that is, at the target—is kept at a minimum. This is not a mere speculation; in a number of empirical cases the prediction matches observed behavior, as in Bernstein's example of hitting a nail with a hammer. However, in the experiments of Mosier et al. (2005) things turned out differently. As subjects became expert in the task of moving the cursor by shaping their hand, they displayed three significant trends with practice that were spontaneous and not explicitly instructed: 1. They executed increasingly straighter trajectories in task-space (Fig. 3a). 2. They reduced the amount of motion in the null-space of the hand-to-cursor map (Fig. 3b). 3. They reduced variability of motion in both the null-space and the task-space (Fig. 3c). Taken together, these three observations suggest that during training, subjects were learning an inverse geometric model of task-space. Consider that among all the possible right inverses of A, the MP pseudoinverse 1 ð5Þ Aþ ¼ AT AAT
selects the glove-signal solution with minimum Euclidean norm. This is the norm calculated as a sum of squares: qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð6Þ jjhjj ¼ h21 þ h22 þ þ h219 Passing through each point B in the signal space (Fig. 2), there is one and only one 2D plane that contains all inverse images of the points in the screen that are at a minimum Euclidean distance from B. The subjects in the experiment of Mosier et al. (2005) demonstrated a learning trend to move over these planes and to reduce the variance orthogonal to them—both at the targets and along the movement trajectory. We consider this to be evidence that the learning process is not only driven by the explicit goal of reaching the targets but also by the goal of forming an inverse model of the target space and its metric properties. This internal representation of space is essential to generalize learning beyond the training set. In a second set of experiments, Liu and Scheidt (2008) controlled the type and amount of taskrelated visual feedback available to different groups of subjects as they learned to move the cursor using finger motions. Subjects rapidly learned to associate certain screen locations with desired hand shapes when cued by small pictures of hand postures at screen locations defined by the mapping A. Although these subjects were also competent to form the gestures with minimal error when cued by simple spatial targets (small discs at the same locations as the pictures), they failed to generalize this learning to untrained target locations (pictorial cue group; Fig. 4). Subjects in a second group also learned to reduce taskspace errors when provided with knowledge of results in the form of a static display of final cursor position at the end of each movement; however, this learning also failed to generalize beyond the training target set (terminal feedback group; Fig. 4). Only subjects provided with continuous visual feedback of cursor motion learned to generalize beyond their training set
53
0.45
(b)
0.4
Null motion (G.S.U.)
Aspect ratio
(a)
0.35 0.3 0.25 0.2
1
2
3
240 220 200 180
24%
160 140
4
1
(c)
3
4
40
60
35
55 D1
45 40
D2
35
D3
30
D4
30 STD (G.S.U.)
50 STD (G.S.U.)
2 Day
Day
D1
25
D2 D3
20 15
D4 10
25 5
20 15
0
0.2
0.4 0.6 Null-space
0.8
1
0
0
0.2
0.4 0.6 Task-space
0.8
1
Fig. 3. Behavioral results of the hand-to-cursor mapping experiment. (a) Subjects execute progressively straighter trajectories of the cursor on the screen. This is measured by the aspect ratio, the maximum perpendicular excursion from the straight-line segment joining the start and end of the movement divided by the length of that line segment. The aspect ratio of perfectly straight lines is zero. (b) Length of subject movements in the null-space of the task, hand motion that does not contribute to cursor movement, decreases through training. (c) Average variability of hand movements over four consecutive days (D1, D2, D3, D4). Left: average standard deviation across subjects of the null-space component over the course of a single movement. Right: average standard deviation across subjects of the task-space component over a single movement. Standard deviations are in glove-signal units (G.S.U.), that is, the numerical values generated by the CyberGlove sensors, each ranging between 0 and 255. The x axes units are normalized time (0: movement start; 1: movement end). The overall variance decreases with practice both in the task- and in the null-space (from Mosier et al., 2005).
(continuous feedback group; Fig. 4) and so, visual feedback of endpoint motion appears necessary for learning an inverse geometrical model of the space of cursor motion. Of all the feedback conditions tested, only continuous visual feedback
provides explicit gradient information that can ^ of the facilitate estimation of an inverse model B hand-to-screen mapping A. Liu and colleagues further examined the learning of an inverse geometric representation
140 Euclidean distance [pixel]
Generalization target performance
54
110
80
Continuous feedback Pictorial cues Terminal feedback
50
3
9
15
21
27
33
Trial cycle number
Fig. 4. The ability to generalize beyond the trained target set depends on the type and amount of task-related visual feedback available during practice in moving the cursor using finger motions. Subjects performed 33 cycles of six movements, wherein a cycle consisted of one movement to each of five training targets (performed with visual feedback) plus a movement to one of three generalization targets (performed entirely without visual feedback). Each generalization target was visited once every three cycles. Each trace represents the across-subject average generalization error for subjects provided with continuous visual feedback of target capture errors (black squares), subjects provided with feedback of terminal target capture errors only (gray diamonds), and subjects provided with pictorial cues of desired hand shapes (gray circles). Error bars represent 1 SEM. We evaluated whether performance gains in generalization trials were consistent with the learning of an inverse hand-to-screen mapping or whether the different training conditions might have promoted another form of learning, such as the formation of associations between endpoint targets and hand gestures projecting onto them (i.e., a look-up table). Look-up table performance was computed as the across-subject average of the mean distance between the three generalization targets and their nearest training target on the screen. Because each subject's A matrix was unique, the locations of generalization and training targets varied slightly from one subject to the next. The gray band indicates the predicted mean 1 SD look-up table performance. Only those subjects provided with continuous visual feedback of cursor motion demonstrated generalization performance consistent with learning an inverse map of task-space (adapted from Liu and Scheidt, 2008).
of task-space by studying how subjects reorganize finger coordination patterns while adapting to rotation and scaling distortions of a newly learned hand-to-screen mapping (Liu et al., 2011). After learning a common hand-to-screen mapping A by practicing a target capture task on one day and refreshing that learning early on the next day, subjects were then exposed to either a rotation y of cursor motion about the origin (TR): xrotated cosðyÞ sinðyÞ x ¼ TR P ¼ PT ¼ sinðyÞ cosðyÞ y yrotated ð7Þ or a scaling k of cursor motion in task-space (TS): xscaled k 0 x PT ¼ ¼ TS P ¼ ð8Þ yscaled 0 k y The distortion parameters y and k were selected such that uncorrected error magnitudes were identical on initial application of T in both cases. The question Liu and colleagues asked was whether step-wise application of the two task-space distortions would induce similar or different reorganization of finger movements. Both distortions required a simple reweighting of the finger coordination patterns acquired during initial learning of A (Fig. 5a), while neither required reorganization of null-space behavior. Because A is a rectangular matrix with 2 rows and 19 columns, it does not have a unique inverse; rather, there are infinite 19 2 matrices B such that AB¼I2
ð9Þ
where I2 is the 2 2 unit matrix. These are “right inverses” of A, each one generating a particular glove-signal vector H mapping onto a common screen coordinate P. Liu et al. (2011) estimated ^ used the inverse hand-to-screen transformation B to solve the target acquisition task before and after adaptation to TR and TS by a least squares fit to the data:
55
^ ¼ HPT PPT 1 B
ð10Þ
^ obtained after They then evaluated how well B adaptation (BADAPT) was predicted by rotation ^ obtained just prior (TR) or scaling (TS) of the B to imposing the distortion (BBEFORE) by computing a difference magnitude DBADAPT: DBADAPT ¼ jjBADAPT BBEFORE T1 jj
ð11Þ
They compared this to the difference magnitude obtained from data collected in two separate time intervals during baseline training on the second day (i.e., before imposing the distortion; BL1 and BL2). Here, T 1 of Eq. (9) is assumed to be the identity matrix: DBNOISE ¼ jjBBL1 BBL2 jj
ð12Þ
Importantly, Liu and colleagues found that adaptation to the rotation induced a significant change in the subject's inverse geometric model of Euclidean task-space whereas adaptation to a scaling did not (Fig. 5b). Because the magnitude of initial exposure error was virtually identical in the two cases, the different behaviors cannot be accounted for by error magnitude. Instead, the results provide compelling evidence that in the course of practicing the target capture task, subjects learned to invoke categorically different compensatory responses to errors of direction and extent. To do so, they must have internalized the inner product structure imposed by the linear hand-to-screen mapping, which establishes the independence of vector concepts of movement direction and extent in task-space. Under the assumption that the brain minimizes energetic costs in addition to kinematic errors (see Shadmehr and Krakauer, 2008 for a review), subjects in the current study should at all times have used their baseline inverse map to constrain command updates to only those degrees of freedom contributing to task performance. This was not the case. The findings were also inconsistent with the general proposition that once the “structure” of a redundant task is learned, such dimensionality reduction is used to improve the
efficiency of learning in tasks sharing a similar structure (Braun et al., 2010). Instead, the findings of Mosier et al. (2005) and colleagues (Liu and Scheidt, 2008) demonstrate that as the subjects learned to remap the function of their finger movements for controlling the motion of the cursor, they also did something that was not prescribed by their task instructions. They formed a motor representation of the space in which cursor was moving and, in the process of learning, they imported the Euclidean structure of the computer monitor into the space of their control signals. This differs sharply from the trend predicted by the uncontrolled manifold theory, where a reduction in the variance at the target should have been accompanied by no such decrease in performance variance in redundant degrees of freedom. The experimental observations of Bernstein, Scholz, Latash, and others (Bernstein, 1967; Latash et al., 2001; Scholz and Schoner, 1999) can be reconciled with the observations of Mosier and colleagues if one considers that the glove task is completely novel, whereas tasks such as hitting a nail with a hammer are performed within the domain of a well learned control system. Because the purpose of learning is to form a map for executing a given task over a broad target space in many different situational contexts, it is possible that once a baseline competency and confidence in the mapping is established, the abundance of degrees of freedom becomes an available resource to achieve a more flexible performance, with higher variability in the null-space.
The dual-learning problem A HMI sets a relation from body-generated signals to control signals or commands for an external device. This relation does not need to be fixed. Intuition suggests that it should be possible to modify the map implemented by the interface so as to facilitate the learning process. In this spirit, Taylor et al. (2002) have employed a
56 (a)
Baseline
Early
Late
Early
Late
Rotation
Scaling
(b)
1.0
^ || DB || (GSU/pixel)
Adaptation
0.5
Washout
Rotation
DB D B NO IS D ADA E iff P er T en ce
DB D B NO IS D ADA E iff P er T en ce
0
Scaling
Fig. 5. Adaptation to rotation and scaling distortions of task-space 1 day after initially learning the manual target capture task. (a) Patterns of cursor trajectory errors are similar to those typically observed in studies of horizontal planar reaching with the arm. Here, we show data from representative subjects exposed to a ROTATION (top) or SCALING (bottom) of task-space during baseline (left) adaptation (early and late) as well as washout (early and late) blocks of trials. Shading indicates the adaptation block of trials. During preadaptation practice with the baseline map, cursor trajectories were well directed to the target. Imposing the step-wise counterclockwise (CCW) rotation caused cursor trajectories to deviate CCW initially but later “hook back” to the desired final position (Fig. 4a, top). With practice under the altered map, trajectories regained their original rectilinearity. When the baseline map was suddenly restored, initial trajectories deviated clockwise (CW) relative to trajectories made at the start of Session 2, indicating that subjects used an adaptive feedforward strategy to compensate for the rotation. These aftereffects were eliminated by the end of the washout period. Similarly, initial exposure to a step-wise increase in the gain of the hand-to-screen map resulted in cursor trajectories that far overshot their goal. Further practice under the altered map reduced these extent errors. Restoration of the baseline map resulted in initial cursor movements that undershot their goal. These targeting errors were virtually eliminated by the end of the washout period. (b) Adaptation to the rotation induced a significant change in the subject's inverse geometric model of Euclidean task-space whereas adaptation to a scaling did not. ^ is our measure of reorganization within the redundant articulation space, for subjects exposed to a rotation (red) and Here, DB scaling (black) of task-space, both before (solid bars) and after (unfilled bars) visuomotor adaptation. For the subjects exposed ^ after adaptation could not reasonably be characterized as a rotated version of the baseline map to the rotation distortion, B because DBADAPT far exceeded DBNOISE for these subjects. The within-subject difference between DBADAPT and DBNOISE was 0.44 0.32 G.S.U./pixel (red solid bar), from which we conclude that the rotational distortion induced these subjects to form a new inverse hand-to-screen map during adaptation. In contrast, DBADAPT did not exceed DBNOISE, for scaling subjects
57
where ^s ¼ ½l1 ; l2 ; x0 ; y0 T is a constant parameter vector that includes the link lengths and the origin of the shoulder joint. The virtual arm was not displayed except for the arm's endpoint, which was represented by a 0.5-cm-radius circle. Subjects
h1 aq1,1 aq1,2
…
aq2,1 aq2,2
…
aq1,19 aq2,19
.
h2 …
coadaptive movement prediction algorithm in rhesus macaques to improve cortically controlled 3D cursor movements. Using an extensive set of empirically chosen parameters, they updated the system weights through a normalized balance between the subject's most successful trials and their most recent errors, resulting in quick initial error reductions of about 7% daily. After significant training with exposure to the coadaptive algorithm, subjects performed a series of novel point-to-point reaching movements. They found that subjects’ performance in the new task was not appreciably different from the training task. This is evidence of successful generalization. Danziger et al. (2009) modified the glove-cursor paradigm by introducing a nonlinear transformation between the hand signals and the cursor (Fig. 6). In their experiment, the 19D vector of sensor values was mapped to the position of a cursor presented on a computer monitor. First, the glove signals were multiplied by a 2 19 transformation matrix to obtain a pair of angles. These angles then served as inputs to a forward kinematics equation of a simulated 2-link planar arm to determine the end-effector location: 2 3 h1 6 h2 7 y1 6 7 ¼ A6 .. 7 y2 4 . 5 ð13Þ h19 x cosðy1 Þ cosðy1 þ y2 Þ 1 0 ^s ¼ sinðy2 Þ sinðy1 þ y2 Þ 0 1 y
q1 =
z(q,s^)
q2
x y
h19 A
q2 q1
Fig. 6. Hand posture represented as a point in “hand space,” h, is mapped by a linear transformation matrix, A, into twojoint angles of a simulated planar revolute-joint kinematic arm on a monitor. The endpoint of the simulated arm was determined by the nonlinear forward kinematics, z. Subjects placed the arm's endpoint into displayed targets through controlled finger motions. During training, the elements of the A matrix were updated to eliminate movement errors and assist subjects in learning the task (from Danziger et al., 2009).
were given no information about the underlying mapping of hand movement to cursor position. The mapping matrix, A, was initially determined by having the subject generate four preset hand postures. Each one of these postures was placed in correspondence with a corner of a rectangle inside the joint angle workspace. The A matrix was then calculated as, A ¼ Y Hþ, where Y is a 2 4 matrix of angle pairs that represent the corners of the rectangle, and Hþ is the MP pseudoinverse of H (Ben-Israel and Greville, 1980), the 19 4 matrix whose columns are signal vectors corresponding to the calibration postures. Using the MP pseudoinverse corresponded to
(black gradient bars; p ¼ 0.942), yielding an average within-subject difference between DBADAPT and DBNOISE of 0.03 0.10 G.S.U./ pixel (black solid bar). We, therefore, found no compelling reason to reject the hypothesis that after adaptation, scaling subjects simply contracted their baseline inverse map to compensate for the imposed scaling distortion. Taken together, the results demonstrate that applying a rotational distortion to cursor motion initiated a search within redundant degrees of freedom for a new solution to the target capture task whereas application of the scaling distortion did not (adapted from Liu et al., 2010).
58
minimizing the norm of the A matrix in the Euclidean metric. As a result of this redundant geometry, each point of the workspace was reachable by many anatomically attainable hand postures. Danziger et al. asked subjects to shape their hands so as to move the tip of the simulated arm into a number of targets. The experiment proceeded in sets of training epochs. In each epoch, the mapping between the hand joint angles and the arm's free-moving tip (the “endeffector”) was updated so as to cancel the mean endpoint error in the previous set of movements. This was done in two ways by two separate subject groups: (a) by a least mean squares (LMS) gradient descent algorithm which takes steps in the direction of the negative gradient of the endpoint error function, or (b) by applying the MP pseudoinverse which offers an analytical solution for error elimination while minimizing the norm of the mapping. LMS (Widrow and Hoff, 1960) is an iterative procedure, which seeks to minimize the square of the performance error norm by iteratively modifying the elements of the A matrix in Eq. (13). The minimization procedure terminated when the difference between the old and the new matrix exceeded a preset threshold. In contrast, the MP procedure was merely a recalibration of the A matrix, which canceled the average error after each epoch. Therefore, both LMS and MP algorithms had identical goals, to abolish the mean error in each training epoch, and each method found a different solution. The result was that subjects exposed to the LMS adaptive update outperformed their control counterparts who had a constant mapping. But, surprisingly, the MP update procedure was a complete failure, and subjects exposed to this method failed to improve their skill levels at all (Fig. 7, left). We hypothesize that this was because the LMS procedure finds local solutions to the error elimination problem (because it is a gradient decent algorithm), while the MP update may lead to radically different A-matrices across epochs. This finding highlights a trade-off
between maintaining a constant structure of the map and altering the structure of the map so as to assist subjects in their learning. But perhaps the most important finding in that study was a negative result. In spite of the more efficient learning over the training set, subjects in the LMS group did not show any significant improvement over the control group on a different set of targets, which were not practiced during the training session (Fig. 7, right). The implication is that the LMS algorithms facilitated subjects’ creation of an associative map from the training targets to a set of corresponding hand configurations. However, this did not improve learning the geometry of the control space itself. Had this been the case, we would expect to see greater improvement in generalization. Finding machine learning methods that facilitate “space learning” as distinct from improving performance over a training set remains an open and important research goal in human–machine interfacing.
A clinical perspective: the body–machine interface The experiments of Mosier et al. (2005) and Danziger et al. (2009) demonstrated the ability of the motor system to reorganize motor coordination so as to match the low-dimensional geometrical structure of a novel control space. Subjects learned to redistribute the variance of the many degrees of freedom in their fingers over a 2D space that was effectively an inverse image of the computer monitor under the hand-to-cursor map. We now consider in the same framework the problem of controlling a powered wheelchair by coordinated upper body motions. People suffering from paralysis, such as spinal cord injury (SCI) survivors are offered a variety of devices for operating electrically powered wheelchairs. These include joysticks, head and neck switches, sip-and-puff devices, and other interfaces. All these devices are designed to match the motor control functions that are available to their users.
59 Normalized average endpoint reaching error MP compared with LMS and control performance
Movement errors in control and LMS subjects during generalization averages over many subjects
1.6 MP Control LMS
1.4
1.2 Control LMS
Average normalized error
1.1 Normalized error
1.2 1 0.8 0.6
1 0.9 0.8 0.7 0.6 0.5 0.4
0.4
0.3 0.2
1
2
3
4 5 6 Epoch number
7
8
0.2
1
2
3
4 5 6 Epoch number dashed lines separate days
7
8
9
Fig. 7. (Left) Average normalized movement errors for three subject groups in the experiment outlined in Fig. 6. The mapping for MP subjects was updated to minimize prior movement errors by an analytical method, which resulted in large mapping changes. The mapping for LMS subjects was also updated to minimize prior error but with a gradient descent algorithm that resulted in small mapping changes. Control subjects had a constant mapping. LMS subjects outperformed controls, while MP subjects failed to learn the task at all. (Right) Movement errors on untrained targets for control and LMS groups show that adaptive mapping updates does not facilitate spatial generalization (from Danziger et al., 2008, 2009).
However, they have a fixed structure and ultimately they present the users with challenging learning problems (Fehr et al., 2000). In general, the lack of customizability of these devices creates various difficulties across types and levels of disability (Hunt et al., 2004) and subjects with poor control of the upper body are at a greater risk of incurring accidents. Decades of research and advances in robotics and machine learning offer now the possibility to shift the burden of learning from the human user to the device itself. In a simple metaphor, instead of having the user of the wheelchair learning how to operate a joystick, we may have the wheelchair interface looking at the user's body as if it were a joystick. The controller of a powered wheelchair is a 2D device, setting the forward speed and the rotation about a vertical axis. Most paralyzed SCI
survivors have residual mobility much in excess of 2 degrees of freedom. Therefore, from a computational point of view one can see the control problem as a problem of embedding a 2D control surface within a higher-dimensional “residual motor space.” This is analogous to the problem of embedding the control space of a robotic arm within the signal space associated with a multiunit neural signal from a cortical area. From a geometrical standpoint, the embedding operation is facilitated by the ability of the motor control system to learn Euclidean metrics in a remapping operation, as shown in Mosier et al. (2005). While control variables may have a nonEuclidean Riemannian structure, a powerful theorem by Nash (1956) states that any Riemannian surface can be embedded within a Euclidean space of higher dimension. A simple way to
60
construct a Euclidean space from body motions is by principal component analysis (PCA; Jolliffe, 2002). This is a standard technique to represent a multidimensional signal in a Cartesian reference frame, whose axes are ordered by decreasing variance. Using PCA, Casadio et al. (2010) developed a camera-based system to capture upper body motions and control the position of a cursor on a computer monitor (Fig. 8). Both SCI injured subjects—at or above C5—and unimpaired control subjects participated in this study. Four small cameras monitored the motions of four small infrared active markers that were placed on the subjects’ upper arms and shoulders. Since each marker had a 2D image on a camera, the net signal was an 8D vector of marker coordinates. This vector defined the “body space.” The control space was defined by the two coordinates (x, y) of the cursor on the monitor. Unlike the hand-to-cursor map of the previous study, the body-to-cursor map was not based on a set of predefined calibration points. Instead, in the first part of the experiment subjects performed free
Fig. 8. Controlling a cursor by upper-body motion: experimental apparatus. Four infrared cameras capture the movements of four active markers attached to the subject's arm and shoulder. Each camera outputs the instantaneous x, y coordinates of a marker. The eight coordinates from the four cameras are mapped by linear transformation into the coordinates of a cursor, presented as a small dot on the monitor. The subject is asked to move the upper body so as to guide the dot inside a target (from Casadio et al., 2010).
upper body motions for 1 min. This was called the “dance” calibration. A rhythmic music background facilitated the subjects’ performance in this initial phase. The purpose of the dance was to evaluate how subjects naturally distributed motor variance over the signal space. The two principal component vectors, generating the highest variance of the calibration signals, defined two Cartesian axes over the signal space. In the calibration phase, subjects could scale the axis to compensate for the difference in variance associated with them. They were also allowed to rotate and/or reflect the axis to match the natural right-left, front-back directions of body space. After the calibration, subjects were engaged in a set of reaching movements. Both control and SCI subjects learned to execute efficiently the required motions of the cursor on the computer monitor by controlling their upper body movements (Fig. 9). Learning in terms of error reduction, increase in movement speed, and trajectory smoothness was evident both in controls and SCI subjects. In particular, all SCI subjects were able to use their shoulder movements for piloting the cursor for about 1 h. Importantly, subjects did not merely learn to track the cursor on the monitor. Instead, they acquired the broader skill of organizing their upper-body motions in “feedforward” motor programs, analogous to the natural reaching by hand. No statistically significant effect of vision could be detected, as well as no interaction between vision and practice when comparing movement executed under continuous visual feedback of the cursor, with movements where the cursor feedback was suppressed. Moreover, PCA succeeded in capturing the main characteristics of the upper-body movements for both control and SCI subjects. During the calibration phase, for all high-level SCI subjects it was possible to extract at least two principal components with significant variance from the 8D signals. Their impairment constrained and shaped the movements. Compared to control, they had on average a bigger variance associated with the first component and
61 Early Training
Late Training
Control
SCI 1
SCI 2
SCI 3
SCI 4
Fig. 9. Movement trajectories in early (left) phases of learning, for a control subject subjects. Calibration lines on bottom right panel: 1 cm on the computer screen (from 2010).
and late (right) and four SCI corner of each Casadio et al.,
smaller variances associated with the second through fourth components. Otherwise stated, the SCI subjects had a lower-dimensional upper body motor space. At the end of training, for all subjects the first three principal components accounted for more than 95% of the overall variance. Furthermore, the variance accounted for (VAF) by the two first principal components slightly increased with practice. However, there was a significant difference
between controls and SCI subjects. Controls mainly changed the movements associated with their degrees of freedom in order to use two balanced principal movements. They learned to increase the variance associated with the second principal component (Fig. 10), thus achieving a better balance between the variance explained by the first two components. This behavior was consistent with the consideration that subjects practiced a 2D task, with a balanced on-screen excursion in both dimensions. In contrast, at the end of the training, SCI subjects maintained the predominance of the variance explained by the first component: they increased the variance explained by the first component and decreased the fourth. Their impairment effectively constrained their movements during the execution of the reaching task as well as during the free exploration of the space. The most relevant findings of Casadio et al. (2010) concerned the distribution of variance across task-relevant and task-irrelevant dimensions. For control subjects, the VAF by the task-space with respect to the overall variance significantly increased with practice. In spite of the reduced number of training movements, the same trend was present in most SCI subjects. Therefore, as in the hand-cursor glove experiments of Mosier et al. (2005), subjects learned to reduce the variance that did not contribute to the motion of the cursor and demonstrated the ability to form an inverse model of the body-to-cursor transformation. As subjects reduced the dimensionality of their body motions, they also showed a marked tendency to align their movement subspace with the 2D space established by the body-cursor map (Fig. 11). It is important to observe that this was by no means an expected result. In principle, one could be successful at the task while confining one's movements to a 2D subspace that differs from the 2D subspace defined by the calibration. To see this, consider the task of drawing on a wall with the shadow of your hand. You can move the hand on any invisible surface with any orientation
62 80
80 First target set Last target set
First target set Last target set 60 Variance (%)
Variance (%)
60
40
20
20
0
40
1
2
3
4
5
6
7
8
0
1
2
3
4
5
6
7
8
Space dimensions
Principal component
Fig. 10. Distribution of motor variance across learning. Left panel: Results of principal component analysis on the first (gray) and last movement set (black) for control subjects (mean þ SE). In the first movement set (gray) more than 95% of variance was explained by four principal components. At the end of the training session (black), unimpaired controls mainly tended to increase the variance associated with the second principal component. Right panel: Control subjects (mean þ SE). Results of the projection of the data of the first (gray) and last movement set (black) over the 8D space defined by the body-cursor map. This transformation defines an orthonormal basis, where the “task-space” components a1, a2 determine the cursor position on the screen, and the orthogonal vectors a3, . . ., a8 represent the “null-space” components that do not change the control vector. For most of the control subjects, the fraction of movement variance in the null-space decreased with training in favor of the variance associated in the task-space (from Casadio et al., 2010).
(except perpendicular to the wall!). The result of Casadio and collaborators is analogous to finding that one would prefer to move the hand on an invisible plane parallel to the wall. Taken together, these results indicate that subjects were able to capture the structure of the task-space and to align their movements with it.
a3
a3
a1
a2
a1
a2
Conclusions Fig. 11. Matching the plane of the task. The limited number of dimensions involved in the task allowed us to project the body movement signals in a 3D subspace where the vectors a1,a2 define the “task-space” and a3 is the most significant nullspace component in terms of variance accounted for. In the first movement set (early phase of learning, left panel) there was a relevant movement variance associated with the nullspace dimension a3. That component was strongly reduced in the last target set (late phase of learning, right panel) where the movement's space became more planar, with the majority of the movement variance accounted by the task-space components a1, a2 (from Casadio et al., 2010).
The concept of motor redundancy has attracted consistent attention since the early studies of motor control. Bernstein (1967) pioneered the concept of “motor equivalence” at dawn of the past century by observing the remarkable ability of the motor system to generate a variety of movements achieving a single well-defined goal. As aptly suggested by Latash (2000), the very term “redundancy” is a misnomer as it implies
63
an excess of elements to be controlled instead of a fundamental resource of biological systems. We agree with Latash, and stick opportunistically with the term redundancy simply because it is commonly accepted and well understood. There is a long history of studies that have addressed the computational tasks associated with kinematic redundancy while others have considered the advantage of large kinematic spaces in providing ways to improve accuracy in the reduced space defined by a task. Here, we have reviewed a new point of view on this issue. We considered how the abundance of degrees of freedom may be a fundamental resource in the learning and remapping problems that are encountered in human–machine interfacing. We focused on two distinctive features: 1. The HMI often poses new learning problems and these problems may be burdensome to users that are already facing the challenges of disability. 2. By creating an abundance of signals—either neural recordings or body motions—one can cast a wide net over which a lower-dimensional control space can be optimally adapted. Work on remapping of finger and body movements over 2D task-spaces have highlighted the existence of learning mechanisms that capture the structure of a novel map relating motor commands to their effect on task-relevant variables. Both unimpaired and severely paralyzed subjects were able with practice not only to perform what they were asked to do but they also adapted their movements to match the structure of the novel geometrical space over which they operated. This may be seen as “suboptimal” with respect to a goal of maximal accuracy. Subjects did not shift their variance from the low-dimensional task to the null-space (or uncontrolled manifold). Instead, as learning progressed, variance in the null-space decreased as well as variance in the task-relevant variables. This is consistent with the hypothesis that through learning, the motor system strives to form an inverse map of
the task. This must be a function from the lowdimensional target space to the high-dimensional space of control variables. It is only after such a map is formed that a user may begin to exploit the possibility of achieving the same goals through a multitude of equivalent paths.
Acknowledgments This work was supported by the NINDS grants 1R21HD053608 and 1R01NS053581-01A2, by Neilsen Foundation, and Brinson Foundation.
References Bach-y-Rita, P. (1999). Theoretical aspects of sensory substitution and of neurotransmission-related reorganization in spinal cord injury. Spinal Cord, 37, 465–474. Ben-Israel, A., & Greville, T. N. E. (1980). Generalized inverses: Theory and application. New York, NY: John Wiley and Sons. Bernstein, N. (1967). The coordination and regulation of movement. Oxford: Pegammon Press. Braun, D., Mehring, C., & Wolpert, D. (2010). Structure learning in action. Behavioural Brain Research, 206, 157–165. Casadio, M., Pressman, A., Fishbach, A., Danziger, Z., Acosta, S., Chen, D., et al. (2010). Functional reorganization of upper-body movement after spinal cord injury. Experimental Brain Research, 207, 233–247. Danziger, Z., Fishbach, A., & Mussa-Ivaldi, F. (2008). Adapting Human-Machine Interfaces to User Performance. IEEE EMBC. Canada: Vancouver British Columbia. Danziger, Z., Fishbach, A., & Mussa-Ivaldi, F. A. (2009). Learning algorithms for human–machine interfaces. IEEE Transactions on Biomedical Engineering, 56, 1502–1511. Dingwell, J. B., Mah, C. D., & Mussa-Ivaldi, F. A. (2002). Manipulating objects with internal degrees of freedom: Evidence for model-based control. Journal of Neurophysiology, 88, 222–235. Dingwell, J. B., Mah, C. D., & Mussa-Ivaldi, F. A. (2004). An experimentally confirmed mathematical model for human control of a non-rigid object. Journal of Neurophysiology, 91, 1158–1170. Fehr, L., Langbein, W. E., & Skaar, S. B. (2000). Adequacy of power wheelchair control interfaces for persons with severe disabilities: A clinical survey. Journal of Rehabilitation Research and Development, 37, 353–360.
64 Flash, T., & Hogan, N. (1985). The coordination of arm movements: An experimentally confirmed mathematical model. The Journal of Neuroscience, 5, 1688–1703. Houweling, A. R., & Brecht, M. (2007). Behavioural report of single neuron stimulation in somatosensory cortex. Nature, 451, 65–68. Hunt, P. C., Boninger, M. L., Cooper, R. A., Zafonte, R. D., Fitzgerald, S. G., & Schmeler, M. R. (2004). Demographic and socioeconomic factors associated with disparity in wheelchair customizability among people with traumatic spinal cord injury. Archives of Physical Medicine and Rehabilitation, 85, 1859–1864. Jolliffe, I. T. (2002). Principal component analysis. New York, NY: Springer. Kuiken, T. A., Li, G., Lock, B. A., Lipcshutz, R. D., Miller, L. A., Subblefield, K. A., et al. (2009). Targeted muscle reinnervation for real-time myoelectric control of multifunction artificial arms. JAMA, 301, 619–628. Lackner, J., & Dizio, P. (1994). Rapid adaptation to Coriolis force perturbations of arm trajectory. Journal of Neurophysiology, 72, 299–313. Latash, M. (2000). There is no motor redundancy in human movements. There is motor abundance. Motor Control, 4, 259–261. Latash, M. L., Scholz, J. F., Danion, F., & Schoner, G. (2001). Structure of motor variability in marginally redundant multifinger force production tasks. Experimental Brain Research, 141, 153–165. Latash, M. L., Scholz, J. P., & Schoner, G. (2002). Motor control strategies revealed in the structure of motor variability. Exercise and Sport Sciences Reviews, 30, 26–31. Libet, B., Alberts, W. W., & Wright, E. W. (1964). Production of threshold levels of conscious sensation by electrical stimulation of human somatosensory cortex. Journal of Neurophysiology, 27, 546. Liu, X., & Scheidt, R. (2008). Contributions of online visual feedback to the learning and generalization of novel finger coordination patterns. Journal of Neurophysiology, 99, 2546–2557. Liu, X., Mosier, K. M., Mussa-Ivaldi, F. A., Casadio, M., & Scheidt, R. A. (2011). Reorganization of finger coordination patterns during adaptation to rotation and scaling of a newly learned sensorimotor transformation. Journal of Neurophysiology, 105, 454–473. Loeb, G. E. (1990). Cochlear prosthetics. Annual Review of Neuroscience, 13, 357–371. Morasso, P. (1981). Spatial control of arm movements. Experimental Brain Research, 42, 223–227.
Mosier, K. M., Scheidt, R. A., Acosta, S., & MussaIvaldi, F. A. (2005). Remapping hand movements in a novel geometrical environment. Journal of Neurophysiology, 94, 4362–4372. Mussa-Ivaldi, F. A., & Danziger, Z. (2009). The remapping of space in motor learning and human-machine interfaces. Journal of Physiology-Paris, 103(3-5), 263–275. Nash, J. (1956). The imbedding problem for Riemannian manifolds. Annals of Mathematics, 63, 20–63. Poggio, T., & Smale, S. (2003). The mathematics of learning: Dealing with data. Notices of the American Mathematical Society, 50, 537–544. Romo, R., Hernandez, A., Zainos, A., Brody, C. D., & Lemus, L. (2000). Sensing without touching: Psychophysical performance based on cortical microstimulation. Neuron, 26, 273–278. Scholz, J. P., & Schoner, G. (1999). The uncontrolled manifold concept: Identifying control variables for a functional task. Experimental Brain Research, 126, 289–306. Shadmehr, R., & Krakauer, J. (2008). A computational neuroanatomy for motor control. Experimental Brain Research, 185, 359–381. Shadmehr, R., & Mussa-Ivaldi, F. A. (1994). Adaptive representation of dynamics during learning of a motor task. The Journal of Neuroscience, 14, 3208–3224. Soechting, J. F., & Lacquaniti, F. (1981). Invariant characteristics of a pointing movement in man. The Journal of Neuroscience, 1, 710–720. Taylor, D. M., Tillery, S. I., & Schwartz, A. B. (2002). Direct cortical control of 3D neuroprosthetic devices. Science, 296, 1829–1832. Todorov, E., & Jordan, M. I. (2002). Optimal feedback control as a theory of motor coordination. Nature Neuroscience, 5, 1226–1235. Uno, Y., Kawato, M., & Suzuki, R. (1989). Formation and control of optimal trajectory in human multijoint arm movement. Biological Cybernetics, 61, 89–101. Widrow, B., & Hoff, M. (1960). Adaptive switching circuits. WESCON Conv Rec, 4, 99. Wolpaw, J. R., & McFarland, D. J. (2004). Control of a two-dimensional movement signal by a noninvasive brain–computer interface in humans. Proceedings of the National Academy of Sciences of the United States of America, 101, 17849–17854. Zrenner, E. (2002). Will retinal implants restore vision? Science, 295, 1022–1025.
A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 191 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.
CHAPTER 4
Locomotor adaptation Gelsy Torres-Oviedo{,{, Erin Vasudevan{,{,1, Laura Malone{,} and Amy J. Bastian*,{,{ {
Department of Motor Learning Lab, Kennedy Krieger Institute, Baltimore, Maryland, USA Neuroscience Department of Johns Hopkins School of Medicine, Baltimore, Maryland, USA Biomedical Engineering Department of Johns Hopkins School of Medicine, Baltimore, Maryland, USA {
}
Abstract: Motor learning is an essential part of human behavior, but poorly understood in the context of walking control. Here, we discuss our recent work on locomotor adaptation, which is an error driven motor learning process used to alter spatiotemporal elements of walking. Locomotor adaptation can be induced using a split-belt treadmill that controls the speed of each leg independently. Practicing split-belt walking changes the coordination between the legs, resulting in storage of a new walking pattern. Here, we review findings from this experimental paradigm regarding the learning and generalization of locomotor adaptation. First, we discuss how split-belt walking adaptation develops slowly throughout childhood and adolescence. Second, we demonstrate that conscious effort to change the walking pattern during split-belt training can speed up adaptation but worsens retention. In contrast, distraction (i.e., performing a dual task) during training slows adaptation but improves retention. Finally, we show the walking pattern acquired on the split-belt treadmill generalizes to natural walking when vision is removed. This suggests that treadmill learning can be generalized to different contexts if visual cues specific to the treadmill are removed. These findings allow us to highlight the many future questions that will need to be answered in order to develop more rational methods of rehabilitation for walking deficits. Keywords: locomotion; motor learning; adaptation; generalization of learning; rehabilitation.
Walking is a fundamental motor act. As such, it must be flexible enough to accommodate different environments, yet automatic enough so that we do not have to consciously focus on every step. Recently, we, and others, have been exploring the adaptability of locomotion with an eye toward improving rehabilitation of walking for people with brain lesions (e.g., Choi et al., 2009;
*Corresponding author. Tel.: þ443-923-2718; Fax: þ443-923-2715 E-mail:
[email protected]
1 Present address: Moss Rehabilitation Research Institute, Pennsylvania, USA.
DOI: 10.1016/B978-0-444-53752-2.00013-8
65
66
Reisman et al., 2007, 2009). This review will focus on what we know about adaptive processes for human walking control, and perhaps more importantly, what we do not know. Adaptive processes allow us to modify our locomotor patterns to suit changing environments. Since this is a critical ability for navigating the world, it is possible that adaptation develops at a very early age in humans. Conversely, the development of adaptation could follow a more protracted time course. This may be particularly true in human children, since they take much longer to learn how to walk independently than most other mammals. While humans typically begin walking 1 year after birth, many other mammals (e.g., horses, elephants) walk on the day that they are born. However, a recent study suggests that the late onset of human walking might be related to large brain mass, which takes extra time to develop (Garwicz et al., 2009). Indeed, if one considers the time from conception (rather than birth) to onset of walking, mammals with large brains relative to their body take longest to walk: humans ( 19–25 months) and elephants ( 22 months). Both animals have large brains—an adult human brain weighs 1350 g and an adult elephant brain weighs 4400 g. However, the percentage of brain mass with respect to the body is larger in humans than in elephants. Thus, brain development seems to be an important influence in dictating the onset of walking in mammals. Given the dependence of onset of walking on brain development, we wondered if other elements of walking control would follow a protracted developmental time course in humans as the nervous system matures. Specifically, we have been interested in understanding whether children can learn novel walking patterns through adaptive learning mechanisms. Although children are able to walk independently, we predicted that processes to adapt locomotor patterns would not be fully developed since human brain development continues well after birth, through childhood, and even into adulthood (LeBel et al., 2008).
We use a motor learning paradigm to study walking adaptation involving a split-belt treadmill, with independent belts under each leg (Reisman et al., 2005). Using this device, we can study people walking with the belts moving at the same speed, or “tied,” and with the belts moving at different speeds, or “split.” Figure 1a illustrates the general paradigm that is used for these studies. We have previously reported that adults adapt their walking pattern when walking in the split-belt condition over the course of 10–20 min. They specifically change step symmetry (i.e., the normalized difference in step sizes of the two legs; Fig. 1b), using both spatial and temporal strategies as described in Fig. 1c and d. When returning to tied belts, they show aftereffects in both domains, indicating that the nervous system learned and stored a new locomotor pattern that had to be actively unlearned. Recent work in our lab suggests that young children can adapt their walking pattern, but appear to show different developmental patterns for spatial versus temporal adaptation of walking (Vasudevan et al., 2011). Our initial intuition was that children might be more flexible in their ability to learn and, therefore, might adapt faster or more completely. Instead, we found that 3- to 5-year-old children adapt step symmetry slowly (Fig. 2a), and this ability does not fully develop until after age 12–14. Similar findings were present for the center of oscillation difference, which is defined as the difference in the midpoint position between heel strike (HS) and toe-off of each leg. Since the center of oscillation is dependent upon where the foot is placed at HS and where it is lifted off at toe-off, this measure reflects spatial locomotor control (Fig. 2b). In contrast, all ages could adapt the temporal parameter of phase at normal rates (Fig. 2c). Our interpretation of this finding is that the ability to adapt spatial control of walking depends on brain functions that are still developing through adolescence. Candidate sites are the cerebellum and motor cortex, though we consider the former to be more likely (Morton and Bastian, 2006).
67 (a)
Shoulder Pelvis Hip Limb angle
Knee
Baseline tied
Adaptation split
Postadapt tied
0.5 m/s 1.5 m/s
0.5 and 1.5 m/s
0.5 m/s
16 min
5 min
2 min
+
2 min
Ankle MT
Step symmetry
(c)
Center of oscillation shift 30
Degrees
Slow HS
0
−30
Step sym =
Phase shift 30
Slow HS Fast HS
0
−30
1s Fast step
(d)
Fast HS Degrees
(b)
1s
Slow step Fast step − Slow step Fast step + Slow step
Fast step
Slow step
Fast step
Slow step
Fig. 1. (a) Diagram of marker locations and an example of the paradigm structure. Limb angle convention is shown on the stick figure (left panel). Panel on the right shows an example experimental paradigm indicating the periods of split and tied-belt walking. The walking pattern is first recorded during a baseline period in which both treadmill belts move at the same speed. Then, changes to the walking pattern are recorded during an adaptation period in which one belt moves two to four times faster than the other. Finally, stored changes to the walking pattern are assessed during a deadaptation period in which the treadmill belts move at the same speed as in the baseline period. (b) An example of kinematic data of two consecutive steps is shown. Kinematic data for every two steps were used to calculate step symmetry, defined as the difference in step lengths normalized by the step lengths sum. (c) Figure adapted from Malone and Bastian (2010). Limb angle trajectories plotted as a function of time in late split-belt adaptation—two cycles are shown. Gray trajectory represents the movement in the slow limb in early adaptation. Positive limb angles are when the limb is in front of the trunk (flexion). Two time points are marked—slow heel strike (HS) in black and fast HS in gray. The spread between the limb angles is directly proportional to the step lengths shown in the bottom. Step lengths can be equalized by changing the position of the foot at landing (i.e., the “spatial” placement of the foot). This spatial strategy is known as a shift in the center of oscillation difference since subjects change midpoint angle around which each leg oscillates, with respect to the other leg. (d) Step lengths can also be equalized by changing the timing of foot landing, as shown by the change in phasing of the slow limb from the gray trajectory (early adaptation) to the black trajectory. This purely temporal strategy is known as phase shift since subjects equalize step lengths by changing the timing of foot landings with respect to each other.
This result is interesting and raises many issues about development of movement adaptability. First, it suggests that the nervous system gains some adaptive abilities in late childhood. This is counter to the belief that, because children are developing, they are “more plastic” and should
adapt faster. Of course, an important question is whether there are advantages to adapting slower as a child—since children adapt more slowly, do they also deadapt slower and does this make them retain more from day to day, for example? A second issue is whether this result would be
68
Step symmetry
Adaptation
Deadaptation
0 0.2
−0.1
0.1
−0.2
0
−0.3
10
3–5 yo 12–14 yo Adults
Center of oscillation
0 5 −5 0 −10 0.08
0.02
Phasing
0.06
0
0.04
−0.02
0.02
−0.04
0
−0.06
−0.02 0
100
200
300
400
−0.08 500 0 Number of steps
100
200
300
Fig. 2. Rates of adaptation (left column) and deadaptation (right column) in 3- to 5-year olds (red; n ¼ 10), 12- to 14-year olds (blue; n ¼ 10), and adults (black; n ¼ 10). Step symmetry data are shown in the top row, center of oscillation difference in the middle and phasing on the bottom. Shaded regions indicate standard error. Data were fit with linear, single-exponential, or double-exponential functions depending on which fit resulted in the highest r2 values. For 3- to 5-year-old step symmetry and center of oscillation difference, linear fits were best; double-exponential fits were best for the phasing data. A single exponential fit was used for 12to 14-year-old center of oscillation difference adaptation data and all remaining 12- to 14-year-old data were best fit by doubleexponential functions. All adult data were fit by double-exponential functions.
observed in adaptation of other kinds of movements, such as finger control. Clearly, there are differences in which brain areas are involved in these different kinds of movements. Walking heavily engages brainstem circuits, which may make its control more unique. Along this line, a third question is what neural substrates are important for adapting temporal versus spatial control of walking and do they control other movements (i.e., reaching)? We are particularly interested in knowing whether spinal circuits are involved in this adaptive process. Previously, we have shown that the cerebellum is necessary for
walking adaptation (Morton and Bastian, 2006), but have not been able to probe spinal contributions directly. Finally, do children learn better or faster when trained for longer periods of time (days rather than minutes)? This would obviously be more relevant for rehabilitation, since training is done over days to weeks. Another set of recent studies from our group has used a similar split-belt treadmill paradigm in healthy adults to explore whether we can change the rate of walking adaptation, and whether we can promote generalization of the adapted pattern to overground walking. These
69
questions are important not only to understand the adaptive process but also to determine how best to leverage this type of learning for rehabilitation. We would like to optimize the amount of adaptation, how long it lasts, and its transfer to more natural walking conditions. We first tested whether adaptation and deadaptation rates could be altered by (a) asking people to consciously correct their walking pattern to aid adaptation, or (b) distracting them with a dual task during adaptation (Malone and Bastian, 2010). Figure 3a shows the basic paradigm—subjects were tested in baseline tied-belt conditions with no instruction. We then asked each of the three groups to (1) consciously correct their step sizes to be equal by watching their feet on a video screen, (2) perform a secondary task while watching a video, or (3) simply walk with no instructions or distraction. Here, we assessed the adaptation and deadaptation rates. The deadaptation rate is perhaps more interesting in this particular study because all manipulations (e.g., distraction, conscious corrections) were removed in the deadaptation period. Figure 3b illustrates the main result from this study—adaptation and deadaptation of step symmetry were faster with conscious corrections and slower with distraction (Malone and Bastian, 2010). Thus, conscious corrections during adaptation sped the process up, but this did not lead to better retention in deadaptation. In contrast, distraction slowed the adaptation process, but resulted in better retention since deadaptation was also slower. This demonstrates that the conditions under which the nervous system learns are important, as they strongly influence the pattern of unlearning. In this work, we also found that the conscious correction and distraction effects were due to changes in the rate of adapting the spatial pattern, but not the temporal pattern (Fig. 3c and d). In other words, conscious corrections to change the step size were implemented by changing where the foot was placed, and not when it was moved there. Interestingly, distraction slowed spatial adaptation
only, despite the fact that there was no indication of what to change in this condition—subjects could have changed either the spatial or temporal components of walking. These results suggest that adaptation of spatial locomotor control is more flexible and accessible than temporal control. One interpretation of this finding is that different neural structures are involved in these two control processes, and that spatial control is more easily accessed using conscious cerebral resources. However, timing control may operate at a lower level in the nervous system, such as the brainstem or spinal cord, and is therefore less accessible through cerebral resources. The cerebellum, which is known to influence both spatial and temporal control, has projections to both cerebral motor areas and brainstem regions. Thus, there may be distinct anatomical circuits for these adaptive learning processes. These results bring up several important questions. First, does distraction lead to better day-to-day retention of newly learned movement patterns? In other words, if a person is distracted during training, will the effects last longer? Second, in rehabilitation, people are often instructed how to move and asked to “try” to move in the desired way. However, our results suggest that patients would retain more of what they learn if they do not use conscious or voluntary resources. Therefore, it is possible that a more effective rehabilitation strategy may be to put patients into a situation that drives the learning of a new pattern without having to use voluntary effort. In other words, perhaps patients would learn better if they were not “trying” so hard. Given our interest in patient rehabilitation, a third interesting question is whether similar effects of conscious correction versus distraction would be observed in patient populations. Can people who have a cerebral stroke, for example, benefit in any way from distraction? Do they even respond in the same way to conscious efforts? In sum these issues have important significance for rehabilitation of walking. Another important aspect of motor learning is how well the adapted pattern transfers to
70 Baseline tied
Adaptation split
Deadapt tied
0.5 m/s
1.5 m/s
0.5 and 1.5 m/s
0.5 m/s
2 min
2 min
16 min
5 min
control: no instructions conscious correction: feedback corrections distraction: dual task
all groups: no instructions
all groups: no instructions
Adaptation
(b)
Deadaptation
0
−0.2 −0.3 −0.4 −0.5 −0.6
(c)
400
Plateau *
300 200 100 0
Plateau 200 150 100
*
50 0
14 12
Center of oscillation difference (degrees)
Control Conscious correction Distraction
Number of strides
Step symmetry
−0.1
0.6 0.4 0.2 0 0.6 0.4 0.2 0 0.6 0.4 0.2 0
Number of strides
(a)
0
200
−5
150
−10
100
100
0
50
0
−5
0
400
*
300
10 200
8 6
−10
4
0
2
−5 −10
0
(d)
0.60
Phasing
0.56
200
400
0.50
300
0.45
200
0.40
100
0.50
50
0
0.45
0
0.58
0.54 0.52
150 100
0.40 0.50
0.50 0.48
0.45 0.40 60 120 180 240 300 360 420 480 540
30 60 90 120
Stride number
Fig. 3. (a) Experimental paradigm showing the periods of split-belt walking and conditions. In baseline, tied walking all groups were given no specific instructions. Subjects were divided into three groups for adaptation (split belts). The conscious correction group (N ¼ 11) was instructed on how to step more symmetrically and given intermittent visual feedback of their stepping during adaptation. The distraction group (N ¼ 11) was given an auditory and visual dual-task they were asked to focus on. The control
71
untrained environments or situations. The amount of transfer, or generalization, indicates how much of the adapted circuit is used in different situations. This question of generalization of device-induced motor learning across different environments has been addressed in recent studies (e.g., Berniker and Kording, 2008; Cothros et al., 2009; Kluzik et al., 2008; McVea and Pearson, 2007; Wolpert et al., 1998 ). Here, we discuss it in the context of human locomotion. Our prior work has shown that healthy subjects transfer little of the split-belt adaptation to overground walking (Reisman et al., 2009). Instead, it seems that they link the adapted pattern to the context of being on the treadmill. Given our interest in using split-belt treadmills to rehabilitate walking patterns in people with brain lesions, we wanted to understand if we could improve the generalization of split-belt treadmill adaptation to more natural walking situations. We hypothesized that treadmill walking has some unique features that provide very strong contextual cues to people as they walk on it, the main one being the mismatch between vision and proprioception. Specifically, when walking on a treadmill, proprioception tells us that we are moving, but vision tells us that we are not. This is a highly unusual situation, and the nervous system may therefore link the adapted pattern to this particular context. We tested whether removing vision during split-belt treadmill adaptation could improve overground transfer of the new walking pattern
(Torres-Oviedo and Bastian, 2010). Subjects walked with or without vision during an adaptation and transfer experiment. Figure 4a illustrates the basic paradigm—subjects walked overground and on the treadmill before and after split-belt adaptation. They were given a “catch” trial of tied-belt walking during adaptation so that we could assess how much they had learned prior to testing the transfer of adaptation effects to overground walking. Figure 4b shows individual subject data for step symmetry from periods of this experiment. Both subjects adapted, though the aftereffects during the catch trial in the subject from the no-vision group were larger than the one from the vision group, indicating that this first subject learned more. Transfer to overground walking was also markedly different between these subjects—the one without vision transferred much more than the one with vision. When subjects returned to the treadmill there was again a striking difference—the subject with no vision showed much greater washout of the adapted pattern compared to the subject with vision. Group data for step symmetry are shown in Fig. 4c–e. Similar changes were observed in phasing (i.e., temporal control). This work demonstrates that altering the sensory context can change the extent to which treadmill learning transfers to natural overground walking. We speculate that this could be for a couple of reasons. One possibility is that it changes a person's perception of the source of the error during adaptation (i.e., credit
group (N ¼ 11) was given no specific instructions. In deadaptation (tied belts), all groups walked under “Control” conditions, where the visual feedback and distracter were removed. (b) Adaptation and deadaptation curves for step symmetry. Average adaptation curves for the three groups, with standard errors indicated by the shaded area. Baseline values are subtracted out from curves (i.e., symmetry is indicated by a value of 0). Average deadaptation curves for the three groups. Recall that all groups deadapted under the same condition (no feedback or distraction). Curves are shown individually to more clearly illustrate the plateau level. Bar graphs represent group averages for adaptation and deadaptation rate, assessed by the number of strides until plateau is reached (i.e., behavior is level and stable). Note that with step symmetry, the conscious correction group adapted faster, and the Distraction group adapted slower. However, retention was improved with the Distraction group because they took longer to deadapt, despite removal of the distracter. (c) Adaptation and deadaptation curves for the center of oscillation difference. Average adaptation curves for the three groups plotted as in (b). Trends seen in the center of oscillation difference are comparable to those seen in step symmetry. (d) Average adaptation and deadaptation curves for phasing, plotted as similar to (b). Note that our interventions did not significantly affect the rate of adaptation or deadaptation of phasing.
72 (a)
Paradigm
Baseline over ground
baseline tied
self-selected speed all groups
Adaptation split
0.5 m/s
1.0 m/s
0.5 and 1.0 m/s
2 min
2 min
10 min
all groups
5 min
group1: Vision group 2: noVision
Postadapt over ground
Postadapt tied
self-selected speed
0.5 m/s
all groups
all groups
5 min
Catch (tied-belts) 0.5 m/s, 10 sec (b)
Sample subjects Baseline OG TM
Catch TM
Postadaptation OG TM Vision No-vision
Step symmetry
0.8 0.6 0.4 0.2 0 −0.2 0 30 Steps
(c)
2472 Steps
Learning
0.6
(d)
*
2640
Transfer
3780
(e)
Washout
100
100
80
80
*
% Washout
% Transfer
0.4
40
60
40
si Vi
si ovi N
si Vi
si
on si Vi
on
0 on
0 on
20
ovi
N
ov
is
io n
0
20
on
0.2
N
Step sym catch
* 60
Fig. 4. (a) Overall paradigm. In all groups, baseline behavior was recorded overground (OG) and subsequently on the treadmill with the two belts moving at 0.7 m/s. Then subjects were adapted for a total of 15 min, during which one belt was moving at 0.5 m/s and the other belt at 1 m/s. After 10 min of adaptation, a 10-s catch trial was introduced, in which both belts moved at 0.7 m/s. Subjects were readapted (i.e., belts’ ratio at 2:1) for five more minutes before they were asked to walk OG, where we tested the transfer of treadmill adaptation to natural walking. Subjects were transported on a wheelchair to a 6-m walkway where they walked back-and-forward 15 times. All steps on the walkway were recorded except for those when subjects were
73
assignment) from the treadmill to the person (Berniker and Kording, 2008). If this were the case, the person would learn to associate the newly learned calibration to one's faulty movements, rather than to being on the treadmill. A second is that closing the eyes may have led to an upweighting of proprioceptive information. It is possible that errors derived from proprioceptive signals encode learning in intrinsic (i.e., body centered) coordinates and thus learning could be more easily carried with the person when they move off of the treadmill. These results also lead to several questions. First, is it necessary to actually remove vision to improve transfer, or can this be done through other means? For example, if visual and proprioceptive information were congruent during splitbelt adaptation, would transfer to overground walking improve? We have started to study this using optic flow patterns displayed to the individuals as they walk. We can manipulate optic flow to match or oppose the proprioceptive signals and would like to be able to understand how these two sources of information are integrated. If it is important to upweight proprioceptive information from the legs to improve transfer to natural walking, adding congruent vision may not help. However, if it is important to remove the sensory mismatch and make the
adaptation context more similar to natural walking situations, then adding optic flow may improve it. Another important question is whether individuals with stroke will show a similar effect from changing the sensory context during split-belt treadmill adaptation. Our previous work has shown that people with cerebral lesions caused by stroke (e.g., middle cerebral artery distribution), can adapt their walking pattern and show better transfer to overground walking than controls (Reisman et al., 2009), even with eyes open. Will changing the visual information to match the proprioceptive inputs improve this transfer? We think that it is unrealistic to adapt stroke patients without vision and, therefore, would like to use visual displays to manipulate visual information during this task. Finally, it is not understood whether credit assignment or the ability to assign errors to the environment versus the body is developed throughout childhood. Therefore, we would like to know how children transfer split-belt treadmill adaptation. Does an immature nervous system transfer newly adapted patterns more readily? If so, does this mean that they have difficulty learning context-dependent walking calibrations? These questions are important for reaching our ultimate goal of optimizing this process for long-term training of adults and children with brain damage.
turning to return to the initial position. Finally, subjects returned to the treadmill where they walked for 5–10 min at 0.7 m/s to determine form the remaining aftereffects the extent to which walking without the device washed out the learning specific to the treadmill. (b) Spatial symmetry (i.e., symmetry in step lengths of the two legs) of sample subjects of the vision and no-vision group when walking on the treadmill (TM) and OG during baseline, catch, and deadaptation periods. Behavior of two sample subjects is shown: one walking with vision (gray trace) and one walking without vision (black trace). Lines represent the running average using a three-step window SD (shaded area). No differences in step symmetry were observed preadaptation when subjects walked with and without vision on the treadmill or OG. However, the subject that walked without vision had larger aftereffects on the treadmill during the catch trial (i.e., more learning), more transfer of treadmill learning to OG walking, and more washout of learning specific to the treadmill than subject that walked with vision. (c) Aftereffects on treadmill during catch trial for vision and no-vision groups. Subjects that trained without vision had significantly larger aftereffects—greater learning, than subjects that trained with vision. Bars’ height indicates the averaged aftereffects of the first three steps during the catch trial across subjects SE. (d) Transfer of adaptation effects to OG walking. (e) Washout of treadmill spatial aftereffects following OG walking. Removing vision during training had a significant effect on the washout of step symmetry aftereffects specific to the treadmill. Step symmetry transfer and washout are expressed as a percentage of the aftereffects on the treadmill during catch. Bars’ height indicates the average across subjects SE of % transfer and % washout for the first three steps OG or when returning to the treadmill. Figures in all panels were adapted from Torres-Oviedo and Bastian (2010). *p<0.01.
74
To summarize, we discussed three experiments about walking adaptation. First, humans develop the ability to adapt walking patterns throughout childhood. Children are slower to adapt spatial elements of the walking pattern, and this improves throughout childhood until adolescence. In contrast, temporal adaptation is remarkably conserved even in 3-year olds. This distinction suggests that neural circuits that develop at different times might be involved in spatial versus temporal adaptive processes. Walking adaptation is also a highly automatic process. However, it can be sped up when healthy adults try to consciously modify their walking pattern, but this does not improve retention. Conversely, when distracted during adaptation, healthy adults learn slower, but retain the walking pattern longer. These conscious correction versus distraction effects are due to changes in spatial control of the walking pattern, suggesting that it is accessible through cerebral mechanisms. Finally, we show that transfer of learning from the treadmill to natural overground walking is greatly enhanced by removing visual cues specific to the treadmill context. This may be due to changes in sensory reweighting or to changes in credit assignment. Many other questions remain to be answered, as illustrated throughout this review, which we hope will ultimately lead to more rational bases for walking rehabilitation.
Acknowledgments This work was supported by NIH Grants F32 NS063642 and R01 HD048741.
References Berniker, M., & Kording, K. (2008). Estimating the sources of motor errors for adaptation and generalization. Nature Neuroscience, 11(12), 1454–1461.
Choi, J. T., Vining, E. P., Reisman, D. S., & Bastian, A. J. (2009). Walking flexibility after hemispherectomy: Split-belt treadmill adaptation and feedback control. Brain, 132(Pt 3), 722–733. Cothros, N., Wong, J., & Gribble, P. L. (2009). Visual cues signaling object grasp reduce interference in motor learning. Journal of Neurophysiology, 102, 2112–2120. Garwicz, M., Christensson, M., & Psouni, E. (2009). A unifying model for timing of walking onset in humans and other mammals. Proceedings of the National Academy of Sciences USA, 106(51), 21889–21893. Kluzik, J., Diedrichsen, J., Shadmehr, R., & Bastian, A. J. (2008). Reach adaptation: What determines whether we learn an internal model of the tool or adapt the model of our arm? Journal of Neurophysiology, 100, 1455–1464. LeBel, C., Walker, L., Leemans, A., Phillips, L., & Beaulieu, C. (2008). Microstructural maturation of the human brain from childhood to adulthood. Neuroimage, 40, 1044–1055. Malone, L. A., & Bastian, A. J. (2010). Thinking about walking: Effects of conscious correction versus distraction on locomotor adaptation. Journal of Neurophysiology, 103(4), 1954–1962. McVea, D., & Pearson, K. (2007). Contextual learning and obstacle memory in the walking cat. Integrative and Comparative Biology, 47, 457–464. Morton, S. M., & Bastian, A. J. (2006). Cerebellar contributions to locomotor adaptations during splitbelt treadmill walking. The Journal of Neuroscience, 26, 9107–9116. Reisman, D. S., Block, H. J., & Bastian, A. J. (2005). Interlimb coordination during locomotion: What can be adapted and stored? Journal of Neurophysiology, 94, 2403–2415. Reisman, D. S., Wityk, R., Silver, K., & Bastian, A. J. (2007). Locomotor adaptation on a split-belt treadmill can improve walking symmetry post-stroke. Brain, 130(Pt. 7), 1861–1872. Reisman, D. S., Wityk, R., Silver, K., & Bastian, A. J. (2009). Split-belt treadmill adaptation transfers to overground walking in persons poststroke. Neurorehabilitation and Neural Repair, 23(7), 735–744. Torres-Oviedo, G., & Bastian, A. J. (2010). Seeing is believing: Effects of visual contextual cues on learning and transfer of locomotor adaptation. The Journal of Neuroscience, 30(50), 17015–17022. Vasudevan, E. V., Torres-Oviedo, G., Morton, S. M., Yang, J. F., & Bastian, A. J. (2011). Younger is not always better: development of locomotor adaptation from childhood to adulthood. Journal of Neuroscience, 31, 3055–3065. Wolpert, D., Miall, R., & Kawato, M. (1998). Internal models in the cerebellum. Trends in Cognitive Sciences, 2, 338–347.
A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 191 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.
CHAPTER 5
Age-related changes in the cognitive function of sleep Edward F. Pace-Schott{,{ and Rebecca M. C. Spencer{,{,* {
Department of Psychology and Neuroscience, University of Massachusetts, Amherst, USA { Neuroscience and Behavior Program, University of Massachusetts, Amherst, USA
Abstract: Healthy aging is characterized by a diminished quality of sleep with decreased sleep duration and increased time awake after sleep onset. Older adults awaken more frequently and tend to awaken less from rapid eye movement (REM) sleep and more from non-REM (nREM) sleep than young adults. Sleep architecture also begins changing in middle age leading to a dramatic decrease in the deepest stage of nREM—slow wave sleep (SWS)—as aging progresses. Other less marked nREM changes include reduced numbers of sleep spindles and K-complexes. In contrast, the amount of REM diminishes only slightly. Both circadian and homeostatic sleep-regulatory processes are affected by aging. Circadian rhythms of temperature, melatonin, and cortisol are phase advanced and their amplitude diminished. An increased number of nocturnal awakenings and diminished daytime sleepiness suggest diminished homeostatic sleep pressure. A variety of endocrine and neuromodulatory changes (e.g., reduced growth hormone and dopamine levels) also accompany healthy aging. Healthy aging is characterized by declines in working memory and new episodic memory performance with relative sparing of semantic memory, recognition memory, and priming. Memory systems impacted by aging are associated with volumetric and functional changes in fronto-striatal circuits along with more limited changes in medial temporal structures (in which larger aging-related changes suggest neuropathology). Cross-sectional studies generally associate poorer sleep quality with poorer neuropsychological functioning. However, paradoxically, older adults appear to be more resistant to the cognitive effects of sleep deprivation, restriction, and fragmentation than younger adults. A new and expanding field examines the interaction between aging and sleep-dependent memory consolidation. Among forms of learning displaying prominent sleep-dependent consolidation in young adults, motor-sequence learning displays loss of sleep-dependent consolidation with aging whereas sleep-dependent consolidation of verbal declarative memory appears spared. Findings suggest that improving sleep through behavioral or pharmacological treatments may enhance cognition and performance in older adults. *Corresponding author. Tel.: þ1 413 545 5987; Fax: þ1 413 545 0996 E-mail:
[email protected] DOI: 10.1016/B978-0-444-53752-2.00012-6
75
76
Keywords: sleep; aging; memory; consolidation; motor skill.
Introduction As the population ages, increasing effort has been invested in understanding and improving the wellbeing of healthy older adults. Much of this research has focused on the decline in cognitive function associated with normal aging. In parallel, research has provided insight into age-related changes in sleep. In the present chapter, we will review evidence of an interaction between these two functions in healthy aging, that is, a building literature suggesting that changes in sleep may, in part, contribute to changes in cognition in healthy older adults. Such data suggest that by improving sleep through behavioral or pharmacological treatments, cognition and performance may be enhanced in older adults. Improving sleep quality represents an attractive potential strategy to ameliorate sensory, motor, cognitive, and emotional deficits that accompany aging because interventions can be designed to optimize a naturally occurring restorative process. Moreover, although primary and secondary sleep disorders increase in frequency with aging, it is also often the case that sleep quality in older adults is poor not because of specific pathologies but because of lifestyle factors (e.g., daytime napping, alcohol consumption) that oppose optimum nocturnal sleep. In such cases, behavioral and psychoeducational interventions (“sleep hygiene”) can prove extremely effective in reducing the effects of insufficient sleep on waking performance.
Changes in sleep with healthy aging Aging is associated with distinct and interrelated changes in sleep quality and architecture as well as changes in sleep's circadian timing and homeostatic regulation. The current section briefly reviews sleep changes in healthy aging. Not
reviewed, however, are sleep changes associated with major medical, neuropsychiatric, and sleep disorders, the incidence of which dramatically increases in middle to late adulthood (Bliwise, 1993) with perhaps greater than 50% of the population over 65 facing compromised sleep (Neikrug and Ancoli-Israel, 2010; Vitiello, 2006). Of those with primary sleep disorders, insomnia, obstructive sleep apnea, and periodic leg movements are among the most common (for reviews see Ancoli-Israel et al., 2008; Neikrug and Ancoli-Israel, 2010; Wolkove et al., 2007). Sleep is also degraded in older adults with certain cardiovascular conditions (Riegel and Weaver, 2009), and disruption of circadian rhythms accompanies many neurodegenerative diseases of aging such as Alzheimer's disease (Wu and Swaab, 2007) and Parkinson's disease (Menza et al., 2010). The most obvious change in sleep of healthy older adults is a decrease in total sleep time (TST). This is paralleled by increased time spent awake during the night, or wake after sleep onset (WASO; Bliwise, 1993; Buysse et al., 2005; O'Donnell et al., 2009) and thereby decreased sleep efficiency, that is, proportion of total time in bed spent asleep (Buysse et al., 2005; Huang et al., 2002). There may be as much as a 100% increase in WASO (Bliwise, 1993). Sleep efficiency declines from approximately 86% at ages 37–54 to 79% over age 70 (Bliwise, 2005). In a longitudinal study, Hoch et al. (1994, 1997) found that while general measures of sleep quality (sleep efficiency, WASO, sleep onset latency) appear stable over time in “young-old” (aged 60–74), there is evidence of sustained decline over time in “old–old” (75–87) individuals. More frequent awakenings in the elderly are associated with a lesser tendency to awaken predominantly from rapid eye movement (REM) sleep and relatively more frequent
77
awakenings from nonrapid eye movement (nREM) sleep, especially nREM2 as described below (Dijk et al., 2001; Salzarulo et al., 1999). Because awakenings from nREM produce more severe “sleep inertia” (immediately postawakening grogginess that is believed to be caused by carryover of a deactivated brain state into waking), older adults may be more susceptible to the cognitive effects of sleep inertia (Silva and Duffy, 2008). Also contributing to sleep disruption in the elderly are normal behavioral changes that often accompany aging such as increased nighttime urination (nocturia; Asplund and Aberg, 1996) and increased daytime napping (Carskadon et al., 1982) which can diminish subsequent overnight sleep.
Changes in sleep architecture accompanying healthy aging In addition to general changes in sleep patterns, the architecture of sleep (the sequential
physiological stages which occur during sleep) changes through the course of aging (Fig. 1). A key feature distinguishing sleep stages is the frequency of oscillations in the EEG signal. These oscillations reflect more or less synchronous changes in membrane potential among large assemblages of cortical neurons. With deepening of nREM sleep through its three stages, nREM1, nREM2, and SWS, these oscillations become progressively slower in frequency and higher in amplitude reflecting greater degrees of synchronous change in membrane potential among cortical neurons. Greater degrees of synchrony are believed to reflect lesser brain activation and greater attenuation of consciousness with the slowest frequency and highest amplitude occurring in the delta (0.75–4.5 Hz) oscillations of SWS. nREM2, a sleep stage characterized by a background EEG frequency in the theta (4–7 Hz) range, is also marked by sleep spindles, groups of faster oscillations lasting around 0.5 s, and K-complexes, high amplitude negative-thenpositive deflections in the EEG signal. The slow
Sleep stage
wake (W) REM (R) nREM1 (N1) nREM2 (N2) SWS (N3)
0
1
2
0
1
2
3
4
5
6
7
6
7
Sleep stage
wake (W) REM (R) nREM1 (N1) nREM2 (N2) SWS (N3)
3 4 5 Time after lights off (h)
Fig. 1. Hypnograms for young (top panel) and older (lower panel) adults illustrating the timing and transitions of sleep stages across an overnight sleep bout. REM: rapid eye movement sleep; nREM1: Stage 1 nonrapid eye movement sleep; nREM2: Stage 2 nonrapid eye movement sleep; SWS: slow wave sleep. Abbreviations in parentheses reflect the new (2007) conventions of the American Academy of Sleep Medicine.
78
EEG rhythms of nREM reflect internal neural activity between the thalamus and cortex (delta, spindle) as well as, in the case of what is termed the slow (< 1 Hz) oscillation, between different regions of the cortex itself (Steriade, 2006). It has been suggested that changes in these circuits and their endogenous activity during sleep may differentiate healthy versus cognitively impaired aging (Cantero et al., 2009). Brain deactivation of nREM is reversed during REM when mixed high-frequency, low-amplitude EEG rhythms reflect desynchronization of cortical networks, increased brain activation, and the return of consciousness in the form of enhanced dreaming. The best replicated sleep architectural change associated with aging is a reduction in SWS and a broader distribution of this stage across the sleep bout rather than being concentrated in early overnight sleep as in younger adults (see Fig. 1; Lombardo et al., 1998). Reduced SWS with aging is accompanied by reduced EEG spectral power in the delta range, a quantity termed slow wave activity (SWA; Cajochen et al., 2006; Carrier et al., 2001; Ohayon et al., 2004). Although there appear to be few gender differences in the impact of healthy aging on sleep (Carrier et al., 2001), SWS is markedly better preserved with aging in females (Bliwise, 2005; Fukuda et al., 1999). With the relative loss of SWS, the proportion of total sleep spent in nREM1 and nREM2 is increased in healthy older adults (Ohayon et al., 2004) along with an increase in EEG spectral power in the faster (beta, 15–30 Hz) frequencies that are also characteristic of waking (Munch et al., 2010). Additionally, there is a decrease in the number of sleep spindles (Carrier et al., 2001; Crowley et al., 2002; Landolt et al., 1996; Wei et al., 1999) and K-complexes in older adults (Crowley et al., 2002). Unlike SWS, the amount of REM is relatively unchanged with aging declining by less than 1% per decade and, at times, increasing in individuals over 70 years (Floyd et al., 2007). Despite the relative stability of REM percent, REM density
(number of REMs per unit time) is reduced in older adults (Darchia et al., 2003). Fast gamma frequency rhythms (30–80 Hz), although much investigated as correlates of higher order and effortful cognition in waking (Jensen et al., 2007) and of REM sleep (e.g., Abe et al., 2008; Clemens et al., 2009) have been little studied in relation to comparative sleep of older and younger adults although observations under anesthesia (Pritchett et al., 2008) suggest differences may be present. Similarly, the slow (< 1 Hz) oscillation (for review see Crunelli and Hughes, 2010) that has received much recent interest with regard to human sleep and cognition (Marshall et al., 2006; Marshall and Born, 2007) has been investigated little in relation to sleep and aging (but see Anderson and Horne, 2003).
Changes in circadian and homeostatic regulation of sleep accompanying healthy aging Borbely's two-process model of sleep suggests that human sleep propensity results from the interaction of circadian and homeostatic mechanisms (Borbely, 1982). Circadian processes are physiological and behavioral rhythms with a 24-h period that are normally entrained to (synchronized with) the local astronomical day/ night cycle. In contrast, homeostatic mechanisms increase sleep propensity proportionately to the duration of prior waking. The master regulator of circadian processes is the suprachiasmatic nucleus (SCN) of the anterior hypothalamus, whereas homeostatic mechanisms are believed to act within basal forebrain structures (for reviews see Pace-Schott and Hobson, 2002; Saper et al., 2005). Circadian and homeostatic processes acting in concert allow consolidated nocturnal sleep and daytime waking (Dijk and von Schantz, 2005). As detailed in the next sections, both circadian and homeostatic processes are affected by aging (Bliwise, 2005). By isolating subjects from all external time cues and placing them on a sleep–wake schedule
79
outside of the range of times to which the circadian oscillator (in the SCN) can be entrained (e.g., on 20 or 28-h “days”), the influences of circadian and homeostatic processes can be dissociated (Czeisler et al., 1999). This causes the circadian clock to “free run” thereby allowing sleep to be studied at all endogenous circadian phases as well as circadian effects to be differentiated from homeostatic effects. Under this schedule, older adults were found to have 2.7 times more awakenings from sleep but a similar propensity to fall back asleep as younger adults, a pattern that occurred throughout bed rest periods at most circadian phases. Because these older individuals returned to sleep as readily as young ones, Klerman et al. (2004) conclude that it is difficulty remaining asleep versus difficulty falling asleep that accounts for poorer sleep continuity in older adults.
activity patterns are phase advanced in relation to melatonin secretion (Knoblauch et al., 2005). In addition, although showing similar circadian variation in core body temperature relative to younger adults, older adults show decreased amplitude of circadian variation in objective and subjective sleepiness (Buysse et al., 2005) and poorer ability to tolerate rapid phase shifts (Monk, 2005). The underlying physiological basis of these age-related changes may be a weakening of the master circadian signal (Hofman and Swaab, 2006), a change that may be exacerbated by increased time indoors away from natural light cues (Van Someren, 2000) or degenerative changes in the eye itself (Bliwise, 2005). Nonetheless, the response of the circadian clock to resetting by bright light remains similar in young and older adults (Benloucif et al., 2006; Van Someren et al., 2002).
Changes in circadian regulation of sleep accompanying healthy aging
Changes in homeostatic regulation of sleep accompanying healthy aging
Although the period of the circadian oscillator remains remarkably stable at about 24.2 h across the lifespan (Czeisler et al., 1999), other aspects of circadian rhythms have been shown to change with aging. For example, in older versus younger adults, the circadian rhythms of core body temperature, melatonin, and cortisol are phase advanced by approximately 1 h, and the amplitude of these rhythms may be 20–30% lower (Dijk et al., 2000; Monk, 2005). Compared with younger adults, older adults both fall asleep and awaken earlier with respect to rising and falling levels of melatonin (Duffy et al., 2002). Therefore, compared to younger adults, during their more frequent awakenings and especially at their characteristically earlier final morning awakening, older adults are awakening at an earlier circadian phase (Dijk et al., 1999, 2000; Dijk and Duffy, 1999; Duffy et al., 1998, 2002). In older versus younger adults, sleep spindle
The two-process model of sleep suggests that human sleep propensity results from the interaction of circadian and homeostatic mechanisms, the latter being proportional to the duration of prior wakefulness (Borbely, 1982). In addition to changes observed in the circadian regulation of sleep with aging, decreased homeostatic sleep pressure in older adults has also been hypothesized as cause of reduced TST and sleep efficiency. An increased number of nocturnal awakenings and diminished daytime sleepiness in older adults suggest diminished homeostatic sleep pressure (Carrier et al., 2001; Dijk et al., 2010). For example, both objective (multiple sleep latency test) and subjective (Karolinska sleepiness scale) diurnal sleepiness measures are reduced in older compared to younger adults (Dijk et al., 2010). Paradoxically, reduced sleepiness coexists with a high frequency of daytime napping in many older adults resulting from
80
lifestyle factors combined with an unawareness of the detrimental effects of daytime napping on nocturnal sleep (Ancoli-Israel et al., 2008). However, in older and younger adults, homeostatic responses are proportionately similar when sleep pressure is experimentally increased by sleep deprivation or decreased by napinduced sleep saturation (Cajochen et al., 2006).
Sleep-related neuroendocrine changes accompanying healthy aging In addition to changes in circadian mechanisms (e.g., reduced melatonin secretion; Pandi-Perumal et al., 2005) and sleep-homeostatic effects of aging, changes in other neuroendocrine and neuromodulatory systems may contribute to alterations in sleep quality and architecture (for review see Bliwise, 2005). Briefly, lowered SWS with aging is associated with lowered levels of growth hormone, which is secreted during this sleep stage (Bliwise, 2005). An increase in mean daily cortisol level may also contribute to changes in sleep duration and sleep stage percentages with aging (Bliwise, 2005). Interestingly, however, CNS levels of orexin, reduced levels of which are responsible for rapid state transitions in narcolepsy (Saper et al., 2001), appear to remain constant across the lifespan (Kanbayashi et al., 2002). Nonetheless, baseline levels of dopamine, serotonin, and norepinephrine, all of which are involved in arousal and behavioral state control (Pace-Schott and Hobson, 2002), diminish with aging (Hedden and Gabrieli, 2004) and such changes may, in turn, contribute to changes in the sleep–wake and REM–nREM cycles associated with aging.
Changes in cognition with healthy aging Memory decline with healthy aging involves both working memory and new episodic memory with relative sparing of semantic memory (e.g.,
vocabulary), autobiographical memory, recognition memory, emotional memory, and priming (Buckner, 2004; Drag and Bieliauskas, 2010; Hedden and Gabrieli, 2004). Such changes may be associated with diminished capacity of two dissociable systems: the fronto-striatal circuits and the medial temporal memory system (Buckner, 2004; Hedden and Gabrieli, 2004). Changes in fronto-striatal systems are believed to be the characteristic substrate of cognitive changes in healthy aging, whereas large changes in medial temporal systems are characteristic of Alzheimer's disease and other dementias (Buckner, 2004; Hedden and Gabrieli, 2004). Nonetheless, smaller volumetric and functional changes are observed in the normally aging hippocampal formation (Hedden and Gabrieli, 2004). Normal aging within fronto-striatal circuits is accompanied by white matter changes and declines in neurotransmitters such as dopamine, serotonin, and norepinephrine (Buckner, 2004; Hedden and Gabrieli, 2004). Structural studies indicate reduction in gray matter with aging, especially in prefrontal cortex and to a lesser extent in striatum and the medial temporal areas (Hedden and Gabrieli, 2004). Among functional studies, a recent meta-analysis, comparing younger and older adults performing tasks spanning multiple cognitive domains, emphasizes much greater recruitment of prefrontal areas in older adults (Spreng et al., 2010). In contrast, young adults show greater occipital activity and this frontal shift in functional activation with aging has been dubbed “the posterior–anterior shift in aging” (Davis et al., 2008). Regions more activated in older adults performing the same tasks as younger adults comprise anterior portions of a “task positive network,” a group of largely lateral cortical structures involved in cognitive control and attention, suggesting compensatory recruitment of these areas when older adults perform tasks that can be accomplished by more limited networks in younger adults (Spreng et al., 2010). It has been suggested that such added,
81
compensatory recruitment often occurs in analogous areas contralateral to those preferentially recruited when young adults perform similar tasks resulting in a more bilateral pattern of activation (Hedden and Gabrieli, 2004). It is particularly notable that putative compensatory recruitment that includes lateral prefrontal areas is also associated with performance of cognitive tasks following 36 h sleep deprivation compared to rested waking (Drummond et al., 2005). Cross-sectional studies of cognitive change with aging typically report steady declines in a number of domains, including processing speed, encoding new episodic memories, and working memory from early adulthood onward, whereas longitudinal studies suggest stability of such functions up until approximately age 60 after which decline commences (Hedden and Gabrieli, 2004; Salthouse, 2009). Similarly, cross-sectional studies using a battery of 12 diverse tasks (Salthouse, 2009) estimated average annual declines of 0.02–0.03 standard deviations prior to age 60 and 0.04–0.05 between ages 61 and 96. Interactions between sleep and cognition in healthy aging Given the parallel decreases in sleep and cognitive measures in older populations, recent studies have begun to consider an association between these factors. While this work has predominantly used cross-sectional correlations and sleep-deprivation paradigms, more recent attention has been directed at measures of changes in cognition that preferentially occur over sleep compared to waking that are termed “sleep-dependent consolidation.” We will review these literatures in turn. Sleep deprivation and restriction studies in healthy aging Although it has been suggested that sleep deprivation in young adults may temporarily induce the same cognitive changes typical of
nonsleep-deprived, healthy older adults (Harrison et al., 2000), paradoxically, cognitive performance is less disrupted in older versus younger adults following sleep fragmentation (Bonnet, 1989), total sleep deprivation (Bonnet and Arand, 1989), and sleep restriction (Bliese et al., 2007; Stenuit and Kerkhofs, 2005). Moreover, recovery of baseline function following sleep deprivation does not differ for older versus young adults (Bonnet and Rosa, 1987). Therefore, older adults appear to be more resistant to the cognitive effects of sleep deprivation and sleep restriction than younger adults. For example, over a 26h wake episode, young and older adults performed similarly on a psychomotor vigilance test during normal waking hours, but during normal sleeping hours, older subjects were less impaired, showing faster reaction times and fewer lapses in performance (Duffy et al., 2009). Similarly, under more naturalistic conditions, greater impairment of psychomotor vigilance was seen in young (21–30) compared to older (61–70) individuals following 40 h of total sleep deprivation (Adam et al., 2006). Likewise, young adults were also more impaired on a simple reaction time task relative to older adults 52–62 years of age (Philip et al., 2004). More recent studies have shown less subjective sleepiness and maintained reaction time in older versus younger subjects during the extended exposure to experimentally altered circadian phase (O'Donnell et al., 2009; Silva et al., 2010).
Cross-sectional and longitudinal studies of sleep and cognition in healthy aging Cross-sectional studies in the elderly have generally found negative associations between sleep quality and neuropsychological functioning (Blackwell et al., 2006; Nebes et al., 2009). Paradoxically, however, a recent study of 3212 older adults in Spain clearly associated longer sleep duration with poorer cognitive functioning (Faubel et al., 2009). Therefore, longitudinal studies
82
that associate sleep quality and cognitive changes may be preferable. For example, in a longitudinal study of a community sample of over 6000 persons age 65 or older who were cognitively intact at baseline, presence of liberally defined, self-reported insomnia at baseline independently predicted cognitive decline at 3-year follow-up in men but not women (Cricco et al., 2001). Moreover, it has been suggested that cognitive decline that appears in such longitudinal studies may in fact be underestimated due to masking of such declines by test–retest (practice) effects (Salthouse, 2009). Therefore, poorly sleeping older adults may be at even greater risk of cognitive decline than is suggested by such longitudinal studies. Given that SWS most substantially declines in healthy aging, a number of studies have crosssectionally linked SWS with cognitive performance, revealing changes in this relationship across the lifespan. Young adults show a clear positive relationship between proportion of sleep spent in SWS and reaction time on a variety of vigilance tasks (Jurado et al., 1989). In contrast, in middle-aged individuals (40–59 years), only some but not other vigilance measures correlated with SWS (Edinger et al., 2000). Among older adults (over 60 years), a positive relationship between spectral power in the delta band and reaction time on a variety of vigilance tasks was only seen among individuals with insomnia complaints but not in those without such complaints (Crenshaw and Edinger, 1999). Therefore, the relationship between SWS and cognitive performance may weaken as the amount of SWS diminishes with aging. Cross-sectionally measured relationships between cognition and the cortico-cortical slow (< 1 Hz) oscillation may also exist. For example, spectral power at slow oscillation frequencies correlates with performance on verbal tasks in older adults (Anderson and Horne, 2003). Whereas younger adults are predominantly evening chronotypes (“night owls”), older adults
are predominantly morning chronotypes (Schmidt et al., 2007). While the relationship between cognitive performance and testing at favorable versus unfavorable times of day (based on chronotype) differs with age in some studies (Hasher et al., 2002; May et al., 2005; May and Hasher, 1998), others have failed to find such relationships (Brown et al., 1999; Hogan et al., 2009).
Sleep-dependent consolidation studies in healthy aging Following learning (encoding), memories continue to be processed. Processing of memories when no longer engaged in learning is termed memory consolidation and typically involves the stabilization and strengthening of these memory traces (Diekelmann et al., 2009; Diekelmann and Born, 2010). Whereas consolidation can occur over intervals of waking, in healthy young adults, consolidation is optimized over sleep. Sleepdependent memory consolidation has been observed following a wide range of learning tasks in young adults and these findings have been extensively reviewed (Diekelmann et al., 2009; Diekelmann and Born, 2010; Stickgold, 2005; Walker, 2009). In young adults, sleep-dependent consolidation has been shown to be greatest for tasks that engage the hippocampus during learning (Ferrara et al., 2008; Spencer et al., 2006; Wilson, 2002). This has been demonstrated extensively with the paired associates task in which participants learn semantically related word pairs (Plihal and Born, 1997, 1999; Rasch et al., 2007; Tucker et al., 2006) and, more recently, on learning novel associations (Ellenbogen et al., 2006, 2009). However, procedural learning also benefits from sleep. Motorsequence learning is considered representational of procedural learning. Walker et al. (2002) and others (Fischer et al., 2002; Spencer et al., 2006, 2007) have illustrated as much as 18% enhancement in motor-sequence learning following sleep.
83
A handful of recent studies have probed whether sleep-dependent memory processes also change with increasing age. Prior to experimental assessments, two theoretical papers hypothesized that sleep-dependent consolidation would be diminished in older adults. Hornung et al. (2005) predicted a relationship between the reduction in memory and the changes in sleep physiology that are commonly observed in older adults. According to this theoretical paper, reductions in sleep and memory may share a common source such as reduced brain volume or cardiovascular or neurochemical changes. Conversely, the relationship may be direct: changes in sleep— such as reduced SWS, REM, or spindle density—may result in reduced memory. According to the hypothesis of Buckley and (2005), changes in the Schatzberg hypothalamo–pituitary–adrenal (HPA) axis in older adults underlie the reduction in sleep qualities and thereby reduction of memory in older adults. The HPA axis is a feedback loop that controls the stress response and other processes via cortisol release. Cortisol also fluctuates in a circadian rhythm with a peak in the morning and falling throughout the day to an evening nadir. The circadian rhythm is flattened in older adults, and the HPA axis becomes hyperactive. The combination of these two changes, according to Buckley and Schatzberg (2005), results in the age-related changes in sleep that, in turn, reduce sleep-dependent memory consolidation. In the first direct test of whether older adults preferentially consolidate learning over a period of sleep, older (45–80 years) and younger adults (18–24 years) adults were compared using a motor-sequence learning task (Spencer et al., 2007). This task is a variant of the serial reaction time task (Nissen and Bullemer, 1987) in which participants press response keys based on the spatial position of visual stimuli presented on a computer monitor which appear in a repeating sequence, with occasional presentations of unsequenced stimuli that allow sequence versus general learning to be differentiated. This task
(Fig. 2) as well as similar tasks have clearly demonstrated sleep-dependent memory consolidation in young adults (Spencer et al., 2006; see also Robertson et al., 2004; Walker et al., 2002). However, performance changes in the older adult group following an overnight interval with sleep did not differ from performance changes following a daytime interval spent awake (Fig. 2). Notably, this was the case either under conditions in which participants were aware that stimuli were arranged in a repeating series (explicit learning) or conditions under which they implicitly learned sequences of which they were unaware. Moreover, there was no difference in initial learning between younger and older individuals regardless of the time of day at which they were tested indicating that the observed changes were not due to circadian influences on performance. Hence, the added benefit to motor-sequence learning accrued from posttraining sleep in younger adults was unavailable to older adults. Siengsukon and Boyd (2008, 2009) examined whether sleep-related changes in performance would be evident in individuals poststroke. Healthy adult participants, approximately 55–75 years of age, served as controls. They, too, found that performance was unchanged following sleep in healthy older adults. In fact, in a second study comparing off-line changes on explicit and implicit motor learning tasks, healthy older adults showed similar changes in performance over intervals with sleep and intervals with wake regardless of awareness (Siengsukon and Boyd, 2009). However, performance of the poststroke group did show significant performance changes following sleep. The authors suggest that this preserved sleep benefit, which was absent in age-matched healthy controls, is related to increased nREM2 sleep and spindle density which has been observed poststroke (Vock et al., 2002). Two recent studies have examined sleepdependent consolidation of nonmotor learning in older adults. Aly and Moscovitch (2010) compared episodic recall following an interval with
84 Young adults 300
sleep
sleep
Reaction time (ms)
250 200 150 100 50 0 p.m.
a.m.
p.m.
a.m.
p.m.
a.m.
Older adults 550
sleep
sleep
Reaction time (ms)
500 450 400 350 300 250 p.m.
a.m.
p.m.
a.m.
p.m.
a.m.
Fig. 2. Reaction time on an explicit sequence learning task following 12-h intervals either containing overnight sleep (e.g., 8 p.m. to 8 a.m.) or spent awake (e.g., 8 a.m. to 8 p.m.). Young adults (top panel) consistently show the largest performance improvements (decrease in reaction time) following sleep bouts regardless of whether the first of three sessions occurs in the evening (left side) or in the morning (right). While older adults show an improvement following the first session, this change is not sleep-dependent.
overnight sleep relative to recall following a daytime interval spent awake. While older adults (69–80 years) exhibited greater forgetting over both intervals relative to the young adult group (19–29 years), the protection of the memory provided by sleep did not differ across the age groups, indicating sparing of the sleep-dependent memory consolidation process. Likewise, we found spared sleep-dependent memory consolidation in older adults on a word-pair learning task. We compared changes in sleep-dependent
memory processing of a motor-sequence learning task with off-line changes in word-pair learning (Wilson et al., submitted). Again, older adults (55–70 years) showed no sleep-dependent performance changes on the motor learning task. However, these same individuals exhibited similar sleep-dependent protection of performance on the word-pair task. This study (Wilson et al., submitted) also included middle-aged adults (35–50 years), and we report their performance changes over sleep
85
and wake were similar to young adults on both tasks. However, Backhaus et al. (2007) suggest that sleep-dependent consolidation of declarative memories is reduced in this age group (48–55 years) relative to young adults. Young adults recalled more word pairs in the associated word-pair learning task following early-night (SWS-rich) sleep than late-night (REM-rich) sleep. The middle-age group showed no significant change across either interval. However, when they considered physiological measures in these subjects, they found that middle-aged individuals who garnered the same amount of SWS as the young adults did show greater consolidation following sleep than middle-aged individuals remaining awake.
Conclusions and future directions Healthy aging is prominently marked by declines in both sleep and cognitive function. Older adults with greater sleep quantity and quality generally perform better on cognitive tasks. Moreover, the benefit of intervening sleep on recent memories is diminished with age, at least for motor tasks. An understanding of the relationship between these two processes, however, is just beginning to emerge. Although much of this work has been correlative, current work is being directed at the causal relationships which may exist. For instance, given the importance of thalamocortical and cortico-cortical rhythms to sleep-dependent memory consolidation (Marshall et al., 2006; Marshall and Born, 2007; Steriade, 2006), changes in these rhythms (including delta waves, spindles, and slow oscillations; Cajochen et al., 2006) may underlie reduced sleep-related performance changes in older adults. Alternatively, such changes may be accounted for by age-related changes in cortico-striatal circuits (Buckner, 2004; Spreng et al., 2010). Frontal areas, important for strategic retrieval (Buckner, 2004;
Hedden and Gabrieli, 2004), are sensitive to increased homeostatic sleep pressure which, as noted, is diminished with aging (Cajochen et al., 2006). Thus, it is possible that diminished restorative effects of sleep on brain regions critical for retrieval may reduce sleep's benefit on memory in older adults. We note, however, that these explanations fail to account for a possible task-dependence of the age-related decline in sleep-dependent memory processing as recently suggested (Aly and Moscovitch, 2010; Wilson et al., submitted). As stated in the Introduction, there are many instances in which psychoeducational interventions emphasizing sleep hygiene can improve sleep and, potentially, waking performance, cognition, and emotional stability in healthy older adults. For more entrenched sleep issues such as psychophysiological insomnia, sleep hygiene can be augmented by additional behavioral treatments such as cognitive behavioral therapy. It is important, however, to reiterate that the prevalence of other primary sleep disorders such as obstructive sleep apnea increases with age as does the prevalence of neuropsychiatric and major medical illnesses and use of medications which can disturb sleep (Neikrug and Ancoli-Israel, 2010). In the latter cases, it may be even more important to determine possible effects of sleep disturbance in order to assess the efficacy of medical interventions targeted at primary symptoms of illnesses receiving treatment. In treating symptoms of neurodegenerative illnesses, it is especially crucial to assess sleep quality as the effects of insufficient sleep (e.g., drowsiness) may overlap with those of the illness itself. For example, in movement disorders, inattention due to sleep loss may exacerbate incoordination symptoms that can lead to falls. Importantly, if a direct association between sleep quality and cognitive decline is revealed, sleep may be considered as a behavioral or pharmacological target for improving cognitive performance late in life.
86
References Abe, T., Ogawa, K., Nittono, H., & Hori, T. (2008). Gamma band EEG activity is enhanced after the occurrence of rapid eye movement during human REM sleep. Sleep and Biological Rhythms, 6, 26–33. Adam, M., Retey, J. V., Khatami, R., & Landolt, H. P. (2006). Age-related changes in the time course of vigilant attention during 40 hours without sleep in men. Sleep, 29 (1), 55–57. Aly, M., & Moscovitch, M. (2010). The effects of sleep on episodic memory in older and younger adults. Memory, 18(3), 327–334. Ancoli-Israel, S., Ayalon, L., & Salzman, C. (2008). Sleep in the elderly: Normal variations and common sleep disorders. Harvard Review of Psychiatry, 16(5), 279–286. Anderson, C., & Horne, J. A. (2003). Prefrontal cortex: Links between low frequency delta EEG in sleep and neuropsychological performance in healthy, older people. Psychophysiology, 40(3), 349–357. Asplund, R., & Aberg, H. (1996). Nocturnal micturition, sleep and well-being in women of ages 40–64 years. Maturitas, 24 (1–2), 73–81. Backhaus, J., Born, J., Hoeckesfeld, R., Fokuhl, S., Hohagen, F., & Junghanns, K. (2007). Midlife decline in declarative memory consolidation is correlated with a decline in slow wave sleep. Learning and Memory, 14(5), 336–341. Benloucif, S., Green, K., L'Hermite-Baleriaux, M., Weintraub, S., Wolfe, L. F., & Zee, P. C. (2006). Responsiveness of the aging circadian clock to light. Neurobiology of Aging, 27(12), 1870–1879. Blackwell, T., Yaffe, K., Ancoli-Israel, S., Schneider, J. L., Cauley, J. A., Hillier, T. A., et al. (2006). Poor sleep is associated with impaired cognitive function in older women: The study of osteoporotic fractures. The Journals of Gerontology. Series A: Biological Sciences and Medical Sciences, 61 (4), 405–410. Bliese, P. D., McGurk, D., Thomas, J. L., Balkin, T. J., & Wesensten, N. (2007). Discontinuous growth modeling of adaptation to sleep setting changes: Individual differences and age. Aviation Space and Environmental Medicine, 78 (5), 485–492. Bliwise, D. L. (1993). Sleep in normal aging and dementia. Sleep, 16(1), 40–81. Bliwise, D. L. (2005). Normal aging. In M. H. Kryger, T. Roth & W. C. Dement (Eds.), Principles and practice of sleep medicine (pp. 24–38). (4th ed.). Philadelphia: Elsevier. Bonnet, M. H. (1989). The effect of sleep fragmentation on sleep and performance in younger and older subjects. Neurobiology of Aging, 10(1), 21–25. Bonnet, M. H., & Arand, D. L. (1989). Sleep loss in aging. Clinics in Geriatric Medicine, 5(2), 405–420.
Bonnet, M. H., & Rosa, R. R. (1987). Sleep and performance in young adults and older normals and insomniacs during acute sleep loss and recovery. Biological Psychology, 25 (2), 153–172. Borbely, A. A. (1982). A two process model of sleep regulation. Human Neurobiology, 1(3), 195–204. Brown, L. N., Goddard, K. M., Lahar, C. J., & Mosley, J. L. (1999). Age-related deficits in cognitive functioning are not mediated by time of day. Experimental Aging Research, 25 (1), 81–93. Buckley, T. M., & Schatzberg, A. F. (2005). Aging and the role of the HPA axis and rhythm in sleep and memory-consolidation. The American Journal of Geriatric Psychiatry, 13 (5), 344–352. Buckner, R. L. (2004). Memory and executive function in aging and AD: Multiple factors that cause decline and reserve factors that compensate. Neuron, 44(1), 195–208. Buysse, D. J., Monk, T. H., Carrier, J., & Begley, A. (2005). Circadian patterns of sleep, sleepiness, and performance in older and younger adults. Sleep, 28(11), 1365–1376. Cajochen, C., Munch, M., Knoblauch, V., Blatter, K., & Wirz-Justice, A. (2006). Age-related changes in the circadian and homeostatic regulation of human sleep. Chronobiology International, 23(1–2), 461–474. Cantero, J. L., Atienza, M., Gomez-Herrero, G., CruzVadell, A., Gil-Neciga, E., Rodriguez-Romero, R., et al. (2009). Functional integrity of thalamocortical circuits differentiates normal aging from mild cognitive impairment. Human Brain Mapping, 30(12), 3944–3957. Carrier, J., Land, S., Buysse, D. J., Kupfer, D. J., & Monk, T. H. (2001). The effects of age and gender on sleep EEG power spectral density in the middle years of life (ages 20–60 years old). Psychophysiology, 38(2), 232–242. Carskadon, M. A., Brown, E. D., & Dement, W. C. (1982). Sleep fragmentation in the elderly: Relationship to daytime sleep tendency. Neurobiology of Aging, 3(4), 321–327. Clemens, Z., Weiss, B., Szucs, A., Eross, L., Rasonyi, G., & Halasz, P. (2009). Phase coupling between rhythmic slow activity and gamma characterizes mesiotemporal rapideye-movement sleep in humans. Neuroscience, 163(1), 388–396. Crenshaw, M. C., & Edinger, J. D. (1999). Slow-wave sleep and waking cognitive performance among older adults with and without insomnia complaints. Physiology and Behavior, 66(3), 485–492. Cricco, M., Simonsick, E. M., & Foley, D. J. (2001). The impact of insomnia on cognitive functioning in older adults. Journal of the American Geriatrics Society, 49(9), 1185–1189. Crowley, K., Trinder, J., Kim, Y., Carrington, M., & Colrain, I. M. (2002). The effects of normal aging on sleep spindle and K-complex production. Clinical Neurophysiology, 113(10), 1615–1622.
87 Crunelli, V., & Hughes, S. W. (2010). The slow (<1 Hz) rhythm of non-REM sleep: A dialogue between three cardinal oscillators. Nature Neuroscience, 13(1), 9–17. Czeisler, C. A., Duffy, J. F., Shanahan, T. L., Brown, E. N., Mitchell, J. F., Rimmer, D. W., et al. (1999). Stability, precision, and near-24-hour period of the human circadian pacemaker. Science, 284(5423), 2177–2181. Darchia, N., Campbell, I. G., & Feinberg, I. (2003). Rapid eye movement density is reduced in the normal elderly. Sleep, 26 (8), 973–977. Davis, S. W., Dennis, N. A., Daselaar, S. M., Fleck, M. S., & Cabeza, R. (2008). Que PASA? The posterior–anterior shift in aging. Cerebral Cortex, 18(5), 1201–1209. Diekelmann, S., & Born, J. (2010). The memory function of sleep. Nature Reviews. Neuroscience, 11(2), 114–126. Diekelmann, S., Wilhelm, I., & Born, J. (2009). The whats and whens of sleep-dependent memory consolidation. Sleep Medicine Reviews, 13(5), 309–321. Dijk, D. J., & Duffy, J. F. (1999). Circadian regulation of human sleep and age-related changes in its timing, consolidation and EEG characteristics. Annals of medicine, 31(2), 130–140. Dijk, D. J., Duffy, J. F., & Czeisler, C. A. (2000). Contribution of circadian physiology and sleep homeostasis to age-related changes in human sleep. Chronobiology International, 17(3), 285–311. Dijk, D. J., Duffy, J. F., & Czeisler, C. A. (2001). Age-related increase in awakenings: Impaired consolidation of nonREM sleep at all circadian phases. Sleep, 24(5), 565–577. Dijk, D. J., Duffy, J. F., Riel, E., Shanahan, T. L., & Czeisler, C. A. (1999). Ageing and the circadian and homeostatic regulation of human sleep during forced desynchrony of rest, melatonin and temperature rhythms. The Journal of Physiology, 516(Pt 2), 611–627. Dijk, D. J., Groeger, J. A., Stanley, N., & Deacon, S. (2010). Age-related reduction in daytime sleep propensity and nocturnal slow wave sleep. Sleep, 33(2), 211–223. Dijk, D. J., & von Schantz, M. (2005). Timing and consolidation of human sleep, wakefulness, and performance by a symphony of oscillators. Journal of Biological Rhythms, 20 (4), 279–290. Drag, L. L., & Bieliauskas, L. A. (2010). Contemporary review 2009: Cognitive aging. Journal of Geriatric Psychiatry and Neurology, 23(2), 75–93. Drummond, S. P., Meloy, M. J., Yanagi, M. A., Orff, H. J., & Brown, G. G. (2005). Compensatory recruitment after sleep deprivation and the relationship with performance. Psychiatry Research, 140(3), 211–223. Duffy, J. F., Dijk, D. J., Klerman, E. B., & Czeisler, C. A. (1998). Later endogenous circadian temperature nadir relative to an earlier wake time in older people. The American Journal of Physiology, 275(5 Pt 2), R1478–R1487.
Duffy, J. F., Willson, H. J., Wang, W., & Czeisler, C. A. (2009). Healthy older adults better tolerate sleep deprivation than young adults. Journal of the American Geriatrics Society, 57(7), 1245–1251. Duffy, J. F., Zeitzer, J. M., Rimmer, D. W., Klerman, E. B., Dijk, D. J., & Czeisler, C. A. (2002). Peak of circadian melatonin rhythm occurs later within the sleep of older subjects. American Journal of Physiology. Endocrinology and Metabolism, 282(2), E297–E303. Edinger, J. D., Glenn, D. M., Bastian, L. A., & Marsh, G. R. (2000). Slow-wave sleep and waking cognitive performance II: Findings among middle-aged adults with and without insomnia complaints. Physiology and Behavior, 70(1–2), 127–134. Ellenbogen, J. M., Hulbert, J. C., Jiang, Y., & Stickgold, R. (2009). The sleeping brain's influence on verbal memory: Boosting resistance to interference. PLoS ONE, 4(1), e4117. Ellenbogen, J. M., Hulbert, J. C., Stickgold, R., Dinges, D. F., & Thompson-Schill, S. L. (2006). Interfering with theories of sleep and memory: Sleep, declarative memory, and associative interference. Current Biology, 16(13), 1290–1294. Faubel, R., Lopez-Garcia, E., Guallar-Castillon, P., Graciani, A., Banegas, J. R., & Rodriguez-Artalejo, F. (2009). Usual sleep duration and cognitive function in older adults in Spain. Journal of Sleep Research, 18(4), 427–435. Ferrara, M., Iaria, G., Tempesta, D., Curcio, G., Moroni, F., Marzano, C., et al. (2008). Sleep to find your way: The role of sleep in the consolidation of memory for navigation in humans. Hippocampus, 18(8), 844–851. Fischer, S., Hallschmid, M., Elsner, A. L., & Born, J. (2002). Sleep forms memory for finger skills. Proceedings of the National Academy of Sciences of the United States of America, 99(18), 11987–11991. Floyd, J. A., Janisse, J. J., Jenuwine, E. S., & Ager, J. W. (2007). Changes in REM-sleep percentage over the adult lifespan. Sleep, 30(7), 829–836. Fukuda, N., Honma, H., Kohsaka, M., Kobayashi, R., Sakakibara, S., Kohsaka, S., et al. (1999). Gender difference of slow wave sleep in middle aged and elderly subjects. Psychiatry and Clinical Neurosciences, 53(2), 151–153. Harrison, Y., Horne, J. A., & Rothwell, A. (2000). Prefrontal neuropsychological effects of sleep deprivation in young adults—A model for healthy aging? Sleep, 23(8), 1067–1073. Hasher, L., Chung, C., May, C. P., & Foong, N. (2002). Age, time of testing, and proactive interference. Canadian Journal of Experimental Psychology, 56(3), 200–207. Hedden, T., & Gabrieli, J. D. (2004). Insights into the ageing mind: A view from cognitive neuroscience. Nature Reviews. Neuroscience, 5(2), 87–96. Hoch, C. C., Dew, M. A., Reynolds 3rd., C. F., Buysse, D. J., Nowell, P. D., Monk, T. H., et al. (1997). Longitudinal changes in diary- and laboratory-based sleep measures in
88 healthy “old old” and “young old” subjects: A three-year follow-up. Sleep, 20(3), 192–202. Hoch, C. C., Dew, M. A., Reynolds 3rd., C. F., Monk, T. H., Buysse, D. J., Houck, P. R., et al. (1994). A longitudinal study of laboratory- and diary-based sleep measures in healthy “old old” and “young old” volunteers. Sleep, 17(6), 489–496. Hofman, M. A., & Swaab, D. F. (2006). Living by the clock: The circadian pacemaker in older people. Ageing Research Reviews, 5(1), 33–51. Hogan, M. J., Kelly, C. A., Verrier, D., Newell, J., Hasher, L., & Robertson, I. H. (2009). Optimal time-of-day and consolidation of learning in younger and older adults. Experimental Aging Research, 35(1), 107–128. Hornung, O. P., Danker-Hopfe, H., & Heuser, I. (2005). Agerelated changes in sleep and memory: Commonalities and interrelationships. Experimental Gerontology, 40(4), 279–285. Huang, Y. L., Liu, R. Y., Wang, Q. S., Van Someren, E. J., Xu, H., & Zhou, J. N. (2002). Age-associated difference in circadian sleep-wake and rest-activity rhythms. Physiology and Behavior, 76(4–5), 597–603. Jensen, O., Kaiser, J., & Lachaux, J. P. (2007). Human gammafrequency oscillations associated with attention and memory. Trends in Neurosciences, 30(7), 317–324. Jurado, J. L., Luna-Villegas, G., & Buela-Casal, G. (1989). Normal human subjects with slow reaction times and larger time estimations after waking have diminished delta sleep. Electroencephalography and Clinical Neurophysiology, 73 (2), 124–128. Kanbayashi, T., Yano, T., Ishiguro, H., Kawanishi, K., Chiba, S., Aizawa, R., et al. (2002). Hypocretin-1 (orexinA) levels in human lumbar CSF in different age groups: Infants to elderly persons. Sleep, 25(3), 337–339. Klerman, E. B., Davis, J. B., Duffy, J. F., Dijk, D. J., & Kronauer, R. E. (2004). Older people awaken more frequently but fall back asleep at the same rate as younger people. Sleep, 27(4), 793–798. Knoblauch, V., Munch, M., Blatter, K., Martens, W. L., Schroder, C., Schnitzler, C., et al. (2005). Age-related changes in the circadian modulation of sleep-spindle frequency during nap sleep. Sleep, 28(9), 1093–1101. Landolt, H. P., Dijk, D. J., Achermann, P., & Borbely, A. A. (1996). Effect of age on the sleep EEG: Slow-wave activity and spindle frequency activity in young and middle-aged men. Brain Research, 738(2), 205–212. Lombardo, P., Formicola, G., Gori, S., Gneri, C., Massetani, R., Murri, L., et al. (1998). Slow wave sleep (SWS) distribution across night sleep episode in the elderly. Aging (Milano), 10(6), 445–448. Marshall, L., & Born, J. (2007). The contribution of sleep to hippocampus-dependent memory consolidation. Trends in Cognitive Sciences, 11(10), 442–450.
Marshall, L., Helgadottir, H., Molle, M., & Born, J. (2006). Boosting slow oscillations during sleep potentiates memory. Nature, 444(7119), 610–613. May, C. P., & Hasher, L. (1998). Synchrony effects in inhibitory control over thought and action. Journal of Experimental Psychology. Human Perception and Performance, 24(2), 363–379. May, C. P., Hasher, L., & Foong, N. (2005). Implicit memory, age, and time of day: Paradoxical priming effects. Psychological Science, 16(2), 96–100. Menza, M., Dobkin, R. D., Marin, H., & Bienfait, K. (2010). Sleep disturbances in Parkinson's disease. Movement Disorders, 25(Suppl. 1), S117–S122. Monk, T. H. (2005). Aging human circadian rhythms: Conventional wisdom may not always be right. Journal of Biological Rhythms, 20(4), 366–374. Munch, M., Silva, E. J., Ronda, J. M., Czeisler, C. A., & Duffy, J. F. (2010). EEG sleep spectra in older adults across all circadian phases during NREM sleep. Sleep, 33(3), 389–401. Nebes, R. D., Buysse, D. J., Halligan, E. M., Houck, P. R., & Monk, T. H. (2009). Self-reported sleep quality predicts poor cognitive performance in healthy older adults. The Journals of Gerontology. Series B: Psychological Sciences and Social Sciences, 64(2), 180–187. Neikrug, A. B., & Ancoli-Israel, S. (2010). Sleep disorders in the older adult—A mini-review. Gerontology, 56(2), 181–189. Nissen, M. J., & Bullemer, P. (1987). Attentional requirements of learning: Evidence from performance measures. Cognitive Psychology, 19(1), 1–32. O'Donnell, D., Silva, E. J., Munch, M., Ronda, J. M., Wang, W., & Duffy, J. F. (2009). Comparison of subjective and objective assessments of sleep in healthy older subjects without sleep complaints. Journal of Sleep Research, 18(2), 254–263. Ohayon, M. M., Carskadon, M. A., Guilleminault, C., & Vitiello, M. V. (2004). Meta-analysis of quantitative sleep parameters from childhood to old age in healthy individuals: Developing normative sleep values across the human lifespan. Sleep, 27(7), 1255–1273. Pace-Schott, E. F., & Hobson, J. A. (2002). The neurobiology of sleep: Genetics, cellular physiology and subcortical networks. Nature Reviews. Neuroscience, 3(8), 591–605. Pandi-Perumal, S. R., Zisapel, N., Srinivasan, V., & Cardinali, D. P. (2005). Melatonin and sleep in aging population. Experimental Gerontology, 40(12), 911–925. Philip, P., Taillard, J., Sagaspe, P., Valtat, C., SanchezOrtuno, M., Moore, N., et al. (2004). Age, performance and sleep deprivation. Journal of Sleep Research, 13(2), 105–110. Plihal, W., & Born, J. (1997). Effects of early and late nocturnal sleep on declarative and procedural memory. Journal of Cognitive Neuroscience, 9(4), 534–547. Plihal, W., & Born, J. (1999). Memory consolidation in human sleep depends on inhibition of glucocorticoid release. Neuroreport, 10(13), 2741–2747.
89 Pritchett, S., Zilberg, E., Xu, M., Burton, D., Brown, I., & Myles, P. (2008). Power analysis of gamma frequencies (30–47 Hz), adjusting for muscle activity (80–97 Hz), in anesthesia: A comparison between young adults, middleaged and the elderly. Conference proceedings: Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2008, 825–830. Rasch, B., Buchel, C., Gais, S., & Born, J. (2007). Odor cues during slow-wave sleep prompt declarative memory consolidation. Science, 315(5817), 1426–1429. Riegel, B., & Weaver, T. E. (2009). Poor sleep and impaired self-care: Towards a comprehensive model linking sleep, cognition, and heart failure outcomes. European Journal of Cardiovascular Nursing, 8(5), 337–344. Robertson, E. M., Pascual-Leone, A., & Press, D. Z. (2004). Awareness modifies the skill-learning benefits of sleep. Current Biology, 14(3), 208–212. Salthouse, T. A. (2009). When does age-related cognitive decline begin? Neurobiology of Aging, 30(4), 507–514. Salzarulo, P., Fagioli, I., Lombardo, P., Gori, S., Gneri, C., Chiaramonti, R., et al. (1999). Sleep stages preceding spontaneous awakenings in the elderly. Sleep Research Online, 2(3), 73–77. Saper, C. B., Chou, T. C., & Scammell, T. E. (2001). The sleep switch: Hypothalamic control of sleep and wakefulness. Trends in Neurosciences, 24(12), 726–731. Saper, C. B., Scammell, T. E., & Lu, J. (2005). Hypothalamic regulation of sleep and circadian rhythms. Nature, 437 (7063), 1257–1263. Schmidt, C., Collette, F., Cajochen, C., & Peigneux, P. (2007). A time to think: Circadian rhythms in human cognition. Cognitive Neuropsychology, 24(7), 755–789. Siengsukon, C. F., & Boyd, L. A. (2008). Sleep enhances implicit motor skill learning in individuals poststroke. Topics in Stroke Rehabilitation, 15(1), 1–12. Siengsukon, C. F., & Boyd, L. A. (2009). Sleep to learn after stroke: Implicit and explicit off-line motor learning. Neuroscience Letters, 451(1), 1–5. Silva, E. J., & Duffy, J. F. (2008). Sleep inertia varies with circadian phase and sleep stage in older adults. Behavioral Neuroscience, 122(4), 928–935. Silva, E. J., Wang, W., Ronda, J. M., Wyatt, J. K., & Duffy, J. F. (2010). Circadian and wake-dependent influences on subjective sleepiness, cognitive throughput, and reaction time performance in older and young adults. Sleep, 33(4), 481–490. Spencer, R. M., Gouw, A. M., & Ivry, R. B. (2007). Age-related decline of sleep-dependent consolidation. Learning and Memory, 14(7), 480–484. Spencer, R. M., Sunm, M., & Ivry, R. B. (2006). Sleep-dependent consolidation of contextual learning. Current Biology, 16(10), 1001–1005. Spreng, R. N., Wojtowicz, M., & Grady, C. L. (2010). Reliable differences in brain activity between young and old adults:
A quantitative meta-analysis across multiple cognitive domains. Neuroscience and Biobehavioral Reviews, 34(8), 1178–1194. Stenuit, P., & Kerkhofs, M. (2005). Age modulates the effects of sleep restriction in women. Sleep, 28(10), 1283–1288. Steriade, M. (2006). Grouping of brain rhythms in corticothalamic systems. Neuroscience, 137(4), 1087–1106. Stickgold, R. (2005). Sleep-dependent memory consolidation. Nature, 437(7063), 1272–1278. Tucker, M. A., Hirota, Y., Wamsley, E. J., Lau, H., Chaklader, A., & Fishbein, W. (2006). A daytime nap containing solely non-REM sleep enhances declarative but not procedural memory. Neurobiology of Learning and Memory, 86(2), 241–247. Van Someren, E. J. (2000). Circadian and sleep disturbances in the elderly. Experimental Gerontology, 35(9–10), 1229–1237. Van Someren, E. J., Riemersma, R. F., & Swaab, D. F. (2002). Functional plasticity of the circadian timing system in old age: Light exposure. Progress in Brain Research, 138, 205–231. Vitiello, M. V. (2006). Sleep in normal aging. Sleep Medicine Clinics, 1, 171–176. Vock, J., Achermann, P., Bischof, M., Milanova, M., Muller, C., Nirkko, A., et al. (2002). Evolution of sleep and sleep EEG after hemispheric stroke. Journal of Sleep Research, 11(4), 331–338. Walker, M. P. (2009). The role of sleep in cognition and emotion. Annals of the New York Academy of Sciences, 1156, 168–197. Walker, M. P., Brakefield, T., Morgan, A., Hobson, J. A., & Stickgold, R. (2002). Practice with sleep makes perfect: Sleep-dependent motor skill learning. Neuron, 35(1), 205–211. Wei, H. G., Riel, E., Czeisler, C. A., & Dijk, D. J. (1999). Attenuated amplitude of circadian and sleep-dependent modulation of electroencephalographic sleep spindle characteristics in elderly human subjects. Neuroscience Letters, 260(1), 29–32. Wilson, M. A. (2002). Hippocampal memory formation, plasticity, and the role of sleep. Neurobiology of Learning and Memory, 78(3), 565–569. Wilson, J. K., Baran, B., Pace-Schott, E., Ivry, R. B., and Spencer, R. M. C. Offline processing in aging: Sleep modulates declarative but not procedural learning in healthy older adults. (submitted). Wolkove, N., Elkholy, O., Baltzan, M., & Palayew, M. (2007). Sleep and aging: 1. Sleep disorders commonly found in older people. Canadian Medical Association Journal, 176(9), 1299–1304. Wu, Y. H., & Swaab, D. F. (2007). Disturbance and strategies for reactivation of the circadian rhythm system in aging and Alzheimer's disease. Sleep Medicine, 8(6), 623–636.
A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 191 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.
CHAPTER 6
Motor adaptation and proprioceptive recalibration Erin K. Cressman{,* and Denise Y. P. Henriques{,} {
School of Human Kinetics, University of Ottawa, Ottawa, Ontario, Canada Center for Vision Research, York University, Toronto, Ontario, Canada School of Kinesiology and Health Science, York University, Toronto, Ontario, Canada {
}
Abstract: Goal-directed reaches are rapidly adapted after reaching with misaligned visual feedback of the hand. It has been suggested that reaching with misaligned visual feedback of the hand also results in proprioceptive recalibration (i.e., realigning proprioceptive estimates of hand position to match visual estimates). In this chapter, we review a series of experiments conducted in our lab which examine this proposal. We assessed proprioceptive recalibration by comparing subjects’ estimates of the position at which they felt their hand was aligned with a reference marker (visual or proprioceptive) before and after aiming with a misaligned cursor that was typically rotated 30 clockwise (CW) with respect to the hand. In general, results indicated that subjects recalibrated proprioception such that their estimates of felt hand position were shifted in the same direction that they adapted their reaches. Moreover, proprioception was recalibrated to a similar extent of motor adaptation ( 30%), regardless of how the hand was positioned during the estimate trials (active or passive placement), the location or modality of the reference marker (visual or proprioceptive), the hand used during reach training (right or left), how the distortion was introduced (gradual or abrupt), and age (young or older subjects) and the magnitude of the visuomotor distortion introduced (30 or 50 or 70 ). These results suggest that in addition to recalibrating the sensorimotor transformations underlying reaching movements, visuomotor adaptation results in partial proprioceptive recalibration. Keywords: visuomotor adaptation; sensory recalibration; proprioception. 1988). If these sensory cues conflict and one is reaching to a visual target, one tends to rely more on the visual estimate of the hand than on the actual or felt position. For example, it has been shown that when reaching to a target with misaligned visual feedback of the hand (i.e., reaching in a virtual reality environment or while wearing prism goggles), one adjusts the movement in order for the visual
Introduction When reaching to visual targets, one uses vision and proprioception to plan movements (e.g., Jeannerod,
*Corresponding author. Tel.: þ1-613-562-5800x4264 E-mail:
[email protected] DOI: 10.1016/B978-0-444-53752-2.00011-4
91
92
representation of the hand to achieve the desired endpoint (Baraduc and Wolpert, 2002; Ghahramani et al., 1996; Klassen et al., 2005; Krakauer et al., 1999, 2000; Magescas and Prablanc, 2006; Redding and Wallace, 1996; Simani et al., 2007, Vetter et al., 1999; Wang and Sainburg, 2005). This process is referred to as visuomotor adaptation and results in the formation of a new visuomotor mapping to guide one's movements. In the research presented below, we examine how the brain deals with conflicting sensory signals during visuomotor adaptation. Based on changes observed in the motor system, it has been proposed that changes in reaches arise after reaching with misaligned visual feedback of the hand due to a difference between the desired (predicted) and actual sensory feedback arising from a given motor command. For example, when first reaching with altered visual feedback of the hand, one expects to see the visual representation of the hand head to the target. However, because visual feedback of the hand is misaligned from the actual hand location, the hand is seen to head off on an angle. This gives rise to an error signal, and it is thought that this signal (i.e., the sensory discrepancy between the predicted and actual sensory feedback) is used to amend the motor command (Miall and Wolpert, 1996; Wolpert, 1997; Wolpert et al., 1995). While models accounting for motor learning include a role for sensory feedback, it is unclear what happens to one's sense of felt hand position during motor learning. How does the brain resolve the spatial conflict between seen and felt hand location? To look at this issue, we (1) asked if proprioceptive estimates of felt hand position are remapped (i.e., recalibrated) to match the visual representation of one's hand and (2) examined whether changes in proprioception contribute to changes in one's reaches. Previous research examining sensory recalibration has typically asked subjects to reach to visual and proprioceptive targets with their adapted hand following visuomotor adaptation (Harris, 1963; Hay and Pick, 1966; Simani et al., 2007; van Beers et al., 2002). While subjects’ reaches are altered
following visuomotor adaptation, it is unclear if these changes reflect sensory recalibration (specifically proprioceptive recalibration) per se. Subjects were allowed to freely move their adapted arm. Thus errors in reaches could have arisen because subjects were using the adapted sensorimotor mapping to program their movements. In the research discussed below, we examined sensory recalibration by determining changes in felt hand position following visuomotor adaptation in perceptual, nonreaching tasks. Specifically, proprioceptive estimates of hand position were measured in a task, in which subjects did not reach, aim, align—or otherwise freely move—their adapted hand to a target (i.e., subjects did not complete a goal-directed movement). Thus, these proprioceptive estimates provide insight into sensory recalibration processes, independent of possible motor changes. To start, we will provide a general description of our proprioceptive estimation task and then outline the visuomotor tasks subjects completed before we assessed felt hand position. Overall, our results indicate that proprioception is recalibrated following visuomotor adaptation. As well, our results begin to reveal the relationship between sensory plasticity and adaptation of motor commands.
General methodology We assessed changes in subjects’ felt hand position by determining the position at which they perceived their hand to be aligned with a reference marker. On proprioceptive estimation trials, subjects grasped the handle of a robot manipulandum (Interactive Motion Technologies; Fig. 1a) and pushed it out from the home position along a robot-generated constrained linear path (i.e., a slot) to a location somewhere along the dotted line shown in Fig. 1b. If subjects attempted to move outside of the established path, the robot generated a resistance force (proportional to the depth of penetration with a stiffness of 2 N/mm and a viscous damping of 5 N/ (mm/s)) perpendicular to the linear path (Henriques and Soechting, 2003; Jones et al., 2010).
93 (a)
(b)
(c)
Fig. 1. Experimental setup and design. (a) Side view of the experimental setup. (b and c) Top view of experimental surface. (b) In general, reach targets (open black circles) and reference markers (filled white circles) used in the proprioceptive estimation task were located along a circular arc, 10 cm from the home position (black filled circle). Note that the black dotted line is provided as a reference to indicate the locations of the targets and reference markers and illustrate potential positions that the hand could have been moved to during the proprioceptive estimation trials. (c) Possible visuomotor distortion introduced in the reach training task when subjects reached with a misaligned cursor. In this example, the black cursor (representing the hand) was rotated 30 clockwise with respect to the actual hand location.
Once the hand reached its final position, a reference marker appeared (e.g., white circles in Fig. 1b, from Cressman et al., 2010) and subjects made a two-alternative forced-choice judgment about the position of their hand (left or right) relative to the reference marker. Because subjects actively moved their hand into position by pushing the robot handle out along a constrained path, we refer to these estimation trials as active placement estimates. As discussed below, one of the first questions we asked with respect to proprioceptive recalibration was whether shifts in
subjects’ felt hand position were dependent on how their hand was moved into position. Specifically, we asked if changes in subjects’ felt hand position differed depending on whether subjects actively moved the robot handle into position (active placement estimates) or the robot passively moved subjects’ hands into position along the same constrained path (passive placement estimates). In general, results reported are from active placement estimate trials, unless otherwise noted. To determine the locations at which subjects felt their hand was aligned with a reference marker, we adjusted the position of the hand with respect to each reference marker over proprioceptive estimation trials (50 for each reference marker) using an adaptive staircase algorithm that was dependent on a subject's pattern of responses (Cressman and Henriques, 2009; Kesten, 1958; Treutwein, 1995). We then fit a logistic function to each subject's left–right responses for each reference marker for each testing session that they completed (Cressman and Henriques, 2009, 2010a; Cressman et al., 2010; Jones et al., 2010). Based on each logistic function, we calculated the bias (accuracy: the point of responding left (or right) 50% of the time) and uncertainty (precision: the difference between the values at which the response probability of responding left (or right) was 25% and 75%). In general, we measured proprioceptive estimates of hand position after subjects trained to reach to targets with a free-moving robot manipulandum (i.e., the robot's motion was not constrained to a specific path as in the proprioceptive estimation trials). In the first training session, subjects freely reached to targets with a cursor that was aligned with the hand. In the second training session, subjects reached to targets with a misaligned cursor, for example, a cursor that was either gradually or abruptly rotated CW with respect to the hand. To introduce the distortion gradually, the cursor was aligned with the hand on the first reaching trial. Over
94
subsequent trials, the cursor was rotated CW with respect to the hand in increments of 0.75 until the goal distortion was achieved (typically 30 ). In testing sessions in which the distortion was introduced abruptly, the cursor was rotated 30 CW with respect to the hand on the first reaching trial. Details regarding the order of the reach and proprioceptive estimation trials for each of the two testing sessions are provided in Fig. 2. To assess the extent of motor adaptation (or motor recalibration), subjects reached to targets without any visual feedback of their hand (i.e., without a cursor) so that we could measure reach aftereffects (i.e., changes in reach errors).
Proprioception is recalibrated following visuomotor adaptation In all of our studies to date (Cressman and Henriques, 2009, 2010a; Cressman et al., 2010), we have consistently found proprioceptive recalibration at all reference marker locations following visuomotor adaptation. An example of such findings from a recent study (Cressman et al., 2010) is shown in Fig. 3. Here, proprioceptive estimates were shifted 6 more left after reach training with a cursor that was gradually rotated 30 CW with respect to the hand (black squares in Fig. 3a) compared to estimates obtained after reaching with
an aligned cursor (black triangles in Fig. 3a). Further, from Fig. 3b, we see that these shifts in proprioceptive estimates were in the same direction that subjects adapted their reaches. We found similar changes in proprioceptive estimates of hand position (i.e., shifts of 6 ) when the robot passively moved the hand into position during the estimation trials (Fig. 4, passive placement estimates (filled black circles) vs. active placement estimates (open white circles); Cressman and Henriques, 2009). As well, changes in proprioceptive estimates generalized to novel locations at which subjects did not have any practice reaching to (i.e., reference markers that were deviated 15 from the reach targets during training with the aligned or misaligned cursor (data not shown; Cressman and Henriques, 2009)). However, at all reference markers, proprioception was only recalibrated a fraction (about one-third) of the extent that subjects adapted their reaches (Cressman and Henriques, 2009; Cressman et al., 2010), regardless of whether subjects actively moved their hand along a constrained path or the robot passively moved the hand into position during the proprioceptive estimate trials. In addition to finding proprioceptive recalibration in the right hand, we have also found similar shifts in felt hand position of the left hand, after subjects trained with the left hand (Salomonczyk et al., 2010a). In fact, the only difference between
Tasks completed in both testing sessions Proprioceptive estimate + Reach task
Reach task Reach training with cursor
No cursor reaches
Reach training with cursor
Proprioceptive estimates
No cursor reaches
Repeat
Fig. 2. Schematic showing the order in which different tasks were typically completed within a testing session. In the first testing session, subjects reached with an aligned cursor on all reach training trials. On the second day of testing, the cursor was misaligned from the actual hand location (e.g., rotated clockwise with respect to the actual hand location, Box 1). After the visually guided reach training trials, subjects next reached to the reach targets without a cursor to assess motor adaptation (No cursor reaches, Box 2). This was followed by interleaving proprioceptive estimation trials (Box 3) and visually guided reaches (Box 4). Finally, subjects completed an additional set of no cursor reaches to end the testing session (Box 5).
95 (b)
Mean change (°)
30
CCW/left of target
100
75 20 50 10 25
Mean change (%)
(a)
Right
Left 0
5 Aligned
5 cm Misaligned
0
0 Prop recal
Reach adapt
Target
Fig. 3. (a) Mean 2D biases on the proprioceptive estimation task after subjects reached with an aligned (triangles) or misaligned (squares) cursor during the reach training task. In (b), we show mean changes in proprioceptive estimates (black bar) and reach errors (i.e., motor adaptation, white bar) after subjects reached with a misaligned compared to aligned cursor. Results are shown in degrees and as a percentage of the distortion introduced during reach training trials. Error bars reflect standard error of the mean (from Cressman et al., 2010).
the right and left hand proprioceptive estimates was a hand-dependent bias in estimated position, such that right-hand biases were more left of a reference marker than left-hand biases. These results are consistent with work from our lab in which we specifically compared proprioceptive acuity between the two hands and found that, in general, subjects judged their left hand to be more left than it actually was and their right hand to be more right than it actually was (Jones et al., 2010). Having established that proprioception was recalibrated in the trained hand (either right or left) with respect to a visual reference marker, we then determined if this change in hand position reflected an overall shift in felt hand position. To do this, we manipulated the modality of the center reference marker such that subjects estimated the position of their hand with respect to body midline (i.e., a proprioceptive reference marker). We found shifts in the position at which subjects perceived the hand was aligned with the proprioceptive reference marker that were similar in magnitude and direction to the shifts observed when a visual reference
marker was displayed at the same location (Cressman and Henriques, 2009). Given that we found that hand-reference marker alignment biases were shifted regardless of reference marker modality (i.e., regardless of whether the reference marker was an extrinsic cue (a visual reference marker) or an intrinsic, egocentric cue (the body midline)), our results suggest that proprioception is recalibrated such that there is a general shift in felt hand position as opposed to a visual–proprioceptive realignment. This shift is only evident in the trained hand (i.e., the hand that performs the reaching trials), as we did not find evidence of proprioceptive recalibration in the untrained (left or right) hand (Salomonczyk et al., 2010a).
Proprioceptive recalibration is not dependent on motor learning conditions Given differences in reaching errors to visual and proprioceptive targets following prism exposure, it has been suggested that the sensory processes
96 120
Proprioceptive recalibration (%)
100 80 60 40 20
−20 −20
−40
0
20
40
60
80
100
120
Reach adaptation (%) Control subjects (Cressman et al., 2010) Passive placement estimates (Cressman and Henriques, 2009) Active placement estimates (Cressman and Henriques, 2009)
Fig. 4. The changes in proprioceptive estimates are plotted as a function of changes in reach aftereffects for each subject after training with a misaligned cursor compared to an aligned cursor. Results are shown for (1) the control (i.e., young) subjects in Cressman et al. (2010) and (2) subjects from Cressman and Henriques (2009), when the robot positioned their hands during the estimation trials (passive placement estimates, filled black circles) and when they actively moved their hand into position (active placement estimates, open white circles). The dashed line is a unit slope and thus indicates equivalent levels of proprioceptive recalibration and motor adaptation.
engaged during visuomotor adaptation are dependent on how the visuomotor distortion is introduced (i.e., gradual vs. abrupt) and age (Bock, 2005; Bock and Girgenrath, 2006; Heuer and Hegele, 2008; McNay and Willingham, 1998; Redding and Wallace, 1996; Redding et al., 2005). In particular, it has been suggested that strategic, cognitive processes are engaged early during the learning process when a visuomotor distortion is introduced abruptly to produce rapid corrections in motor performance (Redding and Wallace, 1996; Redding et al., 2005). Moreover,
it is proposed that older individuals have difficulty engaging these processes, leading to motor learning deficits (Bock, 2005; Bock and Girgenrath, 2006). In contrast, spatial realignment processes (i.e., proprioceptive recalibration) are thought to be responsible for motor adaptation when a visuomotor distortion is introduced gradually, and these processes are proposed to be maintained with advancing age (Heuer and Hegele, 2008; McNay and Willingham, 1998; Redding and Wallace, 1996). While motor adaptation results indicate differences in performance depending on processes engaged during motor learning, our results indicate that this is not the case when assessing estimates of hand position. We found no change in our estimate results: proprioception was consistently recalibrated and recalibrated to a similar extent ( 6 ), regardless of whether the distortion was introduced gradually or abruptly (Salomonczyk et al., 2010a). This similarity was found despite the fact that there was some decay in reaching errors (i.e., motor adaptation) over time following reach training when the distortion was introduced abruptly. To begin to investigate the influence of agerelated changes in motor learning on proprioceptive recalibration, we had a group of elderly subjects (mean age ¼ 66.3, SD ¼ 6.0 years) adapt to a gradually introduced visuomotor distortion and then complete our proprioceptive estimation task (Cressman et al., 2010). Results revealed that elderly subjects recalibrated proprioception to the same extent as younger (i.e., control) subjects ( 6 left of the marker). These results indicate that proprioception is recalibrated to a similar extent throughout the lifespan, when a visuomotor distortion is introduced gradually and similar levels of motor adaptation are achieved. Taken together, these results indicate that sensory recalibration processes are similar regardless of how a visuomotor discrepancy is introduced and one's age. These results, as determined by our proprioceptive estimation task, do not follow the same trend as reaching results
97
(achieved in the prism literature), suggesting that sensory and motor changes may be two independent processes arising from visuomotor learning.
The relationship between proprioceptive recalibration and motor adaptation As shown in Fig. 4, we found that (1) almost all subjects recalibrated proprioception to some extent following visuomotor adaptation; (2) in almost all instances, proprioceptive recalibration was less than motor adaptation; and (3) the magnitude of proprioceptive recalibration was similar regardless of the level of motor adaptation achieved. In accordance with this last observation, our results to date have not revealed a significant correlation between the magnitude of proprioceptive recalibration and the level of motor adaptation attained, as measured in open-loop reaches with no cursor feedback (Cressman and Henriques, 2009; Cressman et al., 2010). These results further suggest that proprioceptive recalibration and motor adaptation may be two independent processes arising after training with misaligned visual feedback of the hand. To investigate the relationship between proprioceptive recalibration and motor adaptation in more detail, we manipulated the extent of motor adaptation achieved by changing the magnitude of the cursor distortion on the reach training trials. Specifically, we examined changes in sense of felt hand position with increasing levels of motor adaptation (Salomonczyk et al., 2010b). To increase levels of motor adaptation, we had subjects complete three testing sessions with a rotated cursor. In the first session, the cursor was gradually rotated 30 CW with respect to the hand during the reach training trials. In the second session, the distortion was increased to 50 and then finally to 70 in the third session. We found that motor adaptation increased in the expected direction over reach training blocks. Subjects reached on average 16 more left of the target after training with a 30 CW rotated cursor
compared to an aligned cursor. These reach errors increased to 27.6 and 33.8 after training with a 50 CW rotated cursor and 70 CW rotated cursor, respectively, compared to training with an aligned cursor. Similar to this increase in motor adaptation, we found that proprioception was also recalibrated to a greater extent following reach training with increasing visuomotor distortions (from 7 after reach training with a 30 rotated cursor to 12 after training with a 50 cursor to 15 after training with a 70 cursor). However, even though both motor and sensory processes were adapted to a greater extent across the experiment, motor adaptation was approximately 50% of the visuomotor distortion across all blocks of trials and proprioceptive recalibration was maintained at a constant percentage of motor adaptation ( 40%, or about 20% of the visuomotor distortion introduced). While the magnitude of the visuomotor distortion was correlated with both changes in movement aftereffects and proprioceptive bias, no significant correlation between these motor and sensory changes was observed overall or within training blocks. Thus it appears that while sensory recalibration and motor adaptation do occur simultaneously and are similarly affected by the size of distortion (and thus the size of the respective error signals), the mechanisms underlying these processes may arise independently following visuomotor adaptation. Taken together, our findings suggest that when one learns a new visuomotor mapping, one also recalibrates proprioception in the trained hand (Salomonczyk et al., 2010a). In accordance with our results, Ostry and colleagues (2010) have recently reported changes in felt hand position after subjects learned to reach in a velocitydependent force field. Similar to our results, they found that proprioception was recalibrated to about 33% of motor adaptation. Thus this consistency in proprioceptive recalibration is found regardless of reaching task (i.e., visuomotor distortion vs. velocity-dependent force field), how the hand is positioned during the estimate trials
98
(active vs. passive placement), the location or modality of the reference marker (visual or proprioceptive), the hand used during reach training (right vs. left), how the distortion is introduced (gradual vs. abrupt), age (young vs. older subjects), and the magnitude of the visuomotor distortion introduced (30 vs. 50 and 70 ). It is important to keep in mind that these changes in the estimate of felt hand position are only a fraction of the motor learning-related changes observed in the unseen trained hand (i.e., motor adaptation). Thus it is unlikely that sensory recalibration is the sole source driving adaptive changes in reaching movements. In fact, given that sensory and motor adaptation are differentially influenced by the processes engaged during reach training (i.e., strategic vs. realignment processes) and that we have found no significant correlation between changes in these systems, sensory and motor changes could be two independent processes. This proposal is supported by studies that have demonstrated motor adaptation in the absence of proprioceptive recalibration, for example, deafferented individuals have been shown to adapt their reaches in response to altered visual feedback of the hand (see Bernier et al., 2006; Ingram et al., 2000; Miall and Cole, 2007). As well, Henriques and colleagues have shown that under some conditions in which motor adaptation is observed, subjects do not recalibrate their sense of hand path geometry (Wong and Henriques, 2009) or path length (Cressman and Henriques, 2010b). To determine the relationship between proprioceptive recalibration and motor adaptation in more detail, we are now investigating how sensory and motor changes generalize to different areas of the workspace. Specifically, we are asking if proprioceptive recalibration extends from a final position (e.g., at the distance of the target) to other positions along the trajectory and how this relates to observed changes in reach trajectories at these positions. Answers to this question may also help explain why we did not find evidence of proprioceptive recalibration
when subjects had to assess a path's shape or length. Moreover, we are looking to see if proprioceptive recalibration generalizes across the workspace after a subject reaches to just one target with a misaligned cursor. Work by Krakauer et al. (2000) suggests that motor adaptation can be limited to certain areas of the workspace after subjects train to just one target. We will look to determine if proprioceptive recalibration follows the same trend. For now, our results suggest that proprioception regarding final endpoint position is recalibrated under task conditions in which motor learning arises. It remains to be determined if motor adaptation and proprioceptive recalibration are two independent adjustments arising from learning to reach with misaligned visual feedback of the hand. References Baraduc, P., & Wolpert, D. M. (2002). Adaptation to a visuomotor shift depends on the starting posture. Journal of Neurophysiology, 88, 973–981. Bernier, P. M., Chua, R., Bard, C., & Franks, I. M. (2006). Updating of an internal model without proprioception: A deafferentation study. Neuroreport, 17, 1421–1425. Bock, O. (2005). Components of sensorimotor adaptation in young and elderly subjects. Experimental Brain Research, 160, 259–263. Bock, O., & Girgenrath, M. (2006). Relationship between sensorimotor adaptation and cognitive functions in younger and older subjects. Experimental Brain Research, 169, 400–406. Cressman, E. K., & Henriques, D. Y. P. (2009). Sensory recalibration of hand position following visuomotor adaptation. Journal of Neurophysiology, 102, 3505–3518. Cressman, E. K., & Henriques, D. Y. P. (2010a). Reach adaptation and proprioceptive recalibration following exposure to misaligned sensory input. Journal of Neurophysiology, 103, 1888–1895. Cressman, E. K., & Henriques, D. Y. P. (2010b). Proprioceptive sensitivity and recalibration of hand movement amplitude. In: Paper presented at the Enhancing performance for action and perception symposium, Montreal, QC. Cressman, E. K., Salmononczyk, D., & Henriques, D. Y. P. (2010). Visuomotor adaptation and proprioceptive recalibration in older adults. Experimental Brain Research, 205, 533–544.
99 Ghahramani, Z., Wolpert, D. M., & Jordan, M. I. (1996). Generalization to local remappings of the visuomotor coordinate transformation. The Journal of Neuroscience, 16, 7085–7096. Harris, C. S. (1963). Adaptation to displaced vision: Visual, motor, or proprioceptive change? Science, 140, 812–813. Hay, J. C., & Pick, H. L. (1966). Gaze-contingent prism adaptation: Optical and motor factors. Journal of Experimental Psychology, 72, 640–648. Henriques, D. Y. P., & Soechting, J. F. (2003). Bias and sensitivity in haptic perception of geometry. Experimental Brain Research, 150, 95–108. Heuer, H., & Hegele, M. (2008). Adaptation to visuomotor rotations in younger and older adults. Psychology and Aging, 23, 190–202. Ingram, H. A., van Donkelaar, P., Cole, J., Vercher, J. L., Gauthier, G. M., & Miall, R. C. (2000). The role of proprioception and attention in a visuomotor adaptation task. Experimental Brain Research, 132, 114–126. Jeannerod, M. (1988). The neural and behavioural organization of goal-directed movements. Oxford, UK: Clarendon Press. Jones, S. H., Cressman, E. K., & Henriques, D. Y. P. (2010). Proprioceptive localization of the left and right hands. Experimental Brain Research, 204, 373–383. Kesten, H. (1958). Accelerated Stochasitc Approximation. The Annals of Mathematical Statistics, 29, 41–59. Klassen, J., Tong, C., & Flanagan, J. R. (2005). Learning and recall of incremental kinematic and dynamic sensorimotor transformations. Experimental Brain Research, 164, 250–259. Krakauer, J. W., Ghilardi, M. F., & Ghez, C. (1999). Independent learning of internal models for kinematic and dynamic control of reaching. Nature Neuroscience, 2, 1026–1031. Krakauer, J. W., Pine, Z. M., Ghilardi, M. F., & Ghez, C. (2000). Learning of visuomotor transformations for vectorial planning of reaching trajectories. The Journal of Neuroscience, 20, 8916–8924. Magescas, F., & Prablanc, C. (2006). Automatic drive of limb motor plasticity. Journal of Cognitive Neuroscience, 18, 75–83. McNay, E. C., & Willingham, D. B. (1998). Deficit in learning of a motor skill requiring strategy, but not of perceptuomotor recalibration, with aging. Learning & Memory, 4, 411–420. Miall, R. C., & Cole, J. (2007). Evidence for stronger visuomotor than visuo-proprioceptive conflict during mirror drawing performed by a deafferented subject and control subjects. Experimental Brain Research, 176, 432–439.
Miall, R. C., & Wolpert, D. M. (1996). Forward models for physiological motor control. Neural Networks, 9, 1265–1279. Ostry, D., Darainy, M., Mattar, A., Wong, J., & Gribble, P. L. (2010). Somatosensory plasticity and motor learning. The Journal of Neuroscience, 30, 5384–5393. Redding, G. M., Rossetti, Y., & Wallace, B. (2005). Applications of prism adaptation: A tutorial in theory and method. Neuroscience and Biobehavioral Reviews, 29, 431–444. Redding, G. M., & Wallace, B. (1996). Adaptive spatial alignment and strategic perceptual-motor control. Journal of Experimental Psychology. Human Perception and Performance, 22, 379–394. Salomonczyk, D., Cressman, E. K., & Henriques, D. Y. P. (2010). Proprioceptive recalibration increases with greater visuomotor distortions. In Paper presented at the Canadian Society for Psychomotor Learning and Sport Psychology, Ottawa, ON. Salomonczyk, D., Henriques, D. Y. P., & Cressman, E. K. (2010). Intermanual transfer of visuomotor adaptation without sensory recalibration. In Paper presented at the Enhancing performance for action and perception symposium, Montreal, QC. Simani, M. C., McGuire, L. M., & Sabes, P. N. (2007). Visualshift adaptation is composed of separable sensory and taskdependent effects. Journal of Neurophysiology, 98, 2827–2841. Treutwein, B. (1995). Adaptive psychophysical procedures. Vision Research, 35, 2503–2522. van Beers, R. J., Wolpert, D. M., & Haggard, P. (2002). When feeling is more important than seeing in sensorimotor adaptation. Current Biology, 12, 834–837. Vetter, P., Goodbody, S. J., & Wolpert, D. M. (1999). Evidence for an eye-centered spherical representation of the visuomotor map. Journal of Neurophysiology, 81, 935–939. Wang, J., & Sainburg, R. L. (2005). Adaptation to visuomotor rotations remaps movement vectors, not final positions. The Journal of Neuroscience, 25, 4024–4030. Wolpert, D. M. (1997). Computational approaches to motor control. Trends in Neurosciences, 1, 209–216. Wolpert, D. M., Ghahramani, Z., & Jordan, M. I. (1995). An internal model for sensorimotor integration. Science, 269, 1880–1882. Wong, T., & Henriques, D. Y. P. (2009). Effect of visuomotor adaptation on felt hand path. Journal of Neurophysiology, 101, 614–623.
A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 191 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.
CHAPTER 7
Perception and action in singing Sean Hutchins* and Isabelle Peretz BRAMS Laboratory and Department of Psychology, Université de Montréal, Montréal, Québec, Canada
Abstract: Singing is an important cultural activity, yet many people are hesitant to sing, because they feel they do not sing well. This chapter reviews the work that has been done concerning singing among nonmusicians, focusing on pitch accuracy, which is one of the most important aspects of singing. First, we look at the prevalence of poor pitch singing and examine what it means to be a poor singer. Next, we look at the possible causes of poor singing and examine the possible roles of perceptual deficits, sensorimotor translation deficits, motor control deficits, and feedback deficits. Whereas many previous studies have tried to explain poor singing by positing perceptual problems, we argue that the current evidence supports sensorimotor translation deficits and motor control deficits as the more likely causes. Finally, we examine the neural bases of singing and the possibility of a dualpathway basis for pitch perception and production. Based on these studies, we suggest changes to improve the singing abilities of poor singers. Keywords: singing; voice perception; vocal pitch matching. Happy Birthday, even at a crowded restaurant, they would not opt to do so in public without the reinforcing effect of others, even though these same people might not hesitate to sing alone in the car or the shower. Singing occupies an odd place in our culture. Whereas in other cultures, people sing regularly as part of their work or in gatherings, in Western society, singing has largely been relegated to a professional activity, where well-trained singers produce music with the benefit of microphones, recording studios, and even auto-tuning, in which the notes produced by the singer are automatically corrected by a computer. Thus, despite the fact that most people listen to music frequently and with pleasure, they
Introduction Sing the first few lines of your national anthem right now. Go ahead, sing it. Did you? I bet you didn't. I bet you thought about your national anthem, maybe sung it in your head, maybe whistled it, or even hummed it under your breath, but you probably did not sing it. Why? Most people feel self-conscious about singing, and especially singing alone in front of others. While most people would not hesitate to join into a chorus of *Corresponding author. Tel.: þ1 514 343 3432; Fax: þ1 514 343 2175. E-mail:
[email protected] DOI: 10.1016/B978-0-444-53752-2.00010-2
103
104
rarely hear live, unadulterated singing. Most instances of singing are heard with a type of perfection imposed upon them, often with a vocal quality that is impossible for a real human voice to reproduce without technological aid. Even more surprising, to an outsider, would be the fact that most Westerners rarely engage with music in a participatory setting, and if you ask people why they do not sing more often, the most common response you will hear is “I don't sing very well.” To the best of our knowledge, no serious crosscultural study has been performed comparing the singing abilities of people from societies with and without large-scale exposure to professionally recorded music. However, participatory singing occupies a much larger part of people's activities in non-Westernized cultures, and there is certainly not this widespread belief among people that they are unable to sing well. For example, an informal class survey showed that nearly three out of five students indicated that they could not imitate a melody with their voice (Pfordresher and Brown, 2007). It is possible that people from Western societies sing worse, on average, than those from non-Western societies, in which case this would likely be attributable to comparative lack of practice. It is also possible that there is no difference in singing abilities between the two groups. In this case, we would need to explain why so many people think that they sing so poorly. One of the main explanations for this effect could be the professionalization of singing. By hearing mostly audioengineered examples of trained singers, it is likely that our definition of what constitutes good, or even adequate singing has been shifted. Regardless of the nature of Western people's singing inadequacies, be they real or only perceived, one simple way of making people feel more comfortable in participating in singing activities is improving their singing abilities. But what does good singing ability actually entail? In a survey of over 1000 music educators, Watts et al. (2003) found that the pitch intonation was the most important factor of determining someone's singing ability. This factor was rated above other factors such as
musicality, vocal quality (timbre), range, diction, and others. The importance of intonation was later confirmed by Dalla Bella et al. (2007). It is telling that the most salient examples of vocal audio engineering involve pitch correction, especially autotuning, which is now even a part of many live shows. It is also telling that, in shows such as American Idol, the most jarring examples of poor singing tend to come from participants who do not sing in tune. Thus, one of the easiest ways to enhance people's musical performance, and to get people interested in performing, as well as consuming music, is to help them to improve their pitch intonation.
How well do people sing? How well do people sing? Or, more specifically, how correct or incorrect are people when singing an intended pitch? A few recent studies have attempted to quantify the pitch errors of musically untrained individuals. One such study was conducted by Dalla Bella et al. (2007). They recruited musically untrained singers and asked them to sing “Gens du Pays,” a very well known Québecois song typically sung for birthday celebrations. Some of these participants were recruited to come in to sing in a laboratory setting, but the majority of them were approached and sang for the experimenter in a more ecologically valid situation. In these cases, the experimenter approached the participants at a public park and asked them to sing “Gens du Pays” to him for his birthday. These were recorded with a portable microphone and recorder and later brought back to the laboratory for analysis. Among both those who sang in the lab and outdoors, an average of five to six intervals (an interval is the pitch distance between two successive notes) out of 31 were classified as errors. However, these data were not normally distributed: about 30% of singers made 0- or 1-pitch interval errors, whereas a small number of singers made significant errors on over half of all intervals, which would make
105
the song practically unrecognizable without words. Overall, about 80% of all of the song's intervals were sung within one semitone of the intended interval, which the authors interpreted as a reasonably high level of accuracy. However, interestingly, they found that the singers recruited in the park performed at a significantly lower level than those brought into the lab, despite the fact that the laboratory participants were specifically recruited to be nonmusicians, whereas the outdoors singers were not screened for musical ability. This may speak to a selection bias for participants coming to the laboratory, or less motivation for accurate singing among the outdoors singers, but it is worth keeping in mind as an important caveat for other studies of vocal pitch matching that take place in a laboratory. Another study by Pfordresher and Brown (2007) looked at singing ability through the lens of simpler tasks. They asked 79 participants to reproduce four-note sequences. These sequences were of three types: all the same pitch, two different pitches that formed one interval, or four unique pitches. In this sample, 87% of the participants could sing back the pitches within plus or minus one semitone. They also found that errors in producing the correct pitch interval tended to co-occur with errors in producing the correct absolute pitch. Singers who sang the wrong pitches tended to reduce pitch interval size, and Pfordresher and Brown (2007) concluded that poor singing was primarily related to vocal pitchmatching—rather than interval-matching—ability. In our own work, we have endeavored to examine vocal pitch-matching ability in musicians and nonmusicians. We have used simple pitchmatching tasks to measure this ability without any intervening complications, such as time pressure or memory effects. Our results show that almost 50% of nonmusicians failed to match a target pitch to within half a semitone on half of their attempts (Hutchins and Peretz, in preparation-a). This is a considerably higher rate of poor singing than was found in Pfordresher and Brown (2007) and Dalla Bella et al. (2007), who
estimated that only about 10–20% of the population should be classified as poor singers. There are a few reasons why this might be the case. First, our task was simpler in nature and, because of this, may have provided fewer cues to aid singers in finding the correct pitch. Whereas the other studies provide tonal contexts in which singers could use their implicit knowledge of key structure and the relationship between notes in a key, the tones in our study were not tonally related, and singers could not use intertrial relationships to help them determine the correct pitch of a trial. In fact, in Pfordresher and Brown's study (2007), where stimulus complexity was manipulated, there was a tendency for better pitch matching in the more complex stimuli among the poorer singers, which indicates that we should see more instances of poor singing in less complex tasks. In addition, poor singing was defined in slightly different ways across these studies. Dalla Bella et al. (2007) studied impromptu song performance and so could not measure pitch matching per se, but used only relative interval sizes to determine accuracy. Pfordresher and Brown (2007) measured both absolute and relative accuracy but used absolute accuracy to label singers as good or poor. Both of these, however, used plus or minus one semitone as the demarcation between good and poor singing, whereas we used plus or minus half a semitone. Both of these metrics have some validity to them, but the full semitone demarcation is significantly more forgiving. This can explain some of the difference in measured prevalence of poor singing between the studies, but even if we use the more liberal criterion of a full semitone, our data still show approximately 40% of singers as making more than 50% errors. This criterion difference does raise an important issue, however, namely, what counts as poor singing? Do listeners think that a note that is half a semitone off from its intended target is out of tune? Or is it closer to a full semitone? One way to test this is to examine the minimum amount of pitch deviation that can be detected by listeners. Just noticeable differences for pitch can vary but are typically
106
estimated at around 5 cents (i.e., 1/20th of a semitone; 100 cents ¼ one semitone; Zwicker and Fastl, 1999). However, these can vary, with some expert listeners reporting just-noticeable differences for pitch even lower and many nonmusicians reporting thresholds closer to 15 or 20 cents. Klatt (1973) estimated the just-noticeable difference of the synthetic vowel / e / at just under 5 cents. However, naturally sung tones are much more complex: they generally include many upper harmonics, some nonharmonic frequencies, and pitch fluctuations, such as vibrato (sinusoidal pitch variation) and unintended pitch change. These complexities will serve to make pitch judgments more difficult, and it is likely that the just-noticeable differences for sung tones are higher in these cases. We have examined these perceptual limits for voice recordings. Participants listened to short melodies and single tones, either played on a violin or sung by a trained singer on the syllable/ba/. In the melody trials, the final tone of the melody was pitch-shifted anywhere from 100 to þ 100 cents, by 10-cent steps. In the single tone trials, we presented two tones, which could be the same or shifted by the same amounts as in the melody condition. In both conditions, the participants decided whether the final note was in tune (i.e., the same note, in single tone trials), the right note, but out of tune, or a wrong note altogether. Note that these types of judgments were shown to be equivalent for single tone and melodic contexts by Warrier and Zatorre (2002). Figure 1 shows the pattern of each type of judgment by nonmusicians for single tones which were sung versus played on the violin. On average, nonmusicians do not notice the tuning difference between two sung tones until they are different by at least 40 cents, whereas they can tell the difference between two violin tones that are only 30 cents apart. We have also replicated these single tone results in tuning judgments of untrained singers and synthesized vocal tones, showing that the higher thresholds for noticing tuning differences are not specific to trained singing voice or the violin.
However, most musical judgments of tuning are not made by asking whether two versions of the same tone are identical. They instead consist of judgments of whether a particular note fits into the musical context, which is what we measured with the melodic context trials. In their tuning judgments, nonmusicians needed the final tone to be mistuned by at least 60 cents before they recognized any mistuning, when the melody was produced by a voice. The smallest consistently noticeable difference was much lower, only 30 cents, when the melody was played on the violin. This difference tells us two things. First, the same tuning deviances are less noticeable when sung, compared with being played on a violin. Listeners seem to be either less likely to notice vocal mistuning than instrumental mistuning or more forgiving to singers than instrumentalists. This may be due to the properties of the voice, such as large amounts of vibrato; however, the violin uses vibrato as well (although not to the same extent). Second, these measurements confirm that one-half semitone (50 cents) is more likely a better criterion than 100 cents (one semitone) for determining whether a sung tone is correct or not. Most tones in the 50–100 cents error range are indeed heard as mistuned, both in an exact pitch-matching context and in a melodic tuning context, and it seems to be the case that if a tone is not produced within one-half semitone of the intended target, it will be heard as mistuned. Using these criteria, it seems the estimates of poor or unreliable singers may be nearly half the population (Hutchins and Peretz, in preparation-a). Thus, it is no surprise that people may be hesitant to sing in public.
What causes poor singing? To improve adults’ singing ability, it is important to know the root of their problem. A question like the causes of poor singing can have two types of answers, however. It is tempting to argue that inadequate practice is the cause of poor singing,
107
Proportion judged in category
Voice single tones 1.00 0.90 0.80 0.70 0.60 0.50 0.40 0.30 0.20 0.10 0.00
In-tune Mistuned Wrong note
0
10
20
30
40
50
60
70
80
90
100
Deviation of final tone in cents
Proportion judged in category
Violin single tones 1.00 0.90 0.80 0.70 0.60 0.50 0.40 0.30 0.20 0.10 0.00
In-tune Mistuned Wrong note
0
10
20
30
40
50
60
70
80
90
100
Deviation of final tone in cents Fig. 1. The pattern of each type of judgment (in tune, the right note but mistuned, or wrong note altogether) by nonmusicians in the single tone context for sung melodies (a) and played on the violin (b).
and it is most likely true that greater amounts of practice, especially among children, will lead to better vocal performance. However, this teleological type of reasoning does not tell us the immediate causes of poor singing and fails to explain many salient cases, such as naturally gifted untrained singers, or people who have been through a comprehensive music education and still fail to match pitch adequately. It also does not give us an idea of what specific benefits practice can have or provide practical advice to music
educators who would like to improve their pedagogical techniques and are unsure how to do so. Therefore, we will restrict our discussion to proximal causes of poor singing: the immediate physical and psychological impediments which lead to an erroneous response on a particular trial or set of trials. Until recently, research on this topic had been quite limited, due to the technological constraints, and there were few studies of normal adult singing ability using acoustical measurements.
108
Several music education researchers tested children's singing abilities (e.g., Bentley, 1969; Cleall, 1979; Joyner, 1969) as a pedagogical tool, but many of these studies did not use rigorous scientific designs. However, there were some insights into the processes causing poor singing ability. For example, Bentley (1969) noted that monotone singers tended to have lower pitch-discrimination scores. Joyner (1969) followed on this work, evaluating the singing of a sample of “monotone” singers. He posited that three processes were necessary for good singing ability. First, singers must be able to discriminate pitches from each other. Second, singers must be able to recall the organization of pitches in a melody. Finally, singers must possess a vocal instrument capable of producing the intended pitches and able to respond quickly and accurately to their intentions. Joyner also suggested that “monotonism” was caused by a motor dysfunction and that perceptual problems may be a by-product, rather than a cause, of poor singing ability. Since the 1960s, other music educators have continued to research singing abilities in grade school children, using similar methods and elaborating upon these conclusions, often with the explicit aim of improving teaching techniques. Since the turn of the century, though, there has been a revitalization of singing research. Because of the ease of applying acoustical analyses to digitally recorded data, there have been many studies on the prevalence of good and poor singing, the factors that count as good or poor singing, the root causes of poor singing, and external factors that can affect singing ability. The question of why some people cannot sing in tune is a particularly interesting one because singing is generally not a consciously accessible ability.
External sound
Pitch perception
That is, vocal production is such an automatic process, even among poor singers, that most people do not understand how they are using their bodies when they produce a note or how to adjust their vocal mechanism to change their sound. Those who can sing well typically cannot understand how someone could fail to sing well, and those who cannot sing well have no idea how to improve. Vocal pitch production, in fact, can be even harder to learn about than other automatic tasks, such as walking, because the relevant mechanisms to create vocal pitch are inside the body, and we only have direct access to the final outcome. This is why a lot of singing teachers use metaphor and imagery to try to improve their students’ sound, with instructions like “breathe through your navel” and “pretend you are yawning”; these are simply more useful to many singers than “expand your pharyngeal cavity” or “tighten your thyroarytenoid muscles.” These types of pedagogical strategies are helpful because of the lack of conscious knowledge of our vocal mechanism, which makes understanding the causes of poor singing all the more difficult to discover. In their 2007 paper, Pfordresher and Brown listed four possible causes of inaccurate singing. To the three causes listed by Joyner—a perceptual deficit, a production deficit, and a memory deficit—they added a fourth, a sensorimotor mismapping. This refers to an inability to connect an accurately heard pitch to the same vocal response. Taking these into consideration, Fig. 2 shows the steps that must be undertaken in order for someone to sing back a pitch they hear. This figure models immediate reproduction and thus does not account for any possible effects of pitch or song memory. First, a sound is heard and the
Motor plan selection
Motor plan execution
Fig. 2. The steps necessary to accurately sing back the pitch of a presented sound.
Output evaluation
109
brain determines the pitch of the sound (note that this is often, but not always, the same as the fundamental frequency, the primary acoustic correlate of pitch). Second, an appropriate motor plan for producing that pitch must be accessed. Third, the motor plan must be acted upon and must be executed properly. Finally, the output can be evaluated, either from acoustical feedback or feedback from an efference copy of the motor command, and adjustments to the motor plan can be made accordingly. The next section of this review will deal with evidence for deficits at each stage in this process.
Perceptual deficits Musicians and researchers both have long entertained the hypothesis that the reason some people do not sing in tune is that they cannot perceive the pitch accurately. However, in vocal pitch matching, one cannot separate the pitch perception and pitch production stages; it is necessary for a singer to perceive the intended pitch before it can be matched (excluding the relatively rare case of singers with absolute pitch (perfect pitch), who use long-term memory as a basis for their pitch matching). Therefore, one technique used to determine the relationship between the two processes is to measure perception separately, without invoking a vocal matching response and the vocal–motor code associated with it, and correlate this measurement with vocal pitch production abilities. However, the problem with this design is that polling perceptual abilities generally means requiring the use of other cognitive abilities which are not present in vocal pitch matching. Most often, these concern making same/different judgments about a pair of pitches. Several studies have reported correlations between pitch-discrimination abilities and vocal pitch accuracy (e.g., Estis et al., 2009, in press; Moore et al., 2007; Watts et al., 2005). However, there have been many other studies which failed to find such a relationship (e.g., Bradshaw and
McHenry, 2005; Dalla Bella et al., 2007; Moore et al., 2008; Pfordresher and Brown, 2007). The conflicting results make for a complicated story overall, and it is still in doubt whether there actually is a true correlation between pitch-discrimination abilities and vocal pitch accuracy. However, there are some major problems which are endogenous to the paradigm. First, there are some large variations between the methodologies used to measure perception and production abilities between experiments, with different studies using different comparison pitches, different timbres (i.e., the color of a note—this is what makes a piano sound different from a guitar), and different types of subjects. These differences between studies make it difficult to discover what type of consensus may underlie them; the variability in their findings may be more reflective of the methodological differences than the underlying relationship between perception and production. Another major problem with these studies is their use of percentage of correct responses on the perception task as the variable to correlate with the pitch-matching ability. Percentage of correct responses does not reflect the same kind of measurement as error in vocal pitch matching. Vocal pitch-matching error should be correlated with perceptual error, rather than a discrete correct or incorrect measurement of perception. This average perceptual judgment error is equivalent to the just-noticeable difference for pitch (pitchdiscrimination threshold). The measurement of just-noticeable differences for pitch was done in two different studies. Amir et al. (2003) showed a significant correlation between pitch perception thresholds and vocal pitch-matching ability; this represents a more appropriate type of perceptual measurement than other percent correct measurements. However, this study had some limitations. The pitch-discrimination thresholds were measured at a much higher pitch level than the vocal pitch-matching trials and used unnatural-sounding sine tones. Furthermore, each participant was asked to sing
110
only nine times, which is probably not enough to specify the mean pitch-matching error, especially among poor singers, who tend to be more variable in their responses (Hutchins and Peretz, in preparation-a; Pfordresher et al., in press). Amir et al.'s (2003) measured pitch perception thresholds were also considerably lower than the production errors, making it unlikely that perception error was the limiting factor in vocal pitchmatching ability. Nikjeh et al. (2009) improved upon this design, using complex tones instead of sine tones and measuring perceptual ability at the same pitch height as vocal pitch matching. However, they found a correlation between these two abilities only among trained instrumentalists, but not trained singers or nonmusicians. In addition, their measured pitch perception thresholds were quite high (see, e.g., Hyde and Peretz, 2004; Zwicker and Fastl, 1999 for comparison); they attributed this discrepancy to their stimulus timbre and procedure. Another problem with these studies of perception is their use of a correlational design to evaluate the perception–production relationship. Even a significant, positive correlation does not give a solid answer as to whether poor perceptual ability causes poor singing performance. It is just as possible that poor singing ability causes poor performance on these perceptual comparison tests (perhaps listeners are using their voice to evaluate whether the two tones are the same) or that both abilities are regulated by some third factor (e.g., attention). Zarate et al. (2010) used a true experimental design to evaluate whether improving perceptual abilities would improve people's singing abilities. Over six sessions, they trained participants to perceive micromelodies, which use very small, nontraditional intervals. These participants showed significant improvements in pitch-discrimination abilities; however, there was no evidence that their vocal pitch-matching abilities had improved. This demonstrates that pitch-discrimination ability was likely not the limiting factor for their singing ability.
In a new study (Hutchins and Peretz, in preparation-a), we measured pitch perception abilities in a different way. Rather than taking a set of same or different judgments, we let participants use a physical slider to adjust a comparison pitch to match a target pitch. This method measures pitch matching with an active process, rather than a passive decision comparing a set of tones. Using the slider approximates the task of vocal pitch matching much better than other types of decision-based tasks and allows us to compare error in a vocal pitch-matching task to error in an instrumental pitch-matching task using the same types of measurements. Both musicians and nonmusicians were very accurate in pitch matching using the slider, to a much greater degree than they were using their voice. Only 6% of our participants (2 out of 31) showed any difficulty using the slider (see Table 1), and we concluded from this data that participants were capable of matching a pitch on a different apparatus, and pitch perception deficits were not responsible for their poor singing. Although it is a popular theory, the majority of the evidence seems to indicate that poor pitch perception ability does not seem to be a major cause of poor singing. At best, poor singing ability may tend to co-occur with poor performance on pitch-discrimination tasks, but in general, pitchdiscrimination abilities tend to be much better than pitch production abilities. This is not what you would expect if the former were the major cause of poor singing and suggests that the real Table 1. The potential causes of poor singing and the percentage of the population estimated to be affected by deficits in each of these areas (from Hutchins and Peretz, in preparation-a) Cause of singing problems
Percent of population
Perceptual Sensorimotor Motor control Feedback No problems
6% 35% 19% Not assessed 39%
111
cause is likely in the accessing or execution of the proper motor plan. However, we would be remiss if we did not mention the notable case where perceptual disability does seem to be at the heart of singing problems. Congenital amusia, commonly referred to as tone deafness, is estimated to affect around 4% of the population (Kalmus and Fry, 1980). Congenital amusics have very poor musical abilities and lack the ability to recognize the tune of songs or perceive wrong notes in familiar melodies, without any history of brain trauma or other neurological disorders. It is thought to be a neurogenetic disorder (Peretz, 2008) which leads to severe pitch perception difficulties (Hyde and Peretz, 2004). Congenital amusics generally have very poor singing abilities (Ayotte et al., 2002; Dalla Bella et al., 2009) and poor pitch-matching abilities (Hutchins et al., 2010), although a small number retains some ability to sing better than one might expect given their perceptual difficulties. In the case of congenital amusia, it does seem that their impaired pitch perception is at least related to their poor singing ability.
Sensorimotor translation deficits Another possible cause of poor pitch matching is that an appropriate motor plan is not selected. In this account, poor singers correctly perceive the intended target pitch but select a motor plan that, even if correctly executed, still results in an errorful output. There are no errors in perception or motor control, but there is an error with the motor plan they choose to enact. There are in fact many different ways to produce the same pitch, so the process of selecting a motor plan is not straightforward, making this a potentially difficult task. Because they could not find any evidence of problems within the perception or motor systems alone, Pfordresher and Brown (2007) proposed that there was a mismapping between auditory representations and motor representations of
pitches. They proposed that among inaccurate singers, there was a regular transformation between the two, such that the inaccuracies are constant. This would suggest that when participants fail to match a pitch, they should always sing the same incorrect pitch in response to the same target tone. However, other studies have shown that inaccurate singers tend to be more variable in their responses than good pitch matchers (Pfordresher et al., in press; Hutchins and Peretz, in preparation-a), which argues against this constant transformation view. Another factor may be at the root of a possible mismapping problem, though. The timbre of the target tone (i.e., the color or type of the sound) can make a big difference to people's pitchmatching abilities. It is clear that the timbre of a tone can affect its perceived pitch (Krumhansl and Iverson, 1992; Melara and Marks, 1990a,b,c; Pitt, 1994; Warrier and Zatorre, 2002), and prior work has shown that people are better at matching the pitch of target tones when they are more similar to their own voice (e.g., Watts and Hall, 2008), and especially so to actual recordings of their own voice (Moore et al., 2008). In most pitch-matching studies, however, participants are asked to match either an instrumental timbre or a complex-synthesized sound. Thus, it may be that pitch-matching errors occur because the singer cannot determine the proper relationship between the target tone and their vocal response. It is as if different timbres constitute different languages, and poor singers cannot translate between them, unable to find which vocal pitch is equivalent to which target pitch. In one of the experiments of our recent study (Hutchins and Peretz, in preparation-a), we showed that 35% of singers (Table 1) are able to match synthesized vocal tones (which bear a resemblance to the voice but are not identical) with a slider but are not able to match these synthesized vocal tones with their own voice. These same participants, however, are able to match recordings of their own voice quite accurately. This indicates that neither do they have any problems perceiving the pitch of a tone, nor
112
do they have any motor control issues to prevent them from accurate singing. However, they fail to translate the synthesized vocal tone to the appropriate vocal–motor plan. This is evidence that these participants do have such a sensorimotor translation problem. Because it is due to a lack of translation between timbres, rather than a fixed mismapping, these participants do not make constant errors but produce errors with a great deal of variability. Interestingly, even though they often realize they are matching pitch incorrectly, they do not realize how to improve their response. This leads to the interesting conundrum that incorrect singers will sing the same incorrect pitch multiple times when given the opportunity to correct it during a single trial but will sing a different incorrect pitch on a subsequent trial at the same target pitch level. In a follow-up experiment in which we required at least 20 pitch-matching attempts per trial from singers, none showed any significant within-trial pitch changes, but poor singers did show between-trial variability. It is important to note that this timbral translation explanation is distinct from a pure perceptual problem. Within a timbre, listeners can resolve pitch quite well, easily determining higher–lower and same–different judgments. The problem only arises when the listener needs to make comparisons across timbres, in this case, from their representation of the target tone timbre to their representation of their own voice, and the correct motor plan to produce this. Another type of translation problem that commonly occurs concerns pitch transposition. It has been shown that singers with a high voice can have a hard time imitating singers with a low voice, and vice versa (Clegg, 1966; Goetze et al., 1990; Pfordresher and Brown, 2007). These types of singing errors seem to have to do mainly with a problem in transposing up or down an octave (a common task requirement for singing in groups) and constitute a similar difficulty of finding equivalent pitches, across range rather than timbre. Other translation types of errors may also exist, but these have yet to be identified.
Motor control deficits A third type of explanation for poor singing abilities is a pure motor problem. That is, poor singers may accurately perceive a pitch and select the correct motor plan but may not have the range or coordination to faithfully execute this plan. The possibility of overt vocal problems is sometimes assessed in studies of vocal pitch matching, but these typically address problems such as vocal range and the ability to adequately sustain a tone. However, motor control problems may in fact lie in other areas, specifically the ability to coordinate one's vocal–motor apparatus to achieve a desired frequency. These types of motor control issues are rarely addressed directly and sometimes may be seen only by ruling out other types of problems. For example, in our recent work on pitch matching (Hutchins and Peretz, in preparation-a), we showed that about 20% of participants (Table 1) could match pitch well on the slider but were impaired in vocal pitch matching both in the case where the target tone was a synthesized vocal tone and where it was a recording of their own voice. Because they neither had direct pitch perception problems nor were good at matching the target in the own voice condition, showing little evidence for a sensorimotor translation deficit, we concluded that these participants had a problem with their motor control of their vocal apparatus. This type of lack of coordination is often observed among amateurs in other physical activities as well. For example, a poor bowler may perceive exactly where he wishes to aim a ball, but may simply not be coordinated enough to do so consistently. It is not surprising that individuals without much practice at using their voice in this way would lack the coordination to sing consistently on pitch, and the finding that inaccurate singers tend to be inconsistent with their productions (Hutchins and Peretz, in preparation-a; Pfordresher and Brown, 2007, Pfordresher et al., in press) is consistent with this account. However, people do tend to have a great deal of practice
113
using their voice in another manner, that is, speech. Being another vocal method for communication, speech necessarily involves some of the same types of properties of the acoustic signal as singing. Both speech and singing use changes in timbre, amplitude, timing, and pitch to create a meaningful signal. However, unlike singing, the vast majority of people are very skilled at using their voice in precise ways to express what they wish to say. Although pitch is not as important in speech as it is in singing (with the exception of tone languages such as Mandarin, which can use pitch to distinguish between otherwise identical words), pitch is still used to convey meaning in speech in several ways, including marking the focus of a sentence, or indicating whether a sentence is a statement or a question. Most people do in fact manipulate the pitch of their voice in such a way as to create exactly the nuances in meaning that they wish to convey, and it should come as no surprise that vocal communication is for the large part successful. This raises the issue, then, of how people who are good at using pitch cues with their voice in speech could lack the coordination and/or motor control to sufficiently control their vocal pitch in singing. The answer to this apparent puzzle lies in the realization that singing and speech may require two different types of vocal control. Just as a scratch bowler might be a terrible softball pitcher despite the similarities between the two types of movements, the practice at controlling vocal pitch in speech might not carry over to the music domain. Therefore, it is reasonable that many people may be very skilled at using their voice for speech, but not coordinated in using their voice for singing. We have shown that congenital amusics have retained abilities to imitate language just as well as controls, despite reduced pitch sensitivity and reduced singing abilities (Hutchins and Peretz, in preparation-b). In addition, we have found a case of acquired amusia, IR, who after brain damage has lost the ability to produce any sung tone and can only speak the words of songs. IR, however, was unaware that she was not singing during the
singing tasks, which implies that she may have a singing-specific motor deficit.
Feedback deficits The final step of singing production that may be impaired is evaluating feedback and making adjustments based on this feedback. That is, poor singers may perceive a pitch correctly, select an appropriate motor plan, and produce it with reasonable accuracy, but misconstrue their own accuracy and adjust their pitch afterwards in an errorful way. Or, it may be that good and poor singers both tend to make the same initial inaccuracies, but good singers are able to use their feedback to correct their responses, whereas poor singers are unable to do so. We know that feedback from the vocal signal is monitored by singers and speakers from the way that they react to alterations to this vocal feedback. For example, delayed auditory feedback causes severe disruptions to speech (MacKay, 1987), and pitch shifting the auditory feedback of speakers and singers evokes automatic compensatory responses in the opposite direction (Burnett and Larson, 2002; Burnett et al., 1998; Natke et al., 2003), even when these differences are not consciously perceived (Hafke, 2008). Masking the auditory feedback of singers also reduces their pitchmatching accuracy (Anstis and Cavanagh, 1979; Elliott and Niemoeller, 1970; Mürbe et al., 2002; Ternström et al., 1988), which shows that feedback evaluation and adjustments are indeed important to accurate singing. In addition, there is some evidence that enhancing feedback through accompaniment (to provide a concurrent comparison) can aid singers. Wise and Sloboda (2008) found that singers were better at matching pitch when singing along with another singer. However, several studies of elementary school children seem to show that they perform less accurately in a choral context than alone (Clayton, 1986; Goetze, 1986; Joyner, 1971; cited in Goetze et al., 1990), possibly due to the
114
masking effects of the other singers, which may in fact be reducing, rather than enhancing feedback. However, the majority of evidence we have indicates that deficits in feedback evaluation and adjustment are not a primary cause of poor singing. Pfordresher and Brown (2007) manipulated the feedback they provided to their participants in their pitch-matching tasks and found that masking the feedback made no difference to either good or poor singers, and accompaniment actually led to a small decrement in performance among poor singers (who should have been most aided by this, if their problems were due to feedback). In our study on pitch matching (Hutchins and Peretz, in preparation-a), we showed that neither musicians nor nonmusicians made any large corrections to their pitch across multiple attempts to match the same pitch, but that good and poor singers alike tend to persist on a pitch, whether it is accurate or not, which adds more evidence against the feedback account of poor singing. Although feedback is an important process in the singing system, it is most likely not a major cause of inaccurate singing. In the final section of this chapter, we will briefly examine the neural structures which underlie the steps necessary to singing that we have outlined above.
Neural bases of singing Several studies have examined brain activity during singing tasks. Perry and collaborators (1999) used positron emission tomography during a simple singing task to identify regions involved in perceiving sounds and in producing sung tones. This study showed that a network involving the right Heschl's gyrus and a posterior region in the right superior temporal plane, which contain primary and higher-order auditory areas, were engaged during singing. In addition, several motor control areas were recruited for the singing task, including the supplementary motor area, the anterior cingulate cortex, the precentral gyrus, the anterior insula, and the cerebellum. These regions
are very similar to those recruited during speech production (Özdemir et al., 2006; Paus et al., 1993, 1996). Activity in these same regions has been confirmed in several subsequent studies using more complex singing tasks and involving participants with varying degrees of musical training (Brown et al., 2004; Kleber et al., 2007; Zarate and Zatorre, 2008). Although the basic elements of the singing network have been identified, it is still difficult to map all of the steps required for singing identified in Fig. 2 to particular brain regions. While perception and motor control areas in the singing network are easily identifiable due to their recruitment in other tasks involving those functions, it is harder to identify areas involved in motor plan selection and feedback evaluation. Zarate and Zatorre (2008) showed that the intraparietal sulcus and the dorsal premotor cortex were both activated in response to changes in auditory feedback during singing and so may be involved in both feedback evaluation and motor plan selection. Foster and Zatorre (2010) also showed intraparietal sulcus activity in response to a pitch task in which transposed melodies are compared, which makes it likely that this region may be involved in pitch translation tasks. The dorsal premotor cortex, however, has been implicated in other types of auditory–motor interactions (Chen et al., 2006; Zatorre et al., 2007) and so may be responsible for adjusting motor output in response to auditory events. Finally, there might a qualitative difference in the functioning of the ventral and dorsal pathways connecting auditory and motor regions, which underlies the traditional distinction between perception and production areas. In a behavioral study, Hafke (2008) showed that trained singers will make unconscious vocal adjustments in response to very small pitch shifts, even when those shifts are not consciously perceived. A subset of congenital amusics show relatively preserved singing (Dalla Bella et al., 2009) and pitch-matching ability (Hutchins et al., 2010). Some amusics can even reproduce the
115
direction of a pitch interval with their voice but yet be unable to name the direction of the same interval (Loui et al., 2008), and in rare cases, can even sing whole songs without being able to distinguish tone pairs which are easy for normal listeners (Hutchins and Peretz, 2010). This action–perception dissociation bears a resemblance to a type of blindsight in which patients who experience no conscious visual input nonetheless perform much better than chance at goal-directed tasks, such as pointing at an object (Danckert and Rossetti, 2005). Such dissociations lend support to dual-route models (e.g., Hickok and Poeppel, 2007; Milner and Goodale, 1995; Rauschecker and Scott, 2009; Warren et al., 2005), in which the ventral and dorsal pathways support (conscious) perception and (unconscious) action, respectively. Supporting evidence for this was found for music, as well, by Loui et al. (2009), who suggested that the superior and inferior pathways in the arcuate fasciculus, connecting auditory and frontal premotor regions, are separately responsible for pitch perception and production. This may explain some of the dissociations seen in both amusics and normal participants.
Conclusion We began this review by asking “Why don't people sing more often?” One of the most common answers to this question is that they feel they are not a strong enough singer. Many people would like to be able to sing better, and people often feel very self-conscious about the quality of their voices. However, it also seems to be the case that few poor singers become noticeably better later in life, and one of the reasons is that many voice teachers do not truly understand the roots of poor singing, and especially of poor pitch accuracy—one of the primary determinants of poor singing (Watts et al., 2003). Although most of the research focus lately has been on the role of impoverished pitch perception on poor singing,
we believe that the recent evidence points toward poor sensorimotor translation and motor control as being the primary determinants of poor singing. Music teachers are trained to mainly help young children and can sometimes help them to improve their pitch-matching ability, but teachers have little research to help guide their pedagogical strategy. If we can more fully understand what processes are responsible for poor singing behavior, we can adapt our music pedagogy to be able to better help poor singers of all age levels improve. Our research implies that poor singers need more help in recognizing pitch equivalents between instruments and voices, and in developing better vocal–motor coordination, rather than practice in simple pitch perception. This research may also lead to the creation of other tools to aid training; adapted versions of our slider could prove to be effective teaching tools. Imaging work may also give us further insight as to how these processes of perception, sensorimotor translation, motor control, and use of feedback are carried out in the brain and may lead to further insights on the causes of poor singing and may even help identify people with as yet unexpressed musical talent. In sum, our understanding of poor singing is an important step in helping people to not only enjoy music they get from society but also contribute to the music of society and gain the pleasure of performing.
Acknowledgments This work was supported by grants from the Canadian Institutes of Health Research, the Natural Sciences and Engineering Research Council of Canada, and a Canada Research Chair in neurocognition of music to I. P. and a fellowship in Auditory Cognitive Neuroscience from the CREATE program of the Natural Sciences and Engineering Research Council of Canada to S. H.
116
References Amir, O., Amir, N., & Kishon-Rabin, L. (2003). The effect of superior auditory skills on vocal accuracy. The Journal of the Acoustical Society of America, 113, 1102–1108. Anstis, S. M., & Cavanagh, P. (1979). Adaptation to frequency-shifted auditory feedback. Perception & Psychophysics, 26, 449–458. Ayotte, J., Peretz, I., & Hyde, K. (2002). Congenital amusia: A group study of adults afflicted with a music-specific disorder. Brain, 125, 238–251. Bentley, A. (1969). Monotones. Music Education Research Papers (1) London: Novello and Company. Bradshaw, E., & McHenry, M. A. (2005). Pitch discrimination and pitch matching abilities of adults who sing inaccurately. Journal of Voice, 19, 431–439. Brown, S., Martinez, M. J., Hodges, D. A., Fox, P. T., & Parsons, L. M. (2004). The song system of the human brain. Cognitive Brain Research, 20, 363–375. Burnett, T. A., Freedland, M. B., Larson, C. R., & Hain, T. C. (1998). Voice F0 responses to manipulations in pitch feedback. The Journal of the Acoustical Society of America, 103, 3153–3161. Burnett, T. A., & Larson, C. R. (2002). Early pitch shift response is active in both steady and dynamic voice pitch control. The Journal of the Acoustical Society of America, 112, 1058–1063. Chen, J. L., Zatorre, R. J., & Penhune, V. B. (2006). Interactions between auditory and dorsal premotor cortex during synchronization to musical rhythms. Neuroimage, 32, 1771–1781. Clayton, L. S. (1986). An investigation of the effect of a simultaneous pitch stimulus on vocal pitch accuracy. Unpublished master's thesis, Indiana University, Bloomington. Cited in Goetze, M., Cooper, N., & Brown, C.J. (1990). Singing in the general music classroom. Bulletin of the Council for Research in Music Education, 104, 16–37.. Cleall, C. (1979). Vocal range in young children. Paper presented at the course in Development of Young Children's Music Skills, University of Reading. Cited in Goetze, M., Cooper, N., & Brown, C.J. (1990). Singing in the general music classroom. Bulletin of the Council for Research in Music Education, 104, 16–37. Clegg, B. W. (1966). A comparative study of primary grade children's ability to match tones. Unpublished master's thesis, Brigham Young University, Provo, UT. Cited in Goetze, M., Cooper, N., & Brown, C.J. (1990). Singing in the general music classroom. Bulletin of the Council for Research in Music Education, 104, 16–37. Dalla Bella, S., Giguère, J. F., & Peretz, I. (2007). Singing proficiency in the general population. The Journal of the Acoustical Society of America, 121, 1182–1189.
Dalla Bella, S., Giguère, J.-F., & Peretz, I. (2009). Singing in congenital amusia: An acoustical approach. The Journal of the Acoustical Society of America, 126, 414–424. Danckert, J., & Rossetti, Y. (2005). Blindsight in action: What can the different sub-types of blindsight tell us about the control of visually guided actions? Neuroscience and Biobehavioral Reviews, 29, 1035–1046. Elliott, L. L., & Niemoeller, A. F. (1970). The role of hearing in controlling voice fundamental frequency. International Journal of Audiology, 9, 47–52. Estis, J. M., Coblentz, J. K., & Moore, R. E. (2009). Effects of increasing time delays on pitch-matching accuracy in trained singers and untrained individuals. Journal of Voice, 23, 439–445. Estis, J. M., Dean-Claytor, A., Moore, R. E., & Rowell, T. L. (2010). Pitch-matching accuracy in trained singers and untrained individuals: The impact of musical interference and noise. Journal of Voice, 25, 173–180. Foster, N. E. V., & Zatorre, R. J. (2010). A role for the intraparietal sulcus in transforming musical pitch information. Cerebral Cortex, 20, 1350–1359. Goetze, M. (1986). Factors affecting accuracy in children's singing. Doctoral dissertation, University of Colorado. Cited in Goetze, M., Cooper, N., & Brown, C. J. (1990). Singing in the general music classroom. Bulletin of the Council for Research in Music Education, 104, 16–37. Goetze, M., Cooper, N., & Brown, C. J. (1990). Recent research on singing in the general music classroom. Bulletin of the Council for Research in Music Education, 104, 16–37. Hafke, H. Z. (2008). Nonconscious control of fundamental voice frequency. The Journal of the Acoustical Society of America, 123, 273–278. Hickok, G., & Poeppel, D. (2007). The cortical organization of speech processing. Nature Reviews. Neuroscience, 8, 393–402. Hutchins, S., & Peretz, I. (2010). Double dissociation of pitch production and perception. In: Poster presented at 17th annual meeting of the Cognitive Neuroscience Society, Montreal, April, 2010. Hutchins, S., & Peretz, I. (in preparation-a). Why do some people fail to sing in tune? A comparison of pitch perception and production abilities of musicians and nonmusicians. Hutchins, S., & Peretz, I. (in preparation-b). Perceiving and producing pitch in language. Hutchins, S., Zarate, J. M., Zatorre, R. J., & Peretz, I. (2010). An acoustical study of vocal pitch matching in congenital amusia. The Journal of the Acoustical Society of America, 127, 504–512. Hyde, K. L., & Peretz, I. (2004). Brains that are out of tune but in time. Psychological Science, 15, 356–360. Joyner, D. R. (1969). The monotone problem. Journal of Research in Music Education, 31, 115–124.
117 Joyner, D. R. (1971). Pitch discrimination and tonal memory and their association with singing and the larynx. Master's thesis, University of Reading. Cited in Goetze, M., Cooper, N., & Brown, C. J. (1990). Singing in the general music classroom. Bulletin of the Council for Research in Music Education, 104, 16–37. Kalmus, H., & Fry, D. B. (1980). On tune deafness (dysmelodia): Frequency, development, genetics and musical background. Annals of Human Genetics, 43, 369–382. Klatt, D. H. (1973). Discrimination of fundamental frequency contours in synthetic speech: Implications for models of pitch perception. The Journal of the Acoustical Society of America, 53, 8–16. Kleber, B., Birbaumer, N., Veit, R., Trevorrow, T., & Lotze, M. (2007). Overt and imagined singing of an Italian aria. Neuroimage, 36, 889–900. Krumhansl, C. L., & Iverson, P. (1992). Perceptual interactions between musical pitch and timbre. Journal of Experimental Psychology. Human Perception and Performance, 18, 739–751. Loui, P., Alsop, D., & Schlaug, G. (2009). Tone deafness: A new disconnection syndrome? The Journal of Neuroscience, 29, 10215–10220. Loui, P., Guenther, F., Mathys, C., & Schlaug, G. (2008). Action-perception mismatch in tone-deafness. Current Biology, 18(8), R331–R332. MacKay, D. G. (1987). The organization of perception and action. New York: Springer. Melara, R. D., & Marks, L. E. (1990a). HARD and SOFT interacting dimensions: Differential effects of dual context on classification. Perception & Psychophysics, 47, 307–325. Melara, R. D., & Marks, L. E. (1990b). Interaction among auditory dimensions: Timbre, pitch, and loudness. Perception & Psychophysics, 48, 169–178. Melara, R. D., & Marks, L. E. (1990c). Perceptual primacy of dimensions: Support for a model of dimensional interaction. Journal of Experimental Psychology. Human Perception and Performance, 16, 398–414. Milner, A. D., & Goodale, M. A. (1995). The visual brain in action. Oxford: Oxford University Press. Moore, R. E., Estis, J., Gordon-Hickey, S., & Watts, C. (2008). Pitch discrimination and pitch matching abilities with vocal and nonvocal stimuli. Journal of Voice, 22, 399–407. Moore, R. E., Keaton, C., & Watts, C. (2007). The role of pitch memory in pitch discrimination and pitch matching. Journal of Voice, 21, 560–567. Mürbe, D., Pabst, F., Hofmann, G., & Sundberg, J. (2002). Significance of auditory and kinesthetic feedback to singers’ pitch control. Journal of Voice, 16, 44–51. Natke, U., Donath, T. M., & Kalveram, K. T. (2003). Control of voice fundamental frequency in speaking versus singing. The Journal of the Acoustical Society of America, 113, 1587–1593.
Nikjeh, D. A., Lister, J. J., & Frisch, S. A. (2009). The relationship between pitch discrimination and vocal production: Comparison of vocal and instrumental musicians. The Journal of the Acoustical Society of America, 125, 328–338. Özdemir, E., Norton, A., & Schlaug, G. (2006). Shared and distinct neural correlates of singing and speaking. Neuroimage, 33, 628–635. Paus, T., Petrides, M., Evans, A. C., & Meyer, E. (1993). Role of the human anterior cingulate cortex in the control of oculomotor, manual, and speech responses: a positron emission tomography study. Journal of Neurophysiology, 70, 453–469. Paus, T., Perry, D. W., Zatorre, R. J., Worsley, K., & Evans, A. C. (1996). Modulation of cerebral blood-flow in the human auditory cortex during speech: role of motor-tosensory discharges. European Journal of Neuroscience, 8, 2236–2246. Peretz, I. (2008). Musical disorders: From behavior to genes. Current Directions in Psychological Science, 17, 329–333. Perry, D. W., Zatorre, R. J., Petrides, M., Alivisatos, B., Meyer, E., & Evans, A. C. (1999). Localization of cerebral activity during simple singing. Neuroreport, 10, 3979–3984. Pfordresher, P. Q., & Brown, S. (2007). Poor-pitch singing in the absence of “tone deafness” Music Perception, 25, 95–115. Pfordresher, P. Q., Brown, S., Meier, K. M., Belyk, M., & Liotti, M. (2010). Imprecise singing is widespread. Journal of the Acoustical Society of America, 128, 2182–2190. Pitt, M. A. (1994). Perception of pitch and timbre by musically trained and untrained listeners. Journal of Experimental Psychology. Human Perception and Performance, 20, 976–986. Rauschecker, J. P., & Scott, S. K. (2009). Maps and streams in the human auditory cortex: Nonhuman primates illuminate human speech processing. Nature Neuroscience, 12, 718–724. Ternström, S., Sundberg, J., & Colldén, A. (1988). Articulatory F0 perturbations and auditory feedback. Journal of Speech, Language, and Hearing Research, 31, 187–192. Warren, J. E., Wise, R. J. S., & Warren, J. D. (2005). Sounds do-able: Auditory-motor transformations and the posterior temporal plane. Trends in Neurosciences, 28, 636–643. Warrier, C. M., & Zatorre, R. J. (2002). Influence of tonal context and timbral variation on perception of pitch. Perception & Psychophysics, 64, 198–207. Watts, C., Barnes-Burroughs, K., Adrianopoulos, M., & Carr, M. (2003). Potential factors related to untrained singing talent: A survey of singing pedagogues. Journal of Voice, 17, 298–307. Watts, C. R., & Hall, M. D. (2008). Timbral influences on vocal pitch-matching accuracy. Logopedics, Phoniatrics, Vocology, 33, 74–82. Watts, C., Moore, R., & McCaghren, K. (2005). The relationship between vocal pitch matching skills and pitch
118 discrimination skills in untrained accurate and inaccurate singers. Journal of Voice, 19, 534–543. Wise, K. J., & Sloboda, J. A. (2008). Establishing an empirical profile of self-defined “tone deafness”: Perception, singing performance and self-assessment. Music Scientiae, 12, 3–26. Zarate, J. M., Delhommeau, K., Wood, S., & Zatorre, R. J. (2010). Vocal accuracy and neural plasticity following micromelody-discrimination training. PLoS ONE, 5, e11181.
Zarate, J. M., & Zatorre, R. J. (2008). Experience-dependent neural substrates involved in vocal pitch regulation during singing. Neuroimage, 40, 1871–1887. Zatorre, R. J., Chen, J. L., & Penhune, V. B. (2007). When the brain plays music: Auditory-motor interactions in music perception and production. Nature Reviews. Neurosience, 8, 547–558. Zwicker, E., & Fastl, H. (1999). Psychoacoustics: Facts and models. Berlin: Springer-Verlag.
A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 191 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.
CHAPTER 8
Lifelong plasticity in the rat auditory cortex: Basic mechanisms and role of sensory experience Etienne de Villers-Sidani{,* and Michael M. Merzenich{,} {
W.M. Keck Center for Integrative Neuroscience, Coleman Laboratory, Department of Otolaryngology, University of California, San Francisco, San Francisco, California, USA { Department of Neurology and Neurosurgery, Montreal Neurological Institute, McGill University, Montréal, Québec, Canada } Brain Plasticity Institute, San Francisco, California, USA
Abstract: The rodent auditory cortex has provided a particularly useful model for studying cortical plasticity phenomenology and mechanisms, both in infant and in adult animal models. Much of our initial understanding of the neurological processes underlying learning-induced changes in the cortex stems from the early exploitation of this model. More recent studies have provided a rich and elaborate demonstration of the “rules” governing representational plasticity induced during the critical period (CP) and in the longer post-CP “adult” plasticity epoch. These studies have also contributed importantly to the application of these “rules” to the development of practical training tools designed to improve the functional capacities of the auditory, language, and reading capacities of both children with developmental impairments and adults with acquired impairments in the auditory/aural speed and related cognitive domains. Using age as a connecting thread, we review recent studies performed in the rat primary auditory cortex (A1) that have provided further insight into the role of sensory experience in the shaping auditory signal representations, and into their possible role in shaping the machinery that regulates “adult” plasticity in A1. With this background, the role of auditory training in the remediation of auditory processing impairments is briefly discussed. Keywords: auditory; plasticity; critical period; aging; cortex; training; sensory.
Introduction The existence of thousands of phonologically different spoken languages on our planet is most certainly a consequence of the phenomenal plastic
*Corresponding author. Tel.: þ1 (415) 476 1362; Fax: þ1 (415) 502 4848 E-mail:
[email protected] DOI: 10.1016/B978-0-444-53752-2.00009-6
119
120
potential of the central auditory system (Goodman and Nusbaum, 1994). A genetic predisposition for language, thought to be unique to humans (Chomsky, 1965; Pinker, 1994), cannot alone explain the ease with which infants raised in different cultures can naturally acquire the language of their surroundings. This feat is believed to be achieved primarily through the native languagespecific experience-dependent remodeling of perceptual cortical representations during brain development (Doupe and Kuhl, 1999; Goodman and Nusbaum, 1994; Pinker, 1994). In humans, this developmental plasticity is not restricted to language and early musical training has been associated with the development of absolute pitch, the perception of harmonic sound and leads to an expansion of the primary auditory cortex (A1) (Pantev et al., 1998; Schlaug et al., 1995). Perceptual and neurological “specialization” is on an especially grand scale during early development; an infant's brain refines its processing machinery to selectively represent the sounds of its native language over the first several months of postnatal life (Chomsky, 1965; Goodman and Nusbaum, 1994; Pinker, 1994). Before 3 or 4 months of age, no clear language specificity is established and infants can discriminate phonetic contrasts of all languages. By the end of the first year of life, however, recurrent and stable phonemic inputs will be preferentially consolidated and infants will then fail to discriminate foreign language consonant contrasts. This phenomenon has been well described in 1-year-old Japanese and American infants: by this age, the Japanese infants have lost the ability to distinguish the American English /r/ and /l/ sounds; at the same time, the American infants have lost the ability to distinguish between the Spanish /b/ and / p/ sounds (Yamada and Tohkura, 1992). A crystallization of these phonemic representations is necessary for the subsequent elaboration of auditory semantic (word) representations, which use phonemes as building blocks. Powerful experience plasticity during early development infers important risks for normal child development, if early auditory inputs are distorted.
For example, it has been shown that a history of chronic otitis media, which prevents high-fidelity sound transmission at the level of the middle ear, can affect the rate and quality of acquisition of normal language abilities (Moore et al., 1991). In a more exaggerated example, infants raised with continuously closed Eustachian tubes attributable to a deep cleft palate have both middle ears chronically filled with fluid. All speech inputs are muffled and acoustically distorted in such a child, and it is not surprising that they develop degraded language abilities—and ultimately, cognitive and reading impairments. If their cleft palate is surgically repaired at a young developmental age, language, cognitive, and reading progressions are relatively normal. With their inherited defect in place, all language inputs received by these infants are grossly degraded; with surgical correction, high-fidelity speech reception is restored (Dorf and Curtin, 1982; Richman et al., 1988). The classical studies of Hubel and Wiesel on the developing cat visual cortex were the first to confirm the existence of time-limited epochs of stimulus exposure-induced plasticity in an animal model (Hubel and Wiesel, 1962). These windows of “unregulated” plasticity, also commonly referred to as “critical periods” (CPs), have now been described in all major sensory systems in a variety of animal species, including the rat auditory cortex, and their identification has been instrumental in the discovery of the cortical machinery involved in its regulation (see Hensch, 2005 for a review). But representational plasticity in sensory cortices is not limited to early development. Despite qualitatively and quantitatively different rules of regulation (Dorrn et al., 2010; Maffei and Turrigiano, 2008), profound functional and structural experience-dependent changes can still be achieved in the adult brain (Allard et al., 1991; Buonomano and Merzenich, 1998; Jenkins et al., 1990; Metherate and Weinberger, 1990; Recanzone et al., 1993; Xerri et al., 1998). Animal models have demonstrated that, of all the mechanisms involved in the regulation of representational plasticity, sensory experience, in the
121
form of passive sensory exposure (early in life) or reinforced associative learning (during adult life), is certainly one of the most powerful (Buonomano and Merzenich, 1998). In the developing cortex, changes in sensory input-driven neural activity trigger a cascade of events including the elaboration of neurotrophic factors (Huang et al., 1999), the stabilization of synapses (Berardi et al., 2004; Daw, 1995), and, finally, the elaboration of myelin (McGee et al., 2005) that signal the closure of CP windows. Similar rearrangements, albeit of different proportions or in different contexts (e.g., during attentive behavior only), have been observed in the adult cortex (Buonomano and Merzenich, 1998; de Villers-Sidani et al., 2010). A better understanding of how sensory experience shapes cortical circuits will undoubtedly have a broad impact on how we approach and treat a wide range of neurological or neurodevelopmental disorders. Intensive perceptual training based on the principles of neuroplasticity has already shown great promise in the remediation of age-related cognitive decline (Mahncke et al., 2006) and dyslexia (Merzenich et al., 1996; Tallal et al., 1998; Temple et al., 2003), two conditions tightly linked to a dysfunction of sensory processing. In this chapter, we shall focus on the functional aspects of cortical plasticity in the developing and aging A1. First, we will discuss the role of auditory input patterns in the regulation of the CP plasticity and the establishment of sound representations in A1, and then we review recent evidence showing how chronic distortions in sensory inputs and negative plasticity might contribute to the emergence of age-related cortical impairments. The use of an auditory training strategy for the identification and remediation of age-related cortical processing impairment in A1 is explored.
A succession of cortical sensitive periods of plasticity during early development Several thousand studies documenting many aspects of CP plasticity were conducted in the
primary visual and somatosensory cortices before auditory neuroscientists began the serious documentation of early postnatal plasticity in A1, initiating those auditory system experiments less than a decade ago. A1 studies quickly demonstrated great advantages for studying the “CP” of cortical development in this model system and very rapidly led to a significant reinterpretation of the developmental role of this “unregulated” early period of plasticity. As in the other major sensory systems, studies of CP plasticity were paralleled by and, in some cases, preceded by demonstrations that the representations of subcortical auditory nuclei were also easily plastically remodeled over the same developmental period. Most of those studies documented this subcortical plasticity within the central nucleus of the midbrain inferior colliculus, which, in contradistinction to the superior colliculus, is a key subcortical processing center on primary auditory projection pathway that ultimately feeds into A1. For example, it has been shown that exposing rat pups during the first weeks of life to pure tones or unidirectional frequency-modulated (FM) sweeps resulted in a significant increase in inferior collicular neurons most responsive to the sound used for exposure (Clopton and Winfield, 1976; Poon and Chen, 1992). Inversely, if mouse pups are exposed to broadband sounds such as clicks during an equivalent period, neurons of the inferior colliculus develop broad, poorly selective tuning curves (Sanes and Constantine-Paton, 1983). Elegant studies in the barn owl have also demonstrated that sensory experience during the first few weeks of life precisely directed the visual calibration of the auditory space in the optic tectum of barn owls (Knudsen, 1998). These studies showed that as in the visual and somatosensory systems, CP changes documented at the cortical level arise in part from feed-forward response remodeling, that is, expressed subcortically. Because subcortical nuclei are strongly influenced in their plastic organization, in turn, by centrifugal projections from the cortex (Suga and Ma, 2003), the relative
122
contributions of changes localized to the brainstem, midbrain, thalamus, or cortex, and a determination of the sources of primary plastic change are still very incompletely resolved in these developmental models. What we can say with certainty is that CP plasticity in the auditory system manifested at the cortical level is powerful and that it contributes richly and crucially to the early development of environmentally specialized acoustic signal representation in the young animal brain. Most studies of CP plasticity in the auditory cortex have focused on A1 studied in the rat. The domestic rat's cochlea matures at P10–P11 (Geal-Dor et al., 1993); A1 is almost immediately active, and as in the auditory brainstem nuclei, neurons within it over 1–2 days after hearing onset are frequency-selective (“tuned”), with
neurons with different “best frequencies” organized tonotopically (de Villers-Sidani et al., 2007) (Fig. 1). Despite this early organization, A1 frequency tuning is very plastic at the onset of hearing, but only for a few days in the presence of structured auditory inputs, and several-fold overrepresentations in A1 frequency tuning can be obtained with a mere passive exposure to sounds with a predominant spectral content (de Villers-Sidani et al., 2007). Such distortions are long lasting, strongly persisting more than 2 months after the exposure (the oldest postnatal ages studied), as would be expected for a typical CP in a sensory cortical field. Neurons in A1 initially respond sluggishly to modulated inputs; mature temporal dynamics emerge over a 2- to 4-week postnatal epoch (Chang et al., 2005). Neurons in the immature A1 also show (b) A1 characteristic frequency maps
Frequency tuning plasticity in the rat A1
(a)
Control
Hearing onset
Pulsed 7 kHz tones
28 14 7 3.5 1.75 kHz
Exposed P9
P10
P11
P12
P13
P14
P15
Time (post natal days)
D R
C
(c)
V 28 14 7 3.5 1.75 kHz
Controls P11
P12
P13
P14
Fig. 1. A critical period window for frequency tuning in A1 during development. (a) Time window for frequency tuning plasticity in A1 when measured using a passive exposure to pulsed tones. (b) Representative A1 characteristic frequency (CF) maps obtained in an unexposed control and in a rat pup exposed to pulsed 7 kHz tones during P10–P14. (c) Maturation of excitatory tone-evoked responses in rat A1 during the first few days after hearing onset and coinciding with the critical period for frequency tuning. Representative A1 CF maps obtained at different postnatal ages in naïve controls between P11 and P14 (de Villers-Sidani et al., 2007). Scale bar represents 0.75 mm. D, dorsal; C, caudal; R, rostral; V, ventral.
123
broad and nonselective “sideband” inhibition; adult-like profiles again evolve over the first 2–4 weeks of postnatal hearing life. Both of these distinctions reflect a slow maturation of inhibitory processes in A1. Put another way, the surprisingly sharp tuning recorded in A1 in the initial epoch of postnatal hearing does not appear to be accounted for, or substantially contributed to by cortical network inhibition, and plasticity in postexcitatory inhibitory processes and synaptic recovery processes appear to contribute to the improvements in response dynamics and emergent complex feature selectivity recorded in the temporally livelier “adult” cortex. A1 CP windows for tuning bandwidth and FM sweep direction selectivity coincide with the maturation of sideband inhibition. These were precisely mapped by exposing rat pups to unidirectional downward FM sweeps during four predetermined postnatal epochs (Insanally et al., 2009). In these experiments, exposure to a single complex stimulus had very different impacts on A1 sound representations depending on the period of exposure. Between P10 and P17, FM sweep exposure led to an overrepresentation of high frequencies in A1. Strong onset responses to the high-frequency edge of the downward FM stimuli are probably at the origin of this effect
(A1 neurons typically respond more strongly to the very beginning of a sound stimulus). Between P17 and P24, the same exposure resulted in a broadening of A1 neurons receptive fields (RFs), with no overall impact on frequency tuning or other parameter measured. Exposure between P25 and P40 primarily biased FM sweep direction selectivity in favor of downward sweeps but had no observable effect on frequency selectivity or tuning bandwidth. Importantly, A1 neurons’ sweep selectivity tuning was not affected in the groups exposed before P25, suggesting that a stable representation of basic sound parameters in A1 such as frequency tuning or response bandwidth was prerequisite for establishing FM sweep selectivity. Persistent alterations in the representations of several other sound parameters such as intensity tuning, amplitude modulation rate selectivity, or temporal order preference have been demonstrated with exposures to pulsed pure tones (de Villers-Sidani et al., 2007) at a fixed sound intensity, pulsed noises at a fixed rate of presentation (Zhou and Merzenich, 2008), or repeated sequences of tones during the same P10–P30 developmental period (Nakahara et al., 2004). Figure 2 summarizes the timeline of CP windows so far documented in the rat A1.
Frequency tuning Tuning bandwidth Temporal following limit Sound intensity Temporal order FM sweeps
Birth
10
20
30 Age (post natal days)
Hearing onset
Fig. 2. Succession of critical periods during rat A1 development. Passive exposure experiments using a variety of simple or complex stimuli have demonstrated the existence of different mostly overlapping windows of A1 plasticity for practically every sound parameter examined. By definition, the representational distortions resulting from CP exposures are long lasting and persist for a significant portion of the animal's life. The CP for more complex sound representations tends to occur later during development.
124
These results collectively demonstrate that A1 representations for most if not all sound parameters known to be coded by auditory neurons are highly dependent on the statistics of sounds present in the infant's sensory environment. They also indicate (see below) that the CP in any given cortical area may comprise a series of “sensitive periods” (see references), which apply for the achievement of reliable representations of that parameter. In turn, the “consolidation” of that representational parameter, now reliably fed-forward, makes possible the opening of the next CP locally or in a downstream cortical area dependent on the information it carries. Jean Piaget used the term “consolidation” to describe the achievement of stable levels of development in his important theory of cognitive development elaborated more than five decades ago (Piaget, 1950). He posited that the acquisition and consolidation of cognitive skills occur in a tightly orchestrated sequence in which more complex skills are elaborated following the progressive crystallization (“consolidation”) of the simpler ones on which they depend.
Local regulation of CP plasticity in A1 by sensory input patterns The studies described above provide a compelling case for the idea (as has been shown to be the case in V1; see Crair et al., 1998; Hensch, 2004) that the maturation of A1 is dependent upon stimulation with temporally structured acoustic inputs. The rapid transition from sound exposure-based plasticity to the epoch of “adult” plasticity is also marked by a dramatic change of the impacts of neuromodulators that have been shown to regulate plasticity in the older brain (Bao et al., 2001; Kilgard and Merzenich, 1998; Mercado et al., 2001; Metherate and Weinberger, 1990; Weinberger, 2003). Plasticity is readily achieved in older brains with short stimulus exposures, but only in the right behavioral-context (neuromodulator release) conditions. If the
older brain is not attending to stimuli, and/or if there is no systematic association of behavioral “rewards,” perceived goal achievement or novelty for even stimuli delivered continuously for 1–2 weeks periods in the “mature” brain, no changes are generated in A1 (Bao et al., 2003, 2004; Blake et al., 2006; Buonomano and Merzenich, 1998; Jenkins et al., 1990; Kilgard and Merzenich, 1998; Recanzone et al., 1993). That “attentional/ reward-system” maturation could be plausibly achieved by changes induced in the nuclei that deliver the modulatory neurotransmitters that enable plasticity as a function of these behavioral variables, they could be induced by changes that occur locally within A1, or they could be induced because A1 maturation enables feed-forward effects that result in feedback that now strictly controls when A1 plasticity can occur. We have conducted a simple CP plasticity study that eliminates the first of these possibilities. By raising rat pups in the presence of a continuous, moderate-level band-limited or notched white noise, we can block cortical maturation for a sector(s) of A1, while we keep it open for another sector(s) (de Villers-Sidani et al., 2008) (see Fig. 3). By that strategy, we have shown that we can close the CP in the A1 zone representing frequencies outside of the noiseband frequency ranges, while we keep it open for those sectors that are within this range. An examination of the physical maturation of the cortex within these zones—for example, as manifested by documenting the state of maturation of parvalbumin (PV) inhibitory interneurons— reveals a very sharp (minicolumn by minicolumn) boundary separating mature from immature A1 sectors. Notched-noise exposure also suspended the maturation of a number of other A1 functional properties in a sector-specific manner. In the noise-exposed sector, responses to amplitudemodulated stimuli remained sluggish and neural synchronization, or the likelihood of neighboring neurons firing simultaneously, were low as in the immature cortex. Decreased neural synchrony
125 P7
P20
Noise exposure A1 mapping
Band-limited noise
Naive control
D C
1.75 7 28 Frequency (kHz)
R V
Parvalbumin staining in A1
5–20 kHz Sound energy level (dB SPL)
1
40
Frequency (kHz) Caudal
Center
Rostral
Caudal
Center
Rostral
Fig. 3. Band-limited noise exposure during early development prevents the maturation of the noise-engaged A1 sector. Left, spectrogram of the band-limited noise stimulus used for the studies and its relationship to a typical A1 CF map. Right above, representative A1 characteristic-frequency (CF) map from a band-limited noise-exposed rat and naive control. Note the contraction of the area of A1 tuned to frequencies falling in the noise band (hatched polygons). The color of each polygon represents the frequency tuning of the neuron recorded in its center. Right below, photomicrographs of A1 sections stained for PV. PV positive interneurons play an important role in the regulation of cortical plasticity during development. Band-limited noise exposure prevented the maturation of these cells, only in the noise-exposed sector of A1. Scale bar represents 0.75 mm. D, dorsal; C, caudal; R, rostral; V, ventral.
when measured in the absence of sensory stimulation (in silence in this case) most likely implies immature or inefficient cortico-cortical projections, which normally develop during the same developmental period and which promote synchronous activity in the cortex (Durack and Katz, 1996). Similar experiment using broadband noise covering the whole rat hearing spectrum prevented the closure of the CP window for frequency tuning for the entire A1 field; it resumed normally more than 3 months later after discontinuation of the noise (Chang and Merzenich, 2003). This fine-grained control of A1 maturation by sensory experience supports the view that its regulation is orchestrated by maturational changes in the cortex itself. It also indicates that the end of the period during which mere exposure to environmental acoustic stimulation will drive largescale plastic changes in the cortex may be attributable to the achievement of patterned cortical activity (as opposed to chaotic neural firing
during noise exposure). What specific patterns might be necessary or sufficient to trigger such changes remains unknown. Neuromodulator nuclei, which have been shown to drive adult plasticity and could conceivably gate CP windows during development, have very diffuse cortical projections that probably cannot target precise sectors of A1 (Baskerville et al., 1993; Fournier et al., 2004; Sarter and Bruno, 1997), much less subcircuits specifically dedicated to the processing of specific sound parameters. Moreover, A1 is probably the brain structure the most apt at detecting the specific activity patterns required to trigger the various CP closures.
Patterned cortical activity maintains stimulus selectivity in the adult A1 Short passive sound exposures that can drive profound representational changes in the developing cortex have little or no measurable impact on the
126
post-CP A1 (Bao et al., 2003, 2004; Blake et al., 2006; Buonomano and Merzenich, 1998; Jenkins et al., 1990; Kilgard and Merzenich, 1998; Recanzone et al., 1993). However, prolonged, several weeks or months long, consistent changes in sensory input statistics will lead to substantial functional adjustments in the adult cortex (Allard et al., 1991). This has been shown first in A1 by studies that examined the impact of cochlear lesions on A1 frequency maps (Irvine and Rajan, 1997; Rajan, 2001; Rajan and Irvine, 1998). An experimentally induced lesion at the base of the cochlea where auditory hair cells are tuned to high frequencies results within a few months in a progressive retuning of the high-frequency area of A1 to lower frequencies, fed by the intact apical hair cells. The lesion causes a sudden loss of auditory inputs accompanied in the corresponding A1 sector by a downregulation of inhibitory circuits manifested by reduced sideband inhibition and altered responses to successive signals (Rajan, 2001). The absence of significant unmasking of neighboring inputs in the affected A1 sector suggests that the final A1 reorganization observed months later cannot be explained solely by an unmasking of already existing inputs and is therefore almost certainly due to a progressive plastic rewiring within the auditory system (Irvine and Rajan, 1997). A chronic reduction in signal-to-noise ratio (rather than an absence of inputs) appears to be the main drive for the induction of these lesion-induced A1 plastic rearrangements. A similar—but reversible—A1 retuning is observed in the adult cat cortex after a 5-month exposure to a dense random band-limited stimulus of moderate intensity (Norena et al., 2006). In that study, the stimulus used consisted of a dense stream of random tone pips chosen within a specific range of frequencies (a form of noise because of its inherently random nature). The exposure resulted in a massive reorganization of A1 tuning to frequencies outside of the stimulus spectrum. It thus appears that prolonged chaotic neural activity caused by the presence of “noisy” sensory inputs alone is sufficient
to induce plasticity and profoundly alter sensory representations in A1. Recent results for our laboratory also substantiate the claim that random “noisy” cortical activity degrades stimulus selectivity in A1. We exposed passively young adult rats to random broadband noise at a low conversation-level intensity (55 dB SPL) for 3 months and then mapped their A1 responses (see Fig. 4, unpublished results). This low level but prolonged noise exposure resulted among other things in a significant scatter of frequency representation along A1's tonotopic axis and in 30–40% broader (less selective) RFs compared to controls. The negative impact of consistently lowered signal-to-noise ratios on sensory processing in the brain is of direct relevance to cortical aging. In the aging A1 and V1, the most frequently observed functional cortical impairments are a decrease in RF selectivity and a dysregulation of inhibitory processes (Caspary et al., 2008; de Villers-Sidani et al., 2010; Hua et al., 2008; Schmolesky et al., 2000), which are directly linkable to the most frequent cognitive deficits observed in the human population (Gazzaley et al., 2005; Hasher et al., 1991; Salthouse, 1996). The striking similarity between age and cochlear lesion-induced brain changes have led to the speculation that age-related cortical impairments might largely represent slow plastic adjustments to progressive peripheral deafferentation (Caspary et al., 2008). The predominantly secondary nature of agerelated cortical impairments was confirmed by a recently published study from our laboratory showing their partial to complete reversibility using targeted operant training (de Villers-Sidani et al., 2010). In this study performed on 26- to 32-month-old Brown-Norway aging rats and young adult controls, we initially performed a detailed documentation of spectral and temporal processing impairments in the aged A1. In the aged A1, RFs were poorly tuned, 45% broader across in the entire A1 field and responses to successive stimuli presented at various rates were relatively noisy and imprecise, lacking the
127 Adult exposed to noise for 3 months
Adult control
(b) 28 7
A1 CF maps
1.75 Frequency (kHz)
2.5 1.5 0.5 BW10 (octaves)
A1 BW10 maps D C
R
* BW10 (octaves)
(a)
*
1.5 * * 1.0 0.5 0 4.6 8.5 15.7 29.0 Characteristic frequency (kHz) Adult Adult-noise exposed
V
Fig. 4. Impact of 8 weeks of moderate-level noise exposure on A1 frequency tuning. (a) Above, representative A1 characteristicfrequency (CF) map from a young adult rat exposed for 8 weeks to 55 dB SPL of white noise. Note the increased scatter in frequency representation in the noise-exposed cortex. (a) Below, A1 maps from the same animals showing RF selectivity expressed as tuning bandwidth 10 dB above threshold (BW10). Higher numbers imply a decrease in tuning selectivity. (b) Quantification of BW10 for the two groups and separated by tuning frequency. Note the significant decrease in frequency tuning selectivity in the noise-exposed group. Scale bar represents 0.75 mm. (Young control, n ¼ 3, neurons ¼ 55; young noise exposed, n ¼ 3, number of site pairs ¼ 52.) Values shown are mean SE. *P < 0.05: t-test.
rate-specific modulation seen in young A1 neurons. Poor sensory adaptation also resulted in a decreased contrast of rare stimuli presented in an oddball sequence, consistent with what has been observed in human psychophysical experiments. These functional deficits, which appear to be mostly contributed to by poor inhibition, were accompanied by a decrease in the density and the complexity of PV positive GABAergic interneurons in A1. PV positive cells have been shown to be important for both noise suppression and novel stimulus detection (Contreras and Palmer, 2003). A decreased density of cortical myelin, critical for the rapid and reliable transfer of information (Peters, 2002), was also found in the aging cortex in our study. As a second step, we trained a group of aged rats for 31-h sessions on an auditory oddball detection task specifically designed to improve novel stimulus resolution in A1. At the completion of training, when behavioral improvements were plateauing, aged rats’ frequency discrimination thresholds were close to young pretraining levels. A1 responses were then mapped in detail in all groups (see Fig. 5). Training resulted in a partial
to complete recovery of most of the age-related cortical deficits observed. A1 neurons in the aged group had recovered narrowly tuned RFs with quasi-normal tonopic organization, and the reliability of temporal coding and responses to novel stimuli had both significantly improved. Perhaps, the most surprising result of the study was the partial recovery of PVþ cell counts and myelin density in A1 after training. The fact that we saw, using a simple training strategy, a significant recovery of almost all functional and structural age-related A1 deficits raises the possibility that none of the effects of age on the brain are really truly degenerative. In fact, in many respects, the functional and structural state of the aged cortex is strikingly similar to the state of the immature or noise-exposed cortex (broad RFs, weak inhibition, low PV cells numbers, and low myelin density). Age-related “impairments” could possibly all represent slow plastic, and thus fundamentally reversible, adjustments to growing peripheral or internal noise. That noise returned the aged cortex to an “immature” or dedifferentiated state where it is waiting for coherent patterns to reestablish more refined
128 Young
(a)
Young trained
Aged
Aged trained 28
Characteristic frequency maps
1
7
4
3
2
1.75 Frequency (kHz)
2.5 1.5
BW10 maps
0.5 BW10 (octaves) D C
R V
(b)
(c)
Sound intensity (dB SPL)
1
2
3
4
70 50 30 10 1.8 3.5 7 14 28
1.8 3.5 7 14 28
1.8 3.5 7 14 28
1.8 3.5 7 14 28
Frequency (kHz) Frequency (kHz) Frequency (kHz) Frequency (kHz)
BW10 (ocytaves)
**
Young
**
**
*
**
** *
1.5
Young - T Aged Aged - T
** 1.0 0.5 0 4.6
8.5
15.7 29.0
Characteristic frequency (kHz)
Fig. 5. Age-related changes in A1 frequency representation and impact of training. (a) Top row, representative A1 characteristicfrequency (CF) maps from naïve young (6- to 12-months old), naïve aged (24- to 32-months old), and age-matched rats trained for 30 days on an auditory oddball discrimination task (de Villers-Sidani et al., 2010). (a) Bottom row, A1 maps from the same animals showing the A1 representation of tuning curve width (BW10). (b) Representative cortical receptive fields from the CF maps shown in (a). (c) Distribution of BW10 by CF in all experimental groups. Scale bar 0.75 mm; D, dorsal; C, caudal; R, rostral; V, ventral. Young: n ¼ 14, neurons ¼ 387; young trained: n ¼ 5, neurons ¼ 211; aged: n ¼ 12, neurons ¼ 291; age trained, n ¼ 5, neurons ¼ 201. Values shown are mean SEM. *P < 0.05, **P < 0.001: t-test.
sensory representation. Experiments are currently being conducted in our laboratory to examine this interesting possibility. Conclusion The infinitely diverse expressions of human cognition and behavior can only be the result of an equally large number of unique life experiences. While the discovery of DNA has revolutionized biological sciences by providing us the code to
decipher the molecular makeup of every single cell in our body, it is becoming abundantly clear that this molecular template is not sufficient to explain how cortical circuits precisely process information in any given individual. During early development, the patterns present in our sensory environment determine stimulus selectivity and regulate the rate of cortical maturation. Preferential tuning of cortical neurons to highly coherent, repeated stimuli—and less so to noisy stimuli— ensures the consolidation of a representational repertoire that mirrors our own sensory world.
129
As our brain matures, a tightly orchestrated sequence of sensitive periods leads to the emergence of increasingly complex cortical filters founded on progressively stabilized basic sensory representations. With age, profound adjustments of sensory representations are still possible even with mostly passive sensory exposure but at a slower pace and appear primarily driven by prolonged significant changes in the overall statistics of sensory inputs such as seen with sensory deprivation or noise exposure. The plastic nature of these adjustments makes them inherently reversible and, as seen in the aged A1, intensive perceptual training is a powerful noninvasive method to drive their correction, at both functional and structural levels. In this review, we demonstrated that sensory representations in A1 are for the most part the product of our lifelong experiences. Refining further our understanding of how sensory experience shapes functional brain circuits and how it can be harnessed for their repair will transform how we approach, prevent, and treat a wide variety of neurological disorders involving perceptual impairments. Acknowledgments Research embodied in this review is supported by grants from the National Institutes of Health Grants (Conte Grant 5P50MH077970-03) and the Canadian Institutes of Health Research (Clinician-Scientist Award Phase I). Abbreviations A1 BW10 CF CP dB SPL DNA
primary auditory cortex tuning bandwidth 10 decibels above threshold characteristic frequency critical period decibels sound pressure level deoxyribonucleic acid
FM GABA P PV RF
frequency modulated gamma-amino butyric acid postnatal day parvalbumin receptive field
References Allard, T., Clark, S. A., Jenkins, W. M., & Merzenich, M. M. (1991). Reorganization of somatosensory area 3b representations in adult owl monkeys after digital syndactyly. J Neurophysiol, 66, 1048–1058. Bao, S., Chan, V. T., & Merzenich, M. M. (2001). Cortical remodelling induced by activity of ventral tegmental dopamine neurons. Nature, 412, 79–83. Bao, S., Chang, E. F., Davis, J. D., Gobeske, K. T., & Merzenich, M. M. (2003). Progressive degradation and subsequent refinement of acoustic representations in the adult auditory cortex. J Neurosci, 23, 10765–10775. Bao, S., Chang, E. F., Woods, J., & Merzenich, M. M. (2004). Temporal plasticity in the primary auditory cortex induced by operant perceptual learning. Nat Neurosci, 7, 974–981. Baskerville, K. A., Chang, H. T., & Herron, P. (1993). Topography of cholinergic afferents from the nucleus basalis of Meynert to representational areas of sensorimotor cortices in the rat. J Comp Neurol, 335, 552–562. Berardi, N., Pizzorusso, T., & Maffei, L. (2004). Extracellular matrix and visual cortical plasticity: freeing the synapse. Neuron, 44, 905–908. Blake, D. T., Heiser, M. A., Caywood, M., & Merzenich, M. M. (2006). Experience-dependent adult cortical plasticity requires cognitive association between sensation and reward. Neuron, 52, 371–381. Buonomano, D. V., & Merzenich, M. M. (1998). Cortical plasticity: from synapses to maps. Annu Rev Neurosci, 21, 149–186. Caspary, D. M., Ling, L., Turner, J. G., & Hughes, L. F. (2008). Inhibitory neurotransmission, plasticity and aging in the mammalian central auditory system. J Exp Biol, 211, 1781–1791. Chang, E. F., Bao, S., Imaizumi, K., Schreiner, C. E., & Merzenich, M. M. (2005). Development of spectral and temporal response selectivity in the auditory cortex. Proc Natl Acad Sci USA, 102, 16460–16465. Chang, E. F., & Merzenich, M. M. (2003). Environmental noise retards auditory cortical development. Science, 300, 498–502. Chomsky, N. (1965). Aspects of the theory of syntax. Cambridge: M.I.T. Press.
130 Clopton, B. M., & Winfield, J. A. (1976). Effect of early exposure to patterned sound on unit activity in rat inferior colliculus. J Neurophysiol, 39, 1081–1089. Contreras, D., & Palmer, L. (2003). Response to contrast of electrophysiologically defined cell classes in primary visual cortex. J Neurosci, 23, 6936–6945. Crair, M. C., Gillespie, D. C., & Stryker, M. P. (1998). The role of visual experience in the development of columns in cat visual cortex. Science, 279, 566–570. Daw, N. (1995). Visual development. New York: Plenum Press. de Villers-Sidani, E., Alzghoul, L., Zhou, X., Simpson, K. L., Lin, R. C., & Merzenich, M. M. (2010). Recovery of functional and structural age-related changes in the rat primary auditory cortex with operant training. Proc Natl Acad Sci USA, 107, 13900–13905. de Villers-Sidani, E., Chang, E. F., Bao, S., & Merzenich, M. M. (2007). Critical period window for spectral tuning defined in the primary auditory cortex (A1) in the rat. J Neurosci, 27, 180–189. de Villers-Sidani, E., Simpson, K. L., Lu, Y. F., Lin, R. C., & Merzenich, M. M. (2008). Manipulating critical period closure across different sectors of the primary auditory cortex. Nat Neurosci, 11, 957–965. Dorf, D. S., & Curtin, J. W. (1982). Early cleft palate repair and speech outcome. Plast Reconstr Surg, 70, 74–81. Dorrn, A. L., Yuan, K., Barker, A. J., Schreiner, C. E., & Froemke, R. C. (2010). Developmental sensory experience balances cortical excitation and inhibition. Nature, 465, 932–936. Doupe, A. J., & Kuhl, P. K. (1999). Birdsong and human speech: common themes and mechanisms. Annu Rev Neurosci, 22, 567–631. Durack, J. C., & Katz, L. C. (1996). Development of horizontal projections in layer 2/3 of ferret visual cortex. Cereb Cortex, 6, 178–183. Fournier, G. N., Semba, K., & Rasmusson, D. D. (2004). Modality- and region-specific acetylcholine release in the rat neocortex. Neuroscience, 126, 257–262. Gazzaley, A., Cooney, J. W., Rissman, J., & D'Esposito, M. (2005). Top-down suppression deficit underlies working memory impairment in normal aging. Nat Neurosci, 8, 1298–1300. Geal-Dor, M., Freeman, S., Li, G., & Sohmer, H. (1993). Development of hearing in neonatal rats: air and bone conducted ABR thresholds. Hear Res, 69, 236–242. Goodman, J., & Nusbaum, H. C. (1994). The Development of speech perception : the transition from speech sounds to spoken words. Cambridge, Mass: MIT Press. Hasher, L., Stoltzfus, E. R., Zacks, R. T., & Rypma, B. (1991). Age and inhibition. J Exp Psychol Learn Mem Cogn, 17, 163–169. Hensch, T. K. (2004). Critical period regulation. Annu Rev Neurosci, 27, 549–579.
Hensch, T. K. (2005). Critical period plasticity in local cortical circuits. Nat Rev Neurosci, 6, 877–888. Hua, T., Kao, C., Sun, Q., Li, X., & Zhou, Y. (2008). Decreased proportion of GABA neurons accompanies age-related degradation of neuronal function in cat striate cortex. Brain Res Bull, 75, 119–125. Huang, Z. J., et al. (1999). BDNF regulates the maturation of inhibition and the critical period of plasticity in mouse visual cortex. Cell, 98, 739–755. Hubel, D. H., & Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. J Physiol, 160, 106–154. Insanally, M. N., Kover, H., Kim, H., & Bao, S. (2009). Feature-dependent sensitive periods in the development of complex sound representation. J Neurosci, 29, 5456–5462. Irvine, D. R., & Rajan, R. (1997). Injury-induced reorganization of frequency maps in adult auditory cortex: the role of unmasking of normally-inhibited inputs. Acta Otolaryngol Suppl, 532, 39–45. Jenkins, W. M., Merzenich, M. M., & Recanzone, G. (1990). Neocortical representational dynamics in adult primates: implications for neuropsychology. Neuropsychologia, 28, 573–584. Kilgard, M. P., & Merzenich, M. M. (1998). Cortical map reorganization enabled by nucleus basalis activity. Science, 279, 1714–1718. Knudsen, E. I. (1998). Capacity for plasticity in the adult owl auditory system expanded by juvenile experience. Science, 279, 1531–1533. Maffei, A., & Turrigiano, G. (2008). The age of plasticity: developmental regulation of synaptic plasticity in neocortical microcircuits. Prog Brain Res, 169, 211–223. Mahncke, H. W., et al. (2006). Memory enhancement in healthy older adults using a brain plasticity-based training program: a randomized, controlled study. Proc Natl Acad Sci USA, 103, 12523–12528. McGee, A. W., Yang, Y., Fischer, Q. S., Daw, N. W., & Strittmatter, S. M. (2005). Experience-driven plasticity of visual cortex limited by myelin and Nogo receptor. Science, 309, 2222–2226. Mercado, E., Bao, S., Orduna, I., Gluck, M. A., & Merzenich, M. M. (2001). Basal forebrain stimulation changes cortical sensitivities to complex sound. Neuroreport, 12, 2283–2287. Merzenich, M. M., Jenkins, W. M., Johnston, P., Schreiner, C., Miller, S. L., & Tallal, P. (1996). Temporal processing deficits of language-learning impaired children ameliorated by training. Science, 271, 77–81. Metherate, R., & Weinberger, N. M. (1990). Cholinergic modulation of responses to single tones produces tone-specific receptive field alterations in cat auditory cortex. Synapse, 6, 133–145.
131 Moore, D. R., Hutchings, M. E., & Meyer, S. E. (1991). Binaural masking level differences in children with a history of otitis media. Audiology, 30, 91–101. Nakahara, H., Zhang, L. I., & Merzenich, M. M. (2004). Specialization of primary auditory cortex processing by sound exposure in the "critical period". Proc Natl Acad Sci USA, 101, 7170–7174. Norena, A. J., Gourevitch, B., Aizawa, N., & Eggermont, J. J. (2006). Spectrally enhanced acoustic environment disrupts frequency representation in cat auditory cortex. Nat Neurosci, 9, 932–939. Pantev, C., Oostenveld, R., Engelien, A., Ross, B., Roberts, L. E., & Hoke, M. (1998). Increased auditory cortical representation in musicians. Nature, 392, 811–814. Peters, A. (2002). The effects of normal aging on myelin and nerve fibers: a review. J Neurocytol, 31, 581–593. Piaget, J. (1950). The Psychology of intelligence. New York: Harcourt, Brace. Pinker, S. (1994). The language instinct. W. Morrow and Co.: New York. Poon, P. W., & Chen, X. (1992). Postnatal exposure to tones alters the tuning characteristics of inferior collicular neurons in the rat. Brain Res, 585, 391–394. Rajan, R. (2001). Plasticity of excitation and inhibition in the receptive field of primary auditory cortical neurons after limited receptor organ damage. Cereb Cortex, 11, 171–182. Rajan, R., & Irvine, D. R. (1998). Neuronal responses across cortical field A1 in plasticity induced by peripheral auditory organ damage. Audiol Neurootol, 3, 123–144. Recanzone, G. H., Schreiner, C. E., & Merzenich, M. M. (1993). Plasticity in the frequency representation of primary auditory cortex following discrimination training in adult owl monkeys. J Neurosci, 13, 87–103. Richman, L. C., Eliason, M. J., & Lindgren, S. D. (1988). Reading disability in children with clefts. Cleft Palate J, 25, 21–25. Salthouse, T. A. (1996). The processing-speed theory of adult age differences in cognition. Psychol Rev, 103, 403–428.
Sanes, D. H., & Constantine-Paton, M. (1983). Altered activity patterns during development reduce neural tuning. Science, 221, 1183–1185. Sarter, M., & Bruno, J. P. (1997). Cognitive functions of cortical acetylcholine: toward a unifying hypothesis. Brain Res Brain Res Rev, 23, 28–46. Schlaug, G., Jancke, L., Huang, Y., & Steinmetz, H. (1995). In vivo evidence of structural brain asymmetry in musicians. Science, 267, 699–701. Schmolesky, M. T., Wang, Y., Pu, M., & Leventhal, A. G. (2000). Degradation of stimulus selectivity of visual cortical cells in senescent rhesus monkeys. Nat Neurosci, 3, 384–390. Suga, N., & Ma, X. (2003). Multiparametric corticofugal modulation and plasticity in the auditory system. Nat Rev Neurosci, 4, 783–794. Tallal, P., Merzenich, M., Miller, S., & Jenkins, W. (1998). Language learning impairment: integrating research and remediation. Scand J Psychol, 39, 197–199. Temple, E., et al. (2003). Neural deficits in children with dyslexia ameliorated by behavioral remediation: evidence from functional MRI. Proc Natl Acad Sci USA, 100, 2860–2865. Weinberger, N. M. (2003). The nucleus basalis and memory codes: auditory cortical plasticity and the induction of specific, associative behavioral memory. Neurobiol Learn Mem, 80, 268–284. Xerri, C., Merzenich, M. M., Peterson, B. E., & Jenkins, W. (1998). Plasticity of primary somatosensory cortex paralleling sensorimotor skill recovery from stroke in adult monkeys. J Neurophysiol, 79, 2119–2148. Yamada, R. A., & Tohkura, Y. (1992). The effects of experimental variables on the perception of American English /r/ and /l/ by Japanese listeners. Percept Psychophys, 52, 376–392. Zhou, X., & Merzenich, M. M. (2008). Enduring effects of early structured noise exposure on temporal modulation in the primary auditory cortex. Proc Natl Acad Sci USA, 105, 4423–4428.
A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 191 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.
CHAPTER 9
Enhancing visual cues to orientation: Suggestions for space travelers and the elderly Laurence R. Harris*, Michael Jenkin, Richard T. Dydew and Heather Jenkin Centre for Vision Research, York University, Toronto, Ontario, Canada
Abstract: Establishing our orientation in the world is necessary for almost all aspects of perception and behavior. Gravity usually defines the critical reference direction. The direction of gravity is sensed by somatosensory detectors indicating pressure points and specialized organs in the vestibular system and viscera that indicate gravity's physical pull. However, gravity's direction can also be sensed visually since we see the effects of gravity on static and moving objects and also deduce its direction from the global structure of a scene indicated by features such as the sky and ground. When cues from either visual or physical sources are compromised or ambiguous, perceptual disorientation may result, often with a tendency to replace gravity with the body's long axis as a reference. Orientation cues are compromised while floating in the weightlessness of space (which neutralizes vestibular and somatosensory cues) or while suspended at neutral buoyancy in the ocean (which neutralizes somatosensory cues) and the ability to sense orientation cues may also be compromised in the elderly or in clinical populations. In these situations, enhancing the visual cues to orientation may be beneficial. In this chapter, we review research using specially constructed virtual and real environments to quantify the contribution of various visual orientation cues. We demonstrate how visual cues can counteract disorientation by providing effective orientation information. Keywords: microgravity; levitation illusion; field of view; visual gravity; perceived direction of gravity; falls; floor; support surface; balance; cue weighting.
Introduction *Corresponding author. Tel.: þ1 416-736-2100x66108; Fax: þ1 416-736-5814 E-mail:
[email protected]
Vision is important to most people's conscious perception of the world, but vision also has an important proprioceptive function (Nakayama, 1985) in which it provides information about the orientation and movement of the body. Nonvisual cues to
w
Deceased.
DOI: 10.1016/B978-0-444-53752-2.00008-4
133
134
orientation are provided by systems that report the direction of the physical force of gravity. These include the otoliths of the vestibular system of the inner ear (Mittelstaedt, 1991, 1999), specialized detectors in the viscera (Mittelstaedt, 1992), and the somatosensory system (Lechner-Steinleitner et al., 1979; Yardley, 1990) that reports the location of pressure from points of contact with the support surface. These sources of information normally work together with visual proprioception to provide robust information about a person's orientation in their environment. Here, we concentrate on visual proprioceptive cues to orientation and how they can potentially be strengthened to compensate for the loss of other cues.
Visual cues to orientation Figure 1 illustrates some of the visual cues that specify orientation in a typical scene. These cues include: (i) The visual frame (indicated by a rectangle in the figure), comprising the ground plane and features known to be approximately earth vertical or earth horizontal such as trees, walls, ceilings, and floors. The frame cue is inherently ambiguous.
Each of the directions indicated by the four arrows could potentially be the direction of “up.” (ii) The visual horizon. This cue indicates two possible directions (opposite to each other). (iii) The assumption that light comes from above (Mamassian and Goutcher, 2001). Although this is generally true, it cannot be precise as light can of course come from many directions (Morgenstern et al., submitted for publication). (iv) Support relationships between objects, determined by the laws of physics, such as objects resting on other objects or supported by the ground. Again, this clue is generally true, but even unattached objects can rest on surfaces that are quite tilted relative to gravity, depending on friction to stay in place. (v) The visual polarity cues of objects with a recognizable top and bottom such as people, lamps, and chairs that have a “most-familiar” orientation relative to gravity (Cian et al., 2001). But these objects can be present in an unfamiliar orientation, such as a when a chair is lying down. (vi) Movement such as objects moving on the ground plane or falling through the air. The brain can use visual movement to build an internal representation of gravity (McIntyre et al., 2001; Indovina et al., 2005). In fact, of all these cues, only the direction in which Light from above
Frame
Horizon
Objects supported by other objects Fig. 1. A typical scene showing examples of the various visual cues to the direction of gravity. On the right, the possible directions reported by the cues in the photograph are detailed. The four possible directions of gravity signaled by the frame, defined by the orientation of the walls, floor, ceiling, and window stripped of their other cues, are indicated by arrows. The two directions indicated by the horizon are shown in black, and the unique direction signaled by the relationships of objects in the world (e.g., the fruit bowl on the stool), the expected orientation of objects (fruit bowl curved side down), and light coming from above is indicated by the pale vertical arrow.
135
something falls demonstrates the direction of gravity unambiguously and even then the path of lighter falling objects, such as rain or leaves, may be diverted by wind or air resistance. The use of vision therefore requires a best guess based on all available cues. In order to make recommendations for enhancing vision's effectiveness for orientation, we need to quantify the influence of each. How can perception be influenced by visual cues to orientation? The power of visual cues in specifying orientation can be demonstrated by separating the directions indicated by vision and other cues. Under some circumstances, vision seems to dominate completely as it does in the “levitation illusion” (Howard and Hu, 2001, Howard et al., 1997), for example. In the levitation illusion subjects are pitched onto their backs while the entire visual environment is moved with them.1 If they are pitched slowly and smoothly enough so that they are unaware of their change in orientation with respect to gravity, they continue to feel upright and the direction of the floor, indicated entirely visually, is perceived as remaining orthogonal to gravity: vision dominates their overall perceptual experience. However, this illusion speaks more to how vestibular mechanisms for detecting gravity can be fooled rather than the normal importance of vision. Careful measurement of the quantitative contribution of visual cues under cue conflict conditions has ascertained that visual cues do not usually dominate but rather contribute different amounts of information to different proprioceptive functions. Tasks can be broadly divided into those that are perceptually based and those that involve physically interacting with the world.
1
This is achieved using a tumbling room (Allison et al., 1999) in which an observer is firmly held so that their relationship with the room does not change during the maneuver. It is called the “levitation illusion” because unrestrained objects within the room appear to levitate as the room pitches.
The influence of visual orientation cues on perceptual tasks The influence of the visual cues to orientation contained in a scene on perceptual tasks, such as recognizing faces or objects, has been measured by assessing the influence of visual cues on the perceptual upright. The perceptual upright is defined as the orientation at which objects appear the right way up (Jolicoeur, 1985; Maki, 1986; McMullen and Jolicoeur, 1992). The direction of the perceptual upright can be assessed using the Oriented Character Recognition Test (OCHART; Dyde et al., 2006). OCHART presents an observer with a character with an identity that depends on its orientation. For example, deciding whether a character is the letter “p” or the letter “d” depends on an independent estimate of which way is up. By assessing the orientations at which this character is most ambiguous, the perceptual upright (where it is least ambiguous) can be inferred. The different cues to up (vision, the body, and gravity) can then be put in conflict to assess their relative effect on the perceptual upright. For example, lying on one's side separates gravity from the long axis of the body. The orientation of visual cues can then be manipulated independently by having subjects look at an image viewed through a shroud to remove other visual cues to orientation. Such tests have shown that vision normally contributes about 25% of the information needed to determine the perceptual upright compared to about 25% from gravity, the remainder coming from the orientation of the long axis of the observer's own body (Dyde et al., 2006). This means that if a compelling visual environment is tilted by 90 relative to an earthvertical observer, the perceptual upright is tilted about 18 away from gravity.2
2
To a first approximation, the cues appear to combine by a simple vector summation. In this example, gravity and the body form one vector with vision at 90 to this yielding angle of tan 1 (25/75) ¼ 18 . See Mittelstaedt (1991) for a model that includes additional terms.
136
The influence of visual orientation cues on estimating the direction of gravity
their selection heavily toward larger surfaces leading to a larger surface sometimes being chosen as the preferred support surface even if a smaller surface is actually closer to earth horizontal (Harris et al., 2010). This phenomenon is summarized in Fig. 2. Thus, in order to provide optimal cues to correctly perceiving the floor surface, it is important to make sure that one potential floor surface is clearly larger than the other possible choices.
The influence of vision on direct estimates of the direction of gravity can be measured using the subjective visual vertical test: aligning a line to the perceived direction of gravity (Asch and Witkin, 1948; Mittelstaedt, 1986). This probe measures the direct perception of the direction of gravity rather than the consequences of gravity on more perceptual tasks. The influence of vision on this task is considerably less than it is on a perceptual task. Here, vision only contributes about 8% of the information compared to 77% from gravity and 16% from the body (Dyde et al., 2006). This means that if the environment is tilted by 90 relative to an upright observer, the subjective visual vertical is tilted by only about 5 . What does this mean in practice?
When is it desirable to use vision to influence perceived orientation? If gravity is not available, or its direction is unreliably perceived, vision becomes a more important cue. SCUBA divers often regard tilted surfaces that they see underwater as being earth horizontal (Ross et al., 1969) and astronauts can suffer sudden unsettling feelings of inversion if they see a crew member upside down relative to them (Oman, 2007). However, in such circumstances, the dominant determinant of the perceptual upright is usually the orientation of the body.
The influence of visual orientation cues on estimating which surface is the floor Imagine you are entering a strange environment. On which surface should you place your feet? It actually takes surprisingly little visual information to influence the choice of which surface is chosen as the floor. Although the direction of gravity restricts the choice, surprisingly the surface closest to orthogonal to gravity is not the surface always chosen as the floor. Rather, subjects bias 1:1
50%
The use of vision for orientation when gravity is not present Humans have been experiencing the weightless environment of space since 1961. In microgravity, 2:1
3:1
75%
60%
50% 40%
25%
Fig. 2. The effect of changing room aspect ratio on determining the choice of which surface to choose as the support surface. Each of these rooms is tilted at 45 and either of the lower surfaces is thus equally valid as the choice of support surface. The percentages by each surface show the percentage of times each surface was chosen. As the aspect ratio changes, the larger surface is increasingly likely to be chosen, based on visual factors alone. Data from Harris et al. (2010).
137
the perceived direction of “up” can no longer depend on gravity and thus must be defined by a combination of cues from vision and the body. Paradoxically however, when subjects are exposed to brief periods of microgravity (created by using parabolic flight) the weighting assigned to vision is significantly reduced3 (Dyde et al., 2009). In fact, many visual effects seem to be reduced under microgravity: the rod-and-frame effect4 (Villard et al., 2005), the horizontal/vertical illusion5 and Ponzo Illusion6 (Clement et al., 2007), and the influence of a tilted background on interpreting shape from the pattern of shading over an object's surface (Jenkin et al., 2004) are altered suggesting a reduced influence of visual cues to orientation under microgravity. Exposure to microgravity can evoke crippling feelings of disorientation (see Oman, 2007 for a review) and a long-duration interplanetary spaceflight without some kind of artificial gravity is therefore highly undesirable. Attempts to provide artificial gravity to counteract this disorientation have concentrated on regular sessions in a shortarm centrifuge (Young et al., 2001). However, an additional form of “artificial gravity” could be provided by the careful construction of a visual environment to provide consistent “visual gravity” cues. The visual cues provided by a typical spacecraft environment are not in themselves
3 The weighting assigned to vision was ascertained by measuring the perceptual upright using an OCHART probe superimposed on a highly polarized visual background presented in different orientations. The influence of the background on the perception of the probe could thus be measured (Dyde et al., 2009). 4 The rod-and-frame effect is where the orientation of a rod is influenced by the orientation of a surrounding frame. 5 The horizontal/vertical illusion is where a horizontal line of the same length as a vertical one appears longer (Prinzmetal and Gettleman, 1993). 6 The Ponzo illusion is an optical illusion that was first demonstrated by the Italian psychologist Mario Ponzo (1882–1960) in 1913. The upper of two horizontal lines of identical length drawn above each other on converging lines (like railway lines) appears longer.
particularly effective at providing an artificial gravity cue because they do not provide a consistent “gravity” direction: no one surface has visual cues to distinguish it from any other as all surfaces are used for mounting equipment and no one surface is larger than the others (see section “The influence of visual orientation cues on estimating which surface is the floor”). The use of vision for orientation by the elderly A third of people over 65 years of age experience one or more falls every year (Fuller, 2000). The tendency to fall seems to be related to deterioration in the peripheral or central vestibular systems (Matheson et al., 1999) which in turn may lead to postural instability (Campbell et al., 1995). Some aspects of vision, including visual acuity, contrast sensitivity, depth perception, and size of visual field, do not seem to be significantly correlated with the tendency to fall in the elderly (Lamoureux et al., 2010). However, selectively providing active older people with glasses does significantly reduce their tendency to fall (Haran et al., 2010). This implies that some visual cues, especially the higher spatial frequencies (Dyde et al., 2005), are important for the elderly. Providing enhanced visual information may therefore be beneficial in overcoming postural instability in this group also. How can effective visual cues be provided to this population? How can the contribution of vision be enhanced? An obvious way to enhance visual cues to orientation is to provide additional information about the direction that we wish to be perceived as “up.” For example, arrows or the words “this way up” might be helpful. However, such signs place additional cognitive demands on people that might already be cognitively loaded, such as astronauts or the cognitively impaired, as may be the case for some elderly or clinical populations. Therefore, here we concentrate on enhancing the existing natural cues to
138
orientation, the processing of which does not impose additional cognitive loads. Our central question is: which visual factors are important to enable someone to interact comfortably with their environment and to help them to feel correctly oriented? Enhancing visual orientation with polarized cues In order to assess the contribution of various visual cues that contribute to specifying the orientation of a room, we used virtual reality to simulate a room in which we could present just the room (walls only), furniture (polarized cues only), or a fully furnished room (both cues) (Harris et al., 2007). The presence of furniture enhanced the ability of the room to determine the perceptual upright. This is summarized in Fig. 3. Polarized cues help most. Can these polarized cues be further enhanced? Enhancing visual orientation cues by motion Most natural scenes contain movement and most moving things demonstrate the principles and more relevantly the direction of gravity. We compared the effectiveness of static and dynamic scenes in influencing the perceptual upright (Jenkin et al., 2011). The addition of motion cues significantly increased the influence of the visual background in determining the perceptual upright. The results are summarized in Fig. 4. Polarized objects that move in a way that demonstrates a gravity field can be an effective aid to perceiving orientation. Enhancing visual cues by compression It might be assumed that being able to see further in all directions (i.e., as a result of increasing the field of view) would be advantageous in providing better visually defined orientation cues as more polarized objects might be in view. By the same token, having a restricted field of view (brought on by a clinical condition or by wearing a helmet, for example) might be disadvantageous. We assessed the effect
of field of view on the power of vision to override gravity cues using the levitation illusion described above. The levitation illusion (Howard and Hu, 2001, Howard et al., 1997) depends, of course, on being able to see the visual environment to suppress any feeling of tilting over backward. How wide a field of view is necessary for visual cues to continue to override gravity cues? Experiments with fieldrestricting glasses showed that the incidence of the levitation illusion did not depend on the size of the visual field of view per se, but rather depended upon what was visible within the field (Jenkin et al., 2007). If what was normally seen in a larger field was made smaller so it could still be seen in a smaller field (Fig. 5), the illusion remained as strong as ever. Wearing an optical device that had the effect of shrinking the visible world so that more of it could be seen at once should thus have an enhancing effect.7 The visible world also needs to provide strong and effective polarizing cues rather than just being large. Conclusions Visual cues to orientation are very important and can be manipulated to make them more important still. To enhance a visual environment so that it provides stronger-than-normal cues to orientation we suggest providing strongly polarized cues that move around in a way that is consistent with gravity in a room with a respectable aspect ratio and viewing it through minifying lenses that expand what can be seen with a given field of view. Environments designed for people with a tendency to lose their balance should avoid tall narrow rooms. Such visually enhanced environments could be helpfully implemented in spacecraft design, in in-helmet visual displays for use during extravehicular space walks or during diving, and in living environments for people with impaired orientation perception.
7 One would need to adapt, however, to the changes in perceived distance that such lenses would introduce.
139
Furniture only
Full room
Empty room
Perceptual upright (deg)
30 Empty room Furniture filled Furniture alone Wireframe
20 10 0 −10 −20 −30 −90
0
180 270 90 Orientation of room (deg)
360
450
Wire frame
Fig. 3. The relative effectiveness of simulated rooms (left) experienced in virtual reality on the orientation of the perceptual upright. The perceptual upright is plotted in the right hand panel as a function of the orientation of the room where 0 is upright with respect to gravity. The perceptual upright is tilted by 20 by the furniture, whether or not it is in a room at all! An empty room has less effect. A wire frame room has a small effect which repeats every 90 that it is rotated because of the inherent ambiguity of this component of a room. Lines fitted through the data are the output of a simple vector sum model including the directions indicated by vision, gravity, and the long axis of the body (see Dyde et al., 2006). Data are from Harris et al. (2007).
A caveat: Enhancing vision might not be all good If visual cues misalign with gravity they tend to pull the perceptual upright (Dyde et al., 2006) and even egocentric judgments about the orientation of the head and body (Barnett-Cowan and Harris, 2008) away from veridical to some intermediate direction that may not actually correspond with the direction of any of the individual cues. Placing too much emphasis on vision when the information they carry is not aligned with
other cues can therefore be counterproductive. Misalignment arises in many natural situations such as when getting up from a chair or even reading a newspaper. Patients with Parkinson's disease do tend to put such an overemphasis on vision (Barnett-Cowan et al., 2010) which might contribute to their tendency to fall under such circumstances. When enhancing a visual environment, it is important to ensure that the cues align with the desired direction of the perceptual upright and with gravity.
140 (a)
P
P
Static
Size of visual influence
30°
Moving
60°
(b)
0° Fig. 4. The effect of adding movement to a visual scene. Backgrounds were presented tilted to the left or right (a) and could be either movies of people walking or stills from the same movie. Also shown is the OCHART probe used for measuring the orientation of the perceptual upright. The moving images had a significantly larger influence on the perceptual upright (b). Data taken from Jenkin et al. (2011).
Acknowledgments The microgravity flights referred to in this chapter were made possible by the generous support of the Canadian Space Agency. M. J. and L. R. H. hold NSERC Discovery Grants. The use of the Falcon-20 aircraft for microgravity flights in Canada was provided by the NRC/CNRC Microgravity Facility in Ottawa, Canada. We thank Carolee Orme for her helpful comments on this chapter. References Allison, R. S., Howard, I. P., & Zacher, J. E. (1999). Effect of field size, head motion, and rotational velocity on roll vection and illusory self-tilt in a tumbling room. Perception, 28, 299–306.
Asch, S. E., & Witkin, H. A. (1948). Studies in space perception. II. Perception of the upright with displaced visual fields and with body tilted. Journal of Experimental Psychology, 38, 455–477. Barnett-Cowan, M., Dyde, R. T., Fox, S. H., Moro, E., Hutchison, W. D., & Harris, L. R. (2010). Multisensory determinants of orientation perception in Parkinson's disease. Neuroscience, 167, 1138–1150. Barnett-Cowan, M., & Harris, L. R. (2008). Perceived selforientation in allocentric and egocentric space: Effects of visual and physical tilt on saccadic and tactile measures. Brain Research, 1242, 231–243. Campbell, A. J., Robertson, M. C., & Gardner, M. M. (1995). Elderly people who fall: Identifying and managing the causes. British Journal of Hospital Medicine, 54, 520–523. Cian, C., Raphel, C., & Barraud, P. A. (2001). The role of cognitive factors in the rod-and-frame effect. Perception, 30, 1427–1438.
141 (a)
(b)
(c)
(d)
27%
68%
75%
Fig. 5. What is seen is more important than the field of view. When on one's back in a tilted room (a) one may experience the perception of being upright in an upright room (the Levitation Illusion). Under normal viewing conditions (b) the illusion is experienced 75% of the time. When the field is restricted (c) the effectiveness of the illusion is reduced. But if more of the room is visible within the same restricted field of view, the illusion returns. Data from Jenkin et al. (2007). Clement, G., Arnesen, T. N., Olsen, M. H., & Sylvestre, B. (2007). Perception of longitudinal body axis in microgravity during parabolic flight. Neuroscience Letters, 413, 150–153. Dyde, R. T., Jenkin, M. R., & Harris, L. R. (2005). Cues that determine the perceptual upright: Visual influences are dominated by high spatial frequencies. Journal of Vision, 5, 193a. Dyde, R. T., Jenkin, M. R., & Harris, L. R. (2006). The subjective visual vertical and the perceptual upright. Experimental Brain Research, 173, 612–622. Dyde, R. T., Jenkin, M. R., Jenkin, H. L., Zacher, J. E., & Harris, L. R. (2009). The effect of altered gravity states on the perception of orientation. Experimental Brain Research, 194, 647–660. Fuller, G. F. (2000). Falls in the elderly. American Family Physician, 61(2159–2168), 2173–2174. Haran, M. J., Cameron, I. D., Ivers, R. Q., Simpson, J. M., Lee, B. B., Tanzer, M., et al. (2010). Effect on falls of providing single lens distance vision glasses to multifocal glasses wearers: VISIBLE randomised controlled trial. British Medical Journal, 340, c2265. Harris, L. R., Dyde, R. T., & Jenkin, M. (2007). The relative contributions of the visual components of a natural scene in defining the perceptual upright. Journal of Vision, 7, 303. Harris, L. R., Jenkin, M., Jenkin, H., Dyde, R. T., & Oman, C. M. (2010). Where's the floor? Seeing and Perceiving, 23, 81–88.
Howard, I. P., & Hu, G. (2001). Visually induced reorientation illusions. Perception, 30, 583–600. Howard, I. P., Groen, E., & Jenkin, H. L. (1997). Visually induced self inversion and levitation. Investigative Ophthalmology and Visual Science, 38, S80. Indovina, I., Maffei, V., Bosco, G., Zago, M., Macaluso, E., & Lacquaniti, F. (2005). Representation of visual gravitational motion in the human vestibular cortex. Science, 308, 416–419. Jenkin, H. L., Jenkin, M. R., Dyde, R. T., & Harris, L. R. (2004). Shape-from-shading depends on visual, gravitational, and body-orientation cues. Perception, 33, 1453–1461. Jenkin, H. L., Zacher, J. E., Jenkin, M. R., Oman, C. M., & Harris, L. R. (2007). Effect of field of view on the levitation illusion. Journal of Vestibular Research, 17, 271–277. Jenkin, M., Dyde, R. T., Jenkin, H. L., Zacher, J. E., & Harris, L. R. (2011). Perceptual upright: The relative effectiveness of dynamic and static images under different gravity fields. Seeing and Perceiving, 24, 53–64. Jolicoeur, P. (1985). The time to name disoriented natural objects. Memory Cognition, 13, 289–303. Lamoureux, E., Gadgil, S., Pesudovs, K., Keeffe, J., Fenwick, E., Dirani, M., et al. (2010). The relationship between visual function, duration and main causes of vision loss and falls in older people with low vision. Graefe's Archive for Clinical and Experimental Ophthalmology, 248, 527–533.
142 Lechner-Steinleitner, S., Schone, H., & Wade, N. J. (1979). Perception of the visual vertical: Utricular and somatosensory contributions. Psychological Research, 40, 407–414. Maki, R. H. (1986). Naming and locating the tops of rotated pictures. Canadian Journal of Psychology, 40, 368–387. Mamassian, P., & Goutcher, R. (2001). Prior knowledge on the illumination position. Cognition, 81, B1–B9. Matheson, A. J., Darlington, C. L., & Smith, P. F. (1999). Further evidence for age-related deficits in human postural function. Journal of Vestibular Research, 9, 261–264. McIntyre, J., Zago, M., Berthoz, A., & Lacquaniti, F. (2001). Does the brain model Newton's laws? Nature Neuroscience, 4, 693–694. McMullen, P. A., & Jolicoeur, P. (1992). Reference frame and effects of orientation of finding the tops of rotated objects. Journal of Experimental Psychology: Human Perception and Performance, 3, 807–820. Mittelstaedt, H. (1986). The subjective vertical as a function of visual and extraretinal cues. Acta Psychologica, 63, 63–85. Mittelstaedt, H. (1991). The role of the otoliths in the perception of the orientation of self and world to the vertical. Yearbooks Zoological Department of Zoology and Physiology of Animals, 95, 419–425. Mittelstaedt, H. (1992). Somatic versus vestibular gravity reception in man. Annals of the New York Academy of Sciences, 656, 124–139.
Mittelstaedt, H. (1999). The role of the otoliths in perception of the vertical and in path integration. Annals of the New York Academy of Sciences, 871, 334–344. Morgenstern, Y., Murray, R. F., & Harris, L. R. (submitted for publication). The light-from-above prior is weak. Nakayama, K. (1985). Biological image motion processing: A review. Vision Research, 25, 625–660. Oman, C. M. (2007). Spatial orientation and navigation in microgravity. In F. Mast & L. Jancke (Eds.), Spatial Processing in Navigation, Imagery and Perception (pp. 209–247). New York: Springer. Prinzmetal, W., & Gettleman, L. (1993). Vertical-Horizontal Illusion—One eye is better than two. Perception and Psychophysics, 53, 81–88. Ross, H. E., Crickmar, S. D., Sills, N. V., & Owen, E. P. (1969). Orientation to the vertical in free divers. Aerospace Medicine, 40, 728–732. Villard, E., Garcia-Moreno, F. T., Peter, N., & Clement, G. (2005). Geometric visual illusions in microgravity during parabolic flight. Neuroreport, 16, 1395–1398. Yardley, L. (1990). Contribution of somatosensory information to perception of the visual vertical with body tilt and rotating visual-field. Perception and Psychophysics, 48, 131–134. Young, L. R., Hecht, H., Lyne, L. E., Sienko, K. H., Cheung, C. C., & Kavelaars, J. (2001). Artificial gravity: Head movements during short-radius centrifugation. Acta Astronautica, 49, 215–226.
A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 191 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.
CHAPTER 10
Organization and plasticity in multisensory integration: early and late experience affects its governing principles Barry E. Stein* and Benjamin A. Rowland Department of Neurobiology and Anatomy, Wake Forest School of Medicine, Winston-Salem, North Carolina, USA
Abstract: Neurons in the midbrain superior colliculus (SC) have the ability to integrate information from different senses to profoundly increase their sensitivity to external events. This not only enhances an organism's ability to detect and localize these events, but to program appropriate motor responses to them. The survival value of this process of multisensory integration is self-evident, and its physiological and behavioral manifestations have been studied extensively in adult and developing cats and monkeys. These studies have revealed, that contrary to expectations based on some developmental theories this process is not present in the newborn's brain. The data show that is acquired only gradually during postnatal life as a consequence of at least two factors: the maturation of cooperative interactions between association cortex and the SC, and extensive experience with cross-modal cues. Using these factors, the brain is able to craft the underlying neural circuits and the fundamental principles that govern multisensory integration so that they are adapted to the ecological circumstances in which they will be used. Keywords: visual; auditory; somatosensory; cross-modal; development.
The senses serve as portals thorough which the brain samples the environment. Each transduces a different form of energy, and thus provides an independent sample of the same event. The senses can compensate for one another when necessary and complement one another when reporting
about the same event. But, most impressive is that the brain can integrate the information it gathers from these different sensory channels to provide real-time benefits in event detection and scene analysis that would otherwise be impossible. The survival value of such a system is substantial. Biologic systems devised this strategy of “multisensory integration” very early in evolution, even before a brain was invented. They have extended and refined this process during evolution under
*Corresponding author. Tel.: 336-716-4368; Fax: 336-716-4534 E-mail:
[email protected] DOI: 10.1016/B978-0-444-53752-2.00007-2
145
146
most challenging circumstances (see Stein and Meredith, 1993). Nature is harsh in its assessment of innovation, and selection has yielded a striking diversity in the functional capabilities of sensory systems in different ecological niches of their host organisms. The same is true of the brains that process this information. Yet, despite this diversity, there are species-independent similarities in how multiple sensory inputs are integrated (e.g., Angelaki et al., 2011; Ernst and Banks, 2002; Frens and Van Opstal, 1998; King and Palmer, 1985; Meredith and Stein, 1983; Wallace et al., 1996). For example, in the detection of salient environmental events, one common strategy is to pool information across the senses according to the spatial and temporal relationships between observed crossmodal stimuli despite variation in the types of sensors providing this information. The commonality of these principles across species may reflect the fact that the constancies of space and time supersede variations in biology and ecological niche. Presumably, the ability to determine that stimuli accessing different sensory channels are linked to the same event is unavailable to the naïve brain, and yet such information is essential to construct a neural circuit that uses sensory systems synergistically to optimally detect and disambiguate environmental events. This determination results from early life experiences which, via simple learning rules based on the spatial and temporal congruence of crossmodal cues, could guide the formation of the underlying structural and functional architecture for making such determinations. Indeed, early life appears to be a time during which the brain uses its own experience to determine the integrative principles that will enhance the detection and identification of biologically significant events. There are several parallel objectives in this discussion: first to describe multisensory integration at the level of the single neuron and its implications for overt behavior, then to detail the development and maturation of this process
and its early plasticity, and finally to discuss the plasticity of multisensory neurons in adulthood. The mature superior colliculus A good deal of information has been obtained about the principles that govern multisensory integration at the level of the single neuron. Most of this has been derived from multisensory neurons in the cat superior colliculus (SC), although other species have also been studied (see, e.g., Gaither and Stein, 1979; Hartline et al., 1978; Stein and Gaither, 1981; Zahar et al., 2009). The SC has approximately the same location and function (orientation and localization) in all mammals (see Fig. 1). One of the many attractive features of the cat SC as a model for studying this process is that it is a site at which unisensory visual, auditory, and somatosensory projections converge onto individual neurons (Edwards et al., 1979; Fuentes-Santamaria et al., 2009; Huerta and Harting, 1984; Stein and Meredith, 1993; Wallace et al., 1993). As a result, its different constituent neurons become unisensory, bisensory, or trisensory, and the multisensory responses of these neurons reflect an operation that is taking place on-site, not one that happened elsewhere and was referred to the SC. The majority of neurons in the multisensory (i.e., deep) layers are visually responsive. Most common is the bisensory visual–auditory neuron, followed by the visual–somatosensory neuron. These have generally served as the exemplars for understanding the circuitry, principles, and underlying computations of multisensory integration, with use of the former type far more common than the latter. This is most likely due to the ease of finding visual–auditory neurons, and the ease of providing appropriate stimuli. However, the principles of multisensory integration appear to apply equally well to all modality convergence patterns.
147
Fig. 1. The superior colliculus (SC). Left: a schematic of the SC and the cortical afferents critical for its multisensory capabilities in the cat. Right: a photograph of a cat brain. The SC appears as a pair of hillocks rostral/superior to the inferior colliculus and caudal/ medial to the lateral geniculate nucleus.
Another attractive feature of this model is that SC activity is directly coupled to specific behaviors: detection, orientation, and localization (Lomber et al., 2001; Stein and Meredith, 1993). This is achieved through its descending projections to the motor regions of the brainstem and spinal cord (Moschovakis and Karabelas, 1985; Peck and Baro, 1997; Stein and Meredith, 1993). This affords one the opportunity to compare neurophysiological observations from individual SC neurons to SC-mediated overt behaviors. This is not a simple task in higherorder centers where the link between a simple behavioral response and a neural response is not as evident. As one would predict, the same principles that govern the responses of individual SC neurons to crossmodal stimuli also govern SC-mediated responses to them (Bell et al., 2005; Burnett et al., 2000, 2004; Frens and Van Opstal, 1998; Gingras et al., 2009; Jiang et al., 2002, 2007; Stein et al., 1988). Another benefit to this model is that the cat is born at an early maturational stage (see below), and requires substantial postnatal development before achieving its adult-like status. As a result, one can observe functional changes as they appear and are elaborated over time. The mass of information that had already been accumulated
about the maturation of its unisensory (largely visual) neuronal properties (e.g., Stein, 1984) provides a benchmark for sensory maturation that has also been extremely helpful. Thus, except where other species are noted, the discussion below will be referring to this animal. Semantic issues in multisensory integration Multisensory integration refers to the process by which a combination of stimuli from different senses (i.e., “crossmodal” stimulus) produces a neural response product that differs significantly from that evoked by the individual component stimuli, indicating a fusion of information. Multisensory integration has been defined at the level of the individual neuron as: a statistically significant difference between the number of impulses evoked by a crossmodal combination of stimuli and the number evoked by the most effective of these stimuli individually (see Stein and Meredith, 1993). This definition incorporates both multisensory response enhancement and response depression, but the former is more often used as an index of this process than the latter. This is because it is found in all the neurons exhibiting multisensory integration, whereas the latter is
148
found only in a subset of the neurons showing response enhancement (Kadunce et al., 1997). It does not refer to other multisensory processes such as those involved in crossmodal matching, where the individual sensory components must retain their independence so that they can be compared, as for example, when one compares the sight or feel of a given object. Nor does it refer to “amodal” processes; for example, engaged in comparing equivalencies in size, intensity, or number across senses (see Stein et al., 2010).
The underlying computation It is important to note that multisensory integration, as indicated by response enhancement, can reflect a variety of underlying computations (e.g., subadditive, additive, superadditive) that expose how a given neuron has integrated two or more different sensory inputs. Although superadditive responses most dramatically illustrate this phenomenon, superadditivity is not a prerequisite for multisensory integration. This important distinction is sometimes misunderstood. All cases of multisensory enhancement increase the physiological salience of the signal and thereby the
probability that the organism will respond appropriately to an event. Multisensory depression does the opposite (Calvert et al., 2004; Gillmeister and Eimer, 2007; Spence et al., 2004; Stein and Stanford, 2008), and in the case of the SC, may reflect a competition among stimuli for access to the motor circuitry (Stanford and Stein, 2007).
The principles of multisensory integration Multisensory SC neurons and SC-mediated behavior appear to follow a common set of operational principles (Stein and Meredith, 1993). Generally, crossmodal stimuli that are presented at the same time and place within their respective receptive fields enhance response magnitude, while crossmodal stimuli presented at different locations or times degrade or do not affect responses (Fig. 2). These are described by the “spatial” and “temporal” principles of integration. The degree to which a response is enhanced or depressed is inversely related to the effectiveness of the individual component stimuli (the “principle of inverse effectiveness”). This makes intuitive sense, as potent responses have less “room” for enhancement than do weak responses. The challenges and successes
10
V A
V Ai
100
A only
V only
VA
VA {ipsilateral}
x- Impulses/trail
30 V A
20
10
A
V
VA VAi
Fig. 2. Multisensory enhancement and depression: The spatial principle. Left: A square-wave broadband auditory stimulus (A, first panel) and a moving visual stimulus (ramp labeled V, second panel) evoked unisensory responses from this neuron (illustrated below the stimulus by rasters and peristimulus time histograms). Each dot in the raster represents a single impulse and each row a single trial. The third panel shows the response to their presentation at the same time and location, which is much more robust than either unisensory response. But, when the auditory stimulus was moved into ipsilateral auditory space (Ai) and out of the receptive field, its combination with the visual stimulus elicited fewer impulses than did the visual stimulus individually. This “response depression” is illustrated within the fourth panel. Right: The mean number of impulses/trial elicited by each of the four stimulus configurations. Note the difference between multisensory enhancement (VA) and multisensory depression (VAi). From Meredith et al. (1987).
149
in defining and using these principles have recently been discussed in detail (Stein and Stanford, 2008; Stein et al., 2009, 2010) and the operational definitions of “same place” and “same time,” as well as the interactions between spatial and temporal factors have been detailed empirically (Kadunce et al., 1997, 2001; Meredith and Stein, 1983, 1986, 1996; Meredith et al., 1987; Royal et al., 2009).
Simple heuristics A “rule of thumb” in remembering the principles of multisensory integration is that crossmodal stimuli likely to be linked to a common event enhance activity and behavioral performance, whereas those likely to be linked to different events degrade activity and performance. Further, the proportionate multisensory enhancement benefits that accrue are greatest when the integrated stimuli are least effective. The term “proportionate” is of key importance when considering the impact of multisensory integration. Multisensory responses are based on their magnitude relative to that of their presumptive component responses. Thus, even superadditive multisensory responses may be less robust in a given circumstance than those evoked by a highly effective modality-specific stimulus (see Stein and Stanford, 2008; Stein et al., 2009). The generality of the SC model These fundamental principles of multisensory integration, though derived from studies of the SC, appear to be general ones. They apply to multisensory neurons in other structures (e.g., Wallace et al., 1992) and are helpful in understanding behaviors other than those mediated by the SC. But it is important to remember that the principles discussed here are basic, they take into account neither higher-order cognitive issues such as semantic congruence and expectation nor issues relevant to the state of the organism, such
as hunger, thirst, fear, etc. The higher-order influences over this process are likely to differ in different brain circuits to best suit their functions. The impact of higher-order factors on the cellular processes underlying multisensory integration are poorly understood at present, but are being examined in a number of laboratories. It is also important to remember that the impact of each of these principles may not always be obvious or relevant in all situations, especially when examining their applicability to behaviors or perceptions uninvolved in the detection and localization of events from which they have been derived. For example, tasks involving no spatial component are unlikely to reflect the spatial principle. Although this may seem self-evident, it is sometimes overlooked. The essential circuit for SC multisensory integration Implementing multisensory integration may seem as simple as connecting different sensory inputs to a given neuron, but is not. This common assumption reflects a failure to appreciate the nature of the circuit that implements SC multisensory integration, and has hindered efforts to extract its features for nonbiologic uses. Although crossmodal convergence is necessary for this process, it is not sufficient for its implementation. The essential circuit includes converging descending inputs from two or more unisensory regions of association cortex (e.g., the anterior ectosylvian sulcus, AES) as shown in Fig. 3 (see also Alvarado et al., 2009; FuentesSantamaria et al., 2008, 2009). In the absence of this cortical input, SC neurons still respond to multiple sensory inputs (because they receive multiple inputs from many noncortical sources) but cannot integrate those inputs to produce signal enhancement (Alvarado et al., 2007, 2008, 2009; Jiang et al., 2001; Wallace and Stein, 1994). Instead, they respond no better to the combined input than they do to the most effective of them
150
individually (see Fig. 4). This physiological result is paralleled in behavioral observations as a loss of the performance benefit of multisensory integration in detection and localization (Jiang et al., 2002, 2006, 2007; Wilkinson et al., 1996). In short, the combination of modality-specific cues no longer provides significant operational benefits.
AE
S
Superi
orcol
+
licu
lus
+ +
+
+
+
The development of multisensory integration The significance of multisensory integration discussed above, as well as its seeming ubiquity across species, might lead one to assume that it is an inherent brain process that is present or prescribed at birth, especially given the newborn's heightened vulnerability. However, the data from experiments detailing the maturation of multisensory neurons in SC or cortex of animals indicated that the capability to engage in multisensory integration was not innate, but was acquired gradually during postnatal life as a consequence of experience with crossmodal stimuli (Carriere et al., 2007; Wallace and Stein, 1997, 2001, 2007; Wallace et al., 1993, 2006). This is in keeping with the finding that many sensory systems are poorly developed at birth and require substantial postnatal refinement for optimum function. Indeed, integrating information across them is even more complex than using them independently and thus may require longer periods of postnatal maturation. The cat is an altricial species, which makes it advantageous as a model because one can observe functional changes over its protracted postnatal developmental time period. The unisensory (particularly visual) response
− PA
G
+
+
Fig. 3. Schematic model of the SC multisensory circuit. Here the auditory-visual neuron is the exemplar multisensory neuron. Its inputs are derived from numerous unisensory sources. Of particular importance are the inputs that descend from an area of association cortex, the anterior ectosylvian sulcus or AES (top). The host of other inputs it receives (bottom) such as those ascending from sensory organs, relayed from other subcortical structures, or projecting from non-AES cortical areas have been collapsed here for illustrative and computational purposes. They form the ascending component of the input pathways illustrated here. Both the AES and non-AES inputs have a dual projection. Thus, they project also to a population of inhibitory interneurons. Together these afferents and target neurons constitute a circuit through which SC responses are a result of an excitatory (þ)–inhibitory () balance of inputs. From Rowland et al. (2007).
151 A
V
VA Mean impulses
rLs
Impulses
15
AES
10 5 0 200 ms
B Deactivate AES
7.0
3.5
*
200
Sum 100
0
0.0 V
A
VA
±0%
C Reactivate AES
*
+177%
270
35
Control Deactivate FAES Deactivate AEV Deactivate both
240 30
210
25
Enhancement index
Combined response (mean impulses)
+147%
Enhancement (%)
A Control
20 15
180 150 120 90 60 30
10 Control Deactivate FAES Deactivate AEV Deactivate both
5
0 -30 -60
0 0
5
10
15
20
25
30
Best unisensory response (mean impulses)
35
1
2
3
4
5>
Best unisensory response (mean impulses)
Fig. 4. The dependence of SC multisensory integration on cortex. The top-left figure shows the placement of cryogenic coils in the relevant cortical areas. Cold fluid circulated through the coils reduces the temperature of the surrounding cortex and inhibits activity. The top-right figure shows the area deactivated and then reactivated in this procedure (shaded region) and sample responses from a visual–auditory neuron to visual (V), auditory (A), and spatiotemporally concordant visual–auditory stimuli. Prior to cooling (control), the neuron shows an enhanced response to the visual–auditory stimulus complex. However, when cortex is cooled (deactivate AES), the multisensory response is no longer statistically greater than the best unisensory response. Reactivating cortex (reactivate AES) returns the neuron's integrative capabilities. The bottom-left figure plots the multisensory response versus the best unisensory response for a population of similar visual–auditory neurons before deactivation (green), when only one subregion of AES is deactivated (red, FAES; blue, AEV), or when both are deactivated (yellow). The bottomright plots the enhancement index (percent difference between the multisensory and best unisensory response) for these four conditions against the best unisensory response. The results of this study indicate that there is a true “synergy” between the subregions of AES cortex in producing multisensory integration in the SC: deactivating one or the other subregion often yields results equivalent to deactivating both.
152
properties of these neurons are well-studied and provide a benchmark for sensory development (Stein, 1984). The newborn cat has poor motor control and is both blind and deaf, and its SC contains only tactile-responsive neurons (Stein et al., 1973), which presumably aid the neonate in suckling (Larson and Stein, 1984). SC responses to auditory stimuli are first apparent at 5 dpn (days postnatal; Stein, 1984) and visual responses in the multisensory (deep) layers after several additional weeks (Kao et al., 1994). Obviously, prior to this time, these neurons cannot engage in multisensory integration. It is necessary to be specific about the appearance of visual sensitivity in the multisensory layers of the SC, because the overlying superficial layers, which are purely visual, develop their visual sensitivity considerably earlier (Stein, 1984; Stein et al., 1973). Although superficial layer neurons are not directly involved in multisensory processes (their function is believed to more closely approximate that of neurons in the primary projection pathway), this superficial-deep developmental lag is still somewhat surprising because superficial layer neurons provide some of the visual input to the multisensory layers (Behan and Appell, 1992; Grantyn and Grantyn, 1984; Moschovakis and Karabelas, 1985). Apparently, the functional coupling of superficial neurons with their deep layer target neurons has not yet developed. The maturational distinction between visually responsive neurons within the same structure underscores a key difference between unisensory neurons and those that will be involved in integrating inputs from different senses. The chronology of multisensory neurons parallels but is delayed with respect to the chronology of unisensory development. The earliest multisensory neurons are somatosensory-auditory, appearing at 10–12 days after birth. The first visual–nonvisual neurons take 3 weeks to appear (Kao et al., 1994; Stein et al., 1973; Wallace and Stein, 1997). However, the incidence of these multisensory neurons does not reach adult-
like proportions until many weeks later. Visual, auditory, and somatosensory receptive fields are all initially very large and contract significantly over months of development, thereby enhancing the resolution of their individual maps, the concordance among the maps, and of special importance in this context, the spatial concordance of the multiple receptive fields of individual neurons (Fig. 5). The changes are accompanied by increases in the vigor of neuronal responses to sensory stimuli, increases in response reliability, decrease in response latency, and an increase in the ability to respond to successive stimuli (Kao et al., 1994; Stein et al., 1973; Wallace and Stein, 1997). These functional changes reflect the maturation of the intrinsic circuitry of the structure, as well as the maturation and selection of its afferents resulting from selective strengthening and pruning of synapses. However, these neonatal multisensory neurons are incapable of integrating their multiple sensory inputs. SC neurons do not show multisensory integration until at least a month of age, long after they have developed the capacity to respond to more than one sensory modality (Wallace and Stein, 1997). In other words, they respond to crossmodal stimulations as if only one (typically the more effective) stimulus is present. Once multisensory integration begins to appear, only a few neurons show it at first. Gradually, more and more multisensory neurons begin to show integration, but it takes many weeks before the normal complement of neurons capable of multisensory integration is achieved. The inability of neonatal multisensory neurons to integrate their different sensory inputs is not limited to the kitten, nor is it restricted to altricial species. The Rhesus monkey is much more mature at birth than is the cat, and already has many multisensory SC neurons. Apparently, the appearance of mulitsensory neurons during development does not depend on postnatal experience, but on developmental stage, an observation we will revisit below. However, the multisensory neurons in the newborn primate,
153 Visual
Auditory
Somatosensory
Visual
22 dpn
N
20 dpn S
S
-
N
I
S
T
N
I
I
S
S
T
42 dpn S
N
T
42 dpn
N
I
T
N
I
I
S
S
T
135 dpn S
N
I
-
135 dpn
N
I
T
N
T
I
Fig. 5. Receptive fields of SC neurons shrink during development. Illustrated are the receptive fields of two types of neurons (left: visual–auditory, right: visual–somatosensory) at different developmental ages (increasing from top to bottom). As depicted, the receptive fields shrink in size as the animal ages, increasing their specificity and also their spatial concordance across the senses. Adapted from Wallace and Stein (1997).
just like those in the cat, are incapable of integrating their different sensory inputs, and, in this regard, are distinctly different than their adult counterparts (Wallace and Stein, 2001; Wallace et al., 1996). Presumably this is because they have not yet had the requisite experience with crossmodal events. Recent observations in human subjects (Gori et al., 2008; Neil et al., 2006; Putzar et al., 2007) also suggest that there is a gradual postnatal acquisition of this capability, but there is no unequivocal information regarding the newborn. However, this does not mean that newborns have no multisensory processing capabilities, only that
they cannot use crossmodal information in a synergistic way (i.e., do not engage “multisensory integration” as defined above). Those studying human development sometimes include other multisensory processes under this umbrella. The best example of this is crossmodal matching, a capacity that appears to be present early in life. However, as noted earlier, this process does not yield an integrated product. While it is clearly a multisensory process, it is not an example of multisensory integration (see Stein et al., 2010 for more discussion). But, some caution should still be exercised here, as the brains and/or behaviors of only a limited number of species
154
have been studied thus far. It may still turn out that some examples of multisensory integration, such as those involving chemical senses, and/or species that are rarely examined in the laboratory, develop prenatally and independent of experience. After all, it is probably an inherent characteristic of single-celled organisms that have multiple receptors embedded in the same membrane.
1988). Presumably, the inability of neonatal SC multisensory neurons to integrate their crossmodal inputs is because the AES–SC synaptic coupling is not properly functional (just as those from superficial layers are not). This is only a supposition, for at this point we know little about how this projection changes over time. Some of the AES inputs to the SC certainly become functional at about 1 month of age, for soon after individual SC neurons exhibit multisensory integration, this capability can be blocked by deactivating AES (see Fig. 6; Wallace and Stein, 2000). These relationships strengthen over the next few months. This is also a period during which the brain is exposed to a variety of sensory stimuli, some of which are linked to the same event and some of which are not. Crossmodal cues that are derived
How experience changes the circuit for multisensory integration Inputs from AES have already reached the multisensory SC at birth, even before its constituent neurons become multisensory (McHaffie et al.,
15–20 weeks
75
control AES deactivated **
(12)
7–10 weeks
50
(7)
* (5)
4–6 weeks
(15)
200
25
Multisensory enhancement (%)
% of deep SC sensory population
11–14 weeks
** (10)
150
(12) 100
(11) **
50
(9)
0
Multisensory integrating neurons Multisensory integrating neurons affected by AES deactivation
(8)
−50
0 0
4
6
8
10
12
14
16
20
Adult
Postnatal age (weeks) Fig. 6. The developmental appearance o f multisensory integration coincides with the development of AES–SC influences. There is a progressive increase in the percent of SC neurons exhibiting multisensory integration capabilities as revealed by the graph. Note that whenever a neuron with integrating capabilities was located, the effect of AES deactivation was examined. Regardless of age, nearly all neurons lost this capability during cryogenic block of AES activity (numbers in parentheses show the number of neurons examined). Presumably, those SC neurons that were not affected by AES blockade were dependent on adjacent areas (e.g., rostral lateral suprasylvian cortex, see Jiang et al., 2001). From Wallace and Stein (2000).
155
from the same event often occur in spatiotemporal concordance, while unrelated events are far less tightly linked in space and time. Presumably, after sufficient experience, the brain has learned the statistics of those sensory events, which, via Hebbian learning rules, have been incorporated into the neural architecture underlying the capacity to integrate different sensory inputs. Such experience provides the foundation for a principled way of perceiving and interacting with the world so that only some stimulus configurations will be integrated and yield response enhancement or response depression. Put another way, the experience leads the animal to expect that certain crossmodal physical properties covary (e.g., the timing and/or spatial location of visual and auditory stimuli) and this “knowledge” is used to craft the principles for discriminating between those stimuli derived from the same event and those derived from different events. The first test of this hypothesis was aimed at determining whether experience is essential for the maturation of this process. Visual–nonvisual experiences were precluded by rearing animals in darkness from birth to well after the maturation of multisensory integration is normally achieved (i.e., 6 months or more). Interestingly, this rearing condition did not prevent the development of visually responsive neurons. In fact, in addition to unisensory neurons, each of the crossmodal convergence and response patterns characteristic of normal animals was evident in neurons within the SC of dark-reared animals, though their incidence was slightly different (Wallace et al., 2001, 2004). This parallels the observations in monkey, which is born later in development than the cat but already has visual–nonvisual SC neurons. Visual experience is obviously not essential for the appearance of such neurons. The receptive fields of these neurons in darkreared cats were very large, more like neonatal SC neurons than those in the adult. Like neonatal neurons, they could not integrate their crossmodal inputs and their responses to crossmodal pairs
of visual–nonvisual stimuli were no more vigorous than were their responses to the best of the modality-specific component stimuli (Fig. 7). As postulated, experience with visual–nonvisual stimuli proved to be necessary to develop the capacity to engage in multisensory integration. This is also consistent with observations in human subjects who had congenital cataracts removed during early life. Their vision seemed reasonably normal, but they were compromised in their ability to integrate visual and nonvisual cues, despite having years of experience after surgical correction (Putzar et al., 2007). The next test of this hypothesis was to rear animals in conditions in which the spatiotemporal relationships of crossmodal stimuli were altered from “normal” experience, in which they are presumably in spatiotemporal concordance when derived from the same event. If crossmodal experience determines the governing principles of multisensory integration, then changing it should change the principles. This possibility was examined after rearing animals in special dark environments in which their only experience with simultaneous visual and auditory stimuli was when they were spatially displaced (Wallace and Stein, 2007). They were raised to 6 months or more in this condition and then the multisensory integration characteristics of SC neurons were examined. Similar to simply dark-reared animals, these animals possessed the full range of multisensory convergence patterns and there were many visual–nonvisual neurons. However, the properties of visually responsive neurons were atypical: their receptive fields were very large, and many were unable to integrate visual–nonvisual cues. There was, however, a sizable minority of visual–auditory neurons that were fully capable of multisensory integration, but the stimulus configurations eliciting response enhancement or no integration were significantly different from those of normally reared animals (Fig. 8). Their receptive fields, unlike those of many of their neighbors, had contracted partially, but were in
156 NORMAL
Mean impulses/trial
VA
A
V
10
A
VA
V
0
A VA
100
10 Mean impulses/trial
V
Visual–auditory
Sum 100
0
Auditory
200 +165%
5
DARK-REARED Visual
**
15
Interaction (%)
Visual–auditory
Sum 5
50
Interaction (%)
Auditory
Visual
+8% 0
V
A
VA
0
Fig. 7. Comparison between normal and dark-reared animals. The sample neuron illustrated on top was recorded from the SC of a normally reared animal. Its visual and auditory receptive fields are relatively small and in good spatial register with one another. The summary figure on the right indicates its responses to visual, auditory, and spatiotemporally concordant visual–auditory stimuli, which yields typical multisensory response enhancement. The sample neuron on the bottom was recorded from the SC of an animal reared in the dark. Its receptive fields are much larger, and while it responds to both visual and auditory stimuli, its response to a spatiotemporally concordant visual–auditory stimulus complex is statistically no greater than the response to the visual stimulus alone. Adapted from Wallace et al. (2004).
poor spatial alignment with one another. Some were totally out of register, a feature that is exceedingly rare in normal animals, but one that clearly reflects the early experience of these animals with visual–auditory cues. Most important in the current context is that those neurons integrated spatially disparate stimuli to produce response enhancement—not spatially concordant stimuli. This is because their receptive fields were misaligned and only spatially disparate stimuli could fall simultaneously within them. Taken together, the dark rearing and disparity rearing conditions demonstrate that not only is experience critical for the maturation of multisensory integration, but that the nature of the experience directs formation of the neural circuits that engage in this process. In both normal and disparity-reared animals, the basis for multisensory response enhancement is defined by early experience. Whether this reflects a simple adaptation to
specific crossmodal stimulus configurations, or the general statistics of multisensory experience, is a subject of ongoing experimentation. Parallel experiments in AES cortex revealed that multisensory integration develops more slowly in cortex than in the SC. These multisensory neurons in AES populate the border regions between its visual (AEV), auditory (FAES), and somatosensory (SIV) subregions. This is perhaps not surprising, as in general, the development of the cortex is thought to be more protracted than that of the midbrain. These multisensory cortical neurons are involved in a circuit independent of the SC, as they do not project into the cortico-SC pathway (Wallace et al., 1992). Despite this, they have properties very similar to those found in the SC. They too fail to show multisensory integration capabilities during early neonatal stages, and develop this capacity gradually, and after SC neurons (Wallace
157 Visual receptive field
A
V
A
A V
35
A V
V
35
30
30
30
25
25
25
20
20
20
15
15
15
10
10
10
5
5
5
0
0
0
Mean impulses/trial
A V
A V
ns
8
100
A V
P < 001
200
16
+144%
interaction (%)
Impulses
35
Auditory receptive field
sum ns
+16% 0
0 V
A
VA
V
A
VA
-3%
V
A
VA
Fig. 8. Rearing animals in environments with spatially disparate visual–auditory stimulus configurations yields abnormal multisensory integration. Illustrated is a sample neuron recorded from the SC of an animal reared in an environment in which simultaneous visual and auditory stimuli were always spatially displaced. This neuron developed spatially misaligned receptive fields (i.e., the visual receptive field is central while the auditory receptive field is in the periphery). When presented with spatiotemporally concordant visual–auditory stimuli in the visual (left plots) or auditory (center plots) receptive fields, the multisensory response is no larger than the largest unisensory response (the identity of which is determined by which receptive field served as the stimulus location). However, if temporally concordant but spatially disparate visual and auditory stimuli are placed within their respective receptive fields, the multisensory response shows significant enhancement. In other words, this neuron appears to integrate spatially disparate crossmodal cues as a normal animal integrates spatially concordant cues. It fails to integrate spatially concordant, just as a normal animal might fail to integrate spatially discordant cues, an apparent “reversal” of the spatial principle. Adapted from Wallace and Stein (2007).
et al., 2006). Just as is the case for SC neurons, these AES neurons also require sensory experience and fail to develop multisensory integration capabilities when animals are raised in the dark (Carriere et al., 2007). Although the above observations suggest that the development of multisensory integration in the SC and cortex is dependent on exposure to
crossmodal stimuli and its principles adapt to their configurations, they provide no insight as to the underlying circuitry governing its development and adaptation. However, for multisensory SC neurons, the cortical deactivation studies described above coupled with the maturational time course of the AES–SC projection suggests that AES cortex is likely to play a critical role.
158
Evaluating this idea began with experiments in which chronic deactivation of association cortex (both AES and its adjacent area rLS) was induced on one side of the brain for 8 weeks (between 4 and 12 weeks postnatal) during the period in which multisensory integration normally develops (Stein and Stanford, 2008; Wallace and Stein, 1997), thereby rendering them unresponsive to sensory (in particular, crossmodal) experience. The deactivation was induced with muscimol, a GABAa agonist. It was embedded in a polymer that was placed over association cortex from which it was slowly released over this period. After the polymer released its stores of muscimol or was physically removed, the cortex became active and responsive to environmental stimuli. Animals were then tested behaviorally and physiologically when adults (1 year of age), long after cortex had reactivated. These experiments are still ongoing, but preliminary results are quite clear. Their ability of these animals to detect and locate visual stimuli was indistinguishable from that of normal animals, and was equally good in both hemifields. Further, behavioral performance indicated that they significantly benefited from the presentation of spatiotemporally concordant but task-irrelevant auditory stimuli in the ipsilateral hemifield (as do normal animals). However, in the contralateral hemifield, they were abnormal: responses to spatiotemporally concordant visual–auditory stimuli were no better than when the visual stimulus was presented alone. Apparently, deactivating ipsilateral association cortex during early life disrupted the maturation of multisensory integration capabilities in the contralateral hemifield. SC neurons in these animals also appeared incapable of synthesizing spatiotemporally concordant crossmodal stimuli to produce multisensory response enhancement. These data strongly support the hypothesis that the AES–SC projection is principally engaged in the instantiation of multisensory integration in the SC. The fact that the deficits in multisensory integration were observed
long after the deprivation period, regardless of whether they were induced by dark rearing or chronic cortical deactivation, suggests that there is a “critical” or “sensitive” period for acquiring this capacity. Such a period would demarcate the period in which the capacity could be acquired. Multisensory plasticity in adulthood Despite these observations, it is possible that multisensory integration and its principles may be plastic in adulthood, but may operate on different time scales or be sensitive to different types of experience. Animals involved in the studies described above entailing chronic deactivation during early life were retained, and were available for experimentation several years later. The results were striking: whereas they had shown deficits before, now they appeared to show normal multisensory integration capabilities both in behavior and physiology on both sides of space. It is possible that experience was gradually incorporated into the AES–SC projection over such a long period of time. Another possibility is that entirely new circuits, not involving the AES, formed to support the instantiation of multisensory integration in the SC, although this seems less likely, as such circuits do not form in response to neonatal cortical ablation of cortex (Jiang et al., 2006). Ongoing experiments are investigating this issue. However, it is possible that multisensory integration might also be plastic on shorter time scales in the adult under the proper conditions. Yu et al. (2009) examined whether multisensory SC neurons in anesthetized animals would alter their responses if repeatedly presented with temporally proximal sequential crossmodal stimuli. Because the stimuli were separated by hundreds of milliseconds, they initially generated what would be generally regarded as two distinct unisensory responses (separated by a “silent” period) rather than a single integrated response. They found
159
that SC neurons rapidly adapted to the repeated presentation of these stimulus configurations (Yu et al., 2009). After only a few minutes of exposure to the repeated sequence, the initial silent period between the two responses began to be populated by impulses. Soon it appeared as if the two responses were merging (see Fig. 9). This resulted from an increase in the magnitude and duration of the first response and a shortening of the latency of the second response when they were presented in sequence. The stimuli were either visual or auditory, but it did not seem to matter which was the first or second in the sequence. Interestingly, similar sequences of stimuli belonging to the same modality did not generate the same consistent results. These observations confirmed the presence of plasticity in adult multisensory SC neurons, and revealed that this plasticity could be induced even when the animal was anesthetized. Presumably, similar changes could be induced by temporally proximate crossmodal cues like those used in studies examining multisensory integration. The results also raise the question of whether a darkreared animal's SC neurons could acquire the capacity to develop multisensory integration capabilities even after the animal is raised to adulthood in the dark. To test this possibility, Yu et al. (2010) raised animals from birth to maturity in darkness and then provided them with spatiotemporally concordant visual–auditory stimuli during daily exposure periods (Fig. 10). Once again, the animals were anesthetized during these exposure periods. Comparatively soon, SC neurons began showing multisensory integration capabilities, and the magnitude of these integrated responses increased over time to reach the level of normal adults (Yu et al., 2010). Of particular interest was the speed of acquisition of this capability. It was far more rapid than the acquisition in normally reared animals, suggesting that much of the delay in normal maturation is related to the development of the neural architecture that
encodes these experiences. Interestingly, with only a few exceptions, only those neurons that had both receptive fields encroaching on the exposure site acquired this capability. This finding indicates that the crossmodal inputs to the neuron had to be activated together for this experience to have influence; that is, the influence of experience was not generalized across the population of neurons. However, within a given neuron, the experience was generalized to other overlapping areas of the receptive field, even those that did not exactly correspond to the exposure site. It is not clear from these observations whether this is a general finding or one specific to the exact stimuli and stimulus configurations (e.g., the fact that the exposure stimuli have precise spatiotemporal relationships) used to initiate the acquisition in multisensory integration. This may be the reason that the cats given chronic cortical deactivation do not develop multisensory integration capabilities even as young adults, and humans with congenital cataracts that have undergone corrective surgery do not immediately develop this capacity (Putzar et al., 2007). Though seemingly reasonable, this supposition requires empirical validation. The continued plasticity of multisensory integration into adulthood also suggests that its characteristics may be adaptable to changes in environmental circumstances, specifically, changes in crossmodal statistics. This promises to be an exciting issue of future exploration. The possibility that it is not too late to acquire this fundamental capacity during late childhood or adulthood promises an ability to ameliorate the dysfunctions in this capacity induced by early deprivation via congenital vision and/or early hearing impairments. Perhaps by better understanding the requirements for its acquisition, better rehabilitative strategies can be developed. It may also be possible to significantly enhance the performance of people with normal developmental histories, especially in circumstances in which detection and localization of events is of critical importance.
(a)
(b)
mean response (impulses)
1.0
0 1.0
1.0
The first 15 trials
0 1.0
The last 15 trials
0 -1000
-500
The first 15 trials
0
500
1000
0 -500
1500
180
The last 15 trials
0
500
1000
1500
0
500
1000
1500
150
Sequential cross-modal exposure (trials)
150 120
100
90 60
50
30 0 -1000
-500
0
500
1000
0 -500
1500
Time after stimulus onset (ms)
(c)
(d)
mean response (impulses)
1.6
0 1.6
2
The first 15 trials
The first 15 trials
0 2
The last 15 trials
0 -500
0
500
1000
1500
0 -500
The last 15 trials
0
500
1000
1500
0
500
1000
1500
100 160 140
Sequential cross-modal exposure (Trials)
80
120 60
100 80
40 60 40
20
20 0 -500
0
500
1000
1500
0 -500
Time after stimulus onset (ms)
Fig. 9. A repeated sequence of auditory and visual stimuli in either order led to a merging of their responses. Shown are the responses of four neurons. In each display, the responses are ordered from bottom to top. The stimuli are represented by bars above the rasters: the short one refers to the auditory stimulus and the long one to the visual stimulus. The first and last series of trials (n ¼ 15 in each) are shaded in the rasters and displayed at the top as peristimulus time histograms (20 ms bin width). Arrows indicate the time period between the responses to the stimuli. Note that the period of relative quiescence between the two distinct unisensory responses is lost after a number of trials and the responses begin to merge. This is most obvious when comparing the activity in the first and last 15 trials. From Yu et al. (2009).
3600 exposure trials (30°)
(b)
A V
A
250 ms V
A
A V
37,800 exposure trials (30°)
(c) V
50,400 exposure trials (0°)
A
A V
Mean impulses/trial
V
8
Mean impulses/trial
(a)
12
Mean impulses/trial
161
12
4
0
6
0
46.0% *
V A VA
164.7% **
V A VA
224.1% **
6
0
V A VA
Fig. 10. Exposure to spatiotemporally concordant visual–auditory stimuli leads to the maturation of multisensory integration capabilities in dark-reared animals. Shown above are the receptive fields, exposure sites, and responses of three SC neurons (a–c). Left: receptive fields (visual, black; auditory, gray) are shown on schematics of visual–auditory space. The numbers below refer to crossmodal exposure trials provided before testing the neuron's multisensory integration capability. The exposure site (0 or 30 ) is also on the schematic and designated by a light gray square. Both receptive fields of each neuron overlapped the exposure site. Middle: each neuron responded to the crossmodal stimuli with an integrated response that exceeded the most robust unisensory response and, in 2/3 cases, exceeded their sum. Right: the summary bar graphs compare the average unisensory and multisensory responses. From Yu et al. (2010).
Acknowledgments The research described here was supported in part by NIH grants NS36916 and EY016716. References Alvarado, J. C., Rowland, B. A., Stanford, T. R., & Stein, B. E. (2008). A neural network model of multisensory integration also accounts for unisensory integration in superior colliculus. Brain Research, 1242, 13–23. Alvarado, J. C., Stanford, T. R., Rowland, B. A., Vaughan, J. W., & Stein, B. E. (2009). Multisensory integration in the superior colliculus requires synergy among corticocollicular inputs. The Journal of Neuroscience, 29, 6580–6592. Alvarado, J. C., Stanford, T. R., Vaughan, J. W., & Stein, B. E. (2007). Cortex mediates multisensory but not unisensory
integration in superior colliculus. The Journal of Neuroscience, 27, 12775–12786. Angelaki, D., Gu, Y., & Deangelis, G. (2011). Visual and vestibular cue integration for heading perception in extrastriate visual cortex. Journal of Physiology, 589, 825–833. Behan, M., & Appell, P. P. (1992). Intrinsic circuitry in the cat superior colliculus: Projections from the superficial layers. The Journal of Comparative Neurology, 315, 230–243. Bell, A. H., Meredith, M. A., Van Opstal, A. J., & Munoz, D. P. (2005). Crossmodal integration in the primate superior colliculus underlying the preparation and initiation of saccadic eye movements. Journal of Neurophysiology, 93, 3659–3673. Burnett, L. R., Stein, B. E., Chaponis, D., & Wallace, M. T. (2004). Superior colliculus lesions preferentially disrupt multisensory orientation. Neuroscience, 124(3), 535–547. Burnett, L., Stein, B. E., & Wallace, M. T. (2000). Ibotenic acid lesions of the superior colliculus disrupt multisensory orientation behaviors. Society for Neuroscience, Abstracts, 26, 1220. New Orleans, LA: Society for Neuroscience.
162 Calvert, G. A., Spence, C., & Stein, B. E. (2004). The handbook of multisensory processes. Cambridge, MA: MIT Press. Carriere, B. N., Royal, D. W., Perrault, T. J., Morrison, S. P., Vaughan, J. W., Stein, B. E., et al. (2007). Visual deprivation alters the development of cortical multisensory integration. Journal of Neurophysiology, 98, 2858–2867. Edwards, S. B., Ginsburgh, C. L., Henkel, C. K., & Stein, B. E. (1979). Sources of subcortical projections to the superior colliculus in the cat. The Journal of Comparative Neurology, 184, 309–329. Ernst, M. O., & Banks, M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 415, 429–433. Frens, M. A., & Van Opstal, A. J. (1998). Visual-auditory interactions modulate saccade-related activity in monkey superior colliculus. Brain Research Bulletin, 46, 211–224. Fuentes-Santamaria, V., Alvarado, J. C., McHaffie, J. G., & Stein, B. E. (2009). Axon morphologies and convergence patterns of projections from different sensory-specific cortices of the anterior ectosylvian sulcus onto multisensory neurons in the cat superior colliculus. Cerebral Cortex, 19, 2902–2915. Fuentes-Santamaria, V., Alvarado, J. C., Stein, B. E., & McHaffie, J. G. (2008). Cortex contacts both output neurons and nitrergic interneurons in the superior colliculus: Direct and indirect routes for multisensory integration. Cerebral Cortex, 18, 1640–1652. Gaither, N. S., & Stein, B. E. (1979). Reptiles and mammals use similar sensory organizations in the midbrain. Science, 205, 595–597. Gillmeister, H., & Eimer, M. (2007). Tactile enhancement of auditory detection and perceived loudness. Brain Research, 1160, 58–68. Gingras, G., Rowland, B. A., & Stein, B. E. (2009). The differing impact of multisensory and unisensory integration on behavior. The Journal of Neuroscience, 29, 4897–4902. Gori, M., Del, V. M., Sandini, G., & Burr, D. C. (2008). Young children do not integrate visual and haptic form information. Current Biology, 18, 694–698. Grantyn, A., & Grantyn, R. (1984). Generation of grouped discharges by tectal projection cells. Archives of Italian Biology, 122, 59–71. Hartline, P. H., Kass, L., & Loop, M. S. (1978). Merging of modalities in the optic tectum: Infrared and visual integration in rattlesnakes. Science, 199, 1225–1229. Huerta, M. F., & Harting, J. K. (1984). The mammalian superior colliculus: Studies of its morphology and connections. In H. Vanegas (Ed.), Comparative neurology of the optic tectum (pp. 687–773). New York, NY: Plenum Publishing Corporation. Jiang, W., Jiang, H., Rowland, B. A., & Stein, B. E. (2007). Multisensory orientation behavior is disrupted by neonatal cortical ablation. Journal of Neurophysiology, 97, 557–562. Jiang, W., Jiang, H., & Stein, B. E. (2002). Two corticotectal areas facilitate multisensory orientation behavior. Journal of Cognitive Neuroscience, 14, 1240–1255.
Jiang, W., Jiang, H., & Stein, B. E. (2006). Neonatal cortical ablation disrupts multisensory development in superior colliculus. Journal of Neurophysiology, 95, 1380–1396. Jiang, W., Wallace, M. T., Jiang, H., Vaughan, J. W., & Stein, B. E. (2001). Two cortical areas mediate multisensory integration in superior colliculus neurons. Journal of Neurophysiology, 85, 506–522. Kadunce, D. C., Vaughan, J. W., Wallace, M. T., Benedek, G., & Stein, B. E. (1997). Mechanisms of within- and crossmodality suppression in the superior colliculus2. Journal of Neurophysiology, 78, 2834–2847. Kadunce, D. C., Vaughan, J. W., Wallace, M. T., & Stein, B. E. (2001). The influence of visual and auditory receptive field organization on multisensory integration in the superior colliculus1. Experimental Brain Research, 139, 303–310. Kao, C. Q., Stein, B. E., & Coulter, D. A. (1994). Postnatal development of excitatory synaptic function in deep layers of superior colliculus. Society for Neuroscience Abstract, 20, 1186. King, A. J., & Palmer, A. R. (1985). Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Experimental Brain Research, 60, 492–500. Larson, M. A., & Stein, B. E. (1984). The use of tactile and olfactory cues in neonatal orientation and localization of the nipple. Developmental Psychobiology, 17, 423–436. Lomber, S. G., Payne, B. R., & Cornwell, P. (2001). Role of the superior colliculus in analyses of space: Superficial and intermediate layer contributions to visual orienting, auditory orienting, and visuospatial discriminations during unilateral and bilateral deactivations. The Journal of Comparative Neurology, 441, 44–57. McHaffie, J. G., Kruger, L., Clemo, H. R., & Stein, B. E. (1988). Corticothalamic and corticotectal somatosensory projections from the anterior ectosylvian sulcus (SIV cortex) in neonatal cats: An anatomical demonstration with HRP and 3H-leucine. The Journal of Comparative Neurology, 274, 115–126. Meredith, M. A., Nemitz, J. W., & Stein, B. E. (1987). Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. Journal of Neuroscience, 7, 3215–3229. Meredith, M. A., & Stein, B. E. (1983). Interactions among converging sensory inputs in the superior colliculus. Science, 221, 389–391. Meredith, M. A., & Stein, B. E. (1986). Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Research, 365, 350–354. Meredith, M. A., & Stein, B. E. (1996). Spatial determinants of multisensory integration in cat superior colliculus neurons. Journal of Neurophysiology, 75, 1843–1857. Moschovakis, A. K., & Karabelas, A. B. (1985). Observations on the somatodendritic morphology and axonal trajectory of intracellularly HRP-labeled efferent neurons located in the deeper layers of the superior colliculus of the cat. The Journal of Comparative Neurology, 239, 276–308.
163 Neil, P. A., Chee-Ruiter, C., Scheier, C., Lewkowicz, D. J., & Shimojo, S. (2006). Development of multisensory spatial integration and perception in humans. Developmental Science, 9, 454–464. Peck, C. K., & Baro, J. A. (1997). Discharge patterns of neurons in the rostral superior colliculus of cat: Activity related to fixation of visual and auditory targets. Experimental Brain Research, 113, 291–302. Putzar, L., Goerendt, I., Lange, K., Rosler, F., & Roder, B. (2007). Early visual deprivation impairs multisensory interactions in humans. Nature Neuroscience, 10, 1243–1245. Rowland, B. A., Stanford, T. R., & Stein, B. E. (2007). A model of the neural mechanisms underlying multisensory integration in the superior colliculus. Perception, 36, 1431–1443. Royal, D. W., Carriere, B. N., & Wallace, M. T. (2009). Spatiotemporal architecture of cortical receptive fields and its impact on multisensory interactions. Experimental Brain Research, 198, 127–136. Spence, C., Pavani, F., & Driver, J. (2004). Spatial constraints on visual-tactile cross-modal distractor congruency effects. Cognitive, Affective and Behavioral Neuroscience, 4, 148–169. Stanford, T. R., & Stein, B. E. (2007). Superadditivity in multisensory integration: Putting the computation in context. Neuroreport, 18, 787–792. Stein, B. E. (1984). Development of the superior colliculus. In Annual review of neuroscience (pp. 95–125). Palo Alto, CA: Annual Reviews, Inc. Stein, B. E., Burr, D., Constantinidis, C., Laurienti, P. J., Alex, M. M., Perrault, T. J. Jr., et al. (2010). Semantic confusion regarding the development of multisensory integration: A practical solution. The European Journal of Neuroscience, 31, 1713–1720. Stein, B. E., & Gaither, N. S. (1981). Sensory representation in reptilian optic tectum: Some comparisons with mammals. The Journal of Comparative Neurology, 202, 69–87. Stein, B. E., Huneycutt, W. S., & Meredith, M. A. (1988). Neurons and behavior: The same rules of multisensory integration apply. Brain Research, 448, 355–358. Stein, B. E., Labos, E., & Kruger, L. (1973). Sequence of changes in properties of neurons of superior colliculus of the kitten during maturation. Journal of Neurophysiology, 36, 667–679. Stein, B. E., & Meredith, M. A. (1993). The merging of the senses. Cambridge, MA: MIT Press. Stein, B. E., & Stanford, T. R. (2008). Multisensory integration: Current issues from the perspective of the single neuron. Nature Reviews. Neuroscience, 9, 255–266. Stein, B. E., Stanford, T. R., Ramachandran, R., Perrault, T. J., Jr., & Rowland, B. A. (2009). Challenges in quantifying multisensory integration? Alternative criteria, models, and inverse effectiveness. Experimental Brain Research, 198, 113–126.
Wallace, M. T., Carriere, B. N., Perrault, T. J., Jr. Vaughan, J. W., & Stein, B. E. (2006). The development of cortical multisensory integration. The Journal of Neuroscience, 26, 11844–11849. Wallace, M. T., Hairston, W. D., & Stein, B. E. (2001). Longterm effects of dark-rearing on multisensory processing. Program No. 511.6. Wallace, M. T., Meredith, M. A., & Stein, B. E. (1992). Integration of multiple sensory modalities in cat cortex. Experimental Brain Research, 91, 484–488. Wallace, M. T., Meredith, M. A., & Stein, B. E. (1993). Converging influences from visual, auditory, and somatosensory cortices onto output neurons of the superior colliculus. Journal of Neurophysiology, 69, 1797–1809. Wallace, M. T., Perrault, T. J., Jr., Hairston, W. D., & Stein, B. E. (2004). Visual experience is necessary for the development of multisensory integration1. The Journal of Neuroscience, 24, 9580–9584. Wallace, M. T., & Stein, B. E. (1994). Cross-modal synthesis in the midbrain depends on input from cortex. Journal of Neurophysiology, 71, 429–432. Wallace, M. T., & Stein, B. E. (1997). Development of multisensory neurons and multisensory integration in cat superior colliculus. The Journal of Neuroscience, 17, 2429–2444. Wallace, M. T., & Stein, B. E. (2000). Onset of cross-modal synthesis in the neonatal superior colliculus is gated by the development of cortical influences. Journal of Neurophysiology, 83, 3578–3582. Wallace, M. T., & Stein, B. E. (2001). Sensory and multisensory responses in the newborn monkey superior colliculus. The Journal of Neuroscience, 21, 8886–8894. Wallace, M. T., & Stein, B. E. (2007). Early experience determines how the senses will interact. Journal of Neurophysiology, 97, 921–926. Wallace, M. T., Wilkinson, L. K., & Stein, B. E. (1996). Representation and integration of multiple sensory inputs in primate superior colliculus. Journal of Neurophysiology, 76, 1246–1266. Wilkinson, L. K., Meredith, M. A., & Stein, B. E. (1996). The role of anterior ectosylvian cortex in cross-modality orientation and approach behavior. Experimental Brain Research, 112, 1–10. Yu, L., Rowland, B. A., & Stein, B. E. (2010). Initiating the development of multisensory integration by manipulating sensory experience. The Journal of Neuroscience, 30, 4904–4913. Yu, L., Stein, B. E., & Rowland, B. A. (2009). Adult plasticity in multisensory neurons: Short-term experience-dependent changes in the superior colliculus. The Journal of Neuroscience, 29, 15910–15922. Zahar, Y., Reches, A., & Gutfreund, Y. (2009). Multisensory enhancement in the optic tectum of the barn owl: Spike count and spike timing. Journal of Neurophysiology, 101, 2380–2394.
A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 191 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.
CHAPTER 11
Multisensory object representation: Insights from studies of vision and touch Simon Lacey{ and K. Sathian{,{,},},* { }
{ Department of Neurology, Emory University, Atlanta, Georgia, USA Department of Rehabilitation Medicine, Emory University, Atlanta, Georgia USA } Department of Psychology, Emory University, Atlanta, Georgia, USA Rehabilitation R&D Center of Excellence, Atlanta VAMC, Decatur, Georgia, USA
Abstract: Behavioral studies show that the unisensory representations underlying within-modal visual and haptic object recognition are strikingly similar in terms of view- and size-sensitivity, and integration of structural and surface properties. However, the basis for these attributes differs in each modality, indicating that while these representations are functionally similar, they are not identical. Imaging studies reveal bisensory, visuo-haptic object selectivity, notably in the lateral occipital complex and the intraparietal sulcus, that suggests a shared representation of objects. Such a multisensory representation could underlie visuo-haptic cross-modal object recognition. In this chapter, we compare visual and haptic within-modal object recognition and trace a progression from functionally similar but separate unisensory representations to a shared multisensory representation underlying cross-modal object recognition as well as view-independence, regardless of modality. We outline, and provide evidence for, a model of multisensory object recognition in which representations are flexibly accessible via top-down or bottom-up processing, the choice of route being influenced by object familiarity and individual preference along the object–spatial continuum of mental imagery. Keywords: haptic; cross-modal; visual imagery; fMRI. by Klatzky, Lederman, and their colleagues (e.g., Klatzky and Lederman, 1995; Klatzky et al., 1985, 1987; Lederman and Klatzky, 1987), there is now a substantial literature on haptic object recognition. The representations underlying visual and haptic within-modal object recognition are strikingly similar: each, for example, is sensitive to changes in orientation, size, and surface properties. However,
Introduction Object recognition research has typically focused on the visual modality but, following pioneering work
*Corresponding author. Tel.: þ1-404-712-1366; Fax: þ1-404-727-3157 E-mail:
[email protected] DOI: 10.1016/B978-0-444-53752-2.00006-0
165
166
these similarities should not be taken to mean that vision and haptics necessarily access only a single, shared representation: the basis for similar representational characteristics differs between modalities. It is a first theme of this chapter that, though functionally similar, within-modal visual and haptic recognition are supported at a basic level by separate, unisensory representations. Vision and haptics do, however, converge on a shared representation in the service of higher-order recognition. We will review studies indicating that a single, shared representation supports within-modal, view-independent recognition in both vision and touch, and also visuo-haptic, cross-modal recognition. A shared representation suggests a shared neural substrate between vision and touch; this and the implications for the nature of the underlying representation constitute a second theme of this chapter. The final part of the chapter links these earlier themes by outlining a model of multisensory object recognition in which visuo-haptic access to multisensory representations is modulated by object familiarity and individual differences on the object–spatial dimension of mental imagery. (a)
View-dependence A major challenge for object recognition is achieving perceptual constancy, which insulates it from potentially disruptive transformations in the sensory input caused by changes in orientation, size, etc. Visual object recognition is viewdependent under certain circumstances, since rotating an object away from its original orientation impairs subsequent recognition (see Peissig and Tarr, 2007, for a review). Although the hands can explore an object from different sides simultaneously and therefore might be expected to be capable of acquiring information about different “views” at the same time, several studies have now shown, counter-intuitively, that haptic object recognition is also view-dependent (Craddock and Lawson, 2008, 2010; Lacey et al., 2007, 2009a; Lawson, 2009; Newell et al., 2001). The extent to which visual recognition is impaired by changes in orientation appears to depend on the axis around which an object is rotated (Gauthier et al., 2002; Lacey et al., 2007). Rotations in depth, about the x- and y-axes (Fig. 1), have more (b)
y-axis x-axis z-axis (c)
(d)
Fig. 1. Example 3D unfamiliar object shown in the original orientation (a) and rotated 180º about the z-axis (b), x-axis (c), and y-axis (d): rotation about the x- and y-axes are rotations in depth, rotation about the z-axis is a rotation in the picture plane. From Lacey et al., 2007.
167
disruptive effects than rotations in the picture plane, about the z-axis, resulting in slower and less accurate visual recognition for both 2D (Gauthier et al., 2002) and 3D stimuli (Lacey et al., 2007). However, haptic recognition is equally impaired by rotation about any axis (Lacey et al., 2007). This suggests that, although visual and haptic recognition are similar in being view-dependent, the basis for this is different in each modality. A possible explanation is that, unless the observer physically changes position relative to an object (e.g., Pasqualotto et al., 2005; Pasqualotto and Newell, 2007), a change in orientation typically means that visual recognition has to contend not only with a transformation in the perceptual shape but also with the occlusion of one or more surfaces. For example, compare Fig. 1a to c: here, rotation about the x-axis means that the object is turned upside-down and that the former top surface becomes occluded. But the hands are free to move over all surfaces of an object, and to manipulate it into different orientations relative to the hand, so that no surface is necessarily occluded in any given orientation. Haptic recognition therefore only has to deal with a shape transformation; thus, no single axis of rotation should be more or less disruptive than another due to surface occlusion. Further work is required to examine whether this explanation is, in fact, correct. The studies reviewed so far have largely concentrated on unfamiliar objects. As objects become more familiar, however, visual recognition becomes view-independent (Bülthoff and Newell, 2006; Tarr and Pinker, 1989). Many familiar objects are typically seen in one particular orientation known as a canonical view, for example, the front view of a house (Palmer et al., 1981). View-independence may hold for a range of orientations around the canonical view, but when objects are presented in radically noncanonical views, for example, an upside-down house, visual recognition can be impaired (Bülthoff and Newell, 2006; Palmer et al., 1981; Tarr and Pinker, 1989). Similarly, haptic
recognition of familiar objects is also view-independent, with unusual, non-canonical orientations incurring a cost in that they are recognized more slowly (Craddock and Lawson, 2008). However, what constitutes a canonical view depends on the modality: visually, a 3/4 view is preferred in which the object is aligned at 45 to the observer (Palmer et al., 1981). But in haptic canonical views, objects are generally aligned either parallel or orthogonal to the body midline (Woods et al., 2008). Canonical views may facilitate viewindependent recognition either because they provide the most structural information about an object or because they most closely match a stored representation, but the end result is the same for both vision and haptics (Craddock and Lawson, 2008; Woods et al., 2008). Haptic representations of familiar objects also maintain object constancy across changes in orientation even where there is a change in the hand used to explore the object (Craddock and Lawson, 2009a). In contrast to within-modal recognition, we found that visuo-haptic cross-modal recognition is view-independent even for unfamiliar objects that are highly similar and lack distinctive parts (see Fig. 1), regardless of the axis of rotation and whether visual study is followed by haptic test or vice versa (Lacey et al., 2007, 2010b). Cross-modal view-independence has also been demonstrated for familiar objects when haptic study was followed by visual test, although recognition was viewdependent in the reverse condition (Lawson, 2009). The reason for this asymmetry is not clear, but the familiar objects employed were a mixture of scale models of larger objects (e.g., bed, bath, and shark) and more or less actual size objects (e.g., jug, pencil). Possibly, some of these objects might have been more familiar visually than haptically, contributing to uncertainty when visually familiar objects had to be recognized by touch. To the best of our knowledge, however, the effect of differential familiarity depending on modality has not been investigated. There are two ways in which cross-modal viewindependence could arise. The simplest possibility
168
is that the view-dependent visual and haptic unisensory representations are directly integrated into a view-independent multisensory representation (Fig. 2a). Alternatively, cross-modal view-independence might be gated through unisensory view-independent representations, separately in vision and haptics (Fig. 2b). We sought to distinguish between these two possibilities in a perceptual learning study (Lacey et al., 2009a). First, we established that view-independence induced by learning in one modality transferred completely and symmetrically to the other; thus, within-modal view-independence, whether visual or haptic, is supported by a single view-independent representation. Second, we addressed whether this representation was the same as that underlying cross-modal (a)
Bisensory (visuo-haptic) View-independent
Unisensory (visual) View-dependent
(b)
Unisensory (haptic) View-dependent
Bisensory (visuo-haptic) View-independent
Unisensory (visual) View-independent
Unisensory (haptic) View-independent
Unisensory (visual) View-dependent
Unisensory (haptic) View-dependent
Fig. 2. Visuo-haptic view-independence might be (a) derived from directly integrating the unisensory view-dependent representations, alternatively (b) unisensory viewindependence might be necessary for bisensory viewindependence. From Lacey et al. (2009a,b).
view-independence. Since both visual and haptic within-modal view-independence were acquired following training on cross-modal object recognition (whether haptic-visual or visual-haptic), we concluded that visuo-haptic view-independence relies on a single multisensory representation that directly integrates the unisensory view-dependent representations (Fig. 2a), similar to models that have been proposed for vision (Riesenhuber and Poggio, 1999). Thus, the same representation underlies both cross-modal recognition and viewindependence, even if view-independence is tested within-modally.
Size-dependence In addition to achieving object constancy across orientation changes, the visual system also has to recognize the physical size of an object across variations in the size of the retinal image that arise from changing object–observer distance: the same object can produce retinal images that vary in size depending on whether it is near to or far from the observer. Presumably, this is compensated by cues arising from depth or motion perception, accounting for the fact that a change in size does not disrupt visual object identification (Biederman and Cooper, 1992; Uttl et al., 2007). However, size change does produce a cost in visual recognition for both unfamiliar (Jolicoeur, 1987) and familiar objects (Jolicoeur, 1987; Uttl et al., 2007). Interestingly, changes in retinal size due to movement of the observer result in better size constancy than those due to movement of the object (Combe and Wexler, 2010). Haptic perception of size is a product of both cutaneous (contact area and force) and proprioceptive (finger spread and position) information at initial contact (Berryman et al., 2006). Integration of these information sources achieves initial size constancy in that we do not perceive the size of an object as changing simply by gripping it harder, which increases contact area, or altering the spread of the fingers (Berryman et al., 2006),
169
thus, in touch, physical size is perceived directly. Thus, both haptic (Craddock and Lawson, 2009b,c) and cross-modal (Craddock and Lawson, 2009c) recognition are size-dependent. Haptic representations may store a canonical size for familiar objects, as has recently been proposed for visual representations (Konkle and Oliva, 2011), and deviations from this could impair recognition. Further work is required to examine this and to investigate perceived object constancy across size changes of unfamiliar objects.
Integration of structural and surface properties Although visual shape, color, and texture are processed in different cortical areas (Cant et al., 2009; Cant and Goodale, 2007), behavioral evidence suggests that visual object representations integrate structural and surface properties because changing the color of an object or its part-color combinations between study and test resulted in longer response times for a shape recognition task (Nicholson and Humphrey, 2003). Since altering the background color against which objects were presented did not impair recognition, this effect could be isolated to the object representation and indicated that this contained information about color as well as shape (Nicholson and Humphrey, 2003). Both visual and haptic within-modal object discrimination are impaired when surface texture is altered, showing first that information about surface properties in visual representations is not limited to modality-specific properties like color, and second that haptic representations also integrate structural and surface properties (Lacey et al., 2010b). However, the question whether surface properties are integrated into the multisensory representation underlying cross-modal object discrimination does not have a straightforward answer. We conducted a study requiring object discrimination across changes in orientation (in order to ensure that participants were accessing the view-independent multisensory
representation), texture, or both. While object discrimination was view-independent when texture did not change, replicating earlier findings, performance reduced to chance levels with a change in texture, whether orientation also changed or not (Lacey et al., 2010b). However, performance was heterogeneous, with some participants being more affected by the texture changes than others. We wondered whether this reflected the recent description of two kinds of visual imagery: “object imagery” (images that are pictorial and deal with the literal appearance of objects in terms of shape, color, brightness, etc.) and “spatial imagery” (more schematic images dealing with the spatial relations of objects, their component parts, and their spatial transformations; Blajenkova et al., 2006; Kozhevnikov et al., 2002, 2005). Both visual and haptic imagery can potentially be subdivided into object imagery dealing with the appearance or feel of objects, and spatial imagery dealing with spatial relationships between objects or between parts of objects. Hence, this object–spatial dimension of imagery might also apply to haptically derived representations. A major difference between object and spatial imagery is that the former includes surface property information while the latter does not. Further analysis of the texture-change condition showed that performance was indeed related to imagery preference such that object imagers were more likely to be impaired by texture changes than spatial imagers; thus, surface properties are likely only integrated into the multisensory representation by object imagers (Lacey et al., 2011). In a follow-up experiment, we asked participants to discriminate shape across changes in texture and texture across changes in shape in both visual and haptic within-modal conditions. As expected, spatial imagers were able to discriminate shape despite concomitant changes in texture but not vice versa, presumably because they abstract away from surface properties. By contrast, object imagers could discriminate texture despite concomitant changes in shape, but not
170
the reverse. Importantly, there was no significant difference between visual and haptic performance on either task. Thus, we concluded that the object–spatial imagery dimension occurs in haptics as well as vision, and that individual variations along this dimension affect the extent to which surface properties are integrated into object representations (Lacey et al., 2011).
Multisensory cortical processing Many studies have shown that visual cortical areas are functionally involved during haptic tasks (reviewed in Amedi et al., 2005; Sathian and Lacey, 2007). The lateral occipital complex (LOC) in the ventral visual pathway responds selectively to objects (Malach et al., 1995) and a subregion responds selectively to objects in both vision and touch (Amedi et al., 2001, 2002; Stilla and Sathian, 2008). Tactile responsiveness in the LOC has been found for both 3D (Amedi et al., 2001; Stilla and Sathian, 2008; Zhang et al., 2004) and 2D stimuli (Prather et al., 2004; Stoesz et al., 2003). The LOC does not respond during conventional auditory object recognition triggered by object-specific sounds (Amedi et al., 2002), but it does respond to shape information created by a visual–auditory sensory substitution device (SSD) (Amedi et al., 2007). SSDs convert visual information into an auditory stream or “soundscape” via a specific algorithm that conveys the visual horizontal axis via auditory time and stereo panning, the visual vertical axis by varying tone frequency, and pixel brightness by varying tone loudness: both sighted and blind humans can learn to recognize objects by extracting shape information from such soundscapes (Amedi et al., 2007). However, for participants trained in the use of the SSD, the LOC only responds to soundscapes created according to the algorithm and not to soundscapes associated arbitrarily with specific objects through learning (Amedi et al., 2007). Thus, LOC can be regarded as processing
geometric shape information, regardless of the sensory modality used to acquire it. Parietal cortical regions also show multisensory shape-selectivity: In particular, the intraparietal sulcus (IPS) is involved in perception of both the shape and location of objects, with coactivation of LOC during shape discrimination and the frontal eye fields during location discrimination (Sathian et al., 2011; Stilla and Sathian, 2008). Visuo-haptic shape selectivity has also been reported in the postcentral sulcus (PCS; Stilla and Sathian, 2008), corresponding to BA2 of primary somatosensory cortex (S1; Grefkes et al., 2001). This is an area that is generally thought to be exclusively somatosensory; however, our observation of multisensory shape selectivity in this region (Stilla and Sathian, 2008) is consistent with earlier neurophysiological studies that suggested visual responsiveness in parts of S1 (Iriki et al., 1996; Zhou and Fuster, 1997). Some case studies suggest that multisensory convergence in the LOC is necessary for both visual and haptic shape perception. For example, one patient with bilateral lesions of the LOC was unable to recognize novel objects by either vision or touch (James et al., 2006) while another, with a lesion to the left occipito-temporal cortex that likely included the LOC, exhibited both tactile and visual agnosia although somatosensory cortex and basic somatosensation were spared (Feinberg et al., 1986). Another patient with a ventrolateral somatosensory lesion showed tactile but not visual agnosia (Reed et al., 1996). These case studies are consistent with the behavioral literature reviewed above, indicating the existence of separate visual and haptic unisensory representations, with evidence for the shared multisensory representation being in the LOC. An important question is whether multisensory responses in the LOC and elsewhere reflect visuo-haptic integration at the neuronal level or separate, interdigitated populations of unisensory neurons receiving information from either visual or haptic inputs. To investigate this, Tal and
171
Amedi (2009) used an fMRI-based adaptation paradigm (fMR-A). fMR-A takes advantage of the repetition suppression effect, that is, attenuation of the blood-oxygen-level-dependent signal when the same stimulus is repeated; since similar repetition suppression is observed in single neurons, this method can be used to reveal neuronal selectivity profiles (reviewed by Grill-Spector et al., 2006; Krekelberg et al., 2006). Robust cross-modal adaptation from vision to touch was observed not only in the LOC and anterior IPS but also bilaterally in the precentral sulcus (corresponding to ventral premotor cortex) and the right anterior insula, suggesting that these areas have multisensory responses at the neuronal level. Multisensory regions which did not show fMR-A included the PCS and posterior parts of the IPS, suggesting that multisensory convergence in these zones arises from separate unisensory populations. Note, however, the concern that fMR-A effects may not necessarily reflect neuronal selectivity (Mur et al., 2010); thus, converging evidence using other methods would be helpful to confirm the conclusions from the study of Tal and Amedi (2009). The cortical localization of the multisensory, view-independent representation is not known. For visual stimuli, the LOC has been reported to show view-dependent responses in some studies and view-independent responses in other studies. As might be expected, view-dependence has been observed for unfamiliar objects (Gauthier et al., 2002) and view-independence for familiar objects (Eger et al., 2008a; Pourtois et al., 2009; Valyear et al., 2006). However, view-dependence has been found in the LOC even for familiar objects in one study, although in this study there was positionindependence (Grill-Spector et al., 1999), whereas another study found view-independence for both familiar and unfamiliar objects (James et al., 2002a). A recent study using transcranial magnetic stimulation (TMS) suggests that the LOC is causally involved in view-independent recognition, at least for 2D shape (Silvanto et al., 2010). However, further work is required to substantiate this since
only two rotations (20 and 70 ) were tested and TMS effects were only observed for the smaller rotation. View-dependent responses to visual stimuli have been reported in parietal regions, for example, the IPS (James et al., 2002a) and parieto-occipital junction (PO; Valyear et al., 2006). This is perhaps not surprising since these regions are in the dorsal pathway, which is concerned more with object location and sensory processing for action rather than object identity and sensory processing for perceptual recognition (Goodale and Milner, 1992; Ungerleider and Mishkin, 1982). Changes in orientation might be expected to affect associated actions, and indeed, the lateral PO demonstrates view-dependent responses for graspable, but not for nongraspable objects (Rice et al., 2007). Superior parietal cortex exhibits view-dependent responses during mental rotation but not during visual object recognition (Gauthier et al., 2002). Haptic and multisensory processing of stimuli across changes in orientation have not been examined in regard to cortical responses. Although James et al. (2002b) varied object orientation, their study concentrated on haptic-to-visual priming effects rather than the cross-modal response to same versus different orientations. It will be interesting to examine the effect of orientation changes when shape information is derived from the auditory soundscapes produced by SSDs and also when orientation changes alter the affordances and possibilities for haptically interacting with an object. Although haptic and multisensory processing of stimuli across changes in size have also not been investigated, visual size-independence has been consistently observed in the LOC (Eger et al., 2008a,b; Ewbank et al., 2005; Grill-Spector et al., 1999), with anterior regions showing more size-independence than posterior regions (Eger et al., 2008b; Sawamura et al., 2005). What role does visual imagery play? Haptic activation of visual cortex might arise either because haptic exploration of an object
172
evokes visual imagery (Sathian et al., 1997), presumably processed in the LOC, or because the LOC can be directly activated by somatosensory input. A recent electrophysiological study of tactile discrimination of simple geometric shapes applied to the fingerpad shows that activity propagates from somatosensory cortex to LOC as early as 150 ms after stimulus onset, a time frame which is consistent with “bottom-up” somatosensory projections to the LOC (Lucan et al., 2010). In addition, a recent case study examined a patient with visual agnosia arising from bilateral ventral occipito-temporal lesions, but with sparing of the dorsal part of the LOC that likely included the multisensory subregion (Allen and Humphreys, 2009). This patient's haptic object recognition was intact and was associated with activation of the intact dorsal part of the LOC, suggesting that this region can be activated directly by somatosensory input (Allen and Humphreys, 2009). Consistent with the visual imagery hypothesis, however, many studies have demonstrated LOC activity during visual imagery. For example, left LOC activity was observed in both blind and sighted participants during auditorily cued mental imagery of familiar object shape, where shape information would stem mainly from haptic experience in the case of the blind and mainly from visual experience in the sighted (De Volder et al., 2001). The left LOC is also active during a task requiring retrieval of geometric and material object properties from memory (Newman et al., 2005). In the right LOC, individual differences in haptic shape-selective activation magnitude were highly correlated with individual differences in ratings of visual imagery vividness (Zhang et al., 2004). By contrast, a lesser role for visual imagery has been suggested because LOC activity was substantially lower during visual imagery compared to haptic shape perception (Amedi et al., 2001), although there was no attempt to verify that participants maintained their images online during the imaging session. Some researchers have concluded that visual imagery does not explain
haptically evoked LOC activity because early- as well as late-blind individuals show shape-related LOC activation via both touch (reviewed in Pascual-Leone et al., 2005; Sathian, 2005; Sathian and Lacey, 2007) and hearing using SSDs (Amedi et al., 2007; Arno et al., 2001; Renier et al., 2004, 2005). While this argument is clearly true for the early blind, it does not necessarily exclude visual imagery as an explanation in the sighted, given the extensive evidence for cross-modal plasticity demonstrated in studies of visual deprivation (reviewed in Pascual-Leone et al., 2005; Sathian, 2005; Sathian and Lacey, 2007). The weight of the evidence is therefore that visual imagery is likely involved; in the next section, we show that this involvement depends on an interaction with object familiarity.
A model of multisensory object representation In this section, we draw together some of the threads reviewed earlier by outlining, and reviewing the evidence for, a preliminary conceptual model of visuo-haptic multisensory object representation that we detailed previously (Lacey et al., 2009b). In this model, the LOC contains a representation of object form that can be flexibly accessed either bottom-up or top-down, independently of the input modality, but with the choice of the bottom-up versus top-down route depending on object familiarity. For haptic recognition of unfamiliar objects, global shape has to be computed by exploring the entire object and relating the component parts to one another. The model therefore incorporates a bottom-up pathway from somatosensory cortex to the LOC, together with recruitment of the IPS to compute the spatial relationships between component parts and arrive at global shape, facilitated by spatial imagery processes. For haptic recognition of familiar objects, global shape can be computed more easily, perhaps from only a subset of parts or one diagnostic part, and we suggest that haptic sensing rapidly acquires enough information to
173
trigger a visual image of the object and generate a hypothesis about its identity, as has been proposed for visual sensing (e.g., Bar, 2007). The model therefore calls for top-down processing from prefrontal cortex into the LOC, associated with object imagery processes (this does not, however, exclude spatial imagery for familiar objects, e.g., in enabling view-independent recognition). In a pair of recent papers that provide evidence for this model, we examined the roles of visual object imagery and object familiarity in haptic shape perception using analyses of correlations of activation magnitude between visual object imagery and haptic shape perception (Lacey et al., 2010a) and analyses of effective connectivity (Deshpande et al., 2010). In the imagery task, participants heard pairs of words and decided whether the objects represented by those words had similar or different shapes. In contrast to previous studies, this ensured that participants were engaging in visual imagery throughout the scan, verifiable by reference to their recorded task performance. In a separate session, participants performed a haptic shape discrimination task using either familiar or unfamiliar objects. If haptic perception of shape depends on visual object imagery, we expected that activation magnitudes during the imagery task would be correlated with activation magnitudes during haptic perception of shape, with effective connectivity analyses showing similar top-down networks for object imagery and haptic shape perception. However, a lack of correlation between activation magnitudes for visual object imagery and haptic shape perception, and different networks in the two tasks reflecting bottom-up paths for haptic shape perception but top-down paths for imagery, would argue against imagery mediation of haptic shape perception. We found that object familiarity modulated both intertask correlations and effective connectivity. Visual object imagery and both haptic shape perception tasks activated the LOC bilaterally; familiar but not unfamiliar haptic shape perception recruited prefrontal cortical areas. Importantly, imagery activation
magnitudes in the LOC correlated with those for haptic shape perception for familiar, but not unfamiliar objects, and there were more regions showing such intertask correlations with imagery for the familiar compared to the unfamiliar shape task (Lacey et al., 2010a). The effective connectivity analyses showed similar networks for visual object imagery and haptic perception of familiar shape, dominated by top-down connections from prefrontal cortical regions into the LOC (Deshpande et al., 2010). Haptic perception of unfamiliar shape engaged a network that was very different from either of these; consistent with earlier connectivity analyses (Deshpande et al., 2008; Peltier et al., 2007), this network showed mainly bottom-up connections to the LOC from somatosensory cortex (PCS) (Deshpande et al., 2010). Thus, we have evidence for the first part of the model; in ongoing work, we are investigating the interaction between object familiarity and spatial imagery in order to provide similar evidence for the second part.
Summary In this chapter, we have shown that visual and haptic within-modal object recognition initially rely on separate representations that are nonetheless functionally similar in being view- and size-dependent. Further work is required to investigate the different mechanisms by which these similarities arise in each modality. These unisensory representations feed forward into a multisensory, view-independent representation that supports cross-modal object recognition. Here, further work is needed to examine the neural basis of both haptic and cross-modal view-independence. Finally, cross-modal object recognition depends on complex interactions between modalities, object and spatial dimensions of imagery, and object familiarity. A fruitful avenue for future research will be to examine how these interactions differ between sighted and blind individuals.
174
Acknowledgments Support to K. S. from the National Eye Institute at the NIH, the National Science Foundation, and the Veterans Administration is gratefully acknowledged. References Allen, H. A., & Humphreys, G. W. (2009). Direct tactile stimulation of dorsal occipito-temporal cortex in a visual agnosic. Current Biology, 19, 1044–1049. Amedi, A., Jacobson, G., Hendler, T., Malach, R., & Zohary, E. (2002). Convergence of visual and tactile shape processing in the human lateral occipital complex. Cerebral Cortex, 12, 1202–1212. Amedi, A., Malach, R., Hendler, T., Peled, S., & Zohary, E. (2001). Visuo-haptic object-related activation in the ventral visual pathway. Nature Neuroscience, 4, 324–330. Amedi, A., Stern, W. M., Camprodon, J. A., Bermpohl, F., Merabet, L., Rotman, S., et al. (2007). Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex. Nature Neuroscience, 10, 687–689. Amedi, A., von Kriegstein, K., van Atteveldt, N. M., Beauchamp, M. S., & Naumer, M. J. (2005). Functional imaging of human crossmodal identification and object recognition. Experimental Brain Research, 166, 559–571. Arno, P., De Volder, A. G., Vanlierde, A., WanetDefalque, M.-C., Streel, E., Robert, A., et al. (2001). Occipital activation by pattern recognition in the early blind using auditory substitution for vision. Neuroimage, 13, 632–645. Bar, M. (2007). The proactive brain: Using analogies and associations to generate predictions. Trends in Cognitive Sciences, 11, 280–289. Berryman, L. J., Yau, J. M., & Hsiao, S. S. (2006). Representation of object size in the somatosensory system. Journal of Neurophysiology, 96, 27–39. Biederman, I., & Cooper, E. E. (1992). Size invariance in visual object priming. Journal of Experimental Psychology: Human Perception and Performance, 18, 121–133. Blajenkova, O., Kozhevnikov, M., & Motes, M. A. (2006). Object-spatial imagery: A new self-report imagery questionnaire. Applied Cognitive Psychology, 20, 239–263. Bülthoff, I., & Newell, F. N. (2006). The role of familiarity in the recognition of static and dynamic objects. Progress in Brain Research, 154, 315–325. Cant, J. S., Arnott, S. R., & Goodale, M. A. (2009). fMR-adaptation reveals separate processing regions for the perception of form and texture in the human ventral stream. Experimental Brain Research, 192, 391–405.
Cant, J. S., & Goodale, M. A. (2007). Attention to form or surface properties modulates different regions of human occipitotemporal cortex. Cerebral Cortex, 17, 713–731. Combe, E., & Wexler, M. (2010). Observer movement and size constancy. Psychological Science, 21, 667–675. Craddock, M., & Lawson, R. (2008). Repetition priming and the haptic recognition of familiar and unfamiliar objects. Perception and Psychophysics, 70, 1350–1365. Craddock, M., & Lawson, R. (2009a). Do left and right matter for haptic recognition of familiar objects? Perception, 38, 1355–1376. Craddock, M., & Lawson, R. (2009b). The effect of size changes on haptic object recognition. Attention, Perception and Psychophysics, 71, 910–923. Craddock, M., & Lawson, R. (2009c). Size-sensitive perceptual representations underlie visual and haptic object recognition. PLoS ONE, 4, e8009. doi:10.1371/journal.pone.0008009. Craddock, M., & Lawson, R. (2010). The effects of temporal delay and orientation on haptic object recognition. Attention, Perception and Psychophysics, 72, 1975–1980. De Volder, A. G., Toyama, H., Kimura, Y., Kiyosawa, M., Nakano, H., Vanlierde, A., et al. (2001). Auditory triggered mental imagery of shape involves visual association areas in early blind humans. Neuroimage, 14, 129–139. Deshpande, G., Hu, X., Lacey, S., Stilla, R., & Sathian, K. (2010). Object familiarity modulates effective connectivity during haptic shape perception. Neuroimage, 49, 1991–2000. Deshpande, G., Hu, X., Stilla, R., & Sathian, K. (2008). Effective connectivity during haptic perception: A study using Granger causality analysis of functional magnetic resonance imaging data. Neuroimage, 40, 1807–1814. Eger, E., Ashburner, J., Haynes, J.-D., Dolan, R. J., & Rees, G. (2008a). fMRI activity patterns in human LOC carry information about object exemplars within category. Journal of Cognitive Neuroscience, 20, 356–370. Eger, E., Kell, C. A., & Kleinschmidt, A. (2008b). Graded size-sensitivity of object-exemplar-evoked activity patterns within human LOC regions. Journal of Neurophysiology, 100, 2038–2047. Ewbank, M. P., Schluppeck, D., & Andrews, T. J. (2005). fMR-adaptation reveals a distributed representation of inanimate objects and places in human visual cortex. Neuroimage, 28, 268–279. Feinberg, T. E., Rothi, L. J., & Heilman, K. M. (1986). Multimodal agnosia after unilateral left hemisphere lesion. Neurology, 36, 864–867. Gauthier, I., Hayward, W. G., Tarr, M. J., Anderson, A. W., Skudlarski, P., & Gore, J. C. (2002). BOLD activity during mental rotation and view-dependent object recognition. Neuron, 34, 161–171.
175 Goodale, M. A., & Milner, A. D. (1992). Separate visual pathways for perception and action. Trends in Neuroscience, 15, 20–25. Grefkes, C., Geyer, S., Schormann, T., Roland, P., & Zilles, K. (2001). Human somatosensory area 2: Observer-independent cytoarchitectonic mapping, interindividual variability, and population map. Neuroimage, 14, 617–631. Grill-Spector, K., Henson, R., & Martin, A. (2006). Repetition and the brain: Neural models of stimulus-specific effects. Trends in Cognitive Sciences, 10, 14–23. Grill-Spector, K., Kushnir, T., Edelman, S., Avidan, G., Itzchak, Y., & Malach, R. (1999). Differential processing of objects under various viewing conditions in the human lateral occipital complex. Neuron, 24, 187–203. Iriki, A., Tanaka, M., & Iwamura, Y. (1996). Attentioninduced neuronal activity in the monkey somatosensory cortex revealed by pupillometrics. Neuroscience Research, 25, 173–181. James, T. W., Humphrey, G. K., Gati, J. S., Menon, R. S., & Goodale, M. A. (2002a). Differential effects of view on object-driven activation in dorsal and ventral streams. Neuron, 35, 793–801. James, T. W., Humphrey, G. K., Gati, J. S., Servos, P., Menon, R. S., & Goodale, M. A. (2002b). Haptic study of three-dimensional objects activates extrastriate visual areas. Neuropsychologia, 40, 1706–1714. James, T. W., James, K. H., Humphrey, G. K., & Goodale, M. A. (2006). Do visual and tactile object representations share the same neural substrate? In M. A. Heller & S. Ballesteros (Eds.), Touch and blindness: Psychology and neuroscience (pp. 139–155). Mahwah, NJ: Lawrence Erlbaum Associates. Jolicoeur, P. (1987). A size-congruency effect in memory for visual shape. Memory and Cognition, 15, 531–543. Klatzky, R. L., & Lederman, S. J. (1995). Identifying objects from a haptic glance. Perception and Psychophysics, 57, 1111–1123. Klatzky, R. L., Lederman, S. J., & Metzger, V. A. (1985). Identifying objects by touch: An ‘expert system’. Perception and Psychophysics, 37, 299–302. Klatzky, R. L., Lederman, S. J., & Reed, C. L. (1987). There's more to touch than meets the eye: The salience of object attributes for haptics with and without vision. Journal of Experimental Psychology: General, 116, 356–369. Konkle, T., & Oliva, A. (2011). Canonical visual size for realworld objects. Journal of Experimental Psychology: Human Perception and Performance, 37, 23–37. Kozhevnikov, M., Hegarty, M., & Mayer, R. E. (2002). Revising the visualiser-verbaliser dimension: Evidence for two types of visualisers. Cognition and Instruction, 20, 47–77. Kozhevnikov, M., Kosslyn, S. M., & Shephard, J. (2005). Spatial versus object visualisers: A new characterisation of cognitive style. Memory and Cognition, 33, 710–726.
Krekelberg, B., Boynton, G. M., & van Wezel, R. J. A. (2006). Adaptation: From single cells to BOLD signals. Trends in Neurosciences, 29, 250–256. Lacey, S., Flueckiger, P., Stilla, R., Lava, M., & Sathian, K. (2010a). Object familiarity modulates the relationship between visual object imagery and haptic shape perception. Neuroimage, 49, 1977–1990. Lacey, S., Hall, J., & Sathian, K. (2010b). Are surface properties integrated into visuo-haptic object representations? The European Journal of Neuroscience, 31, 1882–1888. Lacey, S., Lin, J. B., & Sathian, K. (2011). Object and spatial imagery dimensions in visuo-haptic representations. Experimental Brain Research doi: 10.1007/s00221-011-2623-1 (in press). Lacey, S., Pappas, M., Kreps, A., Lee, K., & Sathian, K. (2009a). Perceptual learning of view-independence in visuo-haptic object representations. Experimental Brain Research, 198, 329–337. Lacey, S., Peters, A., & Sathian, K. (2007). Cross-modal object representation is viewpoint-independent. PLoS ONE, 2(9), e890. doi:10.1371/journal.pone0000890. Lacey, S., Tal, N., Amedi, A., & Sathian, K. (2009b). A putative model of multisensory object representation. Brain Topography, 21, 269–274. Lawson, R. (2009). A comparison of the effects of depth rotation on visual and haptic three-dimensional object recognition. Journal of Experimental Psychology: Human Perception and Performance, 35, 911–930. Lederman, S. J., & Klatzky, R. L. (1987). Hand movements: A window into haptic object recognition. Cognitive Psychology, 19, 342–368. Lucan, J. N., Foxe, J. J., Gomez-Ramirez, M., Sathian, K., & Molholm, S. (2010). Tactile shape discrimination recruits human lateral occipital complex during early perceptual processing. Human Brain Mapping, 31, 1813–1821. Malach, R., Reppas, J. B., Benson, R. R., Kwong, K. K., Jiang, H., Kennedy, W. A., et al. (1995). Object-related activity revealed by functional magnetic resonance imaging in human occipital cortex. Proceedings of the National Academy of Science of the United States of America, 92, 8135–8139. Mur, M., Ruff, D. A., Bodurka, J., Bandettini, P. A., & Kriegeskorte, N. (2010). Face-identity change activation outside the face system: “Release from adaptation” may not always indicate neuronal selectivity. Cerebral Cortex, 20, 2027–2042. Newell, F. N., Ernst, M. O., Tjan, B. S., & Bülthoff, H. H. (2001). View dependence in visual and haptic object recognition. Psychological Science, 12, 37–42. Newman, S. D., Klatzky, R. L., Lederman, S. J., & Just, M. A. (2005). Imagining material versus geometric properties of objects: An fMRI study. Cognitive Brain Research, 23, 235–246.
176 Nicholson, K. G., & Humphrey, G. K. (2003). The effect of colour congruency on shape discriminations of novel objects. Perception, 32, 339–353. Palmer, S. E., Rosch, E., & Chase, P. (1981). Canonical perspective and the perception of objects. In J. Long & A. Baddeley (Eds.), Attention and performance IX (pp. 135–151). Hillsdale, NJ: Lawrence Erlbaum Associates. Pascual-Leone, A., Amedi, A., Fregni, F., & Merabet, L. B. (2005). The plastic human brain. Annual Review of Neuroscience, 28, 377–401. Pasqualotto, A., Finucane, C., & Newell, F. N. (2005). Visual and haptic representations of scenes are updated with observer movement. Experimental Brain Research, 166, 481–488. Pasqualotto, A., & Newell, F. N. (2007). The role of visual experience on the representation and updating of novel haptic scenes. Brain and Cognition, 65, 184–194. Peissig, J. J., & Tarr, M. J. (2007). Visual object recognition: Do we know more now than we did 20 years ago? Annual Review of Psychology, 58, 75–96. Peltier, S., Stilla, R., Mariola, E., LaConte, S., Hu, X., & Sathian, K. (2007). Activity and effective connectivity of parietal and occipital cortical regions during haptic shape perception. Neuropsychologia, 45, 476–483. Pourtois, G., Schwarz, S., Spiridon, M., Martuzzi, R., & Vuilleumier, P. (2009). Object representations for multiple visual categories overlap in lateral occipital and medial fusiform cortex. Cerebral Cortex, 19, 1806–1819. Prather, S. C., Votaw, J. R., & Sathian, K. (2004). Task-specific recruitment of dorsal and ventral visual areas during tactile perception. Neuropsychologia, 42, 1079–1087. Reed, C. L., Caselli, R. J., & Farah, M. J. (1996). Tactile agnosia: Underlying impairment and implications for normal tactile object recognition. Brain, 119, 875–888. Renier, L., Collignon, O., Poirier, C., Tranduy, D., Vanlierde, A., Bol, A., et al. (2005). Cross modal activation of visual cortex during depth perception using auditory substitution of vision. Neuroimage, 26, 573–580. Renier, L., Collignon, O., Tranduy, D., Poirier, C., Vanlierde, A., Veraart, C., et al. (2004). Visual cortex activation in early blind and sighted subjects using an auditory visual substitutiondevicetoperceivedepth.Neuroimage, 22, S1. Rice, N. J., Valyear, K. F., Goodale, M. A., Milner, A. D., & Culham, J. C. (2007). Orientation sensitivity to graspable objects: An fMRI adaptation study. Neuroimage, 36, T87–T93. Riesenhuber, M., & Poggio, T. (1999). Hierarchical models of object recognition in cortex. Nature Neuroscience, 2, 1019–1025. Sathian, K. (2005). Visual cortical activity during tactile perception in the sighted and the visually deprived. Developmental Psychobiology, 46, 279–286. Sathian, K., & Lacey, S. (2007). Journeying beyond classical somatosensory cortex. Canadian Journal of Experimental Psychology, 61, 254–264.
Sathian, K., Lacey, S., Stilla, R., Gibson, G. O., Deshpande, G., Hu, X., LaConte, S. & Glielmi, C. (2011). Dual pathways for haptic and visual perception of spatial and texture information. NeuroImage, doi: 10.1016/j.neuroimage.2011.05.011 (in press). Sathian, K., Zangaladze, A., Hoffman, J. M., & Grafton, S. T. (1997). Feeling with the mind's eye. Neuroreport, 8, 3877–3881. Sawamura, H., Georgieva, S., Vogels, R., Vanduffel, W., & Orban, G. A. (2005). Using functional magnetic resonance imaging to assess adaptation and size invariance of shape processing by humans and monkeys. The Journal of Neuroscience, 25, 4294–4306. Silvanto, J., Schwarzkopf, D. S., Gilaie-Dotan, S., & Rees, G. (2010). Differing causal roles for lateral occipital complex and occipital face area in invariant shape recognition. The European Journal of Neuroscience, 32, 165–171. Stilla, R., & Sathian, K. (2008). Selective visuo-haptic processing of shape and texture. Human Brain Mapping, 29, 1123–1138. Stoesz, M., Zhang, M., Weisser, V. D., Prather, S. C., Mao, H., & Sathian, K. (2003). Neural networks active during tactile form perception: Common and differential activity during macrospatial and microspatial tasks. International Journal of Psychophysiology, 50, 41–49. Tal, N., & Amedi, A. (2009). Multisensory visual-tactile object related network in humans: Insights gained using a novel crossmodal adaptation approach. Experimental Brain Research, 198, 165–182. Tarr, M. J., & Pinker, S. (1989). Mental rotation and orientation dependence in shape recognition. Cognitive Psychology, 21, 233–282. Ungerleider, L. G., & Mishkin, M. (1982). Two cortical visual systems. In D. J. Ingle, M. A. Goodale & R. J. W. Mansfield (Eds.), Analysis of visual behavior (pp. 549–586). Cambridge, MA: MIT Press. Uttl, B., Graf, P., & Siegenthaler, A. L. (2007). Influence of object size on baseline identification, priming, and explicit memory. Scandinavian Journal of Psychology, 48, 281–288. Valyear, K. F., Culham, J. C., Sharif, N., Westwood, D., & Goodale, M. A. (2006). A double dissociation between sensitivity to changes in object identity and object orientation in the ventral and dorsal streams: A human fMRI study. Neuropsychologia, 44, 218–228. Woods, A. T., Moore, A., & Newell, F. N. (2008). Canonical views in haptic object perception. Perception, 37, 1867–1878. Zhang, M., Weisser, V. D., Stilla, R., Prather, S. C., & Sathian, K. (2004). Multisensory cortical processing of object shape and its relation to mental imagery. Cognitive, Affective and Behavioral Neuroscience, 4, 251–259. Zhou, Y.-D., & Fuster, J. M. (1997). Neuronal activity of somatosensory cortex in a cross-modal (visuo-haptic) memory task. Experimental Brain Research, 116, 551–555.
A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 191 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.
CHAPTER 12
Adaptation and maladaptation: insights from brain plasticity Elena Nava* and Brigitte Röder Department of Biological Psychology and Neuropsychology, University of Hamburg, Hamburg, Germany
Abstract: Evolutionary concepts such as adaptation and maladaptation have been used by neuroscientists to explain brain properties and mechanisms. In particular, one of the most compelling characteristics of the brain, known as neuroplasticity, denotes the ability of the brain to continuously adapt its functional and structural organization to changing requirements. Although brain plasticity has evolved to favor adaptation, there are cases in which the same mechanisms underlying adaptive plasticity can turn into maladaptive changes. Here, we will consider brain plasticity and its functional and structural consequences from an evolutionary perspective, discussing cases of adaptive and maladaptive plasticity and using examples from typical and atypical development. Keywords: crossmodal plasticity; maladaptive plasticity; phantom limb pain; tinnitus; cochlear implants; evolution. Lessons from evolution
brain, and its particular adaptive properties, into a broader evolutionary perspective, according to which some structural and functional properties of an individual's brain are considered to be the result of natural selection. In this context, we will discuss the capacity of the brain to change its functional and structural organization (called plasticity or neuroplasticity) and particularly the resulting beneficial (adaptive) as well as possible detrimental (maladaptive) outcomes. Adaptation defines a dynamic process in structure, function, and behavior by which a species or individual improves its chance of survival in a specific environment as a result of natural
A number of terms used to characterize the evolutionary process have also been adopted by neuroscientists to define brain mechanisms, processes, and abilities. The following short definitions of fundamental evolutionary terms will aid in understanding the context which “inspired” neuroscientists in defining their own terms. In drawing some parallels between these commonly adopted terms, our attempt will be to put the *Corresponding author. Tel.: þ49-40-428385838; Fax: þ49-40-428386591 E-mail:
[email protected] DOI: 10.1016/B978-0-444-53752-2.00005-9
177
178
selection. While the term adaptation speaks for the evolutionary process, an adaptive trait is an aspect of the developmental pattern of the organism that enables or enhances the probability of that organism to survive and reproduce during certain stages of the lifespan (Dobzhansky, 1956). Adaptation became the root concept of Darwin's theory (1859), in that it provided the mechanism to explain why things change in the course of time, and how these affect all aspects of the life of an organism. Natural selection acts on phenotypes (i.e., an observable trait of an organism, which includes physiological as well as behavioral changes), and a particular trait will survive if best suited to the environment. Most importantly, though, only a change in genotype (i.e., the complete set of genes within an organism) will define evolution. Natural selection typically produces fitness, a commonly used but nonetheless controversial term that describes how successful an organism has been at passing its genes. Adaptive traits have continuously evolved as a response to environmental demands. The mechanism underpinning all environmentally induced phenotypic variations is called phenotypic plasticity (Via et al., 1995). This mechanism allows a single genotype to produce more than one response (in terms of morphology, physiological state, etc.) to environmental changes, including learned behaviors as well as reaction to diseases. When an organism produces a phenotype that can continuously change as a function of environmental change (e.g., the ability of the marine snail to increase shell thickness in response to new predators; see Trussell and Smith, 2000), the relationship between these two is termed reaction norm. These reactions can be flexible or more inflexible with the term flexible indicating the ability of the phenotypic trait to change throughout the organism's lifespan. In contrast, the term inflexible indicates an inability to change so that any determined characteristic remains fixed. Phenotypic plasticity likely evolved to allow different organisms a greater chance of survival in their
ever-changing surroundings. Finally, it is as a result of plasticity that the environment directly influences which phenotypes are exposed to selection. In our view, brain plasticity can be seen as an example of phenotypic plasticity. In particular, its many possible outcomes can be seen as phenotypes that react to the environmental changes. Changes in behavior occur at an ontogenetic level, but plasticity itself may have evolved phylogenetically. At the same time, the importance of phenotypic plasticity in driving genetic evolution (Price et al., 2003) suggests the importance of considering brain plasticity within the larger framework of evolutionary processes. The vision from the brain The term plasticity, as is true of most scientific terms, has undergone debates and revisions for the past 100 years (Berlucchi and Buchtel, 2009). In his seminal paper entitled “Réflexions sur l'usage du concept de plasticité en neurobiologie,” Paillard (1976; see Will et al., 2008 for the English translation and commentaries) stated that not every change in the neural system should be considered plastic. Only those resulting from a structural and functional change should be considered as such. Also, structural and functional changes should be long-lasting and not transient events (to distinguish plasticity from “elasticity”). Finally, only changes resulting from an adaptation of the system to environmental pressures should be considered plastic, therefore excluding those mechanisms responsible for the “natural” maturation of the early developing system. Recently, Lövdén et al. (2010), presenting a new theoretical framework for the study of adult plasticity and inspired by Paillard's ideas, has proposed that plasticity occurs as a consequence of a prolonged mismatch between supply (i.e., the actual capacities of the brain resulting from biological constraints and environmental influences) and environmental demands. Plasticity is then the ability of the brain to react to this mismatch
179
undergoing anatomical as well as functional changes to best fit an adaptive demand. In this view, the resulting structural and functional change that accompanies plasticity can be seen as a phenotypic plastic change. Adaptation In referring to brain mechanisms, adaptation commonly refers not only to plasticity, which is the capacity of the brain to change to suit external environmental as well as inner changes, but also to any experience acquired throughout development (for reviews, see Kolb et al., 2003; PascualLeone et al., 2005). Adaptive plasticity is also known as experiencedependent plasticity (Greenough et al., 1987). This type of plasticity refers to the ability of the brain to learn throughout its lifespan by means of processes involving structural and functional changes. Although experience-dependent plasticity refers to the ability to learn any new perceptual, motor, or cognitive skill, a particularly spectacular example is provided by musicians, whose extensive practice on a particular task (i.e., playing an instrument) has been shown to modify tactile, motor, and auditory brain regions (for reviews, see Johansson, 2006; Münte et al., 2002). Most of these studies were conducted on adult musicians, leaving the question of whether these structural brain changes could be innate (therefore predisposing the individual to learn music) or acquired through training (i.e., “real” plastic adaptation of the brain to the greater use of particular regions). Recently, some studies (Hyde et al., 2009; Moreno et al., 2009) have precisely addressed this question by investigating structural brain and behavioral changes in children trained on music skills compared to nontrained children. Hyde et al. (2009) trained fifteen 6-year-old children for 15 months on playing the keyboard, while the control group consisted of age-matched children who only had weekly music classes in school. Both groups were tested on behavioral
tasks as well as scanned with MRI before and after training. Results showed that trained children had increased activity in motor hand areas and primary auditory areas compared to controls, which correlated with behavioral improvements on motor and auditory-musical tasks. The fact that no structural brain difference was found between the two groups before training strongly suggests that training itself triggers adaptive changes. Although studies on adults and children have not directly tested whether these plastic changes can persist longer in life even if musical training is suspended, there may be a sensitive period (which refers to the limited period during development in which effects of experience are particularly strong in shaping the brain, see Knudsen, 2004) in childhood in which musical practice may result in long-lasting benefits in performance later in life. For example, brain-imaging studies highlighting plastic changes occurring as a consequence of musical training have found that the degree of these changes appears to decrease as a function of age, so that musical training experienced very early in life triggers larger plastic changes (Elbert et al., 1995). Given the particular nature of early developmental plasticity (Greenough et al., 1987; Knudsen, 2004), it could be hypothesized that musical training early in life changes the brain structurally and functionally in a hierarchical and long-lasting fashion. Although only investigated by means of a behavioral task, Watanabe et al. (2007) addressed this question by comparing performance of two groups of adults who started their musical training at different ages: early (i.e., before 7 years of age) or late (i.e., after 7 years of age). Participants of the two groups were matched for years of musical experience and practice, so that they only differed in the age when training began. The task consisted in learning to reproduce a temporally complex motor sequence by tapping in synchrony with sequentially presented visual stimuli. Results showed that early-trained musicians had an overall better performance compared to late-trained
180
musicians, suggesting that musical training started early in life (i.e., during sensitive periods) can have long-term effects on the ability to learn novel motor tasks. While the case of musicians speaks for the ability of the typically developing brain to change as a function of increased demand, there are cases in which changes in supply (i.e., the brain) cause plasticity to take place to functionally adapt to the new environment. In other words, in the case of direct or indirect brain insult (i.e., brain lesions or sensory loss, respectively), plasticity will act to reorganize the brain. In particular, plastic changes after sensory deafferentation (i.e., blindness, deafness) trigger the system to reorganize in a compensatory fashion to enable sensory-deprived individuals to better suit new environmental pressures. The following section will discuss this particular type of plasticity mechanism, which we will compare to an evolutionary concept known as exaptation. Crossmodal plasticity after sensory deafferentation: a case of exaptation? Exaptation refers to the shifts in functions of a trait during evolution, so that one trait originally serving a particular function may evolve and serve another one, achieving complete fitness for that trait (Gould, 1991; Gould and Lewontin, 1979). The classical example is bird feathers, which initially evolved for temperature regulation and only later were adapted for flight. Moreover, Gould (1991) suggested that there are two types of exaptation. The first type characterizes features that evolved by natural selection to process one function but are then co-opted for another function (i.e., the example of the bird's feathers); the second type refers to features that did not evolve as adaptations through natural selection but are rather side effects of adaptive processes, features that Gould defined spandrels. Arguing against the rigidity of concepts such as adaptation and natural selection, which cannot fully explain the complexity of some human
behaviors, he described the concept of spandrels making a parallel from the architectural spandrels present in the church of San Marco in Venice: “Every fan-vaulted ceiling must have a series of open spaces along the midline of the vault, where the sides of the fans intersect between the pillars. As the spaces must exist, they are often used for ingenious ornamental effect.” In other words, those spaces between vaults, which originally had purely structural functions, ended up being used to enhance esthetic characteristics (i.e., a by-product of their original function).
Spandrels in the brain The term exaptation, if considered in its conceptual form, well suits a particular type of plasticity called crossmodal plasticity. The term crossmodal plasticity has been adopted particularly when describing compensatory plasticity that emerges in some cases of sensory deprivation, such as blindness and profound deafness (Bavelier and Neville, 2002; Pascual-Leone et al., 2005). In particular, some studies have suggested that the absence of the stream of information coming from one sensory modality causes the brain to reorganize in a crossmodal fashion, so that the deafferented cortex responds to input coming from the intact sensory modalities. These types of changes have been also called intermodal changes in both animal (Rauschecker, 1995) and human studies (Röder et al., 1999) because of their “between-senses” interactions. In this view, intermodal changes share commonalities with the concept of exaptation, in that regions subserving the deafferented modality take over new functions originally exclusively mediated by other brain areas. Specifically, a subset of the neurons that are usually responsive to a particular stimulation in a region of the brain will now respond to stimulation of another modality or in the context of a new function. The rationale behind drawing parallels between crossmodal plasticity after sensory deprivation and the concept of exaptation is that the former
181
has been enthusiastically advanced as the mechanism responsible for the enhanced performance found, for example, in tactile and auditory tasks in blind individuals (Amedi et al., 2010; Gougoux et al., 2005; Röder et al., 1999). However, this explanation has been challenged by the diversity of tasks eliciting visual cortex activation after congenital blindness (differing in modality and complexity, see, e.g., Pavani and Röder, 2011) and by studies that found similar crossmodal activity in sighted individuals blindfolded a few days only (Pascual-Leone and Hamilton, 2001), suggesting that this process may not exclusively emerge as a consequence of early sensory deprivation per se. Where does the idea of functional crossmodal plasticity come from? Around 20 years ago, animal studies began to address the question of whether the functional properties of cortical tissue are determined by the inputs they receive rather than being innate. In these experiments, input from one sensory modality was rerouted to the primary cortex of another modality (Sur et al., 1990; von Melchner et al., 2000). For instance, Sur et al. (1990) rerouted retinal axons of newborn ferrets into the auditory pathway by removing ascending auditory projections through deafferentation of the medial geniculate nucleus (MGN) (and by removing the visual cortical targets by ablating visual cortex). This caused retinal fibers to innervate the MGN, so that MGN was now “invaded” by visual input. These inputs were then transferred to auditory cortex via intact MGN projections. The physiological and anatomical consequence of this rerouting was the development of visual networks in auditory cortex, so that a map of visual space emerged in the auditory cortex (i.e., a change in receptive field properties including the development of visual orientation-selective cells). How were these structural changes then interpreted by the animal? In other words, were the rewired projections interpreted as a visual input or an auditory one? If the behavioral role of a cortical area is independent of its input, then activation of the auditory cortex by any stimulus
would be interpreted as auditory. In contrast, if the nature of the input has a role in determining the function of a cortical area, then rewired animals should interpret visual activation in the auditory cortex as a visual stimulus. Von Melchner et al. (2000) addressed this question by training neonatal ferrets to discriminate between visual and auditory stimuli. A group of ferrets were “rewired” by directing their retinal axons to the left MGN, thus providing visual information to the auditory cortex in the left hemisphere. When the auditory cortex in the left hemisphere was lesioned, these animals were no longer able to discriminate visual stimuli, indicating that they became blind in the right visual field because the auditory cortex had mediated visual processing for this part of visual field. These experiments suggest that visual inputs routed to the auditory thalamus are capable of inducing auditory pathways to mediate vision, which crucially means that cortical areas process their functions under the input control. The fact that rewired cortices functionally mediate functions originally belonging to another region leads to the suggestion that even after sensory deprivation (i.e., without artificial rerouting), crossmodal plasticity may take place. In addition, would crossmodal plasticity correspond to an enhancement in performance in some behavioral tasks? To address this issue, Rauschecker and Kniepert (1994) tested visually deprived cats in a localization task in which animals had to walk toward a target sound source that was continuously manipulated in azimuth location. Deprived cats showed better auditory localization abilities compared to nondeprived cats, particularly for lateral and more peripheral locations, suggesting that compensatory plastic changes could underlie enhanced performance in the intact modality after sensory deprivation. Similar findings also come from King and Parsons (1999), who investigated auditory spatial acuity in visually deprived ferrets and documented improved performance in the lateral sound field for both juvenile and adult animals that were deprived early in life. However, these
182
studies might also be partially explained by intramodal changes, for example, by a higher functionality of cortical networks associated with the auditory system. Therefore, they did not provide complete answers to the functional meaning of the deafferented cortical activity. Recently, in reviewing their experiments on deaf cats conducted over several years, Lomber et al. (2010) have advanced a new hypothesis on crossmodal reorganization after sensory deprivation. In a number of experiments, the performance of congenitally deaf cats and hearing controls was compared for a number of visual psychophysical tasks (i.e., visual localization, movement detection, orientation and velocity discrimination, and visual acuity). Deaf cats were found to have enhanced performance only on the visual localization task (particularly for peripheral locations) and on the movement detection task. To investigate which cortical area could mediate the enhanced visual abilities, portions of auditory cortex were deactivated by means of a cryoloop device, which applied cold temperatures to a specific region of the brain and temporarily inactivated its functions. Interestingly, results showed that cooling of different areas could undermine the enhanced performance of deaf cats selectively for one task only, suggesting that perceptual enhancements were processed in specific cortical areas. In sum, crossmodal reorganization does not seem to be a unitary process involving reorganization of the whole (auditory) cortex; rather, it seems to involve changes in specific cortical loci. What are the characteristics of these reorganized loci? Why should they be so “special”? Lomber et al. (2010) suggested that only those regions subserving supramodal functions might undergo reorganization, while leaving modality-specific functions unaltered. In other words, skills that are shared across senses have greater potential to undergo enhancement and reorganization. For example, while color discrimination is an exclusively visual ability, and pitch discrimination an exclusively auditory ability, information on the spatial location of an object is
provided by both vision and audition. In this supramodal view, auditory deprivation will lead to crossmodal changes in those regions that “naturally” engage multisensory processing, thus leaving unchanged regions that functionally process a modality-specific feature (such as color or tone). Interestingly, crossmodal plasticity after auditory deprivation in humans appears to have a similar behavioral pattern as shown in Lomber et al. (2010). For instance, from a behavioral point of view, deaf individuals show enhanced performance in highly task-specific contexts, suggesting that not all aspects of the visual system are reorganized following sensory loss (for reviews, see Bavelier et al., 2006). In particular, deaf individuals have proven to have comparable performance to hearing controls in most visual tasks involving accuracy and sensitivity thresholds. These include brightness discrimination (Bross, 1979), visual contrast sensitivity (Finney and Dobkins, 2001), luminance change detection (Bavelier et al., 2000, 2001), motion direction (Bosworth and Dobkins, 2002a, b), motion velocity (Brozinsky and Bavelier, 2004), and temporal order perception (Nava et al., 2008). By contrast, deaf individuals appear to have enhanced performance for detection or discrimination of stimuli presented in the periphery of the visual field (Bavelier et al., 2000; Loke and Song, 1991; Neville and Lawson, 1987; but see Bottari et al., 2010 for contrasting results). In addition, found enhanced tactile sensitivity in congenitally deaf individuals when detecting suprathreshold tactile changes within a monotonous sequence of vibratory stimuli (Levänen et al., 1998; Levänen and Hamdorf, 2001). In contrast, studies in blind individuals have shown more consistent results with regards to enhanced performance compared to sighted controls in several different domains (Amedi et al., 2010; Collignon et al., 2009; Gougoux et al., 2009; Röder et al., 1996), For example, blind individuals outperform sighted controls on tactile tasks (Amedi et al., 2010; Sadato et al., 1996), auditory tasks (Rauschecker, 1995; Röder et al., 1996), sound localization tasks (Collignon
183
et al., 2009; Rauschecker, 1995; Voss et al., 2004), spatial imagery (Röder et al., 1997; Vanlierde et al., 2003), voice perception (Gougoux et al., 2009), and language perception (Röder et al., 2002). Some studies have put forward the possibility that the enhanced performance in deaf and blind individuals may be a result of recruitment of the deafferented sensory cortices by the intact senses to functionally compensate for the loss (Cohen et al., 1997; Levänen et al., 1998; Sadato et al., 1996). However, these studies remain very controversial due to several possible confounding factors (i.e., different experimental paradigms, individuals’ high variability). The most important factor concerns the limited spatial resolution of the employed neuroimaging techniques, which may not be sufficiently precise to identify the subregions of the deafferented cortex involved. In sum, the data discussed above show that the functional meaning of the cortical activity in the sensory-deprived cortex still needs to be further investigated. However, they also suggest that at least a portion of the cortical tissue that has become dominated by the intact senses may reorganize to now subserve functions of the intact modalities. In this sense, the possibility that brain regions that originally evolved to process specific modalities may partially take on new functions to better suit the environment can be seen as a case of exaptation; namely, as a mechanism that has new biological functions different from the ones that caused the original selection of that mechanism. The following section will discuss how these same spandrels can sometimes lead to maladaptive changes, therefore suggesting that plasticity may have mixed consequences: “positive” ones and “negative” ones. Maladaptation So far, plasticity has been viewed as a highly evolved feature of the brain to allow the organism to best adapt to the challenges imposed by the
environment. However, the same mechanisms that promote adaptation can sometimes turn into maladaptive changes in structure and behavior. In evolutionary biology, maladaptation has been defined as a deviation from adaptive peaks (Crespi, 2000). Adaptive peaks refer to the notion of an adaptive landscape introduced by Sewell Wright in 1931. The metaphor of the adaptive landscape was adopted to graphically summarize a theory concerning population genetics, by which “hills” represent the fittest populations (in terms of combination of genes) and the “valleys” represent the less fit populations. Natural selection tends to move the populations toward the peaks of the hills, but as the environment continuously changes, the populations are forced to adapt to these changes to maintain or build fitness. Assuming, hypothetically, that plasticity may be encoded in a group of genes, its phenotypic expression can be either adaptive or maladaptive. In this view, maladaptive plasticity can be seen as a phenotype placed in a valley of the adaptive landscape. Thus, it could be hypothesized that adaptive plasticity has evolved while leaving behind maladaptive plasticity. However, the following paragraphs will show that in some cases, the same mechanisms allowing adaptive changes can sometimes lead to maladaptive changes, thus narrowing the border between adaptive and maladaptive plasticity. Maladaptive brain plasticity, the other side of the coin Adaptive plastic changes in the cases we have described in the previous paragraph have a positive nature, in that they aid typically and atypically developing brains to functionally best fit the environment. However, there is also the other side of the coin of plasticity, which Elbert and Heim (2001) called “the dark side” of cortical reorganization, and what is commonly known as maladaptive plasticity. This can be seen as an excess of brain reorganization but might actually
184
consist of only a small structural change. In both cases, the outcomes are highly dysfunctional. If seen in the perspective of the mismatch between supply and demand, maladaptive changes even go beyond this mismatch, in that the supply (i.e., the brain) abnormally interprets the environmental demands and does not adjust to a more suitable and optimal condition. Curiously, in some cases, the same adaptive plastic changes that have aided the brain to best suit the environment are also those that can lead to maladaptive changes. For example, musicians, whose differences in brain structure with respect to nonmusicians may likely represent plastic brain adaptations in response to skill acquisition and repetitive rehearsal of those skills, can also sometimes develop the so-called musician's cramp, which is very similar to the well-known “writer's cramp” (Quartarone et al., 2003). Both maladaptive syndromes lead to focal dystonia, a movement disorder that causes the muscles to contract and spasm involuntarily. This debilitating disease finds its explanation in a dysfunctional reorganization of the brain (Tamura et al., 2009), particularly in the reorganization of the digits in the primary somatosensory cortex in these cases. More precisely, the topographic map represented in the somatosensory cortex is altered during the learning of sensorimotor skills, and those parts of the body (i.e., fingers, hand) that are stimulated the most drive the homologous cortical representations to expand (for classical animal studies, see Kaas, 1991). In support of the findings that cases of focal dystonia are triggered by maladaptive plastic changes, Candia et al. (2003) have developed a new treatment for focal hand dystonia in musicians based on the assumption that if the dysfunction arises as a consequence of maladaptive shifts of cortical reorganization, retuning the sensorimotor representations could likely treat these patients. During this training, dystonic patients have one or more nondystonic fingers immobilized in a splint device. The therapy consists in making sequential movements of two or three digits in extension, including the dystonic
finger, for a prolonged period and increasing time of training each day. In particular, in their fMRI experiment, Candia et al. (2003) showed a reduction in distances between cortical finger representations, suggesting a normalization of functional topography associated with the therapy. Most importantly, this cortical shift correlated with behavioral motor benefits, thus corroborating the notion that the underlying maladaptive mechanisms of dystonia may find their roots in cortical reorganization. The following paragraphs will focus on three particular cases for which plasticity operates in a maladaptive fashion: pain following amputation, tinnitus following hearing loss, and absence of benefits following cochlear implantation. While for the first two cases, the notion of maladaptive plasticity has a more intuitive connotation, maladaptive plasticity after cochlear implantation has a different nature. Nonetheless, all three cases represent the other side of the coin of beneficial adaptive changes, suggesting that plasticity can exert its influence in different ways. Phantoms after sensory deafferentation: phantom limb pain and tinnitus Phantom limb pain and tinnitus share common characteristics that allow, to some extent, for a direct comparison. First of all, both syndromes are characterized by a “phantom” sensation, sometimes very painful, arising from a lesion (in case of amputation) or a disease (in some cases of tinnitus following hearing loss). This, in turn, results in perceived pain although no stimulus is actively triggering it. Also, both maladaptive sensations are subjective and can change in quality throughout life, and for both conditions, similar recent training procedures have been shown to provide beneficial effects (Flor and Diers, 2009). In particular, the rationale behind the training is the assumption that pain is triggered, in both cases, by a reorganization of cortical maps, and by an “expansion” of some frequencies (in tinnitus) or somatosensory representations (in phantom pain) at the expense of others.
185
Phantom limb pain After amputation of a body part, the sensation of the presence of the missing part is reported by almost all amputees. The reported prevalence of phantom pain varies considerably in the literature, but most studies agree that around 60–80% of all amputees experience phantom pain following amputation. Phantom pain seems to be independent of age, gender, and cause of amputation. Very interestingly, phantom limb pain mostly occurs in late-amputated individuals (i.e., amputated in adulthood), being instead very infrequent in amputated children and almost absent in congenital amputees (for reviews, see Flor, 2002; Flor et al., 2006). The mechanisms underlying phantom limb pain are not fully understood and may involve complex interactions between morphologic, physiologic, and chemical changes at central and/or peripheral levels (Flor et al., 2006). However, similarly to the musicians’ case, the experience of pain correlates with reorganization of the somatosensory map. The possibility that pain, the maladaptive component following amputation, could be directly related to cortical reorganization of the primary somatosensory cortex, has only recently found major acceptance in the literature. As plastic reorganization has commonly been seen (as discussed in the previous paragraphs) as a beneficial and functional response of the brain to adaptive needs, the possibility that the same mechanism could trigger maladaptive outcomes has somehow been viewed as counterintuitive. However, nearly 15 years ago, along with other causal mechanisms that can explain phantom limb pain, the possibility that this maladaptive plastic change could additionally result from cortical reorganization started emerging (Birbaumer et al., 1997; Flor et al., 1995; Knecht et al., 1996). The relationship between cortical reorganization and phantom limb pain started with the notion that deafferentation of digits or the hand leads to plastic changes in the somatosensory
cortex (Pons et al., 1991). In addition, findings on chronic back pain revealed a strong correlation between cortical alteration and pain (Flor et al., 1997), with patients exhibiting more cortical reorganization as a function of felt pain. These two factors led researchers to point to cortical reorganization as a structural correlate of phantom limb pain. For example, Flor et al. (1995) and Birbaumer et al. (1997) determined cortical reorganization in a group of adult amputees by means of neuroelectric source imaging (a technique that combines evoked potential recordings with structural magnetic resonance imaging). In particular, Birbaumer et al. (1997) compared the representations of hand and mouth in both hemispheres of the somatosensory cortex. As amputees without pain were found to have mirrored representations of mouth and hand, any asymmetry found in amputees with pain would become a marker of cortical reorganization. As hypothesized, the cortical representation in amputees with pain showed a shift of the lip representation into the cortical region, which previously belonged to the amputated hand. An intriguing explanation of phantom limb pain has also been put forward, namely, the possibility that the maladaptive outcome could be elicited by the memory of the pain experienced prior to amputation (Flor, 2002; Katz and Melzack, 1990). In other words, if the preamputated limb had received prolonged and intense noxious input, it would have developed enhanced excitability for pain and therefore exhibited an alteration in cortical somatosensory processing. Subsequent amputation and invasion of the cortical region by neighboring inputs would then activate cortical neurons coding for pain, leading to the perception of pain. In support to this view, Nikolajsen et al. (1997) have shown that pain experienced before amputation can sometimes even predict phantom limb pain after deafferentation, supporting the importance of the memory of pain in making the phantom persist over time.
186
A particularly interesting finding concerns the lack of reorganization of somatosensory cortical maps in congenital amputees, which also correlates with their lack of reported pain (though the sensation of the missing limb persists in many cases). However, only in recent times has this correlation been investigated. So, for example, Flor et al. (1998) investigated cortical reorganization in primary somatosensory cortex in a group of congenital amputees and a group of traumatic amputees with or without pain determined by neuromagnetic source imaging. Results showed that the most cortically reorganized individuals were the traumatic amputees reporting pain. In contrast, the congenital amputees and amputees without pain presented very little reorganization and the small amounts of reorganization observed in each case were similar. In addition, phantom limb pain was found to positively correlate with cortical reorganization and with no other factor (i.e., time as amputation) or sensation (i.e., phantom limb sensation per se). The fact that congenitally limb-deprived individuals do not experience pain and do not present cortical reorganization opens an additional issue concerning adaptive and maladaptive plasticity that should be further explored, namely, the possibility that these two outcomes are influenced by development. In other words, while congenital or early deprivation may favor overall adaptation, deprivation experienced in adulthood may lead to maladaptation. Curiously, the presence or absence of beneficial versus detrimental cortical reorganization differs between types of developmental deprivations, as the following section will suggest.
Adaptation early in life: a comparison between congenitally deprived sensory modalities While congenital amputees have been shown to have a lack of cortical reorganization compared to late amputees, some studies in blind individuals show the opposite pattern (Cohen et al., 1999; Sadato et al., 2002). For example, Fieger et al.
(2006) compared his results in late-blind individuals with the findings of Röder et al. (1999) in congenitally blind individuals and showed that despite comparable performance, the brain mechanisms differed between the two groups. While a more precise spatial tuning of early auditory processes was observed in the congenitally blind (indexed by the event-related potential (ERP) called N1), later processing stages (indexed by the ERP called P3) seemed to mediate the improved behavior in the late blind. Overall, these results showed that the neural mechanisms underlying crossmodal changes differ in the developing and adult brain, further corroborating the notion that plastic changes that occur early in life can lead to functional advantages throughout life. In sum, in congenital blindness, the presence of crossmodal reorganization appears to be functionally adaptive, while in congenitally limbdeafferented individuals, the absence of crossmodal reorganization appears to be one of the preconditions for avoiding maladaptive outcomes (i.e., pain). What can this differential pattern of plasticity suggest? A hypothesis could be that plastic changes early in life as a consequence of congenital deafferentation may be more adaptive compared to changes at later developmental stages. In other words, the flexibility of the brain after either direct or indirect damage during early development may be the expression of normal ontogenetic mechanisms that instead of “repairing” (as in the case of adult brains) simply make the young brain optimally adjust to the insult. The fact that positive adaptive plasticity is expressed differentially (i.e., reorganization vs. nonreorganization) in the two cases (i.e., blindness vs. phantom limb pain) could possibly be due to the specific type of damage or exceptional experience.
Tinnitus Tinnitus can be “objective” or “subjective.” The former refers to a perceived sensation of sound
187
elicited by internal stimulation (i.e., abnormal blood flow pulsations or muscle contraction) that can be heard (therefore objectively measured) by a clinician (e.g., by placing a stethoscope over the patient's external auditory canal). Here, we will focus on subjective tinnitus, which causes the affected person to experience “phantom sounds,” commonly reported to be ringing noises, buzzes, clicks, pure tones, and even songs. Tinnitus has many different causes, otologic, neurologic, and drug related, making the understanding and treatment of the disease difficult to handle (for a clinical review of tinnitus, see Lockwood et al., 2002; M ller et al., 2010). The prevailing opinion is that tinnitus is generated as a consequence of altered patterns of intrinsic neural activity generated along the central auditory pathway following damage to peripheral auditory structures (Eggermont and Roberts, 2004), making it a prevailing symptom following hearing loss. But what does this altered neural activity precisely refer to? Electrophysiological recordings in animals have identified three types of abnormal activity in the auditory system following sensory deprivation, which could also account for causes of tinnitus when associated with hearing loss (for a comparison between animal and human studies, see Adjamian et al., 2009). The first type refers to changes in the spontaneous neural firing rate, by which neurons at rest fire even in the absence of sound stimulation (Seki and Eggermont, 2003). The second type refers to changes in the temporal firing pattern of a single neuron as well as the synchronous activity between neurons. After highnoise exposure or hearing loss, their impulses tend to become pathologically synchronous. This synchronic firing would then become more salient compared to more dispersed firing and be interpreted by the brain as a real sound. Moreover, it is precisely this prolonged synchronization that would induce the perception of tinnitus (Noreña and Eggermont, 2003; Seki and Eggermont, 2003; Weisz et al., 2005, 2007). Weisz et al. (2007) have proposed that gamma band
activity, which is increased in tinnitus patients, may reflect the synchronous firing of neurons within the auditory cortex and constitute the neural code of tinnitus. The reason why gamma band activity has been viewed with such excitement in explaining tinnitus is because a series of previous studies have shown that gamma band synchronous oscillations of neuroelectrical activity may be a mechanism used by the brain to generate and bind conscious sensations to represent distinct objects (for a review, see Sauvé, 1999). This functional significance of gamma band activity would, therefore, explain why tinnitus patients consciously experience a phantom sensation. Finally, the third type of abnormal activity in auditory system following sensory the deafferentation has been shown to result in reorganization of the cortical tonotopic representation. This third type clearly parallels mechanisms of cortical reorganization reviewed for phantom limb pain. As in the latter case, the tonotopic map becomes distorted for those sound frequencies where the hearing loss occurred. This results in an expansion of the representation of the frequencies that border on the lost frequencies, so that the deprived neurons now become responsive to frequencies adjacent to those at which hearing loss has taken place. To investigate this issue, Mühlnickel et al. (1998) used magnetoencephalographic recordings on 10 individuals with tinnitus to establish whether there could be any reorganization of the tonotopic map in the auditory cortex. The rationale was to observe whether tinnitus could be related to a shift of frequency representations in the auditory cortex. To this end, four sets of pure tones above an individual's hearing level were selected and presented to each ear to form a trajectory representing the tonotopic map in healthy controls. For tinnitus patients, three tones were distant from the tinnitus frequency and the fourth was close to the tinnitus frequency. The three tones served to reconstruct the tonotopic map of each patient. Results showed that the tinnitus frequency had “invaded” the neighboring
188
frequency regions. Further, this invasion correlated with tinnitus strength, so that patients reporting more symptoms were also the ones who presented more cortical reorganization. It is worth noting that the three types of changes described seldom occur independently of each other, as suggested by animal (Seki and Eggermont, 2003) and human studies (Weisz et al., 2005, 2007), pointing to their correlational rather than causal nature. That these three factors may be simultaneously present has been highlighted in studies that are investigating which treatments can exert the most beneficial and prolonged effects on tinnitus. In other words, several studies have particularly manipulated cortical reorganization with the assumption that, as in the case of dystonic patients, retuning the tonotopic maps could relieve patients of the phantom sensation. Recently, Okamoto et al. (2010) exposed eight chronic tinnitus patients to music they chose themselves and which they were asked to listen to for 12 months regularly. The music was then frequency modified, so that it did not contain frequencies in the range neighboring the tinnitus frequency. After a 1-year exposure, tinnitus patients reported a reduction in tinnitus loudness. There was also a corresponding decrease in evoked activity in auditory cortex areas corresponding to the tinnitus frequency. The authors speculated that lateral inhibition from the neighboring parts of the tonotopic map were responsible for the beneficial effects on tinnitus. “Rewiring” cortical reorganization through prostheses: to what extent is plasticity malleable? Considering the lessons learned from maladaptive plastic changes strictly linked to cortical reorganization, one could ask whether restoring sensory input to the deafferented region by means of a prosthesis would provide substantial relief to tinnitus and phantom limb pain patients. The rationale behind reafferentation is that either tactile
(for phantom limb pain) or auditory (for tinnitus) stimulation will expand the cortical representation of the stimulated body region, thus “rewiring” cortical maps back to their original state. According to this view, prostheses for phantom limb pain and cochlear implants for tinnitus patients could potentially help in “blocking” or even “rewiring” the effects of maladaptive plasticity. A cochlear implant is a neuroprosthetic device consisting of a microelectrode array inserted in the cochlea that directly stimulates the auditory nerve (for reviews, see Moore and Shannon, 2009; Rauschecker and Shannon, 2002). Although limb prostheses and cochlear implants cannot be directly compared because they are based on different principles (i.e., on somatosensory feedback in the former case, and on nerve stimulation in the latter), they nonetheless represent good models to investigate how and to what extent the brain learns to interpret new information. In particular, several studies have shown that these devices can, in some cases, relieve phantom limb pain and tinnitus. For example, Lotze et al. (1999) examined the effects of the use of a myoelectric device in a group of unilateral amputees using fMRI. The groups were split into myoelectric versus nonmyoelectric users based on the extent of wearing time and average usage. The myoelectric users showed a symmetrical lip representation in the somatosensory cortex (in accordance with previous studies showing that symmetrical body representations are an index of a lack of cortical reorganization), which correlated with a reduction of phantom limb pain. In contrast, the nonmyoelectric users showed the exact opposite pattern, namely, a reported intense pain that correlated with massive cortical reorganization. Similarly, for tinnitus patients, several studies have documented a reduction of tinnitus after cochlear implantation (Miyamoto et al., 1997; Ruckenstein et al., 2001). However, it should be noted that results for both treatments are controversial, in that not all patients have systematically reported benefits. To date, it is not
189
known whether this difficulty in “undoing” or “rewiring” previous plastic changes relates to the technical limits of the devices and/or to the limits of plasticity itself. It is likely, though, that both factors interact to make reafferentation a challenging issue. The particular case of cochlear implants failing to suppress and reduce tinnitus leads to our discussion of the last example of maladaptive plasticity. In which sense can a cochlear implant be maladaptive? As cochlear implantation has become routine therapy for partially restoring auditory function in profoundly deaf individuals, most studies have emphasized the beneficial outcomes of this device following auditory deprivation (Litovsky et al., 2006; Svirsky et al., 2000; Van Hoesel, 2004). The extent to which a cochlear implant exerts its benefits on single individuals appears to be determined by several factors. These factors include the age at which implantation takes place (Sharma et al., 2002, 2005), and the previous experience with auditory cues (Nava et al., 2009a,b). Clearly, cochlear implantation per se does not create any phantom sensation, so that a direct comparison to tinnitus and phantom limb pain is not feasible. However, the outcome of a cochlear implant is related to the amount of cortical reorganization that has taken place prior to implantation. In other words, precisely what we have defined as “spandrels” after sensory deafferentation may be detrimental in case of reafferentation. The following examples show that some plastic changes can be maladaptive because they do not allow the brain to “rewire” once the reafferented sensory cortices have been taken over by other modalities. Lee et al. (2001) were the first to suggest such a possibility by examining glucose metabolism (used as an index of brain activity) in a group of prelingually deafened individuals before cochlear
implantation. The degree of hypometabolism before implantation correlated with the hearing abilities achieved after implantation, so that those patients with higher hypometabolism in temporal areas (including auditory cortex) were also the ones who gained more from auditory restoration. Conversely, those with lower hypometabolism did not achieve the same auditory capabilities, as measured with a speech perception test administered at several follow-up sessions after implantation. Results were interpreted as a being related to a possible increase in visual or somatosensory afferents to these temporal regions due to auditory deafferentation. Therefore, if crossmodal plasticity takes place in the auditory cortex before implantation, improvement in hearing after implantation will be less pronounced. Beneficial outcomes after cochlear implantation have commonly been measured by evaluating speech recognition over time (for review, see Peterson et al., 2010). Reasoning that responses to visual stimulation in cochlear implant recipients may be related to their speech recognition abilities; Doucet et al. (2006) compared visual processing in two groups of cochlear implant recipients. The subjects were divided into “good” and “bad” performers according to their auditory speech perception skills, in that the former were able to recognize speech without visual cues, and the latter only relied on sign language and lip-reading to communicate efficiently. All participants were simply asked to fixate a visual stimulus presented several times while evoked potentials were recorded. Results showed that, while “good” performers had similar activation compared to hearing controls (i.e., evoked activity measured with ERPs was circumscribed around the primary visual cortex), “bad” performers exhibited extended cortical activity, suggesting recruitment of auditory cortical areas for visual processing. This result further suggests that once crossmodal plastic changes have taken place, speech perception performance after cochlear implantation might be undermined as a consequence of cortical reorganization.
190
The fact that crossmodal changes can undermine the good outcome of cochlear implants is relevant to the issue of when (in terms of age) plastic changes take place, and therefore when a device should be implanted. In this view, the existence of sensitive periods early in life for the typical development of the auditory system suggests that crossmodal plasticity may occur within these phases, and only to a lesser extent, or not at all, later in life. For example, Sharma et al. (2002) examined P1 latencies in congenitally deaf children who received a cochlear implant and found that those implanted before 3.5 years of age had normal P1 latencies, while children who received their implant after 7 years of age had abnormal latencies. This suggests a sensitive period for central auditory development that persists up to 3.5 years of age. In a further study, Sharma et al. (2005) assessed the time course of central auditory development in early and late congenitally deaf children implanted unilaterally either before 3.5 years of age or after 7 years of age. The results showed a different pattern of P1 development for early and late implanted children. While early implanted children reached almost normal P1 latencies within a week of implant use, late implanted children showed atypical response that remained atypical until the 18-month follow-up. Overall, these results suggest that, in line with what we have previously mentioned for congenitally blind individuals, plastic changes that occur within sensitive periods early in life might be particularly strong and long-lasting, therefore preventing the brain from reorganizing at a later time. In this sense, some plastic changes can be maladaptive from the perspective of reafferenting the auditory pathways later in life. Finally, it should be mentioned that, comparable to the case of phantom limb pain after amputation later in life, crossmodal changes in the auditory cortex can occur also as a function of years of deprivation. For example, Lee et al. (2003) showed that there is a correlation between years of auditory deprivation and cortical reorganization that goes beyond sensitive periods. In his
study (Lee et al., 2003), a group of postlingually deafened adults with years of auditory deprivation ranging from 2 months to 23 years underwent PET scans to evaluate their regional cerebral metabolism (similar to Lee et al., 2001). Results showed that glucose metabolism in the auditory cortex decreased after auditory deprivation, but increased as a function of years of deprivation, suggesting that functional crossmodal reorganization also takes place in the adult brain. What does this study suggest? First, it is compatible with the view that plasticity and crossmodal changes can also occur during adulthood (Pascual-Leone and Hamilton, 2001; Voss et al., 2004). Second, it corroborates the criterion expressed by Lövdén et al. (2010) by which adult plasticity is driven by a prolonged mismatch between supply and demand. The longer the mismatch is, the higher the probability that the change will result in a plastic change. Final remarks We started this review by defining some evolutionary terms adopted by neuroscientists to highlight some properties, mechanisms, and behaviors of the brain. As much as phenotypic plasticity represents an important factor in evolution, has a genetic basis, and may be altered by natural selection (Price et al., 2003), we suggest that brain plasticity could mimic this evolutionary pattern, so that it becomes worth asking why and how this characteristic of the brain has evolved. Here, we have discussed how adaptive plasticity can lead the brain to structural and functional changes, in typical and atypical development, to best suit environmental demands. However, we have also challenged the view that plasticity consists only of beneficial adaptive changes, by emphasizing how it can sometimes result in highly dysfunctional outcomes that we have generally described here as being maladaptive. From an evolutionary perspective, maladaptive plasticity arises as a phenotype that has reduced
191
fitness or is distant from an adaptive peak (Ghalambor et al., 2007). Brain plasticity has likely evolved to accommodate continuous environmental changes, suggesting that what we define as adaptive or maladaptive at any given time could also exchange roles as a function of changing environmental demands. An additional important consideration is whether in the modern era making a distinction between adaptive and maladaptive plasticity is actually relevant. Advances in technology and medicine have clearly increased our chances of survival and have, therefore, changed the pressure of natural selection on our genes by changing the extent to which we must adapt to environmental demands. In this context of less selective pressures, an adaptive landscape may be more difficult to draw, as “hills” and “valleys” effectively become less distinct. In conclusion, the environmental manipulations carried out by humans may slowly shape natural selection, may even change the rate of evolutionary dynamics, and finally also the trait of plasticity. References Adjamian, P., Sereda, M., & Hall, D. A. (2009). The mechanisms of tinnitus: Perspectives from human functional neuroimaging. Hearing Research, 253, 15–31. Amedi, A., Raz, N., Azulay, H., Malach, R., & Zohary, E. (2010). Cortical activity during tactile exploration of objects in blind and sighted humans. Restorative Neurology and Neuroscience, 28, 143–156. Bavelier, D., Brozinsky, C., Tomann, A., Mitchell, T., Neville, H., & Liu, G. (2001). Impact of early deafness and early exposure to sign language on the cerebral organization for motion processing. The Journal of Neuroscience, 21, 8931–8942. Bavelier, D., Dye, M. W. G., & Hauser, P. C. (2006). Do deaf individuals see better? Trends in Cognitive Sciences, 10, 512–518. Bavelier, D., & Neville, H. J. (2002). Cross-modal plasticity: Where and how? Nature Reviews. Neuroscience, 3, 443–452. Bavelier, D., Tomann, A., Hutton, C., Mitchell, T., Corina, D., Liu, G., et al. (2000). Visual attention to the periphery is enhanced in congenitally deaf individuals. The Journal of Neuroscience, 20, 1–6.
Berlucchi, G., & Buchtel, H. A. (2009). Neuronal plasticity: Historical roots and evolution of meaning. Experimental Brain Research, 192, 307–319. Birbaumer, N., Lutzenberger, W., Montoya, P., Larbig, W., Unertl, K., Töpfner, S., et al. (1997). Effects of regional anesthesia on phantom limb pain are mirrored in changes in cortical reorganization. The Journal of Neuroscience, 17, 5503–5508. Bosworth, R. G., & Dobkins, K. R. (2002a). The effect of spatial attention on motion processing in deaf signers, hearing signers, and hearing non-signers. Brain and Cognition, 4, 152–169. Bosworth, R. G., & Dobkins, K. R. (2002b). Visual field asymmetries for motion processing in deaf and hearing signers. Brain and Cognition, 49, 170–181. Bottari, D., Nava, E., Ley, P., & Pavani, F. (2010). Enhanced reactivity to visual stimuli in deaf individuals. Restorative Neurology and Neuroscience, 28, 167–179. Bross, M. (1979). Residual sensory capacities of the deaf: A signal detection analysis of a visual discrimination task. Perceptual and Motor Skills, 48, 187–194. Brozinsky, C. J., & Bavelier, D. (2004). Motion velocity thresholds in deaf signers: Changes in lateralization but not in overall sensitivity. Cognitive Brain Research, 21, 1–10. Candia, V., Wienbruch, C., Elbert, T., Rockstroh, B., & Ray, W. (2003). Effective behavioral treatment of focal hand dystonia in musicians alters somatosensory cortical organization. Proceedings of the National Academy of Sciences of the United States of America, 100, 7942–7946. Cohen, L., Celnik, P., Pascual-Leone, A., Corwell, B., Faiz, L., Dambrosia, J., et al. (1997). Functional relevance of crossmodal plasticity in blind humans. Nature, 389, 180–183. Cohen, L. G., Weeks, R. A., Sadato, N., Celnik, P., Ishii, K., & Hallett, M. (1999). Period of susceptibility for cross-modal plasticity in the blind. Annals of Neurology, 45, 451–460. Collignon, O., Voss, P., Lassonde, M., & Lepore, F. (2009). Cross-modal plasticity for the spatial processing of sounds in visually deprived subjects. Experimental Brain Research, 192, 343–358. Crespi, B. J. (2000). The evolution of maladaptation. Heredity, 84, 623–629. Darwin, C. (Ed.), (1859). On the origin of species by means of natural selection, or the preservation of favoured races in the struggle for life. Albemarle Street, London: John Murray. Dobzhansky, T. (1956). What is an adaptive trait? The American Naturalist, 90, 337–347. Doucet, M. E., Bergeron, F., Lassonde, M., Ferron, P., & Lepore, F. (2006). Cross-modal reorganization and speech perception in cochlear implant users. Brain, 129, 3376–3383. Eggermont, J. J., & Roberts, L. E. (2004). The neuroscience of tinnitus. Trends in Neurosciences, 27, 676–682. Elbert, T., & Heim, S. (2001). A light and a dark side. Nature, 411, 139.
192 Elbert, T., Pantev, C., Wienbruch, C., Rockstroh, B., & Taub, E. (1995). Increased cortical representation of the fingers of the left hand in string players. Science, 270, 305–307. Fieger, A., Röder, B., Teder-Sälejärvi, W., Hillyard, S. A., & Neville, H. J. (2006). Auditory spatial tuning in late-onset blindness in humans. Journal of Cognitive Neuroscience, 18, 149–157. Finney, E. M., & Dobkins, K. R. (2001). Visual contrast sensitivity in deaf versus hearing populations: Exploring the perceptual consequences of auditory deprivation and experience with a visual language. Cognitive Brain Research, 11, 171–183. Flor, H. (2002). Phantom-limb pain: Characteristics, causes, and treatment. Lancet Neurology, 1, 182–189. Flor, H., Braun, C., Elbert, T., & Birbaumer, N. (1997). Extensive reorganization of primary somatosensory cortex in chronic back pain patients. Neuroscience Letters, 224, 5–8. Flor, H., & Diers, M. (2009). Sensorimotor training and cortical reorganization. NeuroRehabilitation, 25, 19–27. Flor, H., Elbert, T., Knecht, S., Wienbruch, C., Pantev, C., Birbaumer, N., et al. (1995). Phantom-limb pain as a perceptual correlate of cortical reorganization following arm amputation. Nature, 375, 482–484. Flor, H., Elbert, T., Mühlnickel, W., Pantev, C., Wienbruch, C., & Taub, E. (1998). Cortical reorganization and phantom phenomena in congenital and traumatic upper-extremity amputees. Experimental Brain Research, 119, 205–212. Flor, H., Nikolajsen, L., & Jensen, T. S. (2006). Phantom limb pain: A case of maladaptive CNS plasticity? Nature Reviews. Neuroscience, 7, 873–881. Ghalambor, C. K., McKay, J. K., Carroll, S. P., & Reznick, D. N. (2007). Adaptive versus non-adaptive phenotypic plasticity and the potential for contemporary adaptation in new environments. Functional Ecology, 21, 394–407. Gougoux, F., Belin, P., Voss, P., Lepore, F., Lassonde, M., & Zatorre, J. R. (2009). Voice perception in blind persons: A functional magnetic resonance imaging study. Neuropsychologia, 47, 2967–2974. Gougoux, F., Zatorre, R. J., Lassonde, M., Voss, P., & Lepore, F. (2005). A functional neuroimaging study of sound localization: Visual cortex activity predicts performance in early-blind individuals. PLoS Biology, 3, e27. Gould, S. J. (1991). Exaptation: A crucial tool for an evolutionary psychology. Journal of Social Issues, 47, 43–65. Gould, S. J., & Lewontin, R. C. (1979). The sprandels of San Marco and the Panglossian Paradigm: A critique of the adaptationist programme. Proceedings of the Royal Society of London, 205, 581–598. Greenough, W. T., Black, J. E., & Wallace, C. S. (1987). Experience and brain development. Child Development, 58, 539–559.
Hyde, K. L., Lerch, J., Norton, A., Forgeard, M., Winner, E., Evans, A. C., et al. (2009). Musical training shapes structural brain development. The Journal of Neuroscience, 29, 3019–3025. Johansson, B. B. (2006). Music and brain plasticity. European Review, 14, 49–64. Kaas, J. H. (1991). Plasticity of sensory and motor maps in adult mammals. Annual Review of Neuroscience, 14, 137–167. Katz, J., & Melzack, R. (1990). Pain ‘memories’ in phantom limbs: Review and clinical observations. Pain, 43, 319–336. King, A. J., & Parsons, C. (1999). Improved auditory spatial acuity in visually deprived ferrets. The European Journal of Neuroscience, 11, 3945–3956. Knecht, S., Henningsen, H., Elbert, T., Flor, H., Hohling, C., Pantev, C., et al. (1996). Reorganizational and perceptional changes after amputation. Brain, 119, 1213–1219. Knudsen, E. I. (2004). Sensitive periods in the development of the brain and behavior. Journal of Cognitive Neuroscience, 16, 1412–1425. Kolb, B., Gibb, R., & Robinson, T. E. (2003). Brain plasticity and behavior. Current Directions in Psychological Science, 12, 1–5. Lee, D. S., Lee, J. S., Kim, S. H., Kim, J. W., Chung, J. K., Lee, M. C., et al. (2001). Cross-modal plasticity and cochlear implants. Nature, 409, 149–150. Lee, J. S., Lee, D. S., Oh, S. H., Kim, C. S., Kim, J. W., Hwang, C. H., et al. (2003). PET evidence of neuroplasticity in adult auditory cortex of postlingual deafness. Journal of Nuclear Medicine, 9, 1435–1439. Levänen, S., & Hamdorf, D. (2001). Feeling vibrations: Enhanced tactile sensitivity in congenitally deaf humans. Neuroscience Letters, 301, 75–77. Levänen, S., Jousmaki, V., & Hari, R. (1998). Vibrationinduced auditory-cortex activation in a congenitally deaf adult. Current Biology, 8, 869–873. Litovsky, R. Y., Johnstone, P. M., & Godar, S. P. (2006). Benefits of bilateral cochlearimplants and/or hearing aids in children. International Journal of Audiology, 45, 78–91. Lockwood, A. H., Salvi, R. J., & Burkhard, R. F. (2002). Tinnitus. The New England Journal of Medicine, 347, 904–910. Loke, W. H., & Song, S. (1991). Central and peripheral visual processing in hearing and nonhearing individuals. Bulletin of the Psychonomic Society, 29, 437–440. Lomber, S. G., Meredith, M. A., & Kral, A. (2010). Crossmodal plasticity in specific auditory cortices underlies visual compensations in the deaf. Nature Neuroscience, 13, 1421–1427. Lotze, M., Grodd, W., Birbaumer, N., Erb, M., Huse, E., & Flor, H. (1999). Does use of a myoelectric prosthesis prevent cortical reorganization and phantom limb pain? Nature, 2, 501–502.
193 Lövdén, M., Bäckman, L., Lindenberger, U., Schaefer, S., & Schmiedek, F. (2010). A theoretical framework for the study of adult cognitive plasticity. Psychological Bulletin, 136, 659–676. Miyamoto, R. T., Wynne, M. K., McKnight, C., & Bichey, B. (1997). Electrical suppression of tinnitus via cochlear implants. The International Tinnitus Journal, 3, 35–38. M ller, A. R., Langguth, B., DeRidder, D., & Kleinjung, T. (Eds.), (2010). Textbook of Tinnitus. Springer: New York, USA. Moore, D. R., & Shannon, R. V. (2009). Beyond cochlear implants: Awakening the deafened brain. Nature Neuroscience, 12, 686–691. Moreno, S., Marques, C., Santos, A., Santos, M., Castro, S. L., & Besson, M. (2009). Musical training influences linguistic abilities in 8-year-old children: More evidence for brain plasticity. Cerebral Cortex, 19, 712–723. Mühlnickel, W., Elbert, T., Taub, E., & Flor, H. (1998). Reorganization of auditory cortex in tinnitus. Proceedings of the National Academy of Sciences of the United States of America, 95, 10340–10343. Münte, T. F., Altenmüller, E., & Jäncke, L. (2002). The musician's brain as a model of neuroplasticity. Nature Neuroscience. Reviews, 3, 473–478. Nava, E., Bottari, D., Bonfioli, F., Beltrame, M. A., & Pavani, F. (2009a). Spatial hearing with a single cochlear implant in late-implanted adults. Hearing Research, 255, 91–98. Nava, E., Bottari, D., Portioli, G., Bonfioli, F., Beltrame, M. A., Formigoni, P., et al. (2009b). Hearing again with two ears: Recovery of spatial hearing after bilateral cochlear implantation. Neuropsychologia, 47, 928–932. Nava, E., Bottari, D., Zampini, M., & Pavani, F. (2008). Visual temporal order judgment in profoundly deaf individuals. Experimental Brain Research, 190, 179–188. Neville, H. J., & Lawson, D. S. (1987). Attention to central and peripheral visual space in a movement detection task: An event related potential and behavioral study. II. Congenitally deaf adults. Brain Research, 405, 268–283. Nikolajsen, L., Ilkjaer, S., Kroner, K., Christensen, J. H., & Jensen, T. S. (1997). The influence of preamputation pain on postamputation stump and phantom pain. Pain, 72, 393–405. Noreña, A. J., & Eggermont, J. J. (2003). Changes in spontaneous neural activity immediately after an acoustic trauma: Implications for neural correlates of tinnitus. Hearing Research, 183, 137–153. Okamoto, H., Stracke, H., Stoll, W., & Pantev, C. (2010). Listening to tailor-made notched music reduces tinnitus loudness and tinnitus-related auditory cortex activity. Proceedings of the National Academy of Sciences of the United States of America, 107, 1207–1210. Pascual-Leone, A., Amedi, A., Fregni, F., & Merabet, L. B. (2005). The plastic human brain cortex. Annual Review of Neuroscience, 28, 377–401.
Pascual-Leone, A., & Hamilton, R. (2001). The metamodal organization of the brain. Progress in Brain Research, 134, 427–445. Pavani, F., & Röder, B. (2011). Crossmodal plasticity as a consequence of sensory loss: Insights from blindness and deafness. In B. E. Stein (Ed.), The new handbook of multisensory processes. Cambridge, MA: MIT Press. Peterson, N. R., Pisoni, D. B., & Miyamoto, R. T. (2010). Cochlear implants and spoken language processing abilities: Review and assessment of the literature. Restorative Neurology and Neuroscience, 28, 237–250. Pons, T. P., Garraghty, P. E., Ommaya, A. K., Kaas, J. H., Taub, E., & Mishkin, M. (1991). Massive cortical reorganization after sensory deafferentation in adult macaques. Science, 28, 1857–1860. Price, T. D., Qvarnström, A., & Irwin, D. E. (2003). The role of phenotypic plasticity in driving genetic evolution. Proceedings: Biological Sciences, 270, 1433–1440. Quartarone, A., Bagnato, S., Rizzo, V., Siebner, H. R., Dattola, V., Scalfari, A., Morgante, F., Battaglia, F., Romano, M., & Girlanda, P. (2003). Abnormal associative plasticity of the human motor cortex in writer's cramp. Brain, 126, 2586–2596. Rauschecker, J. P. (1995). Compensatory plasticity and sensory substitution in the cerebral cortex. Trends in Neurosciences, 18, 36–43. Rauschecker, J. P., & Kniepert, U. (1994). Auditory localization behaviour in visually deprived cats. The European Journal of Neuroscience, 6, 149–160. Rauschecker, J. P., & Shannon, R. V. (2002). Sending sound to the brain. Science, 295, 1025–1029. Röder, B., Rösler, F., & Henninghausen, E. (1997). Different cortical activation patterns in blind and sighted humans during encoding and transformation of haptic images. Psychophysiology, 34, 292–307. Röder, B., Rösler, F., Henninghausen, E., & Näcker, F. (1996). Event-related potentials during auditory and somatosensory discrimination in sighted and blind human subjects. Cognitive Brain Research, 4, 77–93. Röder, B., Stock, O., Bien, S., Neville, H., & Rösler, F. (2002). Speech processing activates visual cortex in congenitally blind humans. The European Journal of Neuroscience, 16, 930–936. Röder, B., Teder-Sälejärvi, W., Sterr, A., Rösler, F., Hillyard, S. A., & Neville, H. J. (1999). Improved auditory spatial tuning in blind humans. Nature, 400, 162–166. Ruckenstein, M. J., Hedgepeth, C., Rafter, K. O., Montes, M. L., & Bigelow, D. C. (2001). Tinnitus suppression in patients with cochlear implants. Otology & Neurotology, 22, 200–204. Sadato, N., Okada, T., Honda, M., & Yonekura, Y. (2002). Critical period for cross-modal plasticity in blind humans: A functional MRI study. Neuroimage, 16, 389–400.
194 Sadato, N., Pascual-Leone, A., Grafman, J., Ibañez, V., Deiber, M. P., Dold, G., et al. (1996). Activation of the primary visual cortex by Braille reading in blind subjects. Nature, 380, 526–528. Sauvé, K. (1999). Gamma-band synchronous oscillations: Recent evidence regarding their functional significance. Consciousness and Cognition, 8, 213–224. Seki, S., & Eggermont, J. J. (2003). Changes in spontaneous firing rate and neural synchrony in cat primary auditory cortex after localized tone-induced hearing loss. Hearing Research, 180, 28–38. Sharma, A., Dorman, M. F., & Kral, A. (2005). The influence of a sensitive period on central auditory development in children with unilateral and bilateral cochlear implants. Hearing Research, 203, 134–143. Sharma, A., Dorman, M. F., & Spahr, A. J. (2002). A sensitive period for the development of the central auditory system in children with cochlear implants: Implications for age of implantation. Ear and Hearing, 23, 532–539. Sur, M., Pallas, S. L., & Roe, A. (1990). Cross-modal plasticity in cortical development: Differentiation and specification of sensory neocortex. Trends in Neurosciences, 13, 227–233. Svirsky, M. A., Robbins, A. M., Iler Kirk, K., Pisoni, D. B., & Miyamoto, R. T. (2000). Language development in profoundly deaf children with cochlear implants. Psychological Science, 11, 153–158. Tamura, Y., Ueki, Y., Lin, P., Vorbach, S., Mima, T., Kakigi, R., et al. (2009). Disordered plasticity in the primary somatosensory cortex in focal hand dystonia. Brain, 132, 749–755. Trussell, G. C., & Smith, L. D. (2000). Induced defenses in response to an invading crab predator: An explanation of historical and geographic phenotypic change. Proceedings of the National Academy of Sciences of the United States of America, 97, 2123–2127.
Van Hoesel, R. J. M. (2004). Exploring the benefits of bilateral cochlear implants. Audiology & Neurotology, 9, 234–246. Vanlierde, A., De Volder, A. G., Wanet-Defalque, M. C., & Veraart, C. (2003). Occipito-parietal cortex activation during visuo-spatial imagery in early blind humans. Neuroimage, 19, 698–709. Via, S., Gomulkiewicz, R., De Jong, G., Scheiner, S. M., Schlichting, C. D., & Van Tienderen, P. H. (1995). Adaptive phenotypic plasticity: Consensus and controversy. Trends in Ecology & Evolution, 10, 212–217. Von Melchner, L., Pallas, S. L., & Sur, M. (2000). Visual behaviour mediated by retinal projections directed to the auditory pathway. Nature, 404, 871–876. Voss, P., Lassonde, M., Gougoux, F., Fortin, M., Guillemot, J. P., & Lepore, F. (2004). Early- and late-onset blind individuals show supra-normal auditory abilities in far-space. Current Biology, 14, 1734–1738. Watanabe, D., Savion-Lemieux, T., & Penhune, V. B. (2007). The effect of early musical training on adult motor performance: Evidence for a sensitive period in motor learning. Experimental Brain Research, 176, 332–340. Weisz, N., Müller, S., Schlee, W., Dohrmann, K., Hartmann, T., & Elbert, T. (2007). The neural code of auditory phantom perception. The Journal of Neuroscience, 27, 1479–1484. Weisz, N., Wienbruch, C., Dohrmann, K., & Elbert, T. (2005). Neuromagnetic indicators of auditory cortical reorganization of tinnitus. Brain, 128, 2722–2731. Will, B., Dalrymple-Alford, J., Wolff, M., & Cassel, J. C. (2008). The concept of brain plasticity—Paillard's systemic analysis and emphasis on structure and function (followed by the translation of a seminal paper by Paillard on plasticity). Behavioral Brain Research, 192, 2–7.
A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 191 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.
CHAPTER 13
Sensory integration for reaching: Models of optimality in the context of behavior and the underlying neural circuits Philip N. Sabes* Department of Physiology, Keck Center for Integrative Neuroscience, University of California, San Francisco, California, USA
Abstract: Although multisensory integration has been well modeled at the behavioral level, the link between these behavioral models and the underlying neural circuits is still not clear. This gap is even greater for the problem of sensory integration during movement planning and execution. The difficulty lies in applying simple models of sensory integration to the complex computations that are required for movement control and to the large networks of brain areas that perform these computations. Here I review psychophysical, computational, and physiological work on multisensory integration during movement planning, with an emphasis on goal-directed reaching. I argue that sensory transformations must play a central role in any modeling effort. In particular, the statistical properties of these transformations factor heavily into the way in which downstream signals are combined. As a result, our models of optimal integration are only expected to apply “locally,” that is, independently for each brain area. I suggest that local optimality can be reconciled with globally optimal behavior if one views the collection of parietal sensorimotor areas not as a set of task-specific domains, but rather as a palette of complex, sensorimotor representations that are flexibly combined to drive downstream activity and behavior. Keywords: sensory integration; reaching; neurophysiology; parietal cortex; computational models; vision; proprioception. parameter, for example, when one can feel and see an object touching one's arm. Understanding how the brain combines these signals has been an active area of research. As described below, models of optimal integration have been successful at capturing psychophysical performance in a variety of tasks. Further, network models have shown how optimal integration could be instantiated in neural circuits.
Introduction Multiple sensory modalities often provide “redundant” information about the same stimulus *Corresponding author. Tel.: þ415-476-0364; Fax: þ415-502-4848 E-mail:
[email protected] DOI: 10.1016/B978-0-444-53752-2.00004-7
195
196
However, strong links have yet to be made between these bodies of work and neurophysiological data. Here we address how models of optimal integration apply to the context of a sensory-guided movement and its underlying neural circuitry. This chapter focuses on the sensory integration required for goal-directed reaching and how that integration is implemented in the parietal cortex. We show that normative models developed for perceptual tasks and simple neural network models cannot, on their own, explain behavioral and physiological observations. These principles may nonetheless apply at a “local level” within each neuronal population. The link between local optimality and globally optimal behavior is then considered in the context of the broad network of sensorimotor areas in parietal cortex.
Modeling the psychophysics of sensory integration The principal hallmark of sensory integration should be the improvement of performance when multiple sensory signals are combined. In order to test this concept, we must choose a performance criterion by which to judge improvement. In the case of perception for action, the goal is often to estimate a spatial variable from the sensory input, for example, the location of the hand or an object in the world. In this case, the simplest and most commonly employed measure of performance is the variability of the estimate. It is not difficult to show that the minimum variance combination of two unbiased estimates of a variable x is given by the expression: ^ 1 1 1 x1 ^ x2 2 2 ^xinteg ¼ s þ ; s ¼ þ ; integ s21 s22 s21 s22 integ ð1Þ where ^ xi ; i ¼ 1; 2, are the unimodal estimates and si2 are their variances. In other words, the integrated estimate ^ xinteg is the weighted sum of
the two unimodal estimates, with weights inversely proportional to the respective variances. Importantly, the variance of the integrated estimate, sinteg2, is always less than either of the unimodal variances. While Eq. (1) assumes that the unimodal estimates ^xi are scalar and independent (given x), the solution is easily extended to correlated or multidimensional signals. Further, since the unimodal estimates are often well approximated by independent, normally distributed random variables, ^xinteg can also be viewed as the Maximum Likelihood (ML) integrated estimate (Ernst and Banks, 2002; Ghahramani et al., 1997). This model has been tested psychophysically by measuring performance variability with unimodal sensory cues and then predicting either variability or bias with bimodal cues. Numerous studies have reported ML-optimal or near-optimal sensory integration in human subjects performing perceptual tasks (e.g., Ernst and Banks, 2002; Ghahramani et al., 1997; Jacobs, 1999; Knill and Saunders, 2003; van Beers et al., 1999). Sensory integration during reach behavior Sensory integration is more complicated for movement planning than for a simple perceptual task. The problem is that movement planning and execution rely on a number of different computations, and estimates of the same spatial variable may be needed for several of these. For example, there is both psychophysical (Rossetti et al., 1995) and physiological (Batista et al., 1999; Buneo et al., 2002; Kakei et al., 1999, 2001) evidence for two separate stages of movement planning, as illustrated in Fig. 1. First, the movement vector is computed as the difference between the target location and the initial position of the hand. Next, the initial velocity along the planned movement vector must be converted into joint angle velocities (or other intrinsic variables such as muscle activations), which amounts to evaluating an inverse kinematic or
197 Movement vector planning
Inverse model evaluation
Execution
Target
Initial position
Actual movement direction
Planned movement vector
Planned joint angle changes
Fig. 1. Two separate computations required for reach planning. Adapted from Sober and Sabes (2005).
dynamic model. This evaluation also requires knowing the initial position of the arm. When planning a reaching movement, humans can often both see and feel the location of their hand. The ML model of sensory integration would seem to predict that the same weighting of vision and proprioception should be used for both of the computations illustrated in Fig. 1. However, we have previously shown that when reaching to visual targets, the relative weighting of these signals was quite different for the two computations: movement vector planning relied almost entirely on vision of the hand, and the inverse model evaluation relied more strongly on proprioception (Sober and Sabes, 2003). We hypothesized that the difference was due to the nature of the computations. Movement vector planning requires comparing the visual target location to the initial hand position. Since proprioceptive signals would first have to be transformed, this computation favors vision. Conversely, evaluation of the inverse model deals with intrinsic properties of the arm, favoring proprioception. Indeed, when subjects are asked to reach to a proprioceptive target (their other hand), the weighting of vision is significantly reduced in the movement vector calculation (Sober and Sabes, 2005). We hypothesized that these results are consistent with “local” ML integration, performed separately for each computation, if sensory transformations inject variability into the transformed signal. In order to make this hypothesis quantitative, we must understand the role of sensory
transformations during reach planning and their statistical properties. We developed and tested a model for these transformations by studying patterns of reach errors (McGuire and Sabes, 2009). Subjects made a series of interleaved reaches to visual targets, proprioceptive targets (the other hand, unseen), or bimodal targets (the other hand, visible), as illustrated in Fig. 2a. These reaches were made either with or without visual feedback of the hand prior to reach onset, during an enforced delay period after target presentation (after movement onset, feedback was extinguished in all trials). We took advantage of a bias in reaching that naturally occurs when subjects fixate a location distinct from the reach target. Specifically, when subjects reach to a visual target in the peripheral visual field, reaches tend to be biased further from the fixation point (Bock, 1993; Enright, 1995). This pattern of reach errors is illustrated in the left-hand panels of Fig. 2b: when reaching left of the fixation point, a leftward bias is observed, and similarly for the right. Thus, these errors follow a retinotopic pattern, that is, the bias curves shift with the fixation point. The bias pattern changes, but remains retinotopic, when reaching to bimodal targets (Fig. 2c) or proprioceptive targets (Fig. 2d). Most notably, the sign of the bias switches for proprioceptive reaches: subjects tend to reach closer to the point of fixation. Finally, the magnitude of these errors depends on whether visual feedback of the reaching hand is available prior to movement onset (compare the top and bottom panels of Fig. 2b–d; see also Beurze et al. (2007)).
198 (b)
Visual targets Fixation points
Unseen proprioceptive target (left fingertip)
No feedback Feedback Bias angle (°)
(a)
(c)
Visual targets
Bimodal targets
(d) Proprioceptive targets
2 0 2
2 0 2 20
0 Left fix
Reaching hand, shown with visual feedback of fingertip location
20 Right fix
(e)
20
0
20
Proprioception Target (t) Hand (h)
20
2 Retinotopic representation t ret, h ret
Transformation
Retinotopic movement vector MV ret t ret h ret
xret xbody g
Integrated movement vector
xbody xret g
Body-centered Representation t body, h body
Body-centered movement vector MV body t body h body
Standard deviation (°, mean subtracted)
Gaze (g)
0
Reach variability
(f) Vision Target (t) Hand (h)
20
Target angle (°)
1 0 1 2
Vis Bimod Prop No feedback
Vis Bimod Prop Feedback
Fig. 2. (a) Experimental setup. Left subpanel: Subjects sat in a simple virtual reality apparatus with a mirror reflecting images presented on a rear projection screen (Simani et al., 2007). View of both arms was blocked, but artificial feedback of either arm could be given in the form of a disk of light that moved with the fingertip. The right and left arms were separated by a thin table, allowing subject to reach to their left hand without tactile feedback. We were thus able to manipulate both the sensory modality of the target (visual, proprioceptive, or bimodal) and the presence or absence of visual feedback of the reaching hand. Right subpanel: For each target and feedback condition, reaches were made to an array of targets (displayed individually during the experiment) with the eyes fixated on one of two fixation points. (b–d) Reach biases. Average reach angular errors are plotted as a function of target and fixation location, separately for each trial condition (target modality and presence of visual feedback). Target modalities were randomly interleaved across two sessions, one with visual feedback and one without. Solid lines: average reach errors (with standard errors) across eight subjects for each trial condition. Dashed lines: model fits to the data. The color of the line indicates the gaze location. (d) Schematic of the Bayesian parallel representations model of reach planning. See text for details. (e) Reach variability. Average variability of reach angle plotted for each trial condition. Solid lines: average standard deviation of reach error across subjects for each trial condition. Dashed lines: model predictions. Adapted from McGuire and Sabes (2011).
While these bias patterns might seem arbitrary, they suggest an underlying mechanism. First, the difference in the sign of errors for visual and proprioceptive targets suggests that the bias arises in the transformation from a retinotopic (or eyecentered) representation to a body-centered representation. To see why, consider that in its simplified one-dimensional form, the transformation requires only adding or subtracting the gaze location (see the box labeled “Transformation”
in Fig. 2e). This might appear to be a trivial computation. However, the internal estimate of gaze location is itself an uncertain quantity. We argued that this estimate relies on current sensory signals (proprioception or efference copy) as well as on an internal prior that “expects” gaze to be coincident with the target. Thus, the estimate of gaze would be biased toward a retinally peripheral target. Since visual and proprioceptive information about target location travels in different directions
199
through this transformation, a biased estimate of gaze location results in oppositely signed errors for the two signals, as observed in Fig. 2b and d. Further, because the internal estimate of gaze location is uncertain, the transformation adds variability to the signal (see also Schlicht and Schrater, 2007), even if the addition or subtraction operation itself can be performed without error (not necessarily the case for neural computations, Shadlen and Newsome, 1994). One consequence of this variability is that access to visual feedback of the hand would improve the reliability of an eye-centered representation (upper pathway in Fig. 2e) more than it would improve the reliability of a body-centered representation (low pathway in Fig. 2e), since the latter receives a transformed, and thus more variable, version of the signal. Therefore, if the final movement plan were constructed from the optimal combination of an eye-centered and body-centered plan (rightmost box in Fig. 2e), the presence of visual feedback of the reaching hand should favor the eye-centered representation. This logic explains why the visual feedback of the reaching hand decreases the magnitude of the bias for visual targets (when the eye-centered space is unbiased; Fig. 2b) but increases the magnitude of the bias for proprioceptive targets (when the eyecentered space is biased; Fig. 2d). Together, these ideas form the Bayesian integration model of reach planning with “parallel representations,” illustrated in Fig. 2e. In this model, all sensory inputs related to a given spatial variable are combined with weights inversely proportional to their local variability (Eq. 1), and a movement vector is then computed. This computation occurs simultaneously in an eye-centered and a body-centered representation. The two resultant movement vectors have different uncertainties, depending on the availability and reliability of the sensory signals they receive in a given experimental condition. The final output of the network is itself a weighted sum of these two representations. We fit the four free parameters of the model (corresponding to values of sensory variability) to the reach error data
shown in solid lines in Fig. 2b–d. The model captures those error patterns (dashed lines in Fig. 2b–d) and predicts the error patterns from two similar studies described above (Beurze et al., 2007; Sober and Sabes, 2005). In addition, the model predicts the differences we observed in reach variability across experimental conditions (Fig. 2f). These results challenge the idea that movement planning should begin by mapping the relevant sensory signals into a single common reference frame (Batista et al., 1999; Buneo et al., 2002; Cohen et al., 2002). The model shows that the use of two parallel representations of the movement plan yields a less variable output in the face of variable and sometimes missing sensory signals and noisy internal transformations. It is not clear whether or how this model can be mapped onto the real neural circuits that underlie reach planning. For example, the two parallel representations could be implemented by a single neuronal population (Pouget et al., 2002; Xing and Andersen, 2000; Zipser and Andersen, 1988). Before addressing this issue, though, we consider the question of how single neurons or populations of neurons should integrate their afferent signals.
Modeling sensory integration in neural populations Stein and colleagues have studied multimodal responses in single neurons in the deep layers of cat superior colliculus and have found both enhancement and suppression of multimodal responses (Meredith and Stein, 1983, 1986; Stein and Stanford, 2008). Based on this work, they suggest that the definition of sensory integration at the level of the single unit is for the responses to be significantly enhanced or suppressed relative to the preferred unimodal stimulus (Stein et al., 2009). However, this definition is overly broad and includes computations that are not
200
typically thought of as integration. For example, Kadunce et al. (1997) showed that cross-modal suppressive effects in the superior colliculus often mimic those observed for paired within-modality stimuli. These effects are most likely due not to integration but rather to competition within the spatial map of the superior colliculus, similar to the process seen during saccade planning in primate superior colliculus (Dorris et al., 2007; Trappenberg et al., 2001). The criterion for signal integration should be the presence of a shared representation that offers improved performance (e.g., reduced variability) when multimodal inputs are available. Here, a “shared” representation is one that encodes all sensory inputs similarly. Using the notation of Eq. (1), the strongest form a shared representation is one in which neural activity is function only of xinteg and sinteg2, rather than being a function of the independent inputs, x1, s12 and x2, s22. The advantage of such a representation is that downstream areas need not know about which sensory signals were available in order to use the information. Ma et al. (2006) suggest a relatively simple approach to achieving such an integrated representation. They show that a population of neurons that simply adds the firing rates of independent input populations (or their linear transformations) effectively implements ML integration, at least when firing rates have Poisson-like distributions. This result can be understood intuitively for Poisson firing rates. The variance of the ML decode from each population is inversely proportional to its gain. Therefore, summing the inputs yields a representation with the summed gains, and thus with variance that matches the optimum defined in Eq. (1) above. Further, because addition preserves information about variability, this operation can be repeated hierarchically, a desirable feature for building more complex circuits like those required for sensory-guided movement. It remains unknown whether real neural circuits employ such a strategy, or even if they
combine their inputs in a statistically optimal manner. In practice, it can be difficult to quantitatively test the predictions of this and similar models. For example, strict additivity of the inputs is not to be expected in many situations, such as in the presence of inputs that are correlated or non-Poisson, or if activity levels are normalized within a given brain area (Ma et al., 2006; Ma and Pouget, 2008). These difficulties are compounded for recordings from single neurons. In this case, biases in the unimodal representations of space would lead to changes in firing rates across modalities even in the absence of integration-related changes in gain. Nonetheless, several hallmarks of optimal integration have been observed in the responses of bimodal (visual and vestibular) motion encoding neurons in macaque area MST: bimodal activity is well modeled as a weighted linear sum of unimodal responses; the visual weighting decreases when the visual stimulus is degraded; and the variability of the decoding improves in the bimodal condition (Morgan et al., 2008). Similarly, neurons in macaque Area 5 appear to integrate proprioceptive and visual cues of arm location (Graziano et al., 2000). In particular, the activity for a mismatched bimodal stimulus is between that observed for location-matched unimodal stimuli (weighting), and the activity for matched bimodal stimuli is greater than that observed for proprioception alone (variance reduction).
Sensory integration in the cortical circuits for reaching In the context of the neural circuits for reach planning, locally optimal integration of incoming signals could be sufficient to explain behavior, as illustrated in the parallel representations model of Fig. 2. Here we ask whether the principles in this model can be mapped onto the primate cortical reach network.
201
Volitional arm movements in primate involve a large network of brain areas with a rich pattern of interarea connectivity. Within this larger circuit, there is a subnetwork of areas, illustrated in Fig. 3, that appear to be responsible for the complex sensorimotor transformations required for goaldirected reaches under multisensory guidance. Visual information primarily enters this network via the parietal–occipital area, particularly Area V6 (Galletti et al., 1996; Shipp et al., 1998). Proprioceptive information primarily enters via Area 5, which receives direct projections from primary somatosensory cortex (Crammond and Kalaska, 1989; Kalaska et al., 1983; Pearson and Powell, 1985). These visual and proprioceptive signals converge on a group of parietal sensorimotor areas in or near the intraparietal sulcus (IPS): MDP and 7m (Ferraina et al., 1997a,b; Johnson et al., 1996), V6a (Galletti et al., 2001; Shipp et al., 1998), and MIP and VIP (Colby et al., 1993; Duhamel et al., 1998). The parietal reach region (PRR), characterized physiologically by Andersen and colleagues (Batista et al., 1999; Snyder et al., 1997), includes portions of MIP, V6a, and MDP (Snyder et al., 2000a). These parietal areas project forward to the dorsal premotor cortex (PMd) and, in some cases, the primary motor cortex (M1), and they all exhibit some degree of activity related to visual and proprioceptive movement cues, the pending movement plan (“set” or “delay” activity), and the ongoing movement kinematics or dynamics. While the network illustrated in Fig. 3 is clearly more complex than the simple computational
V2, MT,
Vision MST, etc.
PO (V6)
schematic of Fig. 2e, there is a suggestive parallel. While both Area 5 and MIP integrate multimodal signals and project extensively to the rest of the reach circuit, they differ in their anatomical proximity to their visual versus proprioceptive inputs: Area 5 is closer to somatosensory cortex, and MIP is closer to the visual inputs to reach circuit. Further, Area 5 uses more body- or hand-centered representations compared to the eye-centered representations reported in MIP (Batista et al., 1999; Buneo et al., 2002; Chang and Snyder, 2010; Colby and Duhamel, 1996; Ferraina et al., 2009; Kalaska, 1996; Lacquaniti et al., 1995; Marconi et al., 2001; Scott et al., 1997). Thus, these areas are potential candidates for the parallel representations predicted in the behavioral model. To test this possibility, we recorded from Area 5 and MIP (Fig. 4a) as macaque monkeys performed the same psychophysical task that was illustrated in Fig. 2a for human subjects (McGuire and Sabes, 2011). One of the questions we addressed in this study is whether there is evidence for parallel representations of the movement plan in body and eye-centered reference frames. We performed several different analyses to characterize neural reference frames; here we focus on the tuning-curve approach illustrated in Fig. 4b. Briefly, we fit a tuning curve to the neural responses for a range of targets with two different fixation points (illustrated schematically as the red and blue curves in Fig. 4b). Tuning was assumed to be a function of TdE, where T and E are the target and eye locations in absolute (or body-centered) space and d is a dimensionless quantity. If d ¼ 0,
MDP, 7m V6a, MIP
VIP
Proprioception
Area 3b
Areas 1,2
Fig. 3. Schematic illustration of the cortical reach circuit.
Area 5
PMd
M1
Motor
202 (c)
(a)
Fraction of recorded tuning curves VIS n = 95 BIMODAL n = 121 PROP n = 144
CS PCD
0.2
Are
Area 5
a5
0.1 10 mm IPS
P↔A
0
0
1
Schematic tuning curves VIS n = 195
(d)
(b) Body-centered
Intermediate
BIMODAL n = 185
Eye-centered
PROP n = 168
d0
d 0.5
MIP
Firing rate
0.2
d1
0.1
Left Right Fix Fix 0
0 Target location (with respect to body)
0
0
0
1 Shift (d)
Fig. 4. (a) Recording locations. Approximate location of neural recordings in Area 5 with respect to sulcal anatomy. Recordings in MIP were located in the same region of cortex, but at a deeper penetration depth. The boundary between Area 5 and MIP was set to a nominal value of 2000 mm below the dura. This value was chosen based on anatomical MR images and the stereotactic coordinates of the recording cylinder. (b) Schematic illustration of the tuning-curve shift, d. Each curve represents the firing rate for an idealized cell as a function of target for either the left (red) or right (blue) fixation points. Three idealized cells are illustrated, with shift values of d ¼ 0, 0.5, and 1. See text for more details. (c, d) Distribution of shift values estimated from neural recordings in Area 5 (c) and MIP (d). Each cell may be included up to three times, once for the delay, reaction time, and movement epoch tuning curves. The number of tuning curves varies across modalities because tuning curves were only included when the confidence limit on the best-fit value of d had a range of less than 1.5, a conservative value that excluded untuned cells. Adapted from McGuire and Sabes (2011).
firing rate depends on the body-centered location of the target (left panel of Fig. 4b), and if d ¼ 1, firing rate depends on the eye-centered location of the target (right panel of Fig. 4b). We found a large degree of heterogeneity in the shift values across cells, but there was no difference in the mean or distribution of shift values across target modality for either cortical area
(Fig. 4c and d), that is, these are shared (modality-invariant) representations. Although some evidence for other shared movement-related representations have been found in the parietal cortex (Cohen et al., 2002), many studies of multisensory areas in the parietal cortex and elsewhere have found that representations are determined, at least in part, by the representation
203
of the current sensory input (Avillac et al., 2005; Fetsch et al., 2007; Jay and Sparks, 1987; Mullette-Gillman et al., 2005; Stricanne et al., 1996). Shared representations such as those we have observed have the advantage that downstream areas do not need to know which sensory signals are available in order to use the representation. We also observed significant differences in the mean and distribution of shift values across cortical areas, with MIP exhibiting a more eye-centered representation (mean d ¼ 0.51), while Area 5 has a more body-centered representation (mean d ¼ 0.25). In a separate analysis, we showed that more MIP cells encode target location alone, compared to Area 5, where more cells encode both target and hand location (McGuire and Sabes, 2011). These inter-area differences parallel observations from Andersen and colleagues of eye-centered target coding in PRR (Batista et al., 1999) and eye-centered movement vector representation for Area 5 (Buneo et al., 2002). However, where those papers report consistent, eye-centered reference frames, we observed a great deal of heterogeneity in representations within each area, with most cells exhibiting “intermediate” shifts between 0 and 1. We think this discrepancy lies primarily in the analyses used: the shift analysis does not force a choice between alternative reference frames, but rather allows for a continuum of intermediate reference frames. When an approach very similar to ours was applied to recordings from a more posterior region of the IPS, a similar spread of shift values was obtained, although the mean shift value was somewhat closer to unity (Chang and Snyder, 2010). While we did not find the simple eye- and bodycentered representations that were built into the parallel representations model of Fig. 4d, these physiological results can nonetheless be interpreted in light of that model. We found that both Area 5 and MIP use modality-invariant representations of the movement plan, an important feature of the model. Further, there are multiple integrated representations of the movement plan within the superior parietal lobe, with an anterior to posterior gradient in the magnitude of
gaze-dependent shifts (Chang and Snyder, 2010; McGuire and Sabes, 2011). A statistically optimal combination of these representations, dynamically changing with the current sensory inputs, would likely provide a close match to the output of the model. The physiological recordings also revealed a great deal of heterogeneity in shift values, suggesting an alternate implementation of the model. Xing and Andersen (2000) have observed that a network with a broad distribution of reference-frame shifts can be used to compute multiple simultaneous readouts, each in a different reference frame. Indeed, a broad distribution of gazedependent tuning shifts has been observed within many parietal areas (Avillac et al., 2005; Chang and Snyder, 2010; Duhamel et al., 1998; Mullette-Gillman et al., 2005; Stricanne et al., 1996). Thus, parallel representations of movement planning could also be implemented within a single heterogeneous population of neurons.
From local to global optimality We have adopted a simple definition of sensory integration, namely, improved performance when multiple sensory modalities are available— whether in a behavioral task or with respect to the variability of neural representations. This definition leads naturally to criteria for optimal integration such as the minimum variance/ML model of Eq. (1), and a candidate mechanism for achieving such optimality was discussed above (Ma et al., 2006). In the context of a complex sensorimotor circuit, a mechanism such as this could be applied at the local level to integrate the afferent signals at each cortical area, independently across areas. However, these afferent signals will include the effects of the specific combination of upstream transformations, and so such a model would only appear to be optimal at the local level. It remains an open question as to how locally optimal (or near-optimal) integration could lead to globally optimal (or near-optimal) behavior.
204
The parietal network that underlies reaching is part of a larger region along the IPS that subserves a wide range of sensorimotor tasks (reviewed, e.g., in Andersen and Buneo, 2002; Burnod et al., 1999; Colby and Goldberg, 1999; Grefkes and Fink, 2005; Rizzolatti et al., 1997). These tasks make use of many sensory inputs, each naturally linked to a particular reference frame (e.g., visual signals originate in a retinotopic reference frame), as well as an array of kinematic feedback signals needed to transform from one reference frame to another. In this context, it seems logical to suggest a series of representations and transformations, for example, from eye-centered to hand-centered space, as illustrated in Fig. 5a. This schema offers a great degree of flexibility, since the “right” representation would be available for any given task. An attractive hypothesis is that a schema such as this could be mapped onto the series of sensorimotor representations that lie along the IPS, for example, from the retinotopic visual maps in Area V6 (Fattori et al., 2009; Galletti et al., 1996) to the hand-centered grasp-related activity in AIP. The pure reference-frame representations illustrated in the schema of Fig. 5a are not consistent with the evidence for heterogeneous “intermediate” representations. However, the general schema of a sequence of transformations and representations might still be correct, since the neural circuits implementing these transformations need not represent these variables in the reference frames of their inputs, as illustrated by several network models of reference-frame transformations (Blohm et al., 2009; Deneve et al., 2001; Salinas and Sejnowski, 2001; Xing and Andersen, 2000; Zipser and Andersen, 1988). The use of network models such as these could reconcile the schema of Fig. 5a with the physiological data (Pouget et al., 2002; Salinas and Sejnowski, 2001). While this schema is conceptually attractive, it has disadvantages. As described above, each transformation will inject variability into the circuit. This variability would accrue along the
sequence of transformations, a problem that could potentially be avoided by “direct” sensorimotor transformations such as those proposed by Buneo et al. (2002). Further, in order not to lose fidelity along this sequence, all intermediate representations require comparably sized neuronal populations, even representations that are rarely directly used for behavior. Ideally, one would be able to allocate more resources to a retinotopic representation, for example, than an elbow-centered representation. An alternative schema is to combine many sensory signals into each of a small number of representations; in the limit, a single complex representational network could be used (Fig. 5b). It has been shown that multiple reference frames can be read out from a single network of neurons when those neurons use “gain-field” representations, that is, when their responses are multiplicative in the various input signals (Salinas and Abbott, 1995; Salinas and Sejnowski, 2001; Xing and Andersen, 2000). More generally, nonlinear basis functions create general purpose representations that can be used to compute (at least approximately) a wide range of task-relevant variables (Pouget and Sejnowski, 1997; Pouget and Snyder, 2000). In particular, this approach would allow “direct” transformations from sensory to motor variables (Buneo et al., 2002) without the need for intervening sequences of transformations. However, this schema also has limitations. In order to represent all possible combinations of variables, the number of required neurons increases exponentially with the number of input variables (the “curse-of-dimensionality”). Indeed, computational models of such generic networks show a rapid increase in errors as the number of input variables grow (Salinas and Abbott, 1995). This limitation becomes prohibitive when the number of sensorimotor variables approaches a realistic value. A solution to this problem, illustrated in Fig. 5c, is to have a large number of networks, each with only a few inputs or encoding only a small subspace of the possible outputs. These representations
205
(a) Saccade
Gaze shift
Trunk movements
Eye-centered representation
Head-centered representation
Trunk-centered representation
Eye position
Vision
(b)
Saccade
Head position
Audition
Tactile inputs
Gaze shift
Trunk movements
Grasp
...
Tasks
Hand-centered representation Kinematic feedback for transformations
Wrist posture
Tactile inputs
Other sensory inputs
Grasp
Tasks
Tactile inputs
Kinematic feedback and other sensory inputs
Grasp
Tasks
High-dimensional integrated representation
Eye position Vision
(c)
Head position Audition
Saccade
Wrist posture
Tactile inputs
Gaze shift
Trunk movements
Low-dimensional integrated representations
...
Eye position Vision
Head position Audition
Tactile inputs
Wrist posture Tactile inputs
Kinematic feedback and other sensory inputs
Fig. 5. Three schematic models of the parietal representations of sensorimotor space. (a) A sequence of transformations that follows the kinematics of the body. Each behavior uses the representation that most closely matches the space of the task. (b) A single high-dimensional representation that integrates all of the relevant sensorimotor variables and subserves the downstream computations for all tasks. (c) A large collection of low-dimensional integrated representations with overlapping sensory inputs and a high degree of interconnectivity. Each white box represents a different representation of sensorimotor space. The nature of these representations is determined by their inputs, and their statistical properties (e.g., variability, gain) will depend on the sensory signals available at the time. The computations performed for any given task make use of several of these representations, with the relative weighting dynamically determined by their statistical properties.
206
would likely have complex or “intermediate” representations of sensorimotor space that would not directly map either to particular stages in the kinematic sequence (5A) or to the “right” reference frames for a set of tasks. Instead, the downstream circuits for behavior would draw upon several of these representations. This schema is consistent with the large continuum of representations seen along the IPS (reviewed, e.g., in Burnod et al., 1999), and the fact that the anatomical distinctions between nominal cortical areas in this region are unclear and remain a matter of debate (Cavada, 2001; Lewis and Van Essen, 2000). It is also consistent with the fact that there is a great deal of overlap in the pattern of cortical areas that are active during any given task, for example, saccade and reach activity have been observed in overlapping cortical areas (Snyder et al., 1997, 2000b) and grasp-related activity can be observed in nominally reach-related areas (Fattori et al., 2009). This suggests that brain areas around the IPS should not be thought of a set of task-specific domains (e.g., Andersen and Buneo, 2002; Colby and Goldberg, 1999; Grefkes and Fink, 2005), but rather as a palette of complex, sensorimotor representations. This picture suggests a mechanism by which locally optimal integration could yield globally optimal behavior, essentially a generalization of the parallel representations model of Fig. 4d. In both the parallel representations model and the schema of Fig. 5c, downstream motor circuits integrate overlapping information from multiple sensorimotor representations of space. For any specific instance of a behavior, the weighting of these representations should depend on their relative variability, perhaps determined by gain (Ma et al., 2006), and this variability would depend on the sensory and motor signals available at that time. If each of the representations in this palette contains a locally optimal mixture of its input signals, optimal weighting of the downstream projections from this palette could drive statistically efficient behavior.
Acknowledgments This work was supported by the National Eye Institute (R01 EY-015679) and the National Institute of Mental Health (P50 MH77970). I thank John Kalaska, Joseph Makin, and Matthew Fellows for reading and commenting on earlier drafts of this chapter.
Abbreviations ML MST MDP MIP VIP PMd M1 PRR IPS
maximum likelihood medial superior temporal area medial dorsal parietal area medial intraparietal area ventral intraparietal area dorsal premotor cortex primary motor cortex parietal reach region intraparietal sulcus
References Andersen, R. A., & Buneo, C. A. (2002). Intentional maps in posterior parietal cortex. Annual Review of Neuroscience, 25, 189–220. Avillac, M., Deneve, S., Olivier, E., Pouget, A., & Duhamel, J. R. (2005). Reference frames for representing visual and tactile locations in parietal cortex. Nature Neuroscience, 8, 941–949. Batista, A. P., Buneo, C. A., Snyder, L. H., & Andersen, R. A. (1999). Reach plans in eye-centered coordinates. Science, 285, 257–260. Beurze, S. M., de Lange, F. P., Toni, I., & Medendorp, W. P. (2007). Integration of target and effector information in the human brain during reach planning. Journal of Neurophysiology, 97, 188–199. Blohm, G., Keith, G. P., & Crawford, J. D. (2009). Decoding the cortical transformations for visually guided reaching in 3D space. Cerebral Cortex, 19(6), 1372–1393. Bock, O. (1993). Localization of objects in the peripheral visual field. Behavioural Brain Research, 56, 77–84. Buneo, C. A., Jarvis, M. R., Batista, A. P., & Andersen, R. A. (2002). Direct visuomotor transformations for reaching. Nature, 416, 632–636.
207 Burnod, Y., Baraduc, P., Battaglia-Mayer, A., Guigon, E., Koechlin, E., Ferraina, S., et al. (1999). Parieto-frontal coding of reaching: An integrated framework. Experimental Brain Research, 129, 325–346. Cavada, C. (2001). The visual parietal areas in the macaque monkey: Current structural knowledge and ignorance. Neuroimage, 14, S21–26. Chang, S. W., & Snyder, L. H. (2010). Idiosyncratic and systematic aspects of spatial representations in the macaque parietal cortex. Proceedings of the National Academy of Sciences of the United States of America, 107, 7951–7956. Cohen, Y. E., Batista, A. P., & Andersen, R. A. (2002). Comparison of neural activity preceding reaches to auditory and visual stimuli in the parietal reach region. Neuroreport, 13, 891–894. Colby, C. L., & Duhamel, J. R. (1996). Spatial representations for action in parietal cortex. Brain Research. Cognitive Brain Research, 5, 105–115. Colby, C. L., Duhamel, J. R., & Goldberg, M. E. (1993). Ventral intraparietal area of the macaque: Anatomic location and visual response properties. Journal of Neurophysiology, 69, 902–914. Colby, C. L., & Goldberg, M. E. (1999). Space and attention in parietal cortex. Annual Review of Neuroscience, 22, 319–349. Crammond, D. J., & Kalaska, J. F. (1989). Neuronal activity in primate parietal cortex area 5 varies with intended movement direction during an instructed-delay period. Experimental Brain Research, 76, 458–462. Deneve, S., Latham, P. E., & Pouget, A. (2001). Efficient computation and cue integration with noisy population codes. Nature Neuroscience, 4, 826–831. Dorris, M. C., Olivier, E., & Munoz, D. P. (2007). Competitive integration of visual and preparatory signals in the superior colliculus during saccadic programming. The Journal of Neuroscience, 27, 5053–5062. Duhamel, J. R., Colby, C. L., & Goldberg, M. E. (1998). Ventral intraparietal area of the macaque: Congruent visual and somatic response properties. Journal of Neurophysiology, 79, 126–136. Enright, J. T. (1995). The non-visual impact of eye orientation on eye-hand coordination. Vision Research, 35, 1611–1618. Ernst, M. O., & Banks, M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 415, 429–433. Fattori, P., Pitzalis, S., & Galletti, C. (2009). The cortical visual area V6 in macaque and human brains. Journal of Physiology (Paris), 103, 88–97. Ferraina, S., Brunamonti, E., Giusti, M. A., Costa, S., Genovesio, A., & Caminiti, R. (2009). Reaching in depth: Hand position dominates over binocular eye position in the rostral superior parietal lobule. The Journal of Neuroscience, 29, 11461–11470.
Ferraina, S., Garasto, M. R., Battaglia-Mayer, A., Ferraresi, P., Johnson, P. B., Lacquaniti, F., et al. (1997a). Visual control of hand-reaching movement: Activity in parietal area 7m. The European Journal of Neuroscience, 9, 1090–1095. Ferraina, S., Johnson, P. B., Garasto, M. R., BattagliaMayer, A., Ercolani, L., Bianchi, L., et al. (1997b). Combination of hand and gaze signals during reaching: Activity in parietal area 7 m of the monkey. Journal of Neurophysiology, 77, 1034–1038. Fetsch, C. R., Wang, S., Gu, Y., Deangelis, G. C., & Angelaki, D. E. (2007). Spatial reference frames of visual, vestibular, and multimodal heading signals in the dorsal subdivision of the medial superior temporal area. The Journal of Neuroscience, 27, 700–712. Galletti, C., Fattori, P., Battaglini, P., Shipp, S., & Zeki, S. (1996). Functional demarcation of a border between areas V6 and V6A in the superior parietal gyrus of the macaque monkey. The European Journal of Neuroscience, 8, 30–52. Galletti, C., Gamberini, M., Kutz, D. F., Fattori, P., Luppino, G., & Matelli, M. (2001). The cortical connections of area V6: An occipito-parietal network processing visual information. The European Journal of Neuroscience, 13, 1572–1588. Ghahramani, Z., Wolpert, D. M., & Jordan, M. I. (1997). Computational models of sensorimotor integration. In P. G. Morasso & V. Sanguineti (Eds.), Self-organization, computational maps and motor control (pp. 117–147). Oxford: Elsevier. Graziano, M. S., Cooke, D. F., & Taylor, C. S. (2000). Coding the location of the arm by sight. Science, 290, 1782–1786. Grefkes, C., & Fink, G. R. (2005). The functional organization of the intraparietal sulcus in humans and monkeys. Journal of Anatomy, 207, 3–17. Jacobs, R. A. (1999). Optimal integration of texture and motion cues to depth. Vision Research, 39, 3621–3629. Jay, M. F., & Sparks, D. L. (1987). Sensorimotor integration in the primate superior colliculus. II. Coordinates of auditory signals. Journal of Neurophysiology, 57, 35–55. Johnson, P. B., Ferraina, S., Bianchi, L., & Caminiti, R. (1996). Cortical networks for visual reaching: Physiological and anatomical organization of frontal and parietal lobe arm regions. Cerebral Cortex, 6, 102–119. Kadunce, D. C., Vaughan, J. W., Wallace, M. T., Benedek, G., & Stein, B. E. (1997). Mechanisms of within- and crossmodality suppression in the superior colliculus. Journal of Neurophysiology, 78, 2834–2847. Kakei, S., Hoffman, D. S., & Strick, P. L. (1999). Muscle and movement representations in the primary motor cortex. Science, 285, 2136–2139. Kakei, S., Hoffman, D. S., & Strick, P. L. (2001). Direction of action is represented in the ventral premotor cortex. Nature Neuroscience, 4, 1020–1025.
208 Kalaska, J. F. (1996). Parietal cortex area 5 and visuomotor behavior. Canadian Journal of Physiology and Pharmacology, 74, 483–498. Kalaska, J. F., Caminiti, R., & Georgopoulos, A. P. (1983). Cortical mechanisms related to the direction of two-dimensional arm movements: Relations in parietal area 5 and comparison with motor cortex. Experimental Brain Research, 51, 247–260. Knill, D. C., & Saunders, J. A. (2003). Do humans optimally integrate stereo and texture information for judgments of surface slant? Vision Research, 43, 2539–2558. Lacquaniti, F., Guigon, E., Bianchi, L., Ferraina, S., & Caminiti, R. (1995). Representing spatial information for limb movement: Role of area 5 in the monkey. Cerebral Cortex, 5, 391–409. Lewis, J. W., & Van Essen, D. C. (2000). Mapping of architectonic subdivisions in the macaque monkey, with emphasis on parieto-occipital cortex. Journal of Comparative Neurology, 428, 79–111. Ma, W. J., Beck, J. M., Latham, P. E., & Pouget, A. (2006). Bayesian inference with probabilistic population codes. Nature Neuroscience, 9, 1432–1438. Ma, W. J., & Pouget, A. (2008). Linking neurons to behavior in multisensory perception: A computational review. Brain Research, 1242, 4–12. Marconi, B., Genovesio, A., Battaglia-Mayer, A., Ferraina, S., Squatrito, S., Molinari, M., et al. (2001). Eye-hand coordination during reaching. I. Anatomical relationships between parietal and frontal cortex. Cerebral Cortex, 11, 513–527. McGuire, L. M., & Sabes, P. N. (2009). Sensory transformations and the use of multiple reference frames for reach planning. Nature Neuroscience, 12, 1056–1061. McGuire, L. M. M., & Sabes, P. N. (2011). Heterogeneous representations in the superior parietal lobule are common across reaches to visual and proprioceptive targets. The Journal of Neuroscience, 31(18), 6661–6673. Meredith, M. A., & Stein, B. E. (1983). Interactions among converging sensory inputs in the superior colliculus. Science, 221, 389–391. Meredith, M. A., & Stein, B. E. (1986). Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. Journal of Neurophysiology, 56, 640–662. Morgan, M. L., Deangelis, G. C., & Angelaki, D. E. (2008). Multisensory integration in macaque visual cortex depends on cue reliability. Neuron, 59, 662–673. Mullette-Gillman, O. A., Cohen, Y. E., & Groh, J. M. (2005). Eye-centered, head-centered, and complex coding of visual and auditory targets in the intraparietal sulcus. Journal of Neurophysiology, 94, 2331–2352. Pearson, R. C., & Powell, T. P. (1985). The projection of the primary somatic sensory cortex upon area 5 in the monkey. Brain Research, 356, 89–107.
Pouget, A., Deneve, S., & Duhamel, J. R. (2002). A computational perspective on the neural basis of multisensory spatial representations. Nature Reviews. Neuroscience, 3, 741–747. Pouget, A., & Sejnowski, T. J. (1997). Spatial transformations in the parietal cortex using basis functions. Journal of Cognitive Neuroscience, 9, 222–237. Pouget, A., & Snyder, L. H. (2000). Computational approaches to sensorimotor transformations. Nature Neuroscience, 3, 1192–1198. Rizzolatti, G., Fogassi, L., & Gallese, V. (1997). Parietal cortex: From sight to action. Current Opinion in Neurobiology, 7, 562–567. Rossetti, Y., Desmurget, M., & Prablanc, C. (1995). Vectorial coding of movement: Vision, proprioception, or both? Journal of Neurophysiology, 74, 457–463. Salinas, E., & Abbott, L. F. (1995). Transfer of coded information from sensory to motor networks. The Journal of Neuroscience, 15, 6461–6474. Salinas, E., & Sejnowski, T. J. (2001). Gain modulation in the central nervous system: Where behavior, neurophysiology, and computation meet. The Neuroscientist, 7, 430–440. Schlicht, E. J., & Schrater, P. R. (2007). Impact of coordinate transformation uncertainty on human sensorimotor control. Journal of Neurophysiology, 97, 4203–4214. Scott, S. H., Sergio, L. E., & Kalaska, J. F. (1997). Reaching movements with similar hand paths but different arm orientations. II. Activity of individual cells in dorsal premotor cortex and parietal area 5. Journal of Neurophysiology, 78, 2413–2426. Shadlen, M. N., & Newsome, W. T. (1994). Noise, neural codes and cortical organization. Current Opinion in Neurobiology, 4, 569–579. Shipp, S., Blanton, M., & Zeki, S. (1998). A visuosomatomotor pathway through superior parietal cortex in the macaque monkey: Cortical connections of areas V6 and V6A. The European Journal of Neuroscience, 10, 3171–3193. Simani, M. C., McGuire, L. M., & Sabes, P. N. (2007). Visualshift adaptation is composed of separable sensory and taskdependent effects. Journal of Neurophysiology, 98, 2827–2841. Snyder, L. H., Batista, A. P., & Andersen, R. A. (1997). Coding of intention in the posterior parietal cortex. Nature, 386, 167–170. Snyder, L. H., Batista, A. P., & Andersen, R. A. (2000a). Intention-related activity in the posterior parietal cortex: A review. Vision Research, 40, 1433–1441. Snyder, L. H., Batista, A. P., & Andersen, R. A. (2000b). Saccade-related activity in the parietal reach region. Journal of Neurophysiology, 83, 1099–1102. Sober, S. J., & Sabes, P. N. (2003). Multisensory integration during motor planning. The Journal of Neuroscience, 23, 6982–6992.
209 Sober, S. J., & Sabes, P. N. (2005). Flexible strategies for sensory integration during motor planning. Nature Neuroscience, 8, 490–497. Stein, B. E., & Stanford, T. R. (2008). Multisensory integration: Current issues from the perspective of the single neuron. Nature Reviews. Neuroscience, 9, 255–266. Stein, B. E., Stanford, T. R., Ramachandran, R., Perrault, T. J., Jr.& Rowland, B. A. (2009). Challenges in quantifying multisensory integration: Alternative criteria, models, and inverse effectiveness. Experimental Brain Research, 198, 113–126. Stricanne, B., Andersen, R. A., & Mazzoni, P. (1996). Eyecentered, head-centered, and intermediate coding of remembered sound locations in area LIP. Journal of Neurophysiology, 56, 2071–2076.
Trappenberg, T. P., Dorris, M. C., Munoz, D. P., & Klein, R. M. (2001). A model of saccade initiation based on the competitive integration of exogenous and endogenous signals in the superior colliculus. Journal of Cognitive Neuroscience, 13, 256–271. van Beers, R. J., Sittig, A. C., & Denier van der Gon, J. J. (1999). Integration of proprioceptive and visual positioninformation: An experimentally supported model. Journal of Neurophysiology, 81, 1355–1365. Xing, J., & Andersen, R. A. (2000). Models of the posterior parietal cortex which perform multimodal integration and represent space in several coordinate frames. Journal of Cognitive Neuroscience, 12, 601–614. Zipser, D., & Andersen, R. A. (1988). A back-propagation programmed network that simulates response properties of a subset of posterior parietal neurons. Nature, 331, 679–684.
A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 191 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.
CHAPTER 14
Sensory rehabilitation in the plastic brain Olivier Collignon{,{,*, François Champoux}, Patrice Voss{ and Franco Lepore{ { {
Centre de Recherche en Neuropsychologie et Cognition (CERNEC), Université de Montréal, Montréal, Québec, Canada Centre de Recherche CHU Sainte-Justine, Université de Montréal, Montréal, Québec, Canada } Centre de Recherche Interdisciplinaire en Réadaptation du Montréal Métropolitain, Institut Raymond-Dewar, Montréal, Québec, Canada
Abstract: The purpose of this review is to consider new sensory rehabilitation avenues in the context of the brain's remarkable ability to reorganize itself following sensory deprivation. Here, deafness and blindness are taken as two illustrative models. Mainly, two promising rehabilitative strategies based on opposing theoretical principles will be considered: sensory substitution and neuroprostheses. Sensory substitution makes use of the remaining intact senses to provide blind or deaf individuals with coded information of the lost sensory system. This technique thus benefits from added neural resources in the processing of the remaining senses resulting from crossmodal plasticity, which is thought to be coupled with behavioral enhancements in the intact senses. On the other hand, neuroprostheses represent an invasive approach aimed at stimulating the deprived sensory system directly in order to restore, at least partially, its functioning. This technique therefore relies on the neuronal integrity of the brain areas normally dedicated to the deprived sense and is rather hindered by the compensatory reorganization observed in the deprived cortex. Here, we stress that our understanding of the neuroplastic changes that occur in sensory-deprived individuals may help guide the design and the implementation of such rehabilitative methods. Keywords: blindness; deafness; neuroplasticity; rehabilitation; sensory substitution; neuroprosthesis.
evolution. It is likely that the apparent regularity and homogeneity of cortical anatomy have prolonged this conception of an immutable brain. However, results acquired mainly in the past two decades have led to the recognition that the developing, and even adult, brain has a remarkable ability to remodel and restructure the different circuits within it, based on learning and experience. This concept, called
Introduction It has long been believed that the brain is hard-wired, in a predetermined manner mainly shaped by *Corresponding author. Tel.: þ1-514-343-6111x2667; Fax: þ1-514-343-5787 E-mail:
[email protected] DOI: 10.1016/B978-0-444-53752-2.00003-5
211
212
neuroplasticity, is opening up exciting new fields of research based on the brain's ability to constantly adapt itself to its environment throughout life. Recognizing the dynamic nature of cortical circuitry is important in understanding how the nervous system adapts after sensory deprivation. Pioneering studies of Wiesel and Hubel (1965, 1974) on the development of ocular dominance columns have compellingly demonstrated that alterations in visual experience can influence the normal development of the visual cortex. Other seminal experiments have also shown that cortical maps can change/expand with use; for example, the representation of the finger tips in the somatosensory cortex has been shown to expand after a period of intense stimulation (Kaas et al., 1983), as observed in proficient Braille blind readers (Pascual-Leone and Torres, 1993; Sterr et al., 1998). Similarly, the tonotopic map in the auditory cortex is larger in musicians (Pantev et al., 1998) and visually deprived individuals (Elbert et al., 2002). Aside from such intramodal plasticity, massive crossmodal changes have also been observed in sensory-deprived cortex (Bavelier and Neville, 2002; Pascual-Leone et al., 2005). Striking evidence that external inputs can determine the functional role of a sensory cortex has come from experiments on “rewired” animals. For instance, by making a series of brainstem lesions, researchers surgically rerouted visual input toward primary somatosensory or auditory areas (Frost and Metin, 1985; Frost et al., 2000; Roe et al., 1990; Sur et al., 1988). These experiments demonstrated that cells from the rewired regions shared some structural and functional similarities with cells recorded in the visual cortex of normally raised animals. Moreover, these authors demonstrated that these newly visual cells also mediated visually guided behavior (Frost et al., 2000; von Melchner et al., 2000). Taken together, these data suggest that primary cortical areas can change their functional specificity depending on which inputs they receive. Indeed, the observation that “visual” regions can be recruited for nonvisual processing
in blind subjects (Sadato et al., 1996; WanetDefalque et al., 1988) and that auditory regions can be recruited by nonauditory inputs in deaf subjects (Bavelier et al., 2001; Finney et al., 2001) has led to a change in how we think about the brain and its development in relation to experience. Importantly, these findings also demonstrate that these plastic changes are compensatory in nature because they appear to underlie improved abilities in the remaining senses of sensory-deprived individuals (Amedi et al., 2003; Bavelier et al., 2000, 2006; Collignon et al., 2006, 2009b; Gougoux et al., 2005). Overall, these results point to the important role of sensory experience in the development and the maintenance of sensory brain functions. This has major implications, given current developments in sensory rehabilitation technologies, whether they are of the invasive type or not (Veraart et al., 2004; see Fig. 1). Invasive interventions rely on the integrity of the deprived system. Plastic reorganization that occurs all along the sensory pathway after deprivation is therefore likely to interfere with the reacquisition of the initial function of the system (Merabet et al., 2005). Indeed, in addition to the technical and surgical challenge of sensory restoration, there exists a neuropsychological one: how will the restored sensory input be interpreted by the reorganized sensory cortex? In contrast, sensory substitution refers to the use of one sensory modality to supply information normally gathered from another sense (Bach-y-Rita and Kercel, 2003). In so doing, sensory substitution devices can take advantage of the crossmodal plasticity observed in deprived individuals whereby deafferented areas provide the neural basis for behavioral compensation reported in the preserved senses (Amedi et al., 2003; Gougoux et al., 2005). Indeed, studies on how the brain changes following sensory deprivation are not only central to our understanding of the development of brain function but are also crucial to the development of adequate and successful rehabilitation strategies in case of sensory alterations.
213 Sensory environment
Artificial organ
Artificial transduction
Preserved organ
Restored organ
Altered organ
Neurostimulator
Artificial organ
Primary sensory pathway and brain regions Simplification and coding
Simplification and coding Associative brain regions
Motor output Sensory substitution
Sensory deprived individual
Invasive neuroprostheses
Fig. 1. Model of rehabilitation procedures for sensory-deprived individuals. The middle section represents a sensory-deprived person for whom environmental information can be transmitted to the brain by means of a remaining modality after sensory substitution (left panel), surgical restoration of the defective organ, or by the use of an implanted neuroprosthesis stimulating the deficient sensory system (right panel). With sensory substitution, the environmental inputs usually gathered by the defective sense is simplified and coded in order to be manipulated in a preserved remaining modality. With neuroprostheses, the lacking sensory information is simplified and coded into electrical impulses to stimulate the fully or partly preserved part of the deficient sense.
Rehabilitation in blindness Early visual deprivation causes atrophy in the optic tracts and radiations as well as massive gray and white matter volume reduction in early visual areas (Noppeney et al., 2005; Pan et al., 2007; Park et al., 2009; Ptito et al., 2008b; Shu et al., 2009). Although increased cortical thickness of occipital cortex has also been reported in the blind (Jiang et al., 2009; Park et al., 2009), it is believed to reflect the reduced surface area of the primary and secondary visual cortices (Park et al., 2009). In addition to these structural changes, visual deprivation enables a new role for the visual cortex in that it becomes responsive to nonvisual inputs (Bavelier and Neville, 2002). Moreover, a growing number of studies show that the recruitment of the deafferented visual areas during nonvisual tasks is not simply an epiphenomenon. First, these changes are thought to underpin superior nonvisual abilities often
observed in blind individuals as several studies have shown positive correlations between nonvisual performance and occipital activity: the most efficient blind participants are the ones who recruit occipital regions the most (Amedi et al., 2003; Gougoux et al., 2005). Second, transient disruption of occipital activity induced by transcranial magnetic stimulation (TMS) disrupts nonvisual abilities, further demonstrating the functional role of occipital regions of congenitally blind subjects in nonvisual processing (Amedi et al., 2004; Cohen et al., 1997; Collignon et al., 2007, 2009a). Finally, some aspects of the functional architecture present in the occipital cortex of sighted subjects appear to be preserved in the blind (Collignon et al., 2009b, Dormal et al., 2011). For example, the “visual” dorsal stream appears to maintain its preferential coding for spatial processing (Collignon et al., 2007, 2011; Renier et al., 2010; Fig. 2), the ventral stream for the processing of the identity of the input
(a)
(b)
(c)
Auditory discrimination tasks
Sham rTMS
(d)
(e)
(f)
Intensity
Pitch
15
Localization
15
10
5
0
* Error rate (%)
Error rate (%)
Error rate (%)
15
Real rTMS
10
5
0
Sighted
10
5
0
Blind
Sighted
Blind
Sighted
Blind
PSVA tasks (g)
(h)
(i)
Figure Recognition
Exploration Time
100
95
*
90
*
90 85 80
85 time (s)
% correct
95
80 75 70
75
65
70
60 Sighted
Blind
Sighted
Blind
Fig. 2. Prosthesis substituting vision by audition (PSVA). (a) A head-worn video camera (fixed on glasses) allows online translation of visual patterns into sounds that are transmitted to the subject through headphones. (b) The artificial retina provided by the PSVA. The acquired image is divided into pixels according to a 2-resolution artificial retina scheme. The central part of the processed image or fovea has a four times higher resolution than the periphery. The coding scheme is based on a pixel–frequency association. Pixels in use are drawn with a bold border. Frequency is indicated in hertz in the lower part of the used pixels. A single sinusoidal tone is assigned to each pixel of the multiresolution image. The amplitude of each sine wave (the intensity of each sound) is modulated by the gray level of the corresponding pixel. The pattern moves on the grid according to the head movements of the subject, and the corresponding sounds of the activated pixels are transmitted to the subject in real time. (c) Examples of patterns used in the experiments. The second part of the figure denotes the average error rate in blind and sighted subjects after sham and real TMS targeting the dorsal occipital stream during auditory tasks involving discrimination of intensity (d), pitch (e), and spatial location (f). The data show a significant increase of the error rate after real rTMS only in the blind group and selectively for the sound location task. Also, the figure displays the average percentage of correct pattern recognition (g) and the mean exploration time (h) taken to recognize patterns with the PSVA. The data indicate a significant decrease of recognition score and a significant increase of exploration time after real compared to sham TMS in the blind group only. Panel (i) displays the projection of the site of TMS application. This area corresponds to the right dorsal extrastriate occipital cortex (BA 18). Adapted with permission from Collignon et al. (2007).
215
(Amedi et al., 2007; Gougoux et al., 2009), and hMTþ/V5 for processing movement (Bedny et al., 2010; Poirier et al., 2004; Ricciardi et al., 2007). Taken together, these structural and functional changes in “visual” areas of early-blind individuals are thought to induce permanent changes in visual capabilities (Maurer et al., 2005). For example, the ability to elicit phosphenes with application of TMS over the occipital cortex (a measure of visual cortex excitability) is dramatically reduced in congenitally blind individuals (Gothe et al., 2002). Sight restoration with surgery The study of adult sight-recovery patients after early-onset blindness, even if extremely rare, has served as an important testing ground for hypotheses about the role of experience in shaping the functional architecture of the brain. These studies have demonstrated that early visual deprivation permanently and deeply affects visual functions (Fine et al., 2003; Gregory, 2003; Levin et al., 2010). Probably the most famous case report concerns patient SB, studied by Richard Gregory (Gregory and Wallace, 1963). SB lost his sight at 10 months of age before regaining it at 52 years of age, by means of a corneal graft. Despite the fact that the visual world now mapped correctly on his retina, SB had severe problems interpreting what he saw. Perception of depth was notably problematic (i.e., Necker's cube appeared flat) and he was only able to recognize faces when they moved. SB continued to rely on audition and touch to interact with his environment and situations that he managed very well while blind, like crossing a street in traffic, suddenly became problematic for him because of the presence of concurrent confusing visual information. Shortly after implantation, he became clinically depressed, probably due to his change of status from a successful blind to an unsuccessful sighted person (Gregory and Wallace, 1963). Another fascinating case was documented more
recently in the literature, patient MM, who was blind since the age of 3 years and who had his sight restored at 43 years of age, thanks to stem cell transplant (Fine et al., 2003). MM also had considerable difficulty perceiving depth and perceiving the specific details of objects, including faces. Even 7 years after the intervention, MM still had poor spatial resolution and limited visual abilities that did not allow him to rely on his vision in day-to-day activities (Levin et al., 2010). Imaging studies of MM showed extensive cortical reorganization, even after implantation, which may play a role in his visual difficulties (Fine et al., 2003; Levin et al., 2010; Saenz et al., 2008; Fig. 3). This is hypothesized to be due to an absence of mature cells coding for “fine” details because these cells were still not tuned at 3 years of age when MM lost his sight (Levin et al., 2010). In contrast to visual acuity and form or face perception, visual motion ability appeared relatively preserved after vision restoration in both SB and MM, with robust and specific brain activations for visual motion stimuli having been observed in subject MM (Fine et al., 2003; Levin et al., 2010; Sacks, 1995; Saenz et al., 2008). This is thought to be due to the fact that motion processing develops very early in infancy compared to form processing and might therefore have been more established and robust, allowing its preservation despite many years of visual deprivation (Fine et al., 2003). It was also shown that robust and specific crossmodal auditory motion responses coexist with regained visual motion responses in area hMTþ/V5 after sight restoration in subject MM (Saenz et al., 2008). However, it was not ascertained if the presence of such crossmodal auditory motion responses competes with or improves visual motion perception after recovery, nor whether the interaction between these two senses is enhanced or decreased due to interference (see our related discussion in the cochlear implant (CI) section below). This question is of major importance because the challenge for MM is to use the strong nonvisual skills he developed
216 (1.a)
(1.b) 10 deg 5 deg
Psychophysics
(3)
FMRI
(5.a)
Auditory motion MT + overlapping MT + non overlapping
5
0.2
2
0.5
Projected amplitude (% signal change)
0.1
Threshold (% contrast)
Sensitivity (% contrast)
0.6 10
1
1
(2)
0.4 0.3 0.2 0.1 0 –0.1
3 0.25 1 2 Spatial frequency (c.p.d.) MM + 5 m MM + 11 m MM + 17 m MM + 21 m
Sighted controls
0.5
MM
0.25 1 2 3 Spatial frequency (c.p.d.) V1 MT+
Control
MM Control
(4)
MM
OcP
AB
OcP
(5.b) MM Sighted controls Monocular Blind
% fMRI signal change
Radial (mm2/msec)
1.5
1
Posterior 0.5 1.25
Medial 1.5 1.75 2 2.25 Longitudinal (mm2/msec)
1
0.4 0 0.4 Correlation
1
Visual motion Auditory motion (ILD)
** ** 1
**
** **
**
** **
0.5 0 –0.5 c1 c2 c3 c4 c5 c6
MM MS
Fig. 3. Patchwork of different studies carried out with MM, an early-blind person who recovered sight at 43 years. Altogether, the results show major alteration in visual processing in this subject. (1.a) MM's sensitivity as a function of spatial frequency measured psychophysically 5–21 months after surgery. (1.b) Neural responses as a function of spatial frequency measured using fMRI in MT þ (dashed line) and V1 (solid line). (2) Comparison of radial and longitudinal diffusivities in the optic tracts and optic radiations (a) Three-dimensional rendering of the optic tract fibers (blue) shown superimposed on axial and coronal slices of MM's brain. The optic tracts connect the optic chiasm and the LGN (white sphere). Scatter plot of the radial and longitudinal diffusivities for the average of the right and left optic tracts. Data are from MM (gray star), 10 normal controls (black open circles), two seeing monocular subjects (black asterisks), and one blind subject (black closed circle). The 2 standard deviation covariance ellipsoid (dashed) is shown. (3) Visual field eccentricity representations in medial-ventral and dorsal-lateral cortex visual field eccentricity maps in lateral-occipital surface of MM's left (left panel) and right (right panel) hemispheres. Several extrastriate regions respond unusually to foveal stimuli. The right hemisphere shows some regions and a color map defining the visual field eccentricity representations.(4) Left hemisphere activation in response to faces versus objects with red–orange regions that responded more to faces and green–blue regions that responded more to objects. A control subject (AB) showed a typical pattern of activation, with large contiguous regions that responded more either to faces or objects near the fusiform gyrus (FuG) and lingual gyrus (LiG). In contrast, MM showed little activity to objects, and almost no activity to faces. (5.a) Surface maps of auditory and visual motion responses in MT for MM and sighted controls. Yellow regions responded more to moving versus stationary auditory white noise. Green and blue regions show MT location as determined by a visual MT localizer scans run in the same subjects (green, MT overlapped by auditory ILD motion responses; blue, MT not overlapped by auditory ILD motion responses). Note the nearcomplete overlap (very little blue) in subject MM indicating colocalization of MT for auditory motion processing. Adapted with permission from Fine et al. (2003; parts 1 and 4), Levin et al. (2010; parts 2 and 3), and Saenz et al. (2008; part 5).
as a proficient blind subject (sensory compensation in the remaining senses) in conjunction with his rudimentary vision in order to improve his use of visual functions. Indeed, knowledge of how visual and auditory responses interact in sight-recovery patients is important for optimizing patients’ use of their restored vision (Saenz et al., 2008).
The study of children treated for congenital bilateral cataracts after varying periods of visual deprivation presents the opportunity to examine the fundamental role of visual inputs for the normal development of specific aspects of vision. Particular studies on this topic have shed light on the fact that different visual abilities have various sensitive periods during which the absence
217
of visual inputs permanently impairs the investigated process. For example, even when treated for congenital bilateral cataracts before the first 6 months of age, permanent deficits in sensitivity to global motion have been shown to develop (Ellemberg et al., 2002; Lewis and Maurer, 2005), as well as for holistic face processing (Le Grand et al., 2001, 2004). However, the loss of sight after 6 months of age preserves the global detection of motion even if the period of blindness is extended as shown in patients MM and SB (Fine et al., 2003; Gregory and Wallace, 1963) but still can dramatically impair acuity, peripheral light sensitivity, and object and face processing (Fine et al., 2003; Levin et al., 2010; Lewis and Maurer, 2005; Gregory and Wallace, 1963). Strikingly, in some visual domains, visual input is necessary throughout the period of normal development and even after the age when performance reaches adult levels (Maurer et al., 2005). For instance, a short period of visual deprivation beginning any time before the age of 10 years causes permanent deficits in letter visual acuity, which normally reaches adult levels by the age of 6 years (Lewis and Maurer, 2005). Similarly, short periods of deprivation beginning even in early adolescence cause permanent deficits in peripheral light sensitivity, which normally reaches adult functional levels by 7 years of age (Bowering et al., 1993). It thus appears that visual input is necessary not only for the development but also for the consolidation of some visual connections (Lewis and Maurer, 2005). Regarding multisensory integration abilities, recent studies conducted in bilateral congenital cataract patients treated within the first two years of life demonstrated that visual input in early infancy is also a prerequisite for the normal development of multisensory functions (Putzar et al., 2007, 2010). Even if some studies demonstrated that the human brain retains an impressive capacity for visual learning well into late childhood (Ostrovsky et al., 2006, 2009), an important point raised by these studies in sightrestored patients is that early intervention is often a good predictor of visual abilities in adults. In the
particular case of congenital blindness, sight restoration in adults may be less miraculous than intuitively expected, probably because of the deterioration of visual tracts and massive crossmodal plasticity observed in the visual cortex of these persons (Noppeney, 2007). Sensory substitution in the blind The fact that the crossmodal recruitment of visually deafferented occipital areas effectively contributes to the processing of nonvisual inputs offers a real opportunity for rehabilitation via sensory substitution. Indeed, this fact has been intuitively exploited in numerous rehabilitation programs aimed at promoting nonvisual skills. Since it was discovered that the enrichment of the environment is an effective means of dramatically enhancing crossmodal plasticity associated with blindness (Piche et al., 2004), and because such reorganization mechanisms are thought to underlie enhanced perceptual skills in the blind (Amedi et al., 2003; Gougoux et al., 2005), orientation and mobility programs assume that they can help develop enhanced skills in the remaining senses of blind subjects though rehabilitation. These rehabilitation programs rely on the concept of sensory substitution, which refers to the use of one sensory modality to supply information normally gathered from another sense (Bach-y-Rita et al., 1969). The use of the long-cane as an extension of the body (Serino et al., 2007), the development of refined tactile discrimination in order to fluently read Braille dots (Van Boven et al., 2000; Wong et al., 2011), or the use of the reverberation of sounds to locate obstacles and discriminate object size (Dufour et al., 2005; Rice, 1967; Rice and Feinstein, 1965; Strelow and Brabyn, 1982) are excellent examples of such abilities that appear “supranormal” for a naïve sighted person but which are mastered by blind individuals due to a combination of extensive training programs and neuroplastic mechanisms. The Braille reading system is probably the best
218
example of these effects and massive involvement of the occipital cortex has been demonstrated in blind individuals when reading (Buchel, 1998; Burton et al., 2002; Sadato et al., 1996, 1998). Moreover, it has been shown that TMS over the occipital cortex of early-blind subjects disrupts Braille reading and even induces tactile sensations on the tip of the reading fingers in experienced users (Cohen et al., 1997; Kupers et al., 2007; Ptito et al., 2008a). Such findings demonstrate the functional involvement of the reorganized occipital cortex of blind subjects in Braille reading. This notion is even further supported by the reported case study of an expert blind Braille reader who lost her ability (Braille alexia) following an ischemic stroke which caused bilateral lesions to her occipital cortex (Hamilton et al., 2000). Aside from these classical rehabilitative programs, researchers have also considered providing blind people with new sensory-motor interactions with their environment in order to lower the impact of visual deprivation. Bach-y-Rita can arguably be seen as a visionary in the field since he had the idea in 1969 to design the first sensory substitution devices for the blind by using the preserved sense of touch to supply information usually gathered from vision (Bach-y-Rita et al., 1969). Since this seminal work, and partly due to subsequent technological improvements, several laboratories have been engaged in developing and testing new sensory substitution prosthesis (Bach-y-Rita et al., 1998; Capelle et al., 1998; Cronly-Dillon et al., 1999; Kaczmarek et al., 1985; Meijer, 1992). All these systems are designed to make use of the residual intact senses, mainly audition or touch, to provide blind people with a sample of the visual world that has been coded into another modality via specific algorithms that can be learned through practice (Veraart et al., 2004). These systems have proven their efficiency for the recognition of quite complex two-dimensional shapes (Arno et al., 1999, 2001b), to localize objects (Proulx et al., 2008; Renier and De Volder, 2010) or to navigate in a “virtual” environment
(Segond et al., 2005) and were found to massively and crossmodally recruit the occipital cortex of blind subjects (Amedi et al., 2007; De Volder et al., 1999; Kupers et al., 2010; Merabet et al., 2009; Poirier et al., 2007; Ptito et al., 2005). In our group, we investigated one such system, a prosthesis for substitution of vision by audition (PSVA) (Capelle et al., 1998). Early-blind participants were found to be more accurate when using the PSVA (Arno et al., 2001b) and their occipital cortex was more strongly activated than in the sighted in a pattern recognition task (Arno et al., 2001a). We also demonstrated that TMS interfered with the use of the PSVA when applied over the right dorsal extrastriate cortex of blind participants, probably due to the spatial cognitive components associated with the use of the prosthesis (Collignon et al., 2007). By contrast, TMS targeting the same cortical area had no effect on performance in sighted subjects (Fig. 2). As stated previously, we postulate that occipital regions are recruited in a compensatory crossmodal manner that may account for the superior abilities seen when using the prosthesis. The sensory substitution devices, therefore, constitute interesting noninvasive techniques, in great part because their working principles follow the natural tendency of the brain to reorganize itself in favor of the remaining sensory modalities. That being said, their principal drawback is that they are currently mainly dedicated to fundamental research on crossmodal reorganization; in their present form, there are no realistic opportunities for their introduction into the blind community. This is generally related to the poor ergonomic quality of such human–machine interfaces. In addition, the coding scheme may appear quite difficult, and the visual information gathered by the camera is generally too complex to be entirely recorded in the substitutive modality without creating a “noisy” percept. Indeed, laboratory settings where such systems are tested are extremely impoverished in order to avoid an excessive sensory and cognitive load when using such devices. These experimental situations are
219
usually composed of few target elements having a high figure-ground contrast (i.e., white shape on a black background). In the case of auditory devices, the technology appropriates a sensory channel that blind people already use in a skilful way for their daily-life activities. Modern tactile devices have mainly used the tongue to deliver the substituted information. This body part has been preferred because its sensitivity, spatial acuity, and discrimination abilities are better than other parts of the body (Bach-y-Rita et al., 1998). However, this choice probably adds aesthetic and hygienic problems, which may impact on the willingness of the blind community to introduce the system as a standard aid. Moreover, in order to become a real option for the blind in guiding their navigation, such systems should be complementary and thus provide new information to existing aids like the guide-dog and the white cane. Consequently, it appears evident that more consideration is needed in the design of more ergonometric sensory substitution systems for visual rehabilitation purposes. However, because sensory substitution greatly benefit from the crossmodal changes that occur in the brain of blind individuals they constitute a promising solution especially for early-blind individuals for whom surgical intervention is not possible, particularly if introduced in early infancy when the plasticity of the brain is the highest. Neuroprostheses in the blind Visual prosthetic implants aim to electrically stimulate the remaining functional parts of the previously fully developed visual system in order to restore some visual-like perception, mainly by inducing the perception of patterned spots of light called phosphenes (Merabet et al., 2005; Zrenner, 2002). Such implants would connect a digital camera to a signal processor that would convert visual information into patterned electrical signals (Fig. 1). Several approaches are currently under investigation and involve subretinal (Pardue
et al., 2006a,b; Zrenner et al., 1999), epiretinal (Humayun et al., 2003; Rizzo et al., 2003a,b), optic nerve (Veraart et al., 1998, 2003), or occipital (Schiller and Tehovnik, 2008; Schmidt et al., 1996; Tehovnik et al., 2005) stimulation. Aside from the major issues of electrical safety and biocompatibility of the material (Veraart et al., 2004), knowledge about the selectivity and diffusivity of the stimulation is an essential problem in evaluating the behavioral effects of the stimulated area itself. As a result, researchers are currently trying to combine microstimulation of neural tissue with fMRI in order to provide the unique opportunity to visualize the networks underlying electrostimulation-induced perceptions (Logothetis et al., 2010). In contrast to sensory substitution systems, the visual prostheses do not take advantage of the natural reorganization of the cortex of the blind since such invasive approaches attempt to stimulate the deficient sensory system directly. As such, these prostheses are mainly dedicated to blindness acquired at a later age since the development of the visual system and previous visual experience would be a prerequisite to trigger and interpret the visual percept induced by the stimulation of neural tissues. For example, one study demonstrated that the ability to elicit phosphenes with application of TMS over the occipital area is dramatically reduced in subjects with an early onset of visual deafferentation, especially in those without history of visual experience (Gothe et al., 2002). Indeed, the structural (deterioration of visual tracks) and functional (crossmodal plasticity) changes following early visual deprivation might hamper the reacquisition of the original visual function of a given structure via the prosthetic implant. There are reasons to believe, however, that such devices might work with late-blind individuals since far less alterations in the visual tracks and areas (Jiang et al., 2009; Noppeney et al., 2005; Park et al., 2009) and less-crossmodal recruitment of occipital regions by nonvisual stimuli (Burton et al., 2003; Cohen et al., 1999; Voss et al., 2008) have been observed in subjects
220
who developed late-onset blindness. Moreover, studies of sustained blindfolding in sighted subjects suggest that the crossmodal recruitment of occipital cortex that appears after visual deprivation later in life may be more reversible after the reintroduction of vision (Merabet et al., 2008; Pascual-Leone et al., 2005). In fact, the mechanisms underlying crossmodal occipital recruitment in early- and late-blind individuals may differ considerably (Collignon et al., 2009b). Early deprivation could favor the maintenance of intermodal connections between cortical areas that are normally pruned in infancy, thus preventing the strengthening of typical visual cortical networks. In late blindness, however, these extrinsic connections would not escape the normal developmental synaptic pruning due to the presence of stabilizing visual input. Indeed, crossmodal recruitment of occipital regions observed in late blindness may reflect the strengthening, probably via Hebbian mechanisms1 (Hebb, 1949), of existing intermodal connections also present in sighted subjects. In line with such an assumption, an elegant study combining PET-scan and TMS showed that the application of TMS over the primary somatosensory cortex induced significant activation of the primary visual cortex only in an early-blind group but not in late-blind or sighted subjects (Wittenberg et al., 2004). These results are consistent with the hypothesis of reinforced corticocortical connections between primary sensory cortices in early- but not in late-blind subjects (Collignon et al., 2009b). These results place late-blind individuals as the candidate of choice for visual prosthetic implantation, especially because blindness acquired later in life may prevent the development of all the compensatory mechanisms observed in the early blind; this is also true because in the absence of
1 “When the axon of cell A excites cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells so that A's efficiency as one of the cells firing B is increased.”
enhanced abilities in the remaining senses, the late blind may encounter greater difficulty in coping with the handicap (Wan et al., 2010).
Rehabilitation in deafness While crossmodal plasticity has been less extensively studied in deaf than in blind individuals, research in deaf subjects again leads to the conclusion that crossmodal reorganization occurs, such that cortical territories from the unused auditory modality can be recruited by other senses, in particular vision (Bavelier et al., 2006). Sensory substitution in the deaf These functional changes in the network dedicated to visual processing in the deaf appear to be accompanied by behavioral enhancements in visual attention and visual localization in peripheral visual space (Bavelier et al., 2000; Bosworth and Dobkins, 2002; Neville, 1990; Neville and Lawson, 1987a,b; Proksch and Bavelier, 2002; Rettenbach et al., 1999). Along with these lowlevel processing enhancements (i.e., devoid of phonetics), extensive visual-to-auditory reorganization has also been demonstrated with the presentation of visual stimuli activating the auditory cortex of deaf individuals. Indeed, activation of primary, secondary, and association auditory regions has been observed in early-deaf subjects during the observation of moving dot patterns (Armstrong et al., 2002; Finney et al., 2001) or moving sinusoidal luminance gratings (Finney et al., 2003). Crossmodal changes have also been related to cognitive functions. In normally hearing individuals, speech comprehension is achieved in a multisensory mode that combines auditory and visual (e.g., movement of the lips) speech information. To improve speech recognition or discrimination capabilities, this multisensory process is substituted to favor more exclusively the visual strategies in profoundly
221
deaf individuals. These communication strategies consist mainly of lipreading (Kaiser et al., 2003; Tyler et al., 1997) and sign language reading capabilities (Brozinsky and Bavelier, 2004; Neville et al., 1997; Proksch and Bavelier, 2002). Again, activity in traditionally considered auditory regions has been reported in the deaf during the observation of visual lip motion in the left planum temporale and during the visual presentation of sign language in the superior temporal gyrus and association auditory cortex (Hirano et al., 2000; MacSweeney et al., 2002; Nishimura et al., 1999; Petitto et al., 2000; Sadato et al., 2005). As in the literature on blind subjects, it is believed that the crossmodal plasticity observed in deaf subjects directly leads to a behavioral advantage and improved communication strategies (Bavelier et al., 2006). In those individuals who are trying to achieve some recovery of hearing function, however, such extensive reorganization may represent a challenge that may, in some case, hinder their rehabilitation. Cochlear implant While the visual takeover of the normally auditory cortices represents an impressive cerebral ability to adapt to changes in environment, it begs an important question relative to the recovery of the hearing function. Indeed, once responsive to a new input modality, can the auditory cortices respond to their original auditory input? This question bears special importance given that profound deafness can sometimes be reversed by auditory stimulation via a cochlear implant (CI) (Ponton et al., 1996). Put simply, the device replaces normal cochlear function by converting auditory signals into electrical impulses delivered to the auditory nerve (see Mens, 2007 for a more detailed description). Over the past decade, advances in engineering and surgical implantation techniques have begun to make the CI a standard part of the treatment for hearing loss (Clark, 2006; Fallon et al., 2008). Such success has
allowed researchers to ascertain the consequences of crossmodal plasticity in the deaf population on the success rate of CIs. In deaf individuals, activity in auditory cortical regions is increased following cochlear implantation (Lee et al., 2001; Naito et al., 1995; Wong et al., 1999), as soon as the implant is turned on (Giraud et al., 2001). In their longitudinal electrophysiological investigation, Pantev et al. (2006) showed that the cortical activity in auditory regions had normal component configurations and localizations, confirming that the input from the CI stimulation may be transmitted adequately to auditory structures as soon as the implant is made active in postlingually deaf individuals. The authors also showed that brain activity increased progressively over several months following implantation (Pantev et al., 2006). However, the general outcome of the hearing proficiency following implantation is still highly unpredictable (Green et al., 2007). It has been argued that the level of crossmodal plasticity occurring as a consequence of early deprivation can predict the performance with an auditory prosthesis, with less reorganization leading to greater proficiency with the implant and vice versa (Giraud and Lee, 2007). For instance, it was shown that speech perception performance was positively associated with preoperative activity in frontoparietal networks and negatively associated with activity in occipito-temporal networks (Lee et al., 2005), even when factoring out the confounding effect of age of implantation (Lee et al., 2007). Indeed, the hindering effect of preoperative activity in temporal areas might be a sign that auditory areas may have been taken over by the visual modality, suggesting that crossmodal recruitment can serve as a predictor of the outcome of implantation. Similarly, a recent study compared cortical evoked potentials involved in the processing of visual stimuli between implanted (at least 1 year post-op) and hearing subjects (Doucet et al., 2006). After evaluation of speech perception abilities of the implanted subjects, they were subsequently divided into two groups based on their
222
performance. The results showed that implanted individuals with broader and more anterior scalp distributions (i.e., showing signs of visual processing in the temporal cortices) in response to visual stimuli were those who performed more poorly in the speech perception task and vice versa. In fact, several factors interact and influence crossmodal reorganization in deaf individuals, which in turn impacts auditory perception following implantation. The most influential factors are most likely the duration of deafness, the deafness onset, the time of implantation, and the communication strategy used before implantation. (i) Duration of deafness. Straightforward correlations have been reported between postimplantation auditory-word recognition performance, cortical activity in response to auditory stimulation, and the duration of deafness. Indeed, it appears that implanted deaf individuals who had a longer period of deprivation show less cortical activity in response to auditory stimulation and poorer auditory performance (Lee et al., 2001). The results of this neuroimaging study suggest that a long duration of deafness might lead the higher visual cognitive functions to invade the underutilized areas of the auditory cortex. However, in a retrospective case review, Green et al. (2007) showed that the duration of deprivation only accounted for 9% of the variability in implant outcome, which is substantially less than first thought. In fact, Lee et al. (2001) had already suggested that other factors, such as the onset of deafness or the preimplantation communication strategies, could also have a dramatic impact on auditory perception following implantation. (ii) Onset of deafness. It is in fact commonly acknowledged that postlingually deafened candidates perform better following cochlear implantation in adulthood in all auditory tasks compared to prelingually deaf individuals implanted in later life (Giraud
et al., 2001). Supporting this behavioral evidence, imaging data also suggest more extensive plastic changes in the early-deafened individuals. Indeed, auditory stimuli have been shown to activate both the primary and secondary auditory cortices in postlingually deafened individuals, whereas they merely activate the primary auditory cortex in the prelingually deafened ones following implantation (Naito et al., 1997). Also illustrative of the importance of the age of onset of deafness, Sadato et al. (2004) demonstrated that both early- and late-onset deaf groups showed similar activation of the planum temporale in a visual sentence comprehension task whereas early-deaf subjects showed more prominent activation in the middle superior temporal sulcus (STS), a region thought to be important for the processing of vocalizations (Belin et al., 2000). (iii) Time of implantation. Several studies have shown that if implanted before the age of 2, implanted children can acquire spoken language in a comparable time frame to normal hearing children (Hammes et al., 2002; Waltzman and Cohen, 1998). However, this time window for the recovery of auditory function following deprivation is generally limited to the first few years of life, with the chances of recovery rapidly decreasing afterward (Kral et al., 2005). (iv) Communication strategy before implantation. Hirano et al. (2000) have suggested that crossmodal plasticity may be influenced by the communication strategies (i.e., familiarity with lipreading or sign language ability) used before implantation. Indeed, the authors showed that patients trained to communicate with visual modes of communication are more prone to extensive crossmodal changes compared to individuals trained in a more exclusive auditory mode (i.e., with conventional auditory amplification strategies based on the residual hearing). This last rehabilitation technique seems to prevent visual
223
information from invading the relatively unused cortical regions (Hirano et al., 2000). However, it is worth noting here that the use of this technique in patients with very little or no residual hearing may have a dramatic impact on the communication capabilities of these persons. Although difficult to assess, it is commonly acknowledged that these features (duration of deafness, onset of deafness, time of implantation, and communication strategy before implantation) might also interact in determining the degree to which crossmodal changes might occur, and so, in defining the level of proficiency reached by each participant following cochlear implantation. Multisensory interactions in CI users Since the world around us is made up of events that stimulate several senses simultaneously, it begs the question of how the regained auditory modality might interact with other sensory information during multisensory perception in CI users, especially with regard to speech perception. The integration of congruent cues. Greater visual activity during speech recognition tasks has been reported in deaf individuals with a CI (Giraud et al., 2001). Some evidence even suggests that such visual activity increases progressively with the use of the auditory device (Desai et al., 2008). Indeed, Giraud et al. (2001) suggested that cochlear implantation might result in a mutual reinforcement between vision and hearing. In accordance with this belief of reciprocal enhancement, there seems to be a consensus surrounding the notion that accessing simultaneous visual and auditory information, when both cues are related, is beneficial in CI users (Bergeson and Pisoni, 2004; Geers, 2004; Kaiser et al., 2003; Moody-Antonio et al., 2005; Tyler et al., 1997). Some have even argued that CI users might be better at integrating congruent auditory and visual information when compared to normally hearing individuals (Rouger et al., 2007).
The fusion of incongruent cues. The ability to fuse incongruent audiovisual information has also been studied recently. Schorr et al. (2005) used McGurk-like stimuli, where incongruent lip movements can induce the misperception of spoken syllables (McGurk and MacDonald, 1976), to investigate the ability to integrate incongruent multisensory cues in children with a CI, as a function of experience with spoken language (Schorr et al., 2005). In children aged two and a half years or younger, the authors found normal-like results in the audiovisual task. In contrast, the fusion capability in children implanted later in life was significantly reduced. This is consistent with the notion that an extended duration of deafness might be detrimental to the use of a CI. In addition, typical McGurk-like effects have recently been showed in postlingually deafened candidates (Rouger et al., 2007; Tremblay et al., 2010), in accordance with the idea that crossmodal changes depend of the onset of sensory deprivation. The segregation of incongruent cues. In our laboratory, we investigated the ability of CI users to segregate conflicting auditory and visual inputs (Champoux et al., 2009; see Fig. 4). An auditory speech recognition task was used in the presence of three different incongruent visual stimuli (color-shift, random-dot motion, and lip movement). We showed that the presentation of visual stimuli significantly impairs auditory-word recognition in nonproficient CI users (individuals with poor performance in the speech task without any concurrent visual presentation) while not affecting the performance of proficient CI users and normal hearing subjects. Moreover, this effect was not specific to the presence of linguistic cues (lip movement condition) but was also present during the random-dot motion stimuli. These results are consistent with the notion of extensive changes for the motion-processing dorsal pathway in the deaf (Armstrong et al., 2002) and with our idea that the level of plastic changes consequent to deafferentation might be a crucial factor for auditory rehabilitation through the use of a CI (Doucet et al., 2006). Most
224 Audiovisual interaction in cochlear implant users None
Color-shift
Dot motion
Lip motion
(a) 250 ms
(d)
Auditory stimulus 15 kHz 10 kHz 5 kHz
(b) 500 ms
(c)
500 ms 250 ms
(f)
100
Decrease of performance (%)
Decrease of performance (%)
(e)
Control-paired
75
Proficient CI users
50
25
0
Color-shift
Dot motion
Lip motion
*
100 Control-paired
75
Non-proficient CI users
50
*
25
0
Color-shift
Dot motion
Lip motion
Fig. 4. Audiovisual interaction in CI users. In the top panel is the illustration of the experimental procedure. Each condition began (a) and ended (c) in a static neutral position. In all audiovisual conditions (b), auditory stimuli (d) were simultaneously presented with a visual stimulus change (color, movement, or video sequence). In the bottom panel are plotted the decreases in performance (%) for each audiovisual condition for both proficient (e) and nonproficient (f) CI users. Adapted with permission from Champoux et al. (2009).
important, these data suggest that although visual signals can facilitate speech perception in CI users in congruent audiovisual conditions, they might also hinder speech discrimination performance in some CI users when audiovisual inputs need to be segregated.
Conclusion The immaturity of the human brain at birth is a valuable trait. Delaying the maturation and growth of brain circuits allows initial confrontations with the environment to shape the
225
developing neural architecture in order to create the most adapted circuitry to cope with the external world (Meltzoff et al., 2009). Over the first few years of life, the brain grows rapidly, with each neuron having 2500 synapses at birth and going to 15,000 synapses per neuron after 2–3 years (Gopnik et al., 1999). As we age, experience will drive a process called synaptic pruning, which eliminates or strengthens connections based on the frequency of their use. Indeed, in the same way a gardener would prune a tree in order to give it a desired shape, ineffective connections are pruned in order to adapt the brain to its environment. Even if experience-dependent plasticity appears to be far more pronounced in children, synaptic connection efficiency changes based on experience are also present at more advanced ages. As discussed at length in this chapter, sensory deprivation at early and, to a lesser extent, later ages will induce plastic changes in the structural and functional architecture of sensory cortices. Any severe sensory deafferentation precipitates unexpected sensory access to the affected cortex by the remaining senses. Such crossmodal plasticity is thought to be intrinsically linked to behavioral compensation mechanisms observed in sensory-deprived individuals (Amedi et al., 2003; Gougoux et al., 2005). Indeed, we have argued that rehabilitation based on sensory substitution systems, among which the two most well known are probably the Braille reading system for the blind and the sign language system for the deaf, spontaneously benefit from the natural tendency of the sensory-deprived brain to reorganize itself to optimize the processing of nonvisual inputs. In contrast, rehabilitation techniques aimed at restoring the deprived sense, like neuroprostheses, are based on an opposite principle of rehabilitation and rely on the integrity of the original function of sensory-deprived cortex. In both cases, we strongly believe that a better understanding of the mechanisms underlying experience-dependent crossmodal plasticity is a necessary prerequisite to properly develop new rehabilitation avenues. The task is obviously not
an easy one because the full impact of sensory deprivation is always the result of a complex interaction between the specific etiology, the age of onset, the length of the deprivation, as well as the strategy that has been put in place in order to cope with the handicap. However, some lessons can be learned from the studies described above. For instance, if an invasive intervention for restoring the deprived sense is chosen in the case of congenital or early childhood deprivation, the “the earlier, the better” adage holds true based on the principle that it is easier to build than to rebuild, meaning that when neural circuitry has reached maturity, the possibility of rewiring it by the introduction of a novel input is more limited. The rapid development of neuroimaging tools over the past few decades has allowed us to probe the brain's functioning and anatomy in a noninvasive manner and thus may serve as a standard procedure in order to evaluate the suitability of specific rehabilitation procedures in the future (Merabet et al., 2005). For example, the observation of massive crossmodal recruitment of the deafferented cortex could alert us that the restoration of the deprived function with new rehabilitative interventions may be more problematic than first thought (Gregory and Wallace, 1963). This is reminiscent of a quote from the philosopher Jean-Jacques Rousseau: “With progress, we know what we gain but not what we lose.” We again stress that a better basic comprehension of the underlying mechanisms of crossmodal plasticity will help us better understand and predict the outcome of sensory restoration based on increasingly complex biotechnologies.
Acknowledgments This research was supported in part by the Canada Research Chair Program (F. L.), the Canadian Institutes of Health Research (P. V. and F. L.), and the Natural Sciences and Engineering Research Council of Canada (F. L.).
226
References Amedi, A., Floel, A., Knecht, S., Zohary, E., & Cohen, L. G. (2004). Transcranial magnetic stimulation of the occipital pole interferes with verbal processing in blind subjects. Nature Neuroscience. Amedi, A., Raz, N., Pianka, P., Malach, R., & Zohary, E. (2003). Early “visual” cortex activation correlates with superior verbal memory performance in the blind. Nature Neuroscience, 6, 758–766. Amedi, A., Stern, W. M., Camprodon, J. A., Bermpohl, F., Merabet, L., Rotman, S., et al. (2007). Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex. Nature Neuroscience, 10, 687–689. Armstrong, B. A., Neville, H. J., Hillyard, S. A., & Mitchell, T. V. (2002). Auditory deprivation affects processing of motion, but not color. Brain Research. Cognitive Brain Research, 14, 422–434. Arno, P., Capelle, C., Wanet-Defalque, M. C., CatalanAhumada, M., & Veraart, C. (1999). Auditory coding of visual patterns for the blind. Perception, 28, 1013–1029. Arno, P., De Volder, A. G., Vanlierde, A., WanetDefalque, M. C., Streel, E., Robert, A., et al. (2001). Occipital activation by pattern recognition in the early blind using auditory substitution for vision. Neuroimage, 13, 632–645. Arno, P., Vanlierde, A., Streel, E., Wanet-Defalque, M. C., Sanabria-Bohorquez, S. M., & Veraart, C. (2001). Auditory substitution of vision: Pattern recognition by blind. Applied Cognitive Psychology, 15, 509–519. Bach-y-Rita, P., Kaczmarek, K. A., Tyler, M. E., & GarciaLara, J. (1998). Form perception with a 49-point electrotactile stimulus array on the tongue: A technical note. Journal of Rehabilitation Research and Development, 35, 427–430. Bach-y-Rita, P., & Kercel, S. (2003). Sensory substitution and the human-machine interface. Trends in Cognitive Sciences, 7, 541–546. Bavelier, D., Brozinsky, C., Tomann, A., Mitchell, T., Neville, H., & Liu, G. (2001). Impact of early deafness and early exposure to sign language on the cerebral organization for motion processing. The Journal of Neuroscience, 21, 8931–8942. Bavelier, D., Dye, M. W., & Hauser, P. C. (2006). Do deaf individuals see better? Trends in Cognitive Sciences, 10, 512–518. Bavelier, D., & Neville, H. J. (2002). Cross-modal plasticity: Where and how? Nature Reviews. Neuroscience, 3, 443–452. Bavelier, D., Tomann, A., Hutton, C., Mitchell, T., Corina, D., Liu, G., et al. (2000). Visual attention to the periphery is enhanced in congenitally deaf individuals. The Journal of Neuroscience, 20, RC93. Bedny, M., Konkle, T., Pelphrey, K., Saxe, R., & PascualLeone, A. (2010). Sensitive period for a multimodal
response in human visual motion area MT/MST. Current Biology, 20, 1900–1906. Belin, P., Zatorre, R. J., Lafaille, P., Ahad, P., & Pike, B. (2000). Voice-selective areas in human auditory cortex. Nature, 403, 309–312. Bergeson, T. R., & Pisoni, D. B. (2004). Audiovisual speech perception in deaf adults and children following cochlear implantation. In G. Calvert, C. Sence & B. E. Stein (Eds.), Handbook of multisensory processes (pp. 749–772). Cambridge: MIT Press. Bosworth, R. G., & Dobkins, K. R. (2002). The effects of spatial attention on motion processing in deaf signers, hearing signers, and hearing nonsigners. Brain and Cognition, 49, 152–169. Bowering, E. R., Maurer, D., Lewis, T. L., & Brent, H. P. (1993). Sensitivity in the nasal and temporal hemifields in children treated for cataract. Investigative Ophthalmology and Visual Science, 34, 3501–3509. Brozinsky, C. J., & Bavelier, D. (2004). Motion velocity thresholds in deaf signers: Changes in lateralization but not in overall sensitivity. Brain Research. Cognitive Brain Research, 21, 1–10. Buchel, C. (1998). Functional neuroimaging studies of Braille reading: Cross-modal reorganization and its implications. Brain, 121(Pt. 7), 1193–1194. Burton, H., Diamond, J. B., & McDermott, K. B. (2003). Dissociating cortical regions activated by semantic and phonological tasks: A FMRI study in blind and sighted people. Journal of Neurophysiology, 90, 1965–1982. Burton, H., Snyder, A. Z., Conturo, T. E., Akbudak, E., Ollinger, J. M., & Raichle, M. E. (2002). Adaptive changes in early and late blind: A fMRI study of Braille reading. Journal of Neurophysiology, 87, 589–607. Capelle, C., Trullemans, C., Arno, P., & Veraart, C. (1998). A real-time experimental prototype for enhancement of vision rehabilitation using auditory substitution. IEEE Transactions on Biomedical Engineering, 45, 1279–1293. Champoux, F., Lepore, F., Gagne, J. P., & Theoret, H. (2009). Visual stimuli can impair auditory processing in cochlear implant users. Neuropsychologia, 47, 17–22. Clark, G. M. (2006). The multiple-channel cochlear implant: The interface between sound and the central nervous system for hearing, speech, and language in deaf people—A personal perspective. Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences, 361, 791–810. Cohen, L. G., Celnik, P., Pascual-Leone, A., Corwell, B., Falz, L., Dambrosia, J., et al. (1997). Functional relevance of cross-modal plasticity in blind humans. Nature, 389, 180–183. Cohen, L. G., Weeks, R. A., Sadato, N., Celnik, P., Ishii, K., & Hallett, M. (1999). Period of susceptibility for cross-modal plasticity in the blind. Annals of Neurology, 45, 451–460.
227 Collignon, O., Davare, M., Olivier, E., & De Volder, A. G. (2009). Reorganisation of the right occipito-parietal stream for auditory spatial processing in early blind humans. A transcranial magnetic stimulation study. Brain Topography, 21, 232–240. Collignon, O., Lassonde, M., Lepore, F., Bastien, D., & Veraart, C. (2007). Functional cerebral reorganization for auditory spatial processing and auditory substitution of vision in early blind subjects. Cerebral Cortex, 17, 457–465. Collignon, O., Renier, L., Bruyer, R., Tranduy, D., & Veraart, C. (2006). Improved selective and divided spatial attention in early blind subjects. Brain Research, 1075, 175–182. Collignon, O., Vandewalle, G., Voss, P., Albouy, G., Charbonneau, G., Lassonde, M., & Lepore, F. (2011). Functional specialization for auditory-spatial processing in the occipital cortex of congenitally blind humans. Proceedings of the National Academy of Sciences, 108, 4435–4440. Collignon, O., Voss, P., Lassonde, M., & Lepore, F. (2009). Cross-modal plasticity for the spatial processing of sounds in visually deprived subjects. Experimental Brain Research, 192, 343–358. Cronly-Dillon, J., Persaud, K., & Gregory, R. P. (1999). The perception of visual images encoded in musical form: A study in cross-modality information transfer. Proceedings of Biological Sciences, 266, 2427–2433. De Volder, A. G., Catalan-Ahumada, M., Robert, A., Bol, A., Labar, D., Coppens, A., et al. (1999). Changes in occipital cortex activity in early blind humans using a sensory substitution device. Brain Research, 826, 128–134. Desai, S., Stickney, G., & Zeng, F. G. (2008). Auditory-visual speech perception in normal-hearing and cochlear-implant listeners. The Journal of the Acoustical Society of America, 123, 428–440. Dormal, G., & Collignon O. (2011). Functional selectivity in sensory deprived cortices. Journal of Neurophysiology. In Press. Doucet, M. E., Bergeron, F., Lassonde, M., Ferron, P., & Lepore, F. (2006). Cross-modal reorganization and speech perception in cochlear implant users. Brain, 129, 3376–3383. Dufour, A., Despres, O., & Candas, V. (2005). Enhanced sensitivity to echo cues in blind subjects. Experimental Brain Research, 165, 515–519. Elbert, T., Sterr, A., Rockstroh, B., Pantev, C., Muller, M. M., & Taub, E. (2002). Expansion of the tonotopic area in the auditory cortex of the blind. The Journal of Neuroscience, 22, 9941–9944. Ellemberg, D., Lewis, T. L., Maurer, D., Brar, S., & Brent, H. P. (2002). Better perception of global motion after monocular than after binocular deprivation. Vision Research, 42, 169–179. Fallon, J. B., Irvine, D. R., & Shepherd, R. K. (2008). Cochlear implants and brain plasticity. Hearing Research, 238, 110–117.
Fine, I., Wade, A. R., Brewer, A. A., May, M. G., Goodman, D. F., Boynton, G. M., et al. (2003). Long-term deprivation affects visual perception and cortex. Nature Neuroscience, 6, 915–916. Finney, E. M., Clementz, B. A., Hickok, G., & Dobkins, K. R. (2003). Visual stimuli activate auditory cortex in deaf subjects: Evidence from MEG. Neuroreport, 14, 1425–1427. Finney, E. M., Fine, I., & Dobkins, K. R. (2001). Visual stimuli activate auditory cortex in the deaf. Nature Neuroscience, 4, 1171–1173. Frost, D. O., Boire, D., Gingras, G., & Ptito, M. (2000). Surgically created neural pathways mediate visual pattern discrimination. Proceedings of the National Academy of Sciences of the United States of America, 97, 11068–11073. Frost, D. O., & Metin, C. (1985). Induction of functional retinal projections to the somatosensory system. Nature, 317, 162–164. Geers, A. E. (2004). Speech, language, and reading skills after early cochlear implantation. Archives of Otolaryngology, 130, 634–638. Giraud, A. L., & Lee, H. J. (2007). Predicting cochlear implant outcome from brain organisation in the deaf. Restorative Neurology and Neuroscience, 25, 381–390. Giraud, A. L., Price, C. J., Graham, J. M., & Frackowiak, R. S. (2001). Functional plasticity of language-related brain areas after cochlear implantation. Brain, 124, 1307–1316. Gopnik, A., Meltzoff, A., & Kuhl, P. (1999). The scientist in the crib: What early learning tells us about the mind. New York, NY: HarperCollins Publishers. Gothe, J., Brandt, S. A., Irlbacher, K., Roricht, S., Sabel, B. A., & Meyer, B. U. (2002). Changes in visual cortex excitability in blind subjects as demonstrated by transcranial magnetic stimulation. Brain, 125, 479–490. Gougoux, F., Belin, P., Voss, P., Lepore, F., Lassonde, M., & Zatorre, R. J. (2009). Voice perception in blind persons: A functional magnetic resonance imaging study. Neuropsychologia, 47, 2967–2974. Gougoux, F., Zatorre, R. J., Lassonde, M., Voss, P., & Lepore, F. (2005). A functional neuroimaging study of sound localization: Visual cortex activity predicts performance in early-blind individuals. PLoS Biology, 3, e27. Green, K. M., Bhatt, Y. M., Mawman, D. J., O'Driscoll, M. P., Saeed, S. R., Ramsden, R. T., et al. (2007). Predictors of audiological outcome following cochlear implantation in adults. Cochlear Implants International, 8, 1–11. Gregory, R. L. (2003). Seeing after blindness. Nature Neuroscience, 6, 909–910. Gregory, R. L., & Wallace, J. (1963). Recovery from early blindness: A case study. Experimental Psychology Society Monograph, 2, Heffers, Cambridge. (Reprinted in: Gregory, R. L. (1974). Concepts and mechanisms of perception. Duckworth, London.).
228 Hamilton, R., Keenan, J. P., Catala, M., & Pascual-Leone, A. (2000). Alexia for Braille following bilateral occipital stroke in an early blind woman. Neuroreport, 11, 237–240. Hammes, D. M., Novak, M. A., Rotz, L. A., Willis, M., Edmondson, D. M., & Thomas, J. F. (2002). Early identification and cochlear implantation: Critical factors for spoken language development. The Annals of Otology, Rhinology and Laryngology. Supplement, 189, 74–78. Hebb, D. O. (1949). The organization of behavior. New York: John Wiley. Hirano, S., Naito, Y., Kojima, H., Honjo, I., Inoue, M., Shoji, K., et al. (2000). Functional differentiation of the auditory association area in prelingually deaf subjects. Auris, Nasus, Larynx, 27, 303–310. Humayun, M. S., Weiland, J. D., Fujii, G. Y., Greenberg, R., Williamson, R., Little, J., et al. (2003). Visual perception in a blind subject with a chronic microelectronic retinal prosthesis. Vision Research, 43, 2573–2581. Jiang, J., Zhu, W., Shi, F., Liu, Y., Li, J., Qin, W., et al. (2009). Thick visual cortex in the early blind. The Journal of Neuroscience, 29, 2205–2211. Kaas, J. H., Merzenich, M. M., & Killackey, H. P. (1983). The reorganization of somatosensory cortex following peripheral nerve damage in adult and developing mammals. Annual Review of Neuroscience, 6, 325–356. Kaczmarek, K., Rita, P., Tompkins, W. J., & Webster, J. G. (1985). A tactile vision-substitution system for the blind: Computer-controlled partial image sequencing. IEEE Transactions on Biomedical Engineering, 32, 602–608. Kaiser, A. R., Kirk, K. I., Lachs, L., & Pisoni, D. B. (2003). Talker and lexical effects on audiovisual word recognition by adults with cochlear implants. Journal of Speech, Language, and Hearing Research, 46, 390–404. Kral, A., Tillein, J., Heid, S., Hartmann, R., & Klinke, R. (2005). Postnatal cortical development in congenital auditory deprivation. Cerebral Cortex, 15, 552–562. Kupers, R., Chebat, D. R., Madsen, K. H., Paulson, O. B., & Ptito, M. (2010). Neural correlates of virtual route recognition in congenital blindness. Proceedings of the National Academy of Sciences of the United States of America, 107, 12716–12721. Kupers, R., Pappens, M., de Noordhout, A. M., Schoenen, J., Ptito, M., & Fumal, A. (2007). rTMS of the occipital cortex abolishes Braille reading and repetition priming in blind subjects. Neurology, 68, 691–693. Le Grand, R., Mondloch, C. J., Maurer, D., & Brent, H. P. (2001). Neuroperception. Early visual experience and face processing. Nature, 410, 890. Le Grand, R., Mondloch, C. J., Maurer, D., & Brent, H. P. (2004). Impairment in holistic face processing following early visual deprivation. Psychological Science, 15, 762–768. Lee, H. J., Giraud, A. L., Kang, E., Oh, S. H., Kang, H., Kim, C. S., et al. (2007). Cortical activity at rest predicts
cochlear implantation outcome. Cerebral Cortex, 17, 909–917. Lee, H. J., Kang, E., Oh, S. H., Kang, H., Lee, D. S., Lee, M. C., et al. (2005). Preoperative differences of cerebral metabolism relate to the outcome of cochlear implants in congenitally deaf children. Hearing Research, 203, 2–9. Lee, D. S., Lee, J. S., Oh, S. H., Kim, S. K., Kim, J. W., Chung, J. K., et al. (2001). Cross-modal plasticity and cochlear implants. Nature, 409, 149–150. Levin, N., Dumoulin, S. O., Winawer, J., Dougherty, R. F., & Wandell, B. A. (2010). Cortical maps and white matter tracts following long period of visual deprivation and retinal image restoration. Neuron, 65, 21–31. Lewis, T. L., & Maurer, D. (2005). Multiple sensitive periods in human visual development: Evidence from visually deprived children. Developmental Psychobiology, 46, 163–183. Logothetis, N. K., Augath, M., Murayama, Y., Rauch, A., Sultan, F., Goense, J., et al. (2010). The effects of electrical microstimulation on cortical signal propagation. Nature Neuroscience, 13, 1283–1291. MacSweeney, M., Calvert, G. A., Campbell, R., McGuire, P. K., David, A. S., Williams, S. C., et al. (2002). Speechreading circuits in people born deaf. Neuropsychologia, 40, 801–807. Maurer, D., Lewis, T. L., & Mondloch, C. J. (2005). Missing sights: Consequences for visual cognitive development. Trends in Cognitive Sciences, 9, 144–151. McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746–748. Meijer, P. B. (1992). An experimental system for auditory image representations. IEEE Transactions on Biomedical Engineering, 39, 112–121. Meltzoff, A. N., Kuhl, P. K., Movellan, J., & Sejnowski, T. J. (2009). Foundations for a new science of learning. Science, 325, 284–288. Mens, L. H. (2007). Advances in cochlear implant telemetry: Evoked neural responses, electrical field imaging, and technical integrity. Trends in Amplification, 11, 143–159. Merabet, L. B., Battelli, L., Obretenova, S., Maguire, S., Meijer, P., & Pascual-Leone, A. (2009). Functional recruitment of visual cortex for sound encoded object identification in the blind. Neuroreport, 20, 132–138. Merabet, L. B., Hamilton, R., Schlaug, G., Swisher, J. D., Kiriakopoulos, E. T., Pitskel, N. B., et al. (2008). Rapid and reversible recruitment of early visual cortex for touch. PLoS ONE, 3, e3046. Merabet, L. B., Rizzo, J. F., Amedi, A., Somers, D. C., & Pascual-Leone, A. (2005). What blindness can tell us about seeing again: Merging neuroplasticity and neuroprostheses. Nature Reviews Neuroscience, 6, 71–77. Moody-Antonio, S., Takayanagi, S., Masuda, A., Auer, E. T., Jr.Fisher, L., & Bernstein, L. E. (2005). Improved speech
229 perception in adult congenitally deafened cochlear implant recipients. Otology and Neurotology, 26, 649–654. Naito, Y., Hirano, S., Honjo, I., Okazawa, H., Ishizu, K., Takahashi, H., et al. (1997). Sound-induced activation of auditory cortices in cochlear implant users with post- and pre-lingual deafness demonstrated by positron emission tomography. Acta Otolaryngologica, 117, 490–496. Naito, Y., Okazawa, H., Honjo, I., Takahashi, H., Kawano, M., Ishizu, K., et al. (1995). Cortical activation during sound stimulation in cochlear implant users demonstrated by positron emission tomography. The Annals of Otology, Rhinology and Laryngology. Supplement, 166, 60–64. Neville, H. J. (1990). Intermodal competition and compensation in development. Evidence from studies of the visual system in congenitally deaf adults. Annals of the New York Academy of Sciences, 608, 71–87 discussion, 87–91. Neville, H. J., Coffey, S. A., Lawson, D. S., Fischer, A., Emmorey, K., & Bellugi, U. (1997). Neural systems mediating American sign language: Effects of sensory experience and age of acquisition. Brain and Language, 57, 285–308. Neville, H. J., & Lawson, D. (1987a). Attention to central and peripheral visual space in a movement detection task: An event-related potential and behavioral study. II. Congenitally deaf adults. Brain Research, 405, 268–283. Neville, H. J., & Lawson, D. (1987b). Attention to central and peripheral visual space in a movement detection task. III. Separate effects of auditory deprivation and acquisition of a visual language. Brain Research, 405, 284–294. Nishimura, H., Hashikawa, K., Doi, K., Iwaki, T., Watanabe, Y., Kusuoka, H., et al. (1999). Sign language “heard” in the auditory cortex. Nature, 397, 116. Noppeney, U. (2007). The effects of visual deprivation on functional and structural organization of the human brain. Neuroscience and Biobehavioral Reviews, 31, 1169–1180. Noppeney, U., Friston, K. J., Ashburner, J., Frackowiak, R., & Price, C. J. (2005). Early visual deprivation induces structural plasticity in gray and white matter. Current Biology, 15, R488–R490. Ostrovsky, Y., Andalman, A., & Sinha, P. (2006). Vision following extended congenital blindness. Psychological Science, 17, 1009–1014. Ostrovsky, Y., Meyers, E., Ganesh, S., Mathur, U., & Sinha, P. (2009). Visual parsing after recovery from blindness. Psychological Science, 20, 1484–1491. Pan, W. J., Wu, G., Li, C. X., Lin, F., Sun, J., & Lei, H. (2007). Progressive atrophy in the optic pathway and visual cortex of early blind Chinese adults: A voxel-based morphometry magnetic resonance imaging study. Neuroimage, 37, 212–220. Pantev, C., Dinnesen, A., Ross, B., Wollbrink, A., & Knief, A. (2006). Dynamics of auditory plasticity after cochlear implantation: A longitudinal study. Cerebral Cortex, 16, 31–36.
Pantev, C., Oostenveld, R., Engelien, A., Ross, B., Roberts, L. E., & Hoke, M. (1998). Increased auditory cortical representation in musicians. Nature, 392, 811–814. Pardue, M. T., Ball, S. L., Phillips, M. J., Faulkner, A. E., Walker, T. A., Chow, A. Y., et al. (2006). Status of the feline retina 5 years after subretinal implantation. Journal of Rehabilitation Research and Development, 43, 723–732. Pardue, M. T., Phillips, M. J., Hanzlicek, B., Yin, H., Chow, A. Y., & Ball, S. L. (2006). Neuroprotection of photoreceptors in the RCS rat after implantation of a subretinal implant in the superior or inferior retina. Advances in Experimental Medicine and Biology, 572, 321–326. Park, H. J., Lee, J. D., Kim, E. Y., Park, B., Oh, M. K., Lee, S., et al. (2009). Morphological alterations in the congenital blind based on the analysis of cortical thickness and surface area. Neuroimage, 47, 98–106. Pascual-Leone, A., Amedi, A., Fregni, F., & Merabet, L. B. (2005). The plastic human brain cortex. Annual Review of Neuroscience, 28, 377–401. Pascual-Leone, A., & Torres, F. (1993). Plasticity of the sensorimotor cortex representation of the reading finger in Braille readers. Brain, 116(Pt. 1), 39–52. Petitto, L. A., Zatorre, R. J., Gauna, K., Nikelski, E. J., Dostie, D., & Evans, A. C. (2000). Speech-like cerebral activity in profoundly deaf people processing signed languages: Implications for the neural basis of human language. Proceedings of the National Academy of Sciences of the United States of America, 97, 13961–13966. Piche, M., Robert, S., Miceli, D., & Bronchti, G. (2004). Environmental enrichment enhances auditory takeover of the occipital cortex in anophthalmic mice. The European Journal of Neuroscience, 20, 3463–3472. Poirier, C., Collignon, O., Scheiber, C., & De Volder, A. G. (2004). Auditory motion processing in early blind subjects. Cognitive Processing, 5(4), 254–256. Poirier, C., De Volder, A. G., & Scheiber, C. (2007). What neuroimaging tells us about sensory substitution. Neuroscience and Biobehavioral Reviews, 31, 1064–1070. Ponton, C. W., Don, M., Eggermont, J. J., Waring, M. D., Kwong, B., & Masuda, A. (1996). Auditory system plasticity in children after long periods of complete deafness. Neuroreport, 8, 61–65. Proksch, J., & Bavelier, D. (2002). Changes in the spatial distribution of visual attention after early deafness. Journal of Cognitive Neuroscience, 14, 687–701. Proulx, M. J., Stoerig, P., Ludowig, E., & Knoll, I. (2008). Seeing “where” through the ears: Effects of learning-by-doing and long-term sensory deprivation on localization based on image-to-sound substitution. PLoS ONE, 3, e1840. Ptito, M., Fumal, A., de Noordhout, A. M., Schoenen, J., Gjedde, A., & Kupers, R. (2008). TMS of the occipital
230 cortex induces tactile sensations in the fingers of blind Braille readers. Experimental Brain Research, 184, 193–200. Ptito, M., Moesgaard, S. M., Gjedde, A., & Kupers, R. (2005). Cross-modal plasticity revealed by electrotactile stimulation of the tongue in the congenitally blind. Brain, 128, 606–614. Ptito, M., Schneider, F. C., Paulson, O. B., & Kupers, R. (2008). Alterations of the visual pathways in congenital blindness. Experimental Brain Research, 187, 41–49. Putzar, L., Goerendt, I., Lange, K., Rosler, F., & Roder, B. (2007). Early visual deprivation impairs multisensory interactions in humans. Nature Neuroscience, 10, 1243–1245. Putzar, L., Hötting, K., & Röder, B. (2010). Early visual deprivation affects the development of face recognition and audio-visual speech perception. Restorative Neurology and Neuroscience, 28, 251–257. Renier, L., Anurova, I., De Volder, A. G., Carlson, S., VanMeter, J., & Rauschecker, J. P. (2010). Preserved functional specialization for spatial processing in the middle occipital gyrus of the early blind. Neuron, 68, 138–148. Renier, L., & De Volder, A. G. (2010). Vision substitution and depth perception: Early blind subjects experience visual perspective through their ears. Disability and Rehabilitation: Assistive Technology, 5, 175–183. Rettenbach, R., Diller, G., & Sireteanu, R. (1999). Do deaf people see better? Texture segmentation and visual search compensate in adult but not in juvenile subjects. Journal of Cognitive Neuroscience, 11, 560–583. Ricciardi, E., Vanello, N., Sani, L., Gentili, C., Scilingo, E. P., Landini, L., et al. (2007). The effect of visual experience on the development of functional architecture in hMTþ. Cerebral Cortex, 17, 2933–2939. Rice, C. E. (1967). Human echo perception. Science, 155, 656–664. Rice, C. E., & Feinstein, S. H. (1965). Sonar system of the blind: Size discrimination. Science, 148, 1107–1108. Rita, P., Collins, C. C., Saunders, F. A., White, B., & Scadden, L. (1969). Vision substitution by tactile image projection. Nature, 221, 963–964. Rizzo, J. F., 3rd., Wyatt, J., Loewenstein, J., Kelly, S., & Shire, D. (2003a). Methods and perceptual thresholds for short-term electrical stimulation of human retina with microelectrode arrays. Investigative Ophthalmology and Visual Science, 44, 5355–5361. Rizzo, J. F., 3rd., Wyatt, J., Loewenstein, J., Kelly, S., & Shire, D. (2003b). Perceptual efficacy of electrical stimulation of human retina with a microelectrode array during short-term surgical trials. Investigative Ophthalmology and Visual Science, 44, 5362–5369. Roe, A. W., Pallas, S. L., Hahm, J. O., & Sur, M. (1990). A map of visual space induced in primary auditory cortex. Science, 250, 818–820.
Rouger, J., Lagleyre, S., Fraysse, B., Deneve, S., Deguine, O., & Barone, P. (2007). Evidence that cochlear-implanted deaf patients are better multisensory integrators. Proceedings of the National Academy of Sciences of the United States of America, 104, 7295–7300. Sacks, O. (1995). An anthropolgist on Mars. New York: Knopf. Sadato, N., Okada, T., Honda, M., Matsuki, K., Yoshida, M., Kashikura, K., et al. (2005). Cross-modal integration and plastic changes revealed by lip movement, random-dot motion and sign languages in the hearing and deaf. Cerebral Cortex, 15, 1113–1122. Sadato, N., Pascual-Leone, A., Grafman, J., Deiber, M. P., Ibanez, V., & Hallett, M. (1998). Neural networks for Braille reading by the blind. Brain, 121(Pt 7), 1213–1229. Sadato, N., Pascual-Leone, A., Grafman, J., Ibanez, V., Deiber, M. P., Dold, G., et al. (1996). Activation of the primary visual cortex by Braille reading in blind subjects. Nature, 380, 526–528. Sadato, N., Yamada, H., Okada, T., Yoshida, M., Hasegawa, T., Matsuki, K., et al. (2004). Age-dependent plasticity in the superior temporal sulcus in deaf humans: A functional MRI study. BMC Neuroscience, 5, 56. Saenz, M., Lewis, L. B., Huth, A. G., Fine, I., & Koch, C. (2008). Visual motion area MTþ/V5 responds to auditory motion in human sight-recovery subjects. The Journal of Neuroscience, 28, 5141–5148. Schiller, P. H., & Tehovnik, E. J. (2008). Visual prosthesis. Perception, 37, 1529–1559. Schmidt, E. M., Bak, M. J., Hambrecht, F. T., Kufta, C. V., O'Rourke, D. K., & Vallabhanath, P. (1996). Feasibility of a visual prosthesis for the blind based on intracortical microstimulation of the visual cortex. Brain, 119(Pt 2), 507–522. Schorr, E. A., Fox, N. A., van, W. V., & Knudsen, E. I. (2005). Auditory-visual fusion in speech perception in children with cochlear implants. Proceedings of the National Academy of Sciences of the United States of America, 102, 18748–18750. Segond, H., Weiss, D., & Sampaio, E. (2005). Human spatial navigation via a visuo-tactile sensory substitution system. Perception, 34, 1231–1249. Serino, A., Bassolino, M., Farne, A., & Ladavas, E. (2007). Extended multisensory space in blind cane users. Psychological Science, 18, 642–648. Shu, N., Li, J., Li, K., Yu, C., & Jiang, T. (2009). Abnormal diffusion of cerebral white matter in early blindness. Human Brain Mapping, 30, 220–227. Sterr, A., Muller, M. M., Elbert, T., Rockstroh, B., Pantev, C., & Taub, E. (1998). Perceptual correlates of changes in cortical representation of fingers in blind multifinger Braille readers. The Journal of Neuroscience, 18, 4417–4423. Strelow, E. R., & Brabyn, J. A. (1982). Locomotion of the blind controlled by natural sound cues. Perception, 11, 635–640.
231 Sur, M., Garraghty, P. E., & Roe, A. W. (1988). Experimentally induced visual projections into auditory thalamus and cortex SUR1988. Science, 242, 1437–1441. Tehovnik, E. J., Slocum, W. M., Carvey, C. E., & Schiller, P. H. (2005). Phosphene induction and the generation of saccadic eye movements by striate cortex. Journal of Neurophysiology, 93, 1–19. Tremblay, C., Champoux, F., Lepore, F., & Theoret, H. (2010). Audiovisual fusion and cochlear implant proficiency. Restorative Neurology and Neuroscience, 28, 283–291. Tyler, R. S., Fryauf-Bertschy, H., Kelsay, D. M., Gantz, B. J., Woodworth, G. P., & Parkinson, A. (1997). Speech perception by prelingually deaf children using cochlear implants. Otolaryngology—Head and Neck Surgery, 117, 180–187. Van Boven, R. W., Hamilton, R. H., Kauffman, T., Keenan, J. P., & Pascual-Leone, A. (2000). Tactile spatial resolution in blind braille readers. Neurology, 54, 2230–2236. Veraart, C., Duret, F., Brelen, M., Oozeer, M., & Delbeke, J. (2004). Vision rehabilitation in the case of blindness. Expert Review of Medical Devices, 1, 139–153. Veraart, C., Raftopoulos, C., Mortimer, J. T., Delbeke, J., Pins, D., Michaux, G., et al. (1998). Visual sensations produced by optic nerve stimulation using an implanted selfsizing spiral cuff electrode. Brain Research, 813, 181–186. Veraart, C., Wanet-Defalque, M. C., Gerard, B., Vanlierde, A., & Delbeke, J. (2003). Pattern recognition with the optic nerve visual prosthesis. Artificial Organs, 27, 996–1004. von Melchner, L., Pallas, S. L., & Sur, M. (2000). Visual behaviour mediated by retinal projections directed to the auditory pathway. Nature, 404, 871–876. Voss, P., Gougoux, F., Zatorre, R. J., Lassonde, M., & Lepore, F. (2008). Differential occipital responses in earlyand late-blind individuals during a sound-source discrimination task. Neuroimage, 40, 746–758.
Waltzman, S. B., & Cohen, N. L. (1998). Cochlear implantation in children younger than 2 years old. The American Journal of Otology, 19, 158–162. Wan, C. Y., Wood, A. G., Reutens, D. C., & Wilson, S. J. (2010). Early but not late-blindness leads to enhanced auditory perception. Neuropsychologia, 48, 344–348. Wanet-Defalque, M. C., Veraart, C., De Volder, A., Metz, R., Michel, C., Dooms, G., et al. (1988). High metabolic activity in the visual cortex of early blind human subjects. Brain Research, 446, 369–373. Wiesel, T. N., & Hubel, D. H. (1965). Extent of recovery from the effects of visual deprivation in kittens. Journal of Neurophysiology, 28, 1060–1072. Wiesel, T. N., & Hubel, D. H. (1974). Ordered arrangement of orientation columns in monkeys lacking visual experience. The Journal of Comparative Neurology, 158, 307–318. Wittenberg, G. F., Werhahn, K. J., Wassermann, E. M., Herscovitch, P., & Cohen, L. G. (2004). Functional connectivity between somatosensory and visual cortex in early blind humans. The European Journal of Neuroscience, 20, 1923–1927. Wong, M., Gnanakumaran, V., & Goldreich, D. (2011). Tactile spatial acuity enhancement in blindness: Evidence for experience-dependent MechanismsTactile spatial acuity enhancement in blindness: Evidence for experience-dependent mechanisms. The Journal of Neuroscience, 31, 7028–7037. Wong, D., Miyamoto, R. T., Pisoni, D. B., Sehgal, M., & Hutchins, G. D. (1999). PET imaging of cochlear-implant and normal-hearing subjects listening to speech and nonspeech. Hearing Research, 132, 34–42. Zrenner, E. (2002). Will retinal implants restore vision? Science, 295, 1022–1025. Zrenner, E., Stett, A., Weiss, S., Aramant, R. B., Guenther, E., Kohler, K., et al. (1999). Can subretinal microphotodiodes successfully replace degenerated photoreceptors? Vision Research, 39, 2555–2567.
A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 191 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.
CHAPTER 15
Crossmodal plasticity in sensory loss Johannes Frasnelli{,*, Olivier Collignon{,}, Patrice Voss{ and Franco Lepore{ {
{
Département de Psychologie, Centre de Recherche en Neuropsychologie et Cognition, Université de Montréal, Montréal, Québec, Canada International Laboratory for Brain, Music and Sound Research, Université de Montréal, Montréal, Québec, Canada } Centre de Recherche CHU Sainte-Justine, Université de Montréal, Montréal, Québec, Canada
Abstract: In this review, we describe crossmodal plasticity following sensory loss in three parts, with each section focusing on one sensory system. We summarize a wide range of studies showing that sensory loss may lead, depending of the affected sensory system, to functional changes in other, primarily not affected senses, which range from heightened to lowered abilities. In the first part, the effects of blindness on mainly audition and touch are described. The latest findings on brain reorganization in blindness are reported, with a particular emphasis on imaging studies illustrating how nonvisual inputs recruit the visually deafferented occipital cortex. The second part covers crossmodal processing in deafness, with a special focus on the effects of deafness on visual processing. In the last portion of this review, we present the effects that the loss of a chemical sense have on the sensitivity of the other chemical senses, that is, smell, taste, and trigeminal chemosensation. We outline how the convergence of the chemical senses to the same central processing areas may lead to the observed reduction in sensitivity of the primarily not affected senses. Altogether, the studies reviewed herein illustrate the fascinating plasticity of the brain when coping with sensory deprivation. Keywords: blindness; deafness; anosmia; crossmodal plasticity.
senses during their lifetime. Still, persons with sensory loss are often able to live independently and can achieve an impressive degree of accomplishments. In fact, there is a plethora of reports (though often anecdotic) of persons with a sensory loss demonstrating extraordinary abilities with one or several of their remaining senses, with the large number of successful blind musicians being the most prominent example. Going back several decades, Diderot, in his “Lettre
Introduction While most humans can rely on several sensory systems to appropriately interact with the environment, some individuals are born without one or more senses while others may lose one or more *Corresponding author. Tel.: þ1-514-343-6111x0705; Fax: þ1-514-343-5787 E-mail:
[email protected] DOI: 10.1016/B978-0-444-53752-2.00002-3
233
234
sur les aveugles” (Diderot, 1749), reported the famous case of a blind mathematician who could recognize fake from real money coins just by touching them. Similarly, William James explained blind individuals’ remarkable ability to navigate through their environment without colliding with obstacles as resulting from a form of “facial perception” (James, 1890). At first glance, such performance may seem somewhat “supranormal.” However, over the past decades, we have acquired extensive knowledge on compensatory and adaptive changes in primarily unaffected senses occurring after sensory loss and have a better understanding as to how and why they occur. The substantial literature on such compensatory mechanisms that are observed in the blind has often attributed these enhancements to some form of “crossmodal plasticity.” Crossmodal plasticity generally refers to the adaptive reorganization of neurons to integrate the function of a new sensory modality following the loss of another. In fact, such crossmodal plasticity appears to at least partly explain many extraordinary abilities observed in persons with sensory loss. In the following sections, we provide an overview of crossmodal plastic changes that follow sensory loss. We specifically focus on three major topics, that is, blindness, deafness, and loss of chemical senses and how these states affect the other sensory systems.
Blindness Behavioral reorganization in blindness It has long been debated whether blind individuals have perceptual advantages or disadvantages in processing information received via the intact modalities. The fundamental question has been whether the lack of vision disrupts the proper development of nonvisual skills or if, in contrast, blindness enables above-normal performance in the preserved modalities. Even if
several studies support the notion that vision may be required to adequately calibrate other sensory modalities (Axelrod, 1959; Lewald, 2002; Zwiers et al., 2001), a substantial number of recent experiments have demonstrated that blind people are able to compensate for their lack of vision through efficient use of their remaining senses. In studies exploring sharpened nonvisual skills in blind people, spatial processing has been extensively investigated (Collignon et al., 2009c). This observation is probably due to the predominant role of vision in this cognitive ability and the importance for blind people to efficiently extract spatial information from the remaining senses in order to properly and safely navigate in their environment. In a seminal study, Lessard et al. (1998) investigated the auditory localization abilities of early blind individuals under binaural and monaural listening conditions. They first demonstrated that blind subjects can localize binaurally presented sounds as well as sighted individuals, suggesting that vision is not necessary for the construction of a three-dimensional auditory map of space. Moreover, half of the blind subjects significantly outperformed the sighted ones when they had to localize the sounds with one ear occluded (monaural localization). This finding strongly suggests that some blind individuals can use subtle spatial cues (i.e., spectral cues) more efficiently than sighted controls. Another consistent finding is that blind individuals typically outperform sighted ones in binaural localization tasks when the sound sources are located in more peripheral positions as opposed to when they are presented centrally (Roder et al., 1999; Simon et al., 2002, Voss et al., 2004). In recent experiments, we investigated the ability of blind participants to sharply focus their attention and quickly react to auditory or tactile spatial targets (Collignon and De Volder, 2009; Collignon et al., 2006). These studies demonstrated that blind subjects reacted faster than sighted controls to non visual spatial targets in selective and divided attention tasks further extending the
235
view that blind individuals are able to compensate their lack of vision by developing capacities in their remaining senses that exceed those of sighted individuals. The studies described above examined spatial hearing in near space, a region where auditory representations can be calibrated through sensory-motor feedback in blind subjects, such as touching the source of the sound or through the use of a cane, for example. In a later study, we evaluated sound localization in far space, a region of space where sensori-motor feedback could not contribute to the calibration of auditory spatial maps. We showed not only that blind individuals properly mapped their auditory distant space, but actually outperformed their sighted counterparts under specific conditions (Voss et al., 2004). Moreover, we examined whether late-onset blind subjects can manifest sensory compensation, since only a few studies have investigated this point. We thus carried out the task in late-blind subjects and showed that this group could also develop above-normal spatial abilities (Voss et al., 2004), as confirmed in another study (Fieger et al., 2006). However, a recent experiment showed that early but not late-blind participants showed better performance than that of sighted participants on a range of auditory perception tasks (Wan et al., 2010). Interestingly, in the above-mentioned studies, the superiority of early- and late-blind subjects was only present when sounds were presented in the periphery, where more subtle (e.g., spectral) auditory cues have to be exploited to efficiently resolve the task (Fieger et al., 2006; Roder et al., 1999; Simon et al., 2002; Voss et al., 2004). Similarly, when behavioral compensations are observed for the processing of visuospatial stimuli in deaf subjects, they also mainly concern inputs originating in the peripheral visual field (Bavelier et al., 2000; Neville and Lawson, 1987). These compensations observed specifically for peripheral stimuli may be related to the fact that differences in performance may emerge preferentially in conditions where the task is difficult
(i.e., the sighted subjects are not performing at near perfect levels). Recent studies have also pointed out that visual deprivation during early development results in important qualitative changes in nonvisual spatial perception (Eimer, 2004). Other experiments with blind people have suggested that the default localization of touch and proprioception in external space is in fact dependent on early visual experience (Hotting and Roder, 2009; Roder et al., 2004, 2008). For example, Roder et al. (2004) asked participants to judge the temporal order in which two tactile stimuli were delivered to their left and right hands. As expected, they found that temporal order judgments of sighted participants were less accurate with crossed than with uncrossed hands, which would result from the conflict between external and somatotopic spatial codes. By contrast, a congenitally blind group was completely unaffected by crossing the hands. Thus, it seems that sighted persons always use a visually defined reference frame to localize tactile events in external space (Kitazawa, 2002), and are impaired by conflicting external and somatotopic spatial information. By contrast, congenitally blind subjects do not use external spatial coordinates and thus remain unaffected by this conflict. Moreover, the fact that there is no need, in the case of early blindness, to make a correspondence between a nonvisual frame of reference and a visual one would contribute to a faster processing of nonvisual spatial information (Roder et al., 2004). This explanation was supported by an electroencephalographic study showing that the detection of deviant tactile stimuli at the hand induced event-related potentials that varied in crossed when compared to uncrossed postural conditions in sighted subjects, whereas changing the posture of the hand had no influence on the early blind subjects’ brain activity (Roder et al., 2008). In a recent study, we extended this finding by demonstrating that the use of an anatomically anchored reference system for touch and proprioception in subjects visually deprived since birth
236
impaired their ability to integrate audio-tactile information across postural changes (Collignon et al., 2009a). Altogether, these results thus demonstrate that the default remapping of touch/proprioception into external coordinates is acquired during early development as a consequence of visual input. It is, however, important to note that compensatory mechanisms following visual deprivation could extend beyond the auditory spatial domain. For example, enhanced performance in blind participants was also observed in auditory tasks involving pitch (Gougoux et al., 2004; Wan et al., 2010), echoes (Rice and Feinstein, 1965; Rice et al., 1965), or verbal (Amedi et al., 2003) discrimination. The tactile modality has also been studied in blind individuals and is especially interesting given its importance in Braille reading. Compared to sighted controls, blind subjects showed superior abilities in some tactile tasks, such as a haptic angle discrimination task (Alary et al., 2008) and a texture discrimination task, but exhibited similar grating orientation thresholds and vibrotactile frequency discrimination thresholds as the sighted subjects (Alary et al., 2009). A carefully designed study demonstrated that when age and sex of the two groups were carefully matched, the average blind subject had the acuity of an average sighted person of the same gender but 23 years younger (Goldreich and Kanics, 2003). A recent study by Wong and collaborators (2011) observed this heightened tactile acuity in blind subjects to depend on braille readings skills suggesting the sensory compensation to be a direct consequence of the practice of the blind subjects with the braille system. With regard to the chemical senses, several studies suggest that blind subjects outperform sighted subjects in difficult higher-order olfactory tasks, such as free odor identification and odor labeling (Murphy and Cain, 1986; Rosenbluth et al., 2000; Wakefield et al., 2004), but not in simpler and more basic olfactory tasks such as odor threshold or odor discrimination (Diekmann et al., 1994; Schwenn et al., 2002; Smith et al., 1993; Wakefield et al., 2004).
Brain reorganization in blindness Researchers have hypothesized for a long time that brain reorganization could underlie the changes in behavior observed in blind individuals. In particular, it was postulated that the functioning of visual structures changed dramatically following visual deprivation, and increasing evidence points now to the extensive colonization of the occipital cortex (OC)—traditionally considered as visual—by nonvisual inputs in blind individuals (Collignon et al., 2009c). In pioneering studies using positron emission tomography (PET), Veraart and collaborators demonstrated elevated metabolic activity in OC of early blind individuals at rest, which was at about the same level as in sighted subjects involved in a visual task (Veraart et al., 1990; Wanet-Defalque et al., 1988). Following the advent of more powerful neuroimaging techniques, a plethora of studies have demonstrated task-dependent activations of the OC during auditory (Kujala et al., 1997; Roder et al., 1999; Weeks et al., 2000), olfactory (Kupers et al., 2011) and tactile (Buchel et al., 1998; Burton et al., 2004; Gizewski et al., 2003) processing in early blind subjects. It is, however, possible that these results simply reflect an association between stimulus presentation and cortical activation, without there being any functional involvement of occipital areas in nonvisual processing. Transcranial magnetic stimulation (TMS), which induces a focal and transient disruption of the proper functioning of a targeted area, has been used to demonstrate the necessity of the OC of the blind for Braille reading (Cohen et al., 1997; Kupers et al., 2007) and verbal (Amedi et al., 2004) processing. We also demonstrated that TMS applied over the right dorsal extrastriate cortex interfered with the use of a prosthesis substituting vision by audition and with the localization of sounds in blind subjects (Collignon et al., 2007). By contrast, TMS targeting the same cortical area had no effect on any auditory performance in sighted subjects and did not interfere with pitch and intensity discriminations in the blind. The demonstration that transient perturbation of OC with TMS selectively
237
disrupted specific auditory processing in the blind compared to sighted subjects illustrates that this “visual” area is functionally linked to the neural network that underlies this auditory ability. We thus concluded that early visual deprivation leads to functional cerebral reorganization such that the right dorsal visual stream is recruited for the spatial processing of sounds, a result which is in clear agreement with previous neuroimaging studies on nonvisual space processing in this population (Arno et al., 2001; Poirier et al., 2006; Ricciardi et al., 2007; Vanlierde et al., 2003; Weeks et al., 2000). In a recent fMRI study we compared brain activity of congenitally blind and sighted participants processing either the spatial or the pitch properties of sounds carrying information in both domains (the same sounds were used in both tasks), using an adaptive procedure specifically designed to adjust for performance level. In addition to showing a substantial recruitment of the occipital cortex for sound processing in the blind, we also demonstrated that auditory-spatial processing mainly recruited regions of the dorsal occipital stream. Moreover, functional connectivity analyses revealed that these reorganized occipital regions are part of an extensive brain network including regions known to underlie audio-visual spatial abilities in sighted subjects (Collignon et al., 2011). It is worth noting that dorsal occipital regions have previously been shown to be involved in visuospatial processing in sighted subjects (Haxby et al., 1991). The similarity in the activation foci between visuospatial processing in the sighted and auditory spatial processing in the blind suggests that these areas may retain their functional and neuronal coding ability, which would enable them to process input from a different sensory modality. These results suggest that spatial processing in the blind maps onto specialized subregions of the OC known to be involved in the spatial processing of visual input in sighted people (Haxby et al., 1991). Interestingly, a recent study reported activation of a subregion of the lateraloccipital complex normally responsive to visual and tactile object-related processing when blind subjects extracted shape information from visualto-auditory sensory substitution soundscapes
(Amedi et al., 2007; see also Pietrini et al., 2004 for ventral activations in tactile shape recognition in the blind). In a similar manner, mental imagery of object shape recruited more ventral occipital areas (De Volder et al., 2001), whereas mental imagery of object position recruited more dorsal occipital regions (Vanlierde et al., 2003) in the blind. It thus appears that a functional dissociation between a ventral “what?” stream for the processing of object shape and a dorsal “where?” stream for the processing of space may also exist for nonvisual stimuli processed in the OC of blind subjects (Collignon et al., 2009c; Dormal and Collignon, 2011). In order to further understand whether occipital activity levels leads to differences in behavioral performance, several studies correlated individual levels of occipital activity in blind participants with performance in nonvisual tasks. In a study conducted in early blind individuals using a speaker array that permitted pseudo-free-field presentations of sounds during PET scanning, Gougoux and collaborators (Gougoux et al., 2005) observed that during monaural sound localization (one ear plugged), the degree of activation of several foci in the striate and extrastriate cortex correlated with sound localization accuracy (Fig. 1). This result not only confirms an enhanced recruitment of occipital regions in auditory spatial processing in blind subjects but also suggests that such restructuring of the auditory circuit may underlie their superior abilities. The above-mentioned studies undoubtedly demonstrate the presence of crossmodal plasticity in blind individuals, as cortical territories normally involved in visual processing are recruited for nonvisual functions. Still, questions remain about the nature of the mechanisms mediating such massive reorganizations. Top-down processing from associative cortices, feed-forward connections between primary sensory regions, or subcortical reorganizations are putative pathways that could explain how nonvisual inputs enter occipital areas of visually deprived subjects (Bavelier and Neville, 2002; Pascual-Leone et al., 2005). In order to further understand such
238
Percent CBF change
20 15
r = –0.81
10 5 0 –5
10
20
30
40
50
60
70
80
–10 –15
Y = –67
–20 20
Percent CBF change
15
r = –0.77
10 5 0 –5
10
20
30
40
50
60
70
80
–10 –15 –20
Y = –64
20
r = –0.68
Percent CBF change
15
Y = –83
10 5 0 –5
10
20
30
40
50
60
70
80
–10 –15 –20
Mean absolute error score (∞)
Fig. 1. Data of a correlational analysis between performance (mean absolute error) in a pointing task to monaurally presented sounds and cerebral blood flow (as measured by PET) in a group of blind subjects. The column of brain images illustrates regions in the ventral extrastriate (top), in the dorsal extrastriate (middle), and striate (bottom) cortices that correlate with monaural sound location performance in early blind subjects. Arrows point to the regions of interest. The scattergram shows the individual values extracted from each of these regions; closed circles indicate blind subjects; open circles indicate sighted controls; regression lines were fitted to data from blind subjects. Y coordinates refer to standardized stereotaxic space. With permission from Gougoux et al. (2005).
mechanisms, we used event-related TMS to disclose the time course of the spatial processing of sounds in the dorsolateral “where” stream of blind and sighted individuals (Collignon et al.,
2008, 2009b). To address this issue, we induced a virtual lesion of either the right intraparietal sulcus (rIPS) or the right dorsal extrastriate occipital cortex (rOC) at different delays in blind and
239
sighted subjects performing a sound lateralization task. We observed that TMS applied over rIPS 100–150 ms after sound onset disrupted the spatial processing of sound in sighted subjects but surprisingly had no influence on the task performance in blind individuals at any timing. In contrast, TMS applied over rOC 50 ms after sound onset disrupted the spatial processing of sounds in blind and in sighted participants. These studies suggest an early contribution of rOC in the spatial processing of sound in blind but also, to some extent, in sighted participants and also point to a lesser involvement of rIPS in this ability in blind participants. Given the very short latency of the disruptive effect of TMS applied over rOC on auditory spatial processing and considering the absence of rIPS contribution to this function in the blind, we suggested that sounds may reach the OC in blind subjects either via subcortical connections (Piche et al., 2007) or direct “feedforward” afferent projections arising from the auditory cortex (Falchier et al., 2002). However, further studies are needed to better understand how these mechanisms combine together and the influence of age of onset of blindness on the installation of such mechanisms.
Deafness The previous section provided evidence as to why the study of blind individuals constitutes an excellent model of the adaptability of the human brain, and how its plastic properties can in turn influence behavior and often improve sensory and cognitive abilities in these individuals. While crossmodal plasticity has been less extensively studied in the deaf, with the advent of small and efficient cochlear implants, it will become more and more important to understand crossmodal plasticity in deafness in order to comprehend the brain's ability to reverse the changes that followed sensory loss. Here, we will briefly review some of the main findings in the literature regarding crossmodal processing and plasticity in the deaf.
Behavioral reorganization in deafness Deaf individuals must rely more heavily on their remaining senses to carry out their everyday activities. The fine input they receive from the outside world is essentially limited to the binocular visual field, whereas precious information obtained from the auditory system can capture precepts from all directions in space covering 360 along any axis. Given this loss of information, do deaf individuals compensate for their deficit via heightened visual abilities? In other words, do they “see better” than hearing individuals? While some of the earlier studies produced very conflicting results, recent findings suggesting improved visual skills in the deaf tend to be more homogenous, in part because the individuals studied were themselves more homogenous as groups than in the past (see Bavelier et al., 2006). In recent studies, these groups were generally composed exclusively of deaf native signers, a subsample of the deaf population known to not suffer from comorbidity confounds related to language and communication deficits often associated with deafness (Meier, 1991). The heightened visual abilities in deaf native signers do not appear to be widespread, however, but rather seem limited to specific areas of visual cognition. For instance, basic sensory thresholds, such as contrast sensitivity (Finney and Dobkins, 2001), motion velocity (Brozinsky and Bavelier, 2004), motion sensitivity (Bosworth and Dobkins, 1999), brightness discrimination (Bross, 1979), and temporal resolution (Nava et al., 2008; Poizner and Tallal, 1987), do not appear to be enhanced in deaf individuals. Enhanced visual skills have rather revealed themselves in more complex tasks, where visual attention and/or processing of the peripheral visual field are manipulated (Bavelier et al., 2001; Dye et al., 2007; Loke and Song, 1991; Neville and Lawson, 1987; Neville et al., 1983; Proksch and Bavelier, 2002; Sladen et al., 2005; Stevens and Neville, 2006). It has thus been proposed that the loss of hearing leads to changes in higher-level attentional processing, with a redistribution of attentional resources to the periphery (see Bavelier
240
et al., 2006). However, this hypothesis has been challenged by the results of a recent study showing faster reactivity to visual events in the deaf compared to hearing individuals, regardless of spatial location (both peripheral and central; Bottari et al., 2010). Moreover, while hearing subjects were substantially slower for peripheral targets (in relation to central ones), deaf subjects were equally efficient across all spatial locations, suggesting functional enhancements for the peripheral visual field that cannot be explained by different attentional gradients alone.
Brain reorganization in deafness When considering the above-highlighted changes in visual processing, it naturally follows to ask whether we can observe an associated neuronal substrate to these improvements. There is now a substantial body of work looking at compensatory changes in the brain following early auditory deprivation; several studies have focused their attention on the middle temporal (MT) and middle superior temporal (MST) areas known to be not only involved in visual motion processing but also known to be heavily modulated by attentional processes. Consistent with the behavioral data, neuroimaging has revealed that differences in MT/MST between deaf and hearing individuals in response to motion stimuli only emerge when they are attended to in the peripheral field (Bavelier et al., 2001; Fine et al., 2005). However, one could argue that given the substantial role of motion in sign language, this difference could be due to the acquisition of this visuospatial language rather than to auditory deprivation per se. Bavelier et al. (2001) addressed this issue by including a second control group, one composed of hearing native signers, and showed that only early deafness and not early exposure to sign language lead to an increase of MT/MST activation. Other notable areas of interest are the auditory cortices that are deprived of their normal input following deafness. Early animal studies showed
that neurons in the primary auditory cortex could reorganize themselves to process visual information in the absence of auditory input (Pallas et al., 1990; Roe et al., 1992). More recently, several groups have shown BOLD changes in the auditory cortex of deaf individuals in response to visual motion (Finney and Dobkins, 2001; Finney et al., 2003; Sadato et al., 2004; Shibata, 2007). We have also recently investigated BOLD signal changes in both deaf and sighted individuals using global motion and forms defined by motion stimuli previously validated in healthy hearing individuals (see Vachon et al., 2009). Our preliminary results with deaf individuals are consistent with the current literature and show the involvement of higher-order auditory areas in the processing of the stimuli, most notably the right supratemporal gyrus (P. Vachon et al., unpublished). Similarly, several other groups have shown recruitment of the auditory cortex by visually presented sign language in deaf subjects (Nishimura et al., 1999; Petitto et al., 2000), and importantly, it was also shown that this crossmodal recruitment is not a by-product of signing, but rather of being auditorily deafferented (Fine et al., 2005). There are several potential ways in which crossmodal reorganization could lead to the observed functional changes in the deaf. First, anatomical support for visual processing in the auditory cortex comes from animal studies showing direct connections between both primary cortices (Falchier et al., 2002; Rockland and Ojima, 2003). However, corresponding pathways have yet to be identified in humans. Other anatomical findings have focused on the auditory cortex and the superior temporal gyrus, where morphometry and diffusion tensor imaging studies have shown a reduction in white matter as well as reduced diffusion anisotropy within remaining white matter in deaf individuals compared to hearing individuals (Emmorey et al., 2003; Kim et al., 2009; Shibata, 2007). While finding no differences within the auditory cortices, Penhune et al. (2003) did reveal an increase in gray matter density within the left motor
241
hand area, possibly related to more active use of the dominant hand in sign language. Finally, an important point worth discussing is the impact of the age of onset of deafness on crossmodal processing and plasticity. While studies with blind individuals have clearly shown the age of acquisition of blindness to modulate the observed plastic changes, only one study, to our knowledge, has specifically attempted to address this important issue in the deaf (Sadato et al., 2004). Both early and late-onset deaf groups showed similar activation of the planum temporale, but differed with respect to the activation in the middle superior temporal sulcus (STS), which was more prominent in the early deaf. Given that the middle STS corresponds to the main voice sensitive area, the authors argued that exposure to voices had hindered the region's ability to ultimately process sign language in the late deaf.
Anosmia, ageusia, loss of trigeminal chemosensation The chemical senses, that is, smell, taste, and the chemosensory trigeminal system, have obtained considerably less attention when compared to vision or audition. As opposed to physical senses, such as vision, audition, and touch, they allow us to experience our chemical environment via the interaction of substances with sensory organs, mostly, but not exclusively (Lindemann, 1996), via ligand–receptor interactions (Alimohammadi and Silver, 2000; Buck and Axel, 1991). Together, the three chemical senses constitute the main components of flavor perception (Small et al., 1997b). In the following paragraph, we will briefly outline the physiology of the chemical senses, in order to better understand the adaptive changes that occur when one of these senses is impaired or lost. Gustation, better known as the sense of taste, allows us to perceive five distinct taste qualities. In addition to the four classical ones (bitterness, sourness, saltiness, and sweetness; Lindemann,
2000), a fifth taste quality, umami, allows for the perception of the savory aspects of protein-rich food (Chaudhari et al., 2000). Taste receptors are located mostly on the tongue, although elsewhere in the oral cavity as well. In contrast to the sense of taste, the sense of smell allows us to perceive a virtually unlimited number of different odors. Volatile substances reach the olfactory receptor neurons, which are located in the upper portions of the nasal cavity, either orthonasally via the nostrils (while sniffing) or retronasally via the nasopharynx (Burdach et al., 1984). The latter is of utmost importance when perceiving the olfactory components of flavors from the oral cavity (Frasnelli et al., 2005). The chemosensory trigeminal system, finally, allows for the perception of burning, cooling, stinging, and other sensations originating from chemical substances (Laska et al., 1997). Here, trigeminal stimuli interact with receptors and free nerve endings of the trigeminal nerve throughout the oral and the nasal cavities. Since the chemical senses are perceptually interconnected so tightly (Small et al., 1997b), some have put forward the idea of a unique flavor sense (Auvray and Spence, 2008). In fact, a major complaint of individuals who lose one of their chemical senses relates to their reduced ability to appreciate foods.
Behavioral reorganization in chemosensory loss Olfactory dysfunctions can be categorized into quantitative dysfunctions (reduced sense of smell—hyposmia; loss of sense of smell—anosmia) and qualitative dysfunctions (altered perception of existing odors—parosmia; perception of inexistent odors—phantosmia; Leopold, 2002). These are relatively common conditions as up to 5% and 15% of the population are thought to exhibit anosmia and hyposmia, respectively (Bramerson et al., 2004; Landis and Hummel, 2006; Landis et al., 2004). Next to the physiological age related decline of olfactory function, the major etiologies of olfactory dysfunction are sinunasal diseases (polyps,
242
chronic rhino-sinusitis), viral infections (persisting dysfunction after upper respiratory tract infection), traumatic brain injury, neurodegenerative diseases (Parkinson's and Alzheimer's disease, etc.), and others. Up to 1% of the anosmic individuals exhibit congenital anosmia (Kallmann's syndrome, isolated congenital anosmia; Temmel et al., 2002). There are several reports on crossmodal effects of olfactory dysfunctions, mainly on other chemosensory systems. There is an established detrimental effect of olfactory dysfunction on trigeminal perception. When compared to controls, individuals with reduced olfactory function can perceive trigeminal stimuli only at higher concentrations (Frasnelli et al., 2010; Gudziol et al., 2001) and perceive suprathreshold stimuli as less intense (Frasnelli et al., 2007a). This reduced trigeminal sensitivity is, however, restricted to chemosensory trigeminal fibers (Frasnelli et al., 2006). A specific method to test trigeminal sensitivity is the odor lateralization task. In this test, subjects have to determine which of their two nostrils had been stimulated by an odorant in a monorhinal stimulation paradigm. We are only able to do so if the odorant also stimulates the trigeminal system (Kobal et al., 1989). Anosmic individuals have been shown to perform worse than healthy controls in the odor localization task (Hummel et al., 2003). With regard to effects of olfactory dysfunction on taste perception, it is important to note that most of the individuals suffering from an olfactory dysfunction complain about a taste disturbance (Deems et al., 1991). This is because they mainly experience the reduced retronasal olfactory sensation during flavor perception (Deems et al., 1991). This phenomenon can be very impressive as some persons with olfactory dysfunction do not believe their olfactory system to be disturbed at all. However, when referring specifically to gustation, that is, the perception of the five taste qualities, effects of olfactory loss on gustation are more debated. Some studies have reported that, in analogy to trigeminal function, gustatory function is also reduced in individuals with olfactory dysfunction (Gudziol et al., 2007; Landis
et al., 2010), while a recent report failed to confirm this finding (Stinton et al., 2010). As opposed to the commonly observed olfactory dysfunctions, a loss of trigeminal chemosensation is a very rare condition. In a case report, olfactory function was assessed in a woman who suffered from unilateral loss of trigeminal function on the left side resulting from a meningeoma. She also exhibited reduced olfactory function, as assessed with a behavioral test and the measurement of olfactory event-related potentials, but only ipsilaterally to the affected side. Her gustatory function was, however, similar on both sides of the tongue (Husner et al., 2006). While patients seeking help with a medical specialist often complain about a qualitatively altered taste perception (dysgeusia), a complete loss of gustatory sensation (ageusia) is a very rare condition (Deems et al., 1991). No reports of crossmodal effects of loss of gustatory function are known. In summary, a dysfunction or loss of one of the chemical senses is a relatively common finding. Olfaction is by far the most affected sensory system. However, no compensatory mechanisms appear to take place, where another (chemical) sense becomes more sensitive. Rather, the loss of a chemical sense (which in most cases is the loss of olfactory function) is usually accompanied by a reduced sensitivity in the other chemical senses. This is in sharp contrast to blindness and deafness, as described above. A possible explanation for this may be the tight connection of the different chemical senses, an expression of which is the perception of flavor. As stated above, some researchers have in fact put forward the idea of a unique “flavor sense,” consisting of inputs of all different contributing sensory channels (Auvray and Spence, 2008). The loss of one sense would therefore lead to a breakdown of the whole flavor system. There is indeed also evidence from imaging studies for such a flavor sense. The chemical senses share important central processing areas. For example, it has been shown that the orbitofrontal cortex (OFC) and its different subdivisions are activated by olfactory (e.g., Gottfried and Zald,
243
2005; Savic and Gulyas, 2000; Zatorre et al., 1992), gustatory (e.g., Hummel et al., 2007; Small et al., 1997a, 2003; Veldhuizen et al., 2007), and trigeminal (e.g., Albrecht et al., 2010; Boyle et al., 2007b) stimulation. Similarly, the insula is activated following olfactory (e.g., Bengtsson et al., 2001; Cerf-Ducastel and Murphy, 2003; Savic and Gulyas, 2000), gustatory (e.g., Small et al., 1999, 2003; Veldhuizen et al., 2007), and trigeminal (e.g., Albrecht et al., 2010; Boyle et al., 2007b; Iannilli et al., 2008) stimulation. More importantly, combined stimuli consisting of mixtures of gustatory, olfactory, and/or trigeminal stimuli have been shown to activate “chemosensory” brain regions to a higher degree than their single constituents. In their seminal paper, Small and collaborators (1997b) showed that the administration of matching gustatory and olfactory stimuli together evoked different changes in cerebral blood flow in the insula, the opercula, and the OFC than the administration of both kinds of stimuli on their own. Similarly, using the trigeminal stimulus CO2 together with the pure olfactory stimulus phenyl ethanol, we showed that a mixture of both activated chemosensory centers (left OFC) and integration areas (left STS, rIPS) to a higher degree than the mathematical sum of the single components (Boyle et al., 2007a). Cerf-Ducastel et al. (2001) finally showed that both gustatory and lingual trigeminal stimuli showed a striking overlap in their activation of the insula as well as the rolandic, frontal, and temporal opercula. Again, these studies support the existence of a cerebral network for flavor consisting mainly of the OFC as well as the insula and surrounding cortex.
Brain reorganization in chemosensory loss Unfortunately, only few reports are available on changes in brain activations due to chemosensory loss. In accordance with the behavioral findings, anosmic and hyposmic individuals exhibit smaller trigeminal event-related potentials (Frasnelli
et al., 2007a; Hummel et al., 1996). Similarly, following trigeminal stimulation with the trigeminal stimulus carbon dioxide, persons suffering from anosmia were described to exhibit smaller activations in “chemosensory” brain regions when compared to controls with a normal sense of smell. The anosmia group, however, exhibited larger responses in other regions in the frontal and temporal lobe, which usually are not involved in chemosensory perception (Iannilli et al., 2007). However, there appears to be a dissociation between peripheral and central levels of trigeminal processing. When the negative mucosal potential (NMP)—a measure of peripheral responsiveness—is assessed, individuals with anosmia or hyposmia exhibit larger responses than healthy controls, which is in striking contrast to the findings in central responses (Frasnelli et al., 2007a,b). Thus, a model of mixed sensory adaptation/compensation in the interaction between the olfactory and the trigeminal system has been put forward. In normal functioning systems, peripheral trigeminal responsiveness is constantly inhibited; consequently, the periphery of the trigeminal system is functionally downregulated. On central levels, trigeminal input is increased by olfactory costimulation resulting in larger signals. In olfactory loss, however, a release of peripheral inhibition occurs, resulting in increased peripheral susceptibility. However, there is no olfactory costimulation to be integrated, resulting in relatively smaller central signals (Frasnelli et al., 2007a,b; Fig. 2). These data therefore suggest the mechanisms in chemosensory loss to be different from other sensory systems. A first difference is that the chemical senses converge, at least partly, to the same processing areas. Second, sensory loss leads to a reduction in sensitivity in the other senses as well, in addition to the loss in the primarily affected sense. More studies are needed to confirm a causal connection between these consistent observations and to deepen our understanding of crossmodal effects of a loss in the chemical senses.
244 (a) -5
(b1)
(b2)
mV
-15
1s
-200
mV
150
1s
Fig. 2. Effects of loss of olfactory function on the trigeminal chemosensory system. (A) Grand means of trigeminal event-related potentials (central measure; top) and negative mucosal potential (NMP; peripheral measure; bottom) following stimuli of 60% (v/v) CO2 in subjects with acquired anosmia (black) and controls (gray). The black horizontal bars indicate the onset and duration of the CO2 stimulus. (B) Model of the interaction between olfactory (gray arrows) and trigeminal (black arrows) systems. (B1) Normal conditions. Peripheral responsiveness is decreased due to constant activation of intrabulbar trigeminal collaterals and consequent functional downregulation in the periphery of the trigeminal system. Functional integration of olfactory and trigeminal processes leads to augmented cortical signal. (B2) Olfactory loss. Increased NMP due to top downregulation; decreased event-related potential due to missing olfactory augmentation. With permission from Frasnelli et al. (2007b).
Conclusion
References
Loss of a sensory system has vast consequences for the affected person and his interactions with environment. Here, we have outlined how sensory loss leads to changes in primarily unaffected sensory systems. This crossmodal plasticity shows in a fascinating way how the brain copes with sensory deprivation. Only the proper understanding of the mechanisms of crossmodal plasticity will allow us to develop tools to help persons with sensory loss to better experience the world with the unaffected senses and thus enable them to live more independently.
Alary, F., Duquette, M., Goldstein, R., Elaine Chapman, C., Voss, P., La Buissonniere-Ariza, V., et al. (2009). Tactile acuity in the blind: A closer look reveals superiority over the sighted in some but not all cutaneous tasks. Neuropsychologia, 47, 2037–2043. Alary, F., Goldstein, R., Duquette, M., Chapman, C. E., Voss, P., & Lepore, F. (2008). Tactile acuity in the blind: A psychophysical study using a two-dimensional angle discrimination task. Experimental Brain Research, 187, 587–594. Albrecht, J., Kopietz, R., Frasnelli, J., Wiesmann, M., Hummel, T., & Lundstrom, J. N. (2010). The neuronal correlates of intranasal trigeminal function—An ALE metaanalysis of human functional brain imaging data. Brain Research Reviews, 62, 183–196.
245 Alimohammadi, H., & Silver, W. L. (2000). Evidence for nicotinic acetylcholine receptors on nasal trigeminal nerve endings of the rat. Chemical Senses, 25, 61–66. Amedi, A., Floel, A., Knecht, S., Zohary, E., & Cohen, L. G. (2004). Transcranial magnetic stimulation of the occipital pole interferes with verbal processing in blind subjects. Nature Neuroscience, 7, 1266–1270. Amedi, A., Raz, N., Pianka, P., Malach, R., & Zohary, E. (2003). Early ‘visual’ cortex activation correlates with superior verbal memory performance in the blind. Nature Neuroscience, 6, 758–766. Amedi, A., Stern, W. M., Camprodon, J. A., Bermpohl, F., Merabet, L., Rotman, S., et al. (2007). Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex. Nature Neuroscience, 10, 687–689. Arno, P., De Volder, A. G., Vanlierde, A., WanetDefalque, M. C., Streel, E., Robert, A., et al. (2001). Occipital activation by pattern recognition in the early blind using auditory substitution for vision. Neuroimage, 13, 632–645. Auvray, M., & Spence, C. (2008). The multisensory perception of flavor. Consciousness and Cognition, 17, 1016–1031. Axelrod, S. (Ed.), (1959). Effect of early blindness: Performance of blind and sighted children on tactile and auditory tasks. (Research Series No. 7) New York: American Foundation for the Blind. Bavelier, D., Brozinsky, C., Tomann, A., Mitchell, T., Neville, H., & Liu, G. (2001). Impact of early deafness and early exposure to sign language on the cerebral organization for motion processing. The Journal of Neuroscience, 21, 8931–8942. Bavelier, D., Dye, M. W. G., & Hauser, P. C. (2006). Do deaf individuals see better? Trends in Cognitive Sciences, 10, 512–518. Bavelier, D., & Neville, H. J. (2002). Cross-modal plasticity: Where and how? Nature Reviews. Neuroscience, 3, 443–452. Bavelier, D., Tomann, A., Hutton, C., Mitchell, T., Corina, D., Liu, G., et al. (2000). Visual attention to the periphery is enhanced in congenitally deaf individuals. The Journal of Neuroscience, 20, RC93. Bengtsson, S., Berglund, H., Gulyas, B., Cohen, E., & Savic, I. (2001). Brain activation during odor perception in males and females. Neuroreport, 12, 2027–2033. Bosworth, R. G., & Dobkins, K. R. (1999). Left-hemisphere dominance for motion processing in deaf signers. Psychological Science, 10, 256–262. Bottari, D., Nava, E., Ley, P., & Pavani, F. (2010). Enhanced reactivity to visual stimuli in deaf individuals. Restorative Neurology and Neuroscience, 28, 167–179. Boyle, J. A., Frasnelli, J., Gerber, J., Heinke, M., & Hummel, T. (2007). Cross-modal integration of intranasal stimuli: A functional magnetic resonance imaging study. Neuroscience, 149, 223–231.
Boyle, J. A., Heinke, M., Gerber, J., Frasnelli, J., & Hummel, T. (2007). Cerebral activation to intranasal chemosensory trigeminal stimulation. Chemical Senses, 32, 343–353. Bramerson, A., Johansson, L., Ek, L., Nordin, S., & Bende, M. (2004). Prevalence of olfactory dysfunction: The skovde population-based study. The Laryngoscope, 114, 733–737. Bross, M. (1979). Residual sensory capacities of the deaf—Signal-detection analysis of a visual-discrimination task. Perceptual and Motor Skills, 48, 187–194. Brozinsky, C. J., & Bavelier, D. (2004). Motion velocity thresholds in deaf signers: Changes in lateralization but not in overall sensitivity. Cognitive Brain Research, 21, 1–10. Buchel, C., Price, C., Frackowiak, R. S. J., & Friston, K. (1998). Different activation patterns in the visual cortex of late and congenitally blind subjects. Brain, 121, 409–419. Buck, L., & Axel, R. (1991). A novel multigene family may encode odorant receptors: A molecular basis for odor recognition. Cell, 65, 175–187. Burdach, K. J., Kroeze, J. H., & Koster, E. P. (1984). Nasal, retronasal, and gustatory perception: An experimental comparison. Perception & Psychophysics, 36, 205–208. Burton, H., Sinclair, R. J., & McLaren, D. G. (2004). Cortical activity to vibrotactile stimulation: An fMRI study in blind and sighted individuals. Human Brain Mapping, 23, 210–228. Cerf-Ducastel, B., & Murphy, C. (2003). FMRI brain activation in response to odors is reduced in primary olfactory areas of elderly subjects. Brain Research, 986, 39–53. Cerf-Ducastel, B., Van de Moortele, P. F., MacLeod, P., Le Bihan, D., & Faurion, A. (2001). Interaction of gustatory and lingual somatosensory perceptions at the cortical level in the human: A functional magnetic resonance imaging study. Chemical Senses, 26, 371–383. Chaudhari, N., Landin, M. A., & Roper, S. D. (2000). A metabotropic glutamate receptor variant functions as a taste receptor. Nature Neuroscience, 3, 113–119. Cohen, L. G., Celnik, P., PascualLeone, A., Corwell, B., Faiz, L., Dambrosia, J., et al. (1997). Functional relevance of crossmodal plasticity in blind humans. Nature, 389, 180–183. Collignon, O., Charbonneau, G., Lassonde, M., & Lepore, F. (2009). Early visual deprivation alters multisensory processing in peripersonal space. Neuropsychologia, 47, 3236–3243. Collignon, O., Davare, M., De Volder, A. G., Poirier, C., Olivier, E., & Veraart, C. (2008). Time-course of posterior parietal and occipital cortex contribution to sound localization. Journal of Cognitive Neuroscience, 20, 1454–1463. Collignon, O., Davare, M., Olivier, E., & De Volder, A. G. (2009). Reorganisation of the right occipito-parietal stream for auditory spatial processing in early blind humans. A transcranial magnetic stimulation study. Brain Topography, 21, 232–240.
246 Collignon, O., & De Volder, A. G. (2009). Further evidence that congenitally blind participants react faster to auditory and tactile spatial targets. Canadian Journal of Experimental Psychology, 63, 287–293. Collignon, O., Lassonde, M., Lepore, F., Bastien, D., & Veraart, C. (2007). Functional cerebral reorganization for auditory spatial processing and auditory substitution of vision in early blind subjects. Cerebral Cortex, 17, 457–465. Collignon, O., Renier, L., Bruyer, R., Tranduy, D., & Veraart, C. (2006). Improved selective and divided spatial attention in early blind subjects. Brain Research, 1075, 175–182. Collignon, O., Vandewalle, G., Voss, P., Albouy, G., Charbonneau, G., Lassonde, M., & Lepore, F. (2011). Functional specialization for auditory-spatial processing in the occipital cortex of congenitally blind humans. Proceedings of the National Academy of Sciences, 108, 4435–4440. Collignon, O., Voss, P., Lassonde, M., & Lepore, F. (2009). Cross-modal plasticity for the spatial processing of sounds in visually deprived subjects. Experimental Brain Research, 192, 343–358. De Volder, A. G., Toyama, H., Kimura, Y., Kiyosawa, M., Nakano, H., Vanlierde, A., et al. (2001). Auditory triggered mental imagery of shape involves visual association areas in early blind humans. Neuroimage, 14, 129–139. Deems, D. A., Doty, R. L., Settle, R. G., Moore-Gillon, V., Shaman, P., Mester, A. F., et al. (1991). Smell and taste disorders, a study of 750 patients from the University of Pennsylvania Smell and Taste Center. Archives of Otolaryngology—Head & Neck Surgery, 117, 519–528. Diderot, D., 1749. Lettre sur les aveugles à l'usage de ceux qui voient (London). Diekmann, H., Walger, M., & von Wedel, H. (1994). Sense of smell in deaf and blind patients. HNO, 42, 264–269. Dormal, G., & Collignon, O. (2011). Functional selectivity in sensory deprived cortices. Journal of Neurophysiology, doi:10.1152/jn.00109.2011. Dye, M. W. G., Baril, D. E., & Bavelier, D. (2007). Which aspects of visual attention are changed by deafness? The case of the Attentional Network Test. Neuropsychologia, 45, 1801–1811. Eimer, M. (2004). Multisensory integration: How visual experience shapes spatial perception. Current Biology, 14, R115–R117. Emmorey, K., Allen, J. S., Bruss, J., Schenker, N., & Damasio, H. (2003). A morphornetric analysis of auditory brain regions in congenitally deaf adults. Proceedings of the National Academy of Sciences of the United States of America, 100, 10049–10054. Falchier, A., Clavagnier, S., Barone, P., & Kennedy, H. (2002). Anatomical evidence of multimodal integration in primate striate cortex. The Journal of Neuroscience, 22, 5749–5759.
Fieger, A., Roder, B., Teder-Salejarvi, W., Hillyard, S. A., & Neville, H. J. (2006). Auditory spatial tuning in late-onset blindness in humans. Journal of Cognitive Neuroscience, 18, 149–157. Fine, I., Finney, E. M., Boynton, G. M., & Dobkins, K. R. (2005). Comparing the effects of auditory deprivation and sign language within the auditory and visual cortex. Journal of Cognitive Neuroscience, 17, 1621–1637. Finney, E. M., Clementz, B. A., Hickok, G., & Dobkins, D. R. (2003). Visual stimuli activate auditory cortex in deaf subjects: Evidence from MEG. Neuroreport, 14, 1425–1427. Finney, E. M., & Dobkins, K. R. (2001). Visual contrast sensitivity in deaf versus hearing populations: Exploring the perceptual consequences of auditory deprivation and experience with a visual language. Cognitive Brain Research, 11, 171–183. Frasnelli, J., Schuster, B., & Hummel, T. (2007a). Interactions between olfaction and the trigeminal system: What can be learned from olfactory loss. Cerebral Cortex, 17, 2268–2275. Frasnelli, J., Schuster, B., & Hummel, T. (2007b). Subjects with congenital anosmia have larger peripheral but similar central trigeminal responses. Cerebral Cortex, 17, 370–377. Frasnelli, J., Schuster, B., & Hummel, T. (2010). Olfactory dysfunction affects thresholds to trigeminal chemosensory sensations. Neuroscience Letters, 468, 259–263. Frasnelli, J., Schuster, B., Zahnert, T., & Hummel, T. (2006). Chemosensory specific reduction of trigeminal sensitivity in subjects with olfactory dysfunction. Neuroscience, 142, 541–546. Frasnelli, J., van Ruth, S., Kriukova, I., & Hummel, T. (2005). Intranasal concentrations of orally administered flavors. Chemical Senses, 30, 575–582. Gizewski, E. R., Gasser, T., de Greiff, A., Boehm, A., & Forsting, M. (2003). Cross-modal plasticity for sensory and motor activation patterns in blind subjects. Neuroimage, 19, 968–975. Goldreich, D., & Kanics, I. M. (2003). Tactile acuity is enhanced in blindness. The Journal of Neuroscience, 23, 3439–3445. Gottfried, J. A., & Zald, D. H. (2005). On the scent of human olfactory orbitofrontal cortex: Meta-analysis and comparison to non-human primates. Brain Research Reviews, 50, 287–304. Gougoux, F., Lepore, F., Lassonde, M., Voss, P., Zatorre, R. J., & Belin, P. (2004). Neuropsychology: Pitch discrimination in the early blind. Nature, 430, 309. Gougoux, F., Zatorre, R. J., Lassonde, M., Voss, P., & Lepore, F. (2005). A functional neuroimaging study of sound localization: Visual cortex activity predicts performance in early-blind individuals. PLoS Biology, 3, 324–333. Gudziol, H., Rahneberg, K., & Burkert, S. (2007). Anosmiker schmecken schlechter als Gesunde. Laryngo-rhino-otologie, 86, 640–643.
247 Gudziol, H., Schubert, M., & Hummel, T. (2001). Decreased trigeminal sensitivity in anosmia. ORL; Journal of Otorhinolaryngology and its Related Specialties, 63, 72–75. Haxby, J. V., Grady, C. L., Horwitz, B., Ungerleider, L. G., Mishkin, M., Carson, R. E., et al. (1991). Dissociation of object and spatial visual processing pathways in human extrastriate cortex. Proceedings of the National Academy of Sciences of the United States of America, 88, 1621–1625. Hotting, K., & Roder, B. (2009). Auditory and auditory-tactile processing in congenitally blind humans. Hearing Research, 258, 165–174. Hummel, T., Barz, S., Lotsch, J., Roscher, S., Kettenmann, B., & Kobal, G. (1996). Loss of olfactory function leads to a decrease of trigeminal sensitivity. Chemical Senses, 21, 75–79. Hummel, C., Frasnelli, J., Gerber, J., & Hummel, T. (2007). Cerebral processing of gustatory stimuli in patients with taste loss. Behavioural Brain Research, 185, 59–64. Hummel, T., Futschik, T., Frasnelli, J., & Huttenbrink, K. B. (2003). Effects of olfactory function, age, and gender on trigeminally mediated sensations: A study based on the lateralization of chemosensory stimuli. Toxicology Letters, 140–141, 273–280. Husner, A., Frasnelli, J., Welge-Lussen, A., Reiss, G., Zahnert, T., & Hummel, T. (2006). Loss of trigeminal sensitivity reduces olfactory function. The Laryngoscope, 116, 1520–1522. Iannilli, E., Del Gratta, C., Gerber, J. C., Romani, G. L., & Hummel, T. (2008). Trigeminal activation using chemical, electrical, and mechanical stimuli. Pain, 139, 376–388. Iannilli, E., Gerber, J., Frasnelli, J., & Hummel, T. (2007). Intranasal trigeminal function in subjects with and without an intact sense of smell. Brain Research, 1139, 235–244. James, W. (1890). Principles of psychology (Vol. 1). New York: Henry Holt and Company. Kim, D. J., Park, S. Y., Kim, J., Lee, D. H., & Park, H. J. (2009). Alterations of white matter diffusion anisotropy in early deafness. Neuroreport, 20, 1032–1036. Kitazawa, S. (2002). Where conscious sensation takes place. Consciousness and Cognition, 11, 475–477. Kobal, G., Van Toller, S., & Hummel, T. (1989). Is there directional smelling? Experientia, 45, 130–132. Kujala, T., Alho, K., Huotilainen, M., Ilmoniemi, R. J., Lehtokoski, A., Leinonen, A., et al. (1997). Electrophysiological evidence for cross-modal plasticity in humans with earlyand late-onset blindness. Psychophysiology, 34, 213–216. Kupers, R., Beaulieu-Lefebvre, M., Schneider, F. C., Kassuba, T., Paulson, O. B., Siebner, H. R., & Ptito, M. (2011). Neural correlates of olfactory processing in congenital blindness. Neuropsychologia, doi:10.1016/j. neuropsychologia.2011.03.033. Kupers, R., Pappens, M., de Noordhout, A. M., Schoenen, J., Ptito, M., & Fumal, A. (2007). rTMS of the occipital cortex
abolishes Braille reading and repetition priming in blind subjects. Neurology, 68, 691–693. Landis, B. N., & Hummel, T. (2006). New evidence for high occurrence of olfactory dysfunctions within the population. The American Journal of Medicine, 119, 91–92. Landis, B. N., Konnerth, C. G., & Hummel, T. (2004). A study on the frequency of olfactory dysfunction. The Laryngoscope, 114, 1764–1769. Landis, B. N., Scheibe, M., Weber, C., Berger, R., Bramerson, A., Bende, M., et al. (2010). Chemosensory interaction: Acquired olfactory impairment is associated with decreased taste function. Journal of Neurology, 257, 1303–1308. Laska, M., Distel, H., & Hudson, R. (1997). Trigeminal perception of odorant quality in congenitally anosmic subjects. Chemical Senses, 22, 447–456. Leopold, D. (2002). Distortion of olfactory perception: Diagnosis and treatment. Chemical Senses, 27, 611–615. Lessard, N., Pare, M., Lepore, F., & Lassonde, W. (1998). Early-blind human subjects localize sound sources better than sighted subjects. Nature, 395, 278–280. Lewald, J. (2002). Vertical sound localization in blind humans. Neuropsychologia, 40, 1868–1872. Lindemann, B. (1996). Taste reception. Physiological Reviews, 76, 719–766. Lindemann, B. (2000). A taste for umami. Nature Neuroscience, 3, 99–100. Loke, W. H., & Song, S. R. (1991). Central and peripheral visual processing in hearing and nonhearing individuals. Bulletin of the Psychonomic Society, 29, 437–440. Meier, R. P. (1991). Language-acquisition by deaf-children. American Scientist, 79, 60–70. Murphy, C., & Cain, W. S. (1986). Odor identification: The blind are better. Physiology & Behavior, 37, 177–180. Nava, E., Bottari, D., Zampini, M., & Pavani, F. (2008). Visual temporal order judgment in profoundly deaf individuals. Experimental Brain Research, 190, 179–188. Neville, H. J., & Lawson, D. (1987). Attention to central and peripheral visual space in a movement detection task: An event-related potential and behavioral study. II. Congenitally deaf adults. Brain Research, 405, 268–283. Neville, H. J., Schmidt, A., & Kutas, M. (1983). Altered visual evoked-potentials in congenitally deaf adults. Brain Research, 266, 127–132. Nishimura, H., Hashikawa, K., Doi, K., Iwaki, T., Watanabe, Y., Kusuoka, H., et al. (1999). Sign language ‘heard’ in the auditory cortex. Nature, 397, 116. Pallas, S. L., Roe, A. W., & Sur, M. (1990). Visual projections induced into the auditory pathway of ferrets. 1. Novel inputs to primary auditory-cortex (Ai) from the Lp pulvinar complex and the topography of the MGN-AI projection. The Journal of Comparative Neurology, 298, 50–68.
248 Pascual-Leone, A., Amedi, A., Fregni, F., & Merabet, L. B. (2005). The plastic human brain cortex. Annual Review of Neuroscience, 28, 377–401. Penhune, V. B., Cismaru, R., Dorsaint-Pierre, R., Petitto, L. A., & Zatorre, R. J. (2003). The morphometry of auditory cortex in the congenitally deaf measured using MRI. Neuroimage, 20, 1215–1225. Petitto, L. A., Zatorre, R. J., Gauna, K., Nikelski, E. J., Dostie, D., & Evans, A. C. (2000). Speech-like cerebral activity in profoundly deaf people processing signed languages: Implications for the neural basis of human language. Proceedings of the National Academy of Sciences of the United States of America, 97, 13961–13966. Piche, M., Chabot, N., Bronchti, G., Miceli, D., Lepore, F., & Guillemot, J. P. (2007). Auditory responses in the visual cortex of neonatally enucleated rats. Neuroscience, 145, 1144–1156. Pietrini, P., Furey, M. L., Ricciardi, E., Gobbini, M. I., Wu, W. H. C., Cohen, L., et al. (2004). Beyond sensory images: Object-based representation in the human ventral pathway. Proceedings of the National Academy of Sciences of the United States of America, 101, 5658–5663. Poirier, C., Collignon, O., Scheiber, C., Renier, L., Vanlierde, A., Tranduy, D., et al. (2006). Auditory motion perception activates visual motion areas in early blind subjects. Neuroimage, 31, 279–285. Poizner, H., & Tallal, P. (1987). Temporal processing in deaf signers. Brain and Language, 30, 52–62. Proksch, J., & Bavelier, D. (2002). Changes in the spatial distribution of visual attention after early deafness. Journal of Cognitive Neuroscience, 14, 687–701. Ricciardi, E., Vanello, N., Sani, L., Gentili, C., Scilingo, E. P., Landini, L., et al. (2007). The effect of visual experience on the development of functional architecture in hMT. Cerebral Cortex, 17, 2933–2939. Rice, C. E., & Feinstein, S. H. (1965). Sonar system of the blind: Size discrimination. Science, 148, 1107–1108. Rice, C. E., Feinstein, S. H., & Schusterman, R. J. (1965). Echo-detection ability of the blind: Size and distance factors. Journal of Experimental Psychology, 70, 246–255. Rockland, K. S., & Ojima, H. (2003). Multisensory convergence in calcarine visual areas in macaque monkey. International Journal of Psychophysiology, 50, 19–26. Roder, B., Focker, J., Hotting, K., & Spence, C. (2008). Spatial coordinate systems for tactile spatial attention depend on developmental vision: Evidence from event-related potentials in sighted and congenitally blind adult humans. The European Journal of Neuroscience, 28, 475–483. Roder, B., Rosler, F., & Spence, C. (2004). Early vision impairs tactile perception in the blind. Current Biology, 14, 121–124. Roder, B., Teder-Salejarvi, W., Sterr, A., Rosler, F., Hillyard, S. A., & Neville, H. J. (1999). Improved auditory spatial tuning in blind humans. Nature, 400, 162–166.
Roe, A. W., Pallas, S. L., Kwon, Y. H., & Sur, M. (1992). Visual projections routed to the auditory pathway in ferrets: Receptive fields of visual neurons in primary auditory cortex. The Journal of Neuroscience, 12, 3651–3664. Rosenbluth, R., Grossman, E. S., & Kaitz, M. (2000). Performance of early-blind and sighted children on olfactory tasks. Perception, 29, 101–110. Sadato, N., Yamada, H., Okada, T., Yoshida, M., Hasegawa, T., Matsuki, K., et al. (2004). Age-dependent plasticity in the superior temporal sulcus in deaf humans: A functional MRI study. BMC Neuroscience, 5, 56. Savic, I., & Gulyas, B. (2000). PET shows that odors are processed both ipsilaterally and contralaterally to the stimulated nostril. Neuroreport, 11, 2861–2866. Schwenn, O., Hundorf, I., Moll, B., Pitz, S., & Mann, W. J. (2002). Do blind persons have a better sense of smell than normal sighted people? Klinische Monatsblätter für Augenheilkunde, 219, 649–654. Shibata, D. K. (2007). Differences in brain structure in deaf persons on MR imaging studied with voxel-based morphometry. American Journal of Neuroradiology, 28, 243–249. Simon, H. J., Divenyi, P. L., & Lotze, A. (2002). Lateralization of narrow-band noise by blind and sighted listeners. Perception, 31, 855–873. Sladen, D. P., Tharpe, A. M., Daniel, A., & Grantham, D. W. (2005). Visual attention in deaf and normal hearing adults: Effects of stimulus compatibility. Journal of Speech, Language, and Hearing Research, 48, 1529–1537. Small, D. M., Gregory, M. D., Mak, Y. E., Gitelman, D., Mesulam, M. M., & Parrish, T. (2003). Dissociation of neural representation of intensity and affective valuation in human gustation. Neuron, 39, 701–711. Small, D. M., Jones-Gotman, M., Zatorre, R. J., Petrides, M., & Evans, A. C. (1997a). A role for the right anterior temporal lobe in taste quality recognition. The Journal of Neuroscience, 17, 5136–5142. Small, D. M., Jones-Gotman, M., Zatorre, R. J., Petrides, M., & Evans, A. C. (1997b). Flavor processing: More than the sum of its parts. Neuroreport, 8, 3913–3917. Small, D. M., Zald, D. H., Jones-Gotman, M., Zatorre, R. J., Pardo, J. V., Frey, S., et al. (1999). Human cortical gustatory areas: A review of functional neuroimaging data. Neuroreport, 10, 7–14. Smith, R. S., Doty, R. L., Burlingame, G. K., & McKeown, D. A. (1993). Smell and taste function in the visually impaired. Perception & Psychophysics, 54, 649–655. Stevens, C., & Neville, H. (2006). Neuroplasticity as a doubleedged sword: Deaf enhancements and dyslexic deficits in motion processing. Journal of Cognitive Neuroscience, 18, 701–714. Stinton, N., Atif, M. A., Barkat, N., & Doty, R. L. (2010). Influence of smell loss on taste function. Behavioral Neuroscience, 124, 256–264.
249 Temmel, A. F., Quint, C., Schickinger-Fischer, B., Klimek, L., Stoller, E., & Hummel, T. (2002). Characteristics of olfactory disorders in relation to major causes of olfactory loss. Archives of Otolaryngology—Head & Neck Surgery, 128, 635–641. Vachon, P., Voss, P., Lassonde, M., Leroux, J. M., Mensour, B., Beaudoin, G., et al. (2009). Global motion stimuli and form-from-motion stimuli: Common characteristics and differential activation patterns. The International Journal of Neuroscience, 119, 1584–1601. Vanlierde, A., De Volder, A. G., Wanet-Defalque, M. C., & Veraart, C. (2003). Occipito-parietal cortex activation during visuo-spatial imagery in early blind humans. Neuroimage, 19, 698–709. Veldhuizen, M. G., Bender, G., Constable, R. T., & Small, D. M. (2007). Trying to detect taste in a tasteless solution: Modulation of early gustatory cortex by attention to taste. Chemical Senses, 32, 569–581. Veraart, C., Devolder, A. G., Wanetdefalque, M. C., Bol, A., Michel, C., & Goffinet, A. M. (1990). Glucose-utilization in human visual-cortex is abnormally elevated in blindness of early onset but decreased in blindness of late onset. Brain Research, 510, 115–121. Voss, P., Lassonde, M., Gougoux, F., Fortin, M., Guillemot, J. P., & Lepore, F. (2004). Early- and late-onset blind individuals show supra-normal auditory abilities in far-space. Current Biology, 14, 1734–1738.
Wakefield, C. E., Homewood, J., & Taylor, A. J. (2004). Cognitive compensations for blindness in children: An investigation using odour naming. Perception, 33, 429–442. Wan, C. Y., Wood, A. G., Reutens, D. C., & Wilson, S. J. (2010). Early but not late-blindness leads to enhanced auditory perception. Neuropsychologia, 48, 344–348. Wanet-Defalque, M. C., Veraart, C., De Volder, A., Metz, R., Michel, C., Dooms, G., et al. (1988). High metabolic activity in the visual cortex of early blind human subjects. Brain Research, 446, 369–373. Weeks, R., Horwitz, B., Aziz-Sultan, A., Tian, B., Wessinger, C. M., Cohen, L. G., et al. (2000). A positron emission tomographic study of auditory localization in the congenitally blind. The Journal of Neuroscience, 20, 2664–2672. Wong, M., Gnanakumaran, V., & Goldreich, D. (2011). Tactile Spatial Acuity Enhancement in Blindness: Evidence for Experience-Dependent MechanismsTactile Spatial Acuity Enhancement in Blindness: Evidence for ExperienceDependent Mechanisms. The Journal of Neuroscience, 31, 7028–7037. Zatorre, R. J., Jones-Gotman, M., Evans, A. C., & Meyer, E. (1992). Functional localization and lateralization of human olfactory cortex. Nature, 360, 339–340. Zwiers, M. P., Van Opstal, A. J., & Cruysberg, J. R. (2001). A spatial hearing deficit in early-blind humans. The Journal of Neuroscience, 21, RC142, 1–5.
A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.) Progress in Brain Research, Vol. 191 ISSN: 0079-6123 Copyright Ó 2011 Elsevier B.V. All rights reserved.
CHAPTER 16
Adaptive crossmodal plasticity in deaf auditory cortex: areal and laminar contributions to supranormal vision in the deaf Stephen G. Lomber{,{,*, M. Alex Meredith} and Andrej Kral} {
Department of Physiology and Pharmacology, Centre for Brain and Mind, The University of Western Ontario, London, Ontario, Canada { Department of Psychology, Centre for Brain and Mind, The University of Western Ontario, London, Ontario, Canada } Department of Anatomy and Neurobiology, School of Medicine, Virginia Commonwealth University, Richmond, Virginia, USA } Department of Experimental Otology, Institute of Audioneurotechnology, Medical University Hannover, Hannover, Germany
Abstract: This chapter is a summary of three interdigitated investigations to identify the neural substrate underlying supranormal vision in the congenitally deaf. In the first study, we tested both congenitally deaf and hearing cats on a battery of visual psychophysical tasks to identify those visual functions that are enhanced in the congenitally deaf. From this investigation, we found that congenitally deaf, compared to hearing, cats have superior visual localization in the peripheral field and lower visual movement detection thresholds. In the second study, we examined the role of “deaf” auditory cortex in mediating the supranormal visual abilities by reversibly deactivating specific cortical loci with cooling. We identified that in deaf cats, reversible deactivation of a region of cortex typically identified as the posterior auditory field (PAF) in hearing cats selectively eliminated superior visual localization abilities. It was also found that deactivation of the dorsal zone (DZ) of “auditory” cortex eliminated the superior visual motion detection abilities of deaf cats. In the third study, graded cooling was applied to deaf PAF and deaf DZ to examine the laminar contributions to the superior visual abilities of the deaf. Graded cooling of deaf PAF revealed that deactivation of the superficial layers alone does not cause significant visual localization deficits. Profound deficits were identified only when cooling extended through all six layers of deaf PAF. In contrast, graded cooling of deaf DZ showed that deactivation of only the superficial layers was required to elicit increased visual motion detection *Corresponding author. Tel.: þ1-519-663-5777x24110; Fax: þ1-519-663-3193 E-mail:
[email protected] DOI: 10.1016/B978-0-444-53752-2.00001-1
251
252
thresholds. Collectively, these three studies show that the superficial layers of deaf DZ mediate the enhanced visual motion detection of the deaf, while the full thickness of deaf PAF must be deactivated in order to eliminate the superior visual localization abilities of the congenitally deaf. Taken together, this combination of experimental approaches has demonstrated a causal link between the crossmodal reorganization of auditory cortex and enhanced visual abilities of the deaf, as well as identified the cortical regions responsible for adaptive supranormal vision. Keywords: reversible deactivation; posterior auditory field; dorsal zone; congenital deafness; cortical plasticity.
Introduction A remarkable feature of the brain is its ability to respond to change. Among other functions, this neuroplastic process endows a complex nervous system with the facility to adapt itself to its environment but, at the same time, also makes it susceptible to impoverished sensory or developmental experiences. For example, the expansion of somatosensory maps following limb amputation often results in spurious perceptual events known as “phantom limb pain” (e.g., Ramachandran and Hirstein, 1998) or untreated amblyopia results in the profound loss of visual acuity (reviewed by Webber and Wood, 2005). Neither of these neuroplastic effects have adaptive significance. However, there is a clear adaptive benefit when the inputs from another, intact modality substitute for those that have been lost (Collignon et al., 2009; Merabet and PascualLeone, 2010). Adaptive crossmodal plasticity can not only provide a form of partial compensation by one modality for another (e.g., auditory spatial localization in the blind) but also enhance perceptual performance within the remaining sensory modalities (but see Brozinsky and Bavelier, 2004; Finney and Dobkins, 2001). Numerous reports document improvement over intact subjects in auditory and somatosensory tasks in blind individuals (D'Anguilli and Waraich, 2002; Grant et al., 2000; Lewald, 2007; Sathian, 2000, 2005; Weeks et al., 2000), as well as enhanced performance in visual and tactile behaviors in
the deaf (Bavelier et al., 2000; Levanen and Hamdof, 2001). Although research has endeavored to identify the brain structures responsible for the behavioral enhancements resulting from adaptive crossmodal plasticity, it has been noted by many of these same studies that the specific neurological substrate for the effect is largely unknown (Doucet et al., 2006; Lambertz et al., 2005; Lee et al., 2003). Furthermore, the scant but growing literature on this topic seems to be fractionated into sides: one which asserts that crossmodal plasticity results in the wholesale reorganization of all of the affected regions, while the other indicates that crossmodal plasticity occurs only at selective regions therein (see review of Bavelier and Neville, 2002). Given that compensatory crossmodal plasticity appears not to affect brainstem structures (Langers et al., 2005, but see Shore et al., 2009), the suggestion that this phenomenon requires the cerebral cortex is supported by numerous studies (Rauschecker, 1995, 2002). Many of these investigations indicate that entire cortical representations vacated by the damaged sensory modality are completely replaced by inputs from the remaining systems (Bavelier and Neville, 2002). For example, imaging studies of crossmodal plasticity in early-deaf individuals have reported visual activation of auditory cortex partially including its core, or primary levels (Finney et al., 2001; Lambertz et al., 2005), and Braille reading or tactile tasks activated visual cortices in blind subjects
253
(Levanen and Hamdof, 2001; Sathian, 2000, 2005). Accordingly, these observations logically led to the general assumption that all cortical areas possess the ability for crossmodal plasticity. Indeed, the potential for such wholesale reorganization is supported by results from studies using a series of neonatal lesions in experimental animals (Roe et al., 1990; Sur et al., 1990). However, support for such global effects is not universal, and several studies (Nishimura et al., 1999; Weeks et al., 2000) specifically noted that primary auditory cortex was not crossmodally reorganized in their early-deaf subjects. Also, these observations favoring selective reorganization have been corroborated more directly by electrophysiological recordings from primary auditory cortices of congenitally deaf cats, which found no evidence of crossmodal plasticity (Kral et al., 2003). Therefore, while a clear and increasing effort has been directed toward investigating the neural bases for adaptive crossmodal plasticity, knowledge of the underlying brain circuitry remains virtually unexplored. A modest number of studies have been directed toward revealing behavioral/perceptual effects of crossmodal plasticity. The most notable of these efforts is the work of Rauschecker and colleagues, who used visual deprivation to examine the effect of crossmodal compensatory plasticity in cortex. These now classic studies revealed that, in cats visually deprived from birth, the extent of the auditory field of the anterior ectosylvian sulcus (FAES) was greatly expanded (Rauschecker and Korte, 1993), its constituent neurons were more sharply spatially tuned (Korte and Rauschecker, 1993), and the behavioral localization of auditory stimuli was enhanced (Rauschecker and Kniepert, 1994). However, this ground-breaking work has not been furthered since the original series of reports and few, if any, other investigators have incorporated this model of crossmodal plasticity in their studies. In contrast, several labs have produced a highly engineered model of crossmodal plasticity through a strategic series of neonatal lesions in
hamsters (Metin and Frost, 1989) and in ferrets (Pallas et al., 1999; Roe et al., 1990; Sur et al., 1990). However, such a model is as contrived as it is ingenious and, as such, it bears little semblance to naturally occurring neurological phenomena, such as blindness or deafness. Most profound examples of crossmodal plasticity result from loss of function in the peripheral sensory receptors or nerves, whereas central lesions that result in sensory loss generally are not available for reorganization because much of the affected area is essentially dead. However, a major effort has been directed toward understanding other forms of crossmodal effects, including plasticity (but not adaptive plasticity) involved in the visual calibration of auditory brainstem responses in barn owls (Gutfreund et al., 2002; Knudsen and Knudsen, 1989) and ferrets (King, 2002; King and Parsons, 1999). However, outside of these important efforts, the knowledge of cortical crossmodal reorganization is meager and a robust, repeatable, and more naturally occurring model of adaptive crossmodal plasticity has yet to be developed.
Congenitally deaf cat: a model for adaptive crossmodal plasticity Like the visual system, auditory development passes through a sensitive period in which circuits and connections are established and then refined by experience (Knudsen, 2004; Kral et al., 2000). During this period, the functional maturation of auditory processing and perception is critically dependent on adequate auditory experience. Cats appear to progress through a critical phase at 2–3 months old, and complete their auditory maturation by 6 months (Kral et al., 2005). A similar, but more prolonged sensitive period seems to apply to humans (up to 13 years; Doucet et al., 2006), as evidenced by congenitally deaf subjects who receive cochlear implants in early childhood and develop complete language competence. In contrast, those who do not receive such treatment
254
until later in life generally do not develop sophisticated language skills. The specific defects in the auditory system that underlie such persistent deficits remain to be identified. Some practitioners using imaging or EEG techniques have asserted that such deficits are the result of crossmodal plasticity that subsumes the nonfunctional parts of the auditory system into other sensory modes (Doucet et al., 2006; Finney et al., 2001; Lee et al., 2001). In contrast, studies done in congenitally deaf animals using single cell recording techniques have failed to show any crossmodal activation of primary auditory cortex (Kral et al., 2003) and that auditory nerve stimulation maintained access to primary auditory cortex even in congenitally deaf adults (Kral et al., 2002, 2005). Field A1 is functionally well characterized in congenitally deaf cats, with extensive deficits in spatiotemporal activity profiles as well as feature representation (Kral et al., 2009, Tillein et al., 2010) and corticocortical connectivity (reviewed in Kral and Eggermont, 2007). Chronic electrostimulation with a cochlear implant is known to show a sensitive period in cortical plasticity (reviewed in Kral et al., 2006). Thus, this model has been successful in demonstrating neurophysiological substrates of functional deficits after cochlear implantation. Ironically, despite the intense scrutiny that AI has received in these studies, with perhaps the exception of Sadato et al. (1996) in the visual cortex, virtually none of the crossmodally reorganized non-primary areas have been specifically identified. Although non-primary areas are “expected” to be reorganized, it is unclear whether these are similarly affected (and to the same degree). Therefore, the crucial debate in this regard is not only if deafness induces crossmodal plasticity, but where such plasticity occurs. To that end, we initiated a series of experiments to examine adaptive crossmodal plasticity in the congenitally deaf cat. The cat is an appealing model system to use for these types of investigations on cerebral networks in auditory
cortex. It is a simplified and tractable version of the more complex networks present in monkeys and humans. Cats are ideal because (1) they can quickly be trained to perform complex auditory tasks; (2) unlike the monkey, the majority of the auditory areas are easily approachable because they are exposed on the surfaces of gyri, rather than being buried in the depths of a sulcus; (3) each area is small enough so that it may be cooled by a single cryoloop (Lomber et al., 1999); and (4) they develop to maturity relatively quickly (over the course of months rather than years). Adult congenitally deaf cats show a Scheibe type of dysplasia in the organ of Corti with no hair cells present, although the spiral ganglion and cochlear bony structure are preserved (Heid et al., 1998). Preservation of the spiral ganglion cells is a major advantage when compared to pharmacologically deafened animals. The central auditory system of the congenitally deaf cat nonetheless shows expected deprivation-induced changes (Heid et al., 1998; Kral et al., 2006) although the central visual system appears normal in structure and function (Guillery et al., 1981; Levick et al., 1980). In the present study, deafness was confirmed by a standard screening method using auditory brainstem responses. In the first study, mature congenitally deaf cats and age-matched hearing cats were trained on a battery of seven visual psychophysical tests to identify those visual functions that are enhanced in the congenitally deaf. In the second study, we examined the role of “deaf” auditory cortex in mediating the superior visual abilities by reversibly deactivating specific cortical loci with cooling. This investigation revealed whether individual areas or collections of areas in deaf auditory cortex were the neural substrates for the superior visual functions. In the third study, graded cooling was applied to the areas identified in the second study to examine the laminar contributions to the superior visual abilities of the deaf. Overall, this combination of experimental approaches has demonstrated a causal link between the crossmodal reorganization of auditory cortex and enhanced
255
visual abilities of the deaf as well as identified the cortical regions responsible for supranormal vision.
(a)
Study 1: supranormal visual abilities of congenitally deaf cats In the first study, the performance of adult hearing (n ¼ 3) and congenitally deaf cats (n ¼ 3) was compared on a battery of seven visual psychophysical tasks. For specific details on the tasks, see Lomber et al. (2010). The cats’ ability to detect and localize flashed visual stimuli was assessed in a visual orienting arena (Fig. 1a) as we have done previously (Lomber and Payne, 2001; Malhotra et al., 2004). The six other tasks were conducted in a two-alternative forced-choice apparatus (Fig. 1b). To determine psychophysical thresholds, a standard staircase procedure was used, with three consecutive correct responses resulting in a decrease in the difference between the two stimuli, while each incorrect response resulted in an increase in the difference between the two comparison stimuli. Statistical significance was assessed using an analysis of variance and follow-up t-tests (p < 0.01). In the first task, we tested visual localization by placing the animals in an arena and examining their ability to accurately localize, by orienting and approaching, the illumination of red lightemitting diodes (LEDs) that were placed at 15 intervals across 180 of azimuth (Fig. 1a). In hearing controls, performance was excellent throughout the central 90 of the visual field (45 to the left and right), but accurate localization declined across the most peripheral targets tested (60–90 ; Fig. 2a). In contrast, visual localization performance of deaf cats was maintained at higher levels throughout the most peripheral visual field (Fig. 2a). Performance of the deaf cats was significantly better for the 60 , 75 , and 90 positions (p < 0.01), while there was no significant difference across the central 90 of the visual field (Fig. 2b). This result was consistent for both
speaker LED food reward
0°
A
30°
60°
90°
B
(b)
Fig. 1. (a) Orienting arena used for visual localization task. A loudspeaker (top circle) and a light-emitting diode (LED, black dot) were located above a food reward locus (lower circle) at each of 13 regularly spaced (15 ) intervals (for sake of clarity, only 30 intervals are labeled). (A) The animal was first required to fixate on the central (0 ) LED. (B) It then had to orient to, and approach, a secondary acoustic (100 ms broad-band noise) or visual (illumination of an LED) stimulus to receive a food reward. Adapted from Lomber et al. (2007). (b) Two-alternative forced-choice (2AFC) apparatus used for visual discrimination training and testing. The testing apparatus was a 52 29 41 cm Plexiglas box with a 14 cm diameter opening at one end. This opening lead to a smaller Plexiglas enclosure into which the animal placed its head. This chamber contained two hinged transparent response keys which the cat could depress with its nose to register a response. The stimuli could be viewed through the response keys. The monitors were located 28 cm from the cat's eyes (thus 1 cm on the screen was 2 visual degrees). Beneath the response keys was the food reward terminal that dispensed a puree of beef liver and ground pork when the animal made a correct response.
(a)
Visual localization
(b)
Visual localization Hearing 100
0° 30°
Deaf 30°
60°
60°
Percent correct
80
* 60
*
*
40 20
90° 100%
50%
0%
90° 100%
50%
Percent correct
0
0°
15°
30°
45°
60°
75°
90°
Eccentricity
Movement detection
(d)
1
*
0
(f)
(e)
10
Hearing
Threshold (cycles deg−1)
Threshold (deg s−1)
2
Grating acuity
Orientation discrimination
8 6 4 2 0
Deaf
(g)
Vernier acuity
2
Threshold (min arc)
(c)
Hearing
0
Deaf
Direction of motion
1
(h)
Hearing
Deaf
Velocity discrimination
5
4
4
3 2
Weber fraction (DV/V)
5
Threshold (degs)
Threshold (degs)
2
3 2
1
1
0
0
1 Hearing
0 Hearing
Deaf
Hearing
Deaf
Deaf 248
16
32
64
Stimulus velocity (deg s−1)
Fig. 2. Performance of hearing and deaf cats on the battery of seven visual psychophysical tasks. (a) Polar plot of the visual localization responses of hearing cats (light gray bars) and the superior performance of deaf cats (dark gray bars). The two concentric semicircles represent 50% and 100% correct response levels and the length of each colored line corresponds to the percentage of correct responses at each location tested. For both the hearing and deaf cats, data represent mean performance for 200 stimulus presentations at each peripheral target location and 400 stimulus presentations for the central target. (b) Histograms of combined data from left and right hemifields showing mean s.e. performance for the hearing (light gray) and deaf (dark gray) cats at each of the tested positions in the visual localization task. For both hearing and deaf cats, data represent mean performance for 400 stimulus presentations at each peripheral target location and 800 stimulus presentations for the central target (0 ). (c–g) Mean threshold s.e. for the hearing and deaf cats on the movement detection (c), grating acuity (d), Vernier acuity (e), orientation (f), and direction of motion (g), discriminations. (h) Performance of the hearing and deaf cats on the velocity discrimination task. Data are presented as Weber fractions for six different stimulus velocities. Asterisks indicate significant differences (p < 0.01) between the hearing and deaf conditions. Sample stimuli are shown for each task. Figure adapted from Lomber et al. (2010).
257
binocular and monocular testing. Overall, the superior visual localization abilities of deaf cats correspond well with findings from prelingually deaf human subjects (Bavelier et al., 2006). Six additional visual tests were all conducted in a two-alternative forced-choice apparatus using standard staircase procedures to determine psychophysical thresholds (Fig. 1b). In hearing cats, movement detection thresholds agreed with earlier reports (Pasternak and Merigan, 1980) and were identified to be 1.3 0.4 s 1 (Fig. 2c). In contrast, movement detection thresholds for the deaf cats were significantly lower (0.5 0.2 s 1; Fig. 2c). For the remaining five tests of visual function (grating acuity, Vernier acuity, orientation discrimination, direction of motion discrimination, and velocity discrimination), performance of the deaf cats was not significantly different from hearing controls (Fig. 2d–h). Overall, in the first study, we found that congenitally deaf, compared to hearing, cats have supranormal visual abilities, specifically, superior visual localization in the peripheral field and lower visual movement detection thresholds. Study 2: contributions of “deaf” auditory cortex to supranormal visual localization and detection In the second study, portions of auditory cortex (Fig. 3a) were collectively and individually deactivated to determine if specific cortical areas mediated the enhanced visual functions. In both the deaf and hearing cats, individual cooling loops (Lomber et al., 1999) were bilaterally placed over the posterior auditory field (PAF), the dorsal zone of auditory cortex (area DZ), and primary auditory field (A1) because of their involvement in auditory localization in hearing cats (Malhotra and Lomber, 2007; Malhotra et al., 2008; Fig. 3b). An additional control cooling loop was placed over the anterior auditory field (AAF) because of its involvement in pattern, but not spatial, processing (Lomber and Malhotra, 2008).
(a) DZ
ss
dPE AAF
A1 PAF
FAES
iPE
A2
aes
IN T
vPE
D
pes
VAF
VPAF
A
P V
(b) ss DZ
A1 AAF PAF 3 mm
aes
pes
Fig. 3. Cortical areas examined in deaf auditory cortex. (a) Illustration of the left hemisphere of the cat cerebrum (adapted from Reinoso-Suárez, 1961) showing all auditory areas (lateral view) compiled from Reale and Imig (1980), de Ribaupierre (1997), and Tian and Rauschecker (1998). For abbreviations, see List. Areas examined are highlighted in gray. The areal borders shown in this figure are based on a compilation of electrophysiological mapping and cytoarchitectonic studies. (b) Cooling loops in contact with areas AAF, DZ, A1, and PAF of the left hemisphere of a congenitally deaf cat at the time of implantation. Left is anterior. The areal borders presented in this figure are based on the postmortem analysis of SMI-32 processed tissue from the brain shown in this photo. For abbreviations, see List. Figure adapted from Lomber et al. (2010).
Reversible cooling deactivation The cooling method to reversibly deactivate neural tissue is an exciting, potent, and appropriate technique for examining cerebral contributions to behavior and has a number of highly beneficial
258
and practical features (Lomber, 1999). (1) Limited regions of the cerebral cortex can be selectively and reversibly deactivated in a controlled and reproducible way. Baseline and experimental measures can be made within minutes of each other (Lomber et al., 1996). (2) Repeated coolings over months or years produce stable, reversible deficits, with little evidence of attenuation or neural compensations (Lomber et al., 1994, 1999). (3) Repeated cooling induces neither local nor distant degenerations that might compromise conclusions (Yang et al., 2006). (4) Compared to traditional ablation studies, fewer animals are needed because within-animal-comparisons and double dissociations are possible, permitting large volumes of high-quality data to be acquired from each animal (Lomber and Malhotra, 2008; Lomber et al., 1996). (5) Finally, as the major effect of cooling is to block synaptic transmission, activity in fibers of passage is not compromised (Bénita and Condé, 1972; Jasper et al., 1970). Overall, the technique induces localized hypothermia in a restricted region of the brain. The locus of the deactivation is kept small by the constant perfusion of warm blood into, and around, the cooled region. The cooling disrupts calcium channel function in the presynaptic terminal and disrupts normal neurotransmitter release (reviewed by Brooks, 1983). We have verified that the surgical procedure to implant cryoloops, their presence in contact with the cerebrum, and their operation disrupts neither the normal structural nor functional integrity of cortex (Lomber et al., 1999; Yang et al., 2006). In every instance, cell and myelin stains are rich, and the cyto- and myelo-architecture of the region are characteristic of the region investigated, with no signs of pathology, as might be revealed by a marked pale staining of neurons or gliosis or light staining of cytochrome oxidase (Lomber and Payne, 1996). However, the lack of damage to the cortex means that it is not possible to use traditional histological techniques to determine the region that was deactivated. In the second study, cortical temperatures surrounding the
cooling loops were measured using multiple microthermocouples (150 mm in diameter; Omega Engineering, Stamford, CT) to determine the region of deactivation (Carrasco and Lomber, 2009). Across the cortical surface, 300–400 thermal measurements were taken from positions 500 mm below the pial surface. From these measurements, thermal cortical maps from cooling each individual cryoloop were constructed (Fig. 4). Depth of the cooling deactivation was also measured at four different coronal levels to provide an assessment of cooling spread in the Z-dimension. This information is provided in the third study.
Cortical loci investigated We used reversible cooling deactivation (Lomber et al., 1999) to examine the contributions of PAF, DZ, A1, and AAF to determine if specific cortical areas mediated the enhanced visual functions. The extent of the cooling deactivations (Fig. 4) was determined from direct cortical temperature recordings that were matched with adjacent sections processed for SMI-32 that permitted the delineation of the different areas of auditory cortex (Mellott et al., 2010) as we have done previously (Lomber and Malhotra, 2008). The positions of these four loci, as well as how they relate to the cortical maps of other investigators, are described below. Cooling loops were placed on PAF (Phillips and Orman, 1984; Reale and Imig, 1980), located caudal and ventral to A1. Loops were 6 mm long and extended from the anterior one-third of the dorsal-posterior ectosylvian gyrus to the fundus of the posterior ectosylvian sulcus (pes). A heat shielding compound was applied to the anterior side of the PAF loops to keep the cooling deactivations localized to the posterior bank of the pes. All deactivations extended down the posterior bank of the pes to the fundus and did not include the anterior bank. Therefore, the deactivated region included all of area PAF or area P (Fig. 4a; Imig et al., 1982; Phillips and Orman, 1984). For all DZ cooling loops, the dorsal
259
edge of the middle ectosylvian gyrus along the lip of the middle suprasylvian sulcus (mss) was deactivated (Fig. 4b). The region of deactivation included the dorsal-most portion of the lateral bank of the mss. However, the cooling did not appear to directly affect either the anterolateral (ALLS) or posterolateral (PLLS) lateral suprasylvian visual areas (Palmer et al., 1978). For each loop, the deactivated region included the totality of the regions previously described as the DZ (Middlebrooks and Zook, 1983) and the suprasylvian fringe (Beneyto et al., 1998; Niimi and Matsuoka, 1979; Paula-Barbosa et al., 1975; Rose, 1949; Woolsey, 1960). For all A1 cryoloops, the central region of the middle ectosylvian gyrus between the dorsal tips of the anterior and pes was deactivated (Fig. 4c). The deactivations were from stereotaxic coronal levels A1–A12. The deactivated region did not include the dorsal-most aspect of the middle ectosylvian gyrus, along the lateral lip of the mss (Fig. 4c). For each loop, the deactivated region included the ventral 2/3's of the classically defined area A1 (Reale and Imig, 1980). The AAF (Knight, 1977; Phillips and Irvine, 1982; Reale and Imig, 1980) cryoloops were 7 mm long and were located on the crown of the anterior suprasylvian gyrus between A10 and A17. All deactivations included the dorsal half of the lateral bank of the anterior suprasylvian sulcus and the dorsal half of the medial bank of the AES. Therefore, the deactivations included all of area AAF or area A (Fig. 4d), as defined by Knight (1977) and Reale and Imig (1980). Visual localization in the peripheral field
Fig. 4. Representative cooling deactivation reconstructions for the four cortical loci examined in the left hemisphere of a deaf cat. Black regions indicate deactivation extent as plotted from direct temperature measurements. The areal borders were determined by using SMI-32 staining criteria as we have done previously (Lomber and Malhotra, 2008). (a) Deactivation reconstruction showing a lateral (left is anterior) view of the left hemisphere with three horizontal sections in
For the visual localization task, the first step was to determine if auditory cortex could be mediating the enhanced visual performance of the deaf cats. the vicinity of the cooling locus. (b–d) Reconstructions showing a lateral (left is anterior) and dorsal (top is anterior) view of the left hemisphere with three coronal sections in the vicinity of the deactivation locus. For abbreviations, see List.
260
Therefore, we simultaneously deactivated all four areas (PAF, DZ, A1, and AAF) bilaterally, which resulted in a significant reduction in visual localization performance restricted to the most peripheral positions (60 , 75 , and 90 positions; Fig. 5a and b). Although the animals often failed to accurately or precisely localize the stimulus in the far periphery, they were not blind to the onset of the stimulus as the illumination of any LED always triggered a response. Therefore, the nature of the deficit was one of localization and not detection. Errors made during bilateral deactivation of all four areas were almost always undershoots of 30–60 (97.8% of all errors). Rarely (4.3% of all errors) were errors made to the incorrect hemifield. These results demonstrated that auditory cortex does have a role in mediating the enhanced visual localization performance of the congenitally deaf cats. In order to ascertain if the enhanced localization skills could be further localized to specific cortical loci, each of the four auditory areas was individually bilaterally deactivated. In the deaf cats, bilateral deactivation of PAF significantly reduced localization performance to the most peripheral targets (60 , 75 , and 90 positions, p < 0.01) while leaving localization performance for the 0 , 15 , 30 , and 45 targets unchanged (Fig. 5c). The reduction in visual localization at the most peripheral locations resulted in performance that was not different from deactivating all four areas simultaneously (Fig. 5b). Moreover, the localization performance of the deaf cats during bilateral cooling of PAF was not different from hearing cats (Fig. 5g). Neither bilateral nor unilateral deactivation of DZ, A1, or AAF modified visual localization performance (Fig. 5d–f). Unilateral deactivation of PAF resulted in reduced visual localization to the same peripheral positions; however, the deficit was specific to the contralateral hemifield (Lomber et al., 2010). Consequently, the neural basis for the enhanced visual localization skills of the deaf cats can be ascribed to PAF. This is an intriguing finding because, in hearing cats, PAF is normally involved in the accurate localization of acoustic stimuli (Fig. 6; Lomber
and Malhotra, 2008; Malhotra and Lomber, 2007). Bilateral deactivation of PAF in hearing cats results in profound acoustic localization deficits across the frontal field (Fig. 6). Therefore, the present results demonstrate that in deafness, PAF maintains a role in localization, albeit visual rather than acoustic. These results demonstrate that crossmodal plasticity can substitute one sensory modality for another while maintaining the functional repertoire of the reorganized region.
Visual motion detection For the supranormal visual motion detection abilities identified in the congenitally deaf cats, a similar experimental approach was taken to ascertain if “deaf” auditory cortex played a role in the enhanced motion detection. To determine if auditory cortex could be mediating the enhanced motion detection performance of deaf cats, we simultaneously deactivated all four areas (PAF, DZ, A1, and AAF). Bilateral deactivation of all four areas significantly increased motion discrimination thresholds from 0.44 0.19 to 1.39 0.35 s 1 (Fig. 7a). This finding established that auditory cortex does have a role in mediating the enhanced motion detection performance of the deaf cats. Next, to determine if a specific auditory region could be mediating the enhanced visual motion detection skills of deaf cats, areas PAF, DZ, A1, and AAF were individually bilaterally cooled. Bilateral deactivation of DZ significantly increased the motion detection thresholds from 0.40 0.15 to 1.46 0.4 s 1 (Fig. 9c). This increase resulted in performance that was not different from deactivating all four areas simultaneously (Fig. 7c). Moreover, the increase in threshold resulted in performance that was not different from performance of the hearing cats (Fig. 7f). There was no evidence of any functional lateralization, as unilateral deactivation of either left or right DZ did not alter performance (Lomber et al., 2010). Neither bilateral (Fig. 7b, d, and e) nor unilateral (Lomber et al.,
Fig. 5. Visual localization task data from deaf cats during bilateral reversible deactivation of PAF, DZ, A1, and AAF. (a) Polar plot of the visual localization responses of deaf cats while cortex was warm (dark gray) and active and during simultaneous cooling deactivation of PAF, DZ, A1, and AAF (black). (b–f) Histogram of combined data from the left and right hemifields showing mean s.e. performance for deaf cats while cortex was warm (dark gray) and active and while it was cooled (black) and deactivated. Asterisks indicate a significant difference (p < 0.01) between the warm and cool conditions. (b) Data from the simultaneous deactivation of PAF, DZ, A1, and AAF. (c–f) Data from individual area deactivations. (g) Visual localization data comparing performance at each position for hearing cats (light gray), deaf cats while PAF was warm (dark gray), and deaf cats while PAF was cooled (black). Asterisks indicate a significant difference (p < 0.01) from the hearing and deaf PAF cool conditions. Figure adapted from Lomber et al. (2010).
262
Fig. 6. Orienting responses to an acoustic stimulus during deactivation of PAF. Lateral view icons of the cat brain indicate the presence and position of a cryoloop (gray shading), and its operational status (black indicates loop was on and cortex was deactivated). For conventions, see Fig. 2. (a) Control data collected: (i) prior to PAF cryoloop implantation, (ii) after PAF cryoloop implantation and prior to cooling in each testing session, and (iii) shortly after termination of cooling. (b). Deactivation data collected: (iv) during cooling of left PAF, (v) during bilateral cooling of PAF, and (vi) during cooling of right PAF. Note that unilateral deactivation of PAF caused sound localization deficits in the contralateral field with no impairments in the ipsilateral hemifield. Bilateral deactivation of PAF resulted in bilateral sound localization deficits. Data summarized from seven animals. Figure adapted from Malhotra and Lomber (2007).
2010) deactivation of PAF, A1, or AAF resulted in any change in motion detection thresholds. These results demonstrate that DZ cortex mediates the superior visual motion detection thresholds of deaf cats. DZ has neuronal properties that are distinct
from A1 (He et al., 1997; Stecker et al., 2005) and is involved in sound source localization (Malhotra et al., 2008) and duration coding (Stecker et al., 2005). Here, we show DZs involvement in visual motion detection in deaf cats. A role for DZ in
263
Deactivation of auditory cortex in hearing cats does not alter visual function As we have demonstrated that “deaf” auditory cortex is the neural substrate for the enhanced visual abilities of the deaf, it was essential to also demonstrate that the auditory cortex of hearing cats does not contribute to visual function. Therefore, for the group of hearing cats, we both simultaneously and individually deactivated the four auditory areas on each of the seven visual tasks. Overall, neither simultaneous nor individual deactivation of the four auditory regions altered the ability of the hearing cats to perform any of the seven visual tasks (Lomber et al., 2010). These results demonstrate that in the presence of functional hearing, the auditory cortex does not contribute to any of the visual tasks examined. Therefore, deficits in visual function identified during bilateral deactivation of PAF or DZ in the deaf cats must be caused by underlying crossmodal adaptive plasticity in each area.
Study 3: laminar contributions to supranormal vision in the deaf Fig. 7. Motion detection thresholds for the deaf cats before and after cooling deactivation and during bilateral reversible deactivation. (a–e) Histograms showing mean s.e. motion detection thresholds for deaf cats while cortex was warm (dark gray) and active and while it was cooled (black) and deactivated. Asterisks indicate a significant difference (p < 0.01) between the warm and cool conditions. (a) Motion detection thresholds from deaf cats during bilateral reversible deactivation of PAF, DZ, A1, and AAF. (b–e) Data from individual area deactivations. (f) Motion detection thresholds to compare performance of hearing cats (light gray), deaf cats while DZ was warm (dark gray), and deaf cats while DZ was cooled (black). Asterisks indicate a significant difference (p < 0.01) from the hearing and deaf DZ cool conditions. Figure adapted from Lomber et al. (2010).
acoustic motion processing has yet to be investigated. Overall, in the second study, we were able to ascribe superior visual localization functions to PAF (Fig. 5g) and the superior motion detection abilities to DZ (Fig. 7f) in the same animals.
As we have demonstrated that individual areas of deaf auditory cortex contribute to supranormal visual localization in the periphery or visual motion detection, we next sought to determine if these functions could then be further localized in the laminar domain (Lomber and Payne, 2000; Lomber et al., 2007). Our approach was to apply lesser or greater levels of cooling to PAF or DZ to deactivate the cortical thickness in a graded, yet consistent, way, the more-superficial layers alone or in combination with the deep layers (Lomber and Payne, 2000; Lomber et al., 2007). With PAF cryoloop temperatures between 10 and 38 C, deaf cats are proficient at accurately reporting the location of a peripheral visual stimulus (Fig. 8a). Cooling to progressively lower temperatures (< 10 C) first initiated and then maximized an impairment in peripheral visual
264
localization, which was reduced to performance levels of hearing animals, at a cryoloop temperature of 3 1 C (Fig. 8a). Similarly, cooling of the DZ loop to progressively lower temperatures resulted in a rise in visual detection threshold (Fig. 8b). Visual motion detection threshold began to rise at cryoloop temperatures of 14 C and continued to rise to performance levels no different from hearing animals, until a temperature of 8 C were reached (Fig. 8b). However, the initiation temperature for the change in performance (14 C) and the temperature producing a maximal deficit (8 C) were both lower in all three deaf cats examined than the respective temperatures identified on the visual localization task for the same animals during PAF cooling. The different temperatures for initiation and maximum deficit for the two cortical areas can potentially be explained by changes in the laminar extent of cooling to disrupt visual localization in PAF rather than visual motion detection in DZ. As 20 C is the critical temperature below which neurons are silenced by blockade of synaptic transmission from afferent fibers (Bénita and Condé, 1972; Jasper et al., 1970; Lomber et al., 1999), we used arrays of microthermocouples to measure temperatures at more than 300 sites below each of the cryoloops (PAF and DZ) to ascertain the position of the 20 C thermocline. The positions of the temperature measurements were reconstructed using microlesions and depth measurements to determine the temperature profiles in the deaf cats from which the recordings were made. For each of the cooling loop locations (PAF and DZ), data were collected from each of the three deaf cats. A compilation of data from multiple tracks with a DZ cryoloop sequentially cooled to two different temperatures (8 C and 3 C) is presented in Fig. 9. Cortex between the 20 C thermocline and the cryoloop (gray field) has temperatures of < 20 C and is deactivated by the cooling, whereas locations more distal from the cryoloop than the 20 C thermocline have temperatures > 20 C and remain active (Fig. 10). Similar laminar deactivations were also determined for PAF (Fig. 11) cooling loops.
Fig. 8. Graphic representation of performance levels of deaf cats on the visual localization task (a) and the motion detection task (b) as a function of PAF or DZ cryoloop temperature, respectively. Each graph shows mean s.e. performance for blocks of trials collected at different cryoloop temperatures. (a) Black diamonds and lines represent mean performance of deaf cats performing the visual orienting task (mean performance across the three peripheral-most positions (60 , 75 , and 90 )) during bilateral cooling of PAF. (b) Black circles and lines represent mean performance of deaf cats performing visual motion detection task during bilateral cooling of DZ. Note that for the motion detection task (b) that thresholds begin to increase at cryoloop temperatures below 16 C and reaches a maximum deficit at 8 C. In contrast, visual localization performance (a) begins to fall at cryoloop temperatures below 10 C and reaches a maximum deficit at 2 C.
265 (a) 0 mss
8 ⬚C
Cryoloop
3 ⬚C
I 200
II
5 mm
mss 35 37 25 20 23 17 17 24 26
37
29
37
34
37 37
35
34
35
36
37
37
37
37
III
800 1000
IV
1200
V
38
VI
38 38
38
600
1400
37
37 38
Cryoloop = 8 ⬚C
32 33 34
23
34
35
Cortical Depth (mm)
400
1600
38
wm D
38
Fig. 10. Summary of deactivation depth when a cryoloop was cooled to 8 and 3 C. Temperatures <20 C, where synaptic transmission is eliminated, are indicated by the gray field. Temperature between 20 and 24 C, where synaptic activity is impaired, is indicated by light stipple. Adapted from Lomber et al. (2007).
L
M V
(b) mss 35 36 23 37
16 15 15 18
19
18
27
37
Cryoloop = 3 ⬚C
19
37 35
31
30
29
37 36
37 37
37
37
35 36
31
33
37 38
36
37
38
38 37
37
2 mm
Fig. 9. Temperature measurements recorded from identical sites in, and around, the dorsal zone (DZ) of auditory cortex when the cooling loop (circle with gray fill) was cooled to 8 C (a) and 3 C (b). Vertical line on the lateral view of the left cerebrum shows the position of the coronal section shown in (a) and (b). Gray region indicates the depth of cortex that was at, or below, 20 C as estimated from these measurements. For abbreviations, see List.
It is readily apparent from Figs. 9 and 11 that the effect of reducing cryoloop temperature from 8 to 3 C pushed the 20 C thermocline from the middle cortical layers to the gray/white matter interface.
Therefore, when the cooling loop was at 8 C, the resulting deactivation silenced the superficial layers (I–III) alone, and when the cooling loop was at 3 C, the resulting deactivation silenced the superficial and deep layers together. However, instead of the change in cortical deactivation depth explaining the behavioral results observed, it is possible that the change in lateral expansion of the deactivation could underlie the behavioral results. There was a slight lateral expansion in the extent of layers I–III that was deactivated as the cryoloop temperature was lowered from 8 to 3 C. For each of the two regions examined (PAF and DZ), in cross-sectional terms, estimates of lateral movement of the 20 C thermocline on the cortical surface as cooling loop temperature was lowered from 8 to 3 C show an increase in surface area of < 25%, while the depth of cortex deactivated by lowering the temperature of the cryoloop in this way was increased by > 140%. Thus, the major effect of the additional cooling was to push the 20 C thermocline across the deep layers of cortex with minor lateral surface movement. The most parsimonious interpretation of the differences in extents of deactivations is that
266
pes 5 mm
(a)
38 38 38 38 37 37 37 36
38 38 38 38 37 37 37 36 36 36 36
36 36
(b)
38 38 38 38 38 37 37 37 36 36
Cryoloop = 8 ⬚C
35 30 26 18 14 12 13 14 13 13 12
36 32 29 26 24 23 24 25 24 24 18 16 12
34 30 24 17 12 11 12 12 14 13 12 10
32 22 19 18 15 16 14 15 15 15 14 13 10
35 34 32 31 31 30 31 30 27 19 15
37 37 36 36 36 36 35 36 33 33 32 31
pes 38 38 38 38 37 37 37 37 36 36 36
Cryoloop = 3 ⬚C 36 33 30 27 33 23 22 22 21 19 18 17 15 12
32 30 30 28 26 24 23 23 22
M A
P L
2 mm Fig. 11. Temperature measurements recorded from identical sites in the posterior ectosylvian suclus (pes) when a PAF cooling loop was cooled to 8 C (a) and 3 C (b). Horizontal line on the lateral view of the left cerebrum (top right) shows the position of the horizontal section shown in (a) and (b). Gray region indicates the depth of cortex that was at, or below, 20 C as estimated from these measurements. Note that temperatures remain high in the anterior bank of the posterior ectosylvian sulcus due to the application of a heat shielding compound to the anterior surface of the cooling loop. For abbreviations, see List.
motion detection processing in deaf DZ is critically dependent upon the superficial cortical layers and that visual localization processing in deaf PAF is critically dependent upon the deep cortical layers. A critical component in acceptance of this
interpretation is the recognition that deep layer neurons remain active when upper layer neurons are silenced. Control physiological measures made in other cats verify deep layer activity in the absence of upper layer activity, and confirm the results of others in the visual system of intact cats that deep layer neurons remain active in the absence of activity in the superficial layers (Ferster et al., 1996; Schwark et al., 1986; Weyand et al., 1986, 1991). In the deaf cats, we observed deactivation of PAF eliminates supranormal visual localization abilities. We further observed that it is necessary to cool both the superficial and deep layers of PAF in order to completely eliminate the visual localization sensory enhancements. These results are interesting for two reasons. First, in hearing cats, PAF is normally involved in the accurate localization of acoustic stimuli (Fig. 6; Lomber and Malhotra, 2008; Malhotra and Lomber, 2007). This suggests that in deafness, PAF maintains a role in localization, albeit visual rather than acoustic. This is consistent with the hypothesis that the behavioral role of a crossmodally reorganized area is related to its role in hearing/ sighted individuals (Lomber et al., 2010; Meredith et al., 2011). Second, in hearing cats, in order to eliminate accurate acoustic localization, it is only necessary to deactivate the superficial layers of PAF (Lomber et al., 2007). Therefore, only the superficial layers of PAF need to be deactivated to disrupt acoustic localization in hearing animals, while both the superficial and deep layers of PAF must be deactivated in order to disrupt the supranormal visual localization abilities of congenitally deaf cats. Taken together, it will be interesting to examine possible differences in the input and output circuitry of the superficial and deep layers of PAF in congenitally deaf cats compared to hearing animals. Identification of the circuitry underlying crossmodal plasticity is essential toward providing a substrate on which the phenomenon can be studied and manipulated to reveal the fundamental principles governing its organization, function, and potential for therapeutic intervention.
267
Significance Collectively, these results provide new and comprehensive insight into the crossmodal effects induced by congenital deafness to a level that is essentially unobtainable through other methods. In addition, these observations form the basis for a robust and repeatable model of adaptive crossmodal plasticity that will be used to uncover the basic principles that characterize this phenomenon as well as better understand its relation to neuroplastic processes as a whole. By characterizing the regions of auditory cortex that are susceptible to crossmodal plasticity following deafness, we may be able to reveal the roles of intrinsic constraints and environmental input in determining cortical functional specificity. Such information will be critical for predicting and evaluating the success of sensory implants in humans (Kral and O'Donoghue, 2010; Rauschecker and Shannon, 2002; Zrenner, 2002). Specifically, crossmodal reorganization in deprived auditory cortex, like that identified in the present investigations, may hinder the ability of auditory cortex to process new auditory input provided by a cochlear implant (Bavelier and Neville, 2002; Kral and Eggermont, 2007). Studies suggest that deaf subjects, in whom crossmodal plasticity was the most extensive, were the least likely to benefit from cochlear prosthetics (Lee et al., 2001). Therefore, further investigations are necessary in order to more closely examine the link between crossmodal plasticity in deprived auditory cortex and the functional outcomes of cochlear prosthetics. Ultimately, future experiments could use this model of crossmodal plasticity to empirically assess potential windows for therapeutic interventions.
Acknowledgments We thank Amee McMillan for preparing all the figures and for help with the preparation of the chapter. We gratefully acknowledge the support of the Canadian Institutes of Health Research (CAN),
Deutsche Forschungsgemeinschaft (GER), and the National Institutes of Health (USA).
Abbreviations A AAF aes AI or A1 AII or A2 D dPE DZ FAES IN iPE L M mss P pes PAF ss T V VAF VPAF vPE
anterior anterior auditory field anterior ectosylvian sulcus primary auditory cortex second auditory cortex dorsal dorsal-posterior ectosylvian area dorsal zone of auditory cortex auditory field of the anterior ectosylvian sulcus insular region intermediate posterior ectosylvian area lateral medial middle suprasylvian sulcus posterior posterior ectosylvian sulcus posterior auditory field suprasylvian sulcus temporal region ventral ventral auditory field ventral posterior auditory field ventral posterior ectosylvian area
References Bavelier, D., Dye, M. W. G., & Hauser, P. C. (2006). Do deaf individuals see better? Trends in Cognitive Science, 10, 512–518. Bavelier, D., & Neville, H. (2002). Brain plasticity: Where and how? Nature Neuroscience, 3, 443–452. Bavelier, D., Tomann, A., Hutton, C., Mitchell, T., Corina, D., Liu, G., et al. (2000). Visual attention to the periphery is enhanced in congenitally deaf individuals. The Journal of Neuroscience, 20, 1–6. Beneyto, M., Winer, J. A., Larue, D. T., & Prieto, J. J. (1998). Auditory connections and neurochemistry of the sagulum. The Journal of Comparative Neurology, 401, 329–351.
268 Bénita, M., & Condé, H. (1972). Effects of local cooling upon conduction and synaptic transmission. Brain Research, 36, 133–151. Brooks, V. B. (1983). Study of brain function by local, reversible cooling. Reviews of Physiology Biochemistry and Pharmacology, 95, 1–109. Brozinsky, C. J., & Bavelier, D. (2004). Motion velocity thresholds in deaf signers: Changes in lateralization but not overall sensitivity. Cognitive Brain Research, 21, 1–10. Carrasco, A., & Lomber, S. G. (2009). Evidence for hierarchical processing in cat auditory cortex: Nonreciprocal influence of primary auditory cortex on the posterior auditory field. The Journal of Neuroscience, 29, 14323–14333. Collignon, O., Voss, P., Lassonde, M., & Lepore, F. (2009). Cross-modal plasticity for the spatial processing of sounds in visually deprived subjects. Experimental Brain Research, 192, 343–358. D'Anguilli, A., & Waraich, P. (2002). Enhanced tactile encoding and memory recognition in congenital blindness. International Journal of Rehabilitation Research, 25, 143–145. de Ribaupierre, F. (1997). Acoustical information processing in the auditory thalamus and cerebral cortex. In G. Ehret & R. Romand (Eds.), The central auditory system (pp. 317–397). New York: Oxford University Press. Doucet, M. E., Bergeron, F., Lassonde, M., Ferron, P., & Lepore, F. (2006). Cross-modal reorganization and speech perception in cochlear implant users. Brain, 129, 3376–3383. Ferster, D., Chung, S., & Wheat, H. (1996). Orientation selectivity of thalamic input to simple cells of cat visual cortex. Nature, 380, 249–252. Finney, E. M., & Dobkins, K. R. (2001). Visual contrast sensitivity in deaf versus hearing populations: Exploring the perceptual consequences of auditory deprivation and experience with a visual language. Cognitive Brain Research, 11, 171–183. Finney, E. M., Fine, I., & Dobkins, K. R. (2001). Visual stimuli activate auditory cortex in the deaf. Nature Neuroscience, 12, 1171–1173. Grant, A. C., Thiagarajah, M. C., & Sathian, K. (2000). Tactile perception in blind Braille readers: A psychophysical study of acuity and hyperacuity using gratings and dot patterns. Perception & Psychophysics, 62, 301–312. Guillery, R. W., Hickey, T. L., & Spear, P. D. (1981). Do blue-eyed white cats have normal or abnormal retinofugal pathways? Investigative Ophthalmology & Visual Science, 21, 27–33. Gutfreund, Y., Zheng, W., & Knudsen, E. I. (2002). Gated visual input to the central auditory system. Science, 297, 1556–1559. He, J., Hashikawa, T., Ojima, H., & Kinouchi, Y. (1997). Temporal integration and duration tuning in the dorsal zone of cat auditory cortex. The Journal of Neuroscience, 17, 2615–2625. Heid, S., Hartmann, R., & Klinke, R. (1998). A model for prelingual deafness, the congenitally deaf white cat–population statistics and degenerative changes. Hearing Research, 115, 101–112.
Imig, T. J., Reale, R. A., & Brugge, J. F. (1982). The auditory cortex: Patterns of corticocortical projections related to physiological maps in the cat. In C. N. Woolsey (Ed.), Cortical sensory organization: Vol. 3: Multiple auditory areas. New Jersey: Humana Press. Jasper, H., Shacter, D. G., & Montplaisir, J. (1970). The effect of local cooling upon spontaneous and evoked electrical activity of cerebral cortex. Canadian Journal of Physiology and Pharmacology, 48, 640–652. King, A. J. (2002). Neural plasticity: How the eye tells the brain about sound location. Current Biology, 12, R393–R395. King, A. J., & Parsons, C. H. (1999). Improved auditory spatial acuity in visually deprived ferrets. The European Journal of Neuroscience, 11, 3945–3956. Knight, P. L. (1977). Representation of the cochlea within the anterior auditory field (AAF) of the cat. Brain Research, 130, 447–467. Knudsen, E. I. (2004). Sensitive periods in the development of the brain and behavior. Journal of Cognitive Neuroscience, 16, 1412–1425. Knudsen, E. I., & Knudsen, P. F. (1989). Vision calibrates sound localization in developing barn owls. The Journal of Neuroscience, 9, 3306–3313. Korte, M., & Rauschecker, J. P. (1993). Auditory tuning of cortical neurons is sharpened in cats with early blindness. Journal of Neurophysiology, 70, 1717–1721. Kral, A., & Eggermont, J. J. (2007). What's to lose and what's to learn: Development under auditory deprivation, cochlear implants and limits of cortical plasticity. Brain Research Reviews, 56, 259–269. Kral, A., Hartmann, R., Tillein, J., Heid, S., & Klinke, R. (2000). Congenital auditory deprivation reduces synaptic activity within the auditory cortex in a layer-specific manner. Cerebral Cortex, 10, 714–726. Kral, A., Hartmann, R., Tillein, J., Heid, S., & Klinke, R. (2002). Hearing after congenital deafness: Central auditory plasticity and sensory deprivation. Cerebral Cortex, 12, 797–807. Kral, A., & O'Donoghue, G. M. (2010). Profound deafness in childhood. The New England Journal of Medicine, 363, 1438–1450. Kral, A., Schroder, J. H., Klinke, R., & Engel, A. K. (2003). Absence of cross-modal reorganization in the primary auditory cortex of congenitally deaf cats. Experimental Brain Research, 153, 605–613. Kral, A., Tillein, J., Heid, J., Hartmann, R., & Klinke, R. (2005). Postnatal cortical development in congenital auditory deprivation. Cerebral Cortex, 15, 552–562. Kral, A., Tillein, J., Heid, S., Klinke, R., & Hartmann, R. (2006). Cochlear implants: Cortical plasticity in congenital deprivation. Progress in Brain Research, 157, 283–313. Kral, A., Tillein, J., Hubka, P., Schiemann, D., Heid, S., Hartmann, R., et al. (2009). Spatiotemporal patterns of cortical activity with bilateral cochlear implants in congenital deafness. The Journal of Neuroscience, 29, 811–827.
269 Lambertz, N., Gizewski, E. R., Greiff, A., & Forsting, M. (2005). Cross-modal plasticity in deaf subjects dependent on extent of hearing loss. Cognitive Brain Research, 25, 884–890. Langers, D. R., van Dijk, P., & Backes, W. H. (2005). Lateralization, connectivity and plasticity in the human central auditory system. Neuroimage, 28, 490–499. Lee, D. S., Lee, J. S., Oh, S. H., Kim, S. K., Kim, J.-W., Chung, J. K., et al. (2001). Cross-modal plasticity and cochlear implants. Nature, 409, 149–150. Lee, J. S., Lee, D. S., Oh, S. H., Kim, C. S., Kim, J.-W., Hwang, Ch., et al. (2003). PET evidence of neuroplasticity in auditory auditory cortex of postlingual deafness. Journal of Nuclear Medicine, 44, 1435–1439. Levanen, S., & Hamdof, D. (2001). Feeling vibrations: Enhanced tactile sensitivity in congenitally deaf humans. Neuroscience Letters, 301, 75–77. Levick, W. R., Thibos, L. N., & Morstyn, R. (1980). Retinal ganglion cells and optic decussation of white cats. Vision Research, 20, 1001–1006. Lewald, J. (2007). More accurate sound localization induced by short-term light deprivation. Neuropsychologia, 5, 1215–1222. Lomber, S. G. (1999). The advantages and limitations of permanent or reversible deactivation techniques in the assessment of neural function. Journal of Neuroscience Methods, 86, 109–117. Lomber, S. G., Cornwell, P., Sun, J.-S., MacNeil, M. A., & Payne, B. R. (1994). Reversible inactivation of visual processing operations in middle suprasylvian cortex of the behaving cat. Proceedings of the National Academy of Sciences of the United States of America, 91, 2999–3003. Lomber, S. G., & Malhotra, S. (2008). Double dissociation of “what” and “where” processing in auditory cortex. Nature Neuroscience, 11, 609–616. Lomber, S. G., Malhotra, S., & Hall, A. J. (2007). Functional specialization in non-primary auditory cortex of the cat: Areal and laminar contributions to sound localization. Hearing Research, 229, 31–45. Lomber, S. G., Meredith, M. A., & Kral, A. (2010). Crossmodal plasticity in specific auditory cortices underlies visual compensations in the deaf. Nature Neuroscience, 13, 1421–1427. Lomber, S. G., & Payne, B. R. (1996). Removal of two halves restores the whole: Reversal of visual hemineglect during bilateral cortical or collicular inactivation in the cat. Visual Neuroscience, 13, 1143–1156. Lomber, S. G., & Payne, B. R. (2000). Translaminar differentiation of visually-guided behaviors revealed by restricted cerebral cooling deactivation. Cerebral Cortex, 10, 1066–1077. Lomber, S. G., & Payne, B. R. (2001). Task-specific reversal of visual hemineglect following bilateral reversible deactivation of posterior parietal cortex: A comparison with deactivation of the superior colliculus. Visual Neuroscience, 18, 487–499.
Lomber, S. G., Payne, B. R., Cornwell, P., & Long, K. D. (1996). Perceptual and cognitive visual functions of parietal and temporal cortices in the cat. Cerebral Cortex, 6, 673–695. Lomber, S. G., Payne, B. R., & Horel, J. A. (1999). The cryoloop: An adaptable reversible cooling deactivation method for behavioral or electrophysiological assessment of neural function. Journal of Neuroscience Methods, 86, 179–194. Malhotra, S., Hall, A. J., & Lomber, S. G. (2004). Cortical control of sound localization in the cat: Unilateral cooling deactivation of 19 cerebral areas. Journal of Neurophysiology, 92, 1625–1643. Malhotra, S., & Lomber, S. G. (2007). Sound localization during homotopic and heterotopic bilateral cooling deactivation of primary and non-primary auditory cortical areas in the cat. Journal of Neurophysiology, 97, 26–43. Malhotra, S., Stecker, G. C., Middlebrooks, J. C., & Lomber, S. G. (2008). Sound localization deficits during reversible deactivation of primary auditory cortex and/or the dorsal zone. Journal of Neurophysiology, 99, 1628–1642. Mellott, J. G., Van der Gucht, E., Lee, C. C., Carrasco, A., Winer, J. A., & Lomber, S. G. (2010). Areas of the cat auditory cortex as defined by neurofilament proteins expressing SMI-32. Hearing Research, 267, 119–136. Merabet, L. B., & Pascual-Leone, A. (2010). Neural reorganization following sensory loss: The opportunity of change. Nature Reviews. Neuroscience, 11, 44–52. Meredith, M. A., Kryklywy, J., McMillan, A. J., Malhotra, S., Lum-Tai, R., & Lomber, S. G. (2011). Crossmodal reorganization in the early-deaf switches sensory, but not behavioral roles of auditory cortex. Proceedings of the National Academy of Sciences of the United States of America, 108, 8856–8861. Metin, C., & Frost, D. O. (1989). Visual responses of neurons in somatosensory cortex of hamsters with experimentally induced retinal projections to somatosensory thalamus. Proceedings of the National Academy of Sciences of the United States of America, 86, 357–361. Middlebrooks, J. C., & Zook, J. M. (1983). Intrinsic organization of the cat's medial geniculate body identified by projections to binaural response-specific bands in the primary auditory cortex. The Journal of Neuroscience, 1, 203–224. Niimi, K., & Matsuoka, H. (1979). Thalamocortical organization of the auditory system in the cat studied by retrograde and axonal transport of horseradish peroxidase. Advances in Anatomy, Embryology and Cell Biology, 57, 1–56. Nishimura, H., Hashikawa, K., Doi, K., Ixaki, T., Watanabe, Y., Kusuoka, H., et al. (1999). Sign language ‘heard’ in the auditory cortex. Nature, 397, 16. Pallas, S. L., Littman, T., & Moore, D. R. (1999). Cross-modal reorganization of callosal connectivity without altering thalamocortical projections. Proceedings of the National Academy of Sciences of the United States of America, 96, 8751–8756.
270 Palmer, L. A., Rosenquist, A. C., & Tusa, R. J. (1978). The retinotopic organization of lateral suprasylvian visual areas in the cat. The Journal of Comparative Neurology, 177, 237–256. Pasternak, T., & Merigan, W. H. (1980). Movement detection by cats: Invariance with direction and target configuration. Journal of Comparative and Physiological Psychology, 94, 943–952. Paula-Barbosa, M. M., Feyo, P. B., & Sousa-Pinto, A. (1975). The association connexions of the suprasylvian fringe (SF) and other areas of the cat auditory cortex. Experimental Brain Research, 23, 535–554. Phillips, D. P., & Irvine, D. R. (1982). Properties of single neurons in the anterior auditory field (AAF) of cat cerebral cortex. Brain Research, 248, 237–244. Phillips, D. P., & Orman, S. S. (1984). Responses of single neurons in posterior field of cat auditory cortex to tonal stimulation. Journal of Neurophysiology, 51, 147–163. Ramachandran, V. S., & Hirstein, W. (1998). The perception of phantom limbs: The D.O. Hebb lecture. Brain, 9, 1603–1630. Rauschecker, J. P. (1995). Compensatory plasticity and sensory substitution in the cerebral cortex. Trends in Neuroscience, 18, 36–43. Rauschecker, J. P. (2002). Cortical map plasticity in animals and humans. Progress in Brain Research, 138, 73–88. Rauschecker, J. P., & Kniepert, U. (1994). Auditory localization behaviour in visually deprived cats. The European Journal of Neuroscience, 6, 149–160. Rauschecker, J. P., & Korte, M. (1993). Auditory compensation for early blindness in cat cerebral cortex. The Journal of Neuroscience, 13, 4538–4548. Rauschecker, J. P., & Shannon, R. V. (2002). Sensing sound to the brain. Science, 295, 1025–1029. Reale, R. A., & Imig, T. J. (1980). Tonotopic organization in auditory cortex of the cat. The Journal of Comparative Neurology, 192, 265–291. Reinoso-Suárez, F. (1961). Topographical atlas of the cat brain for experimental-physiological research [in German]. Merck, Darmstad: Federal Republic of Germany. Roe, A. W., Pallas, S. L., Hahm, J.-O., & Sur, M. (1990). A map of visual space induced in primary auditory cortex. Science, 250, 818–820. Rose, J. E. (1949). The cellular structure of the auditory region of the cat. The Journal of Comparative Neurology, 91, 409–440. Sadato, N., Pascual-Leone, A., Grafman, J., Ibanez, V., Deiber, M. P., Dold, G., et al. (1996). Activation of the primary visual cortex by Braille reading in blind subjects. Nature, 380, 526–528. Sathian, K. (2000). Practice makes perfect: Sharper tactile perception in the blind. Neurology, 54, 2203–2204.
Sathian, K. (2005). Visual cortical activity during tactile perception in the sighted and the visually deprived. Developmental Psychobiology, 46, 279–286. Schwark, H. D., Malpeli, J. G., Weyand, T. G., & Lee, C. (1986). Cat area 17. II. Response properties of infragranular layer neurons in the absence of supragranular layer activity. Journal of Neurophysiology, 56, 1074–1087. Shore, S. E., Koehler, S., Oldakowski, M., Hughes, L. F., & Syed, S. (2009). Dorsal cochlear nucleus responses to somatosensory stimulation are enhanced after noise-induced hearing loss. The European Journal of Neuroscience, 27, 155–168. Stecker, G. C., Harrington, I. A., Macpherson, E. A., & Middlebrooks, J. C. (2005). Spatial sensitivity in the dorsal zone (area DZ) of cat auditory cortex. Journal of Neurophysiology, 94, 1267–1280. Sur, M., Pallas, S. L., & Roe, A. W. (1990). Cross-modal plasticity in cortical development: Differentiation and specification of sensory neocortex. Trends in Neuroscience, 13, 227–233. Tian, B., & Rauschecker, J. P. (1998). Processing of frequencymodulated sounds in the cat's posterior auditory field. Journal of Neurophysiology, 79, 2629–2642. Tillein, J., Hubka, P., Syed, E., Hartmann, R., Engel, A. K., & Kral, A. (2010). Cortical represetnation of onteraural time difference in congenital deafness. Cerebral Cortex, 20, 492–506. Webber, A. L., & Wood, J. (2005). Amblyopia: Prevalence, natural history, functional effects, and treatment. Clinical & Experimental Optometry, 88, 365–375. Weeks, R., Horwitz, B., Aziz-Sultan, A., Tian, B., Wessinger, M., Cohen, L., et al. (2000). A positron emission tomographic study of auditory localization in the congenitally blind. The Journal of Neuroscience, 20, 2664–2672. Weyand, T. G., Malpeli, J. G., & Lee, C. (1991). Area 18 corticotectal cells: Response properties and identification of sustaining geniculate inputs. Journal of Neurophysiology, 65, 1078–1088. Weyand, T. G., Malpeli, J. G., Lee, C., & Schwark, H. D. (1986). Cat area 17. IV. Two types of corticotectal cells defined by controlling geniculate inputs. Journal of Neurophysiology, 56, 1102–1108. Woolsey, C. N. (1960). Organization of cortical auditory system: A review and synthesis. In G. L. Rasmussen & W. F. Windle (Eds.), Neural mechanisms of the auditory and vestibular systems (pp. 165–180). Springfield: C.C. Thomas. Yang, X. F., Kennedy, B. R., Lomber, S. G., Schmidt, R. E., & Rothman, S. M. (2006). Cooling produces minimal neuropathology in neocortex and hippocampus. Neurobiology of Disease, 23, 637–643. Zrenner, E. (2002). Will retinal implants restore vision? Science, 295, 1022–1025.
Subject Index
Adaptation, brain plasticity definition, 177–178 exaptation, 180–183 experience-dependent plasticity, 179 Adaptive crossmodal plasticity behavioral/perceptual effects, 253 congenitally deaf cat cortical loci, 258–259 deactivation, auditory cortex, 263 laminar contributions, 263–266 reversible cooling deactivation, 257–258 supranormal visual abilities, 255–257 visual localization, peripheral field, 259–260 visual motion detection, 260–263 phantom limb pain, 252 significance, 267
occipital activity, 237 right dorsal extrastriate occipital cortex (rOC), 238–239 right intraparietal sulcus (rIPS), 238–239 transcranial magnetic stimulation (TMS), 236 sensory rehabilitation auditory discrimination tasks, 214 neuroprostheses, 219–220 prosthesis substituting vision by audition (PSVA), 214 sensory substitution, 217–219 sight restoration, 215–217 visual deprivation, 213 Brain plasticity adaptation definition, 177–178 exaptation, 180–183 experience-dependent plasticity, 179 environmental manipulations, 191 evolutionary process, 177 maladaptation cochlear implant, 189–190 congenitally deprived sensory modalities comparison, 186 cortical reorganization rewiring, 188–189 phantom limb pain, 185–186 tinnitus, 186–188 natural selection, 178 structural and functional change, 178 Brain reorganization blindness correlational analysis, 238 occipital activity, 237
Behavioral reorganization blindness external and somatotopic spatial codes, 235 peripheral visual field, 235 spatial processing, 234 tactile modality, 236 chemosensory loss, 241–243 deafness, 239–240 Bilateral deactivation, 260 Blindness behavioral reorganization external and somatotopic spatial codes, 235 peripheral visual field, 235 spatial processing, 234 tactile modality, 236 brain reorganization correlational analysis, 238 271
272
Brain reorganization (Continued) right dorsal extrastriate occipital cortex (rOC), 238–239 right intraparietal sulcus (rIPS), 238–239 transcranial magnetic stimulation (TMS), 236 chemosensory loss, 243–244 deafness, 240–241
Deafness crossmodal plasticity behavioral reorganization, 239–240 brain reorganization, 240–241 sensory rehabilitation cochlear implant (CI), 221–224 sensory substitution, 220–221
Chemosensory loss behavioral reorganization, 241–243 brain reorganization, 243–244 Cochlear implant (CI) audiovisual interaction, 224 auditory input, 221 influential factors, 222 maladaptation, 189–190 multisensory interactions, 223 occipito-temporal networks, 221 Congenitally deaf cat, crossmodal plasticity cortical loci, 258–259 deactivation, auditory cortex, 263 laminar contributions accurate acoustic localization, 266 deactivations, 264 PAF cryoloop temperatures, 263–264 performance levels, 264 reversible cooling deactivation, 257–258 supranormal visual abilities, 255–257 visual localization, peripheral field, 259–260 visual motion detection bilateral deactivation, 260 motion detection thresholds, 263 orienting responses, 262 Crossmodal plasticity, sensory loss. See also Adaptive crossmodal plasticity blindness behavioral reorganization, 234–236 brain reorganization, 236–239 chemosensory loss behavioral reorganization, 241–243 brain reorganization, 243–244 deafness behavioral reorganization, 239–240 brain reorganization, 240–241
Euclidean space definition, 48 distance, 49 metric properties, 50 Haptic discrimination task, 20 Human–machine interfaces (HMIs) clinical perspective average normalized movement errors, 59 distribution, motor variance, 61–62 low-dimensional geometrical structure, 58 movement trajectories, 61 principal component analysis (PCA), 60 2D task-spaces, 63 dual-learning problem, 55–58 features, 63 intrinsic geometry, central nervous system, 49 inverse geometrical model complementary subspaces, 50 glove-signal space, 51 hand-to-cursor mapping experiment, 53 minimum Euclidean norm, 52 task-space, 55 metric properties, Euclidean space, 50 motor learning, 46–47 ordinary space electromyographic (EMG) signals, 47 Euclidean properties, 48–49 Human sensorimotor control. See Sensorimotor control Inverse geometrical model, HMIs complementary subspaces, 50 glove-signal space, 51
273
hand-to-cursor mapping experiment, 53 minimum Euclidean norm, 52 task-space, 55 Locomotor adaptation adaptation and deadaptation rates, 68–69 advantages, 74 anatomical circuits, 69 baseline behavior, 72 conscious correction and distraction effects, 69 errors, 71 individual subject data, 71 late childhood adaptive abilities, 67–68 learning paradigm, 66–67 pattern transfer, 69–70 proprioceptive signals, 73 spatial and temporal strategies, 66 split-belt treadmill paradigm, 68 temporal control, 71 walking rehabilitation, 65–66 Maladaptation cochlear implant, 189–190 congenitally deprived sensory modalities comparison, 186 cortical reorganization rewiring, 188–189 phantom limb pain, 185–186 tinnitus, 186–188 Motion detection thresholds, 263 Motor adaptation and proprioceptive recalibration comparison, 97–98 methodology, 92–94 motor learning conditions, 95–97 visual feedback, 92 visuomotor adaptation, 94–95 Motor learning functional connections, 32 hypothesis, 31 limb movement study, 40–41 plasticity, 31–32 somatosensory perception, upper limb force field learning, 32–33 mean movement curvature, 36
parameter estimation by sequential testing (PEST), 32 perceptual boundary, 34–35 speech sounds experiment, 37 histogram, 40 perceptual classification, 40–41 perceptual psychometric functions, 38–39 sagittal plane view, 38 statistical tests, 40 Multisensory integration adulthood, 158–161 changes experienced, 153–158 development, 150–153 mature superior colliculus, 146–147 objectives, 146 principles, 147–148 SC model, 148–150 semantic issues, 147 senses, 145 simple heuristics, 148 underlying computation, 147 Multisensory object recognition multisensory cortical processing, 170–171 size-dependence, 168–169 structural and surface properties integration, 169–170 view-dependence, 166–168 visual imagery, 171–172 visuo-haptic model, 172–173 Naturalistic approaches, sensorimotor control animal psychology, 4–5 arm movements, 13–15 ethology, object manipulation, 15–16 eye movements, 8–10 hand movements, 11–13 haptic discrimination task, 20 human behavior, 7–8 human cognitive ethology, 5–6 object manipulation task, 20–22 physical objects, familiar dynamics, 16–17 simulated objects, 17–20 visual system, 6–7 Neuroplasticity, 211–212. See also Brain plasticity
274
Parameter estimation by sequential testing (PEST), 32 Perception and action, singing absolute and relative accuracy, 105 deficits, 109–111 feedback deficits, 113–114 limitations, 106 motor control deficits, 112–113 neural bases, 114–115 non-Western societies, 104 pitch errors, 104 poor singing, 106–109 sensorimotor translation deficits, 111–112 tuning, 106 types, 105 Phantom limb pain, 252 Proprioceptive recalibration. See Motor adaptation and proprioceptive recalibration Rat auditory cortex animal model, 120–121 cortical sensitive periods succession A1 frequency tuning, 122 critical periods, 123 progressive crystallization, 124 subcortical processing center, 121 history, 120 local regulation, CP plasticity, 124–125 perceptual and neurological specialization, 120 stimulus selectivity age-related changes, 128 moderate-level noise exposure, 127 plastic rewiring, 126 Reversible cooling deactivation, 257–258 Right dorsal extrastriate occipital cortex (rOC), 238–239 Right intraparietal sulcus (rIPS), 238–239 Sensorimotor control definition, 3 naturalistic approaches animal psychology, 4–5 arm movements, 13–15 ethology, object manipulation, 15–16 eye movements, 8–10
hand movements, 11–13 haptic discrimination task, 20 human behavior, 7–8 human cognitive ethology, 5–6 object manipulation task, 20–22 physical objects, familiar dynamics, 16–17 simulated objects, 17–20 visual system, 6–7 transformations, 3–4 Sensory integration cortical circuits neural reference frames, 201 recording locations, 202 shift analysis, 203 volitional arm movements, 201 local to global optimality complex sensorimotor circuit, 203 parietal representations, 205 pure reference-frame representations, 204 variability, 206 neural populations modeling, 199–200 optimal integration, 196 psychophysics modeling, 196 reach behavior, 196–199 Sensory motor remapping. See Human–machine interfaces (HMIs) Sensory rehabilitation blindness auditory discrimination tasks, 214 neuroprostheses, 219–220 prosthesis substituting vision by audition (PSVA), 214 sensory substitution, 217–219 sight restoration, 215–217 visual deprivation, 213 deafness cochlear implant (CI), 221–224 sensory substitution, 220–221 neuroplasticity, 211–212 Singing, perception and action absolute and relative accuracy, 105 deficits, 109–111 feedback deficits, 113–114 limitations, 106 motor control deficits, 112–113
275
neural bases, 114–115 non-Western societies, 104 pitch errors, 104 poor singing, 106–109 sensorimotor translation deficits, 111–112 tuning, 106 types, 105 Sleep, cognitive function aging neurodegenerative diseases, 76 healthy aging, changes circadian regulation, 79 cognition, 80–81 cross-sectional and longitudinal studies, 81–82 homeostatic regulation, 79–80 sleep architecture, 77–78 sleep-dependent consolidation studies, 82–85 sleep deprivation and restriction studies, 81 sleep-related neuroendocrine, 80 total sleep time (TST), 76 Somatosensory perception, motor learning force field learning, 32–33 mean movement curvature, 36 parameter estimation by sequential testing (PEST), 32 perceptual boundary, 34–35 Speech sounds, motor learning experiment, 37
histogram, 40 perceptual classification, 40–41 perceptual psychometric functions, 38–39 sagittal plane view, 38 statistical tests, 40 Tinnitus abnormal activity, 187 animal and human studies, 188 causes, 187 frequency, 187 objective/subjective, 186 phantom sounds, 187 Transcranial magnetic stimulation (TMS), 236 Visual orientation cues, gravity direction, 134–135 enhancement cognitive demands, 137 compression, 138 motion, 138 polarized cues, 138 gravity direction estimation, 136 perceived orientation, 136–137 perceptual tasks, 135 Walking adaptation. See Locomotor adaptation
Other volumes in PROGRESS IN BRAIN RESEARCH Volume 149: Cortical Function: A View from the Thalamus, by V.A. Casagrande, R.W. Guillery and S.M. Sherman (Eds.) – 2005 ISBN 0-444-51679-4. Volume 150: The Boundaries of Consciousness: Neurobiology and Neuropathology, by Steven Laureys (Ed.) – 2005, ISBN 0-444-51851-7. Volume 151: Neuroanatomy of the Oculomotor System, by J.A. Büttner-Ennever (Ed.) – 2006, ISBN 0-444-51696-4. Volume 152: Autonomic Dysfunction after Spinal Cord Injury, by L.C. Weaver and C. Polosa (Eds.) – 2006, ISBN 0-444-51925-4. Volume 153: Hypothalamic Integration of Energy Metabolism, by A. Kalsbeek, E. Fliers, M.A. Hofman, D.F. Swaab, E.J.W. Van Someren and R.M. Buijs (Eds.) – 2006, ISBN 978-0-444-52261-0. Volume 154: Visual Perception, Part 1, Fundamentals of Vision: Low and Mid-Level Processes in Perception, by S. Martinez-Conde, S.L. Macknik, L.M. Martinez, J.M. Alonso and P.U. Tse (Eds.) – 2006, ISBN 978-0-444-52966-4. Volume 155: Visual Perception, Part 2, Fundamentals of Awareness, Multi-Sensory Integration and High-Order Perception, by S. Martinez-Conde, S.L. Macknik, L.M. Martinez, J.M. Alonso and P.U. Tse (Eds.) – 2006, ISBN 978-0-444-51927-6. Volume 156: Understanding Emotions, by S. Anders, G. Ende, M. Junghofer, J. Kissler and D. Wildgruber (Eds.) – 2006, ISBN 978-0-444-52182-8. Volume 157: Reprogramming of the Brain, by A.R. M ller (Ed.) – 2006, ISBN 978-0-444-51602-2. Volume 158: Functional Genomics and Proteomics in the Clinical Neurosciences, by S.E. Hemby and S. Bahn (Eds.) – 2006, ISBN 978-0-444-51853-8. Volume 159: Event-Related Dynamics of Brain Oscillations, by C. Neuper and W. Klimesch (Eds.) – 2006, ISBN 978-0-444-52183-5. Volume 160: GABA and the Basal Ganglia: From Molecules to Systems, by J.M. Tepper, E.D. Abercrombie and J.P. Bolam (Eds.) – 2007, ISBN 978-0-444-52184-2. Volume 161: Neurotrauma: New Insights into Pathology and Treatment, by J.T. Weber and A.I.R. Maas (Eds.) – 2007, ISBN 978-0-444-53017-2. Volume 162: Neurobiology of Hyperthermia, by H.S. Sharma (Ed.) – 2007, ISBN 978-0-444-51926-9. Volume 163: The Dentate Gyrus: A Comprehensive Guide to Structure, Function, and Clinical Implications, by H.E. Scharfman (Ed.) – 2007, ISBN 978-0-444-53015-8. Volume 164: From Action to Cognition, by C. von Hofsten and K. Rosander (Eds.) – 2007, ISBN 978-0-444-53016-5. Volume 165: Computational Neuroscience: Theoretical Insights into Brain Function, by P. Cisek, T. Drew and J.F. Kalaska (Eds.) – 2007, ISBN 978-0-444-52823-0. Volume 166: Tinnitus: Pathophysiology and Treatment, by B. Langguth, G. Hajak, T. Kleinjung, A. Cacace and A.R. M ller (Eds.) – 2007, ISBN 978-0-444-53167-4. Volume 167: Stress Hormones and Post Traumatic Stress Disorder: Basic Studies and Clinical Perspectives, by E.R. de Kloet, M.S. Oitzl and E. Vermetten (Eds.) – 2008, ISBN 978-0-444-53140-7. Volume 168: Models of Brain and Mind: Physical, Computational and Psychological Approaches, by R. Banerjee and B.K. Chakrabarti (Eds.) – 2008, ISBN 978-0-444-53050-9. Volume 169: Essence of Memory, by W.S. Sossin, J.-C. Lacaille, V.F. Castellucci and S. Belleville (Eds.) – 2008, ISBN 978-0-444-53164-3. Volume 170: Advances in Vasopressin and Oxytocin – From Genes to Behaviour to Disease, by I.D. Neumann and R. Landgraf (Eds.) – 2008, ISBN 978-0-444-53201-5. Volume 171: Using Eye Movements as an Experimental Probe of Brain Function—A Symposium in Honor of Jean BüttnerEnnever, by Christopher Kennard and R. John Leigh (Eds.) – 2008, ISBN 978-0-444-53163-6. Volume 172: Serotonin–Dopamine Interaction: Experimental Evidence and Therapeutic Relevance, by Giuseppe Di Giovanni, Vincenzo Di Matteo and Ennio Esposito (Eds.) – 2008, ISBN 978-0-444-53235-0. Volume 173: Glaucoma: An Open Window to Neurodegeneration and Neuroprotection, by Carlo Nucci, Neville N. Osborne, Giacinto Bagetta and Luciano Cerulli (Eds.) – 2008, ISBN 978-0-444-53256-5. Volume 174: Mind and Motion: The Bidirectional Link Between Thought and Action, by Markus Raab, Joseph G. Johnson and Hauke R. Heekeren (Eds.) – 2009, 978-0-444-53356-2. Volume 175: Neurotherapy: Progress in Restorative Neuroscience and Neurology — Proceedings of the 25th International Summer School of Brain Research, held at the Royal Netherlands Academy of Arts and Sciences, Amsterdam, The Netherlands, August 25–28, 2008, by J. Verhaagen, E.M. Hol, I. Huitinga, J. Wijnholds, A.A. Bergen, G.J. Boer and D.F. Swaab (Eds.) –2009, ISBN 978-0-12-374511-8. Volume 176: Attention, by Narayanan Srinivasan (Ed.) – 2009, ISBN 978-0-444-53426-2. Volume 177: Coma Science: Clinical and Ethical Implications, by Steven Laureys, Nicholas D. Schiff and Adrian M. Owen (Eds.) – 2009, 978-0-444-53432-3. Volume 178: Cultural Neuroscience: Cultural Influences On Brain Function, by Joan Y. Chiao (Ed.) – 2009, 978-0-444-53361-6. Volume 179: Genetic models of schizophrenia, by Akira Sawa (Ed.) – 2009, 978-0-444-53430-9. Volume 180: Nanoneuroscience and Nanoneuropharmacology, by Hari Shanker Sharma (Ed.) – 2009, 978-0-444-53431-6.
278
Other volumes in PROGRESS IN BRAIN RESEARCH
Volume 181: Neuroendocrinology: The Normal Neuroendocrine System, by Luciano Martini, George P. Chrousos, Fernand Labrie, Karel Pacak and Donald W. Pfaff (Eds.) – 2010, 978-0-444-53617-4. Volume 182: Neuroendocrinology: Pathological Situations and Diseases, by Luciano Martini, George P. Chrousos, Fernand Labrie, Karel Pacak and Donald W. Pfaff (Eds.) – 2010, 978-0-444-53616-7. Volume 183: Recent Advances in Parkinson's Disease: Basic Research, by Anders Björklund and M. Angela Cenci (Eds.) – 2010, 978-0-444-53614-3. Volume 184: Recent Advances in Parkinson's Disease: Translational and Clinical Research, by Anders Björklund and M. Angela Cenci (Eds.) – 2010, 978-0-444-53750-8. Volume 185: Human Sleep and Cognition, by Gerard A. Kerkhof and Hans P.A. Van Dongen (Eds.) – 2010, 978-0-444-53702-7. Volume 186: Sex Differences in the Human Brain, their Underpinnings and Implications, by Ivanka Savic (Ed.) – 2010, 978-0-44453630-3. Volume 187: Breathe, Walk and Chew: The Neural Challenge: Part I, by Jean-Pierre Gossard, Réjean Dubuc and Arlette Kolta (Eds.) – 2010, 978-0-444-53613-6. Volume 188: Breathe, Walk and Chew; The Neural Challenge: Part II, by Jean-Pierre Gossard, Réjean Dubuc and Arlette Kolta (Eds.) – 2011, 978-0-444-53825-3. Volume 189: Gene Expression to Neurobiology and Behaviour: Human Brain Development and Developmental Disorders by Oliver Braddick, Janette Atkinson and Giorgio M. Innocenti (Eds.) – 2011, 978-0-444-53884-0. Volume 190: Human Sleep and Cognition Part II: Clinical and Applied Research, by Hans P.A. Van Dongen and Gerard A. Kerkhof (Eds.) – 2011, 978-0-444-53817-8.
