International REVIEW OF
Neurobiology Volume 86 SERIES EDITORS RONALD J. BRADLEY Department of Psychiatry, College of Medicine The University of Tennessee Health Science Center Memphis, Tennessee, USA
R. ADRON HARRIS Waggoner Center for Alcohol and Drug Addiction Research The University of Texas at Austin Austin, Texas, USA
PETER JENNER Division of Pharmacology and Therapeutics GKT School of Biomedical Sciences King’s College, London, UK EDITORIAL BOARD ERIC AAMODT PHILIPPE ASCHER DONARD S. DWYER MARTIN GIURFA PAUL GREENGARD NOBU HATTORI DARCY KELLEY BEAU LOTTO MICAELA MORELLI JUDITH PRATT EVAN SNYDER JOHN WADDINGTON
HUDA AKIL MATTHEW J. DURING DAVID FINK MICHAEL F. GLABUS BARRY HALLIWELL JON KAAS LEAH KRUBITZER KEVIN MCNAUGHT JOSE´ A. OBESO CATHY J. PRICE SOLOMON H. SNYDER STEPHEN G. WAXMAN
Academic Press is an imprint of Elsevier 360 Park Avenue South, New York, NY 10010-1700 525 B Street, Suite 1900, San Diego, California 92101-4495, USA 32 Jamestown Road, London NW1 7BY, UK
This book is printed on acid-free paper.
Copyright ß 2009, Elsevier Inc. All Rights Reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without permission in writing from the Publisher. The appearance of the code at the bottom of the first page of a chapter in this book indicates the Publisher’s consent that copies of the chapter may be made for personal or internal use of specific clients. This consent is given on the condition, however, that the copier pay the stated per copy fee through the Copyright Clearance Center, Inc. (www.copyright.com), for copying beyond that permitted by Sections 107 or 108 of the U.S. Copyright Law. This consent does not extend to other kinds of copying, such as copying for general distribution, for advertising or promotional purposes, for creating new collective works, or for resale. Copy fees for pre-2007 chapters are as shown on the title pages. If no fee code appears on the title page, the copy fee is the same as for current chapters. 0074-7742/2007 $35.00 Permissions may be sought directly from Elsevier’s Science & Technology Rights Department in Oxford, UK: phone: (þ44) 1865 843830, fax: (þ44) 1865 853333, E-mail:
[email protected]. You may also complete your request on-line via the Elsevier homepage (http://elsevier.com), by selecting ‘‘Support & Contact’’ then ‘‘Copyright and Permission’’ and then ‘‘Obtaining Permissions.’’ For information on all Academic Press publications visit our Web site at www.elsevierdirect.com ISBN-13: 978-0-12-374821-8 PRINTED AND BOUND IN THE UNITED STATES OF AMERICA 08 09 10 11 12 8 7 6 5 4 3 2 1
CONTRIBUTORS
Numbers in parentheses indicate the pages on which the authors’ contributions begin.
Dino Accoto (39), Biomedical Robotics and Biomicrosystems Laboratory, Universita` Campus Biomedico di Roma, Roma, Italy F. Aloise (133), IRCCS Fondazione Santa Lucia, Rome, Italy L. Astolfi (133), Dip. Fisiologia e Farmacologia, Univ. La Sapienza, Rome, Italy; and IRCCS Fondazione Santa Lucia, Rome, Italy Claudio Babiloni (67), Department of Biomedical Sciences, University of Foggia, Foggia, Italy; and Hospital San Raffaele Cassino, Cassino, Italy F. Babiloni (133), Dip. Fisiologia e Farmacologia, Univ. La Sapienza, Rome, Italy; and IRCCS Fondazione Santa Lucia, Rome, Italy A. Bengoetxea (171), Laboratory of Neurophysiology and Biomechanics of Movementa, Universite´ Libre de Bruxelles, CP 168, 50 Av. F. Roosevelt, Brussels, Belgium Antonella Benvenuto (39), Biomedical Robotics and Biomicrosystems Laboratory, Universita` Campus Biomedico di Roma, Roma, Italy Alain Berthoz (159), Laboratoire de Physiologie de la Perception et de l’Action, UMR 7152, Colle`ge de France, CNRS, Paris, France Niels Birbaumer (107), Ospedale San Camillo—IRCCS, Istituto di Ricovero e Cura a Carattere Scientifico, Venezia, Lido, Italy; and Institute of Medical Psychology and Behavioral Neurobiology, University of Tuebingen, Tuebingen, Germany Domenico Campolo (39), Biomedical Robotics and Biomicrosystems Laboratory, Universita` Campus Biomedico di Roma, Roma, Italy Federico Carpi (3), University of Pisa, Interdepartmental Research Centre ‘‘E. Piaggio’’, School of Engineering, 56100 Pisa, Italy A. M. Cebolla (171), Laboratory of Neurophysiology and Biomechanics of Movementa, Universite´ Libre de Bruxelles, CP 168, 50 Av. F. Roosevelt, Brussels, Belgium G. Cheron (171), Laboratory of Electrophysiology, Universite´ de Mons-Hainaut, Mons, Belgium; and Laboratory of Neurophysiology and Biomechanics of Movementa, Universite´ Libre de Bruxelles, CP 168, 50 Av. F. Roosevelt, Brussels, Belgium F. Cincotti (133), IRCCS Fondazione Santa Lucia, Rome, Italy
xi
xii
CONTRIBUTORS
Luca Citi (199), School of Computer Science and Electronic Engineering, University of Essex, Wivenhoe Park, CO4 3SQ Colchester, UK B. Dan (171), Department of Neurology, Hopital Universitaire des Enfants Reine Fabiola, Universite´ Libre de Bruxelles, Belgium; and Laboratory of Neurophysiology and Biomechanics of Movementa, Universite´ Libre de Bruxelles, CP 168, 50 Av. F. Roosevelt, Brussels, Belgium F. De Vico Fallani (133), IRCCS Fondazione Santa Lucia, Rome, Italy Antonio Ferretti (67), Department of Clinical Sciences and Biomedical Imaging, Institute for Advanced Biomedical Technologies, University G. D’Annunzio of Chieti, Chieti, Italy Pierre W. Ferrez (189), Idiap Research Institute, Rue Marconi 19, 1920 Martigny, Switzerland Samson Freyermuth (159), Laboratoire de Physiologie de la Perception et de l’Action, UMR 7152, Colle`ge de France, CNRS, Paris, France Cosimo Del Gratta (67), Department of Clinical Sciences and Biomedical Imaging, Institute for Advanced Biomedical Technologies, University G. D’Annunzio of Chieti, Chieti, Italy Eugenio Guglielmelli (39), Biomedical Robotics and Biomicrosystems Laboratory, Universita` Campus Biomedico di Roma, Roma, Italy Dario Izzo (213), Advanced Concepts Team, European Space Agency, Keplerlaan 1, 2201 AZ Noordwijk, The Netherlands Karim Jerbi (159), INSERM U821, Brain Dynamics and Cognition Laboratory, Lyon 69500, France; and Laboratoire de Physiologie de la Perception et de l’Action, UMR 7152, Colle`ge de France, CNRS, Paris, France Philippe Kahane (159), Department of Neurology and INSERM U704, Grenoble Hospital, Grenoble, France Dean J. Krusienski (147), School of Engineering, University of North Florida, Jacksonville, Florida, USA Jean-Philippe Lachaux (159), INSERM U821, Brain Dynamics and Cognition Laboratory, Lyon 69500, France A. Leroy (171), Laboratory of Neurophysiology and Biomechanics of Movementa, Universite´ Libre de Bruxelles, CP 168, 50 Av. F. Roosevelt, Brussels, Belgium M. Marciani (133), IRCCS Fondazione Santa Lucia, Rome, Italy Martina Marinelli (199), Scuola Superiore Sant’Anna, piazza Martiri della Liberta` 33, 56127 Pisa, Italy D. Mattia (133), IRCCS Fondazione Santa Lucia, Rome, Italy Silvestro Micera (23), Institute for Automation, Swiss Federal Institute of Technology, CH-8092 Zurich, Switzerland; and ARTS and CRIM Labs, Scuola Superiore Sant’Anna, I-56127 Pisa, Italy
CONTRIBUTORS
xiii
Jose´ del R. Milla`n (189), Ecole Polytechnique Fe´de´rale de Lausanne (EPFL), Lausanne, Switzerland; and Idiap Research Institute, Rue Marconi 19, 1920 Martigny, Switzerland Lorella Minotti (159), Department of Neurology and INSERM U704, Grenoble Hospital, Grenoble, France Pedro Montoya (107), Department of Psychology, Universidad Illes Baleares, Palma de Mallorca, Spain Gernot R. Mu¨ller-Putz (119), Institute for Knowledge Discovery, Laboratory of Brain–Computer Interfaces, Graz University of Technology, Graz, Austria Ander Ramos Murguialday (107), Fatronik Foundation, San Sebastian, Spain; and Institute of Medical Psychology and Behavioral Neurobiology, University of Tuebingen, Tuebingen, Germany Xavier Navarro (23), Centro de Investigacio´n Biome´dica en Red sobre Enfermedades Neurodegenerativas (CIBERNED), Spain; and Institute of Neurosciences, Universitat Auto`noma de Barcelona, E-08193 Bellaterra, Spain E. Palmero-Soler (171), Laboratory of Electrophysiology, Universite´ de Mons-Hainaut, Mons, Belgium; and Laboratory of Neurophysiology and Biomechanics of Movementa, Universite´ Libre de Bruxelles, CP 168, 50 Av. F. Roosevelt, Brussels, Belgium M. Petieau (171), Laboratory of Neurophysiology and Biomechanics of Movementa, Universite´ Libre de Bruxelles, CP 168, 50 Av. F. Roosevelt, Brussels, Belgium Gert Pfurtscheller (119), Institute for Knowledge Discovery, Laboratory of Brain–Computer Interfaces, Graz University of Technology, Graz, Austria Giovanni Di Pino (39), Biomedical Robotics and Biomicrosystems Laboratory, Universita` Campus Biomedico di Roma, Roma, Italy Vittorio Pizzella (67), Department of Clinical Sciences and Biomedical Imaging, Institute for Advanced Biomedical Technologies, University G. D’Annunzio of Chieti, Chieti, Italy Gian Luca Romani (67), Department of Clinical Sciences and Biomedical Imaging, Institute for Advanced Biomedical Technologies, University G. D’Annunzio of Chieti, Chieti, Italy Danilo De Rossi (3), University of Pisa, Interdepartmental Research Centre ‘‘E. Piaggio’’, School of Engineering, 56100 Pisa, Italy Luca Rossini (213), Biomedical Robotics and Biomicrosystems Laboratory, Universita` Campus Biomedico di Roma, Via Alvaro del Portillo 21, 00128 Roma, Italy Paolo Maria Rossini (39, 81), Department of Neurology, Universita` Campus Biomedico di Roma, Via Alvaro del Portillo 21, 00128 Roma, Italy John Rothwell (51), Sobell Department, Institute of Neurology, Queen Square, London WC1N 3BG, UK
xiv
CONTRIBUTORS
S. Salinari (133), Dip. Informatica e Sistemistica, Univ. La Sapienza, Rome, Italy; and ARTS and CRIM Labs, Scuola Superiore Sant’Anna, Pisa, Italy Reinhold Scherer (119), Computer Science and Engineering, University of Washington, Seattle, Washington, USA; and Institute for Knowledge Discovery, Laboratory of Brain–Computer Interfaces, Graz University of Technology, Graz, Austria Tobias Seidl (39, 189), Advanced Concepts Team, European Space Agency, Keplerlaan 1, 2201 AZ Noordwijk, The Netherlands Francisco Sepulveda (93), Brain–Computer Interfaces Group, Department of Computing and Electronic Systems, University of Essex, Wivenhoe Park, Colchester, Essex CO4 3SQ, United Kingdom Fabrizio Sergi (39), Biomedical Robotics and Biomicrosystems Laboratory, Universita` Campus Biomedico di Roma, Roma, Italy Leopold Summerer (213), Advanced Concepts Team, European Space Agency, Keplerlaan 1, 2201 AZ Noordwijk, The Netherlands Oliver Tonet (199), CRIM Lab, Scuola Superiore Sant’Anna, Pisa, Italy Cornelia Weber (107), Institute of Medical Psychology and Behavioral Neurobiology, University of Tuebingen, Tuebingen, Germany Jonathan R. Wolpaw (147), Laboratory of Neural Injury and Repair, Wadsworth Center, Albany, New York, USA
FOREWORD
In writing this foreword, I am sitting comfortably on my chair in front of my computer and using my fingers to type character by character, word by word as they are formulated in my head. While this is a very common way of transmitting information indeed (emails, papers, books, SMS), it relies on a rather ‘‘un-natural’’ interface, compared to the ‘‘natural’’ ways of human information exchange: gesture, mimic, and speech. We have increased the eYciency of communicating in terms of speed, distance, and reach by adapting our means of communication to the interfaces imposed by the limitations of machines (typewriter, computer, mobile phone, etc.). We have therefore reduced our own interface to our most powerful tools: our hands and its 10 fingers. This has become possible since for many of these tasks, we have freed our hands from any additional manual work to be done at the same time. With the introduction of increasing computational power, we have been gradually introducing more human-like aspects to the interaction with machines and especially computers: the mouse and a graphical interface instead of text-only command-line inputs and more recently touch and multitouch displays as well as gradually speech recognition. In space, we are confronted with a whole range of additional complications and requirements which demand the use of our main tools: our hands! while continuing to interface with computers. In microgravity, we actively use our feet and hands to keep a certain posture and attitude, to perform experiments and manipulate objects. Given the risks associated with handling errors and the underlying complications of some of these activities combined with the high value of experimental and thus astronaut time in space, most of our activities are highly regulated, written down in detailed manuals and we are training most procedures during weeks and months on ground. Once in space, we therefore usually use a laptop close by, displaying the steps of our procedures while being concentrated to performing them with our hands at the same time and trying to keep posture essentially by the use of our feet. Many times we even have two laptops close by, one with the procedure and one that displays expected intermediate results (pressure values, temperatures, etc.) of the operations we perform. Very simple tasks on the laptop, like scrolling down a page, or increasing an image or following a link with further explanations become therefore already a distraction that take time since they require taking at least one hand oV the experiment or maintenance/repair task, find the trackball on the computer keyboard, move the cursor to the right direction and then click, scroll, etc. before being able to turn back to the actual manual task. xv
xvi
FOREWORD
As an astronaut, I am therefore highly interested in any technology that would allow me to keep concentrated on my actual manual tasks and have my two hands freed to perform them. Just to give you an example from my own experience, I even tried sometimes to use the track-pad with my nose just to keep my hands inside the gloves of a glovebox. We are currently experimenting with a range of such tools like speech recognition, gaze tracking, head-mounted cameras, and overlay displays, etc. All of these techniques have their advantages and disadvantages, and at this stage it is not clear which one would prove the most eYcient. Recent advances in brain science in understanding the functioning of our brain and in brain–machine interfaces (BMIs) have opened another possibility. The prospect of being able—with some prior training—to keep our hands and eyes on the experiment and perform some initially even very simple tasks like scrolling in a document, closing windows, increasing font sizes, etc. by ‘‘thinking’’ is very attractive and I am thus welcoming the initiative of this book, where the scientific community working on BMIs attempts to describe the state-of-the-art in BMIs research for the purpose of human space activities and critically reviews the requirements that space puts on the design of these types of interfaces. FRANK DE WINNE ESA Astronaut
PREFACE WHY THE BMI ‘‘RENAISSANCE’’ COULD AFFECT THE FUTURE OF SPACE EXPLORATION
I. Introduction
The Advanced Concepts Team of the European Space Agency has been investigating the potential application of brain–machine interfaces (BMIs) in cooperation with a number of European universities.1 It was a preliminary eVort during which our group collected data, ideas, and started a vivid discussion which confirmed the initial interest and led to a more in-depth analysis of possible solutions oVered by BMIs to some of the limitations astronauts face when operating in space. Hence, this publication on BMIs evolved naturally, targeting primarily the aerospace community, and aiming to provide the knowledge and the reflections that we hope will help overcoming the healthy skepticism sometimes expressed about the application of BMIs in space. While preparing this book we have come to appreciate the importance such a publication could also have for BMI specialists, as it describes the needs and requirements the space environment puts on interface designs and on their performances, and discusses the resulting potential and limitations of the technology. This book is therefore primarily written by, and targeted to, two communities: the space community, always interested in innovative solutions to renowned problems (especially nowadays, as the renewed Moon and Mars human exploration plans are going to put the lights on some unsolved issues related to the human presence in space beyond low Earth orbit), and the wider scientific community (Neuroscientists, Bioinformatics, Bioengineers, Roboticists, Neurorehab scientists) investigating into BMIs and new generation interfaces in general. This community, even if notoriously overloaded with new publications on the topic, is particularly aware of the importance of 1
Brain-Machine Interfaces, Advanced Concepts Team, European Space Agency, March 2009, http:// www.esa.int/gsp/ACT/bng/op/BMI.htm xvii
xviii
PREFACE
understanding the requirements BMI systems need to meet in order to reach new applications, especially those having a huge inspirational value such as space exploration. This book aspires to lay the first foundation and eventually become a reference point for future routine collaborations between space actors and the wide field of neuroprosthesis and BMIs experts.
II. BMIs Renaissance
To share the enthusiasm of many in BMIs, one has to appreciate the diVerence between the potential advantages of a brain interface and other kinds of interfaces. The worldwide amazement generated by the work of Nicolelis in the year 2001 was caused by the clarity with which this diVerence was demonstrated.2 In a long video sequence, the world witnessed a monkey in the act of controlling a robotic arm and feeding itself by doing so. The monkey was sitting and behaving normally while, at the same time, controlling with its neurons the actions of the robotic arm. The message was so clear and revolutionary in its nature that it inevitably started a new fervor we dare to refer to as ‘‘BMIs renaissance.’’ It was suddenly clear that BMIs are not only potentially able to restore lost abilities, but they are also a realistic option to augment capabilities and eventually adding new ‘‘peripherals’’ to the body. An insight that equally generated hopes and fears in the general public and among scientists and opened a healthy discussion on ethical implications related to the use of this technology.3
III. Natural Interfaces Mean Technological Democracy
The proliferation of more and more sophisticated electronic devices is a characteristic of our time. The direct access to informational contents and to applications’ interfacing and controlling is steadily increasing, while the new phenomenon of digital divide (here intended in its general sense) is also growing at the same rate: the use, and thus its related benefits, of new technologies is not accessed by all potential users. Geographical discrimination is only one of the gaps, which the digital revolution is struggling to bridge. Another discrimination regards knowledge. Any new device often requires the know-how related with the interface of its predecessor, plus a learning process associated to the new features oVered. Hence, who does not manage to keep the pace, for reasons related with 2
Nicolelis, M. A. L., ‘‘Actions from thoughts’’, Nature 2001, 409, 403–407 Clausen, J., ‘‘Man, machine and in between’’, Nature 2009, 457, 1080–1081
3
PREFACE
xix
motivation, age, education level, work experience among the others, is eventually overwhelmed by devices each time more complex and advanced. Henry Ford once said that ‘‘true progress is made only when the advantages of a new technology are within reach of everyone’’: a great challenge that lies ahead our digital era. A solution is the development of better interfaces, since they are responsible for our communication with machines, and thus for the definition of a universal language able to provide all with the possibility of controlling present and future devices. It is possible to argue that the best accessibility will derive from interfaces able to exploit at best the natural communication pathways characteristic of the human being. A ‘‘natural’’ interface learns and adapts itself to the user’s interfacing modalities, and not the opposite, extending the use of new technologies towards population sectors which are typically struggling to interface with technologies (i.e., personal computers). Within this context, BMIs are being seen as a technology aimed at improving the everyday life of a large number of common people by providing such a universal language. Many classical interfaces fail to fulfill such an ambitious goal. They are operated by predefined user’s motor actions, and typically diVerent action kinematics and geometries are associated with diVerent interfaces (i.e., computer keyboards and mouse). As a consequence, their use inevitably requires a re-modulation of connections between user’s brain sensorimotor areas involved with the device utilization (i.e., the neuronal networks involved with the actual formulation of a task), and those dedicated to the motor task for the interface operation (i.e., the hand and fingers control to obtain a correct typing of the related task’s commands). Obviously, operating a classical interface carries a high cognitive load which—when several diVerent interfaces have to be operated within a short period, or when the operator is less re-active in its internal re-modulation—can easily decrease the operator’s mental performance due to mental fatigue and increase the error rate.
IV. Interfaces for Astronauts
Astronauts (a particular highly motivated, trained, and skilled category) are also suVering, when in orbit, from what can be seen as a case of digital divide. On Earth, the high versatility of humans allows for a huge range of elaborated behaviors, such as dancing, playing soccer, doing acrobatics, or playing music instruments, some even at the same time. However, the human motor performances are strictly bounded to the physical conditions that govern our planet. Our perception and planning of movement, for example, are related to the identification of the gravity axis. Inevitably, our sensory–motor system encounters a loss of performance in situations of changed or annihilated gravity. This loss is so important that, from certain perspectives and for the purpose of an academic
xx
PREFACE
comparison, we can assume that astronauts are in a similar situation to people aVected by motor disabilities on Earth: both have, for diVerent reasons, a deficit in the performances required to accomplish their motor tasks. On Earth, disabled people can take advantage of assistive systems, which are technologies designed to fill the gap between user’s residual abilities and required ones. The reduction of physical and mental ability suVered by astronauts is addressed (at least partially) by a number of assistive technologies redesigned for functioning in space. As for any activity which relies on motor coordination, weightlessness conditions aVect astronauts uses of any kind of classical interface. It may be argued that a complex integration of ‘‘behavior monitoring technologies’’ like speech recognition, gesture recognition, facial expression recognition, and gaze tracking provides an appropriate supportive system and can provide valid natural interfaces. A correct use of these technologies has been demonstrated to have the potential to handle the control of complex systems, yet their potentials for severely disabled people and—in an analogue way—for astronauts is limited. Gesture and facial expression recognition cannot be eYcient if the user does not perform within strict tolerances and speech recognition reliability presents great challenges when background noise is present (this can be as high as 64 dBA for the air conditioning to 100 dBA for some vent relief valves in case of space stations). Gaze tracking alone is not precise enough and rich in contents to permit the control of complex systems. In this work, we propose to further explore BMI technologies as alternatives to other, more extensively studied devices that are inherently limited by their functioning principle. Completely independent from the user’s physical abilities, BMIs are instead able to access the user’s intentions at a higher level, were they naturally origin: in the brain. They predict directly the user’s motor intentions, not related with users’ abilities, and are thus accessible to the widest range of users. By monitoring the activity of neurons (individually or in networks) and translating them into actions, BMIs have the potential to provide the most direct control over complex systems and since they are expected to operate, in principle, very similar in space and on Earth could assist astronauts by helping them to perform in space as eYciently as on the Earth.
V. BMIs for Space Applications: An Outlook
Understanding the real potentials of BMIs for space applications requires an eVort in correctly framing BMIs themselves in the first place. When a user controls with his thoughts the operations of a robotic hand, for example, his brain is being part of a hybrid system in which living tissues (neural cells among others) and artificial elements (i.e., the robotic hand) are working together. Systems like this are referred to as Hybrid Bionic Systems (HBSs), and BMIs
PREFACE
xxi
are one of their manifestation. The first section is entirely dedicated to these systems, and focuses on HBSs other than BMIs which could be beneficial for space applications, in an increasing ‘‘scale of invasiveness’’ fashion, from the noninvasive electromyographic and gaze-tracking interfaces, where an intact human body controls by means of natural actions diVerent systems, on which Danilo de Rossi and Federico Carpi give an in-depth review; passing through the invasive monitoring of peripheral nerve activity for bidirectional interfaces, able to read the planned limb motions and to stimulate limb sensations at the same time, actually fusing the user’s body with the controlled system, as two of the pioneers of interfaces with the peripheral nervous system, Xavier Navarro and Silvestro Micera, explain in a comprehensive review; to the end point of hybridization, where small portions of an insect’s brain are included in and connected to a complex artificial system, which can then benefit from the amazing and otherwise un-reproducible level of intelligence which was developed in a millenarian evolution towards autonomous and successful behaviors. Tobias Seidl, a behavioral biologist from the Advanced Concepts Team of the European Space Agency, together with Eugenio Guglielmelli and his Biomedical Robotics and Biomicrosystems group, focused for the first time on such an architecture and present their ideas and outcomes in the last chapter of the section on HBS. As for any other HBS, it is first of all essential to understand the biological components included in the architecture, in order to find the right way to integrate it with the artificial part and to foresee the potential applications and possible shortcomings of the complete system. For this reason the second section is fully dedicated to describe the basic neuroscience knowledge, a task addressed by some of the most renown researchers in the field of neurophysiology. John Rothwell explains what a neuron is, how it works, and how it competes with other neurons in neuronal networks and functional areas to the production of the overall brain activity. Claudio Babiloni, Gianluca Romani, and Vittorio Pizzella explain how this activity is globally organized, how we can monitor and analyze it for clinical or research activities, like for the control of BMIs. Finally, Paolo Rossini describes how, thanks to plastic changes, the brain learns and adapts itself under internal and environmental pressures, and how this adaptations could interact with BMIs control performances in the short and in the long time. The next inevitable step towards understanding BMIs is in the interface architecture itself. As Francisco Sepulveda explains, opening the third section specifically dedicated to the architectures of BMIs, BMIs are composed of many diVerent components, each of which playing crucial roles in the final performance of the system. Few diVerent architectures are then presented directly by their inventors, exploring some of the variety of applications to which those systems can be a solution. Clinical applications, like restoring a communication pathway for patients which otherwise would be completely cut out from any possible interaction with the environment and with other people, or controlling a limb on which
xxii
PREFACE
the natural control ways were pathologically interrupted, can be addressed with BMI systems exploiting self-controlled slow cortical potentials, as Niels Birbaumer demonstrated during the last 30 years. Almost the same time separated Gert Pfurtscheller from demonstrating the possibility of correlating event-related potentials (ERP) to limb movement planning and execution, to the current implementation of automatic and real-time ERP interpreter which is able to drive many diVerent BMI systems, like virtual keyboards, virtual navigation systems, and even Google EarthW software. While groups with long-time experience are synonymous of scientific quality, new research teams have in the past shown higher likelihood to introduce unconventional innovation and novelty. Between the new protagonists of BMI research, Francesco Babiloni and his group at Santa Lucia excelled in applying their experience on EEG signal processing in the optimization of a BMI system to work as a remote controller for a complex domotic environment, in which severely paralyzed patients can control devices autonomously. Noteworthy in this respect is also the work of Dean Krusienski, who has built on his experience matured with the BMI group of Jonathan Wolpaw in the Wadsworth Center, one of the most renowned research centers on noninvasive BMIs: Krusienski and Wolpaw present in this book what, in the opinion of the editors, is one of the most advanced BMI systems for humans currently developed. Finally, invasive BMI systems can be evaluated on human subjects, whenever the subjects are already, and for clinical reasons related to other pathologies (like epilepsy), implanted with intracortical electrodes (i.e., for electrocorticography—ECoG). The first results of ECoG–BMI systems are indeed holding promising results, as detailed in the contribution of Karim Jerbi, Alain Berthoz, and Jean-Philippe Lachaux. In the fourth and last section of the book, we wanted to introduce what we think might be pivot elements for future BMI for space application. Despite the many possible solutions presented in the third section, some of which are already able to address questions related to space applications, exposition to microgravity causes some very specific diVerences in the way the human brain works. As the brain is the main element of BMIs, any changes due to short-time or long-time exposition to ‘‘space environments’’ have to be taken in consideration in the specific design and set up of BMI for space applications. In particular, as Guy Cheron demonstrated in tests performed during several experiments on the International Space Station (ISS), there are adaptive changes of rhythmic EEG oscillations under microgravity exposure which can have profound implications for BMI systems’ design. These eVects and implications are presented in the opening chapter of the fourth section. A first step towards appreciating the eVects of microgravity on BMI operations was already done during a parabolic flight campaign of ESA in December 2007. Thanks to parabolic flights the human body can experience repetitive short exposures to microgravity. Together with Jose´ del R. Millan, the Advanced Concepts Team of the European Space Agency tested
PREFACE
xxiii
the eVects of such exposures on the brain activity related with BMI control. The complete results of this study are reported for the first time in this book. Once the potential eVects of microgravity on BMI have been taken under consideration and some of the initial problems overcome, interface engineers are expected to design brain-driven systems for many types of space applications. However, early adopters would probably quickly be lost in trying to rate the diVerent architectures in terms of performance and reliability, since diVerent BMI research groups tend to apply very diVerent approaches to performance evaluation. One of the outcomes of the preliminary studies on BMIs for space applications we performed in 2006 with several European universities came from three young scientists, Oliver Tonet, Luca Citi, and Martina Marinelli. It is a valuable tool to address the matching between BMI systems and space applications, and its results are presented in this book. The book is closed with a short communication chapter from the editors, which intends to highlight the research directions of BMIs with the perceived highest potential impact on future space applications, and to present an overview of the long-term plans with respect to human space flight, and is concluded by suggesting research and development steps considered necessary to include BMI technology in future architectures for human space flight. The research field of BMI grew with progressive impetus during the last 10 years, giving birth to a wide number of publications on the topic. The present book is not trying to complement these. While not introducing new discoveries or interpretations of elements of BMI, it intends to summarize the state-of-art of the field in order to address, for the first time, the possible relationship between BMIs and space exploration. The question, which, at first glance, could be regarded as a mere science fiction dilemma, is indeed well founded in the eVects that microgravity has on the human motor coordination. LEOPOLD SUMMERER DARIO IZZO LUCA ROSSINI
EMG-BASED AND GAZE-TRACKING-BASED MAN–MACHINE INTERFACES
Federico Carpi and Danilo De Rossi University of Pisa, Interdepartmental Research Centre ‘‘E. Piaggio’’, School of Engineering, 56100 Pisa, Italy
I. Introduction II. EMG-Based Interfaces A. Fundamentals B. Characteristics and Issues C. Examples of Applications III. Gaze-Tracking-Based Interfaces A. Fundamentals B. Characteristics and Issues C. Examples of Applications IV. Final Remark References
A great demand for brain–machine and, more generally, man–machine interfaces is arising nowadays, pushed by several promising scientific and technological results, which are encouraging the concentration of eVorts in this field. The possibility of measuring, processing and decoding brain activity, so as to interpret neural signals, is often looked at as a possibility to bypass lost or damaged neural and/or motor structures. Beyond that, such interfaces currently show a potential for applications in other fields, space science being certainly one of them. At present, the concept of ‘‘reading’’ the brain to detect intended actions and use these to control external devices is being studied with several technical and methodological approaches; among these, interfaces based on electroencephalographic signals play today a prominent role. Within such a context, the aim of this section is to present a brief survey on two types of noninvasive man–machine interfaces based on a diVerent approach. In particular, they rely on the extraction of control signals from the user with techniques that adopt electromyography and gaze tracking. Working principles, implementations, typical features, and applications of these two types of interfaces are reported.
INTERNATIONAL REVIEW OF NEUROBIOLOGY, VOL. 86 DOI: 10.1016/S0074-7742(09)86001-7
3
Copyright 2009, Elsevier Inc. All rights reserved. 0074-7742/09 $35.00
4
CARPI AND DE ROSSI
I. Introduction
Integrating human and robotic machines into one system has the potential to oVer multiple opportunities for creating assistive technologies that can be used in space, biomedical, and industrial applications. In this context, the development and use of noninvasive man–machine interfaces is progressively gaining a considerable importance. The most direct and, for certain aspects, fascinating (although challenging as well) implementation of a man–machine interface relies on so-called brainmachine interfaces (BMIs) or brain–computer interfaces (BCI), or neuroprostheses (Wolpaw et al., 2002). Such systems are typically conceived as technological interfaces between a machine (usually a computer) and a human brain. They should allow the user to perform certain tasks without any or with minimal motor action. This implies that neural impulses generated by the user should be detected, elaborated and used by the machine, approximately in real time, to perform specific tasks. Accordingly, BMI might represent fundamental systems functional to rehabilitation, communication or assistance. For instance, they are intended to implement brain controls of devices deputed either to perform external actions (such as active prostheses and orthoses, computer virtual keyboards, home electronic equipment, or aid systems) or to trigger functional electrical stimulations (FES) of muscles and nerves (bypassing degenerated or interrupted biological electrical routes) (Navarro et al., 2005; Wolpaw et al., 2002). Beyond biomedical applications, the availability of reliable, eYcient and noninvasive BMIs may provide advantages for diVerent disciplines, space science being one of them. As an example, extravehicular activities may be performed by robotic systems tele-operated by astronauts by means of noninvasive BMIs. Such interfaces may be used also to perform multitask operations. So far, the concept of ‘‘reading’’ the brain to detect intended actions and to use extrapolated signals to perform tasks has been developed in several ways, by adopting diVerent technical and methodological approaches and achieving diVerent results. Interfaces based on electroencephalography (EEG) are certainly one of the most promising solutions to this problem (Wolpaw et al., 2002), as extensively described in the rest of this book. Nevertheless, in addition to EEG-based interfaces, diVerent kinds of man– machine interfaces are currently drawing considerable attention as well. The main purpose of this section is to provide a brief description of two types of such diVerent systems: those based on electromyography (EMG) recordings and those based on gaze-tracking techniques. These might be regarded as complementary rather than alternative approaches in comparison with EEG-based interfaces.
EMG-BASED AND GAZE-TRACKING-BASED INTERFACES
5
The following sections describe these technologies, by providing for each of them brief insights in the fundamental aspects, typical features and most significant applications. II. EMG-Based Interfaces
A. FUNDAMENTALS Electromyography consists in a recording of bioelectric signals generated by neuromuscular activity. As such, EMG signals are an electrical display of neuromuscular activation associated with a contraction of a skeletal muscle, regulated by the central and peripheral nervous system (De Luca, 1988; Henneberg, 2000). The bioelectric genesis of EMG potentials can be synthetically described as follows. Motoneurons lead nervous pulses from the spinal cord to the neuromuscular junctions, where action potentials trigger muscle contraction, as the final result of a series of electrochemomechanical events (their description goes beyond the scope of this book) (Guyton and Hall, 2005). The electrical signal generated by the activation of the muscular fibers belonging to a certain motor unit is called motor unit action potential (MUAP) and represents an elementary basic component of an EMG signal. A MUAP drives a force twitch of the muscle fibers. To sustain a prolonged contraction, a motor unit should be activated repeatedly; accordingly, muscle fibers belonging to the considered motor unit should be driven by a sequence of MUAP; such a sequence is called motor unit action potential train (MUAPT). As a result, an EMG signal results from a space and time varying superposition of multiple MUAPTs (De Luca, 1988), as schematically represented in Fig. 1. EMG measurements can be performed noninvasively by means of surface electrodes (also known as skin or cutaneous electrodes); in this case, the technique is usually referred to as surface electromyography (SEMG) (De Luca, 1997). Figure 2 reports an example of arrangement of skin electrodes for EMG recordings from the arm (e.g., biceps muscle); as for any other type of bioelectric signal, a single-channel measurement always requires a primary, a secondary, and a ground electrode. EMG signals can have amplitudes up to the order of 1–10 mV, depending on the measurement site and adopted electrodes (De Luca, 1997; Webster, 1997); maximum values of the peak-to-peak amplitude close to 5 mV can be commonly obtained with the appropriate use of surface electrodes. Typically, the most significant frequency components of EMG biopotentials are included approximately between 10 Hz and some hundreds of Hz, as indicative orders of magnitudes; accordingly, usual recommendations suggest a recording bandwidth covering the range 25 – 500 Hz (De Luca, 1997). An example of EMG spectrum is reported in Fig. 3.
6
CARPI AND DE ROSSI
EMG signal
EMG electrode
Muscle MUAPT signals
Motoneuron FIG. 1. Schematic drawing of the bioelectric genesis of EMG signals.
Primary electrode Secondary electrode Ground electrode
EMG recording unit
Vin+ GND Vin−
FIG. 2. Schematic drawing of an EMG recording by means skin electrodes.
EMG-BASED AND GAZE-TRACKING-BASED INTERFACES
7
Amplitude [a.u.]
100 80 60 40 20 0
0
100
200
300
400
500
Frequency [Hz] FIG. 3. Frequency spectrum of an EMG signal recorded with skin electrodes from an anterior tibial muscle during a voluntary isometric activation, about 50% of the maximum value.
Since the half of the 1800 century, electromyographic signals collected from the skin surface have been used as an easy and noninvasive access to the electrophysiological processes that drive muscular contraction. As an application of the knowledge arisen in the following decades, since the early 1960s the use of EMG signals for controlling prosthetic devices has progressively increased. To such an extent, EMG biopotentials can be captured from diVerent body portions, depending on the application (Navarro et al., 2005); for instance, the forearm is typically adopted in the case of prosthetic hands (Castellini and van der Smagt, 2009; Zecca et al., 2002). The simplest strategy of use of an EMG signal for control purposes consists in extracting its amplitude or rate of change. More information can be obtained by using two channels, corresponding to two primary electrodes placed on two antagonist muscles (e.g., biceps and triceps brachii or flexor and extensor of the forearm). Figure 4 reports some examples of signals that can be recorded in such a manner. To extract useful information, the recorded raw signals should undergo an adequate preprocessing, aimed at emphasizing specific features. The estimation of the so-called ‘‘normalized muscle activation level’’ (NAL) is one of the most relevant of those; it is achieved with a signal preprocessing procedure including the following blocks (see, for instance, Rosen et al., 2001): (1) a high-pass filtering (to reject low-frequency motion artifacts); (2) a full signal rectification (to extrapolate the signal absolute value); (3) a low-pass filter (to reduce noise, while limiting the bandwidth to the most useful components); (4) a signal normalization (typically with respect to the mean value during maximal voluntary isometric contraction).
8
CARPI AND DE ROSSI
Elbow flexion
Elbow extension Biceps
Biceps
Triceps
Triceps Forearm pronation
Forearm supination
Biceps
Biceps
Triceps
Triceps
FIG. 4. Examples of EMG signal couples detectable from agonist–antagonist muscle couples for diVerent types of movements.
B 500
Recorded EMG [mV]
3
1000
1500
2000
2500 3
2
2
1
1
0
0
−1
−1
−2
−2
−3 500
1000
1500
Time [n⬚ of samples]
2000
−3 2500
500
Pre-processed EMG [mV]
A
2.0
1000
1500
2000
2500 2.0
1.5
1.5
1.0
1.0
0.5
0.5
0.0
0.0 500
1000
1500
2000
2500
Time [n⬚ of samples]
FIG. 5. Example of an EMG signal recorded from the forearm, before (A) and after (B) a preprocessing.
As an example, Fig. 5 presents an EMG signal, both as-detected and after an elementary preprocessing; in this example, the final part of the preprocessing phase included a leveling of the signal above a certain threshold. Aimed at driving an external device, the controller should then further process the EMG signal, typically according to the following successive actions: (1) action onset detection (i.e., identification of the time instant when the muscle goes from the relaxed to the contracted state); (2) feature extraction; (3) pattern classification. Several algorithms can be used for detecting the movement onset (Micera et al., 1998, 2001) and for extracting features and classifying related patterns (Crawford et al., 2005; Zecca et al., 2002). Real-time pattern discrimination and classification is certainly one of the most delicate issues for EMG signals (as for any other type of bioelectric information); with respect to this, neural network-based algorithms
EMG-BASED AND GAZE-TRACKING-BASED INTERFACES
9
(Bishop, 1995) are today commonly adopted (Castellini and van der Smagt, 2009; Soares et al., 2003; Zecca et al., 2002), in addition to Bayesian classifiers (Zecca et al., 2002) and fuzzy logic systems (Ajiboye and Weir, 2005; Chan et al., 2000; Zecca et al., 2002).
B. CHARACTERISTICS AND ISSUES One of the most significant characteristic of SEMG is represented by its capability of evaluating ongoing muscular activity noninvasively and in a comfortable manner for the user. This feature makes the EMG technology easily and readily applicable for controlling robotic devices (Navarro et al., 2005). Such a use is also favored by the relative ease of detection of EMG signals, due to their quite high amplitudes (as reported above). Moreover, their band-pass spectrum, typically excluding frequencies lower than 25 Hz (De Luca, 1997), provides an advantage for a rejection of motion artifacts; in fact, these contain very low frequencies that can be eVectively filtered out, while keeping most of the useful information. Nevertheless, although surface EMG is a useful measure of muscle activation and assessment, the information that can be extracted from this type of signals is aVected by some intrinsic limitations, briefly reported below. An important issue is that EMGs capture potentials generated mostly by superficial muscles and that are sensitive to the electrical activity of a greater muscular area (large number of motor units), with consequent poor selectivity. With respect to this, the phenomenon known as crosstalk plays a relevant role: it refers to a signal that is recorded over a certain muscle but is actually generated by a nearby muscle, following a conduction through the intervening volume to the recording electrodes (De Luca and Merletti, 1988). In the past, crosstalk signals have been regarded as low-frequency phenomena (due to both their far origin and the low-pass spatial filtering they undergo), so that high-pass filtering has been adopted as a remedy (De Luca, 1997). Nevertheless, more recent investigations have suggested the impossibility of generalizing such an approach, by describing the limitations that these and other types of eVects provide while attempting to use surface EMG to infer several parameters of muscular activations, such as the level of the activation itself, the type of motor unit recruited, the upper limit of motor unit recruitment, the average discharge rate and the degree of synchronization between motor units (Farina et al., 2004). As a second issue, given the complexity of the task and the variability of the EMG signals, EMG controlled systems usually require custom calibrations for each user and specific training for the pattern recognition algorithms. One of the major diYculties with EMG-based interfaces is represented by the needed of considerable mental eVorts during the training phase (Soares et al., 2003).
10
CARPI AND DE ROSSI
An equally important problem is the stochastic nature of EMG signals, resulting in parameter estimation errors that, in turn, cause classification and/ or control diYculties. Moreover, some control errors are generally introduced by the inability of the subject to reliably generate and reproduce the target contraction signals. Besides, current techniques make it typically very diYcult to control more than two degrees of freedom (DoF ) (Castellini and van der Smagt, 2009; Costanza et al., 2004; Zecca et al., 2002). Such a problem is further complicated by the time-variant characteristics of the EMG signal, due to changes in the electrical impedance of the skin, electrode locations, muscular fatigue, sweat, and so on. Such variations can cause significant problems, because of the usually adopted calibration process. In fact, most current EMG controlled devices are tuned only in the oZine phase; the user learns to reproduce one or two diVerent signals and the device is tuned to these signals. With such controllers it is not possible to successfully control more than 1 active DoF, because the diVerences between tuned signals and actual ones tend to increase gradually with time. For practical usage, the number of EMG channels is typically limited to 2, although the implementation of pattern recognition approaches can potentially lead to a much higher number of control commands (Fukuda et al., 2003).
C. EXAMPLES OF APPLICATIONS 1. EMG Controlled Cybernetic Hands EMG signals are today largely used as control inputs for myoelectrically based powered prostheses (Bitzer and van der Smagt, 2006; Castellini and van der Smagt, 2009; Zecca et al., 2002). In such systems, the human operator provides command signals in a feed-forward open-loop mode, utilizing only visual feedback as the primary source of information. So-called myoprocessors for controlling such prosthetic devices are today commercially available (see, for instance, examples mentioned in Castellini and van der Smagt, 2009). These systems are frequently based on one dimension only of the EMG signal (the variance or mean absolute value). Users are trained to produce a constant level of activation of muscles and the prostheses are tuned accordingly. A steady-state EMG signal, however, has very little temporal structure because of the active modification of recruitment and firing patterns needed to sustain a contraction (De Luca, 1988, 1997; Henneberg, 2000). The parameters that could be extracted to quantify its amplitude (e.g., variance or mean absolute value) or its frequency characteristics (e.g., Fourier spectrum or median frequency) are often not suYcient to distinguish between more than two classes of movement. As a result, despite the interesting results being achieved, it is typically
EMG-BASED AND GAZE-TRACKING-BASED INTERFACES
11
very diYcult to control reliably more than two DoF (Castellini and van der Smagt, 2009; Costanza et al., 2004; Zecca et al., 2002). Among the diVerent types of EMG controlled devices and systems studied at present, prosthetic hands (myoelectric hands) and, more generally, cybernetic hands are object of considerable study (Bitzer and van der Smagt, 2006; Castellini and van der Smagt, 2009; Ferguson and Dunlop, 2002; Zecca et al., 2002). We believe that such systems might also have a relevant impact for the space field. As an example, future space robots might benefit from articulated multifinger hands with bioelectric control. However, replicating the performance of the human hand is beyond current technical capabilities. In fact, the human hand is extremely complex: it has 22 DoF, controlled by about 38 muscles in the hand itself (almost twice the number of DoF ). Currently available commercial hand prostheses have a limited number of DoF (1 or 2 for finger movements and thumb opposition), thus typically showing low grasping functionality (Castellini and van der Smagt, 2009). They are controlled either in proportional or on/oV mode, by using a couple of primary EMG electrodes placed on two antagonist muscles. The use of a larger number of electrodes to control more DoF introduces several issues related to the movements coding and to the number of electrodes required that complicates the structure and the use of the socket. 2. EMG Controlled Exoskeletons Powered exoskeletons represent a type of wearable active orthotic systems that, in the opinion of the authors, could find significant applications in the space field. They are typically designed as an external structural mechanism that can be worn by the subject, so that one or more human joints correspond to those of the structure. A powered exoskeleton can be used as a system belonging to one of the following categories of: (1) power amplifier; (2) master device of a master/slave teleoperator system; (3) haptic device (Rosen et al., 2001; Tsuji et al., 2003). They can be synthetically described as follows. Within the use of an exoskeleton as a human power amplifier, the human provides control signals for the exoskeleton, while the actuators of the latter provide most of the power to perform the task. The application of an exoskeleton as a master device (in a master/slave system) enables the operator (master) to control a remote robotic arm (slave). Finally, the adoption of an exoskeleton as a haptic device is aimed at simulating human interactions with virtual objects (virtual reality); in such a case, a computer simulation acts as a virtual slave component of a master/slave system (Rosen et al., 2001). EMG signals can be adopted as useful command inputs for an exoskeleton (Di Cicco et al., 2004; Lucas et al., 2004; Mulas et al., 2005; Rosen et al., 2001). The EMG signal along with joint kinematics of the exoskeleton is fed into a myoprocessor that implements a muscle model, used to predict muscular variables relative to each involved joint (Rosen et al., 2001).
12
CARPI AND DE ROSSI
Nevertheless, as opposed to controlling a myoelectrically powered prosthesis, the operation of a myoelectrically powered exoskeleton is more challenging. In fact, such systems imply the existence of a kinematic link between the human and the exoskeleton; as a result, the human neural control system and exoskeleton control system coexist and have to cooperate by sharing the same kinematic and dynamic constraints. Moreover, when an exoskeleton is used, the muscular force cannot be estimated from the EMG signals; in fact, the angle of each human joint coupled with an exoskeleton joint constantly changes during operation, causing a modification of the length and end points velocities of the muscles attached to that joint. As a result, muscle models implemented by the myoprocessor have to take into account the muscle’s length and velocity, in addition to the EMG signal (which defines the muscle activation level), in order to predict the force that will be developed by the muscle (Rosen et al., 2001).
III. Gaze-Tracking-Based Interfaces
A. FUNDAMENTALS Gaze reflects our attention, intention, and desire. Gaze information plays an important role in identifying a person’s focus of attention; the information can provide useful communication cues to a multimodal interface. For example, it can be used to identify where a person is looking and what he is paying attention to. Thus, detection of gaze direction makes possible to extract such information that is valuable in human–computer interaction. Accordingly, computers integrated with gaze-tracking functions can provide an intuitive and eVective interactive system (Brunner et al., 2007; Ding et al., 2005; Hutchinson et al., 1989; Kaufman et al., 1993; Morimoto and Mimica, 2005). A person’s gaze direction is determined by two factors: the orientation of the head and the orientation of the eyes. While the first determines the overall direction of the gaze, the orientation of the eyes provides the exact gaze direction and is limited by the head orientation. The clear vision of an object is possible only when its image falls on the centre zone of an ocular portion, called fovea. Figure 6 shows the structure of the eye. To explore a scene, it is necessary that the eyes complete the movements that concur to carry and maintain the image stable on the fovea. The ocular movements of a subject, therefore, can tell us where he is watching and what and how long he is observing. Saccades are the faster ocular movements that the oculomotor apparatus can complete and they have the task to move the visual axis during the exploration of the scene. Fixation consists of a pause between two successive saccades and represents the time interval during which the visual information is
13
EMG-BASED AND GAZE-TRACKING-BASED INTERFACES
Cornea Iris Conjunctiva y ar
dy bo
Aqueous
humor
Ciliary muscle
li
Ci
Rectus tendon
Lens Zonal fibers Vitreous humor
Optic axis
Ora serrata
Visual axis
Papilla optic disk blind spot
Sclera
Retina
Choroid Lamina cribrosa Sheath
Fovea
Optic nerve
Macula lutea Center
Right
FIG. 6. Structure of the eye (from Malmivuo and Plonsey, 1995).
acquired. Therefore, visual exploration is made of a succession of saccades and fixations. Gaze determines the user’s current line of sight or point of fixation. The fixation point is defined as the intersection of the line of sight with the surface of the object being viewed. Therefore, gaze tracking may be used to interpret user’s intentions for interactions. Since the beginning of the 1990s, gaze-tracking technologies have progressively become more accurate, less cumbersome and today are available as commercial products. Several techniques for tracking the direction of eye-gaze are
14
CARPI AND DE ROSSI
FIG. 7. Examples of gaze-tracking systems: (A) head mounted device for corneal reflection; (B) EOG electrodes placed around the eyes; and (C) scleral contact lens equipped with an induction coil, to be arranged in contact with the eye (adapted from Matsumoto, 2003).
now available (Duchowski, 2003; Morimoto and Mimica, 2005). The most relevant for the purposes of this book are briefly described below. 1. Corneal Reflection This technique consists of a detection of the eye rotation by measuring the reflection of a light beam that is shone onto the cornea of the eye (Morimoto and Mimica, 2005; Reulen et al., 1988; Yoo et al., 2002). Typically, infrared (IR) light is used, in order to distract the user as little as possible and to avoid interference from other light sources (e.g., lamps). Two types of systems for corneal reflection can be used, depending on the arrangement of the IR video camera: (1) solidary with the subject’s head (Fig. 7A), being typically mounted on a helmet; (2) solidary with the scene to be explored (e.g., a computer screen) (Duchowski, 2003; Morimoto and Mimica, 2005). The higher suitability of one type over the other should be evaluated for each specific application. Corneal reflection techniques typically exhibit an accuracy of 1–2 (Morimoto and Mimica, 2005). 2. Electrooculography Electrooculography (EOG) is today a technique routinely used as a readily applicable diagnostic tool for studying the human oculomotor system (Malmivuo and Plonsey, 1995). It allows a detection of eye rotations, by measuring electric biopotentials from the skin surrounding the eyes, with simple surface electrodes (Fig. 7B) (Kaufman et al., 1993; Malmivuo and Plonsey, 1995; Morimoto and Mimica, 2005). This technique relies on the following bioelectric principle. The cornea of the eye is electrically positive relative to the back of the eye, with a resting potential of the order of 1 mV. This bioelectric source behaves as a corneoretinal dipole, whose orientation is varied by the eyeball movements. Accordingly, ocular rotations can be detected externally with skin electrodes that measure a potential diVerence, whose sign and amplitude depends on the
15
EMG-BASED AND GAZE-TRACKING-BASED INTERFACES
Corneoretinal dipole
Eyeball
−
+ −
EOG [ mV]
+
+ −
200
ϑ
ϑ = 30⬚
100 0
ϑ = 0⬚
ϑ = 0⬚ 1
−100
2 Time [s]
FIG. 8. Schematic drawing of the bioelectric genesis of EOG signals (adapted from Malmivuo and Plonsey, 1995).
rotation: the electrode in the direction of movement becomes positive with respect to the other electrode, as represented in Fig. 8 (Malmivuo and Plonsey, 1995). Two couples of electrodes arranged around the eye (external/internal couple and superior/inferior couple) can be used to detect both horizontal and vertical ocular movements, respectively. Signal amplitudes are typically of the order of 10 mV/degree and the achievable accuracy is 2 (Malmivuo and Plonsey, 1995; Morimoto and Mimica, 2005). Maximum detectable rotations are of approximately 70 , although the linearity of the response progressively worsens beyond 30 (Malmivuo and Plonsey, 1995). Two subdivisions of the EOG can be distinguished: saccadic response and nystagmography (Malmivuo and Plonsey, 1995). A saccadic response consists in a quick rotation of the eye from one fixation point to another, with an angular speed up to 700/s. These fast movements are aimed at rapidly moving the sight to a new visual object in a way that minimizes the transfer time. As an example, in order to follow a target that moves with stepwise jumps, typically the eyes undergo accelerations by reaching the maximum velocity about midway to the target. EOG allows detections of several parameters of saccadic movements, including the maximum angular speed (typically 400/s), the amplitude (typically 20 ), and the duration (typically 80 ms). As an important feature, it is worth stressing that the trajectory and the speed of saccades cannot be altered voluntarily, while they are influenced by fatigue, along with diseases, of course, but also drugs and alcohol (Malmivuo and Plonsey, 1995). Nystagmography is here mentioned for the unique sake of completeness of the discussion, since at present it does not find any use within the field of man–machine interfaces for controlling external devices. In fact, nystagmography has a purely clinical relevance and consist in an evaluation of the response of the visual control system to both vestibular and visual stimuli (Malmivuo and Plonsey, 1995).
16
CARPI AND DE ROSSI
Such a response is known as a nystagmoid movement (or nystagmus), which can be of two diVerent types (according to their origin): vestibular nystagmus and optokinetic nystagmus. The analysis of vestibular nystagmi allows investigations of the vestibular system; in fact, when the vestibular system is damaged, it sends erroneous signals to the oculomotor system and the subject can be aVected by dizziness. As a diVerence, an optokinetic nystagmus originates when multiple targets are in rapid motion with respect to the subject. To keep the image focused on the fovea, a saccadic reflex successively restores the eye to new target positions (Malmivuo and Plonsey, 1995). EOG is today one of the most used gaze-tracking techniques for noninvasive human computer interfaces (Brunner et al., 2007; Ding et al., 2005; Morimoto and Mimica, 2005). 3. Search coil in Scleral Contact Lens This technique consists of a detection of the eye rotation by exploiting electromagnetic induction in a search coil embedded into a flexible contact lens (Fig. 7C). In particular, the user’s gaze is detected by measuring the voltage induced in the search coil by an electromagnetic field generated externally; in fact, the direction and angular displacement of the eye change the polarity and amplitude of the induced voltage (Collewijn et al., 1975; Robinson, 1963). To detect eye movements in two dimension, that is, up/down and left/right rotations, a couple of external electromagnetic sources, arranged along orthogonal directions, can be used; in this case, two electromagnetic fields with diVerent frequency are generated, so as to distinguish (by frequency filtering) each induced voltage within the same search coil. Although very intrusive, search coil-based systems typically have a very high accuracy, about 0.08 (Morimoto and Mimica, 2005). Latest developments of this technology include the useful development of wireless devices (Roberts et al., 2008), so as to avoid the limitations typically introduced by the presence of the wire (Bergamin et al., 2004).
B. CHARACTERISTICS AND ISSUES 1. Corneal Reflection This technique for gaze tracking is highly accurate in suitable conditions (Matsumoto, 2003). Recent improvements have made the setup more accessible and adaptable for desktops and large projection screens. However, head movements are still a limitation for external systems, so that head mounted devices are certainly the best performing (Ciger et al., 2004). In the case of use of a measurement system not solidary with the head, during measurements the head should not move or its movements should be carefully
EMG-BASED AND GAZE-TRACKING-BASED INTERFACES
17
measured and compensated. Alternatively, head mounted devices are uncomfortable, cumbersome and prevent a natural behavior of the user. Another issue worth being reported of this gaze-tracking technique is the need of personalized accurate calibrations (Ciger et al., 2004; Matsumoto, 2003; Morimoto and Mimica, 2005). 2. Electrooculography The above reported description of EOG measurements points out two main features that characterize this technology: it allows recordings with minimal interference with the activities of the subject and it provides minimal discomfort for the subject. Furthermore, EOG recordings can be made in total darkness and/ or with closed eyes. In the opinion of the authors, such advantage might be significant for applications within a manned space module, for instance, since the recoding systems can easily be worn by astronauts and it is not optically aVected by the actual lighting conditions of the environment. Although both horizontal and vertical ocular movements can be recorded, no movements of torsion on the anteroposterior axis can be detected. In fact, these do not modify the dipole and they do not determine therefore potential diVerences them on the derivation electrodes (Patmore and Knapp, 1998). The most important issues relate to the fact that the value of the corneoretinal potential is not constant, since it can vary diurnally and can be aVected by light and muscular fatigue (Malmivuo and Plonsey, 1995). Consequently, there is a need for frequent calibration and recalibration. Additional diYculties may arise, owing to muscle artifacts and the nonlinearity of the technique (Malmivuo and Plonsey, 1995). 3. Scleral Contact Lens/Search Coil This technique is highly accurate, due to the close contact between the eye and the lens equipped with the measurement coil (Collewijn et al., 1975; Morimoto and Mimica, 2005). Despite that, this technique allows a limited measurement range (almost 5 ). Moreover, it is the most intrusive: its discomfort for the user requires the operator to adopt a particular care. Accordingly, in the opinion of the authors it is not realistically applicable for the applications of interest in this book, related to environments like those of spacecrafts.
C. EXAMPLES OF APPLICATIONS Gaze-tracking systems have several application domains, which include psychiatry, cognitive science, behavioral analysis and ophthalmology (Ciger et al., 2004; Conati and Merten, 2007; Herbelin et al., 2007; Malmivuo and Plonsey, 1995). Additionally, applications in the human–computer interaction field are certainly among the most studied at present (Duchowski, 2003; Morimoto and
18
CARPI AND DE ROSSI
Mimica, 2005). A couple of significant examples for the purposes of this book are reported below. 1. Task Selection Systems Computer interfaces represent one of the most studied fields of application of gaze tracking. Many interactive systems based on such a technique have been proposed (Hutchinson et al., 1989; Jacob, 1993). A typical application consists in task selection controlled by eye movements. For example, without using any vocal or manual commands, a user can control a pointing device on a screen and select icons or menus, by looking at them for a while (Norris and Wilson, 1997, Patmore and Knapp, 1998; Zhu and Ji, 2004). The potential benefits for incorporating eye movements into interactions between humans and computers are numerous. For example, reading a user’s gaze may help a computer not only to interpret a user’s request, but also to ascertain some cognitive states of the user (e.g., confusion or fatigue) (Ciger et al., 2004; Conati and Merten, 2007; Herbelin et al., 2007). Corneal reflection and EOG are of course the two technologies of primary choice for such applications. Nevertheless, at present the use of visual-evoked potentials (not discussed in this section) captured from electroencephalography is being investigated as well (Patmore and Knapp, 1998). 2. Attention Tracking Systems Real-time gaze monitoring can also be used to develop so-called attentive interfaces, intended as systems capable of tracking the user’s attention (Vertegaal, 2002; Zhai, 2003). For example, in this type of interfaces gaze can be employed to determine fixation points on a screen, so as to infer what information the user is interested in (Bojko, 2006). Appropriate actions could then be taken; for instance, they might enable the introduction of display changes, according to the spatial or temporal characteristics of eye movements. 3. Vigilance Monitoring Systems Another typical field of application of gaze-tracking systems consists in monitoring the vigilance of specific categories of subjects; for instance, a significant example concerns those at risk of falling suddenly asleep while working, with possibly fatal consequences (e.g., drivers and pilots). The main component of such types of systems consists of a computer vision setup, based on a remotely located device that monitors pupil movements ( Ji and Yang, 2002). Corneal reflection is the most used gaze-tracking techniques in this field, although it can also be advantageously integrated with additional information collected by parallel systems, detecting additional variables such as head position or eye lead movements ( Ji and Yang, 2002).
EMG-BASED AND GAZE-TRACKING-BASED INTERFACES
19
IV. Final Remark
This section described some examples of noninvasive man–machine interfaces, by emphasizing their fundamental features, performances, and applications. In the opinion of the authors, it could be worth studying in depth, for each of these interfaces, its possible use not only as the unique means of bioelectric control of a device, but also in combination with a BMI. In fact a BMI could require and may benefit from an auxiliary system to be used for specific tasks. For instance, this might be the case of astronauts engaged in multitask operations, requiring several activities to be accomplished in parallel. Some tasks may be performed by using signals detected by a BMI, while others, at the same time, may be accomplished by exploiting alternative means of communication enabled by diVerent types of noninvasive interfaces. The challenge is to properly combine these technologies, by making the overall system robust and intelligent. Fulfillment of such issues may open completely new approaches to manage space operations.
References
Ajiboye, A. B., and Weir, R. F. (2005). A heuristic fuzzy logic approach to EMG pattern recognition for multifunctional prosthesis control. IEEE Trans. Neural Syst. Rehabil. Eng. 13(3), 280–291. Bergamin, O., Ramat, S., Straumann, D., and Zee, D. S. (2004). Influence of orientation of exiting wire of search coil annulus on torsion after saccades. Invest. Ophthalmol. Vis. Sci. 45, 131–137. Bishop, C. M. (1995). ‘‘Neural Networks for Pattern Recognition.’’ Oxford University Press, Oxford. Bitzer, S., and van der Smagt, P. (2006). Learning EMG control of a robotic hand: Towards active prostheses. In Proceedings of ICRA, International Conference on Robotics and Automation, Orlando, Florida, USA, May 2006, pp. 2819–2823. Bojko, A. (2006). Using eye tracking to compare Web page designs: A case study. J. Usability Stud. 1(3), 112–120. Brunner, S., Hanke, S., Wassertheuer, S., and Hochgatterer, A. (2007). EOG pattern recognition trial for a human computer interface. In C. Stephanidis, Ed., Universal Access in HCI, Part II, HCII 2007, LNCS 4555, pp. 769–776. Springer, Berlin. Castellini, C., and van der Smagt, P. (2009). Surface EMG in advanced hand prosthetics. Biol. Cybern. 100(1), 35–47. Chan, F. H. Y., Yong-Sheng, Y ., Lam, F. K., Yuan-Ting, Z ., and Parker, P. A. (2000). Fuzzy EMG classification for prosthesis control. IEEE Trans. Rehabil. Eng. 8(3), 305–311. Ciger, J., Herbelin, B., and Thalmann, D. (2004). Evaluation of gaze tracking technology for social interaction in virtual environments. In Proceedings of the Second International Workshop on Modelling and Motion Capture Techniques for Virtual Environments (CAPTECH). Collewijn, H., Van der Mark, F., and Jansen, T. C. (1975). Precise recording of human eye movements. Vis. Res. 15, 447–450. Conati, C., and Merten, C. (2007). Eye-tracking for user modeling in exploratory learning environments: An empirical evaluation. Knowl. Based Syst. 20(6), 557–574.
20
CARPI AND DE ROSSI
Costanza, E., Perdomo, A., Inverso, S. A., and Allen, R. (2004). EMG as a subtle input interface for mobile computing. In Mobile HCI 2004: 6th International Symposium, Glasgow, UK, pp. 426–430. Crawford, B., Miller, K., Shenoy, P., and Rao, R. (2005). Real-time classification of electromyographic signals for robotic control. Technical Report No. 2005-03-05, Department of Computer Science, University of Washington, March 2005. De Luca, C. J. (1988). Electromyography. In ‘‘Encyclopedia of Medical Devices and Instrumentation’’ ( J. G. Webster, Ed.), Wiley, New York. De Luca, C. J. (1997). The use of surface electromyography in biomechanics. J. Appl. Biomechan. 13, 135–163. De Luca, C. J., and Merletti, R. (1988). Surface myoelectric signal cross-talk among muscles of the leg. Electroencephalogr. Clin. Neurophysiol. 69, 568–575. Di Cicco, M., Lucas, L., and Matsuokd, Y. (2004). Comparison of control strategies for an EMG controlled orthotic exoskeleton for the hand. In Proceedings of the 2004 IEEE International Conference on Robotics and Automation, New Orleans, LA, April 2004, pp. 1622–1627. Ding, Q., Tong, K., and Li, G. (2005). Development of an EOG electro-based human-computer interface. In Proceedings of the 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference, Shanghai, China, September 1–4. Duchowski, A. T. (2003). ‘‘Eye Tracking Methodology: Theory and Practice.’’ Springer-Verlag, Berlin. Farina, D., Merletti, R., and Enoka, R. M. (2004). The extraction of neural strategies from the surface EMG. J. Appl. Physiol. 96, 1486–1495. Ferguson, S., and Dunlop, G. R. (2002). Grasp recognition from myoelectric signals. In Proceedings of the Australasian Conference on Robotics and Automation, Auckland, New Zealand. Fukuda, O., Tsuji, T., Kaneko, M., and Otsuka, A. (2003). A human-assisting manipulator teleoperated by EMG signals and arm motions. IEEE Trans. Robot. Autom. 19(2), 210–221. Guyton, A. C., and Hall, J. E. (2005). ‘‘Textbook of Medical Physiology,’’ 10th Edition W. B. Saunders Coompany, Philadelphia. Henneberg, K. (2000). Principles of electromyography. In ‘‘The Biomedical Engineering Handbook’’ ( J. D. Bronzino, Ed.), Second Edition. CRC Press LLC, Boca Raton. Herbelin, B., Grillon, H., De Heras Ciechomski, P., and Thalmann, D. (2007). Coding gaze tracking data with chromatic gradients for VR Exposure Therapy. In Proceedings of the 17th IEEE International Conference on Artificial Reality and Telexistence, pp. 7–14. Hutchinson, T., White, K. Jr., Reichert, K., and Frey, L. (1989). Human–computer interaction using eye-gaze input. IEEE Trans. Syst. Man Cybern. 19, 1527–1533. Jacob, R. (1993). What you look at is what you get. IEEE Comput. 26(7), 65–66. Ji, Q., and Yang, X. (2002). Real-time eye, gaze, and face pose tracking for monitoring driver vigilance. Real Time Imaging Arch. 8(5), 357–377. Kaufman, A., Bandopadhay, A., and Shaviv, B. (1993). An eye tracking computer user interface. In ‘‘Proceedings of the Research Frontier in Virtual Reality Workshop,’’, pp. 78–84. IEEE Computer Society Press, Los Alamitos, CA. Lucas, L., Di Cicco, M., and Matsuoka, Y. (2004). An EMG-controlled hand exoskeleton for natural pinching. J. Robot. Mechatr. 16, 482–488. Malmivuo, J., and Plonsey, R. (1995). ‘‘Bioelectromagnetism – Principles and Applications of Bioelectric and Biomagnetic Fields.’’ Oxford University Press, Oxford. Matsumoto, Y. (2003). ‘‘Real-Time Computer Vision for Human Interfaces.’’ EURON Summer School, Hamburg, Germany. Available online at http://tams-www.informatik.uni-hamburg.de/ lehre/2003ss/euron_summer_school/Matsumoto.09.07.pdf Website visited on October 15, 2008. Micera, S., Sabatini, A. M., and Dario, P. (1998). An algorithm for detecting the onset of muscle contraction by EMG signal processing. Med. Eng. Phys. 20(3), 211–215.
EMG-BASED AND GAZE-TRACKING-BASED INTERFACES
21
Micera, S., Vannozzi, G., Sabatini, A. M., and Dario, P. (2001). Improving detection of muscle activation intervals. IEEE Eng. Med. Biol. Mag. 20(6), 38–46. Morimoto, C. H., and Mimica, M. R. M. (2005). Eye gaze tracking techniques for interactive applications. Comput. Vis. Image Underst. 98(1), 4–24. Mulas, M., Folgheraiter, M., and Gini, G. (2005). An EMG-controlled exoskeleton for hand rehabilitation. In Proceedings of the 9th IEEE International Conference on Rehabilitation Robotics, June 28–July 1, 2005, Chicago, IL, USA, pp. 371–374. Navarro, X., Krueger, T. B., Lago, N., Micera, S., Stieglitz, T., and Dario, P. (2005). A critical review of interfaces with the peripheral nervous system for the control of neuroprostheses and hybrid bionic systems. J. Peripher. Nerv. Syst. 10(3), 229–258. Norris, G., and Wilson, E. (1997). The eye mouse, an eye communication device. In Proceedings of the IEEE 1997 Bioengineering Conference, 21–22 May 1997, pp. 66–67. Patmore, D. W., and Knapp, R. B. (1998). Towards an EOG-based eye tracker for computer control. In Proceedings of the Third International ACM Conference on Assistive Technologies, Marina del Rey, California, pp. 197–203. ACM Press, New York, NY. Reulen, J., Marcus, J. T., Koops, D., de Vries, F., Tiesinga, G., Boshuizen, K., and Bos, J. (1988). Precise recording of eye movement: The iris technique, part 1. Med. Biol. Eng. Comput. 26(1), 20–26. Roberts, D., Shelhamer, M., and Wong, A. (2008). A new ‘‘wireless’’ search-coil system. In Proceedings of the Eye Tracking Research & Application Symposium, ETRA 2008, Savannah, Georgia, USA, March 26–28, pp. 197–204. Robinson, D. A. (1963). A method of measuring eye movements using a scleral search coil in a magnetic field. IEEE Trans. Biomed. Eng. 10, 137–145. Rosen, J., Brand, M., Fuchs, M. B., and Arcan, M. (2001). A myosignal-based powered exoskeleton system. IEEE Trans. Syst. Man Cybern. A 31(3), 210–222. Soares, A., Andrade, A., Lamounier, E., and Carrijo, R. (2003). The development of a virtual myoelectric prosthesis controlled by an EMG pattern recognition system based on neural networks. J. Intell. Inf. Syst. 21(2), 127–141. Tsuji, O. F. T., Kaneko, M., and Otsuka, A. (2003). A human-assisting manipulator teleoperated by EMG signals and arm motions. IEEE Trans. Robot. Autom. 19(2), 210–222. Vertegaal, R. (2002). Designing attentive interfaces. In Proceedings of the Eye Tracking Research & Applications Symposium, New Orleans, LA, pp. 23–30. Webster, J. G. (1997). ‘‘Medical instrumentation: Application and Design,’’ Third Edition Wiley, New York. Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., and Vaughan, T. M. (2002). Brain– computer interfaces for communication and control. Clin. Neurophysiol. 113(6), 767–791. Yoo, D., Kim, J., Lee, B., and Chung, M. (2002). Non contact eye gaze tracking system by mapping of corneal reflections. In Proceedings of the International Conference on Automatic Face and Gesture Recognition, Washington, DC, pp. 94–99. Zecca, M., Micera, S., Carrozza, M. C., and Dario, P. (2002). Control of multifunctional prosthetic hands by processing the electromyographic signal. Crit. Rev. Biomed. Eng. 30(4–6), 459–485. Zhai, S. (2003). What’s in the eyes for attentive input. Commun. ACM 46(3), 34–39. Zhu, Z., and Ji, Q. (2004). Eye and gaze tracking for interactive graphic display. Mach. Vis. Appl. 15(3), 139–148.
BIDIRECTIONAL INTERFACES WITH THE PERIPHERAL NERVOUS SYSTEM
Silvestro Micera*,y and Xavier Navarroz,} *ARTS and CRIM Labs, Scuola Superiore Sant’Anna, I-56127 Pisa, Italy Institute for Automation, Swiss Federal Institute of Technology, CH-8092 Zurich, Switzerland z Institute of Neurosciences, Universitat Auto`noma de Barcelona, E-08193 Bellaterra, Spain } Centro de Investigacio´n Biome´dica en Red sobre Enfermedades Neurodegenerativas (CIBERNED), Spain
y
I. Introduction II. Organization and Function of the PNS III. Nerve Electrodes: Types and Applications A. Extraneural Electrodes B. Intraneural Electrodes C. Regenerative Electrodes IV. Stimulation and Recording Neural Signals A. Stimulation of the PNS B. Recording and Processing Neural Signals V. Biomedical Applications A. Neuroprostheses for CNS Injured Patients B. Cybernetic Prostheses References
Considerable scientific and technological eVorts have been devoted to develop neuroprostheses and hybrid bionic systems that link the human nervous system with electronic or robotic prostheses, with the main aim of restoring motor and sensory functions in disabled patients. Such developments have also the potential to be applied to normal human beings to improve their physical capabilities for bidirectional control and feedback of machines. A number of neuroprostheses use interfaces with peripheral nerves or muscles for neuromuscular stimulation and signal recording. This chapter provides a general overview of the peripheral neural interfaces available and their use from research to clinical application in controlling artificial and robotic prostheses and in developing neuroprostheses. Extraneural electrodes, such as cuV and epineurial electrodes, provide simultaneous interface with many axons in the nerve, whereas intrafascicular, penetrating, and regenerative electrodes may selectively contact small groups of axons within a nerve fascicle. Biological and technical issues are reviewed relative to the problems of electrode design and tissue
INTERNATIONAL REVIEW OF NEUROBIOLOGY, VOL. 86 DOI: 10.1016/S0074-7742(09)86002-9
23
Copyright 2009, Elsevier Inc. All rights reserved. 0074-7742/09 $35.00
24
MICERA AND NAVARRO
injury. The last sections review diVerent strategies for the use of peripheral neural interfaces in biomedical applications.
I. Introduction
The possibility of interfacing and controlling artificial prostheses and machines with biological signals has a long history of multidisciplinary research. Many scientific and technological eVorts have been devoted to develop bionic systems that link, via neural interfaces, the human nervous system with electronic and robotic prostheses, with the main aim of restoring motor and sensory functions in patients with spinal cord injuries, brain injuries, neurodegenerative diseases, or limb amputations (Chapin and Moxon, 2000). A number of such neuroprostheses include interfacing the peripheral nervous system (PNS) by means of electrodes that allow neuromuscular stimulation and neural signal recording. Several architectures have been developed and tested: (1) functional electrical stimulation (FES) systems to artificially replace central motor control and directly stimulate the intact peripheral nerves or muscles of patients with central nervous system (CNS) injuries; (2) artificial prostheses aimed at substituting lost parts of the body; (3) exoskeletons intended to augment or restore reduced human capabilities; (4) teleoperated robots to carry out tasks in environments where it is not possible the direct intervention of human beings. The combination of the artificial system with the human–machine interface (HMI) is often called ‘‘hybrid bionic system’’ (HBS). It is characterized by three main attributes (Fig. 1, Micera et al., 2006): (i) level of Hybridness: the system to be controlled can be a prosthesis, an exoskeleton, a personal computer or a scalable robotic alias; (ii) level of Augmentation: the motor and/or sensory channels of the user that can be involved in the development of the HBS; (iii) level of Connection: multimodal devices or (invasive or noninvasive) interfaces with diVerent parts of the nervous system can be used. Sophisticated prostheses and robotic devices ask for HMIs able to take full advantage of their potentials. Therefore, a fast, intuitive, reliable, and bidirectional flow of information between the nervous system of the user and the robotic device needs to be established. In recent years, various types of HBSs have been developed for such purpose (Dhillon et al., 2004; Hochberg et al., 2006; Navarro et al., 2005; Rutten, 2002; Velliste et al., 2008; Warwick et al., 2003). Among the possible choices, interfaces with the PNS are interesting because they represent a trade-oV between a potentially good ability to restore a natural link with the nervous system and a reduced invasiveness. For example, in the case of a cybernetic prosthesis, the interface should be able to stimulate diVerent aVerent
25
PERIPHERAL NERVE INTERFACES
Hybridness Prosthesis
Exoskeleton
Robotic alias
Connectivity Indirect
Direct to PNS
Direct to CNS
Sensory or motor Sensory and motor Sensory-motor and cognitive
Augmentation FIG. 1. Examples of diVerent systems with diVerent levels of Hybridness, Augmentation, and Connection.
nerves to deliver sensory feedback information originating from sensors in the prosthesis, and to record signals from eVerent nerves or from muscles to derive motor commands to the prosthesis. Similarly, kinematic and kinetic information for the closed-loop control of a neuroprosthesis could be detected from signals originating from natural sensors intercepted by the neural interface. Starting from these needs, several neural interfaces have been developed with diVerent characteristics (Fig. 2). The aim of this chapter is to describe the potentials and limits of the use of PNS neural interfaces to develop advanced HBSs.
II. Organization and Function of the PNS
The PNS is constituted by neurons whose cell bodies are located in the spinal cord or within spinal ganglia, their central connections, and their axons, which extend through peripheral nerves to reach target organs. Peripheral nerves contain several types of nerve fibers (Table I). AVerent sensory fibers terminate
26
MICERA AND NAVARRO
Regenerative
PNS electrodes
Intraneural
Invasivity
d
Intrafascicular
c
b Extraneural
a Selectivity FIG. 2. The diVerent types of electrodes applied to interface peripheral nerves classified regarding invasiveness and selectivity. Micrographs show examples of hybrid silicone–polyimide cuV electrode (FraunhoVer IBMT) polyimide tfLIFE (FraunhoVer IBMT), silicon-based Utah MEA (Cyberkinetics), and polyimide sieve electrode (FraunhoVer IBMT).
TABLE I GENERAL CLASSIFICATION OF THE PERIPHERAL NERVE FIBERS
Fiber type Myelinated A A A A B Unmyelinated sC dC
Function
Diameter (m)
Conduction velocity (m/s)
Alpha-motor eVerents, Proprioceptive aVerents Tactile, proprioceptive aVerents Gamma-motor eVerents Pain, cold aVerents Preganglionic autonomic eVerents
12–22 6–12 3–5 2–5 1–5
60–120 40–70 30–45 10–30 3–15
Postganglionic autonomic eVerents Thermal, pain, mechanical aVerents
0.3–1.3 0.3–1.3
0.7–2.3 0.5–2.0
at the periphery either as free endings or in specialized sensory receptors in the skin, the muscle, and deep tissues. Sensory fibers convey various classes of sensory inputs, mainly mechanical, thermal, and noxious stimuli. EVerent motor fibers
PERIPHERAL NERVE INTERFACES
27
originate from motoneurons in the spinal cord and end in neuromuscular junctions in skeletal muscles. The majority can be divided into alpha-motor fibers that innervate skeletal extrafusal muscle fibers, and gamma-motor fibers that innervate the spindle muscle fibers. EVerent autonomic nerve fibers in somatic peripheral nerves are unmyelinated postganglionic sympathetic fibers that innervate smooth muscle and glandular targets. Most of the somatic peripheral nerves are mixed, providing motor, sensory, and autonomic innervation to the corresponding projection territory. Nerve fibers, both aVerent and eVerent, are grouped in fascicles that eventually give origin to branches that innervate distinct targets, muscular, cutaneous, or visceral. Peripheral nerves are organized somatotopically and functionally at the fascicular level. The fascicular architecture changes throughout the length of the nerve, with an increasing number of fascicles of smaller size in distal with respect to proximal segments. Fascicles innervating a given target remain well localized within the nerve for some long distances (Brushart, 1991), thus facilitating the selective interface of diVerent fascicles within a common nerve (Branner et al., 2001, Veraart et al., 1993). Peripheral nerves are composed of three supportive sheaths: epineurium, perineurium, and endoneurium (Peters et al., 1991) (Fig. 3). The epineurium is the outermost layer, composed of loose connective tissue and carries the blood vessels supplying the nerve. The perineurium that surrounds each fascicle in the nerve is composed of inner layers of flat perineurial cells and an outer layer of collagen fibers. The endoneurium is composed of collagen and reticular fibers and an extracellular matrix occupying the space between nerve fibers within the fascicle. The endoneurial collagen fibrils form the walls of the endoneurial tubules, in which axons are accompanied by Schwann cells, which either myelinate or just surround them. The actions of the body are controlled by neural signals conducted by eVerent nerve fibers to activate diVerent muscles. Each spinal motoneuron makes synaptic contacts with a number of muscle fibers, constituting a motor unit. Graded contraction of each muscle is produced by increasing the number of motor units activated, and by increasing the frequency of impulses to each motor unit. Recruitment of motor units follows a size-dependent order, with slow fatigueresistant motor units activated first and large fast fatigue motor units activated only at high levels of tension (Henneman et al., 1974). On the other hand, the information transduced by the natural receptors is conducted to the CNS by the aVerent nerve fibers. Each somatic sensory neuron is specified to a sensory modality, touch, proprioception, temperature, or pain, depending upon the specialized terminal receptor. Each sensory neurons subsides a receptive field in the peripheral tissue, of variable size according to the body segment. Signals are transmitted by the corresponding axons in series of action potentials, with intensity of the signal mainly coded by impulse frequency.
28
MICERA AND NAVARRO
A
B
C
FIG. 3. Structure of the peripheral nerve. (A) Transverse section of the rat sciatic nerve showing subfascicles of nerve fibers. The endoneurial compartment is encircled by the perineurium and an outer loose epineurium. (B) Semithin cross section showing at high magnification myelinated fibers of diVerent sizes. (C) Electron microscopy section showing a small myelinated fiber surrounded by collagen fibrils and flat fibroblastic cells. Bars ¼ 100 m in (A); 20 m in (B); 2 m in (C).
III. Nerve Electrodes: Types and Applications
Most peripheral nerve interfaces use an electrical coupling method to detect the bioelectrical activity of the nerve fibers and/or to induce their excitation. Thus, most nerve electrodes are implanted around or within a peripheral nerve or
PERIPHERAL NERVE INTERFACES
29
spinal root to reduce tissue resistance and stimulus intensity. Nerve electrodes can be classified into three main classes: extraneural, intraneural, and regenerative. The selectivity of stimulation or recording individual nerve fibers increases with the invasiveness of the electrode implantation (Fig. 2).
A. EXTRANEURAL ELECTRODES CuV electrodes are composed of an insulating tubular sheath that encircles the nerve and contains electrode contacts exposed at the inner surface that are connected to lead wires (Fig. 2a). CuV electrodes placed around the nerves allow for more precise positioning and reduced stimulus intensity compared to surface and epimysial electrodes (Loeb and Peck, 1996). In comparison with more invasive, penetrating and regenerative electrodes, cuV electrodes of spiral type are less prone to damage the nerve and easier to implant (Naples et al., 1990). However, because they are placed around the nerve cuV electrodes have a reduced selectivity; they can record only mixed sensorimotor activity of the enclosed fibers whenever a mixed nerve is utilized. The stimulation of as well as the activity recorded from a nerve with cuV electrodes is dominated by the activation of large myelinated fibers and of those located at superficial locations. Multisite cuV electrodes (Tarler and Mortimer, 2004; Veraart et al., 1993; Walter et al., 1997), innovative cuV structures (Tyler and Durand, 1997), and advanced processing algorithms (Cavallaro et al., 2003; Tesfayesus and Durand, 2006) have increased cuVs selectivity. The flat interface nerve electrode (FINE) is an extraneural electrode designed to reshape peripheral nerves into a favorable geometry for selective stimulation (Leventhal and Durand, 2003; Tyler and Durand, 2002). By flattening the nerve, fascicles become more accessible and central fibers are moved closer to the electrode contacts in comparison with cylindrical cuVs. The surface area of the nerve is also enlarged, increasing the interface surface and allowing more contacts placed around the nerve. Studies in laboratory animals with FINEs implanted over months showed that electrodes applying small forces did not cause detectable changes in nerve physiology and histology. However, high reshaping forces can induce nerve damage (Tyler and Durand, 2003).
B. INTRANEURAL ELECTRODES Longitudinal intrafascicular electrodes (LIFEs), constructed from thin insulated conducting wires (such as Pt–Ir or metalized polymers), are inserted longitudinally into the nerve to lay in-between and parallel to the nerve fibers (Lawrence et al., 2004; Yoshida and Horch, 1993; Yoshida and Stein, 1999). Gathering signals
30
MICERA AND NAVARRO
from only a small number of axons, they allow more selective interfacing than extraneural electrodes that wrap the whole nerve. LIFE electrodes are less invasive than intraneural multielectrode arrays (MEAs) that are inserted transversely into the nerve leading to a higher risk of damage. Recently, thin-film LIFEs (tfLIFE) have been fabricated on micropatterned polyimide substrate, allowing several contacts in one device and selective multiunit nerve recording and stimulation (Navarro et al., 2007). MEAs are composed of tens of needles inserted transversally into the nervous system and developed on materials such as silicon, glass, or polyimide (Kipke et al., 2003; Nordhausen et al., 1996; Rutten et al., 1999). They have been mainly used as CNS microelectrodes to record or stimulate the brain cortex (Hochberg et al., 2006; Velliste et al., 2008). MEAs have been also tested in peripheral nerves in experimental works with animal models and also in a human volunteer (Warwick et al., 2003). Recently, a modified version (named Utah Slanted Electrode Array, USEA) has been used within the PNS, allowing selective recording of single unit responses and low-current highly selective stimulation of motor fibers (Branner et al., 2001, 2004). MEAs present the advantage of a high number of electrical contacts. However, they have also some limitations, such as the rigid structure of the electrodes and the tethering forces by lead wires that may damage the nerves during movements, and the lack of stability of recorded signals over time during chronic studies (Branner et al., 2004; Warwick et al., 2003).
C. REGENERATIVE ELECTRODES Regeneration-type (or sieve) electrodes are designed to interface a high number of nerve fibers by using an array of holes with electrodes around them (Fig. 2d), implanted between the severed stumps of a nerve. Regenerating axons grow through the holes, making it possible to record action potentials from and to stimulate individual axons or small groups. The most logical and challenging application of regenerative electrodes is the implantation in severed nerves of an amputee limb for a bidirectional interface with a prosthesis. They may allow interfacing with the injured axons that originally innervated the lost limb. However, regenerative electrodes can only be applied to transected peripheral nerves and need time for regenerating axons to grow through the interface, thus precluding acute experiments. Although promising results on the use of regenerative electrodes have been achieved in experimental models (Bradley et al., 1997; Kovacs et al., 1994; Navarro et al., 1998), some challenges remain limiting their clinical usability (Lago et al., 2005). Axonal regeneration through polyimide sieve electrodes occurs in most of the animals implanted, with higher quality and quantity than reported for the earlier used silicon-based sieve electrodes, but still in some animals there appeared signs of compressive axonopathy at the
PERIPHERAL NERVE INTERFACES
31
sieve electrode level on the long term (Lago et al., 2005). Nevertheless, stimulation of small numbers of regenerated fibers is feasible using regenerative electrodes, and by matching regenerated axons with original receptive fields an adequate sensory feedback might be delivered (Lago et al., 2007).
IV. Stimulation and Recording Neural Signals
PNS electrodes can be used for stimulation of nerve fibers as well as for recording of neural impulses, constituting a bidirectional interface with the nervous system.
A. STIMULATION OF THE PNS From a functional point of view, cuV electrodes can be used to stimulate the enclosed nerve leading to activation of eVerent motor or autonomic nerve fibers. Simple configurations are bipolar and tripolar, which reduce current leaks out of the cuV. Nerve stimulation produces larger movements and more reproducible than intramuscular or intraspinal stimulation (Aoyagi et al., 2004). In addition, the stimulus current required to activate nerve axons by stimulation with epineurial or cuV electrodes is much lower than that required for intramuscular stimulation. Multichannel cuV electrodes enable selective stimulation of separate axonal fascicles within the nerve, each one supplying innervation to a diVerent muscle (Navarro et al., 2001; Tarler and Mortimer, 2004; Veraart et al., 1993). The reduced size and thickness of polymer cuVs opens the possibility of implanting several small cuVs around diVerent branches of nerves, thus achieving selective functional stimulation of a higher number of muscles (Stieglitz et al., 2000). Short pulse widths and subthreshold transverse currents from a steering anode in the cuV can significantly improve the selectivity of stimulation by restricting the region of excitation of the nerve trunk. Large myelinated fibers are activated before small myelinated and unmyelinated fibers when applying electrical stimulation to the nerve. This is advantageous for stimulation of aVerent fibers to provide sensory feedback, since tactile or position sensations can be elicited without concomitantly evoking pain. On the contrary, large motor fibers innervating fast fatigue motor units are activated earlier than thinner motor fibers controlling slow fatigue-resistant motor units, resulting into an inverse recruitment that causes fast muscle fatigue. Strategies for achieving a more physiological recruitment order include the application of anodal blocking and of quasi-trapezoidal pulses (Fang and Mortimer, 1991).
32
MICERA AND NAVARRO
FINE electrodes have been also used to stimulate the PNS in acute experiments showing that it is possible to selectively activate individual fascicles of the cat sciatic nerve, as well as groups of fibers within the fascicles, and revealed the strong dependency of selectivity on the relative locations of the fascicle and the electrode contacts (Leventhal and Durand, 2003; Tyler and Durand, 2002). Intrafascicular metal and polymer-based electrodes have been shown to be valuable for functional neuromuscular stimulation with high selective characteristics (Yoshida et al., 2000). LIFEs were able to produce equivalent levels of activation using stimulation levels that were an order of magnitude smaller than with hook electrodes. LIFEs can be used to activate nerve fibers in diVerent fascicles independently of each other, and also to activate subsets of axonal populations within a single fascicle (Yoshida and Horch, 1993). Microtechnologies allowing to place multiple contacts in one electrode device for stimulation or recording of individual nerve fibers can considerably increase the number of channels and resolution whereas minimizing the invasiveness and the number of electrodes to implant in a nerve. The nerve stimulation capabilities of multisite USEA (Branner et al., 2001) and of tfLIFE (Navarro et al., 2007) have been assayed in animal nerves.
B. RECORDING AND PROCESSING NEURAL SIGNALS The possibility of extracting potential actions from neural signal recordings is an important issue for the development of HBSs. The dynamic time-variant properties of the musculoskeletal system makes desirable to develop closed-loop FES systems. Feedback information can be gathered by using external (Carpaneto et al., 2003) or implantable (Cavallaro et al., 2005; Johnson et al., 1999) artificial sensors, or by processing electroneurographic (ENG) signals recorded by means of electrodes in the PNS, and used to correct deviations caused by unexpected changes and nonlinearities. The processing of neural signals is related to the type of electrodes used. Because of the insulating properties of perineurial and epineurial layers, electrodes placed inside a peripheral nerve allow enhanced selectivity with respect to extraneural electrodes and increase the signal-to-noise ratio (SNR) of recordings. With extraneural electrodes (e.g., cuV signals), the contribution of single axons cannot be extracted because of the low SNR and the overlapping between the frequency range of the signals (few hundred Hz to a few kHz) and the noise. In most cases, the use of recorded neural activity is limited to the onset detection for the closed-loop control of FES systems (Haugland et al., 1994; Inmann and Haugland, 2004) and for the control of hand prostheses (Stein et al., 1980). Nevertheless, pattern recognition techniques allow identifying complex motor commands from the compound recorded signal, and kinematic information has
PERIPHERAL NERVE INTERFACES
33
been extracted from ENG signals recorded from aVerent activities (Cavallaro et al., 2003; Micera et al., 2001). With intraneural electrodes (e.g., LIFE and MEA), it is possible to record spikes from diVerent nerve fibers. Such signals have been used (Aoyagi et al., 2003; McNaughton and Horch, 1994; Mirfakhraei and Horch, 1997) to diVerentiate single units in multiunit peripheral nerve recordings using diVerent features and classification schemes. Moreover, as individual spikes can be resolved in aVerent activity from natural sensors, the interspike interval can provide meaningful feedback to FES systems (Yoshida and Horch, 1996). The use of discrete and continue wavelet transform was recently applied to ENG signals recorded using LIFEs demonstrating that LIFEs were able to record spikes and to sort diVerent classes of spikes in a robust way for several weeks, and that it was possible to use the extracted spikes to identify diVerent neural stimuli (Citi et al., 2008).
V. Biomedical Applications
Although most interfaces were originally developed for FES systems, they can also be the key component of neurocontrolled prostheses and robotic machines (Table II).
A. NEUROPROSTHESES FOR CNS INJURED PATIENTS Nowadays, the most frequent use of PNS interfaces resides in FES. FES systems have been developed in order to artificially replace the central motor control and directly stimulate the intact peripheral nerves or muscles of spinal cord, or brain injured patients, attempting to generate movements or functions that mimic normal actions. There are clinical applications in a variety of systems designed to control micturition and defecation by stimulating the sacral roots (Brindley, 1994; RijkhoV, 2004), for phrenic nerve pacing in ventilatory assistance (Chervin and Guilleminault, 1997; Creasey et al., 1996), for treating neuropathic pain by stimulating somatic nerves (Stanton-Hicks and Salamon, 1997), for activation of lower extremity movements (Burridge et al., 2007; Triolo et al., 1996; Waters et al., 1985), and for control of hand movements (Kilgore, 1997; Peckham and Keith, 1992) by stimulating paralyzed muscles or nerves. DiVerent studies have demonstrated that patients are able to use such systems for activities of daily living and enhancing their quality of life and independency. The introduction of closed-loop control by recorded sensory neural activity related to the stimulated actions is attempting to improve the usability and functionality of FES systems (Inmann and Haugland, 2004; Sinkjaer et al., 2003).
34
MICERA AND NAVARRO
TABLE II MAIN TYPES OF ELECTRODES USED FOR INTERFACING THE PNS IN BIOMEDICAL APPLICATIONS Electrode Type
Mode
No.
Contact site
Application
Epineurial Epineurial Epineurial Epineurial Helical Book CuV CuV CuV FINE LIFE MEA Regenerative
stim stim stim stim stim stim stim rec stim rec/stim rec/stim rec/stim rec/stim
4 2 2 16–25 1 3 1 1 4 13 1–4 20–100 10–30
Phrenic nerve Peroneal nerve Somatic nerves Retina/ganglion Cells Vagus nerve Sacral spinal roots Peroneal nerve Sural nerve Optic nerve Peripheral nerve Peripheral nerve Nerve, brain cortex Injured nerve
Breathing Foot drop Pain relieve Blindness Epilepsy, sleep apnea Bladder control FES FES control Blindness Artificial limb control Motion control
Mode: rec, recording; stim, stimulation. FINE, flat interface nerve electrode; LIFE, longitudinal intrafascicular electrode; MEA, multielectrode array.
B. CYBERNETIC PROSTHESES The restoration of sensorimotor functions to those who lost limbs due to disease, trauma, or amputation is also an active field of research. Commercial prosthetic limb devices are unable to provide enough functionality and to deliver appropriate sensory feedback to the user so as to functionally replace the lost limb. In recent years, Kuiken and colleagues have developed a novel interface called targeted reinnervation (Kuiken et al., 2007; Miller et al., 2008). The amputated nerves that originally provided innervation to the missing limb are surgically transferred to innervate other arm and chest muscles that remain after the amputation. Once reinnervated, these muscles produce electromyographic (EMG) signals that now correspond to the original arm motor orders and can be used to control several degrees of freedom of the prosthesis. Concurrently, sensory nerves forced to reinnervate the skin overlying the target muscles may provide a pathway for sensory information of the amputated arm. This approach, however, is still dependent on surface electrodes and limited to a few EMG signals. Future perspectives rely on directly interfacing the amputated nerves by multipolar electrodes that may re-create the bidirectional link between the user’s nervous system and the prosthesis in a more physiologically based manner.
PERIPHERAL NERVE INTERFACES
35
In the recent past LIFE electrodes have been used to this aim in selected chronic amputees (Dhillon et al., 2004, 2005). LIFEs were implanted into median or ulnar nerves proximal to the stump. The results indicated that the motor signals recorded using LIFEs could be used to control prosthetic systems. During these trials the possibility of stimulating aVerent nerves to provide sensory feedback to the subjects was also investigated. Subjects were able to feel tactile or proprioceptive sensations localized to individual phantom digits elicited through diVerent electrodes (Dhillon et al., 2005). In some cases plastic stimulus-induced reorganization on somatosensory cortex by the aVerent stimulation made the localization of elicited sensations better defined with time. Interesting results were also achieved with a MEA implanted in the median nerve of an able-bodied subject (Warwick et al., 2003). An eVective bidirectional link between sensorized dexterous hand prosthesis and the nervous system could be achieved. In particular, the ENG-based control was more natural than the standard EMG-based approach. The sensory feedback allowed the subject to minimize the required grasping force after training. However, after the 96-day trial only three channels (over 100) were still working, due to mechanical fatigue of the connection wires.
References
Aoyagi, Y., Stein, R. B., Branner, A., Pearson, K. G., and Normann, R. A. (2003). Capabilities of a penetrating microelectrode array for recording single units in dorsal root ganglia of the cat. J. Neurosci. Methods 128, 9–20. Aoyagi, Y., Mushahwar, V. K., Stein, R. B., and Prochazka, A. (2004). Movements elicited by electrical stimulation of muscles, nerves, intermediate spinal cord, and spinal roots in anesthetized and decerebrate cats. IEEE Trans. Neural Syst. Rehabil. Eng. 12, 1–11. Bradley, R. M., Cao, X., Akin, T., and Najafi, K. (1997). Long term chronic recordings from peripheral sensory fibers using a sieve electrode array. J. Neurosci. Methods 73, 177–186. Branner, A., Stein, R. B., and Normann, R. A. (2001). Selective stimulation of cat sciatic nerve using an array of varying-length microelectrodes. J. Neurophysiol. 85, 1585–1594. Branner, A., Stein, R. B., Ferna´ndez, E., Aoyagi, Y., and Normann, R. A. (2004). Long-term stimulation and recording with a penetrating microelectrode array in cat sciatic nerve. IEEE Trans. Biomed. Eng. 51, 146–157. Brindley, G. S. (1994). The first 500 patients with sacral anterior root stimulator implants: General description. Paraplegia 32, 795–805. Brushart, T. M. (1991). Central course of digital axons within the median nerve of Macaca mulatta. J. Comp. Neurol. 311, 197–209. Burridge, J. H., Haugland, M., Larsen, B., Pickering, R. M., Svaneborg, N., Iversen, H. K., Christensen, P. B., Haase, J., Brennum, J., and Sinkjaer, T. (2007). Phase II trial to evaluate the ActiGait implanted drop-foot stimulator in established hemiplegia. J. Rehabil. Med. 39, 212–218. Carpaneto, J., Micera, S., Zaccone, F., Vecchi, F., and Dario, P. (2003). A sensorized thumb for force closed-loop control of hand neuroprostheses. IEEE Trans. Neural Syst. Rehabil. Eng. 11, 346–353.
36
MICERA AND NAVARRO
Cavallaro, E., Micera, S., Dario, P., Jensen, W., and Sinkjaer, T. (2003). On the intersubject generalization ability in extracting kinematic information from aVerent nervous signals. IEEE Trans. Biomed. Eng. 50, 1063–1073. Cavallaro, E., Cappiello, G., Micera, S., Carrozza, M. C., Rantanen, P., and Dario, P. (2005). On the development of a biomechatronic system to record tendon sliding movements. IEEE Trans. Biomed. Eng. 52, 1110–1119. Chapin, J. K., and Moxon, K. A. (2000). ‘‘Neural Prosthesis for Restoration of Sensory and Motor Function.’’ CRC Press, Boca Raton. Chervin, R. D., and Guilleminault, C. (1997). Diaphragm pacing for respiratory insuYciency. J. Clin. Neurophysiol. 14, 369–377. Citi, L., Carpaneto, J., Yoshida, K., HoVmann, K. P., Koch, K. P., Dario, P., and Micera, S. (2008). On the use of wavelet denoising and spike sorting techniques to process electroneurographic signals recorded using intraneural electrodes. J. Neurosci. Methods 172, 294–302. Creasey, G., Elefteriades, J., DiMarco, A., Talonen, P., Bijak, M., Girsch, W., and Kantor, C. (1996). Electrical stimulation to restore respiration. J. Rehabil. Res. Dev. 33, 123–132. Dhillon, G. S., Lawrence, S. M., Hutchinson, D. T., and Horch, K. (2004). Residual function in peripheral nerve stumps of amputees: Implications for neural control of artificial limbs. J. Hand Surg. 29A, 605–615. Dhillon, G. S., Kruger, T., Sandhu, J., and Horch, K. (2005). EVects of short term training on sensory and motor function in severed nerves of long term human amputees. J. Neurophysiol. 93, 2625–2633. Fang, Z. P., and Mortimer, J. T. (1991). A method to eVect physiological recruitment order in electrically activated muscle. IEEE Trans. Biomed. Eng. 38, 175–179. Haugland, M., HoVer, J., and Sinkjaer, T. (1994). Skin contact force information in sensory nerve signals recorded by implanted cuV electrodes. IEEE Trans. Rehabil. Eng. 2, 18–28. Henneman, E., Clamann, H. P., Gillies, J. D., and Skinner, R. D. (1974). Rank order of motoneurons within a pool: Law of combination. J. Neurophysiol. 37, 1338–1349. Hochberg, L., Serruya, S., Friehs, G., Mukand, J., Saleh, M., Caplan, A., Branner, A., Chen, D., Penn, R., and Donoghue, J. (2006). Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442, 164–171. Inmann, A., and Haugland, M. (2004). Implementation of natural sensory feedback in a portable control system for a hand grasp neuroprosthesis. Med. Eng. Phys. 26, 449–458. Johnson, M. W., Peckham, P. H., Bhadra, N., Kilgore, K. L., Gazdik, M. M., Keith, M. W., and Strojnik, P. (1999). Implantable transducer for two-degree of freedom joint angle sensing. IEEE Trans. Rehabil. Eng. 7, 349–359. Kilgore, K. L. (1997). An implanted upper-extremity neuroprosthesis. J. Bone Joint Surg. 79A, 533–541. Kipke, D. R., Vetter, R. J., Williams, J. C., and Hetke, J. F. (2003). Silicon-substrate intracortical microelectrode arrays for long-term recording of neuronal spike activity in cerebral cortex. IEEE Trans. Rehabil. Eng. 11, 151–155. Kovacs, G. T. A., Storment, C. W., Halks-Miller, M., Belczynski, C. R., Della Santina, C. C., Lewis, E. R., and Maluf, N. I. (1994). Silicon-substrate microelectrode array for parallel recording of neural activity in peripheral and cranial nerves. IEEE Trans. Biomed. Eng. 41, 567–577. Kuiken, T. A., Marasco, P. D., Lock, B. A., Harden, R. N., and Dewald, J. P. (2007). Redirection of cutaneous sensation from the hand to the chest skin of human amputees with targeted reinnervation. Proc. Natl. Acad. Sci. USA 104, 20061–20066. Lago, N., Ceballos, D., Rodrı´guez, F. J., Stieglitz, T., and Navarro, X. (2005). Long term assessment of axonal regeneration through polyimide regenerative electrodes to interface the peripheral nerve. Biomaterials 26, 2021–2031. Lago, N., Udina, E., Ramachandran, A., and Navarro, X. (2007). Neurobiological assessment of regenerative electrodes for bidirectional interfacing injured peripheral nerves. IEEE Trans. Biomed. Eng. 54, 1129–1137.
PERIPHERAL NERVE INTERFACES
37
Lawrence, S. M., Dhillon, G. S., Jensen, W., Yoshida, K., and Horch, K. W. (2004). Acute peripheral nerve recording characteristics of polymer-based longitudinal intrafascicular electrodes. IEEE Trans. Neural Syst. Rehabil. Eng. 12, 345–348. Leventhal, D. K., and Durand, D. M. (2003). Subfascicle stimulation selectivity with the flat interface nerve electrode. Ann. Biomed. Eng. 31, 643–652. Loeb, G. E., and Peck, R. A. (1996). CuV electrodes for chronic stimulation and recording of peripheral nerve. J. Neurosci. Methods 64, 95–103. McNaughton, T. G., and Horch, K. W. (1994). Action potential classification with dual channel intrafascicular electrodes. IEEE Trans. Biomed. Eng. 41, 609–616. Micera, S., Jensen, W., Sepulveda, F., Riso, R. R., and Sinkjaer, T. (2001). Neuro-fuzzy extraction of angular information from muscle aVerents for ankle control during standing in paraplegic subjects: An animal model. IEEE Trans. Biomed. Eng. 48, 787–794. Micera, S., Carrozza, M. C., Beccai, L., Vecchi, F., and Dario, P. (2006). Hybrid bionic systems for the replacement of hand function. Proc. IEEE 94, 1752–1762. Miller, L. A., Stubblefield, K. A., Lipschutz, R. D., Lock, B. A., and Kuiken, T. A. (2008). Improved myoelectric prosthesis control using targeted reinnervation surgery: A case series. IEEE Trans. Neural Syst. Rehabil. Eng. 16, 46–50. Mirfakhraei, K., and Horch, K. (1997). Recognition of temporally changing action potentials in multiunit neural recordings. IEEE Trans. Biomed. Eng. 44, 123–131. Naples, G. G., Mortimer, J. T., and Yuen, T. G. H. (1990). Overview of peripheral nerve electrode design and implantation. In ‘‘Neural Prostheses: Fundamental Studies’’ (W. F. Agnew and D. B. McCreery, Eds.), pp. 107–144. Prentice-Hall, New Jersey. Navarro, X., Calvet, S., Rodrı´guez, F. J., Stieglitz, T., Blau, C., Butı´, M., Valderrama, E., and Meyer, J. U. (1998). Stimulation and recording from regenerated peripheral nerves through polyimide sieve electrodes. J. Peripher. Nerv. Syst. 2, 91–101. Navarro, X., Valderrama, E., Stieglitz, T., and Schuttler, M. (2001). Selective fascicular stimulation of the rat sciatic nerve with mutipolar polyimide cuV electrodes. Restor. Neurol. Neurosci. 18, 9–21. Navarro, X., Krueger, T., Lago, N., Micera, S., Stieglitz, T., and Dario, P. (2005). A critical review of interfaces with the peripheral nervous system for the control of neuroprostheses and hybrid bionic systems. J. Peripher. Nerv. Syst. 10, 229–258. Navarro, X., Lago, N., Vivo´, M., Yoshida, K., Koch, K. P., Poppendieck, W., and Micera, S. (2007). Neurobiological evaluation of thin-film longitudinal intrafascicular electrodes as a peripheral nerve interface. In Proc. IEEE 10th Int. Conf. Rehabil. Robotics, pp. 643–649. Nordhausen, C. T., Maynard, E. M., and Normann, R. A. (1996). Single unit recording capabilities of a 100 microelectrode array. Brain Res. 726, 129–140. Peckham, H. P., and Keith, M. W. (1992). Motor prostheses for restoration of upper extremity function. In ‘‘Neural Prostheses. Replacing Motor Function After Disease or Disability’’ (R. B. Stein, P. H. Peckham, and D. B. Popovic, Eds.), pp. 162–187. Oxford University Press, New York. Peters, A., Palay, S. L., and Webster, H. F. (1991). ‘‘The Fine Structure of the Nervous System: Neurons and their Supporting Cells’’, 3rd Ed. Oxford University Press, New York. RijkhoV, N. J. M. (2004). Neuroprostheses to treat neurogenic bladder dysfunction: Current status and future perspectives. Child Nerv. Syst. 20, 75–86. Rutten, W. L. C. (2002). Selective electrical interfaces with the nervous system. Annu. Rev. Biomed. Eng. 4, 407–452. Rutten, W. L. C., Smit, J. P. A., Frieswijk, T. A., Bielen, J. A., Brouwer, A. L. H., Buitenweg, J. R., and Heida, C. (1999). Neuro-electronic interfacing with multielectrode arrays. IEEE Eng. Med. Biol. 18, 47–55. Sinkjaer, T., Haugland, M., Inmann, A., Hansen, M., and Nielsen, K. D. (2003). Biopotentials as command and feedback signals in functional electrical stimulation systems. Med. Eng. Phys. 12, 29–40.
38
MICERA AND NAVARRO
Stanton-Hicks, M., and Salamon, J. (1997). Stimulation of the central and peripheral nervous system for the control of pain. J. Clin. Neurophysiol. 14, 46–62. Stein, R. B., Charles, D., HoVer, J. A., Arsenault, J., Davis, L. A., Moorman, S., and Moss, B. (1980). New approaches for the control of powered prostheses particularly by high-level amputees. Bull. Prosthet. Res. 10–33, 51–62. Stieglitz, T., Beutel, H., Schuettler, M., and Meyer, J. U. (2000). Micromachined, polyimide-based devices for flexible neural interfaces. Biomed. Microdev. 2, 283–294. Tarler, M., and Mortimer, J. (2004). Selective and independent activation of four motor fascicles using a four contact nerve-cuV electrode. IEEE Trans. Neural Syst. Rehab. Eng. 12, 251–257. Tesfayesus, W., and Durand, D. M. (2006). Blind source separation of neural recordings and control signals. Conf. Proc. IEEE Eng. Med. Biol. Soc. 1, 731–734. Triolo, R. J., Bieri, C., Uhlir, J., Kobetic, R., Scheiner, A., and Marsolais, E. B. (1996). Implanted FNS systems for assisted standing and transfers for individuals with cervical spinal cord injuries. Arch. Phys. Med. Rehabil. 77, 1119–1128. Tyler, D. J., and Durand, D. M. (1997). A slowly penetrating interfascicular nerve electrode for selective activation of peripheral nerves. IEEE Trans. Rehabil. Eng. 5, 51–61. Tyler, D. J., and Durand, D. M. (2002). Functionally selective peripheral nerve stimulation with a flat interface nerve electrode. IEEE Trans. Neural Syst. Rehabil. Eng. 10, 294–303. Tyler, D. J., and Durand, D. M. (2003). Chronic response of the rat sciatic nerve to the flat interface nerve electrode. Ann. Biomed. Eng. 31, 633–642. Velliste, M., Perel, S., Spalding, M. C., Whitford, A. S., and Schwartz, A. B. (2008). Cortical control of a prosthetic arm for self-feeding. Nature 453, 1098–1101. Veraart, C., Grill, W. M., and Mortimer, J. T. (1993). Selective control of muscle activation with a multipolar nerve cuV electrode. IEEE Trans. Biomed. Eng. 40, 640–653. Walter, J. S., GriYth, P., Sweeney, J., Scarpine, V., Bidnar, M., McLane, J., and Robinson, C. (1997). Multielectrode nerve cuV stimulation of the median nerve produces selective movements in a raccoon animal model. J. Spinal Cord. Med. 20, 233–243. Warwick, K., Gasson, M., Hutt, B., Goodhew, I., Kyberd, P., Andrews, B., Teddy, P., and Shad, A. (2003). The application of implant technology for cybernetic systems. Arch. Neurol. 60, 1369–1373. Waters, R. L., McNeal, D. R., Faloon, W., and CliVord, B. (1985). Functional electrical stimulation of the peroneal nerve for hemiplegia: Long-term clinical follow-up. J. Bone Joint Surg. 67, 792–793. Yoshida, K., and Horch, K. (1993). Selective stimulation of peripheral nerve fibers using dual intrafascicular electrodes. IEEE Trans. Biomed. Eng. 40, 492–494. Yoshida, K., and Horch, K. (1996). Closed-loop control of ankle position using muscle aVerent feedback with functional neuromuscular stimulation. IEEE Trans. Biomed. Eng. 43, 167–176. Yoshida, K., and Stein, R. B. (1999). Characterization of signals and noise rejection with bipolar longitudinal intrafascicular electrodes. IEEE Trans. Biomed. Eng. 46, 226–234. Yoshida, K., Jovanovic, K., and Stein, R. B. (2000). Intrafascicular electrode for stimulations and recording from mudpuppy spinal roots. J. Neurosci. Methods 96, 47–55.
INTERFACING INSECT BRAIN FOR SPACE APPLICATIONS
Giovanni Di Pino,* Tobias Seidl,y Antonella Benvenuto,* Fabrizio Sergi,* Domenico Campolo,* Dino Accoto,* Paolo Maria Rossini,z and Eugenio Guglielmelli* *Biomedical Robotics and Biomicrosystems Laboratory, Universita` Campus Biomedico di Roma, Roma, Italy y Advanced Concepts Team, European Space Agency, Keplerlaan 1, 2201 AZ Noordwijk, The Netherlands z Department of Neurology, Universita` Campus Biomedico di Roma, Via Alvaro del Portillo 21, 00128 Roma, Italy
I. Introduction II. Interfaces A. Levels of Interfacing B. Natural Interfaces C. In Vivo Neuronal Bidirectional Interfaces in Insects III. Sensory and Motor Mapping IV. Proposing a Model of Hybrid Control Architecture V. Conclusions and Outlook References
Insects exhibit remarkable navigation capabilities that current control architectures are still far from successfully mimic and reproduce. In this chapter, we present the results of a study on conceptualizing insect/machine hybrid controllers for improving autonomy of exploratory vehicles. First, the diVerent principally possible levels of interfacing between insect and machine are examined followed by a review of current approaches towards hybridity and enabling technologies. Based on the insights of this activity, we propose a double hybrid control architecture which hinges around the concept of ‘‘insect-in-a-cockpit.’’ It integrates both biological/artificial (insect/robot) modules and deliberative/reactive behavior. The basic assumption is that ‘‘low-level’’ tasks are managed by the robot, while the ‘‘insect intelligence’’ is exploited whenever high-level problem solving and decision making is required. Both neural and natural interfacing have been considered to achieve robustness and redundancy of exchanged information.
INTERNATIONAL REVIEW OF NEUROBIOLOGY, VOL. 86 DOI: 10.1016/S0074-7742(09)86003-0
39
Copyright 2009, Elsevier Inc. All rights reserved. 0074-7742/09 $35.00
40
DI PINO et al.
I. Introduction
The present book impressively highlights how astronaut’s capabilities can be enhanced by interfacing brains with machines, but we are well aware that human presence drastically limits space travel beyond earth orbiting. Robotic missions can reach further but require either remote access or a high level of autonomy. Despite the great scientific success of nonhuman missions they still lack important capabilities such as autonomous navigation on ground, problem solving and decision making in conflict situations, learning and adapting to perturbations of the original program. One possible approach is the integration of animal brains into unmanned spacecraft to achieve an intermediate type of mission combining features of the commonly known and strictly separated manned and unmanned mission types. In the framework of an Ariadna-study ESA’s Advanced Concepts Team, the Biomedical Robotics and Biomicrosystems Laboratory and the Neuroscience Department-Area of Neurology of the Campus Bio-Medico University in Rome jointly evaluated the major technologies required and conceptualized a control architecture integrating insect brain tissue in an engineered control architecture (Benvenuto et al., 2008). Among several animal species, insects have been chosen because they developed navigation mechanisms which are optimized in terms of simplicity and robustness ( Dacke and Srinivasan, 2007; Wehner, 2007), that are invaluable features of robotic systems. Since insect neuronal systems diVer considerably from those of humans the approaches and technologies to be used diVer accordingly. Given the inherent technological challenges, some simplifying assumptions are required; in particular we assume that it is feasible to keep alive and functional the animal brain tissue (or the whole insect) for a period of time appropriate for space missions. Moreover, we focus on the use of predeveloped living tissue and do not consider in vitro development of biological neuronal networks. Eventually, we will not take care of control issues which can be solved with a smart (e.g., biomimetic) design. In this chapter, at first the levels of interfacing between living tissues, environment and robot are described and the state-of-the-art technologies for both natural and neural interfacing are critically reviewed. Then elementary behaviors associated to exploration/navigation tasks and their triggers are schematized to better address the sensory and motor mapping issues. Finally, we present a double hybrid control architecture which includes both biological/artificial (insect/robot) modules and deliberative/reactive behaviors.
INTERFACING INSECT BRAIN
41
II. Interfaces
A. LEVELS OF INTERFACING The interface between living tissues and the engineered counterpart of the hybrid controller could be established at several levels involving the whole body of the animal or only part of it. Various solutions diVer also for the invasiveness and for the amount of information exchanged. First of all there is the choice between non-invasive cockpit like interfaces, where the intact animal’s sensors and eVectors are used to transmit information by providing quasi-natural stimuli and invasive interfaces with a direct electric contact between controller and living tissue. The invasive technique then allows choosing between neural interfaces, that is, information is decoded from neurons, or non-neural approaches (i.e., from muscles). Finally, neural interfaces roughly divide into three subgroups: electrodes stimulate and monitor (i) cultured neurons or (ii) pregrown brain tissue but being separated from the organism, and (iii) brain tissue in the organism. Experiments with self-organized neuronal cultures have returned promising results (Novellino et al., 2007) such as sensing and reacting to various stimuli. However, tasks as defined above might not be in the scope of such an arrangement. In consequence, we investigated on more complex neural networks as they can be extracted from living and developed organisms. It is worth noting that the brain stem of a lamprey is already able to control a small mobile device via bidirectional information exchange ( Reger et al., 2000). However, it would be desirable to interface an entire brain and have it maintained by a functional organism. This work has been focused on insect interfacing, that could be achieved either by insertion of microelectrodes in the ganglia or implanting electrodes into the musculature. While the first solution principally lacks mechanical stability, muscular interfacing requires higher electric current (Mavoori et al., 2004) and leads to a loss in information bandwidth. Finally, it is feasible also to invasively interface sensory organs of the organism similar to, for example, cochlear implants in humans.
B. NATURAL INTERFACES Natural interfaces exploit the existing sensorial and eVectorial means of an organism. In insect studies, natural interfacing is part of neurobiological and neuroethological experimental setups focusing on insect flight, visual flight
42
DI PINO et al.
stabilization or navigation. In such setups, the insect is situated in an arena— tethered or freely moving—and provided with visual stimuli by, for example, a circular matrix of light-emitting diodes (compare Fig. 1A; for exemplary description see Dacke and Srinivasan, 2007; Reiser and Dickinson, 2008; Srinivasan et al., 1996). For instance, in freely moving animals, magnetic coils attached to the thorax allow to monitor the insect’s motor response (Schilstra and Van Hateren, 1999) (Fig. 1C), while in tethered insects MEMS-based force sensors register the insect’s attempts and translates them into a visual reaction (Sun et al., 2005) (Fig. 1B). Experimental setups usually focus on the research of certain eVects rather than on achieving bidirectional control and hence the degree of integration of several technologies is kept rather low. There are, however, successful attempts where an insect autonomously controls a robot. The roachbot—developed by Hertz and coworkers (http://www.conceptlab.com/roachbot/)—uses proximity sensors to acquire data about the surrounding. LED panels display the results of these measurements to the tethered cockroach and a trackball monitors the animal’s reactions to the stimuli. The system allows the cockroach to interact with obstacles but is not capable of exploiting high-level autonomous behaviors.
C. IN VIVO NEURONAL BIDIRECTIONAL INTERFACES IN INSECTS Neuronal interfaces can exploit the comparative simplicity of the insect neuronal system, since most of the relevant neurons can be directly approached in the ventral ganglia or the connectives. Moreover, the nervous system presents a straight one-to-one correspondence between nerve stimulation and muscle
FIG. 1. Natural interfaces: (A) Programmable arena (adapted from Reiser and Dickinson, 2008); (B) MEMS-based force sensor (adapted from Sun et al., 2005); (C) sensor coils (adapted from Schilstra and Van Hateren, 1999).
INTERFACING INSECT BRAIN
43
activation and requires only little voltage to excite an axon, due to the sparse insulation. Indeed, direct stimulation of a nerve works with a tenth of the electric current compared to excitation via the musculature ( Mavoori et al., 2004). The technological development in the recent past has led to miniaturized and versatile implantable electrodes that greatly extend the application range. Modern techniques involve flexible polymer carrying multielectrode arrays. They can be wrapped a nerve or an insect appendage enabling multiunit recording and stimulation, without aVecting the kinematics of the animal locomotion (Spence et al., 2007). Other techniques were reported by Ye and coworkers (1995) using cuV-shaped electrodes for chronic recordings in tethered cockroach (Periplaneta americana). The electrode was placed close to the thoracic ganglion and remained for the remarkably long period of 2 months. With the same species Takeuchi and colleagues recently demonstrated a radiotelemetry system that allows recording neural activity in freely walking animal in the range of 16 m ( Takeuchi and Shimoyama, 2004). The system uses a shape memory alloy (SMA) electrode that, when actuated by electric heating, clips around the nerve cord along the thorax (Fig. 2A). Since it was developed only as a recording system, it misses the ability of stimulating and hence to operate as bidirectional interface.
A
Telemeter
Polyimide ribbon cable
B
Polyimide film
SMA microelectrode
C
D
FIG. 2. (A) Telemetric system implanted in freely moving Periplaneta americana (adapted from Takeuchi and Shimoyama, 2004); (B) Neuron-chip implanted in Manduca sexta (adapted from Diorio and Mavoori, 2003); (C) Cyborg beetle microsystem (adapted from Sato et al., 2008); (D) moth–robot (adapted from http://neuromorph.ece.arizona.edu/).
44
DI PINO et al.
Fully integrated bidirectional systems should ideally be able to stimulate and to record from the nervous system of a flying insect, while including an on-board memory and being suYciently small and light to be carried by the animal. Mavoori and coworkers (Diorio and Mavoori, 2003) presented such a microsystem carried by a freely moving Manduca sexta (Fig. 2B). An even higher degree of integration was achieved by Sato and coworkers (2008). The flight of the insect was controlled via an ensemble of muscular stimulators, an embedded microcontroller with batteries, microfluidic tubes and LED visual stimulator, together with a silicon neural probe. The four neural stimulators are implanted in the flight control area of the brain and close to the flight musculature, while the base of the optic flow device is mounted positioning the LED array in front of the head (Fig. 2C). Another remarkable example is the moth–robot (http://neuromorph.ece.arizona.edu/). Its composite bidirectional interfaces consist of (i) a natural interface through a continuous optic flow provided by a revolving wall painted with vertical stripes and (ii) a neural interface for measurements of electrical activity of visual motion neurons. The moth is immobilized inside a plastic tube mounted on a wheeled robot, which turns left or right, according to neural signals from the insect (Fig. 2D).
III. Sensory and Motor Mapping
The operating environment in which an exploratory spacecraft would be situated diVers considerably from the natural habitat of any potential model insect and therefore inputs need to be translated into signals that are readable for the insect. This translation does not only take place at the physical level, that is, presenting stimuli within the signal range of the sensors, but also requires that the neuronal system is able to decipher the meaning of an input and elicit appropriate reaction toward it. Complex behaviors as navigation or exploration may be abstracted as a product of behavioral elements following attraction and repulsion. Depending on their individual characteristics, the behavior elicited has either static or dynamic nature. A typical example of a static attractor is the pursuit of food or the nest, while a predator would elicit fleeting behavior, a dynamic repeller (Table I).
TABLE I ELEMENTARY BEHAVIORS AND THEIR TRIGGERS IN EXPLORATION/NAVIGATION TASKS
Static Dynamic
Attractors
Repellers
Pursuit (food, nest) Hunting (prey)
Avoidance (obstacle) Fleeting (predator)
45
INTERFACING INSECT BRAIN
IV. Proposing a Model of Hybrid Control Architecture
Interfacing insect brains strongly depends on the task the final architecture will be confronted with. In our pilot study, we identified an exploratory scenario where the animal/insect shall explore an unknown environment and autonomously navigate back to, for example, a base station that provides extended functionality such as energy, communication or automated sample analysis. Within such a scenario, high-level tasks like navigation, exploration and maintaining the energy budget would be handed over to the insect, while obstacle avoidance and locomotory issues would be dealt with by the engineered controller. Based on these considerations, a robotic platform including the hybrid control architecture was conceptualized as shown in Fig. 3. The proposed architecture (Benvenuto et al., 2009) comprises three main modules (i) the hybrid controller (HC), (ii) the adaptor (ADP), and (iii) the underlying mechatronic system (UMS). Moreover, two subsets have been defined in sensor modules: low-level sensors (LLS) and embiotic sensors (ES). The LLS are included in the UMS and they can be
Hybrid controller (HC) Sensory mapping
-Visual cues (programmable LEDs) -Olfactory cues
Neural stimulation unit
Neural interface
Environment
Embiotic sensors
Neural registration unit
Sensor modules
Natural stimulation unit
Natural interface
Cockpit Natural registration unit
Motor mapping
-Force sensors -Magnetic coils
Adaptor
Low level sensors Proprioceptors
Low level controller
Underlying mechatronic system (UMS) Actuators
Robot mechanical system
FIG. 3. Scheme of a robotic platform including the hybrid control architecture (Benvenuto et al., 2009).
46
DI PINO et al.
used for executing low-level behaviors; typically they include proximity sensors, inertia modules, wheel\leg encoders and wheel\leg slide sensors for selfstabilization. ‘‘embiotic’’ is a neologism that we introduced to indicate an additional subset of conventional sensors (e.g., vision systems, temperature sensors, polarization sensors, etc.), which complement LLS and depend on the selected insect. The hybrid controller is composed of ES, the cockpit, the interfacing module, and mapping modules. The cockpit surrounds the tethered insect which receives stimuli both via natural and neural interfaces. The neural interface module includes stimulation and registration units as well as sensory and motor mapping modules forming core elements of the proposed architecture. The ADP is designed to process the insect’s motor commands and to give input to the low-level controller. Actually, this is the only module which directly exchanges data between the hybrid controller and the mechatronic system. By changing the ADP module, the proposed hybrid controller could in principle be used with diVerent mechatronic systems, which can be typically tailored to the specific application scenario. The UMS is composed of the LLS, the proprioception sensors, and the low-level controller. The proprioception sensors monitor the internal state of the robot, in particular, for energy budget and mechanical failures. The main task of the low-level controller is to properly weight inputs from the sensor module, the proprioceptors and the adaptor, thus allowing for undisturbed mobility of the robot. V. Conclusions and Outlook
Our studies showed that the integration of predeveloped insect intelligence in robotic platforms could create an intermediate type of mission bridging between purely robotic and human controlled missions capable of performing complex behaviors exploiting the neuronal capabilities of the animal. However, success of automated mission vehicles strongly correlates with the capability of the control architecture to successfully integrate a whole range of decision parameters. The architecture we present here delegates high-level decision making and planning to the insect, while low-level tasks are executed by the robotic platform. The current state of the art of both neural and natural interfaces allows implementing in the short/medium term a ‘‘cockpit’’ interface with tethered insects; while additional challenges have to be considered if an ‘‘arena’’ interface with freely moving insects (i.e., highly ecological environment) will be developed, since dimensions and weight of neural bidirectional interfaces integrating batteries and telemetry systems should be reduced to allow implantation in small insects. Moreover, experiments to assess insect capabilities by reproducing space operating conditions, where the insect brain could also beforehand be trained, should be performed. Eventually, a novel performance/benchmarking metrics should be defined to assess the obtained results and to
INTERFACING INSECT BRAIN
47
compare them with the state-of-the-art autonomous agent performances. Even if much work is required for development and validation of the proposed control architecture, it represents a new challenging approach merging biomimetics, neurophysiology, ethology, and microengineering. References
Benvenuto, A., Di Pino, G., Sergi, F., Campolo, D., Accoto, D., Assenza, G., Rossini, P. M., and Guglielmelli, E. (2008). Machine/animal hybrid controllers for space applications. European Space Agency, the Advanced Concepts Team, Ariadna Final Report (07/6301). Benvenuto, A., Sergi, F., Di Pino, G., Seidl, T., Campolo, D., Accoto, D., Guglielmelli, E. (2009). Beyond biomimetics: Towards insect/machine hybrid controllers for space applications. Advanced Robotics, in press. Dacke, M., and Srinivasan, M. V. (2007). Honeybee navigation: Distance estimation in the third dimension. J. Exp. Biol. 210, 845–853. Diorio, C., and Mavoori, J. (2003). Computer electronics meet animal brains. Computer 36, 69–75. Mavoori, J., Millard, B., Longnion, J., Daniel, T., and Diorio, C. (2004). A miniature implantable computer for functional electrical stimulation and recording of neuromuscular activity. In IEEE International Workshop on Biomedical Circuits & Systems, pp. S1/7/INV–S1/13–16. Novellino, A., D’Angelo, P., Cozzi, L., Chiappalone, M., Sanguinetti, V., and Martinoia, S. (2007). Connecting neurons to a mobile robot: An in vitro bidirectional neural interface. Comput. Intell. Neurosci. 2007, 13. Reger, B. D., Fleming, K. M., Sanguineti, V., Alford, S., and Mussa-Ivaldi, F. A. (2000). Connecting brains to robots: An artificial body for studying the computational properties of neural tissues. Artif. Life 6, 307–324. Reiser, M. B., and Dickinson, M. H. (2008). A modular display system for insect behavioral neuroscience. J. Neurosci. Methods 167, 127–139. Sato, H., Berry, C., Casey, B., Lavella, G., Yao, Y., VandenBrooks, J., and Maharbiz, M. (2008). A cyborg beetle: Insect flight control through an implantable, tetherless microsystem. In MEMS 2008, pp. 164–167. Tucson, AZ, USA. Schilstra, C., and Van Hateren, J. H. (1999). Blowfly flight and optic flow, I. Thorax kinematics and flight dynamics. J. Exp. Biol. 202, 1481–1490. Spence, A. J., Neeves, K. B., Murphy, D., Sponberg, S., Land, B. R., Hoy, R. R., and Isaacson, M. S. (2007). Flexible multielectrodes can resolve multiple muscles in an insect appendage. J. Neurosci. Methods 159, 116–124. Srinivasan, M., Zhang, S., Lehrer, M., and Collett, T. (1996). Honeybee navigation en route to the goal: Visual flight control and odometry. J. Exp. Biol. 199, 237–244. Sun, Y., Fry, S. N., Potasek, D. P., Bell, D. J., and Nelson, B. J. (2005). Characterizing fruit fly behavior using a microforce sensor with a new comb-drive configuration. J. Microelectromech. Syst. 14(1), 4–11. Takeuchi, S., and Shimoyama, I. (2004). A radio-telemetry system with a shape memory alloy microelectrode for neural recording of freely moving insects. IEEE Trans. Biomed. Eng. 51, 133–137. Wehner, R. (2007). The desert ant’s navigational toolkit: Procedural rather than positional knowledge. In Proceedings of 63rd Annual Meeting of the Institute of Navigation, April 23–25, 2007, Cambridge, Massachusetts. Ye, S., Dowd, J. P., and Comer, C. M. (1995). A motion tracking system for simultaneous recording of rapid locomotion and neural activity from an insect. J. Neurosci. Methods 60, 199–210.
MEET THE BRAIN: NEUROPHYSIOLOGY
John Rothwell Sobell Department, Institute of Neurology, Queen Square, London WC1N 3BG, UK
I. II. III. IV.
Introduction How Do Neurons Transmit Information? Synapses The Motor Areas of the Cerebral Cortex A. Historical Background B. Present Day Anatomical and Electrophysiological Definitions of the Motor Areas of Cortex C. Output of Cortical Motor Areas D. Inputs to Cortical Motor Areas E. Activity of Motor Cortical Neurons F. Effect of Lesions of Motor Cortical Areas V. Plasticity of Primary Motor Cortex VI. Conclusions References
The central nervous system is composed of two main types of cells: the most numerous are glia, which have a supportive, protective and regulatory role, and neurons, which are the primary computing element. Neurons transmit information as a pulsed electrical code which is conducted down a specialized process (axon) that connects with other neurons. Each neuron can connect with many others, and each neuron can receive input from many others. At the sites of connection (synapses), information is transmitted across a small gap; small molecules (neurotransmitters) are released from the end of the axon, and these diVuse to receptor molecules on the receiving neuron. The latter then convert the chemical code back into an electrical signal that can be transmitted along the next axon. An important feature of synapses is that they are modifiable according to the prior history of activity in the system. This gives them an important role in learning, memory, and in adaptation to damage. Networks of neurons perform particular tasks. Those controlling movement are located in a number of adjacent areas of cerebral cortex. Some of these have axons that project to the spinal cord where they contact motoneurons that control particular sets of muscles; some have axons that interconnect the areas of the cortex; some have axons that project to subcortical groups of neurons in the basal
INTERNATIONAL REVIEW OF NEUROBIOLOGY, VOL. 86 DOI: 10.1016/S0074-7742(09)86004-2
51
Copyright 2009, Elsevier Inc. All rights reserved. 0074-7742/09 $35.00
52
JOHN ROTHWELL
ganglia and cerebellum. The presence of modifiable synapses in this complex network means that it is capable of learning new tasks and can react to injury by rearranging its connections to optimize function in undamaged parts.
I. Introduction
The central nervous system (CNS) contains two main types of cell: glia and neurons. Even though we always tend to think of the neurons as being the most important part of the CNS, they are in fact vastly outnumbered by the glia. In fact there are about 10–50 times more glia than neurons. The glia, so-called because early anatomists thought they were the ‘‘glue’’ that held the neurons together, come in a variety of forms and perform a large number of diVerent and important functions. The most numerous are the astrocytes, which are roughly star shaped with long processes, some of which contact neurons, where they may have a role in transferring nutrients, whereas others may line the outside of blood capillaries where they form the blood–brain barrier, determining what chemicals are allowed from the blood into the brain. Astrocytes also seem to have an important role in regulating the concentration of potassium ions around the neurons, an important factor in determining the voltage across the neuronal membrane. They are also involved in mopping up neurotransmitters after release from synapses and even manufacturing the raw chemicals that are transformed into neurotransmitters by neurons. Two other forms of glia have a role in insulating the neurons and thus helping them conduct their electrical impulses. These are the oligodendrocytes and the Schwann cells. Oligodendrocytes are found in the CNS where their processes envelope 10 or more diVerent neurons; Schwann cells are found around neurons in the periphery (i.e., in nerves outside the brain and spinal cord). Each Schwann cell contacts only one neuron while each neuron is contacted by many Schwann cells, each of which covers the length between two nodes of Ranvier (see below). Neurons are the active signaling units of the nervous system. Within the CNS they receive input from other neurons, process it, and then send it on to other neurons in a one way flow of information. Neurons in the periphery have other connections: sensory neurons contact (or have specialized terminations themselves) sensory receptors that transform the five senses (touch, vision, hearing, smell, taste) into electrical signals that eventually enter the CNS. Motor neurons form the only output of the CNS: at their ends they contact muscles that move the body. Essentially the CNS receives sensory inputs and uses this information to drive movement (Fig. 1). Each neuron has four main regions. The cell body contains the nucleus and most of the other specialized structures necessary to manufacture proteins. It has
53
NEUROPHYSIOLOGY
Dendrite Sensory neuron Receptor (free nerve ending)
Cell body Axon
Myelin sheath Motor end plate
Axon Motor neuron www.infovisual.info
Dendrite Cell body
FIG. 1. Typical sensory and motor neuron illustrating the main features and connectivity of each. The arrows indicate the direction of information flow.
two sorts of processes, dendrites, of which there may be very many, that branch out over a very large volume in comparison with the volume of the cell body itself, and a single axon that forms the output of the neuron and which may travel considerable distances from the cell body before contacting other neurons at specialized terminations called synapses. The dendrites receive input from the synapses of other neurons and together with the cell body, they integrate input from multiple sources. The axon then transmits the processed information to the next neurons. Each neuron may typically have hundreds of inputs and will synapse with hundreds of other neurons.
II. How Do Neurons Transmit Information?
All neurons have an electrical potential across the membrane that makes the inside about 70 mV negative with respect to the outside. The potential is caused by diVerences in the concentration of charged ions, particularly Naþ and Kþ, in the inside versus the outside. Amino acids and proteins tend to have a negative charge and are located both inside and outside the neuron, but a special pump (the Naþ/Kþ pump) in the cell membrane pumps Naþ out of the cell and pumps Kþ into the cell. The result is that the concentration of Naþ outside is about 10 times that inside whereas the concentration of Kþ is about 20 times higher
54
JOHN ROTHWELL
inside than outside. The membrane itself is fairly impermeable to Naþ (and to the large amino acids and proteins), but is leaky for Kþ. This means that Kþ leaks out of the neuron down its concentration gradient, leaving behind an excess of negatively charged large proteins and amino acids that are responsible for the net negative charge inside the cell. The pumping of ions occurs continuously to maintain the diVerential concentrations of ions; if it stops, the potential disappears. Axons transmit information in an all or none fashion as a series of identical electrical impulses known as action potentials. Thus an axon can only code information in terms of the timing of the pulses that it transmits. The action potential begins at a specialized region of the cell body known as the initial segment, at the start of the axon. It is initiated as the result of processing the inputs collected by the dendrites. This causes the membrane potential at the axon hillock to become less negative than in the resting state (see below). If the potential across the membrane reaches a value that is less negative than a certain value (i.e., more positive than, say, 50 mV ) there is a sudden change in the permeability of the cell membrane at that point and it becomes highly permeable to Naþ (much more permeable than it was to Kþ). Naþ ions then enter the neuron down their concentration gradient and cause the inside at that point to become positive with respect to the outside, eVectively reversing the potential across the membrane. This state lasts a few milliseconds, after which the permeability to Naþ drops back to normal, and the membrane reverts to its original state with exit of Kþ ions. The key to the whole process of changing the membrane permeability to Naþ lies with a specialized protein in the membrane called the voltage sensitive Naþ channel. At a resting potential of 70 mV, the channel is closed; when depolarized beyond a certain value (known as the threshold potential), the Naþ channel opens transiently and then closes. The job of the axon is to transmit this transient change in potential to other neurons that are connected at the ends of the axon. It achieves this via two processes. One is passive spread of the potential to adjacent regions of the neuronal membrane (electrotonic conduction). The neuronal membrane can be regarded as an electrical capacitance in which the depolarized part begins to change the potential on the adjacent membrane as current flows through the resistance of the extra- and intracellular fluids. This change takes time and does not spread very far because the original depolarization is transient. However, the length of the axon is populated by voltage sensitive Naþ channels that open when the potential in their vicinity exceeds the threshold. When they do this, they reinforce the spread of depolarization along the length of the axon. Within the CNS, impulses are transmitted along axons at speeds of up to about 10 m/s. The larger the diameter of the axon, the faster the conduction. This is because a larger axon has a smaller surface:volume ratio than a small axon, and therefore a proportionally smaller membrane capacitance to charge up. It also has a smaller
NEUROPHYSIOLOGY
55
internal electrical resistance which also speeds up the rate at which adjacent sections of membrane are depolarized.
III. Synapses
The synapse is the point at which information is transmitted between two neurons. It consists of a small gap across which information is transferred by diVusion of chemicals (neurotransmitters) that are released from the specialized termination of the axon. They are detected by molecules inserted into the dendrite on the opposite side of the gap. In the presynaptic axon, information transmitted as electrical impulses has to be transformed into a chemical code, while in the postsynaptic dendrite the process must be reversed and the chemical code must be transformed back into an electrical code. Synapses are highly complex parts of the neuron and are designed to be modifiable according to the past history of activity in the pre- and postsynaptic neurons. The details of transmission at the synapse are as follows. At the region of the synapse, the presynaptic axon contains voltage sensitive Ca2þ channels that open when the wave of depolarization produced by the action potential reaches the synapse. Ca2þ exists at a much higher concentration outside than inside the neuron, so that on arrival, the action potential causes an influx of Ca2þ ions. Within the presynaptic terminal, a neurotransmitter is stored in small pockets called vesicles surrounded by a membrane. The sudden Ca2þ influx via a series of intermediate steps causes these vesicles to fuse with the neuronal membrane and release their contents into the synaptic space. In the postsynaptic membrane of the dendrite receptor molecules can bind to the neurotransmitter, and in doing so open ion channels that cause changes in the postsynaptic membrane potential. Exactly which ion channels are open and for how long depends on the neurotransmitter and receptor which it couples to. Typically, postsynaptic membrane polarizations last about 10–20 ms, although some may continue for several hundred milliseconds. In contrast, the depolarization in an action potential lasts about 5 ms (Fig. 2). As an example, glutamate is the most common excitatory neurotransmitter in the brain. When it binds to one of its receptors, the AMPA receptor, an ion channel opens in the postsynaptic dendritic membrane that is permeable to Naþ and Kþ. The net result is that it depolarizes the dendrite. This depolarization spreads electrotonically to the initial segment region of the axon where it can contribute to the initiation of an action potential. GABA is the most common inhibitory neurotransmitter. When bound by its receptor, it increases permeability to Cl ions, which then enter the neuron down a concentration gradient and make the inside of the cell more negative than at rest.
56
JOHN ROTHWELL
Ca++ V-sens
Pre-synaptic Glu
Na+
Mg++ NMDA
K+
AMPA EPSP
Post-synaptic dendritic spine
FIG. 2. Presynaptic release of neurotransmitter from vesicles that fuse with the cell membrane when calcium ions enter flowing depolarization of the membrane. Transmitter molecules (in this case glutamate) diVuse across the synapse and combine with AMPA receptors in the postsynaptic membrane. These then allow sodium and potassium ions to enter the membrane, depolarizing the potential. Another receptor that can bind glutamate is also drawn, the NMDA receptor. This is usually blocked by a magnesium ion at normal membrane potentials, but can be expelled during depolarization and then bind glutamate.
An important feature of the synaptic connections between most neurons is that the input from any one synapse has only a very small eVect on the potential at the trigger zone in the axon hillock; an action potential requires the summation of inputs from many synapses, all active at the same time. In simple terms the neuron
NEUROPHYSIOLOGY
57
adds up the synaptic inputs that occur and if these at any time reach the threshold, an impulse is generated that travels onward to the next set of synapses. There are a large number of refinements to this simple model of a working neuron. The first, as mentioned above is that the eVectiveness of a synapse is not fixed, but can change according to the pattern of activity that has occurred in the recent past. These changes occur over a range of time scales from milliseconds to hours and days, with multiple molecular pathways involved in each stage. In addition, the eVectiveness of inputs to a neuron can be regulated by ‘‘neuromodulatory’’ transmitters such as dopamine, acetylcholine, or serotonin. When these substances are released from synapses, they do not typically make neurons discharge, but instead regulate the response of the neuron to other inputs.
IV. The Motor Areas of the Cerebral Cortex
A. HISTORICAL BACKGROUND Hughlings Jackson was one of the first physicians to speculate that the cortex around the central sulcus contained an organized representation of body movements. He observed that motor epilepsies often began with small twitches in the hand or the corner of the mouth and then spread to involve adjacent muscles and finally the whole body. In a small number of cases that came to pathology, he saw that there was limited damage to part of the cerebral cortex around the central sulcus. He suggested that this indicated that there was a discrete representation of movements of diVerent body parts in this area, and that ‘‘irritation’’ could produce movements of the corresponding part of the contralateral body. He further noted that some parts were likely to have a larger or more excitable eVect than others, explaining the propensity for twitches to begin in the hands or face. His ideas were confirmed later by Fritsch and Hitzig and David Ferrier in the 1870s, who showed that electrical stimulation of the central area in dogs and monkeys could produce movements of the opposite side of the body. Movements of diVerent parts of the body were produced by diVerent locations of the stimulating electrode, with the lowest threshold eVects being observed in the distal limbs. Bartholow carried the first stimulation of the human motor cortex out only a few years later in a patient whose cortex was exposed by a large ulcer on her scalp. These experiments defined the motor cortex as the area from which movements could be elicited at lowest intensity. Within this area there was a map of the body in which movements of the legs were represented medially, with the trunk, arms and face progressively more lateral. As predicted by Hughlings Jackson, movements of the lower face and hands were much more readily evoked, and from a wider area of cortex, than movements of other parts of the body.
58
JOHN ROTHWELL
This arrangement was later popularized in the now-familiar motor ‘‘homunculus’’ drawn by Penfield and coworkers.
B. PRESENT DAY ANATOMICAL AND ELECTROPHYSIOLOGICAL DEFINITIONS OF THE MOTOR AREAS OF CORTEX Neurophysiologists now recognize that there are several representations of the body in what are now termed the ‘‘motor areas’’ of the cerebral cortex. Anatomically, these occupy Brodmann’s areas 4 and 6 on the lateral and medial surfaces of the hemispheres anterior to the central sulcus, together with areas 23 and 24 in the cingulate gyrus. In monkey experiments, electrical stimulation over these areas has revealed a total of seven diVerent representations of the contralateral body. The primary motor cortex has been studied in most detail. This occupies area 4 of Brodmann, which is mostly buried in the anterior bank of the central sulcus. It is distinguished from adjacent sensory areas by its lack of a pronounced cortical layer IV (agranular cortex). It diVers from area 6 by the presence of large pyramidal neurons in layer V (Betz cells). Electrical stimulation of the primary motor cortex has a lower threshold than any other motor area, and produces twitch-like movements of a small number of muscles in the contralateral body. These may be, for example, a flick of the fingers or a twitch of biceps or the corner of the mouth depending on the point of stimulation. The location of the primary motor cortex is frequently mapped out during neurosurgical operations in man. Patients are often awake during such operations, and they point out that the movements feel involuntary, as if imposed by an external force. The implication is that awareness of the eVort of a voluntary movement must arise in other areas of cortex. Patients also note that during stimulation they feel unable to move that part of the body. Presumably activation of the cortex by electrical current prevents patients from using that area in voluntary movements. About one-third of the primary motor cortex is devoted to control of the hand. MRI images show that in most subjects, this region is marked by folding of the central sulcus into an ‘‘omega’’ shape when viewed from the surface. It is a rare example of an anatomical marker for a specific cortical function (Fig. 3). In the monkey two further representations of the body are found anterior to area 4 in the lateral part of area 6 around the arcuate sulcus. These are known as the dorsal and ventral premotor areas. Stimulation here has a higher threshold than for area 4 and provokes more complex movements, often involving more than one part of the body simultaneously. This area has not been extensively studied in humans. One problem is to define the limits of human premotor areas. The precentral sulcus is thought to be the human analogue of the arcuate sulcus, yet human area 6 extends further anterior to this point than it does in the monkey.
59
NEUROPHYSIOLOGY
Cingulate
l
SPC
cu
at
e
ntr al
Ar
Ce
Princ ipal
ta
rie
pa
6
8
4
ra nt
I
6 PCS
CS
44
CMA
SMA PMd
PMv
MCx
Dorsal compartment of area 6 on superior and middle frontal gyrus (PMC d) Ventral compartment of area 6 on precentral gyrus (PMC v) PCS Precentral sulcus CS Central sulcus
FIG. 3. Diagrammatic summary of some of main motor areas of the macaque cortex (left) and homologous areas on the lateral surface of the hemisphere in human (right). The macaque brain on the left shows both the lateral and medial walls of the hemisphere, with the upper panel labeled to show the names of the principal sulci, and the lower panel labeled to indicate the approximate areas of the primary motor cortex (MCx), dorsal and ventral premotor cortex (PMd, PMd), supplementary motor area (SMA), and cingulate motor areas (CMA). The dotted lines indicate the bottom of the central and cingulated sulci. The human brain on the right shows only the lateral surface and indicates the approximate extent of the premotor and primary motor cortices.
Another problem is that there have been no detailed mapping studies to compare with the monkey work. In fact, the original somatotopic maps of human cortex mark this region as part of the trunk representation. The medial portion of area 6 anterior to the leg representation in the primary motor cortex comprises the supplementary area (SMA). This is organized with the legs posterior, adjacent to the primary leg area, and the arms and face anterior. The eVects of stimulation are relatively well described in humans. The threshold is higher than for the primary motor cortex, and the movements more complex, often involving for example combined turning of the head and extension of the arm. Bilateral movements and vocalization can also be produced. In the last 10 years three more motor representations have been described around the cingulate gyrus, approximately ventral to the SMA. These lie in parts of area 6, 23, and 24 of Brodmann, and called the dorsal, ventral, and rostral cingulate motor areas (CMAd, CMAv, CMAr). The main evidence for these representations came initially from anatomical studies that showed they had direct projections to the spinal cord. Electrical stimulation studies are rare,
60
JOHN ROTHWELL
although it has been confirmed in humans that at least one similar representation of the body lies in the cingulate gyrus at this level. Stimulation produced tonic extension of the arm or the leg.
C. OUTPUT OF CORTICAL MOTOR AREAS All the cortical motor areas have a direct output to the spinal cord. This is composed of axons from pyramidal neurons in cortical layer V, which run in the corticospinal (or pyramidal) tract and innervate all levels of the contralateral spinal cord. In humans there are about half a million fibers in the corticospinal tract on each side. Two percent of these are large diameter and conduct impulses rapidly at speeds of up to 80 m/s. They are probably the axons of the large Betz cells of primary motor cortex. The majority of corticospinal fibers are smaller diameter and slower conducting (10–30 m/s). The fibers run in the lateral and anterior columns with the majority traveling in the former. Most of the projection is contralateral, although a small percentage (estimated at about 10%), particularly fibers in the anterior columns, run ipsilaterally. Terminations are mostly onto interneurons in the gray matter of the intermediate zone. However, especially in primates and man, there are extensive monosynaptic projections directly to spinal motoneurons in lamina IX. This projection is particularly prominent to distal muscles of the forearm and hand and is thought to be one factor that contributes to the increased dexterity of humans compared with other species. The pattern of corticospinal projections to antagonist and synergist muscles has been examined in detail for muscles of the hand and arm. In most cases, corticospinal connections excite close synergists and often inhibit the antagonists. However, there are also smaller numbers of neurons that excite agonist and antagonist muscles simultaneously. It is thought that this could be of use when it is necessary to stabilize a joint, such as is needed, for example at the wrist when individual finger movements are made. Although the corticospinal tract is large, it is important to remember that motor cortical areas also communicate with spinal motoneurons via projections to nuclei in the brainstem. Some of these are collaterals of corticospinal fibers, while some project to the brainstem only. These brainstem nuclei have descending fibers that form the reticulospinal tracts and innervate all segments of the cord, sometimes bilaterally. The density of these cortico-reticulospinal projections is higher from premotor, SMA and cingulate motor areas than it is from the primary motor cortex. The relative roles of corticospinal and noncorticospinal pathways are illustrated by experiments in which the corticospinal fibers are surgically cut. This has been performed several times in monkey by lesioning the pyramids in
NEUROPHYSIOLOGY
61
the medulla; in man corticospinal fibers have been cut for relief of involuntary movements by lesioning the middle third of the cerebral peduncle. In both cases, the loss of a million or so fibers is accompanied by little evidence of gross movement deficit. There is no spasticity and little weakness. The main deficit is in control of manipulative movements of the hands: the fingers are no longer used independently and precision grip is lost. The implication is that although the corticospinal system is important for fine distal movement, the noncorticospinal projections are of equal or even greater importance for other types of movement. When projections from cortex to the brainstem nuclei are lesioned, as happens in a capsular stroke, these noncorticospinal projections lose their input from the cerebral cortex. The resulting movement deficit is much greater than seen after pure pyramidal lesions. Motor areas of the cortex also send projections to pontine nuclei that innervate the cerebellum and to the ascending sensory systems in the gracile and cuneate nuclei (in both cases, often as collateral of corticospinal fibers). The cerebellar projection provides a copy of the motor command that could potentially be used to update movement more quickly than relying on sensory feedback. The projection to sensory nuclei is important in controlling the flow of sensory information during movement. D. INPUTS TO CORTICAL MOTOR AREAS The motor areas are also distinguished by diVerences in the inputs they receive from other parts of the brain. The nature of these inputs is presumably an important factor in determining the contribution of each area to specific types of movement (see section below). The primary motor cortex receives input mainly from sensory cortex, and from areas of thalamus that receive input from the cerebellum, and to a lesser extent, the basal ganglia. There are also extensive connections both to and from the other motor areas. The premotor areas receive a major input from regions of the posterior parietal cortex that are involved in the combined processing of visual and somatosensory input, as well as input from cerebellum via the thalamus. The SMA receives a large input from parietal cortex and from thalamic nuclei that receive input from basal ganglia. Cingulate motor areas are thought to have extensive connections with regions in the frontal lobes.
E. ACTIVITY OF MOTOR CORTICAL NEURONS The pattern of input is reflected in the way neurons in each area contribute to diVerent parts of the preparation and execution of movement. Neurophysiological recordings of cell discharge in behaving animals show that neurons in primary motor cortex change their firing rate just before and during a movement.
62
JOHN ROTHWELL
Neurons with monosynaptic connections to motoneurons tend to behave much like the muscles that they drive, being active just before and during contraction. The relation of cortical interneurons or pyramidal neurons without direct connections to spinal motoneurons tends to be more complex. Their activity may reflect stages in cortical processing or the input to spinal interneurons rather than motoneurons. In general, activity in these neurons occurs around the time of movement and tends to be related to parameters such as the force of muscle contraction or the direction of the movement. The activity of neurons in SMA and premotor areas is more likely to be related to preparation, rather than execution of movement. The neurons here may discharge well before a movement as if they were specifying what movement is to be made next rather than specifying the nature of the movement currently underway. There appear to be few diVerences in the contribution of SMA and premotor cortex to simple tasks, such as pushing a button when a light comes on. However, activity in these two areas diVers substantially in more complex tasks, such as those requiring a sequence of movements. Premotor neurons are more likely to change their activity when visual cues are being used to guide the sequence of movements while SMA activity is more intense when the sequence is made from memory, without visual cues. An example would be to point at a series of lights as they are illuminated in random order, or to point to the same positions from memory. The former is said to be an ‘‘externally guided’’ movement and involves activity in premotor cortex; the latter is an ‘‘internally generated’’ movement and involves activity in the SMA. The main input that drives externally guided movements comes from parietal cortex. It can be imagined that processed sensory input from this area can help the premotor regions to select appropriate movements from a set of stored commands. Indeed, there are neurons in this area that have been labeled ‘‘mirror’’ neurons, because they discharge both when a monkey makes a particular movement and also when it observes another animal make the same movement. Such neuronal types could obviously help us to learn movements by observing them.
F. EFFECT OF LESIONS OF MOTOR CORTICAL AREAS Small lesions of the primary motor cortex produce transient weakness that often resolves. This is presumably because neighboring areas of cortex can compensate for the lost function of the damaged area (see Section V ). Larger lesions, particularly if they involve premotor areas, result in permanent weakness and spasticity. The latter is thought to be a consequence of removal of input to reticular centers of the brainstem that influence muscle tone. Pure lesions of the premotor and SMA areas are rare in humans but have been studied extensively in monkey. Lesions of premotor areas aVect the ability of the monkey to
NEUROPHYSIOLOGY
63
retrieve the correct movement on the basis of external cues, whereas SMA lesions aVect the ability of monkeys to perform a series of movements from memory (‘‘internally guided’’ movements). SMA lesions also aVect coordination in actions involving object manipulation in both hands. In humans, rare patients with limited lesions of presumed premotor areas also have diYculty in associating visual cues with particular movements. They cannot learn, for example, to produce diVerent hand postures when they view particular visual symbols. Lesions of the SMA in humans are most commonly associated with diYculty in initiating or maintaining speech, a feature that is not evident in the monkey data. In addition patients with larger midline lesions may exhibit the ‘‘alien hand’’ sign, a condition in which the hand contralateral to the lesion behaves independently of the subject’s will, and may reach out to grasp objects within reach. This can be interpreted as a possible uncoupling of the externally guided premotor system from the internally guided SMA system. Finally, the movements of patients with SMA lesions, even after apparently good recovery, are often slower than normal. Interestingly, bradykinesia is a principal symptom of Parkinson’s disease, and since the SMA is an important output target of the basal ganglia, slowness of movement after a lesion may be due to lack of basal ganglia input to the motor system.
V. Plasticity of Primary Motor Cortex
It has been known for many years that electrical stimulation at the same site in primary motor cortex does not always produce exactly the same movement on every occasion over a period of several hours. However, this capacity of the cortex to change its organization, even in the adult, has only been investigated in detail in the last 10–20 years. For example, in rats, there is a large representation of the whiskers in the primary motor cortex. In the somatotopic representation of the rats’ body, this area is bordered by a region representing the periocular muscles and a region representing the arm muscles. If the VII nerve is lesioned, the rats can no longer use the whiskers. In such circumstances, what happens to that area of motor cortex that was devoted to control of the whiskers? Does it become a ‘‘silent’’ area, from which no movements of any sort can be obtained, or does reorganization occur so that it can contribute to movement of other body parts? In fact the latter seems to be true: stimulation of sites that previously had evoked whisker movement now give rises to movements of the periocular or arm muscles. These changes occur so rapidly that they cannot involve growth of new connections or synapses. Instead it is thought that they rely on changes in horizontal connections between cortical areas. Thus, after VII nerve section, the excitability of connections from the whisker area to other regions becomes more excitable.
64
JOHN ROTHWELL
Electrical stimulation of the whisker area activates these connections and evokes movements of the arm or periocular muscles indirectly via adjacent cortex. The excitability of the horizontal connections that are involved in this remodeling of cortical representation is controlled by GABAergic inhibition. This inhibitory system in turn is regulated by sensory information about the state of the peripheral motor apparatus. Similar remodeling occurs if the damage is in the motor cortex rather than the periphery. After small lesions, healthy surrounding areas can evoke movements previously elicited by stimulation of the damaged part. It is particularly interesting to note that such reorganization is enhanced if animals undergo training of the aVected part of the body. The implication is that physiotherapy may have an important influence in outcome from cortical damage. The situation with larger lesions is unexplored. However, horizontal connections in the cortex are short (2 mm or so) and often do not cross boundaries between cortical areas. Thus recovery from large lesions may require a diVerent from of reorganization. Finally, there is good evidence for similar patterns of reorganization in human primary motor cortex. The recent technique of transcranial magnetic stimulation, in which a magnetic field is used to induce stimulating electrical currents in the brain, allows coarse mapping of the motor cortex to be carried out in intact conscious subjects. Mapping the cortex of amputees, for example, shows that stimulation of the area previously controlling movements of the lost limb can evoke movement in immediately adjacent parts of the body. As in the rat experiments, it seems as if the pattern of representation of the body on the cortex can change after injury. Indeed, recent experiments suggest that there may even be changes in the pattern of representation in motor cortex when subjects learn new tasks. It appears as if the motor cortex map is maintained by a dynamic balance in the excitability of short corticocortical connections.
VI. Conclusions
In conclusion, control of movement depends on the coordinated action of a large number of anatomically separate but interconnected areas of the brain and spinal cord that operate in parallel to determine the final movement outcome. From the viewpoint of a brain–machine interface, this design poses a problem since there is no one point where the final command is represented apart from at its final convergence at the spinal motoneuron. In the case of voluntary movements of the hand and arm, however, a close second point of convergence is the primary motor cortex which is the source of much of the command for grasping and reaching movements. Recording activity here may give a good representation of the final movement that is intended by the brain. However, other types of
NEUROPHYSIOLOGY
65
movement, perhaps involving postural control of legs and trunk are more likely to depend on integration of many inputs from other subcortical areas of the motor system that may be less accessible to noninvasive recording devices.
References
Porter, R., and Lemon, R. N. (1993). ‘‘Corticospinal Function and Voluntary Movement.’’ Oxford University Press, Oxford. Rizzolatti, G., and Luppino, G. (2001). The cortical motor system. Neuron 31, 889–901. Rothwell, J. C. (2001). First studies of the organisation of the human motor cortex. In ‘‘Classics in Movement Science’’ (M. L. Latash and V. M. Zatisiorsky, Eds.), pp. 273–288. Human Kinetics, Champaign, IL. Sanes, J. N., and Donoghue, J. P. (2001). Plasticity and primary motor cortex. Ann. Rev. Neurosci. 23, 393–415.
FUNDAMENTALS OF ELECTROENCEFALOGRAPHY, MAGNETOENCEFALOGRAPHY, AND FUNCTIONAL MAGNETIC RESONANCE IMAGING
Claudio Babiloni,*,z Vittorio Pizzella,y Cosimo Del Gratta,y Antonio Ferretti,y and Gian Luca Romaniy y
*Department of Biomedical Sciences, University of Foggia, Foggia, Italy Department of Clinical Sciences and Biomedical Imaging, Institute for Advanced Biomedical Technologies, University G. D’Annunzio of Chieti, Chieti, Italy z Hospital San Raffaele Cassino, Cassino, Italy
I. II. III. IV. V. VI. VII.
Introduction to Electroencephalography and Magnetoencephalography Physiological Generation of EEG/MEG Signals EEG and MEG Techniques Allow the Study of Brain Rhythms Functional Magnetic Resonance Imaging Physiological Generation of Blood Oxygen Level-Dependent Signal Typical f MRI Experimental Designs BOLD-f MRI Techniques in Clinical Environment References
This review introduces readers to fundamentals of electroencephalography (EEG), magnetoencephalography (MEG), and functional magnetic resonance imaging (fMRI). EEG and MEG signals are mainly produced by postsynaptic ionic currents of synchronically active pyramidal cortical neurons. These signals reflect the integrative information processing of neurons representing the output of cortical neural modules. EEG and MEG signals have a high temporal resolution (<1 ms) ideal to investigate an emerging propriety of brain physiology, namely the brain rhythms. A background spontaneous oscillatory activity of brain neurons at about 10 Hz generates dominant alpha rhythms of restingstate EEG and MEG activity. This background activity is blocked during sensory and cognitive-motor events. Standard EEG shows a low spatial resolution (5–9 cm), which partially improves by high-resolution EEG including 64–128 channels and source estimation techniques (1–3 cm); source estimation of MEG data shows a better spatial resolution (0.5–2 cm). fMRI is an indirect measurement of regional brain activity based on the ratio between deoxyhemoglobin and oxyhemoglobin blood (BOLD) during events referenced to baseline conditions. Event-related BOLD response has low temporal resolution (>1 s) and quite high
INTERNATIONAL REVIEW OF NEUROBIOLOGY, VOL. 86 DOI: 10.1016/S0074-7742(09)86005-4
67
Copyright 2009, Elsevier Inc. All rights reserved. 0074-7742/09 $35.00
68
BABILONI et al.
spatial resolution (<1 cm), and is especially suitable to investigate spatial details of both cortical and subcortical activation.
I. Introduction to Electroencephalography and Magnetoencephalography
The word electroencephalography (EEG) refers to the measurement of brain electrical activity recorded from scalp electrodes. In 1929, EEG data were firstly recorded in human beings by Hans Berger, a German physiologist and psychiatrist. He observed a dominant 10-Hz oscillating voltage diVerence between two electrodes placed on the scalp in healthy subjects posed at wakeful eyes-closed relaxed state (the so-called alpha rhythms). Berger showed that the amplitude of 10-Hz oscillations (10–50 mV) reduces in amplitude when subject opens eyes or performs some cognitive mental activity. In the 1950s, EEG technique was carried out by Edgar Adrian in Europe and by Herbert Jasper in North America. Nowadays, such a technique is largely employed for basic scientific research and clinical applications, since it is easy to use, noninvasive, cheap, and totally safe. An important limitation of EEG technique is the fact that its amplitude measurement depends on the choice of electrical reference considered as the zero voltage value; typical electrical references such as linked earlobes, temporal mastoid processes, nose tip, or cephalic sites can be diVerently contaminated by potentials generated at both cerebral and extracerebral (i.e., eyes, muscles) levels. Another important limitation of EEG technique is its low spatial resolution. DiVerent conductivities of head tissues (brain, meninges, skull, and scalp) attenuate and blur the spatial distribution of neural currents from brain to scalp electrodes. As a consequence, scalp EEG data present enhanced low spatial components and negligible values of high-frequency brain oscillations (>40 Hz, gamma rhythms). Due to these eVects, the amplitude of EEG potentials at a given scalp site cannot reflect the intensity of neural currents in the underlying cerebral generators, posing the problematic issue of the estimation of brain EEG sources. To solve the problem of low spatial resolution and high-frequency filtering of EEG activity, brain potentials can be recorded by thin electrodes inserted into the human head when some special clinical conditions are present. For example, intracranial EEG recordings are used to localize the brain regions generating seizures in epilepsy patients who do not respond to pharmacologic therapy and need neurosurgery. For obvious ethical reasons, the event of intracranial EEG is quite rare, so the above important limitations of EEG techniques have motivated the development of sophisticated mathematical approaches to take into account the eVects of electrical reference and head volume conduction. Furthermore, techniques have been
FUNDAMENTALS OF EEG, MEG, AND FMRI
69
developed to record the magnetic component of brain electromagnetic activity, the so-called magnetoencephalography (MEG). Specifically, MEG technique detects the magnetic field generated by brain electrical activity. MEG activity has been first observed in the late 1960s, but only later it became feasible thanks to the development of superconducting quantum interference devices (SQUIDs). These superconducting devices are extremely sensitive magnetic flux to voltage converters, with an ultimate sensitivity limited only by quantum limit. As a matter of fact, the magnetic field generated by brain electric activity (neuromagnetic field) is extremely weak, often below 100 f T (100 f T ¼ 1013 T ), while the detectors today used in MEG devices (low-Tc SQUIDs operate at 4.2 K, the liquid helium temperature) feature a sensitivity of few f T/sqrt(Hz). Moreover, the neuromagnetic field is much weaker than the ordinary magnetic fields generated by the earth (104 T) or by the 50/60 Hz current flowing inside power lines (about 108 T). The use of a heavy magnetical shielded room is thus required to attenuate these magnetic fields in the sensor area. These technical requirements make the MEG device a relatively expensive one, with an overall present price of more than one million Euro compared to few tens of thousands of Euro needed for EEG equipment. Similarly to EEG, the MEG technique is absolutely noninvasive.
II. Physiological Generation of EEG/MEG Signals
EEG and MEG signals are very large-scale measure of brain source activity, apparently recording synaptic activity synchronized over macroscopic (centimeter), regional, and even whole brain spatial scales (Nunez et al., 2001). Synchrony among neural populations in compact regions of the brain produces localized dipole current sources. Synchrony among neural populations distributed across the cortex can give rise to regional or global networks consisting of many dipole sources. EEG and MEG signals both derive from electric activity of neurons of cerebral cortex. The human cerebral cortex is formed by small macrocolumns (diameter of <1 mm) normally oriented to cortical surface. In the module, granular neurons receive input signals, stellate interneurons process them, and pyramidal neurons deliver the output signals to other cortical modules, thalamus, brainstem, and spinal chord. EEG and MEG signals are mainly produced by postsynaptic ionic currents of synchronically active pyramidal cortical neurons, which have an ordered orientation to palisade ideal for the spatial summation of postsynaptic potentials and for the propagation of the relative neural currents to scalp surface. Postsynaptic potentials of cortical pyramidal neurons reflect the integrative information processing of signals coming from thalamus, brainstem, and other cortical modules.
70
BABILONI et al.
Mathematically, each cortical macrocolumn can be described by a current dipole, that is, a short current segment Q ¼ I L, where I is the current flowing inside the column and L is the length and direction of the column. The current density J associated with a current dipole is J(r) ¼ Q(rr0), where r0 is the dipole coordinate and is the Dirac’s delta function. The macrocolumns are enclosed in a conducting medium, and a return current is elicited together with the column activation according to the diVerential form of Ohm’s law J ¼ E, where E is the electric field generated by the current dipole and is the electric conductivity of the tissue. Thus, in describing bioelectric currents, we can distinguish between active and passive currents. Active or impressed currents take place inside the column and are denoted by Ji, they sustain a potential diVerence along the column, passive or volume currents take place in the extracellular medium. At any point inside the conductor, the total current density is the sum of the impressed current density and the volume current density: JðrÞ ¼ Ji þ sE ¼ Q dðr r0 Þ þ sE
ð1Þ
This is the current density generating both EEG and MEG signals. The actual common source of EEG and MEG signal does not mean that both techniques share the same performances. In fact, the two techniques look at the same phenomenon from a slightly diVerent perspective. To detail the diVerences, we need to move more in details into the physics of neurophysiology. All electromagnetic phenomena are accurately described by Maxwell’s equations, and this is true also for bioelectric phenomena. However, in this last case, a simplified version of the equations can be used: the so-called quasi-static limit. The time variability of bioelectric phenomena corresponds to a frequency range extending from direct current (DC) up to 1 kHz. Inductive and displacement eVects are negligible at these relatively low frequencies. If we consider a sinusoidal component of the signal at frequency f, it can be seen that the displacement current term and the real current term in Maxwell’s equations are in the ratio of 2ef/. Typical values ffi 0.25 1 m1 and e ffi 1010 Fm1 yield for this ratio about 109f, a quantity that is negligible in the range of frequency of interest. The induction term can be shown to be negligible as well. Therefore, true timedependent terms in Maxwell’s equations may be neglected and, considering also that the biological tissue is free from magnetized particles, the Maxwell’s equations reduce to: div E ¼ r=e0
ð2Þ
curl E ¼ 0
ð3Þ
div B ¼ 0
ð4Þ
FUNDAMENTALS OF EEG, MEG, AND FMRI
curl B ¼ m0 J
71 ð5Þ
where E denotes the electric field, B the magnetic induction (or magnetic field since the magnetic permeability m0 is constant), the electric charge density, J the current density, and e0 the dielectric constant. To these we must add the conservation of charge, which in the quasi-static approximation is given by: div J ¼ 0
ð6Þ
Let us now consider an infinite homogeneously conducting medium. Taking the divergence of Equation (1), and using Equation (6), we obtain: div E ¼ div Ji =s
ð7Þ
which gives, using Equation (2): r2 F ¼ div Ji =s
ð8Þ
where is the electric potential measured by EEG. Substituting Equation (1) into Equation (5) and taking the curl gives, after taking (3) and (4) into account: r2 B ¼ m0 curl Ji
ð9Þ
Equations (8) and (9) show that both the electric potential and the magnetic field at any point of an infinite homogeneous conductor depend only on the impressed currents and not on the volume currents. Moreover, the mathematical sources of the field depend on their divergence and curl rather than on the current themselves. In a bounded medium, the explicit solution of Maxwell’s equation depends on the volume conductor model where the sources are constrained. A general solution for a bounded homogeneously conducting medium [using the density current expression in Equation (1)] is: ð m r r0 3 0 d r BðrÞ ¼ 0 Jðr 0 Þ 4p j r r 0 j3 V ð ð10Þ m0 r r0 m0 r r0 0 0 0 Q ¼ þ s Fðr Þnðr Þ ds 4p j r r0 j3 4p j r r 0 j3 S
ð 1 r r0 3 0 FðrÞ ¼ Jðr 0 Þ d r j r r 0 j3 4ps V ð 1 r r0 1 r r0 ¼ þ Fðr 0 Þnðr 0 Þ ds0 Q 3 j r r0 j j r r 0 j3 4ps 4p
S
where V and S describe the volume or the surface of the volume conductor.
ð11Þ
72
BABILONI et al.
The above equations reveal the important diVerences between EEG and MEG techniques. The most important one is related to the larger eVect that tissue conductivity plays on the value of B or in Equation (10). The conductivity value aVects the field value only through the volume current, that is, the last term of equation. Moreover, if the head is modeled by a homogeneously conducting sphere, the magnetic field can be obtained analytically: BðrÞ ¼
m0 ðF Q r0 Q r0 rrF Þ 4pF 2
ð12Þ
where F ¼ a(ra þ r2 r0 r), a ¼ (r r0), a ¼ jaj, r ¼ jrj. In this expression (valid also in the case of an electric conductivity that is a function only of the radius r), the conductivity value does not play any role at all. Although this is strictly true only for a spherical conductor, that is, for a spherical head, the deviation of the head from the spherical shape is generally small and this conclusion is often a good approximation of the real world. Moreover, most of the currently used MEG sensors detect only one component of the vector magnetic field: the component perpendicularly oriented to the scalp surface, namely Br. From Equation (10) or (12) the expression for Br for a spherically symmetric volume conductor is simply: Br ðrÞ ¼
m0 r r0 Q ^e r 4p j r r0 j3
ð13Þ
Again, this is true only for a piecewise homogeneous sphere model, although the brain shape is not spherical. Thus, we can conclude that when a MEG field component is almost perpendicular to the head surface, its dependency on volume currents is very weak. About the electric potential , it is not possible to obtain a true analytical solution of Equation (11), and only a semianalytical expression can be used. Furthermore, the tissue conductivity always aVects the value of regardless of the volume conductor shape. To estimate , an accurate value of for the volume conductor—that is, the scalp, the skull, and the brain—should be used. There are other important diVerences between EEG and MEG techniques. From Equation (12), it is possible to notice that a radial source Q does not produce any MEG field outside the volume conductor. In contrast (11), this generates an EEG potential diVerence over the scalp. This may appear as a limitation of the MEG technique, but this is not the case in several practical applications. In fact, the cortical surface folding causes the larger part of the cortex to be tangentially orientated. Furthermore, this feature simplifies the explanation of surface electromagnetic activity in terms of cortical sources, namely the MEG fields can explained only in terms of tangential brain sources. The loss of sensitivity of the MEG technique to radial sources can be overcome by the combination of EEG and MEG measurements in simultaneous or parallel experimental recordings.
FUNDAMENTALS OF EEG, MEG, AND FMRI
73
EEG and MEG techniques are very diVerent from f MRI, where a true tomographic picture of the brain activity is obtained thanks to the frequency/ phase encoding used in the MRI technique. With EEG and MEG techniques, each sensor (electrode or coil) senses the activity of the whole brain and most of the data analysis is focused on the procedures needed to separate contribution from the diVerent areas. As mentioned above, EEG measurements are diVerential measurements, that is, we can record diVerence in electric potential between the active electrode and a reference signal. Voltage amplitude at a recording site is then aVected by the specific reference electrode considered. High-resolution EEG technologies are available to remove the eVects of electrode reference and to drastically increase the spatial resolution of the raw EEG (Babiloni et al., 2003). Such technologies include the computation of surface Laplacian (SL) of the recorded potentials, as well as the use of realistic head models to estimate the cortical sources via linear inverse procedure (low-resolution brain electromagnetic tomography, LORETA). However, these deblurring procedures are generally used in conjunction with EEG recordings with 64–128 scalp electrodes and with realistic head models obtained via sequential magnetic resonance images (MRIs) of the subjects. Standard EEG shows a low spatial resolution (5–9 cm), which improves by high-resolution EEG including 64–128 channels and source estimation techniques (1–3 cm); source estimation of MEG data shows a better spatial resolution (0.5–2 cm). Figures 1 and 2 show examples of high-resolution EEG and MEG techniques.
III. EEG and MEG Techniques Allow the Study of Brain Rhythms
EEG and MEG signals have a high temporal resolution (<1 ms) ideal to investigate an emerging propriety of brain physiology, namely the awakening brain rhythms. Spectral analysis methods allow the estimation of EEG and MEG dynamics in terms of the dominant frequencies, power (or amplitude), phase, and coherence. A background spontaneous oscillatory activity of brain neurons at about 10 Hz generates dominant alpha rhythms of resting-state EEG and MEG activity. Since classical studies by Jasper and Penfield (1949), alpha rhythms were recorded from nearly the entire upper cortical surface (including frontal and prefrontal areas) in a large population of patients awake prior to surgery. Highresolution EEG and MEG studies have shown long- and short-range correlations of alpha rhythms depending on age, subjects’ conditions, and ongoing task (Babiloni et al., 2004; Nunez et al., 2001; Salenius et al., 1995; Salmelin and Hari, 1994). Such coherence changes often include frontal and prefrontal regions. This background activity is blocked during sensory and cognitive-motor events
74
BABILONI et al.
Distributed source estimation: thousands of dipoles Potential Scalp EEG
−
+
−
+
LE
Right finger movement (EMGo) “Virtual” electrode
FIG. 1. Some general features of advanced high-resolution EEG techniques for the linear or nonlinear inverse estimation (LE) of cortical sources of scalp potentials. EEG data are recorded with more than 64 electrodes. Head compartments (i.e., brain, meninges, skull, scalp) are modeled with thousands of small triangles obtained by magnetic resonance images. Each triangle incorporates an equivalent current dipole as a mathematical model of neural generators of EEG (MEG) fields. In the figure, it is shown the cortical activity modeled by linear inverse source estimation (LE) of scalp potentials (P) recorded at onset of electromyographic response (EMGo) associated with right middle finger extension in a healthy adult.
(Babiloni et al., 2005a,b, 2006a,b,c, 2008; Pfurtscheller and Lopes da Silva, 1999). Oscillations in other frequency bands, for example, theta (4–7 Hz) and gamma bands (30–70 Hz) also exhibit complex patterns of power and coherence that are modulated by cognitive processes such as working memory and perceptual binding (Srinivasan et al., 2006).
IV. Functional Magnetic Resonance Imaging
Since its inception in 1992 (Bandettini et al., 1992; Kwong et al., 1992; Ogawa et al., 1992), functional magnetic resonance imaging (fMRI) has grown explosively to become a standard and indispensable tool in neuroscience research. fMRI provides a sensitive, noninvasive tool for mapping patterns of activation in human brain. Its spatial resolution is of about 1–5 mm (depending of the intensity of fMRI emissions) and its temporal resolution is of 1 s or longer due to intrinsic latency of
75
FUNDAMENTALS OF EEG, MEG, AND FMRI
L R ittl M in e fi In id g fi ng T de dle ng er Nehumx fi fin er Brock b nge ger w r Eye lid Face and eyeb all
ee Kn kle An s Toe
Lips
Sw all
ow ing
Jaw ue ng To
R
Motor cortex
Hip
Trunk r ulde Sho ow Elb t ris W and H
Pros of dipole analysis: somatotopy of SEFs within the “omega” post-central region
L
Thumb
(Sensory cortex)
Median
Little
FIG. 2. Dipole source locations of somatosensory evoked fields (SEFs) relative to electrical stimulation of thumb, middle, and little fingers in a healthy adult. It is noting that these locations successfully distinguish circumscribed cortical activity within specialized zones of the omega of primary hand somatosensory cortex. These zones map a sort of ‘‘homunculus’’ within the postcentral gyrus of cerebral cortex.
hemodynamic response at the basis of fMRI. With respect to fMRI, positron emission tomography (PET) oVers a worse spatial (about 1 cm) and temporal (seconds to minutes) resolution. EEG and MEG techniques oVer better temporal resolution (millisecond), but owing to the nature of the inverse problem the information on spatial localization may be ambiguous; the intrinsic spatial resolution is in any case poorer. On the other side, fMRI can never hope to match the temporal resolution of electrophysiological methods, because the method is based on the indirect measurement of activation via the associated hemodynamic changes.
V. Physiological Generation of Blood Oxygen Level-Dependent Signal
The most widely used fMRI technique is based on the blood oxygen level-dependent (BOLD) contrast. This application uses deoxyhemoglobin as an endogenous tracer, mapping regional brain activation by means of standard T2*weighted MRI sequences, commonly with the echo-planar imaging (EPI) method (Mansfield, 1977). Specifically, a stimulus (or task) change causes a change in
76
BABILONI et al.
neural activity that, in turn, produces a hemodynamic response, altering cerebral blood flow, cerebral blood volume, and oxygen metabolism. The combined hemodynamic and metabolic changes alter the local magnetic susceptibility, creating magnetic field distortions within and around the blood vessels, and this microscopically inhomogeneous field produces a slight alteration in the local MR signal (Buxton et al., 2004). These changes in susceptibility originate from the diVerent magnetic nature of deoxyhemoglobin (paramagnetic) and oxyhemoglobin (diamagnetic). The current explanation of the BOLD eVect takes into account the complex interplay of three physiological parameters: (i) the local rate of metabolic consumption of oxygen. An increase in this parameter will increase the rate of oxygen consumption and, hence, will increase the amount of deoxyhemoglobin; (ii) the regional cerebral blood volume. An increase in blood volume will increase the absolute amount of blood and, hence, of oxyhemoglobin; (iii) the regional cerebral blood flow (rCBF). An increase in rCBF will lead to an increased washout of deoxyhemoglobin. An increase in neural activity will lead to an increase in these parameters. However, the increase in rCBF is dominant over the other changes, with the consequence that increased neural activity leads to an increase in BOLD signal as measured by T2*-weighted imaging. Regarding the neurovascular coupling, there is consensus that in healthy subjects, BOLD fMRI signal is related predominantly to postsynaptic integrative event-related neural activity (Logothetis et al., 2001). Nevertheless, full mechanism is still debated and constitutes a field of active and intense research. To properly design fMRI experiments and correctly interpret the results, diVerent aspects of the BOLD signal have to be considered. First, the functional contrast is relative, meaning that the BOLD contrast always represents a comparison of T2* sensitive signals across two or more behavioral states. Second, BOLD contrast does not reflect a single physiological process, but rather represents the combined eVects of blood flow, blood volume, and oxygen utilization. Third, the sluggish response of the vascular system to neural events delays, disperses, and smoothes over time the underlying neural signal. The temporal profile of the BOLD response to a short stimulus may show three distinct phases: (i) a small negative initial response that attains its minimum value at 2–3 s poststimulus; (ii) the main, positive, BOLD response that is conventionally used in fMRI experiments, with a time to peak of about 5 s and a maximum percent signal change of about 1% at 1.5 T; (iii) a poststimulus undershoot which may take up to a minute to return to baseline.
FUNDAMENTALS OF EEG, MEG, AND FMRI
77
VI. Typical f MRI Experimental Designs
There are two main f MRI experimental designs: ‘‘block’’ and ‘‘event related.’’ In the typical block design, stimuli of the same type are presented in blocks lasting several seconds to minutes, which are alternated with rest blocks (same duration) after brief interstimulus interval of few seconds. In the typical event-related design, stimuli are presented individually at interstimulus intervals long enough to allow a full or partial BOLD response. The same paradigms are used in studies requiring subjects to perform tasks rather than simply receiving physical stimuli. Designing eYcient block designs involves deciding on the number and timing of stimuli to present within a block, the duration of the block, the number of times a specific block type should be repeated, and the number of diVerent classes of stimuli to include during a single run. Block designs typically have more power than event-related designs do when detecting the magnitude of the BOLD response. In contrast, event-related designs are more eYcient at estimating the shape of the hemodynamic response (Liu et al., 2001). Event-related designs also allow investigators to examine the BOLD response based on response types, such as comparing correct with incorrect responses and to randomize the presentation of stimuli from diVerent stimulus classes. In a typical fMRI experiment at 1.5 T, images are collected rapidly with a repetition time between subsequent images of a particular slice of about 2 s, and dynamic images are acquired over several minutes as stimuli are presented to the subject. Then the time course of each image voxel is analyzed to detect BOLD signal time courses that show a significant correlation with the stimulus. The coupling of variations in MR signal response to variations of stimulus presentation can be quantified by a regression model where dummy coding of the stimulus condition is used as an independent variable to predict the MR signal response. The correlation coeYcient from this analysis can be used to measure the statistical significance of activation, whereas the regression weight can be used to measure the magnitude of the BOLD response. The correlation coeYcient is typically calculated voxel by voxel and visualized on structural MRI brain images as a color scale map.
VII. BOLD-f MRI Techniques in Clinical Environment
Although the primary field of application remains the cognitive neurosciences, important applications are also found in clinical studies (Rossini et al., 2004) and surgical treatment planning (Vlieger et al., 2004).
78
BABILONI et al.
FIG. 3. Example of presurgical fMRI mapping. The fMRI measurement allowed to identify the two principal language areas, both in the left hemisphere: the Broca’s area and the Wernicke’s area. The Wernicke’s area is very close to the tumor, indicated in the picture by the black contour.
As an example, a typical presurgical assessment will be described for a patient bearing a tumor. In this case the aim of the fMRI investigation is to localize eloquent areas prior to tumor resection. The task or stimulus is chosen according to the lesion position to characterize the activity in brain areas close to the tumor or, eventually, a plastic reorganization of the brain. In the example the lesion is in the left hemisphere, close to the language areas. During the fMRI acquisition, the patient was required to perform a ‘‘word generation task’’ (e.g., think about words beginning with a certain letter) according to a block paradigm alternating task blocks of 20 s with rest blocks with the same duration. The main activated areas are shown in Fig. 3, superimposed on a structural MR image of the patient. This information is obtained noninvasively and is extremely important for the surgeon when planning the surgical treatment.
References
Babiloni, F., Babiloni, C., Carducci, F., Cincotti, F., and Rossini, P. M. (2003). The stone of madness’ and the search for the cortical sources of brain diseases with non-invasive EEG techniques. Clin. Neurophysiol. 114(10), 1775–1780. Babiloni, C., Binetti, G., Cassetta, E., Cerboneschi, D., Dal Forno, G., Del Percio, C., Ferreri, F., Ferri, R., Lanuzza, B., Miniassi, C., Moretti, D. V., Nobili, F., et al. (2004). Mapping distributed sources of cortical rhythms in mild Alzheimer’s disease. A multicentric EEG study. Neuroimage 22(1), 57–67.
FUNDAMENTALS OF EEG, MEG, AND FMRI
79
Babiloni, C., Babiloni, F., Carducci, F., Cincotti, F., Del Percio, C., Della Penna, S., Franciotti, R., Pienotti, S., Pizzella, V., Rossini, P. M., Sabatini, E., Torquati, K., et al. (2005a). Human alpha rhythms during visual delayed choice reaction time tasks: A magnetoencephalography study. Hum. Brain Mapp. 24(3), 184–192. Babiloni, C., Cassetta, E., Chiovenda, P., Del Percio, C., Ercolani, M., Moretti, D. V., MoVa, F., Pasqualetti, P., Pizzella, V., Romani, G. L., Tecchio, F., Zappasodi, F., et al. (2005b). Alpha rhythms in mild dements during visual delayed choice reaction time tasks: A MEG study. Brain Res. Bull. 65(6), 457–470. Babiloni, C., Benussi, L., Binetti, G., Cassetta, E., Dal Forno, G., Del Percio, C., Ferreri, F., Ferri, R., Frisoni, G., Guidoni, R., Miniassi, C., Rodriguez, G., et al. (2006a). Apolipoprotein E and alpha brain rhythms in mild cognitive impairment: A multicentric electroencephalogram study. Ann. Neurol. 59(2), 323–334. Babiloni, C., Binetti, G., Cassetta, E., Dal Forno, G., Del Percio, C., Ferreri, F., Ferri, R., Frisoni, G., Hirata, K., Lanuzza, B., Miniassi, C., Moretti, D. V., et al. (2006b). Sources of cortical rhythms change as a function of cognitive impairment in pathological aging: A multicenter study. Clin. Neurophysiol. 117(2), 252–268. Babiloni, C., Brancucci, A., Vecchio, F., Arendt-Nielsen, L., Chen, A. C., and Rossini, P. M. (2006c). Anticipation of somatosensory and motor events increases centro-parietal functional coupling: An EEG coherence study. Clin. Neurophysiol. 117(5), 1000–1008. Babiloni, C., Del Percio, C., Brancucci, A., Capotosto, P., Le Pera, D., Marzano, N., Valeriani, M., Romani, G. L., Arendt-Nielsen, L., and Rossini, P. M. (2008). Pre-stimulus alpha power aVects vertex N2-P2 potentials evoked by noxious stimuli. Brain Res. Bull. 75(5), 581–590. Bandettini, P. A., Wong, E. C., Hinks, R. S., Tikofsky, R. S., and Hyde, J. S. (1992). Time course EPI of human brain function during task activation. Magn. Reson. Med. 25, 390–397. Buxton, R. B., Uludag, K., Dubowitz, D. J., and Liu, T. T. (2004). Modeling the hemodynamic response to brain activation. Neuroimage 23(Suppl. 1), 220–233. Jasper, H. D., and Penfield, W. Electrocorticograms in man. (1949). EVects of voluntary movement upon the electrical activity of the precentral gyrus. Arch. Psychiatr. Z. Neurol. 183, 163–174. Kwong, K. K., Belliveau, J. W., Chesler, D. A., Goldberg, I. E., WeisskoV, R. M., Poncelet, B. P., Kennedy, D. N., Hoppel, B. E., Cohen, M. S., and Turner, R. (1992). Dynamic magnetic resonance imaging of human brain activity during primary sensory stimulation. Proc. Natl. Acad. Sci. USA 89, 5675–5679. Liu, T. T., Frank, L. R., Wong, E. C., and Buxton, R. B. (2001). Detection power, estimation eYciency, and predictability in event-related fMRI. NeuroImage 13, 759–773. Logothetis, N. K., Pauls, J., Augath, M., Trinath, T., and Oeltermann, A. (2001). Neurophysiological investigation of the basis of the fMRI signal. Nature 412, 150–157. Mansfield, P. (1977). Multi-planar image formation using NMR spin echoes. J. Phys. C 10, L55–L58. Nunez, P. L., Wingeier, B. M., and Silberstein, R. B. (2001). Spatial-temporal structures of human alpha rhythms: Theory, micro-current sources, multiscale measurements, and global binding of local networks. Hum. Brain Mapp. 13, 125–164. Ogawa, S., Tank, D. W., Menon, R., Ellermann, J. M., Kim, S. G., Merkle, H., and Ugurbil, K. (1992). Intrinsic signal changes accompanying sensory stimulation: Functional brain mapping with magnetic resonance imaging. Proc. Natl. Acad. Sci. USA 89, 5951–5955. Pfurtscheller, G., and Lopes da Silva, F. H. (1999). Event-related EEG/MEG synchronization and desynchronization: Basic principles. Electroencephal. Clin. Neurophysiol. 110, 1842–1857. Rossini, P. M., Altamura, C., Ferretti, A., Vernieri, F., Zappasodi, F., Caulo, M., Pizzella, V., Del Gratta, C., Romani, G. L., and Tecchio, F. (2004). Does cerebrovascular disease aVect the coupling between neuronal activity and local haemodynamics? Brain 127(1), 99–110.
80
BABILONI et al.
Salenius, S., Kajola, M., Thompson, W. L., Kosslyn, S., and Hari, R. (1995). Reactivity of magnetic parieto-occipital alpha rhythm during visual imagery. Electroencephalogr. Clin. Neurophysiol. 95(6), 453–462. Salmelin, R., and Hari, R. (1994). Characterization of spontaneous MEG rhythms in healthy adults. Electroencephalogr. Clin. Neurophysiol. 91(4), 237–248. Srinivasan, R., Winter, W. R., and Nunez, P. L. (2006). Source analysis of EEG oscillations using high-resolution EEG and MEG. Prog. Brain Res. 159, 29–42. Vlieger, E. J., Majoie, C. B., Leenstra, S., and Den Heeten, G. J. (2004). Functional magnetic resonance imaging for neurosurgical planning in neurooncology. Eur. Radiol. 14, 1143–1153.
IMPLICATIONS OF BRAIN PLASTICITY TO BRAIN–MACHINE INTERFACES OPERATION: A POTENTIAL PARADOX?
Paolo Maria Rossini Department of Neurology, Universita` Campus Biomedico di Roma, Via Alvaro del Portillo 21, 00128 Roma, Italy
I. II. III. IV. V.
Introduction Brain Plasticity Brain Plasticity and BMI Systems Monitoring Plasticity During BMI Control Conclusions References
The adult brain has the remarkable ability to plastically reorganize itself in order to record memories (experiences), to add abilities, and to learn skills, significantly expanding the carnet of resources useful for facing and solving the unpredictability of any daily life activity, that is, artistic and cultural activities. Brain plasticity also plays a crucial role in reorganizing central nervous system’s networks after any lesion, being it sudden and localized, or progressive and diVuse, in order to partly or totally restore lost and/or compromised functions. In severely aVected neurological patients unable to move and to communicate with the external environment, technologies implementing brain–machine interfaces (BMIs) can be of valuable help and support. Subjects operating within a BMI frame must learn how to produce a meaningful signal for an external reader; how to increase the signal-to-noise ratio at a level which makes it suitable for rapid communication with the machine; and how to improve the speed and specificity (bit rate) of signal production as a new language for governing and controlling a machine. Since it is of absolute importance for the patient to be able to maintain such a skill for a prolonged lapse of time (i.e., until his/her lost abilities are restored by a therapy and/or a diVerent technology), neurophysiological phenomena at the base of plastic changes are obviously of remarkable importance within any BMI and are the content of the present chapter.
INTERNATIONAL REVIEW OF NEUROBIOLOGY, VOL. 86 DOI: 10.1016/S0074-7742(09)86006-6
81
Copyright 2009, Elsevier Inc. All rights reserved. 0074-7742/09 $35.00
82
PAOLO MARIA ROSSINI
I. Introduction
In 1894 Cajal defined the plasticity of the cellular processes as dynamic throughout life, being maximum during childhood and adolescence, lower in the adult, and almost completely disappearing in the aged (DeFelipe, 2006). We nowadays know that the aging brain maintains a good capacity for regional plastic reorganization (Burke and Barnes, 2006; Del Arco et al., 2003; Hedden and Gabrieli, 2004; Mora et al., 2003, 2007; Segovia et al., 1999, 2001); it must be recognized that Cajal’s work paved the way to the successive popularity that brain plasticity would have received in the 1940s of the last century under the impetus of Donald Hebb’s work which can be summarized in the sentence ‘‘cells that fire together, wire together.’’ Hebb can be considered the father of the first theoretical explanation of the mechanisms for synaptic plasticity. His theory foresaw that presynaptic cell’s repeated and persistent stimulation of a postsynaptic cell produces an increase in synaptic eYcacy. That is, referring to the exciting neuron as the neuron A, and to the excited neuron as the neuron B, Hebb’s theory claimed that when A excited B repeatedly and/or persistently, then the eYciency with which A would excite B in the future would have been increased. Hebb hypothesized diVerent processes responsible for the plastic changes in growth process (like the emission of new synaptic knobs on the axon of A) and/or in metabolic changes taking place in one or both of the cells. By the end of the 1940s, he described another important plastic process, by which two cells would become ‘‘associated’’ if they were repeatedly activated at the same time: the association would have made the further activity in one of the two cells to facilitate activation of the other (Hebb, 1949). Hebbian’s theory of plastic learning was confirmed and enriched with the experiments on long-term potentiation (LTP) mechanisms made in the laboratories of Eric Kandel (1970), which experimentally demonstrated manifestations of associative learning in which simultaneous activation of cells leaded to pronounced increases in synaptic strength. Another supporter of Hebbian’s theories was Jacques Paillard (1976), in his work, recently translated in English (Will et al., 2008), this author firstly reduced the use of the word ‘‘plasticity’’ only to those enduring changes of the organisms internal connectivity or elements, which were actually responsible for new functions or abilities of the organism itself. According to Paillard’s theory, plasticity can be caused by three diVerent processes (Manning, 2008): (1) evolutionary plasticity, indicating some ability for structural mutation of the genome; (2) genetic plasticity, that is, the structural malleability linked with brain maturation; and (3) adaptive plasticity, as the capacity of the fully developed system to change its own structure and expand its behavioral repertoire.
BRAIN PLASTICITY AND BMIs
83
II. Brain Plasticity
Adaptive plasticity in the CNS, that is, what we nowadays refer as brain plasticity, is further summarized into three diVerent forms: habituation, sensitization, and associative learning. We refer to habituation when the nervous system, exposed to prolonged or intense stimuli (e.g., a very loud environment), tends to progressively reduce its response, thus ‘‘habituating’’ to the stimuli. All sensory inputs can, and actually do, lead to habituation, thought the diVerent stimulus’ time lasting or amplitude onset required to trigger the phenomenon, as well as its further persistence (which could last from few seconds till several years), may depend on many variables and may thus diVer substantially between diVerent stimuli. In most of the cases the structural adaptation is at the junction (synapse) between the sensory neuron (or a neuron in the sensory pathway) and the neurons that sustain the stimulus response. The structural changes do not take place, instead, on the sensory neuron’s axon (Kolb and Wishaw, 2005)—even though the sensors themselves can habituate (e.g., retinal photoreceptors). As in the case of the simple network of neuron A and neuron B described above (i.e., any network responsible of a monosynaptic motor reflex, as in Fig. 1), the change would be at the synapse connecting A and B. More specifically, the changes would be in the presynaptic regions, where a reduced concentration of neurotransmitter would be released in the synaptic cleft (the space between the pre- and the postsynaptic sites) as a result of Ca2þ mediated eVects triggered by the prolonged or intense stimulus. How a prolonged stimulus causes a reduction of Ca2þ in the presynaptic cleft is still unknown. It was initially thought that Ca2þ slowly depleted from the synapsis due to a prolonged input, although it does not explain how the phenomenon persists, as Ca2þ concentration in the presynaptic cleft seems to return fairly quickly to normal conditions. Some evidence would instead point to a reduction in responsiveness of the presynaptic Ca2þ channels (Edmonds et al., 1990). Contrary to habituation, sensitization leads to progressively augmented neuronal response to a stable and repetitive stimulus. Specifically, when several stimuli take place and present themselves at the same time (in the form of several, independent inputs) to a single neuron, it may happen the neuron to release more neurotransmitter than if only one of the stimuli was present (i.e., only one input is presented to the neuron). Studies on sensitization (Fig. 2) have found that the ‘‘extra’’ input comes from another neuron, an interneuron, and synapses with the sensory terminal, not with the postsynaptic cell that is linked to response pathways. The activity of the interneuron modulates, through the produced release of serotonine (5HT), the global amount of neurotransmitter which will be released in
84
Procedure
1 Gill withdraws from water jet.
Siphon
2
1
Gill no longer withdraws from water jet, demonstrating habituation.
With habituation, the influx of calcium ions in response to an action potential decreases,...
Ca2+
2
Presynaptic membrane
Water jet
...resulting in less neurotransmitter released at the presynaptic membrane...
Results The sensory neuron stimulates the motor neuron to produce gill withdrawal before habituation.
Postsynaptic membrane
3
Ca2+ Sensory neuron Skin of siphon
Motor neuron
...and less depolarization of the postsynaptic membrane.
Conclusion
Gill muscle
Withdrawal response weakens with repeated presentation of water jet (habituation) owing to decreased Ca2+ influx and subsequently less neurotransmitter release.
FIG. 1. Example of habituation in the simple reflex in a snail (reproduced from Kolb and Wishaw, 2005).
PAOLO MARIA ROSSINI
After repeated stimulation
1
Procedure
Serotonin reduces K+ efflux through potassium channels, prolonging an action potential on the siphon sensory neuron.
Gill withdrawal Interneuron A single shock to the tail enhances the gill-withdrawal response (sensitization).
Serotonin
Motor neuron
K+ 2
Sensory neuron
Shock
Results An interneuron receives input from a “shocked” sensory neuron in the tail and releases serotonin onto the axon of a siphon sensory neuron.
3
Interneuron Ca2+ Sensory neuron Skin of siphon
Motor neuron
The prolonged action potential results in more Ca2+ influx and increased transmitter release,...
...causing greater depolarization of the postsynaptic membrane after sensitization.
BRAIN PLASTICITY AND BMIs
Water jet
Second messenger
Conclusion Gill muscle
Enhancement of the withdrawal response after a shock is due to increased Ca2+ influx and subsequently more neurotransmitter release.
85
FIG. 2. Example of sensitization in the simple reflex in a snail (reproduced from Kolb and Wishaw, 2005).
86
PAOLO MARIA ROSSINI
the synaptic cleft inducing downregulation of Kþ release by the sensory cell, which keeps the sensory terminal highly excited, causing more Ca2þ to enter the terminal, and finally upregulating further neurotransmitter release, which increases the response of the postsynaptic cells. This high responsiveness may persist in time due to the long lasting eVects of serotonine on the Kþ channels (Edmonds et al., 1990), during which any further input coming from the sensor will produce an amplified response of the sensitized neuron. In contrast with the sensitization case, in associative learning the second input does not synapse with the sensory cell. So long as the second (or third, etc.) input aVects the excitation state of the postsynaptic (response) cell when the sensory input comes in, the synapse between the sensory and the response cell (not necessarily motor) will maintain persistent changes, regardless of how far in space the sensory and the other inputs are localized. If both the sensory and the response cells are excited at the same time, the connection between the two cells will be strengthened, that is, the response intensity to a stimulus will increase. Moreover, after this eVect consolidates, the second input alone will cause the same response as if both inputs were arrived, hence the ‘‘associative’’ term. This process has been particularly investigated in the hippocampus (Fig. 3), in the form of LTP or depression (LTD).
Cerebral cortex
3
Lateral geniculate nucleus
Hippocampus (IX) Medial geniculate nucleus
FIG. 3. Established locus of long-term potentiation, an associative learning process (reproduced from Bear et al., 2001).
87
BRAIN PLASTICITY AND BMIs
Any of the above plastic changes can lead to the respective appearance and disappearance of synaptic spines, clefts, and contacts (Fig. 4). Paillard’s theory defines as plastic only those changes that are both structural and functional in nature. The functional structure of the CNS is represented by neurons and their networks of connections, and the wide range of neurobiological mechanism which participates into brain plasticity includes both cellular and anatomical changes which reflect synaptic eYcacy and synaptic redundancy, synaptogenesis, dendritic arborization, and activity-dependent reinforcement of previous existing, but functionally silent synaptic connections (Calabresi et al., 2003). Learning new skills, as well as recovering (via endogenous brain reorganization) from a lesion, are indeed based on neural plasticity (Elbert et al., 1995; Pascual-Leone et al., 1995; Rossi et al., 1998, 2003; Tecchio et al., 2000; Ween, 2008). The external stimuli able to induce cerebral plasticity can be physiological or pathological in nature, and diVerent plastic events can be conveniently distinguished: developmental plasticity mostly elicited by environmental experience in the maturing brain, adaptive plasticity in learning new skills life through (Rossi et al., 1998) and restorative plasticity in response to brain damage.
III. Brain Plasticity and BMI Systems
Any voluntary task is performed by physiological actuators (i.e., subcortical, spinal relays, and muscles) which translate into an action the intentions planned by the subject. When the task is accomplished with good performance, the cortical networks which took part in controlling the actuators are strengthened (in terms of their connections and/or their reciprocal sensitivity to their mutual
Motor neuron
Sensory neuron
Control
Habituated
Sensitized
FIG. 4. Structural eVects of prolonged sensitization and habituation. Notice the modifications of collaterals and synaptic contacts between the two cells compared to the control condition (reproduced from Kolb and Wishaw, 2005).
88
PAOLO MARIA ROSSINI
activation/inhibition) accordingly with the mechanisms foreseen by the above described brain plasticity theoretical framework. The actuators, though, are relatively blind to those plastic modifications in the cortical neuronal networks following learning or training. From the actuators’ point of view, the only eVect of any plastic changes is that they perform the same task ‘‘faster and better.’’ If this is true for any task performed with human’s physiological actuators, it can be radically diVerent, and somehow paradoxical, when the task is executed with a brain–machine interface (BMI): in a BMI operated task planners and actuators coincide, that is, the actuators are the same cortical neural networks which plan the task and, hence, undergo plastic changes during learning. To operate a BMI system, subjects must first of all encode their acquired skill (i.e., how to drive a cursor in a computer display) in a cortical network as accessible as possible to be retrieved by a reader (the imaging techniques implemented in the architecture) which probes the brain through the scalp; depending on the protocol of the BMI, such a cortical network might be quite diVerent from the one routinely activated in physiological conditions to perform a similar task, and hence usually needs to be learned from scratch. The specific activity of the cortical network required to operate he BMI needs to achieve a certain level of automation, so that the mental eVort required to the user to operate the BMI is reduced while a suYcient communication bit-rate is maintained. The level of automation, though, should not be too high, otherwise a reduction of cortical activity in favor of subcortical networks could happen (as normally happens for lot of tasks, like the finger tapping control of a piano player); and subcortical networks’ activity is trickier to access for any reader.
IV. Monitoring Plasticity During BMI Control
Learning how to control a BMI, in other words, can change the actuators in a totally new and unpredictable way. The factual eVects of plasticity on the neuronal networks devoted to BMIs control, and the relative diVerences with the ‘‘normal’’ plasticity, can be explored. Plastic brain changes in healthy humans can be measured and followed up via noninvasive methods, namely those based on local changes of blood flow and metabolism in the brain (i.e., positron emission tomography, PET and functional magnetic resonance imaging, fMRI) and those reflecting electromagnetic signals (EEG, MEG, TMS). Both these sources of information stem from the amount and duration of neuronal firing as well as on the developing and dynamic connectivity of neuronal assemblies which bind and unbind time by time to sustain a given function. Needless to say, combination and integration of the two methodologies represent the best solution since they combine the best of structural and temporal discrimination. It is hence conceivable to build up an instrumental approach through which individual
BRAIN PLASTICITY AND BMIs
89
training schedules could be tailored on the basis of the surviving neural networks which could be utilized in individual patients according to their residual abilities. Training protocols should be rendered appropriate to be carried out within the fMRI environment and under the helmet of MEG sensors. Meanwhile, high density EEG recordings should be acquired and TMS/EEG sessions should also be carried out. From the analysis of the resulting integrated data, we can extract information on: (1) the main neuronal networks utilized to optimally perform the task for BMI control; (2) the localization and chronological hierarchy of the relays constituting individual and distributed networks; (3) time-varying coherence modifications of the examined brain rhythms as well as directions of such modifications; (4) time-varying synchronization likelihood modifications; (5) corticocortical connectivity examined via EEG/MEG as in the previous two points and TMS/EEG recordings. Once the main picture of the brain activities pivotal for sustaining the BMI/ BCI tool is depicted, possible ‘‘reinforcements’’ of the relevant aspects to maintain (or even improve) them along time should be envisaged. They can include rTMS of the cortical relays along the relevant networks (more focal eVects) and/or tDCS of the entire network. Monitoring of the ongoing situation can be carried out mainly via EEG techniques which are cheap and can be easily performed at home and transferred for analysis via a Web-based Internet transmission.
V. Conclusions
It is self-evident that brain plasticity is a tool of pivotal importance for creating an eYcient ‘‘bridge’’ between the subject’s brain and a machine or a computer performing his/her orders and will. In this chapter, we envisaged a possible approach which should permit to build up optimal communication bit-rate and to maintain it stable in time, hence to assure over time eYciency (high specificity and sensitivity with minimal error rate) and speed of reader and interpreter. Nowadays, brain plasticity monitoring thanks to the integration of diVerent technologies aiming to discriminate the best of space and time analysis of the neuronal networks involved in BMI tools is possible and can be organized in a BMI-oriented way.
References
Bear, M. F., Connors, B. W., and Paradiso, M. A. (2001). ‘‘Neuroscience: Exploring the Brain,’’ 2nd ed. Lippincot Williams & Wilkins, Baltimore, MD. Burke, S. N., and Barnes, C. A. (2006). Neural plasticity in the ageing brain. Nat. Neurosci. 7, 30–40.
90
PAOLO MARIA ROSSINI
Calabresi, P., Centonze, D., Pisani, A., Cupini, L., and Bernardi, G. (2003). Synaptic plasticity in the ischaemic brain. Lancet Neurol. 2, 622–629. DeFelipe, J. (2006). Brain plasticity and mental processes: Cajal again. Nat. Rev. Neurosci. 7, 811–817. Del Arco, A., Segovia, G., Fuxe, K., and Mora, F. (2003). Changes of dialysate concentrations of glutamate and GABA in the brain: An index of volume transmission mediated actions? J. Neurochem. 85, 23–33. Edmonds, B., Klein, M., Dale, N., and Kandel, E. R. (1990). Contributions of two types of calcium channels to synaptic transmission and plasticity. Science 250, 1142–1147. Elbert, T., Pantev, C., Wienbruch, C., Rockstroh, B., and Taub, E. (1995). Increased cortical representation of the fingers of the left hand in string players. Science 270, 305–307. Hebb, D. O. (1949). ‘‘The Organization of Behavior.’’ Wiley, New York. Hedden, T., and Gabrieli, J. D. E. (2004). Insights into the ageing mind: A view from cognitive neuroscience. Nat. Rev. Neurosci. 5, 87–96. Kandel, E., Casteixucci, V., Pinsker, H., and Kupfermann, I. (1970). The role of synaptic plasticity in the short-term modification of behavior. In ‘‘Short-Term Changes in Neural Activity and Behavior’’ (G. Horn and R. Hinde, Eds.), Cambridge University Press. Kolb, B., and Wishaw, I. Q. (2005). ‘‘An Introduction to Brain and Behavior,’’ 2nd ed. Worth Publishers, New York. Manning, L. (2008). Do some neurological conditions induce brain plasticity processes? Behav. Brain Res. 192(1), 143–148. Mora, F., Del Arco, A., and Segovia, G. (2003). Glutamate-dopamine interactions in striatum and nucleus accumbens of the conscious rat during aging. In ‘‘Basal Ganglia VI’’ (A. M. Graybiel, Ed.), pp. 615–622. Plenum Press, New York. Mora, F., Segovia, G., and del Arco, A. (2007). Aging, plasticity and environmental enrichment: Structural changes and neurotransmitter dynamics in several areas of the brain. Brain Res. Rev. 55(1), 78–88. Pascual-Leone, A., Dang, N., and Cohen, L. G. (1995). Modulation of muscle responses evoked by transcranial stimulation during the acquisition of new fine motor skills. J. Neurophysiol. 74, 1037–1045. Rossi, S., Pasqualetti, P., Tecchio, F., Pauri, F., and Rossini, P. M. (1998). Corticospinal excitability modulation during mental simulation of wrist movements in human subjects. Neurosci. Lett. 243, 147–151. Rossini, P. M., Calautti, C., Pauri, F., and Baron, J. C. (2003). Post-stroke plastic reorganisation in the adult brain. Lancet Neurol. 2, 493–502. Segovia, G., Del Arco, A., and Mora, F. (1999). EVects of aging on the interaction between glutamate, dopamine and GABA in striatum and nucleus accumbens of the awake rat. J. Neurochem. 73, 2063–2072. Segovia, G., Porras, A., Del Arco, A., and Mora, F. (2001). Glutamatergic neurotransmission in aging: A critical perspective. Mech. Ageing Dev. 122, 1–29. Tecchio, F., Bicciolo, G., De Campora, E., Pasqualetti, P., Pizzella, V., Indovina, I., Cassetta, E., Romani, G. L., and Rossini, P. M. (2000). Tonotopic cortical changes following stapes substitution in otosclerotic patients: A magnetoencephalographic study. Hum. Brain Mapp. 10, 28–38. Ween, J. E. (2008). Functional imaging of stroke recovery: An ecological review from a neural network perspective with an emphasis on motor systems. J. Neuroimag 18, 227–236. Will, B., Dalrymple-Alford, J., WolV, M., and Cassel, J. C. (2008). The concept of brain plasticity-Paillard’s systemic analysis and emphasis on structure and function (followed by the translation of a seminal paper by Paillard on plasticity). Behav. Brain Res. 192(1), 2–7.
AN OVERVIEW OF BMIs
Francisco Sepulveda Brain–Computer Interfaces Group, Department of Computing and Electronic Systems, University of Essex, Wivenhoe Park, Colchester, Essex CO4 3SQ, United Kingdom
I. Introduction II. Main Elements in a BMI A. Mental Task Selection B. Feature Selection C. Translation Algorithms D. Output and Feedback Methods III. BMI Types A. Invasive Versus Non Invasive B. Dependent Versus Independent BMIs C. Spontaneous Versus Evoked Versus Event Related D. Synchronous Versus Asynchronous IV. BMIs and the User’s Ability A. Higher Level Operational Protocol B. Pattern Recognition Versus Operant Conditioning V. Conclusion References
Research in BMIs has grown rapidly in the last few years. However, little attention has been paid to the overall system behavior, most published work being focused on the signal classification (i.e., translation) stage. More recently an increasing amount of work has been centred around the feature selection stage that precedes translation. The emphasis in feature selection and translation has stemmed from the large number of researchers with a machine learning or pattern recognition background who have recently joined the field. While there is an important contribution to BMIs, two crucial elements have not been suYciently explored: the selection of suitable mental tasks and feedback protocols. This review presents an overview of BMIs and its main elements, with a focus on why each stage is important for the overall performance of such systems.
INTERNATIONAL REVIEW OF NEUROBIOLOGY, VOL. 86 DOI: 10.1016/S0074-7742(09)86007-8
93
Copyright 2009, Elsevier Inc. All rights reserved. 0074-7742/09 $35.00
94
FRANCISCO SEPULVEDA
I. Introduction
BMIs may be defined as any system devised to measure brain activity and, from it alone, to translate a person’s intentions into commands to various devices, whether they be external (e.g., a computer) or internal (e.g., an implanted neuromuscular stimulation system) to the user. This implies that even if a mental activity results in signals going to muscles, spinal cord, peripheral nerves, or the autonomic nervous system and its outputs, a BMI will not use any of these and will instead do the intention translation task based exclusively on the brain signals (but, see dependent vs independent BMIs below). From the definition above, it must be understood that BMIs as developed today are unidirectional: the brain controls the machine; the machine does not control the brain, at least not explicitly. While BMIs may include the reverse mode (i.e., the machine controlling the brain) in the future, that is not how the term is used at the moment. We thus start a BMI system by determining which of the various brain monitoring systems available is best suited for the task at hand. Alternatives include, but are not limited to: electroencephalographic signals (EEG), magnetoencephalographic signals (MEG), positron emission tomography (PET), functional magnetic resonance imaging (fMRI), optical imaging (NIRS, near-infrared systems), and implanted methods of recording electrical activity, for example, via electrocorticograms (ECoG) and intracerebral electrodes. Of the above, only EEG has the two properties that are ideal for BMIs: noninvasiveness and portability. NIRS devices are portable as well, but they have two drawbacks: They have an invasive component in the strictest sense of the world: brain function is equated to the amount of infrared light reflected by brain tissue, the emitter, and sensor being near each other on the scalp. Significant nearinfrared light is absorbed by the brain tissue, the long-term eVects of which have not been investigated yet. NIRS have a long response delay (i.e., 10 s or more, which is long compared to a few 100 ms in EEG-based systems) due to the slow nature of the monitored hemodynamics. The other nonimplanted technologies mentioned above suVer more serious problems toward their use in BMIs. PET requires the administration of radioactive substances to the human user, and fMRI still requires very bulky and heavy equipment, both of which are therefore not suitable for BMI applications, although they are very powerful tools for studying brain function in laboratory settings. Further, fMRI, like NIRS, yields long response delays. Finally, MEG
95
AN OVERVIEW OF BMIs
systems are very large as well, although they are noninvasive and yield better spatial resolution than EEG systems. EEG systems are thus by far the most commonly used in BMI research. However, regardless of the type of brain monitoring system chosen, there are a number of elements that are common to all BMIs (Fig. 1). First (stage 1), the experimental protocol must be designed to suit the application and the environment in which the BMI will be used. This includes choice of mental task, stimulus parameters (e.g., visual stimulus timing and constraints), and minimization of unwanted stimuli and distractions that may aVect the properties of the signals to be monitored. Once a suitable stimulus or mental task protocol is designed, signals can be monitored and used. This can be done by means of the bioinstrumentation mentioned above, including the necessary amplifiers and analog filters to remove some of the unwanted interferences. Stage 2: The filtered signals are then further processed to extract the relevant information (i.e., information that correlates well with the mental tasks under investigation) within them. Then (stage 3, which can be combined with stage 2), the extracted features are used by the translation algorithm to send commands to the device to be controlled. Finally (stage 4), feedback can be sent from the translation stage’s output to the user, though this is not done explicitly in many BMIs yet. Further, while we do know that the user adapts to the entire system and that the device’s response plays a role in this process, user adaptation mechanisms have only recently begun to be investigated with regard to BMIs. The following four sections discuss each of the stages in more detail.
A/D conversion
Feature selection
Algorithm selection
SP and feature extraction
Translation algorithm
2 1
Device commands
Recording instrumentation
Stimulus/ mental task protocol
Communication Control
4
Feedback?
FIG. 1. Main elements in a general brain–machine interface system.
3
96
FRANCISCO SEPULVEDA
II. Main Elements in a BMI
A. MENTAL TASK SELECTION While some work has been done concerning the selection of mental tasks to be used in BMIs (e.g., Curran et al., 2004), few groups around the world have focused on this issue. The most widely used mental tasks are motor imagery and counting (usually associated with the blinking of a desired object on a computer screen in P300-based approaches). A great deal of research has thus focused on classifying events (e.g., left hand vs right hand movement, or target vs nontarget blinking letter) without worrying whether these mental states are indeed the most easily separable ones. Part of this tendency is due to the fact that many groups do not have access to experimental facilities and thus rely on borrowing data. However, there is also a strong belief (mostly by the more machine-learning oriented researchers) that intelligent algorithms can classify any signal, provided the data are submitted to some minimum processing (e.g., filtering, including EOG removal, referencing, and feature selection). As the number of publications showing the use of yet another pattern recognition technique rapidly increases, it is becoming clear that the bottle neck in BMI performance is not in the featureselection or translation stages discussed below. The poor signal-to-noise ratios1 and the associated poor separability of the data into the desired classes cannot be suitably addressed unless the process (i.e., the mental state) that generated the data is investigated more carefully. For example, a recent study on self-paced mental activity has shown that it is easier to discriminate between auditory recall and mental calculation than between left hand versus right hand movement imagery (Sepulveda et al., 2007), the latter being long assumed to be one of the easiest pairs of tasks to classify. Future BMI work will need to place a higher priority on the study of suitable mental tasks if major progress is to made in this field.
B. FEATURE SELECTION The signals acquired with the instrumentation are converted to digital format (preferably after applying an analog antialiasing filter) using the standard sampling rate of at least twice the expected highest relevant frequency component in the signal. Most studies assume a maximum frequency of interest of about 50 Hz 1 ‘‘Noise’’ here includes not only the usual environmental electromagnetic sources and motion artifacts but also the myriad of legitimate signals generated by the brain as a result of its intrinsic multitasking nature.
AN OVERVIEW OF BMIs
97
and use a sampling rate of 128 or 256 Hz. Some recent studies suggest that relevant information exists on frequencies well beyond 50 Hz, but further investigation is needed to ascertain this as a fact. The digitized signal can now be further filtered and used for extracting information that is relevant to the mental task under investigation. Before the signals are used, however, some unwanted artifacts need to be removed. These can be, for example, oculomotor signals (very frequent in EEG readings) which cannot be removed by standard digital filters alone and methods based on independent component analysis (ICA) are often used. Once any unwanted signal components are removed as much as possible, features (i.e., specific data parameters or wave shapes) can be extracted from the data. Without exaggeration, thousands of diVerent features have been used in BMIs, falling into one of four general domains: (a) time domain, (b) frequency domain, (c) joint time–frequency ( JTF) domain, and (d) spatial domain (e.g., electrode location, not to be confused with the context of the word ‘‘space’’ used elsewhere in this book). The JTF domain can provide the most detailed view of the signal, but it can also generate enormous amounts of data if used indiscriminately. For example, a typical JTF signal representation may contain 24 time divisions by 64 frequency bands per second for each channel. For a 64-channel setup, this translates into 24 64 64 ¼ 98,304 features per second! Few features within these domains will yield information that correlates well with a given mental task under any protocol conditions. Often no more than 20 or 30 features in a JTF transformation carry task specific information. For example, Vuckovic and Sepulveda (2008) have shown that only 2–3% of all features in the JTF domain have significant information toward discrimination of wrist movements (both real and imagined) in a cue-based protocol. Indiscriminate use of as many features as possible, regardless of the chosen domain, can thus make the subsequent translation task nonrealizable on account of the large amount of irrelevant information, which will make the data nonseparable introduction the desired classes. Thus, proper design of BMIs must include a ‘‘feature selection’’ study for the specific conditions of the envisioned application. Once this selection is done, the overall process only requires extraction of the preselected features from the real-time data. However, some adaptive systems can do some level of feature selection online as well.
C. TRANSLATION ALGORITHMS Once the relevant features have been extracted, they can be fed into a translation algorithm. These are also previously selected among many alternatives, usually by testing the algorithms with previously recorded data, the so-called oV-line analysis approach. Translation algorithms can be as simple as a set of
98
FRANCISCO SEPULVEDA
‘‘if–then’’ rules or a previously determined linear discriminator. However, often the complexity of the data requires the use of nonlinear classifiers, though for some tasks (e.g., discrimination of left hand versus right hand movement imagination) linear discriminant analysis can yield suitable results. The translation algorithm then yields commands to be used by various devices (the ‘‘control’’ case) or to lead to explicit communication of thoughts (the ‘‘communication’’ case). Artificial intelligence methods are common and include linear discriminant analysis, artificial neural networks, genetic algorithms, Kernel-based learning methods (support vector machines, Kernel–Fisher discriminant), Bayesian networks, Hidden Markov models, among many others. Brain signals are nonstationary and suVer from statistical shifts that can be as quick as a few hundreds of a second, depending on the recording ethos used. Thus, to be eYcient, a translation algorithm should adapt to the human user at least at two levels: Adaptation to first order signal statistics: This is important as the most basic attempt to track statistical changes in brain signals. For instance, if the relevant feature is represented by the amplitude of the delta band as recorded above the sensorimotor cortex, the algorithm should adjust to the changes in the amplitude range as well as to possible mean value oVset shifts in this band. Similarly, if the relevant signal in an implanted BMI is represented by the estimated firing rate of an individual neuron, the algorithm should adjust to the changes in firing rate range. Ideally, a BMI should be robust enough not to have its performance degraded due to changes in a feature’s distribution within seconds to a few hours. As simple as this sounds, to date very few real-time BMIs have this level of adaptivity. Higher level changes: Repeated execution of mental tasks or repeated stimulus presentation lead to changes in the brain’s response due to habituation, sensitization, fatigue, and, after a few months, even remapping and the appearance of new strategies, be them conscious or not. This level of brain adaptation may lead to, for example, an algorithm that looks at a single electrode location no longer performing well as the observed location no longer plays the main role. The above levels of adaptation must be tackled whether or not a BMI has explicit feedback to the user. In practice, the user is aware of the BMI’s performance. This implicitly closes the BMI loop, although biofeedback and other feedback techniques can be used to provide explicit feedback to the user. In both cases the brain may adapt toward improving the overall performance of the system if the system is designed for this purpose. For example, suitable tasks and feedback can be designed to engage the user and thus lead him/her to want to perform better. Meanwhile, it is important not to make the task too diYcult as
AN OVERVIEW OF BMIs
99
frustration and fatigue can all aVect a user’s performance (Dibartolo et al., 1997; Lang and Kotchoubey, 2000). Boredom resulting from uninteresting or absent feedback and unchallenging tasks will also most likely lead to a significant drop in performance. In the case of a BMI with explicit performance feedback to the user, the issue of mutual adaptation needs to be considered as well. As the machine (or algorithm) adapts to the brain activity pattern changes, its response shift will also cause the brain to further adapt. Thus, the mutual adaptation loop must be carefully designed to ensure that the whole system progresses toward improved performance. D. OUTPUT AND FEEDBACK METHODS While the output from a BMI has no restrictions and currently includes robotic arm movements and wheel chair control, most BMIs to date present their output on a computer screen in the form of letters, icons, or arrows. Some studies have investigated the possibility of controlling neuroprostheses and/or orthoses via a BMI. In two studies, attempts were made to control either a real hand (via neuromuscular electrical stimulation) or an artificial hand in users with complete section of the cervical spinal cord (Lauer et al., 2000; Pfurtscheller et al., 2000), with some success. However, some researchers have pointed out that BMIs are not yet reliable of fast enough to be used with some neural prostheses. For example, BMIs are not yet useful for controlling walking induced by functional neuromuscular electrostimulation, FES (see e.g., Sinkjaer, et al., 2003) due to their current low reliability (which has not improved much since the 2003 paper by Sinkjaer et al.). Indeed, where a BMI is used for controlling gate at the time of this writing, the user would risk falling every few steps. In general the issue of feedback has not been addressed in BMIs beyond aspects related to visual and or auditory presentation of the output. Recent studies include the use of vibrotactile feedback (Chatterjee et al., 2007; Cincotti et al., 2007). However, as more work is done addressing the (adaptive) interaction between the human user and the machine, this will undoubtedly be a rapidly growing area in the near future. III. BMI Types
BMIs have been categorized in many ways in recent years (for a more detailed review, see Wolpaw et al., 2002). While not all BMI researchers use the same terminology, most subdivisions of BMIs fall under one of the following: Invasive versus noninvasive Dependent versus independent
100
FRANCISCO SEPULVEDA
Spontaneous versus evoked versus event related Synchronous versus asynchronous These have so far been studied mostly in mutually exclusive fashion (e.g., either spontaneous or event-related signals), but future BMIs may combine some or all of the above in one system. All types are discussed below.
A. INVASIVE VERSUS NON INVASIVE Aside from the clear distinction between invasive (implanted) electrodes (e.g., subdural, epidural, or intracortical) and noninvasive ones that go on the skin surface (EEG), there is a wider issue concerning the term ‘‘invasive.’’ Any technology that deposits external elements into the human body may be considered invasive. The deposited element could be a device (e.g., a needle or an electrode), a chemical, atomic particles, and even electromagnetic energy. Thus, in the strictest sense of the world, technologies such as NIRS (which deposit near infrared light on the tissue), fMRI (which applies magnetic fields), and PET (which requires the administration of a radioactive substances) are all invasive as the very mechanisms by which they work requires that the observed tissue (and surrounding ones) be internally disturbed. In the case of NIRS, which may become a useful BMI device in the future, the eVects of the absorbed energy on the brain tissue have not been studied concerning long term or short term but frequent use (e.g., rare occasions, but with hours of use every time). Thus, of the technologies described here, only EEG and MEG are noninvasive. Whether this is good or bad depends on many factors, but this is likely to be an important ethical issue for many potential users, especially able-bodied individuals looking for an enhanced human–machine interface, such as computer games and robotics enthusiasts. EEG-based approaches in particular are by far the most widely studied BMIs as they are also relatively low cost.
B. DEPENDENT VERSUS INDEPENDENT BMIS BCI devices have been classified as dependent and independent (Wolpaw et al., 2002). A dependent BCI does not employ the usual ways of brain output to transport and dispatch the relevant message, but it requires some—even if little— surviving function of such a physiological channels to generate the relevant EEG activity. In other words, a dependent BMI requires the presence but does not use normal output pathways to produce the brain signals that feed the interfacing computer.
AN OVERVIEW OF BMIs
101
For instance, in many BMI studies the user—often able-bodied—is asked to perform a hand or arm movement. If the user has control of the limb and the movement is actually executed, the brain signals will have characteristic patterns related to the executed movement. These brain patterns can be used for translating the user’s motor intentions, but they only took place because the user has control of the limb. The signals would be very diVerent if the limb was paralyzed or if there was significant sensory deficit. An independent BMI, on the other hand, is entirely free from the physiological output pathways of the brain as the relevant signal is not generated by propagating signals along peripheral nerves, muscles, or other physiological outputs. Using the limb movement example above, a user may simply imagine or feel a limb moving without the movement taking place. The brain signals would then reveal the user’s mental task whether or not he/she has the limb or any control over it. Another example can involve eye movement. In for instance, let us consider a matrix of alphabet letters presented in a video display, which are sequentially flashed. In the dependent BMI case, the user is asked to choose a letter via ocular fixation on it. This act produces a visual evoked potential (VEP) on the occipital scalp significantly larger than the responses provoked by the other flashing letters which are not fixated by the user (Sutter, 1992). Therefore, the relevant signal is coming from the EEG, but it is due to the sight focus direction. In the independent BMI case, the user selects a character just by paying attention to the letter he/she wants; he/she does not need to gaze directly at the letter of interest. This attention mechanism elicits an increase in a brain cognitive potential (known as the P300 wave) shortly (300 ms) after the target letter flashes (Donchin, 1981; Donchin et al., 2000; Fabiani et al., 1987; Farwell and Donchin, 1988; Polich, 1999; Sutton et al., 1965). The usual brain outputs do not play an important role in independent BMIs, which can provide a wider variety of brain ‘‘outputs’’ and are therefore of great interest, particularly for users who are completely paralyzed. On the other hand, able-bodied users (e.g., computer game enthusiasts) would benefit from dependent BMIs as well. Thus, which type one chooses depends on the application scenario and population one targets.
C. SPONTANEOUS VERSUS EVOKED VERSUS EVENT RELATED Evoked potentials (EPs) appear in the brain as a direct result of a particular stimulus, for example, a flashing letter, whether or not the user is interested in or even aware of it. EPs are time locked to the stimulus. Other brain signals can be completely spontaneous, such as those related to movement intentions in the sensory motor cortex, and are thus not a result of specific input given to the
102
FRANCISCO SEPULVEDA
human subject. Finally, a third class of signals is referred to as ‘‘event-related potentials’’ (ERPs). These are related to evoked potentials but include brain responses that are not directly elicited by the stimulus. They can include cognitive signals, among other psychological manifestations (Rugg and Coles, 1995). In fact, the term ERP is seen as a more accurate term for all but the most restricted simulation protocols. It is thus the preferred term instead of EP. This type of response is widely used in BMI research, especially in synchronous systems (see below).
D. SYNCHRONOUS VERSUS ASYNCHRONOUS Asynchronous interfaces (Allison and Pineda, 2005) are those that use truly spontaneous signals for its operation (e.g., Graimann et al., 2004; Kennedy et al., 2000; Mason et al., 2004; Townsend et al., 2004; Yom-Tov and Inbar, 2003). In this type of BMI, brain signals relevant to a user’s intention may be produced any time, with or without stimulus, thus making classification of the intention diYcult as the computer must first find when a relevant intention took place, the so-called ‘‘onset detection’’ problem. Synchronous interfaces, on the other hand, will use either evoked or event-related potentials (e.g., Farwell and Donchin, 1988; Gao et al., 2003; Middendorf et al., 2000; Pfurtscheller et al., 2003; Sutter, 1992). Such BMIs look into features that appear at known times after a particular cue (e.g., a flashing letter, or a command to imagine a hand movement). These are ‘‘cuebased’’ and therefore do not have to deal with the onset detection problem. While most work to date has used cue-based (synchronous) interfaces, future BMIs will likely require more asynchronous features as these are more natural and do not require that the user be paying attention to a specific stimulus. Asynchronous interfaces are currently being developed by several groups, including those at the University of Essex (UK) and at IDIAP (Switzerland).
IV. BMIs and the User’s Ability
Successful use of BMIs as developed to date requires that the user maintain his/her ability to learn and retain new abilities in controlling not the usual neuromuscular channels but the EEG pattern that is recognized as relevant by the BMI. This does not imply a training period per se. For instance, the ability to generate an increased P300 (Farwell and Donchin, 1988) wave in response to flashing of the chosen letter is independent from training. It only requires the user to be able to identify the chosen letter as the ‘‘target’’ of his/her attention and to
AN OVERVIEW OF BMIs
103
ignore all the remaining letters. Of course, some degree of learning will take place even scenario. In fact, it has been shown the P300 signal progressively adapts as the user is repeatedly exposed to the visual protocol (Coles and Rugg, 1995; Rosenfeld, 1990). The ability of humans to self-control their EEG rhythms and transient waves has been reported for several decades as well (Black, 1971, 1973; Black et al., 1970; Nowles and Kamiya, 1970). More recently, experiments with primates have shown that the firing rate characteristics of individual neurons can be controlled by learned conditioning. This opened new avenues for controlling BMIs (Fetz and Finocchio, 1975; Schmidt, 1980). In fact, this has led to work on the operant conditioning approach of many groups, notably Wolpaw’s (at the Wadsworth Center in the US) and Birbaumer’s (at the University of Tuebingen in Germany).
A. HIGHER LEVEL OPERATIONAL PROTOCOL Every BMI has its own operational protocol that defines the procedures for (a) switching the whole system (not just the translation/output stages) ‘‘on’’ and ‘‘oV,’’ whether the relevant signal is generated consciously by the user or in response to a stimulus triggered by the BMI (i.e., event related), (b) the exact sequence of interactions between the user and the BMI, and (c) the type of feedback provided to the user. In the future, a BMI user will need to be able to carry out these procedures. Most BMIs produced so far have not addressed these issues, but a few groups have made significant eVorts to this end (e.g., Birbaumer et al., 2000).
B. PATTERN RECOGNITION VERSUS OPERANT CONDITIONING Birbaumer, in Germany, and Wolpaw, in the US, demonstrated that instead of using cognitive tasks, an operant conditioning approach could be used to train users to control a cursor. This approach was based on the idea that it is suYcient to provide an appropriated feedback (i.e., seeing the moving cursor) to let the brain progressively learn which components of its signals are relevant in controlling the BMI device (Pfurtscheller et al., 1993). In Wolpaw et al. (1991) users adopted diVerent strategies to move the cursor (i.e., to lift a weight to move the cursor down and to relax to move the cursor up) during the conditioning period. Thereafter, they did not use the weight lifting strategy any longer as the cursor control could be done just by imagining the task.
104
FRANCISCO SEPULVEDA
V. Conclusion
BMIs comprise various stages from mental task selection to feedback protocol design. While some work has been done in these two stages, much emphasis in recent research has been placed on feature selection and signal classification (or translation) instead. This review discussed the main elements in a BMI as well as the terminology used for describing various BMI types. The message here is that while machine intelligence and pattern recognition have a crucial role in this field, further advancement in BMIs necessitates more focus on mental task selection, feedback design, and the eVects of short- and long-term adaptation on the human user’s side. Acknowledgments
The author thanks Luca Rossini, Oliver Tonet, and Luca Citi for their help putting together various sections of this review.
References
Allison, B. Z., and Pineda, J. A. (2005). EVects of SOA and flash pattern manipulations on ERPs, performance, and preference: Implications for a BCI system. Int. J. Psychophysiol. 59(2), 127–140. Black, A. (1971). The direct control of neural processes by reward and punishment. Am. Sci. 59, 236–245. Black, A. (1973). ‘‘The Operant Conditioning of the Electrical Activity in the Brain as a Method for Controlling Neural and Mental Processes,’’ pp. 35–68. Academic Press, New York. Black, A., Young, G., and Batenchuk, C. (1970). Avoidance training of hippocampal theta waves in flaxedilized dogs and its relation to skeletal movement. J. Comp. Physiol. Psychol. 70, 15–24. Birbaumer, N., Ku¨bler, A., Ghanayim, N., Hinterberger, T., Perelmouter, J., Kaiser, J., Iversen, I., Kotchoubey, B., Neumann, N., and Flor, H. (2000). The thought translation device (TTD) for completely paralyzed patients. IEEE Trans. Rehabil. Eng. 8, 190–192. Chatterjee, A., Aggarwal, V., Ramos, A., Acharya, S., and Thakor, N. V. (2007). A brain-computer interface with vibrotactile biofeedback for haptic information. J. Neuroeng. Rehabil. 4, 40. Cincotti, F., Kauhanen, L., Aloise, F., Paloma¨ki, T., Caporusso, N., Jyla¨nki, P., Mattia, D., Babiloni, F., Vanacker, G., Nuttin, M., Marciani, M. G., Del, R., et al. (2007). Vibrotactile feedback for braincomputer interface operation. Comput. Intell. Neurosci. 2007, 48937. Coles, M., and Rugg, M. (1975). ‘‘Event-related Potentials: An Introduction.’’ Oxford University Press, Oxford. Curran, E. A., Sykacek, P., Stokes, M. J., Robers, S. J., Penny, W., Johnsrude, I., and Owen, A. M. (2004). Cognitive tasks for driving a brain-computer interfacing system: A pilot study. IEEE Trans. Rehab. Eng. 12, 48–54.
AN OVERVIEW OF BMIs
105
Dibartolo, P., Brown, T., and Barlow, D. (1997). EVects of anxiety on attentional allocation and task performance: An information processing analysis. Behav. Res. Ther. 35, 1101–1111. Donchin, E. (1981). Presidential address, 1980. Psychophysiology 18, 493–513. Donchin, E., Spencer, K. M., and Wijesinghe, R. (2000). The mental prosthesis: Assessing the speed of a P300-based brain-computer interface. IEEE Trans. Rehabil. Eng. 8(2), 174–179. Fabiani, M., Gratton, G., Karis, D., and Donchin, E. (1987). ‘‘Definition, Identification and Reliability of the P300 Component of the Event-Related Brain Potential,’’ pp. 1–78. JAI Press, New York. Farwell, L., and Donchin, E. (1988). Talking oV the top of your head: Toward a mental prothesis utilizing event-related brain potentials. Electroenceph. Clin. Neurophysiol. 70, 510–523. Fetz, E., and Finocchio, D. (1975). Correlations between activity of motor cortex cells and arm muscles during operantly conditioned response patterns. Exp. Brain Res. 23, 217–240. Gao, X., Xu, D., Cheng, M., and Gao, S. (2003). A BCI-based environmental controller for the motion-disabled. IEEE Trans. Neural Syst. Rehabil. Eng. 11(2), 137–140. Graimann, B., Huggins, J. E., Levine, S. P., and Pfurtscheller, G. (2004). Toward a direct brain interface based on human subdural recordings and wavelet-packet analysis. IEEE Trans. Biomed. Eng. 51(6), 954–962. Kennedy, P. R., Bakay, R. A., Moore, M. M., Adams, K., and Goldwaithe, J. (2000). Direct control of a computer from the human central nervous system. IEEE Trans. Rehabil. Eng. 8(2), 198–202. Lang, S., and Kotchoubey, B. (2000). Learning eVects on event-related brain potentials. NeuroReport 11, 3327–3331. Lauer, R., Peckham, P., Kilgore, K., and Heetderks, W. (2000). Applications of cortical signals to neuroprosthetic control: A critical review. IEEE Trans. Rehabil. Eng. 8, 205–208. Mason, S. G., Bohringer, R., BorisoV, J. F., and Birch, G. E. (2004). Real-time control of a video game with a direct brain–computer interface. J. Clin. Neurophysiol. 21(6), 404–408. Middendorf, M., McMillan, G., Calhoun, G., and Jones, K. S. (2000). Brain–computer interfaces based on the steady-state visual-evoked response. IEEE Trans. Rehabil. Eng. 8(2), 211–214. Nowles, D., and Kamiya, J. (1970). The control of electroencephalographic alpha rhythms through auditory feedback and the associated mental activity. Psychophysiology 6, 75–84. Pfurtscheller, G., Flotzinger, D. & Kalcher, J. (1993). Brain-computer interface: A new communication device for handicapped persons. J. Microcomput. Appl. 16, 293–299. Pfurtscheller, G., Neuper, C., Guger, C., Harkam, W., Ramoser, H., Schlo¨gl, A., Obermaier, B., and Pregenzer, M. (2000). Current trends in Graz brain-computer interface (BCI) research. IEEE Trans. Rehabil. Eng. 8(2), 216–219. Pfurtscheller, G., Neuper, C., Muller, G. R., Obermaier, B., Krausz, G., Schlogl, A., Scherer, R., Graimann, B., Keinrath, C., Skliris, D., Wortz, M., Supp, G., et al. (2003). Graz–BCI: State of the art and clinical applications. IEEE Trans. Neural Syst. Rehabil. Eng. 11(2), 177–180. Polich, J. (1999). ‘‘P300 in Clinical Applications,’’ pp. 1073–1091. Williams and Wilkins, Baltimore, MD. Rosenfeld, J. (1990). Applied psychophysiology and biofeedback of event-related potentials (brain waves): Historical perspective, review, future directions. Biofeedback Self Regul. 15, 99–119. Rugg, M. D., and Coles, M. H. (1995). ‘‘Electrophysiology of Mind: Event-related Brain Potentials and Cognition.’’ Oxford University Press, Oxford. Schmidt, E. (1980). Single neuron recording from motor cortex as a possible source of signals for control of external devices. Ann. Biomed. Eng. 8, 339–349. Sinkjaer, T., Haugland, M., Inmann, A., Hansen, M., and Nielsen, K. D. (2003). Biopotentials as command and feedback signals in functional electrical stimulation systems. Med. Eng. Phys. 25(1), 29–40. Sepulveda, F., Dyson, M., Gan, J. Q., and Tsui, C. S. L. (2007). A comparison of mental task combinations for asynchronous EEG-based BCIs. In Proceedings of the 29th Annual
106
FRANCISCO SEPULVEDA
International Conference of the IEEE Engineering in Medicine and Biology Society EMBC07, Lyon, August 22–26, pp. 5055–5058. Sutter, E. E. (1992). The brain response interface: Communication through visually-induced electrical brain response. J. Microcomput. Appl. 15, 31–45. Sutton, S., Braren, M., Zubin, J., and John, E. (1965). Evoked correlates of stimulus uncertainty. Science 150, 1187–1188. Townsend, G., Graimann, B., and Pfurtscheller, G. (2004). Continuous EEG classification during motor imagery—Simulation of an asynchronous BCI. IEEE Trans. Neural Syst. Rehabil. Eng. 12(2), 258–265. Vuckovic, A., and Sepulveda, F. (2008). Quantification and visualisation of differences between two motor tasks based on energy density maps for BCI applications. Clin. Neurophys. 119(2), 446–458. Wolpaw, J. R., McFarland, D., Neat, G., and Forneris, C. (1991). An EEG-based brain-computer interface for cursor control. Electroencephalogr. Clin. Neurophysiol. 78, 252–259. Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., and Vaughan, T. M. (2002). Braincomputer interfaces for communication and control. Clin. Neurophysiol. 113(6), 767–791. Yom-Tov, E., and Inbar, G. F. (2003). Detection of movement-related potentials from the electroencephalogram for possible use in a brain–computer interface. Med. Biol. Eng. Comput. 41(1), 85–93.
NEUROFEEDBACK AND BRAIN–COMPUTER INTERFACE: CLINICAL APPLICATIONS
Niels Birbaumer,*,y Ander Ramos Murguialday,*,z Cornelia Weber,* and Pedro Montoya} *Institute of Medical Psychology and Behavioral Neurobiology, University of Tuebingen, Tuebingen, Germany y Ospedale San Camillo—IRCCS, Istituto di Ricovero e Cura a Carattere Scientifico, Venezia, Lido, Italy z Fatronik Foundation, San Sebastian, Spain } Department of Psychology, Universidad Illes Baleares, Palma de Mallorca, Spain
I. II. III. IV. V.
Introduction Functional Magnetic Resonance Imaging: f MRI-BMI BMI in Locked-in Syndrome BMI in Stroke and Spinal Cord Injury Conclusion References
Most of the research devoted to BMI development consists of methodological studies comparing diVerent online mathematical algorithms, ranging from simple linear discriminant analysis (LDA) (Dornhege et al., 2007) to nonlinear artificial neural networks (ANNs) or support vector machine (SVM) classification. Single cell spiking for the reconstruction of hand movements requires diVerent statistical solutions than electroencephalography (EEG)-rhythm classification for communication. In general, the algorithm for BMI applications is computationally simple and diVerences in classification accuracy between algorithms used for a particular purpose are small. Only a very limited number of clinical studies with neurological patients are available, most of them single case studies. The clinical target populations for BMI-treatment consist primarily of patients with amyotrophic lateral sclerosis (ALS) and severe CNS damage including spinal cord injuries and stroke resulting in substantial deficits in communication and motor function. However, an extensive body of literature started in the 1970s using neurofeedback training. Such training implemented to control various EEG-measures provided solid evidence of positive eVects in patients with otherwise pharmacologically intractable epilepsy, attention deficit disorder, and hyperactivity ADHD. More recently, the successful introduction and testing of real-time f MRI and a NIRS-BMI opened an exciting field of interest in patients with psychopathological conditions. INTERNATIONAL REVIEW OF NEUROBIOLOGY, VOL. 86 DOI: 10.1016/S0074-7742(09)86008-X
107
Copyright 2009, Elsevier Inc. All rights reserved. 0074-7742/09 $35.00
108
BIRBAUMER et al.
I. Introduction
Most clinical applications of BMI-research rest on the tradition of neurofeedback and biofeedback, both consequences of technological achievements in rapid computer analysis of EEG patterns that allow online feedback and reward of diVerent types of neuroelectric activity (Elbert et al., 1984). BMIs aimed at restoration of movement, however, were built in the tradition of tuning functions of sensori-motor neurons representing diVerent directions of movements (Georgopoulos et al., 2007). Neurofeedback allowed, for the first time, voluntary self-regulation of brain activity through feedback and reward. Expectancies ran high and many premature announcements of clinical success based on single case studies or uncontrolled observations discredited the field early on. In the 1970s Miller’s demonstrations of operant control of autonomic (and CNS) functions (Miller, 1969) in curarized rats, supposedly proving ‘‘voluntary’’ operant regulation of many bodily functions excluding mediation of the motor system through curarization, turned out to be diYcult to replicate (Dworkin and Miller, 1986). Together with the clinical overstatements in the field of biofeedback, this historic incident virtually halted funding from public sources and blocked large controlled clinical studies despite some indications of its eYciency. However, more recent studies suggested that some patients with drug-resistant epilepsy (mostly with secondarily generalized seizures) experienced a reduction in the number of ictal events during and after training consistent of self-regulation of slow cortical potentials (SCPs) (Kotchoubey et al., 2001; Rockstroh et al., 1993), an eVect also reported using biofeedback of skin conductance responses (GSR) (Nagai et al., 2004). Nagai et al. showed that learned increase in autonomic arousal through reduction of skin conductance decreased negative SCPs at the cortical level and thus increased seizure thresholds confirming earlier reports (Birbaumer et al., 1990; Kotchoubey et al., 2001; Rockstroh et al., 1993). In those studies with training and visual feedback of positive SCPs in focal epilepsies, some patients achieved virtually 100% accuracy in the control of SCPs after extensive training of 30–50 sessions, thus paving the way for application to BMIs for communication. Still, well-controlled trials with larger samples of epileptic patients have not been implemented. Another promising line of neurofeedback in neurology is the self-regulation of SCPs and mu-rhythm (also called sensori-motor-rhythm, SMR) in attention deficit disorder and hyperactivity (ADHD). SMR occurs over the sensorimotor rolandic brain regions with a frequency of 8–15 Hz indicating motor quiescence and a functionally inhibitory mode of the thalamocortical loops (Sterman and Clemente, 1962a). Motor imagery or motor action desynchronizes SMR (eventrelated desynchronization, ERD). Well-controlled studies with relatively small
NEUROFEEDBACK AND BCI
109
samples of ADHD children showed potential, pointing to lasting eVects on attention, and vigilance comparable to those achieved through pharmacological treatment with stimulants (Fuchs et al., 2003; Strehl et al., 2006). All in all, these pioneering studies underlined the possibility to control human electrocortical activity and to modify motor and cognitive functions in health and disease.
II. Functional Magnetic Resonance Imaging: f MRI-BMI
Near-infrared-spectroscopy (NIRS) measuring changes in oxygenation and in deoxygenation of the cortical surface is a relatively cheap noninvasive technology whose regulation can be learned within a few training sessions with contingent feedback only (Sitaram et al., 2007). Sitaram et al. trained healthy human subjects successfully to maximize the diVerence between right and left sensorimotor regions. The regulation of the blood oxygenation level dependent (BOLD) response with real-time fMRI (rt-fMRI) is another development in BMI research (Weiskopf et al., 2005) (see Fig. 1). In contrast to all other noninvasive BMI measures, regulation of circumscribed cortical and subcortical structures is possible. Several experimental studies, mostly with young healthy volunteers, revealed an amazing anatomical resolution in the characterization of the cortical region to be ‘‘trained’’ and a good correlation of these changes with behavioral changes. For example, regulation of premotor and motor areas leads to changes in motor response speed ( Weiskopf et al., 2005), of anterior cingulated regions to downregulation of pain (DeCharms et al., 2005), of parahippocampal areas to changes in explicit memory performance ( Weiskopf et al., 2005) and of the anterior insula to changes in the valence of negative emotional slides without aVecting neutral or positive emotions (Caria et al., 2007). Healthy subjects are able to increase and decrease BOLD activity in a region of interest within one to three 1-h training sessions: usually they receive positive visual feedback within a second after the BOLD change (which itself has a latency of 2–3 s to the neural response). Experiments manipulating the connectivity between diVerent brain areas and real-time control of selected metabolic substances in specific brain regions using magnetic resonance spectroscopy-feedback are underway. A recent experiment (Caria et al., 2007) trained healthy subjects and criminal psychopaths to increase BOLD in the right insula. Before and after training, usually one to three 1-h sessions, aversive and neutral slides were presented. One control group received sham feedback and a second control group received training of emotional imagery. Only in the experimental group and for neutral slides learned BOLD increase in the insula increased subjective aversiveness of
110
BIRBAUMER et al.
Overview of fMRI-BCI system Signal acquisition Custom-built scanner program
Raw images Signal analysis Signal feedback
Participant
BCI program
Preprocessed brain activity
• Visual feedback and reward computation • Pattern classification (support vector machine) • Performance measures
FIG. 1. Experimental condition for f MRI-BCI and BOLD-neurofeedback (see text for explanation).
picture stimuli. This study confirms the anatomical specificity of the f MRI-BMI eVects and underscores the critical importance of contingent feedback and reward. MR-technology use is expensive and applications in large clinical groups may not be feasible but represents a powerful tool to explore the mechanisms underlying BMI eVects and brain–behavior–pathology relationships in emotional disorders such as psychopathy and substance abuse as well as in other neuropsychiatric conditions.
III. BMI in Locked-in Syndrome
Patients with progressive motor neuron disease, particular amyotrophic lateral sclerosis (ALS), Guillain–Barre´ Syndrome, and subcortical stroke, as well as patients with traumatic brain damage in vegetative state (Kotchoubey, 2005) may suVer from locked-in syndrome (LIS) or total locked-in syndrome (TLIS). LIS is defined as complete paralysis with one or a few voluntary functions left (usually
111
NEUROFEEDBACK AND BCI
10 8 6 4 2 R
R Sess 1
R Sess 2
0 Sess 3
FIG. 2. EVect of reward and feedback on BOLD-response of right insula (one person as representative example). Each session’s duration 20 min. Large increase of right insula BOLD after third session.
small eye movements). TLIS consists of complete cessation of volitional control of all voluntary somatic-motor functions. Both, LIS and TLIS show intact auditory and tactile perception and intact cognitive functions, usually measured with event-related brain potentials (ERP, Kotchoubey, 2005) or f MRI (Owen et al., 2006). Visual perception is also frequently compromised through paralysis of eye muscles. Therefore, BMIs using the auditory or tactile modality are mandatory for use in TLIS patients. Since the first report (Birbaumer et al., 1999) of two LIS patients with ALS selecting letters from computer-presented letter strings using learned voluntary decrease of SCPs, several papers with small samples of ALS patients have appeared that demonstrate BMI controlled communication in LIS and advanced stages of ALS. In a thorough review of the literature it was proposed that BMIs using P300 ERPs (Farwell and Donchin, 1988; Sellers and Donchin, 2006), SCPs (Birbaumer et al., 1999) (see Fig. 3), and SMR-control (Pfurtscheller et al., 2005) could provide slow but eVective verbal communication in all stages of ALS, except the TLIS. It is of interest that in two patients with TLIS, not even an invasive BMI controlled from epidural electrodes at left frontal sites improved their ability to communicate (unpublished data, available from the authors). Only one study (Naito et al., 2007) reported more optimistic results from a NIRS-based BMI in 17 patients with TLIS. Patients were trained to respond with an increase in blood oxygenation (‘‘yes’’) or decrease in oxygenation (‘‘no’’) to various questions displayed on a computer screen. Using an elaborate oV-line classification method, a separation of ‘‘yes’’ and ‘‘no’’ of 70% correct was reported in seven out of 17 patients with TLIS. One weakness of this study is the relative lack of quantification and definition of clinical criterions used for the TLIS patients. It remains to be determined whether BMIs using EEG, electrocorticogram (ECoG), or NIRS allow voluntary brain responses and communication in TLIS. One possible explanation for the failure to replicate operant control of autonomic functions in the curarized rat (Dworkin and Miller, 1986; Miller, 1969) and for the lack of learned brain regulation with BMI in TLIS is that goal directed and voluntary
112
BIRBAUMER et al.
FIG. 3. Locked-in patient with amyotrophic lateral sclerosis using a brain–computer interface (BCI) for spelling. Top: Slow cortical potentials (SCPs) during one trial: a voluntary produced positive deflection of the SCP splits the letter string on the screen below in half, if the desired letter is still among the letter string, a new positive SCP splits it again, and so forth until the desired letter is selected. Below: Screen with feedback cursor (not visible), letter string, from which the desired letter has to be selected (lowest box) and already spelled letters above the letter string.
thought processes may over time extinguish in the absence of reinforcement contingencies, a hypothesis worthwhile testing in the future (Birbaumer and Cohen, 2007). If this hypothesis is true a transfer of training success with a BMI from the LIS to the TLIS should be possible.
IV. BMI in Stroke and Spinal Cord Injury
Experimentation with nonhuman primates suggests that intentional goaldirected movements of the upper limbs can be reconstructed and transmitted to external manipulandum or robotic devices controlled from a relatively small number of microelectrodes implanted into movement-relevant brain areas after some training, opening the door for the development of brain–computer interfaces (BCIs) or brain–machine interfaces (BMIs) in humans. While noninvasive BMIs using electroencephalographic recordings EEG or ERPs in healthy individuals and patients with ALS or stroke, can transmit up to 80 bits/min of information, the use of BMIs—invasive or noninvasive—in severely or totally paralyzed patients have met some unforeseen diYculties.
NEUROFEEDBACK AND BCI
113
In 2003, Pfurtscheller et al. reported a tetraplegic patient who, after extensive training to increase and decrease central mu-rhythms, was able to control an electrostimulation device (FES) applied to hand muscles (Pfurtscheller et al., 2000). The patient was able to grasp a glass and bring it to his mouth after he had learned with feedback and reward over a period of 4 months to regulate his mu-rhythm. Hochberg et al. (2006) implanted a 96-microelectrode array into the hand region of the motor cortex of another tetraplegic patient. The patient learned to open and close a prosthetic hand distant from his own hand with intention-driven neuronal ensemble activity. No improvements in voluntary motor function in the paralyzed hand were reported. Motor disability resulting from chronic stroke represents the main cause of long-term disability among adults and has substantial social, financial, and psychological impact on patients, families, and society. Approximately one third of all stroke patients are not able to use the paralyzed hand for activities of daily living 1 year after the stroke. No treatment is available for that condition. A recent study (Buch et al., 2008) using a neuromagnetic BMI showed as a proof-of-principle successful BMI control of opening and closing grasping functions of an orthosis attached to the plegic hand in six out of eight patients. The orthosis was controlled by activity in three of the 275 MEG sensors. Increase of 9–12 Hz mu-rhythm in these three sensors opened the hand while decrease closed it. In six of the eight patients mu activity was derived from central ipsilesional location close to the subcortical lesion (see Fig. 4). After 13–22 1-h training sessions, patients were able to control hand opening– closing functions through the orthosis, in the absence of clinical improvements in the completely paralyzed hands. Training resulted in refocusing of MEG activity, providing first evidence that BMI training may result in well-defined cortical reorganization. Whether an invasive BMI with implanted electrodes and internalized connection to the peripheral nerves, or noninvasive BMIs connected to prosthetic devices or rehabilitation robots may move from these ‘‘bench’’ type of study to the clinic awaits further research. Still, the gap between what can be achieved with implanted microelectrode arrays in motor or parietal cortex (Schwartz, 2006) in healthy nonhuman primates versus a paralyzed human patient is wide: While the monkey learns in relatively short time periods to use a small neural assembly to feed himself without any motor mediation, the human patient needs many training hours to open and close a paralyzed hand. The fact that a pattern of spiking neurons in the appropriate brain region is ‘‘closer’’ to the origin of movement production alone does not explain the explanatory gap: with a dense sensor array of MEG a complex four-directional hand movement was possible to reconstruct with an accuracy of 70% (Waldert et al., 2008) in healthy individuals. The prediction accuracy was only slightly smaller for EEG data.
114
BIRBAUMER et al.
A
C
B
D
PA
P FIG. 4. Brain–computer interface (BCI) for motor restoration in chronic stroke. (A) MEG-BCI for control of mu-rhythm of an orthosis at the paralyzed hand, (B) mu-rhythm density (more yellow–red) before (left) and after (right) 20 sessions of MEG-BCI-training. Black dots indicate MEG-sensors used for the BCI-training, (C) Percent correct responses (opening-closing hand) over 19 sessions of BCItraining in one patient, and (D) MR with subcortical region (right hemisphere) of the patient whose data are presented in (A)–(C).
Experiments with lesioned animals and simultaneous recording of spike patterns, local field potentials, and ECoG are urgently needed to explore the precise parameters at each level of observation necessary to reconstruct movements in the lesioned brain and/or the paralyzed body parts.
V. Conclusion
Despite a growing animal literature demonstrating online control of functional hand movements from spike patterns recorded with microelectrodes in the motor cortex, BMI applications in neurological patients are rare and hampered by methodological diYculties. BMIs using EEG-measures allow verbal communication in paralyzed patients with ALS, BMI-communication in totally locked-in patients, however, awaits experimental confirmation. Movement restoration in chronic stroke without residual movement capacity using noninvasive BMI is possible but generalization of improvement to real life needs further experimentation.
NEUROFEEDBACK AND BCI
115
Invasive and noninvasive BMIs using recordings from nerve cells, large neuronal pools such as ECoG and EEG, or blood flow-based measures such as f MRI and NIRS show potential for communication in LIS and movement restoration in chronic stroke, but controlled phase III clinical trials with larger populations of severely disturbed patients are urgently needed.
Acknowledgments
This work was supported by the Deutsche Forschungsgemeinschaft (DFG), Bundesministerium fu¨r Bildung und Forschung (BMBF, Bernstein-Center for Neurotechnology 01GQ0831), Fatronik, San Sebastian, Spain, Motorike, Cesarea, Israel. Pedro Montoya was supported by Spanish Ministry of Science and European Funds (FEDER) (grant SEJ2007–62312).
References
Birbaumer, N. (2006). Breaking the silence: Brain-computer-interfaces (BCI) for communication and motor control. Psychophysiology 43, 517–532. Birbaumer, N., and Cohen, L. (2007). Brain-computer-interfaces (BCI): Communication and restoration of movement in paralysis. J. Physiol. 579(3), 621–636. Birbaumer, M., Elbert, T., Canavan, A., and Rockstroh, B. (1990). Slow potentials of the cerebral cortex and behavior. Physiol. Rev. 70, 1–41. Birbaumer, N., Ghanayim, N., Hinterberger, T., Iversen, I., Kotchoubey, B., Ku¨bler, A., Perelmouter, J., Taub, E., and Flor, H. (1999). A spelling device for the paralysed. Nature 398, 297–298. Buch, E., Weber, C., Cohen, L. G., Braun, C., Dimyan, M., Ard, T., Mellinger, J., Caria, A., Soekadar, S., Fourkas, A., and Birbaumer, N. (2008). Think to move: A neuromagnetic braincomputer interface (BCI) system for chronic stroke. Stroke 39, 910–917. Caria, A., Veit, R., Sitaram, R., Lotze, M., Weiskopf, N., Grodd, W., and Birbaumer, N. (2007). Regulation of anterior insular cortex activity using real-time fMRI. NeuroImage 35, 1238–1246. DeCharms, R. C., Maeda, F., Glover, G. H., Ludlow, D., Pauly, J. M., Soneji, D., Gabrieli, J. D. E., and Mackey, S. C. (2005). Control over brain activation and pain learned by using real-time functional MRI. Proc. Natl. Acad. Sci. USA 102, 18626–18631. Dornhege, C., Milla´n, J. R., Hinterberger, T., McFarland, D. J., and Mu¨ller, K. R. (Eds.) (2007). ‘‘Toward Brain-Computer Interfacing.’’ MIT Press, Cambridge, MA. Dworkin, B. R., and Miller, N. E. (1986). Failure to replicate visceral learning in the acute curarized rat preparation. Behav. Neurosci. 100, 299–314. Elbert, T., Rockstroh, B., Lutzenberger, W., and Birbaumer, N. (Eds.) (1984). ‘‘Self-Regulation of the Brain and Behavior.’’ Springer, New York. Farwell, L. A., and Donchin, E. (1988). Talking oV the top of your head: Toward a mental prosthesis utilizing event-related brain potentials. Electroenc. Clin. Neurophysiol. 70, 510–523.
116
BIRBAUMER et al.
Felton, E. A., Wilson, J. A., Williams, J. C., and Garell, P. C. (2007). Electrocorticographically controlled brain–computer interfaces using motor and sensory imagery in patients with temporary subdural electrode implants. J. Neurosurg. 106, 495–500. Flor, H., and Birbaumer, N. (1993). Comparison of the eYcacy of EMG biofeedback, cognitive behavior therapy, and conservative medical interventions in the treatment of chronic musculoskeletal pain. J. Consult. Clin. Psychol. 61(4), 653–658. Fuchs, T., Birbaumer, N., Lutzenberger, W., Gruzelier, J. H., and Kaiser, J. (2003). Neurofeedback training for attention-deficit/hyperactivity disorder in children: A comparison with methylphenidate. Appl. Psychophysiol. Biofeedback 28(1), 1–12. Georgopoulos, A. P., Kalaska, J. F., Caminiti, R., and Massey, J. T. (2007). On the relations between the direction of two-dimensional arm movements and cell discharge in primate motor cortex. J. Neurophysiol. 2, 1527–1537. Hinterberger, T., Ku¨bler, A., Kaiser, J., Neumann, N., and Birbaumer, N. (2003). A brain-computerinterface (BCI) for the locked-in: Comparison of diVerent EEG classifications for the thought translation device. Clin. Neurophysiol. 114, 416–425. Hinterberger, T., Veit, R., Wilhelm, B., Weiskopf, N., Vatine, J. J., and Birbaumer, N. (2005). Neuronal mechanisms underlying control of a brain-computer interface. Eur. J. Neurosci. 21, 3169–3181. Hochberg, L. R., Serruya, M. D., Friehs, G. M., Mukand, J. A., Saleh, M., Caplan, A. H., Branner, A., Chen, D., Penn, R. D., and Donoghue, J. P. (2006). Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442, 164–171. Kotchoubey, B. (2005). Event-related potential measures of consciousness: Two equations with three unknowns. Prog. Brain Res. 150, 427–444. Kotchoubey, B., Strehl, U., Uhlmann, C., Holzapfel, S., Ko¨nig, M., Fro¨scher, W., Blankenhom, V., and Birbaumer, N. (2001). Modification of slow cortical potentials in patients with refractory epilepsy: A controlled outcome study. Epilepsia 42(3), 406–416. Ku¨bler, A., and Birbaumer, N. (2008). Brain-computer interfaces and communication in paralysis: Extinction of goal directed thinking in completely paralysed patients? Clin. Neurophysiol. 119(11), 2658–2666. Lal, T. N., Schro¨der, M., Hinterberger, T., Weston, J., Bogdan, M., Birbaumer, N., and Scho¨lkopf, B. (2004). Support vector channel selection in BCI. IEEE Trans. Biomed. Eng. 51(6), 1003–1010. Lebedev, M. A., and Nicolelis, M. A. (2006). Brain machine interfaces: Past, present and future. Trends Neurosci. 29(9), 536–546. Miller, N. E. (1969). Learning of visceral and glandular responses. Science 163, 434–445. Nagai, Y., Goldstein, L. H., Fenwick, P. B. C., and Trimble, M. R. (2004). Clinical eYcacy of galvanic skin response biofeedback training in reducing seizures in adult epilepsy: A preliminary randomized controlled study. Epilepsy Behav. 5, 216–223. Naito, M., Michioka, Y., Ozawa, K., et al. (2007). A communication means for totally locked-in ALS patients based on changes in cerebral blood volume measured with near-infrared light. IEICE Trans. Inf. Syst. E90-D, 1028–1037. Nicolelis, M. A. L., Birbaumer, N., and Mueller, K.-L. (2004). Brain-computer interfaces. Guest Editorial. IEEE Trans. Biomed. Eng. 51, 877–880. Nijboer, F., Furdea, A., Gunst, I., Mellinger, J., McFarland, D. J., Birbaumer, N., and Ku¨bler, A. (2008). An auditory brain-computer-interface (BCI). J. Neurosci. Methods 167, 43–50. Nijboer, F., Sellers, E. W., Mellinger, J., Jordan, M. A., Matuz, T., Halder, S., Mochty, U., Krusienski, D. J., Vaughan, T. M., Wolpaw, J. R., Birbaumer, N., Ku¨bler, A. (2008). A P300based brain-computer interface for people with amyotrophic lateral sclerosis. Clin. Neurophysiol. 119(8), 1909–1916. Owen, A. M., Coleman, M. R., Boly, M., Melanie, B., Matthew, H. D., Steven, L., and John, D. P. (2006). Detecting awareness in the vegetative state. Science 313(5792), 1402.
NEUROFEEDBACK AND BCI
117
Pfurtscheller, G., Guger, C., Mu¨ller, G., Krausz, G., and Neuper, C. (2000). Brain oscillations control hand orthosis in a tetraplegic. Neurosci. Lett. 292(3), 211–214. Pfurtscheller, G., Neuper, C., and Birbaumer, N. (2005). Human brain-computer interface (BCI). In ‘‘Motor Cortex in Voluntary Movements. A Distributed System for Distributed Functions’’ (A. Riehle and E. Vaadia, Eds.), pp. 367–401. CRC Press, Boca Raton. Riehle, A., and Vaadia, E. (Eds.) (2005). ‘‘Motor Cortex in Voluntary Movements. A Distributed System for Distributed Functions.’’ CRC Press, Boca Raton. Rockstroh, B., Elbert, T., Birbaumer, N., Wolf, P., Du¨chting-Ro¨th, A., Reker, M., Daum, I., Lutzenberger, W., and Dichgans, J. (1993). Cortical self-regulation in patients with epilepsies. Epilepsy Res. 14, 63–72. Schwartz, A. B. (2006). Neurobiology: Crossed circuits. Nature 444(7115), 47–48. Schwartz, M., and Andrasik, F. (Eds.) (2003). ‘‘Biofeedback: A Practitioner’s Guide.’’ 3rd ed. Guilford Press, New York. Sellers, E. W., and Donchin, E. (2006). A P300-based brain-computer interface: Initial tests by ALS patients. Clin. Neurophysiol. 117, 538–548. Sitaram, R., Zhang, H., Guan, C., Thulasidas, M., Hoshi, Y., Ishikawa, A., Shimizu, K., and Birbaumer, N. (2007). Temporal classification of multi-channel near-infrared spectroscopy signals of motor imagery for developing a brain-computer interface. NeuroImage 34, 1416–1427. Sterman, M. B., and Clemente, C. D. (1962a). Forebrain inhibitory mechanisms: Cortical synchronization induced by basal forebrain stimulation. Exp. Neurol. 6, 91–102. Strehl, U., Leins, U., Goth, G., Klinger, C., Hinterberger, T., and Birbaumer, N. (2006). Self-regulation of slow cortical potentials—A new treatment for children with attention-deficit/hyperactivity disorder. Pediatrics 118(5), 1530–1540. Velliste, M., Perel, S., Spalding, M. C., et al. (2008). Cortical control of a prosthetic arm for self-feeding. Nature 453, 1098–1101, doi:10.1038/nature06996. Waldert, S., Preissl, H., Demandt, E., Braun, C., Birbaumer, N., Aertsen, A., and Mehring, C. (2008). Hand movement direction decoded from MEG and EEG. J. Neurosci. 28(4), 1000–1008. Walker, J. E., and Kozlowski, G. (2005). Neurofeedback training of epilepsy. Child Adolesc. Psychiatry Clin. N. Am. 14, 163–176. Weiskopf, N., Scharnowski, F., Veit, R., Goebel, R., Birbaumer, N., and Mathiak, K. (2005). Self-regulation of local brain activity using real-time functional magnetic resonance imaging (f MRI). J. Physiol. Paris 98, 357–373.
FLEXIBILITY AND PRACTICALITY: GRAZ BRAIN–COMPUTER INTERFACE APPROACH
Reinhold Scherer,*,y Gernot R. Mu¨ller-Putz,* and Gert Pfurtscheller* *Institute for Knowledge Discovery, Laboratory of Brain–Computer Interfaces, Graz University of Technology, Graz, Austria y Computer Science and Engineering, University of Washington, Seattle, Washington, USA
I. Introduction II. Graz BCI III. Applications A. Operating a (Neuro)Prosthetic Hand—Part I B. Haptic Stimulation: Steady-State Somatosensory-Evoked Potentials C. Operating a (Neuro)Prosthetic Hand—Part II D. The Virtual Keyboard Spelling Device E. Navigation in Virtual Environments F. Operating off-the-Shelf Software IV. Discussion References
‘‘Graz brain–computer interface (BCI)’’ transforms changes in oscillatory electroencephalogram (EEG) activity into control signals for external devices and feedback. Steady-state evoked potentials (SSEPs) and event-related desynchronization (ERD) are employed to encode user messages. User-specific setup and training are important issues for robust and reliable classification. Furthermore, in order to implement small and thus aVordable systems, focus is put on the minimization of the number of EEG sensors. The system also supports the selfpaced operation mode, that is, users have on-demand access to the system at any time and can autonomously initiate communication. Flexibility, usability, and practicality are essential to increase user acceptance. Here, we illustrate the possibilities oVered by now from EEG-based communication. Results of several studies with able-bodied and disabled individuals performed inside the laboratory and in real-world environments are presented; their characteristics are shown and open issues are mentioned. The applications include the control of neuroprostheses and spelling devices, the interaction with Virtual Reality, and the operation of oV-the-shelf software such as Google Earth.
INTERNATIONAL REVIEW OF NEUROBIOLOGY, VOL. 86 DOI: 10.1016/S0074-7742(09)86009-1
119
Copyright 2009, Elsevier Inc. All rights reserved. 0074-7742/09 $35.00
120
SCHERER et al.
I. Introduction
A brain–computer interface (BCI) is a communication system that allows the user to bypass the eVerent pathways of the central nervous system and thus to directly link the human brain with the machine. The motivation for the development of this nonmuscular communication channel is to replace, or at least to somewhat enhance, the lost motor functions of physically disabled persons with intact cortical signals. These include individuals suVering from strokes, spinal cord injuries, or with degenerative diseases like amyotrophic lateral sclerosis. Ablebodied individuals may find BCI-based communication inaccurate and slow compared to their intact motor control abilities. The disabled, however, learned to deal successfully with this technology in their familiar surroundings and to operate spelling devices (Neuper et al., 2006) or neuroprostheses (Mu¨ller-Putz et al., 2005a; Pfurtscheller et al., 2003). Under circumstances or in environments where the body behaves diVerently than under usual conditions, for example, as in space, such an additional ‘‘hands-free’’ communication channel, however, can be advantageous also for able-bodied users. Here, we give an overview of Graz-BCI research and illustrate, by means of diVerent practical applications, the possibilities this kind of technology oVers at the present time. One major aim of our research is to enhance usability, practicality, and flexibility of BCI-based interaction. Important issues in this context are the simplification of the hardware and sensor technology, the reduction of the user training period, and the increased robustness and reliability of the signal processing methods employed.
II. Graz BCI
Graz BCI is based on the real-time detection and classification of transient changes in the ongoing electroencephalogram (EEG) (Pfurtscheller et al., 2006). The EEG signal is in the range of microvolts and consequently is very sensitive to artifacts, that is, to electromagnetic signals not generated by the brain. The most frequent artifacts are muscle activity (electromyogram), eye movements (electrooculogram), and artificial noise generated by nearby electronic devices (e.g., power line interference). Besides artifacts, the nonstationarity and inherent variability of the EEG signal makes a reliable classification of the underlying brain activity diYcult. Steady-state evoked potentials (SSEPs) and event-related desynchronization (ERD) (Pfurtscheller and Lopes Da Silva, 1999) are two neurophysiological phenomena used to encode control messages. SSEPs occur when external sensory
FLEXIBILITY AND PRACTICALITY: GRAZ BCI
121
stimuli are presented in such a rapid sequence that the resulting individually evoked potentials are overlapping and thus reflecting the stimulation frequency (Regan, 1989). We used visual and somatosensory (tactile) stimuli to evoke detectable brain responses. ERD and event-related synchronization (ERS) (Pfurtscheller and Lopes Da Silva, 1999) describe transient changes in on-going oscillatory EEG activity. ERD means a relative power decrease and ERS means a power increase in specific spectral components over defined brain areas. Motor imagery, that is, the mental simulation of movements, is used to induce ERD and/or ERS in sensorimotor rhythms and is the basis of the ERD–BCI method. To convey messages, the user generally changes their brain activity either in response to a cue from the BCI (cue-based BCI) or voluntarily with free will, when an interaction is required (self-paced BCI). While cue-based BCI follow a fixed time scheme and accept messages only within a predefined time window following the cue, self-paced BCIs must continuously analyze the ongoing EEG in order to autonomously detect these messages (Mason et al., 2007). To reach a high level of classification accuracy, training with the BCI is necessary for two reasons: first, to obtain enough EEG-trials in order to set up the classifier, and second, to enable the user to find the best control strategy (this is crucial for motor imagery). The standard training procedure employed is to first adapt the computer to the user’s brain activity by applying machine learning algorithms to samples of diVerent EEG patterns. Usually statistical classifiers such as Fisher’s linear discriminant analysis (Duda et al., 2001) are employed. Next feedback training is performed to enhance and activate these patterns. Finally, the feedback data are again analyzed and if necessary the classifier is updated. In this way the brain and the BCI are mutually adapting (Pfurtscheller and Neuper, 2001; Vidaurre et al., 2006). To make this technology aVordable and thus also accessible to patients, the requirements during optimization and adaptation are not only accuracy and robustness, but also the reduction of the number of EEG sensors. BCI training may last for hours, weeks, or even longer and requires ongoing interaction between user and researcher. To facilitate user training Graz BCI uses telemonitoring (Mu¨ller et al., 2003; Neuper et al., 2003), that is, it provides remote access, audio/video communication capabilities, and file transfer tools. Graz BCI is implemented by using Matlab/Simulink-based (MathWorks, Inc., Natick, MA, USA) rapid prototyping (Guger et al., 2001). The hardware consists of a commercial biosignal amplifier (Guger Technology, Graz, Austria) connected to a data acquisition card and a standard personal computer or laptop. The Graz-BCI open source software package rtsBCI (Scherer et al. 2004b) includes the modules that have been used to successfully realize the results presented here. The package is licensed under the GNU Public License (GPL) and can be downloaded from the BIOSIG homepage that is hosted by the Sourceforge Web site (http://biosig.sourceforge.net).
122
SCHERER et al.
III. Applications
A. OPERATING A (NEURO)PROSTHETIC HAND—PART I When individuals gaze at a flickering light source, steady-state visual evoked potentials (SSVEPs) are evoked over the visual cortex. In our first feasibility study, four lights, each flickering at a diVerent rate, were used to encode control messages for an electromechanical hand prosthesis (Mu¨ller-Putz and Pfurtscheller, 2008). One light on the index finger flickering at 6 Hz and one on the pinky finger flickering at 7 Hz translated to commands for turning the hand in supination and pronation. Two lights on the wrist (flickering at 8 and 13 Hz) represented the commands for opening and closing the hand (Fig. 1A). Four naı¨ve able-bodied subjects followed a given grasping sequence at will. Three out of the four subjects successfully performed the predefined sequence. Erroneous selections had to be undone. The fourth subject was not able to obtain SSVEP control. Two bipolar EEG channels were recorded from four electrodes placed on predefined positions over the visual cortex. The goal of a recent paper of Mu¨ller-Putz et al. (2008) was to investigate optimal electrode positions by evaluating classification accuracies from a set of 21 electrode positions placed over the entire occipital cortex. Results based on data from 10 able-bodied subjects show that the classification accuracy of individually selected electrode positions is significantly higher than those obtained with the ‘‘default’’ electrode positions used in the previous study. Furthermore, a comparison of diVerent signal processing (Mu¨ller-Putz 2005a, 2008) methods suggests that it is possible to detect SSVEPs after a very short training period— basically it is possible to operate a self-paced BCI after a short calibration period of 1 min (Scherer et al., 2007a).
B. HAPTIC STIMULATION: STEADY-STATE SOMATOSENSORY-EVOKED POTENTIALS For real-world applications ongoing acoustic stimulation is not practical. Haptic (sensorimotor) stimulation seems reasonable and thus we researched the usefulness of tactile stimulation in the resonance-like frequency range of the somatosensory system (Mu¨ller et al., 2001). The right index finger of the user was stimulated with the individual specific frequency fT1 (range 25–31Hz). The left index finger was stimulated at fT2 ¼ fT1 5 Hz. A sinusoidal waveform was used to produce a weak tapping stimulation (Fig. 1B). Subjects were asked to focus their attention on the finger indicated by a visual cue and to count the intermittent amplitude twitches of the stimulation signal. The purpose of the counting task was to force the participants to focus on the cued stimulation. Four subjects
FLEXIBILITY AND PRACTICALITY: GRAZ BCI
123
A Turn in supination (6 Hz)
Supination
150
100
100
50 0
Open Pronation
Turn in pronation (7 Hz)
150
50 0
Close
5 10 15 20 25 30 35 40 45
0
0
Hand open (8 Hz)
B
Hand close (13 Hz)
150
150
100
100
50
50
0
0
5 10 15 20 25 30 35 40 45
5 10 15 20 25 30 35 40 45
0
mV2 Hz
0
5 10 15 20 25 30 35 40 45
SIN Transducers
fT2
D fT1
C3, CZ, C4
C
10 s
+10 0 −10 +2 0 −2
mV
15−19 Hz mV
Threshold
2 1
mV2
0
FIG. 1. (A) Hand prosthesis with four light emitting diodes, each associated with a control command. The curves show typical 1-s SSVEP power spectra from one bipolar EEG channel recorded over visual areas. Gray lines indicate the stimulation frequency and the second and third harmonic. (B) The principle of SSSEP-based BCIs—modified from Mu¨ller-Putz et al. (2006). (C) Spinal cord injury (SCI) patient grasping a glass by means of surface functional electrical stimulation (FES). The curves show (from above) one EEG channel recorded bipolarly over sensorimotor foot representation area, the 15–19 Hz band pass filtered EEG, and the power of the filtered signal averaged over the past 1-s period. A threshold detector (horizontal gray line) is used to switch between grasp sequences. (D) SCI patient with implanted neurophrosthesis during a grasp-release performance test.
participated in these feedback experiments. Two of them were unable to focus their attention for the entire duration of an experimental session (usually 160 trials). A selection of their runs with good performance, however, leads to oV-line classification accuracies of about 73%. The performance of the two remaining subjects was better. One subject increased performance from session to session.
124
SCHERER et al.
The online accuracy after three sessions was 71.7%. The last subject was able to focus attention from the very beginning. Online performances ranged between 79.4 and 83.1% (Mu¨ller-Putz et al., 2006). These results suggest that the evoked responses are stable and can be used to encode control messages.
C. OPERATING A (NEURO)PROSTHETIC HAND—PART II Functional electrical stimulation (FES) can be applied to paralyzed limbs and restore motor function. By placing surface electrodes near the motor point of the muscle, or by implanting subcutaneous electrodes and applying stimulation pulses, action potentials are elicited which lead to the contraction of the innervated muscle fibers. In two case studies with spinal cord injury (SCI) patients, we successfully realized self-paced motor imagery-controlled operation of a neuroprosthesis. The grasp function of the left hand of the first patient (29 year, male, SCI at level C5) was restored with FES using surface electrodes. During a 4-month ERD–BCI-training period, the patient learned to induce 17-Hz oscillations by means of foot motor imagery which became suYciently dominant that a threshold detector could be used for the realization of a binary control signal (Pfurtscheller et al., 2003). This trigger signal was used to switch sequentially between grasp phases implemented using the stimulation unit (Fig. 1C). With this grasp the patient was again able to hold for example, a drinking glass. The second patient (42 year, male, SCI sub-C5) had a FreehandW system (Peckam et al., 2001) implanted in his right hand and arm. Within 3 days of feedback training, he learned to reliably induce an ERD pattern during left-hand motor imagery and thus to generate a binary control signal (Mu¨ller-Putz et al., 2005a). In this case, the self-paced BCI system emulated the shoulder joystick which is usually used to operate the FreehandW system. With the BCI-controlled FreehandW system, the patient successfully executed parts of a hand-grasp performance test (Fig. 1D). In the first case, one single bipolar EEG channel, and in the second case, two bipolar EEG channels were recorded from sensorimotor foot and hand representation areas.
D. THE VIRTUAL KEYBOARD SPELLING DEVICE For practical applications, however, one binary control signal might not be suYcient. An increase of the number of brain patterns that can be equally reliably detected also increases the communication speed. To this end a 3-class self-paced ERD–BCI was designed and used to operate the ‘‘Virtual Keyboard (VK)’’ spelling device (Scherer et al., 2004a). Users can write text messages by scrolling through the alphabet and choosing symbols arranged on either side of the screen. Left hand, right hand, and foot motor imagery were used to move the cursor to
FLEXIBILITY AND PRACTICALITY: GRAZ BCI
125
the left, to the right, and to browse the alphabet, respectively. Figure 2A illustrates such a selection process. Three bipolar EEG channels were recorded over sensorimotor hand and foot representation areas. Results of three able-bodied subjects operating the VK, two successfully, showed an improvement of the number of correctly spelled letters per minute (spelling rate) up to ¼ 3.38 (average ¼ 1.99). In the previous 2-class cue-based version of the VK the ability to use the VK varied between ¼ 0.5 and ¼ 0.85 (Obermaier et al., 2003). This performance increase already suggests the inherent potential of BCI technology. Despite the limited information transfer bandwidth, the use of welldesigned human–computer interfaces and optimized selection techniques allowed user to significantly speed up communication.
E. NAVIGATION IN VIRTUAL ENVIRONMENTS The 3-class self-paced ERD–BCI allowed users to reliably switch among three diVerent motor imagery tasks. The classifier, however, was not optimized to reduce the number of erroneous detections (false positive) during periods when BCI control was not needed. To this end an additional classifier was trained to discriminate among the three motor imagery-induced EEG patterns and continuously recorded EEG without motor imagery. Each time the new classifier detected motor imagery, the class identified by the 3-class classifier was the output of the BCI; otherwise the output was ‘‘0’’ and no action was triggered. To further increase the robustness, online methods for eye movement reduction and muscle artifact detection were incorporated (Scherer et al., 2007b). To evaluate the new system, a virtual environment that consisted of a number of labyrinthine arranged hedges with a tree positioned in the middle was created (Fig. 2B). Objects were initially positioned on fixed locations inside the park and users had the task of navigating through the virtual world and collecting them within a time limit. Users could explore the park by moving forward (foot motor imagery) and turning to the left/right (left-/right-hand motor imagery). No directions were given; the subjects could freely choose their trajectory. Three naı¨ve users participated in online experiments. After about 5 h of cue-based, 3-class feedback training, the classification accuracy for each subject reached 80% among the three mental tasks with 17% false positive detections during longer periods when no messages had to be sent. Two out of three subjects succeeded in collecting all three objects; one subject succeeded in collecting only two out of three objects. Figure 2B shows an example of a user chosen trajectory. Because there were no approved performance measures for self-paced operation, the participants were asked to self-report their ability to operate the BCI. The interviews revealed that the ERD–BCI usually detected the control messages correctly (Scherer et al., 2008).
126
SCHERER et al.
A Graz BC
Graz BC
Graz BC
A C Backspace
B D
H J
I
J K
Enter
K
Graz BCI I
G
M
A Enter
L
C Backspace
B D
B Score: 0
Time: 02:09
Starting position Coin Hedge Route
C
FIG. 2. (A) Virtual keyboard letter selection process (left to right): User scroll through the alphabet (symbols move from the bottom to the top of the screen) and pick, by moving the feedback cursor, the desired symbol (letter or control command) shown on the left or on the right half of the screen. This example shows the insertion of the letter ‘‘I’’ in order to spell the word ‘‘BCI.’’ Modified from Scherer et al. (2004a). (B) Screenshot of the virtual environment consisting of a tree and several hedges. The arrows show the current navigation command (here move forward). The trajectory taken by one subject is shown in the map on the right side. Modified from Scherer et al. (2008). (C) Screenshot of the GUI used to operate Google Earth. The user is represented by an icon positioned in the center of the display. The commands at the user’s disposal are placed around this icon and can be selected by moving the feedback cursor (dashed line) into the desired direction. A hierarchical four-level selection procedure allows the user to select the continent, the continental area, the country, and finally to manipulate the virtual camera. The picture shows the experimental setup during a public performance. Modified from Scherer et al. (2007b).
FLEXIBILITY AND PRACTICALITY: GRAZ BCI
127
F. OPERATING OFF-THE-SHELF SOFTWARE Google Earth (Google Inc., Mountain View, CA, USA) is a popular virtual globe program that allows easy access to the world’s geographic information. In contrast to the previously presented applications, the range of functions needed to comfortably operate the software is much higher and, since users have to wait an undefined period of time for the response of the software (e.g., the download of satellite images), a minimization of false positive events is crucial for reasonable operation. Figure 2C shows a screen shot of the specially designed graphical user interface and illustrates the principle of interaction. The user, represented by an icon in the center of the interface, is surrounded by the available commands which can be selected by moving the cursor toward the desired icon for a predefined time. The command ‘‘Scroll’’ (foot motor imagery) was used to browse the available options. As long as this command is enabled, that is, the user continuously performed motor imagery, the options were scrolling from the right to the left side of the screen. The available options were arranged in four levels. These were continent (five options), subcontinent (3–5 options), country (3– 18 options), and camera movement (seven options). This means that first the users have to select the country and then they can position the camera over the desired location. The commands ‘‘Select’’ (left-hand imagery) and ‘‘Back’’ (right-hand imagery) were used to select the current option and to go back to the previous level, respectively. After each selection, Google Earth’s virtual camera moved to the selected position. For more details about this interface see http://www. aksioma.org/brainloop. After about 6 h of feedback training, one subject, previously participating in the virtual environment experiment, successfully operated the application in the lab, as well as in public (Scherer, 2008; Scherer et al., 2007b). The average time required to get from level 1 to level 4 and thus to select one out of 201 available options was about 20 s (minimum 12 s).
IV. Discussion
The presented studies document the advancement of Graz BCI and demonstrate that the system is functioning properly in real-life conditions. The developed system is small, lightweight, robust, and relatively inexpensive because the system complexity has been minimized. On the basis of its open system architecture and rapid prototyping environment, it is highly customizable and incorporating new algorithms is relatively easy. This flexibility and the possibility to remotely adjust parameters and to change the setup allow fast corrections due to unforeseen circumstances (e.g., by suddenly appearing electrical interference
128
SCHERER et al.
caused by a new artificial ventilation system at the patient’s home). It is also easy to combine SSVEPs and ERD/ERS to create a hybrid system, or to measure and process additional biosignals like the electrocardiogram (Pfurtscheller et al., 2008; Scherer et al., 2007a). SSEP-based systems are fast and easy to handle, however, external stimuli are needed. Under some circumstances (e.g., during repair work in outer space which requires the full visual attention or in situations where the user is unable to move) it may not be possible to perceive the required stimulus. On the other hand, motor imagery-based BCIs (ERD–BCI), do not need external stimulation, but the information-transfer-rate is low and reliable classification is diYcult. Whether SSEP or ERD is best for a given application must be decided on a case-by-case basis. Robustness and on-demand operability are extremely important issues. Proper artifact handling is mandatory to ensure that the classifier output is based on voluntary brain activity. So are self-paced operation and self-initiation, that is, the ability of BCI user to autonomously switch the system on and oV (Scherer et al., 2007a). The eVort to reduce the number of channels may cause an increase in the training period. The presented feedback studies, however, prove that the selected ‘‘minimalistic’’ approach achieves satisfactory results within a limited time period. A larger number of channels potentially result in increased classification performances; however, also the probability of electrode failures or of electrode-related error increases. A recent study by Scherer (2008) has shown that the reduction of the number of EEG sensors from 30 to 5 decreases the median classification accuracies, for example, for left hand versus foot motor imagery from 87.1 to 83.2%, that is, only by 4%. Of course higher accuracies result in better performances. The operation of the Virtual Keyboard spelling application and Google Earth, however, clearly indicates the potential of incorporating concepts of the fields’ human–computer interaction and usability. A sound design of the user interface and a sophisticated evaluation of the BCI output may help overcome inaccuracies originated from misclassification and thus supports users and help them to reduce erroneous selections. Once the EEG patterns have been established and the ERD–BCI has been trained, we have found that our system can work reliably for years without the need for any updates. In the case of our first SCI patient, the oscillations have been stable for 9 years; Our Google Earth user’s oscillations have been stable for two years. The long-term stability of trained brain patterns significantly contributes toward the creation of a more reliable communication. Astronauts potentially benefit from BCI-based control of devices in situations where mobility is limited such as during periods of heavy acceleration or during maintenance repairs in outer space. In the former case the self-paced 3-class ERD–BCI could be used to browse through several menus and check the status of the spaceship or execute predefined program sequences; in the latter case a binary
FLEXIBILITY AND PRACTICALITY: GRAZ BCI
129
brain-switch—as developed for neuroprothesis control—could be employed to sequentially flip through a repair manual. Recently it has been shown that the detection accuracy of such a brain switch can be increased when including the post-imagery beta rebound (Pfurtscheller and Solis-Escalante, 2009). Due to the timing of the post-imagery phenomenon, however, the information transfer rate is limited to about 15 bits/min. SSVEP–BCIs may be useful to operate the gripper arm in order to move it closer to the workspace before taking over manual control for fine-tuning. The required visual stimuli, again, could be switched on/oV by means of a brain switch. Another potential field of application for BCI technology is neuromonitoring. Fatigue, stress, increased work load or other mental states which may make individuals error-prone could be detected and used to exemplarily adapt the complexity of the current task or to alert the central station. For any space application, Graz-BCI system already contains a series of methods—both signal processing and experimental strategies—that can be used immediately and can be adapted very easily. The system was engineered so that at the bedside of a patient, the diVerent options can be tested so that the most promising can be applied. Equipped with the experience gained during several months of telemonitoring-based BCI training, a new user can easily be guided through this process. The analogous remote training from a base station on Earth to the International Space Station follows directly.
Acknowledgments
This research was supported in part by the EU project PRESENCCIA (IST-2006-27731), the Fonds zur Fo¨rderung der Wissenschaftlichen Forschung in Austria (project P16326-B02), EU cost action B27, Wings for Life, and the Lorenz-Bo¨hler Foundation. Special thanks to Ru¨diger Rupp (Orthopedic University Hospital II of Heidelberg, Heidelberg, Germany) and Janez Jansˇa (Aksioma—Institute of Contemporary Art, Ljubljana, Slovenia) for their support and to Larry Sorensen for proofreading.
References
Duda, O. R., Hart, P. E., and Stork, D. G. (2001). ‘‘Pattern Classification.’’ 2nd ed. Wiley Interscience, New York. Guger, C., Schlo¨gl, A., Neuper, C., Walterspacher, D., Strein, T., and Pfurtscheller, G. (2001). Rapid prototyping of an EEG-based brain-computer interface (BCI). IEEE Trans. Neural Syst. Rehabil. Eng. 9(1), 49–58. Mason, S. G., Bashashati, A., Fatourechi, M., Navarro, K. F., and Birch, G. E. (2007). A comprehensive survey of brain interface technology designs. Ann. Biomed. Eng. 35(2), 137–169.
130
SCHERER et al.
Mu¨ller, G., Neuper, C., and Pfurtscheller, G. (2001). Resonance-like frequencies of sensorimotor areas evoked by repetitive tactile stimulation. Biomed. Tech. 46, 186–190. Mu¨ller, G. R., Neuper, C., and Pfurtscheller, G. (2003). Implementation of a telemonitoring system for the control of an EEG-based brain-computer interface. IEEE Trans. Neural. Syst. Rehabil. Eng. 11(1), 54–59. Mu¨ller-Putz, G. R., and Pfurtscheller, G. (2008). Control of an electrical prosthesis with an SSVEPbased BCI. IEEE Trans. Biomed. Eng. 55(1), 361–364. Mu¨ller-Putz, G. R., Scherer, R., Pfurtscheller, G., and Rupp, R. (2005a). EEG-based neuroprosthesis control: A step towards clinical practice. Neurosci. Lett. 382(1–2), 169–174. Mu¨ller-Putz, G. R., Scherer, R., Brauneis, C., and Pfurtscheller, G. (2005b). Steady-state visual evoked potential (SSVEP)-based communication: Impact of harmonic frequency components. J. Neural Eng. 2(4), 123–130. Mu¨ller-Putz, G. R., Scherer, R., Neuper, C., and Pfurtscheller, G. (2006). Steady-state somatosensory evoked potentials: Suitable brain signals for brain-computer interfaces? IEEE Trans. Neural Syst. Rehabil. Eng. 14(1), 30–37. Mu¨ller-Putz, G. R., Eder, E., Wriessnegger, S. C., and Pfurtscheller, G. (2008). Comparison of DFT and lock-in amplifier features and search for optimal electrode positions in SSVEP-based BCI. J. Neurosci. Methods 168(1), 174–181. Neuper, C., Mu¨ller, G. R., Ku¨bler, A., Birbaumer, N., and Pfurtscheller, G. (2003). Clinical application of an EEG-based brain-computer interface: A case study in a patient with severe motor impairment. Clin. Neurophysiol. 114(3), 399–409. Neuper, C., Mu¨ller-Putz, G. R., Scherer, R., and Pfurtscheller, G. (2006). Motor imagery and EEGbased control of spelling devices and neuroprostheses. In ‘‘Progress in Brain Research’’ (C. Neuper and W. Klimesch, Eds.), Vol. 159, pp. 403–419. Elsevier, Amsterdam. Obermaier, B., Mu¨ller, G. R., and Pfurtscheller, G. (2003). ‘‘Virtual keyboard’’ controlled by spontaneous EEG activity. IEEE Trans. Neural Syst. Rehabil. Eng. 11(4), 422–426. Peckham, P. H., Keith, M. W., Kilgore, K. L., Grill, J. H., Wuolle, K. S., Thrope, G. B., Gorman, P., Hobby, J., Mulcahey, M. J., Carroll, S., Hentz, V. R., and Wiegner, A. (2001). EYcacy of an implanted neuroprosthesis for restoring hand grasp in tetraplegia: A multicenter study. Arch. Phys. Med. Rehabil. 82(10), 1380–1388. Pfurtscheller, G., and Lopes Da Silva, F. H. (1999). Event-related EEG/MEG synchronization and desynchronization: Basic principles. Clin. Neurophysiol. 110(11), 1842–1857. Pfurtscheller, G., and Neuper, C. (2001). Motor imagery and direct brain-computer communication. Proc. IEEE 89, 123–134. Pfurtscheller, G., and Solis-Escalante, T. (2009). Could the beta rebound in the EEG be suitable to realize a ‘‘brain switch’’? Clin. Neurophysiol. 120(1), 24–29. Pfurtscheller, G., Mu¨ller, G. R., Pfurtscheller, J., Gerner, H. J., and Rupp, R. (2003). ‘‘Thought’’– control of functional electrical stimulation to restore hand grasp in a patient with tetraplegia. Neurosci. Lett. 351(1), 33–36. Pfurtscheller, G., Mu¨ller-Putz, G., Schlo¨gl, A., Graimann, B., Scherer, R., Leeb, R., Brunner, C., Keinrath, C., Lee, F. Y., Townsend, G., Vidaurre, C., and Neuper, C. (2006). 15 years of BCI research at Graz University of Technology: Current projects. IEEE Trans. Neural Syst. Rehabil. Eng. 14, 205–210. Pfurtscheller, G., Leeb, R., Friedman, D., and Slater, M. (2008). Centrally controlled heart rate changes during mental practice in immersive virtual environment: A case study with a tetraplegic. Int. J. Psychophysiol. 68, 1–5. Regan, D. (1989). ‘‘Human Brain Electrophysiology: Evoked Potentials and Evoked Magentic Fields in Science and Medicine.’’ Elsevier, Amsterdam. Scherer, R. (2008). Towards practical brain-computer interfaces: Self-paced operation and reduction of the number of EEG sensors, Ph.D. dissertation. Computer Science Faculty, Graz University of Technology, Austria.
FLEXIBILITY AND PRACTICALITY: GRAZ BCI
131
Scherer, R., Mu¨ller, G. R., Neuper, C., Graimann, B., and Pfurtscheller, G. (2004a). An asynchronously controlled EEG-based virtual keyboard: Improvement of the spelling rate. IEEE Trans. Biomed. Eng. 51(6), 979–984. Scherer, R., Schlo¨gl, A., Mu¨ller-Putz, G., and Pfurtscheller, G. (2004b). Inside the Graz-BCI: rtsBCI. In ‘‘Proceedings of the 2nd International Brain-Computer Interface Workshop and Training Course 2004.’’ Biomedical Technik, Vol. 49(1), pp. 81–82. Scherer, R., Mu¨ller-Putz, G. R., and Pfurtscheller, G. (2007a). Self-initiation of EEG-based braincomputer communication using the heart rate response. J. Neural Eng. 4(4), L23–L29. Scherer, R., Schlo¨gl, A., Lee, F. Y., Bischof, H., Grassi, D., and Pfurtscheller, G. (2007b). The selfpaced Graz brain-computer interface: Methods and applications. Comput. Intell. Neurosci. 2007, 79826. Scherer, R., Lee, F., Schlo¨gl, A., Leeb, R., Bischof, H., and Pfurtscheller, G. (2008). Towards self-paced brain-computer communication: Navigation through virtual worlds. IEEE Trans. Biomed. Eng. 55(2), 675–682. Vidaurre, C., Schlo¨gl, A., Cabeza, R., Scherer, R., and Pfurtscheller, G. (2006). A fully online adaptive BCI. IEEE Trans. Biomed. Eng. 53(6), 1214–1219.
ON THE USE OF BRAIN–COMPUTER INTERFACES OUTSIDE SCIENTIFIC LABORATORIES: TOWARD AN APPLICATION IN DOMOTIC ENVIRONMENTS
F. Babiloni,*,y F. Cincotti,* M. Marciani,* S. Salinari,z,} L. Astolfi,*,y F. Aloise,* F. De Vico Fallani,* and D. Mattia* *IRCCS Fondazione Santa Lucia, Rome, Italy Dip. Fisiologia e Farmacologia, Univ. La Sapienza, Rome, Italy z Dip. Informatica e Sistemistica, Univ. La Sapienza, Rome, Italy } ARTS and CRIM Labs, Scuola Superiore Sant’Anna, Pisa, Italy
y
I. Introduction II. Methodology A. Subjects B. Patients’ Preparation and Training C. Experimental Task D. Experimental Training E. Domotic System Prototype Features F. Estimation of the Cortical Activity from EEG Recordings G. Online Processing H. Off-line Analysis III. Results A. Experimentation with Healthy Subjects B. Experimentation with the Patients IV. Discussion References
Brain–computer interface (BCI) applications were initially designed to provide final users with special capabilities, like writing letters on a screen, to communicate with others without muscular eVort. In these last few years, the BCI scientific community has been interested in bringing BCI applications outside the scientific laboratories, initially to provide useful applications in everyday life and in future in more complex environments, such as space. Recently, we implemented a control of a domestic environment realized with BCI applications. In the present chapter, we analyze the methodological approach employed to allow the interaction between subjects and domestic devices by use of noninvasive EEG recordings. In particular, we analyze whether the cortical activity estimated from noninvasive EEG recordings could be useful in detecting mental states related to imagined limb movements. We estimate cortical activity from
INTERNATIONAL REVIEW OF NEUROBIOLOGY, VOL. 86 DOI: 10.1016/S0074-7742(09)86010-8
133
Copyright 2009, Elsevier Inc. All rights reserved. 0074-7742/09 $35.00
134
BABILONI et al.
high-resolution EEG recordings in a group of healthy subjects by using realistic head models. Such cortical activity was estimated in a region of interest associated with the subjects’ Brodmann areas by use of depth-weighted minimum norm solutions. Results show that the use of the estimated cortical activity instead of unprocessed EEG improves the recognition of the mental states associated with limb-movement imagination in a group of healthy subjects. The BCI methodology here presented has been used in a group of disabled patients to give them suitable control of several electronic devices disposed in a three-room environment devoted to neurorehabilitation. Four of six patients were able to control several electronic devices in the domotic context with the BCI system, with a percentage of correct responses averaging over 63%.
I. Introduction
Brain–computer interfaces (BCI) is an area of research that is rapidly growing in the neuroscience and bioengineering fields. One popular approach to the generation of a BCI system consists of the recognition by a computer of the patterns of electrical activity on the scalp gathered from a series of electrodes. One of the problems related to the use of surface EEG is the blurring eVect owing to the smearing of the skull on the transmission of the potential distribution from the cerebral cortex toward the scalp electrodes. This happens since the skull has very low electrical conductivity compared with the scalp or the brain. The blurring eVect makes the EEG data gathered from the scalp electrodes rather correlated, a problem not observed in the cortical EEG data recorded from the invasive implants in monkeys and people. Such correlation makes the work of classifiers problematical, since the features extracted from the diVerent scalp electrodes tend to be rather similar and this correlation is hard to disentangle with blind methods like principal component analysis. In this last decade, high-resolution EEG technologies have been developed to enhance the spatial information content of EEG activity (Gevins et al., 1990; Nunez, 1995). Furthermore, since the ultimate goal of any EEG recording is to provide useful information about the brain activity, a body of mathematical techniques, known as inverse procedures, has been developed to estimate the cortical activity from raw EEG recordings. Examples of these inverse procedures are dipole localization, distributed source, and cortical imaging techniques (Babiloni et al., 2001; Dale and Sereno, 1993; Gevins et al., 1990; Nunez, 1995). Inverse procedures can use linear and nonlinear techniques to localize putative cortical sources from EEG data, by using mathematical models of the head as volume conductors.
BCI DOMOTIC APPLICATIONS
135
More recently it has been suggested that with the use of the modern highresolution EEG technologies it could be possible to estimate the cortical activity associated with the mental imagery of the upper limb movements in humans better than with the scalp electrodes (Babiloni et al., 2001; Cincotti et al., 2002). We currently use the approach to estimate the cortical current density in a particular region of interest (ROI) on the modeled brain structures from highresolution EEG recordings to provide high-quality signals for the extraction of the features useful for a BCI system. In this chapter, we would like to illustrate how with the use of such advanced high-resolution EEG methods for estimating cortical activity it is possible to run a BCI system able to drive and control several devices in a domotic environment. In particular, we first describe a BCI system used on a group of healthy subjects in which the technology of the estimation of the cortical activity is illustrated. Then we demonstrate use of the BCI system to command several electronic devices within a three-room environment designed for neurorehabilitation. The BCI system was tested by a group of six patients.
II. Methodology
A. SUBJECTS Two groups of subjects were involved in training on the BCI system. One was composed of healthy subjects while the second one was composed of disabled persons who used the BCI system to attempt to drive electronic devices in a threeroom facility at the laboratory of the Fondazione Santa Lucia in Rome. The first group was composed of 14 healthy subjects who voluntarily participated in the study. The second group of subjects comprised six patients aVected by Duchenne muscular dystrophy. According to the Barthel index score (BI) for daily activity, all patients depended almost completely on caregivers, having a BI score <35. In general, all patients were unable to walk since they were already adolescent, and their mobility was possible only by means of a wheelchair. This latter was electric in the cases of all (except two) patients and it was driven by a modified joystick which could be manipulated by either the residual ‘‘fine’’ movements of the first and second fingers or the residual movements at wrist. As for the upper limbs, all patients had a residual muscular strength either of proximal or distal arm muscles that was insuYcient for carrying on any everyday life activity. The neck muscles were so weak as to require a mechanical support to maintain the posture in all of them. Finally, eye movements were substantially preserved in all of them. At the moment of the study, none of the patients was using technologically advanced aids.
136
BABILONI et al.
B. PATIENTS’ PREPARATION AND TRAINING Patients were admitted for a neurorehabilitation program that also included the use of a BCI system on a voluntary basis. Caregivers and patients gave informed consent for the recordings in agreement with the ethical committee rules adopted for this study. The rehabilitation programs aimed to allow patients the use of a versatile system for the control of several domestic devices by using diVerent input devices, tailored to the disability level of the final user. One of the possible inputs for this system was the BCI through EEG modulation. The first step of the clinical procedure consisted of an interview and physical examination performed by the clinicians, wherein several levels of the variables of interest (and possible combinations) were addressed as follows: the degree of motor impairment and of reliance on the caregivers for everyday activities, as assessed by the current standardized scale, that is, the BI for ability to perform daily activities; familiarity with transducers and aids (sip/puV, switches, speech recognition, joysticks) that could be used as input to the system; the ability to speak or communicate and be understood by an unfamiliar person; the level of informatics alphabetization, measured by the number of hours/week spent in front of a computer. Information was structured in a questionnaire administered to the patients at the beginning and end of the training. A level of system acceptance by the users was schematized whereby users were asked to indicate with a number ranging from zero (not satisfactory) to five (very satisfactory) their degree of acceptance relative to each of the controlled output devices. The training consisted of weekly sessions over 3–4 weeks, in which the patient and (when required) patient’s caregivers were practicing with the system. During the whole period, patients had the assistance of an engineer and a therapist in their interaction with the system.
C. EXPERIMENTAL TASK Both healthy volunteers and patients were trained in using the BCI system to control the movement of a cursor on the screen on the base of the modulation of their EEG activity. A description of the experimental task performed by all of them during the training follows. Each trial consisted of four phases: 1. Target appearance: a rectangular target appeared on the right side of the screen, covering either the upper or the lower half of the side. 2. Feedback phase: one second after the target, a cursor appeared in the middle of the left side of the screen and moved at a constant horizontal speed to the right. Vertical speed was determined by the amplitude of sensorimotor rhythms (see Section II.G). A cursor sweep lasted about 3 s.
BCI DOMOTIC APPLICATIONS
137
3. Reward phase. If the cursor successfully hit the target, the latter flashed for about 1 s. Otherwise, it just disappeared. 4. Intertrial interval. The screen stayed blank for about 2 s, in which the subject was allowed to blink and swallow. Subjects were aware that the increase or decrease of a specific rhythm in their EEG produced a movement of the cursor toward the top or the bottom of the screen. They were advised to concentrate on kinesthetic imagination of upper limb movements (e.g., fist clenching) to produce a desynchronization of the mu rhythm on relevant channels (cursor up), and to concentrate on kinesthetic imagination of lower limb movements (e.g., repeated dorsiflexion of ankle joint) to produce a contrasting pattern (with possible desynchronization of mu/beta rhythm over the mesial channels, cursor down). With this simple binary task as a performance measure, training is meant to improve performances from 50–70 to 80–100% of correct hits.
D. EXPERIMENTAL TRAINING The BCI training was performed using the BCI2000 software system (Schalk et al., 2004). An initial screening session was used to define the ideal locations and frequencies of each subject’s spontaneous mu- and beta-rhythm activity. During this session, the subject was provided with any feedback (any representation of her/his mu rhythm), and she/he had to perform motor tasks just in open loop. The screening session consisted of the alternate and random presentation of cues on opposite sides of the screen (either up/down-vertical or left/right-horizontal). In two subsequent runs, the subject was asked to execute (first run) or to image (second run) movements of her/his hands or feet upon the appearance of top or bottom targets, respectively. This sequence was repeated three times. From the seventh run on, the targets appeared on the left or right side of the screen, and the subject was asked to move (odd trials) or to image (even trials) his/her hands for a total of 12 trials. The oV-line analysis based on pairs of contrasts for each task was aimed at detecting two, possibly independent, groups of features, which would be used to train the subject to control two independent dimensions in the BCI. Analysis was carried out by replication of the same signal conditioning and feature extraction that was also used in the online processing (training session). Data sets were divided into epochs (usually 1 s long) and spectral analysis performed by means of a maximum entropy algorithm, with a resolution of 2 Hz. DiVerently from the online processing, when the system only computes the few features relevant for BCI control, all possible features in a reasonable range were extracted and analyzed simultaneously. A feature vector was extracted from each epoch, composed of the spectral value at each frequency bin between 0 and 60 Hz, for each spatially filtered channel. When all features in the two data sets
138
BABILONI et al.
under comparison had been extracted, a statistical analysis was performed to assess significant diVerences in the values (epochs) of each feature in the two conditions. Usually a r2 analysis is performed, but in the case of 2-level independent variables (as in this case: Tasks ¼ {T1, T2}), t-test, ANOVA and other test provide analogous results. At the end of this process, the results were available (channel-frequency matrix and head topography of r2 values) and evaluated to identify the most promising set of features to be enhanced with training. Using information gathered from the oV-line analysis, the experimenter set the online feature extractor so that a ‘‘control signal’’ was generated from the linear combination of the time-varying value of these features and then passed to a linear classifier. The latter’s output controlled how the position of the feedback cursor was updated. During the following training sessions, the subjects were thus fed back with a representation of their mu-rhythm activity, so that they could learn how to improve its modulation. Each session lasted about 40 min and consisted of eight 3-min runs of 30 trials. The task was increased in diYculty during the training, so two broadly diVerent task classes could be defined. During the training sessions, subjects were asked to perform the same kinesthetic imagination movement they were asked to do during the screening session. An upward movement of the cursor was associated with the bilateral decrease of mu rhythm over the hand area (which usually occurs during imagination of upper limb movement). Consequently, the (de)synchronization pattern correlated to imagination of lower limb movements made the cursor move downward. On the same principle, the horizontal movement of the cursor to the left (right) was linked to the lateralization of mu rhythm owing to imagination of movement of the left (right).Two diVerent control signals were defined. The vertical control signal was obtained as the sum of the mu rhythm’s amplitude over both hand motor areas; the value of the mu rhythm’s amplitude over the foot area was possibly subtracted (depending on the individual subject’s pattern). The horizontal control channel was obtained as the diVerence between the mu rhythm’s amplitude over each hand’s motor area. During the first 5–10 training sessions, the user was trained to optimize modulation of one control signal at a time, that is, overall amplitude (‘‘vertical control’’) or lateralization (‘‘horizontal control’’) of the mu rhythm. Each control channel was associated with vertical or horizontal movement of a cursor on the screen, respectively. For the training of ‘‘vertical’’ control, the cursor moved horizontally across the screen from left to right at a fixed rate, while the user controlled vertical movements toward appearing targets, justified at the right side of the screen. Similarly, for the training of ‘‘horizontal’’ control, the cursor moved vertically across the screen from top to bottom at a fixed rate, while the user controlled horizontal movements toward appearing targets, justified at the bottom side of the screen.
BCI DOMOTIC APPLICATIONS
139
This phase was considered complete when the healthy subjects reached a performance of 70–80% correct hits (60–65% for patients) on both monodimensional tasks. In the case of bidimensional tasks, performed only by the healthy subjects, the cursor appeared in the center, and its movement was entirely controlled by the subject, using both control channels (‘‘horizontal’’ and ‘‘vertical’’) simultaneously.
E. DOMOTIC SYSTEM PROTOTYPE FEATURES The core system that disabled patients attempted to use to drive electronic devices in a three-room laboratory was implemented as follows. It received logical signals from several input devices (including the BCI system) and converted them into commands that could be used to drive the output devices. Its operation was organized as a hierarchical structure of possible actions, whose relationship could be static or dynamic. In the static configuration, it behaved like a ‘‘cascaded menu’’ choice system and was used to feed the feedback module only with the options available at the moment (i.e., current menu). In the dynamic configuration, an intelligent agent tried to learn from use the most probable choice the user would make. The user could select the commands and monitor the system behavior through a graphic interface. The prototype system allowed the user to operate electric devices remotely (e.g., TV, telephone, lights, motorized bed, alarm, and a front door opener) as well as monitoring the environment with remotely controlled video cameras. While input and feedback signals were carried over a wireless communication, so that the mobility of the patient was minimally aVected, most of the actuation commands were carried via a powerline-based control system. As described above, the generated system admits the BCI is one possible way to communicate with it, being open to accept command by other signals related to the residual ability of the patient. In this study, however, we report only the performance of the patients with the BCI system in the domotic applications.
F. ESTIMATION OF THE CORTICAL ACTIVITY FROM EEG RECORDINGS For all healthy subjects analyzed in this study, sequential MR images were acquired and realistic head models were generated. For all the patients involved in this study, owing to the lack of their MR images, we used the Montreal average head model. Figure 1 shows a realistic head model generated for a particular experimental subject, together with the high-resolution electrode array that was employed. Scalp, skull, dura mater, and cortical surfaces of the realistic and average head models were obtained. The surfaces of the realistic head models were then used to build the boundary element model of the head as volume
140
BABILONI et al.
FIG. 1. A realistic head model employed for the estimation of the cortical activity. Three layers are displayed namely representing dura mater, skull, and scalp. Also the electrode positions are visible on the scalp surface.
conductor employed in the present study. Conductivity values for scalp, skull, and dura mater were those reported previously (Oostendorp et al., 2000). A cortical surface reconstruction was accomplished for each subject’s head with a tessellation of about 5000 triangles on average, while the average head model had about 3000 triangles. The estimation of cortical activity during the mental imagery task was performed in each subject by use of the depth-weighted minimum norm algorithm (Babiloni et al., 2000, 2003). Such estimation returns a current density estimate for each one of the thousand dipoles constituting the modeled cortical source space. Each dipole returns a time-varying amplitude representing the brain activity of a restricted patch of cerebral cortex during the entire task time-course. This rather large amount of data can be synthesized by computation of the total average of all the dipole magnitudes belonging to the same cortical ROI. Each ROI was defined according to each subject’s cortical model adopted in accordance with its Brodmann areas (BAs). Such areas are regions of the cerebral cortex whose neurons share the same anatomical (and often also functional) properties. Actually, such areas are largely used in neuroscience as a coordinate system for sharing cortical activation patterns found with diVerent neuroimaging techniques. In the present study, the activity in the following ROI was taken into account: the primary left and right motor area, related to the BA 4, the left and right primary somatosensory and supplementary motor areas.
BCI DOMOTIC APPLICATIONS
141
G. ONLINE PROCESSING Digitized EEG data were transmitted in real time to the BCI2000 software system (Schalk et al., 2004), which performed all necessary signal processing and displayed feedback to the user. The processing pipe can consist of several stages, which process the signal in sequence. Only the main ones will be mentioned below: spatial filter, spectral feature extraction, feature combination, and normalization. Spatial filter. A general linear combination of data channels was implemented by defining a matrix of weights multiplied by each time sample of potentials (vector). This allowed implementation of diVerent spatial filters, such as the estimation of cortical current density waveforms on the cortical ROIs, by use of weights derived as explained in Section II. Spectral feature extraction was performed every 40 ms, using the latest 300 ms of data. An autoregressive spectral estimator, based on the maximum entropy algorithm, yielded an amplitude spectrum with resolution of 2 Hz. Maximum frequency was limited to 60 Hz. Feature selection and combination. A small subset of those spectral features (frequency bins EEG channels or ROIs) that were significantly modulated by the motor imagery tasks was linearly combined to form a single control signal. Selection of responsive channels and frequency bins, and determination of combination weights, took place before each online session (see Section II.H). In general, only two or three spectral amplitude values (depending on individual patterns) were generally used to obtain the control signal. Normalization. The control channel was detrended to avoid cursor bias, and scaled so that the resulting vertical deflection of the feedback cursor was visible but not saturated. In fact, the vertical position of the cursor was updated every 40 ms by a number of pixels (positive or negative) equal to the output by this stage. Normalization was adaptive, and based on the estimate of the moving average and standard deviation of the control signal. During the very first session of each subject (screening session), since no oV-line analysis was available to guide feature selection and combination, the subject was given no online feedback (targets only).
H. OFF-LINE ANALYSIS After artifact rejection, the EEG intervals corresponding to the feedback phase were binned into two classes—up or down, depending on the target appearing in each trial. The spatial filtering and feature extraction stages of the online processing were replicated. Since no feedback delay issue had to be considered during the oV-line analysis, spectral estimation was computed on
142
BABILONI et al.
1-s-long epochs, overlapped by 50% (i.e., only five spectral estimates had to be computed for each 3-s-long trial, yielding about 600 spectral estimates per class for the whole session). For each of the EEG channels or ROI waveforms employed, and for each one of the 30 frequency bins in which the EEG spectral interval was divided, a contrast was performed, to assess statistically significant modulations induced in a specific feature. To this end, we computed for each feature (dependent variable) the coeYcient of determination (r2), that is, the proportion of the total variance of the feature samples accounted for by target position. This index had been previously utilized in literature for similar experimental setups (Wolpaw et al., 2002) and allows direct comparison with published results. A fictitious independent variable was created, using values þ1 or 1 in correspondence of ‘‘down’’ or ‘‘up’’ epochs, respectively. A negative sign was attributed to the r2 value when dependent and independent variables were contravariant. If we look at statistical results from a diVerent perspective, features characterized by a high r2 value are those that maximize prediction of the current target. Higher values of r2 indicate that the subject has gained steadier control of EEG rhythms (in fact they generally increase during the training, from values below 0.1 to values above 0.3).
III. Results
A. EXPERIMENTATION WITH HEALTHY SUBJECTS By applying the aforementioned signal processing techniques in the context of the proposed BCI setup, we used the r2 as an index of reliability of the recognition of subjects’ mental activity. The comparisons between the maximum values of the r2 that takes into account the best usable feature (frequency/ROI or scalp channel) were performed for the unprocessed EEG data as well as for the estimated cortical activity by use of the procedure already described above. Mean r2 is 0.20 0.114 SD for the unprocessed EEG case, 0.55 0.16 SD for the cortical current density estimation case. The diVerences are relatively constant across the subjects, and a paired Student’s t test returned a highly significant diVerence between the two conditions (p < 10–5). Once all the healthy subjects had completed the training, we chose the two with the best performance and trained them to use a diVerent BCI application, namely the old game of electronic ping-pong. Figure 2 shows a sequence with two subjects who played a ping-pong game with the use of the BCI system realized through the guidelines provided above. The subjects are able to control the movement of the vertical cursors while the white cursor, simulating the ball, moves across the screen. The sequence reads
BCI DOMOTIC APPLICATIONS
143
FIG. 2. Sequence of two healthy subjects that play the ping-pong with the use of the BCI described in the text. Subjects control the cursor movement along the vertical directions. Sequence from (A) to (B).
from A to B. The two subjects are able to control the device by performing 95 and 96% of successful hits during a game lasting several minutes, with a rate of about five correct hits per minute per subject.
B. EXPERIMENTATION WITH THE PATIENTS As described previously in Section II, all the patients underwent a standard BCI training. In 8–12 sessions of training, four out of six patients were able to develop a sensorimotor reactivity suYciently stable to control the cursor with performance in excess of 63%. They could image either foot or hand movements and the related sensorimotor modulation was mainly located at midline centroparietal electrode positions. Two patients were not able to control the cursor with a percentage superior to 55% and were not taken into consideration further here in the context of the use of the BCI system. At the end of the training, the four patients were able to control the several system outputs, namely the domotic appliances. According to the early results of the questionnaire, these patients were independent in the use of the system at the end of the training and they experienced (as they reported) ‘‘the possibility to interact with the environment by myself.’’ A schematic evaluation of the degree of system acceptance revealed that among the several system outputs, the front door opener was the most accepted controlled device. Such an application that controls the access to the domotic environment in the three-room rehabilitation laboratory is illustrated in the first row of Fig. 3. In particular, the figure shows two sequences of commands realized through the BCI system. In the first row, with (A) and (B) there is a sequence in which the BCI system was able to open a door. The red circles of the first row highlight a person entering through the door that was opened by the successful modulation of the EEG mu rhythm. The second row shows the
144
BABILONI et al.
FIG. 3. Two sequences of commands realized through the BCI systems at the Fondazione Santa Lucia in Rome. In the first row, with (A) and (B) there is a sequence with the BCI system that opens a door. In the red circles of the first row a person enters through a door that was opened with the use of the BCI based on the EEG mu rhythm. The second row (C and D) shows the closure of a light with the use of the same BCI system. The BCI system is controlled with the cursor at the right of the screen.
switching-oV of a light with the use of the same BCI system. The feedback from the BCI system is displayed on the screen with the cursor positioned at the lower right of the screen.
IV. Discussion
The data reported here suggest that it is possible to retrieve the cortical activity related to mental imagery by using sophisticated high-resolution EEG techniques, obtained by solving the inverse linear problem with the use of realistic head models. Of course, the analysis of the distribution of the potential fields associated with the motor imagery in humans has already been described (Babiloni et al., 2001; Cincotti et al., 2002; Wolpaw et al., 2002). In the context of the brain–computer interface, however, it assumes importance if the activity related to the imagination of arm movement could be better detected by the use of such high-resolution EEG techniques than that of unprocessed EEG. It is worth noting that the cortical estimation methodology illustrated above is suitable for
BCI DOMOTIC APPLICATIONS
145
the online applications needed for the BCI device. In fact, despite the use of sophisticated realistic head models for scalp, skull, dura mater, and cortical surface, the estimation of the instantaneous cortical distribution from the acquired potential measures required a limited amount of time for a matrix multiplication. Such multiplication occurs between the vector data gathered and the pseudoinverse matrix, which is stored oV-line before the start of the EEG acquisition process. In the pseudoinverse matrix is enclosed the complexity of the geometrical head modeled with the boundary element or with the finite element modeling technique, as well as the a priori constraints used for the minimum norm solutions. The described methodologies were applied in the context of the neurorehabilitation of a group of six patients aVected by Duchenne muscular dystrophy. Four out of six were also able to control with the BCI system several electronic devices disposed in the three-room facility we described previously. The devices guided by them with an average percentage score of 63% are (i) a simple TV remote commander, with the capabilities to switch the device on and oV as well as the capability to change a TV channel, (ii) the switching of the light in a room on and oV, and (iii) the switching on and oV of a mechanical engine for opening a door of the room. These devices can, of course, also be controlled by diVerent inputs signals that eventually use a residual degree of muscular control of patients. This experiment was reported here because it demonstrates the potential for the patient to accept and adapt themselves to the use of the new technology for the control of their domestic environment. There is a large trend in the modern neuroscience field to move toward invasive electrode implants to record cortical activity in both animals and humans for the realization of an eYcient BCI device (Donoghue, 2002; Kennedy et al., 2000; Taylor et al., 2002). In this chapter, we have presented evidence to suggest an alternative methodology for the estimation of such cortical activity in a noninvasive way, by using the possibilities oVered by an accurate modeling of the principal head structures involved in the transmission of the cortical potential from the brain surface to the scalp electrodes.
Acknowledgments
This work was supported by the Minister for Foreign AVairs and the Department for Scientific and Technological Development in the framework of a bilateral project between Italy and China (Tsinghua University), and by the European Union through COST Action BM0601 NEUROMATH. We acknowledge two other institutions as the FP7 projects TOBI and SM4ALL. It was also supported in part by a grant from NIH (EB006356) in the USA and from a grant of the Fondazione Banca Nazionale Comunicazioni (BNC), Rome. This chapter reflects only the authors’ views, and funding agencies are not responsible for any use that may be made of the information contained herein.
146
BABILONI et al.
References
Babiloni, F., Babiloni, C., Locche, L., Cincotti, F., Rossini, P. M., and Carducci, F. (2000). High resolution EEG: Source estimates of Laplacian-transformed somatosensory-evoked potentials using a realistic subject head model constructed from magnetic resonance images. Med. Biol. Eng. Comput. 38, 512–519. Babiloni, F., Cincotti, F., Carducci, F., Rossini, P. M., and Babiloni, C. (2001). Spatial enhancement of EEG data by surface laplacian estimation: The use of MRI-based head models. Clin. Neurophysiol. 112(5), 724–727. Babiloni, F., Babiloni, C., Carducci, F., Romani, G. L., Rossini, P. M., Angelone, L. M., and Cincotti, F. (2003). Multimodal integration of high resolution EEG and functional magnetic resonance imaging data: A simulation study. Neuroimage 19(1), 1–15. Cincotti, F., Mattia, D., Babiloni, C., Carducci, F., Bianchi, L., Milla´n, J., Mourin˜o, J., Salinari, S., Marciani, M. G., and Babiloni, F. (2002). Classification of EEG mental patterns by using two scalp electrodes and Mahalanobis distance-based classifiers. Methods Inf. Med. 41, 337–341. Dale, A. M., and Sereno, M. (1993). Improved localization of cortical activity by combining EEG and MEG with MRI cortical surface reconstruction: A linear approach. J. Cogn. Neurosci. 5, 162–176. Donoghue, J. P. (2002). Connecting cortex to machines: Recent advances in brain interfaces. Nature Neurosci. 5(Suppl. 1), 1085–1088. Gevins, A., Brickett, P., Costales, B., Le, J., and Reutter, B. (1990). Beyond topographic mapping: Towards functional-anatomical imaging with 124-channel EEG and 3-D MRIs. Brain Topogr. 53–64. Kennedy, P. R., Bakay, R. A. E., Moore, M. M., Adams, K., and Goldwaithe, J. (2000). Direct control of a computer from the human central nervous system. IEEE Trans. Rehabil. Eng. 8(2), 198–202. Nunez, P. L. (1995). ‘‘Neocortical Dynamics and Human EEG Rhythms.’’ Oxford University, Press, Oxford. Oostendorp, T. F., Delbeke, J., and Stegeman, D. F. (2000). The conductivity of the human skull: Results of in vivo and in vitro measurements. IEEE Trans. Biomed. Eng. 47(11), 1487–1492. Schalk, G., McFarland, D. J., Hinterberger, T., Birbaumer, N., and Wolpaw, J. R. (2004). BCI2000: A general purpose brain–computer interface (BCI) system. IEEE Trans. Biomed. Eng. 51(6), 1034–1043. Taylor, D. M., Tillery, S. I. H., and Schwartz, A. B. (2002). Direct cortical control of 3D neuroprosthetic devices. Science 296, 1829–1832. Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., and Vaughan, T. M. (2002). Brain computer interfaces for communication and control. Clin. Neurophysiol. 113, 767–791.
BRAIN–COMPUTER INTERFACE RESEARCH AT THE WADSWORTH CENTER: DEVELOPMENTS IN NONINVASIVE COMMUNICATION AND CONTROL
Dean J. Krusienski* and Jonathan R. Wolpawy y
I. II. III. IV. V.
*School of Engineering, University of North Florida, Jacksonville, Florida, USA Laboratory of Neural Injury and Repair, Wadsworth Center, Albany, New York, USA
Introduction Sensorimotor Rhythm-Based BCI Control P300-Based BCI Control Current and Future Directions Conclusion References
Brain–computer interface (BCI) research at the Wadsworth Center focuses on noninvasive, electroencephalography (EEG)-based BCI methods for helping severely disabled individuals communicate and interact with their environment. We have demonstrated that these individuals, as well as able-bodied individuals, can learn to use sensorimotor rhythms (SMRs) to move a cursor rapidly and accurately in one and two dimensions. We have also developed a practical P300based BCI that enables users to access and control the full functionality of their personal computer. We are currently translating this laboratory-proved BCI technology into a system that can be used by severely disabled individuals in their homes with minimal ongoing technical oversight. Our comprehensive approach to BCI design has led to several innovations that are applicable in other BCI contexts, such as space missions.
I. Introduction
The potential utility of BCI for space applications will be to provide alternative and supplemental control to astronauts for purposes of multitasking during critical mission tasks or when normal physical movement is not possible or restricted, such as during shuttle ascent. Additionally, it would be possible to monitor indicators of alertness and fatigue via brain waves during these critical activities and incorporate these indicators into BCI control. In the foreseeable INTERNATIONAL REVIEW OF NEUROBIOLOGY, VOL. 86 DOI: 10.1016/S0074-7742(09)86011-X
147
Copyright 2009, Elsevier Inc. All rights reserved. 0074-7742/09 $35.00
148
KRUSIENSKI AND WOLPAW
future, any application of BCI technology in space will be noninvasive in nature due to the relative infancy of and obvious risks associated with invasive technology. Currently, noninvasive BCI has evolved to the point where it is accurate and reliable enough to be evaluated for suitable space mission tasks. Since 1986, much of the foundational noninvasive BCI research has been conducted at the Wadsworth Center in Albany, New York. This research continues to focus on the development of practical BCI-based communication and control devices for severely disabled individuals. Nevertheless, many of the findings and developments of this research are directly applicable in other contexts where alternative communication channels are desired, including space missions. All relevant aspects of BCIs are systematically investigated at the Wadsworth Center including signal acquisition and characterization; development and evaluation of hardware, software, algorithms, and applications; user training; system dissemination; and evaluation of eYcacy. We have developed the BCI2000 software platform, a general purpose system that supports and facilitates all reasonable combinations of brain signals, recording methods, processing methods, and output devices (Schalk et al., 2004). To date, BCI2000 has been adopted by more than 350 laboratories worldwide and applied to BCI investigations using sensorimotor rhythms (SMRs) (Krusienski et al., 2007; Wolpaw and McFarland, 2004), slow cortical potentials, P300-evoked potentials (Krusienski et al., 2008; Sellers et al., 2006), steady-state visual-evoked potentials (Allison et al., 2008), and signals recorded from the surface of the cortex (electrocorticographic activity, ECoG) (Leuthardt et al., 2004) in conjunction with a variety of user applications (Moore, 2003). Although we have investigated all of the aforementioned BCI paradigms to various extents, our research continues to focus on the development of select noninvasive paradigms that rely on two of the most promising brain signals for practical BCI control: SMRs and P300-evoked potentials. A personal computer can be considered the ultimate communication device, in the sense that they are ubiquitous, the vast majority are connected to the Internet and/or a local communication network, and they are driven by operating systems that are designed to interact with limitless software applications and external devices. In addition, personal computing devices continue to become more portable and functional. Nearly all computing or electronic devices rely on two basic user interface modalities: continuous or actuated inputs such as a mouse or stylus pad, and discrete selections such as a keyboard or keypad. By developing reliable BCI control schemes that emulate the action of these standard user interface modalities, a BCI user would be able to access the full functionality of a personal computer, including unlimited communication and device control possibilities. To achieve this objective, we are continuing to develop SMR-based paradigms for continuous control (e.g., mouse) and P300-evoked potential-based paradigms for discrete control (e.g., keyboard). Recent progress in these two
BCI RESEARCH AT THE WADSWORTH CENTER
149
paradigms has led to the current development of a BCI home system for disabled users. The evolution and future direction of these two paradigms at Wadsworth, as well as relevant aspects of the BCI home system, are described herein.
II. Sensorimotor Rhythm-Based BCI Control
We have shown that people can learn to use motor imagery (McFarland et al., 2000) to actively modulate scalp-recorded EEG signals to move a cursor on a video screen in a continuous fashion in one (McFarland and Wolpaw, 2003; McFarland et al., 2004) or two dimensions (Wolpaw and McFarland, 1994, 2004). Specifically, the users are trained to modulate SMR amplitudes in the mu (8–12 Hz) and/or beta (18–26 Hz) frequency bands over left and/or right sensorimotor cortex. This is not a normal function of this brain signal, but rather the result of training (McFarland et al., 2004). In our early reports of SMR-BCI control, a single mu- or beta-band spectral feature from a single electrode over the sensorimotor cortex was used to control cursor movement in one dimension to hit a target randomly positioned along the edge of a video monitor (McFarland et al., 1993, 2003; Wolpaw et al., 1991). We have progressed to using two hemispherical channels to control cursor movement independently in two dimensions to hit targets along the periphery of the monitor (Wolpaw and McFarland, 1994, 2004). Examples of the 2D SMR control task, along with representative control signal topographies, are illustrated and described in Fig. 1. We have found that Laplacian spatial filters are well suited for localizing SMR signals and reduce the impact of non-EEG artifacts such as the electromyographic (EMG) and electrooculographic (EOG) activity (McFarland et al., 1997; Goncharova et al., 2003). Our current SMR control protocol employs a regression model to produce the control signals. In contrast to our early studies, this model incorporates a linear combination of spectral features (i.e., amplitudes from 3 Hz autoregressive frequency bins) from multiple Laplacian-filtered channels. We found that this regression approach is well suited for SMR cursor control since, in contrast to a discriminant function, it provides continuous output and generalizes well to novel target configurations (McFarland and Wolpaw, 2005; Wolpaw and McFarland, 2004). The basic linear equation used for control is provided in Equation (1), where A are the EEG features (amplitudes) over the left (L) and right (R) hemispheres at frequencies f, w are the associated feature weights, b is the intercept, and K is the gain. ! X f f X f f w LA L þ w RA R þ b : ð1Þ Dx y ¼ K fL
fR
Target locations
1
2
3
4
Appears on screen
Cursor appears in center and moves
Cursor hits the target
Trial completed
5 Blank screen
Right
Amplitude (mV)
Left 8 mV
14 12 10 8 6 4 2 0 0.8
Right target Left target
0.6
3 mV r2
12 Hz amplitude
Up
Down
0.4 0.2 0
0
10
20
30
40
50
60
Frequency (Hz)
FIG. 1. The 2D sensorimotor rhythm (SMR) paradigm. Top: The screen at left shows the eight possible target locations. The other screens show the sequence of events in one trial. 1, a target appears; 2, 1 s later the cursor appears and moves in two dimensions controlled by the user’s EEG activity; 3, the cursor reaches the target; 4, the target flashes for 1 s; 5, the screen is blank for 1 s and then the next trial begins. Bottom: Representative topographical and spectral properties of 2D EEG control. The four topographies indicate the average amplitudes of the 12-Hz mu-rhythm for the labeled control directions. The correlation of these amplitudes with target direction is highest at locations C3 and C4 over sensorimotor cortex, which are associated with right and left hand imagery. The amplitude spectrum and associated r2 correlation plot for the horizontal dimension are shown to the right. The highest correlations with target direction are observed in the mu- and beta-frequency bands, which can be combined for control. The spectra and correlations for the vertical dimension exhibit similar characteristics.
BCI RESEARCH AT THE WADSWORTH CENTER
151
For our online protocol, this equation translates the EEG features into cursor movement 20 times/s. Recent changes in automatic online adaptations of the gain (K ), intercept (b), and feature weights (w) have resulted in a significant improvement in user performance (McFarland and Wolpaw, 2003, 2005). To attain 2D control, users initially learn cursor control in one dimension (i.e., horizontal) based on a regression function. After achieving reliable 1D control, they are trained on a second dimension (i.e., vertical) using an independent regression function. The two functions are then used simultaneously to produce full 2D control. We have demonstrated that this approach results in simultaneous independent control of horizontal and vertical movement, which is comparable in accuracy and speed to that reported in studies using implanted intracortical electrodes in monkeys (Wolpaw et al., 2002). We perform comprehensive spectral and topographical analyses of 64-channel EEG during BCI operation to guide improvements in online operation. In Fabiani et al. (2004), we determined that 2D linear and nonlinear Bayesian classifiers oVer improved performance over 1D linear classifiers. In Schalk et al. (2000), we showed that time-domain features could be combined with SMR amplitudes to increase accuracy of the cursor control task by detecting errors. In another recent study, we developed an empirically derived matched filter for improved tracking of the mu rhythm based on its amplitude- and phase-coupled harmonic components (Krusienski et al., 2007). To further develop SMR control to emulate the function of a computer mouse, we have recently added an additional transient EEG feature that allows users to select an individual icon if desired after intercepting it with the cursor (McFarland et al., 2008). In this scheme, the user first moves the cursor to hit one of multiple possible targets by controlling two independent EEG features (as described previously) and then selects or rejects the target by performing or withholding hand-grasp imagery. This imagery evokes a transient response that can be detected and used to improve the overall accuracy by reducing unintended target selections. Most recently, because autoregressive (AR) spectrum estimation has gained such wide acceptance in BCI, we have investigated the impact of AR model order on performance (Krusienski et al., 2006b; McFarland and Wolpaw, 2008). These studies show that a properly selected model order can produce superior performance, including reducing the correlation between signals for 2D control. Additionally, these studies demonstrate that a performance-based model order selection criterion should be applied rather than traditional criteria that rely on residual error and do not adequately account for the signal dynamics for BCI purposes. We have also conducted preliminary studies that suggest users are also able to accurately control a robotic arm in two dimensions by applying the same techniques used for cursor control. This demonstrates the potential of the SMR protocol
152
KRUSIENSKI AND WOLPAW
to be extended to a variety of applications, with the level of control obtained for one task directly transferring to another task. Our current research eVorts toward improving the SMR paradigm are refining control procedures with the intention of improving accuracy and progressing to higher dimensional control. This includes the identification and transformation of EEG features so that the resulting control signals are as independent, trainable, stable, and predictable as possible.
III. P300-Based BCI Control
We are also continuing to develop the potential of the P300 matrix class of BCI systems originally introduced by Farwell and Donchin (1988). In the original paradigm, the user views a monitor displaying a 6 6 matrix of 36 symbols (refer to Fig. 2). The user focuses attention on a desired symbol in the matrix while the rows and columns of the matrix are highlighted in a random sequence of flashes. Because flashes of the attended symbol are random and rare in the context of the other flash stimuli, a P300 response occurs when the desired symbol is highlighted. By training a classifier using specific spatiotemporal features of the time-locked EEG responses to the stimuli, the classifier can be used to identify the row and column that contain the desired symbol in an online scenario. By assigning a particular command or function to each symbol in the matrix, a user is able to discreetly select from a variety of commands, similar to using a computer keyboard. Our recent studies have aimed to examine and refine the P300 matrix presentation and classification techniques to improve the speed, accuracy, reliability, and generalizability of the paradigm. We examined the eVects of matrix size and interstimulus interval on classification accuracy (Sellers et al., 2006). The results suggest that these matrix presentation parameters can have a considerable impact on performance and should therefore be carefully designed. We also investigated the impact of various combinations of channel selection, channel referencing, data decimation, and the number of regression model features on classification accuracy using stepwise linear discriminant analysis (SWLDA) (Krusienski et al., 2008). We found that, by adding three occipital electrodes (PO7, PO8, and Oz) to the traditional electrodes used for capturing the P300 (Fz, Cz, and Pz), the classification accuracy increased substantially. Furthermore, these six electrodes provided equivalent classification accuracy to an expanded set of 19 electrodes, indicating that the signals from these six electrodes comprise the majority of unique information for classification purposes. We also found that, in general, the parameters evaluated for channel referencing, data decimation, and number of model features did not have a significant eVect on accuracy. Nevertheless, in addition to the six electrode montage (P3 and P4 were
BCI RESEARCH AT THE WADSWORTH CENTER
153
DICE (D)
B
C
D
E
F
G
H
I
J
K
L
M
N
O
P
Q
R
S
T
U
V
W
X
Y
Z
1
2
3
4
5
6
7
8
9
-
2 1 Amplitude
A
0 −1 −2 −3 0
200
600 400 Time (ms)
800
FIG. 2. The P300 matrix speller paradigm. Left side: The standard 6 6 matrix. The letter in parentheses at the top of the window is the current target letter ‘‘D’’. A P300 should be elicited when the fourth column or first row is intensified. After the intensification sequence for a character epoch, the result is classified and online feedback is provided. Right side: (Top) A representative topography indicating a strong correlation over the central electrodes between the target letter intensifications and the EEG amplitude at 300 ms after the intensifications. (Bottom) The associated averaged P300 temporal response for the targets (red) and the non-targets (green) at electrode Cz.
also added after the study, resulting in a more universal eight electrode montage), we established the following preprocessing and model parameters that produce very consistent and eVective results across users: ear referencing, low-pass filtering followed by data decimation at 20 Hz, and a maximum of 60 features in the regression model. These parameters, in addition to the long-term stability of the P300 response, were validated using online experiments in this and subsequent studies. Due to the data smoothing and statistically derived SWLDA classifier, this methodology is generally resilient to modest artifacts and response latency issues. This protocol forms the processing basis for our current P300 system. To evaluate whether the performance of the channel selection and data preprocessing method established in the aforementioned study could be further improved by using an alternative classifier, we compared SWLDA to a linear support vector machine, a nonlinear support vector machine with a Gaussian kernel, Pearson’s correlation method, and Fisher’s linear discriminant in an off line analysis (Krusienski et al., 2006a). The results revealed marginal diVerences between the classification algorithms, with the exception of the overly simplistic Pearson’s correlation method, which was clearly inferior to all other methods
154
KRUSIENSKI AND WOLPAW
tested. Interestingly, the comparatively simple Fisher’s discriminant and SWLDA linear methods provided superior performance to the support vector machines. We are currently investigating the impact of alternate matrix presentations including various matrix sizes, configurations, flashing schemes, and flash intensities. We are also developing new methods of evaluating the relationship between the number of flashes and classification accuracy to further improve the information transfer rate of the system in practical settings.
IV. Current and Future Directions
Based on our recent progress with the SMR and P300 paradigms, extensive tests in the laboratory and the homes of disabled individuals (Ku¨bler et al., 2005; Nijboer et al., 2008; Sellers and Donchin, 2006), and ultimate goal of developing a practical system for disabled individuals to use in their daily lives, we are currently designing and testing a clinical or ‘‘home’’ BCI system based on these paradigms (Vaughan et al., 2006). The current home system includes a laptop computer, a flat panel display, a custom designed eight-channel electrode cap, and a custom designed digital biosignal amplifier. The amplifier has been reduced to 15 4 9 cm, and we anticipate a smaller amplifier with wireless capabilities in the future. While the current electrode cap with gel application is suYcient for the key goals of this project, we are seeking improved sensor and cap solutions that provide reliable, long-term recordings, even in electrically noisy environments, in addition to maximizing comfort and cosmesis. This includes exploring the possibilities of active and dry electrodes. We have also modified the BCI2000 software to include a configurable, menudriven item selection structure that allows the user to navigate various hierarchical menus to perform specific tasks (e.g., basic communication, environmental controls, etc.). Furthermore, we have configured the P300 matrix to emulate a standard computer keyboard such that users can access the full functionality of a personal computer and associated applications. To further enhance the flexibility and communication rate, we have incorporated a predictive speller, a speech synthesis output option, a function that allows the user to suspend and recommence operation using EEG signals for prolonged or continuous operation, and an auditory mode for users who lack suYcient vision. Our preliminary studies have shown that an auditory mode provides stimuli adequate for eliciting a P300 response that is eVective for BCI operation (Sellers et al., 2006). We are currently testing and evaluating the BCI home system in the homes of several disabled individuals, including a 50-year-old man with amyotrophic lateral sclerosis (ALS) who is totally paralyzed except for limited eye movement. He has successfully used the system for daily work and communication tasks over the
BCI RESEARCH AT THE WADSWORTH CENTER
155
past 3 years at least five times per week for up to 8 h per day (Vaughan et al., 2006). He is currently using the P300 matrix paradigm with a 9 8 matrix representing a computer keyboard with arrows for scrolling and additional customized function calls. This configuration, in addition to the predictive speller, allows him to access and eVectively utilize all of his familiar Microsoft Windows-based applications (e.g., Eudora, Word, Excel, PowerPoint, Acrobat) completely via EEG control.
V. Conclusion
The primary objective of BCI research at the Wadsworth Center is to produce a practical and eVective BCI for disabled individuals who are unable to use existing technology to communicate or perform everyday tasks. By focusing on emulating standard computer interfaces such as the keyboard and mouse, the knowledge gained from our studies and endeavors can be directly transferred to other BCI contexts where continuous and discrete control/communication are desired, such as space missions. In addition to the enhanced accuracy and reliability, recent work on the BCI home system has resulted in several other practical improvements well suited for any BCI application. These improvements include the system’s minimized sensors and hardware; portability; improved comfort; extensive external device interface capabilities; and generalized, configurable software and applications. We will continue to reduce the complexity of our BCI systems and increase their flexibility, capacity, and convenience through systematic testing and evaluation on representative user groups.
References
Allison, B. Z., McFarland, D. J., Schalk, G., Zheng, D., Jackson, M. M., and Wolpaw, J. R. (2008). Towards an independent brain-computer interface using steady state visual evoked potentials. Clin. Neurophysiol. 119(2), 399–408. Fabiani, G. E., McFarland, D. J., Wolpaw, J. R., and Pfurtscheller, G. (2004). Conversion of EEG activity into cursor movement by a brain-computer interface (BCI). IEEE Trans. Neural Syst. Rehabil. Eng. 12(3), 331–338. Farwell, L. A., and Donchin, E. (1988). Talking oV the top of your head: Toward a mental prosthesis utilizing event-related brain potentials. Electroencephalogr. Clin. Neurophysiol. 70, 510–523. Goncharova, I. I., McFarland, D. J., Vaughan, T. M., and Wolpaw, J. R. (2003). EMG contamination of EEG: Spectral and topographical characteristics. Clin. Neurophysiol. 114, 1580–1593. Krusienski, D. J., Sellers, E. W., Cabestaing, F., Bayoudh, S., McFarland, D. J., Vaughan, T. M., and Wolpaw, J. R. (2006a). A comparison of classification techniques for the P300 speller. J. Neural Eng. 3, 299–305.
156
KRUSIENSKI AND WOLPAW
Krusienski, D. J., McFarland, D. J., and Wolpaw, J. R. (2006b). An evaluation of autoregressive spectral estimation model order for brain-computer interfaces. In Proceedings of the 28th International IEEE EMBS Conference, August, 2006. Krusienski, D. J., Schalk, G., McFarland, D. J., and Wolpaw, J. R. (2007). A m-rhythm matched filter for continuous control of a brain-computer interface. IEEE Trans. Biomed. Eng. 54(2), 273–280. Krusienski, D. J., Sellers, E. W., McFarland, D. J., Vaughan, T. M., and Wolpaw, J. R. (2008). Toward enhanced P300 speller performance. J. Neurosci. Methods. 167, 15–21. Ku¨bler, A., Nijboer, F., Mellinger, J., Vaughan, T. M., Pawelzik, H., Schalk, G., McFarland, D. J., Birbaumer, N., and Wolpaw, J. R. (2005). Patients with ALS can use sensorimotor rhythms to operate a brain-computer interface. Neurology 64, 1775–1777. Leuthardt, E. C., Schalk, G., Wolpaw, J. R., Ojemann, J. G., and Moran, D. W. (2004). A braincomputer interface using electrocorticographic signals in humans. J. Neural Eng. 1, 63–71. McFarland, D. J., and Wolpaw, J. R. (2003). EEG-based communication and control: Speed-accuracy relationships. Appl. Psychophysiol. Biofeedback 28, 217–231. McFarland, D. J., and Wolpaw, J. R. (2005). Sensorimotor rhythm-based braincomputer interface (BCI): Feature selection by regression improves performance. IEEE Trans. Neural Syst. Rehabil. Eng. 13(3), 372–379. McFarland, D. J., and Wolpaw, J. R. (2008). Sensorimotor rhythm-based brain-computer interface (BCI): Model order selection for autoregressive spectral analysis. J. Neural Eng. 5(2), 155–162. McFarland, D. J., Neat, G. W., Read, R. F., and Wolpaw, J. R. (1993). An EEG-based method for graded cursor control. Psychobiology 21, 77–81. McFarland, D. J., McCane, L. M., David, S. V., and Wolpaw, J. R. (1997). Spatial filter selection for EEG-based communication. Electroencephalogr. Clin. Neurophysiol. 103, 386–394. McFarland, D. J., Miner, L. A., Vaughan, T. M., and Wolpaw, J. R. (2000). Mu and beta rhythm topographies during motor imagery and actual movements. Brain Topogr. 12(3), 177–186. McFarland, D. J., Sarnacki, W. A., and Wolpaw, J. R. (2003). Brain-computer interface (BCI) operation: Optimizing information transfer rates. Biol. Psychol. 63, 237–251. McFarland, D. J., Sarnacki, W. A., Vaughan, T. M., and Wolpaw, J. R. (2004). Brain-computer interface (BCI) operation: Signal and noise during early training sessions. Clin. Neurophysiol. 116, 56–62. McFarland, D. J., Krusienski, D. J., Sarnacki, W. A., and Wolpaw, J. R. (2008). Emulation of computer mouse control with a noninvasive brain-computer interface. J. Neural Eng. 5, 101–110. Moore, M. M. (2003). Real-world applications for brain-computer interface technology. IEEE Trans. Neural Syst. Rehabil. Eng. 11(2), 162–165. Nijboer, F., Sellers, E. W., Mellinger, J., Jordan, M. A., Matuz, T., Furdea, A., Halder, S., Mochty, U., Krusienski, D. J., Vaughan, T. M., Wolpaw, J. R., Birbaumer, N., et al. (2008). A P300-based brain-computer interface for people with amyotrophic lateral sclerosis (ALS). Clin. Neurophysiol. 119, 1909–1916. Schalk, G., Wolpaw, J. R., McFarland, D. J., and Pfurtscheller, G. (2000). EEG-based communication: Presence of an error potential. Clin. Neurophysiol. 111, 2138–2144. Schalk, G., McFarland, D. J., Hinterberger, T., Birbaumer, N., and Wolpaw, J. R. (2004). BCI2000: A general-purpose brain-computer interface (BCI) system. IEEE Trans. Biomed. Eng. 51(6), 1034–1043. Sellers, E. W., and Donchin, E. (2006). A P300-based brain-computer interface: Initial tests by ALS patients. Clin. Neurophysiol. 117, 538–548. Sellers, E. W., Krusienski, D. J., McFarland, D. J., Vaughan, T. M., and Wolpaw, J. R. (2006). Matrix size and inter-stimulus interval eVects using a P300-based brain-computer interface communication system. Biol. Psychol. 117(3), 538–548.
BCI RESEARCH AT THE WADSWORTH CENTER
157
Vaughan, T. M., McFarland, D. J., Schalk, G., Sarnacki, W. A., Krusienski, D. J., Sellers, E. W., and Wolpaw, J. R. (2006). The Wadsworth BCI research and development program: At home with BCI. IEEE Trans. Neural Syst. Rehabil. Eng. 14(2), 229–233. Wolpaw, J. R., and McFarland, D. J. (1994). Multichannel EEG-based brain–computer communication. Electroencephal. Clin. Neurophysiol. 90, 444–449. Wolpaw, J. R., and McFarland, D. J. (2004). Control of a two-dimensional movement signal by a non-invasive brain-computer interface in humans. Proc. Natl. Acad. Sci. USA 101, 17849–17854. Wolpaw, J. R., McFarland, D. J., Neat, G. W., and Forneris, C. A. (1991). An EEG-based braincomputer interface for cursor control. Electroencephalogr. Clin. Neurophysiol. 78, 252–259. Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., and Vaughan, T. M. (2002). Brain-computer interfaces for communication and control. Clin. Neurophysiol. 113, 767–791.
WATCHING BRAIN TV AND PLAYING BRAIN BALL: EXPLORING NOVEL BCI STRATEGIES USING REAL-TIME ANALYSIS OF HUMAN INTRACRANIAL DATA
Karim Jerbi,*,y Samson Freyermuth,* Lorella Minotti,z Philippe Kahane,z Alain Berthoz,* and Jean-Philippe Lachauxy *Laboratoire de Physiologie de la Perception et de l’Action, UMR 7152, Colle`ge de France, CNRS, Paris, France y INSERM U821, Brain Dynamics and Cognition Laboratory, Lyon 69500, France z Department of Neurology and INSERM U704, Grenoble Hospital, Grenoble, France
I. Introduction II. Materials and Methods A. Subjects and Experimental Paradigms B. Human Intracerebral Recordings III. Results A. Brain TV: Real-Time Detection and Visualization of Brain Power Modulations B. Brain Ball: Cursor Control via Voluntary Brain Power Modulation C. BCI Control via Higher Cognitive Processes D. Decoding Oculomotor Intentions IV. Discussion References
A large body of evidence from animal studies indicates that motor intention can be decoded via multiple single-unit recordings or from local field potentials (LFPs) recorded not only in primary motor cortex, but also in premotor or parietal areas. In humans, reports of invasive data acquisition for the purpose of BCI developments are less numerous and signal selection for optimal control still remains poorly investigated. Here we report on our recent implementation of a real-time analysis platform for the investigation of ongoing oscillations in human intracerebral recordings and review various results illustrating its utility for the development of novel brain–computer and brain–robot interfaces. Our findings show that the insight gained both from oV-line experiments and from online functional exploration can be used to guide future selection of the sites and frequency bands to be used in a translation algorithm such as the one needed for a BCI-driven cursor control. Overall, the findings reported with our online spectral analysis platforms (Brain TV and Brain Ball) indicate the feasibility of online functional exploration via intracranial recordings in humans and outline the direct benefits of this approach for the improvement of invasive BCI strategies INTERNATIONAL REVIEW OF NEUROBIOLOGY, VOL. 86 DOI: 10.1016/S0074-7742(09)86012-1
159
Copyright 2009, Elsevier Inc. All rights reserved. 0074-7742/09 $35.00
160
JERBI et al.
in humans. In particular, our findings suggest that current BCI performance may be improved by using signals recorded from various systems previously unexplored in the context of BCI research such as the oscillatory activity recorded in the oculomotor networks as well as higher cognitive processes including working memory, attention, and mental calculation networks. Finally, we discuss current limitations of the methodology and outline future paths for innovative BCI research.
I. Introduction
A crucial step common to all brain–computer interface (BCI) strategies is the selection of the neural signal to be used in the intention decoding process. In the case of invasive BCI settings, the physiological signal may consist of multiunit activity (MUA) or alternatively of local field potentials (LFPs) recordings that pick up the summed postsynaptic potentials of small populations of neurons. Choosing the optimal recording site is a further challenging issue which directly aVects the performance of the decoding process (Andersen et al., 2004). While previous studies have predominantly addressed direct recordings from single brain structures in animals, only a few studies have explored BCI signal selection using invasive recordings in humans. Recent invasive BCI systems in humans have been demonstrated using microelectrodes recording MUAs (Hochberg et al., 2006; Truccolo et al., 2008) as well as electrocorticographic (ECoG) activity (Felton et al., 2007; Leuthardt et al., 2004, 2006; Schalk et al., 2007, 2008). To date, it is still unclear whether unit recordings or population-level recordings will ultimately prove to be more useful in terms of eYciency and robustness for long-term invasive BCIs in humans. It is also currently unknown whether and to which extent higher cognitive processes can be eYciently used for BCI control (Ramsey et al., 2006). Finally, hardly any system beyond the motor cortex has been explored with direct recordings in humans to assess its potential as novel source for BCI intention decoding signals. The aim of this report is to provide an overview of recent implementations in our lab exploring the utility of real-time spectral analysis of intracerebral data acquired via depth electrodes in human subjects for future invasive BCI research. After discussing novel technical implementations for real-time visualization of neural oscillations (Brain TV), we describe its extensions to an oscillatory amplitude modulated cursor-control setup (Brain Ball) and provide two experimental paradigms that illustrate, as a proof of concept, how novel BCI strategies may be achieved using previously unexplored systems such as higher cognitive processes and oculomotor networks.
ONLINE MONITORING OF OSCILLATIONS FOR INVASIVE BCIs
161
II. Materials and Methods
A. SUBJECTS AND EXPERIMENTAL PARADIGMS The real-time analysis platform and all analysis procedures discussed here were used with several implanted patients who suVered from drug-resistant partial epilepsy and who were therefore candidates for surgery at the Departments of Neurology and Neurosurgery of the Grenoble Hospital, France. All patients underwent intracerebral EEG recordings using stereotactically implanted depth electrodes (SEEG) (Jerbi et al., 2009a; Kahane et al., 2004). All patients provided informed consent prior to their participation in the experiments conducted as part of the routine procedures for presurgical mapping. The experimental online paradigms used in the various experiments described here covered a range of cognitive tasks either inspired by previous oV-line analysis or suggested by preliminary real-time observations (Lachaux et al., 2007a).
B. HUMAN INTRACEREBRAL RECORDINGS For the experiments reported here we used multichannel video-EEG acquisition and monitoring system (Micromed, Treviso, Italy) to simultaneously record the intracerebral activity up to 128 depth-EEG electrodes (512 Hz sampling rate and a 0.1–200 Hz bandpass filter). Twelve to 14 semirigid multilead electrodes were implanted in each patient (Fig. 1A). Each electrode had a diameter of 0.8 mm and, depending on the target structure, comprised between 5 and 18 contact leads 2 mm long and 1.5 mm apart (Alcys, Besanc¸on, France). The electrode contacts were identified on the patient’s individual stereotactic scheme, and subsequently anatomically localized using Talairach and Tournoux’s proportional atlas.
III. Results
A. BRAIN TV: REAL-TIME DETECTION AND VISUALIZATION OF BRAIN POWER MODULATIONS In order to be able to compute and monitor online changes of brain oscillations within a BCI-oriented environment, we implemented a real-time analysis and visualization platform which allows both subject and experimenter to follow
162
JERBI et al.
A SEEG electrodes
B
C Brain TV system
Brain Ball system
5 mm
FIG. 1. Real-time estimation and visualization of ongoing brain oscillations recorded with intracerebral electrodes in humans. (A) Stereotactic-electroencephalography (SEEG) implantation scheme (Grenoble Neurological Hospital, France). (B) Brain TV setup allows online visualization of oscillatory activity associated with various motor and mental imagery tasks. (C) Brain Ball setup: Subjects learn to control the ball position by voluntary modulations of cortical oscillations.
cortical power variations as they unfold. The implemented analysis pipeline consists of the following steps: First, a short-term Fourier transform is applied to the signal to extract the time-course of its spectral energy for a wide range of frequencies up to 150 Hz. Next, the obtained TF map is converted to a z-score display, that is, the value at each frequency is normalized with respect to the mean and variance of baseline power level for that given frequency. The normalized TF maps are then averaged in the frequency bands of interest. Additionally, the timecourse of the power is averaged over a short time window (e.g., 2-s) enhancing the signal-to-noise ratio (SNR) and smoothness of the displayed signal. Finally, the frequency-specific power variations for the selected electrode(s) and frequency band(s) are displayed on a computer screen. The online routines were implemented using Matlab (The Mathworks, inc., Sherborn, MA) on a PC that can access (or receive) the ongoing recordings made by the EEG acquisition system via an Ethernet connection (or via a TCP/IP protocol). The online power modulations are displayed as time-varying curves that depict increases or decreases with respect to baseline levels of spectral power estimated over a 10-s calibration period at the beginning of each session while the patient was asked to relax with eyes open. Note that the real-time platform was dubbed Brain TV as it allows both subject and experimenter to watch ongoing cortical oscillations modulations in various implanted regions (Fig. 1B). Besides, we anecdotally also implemented an electrode-selection option via a remote control device allowing the subject to remotely switch the online visualization to display the power modulations at any electrode implanted in his brain.
ONLINE MONITORING OF OSCILLATIONS FOR INVASIVE BCIs
163
B. BRAIN BALL: CURSOR CONTROL VIA VOLUNTARY BRAIN POWER MODULATION The Brain TV real-time spectral visualization system provides a direct representation of brain power modulations. From a practical perspective a simplified representation of brain activity might be expected to be more useful for the investigation of BCI applications and Neurofeedback strategies. We therefore coupled the same online computations with a more abstract visualization that displays power changes as a moving ball on a screen instead of plotting the actual power curves (Fig. 1C). The instantaneous position of the ball is updated at the refresh rate of the online spectral calculations and given by a simple linear transformation of the estimated instantaneous change in oscillatory power at a specific electrode pair within a selected frequency interval. The aim of the socalled Brain Ball setup is to assess the ability of a subject to perform one and twodimensional control based on direct recordings of self-initiated oscillatory power modulations. The electrodes and frequency ranges that are to be used to operate the Brain Ball interface, as well as the cognitive behavior that triggers the voluntary modulations, all need to be investigated beforehand via oV-line and/ or online experiments that search for task-related and endogenously controlled brain rhythms. Using various experiments ranging from simple motor tasks (finger tapping or fist clenching) to mental imagery, visual attention or higher cognitive tasks such as mental calculations, the Brain TV and Brain Ball systems were successfully used to detect task-specific modulations of brain power in various brain structures probed by the depth electrodes as shown by some illustrative examples discussed in the following sections.
C. BCI CONTROL VIA HIGHER COGNITIVE PROCESSES While an increasing number of studies report the feasibility of achieving BCI communication by training subjects to control the amplitude of certain brain rhythms via motor imagery, we set out to use our online system to address the question whether higher cognitive processes may also be used for BCI control. As an example, we investigated the neural dynamics underlying mental calculation with a 22-year-old subject (female, right handed) who received a rare parietofrontal implantation including a depth electrode in the right hemisphere extending from the supramarginal gyrus (BA 40) externally to the cingulate gyrus (BA 31) medially. We hypothesized that some of the electrode contacts on this trajectory might be located close to or within the parietal areas known to be involved in working memory tasks such as calculations and number processing (Dehaene et al., 2003; Piazza et al., 2007). The oV-line spectral analysis of the data acquired during mental calculation revealed task-related modulations of rhythmic activity at several
164
JERBI et al.
electrodes with a notable increase of high gamma (30–140 Hz) oscillatory activity during the counting epochs compared to baseline activity at a parietal electrode pair in the right supramarginal gyrus. Using Brain TV, we showed that changes in high gamma power during self-initiated mental calculation were not only detectable in oV-line analysis but could also be estimated and displayed in real time ( Jerbi et al., 2007). We then tested the subject’s ability to use self-initiated gamma power (by voluntarily engaging in mental calculation) to control the position of a ball on a screen (Brain Ball setup). The instantaneous position of the ball was refreshed online according to a linear transformation applied to the instantaneous changes in gamma power at the identified parietal electrode pair. By endogenously switching between relaxation (a resting state with eyes open) and number processing the subject was able to slide the ball either to the left or to the right with a degree of accuracy that replicates the recorded single trial gamma power variations ( Jerbi et al., 2007).
D. DECODING OCULOMOTOR INTENTIONS As part of a larger ongoing study investigating the neural dynamics of oculomotor planning and execution, we also analyzed intracerebral recordings from implanted patients while performing a delayed saccade task (Khonsari et al., 2007; Milea et al., 2007). The aim was to scan the data for possible neural markers of oculomotor intention which could be of interest for intention decoding in the context of future BCI applications. Our implementation of the delayed saccade paradigm included a condition in which the subjects were cued to prepare to perform either a left or right saccade. A second cue displayed after a variable time delay (ranging from 3.75 to 7.75 s) indicated the ‘‘go’’ signal requiring the subject to perform the prepared saccade. By contrasting the spectral dynamics of the data acquired preceding the execution of left saccades to that preceding execution of right saccades we attempted to detect specific cerebral signatures encoding directionality of the upcoming eye movement. Figure 2 shows the results obtained by contrasting the time–frequency representations of the activity recorded within human dorsolateral prefrontal cortex (DLPFC, Fig. 2A) for left versus right saccade preparation. A striking diVerence in the high gamma-range activity was apparent around 80 Hz (Fig. 2C). This finding, which is part of a larger study to be reported elsewhere, is an optimal example for the type of oculomotor signals that could be used to decode motor intention. The ability of a subject to endogenously communicate an intended direction without actually performing a movement could be used by a BCI system to infer the required movement for a cursor or a robotic device. Most importantly, such high frequency signals from the prefrontal cortex could be combined with intention-specific signals from other structures such as primary or premotor areas in order to increase the degrees of freedom and/or decoding power of a BCI device.
A
DLPFC depth electrode location
B Hz 140
R
TF-maps for electrode in DLPFC Prep right saccade
L 100
C Statistical difference maps Prep left-prep right
60 +6
20
1 0.999 0.99
−400 0 Prep left saccade Hz 140
0.95
80 Hz
0.05 0.1
−6
0.001 0
100 0 450 ms
60
Peak p-value: p = 3.4 × 10 −4
20 D
D
−400
0
1400 ms
FIG. 2. Detecting directional selectivity in oculomotor intention networks. (A) Anatomical location of an electrode contact in right dorsolateral prefrontal cortex (rDLPFC), overlaid on subject’s MRI. (B) Time–frequency (TF) maps: representation of signal power across time and frequencies for two conditions: subject prepares to execute a saccade to the right (upper panel) and subject prepares to execute a saccade to the left (lower panel). (C) Statistical diVerence between the two maps (Left–Right) displays a significant diVerence in the high gamma band round 80 Hz (Kruskall–Wallis test, p ¼ 3.4 10–4).
166
JERBI et al.
IV. Discussion
Beyond its utility for detecting previously unsuspected correlations between real-time oscillatory modulations in intracerebral recordings on one hand and behavior on the other, the Brain TV system can also be used to test whether previously identified neural correlates of behavior can be detected online. This posits it as an ideal platform for BCI exploration. The extension thereof to the cursor-control application Brain Ball provides an additional tool to test novel BCI algorithms and allow for subject training and performance quantification. Our online detection of parietal gamma modulations during mental calculation can be seen as providing a proof of concept at various levels; First, our findings highlight the feasibility of online control of a basic BCI device using higher cognitive processes. In addition, while numerous studies have reported detection of alpha and beta range brain rhythms in single epochs, gamma-range cortical oscillations in parietal regions have been predominantly only evidenced oV-line. The implementation of Brain TV and Brain Ball have shown that high gamma activity can indeed be estimated online and used as a BCI signal. Interestingly, we have recently shown significant spatial correlations between such high gamma oscillations recorded with intracerebral EEG and the BOLD signal measured by functional MRI (Lachaux et al., 2007b). Furthermore, our ongoing work on the oculomotor system reveals that decoding motor intention might well be possible by real-time estimation of specific spectral components of the oculomotor network. In particular preliminary findings suggest that high gamma signals from the dorsolateral prefrontal cortex appear to be an attractive candidate for decoding the direction of an intended saccade. Future investigations of gamma oscillations in oculomotor networks will have to take into account recent indications that gamma-range artifacts may arise from saccade-related ocular muscle activity in scalp-EEG (Reva and Aftanas, 2004; Trujillo et al., 2005; Yuval-Greenberg et al., 2008) and in intracranial recordings (Ball et al. 2009; Jerbi et al., 2009b). Whether patients suVering from complete loss of voluntary muscle activity (locked-in syndrome condition)—including eye movements—retain the ability to voluntarily activate the same oculomotor intention network still needs to be investigated. However, other studies focusing on the primary motor cortex suggest that such areas may remain functional years after brain injury (Hochberg et al., 2006). More studies will be needed to address the utility of further oscillatory phenomena such as long-range coupling ( Jerbi et al., 2007; Schoffelen and Gross, 2009) or cross-frequency synchronization ( Jensen and Colgin, 2007) for BCI applications. Furthermore, the real-time platforms described here can readily be used for the investigation and optimization of neurofeedback techniques, both
ONLINE MONITORING OF OSCILLATIONS FOR INVASIVE BCIs
167
for performance enhancement and for putative clinical applications. Finally, whether the findings achieved with invasive techniques can also be obtained with noninvasive recordings remains a challenging question that is sure to continue to trigger exciting research and heated debates for many years to come.
Acknowledgments
This work was supported in part by the EU research programs NeuroBotics (FET FP6-IST001917), NeuroProbes (FP6-IST 027017) and la Fondation pour la Recherche Me´dicale (FRM).
References
Andersen, R. A., Musallam, S., and Pesaran, B. (2004). Selecting the signals for a brain-machine interface. Curr. Opin. Neurobiol. 14(6), 720–726. Ball, T., Kern, M., Mutschler, I., Aertsen, A., and Schulze-Bonhage, A. (2009). Signal quality of simultaneously recorded invasive and non-invasive EEG. Neuroimage 46(3), 708–716. Dehaene, S., Piazza, M., Pinel, P., and Cohen, L. (2003). Three parietal circuits for number processing. Cogn. Neuropsychol. 20(3–6), 487–506. Felton, E. A., Wilson, J. A., Williams, J. C., and Garell, P. C. (2007). Electrocorticographically controlled brain-computer interfaces using motor and sensory imagery in patients with temporary subdural electrode implants. Report of four cases. J. Neurosurg. 106(3), 495–500. Hochberg, L. R., Serruya, M. D., Friehs, G. M., Mukand, J. A., Saleh, M., Caplan, A. H., Branner, A., Chen, D., Penn, R. D., and Donoghue, J. P. (2006). Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442(7099), 164–171. Jensen, O., and Colgin, L. L. (2007). Cross-frequency coupling between neuronal oscillations. Trends Cogn. Sci. 11(7), 267–269. Jerbi, K., Bertrand, O., SchoendorV, B., HoVmann, D., Minotti, L., Kahane, P., Berthoz, A., and Lachaux, J. P. (2007). Online detection of gamma oscillations in ongoing intracerebral recordings: From functional mapping to brain computer interfaces. In Noninvasive Functional Source Imaging of the Brain and Heart and the International Conference on Functional Biomedical Imaging. NFSI-ICFBI 2007, Hangzhou, 12–14 October 2007. pp. 330–333. Jerbi, K., Lachaux, J. P., N’Diaye, K., Pantazis, D., Leahy, R. M., Gamero, L., and Baillet, S. (2007). Coherent neural representation of hand speed in humans revealed by MEG imaging. Proc. Natl. Acad. Sci. USA 104(18), 7676–7681. Jerbi, K., Ossando´n, T., Hamame´, C. M., Senova, S., Dalal, S. S., Jung, J., Minotti, L., Bertrand, O., Berthoz, A., Kahane, P., and Lachaux, J. P. (2009a). Task-related gamma-band dynamics from an intracerebral perspective: Review and implications for surface EEG and MEG. Hum. Brain Mapp. In press. Jerbi, K., Freyermuth, S., Dalal, S., Kahane, P., Bertrand, O., Berthoz, A., and Lachaux, J. P. (2009b). Saccade Related Gamma-Band Activity in Intracerebral EEG: Dissociating Neural from Ocular Muscle Activity. Brain Topogr. In press.
168
JERBI et al.
Kahane, P., Minotti, L., HoVmann, D., Lachaux, J. P., and Ryvlin, P. (2004). Invasive EEG in the definition of the seizure onset zone: Depth electrodes. In ‘‘Handbook of Clinical Neurophysiology. Pre-surgical Assessment of the Epilepsies with Clinical Neurophysiology and Functional Neuroimaging’’ (F. Rosenow and H. O. Lu¨ders, Eds.), pp. 109–133. Elsevier Science, Amsterdam. Khonsari, R. H., Lobel, E., Milea, D., Lehe´ricy, S., Pierrot-Deseilligny, C., and Berthoz, A. (2007). Lateralized parietal activity during decision and preparation of saccades. Neuroreport 18(17), 1797–1800. Lachaux, J. P., Jerbi, K., Bertrand, O., Minotti, L., HoVmann, D., SchoendorV, B., and Kahane, P. (2007a). A blueprint for real-time functional mapping via human intracranial recordings. PLoS ONE 2(10), e1094. Lachaux, J. P., Fonlupt, P., Kahane, P., Minotti, L., HoVmann, D., Bertrand, O., and Baciu, M. (2007b). Relationship between task-related gamma oscillations and BOLD signal: New insights from combined fMRI and intracranial EEG. Hum. Brain Mapp. 28(12), 1368–1375. Leuthardt, E. C., Schalk, G., Wolpaw, J. R., Ojemann, J. G., and Moran, D. W. (2004). A braincomputer interface using electrocorticographic signals in humans. J. Neural Eng. 1(2), 63–71. Leuthardt, E. C., Miller, K. J., Schalk, G., Rao, R. P., and Ojemann, J. G. (2006). Electrocorticography-based brain computer interface—The Seattle experience. IEEE Trans. Neural Syst. Rehabil. Eng. 14(2), 194–198. Milea, D., Lobel, E., Lehe´ricy, S., Leboucher, P., Pochon, J. B., Pierrot-Deseilligny, C., and Berthoz, A. (2007). Prefrontal cortex is involved in internal decision of forthcoming saccades. Neuroreport 18(12), 1221–1224. Piazza, M., Pinel, P., Le Bihan, D., and Dehaene, S. (2007). A magnitude code common to numerosities and number symbols in human intraparietal cortex. Neuron 53(2), 293–305. Ramsey, N. F., van de Heuvel, M. P., Kho, K. H., and Leijten, F. S. (2006). Towards human BCI applications based on cognitive brain systems: An investigation of neural signals recorded from the dorsolateral prefrontal cortex. IEEE Trans. Neural Syst. Rehabil. Eng. 14(2), 214–217. Reva, N. V., and Aftanas, L. I. (2004). The coincidence between late non phase-locked gamma synchronization response and saccadic eye movements. Int. J. Psychophysiol. 51, 215–222. Schalk, G., Kuba´nek, J., Miller, K. J., Anderson, N. R., Leuthardt, E. C., Ojemann, J. G., Limbrick, D., Moran, D., Gerhardt, L. A., and Wolpaw, J. R. (2007). Decoding two-dimensional movement trajectories using electrocorticographic signals in humans. J. Neural Eng. 4(3), 264–275. Schalk, G., Miller, K. J., Anderson, N. R., Wilson, J. A., Smyth, M. D., Ojemann, J. G., Moran, D. W., Wolpaw, J. R., and Leuthardt, E. C. (2008). Two-dimensional movement control using electrocorticographic signals in humans. J. Neural Eng. 5(1), 75–84. Schoffelen, J. M., and Gross, J. (2009). Source connectivity analysis with MEG and EEG. Hum. Brain Mapp. in press. Truccolo, W., Friehs, G. M., Donoghue, J. P., and Hochberg, L. R. (2008). Primary motor cortex tuning to intended movement kinematics in humans with tetraplegia. J. Neurosci. 28(5), 1163–1178. Trujillo, L. T., Peterson, M. A., Kaszniak, A. W., and Allen, J. J. (2005). EEG phase synchrony differences across visual perception conditions may depend on recording and analysis methods. Clin. Neurophysiol. 116, 172–189. Yuval-Greenberg, S., Tomer, O., Keren, A. S., Nelken, I., and Deouell, L. Y. (2008). Transient induced gamma-band response in EEG as a manifestation of miniature saccades. Neuron 58, 429–441.
ADAPTIVE CHANGES OF RHYTHMIC EEG OSCILLATIONS IN SPACE: IMPLICATIONS FOR BRAIN–MACHINE INTERFACE APPLICATIONS
G. Cheron,*,y A. M. Cebolla,* M. Petieau,* A. Bengoetxea,* E. Palmero-Soler,*,y A. Leroy,* and B. Dan*,z *Laboratory of Neurophysiology and Biomechanics of Movementa, Universite´ Libre de Bruxelles, CP 168, 50 Av. F. Roosevelt, Brussels, Belgium y Laboratory of Electrophysiology, Universite´ de Mons-Hainaut, Mons, Belgium z Department of Neurology, Hopital Universitaire des Enfants Reine Fabiola, Universite´ Libre de Bruxelles, Belgium
I. II. III. IV. V. VI.
Introduction Spontaneous EEG Fluctuations: Where is the Baseline? How to Manage Alpha and Mu Oscillations in Space From the Identification Process to the Exploitation of Brain Oscillations in Space The Influence of ‘‘Top-Down’’ Dynamics on BCI Approach Gamma EEG Oscillations: A Window into Cognition, Perception, Attention, Binding, or Microsaccadic Eye Movements VII. The Gating of the Somatosensory-Evoked Potentials as a New Tools for BCI References
The dramatic development of brain machine interfaces has enhanced the use of human brain signals conveying mental action for controlling external actuators. This chapter will outline current evidences that the rhythmic electroencephalographic activity of the brain is sensitive to microgravity environment. Experiments performed in the International Space Station have shown significant changes in the power of the astronauts’ alpha and mu oscillations in resting condition, and other adaptive modifications in the beta and gamma frequency range during the immersion in virtual navigation. In this context, the dynamic aspects of the resting or default condition of the awaken brain, the influence of the ‘‘top-down’’ dynamics, and the possibility to use a more constrained configuration by a new somatosensory-evoked potential (gating approach) are discussed in the sense of future uses of brain computing interface in space mission. Although, the state of the art of the noninvasive BCI approach clearly demonstrates their ability and the great expectance in the field of rehabilitation for the restoration of defective communication between the brain and external world, their future application in space mission urgently needs a better understanding of brain neurophysiology, in particular in aspects related to neural network rhythmicity in microgravity. INTERNATIONAL REVIEW OF NEUROBIOLOGY, VOL. 86 DOI: 10.1016/S0074-7742(09)86013-3
171
Copyright 2009, Elsevier Inc. All rights reserved. 0074-7742/09 $35.00
172
CHERON et al.
I. Introduction
The opportunity to exploit the electroencephalographic activity (EEG) for controlling computer (brain–computing interface, BCI) or machine (brain– machine interface, BMI) has opened a new field in neuroscience and has also paved the way for their future application in space missions. However, in spite of their exponential development on Earth, a more comprehensive understanding of brain behavior in microgravity needs to be developed before BCI or BMI can be reliably and fully utilized in space mission. This chapter will outline current understanding of brain EEG rhythms in relation to their potential applications in BMI in space.
II. Spontaneous EEG Fluctuations: Where is the Baseline?
Whatever the detection system used the search of a functional baseline remains a challenge (Gusnard and Raichle, 2001). This necessarily implies the control of the resting or default condition of the awaken brain. This default state cannot be regarded as ‘‘a simple resting state’’ but a complex situation involving dynamic interplay between conscious and unconscious processes. Such a brain state can be considered as a transient equilibrium integrating all aspects of past history for future prediction use (Buzsa´ki, 2006). This ‘‘default’’ mode of the brain is also related to the concept of the ‘‘global work space’’ of consciousness (Baars et al., 2003) in which an ‘‘observing’’ or a ‘‘homunculus’’ function is exerted by the frontal pole of the brain on its own sensory influx (Crick and Koch, 2003). In this framework, the Malach’s group has found with fMRI technique that a large part of the cortex consistently responded when subjects were exposed to an audiovisual movie providing a rich and multidimensional natural stimuli (Hasson et al., 2004) but this activated pattern was at the same time accompanied by the presence of persistent ‘‘cortical islands’’ which failed to respond in a clear time-locked manner to the movie stimulation. It was also demonstrated that these regions were not silent during movie watching, but that they displayed a well-correlated spontaneous activity throughout the diVerent ‘‘islands’’ forming a functional chain of an ‘‘intrinsic’’ system organized in complement to the ‘‘extrinsic’’ system dealing with processing of the sensory influx (Goldberg et al., 2008; Golland et al., 2007, 2008). The ‘‘intrinsic’’ attractor state can also modify the brain’s ability to analyze environmental information and to organize final action. Interestingly, the prefrontal cortex engaged in self-related introspective processes is inhibited during sensorimotor processing (Goldberg et al., 2006). This antagonistic and patchwork-like
ADAPTIVE CHANGES OF RHYTHMIC EEG OSCILLATIONS IN SPACE
173
organization largely complicated the definition of a functional baseline because it necessitates a previous knowledge of the ‘‘extrinsic’’ and ‘‘intrinsic’’ networks localization during a particular task. As the free will is central for the initiation of a BCI process, a better understanding of the ‘‘intrinsic’’ system behavior is urgently required. However, the tendency to diVerent human individuals to present similar patterns of brain activation when they are confronted to natural visual scenes (Hasson and Malach, 2006) encourages the utilization of ecological virtual reality stimulator for deciphering the dynamical dialogue between intrinsic and extrinsic neural network in real space work. Ideally, the baseline should be better defined as a physiological state, rather than just the arbitrary period preceding the task (Buzsa´ki, 2006). The BCI approaches are thus constrained by another emerging pattern of collective network: brain oscillations are inherently linked to the neuronal behavior that gave rise to it and in turn, definitely influence the global behavior of the system. This also implies a high level of control of ‘‘state transitions’’ in EEG activity. It was demonstrated that each transition began with an abrupt phase resetting followed by resynchronization, spatial pattern stabilization, and increase in global pattern amplitude (Freeman, 2004).
III. How to Manage Alpha and Mu Oscillations in Space
In the last few years, EEG applied in the emerging field of neuronal oscillations (Buzsa´ki and Draguhn, 2004) has provided new insights into the neurophysiological activity underlying perceptual processes in the brain and represent the principal noninvasive signals for BCI purpose. Surprisingly, the BCI control obtained with scalp-recorded EEG rhythms is situated in the same performance range in terms of speed and precision as the control obtained with intracortical ensemble of single neurons (Wolpaw, 2007). This is a facilitating aspect for future research in this field. Among the diVerent EEG activities extending from very slow to ultrafast frequencies, the alpha rhythm around 10 Hz (range 8–12 Hz) is the most prominent rhythm observed in awake, relaxed subjects (Berger, 1929) and corresponds to the mu rhythm over the sensorimotor cortex (Gastaut, 1952; Pfurtscheller et al., 1996). A combination of mu and beta oscillations has been successfully used for controlling a cursor on a computer screen (Krusienski et al., 2007; Wolpaw and McFarland, 2004). This was accomplished thanks to an intensive training of the subject and the help of an adaptive algorithm providing a weighting mixture of mu and beta rhythm depending of the desired cursor direction. In the context of space mission (NeuroCog I) we used the arrest reaction (Berger, 1929) of the alpha rhythm for studying the eVects of microgravity on the
174
CHERON et al.
spontaneous alpha dynamics (Cheron et al., 2006). Our working hypothesis was that because alpha rhythm has a functional role in the regulation of network properties, microgravity can be expected to modify the strength of this regulation in order to preserve the network functionality in a novel environment. Figure 1 illustrates the raw EEG recorded in the International Space Station (ISS) in one cosmonaut during the arrest reaction. In the eyes-closed state, the alpha rhythm appears as spindle-shaped episodes of 10-Hz oscillation that dominate the spontaneous activity of the brain. We showed that the power spectrum of the alpha rhythm in the eyes-closed state is significantly increased in microgravity compared to that measured on Earth. The same picture was also found for the mu rhythm (Leroy et al., 2007). We also defined a suppression coeYcient (SC) which was also significantly increased in microgravity (Fig. 1B). (Steriade, 2000). The large amplitude of the alpha rhythm would result from a coherent cortical drive from the thalamus coincident with a lack of other input. In this theoretical framework, that is compatible with Crick’s spotlight-of-attention hypothesis (Crick, 1984), microgravity could facilitate the expression of a single-peak 10-Hz dominant rhythm at rest. The alpha rhythm has also been considered as a mechanism for increasing signal-to-noise ratios within the cortex by means of inhibition of unnecessary or conflicting processes to the current task (Klimesch, 1999; Klimesch et al., 2000). The perturbation of the reference frame and the sensory conflicts produced by microgravity could necessitate such a type of regulation. Whatever the exact mechanisms involved, our finding of enhanced alpha and mu rhythm in microgravity suggests their implication in general mechanisms of multiple sensorimotor conflict solving and integration which should be taken into account in future use of BCI in microgravity.
IV. From the Identification Process to the Exploitation of Brain Oscillations in Space
Classically, a BCI approach consists to firstly identify the brain signals (frequency bands and location of interest) and then to develop appropriate methods to extract and exploit the signals in order to further control an external device (Blankertz et al., 2007; Tonet et al., 2008, for a review). However, paradoxically when the initial identification procedure was finalized and the subject asked to produce the mental task and final goal for controlling the output device, the originally detected brain signals change further inducing some alterations in the BCI performance (Schalk et al., 2008). Such undesirable feedback or paradoxical eVect shows that a better identification and extraction procedure of brain oscillations will not necessarily allow an improved BCI performance. In spite of the helpful adaptive abilities of a new BCI processing named SIGFRIED
A F7 F3 Fz F4 F8 C3 Cz C4 T5 P3 Pz P4 O1 O2
1s
70 mV + −
B 2 dB
W ec EA ec
W eo
EB ec
EA eo 6
8
EB eo
10 Frequency (Hz)
12
14
C 12 W Power (dB)
9
6 EB
EA
3
0
6
8
10 Frequency (Hz)
12
14
FIG. 1. (A) Raw EEG recordings during the arrest reaction in weightlessness. Fourteen EEG channels referenced to linked mastoid (from F7 top) to O2 (bottom). The white arrow indexes to the order to open the eyes. The black arrow points to the onset of the eye movement artifact related to eye opening, mainly recorded by frontal electrodes (F7, F3, FZ, F4, F8). Note that the amplitude of mu rhythm [recorded by central electrodes (C3, CZ, C4)] is reduced before eye opening whereas alpha rhythm (P3, P4, PZ, O1, O2) is only reduced after this movement. (B) Superimposition of grand averaged power spectrum of the EEG recorded from the P3 channel in the eyes-closed state (ec) and in the eyesopened state (eo) on Earth before the flight (EB), in weightlessness (W) and on Earth after the flight (EA). (C) Superimposition of the curve representing the diVerence between the power spectrums recorded in the eyes-closed and eyes-opened states. The peak value represents the suppression coeYcient of the alpha rhythm during the arrest reaction. Adapted with permission from Cheron et al. (2006).
176
CHERON et al.
(Schalk et al., 2008) the problem of baseline definition during the resting period prior to mental action and their generalization ability remain unresolved. If this type of problem is already diYcult to manage on Earth it will became highly critical in space condition. In weightlessness, the adaptation of the default state and the reorganization of the ‘‘top- down’’ control of the neuronal activities may render BCI application very challenged. An example of such reorganization of brain EEG oscillations in space missions is given by the EEG recordings performed during virtual navigation experiments conducted on the ground and in the ISS during NeuroCog ESA experiments. In spite of the fact that the cosmonauts were immerged in the same confined virtual environment, the distribution of the basic EEG rhythms significantly change at the diVerent time of the navigation in comparison to the performance for the same task on Earth. In this protocol the cosmonaut was involved in a 3D navigation task initiated by the presentation of the virtual 3D tunnel that directly contains directional information related to the gravitational frame of reference (Vidal et al., 2003; see Fig. 2 for the description of the task). Preliminary results have shown that the EEG rhythmic activities were modified during the navigational task (Bengoetxea et al., 2006, 2007; Cebolla et al., 2006). On Earth, the presentation of the tunnel was firstly accompanied by a power increase in the theta frequency band during about 300 ms followed by a decrease in the alpha band (Fig. 3B). In weightlessness, the ERSP map is clearly diVerent, the initial increase in the theta band and the decrease in the alpha band were less marked than on Earth. Moreover, additional power increase in alpha, beta, and gamma bands occurred in weightlessness (Fig. 3D–G). This observation definitely signed the emergence of a diVerent oscillating state of the brain while the virtual reality environment has remained the same as on the Earth.
V. The Influence of ‘‘Top-Down’’ Dynamics on BCI Approach
Recent advances in cognitive neuroscience have placed the basic neurophysiological processes upon which the BCI should promote new areas of application. One of the major relevant concepts is a ‘‘top-down’’ processing considered on a dynamical point of view as a large scale neuronal influence exerted on small local group of neurons, the latter being literally absorbed by the global ‘‘top-down’’ dynamics. When neurons are highly synchronized they produce local field potentials (oscillating field at the basis of EEG) which are in turn able to recruit new neurons (Cheron et al., 2008; Servais and Cheron, 2005). This constitutes a positive feedback loop where a specific oscillation would be the cause and the consequence of neuronal synchronicity.
ADAPTIVE CHANGES OF RHYTHMIC EEG OSCILLATIONS IN SPACE
177
A
B
C
FIG. 2. (A) Photograph of Frank De Winne on board ISS doing the Neurocog virtual navigation task in freefloating condition under the control of Sergei Zaletin. (B) Virtual reality scenes representing self-navigation were presented through a tunnel by a barrel frame to the laptop screen. On the press of a button, the subject will appear to either move through a tunnel (C) at constant speed, passing through a single bend between two linear segments. At the end of the trial, by manipulating the trackball, the subject adjusts the magnitude of the turn to reconstructs a planar representation of the virtual tunnel just experienced.
The top-down concept is in accordance with the temporal binding model (Engel and Singer, 2001; Singer and Gray, 1995) in which a neuronal synchrony of about 1 ms is necessary for performing object recognition (binding), attention, response selection, and action. The intrinsic capacity of the neuronal assemblies for synchronization is at the basis of the top-down control allowing interaction on a large scale for the selection of pertinent neuronal correlation (Fries et al., 1997, 2001; Varela et al., 2001). Whatever the BCI communication technology and the learning paradigm used, such approach will remain largely dependent of this neuronal dynamics. Indeed, the weightlessness environment may influence the ‘‘top-down’’ dynamics of the brain by diVerent ways including basic changes in the multiple sensory processing for substitution of graviceptive gating or more global modification in awareness, emotional, and cognitive purposes (Cheron et al., 2006; Leroy et al., 2007; Schneider et al., 2007a,b).
Start
Move
Start
Move
Start
Move
C3 D
20 30 40 50
20 30 40 50
0
CZ
500 1000 Time (ms)
20 30 40 50
500 1000 Time (ms)
0
500 1000 Time (ms)
1
10 20 30 40
40
10
0
500 1000 Time (ms)
30 40 50 500 1000 Time (ms)
On earth before flight
1500
1500
0
500 1000 Time (ms)
1500
0
500 1000 Time (ms)
1500
30 40
I
10
2
20
3
30 40
4
50 0
500 1000 Time (ms)
20
−500
1500
−500
Frequency (Hz)
Frequency (Hz)
20
0
H
F
C 10
−500
30
50
−500
1500
20
−500
1500
50
−500
Frequency (Hz)
0
E Frequency (Hz)
Frequency (Hz)
B 10
ERSP (dB) 8 6 4 2 0 −2 −4 −6 −8
10
50
−500
1500
Frequency (Hz)
−500
C4
G
10
Frequency (Hz)
Frequency (Hz)
Frequency (Hz)
A 10
10 20 30 40 50
0
500 1000 Time (ms)
Weightlessness
1500
−500
On earth after flight
FIG. 3. Event-related spectral perturbation (ERSP) recorded during the first 2 s of the virtual navigation performed on Earth before the flight (A–C), in weightlessness (D–F) and on Earth after the flight (G–I). In these ERSP map (grand average data from five subjects), the baseline was taken during the 500 ms preceding the onset of the navigation which was initiated by the presentation of the entrance in the virtual 3D tunnel, 1 s latter the moving in depth initiated the sensation of a walking navigation. The presentation of the tunnel was firstly accompanied by an increase in the power in the theta frequency band during about 300 ms followed on the ground by a decrease in the alpha band. The ellipses (1–4) indicate the major diVerences observed in weightlessness with respect to Earth condition.
ADAPTIVE CHANGES OF RHYTHMIC EEG OSCILLATIONS IN SPACE
179
VI. Gamma EEG Oscillations: A Window into Cognition, Perception, Attention, Binding, or Microsaccadic Eye Movements
The first demonstration of the functional role of neuronal gamma-band synchronization was provided by invasive microelectrode recordings of spikes and local field potentials in the visual cortex of anesthetized cats (Eckhorn et al., 1988; Gray and Singer, 1989; Gray et al., 1989; Engel et al., 1991) and later confirmed in visual cortex of alert monkey (Fries et al., 2001; Kreiter and Singer, 1996). These important experiments have paved the way for the binding-bygamma oscillation originally promoted by Freeman (1975) in the olfactory bulb and theoretically sustained by the mechanism of synchrony (von der Malsburg, 1985). Novel signal-processing techniques for the detection of transient oscillatory events and the introduction of new source modeling methods (Delorme and Makeig, 2004; Delorme et al., 2007; Lachaux, 2003; Makeig, 1993; PalmeroSoler et al., 2007) have facilitated the study of gamma oscillations in human EEG ( Jensen et al., 2007 for a review). Since pioneering MEG studies (Ribary et al., 1991; Llina´s and Ribary, 1993), the interest in the study of gamma oscillation with EEG and MEG increased strongly and their involvement was described in relation to sensorimotor task (Aoki et al., 1999; Donoghue et al., 1998; Murthy and Fetz, 1992), perception (Tallon-Baudry et al., 1996, 1997, 2005; Keil et al., 1999), attention (Gruber et al., 1999), working memory (Tallon-Baudry et al., 1998; Lutzenberger et al., 2002), and associative memory (Gruber et al., 2001). In the case of gamma EEG recordings, a very recent study (Yuval-Greenberg et al., 2008) has clearly demonstrated by single trial analysis of EEG and concomitant eye movement that the induced gamma-band EEG response occurring about 200–300 ms following stimulus onset correspond to miniature saccade dynamics (spike potentials) rather than neuronal oscillations. However, this study does not raise any doubt regarding the intracranial recordings and the important role of gamma-band activity in neural function, but it urgently demonstrated the necessity to control microsaccade events during any type of BCI protocol. This is also reinforced by the fact that eye movement generation is modified in microgravity (Kornilova et al., 1991).
VII. The Gating of the Somatosensory-Evoked Potentials as a New Tools for BCI
One possible way to avoid the problem of the brain state fluctuation for the definition of the baseline and the optimization of the identification process should be to use the gating approach of the earlier somatosensory-evoked potential (SEP). Dynamic gating is a key aspect of neural processing allowing to select
180
CHERON et al.
and enhance integration of stimuli and events that are most relevant in order to keep the focus on specific goals in the face of numerous environmental distractions (O’Reilly, 2006). In this context, single electrical stimuli on the median nerve on the wrist may be viewed as repetitive sensory events upon which a specific brain state is organized. This also reduces the degree of freedom of the default baseline state for a BCI extraction. As grasping task is a possible outcome for BCI action it is relevant to note that sensory integration during such reaching movement is a dynamic process driven by the task computational demands (Sober and Sabes, 2005) during which gating mechanisms involve multiple parallel circuits including loops between the cortex, the basal ganglia and the thalamus. A possible way for noninvasive investigation of these processes and their possible use for BCI purpose reside firstly in the analysis of the spectral content of neuronal oscillatory activity recorded in EEG signals around the time of sensory stimulus and then to control the dynamic modifications induced on this controlled state by a sensory or mental task (gating condition). The electrical median nerve stimulation provides a convenient and easily quantified sensory input for the study of the gating process and their future use in BCI. We have focused our attention on the frontal N30 SEP component (Cheron et al., 2007; Cebolla et al., 2008). The early occurrence of this negative component (30 ms) after the stimulus and their high sensitivity to gating from concomitant involvement of the brain in sensory, motor, and mental activities (Cheron and Borenstein, 1987, 1991, 1992; Rossini et al., 1999; Rushton et al., 1981) is important for their future used in BCI context. Indeed, the amplitude of the frontal N30 significantly decreases when the stimulated hand performs an actual movement or during movement imagery (Cheron and Borenstein, 1991, 1992; Cheron et al., 2000). Recently, we have demonstrated that the N30 SEP component was accompanied by an increase of the power spectrum of beta/gamma rhythm peaking at 30 Hz and by a concomitant increase of the inter trials coherence (phase-locking) at this frequency band (Cheron et al., 2007) (Fig. 4). Pure phase-locking of the beta/gamma rhythm was found in a large percentage of the trials. The gating of the N30 amplitude was accompanied by a significant reduction of the phaselocking value at 30 Hz (Fig. 5). This disorganization of the beta/gamma oscillation phase-locking by the gating is illustrated by the comparison of the superimposition of the filtered (25–35 Hz) EEG trials at rest (Fig. 4E) and during the gating (Fig. 5D). The application of the recent methodology of Freeman (2007) for the extraction and classification of feature vectors may be applied on the frontal N30 gating. Linear decomposition with the Hilbert transform on the frontal electrodes during the median nerve stimulation at rest and during motor imagery may provide diVerent state variables for adequate BCI application (Freeman, 2007). The great advantage is that after band-pass filtering of the beta–gamma brain wave
STIM. Frequency (Hz)
A
Frequency (Hz)
B
C
ERSP
30 90 150
ITC
30
5 (dB) 0 −5 0.6 0.3
90 150 4
0 SEP at rest
2 0
N30
2 −100
25–35 Hz phasesorted trials (n)
D
25–35 Hz oscillation single trials amplitude (mV)
E
−50
0
50 100 Times (ms)
150
200
STIM.
900
−2 (mV)
500
0
100
2 n = 10
−3 0 3 −3
n = 100
0 3 −200
−100
0 Times (ms)
100
200
FIG. 4. Grand average of time/frequency template from ERSP (A) and ITC (B) analysis in relation to the frontal N30 component recorded at rest during median nerve stimulation at the wrist. Note that the peak ITC value coincided with the N30 component peak (C). (D and E) Median nerve stimulationinduced phase resetting of ongoing EEG activity in the 25–35 Hz frequency band recorded from F4. (D) Each of the 900 horizontal lines represents a single trial (n ¼ 2 subjects). Each trial was sorted from top to bottom according to its relative phase in the 25–35 Hz frequency band. The sigmoidal shape of the phase-sorted poststimulus beta wave front indicating uneven phase distribution. The prominent negative peak (N30) waveform represents the sum of more negative (red) than positive (blue) single trial values. (E) Superimposition of 10 (upper traces group) and 100 (lower traces group) single filtered (25–35 Hz) EEG trials with respect to median nerve stimulation, demonstrating the reorganisation of the ongoing beta rhythm. Adapted with permission from Cheron et al. (2007).
182
CHERON et al.
Frequency (Hz)
Frequency (Hz)
ST IM . A
30
5 (dB )
ERSP
90
0
150
−5
30 B
0.6
ITC
0.3
90
0
150
C
Amplitude (mV)
4 SEP gating 2 0 N 30
2 −100
−50
0
D
100
150
STIM.
−3 25−35 Hz oscillation single trials amplitude (mV)
50 Times (ms)
200
n = 10
0 3 n= 1 00 −3 0 3 −200
−100
0 Times (ms)
100
200
FIG. 5. Grand average of time/frequency template from ERSP (A) and ITC (B) analysis in relation to the frontal N30 component recorded during the movement gating paradigm with the same median nerve stimulation as in Fig. 1. Note that the suppression of both ERSP (A) and ITC (B) accompany the N30 gating (C). (D) Superimposition of 10 (upper traces group) and 100 (lower traces group) single filtered (25–35 Hz) EEG trials with respect to median nerve stimulation illustrating the disorganisation of the beta phase-locking by the movement gating. Adapted with permission from Cebolla et al. (2008).
ADAPTIVE CHANGES OF RHYTHMIC EEG OSCILLATIONS IN SPACE
183
sustaining the N30, the Hilbert transform will decomposes this brain wave into an analytic amplitude, A(t) and analytic phase (t) allowing the possibility to identify EEG goal-oriented state transition for future use as BCI commands. A similar approach based on steady-state visual-evoked potentials has been proposed to control a BCI (Allison et al., 2008).
Acknowledgments
This work was funded by the Belgian Federal Science Policy OYce, the European Space Agency, (AO-2004, 118), the Belgian National Fund for Scientific Research (FNRS) and research funds of the Universite´ Libre de Bruxelles (ULB) and the Universite´ de Mons-Hainaut (Belgium). The authors thank M. Dufief, E. Hortmanns, and M. Toussaint for expert technical assistance.
References
Allison, B. Z., McFarland, D. J., Schalk, G., Zheng, S. D., Jackson, M. M., and Wolpaw, J. R. (2008). Towards an independent brain-computer interface using steady state visual evoked potentials. Clin. Neurophysiol. 119, 399–408. Aoki, F., Fetz, E. E., Shupe, L., Lettich, E., and Ojemann, G. A. (1999). Increased (range activity in human sensorimotor cortex during performance of visuomotor tasks. Clin. Neurophysiol. 110, 524–537. Baars, B. J., Ramsoy, T. Z., and Laureys, S. (2003). Brain, conscious experience and the observing self. Trends Neurosci. 26, 671–675. Bengoetxea, A., Cebolla, A., De Saedeleer, C., Leroy, A., Mcintyre, J., Berthoz, A., and Cheron, G. (2006). Weightlessness eVects on visual evoked potential related to virtual in-depth motion. In Proceedings of the Science on European Soyuz Missions to the International Space Station, Toledo, Spain, p. 63. Bengoetxea, A., De Saedeleer, C., Cebolla, A. M., Leurs, F., Mcintyre, J., Berthoz, A., and Cheron, G. (2007). Adaptation of theta and gamma oscillations to weightlessness condition during a virtual motion in depth task. Soc. Neurosci. Abstr. 791.15/K15. ¨ ber das Elektrenkephalogramm des Menschen. Arch. Psychiatr. Nervenkr. 87, Berger, H. (1929). U 527–570. Blankertz, B., Dornhege, G., Krauledat, M., Mu¨ller, K. R., and Curio, G. (2007). The non-invasive Berlin brain-computer interface: Fast acquisition of eVective performance in untrained subjects. Neuroimage 37, 539–550. Buzsa´ki, G. (2006). ‘‘Rhythms of the Brain.’’ Oxford University Press, Oxford, UK. Buzsa´ki, G., and Draguhn, A. (2004). Neuronal oscillations in cortical networks. Science 304, 1926–1929. Cheron, G., and Borenstein, S. (1987). Specific gating of the early somatosensory evoked potentials during active movement. Electroencephalogr. Clin. Neurophysiol. 67, 537–548.
184
CHERON et al.
Cheron, G., and Borenstein, S. (1991). Gating of the early components of the frontal and parietal somatosensory evoked potentials in diVerent sensory-motor interference modalities. Electroencephalogr. Clin. Neurophysiol. 80, 522–530. Cheron, G., and Borenstein, S. (1992). Mental movement simulation aVects the N30 frontal component of the somatosensory evoked potential. Electroencephalogr. Clin. Neurophysiol. 84, 288–292. Cheron, G., Dan, B., and Borenstein, S. (2000). Sensory and motor interfering influences on somatosensory evoked potentials. J. Clin. Neurophysiol. 17, 280–294. Cheron, G., Leroy, A., De Saedeleer, C., Bengoetxea, A., Lipshits, M., Cebolla, A., Servais, L., Dan, B., Berthoz, A., and McIntyre, J. (2006). EVect of gravity on human spontaneous 10-Hz electroencephalographic oscillations during the arrest reaction. Brain Res. 1121, 104–116. Cheron, G., Cebolla, A. M., De Saedeleer, C., Bengoetxea, A., Leurs, F., Leroy, A., and Dan, B. (2007). Pure phase-locking of beta/gamma oscillation contributes to the N30 frontal component of somatosensory evoked potentials. BMC Neurosci. 18, 8–75. Cheron, G., Servais, L., and Dan, B. (2008). Cerebellar network plasticity: From genes to fast oscillation. Neuroscience 153, 1–19. Cebolla, A., De Saedeleer, C., Bengoetxea, A., Leroy, A., Mcintyre, J., Berthoz, A., and Cheron, G. (2006). Microgravity specifically aVects visual evoked potential related to a virtual 3D navigation tunnel. In Proceedings of the Science on European Soyuz Missions to the International Space Station, Toledo, Spain, p. 63. Cebolla, A. M., De Saedeleer, C., Bengoetxea, A., Leurs, F., Balestra, C., d’Alcantara, P., PalmeroSoler, E., Dan, B., and Cheron, G. (2008). Movement gating of beta/gamma oscillations involved in the N30 somatosensory evoked potential. Hum. Brain Mapp. 25. Crick, F. (1984). Function of the thalamic reticular complex: The searchlight hypothesis. Proc. Natl. Acad. Sci. USA 81, 4586–4590. Crick, F., and Koch, C. (2003). A framework for consciousness. Nat. Neurosci. 6, 119–126. Delorme, A., and Makeig, S. (2004). EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J. Neurosci. Methods 15, 9–21. Delorme, A., Sejnowski, T., and Makeig, S. (2007). Enhanced detection of artifacts in EEG data using higher-order statistics and independent component analysis. Neuroimage 34, 1443–1449. Donoghue, J. P., Sanes, J. N., Hatsopoulos, N. G., and Gaal, G. (1998). Neural discharge and local field potential oscillations in primate motor cortex during voluntary movement. J. Neurophysiol. 79, 159–173. Eckhorn, R., Bauer, R., Jordan, W., Brosch, M., Kruse, W., Munk, M., and Reitboeck, H. J. (1988). Coherent oscillations: A mechanism of feature linking in the visual cortex. Biol. Cybern. 60, 121–130. Engel, A. K., and Singer, W. (2001). Temporal binding and the neural correlates of sensory awareness. Trends Cogn. Sci. 5, 16–25. Engel, A. K., Fries, P., and Singer, W. (2001). Dynamic predictions: Oscillations and synchrony in topdown processing. Nat. Rev. Neurosci. 2, 704–716. Freeman, W. J. (1975). ‘‘Mass Action in the Nervous System.’’ Academic Press, New York. Freeman, W. J. (2004). Derivation of EEG information from rates of change in order parameter and free energy dissipation. Conf. Proc. IEEE Eng. Med. Biol. Soc. 6, 4499–4502. Freeman, W. J. (2007). Definitions of state variables and state space for brain-computer interface. Cogn. Neurodyn. 1, 85–96. Fries, P., Roelfsema, P. R., Engel, A. K., Ko¨nig, P., and Singer, W. (1997). Synchronization of oscillatory responses in visual cortex correlates with perception in interocular rivalry. Proc. Natl. Acad. Sci. USA 94, 12699–12704. Fries, P., Reynolds, J. H., Rorie, A. E., and Desimone, R. (2001). Modulation of oscillatory neuronal synchronization by selective visual attention. Science 291, 1560–1563.
ADAPTIVE CHANGES OF RHYTHMIC EEG OSCILLATIONS IN SPACE
185
Gastaut, H. (1952). Etude e´lectrocorticographique de la re´activite´ des rythmes rolandiques. Rev. Neurol. 87, 176–182. Goldberg, I., Harel, M., and Malach, R. (2006). When the brain loses its self: Prefrontal inactivation during sensorimotor processing. Neuron 50, 329–339. Goldberg, I., Ullman, S., and Malach, R. (2008). Neuronal correlates of ‘‘free will’’ are associated with regional specialization in the human intrinsic/default network. Conscious. Cogn. 17, 587–601. Golland, Y., Bentin, S., Gelbard, H., Benjamini, Y., Heller, R., Nir, Y., et al. (2007). Extrinsic and intrinsic systems in the posterior cortex of the human brain revealed during natural sensory stimulation. Cereb. Cortex 17, 766–777. Golland, Y., Golland, P., Bentin, S., and Malach, R. (2008). Data-driven clustering reveals a fundamental subdivision of the human cortex into two global systems. Neuropsychologia 46, 540–553. Gray, C. M., and Singer, W. (1989). Stimulus-specific neuronal oscillations in orientation columns of cat visual cortex. Proc. Natl. Acad. Sci. USA 86, 1698–1702. Gray, C. M., Ko¨nig, P., Engel, A. K., and Singer, W. (1989). Oscillatory responses in cat visual cortex exhibit inter-columnar synchronization which reflects global stimulus properties. Nature 338, 334–337. Gruber, T., Muller, M. M., Keil, A., and Elbert, T. (1999). Selective visual-spatial attention alters induced gamma band responses in the human EEG. Clin. Neurophysiol. 110, 2074–2085. Gruber, T., Keil, A., and Mu¨ller, M. M. (2001). Modulation of induced gamma band responses and phase synchrony in a paired associate learning task in the human EEG. Neurosci. Lett. 316, 29–32. Gusnard, D. A., and Raichle, M. E. (2001). Searching for a baseline: Functional imaging and the resting human brain. Nat. Rev. Neurosci. 2, 685–694. Hasson, U., and Malach, R. (2006). Human brain activation during viewing of dynamic natural scenes. Novartis Found Symp. 270, 203–212. Hasson, U., Nir, Y., Levy, I., Fuhrmann, G., and Malach, R. (2004). Intersubject synchronization of cortical activity during natural vision. Science 303, 1634–1640. Jensen, O., Kaiser, J., and Lachaux, J. P. (2007). Human gamma-frequency oscillations associated with attention and memory. Trends Neurosci. 30, 317–324. Keil, A., Gruber, T., and Mu¨ller, M. M. (2001). Functional correlates of macroscopic high-frequency brain activity in the human visual system. Neurosci. Biobehav. Rev. 25, 527–534. Klimesch, W. (1999). EEG alpha and theta oscillations reflect cognitive and memory performance: A review and analysis. Brain Res. Rev. 29, 169–195. Klimesch, W., Doppelmayr, M., Rohm, D., Pollhuber, D., and Stadler, W. (2000). Simultaneous desynchronization and synchronization of diVerent alpha responses in the human electroencephalograph: A neglected paradox? Neurosci. Lett. 284, 97–100. Kornilova, L. N., Goncharenko, A. M., Polyakov, V. V., Grigorova, V., and Manev, A. (1991). Modifications of spontaneous oculomotor activity in microgravitational conditions. Acta Astronaut. 23, 79–84. Kreiter, A. K., and Singer, W. (1996). Stimulus-dependent synchronization of neuronal responses in the visual cortex of the awake macaque monkey. J. Neurosci. 16, 2381–2396. Krusienski, D. J., Schalk, G., McFarland, D. J., and Wolpaw, J. R. (2007). A mu-rhythm matched filter for continuous control of a brain-computer interface. IEEE Trans. Biomed. Eng. 54, 273–280. Lachaux, J. P. (2003). A simple measure of correlation across time, frequency and space between continuous brain signals. J. Neurosci. Methods 123, 175–188. Leroy, A., De Saedeleer, C., Bengoetxea, A., Cebolla, A., Leurs, F., Dan, B., Berthoz, A., Mcintyre, J., and Cheron, G. (2007). Mu and alpha rhythms during the arrest reaction in microgravity. Microgravity Sci. Technol. 19, 45–50. Llina´s, R., and Ribary, U. (1993). Coherent 40-Hz oscillation characterizes dream state in humans. Proc. Natl. Acad. Sci. USA 90, 2078–2081.
186
CHERON et al.
Lutzenberger, W., Pulvermu¨ller, F., Elbert, T., and Birbaumer, N. (1995). Visual-stimulation alters local 40-Hz responses in humans—An EEG study. Neurosci. Lett. 183, 39–42. Makeig, S. (1993). Auditory event-related dynamics of the EEG spectrum and eVects of exposure to tones. Electroencephalogr. Clin. Neurophysiol. 86, 283–293. Makeig, S., Westerfield, M., Jung, T. P., EnghoV, S., Townsend, J., Courchesne, E., and Sejnowski, T. J. (2002). Dynamic brain sources of visual evoked responses. Science 295, 690–694. Murthy, V. N., and Fetz, E. E. (1992). Coherent 25- to 35-Hz oscillations in the sensorimotor cortex of awake behaving monkeys. Proc. Natl. Acad. Sci. USA 89, 5670–5674. O’Reilly, R. C. (2006). Biologically based computational models of high-level cognition. Science 314, 91–94. Palmero-Soler, E., Dolan, K., Hadamschek, V., and Tass, P. A. (2007). swLORETA: A novel approach to robust source localization and synchronization tomography. Phys. Med. Biol. 52, 1783–1800. Pfurtscheller, G., Stancak, A. Jr., and Neuper, C. (1996). Event-related synchronization (ERS) in the alpha band—An electrophysiological correlate of cortical idling: A review. Int. J. Psychophysiol. 24, 39–46. Ribary, U., Ioannides, A. A., Singh, K. D., Hasson, R., Bolton, J. P., Lado, F., Mogilner, A., and Llina´s, R. (1991). Magnetic field tomography of coherent thalamocortical 40-Hz oscillations in humans. Proc. Natl. Acad. Sci. USA 88, 11037–11041. Rossini, P. M., Babiloni, C., Babiloni, F., Ambrosini, A., Onorati, P., Carducci, F., and Urbano, A. (1999). ‘‘Gating’’ of human short-latency somatosensory evoked cortical responses during execution of movement. A high resolution electroencephalography study. Brain Res. 843, 161–170. Rushton, D. N., Rothwell, J. C., and Craggs, M. D. (1981). Gating of somatosensory evoked potentials during diVerent kind of movement in man. Brain 104, 465–491. Servais, L., and Cheron, G. (2005). EVect of ketamine and pentobarbitone on fast cerebellar oscillation, Purkinje cell rhythmicity and synchronicity. Neuroscience 134, 1247–1259. Singer, W., and Gray, C. M. (1995). Visual feature integration and the temporal correlation hypothesis. Annu. Rev. Neurosci. 18, 555–586. Schalk, G., Brunner, P., Gerhardt, L. A., Bischof, H., and Wolpaw, J. R. (2008). Brain-computer interfaces (BCIs): Detection instead of classification. J. Neurosci. Methods 167, 51–62. Schneider, S., Brummer, V., Gobel, S., Carnahan, H., Dubrowski, A., and Struder, H. K. (2007a). Parabolic flight experience is related to increased release of stress hormones. Eur. J Appl. Physiol. 100, 301–308. Schneider, S., Brummer, V., Mierau, A., Carnahan, H., Dubrowski, A., and Struder, H. K. (2007b). Increased brain cortical activity during parabolic flights has no influence on a motor tracking task. Exp. Brain Res. 185, 571–579. Sober, S. J., and Sabes, P. N. (2005). Flexible strategies for sensory integration during motor planning. Nat. Neurosci. 8, 490–497. Tallon-Baudry, C., Bertrand, O., Delpuech, C., and Pernier, J. (1996). Stimulus specificity of phaselocked and non-phase-locked 40 Hz visual responses in human. J. Neurosci. 16, 4240–4249. Tallon-Baudry, C., Bertrand, O., Delpuech, C., and Pernier, J. (1997). Oscillatory gamma-band (30–70 Hz) activity induced by a visual search task in humans. J. Neurosci. 17, 722–734. Tallon-Baudry, C., Bertrand, O., Peronnet, F., and Pernier, J. (1998). Induced gamma-band activity during the delay of a visual short-term memory task in humans. J. Neurosci. 18, 4244–4254. Tallon-Baudry, C., Bertrand, O., HenaV, M. A., Isnard, J., and Fischer, C. (2005). Attention modulates gamma-band oscillations diVerently in the human lateral occipital cortex and fusiform gyrus. Cereb. Cortex 15, 654–662. Tonet, O., Marinelli, M., Citi, L., Rossini, P. M., Rossini, L., Megali, G., and Dario, P. (2008). Defining brain-machine interface applications by matching interface performance with device requirements. J. Neurosci. Methods 167, 91–104.
ADAPTIVE CHANGES OF RHYTHMIC EEG OSCILLATIONS IN SPACE
187
Varela, F., Lachaux, J. P., Rodriguez, E., and Martinerie, J. (2001). The brainweb: Phase synchronization and large-scale integration. Nat. Rev. Neurosci. 2, 229–239. Vidal, M., Lipshits, M., McIntyre, J., and Berthoz, A. (2003). Gravity and spatial orientation in virtual 3D-mazes. J. Vestib. Res. 13, 273–286. Von der Malsburg, C. (1985). Nervous structure with dynamical links. Ber. Busenges Phys. Chem. 89, 703–710. Wolpaw, J. R. (2007). Brain-computer interfaces as new brain output pathways. J. Physiol. 579, 613–619. Wolpaw, J. R., and McFarland, D. J. (2004). Control of a two-dimensional movement signal by a noninvasive brain-computer interface in humans. Proc. Natl. Acad. Sci. USA 101, 17849–17854. Yuval-Greenberg, S., Tomer, O., Keren, A. S., Nelken, I., and Deouell, L. Y. (2008). Transient induced gamma-band response in EEG as a manifestation of miniature saccades. Neuron 58, 429–441.
VALIDATION OF BRAIN–MACHINE INTERFACES DURING PARABOLIC FLIGHT
Jose´ del R. Milla`n,*,y Pierre W. Ferrez,* and Tobias Seidlz *Idiap Research Institute, Rue Marconi 19, 1920 Martigny, Switzerland Ecole Polytechnique Fe´de´rale de Lausanne (EPFL), Lausanne, Switzerland z Advanced Concepts Team, European Space Agency, Keplerlaan 1, 2201 AZ Noordwijk, The Netherlands y
I. Introduction II. Methods III. Experimental Results A. Classification of Mental Commands B. Recognition of Error-Related Potentials IV. Discussion References
Here we report on a validation study on brain–machine interfaces (BMIs) performed during the December 2007 ESA parabolic flight campaign. We investigated the feasibility of using BMIs for space applications by performing tests in microgravity. Brain signals were recorded with noninvasive electroencephalography before (calibration sessions) and during the parabolic flights on two subjects with prior BMI experience. The results of our experiments show that an experienced BMI user can achieve stable performance in all gravity conditions examined and, hence, demonstrate the feasibility of operating noninvasive BMIs in space.
I. Introduction
Triggered by the promising review of three Ariadna1 studies (Carpi and De Rossi, 2006; Milla´n et al., 2006; Tonet et al., 2006) initiated by ESA’s Advanced Concepts Team, we experimentally evaluated the functionality of BMIs in diVerent gravity conditions, including microgravity, onboard a parabolic flight 1 Ariadna is the name of a framework for cooperative research between the ESA Advanced Concepts Team and universities (http://www.esa.int/gsp/ACT/ariadna/index.htm).
INTERNATIONAL REVIEW OF NEUROBIOLOGY, VOL. 86 DOI: 10.1016/S0074-7742(09)86014-5
189
Copyright 2009, Elsevier Inc. All rights reserved. 0074-7742/09 $35.00
190
MILLA`N et al.
(47th ESA PFC campaign).2 Brain signals were recorded with noninvasive electroencephalogram (EEG), currently the most promising BMI for space applications (see other chapters for the nature of EEG and the possibilities that BMI open up to astronauts). In this chapter we report the performance of two healthy volunteer subjects with some previous BMI experience during various experimental conditions, including the calibration session run on ground prior to the parabolic flights that is used as a baseline to compare flight performance. The analysis focuses on two diVerent aspects of BMI, the mental commands sent by the user to drive the BMI and the error potentials (ErrP) generated by a feedback that does not match the subject’s intent. These ErrP can be used as a verification procedure: if an ErrP follows the feedback associated to the BMI response, the system can cancel the command and therefore filter errors made by the BMI. Ferrez and Milla´n (2008a,b) describe ErrP for BMI and demonstrate their benefits.
II. Methods
Figure 1 shows the task subjects have to perform. It consists of mentally moving a virtual blue balloon on a standard computer display from a start position at the top of a pyramid to pseudo-randomly selected targets either on the left or on the right bottom of the pyramid. Every 2 s, the balloon goes down one step, either to the left or to the right depending on the BMI’s interpretation of the user’s mental command. The BMI continuously analyzes the subject’s EEG signals to recognize his intent and makes a decision every 2 s. This classification process continues until the balloon reaches the bottom row. In parallel, after each single step of the balloon, the BMI analyzes a small time window to check the presence of an ErrP which would indicate an erroneous feedback (i.e., wrong response of the BMI). During the parabola of the flight, subjects reached two targets per gravity phase (intertrial interval of around 2 s, for a total of around 18 s). As shown in Fig. 2, a parabola consists of five phases of 20 s each: normal gravity (1g), hypergravity (1.8g), microgravity (0g), hypergravity (1.8g), and normal gravity (1g). Each subject executed 15 parabolas. In addition, subjects performed 10 calibration sessions on ground a few days before the flight, each consisting of 10 targets equally distributed. Data from the calibration sessions were used to build a classifier (see Section III for details). Then, during the parabolic flight, EEG preprocessing and classification was done online. However, the feedback delivered to the subjects was not the 2 ESA Parabolic flight campaign: http://www.spaceflight.esa.int/users/index.cfm?act=default. page&level=11&page=paraf
191
VALIDATION OF BMIs DURING PARABOLIC FLIGHT
Balloon Initial position Step 1 Step 2 Step 3 Final step Target
Target to avoid
FIG. 1. Experimental task. The balloon (blue) appears at the top of the pyramid. The goal is to bring it to the green target (left in this example) that is chosen randomly. The subject executes the corresponding mental task (imagination of a left arm movement) until the balloon reached the bottom of the pyramid. The balloon makes a step down every 2 s, either to the left or to the right.
actual response of the BMI. Instead, the balloon moved with a 30% error rate— that is, at each step there was a 0.3 probability that the balloon moved to the wrong direction, thus replicating the performance of the online BMI (see Section III. A). It is our experience that this approach facilitates initial user training (either in early stages or in complex novel conditions) and yields EEG data of higher quality, even if the subjects are aware of the nature of the feedback. The reason is that it helps users to maintain their concentration and avoid frustration or confusion because of a poor performance of the BMI, which in our case can be due to dramatic changes in the EEG induced by hyper- or microgravity (Pletser and Quadens, 2003). Ultimately, this approach eliminates a potential showstopper during the first assessment of BMI for space applications. In order to deliver mental commands, subjects were instructed to execute two mental tasks in a self-paced way—that is, at their own pace without needing any external stimulation. The two mental tasks were imagination of left hand movements, which is associated to the command ‘‘left,’’ and words association, for the command ‘‘right.’’ The words association task consists in searching for words starting with the same letter chosen randomly at the beginning of the trial. EEG signals were processed following the protocol described by Ferrez and Milla´n (2008a) and Milla´n et al. (2008). As a reminder, for recognition of mental tasks, we analyze EEG in the frequency domain and compute 112 EEG samples during the 8 s that lasts a trial; for ErrP detection, analysis is performed in the time domain and there are four EEG samples per trial, one per step of the balloon.
60
40
0
50
10
40
40
0
10 0
10
20
30
40
50
0
10
20 30 40 Frequency [Hz]
1g (ground)
20 30 40 Frequency [Hz]
50
10
20
30
40
30 20
60
10 0
2g 0
40
50
20
0
10
0g
10 0
50 0
30
50
20
60
50 0
10
20 30 40 Frequency [Hz]
50
2g
DP [%]
10 DP [%]
50
20
30
30 20
60
30
40
60
60
DP [%]
DP [%]
DP [%]
DP [%]
50
40 30 20 10 0
50
1g (flight)
0
10
20
30
40
50
1g (flight)
60 50 40 30
60
20
50
10
40
1g (ground)
0
30
60 50 40 30
60
60
60
50
50
40
40
30
30
20
20
10
10
10
0
0
0
1g (flight)
20
20
0g
10
50
10
40
0
0
2g
30
2g
20
1g (flight)
FIG. 2. Stability of features for subject 1 over the diVerent gravity conditions. Top: Degree of relevance of frequencies. Bottom: Degree of relevance of EEG electrodes. Data for subject 2 are similar and omitted due to space restrictions.
VALIDATION OF BMIs DURING PARABOLIC FLIGHT
193
We use machine learning techniques at two levels, namely feature selection and training the two classifiers embedded in the BMI. The approach aims at discovering subject-specific patterns embedded in the continuous EEG signal. At the first level, we select those features that are more relevant for recognizing either the mental tasks or ErrP. Thus, we select spatio-frequency features for mental tasks (relevant electrodes and frequency components) and relevant electrodes for ErrP. Ferrez (2007) and Milla´n et al. (2008) provide details of the diVerent feature selection methods we use. The vector of relevant features is extracted from each EEG sample and fed to a statistical Gaussian classifier. Its output is an estimation of the posterior class probability distribution for a single EEG sample; that is, the probability that the sample belongs to one of the two classes (left or right for mental commands, and error or correct for ErrP). In this statistical classifier, every Gaussian unit represents a prototype of one of the classes to be recognized, and we use several prototypes per class. During learning, the centers of the Gaussian units are pulled toward the samples of the class they represent and pushed away from the samples of the other class (see Milla´n et al., 2004). For the classification of mental commands, the BMI combines the outputs of the Gaussian classifier over 2 s; while for ErrP recognition, the BMI simply takes the output of the classifier to each single sample. No artifact rejection algorithm was applied and all EEG samples were kept for analysis. It is worth noting, however, that after a visual a posteriori check of the samples we found no evidence of eye/muscular artifacts that could have contaminated one condition diVerently from the other.
III. Experimental Results
For each of the two subjects, data from the calibration sessions performed on ground were split in two groups of five consecutive sessions. The first one, training set, was used to select the features and build a classifier. The performance of this classifier was tested on the second group, testing set, to have a baseline against which to compare the subjects’ performance during the parabolic flights. Regarding the data from the parabolic flight, we split it in three groups of five consecutive parabolas. Then, we built a classifier for each group and type of gravity condition, which was tested on the next group. Final performance for each gravity condition is the average for the three groups, which yields a more robust estimation of the BMI performance since we are always testing it on new data recorded on later parabolas than those used for building the classifiers. This procedure is the same for both aspects of the BMI, namely the mental commands and the ErrP. Relevant features, selected on the training set of the calibration sessions, are kept fixed for the parabolic flight sessions. For the recognition of mental
MILLA`N et al.
194
commands, the relevant features are electrodes {C1, C3, C5, CP3, CP5, C2, C4, C6, CP4, CP6} and frequencies {14, 16} Hz for subject 1, and electrodes {FC3, C1, C3, C4} and frequencies {12, 14} Hz for subject 2. These features are in agreement with previous studies where sensorimotor rhythms over the two hemispheres have allowed operating a BMI (Pfurtscheller and Neuper, 2001). Interestingly, the relevant features for ErrP detection are similar for both subjects, namely electrodes FCz and Cz, in accordance with our previous experiments. This is also in agreement with all neurophysiological evidence that ErrP has a centro-frontal focus along the midline (Falkenstein et al., 2000).
A. CLASSIFICATION OF MENTAL COMMANDS Although the overall task for the subjects was to reach the target at the bottom of the pyramid, here we analyze the classification accuracy at the level of each single EEG sample. This is a much harder task, but yields a better picture of the short-time performance and stability of subjects during parabolic flights. Task-level performance is, in general, better than single-sample performance (provided the latter is above chance level), as each step taken by the balloon is a combination of the outputs of the classifier to several consecutive samples. Also, achieving the task only requires getting closer to the target than to the opposite corner. Thus, correct performance at the task level can accommodate errors at the sample level. Performance is above chance level for all gravity conditions (or phases) for both subjects, with a global accuracy in between 72 and 79% (Table I). Despite the stress, noise, and novelty of parabolic flight, performance during the flight does not degrade much with respect to ground (our baseline) for subject 1 and is
TABLE I PERCENTAGES (MEAN AND STANDARD DEVIATION) OF CORRECTLY CLASSIFIED SINGLE SAMPLES FOR THE TWO EXPERIMENTAL SUBJECTS Left arm (%)
1g (ground) 1g (flight) 2g 0g 2g 1g (flight) Average
Words association (%)
Accuracy (%)
Subject 1
Subject 2
Subject 1
Subject 2
Subject 1
Subject 2
84.5 74.8 12.5 77.6 13.0 81.4 3.6 89.7 0.5 88.4 1.3 82.7 5.9
73.5 76.2 4.2 77.4 8.8 74.0 1.4 78.2 7.4 76.0 2.4 75.9 1.8
73.7 69.5 1.5 75.8 6.9 62.8 1.9 68.1 13.9 57.4 3.1 67.9 6.9
73.6 77.6 2.9 80.0 2.8 74.1 0.7 79.3 1.0 81.3 12.9 77.7 3.2
79.1 7.6 72.2 3.7 76.7 1.3 72.1 13.2 78.9 15.3 72.9 21.9 75.3 3.3
73.6 0.1 76.9 1.0 78.7 1.8 74.1 0.1 78.8 0.8 78.7 3.7 76.8 1.2
195
VALIDATION OF BMIs DURING PARABOLIC FLIGHT
even better for subject 2. While subject 2 achieves a well-balanced accuracy among both mental tasks, subject 1 has a bias toward ‘‘left.’’ The stability of the EEG patterns during the diVerent gravity conditions of the flight (and with respect to ground) is a key requirement for a successful and reliable BMI in space applications. To check it, we have run the feature selection algorithms to identify the relevant features characterizing each gravity condition. Remarkably, the relevant features, frequencies and electrodes, are very similar for all conditions (Fig. 2 for subject 1). Indeed, the most relevant frequencies are 14 and 16 Hz, whereas the most relevant electrodes are located around C3 and C4. Subject 2 also exhibits a high stability of relevant features for all conditions.
B. RECOGNITION OF ERROR-RELATED POTENTIALS ErrP are similar for both subjects and, on average, above 80% for both error and correct steps (Tables II and III for subjects 1 and 2, respectively). These recognition rates are similar to the performances of all subjects we have worked with until now (Ferrez, 2007). The benefit of integrating ErrP detection into a BMI becomes obvious since it always improves its bit-rate—that is, how many correct bits it can communicated per step—for any gravity condition (Tables II and III). On average, ErrP detection doubles the bit-rate of the BMI for both subjects (see Ferrez, 2007 for bit-rate computation of a BMI).
TABLE II PERCENTAGES (MEAN AND STANDARD DEVIATION) OF CORRECTLY CLASSIFIED ERROR SAMPLES AND CORRECT SAMPLES, GLOBAL ACCURACY OF THE BMI (FROM TABLE I), BIT-RATE OF THE BMI, AND INCREASE IN PERFORMANCE INTRODUCED BY ERRP DETECTION Bit-rate
1g (ground) 1g (flight) 2g 0g 2g 1g (flight) Average
Error (%)
Correct (%)
BMI (%)
No ErrP
ErrP
Increase (%)
82.8 85.0 7.1 90.6 13.3 90.6 2.4 70.3 9.4 77.5 3.5 82.8 7.9
78.9 91.7 7.1 77.1 3.0 82.7 1.7 89.6 9.5 83.4 7.4 83.9 5.8
79.1 72.2 76.7 72.1 78.9 72.9 75.3
0.260 0.147 0.217 0.146 0.256 0.157 0.193
0.459 0.475 0.477 0.466 0.456 0.374 0.445
76 223 120 219 78 138 130
Performances for subject 1 over all gravity conditions.
MILLA`N et al.
196
TABLE III PERCENTAGES (MEAN AND STANDARD DEVIATION) OF CORRECTLY CLASSIFIED ERROR SAMPLES AND CORRECT SAMPLES, GLOBAL ACCURACY OF THE BMI (FROM TABLE I), BIT-RATE OF THE BMI, AND INCREASE IN PERFORMANCE INTRODUCED BY ERRP DETECTION Bit-rate
1g (ground) 1g (flight) 2g 0g 2g 1g (flight) Average
Error (%)
Correct (%)
BMI (%)
no ErrP
ErrP
Increase (%)
84.6 85.0 7.1 89.9 1.4 86.4 6.4 81.2 1.7 80.8 11.5 84.7 3.4
84.4 87.1 1.3 88.4 1.8 77.6 2.4 68.8 1.1 86.6 5.3 82.2 7.6
73.6 76.9 78.7 74.1 78.8 78.7 76.8
0.167 0.220 0.253 0.175 0.255 0.253 0.218
0.441 0.505 0.578 0.416 0.372 0.496 0.467
163 129 128 138 46 96 113
Performances for subject 2 over all gravity conditions.
IV. Discussion
The results of the December 2007 ESA campaign show that it is possible for a subject with some prior BCI experience to achieve stable performances in normal gravity as well as in microgravity and hypergravity, and hence demonstrate the feasibility of operating noninvasive BMI in space. Both subjects show encouraging performance despite their little experience in microgravity. On average, both of them reached 75% of global accuracy for the recognition of two mental commands and more than 80% of correct classification for ErrP. Although the BMI performance does not achieve the results of experiments run on ground, they are still satisfactory considering the various sensorial stress experienced during parabolic flights. As previous BMI research shows, these performances can be improved with further training and experience of the subjects in the use of BMI during parabolic flights. These results, and hypothesis, need to be confirmed with further experiments in future parabolic flight campaigns that should involve more subjects suYciently trained previously on ground.
Acknowledgments
The experiments were performed onboard the A 300 Zero-G of Novespace during the 47th ESA parabolic flight campaign (December 2007, Bordeux). We are grateful to the organizers of this campaign for acceptance on the flight and great support throughout the entire campaign.
VALIDATION OF BMIs DURING PARABOLIC FLIGHT
197
References
Carpi, F., and De Rossi, D. (2006). Non invasive brain-machine interfaces. European Space Agency, Advanced Concepts Team, Ariadna Final Report (05-6402). Falkenstein, M., Hoormann, J., Christ, S., and Hohnsbein, J. (2000). ERP components on reaction errors and their functional significance: A tutorial. Biol. Psychol. 51, 87–107. Ferrez, P. W. (2007). Error-related EEG potentials in brain-computer interfaces. Ph.D. Thesis, E´cole Polytechnique Fe´de´rale de Lausanne. Ferrez, P. W., and Milla´n, J. d. R. (2008a). Error-related EEG potentials generated during simulated brain-computer interaction. IEEE Trans. Biomed. Eng. 55, 923–929. Ferrez, P. W., and Milla´n, J. d. R. (2008b). Simultaneous real-time detection of motor imagery and error-related potentials for improved BCI accuracy. In ‘‘4th International Brain-Computer Interface Workshop and Training Course,’’ Graz, Austria, September 18–21. Milla´n, J. d. R., Renkens, F., Mourin˜o, J., and Gerstner, W. (2004). Non-invasive brain-actuated control of a mobile robot by human EEG. IEEE Trans. Biomed. Eng. 51, 1026–1033. Milla´n, J. d. R., Ferrez, P. W., and Buttfield, A. (2006). Non invasive brain-machine interfaces European Space Agency, Advanced Concepts Team, Ariadna Final Report (05-6402). Milla´n, J. d. R., Ferrez, P. W., Gala´n, F., Lew, E., and Chavarriaga, R. (2008). Non-invasive brainmachine interaction. Intern. J. Pattern Recognit. Artif. Intell. 22, 959–972. Pfurtscheller, G., and Neuper, C. (2001). Motor imagery and direct brain-computer communication. Proc. IEEE 89, 1123–1134. Pletser, V., and Quadens, O. (2003). Degraded EEG response of the human brain in function of gravity levels by the method of chaotic attractor. Acta Astronaut. 52, 581–589. Tonet, O., Tecchio, F., Sepulveda, F., Citi, L., Rossini, P. M., Marinelli, M., Tombini, M., Laschi, C., and Dario, P. (2006). Non invasive brain-machine interfaces. European Space Agency, Advanced Concepts Team, Ariadna Final Report (05-6402).
MATCHING BRAIN–MACHINE INTERFACE PERFORMANCE TO SPACE APPLICATIONS
Luca Citi,* Oliver Tonet,y and Martina Marinelliz *School of Computer Science and Electronic Engineering, University of Essex, Wivenhoe Park, CO4 3SQ Colchester, UK y CRIM Lab, Scuola Superiore Sant’Anna, Pisa, Italy z Scuola Superiore Sant’Anna, piazza Martiri della Liberta` 33, 56127 Pisa, Italy
I. Introduction II. Methods: Performance Measures of HBSs III. Materials A. Brain–Machine Interfaces B. Robotics and Automation for Space Applications IV. Results: Matching Interfaces and Devices V. Possible Demonstrators VI. Conclusions References
A brain–machine interface (BMI) is a particular class of human–machine interface (HMI). BMIs have so far been studied mostly as a communication means for people who have little or no voluntary control of muscle activity. For able-bodied users, such as astronauts, a BMI would only be practical if conceived as an augmenting interface. A method is presented for pointing out eVective combinations of HMIs and applications of robotics and automation to space. Latency and throughput are selected as performance measures for a hybrid bionic system (HBS), that is, the combination of a user, a device, and a HMI. We classify and briefly describe HMIs and space applications and then compare the performance of classes of interfaces with the requirements of classes of applications, both in terms of latency and throughput. Regions of overlap correspond to eVective combinations. Devices requiring simpler control, such as a rover, a robotic camera, or environmental controls are suitable to be driven by means of BMI technology. Free flyers and other devices with six degrees of freedom can be controlled, but only at low-interactivity levels. More demanding applications require conventional interfaces, although they could be controlled by BMIs once the same levels of performance as currently recorded in animal experiments are attained. Robotic arms and manipulators could be the next frontier for noninvasive BMIs. Integrating smart controllers in HBSs could improve interactivity and boost the use of BMI technology in space applications. INTERNATIONAL REVIEW OF NEUROBIOLOGY, VOL. 86 DOI: 10.1016/S0074-7742(09)86015-7
199
Copyright 2009, Elsevier Inc. All rights reserved. 0074-7742/09 $35.00
200
CITI et al.
I. Introduction
Advances in technology allowed mankind to build machines which are used to interact with the environment in our stead, when direct action is not possible or not desirable. This interaction is mediated by a human–machine interface (HMI). From a control system viewpoint interacting with a HMI implies translating intention into motor commands, dispatching them toward the target muscles, and translating the results of the action, collected through the sensing system, into feedback for the central nervous system (CNS). A brain–machine interface (BMI) allows to break this loop by translating a person’s intentions directly into commands to a device. Some BMIs bypass the musculoskeletal system completely, allowing severely disabled people, who have no voluntary control of muscles, to communicate (Donoghue, 2002; Mussa-Ivaldi and Miller, 2003; Wolpaw et al., 2002). However, to date no technology can provide a viable feedback method by directly stimulating the CNS and therefore the usual approach is to use the natural senses, such as vision or touch, in order to dispatch relevant information to the brain. Information transfer rates of BMIs are low, if compared to conventional HMIs: even the most skilled BMI typewriters can write only few letters per minute. Nevertheless, able-bodied people can still benefit from BMIs, if they are designed as augmenting interfaces, that is, interfaces allowing them to perform actions in addition to what they already can do with their normal abilities. It is precisely in this scenario that BMIs can be gainfully applied for space applications: astronauts are able-bodied and specially trained people, it would therefore make little sense for them to avoid using conventional interfaces, such as keyboards and joysticks, in favor of BMIs, which currently require a high cognitive load, are aVected by artifact signals from other activities, and oVer a poor information transfer rate. Only if astronauts or technical people from Earth will be able to use BMIs together with conventional interfaces, or to achieve some goals for which conventional interfaces are not suitable, it will make sense to introduce BMIs into space applications. This is why we believe that, for space applications, augmenting interfaces will have a dominant role. In this chapter, we hypothesize that performance of HMIs can be roughly compared independently from task, method, and user. After describing HMIs and devices for space applications in terms of latency and throughput, which are used as performance measures, we match the requirements of devices with the performance of available interfaces to point out eVective combinations.
MATCHING BMI PERFORMANCE TO SPACE APPLICATIONS
201
II. Methods: Performance Measures of HBSs
The ensemble user-interface-device, comprising both artificial and biological components, is defined as hybrid bionic system (HBS). A number of parameters and of definitions of the same have been used to characterize performance of HBSs (Kronegg et al., 2005). In this chapter, we will adopt throughput and latency as performance measures. Throughput (also called bit rate, bandwidth, or information transfer rate) is the amount of data that is transferred over a period of time and is measured in bit per second. Latency is a time delay between the moment something is initiated, and the moment one of its eVects begins (onset latency) or reaches the azimuth/ nadir (peak latency). In the following, classes of interfaces and devices are characterized. For each class, a numeric range for throughput and latency is defined. Throughput of devices (TPd) was calculated as the product of the number of bits per unit command b (in bit/command) and the number of commands per second (commands/s) that have to be sent to the device to be able to control it interactively: TPd ¼ bn: The throughput of interfaces (TPi) has been calculated as the Shannon information rate in (Shannon, 1948). This definition of throughput is also popular in the literature on BCIs, having been first suggested by Wolpaw et al. (1998). In a number of BMI papers TPi is not reported; however, the number of symbols, the error probability and the transfer rate (symbols/s), is stated or can be inferred. In calculating TPi, a symmetric N-symbol channel with symbol rate R and error probability (1 P ) has been hypothesized: TPi ¼ R log2 N þ P log2 P þ ð1 PÞ log2 ð1 PÞ=ðN 1Þ The value of latency is usually reported or deducible from the description of the experimental protocol used to generate the physiological signal measured by the interface. The minimum value of latency is limited by physiological characteristics of the neural fibers and relays forming the control loop, by response times of the musculoskeletal and sensory systems, by how interactive the system is designed to be, and by how much feedback is needed to close the control loop. Latency is also bound by the time resolution of the technique used to measure the user’s intent or action. Throughput and latency were chosen as initial measures for determining whether a given interface and device are suitable to be integrated in a HBS. Among the numerous factors that can be pinpointed, they are probably the only ones easily quantifiable and comparable. Therefore, they seem a reasonable
202
CITI et al.
choice in order to perform a first selection allowing to individuate which combinations of interfaces and devices are in principle possible and which ones are surely not. Other important factors, albeit beyond the scope of this chapter, need further to be considered for the final design of the HBS, such as degree of invasiveness, user-friendliness, portability, setup time, need for training, cost/eVectiveness balance, robustness to noise, instantaneous, and cumulative cognitive load required, temporal stability, so on.
III. Materials
A. BRAIN–MACHINE INTERFACES The performance of BMIs presented in this chapter is based on data collected from a number of studies. These studies include all the papers considered in (Tonet et al., 2008) plus a number of additional and more recent articles.1 BMIs have been grouped according to their type, as shown in Fig. 1, first into cortical interfaces, which exploit information collected from the CNS, and noncortical interfaces, in which the information is measured at the peripheral level, and further as explained below. In cortical noninvasive interfaces (C-NIs), brain signals are recorded from the scalp and are attenuated by their transit through the extracerebral layers. This group comprises interfaces based on diVerent types of brain signals: event-related potentials (ERP) and event-related (de)synchronization (ERD/ERS) related to motor imagery, to diVerent mental states or to imagined sensory stimulation; P300-evoked potentials, generated by mental selection of items arranged in a sequence or into square matrices; slow cortical potentials (SCPs) and sensorimotor cortex rhythms, related to 1D and 2D movement tasks; steady-state visual-evoked potentials (SSVEPs), related to 1D movement tasks and nominal selection of targets. Cortical invasive interfaces (C-Is) are characterized by a higher sensitivity than noninvasive ones because they are able to detect directly the voluntary firing of individual neurons in the primary motor cortex. During experiments with 1 For the sake of brevity we refer to (Tonet et al., 2008) for the list of papers already considered in that work. In addition we also used data from recent papers by Acharya et al. (2008), Achtman et al. (2007), Bai et al. (2008), Bell et al. (2008), Brychta et al. (2007), Farina et al. (2007), HoVmann et al. (2008), Karim et al. (2006), Krepki et al. (2007), Momen et al. (2007), Mu¨ller-Putz and Pfurtscheller (2008), Mu¨ller-Putz et al. (2008), Nijboer et al. (2008), Pham et al. (2005), Shenoy et al. (2008), and Truccolo et al. (2008).
MATCHING BMI PERFORMANCE TO SPACE APPLICATIONS
203
Human-machine interfaces
Cortical
Invasive
Non-cortical
Non-invasive
Invasive
Non-invasive
Techniques:
Techniques:
Implantable electrodes:
EMG
Implantable electrodes (microwires, MEA, neurotrophic electrodes), ECoG
EEG, fMRI, MEG, NIRS, PET
Cuff, sieve, LIFE
Conventional:
Signals: Single/multi unit recordings, LFP
Signals: ERP (ERS/ERD), P300, Sensorimotor rhythms, SCP, SSVEP
• Switch Keyboard, touch screen, joystick, button, eye blinking system, fing system • Pointer Mouse, laser pointer, gaze tracker • Speech
FIG. 1. Classification of human–machine interfaces. Examples of signal acquisition techniques and of acquired signals are listed for each class.
primates, the signal recorded has been related to complex 3D movement tasks. So far, during experiment with human subjects, only signals related to 1D or 2D movement tasks and to nominal selection of up four mental states have been recorded. In noncortical invasive interfaces (NC-Is), signals are measured directly from the peripheral nervous system by means of implantable electrodes. Finally, noncortical noninvasive interfaces (NC-NIs) comprise conventional interfaces (e.g., switch-based interfaces, pointing devices, and speech recognition) and interfaces based on electromyographic (EMG) signals.
B. ROBOTICS AND AUTOMATION FOR SPACE APPLICATIONS To protect human beings from the hazard of the hostile environment outside the Earth atmosphere, in manned space flights, astronauts have been enclosed in vehicles (for intravehicular activities) or special suits (for extravehicular activities, EVAs) (Hirzinger et al., 2000). As a complement and alternative, robotics and automation (R&A) is now one of the most attractive areas in space technology, allowing to develop machines that are capable of surviving the rigors of the space environment, performing some activities like exploration and assembly, reducing
204
CITI et al.
EVAs and occasionally improving performance over humans performing the same tasks. They can be sent into situations that are so risky that humans would not be allowed to go (Wilcox et al., 2006). For the purpose of this study, robots for space applications have been grouped into the following categories. Rover robots are vehicles launched by a lander over a planet or a satellite for exploring them and for characterizing soils and rocks. Although the size of rovers can range from larger vehicles for EVA to smaller autonomous vehicles, they share three degrees of freedom (DOF), two translational, and one rotational. EVA rovers are interactive, whereas remote rovers, depending on the time delay, can be teleoperated or be embedded with sensors for autonomous movement control. Manipulator robots are teleoperated robot arms which are useful to deploy or retrieve payloads or satellites on a space craft or station, for assistance in EVA activities such as assembling, maintenance, and repair, and, provided with cameras, as inspection aids. Typical manipulator robots have a complexity comparable with the human arm, though their size can range up to tens of meters, and are teleoperated at an interactive rate by a human operator located on the same space craft or station. An ‘‘astronaut-equivalent’’ robot is designed specifically to work with and around humans. The robot’s considerable mechanical dexterity allows it to use tools and manipulate flexible materials much like a human astronaut would. Moreover, space suits often do not allow astronauts free dexterous movements, a limitation which could be overcome by using an astronaut-equivalent robot. The considerable complexity of these robots, which can have more than 50 DOFs, regarding hardware and control systems makes them suitable only for local teleoperation. To simplify the HMI, their parts (e.g., head, arm, hand, leg, or trunk) may be controlled individually. Free flyers or free floating robots are robots launched in space and able to move in six DOFs, three translational, and three rotational. Their usage scenarios are similar to that of space rovers, that is, inspection and characterization of the atmosphere of planets or satellites. Their higher complexity requires accordingly more complex commands. The base unit may be additionally provided with manipulators for performing dexterous operations. An additional application of R&A to space flight is environmental control, that is, the application of domotics to space, for monitoring of the environmental parameters inside a spaceship or space station. Three key issues have been considered to express the performance of devices for space applications in terms of throughput and latency: first, mobility, that is, moving quickly and accurately between two points without collisions and without risk to the robots, humans, and the work site; second, manipulation, that is, using dexterous robots to manipulate objects safely, accurately, and quickly, without accidentally contacting unintended objects or imparting excessive forces beyond
MATCHING BMI PERFORMANCE TO SPACE APPLICATIONS
205
those needed for the task; third, time delay, that is, allowing a human operator to eVectively command a robot to do useful work. The operator may control the robot from a ‘‘local’’ console, for example, an astronaut inside the pressurized cabin of a space craft, or from a ‘‘remote’’ console, for example, a human operator on Earth, with nonnegligible speed-of-light communication delay with the robot. The requirements, in terms of throughput and latency, of the above space applications have been estimated from data contained in the following studies: Kim et al. (1992), Miller and Machulis (2005), Pen˜ı´n et al. (2000), Sheridan (1993), and Wilcox et al. (2006). Concerning throughput, we computed the throughput as the product of the number of bits per unit command and the number of commands per second that have to be sent to the device to be able to control it directly, as in a master–slave system, and interactively. This is a conservative estimate, since shared control methods can reduce the need for bandwidth: this issue is discussed in Section V. Also, we do not consider here the bandwidth necessary for operator feedback, typically visual feedback, which, though being a considerable consumer of bandwidth, does not aVect the suitability of an interface for a given application. Similarly, the value of latency for space applications was considered to be the acceptable time interval from the user’s intention to the moment the command is received by the device, neglecting the return time needed for feedback. Therefore, only half of the round-trip time reported in the above experiments was considered. For space applications where no literature data were available, requirements have been estimated taking into account related applications, such as ultrasound-based deep ocean teleoperation (Sheridan, 1993) and rehabilitation (Tonet et al., 2006). By slowing down the speed of devices and implementing autonomous control schemes, there is theoretically no upper limit to latency. However, the reported values take into account the maximum time allowed for a typical task, for example, a payload positioning task should be completed in minutes, not hours.
IV. Results: Matching Interfaces and Devices
In this section the performance of the interfaces described in Section III.A is matched with the needs of the applications presented in Section III.B. Identifying the regions of overlap allows to define realistically which applications could in principle be driven by means of a given BMI and also which types of BMI could be suitable for a given application. As said, this matching represents a necessary, but not suYcient, condition and other factors must be considered in the final design of a HBS.
206
CITI et al. Environment control Rover Free−flyer Manipulator arm Astronaut equivalent hand Astronaut equivalent arm Astronaut equivalent leg Astronaut equivalent head Astronaut equivalent C−NI: ERD/ERS C−NI: P300 C−NI: SCP C−NI: SensMot C−NI: SSVEP C−I: Human C−I: Monkey NC−NI: Conventional NC−NI: EMG NC−I
Latency (s)
102
101
100
10−2
10−1
100 101 Throughput (bit/s)
102
103
FIG. 2. Graphical representation, in terms of latency and throughput, of the requirements of space applications (gray boxes) and of the performance of separate subclasses of human–machine interfaces (areas delimited by colored convex hulls).
Figure 2 shows the overlap of application needs (rectangles) and interface performance (convex hulls). Figure 3 is similar, but the diVerent HMIs are grouped according to invasiveness (invasive/noninvasive) and to the location of the hybrid link (cortical/noncortical). At a first glance, it can be pointed out that applications that require little throughput and tolerate higher latency could be driven by any of the interfaces considered. These applications comprise devices for environmental control, an astronaut-equivalent head, and rovers. In Section V, we will present three possible demonstrators of BMI-controlled space applications. Furthermore, to some extent, control of free flyers and of an astronaut-equivalent leg is also possible by means of several separate interfaces in all four groups, even though for some interfaces the overlap is limited to the lower part of the required throughput range. The requirements of more demanding devices, namely the manipulator arm and the astronaut-equivalent hand, are met only by conventional interfaces. Also an EMG-based interface could allow some form of control of an astronautequivalent hand, probably a smart underactuated one requiring less throughput
MATCHING BMI PERFORMANCE TO SPACE APPLICATIONS
207
Environment control Rover Free−flyer Manipulator arm Astronaut equivalent hand Astronaut equivalent arm Astronaut equivalent leg Astronaut equivalent head Astronaut equivalent C−NI C−I NC−NI NC−I
Latency (s)
101
100
10−2
10−1
100 101 Throughput (bit/s)
102
FIG. 3. Graphical representation, in terms of latency and throughput, of the requirements of space applications (gray boxes) and of the performance of the four main classes of human–machine interfaces (areas delimited by colored convex hulls).
than conventional robotic hands. The same could apply to invasive cortical interfaces, once the performance of human subjects reaches the one obtained by monkeys. In fact, performance measured in monkeys suggests that cortical invasive interfaces could be used successfully for controlling prosthetic hands with greater interactivity. However, with cortical invasive interfaces, humans have not reached the same performance as monkeys. In Hochberg et al. (2006), the quadriplegic human subject that received the 96-multielectrode array was able to control a computer cursor to interact with home appliances, operate the opening and closing of a prosthetic hand, and perform rudimentary actions with a multijointed robot arm. It is worth noting that he could perform these actions even while conversing, which suggests that invasive interfaces have greater capabilities of discriminating shared output, that is, simultaneous orders of diVerent content, than noninvasive ones. Complex compound devices, namely the astronaut-equivalent arm and the whole astronaut-equivalent robot, require performance that is currently not attained by any of the interfaces. While the latency requirement is well accomplished by a few interfaces—invasive ones, conventional ones, and EMG—the
208
CITI et al.
limiting factor is the throughput. In fact, the control of a robot with many independent DOF requires an overall throughput well above the capabilities of the state-of-the-art interfaces.
V. Possible Demonstrators
Based on the regions of overlap between the performance of interfaces and requirements of applications in Figs. 2 and 3, a few demonstrators can be envisaged. Three of them are briefly described and discussed to verify their feasibility beyond the mere numerical comparison of throughput and latency shown in Figs. 2 and 3. A first demonstrator migrates the concept of domotics to space applications. Several BMIs are suitable for operating environmental controls. This result is not surprising: indeed, the control panel of domotic applications is usually a simple interface composed of switches and sliders, controls that are easily implemented by means of a BMI (Cincotti et al., 2006; Gao et al., 2003). Nevertheless they should not be the first choice. It is obvious that mechanical buttons and sliders, or their equivalent on a graphical user interface, are the most intuitive way to toggle switches or set ranges. Nevertheless, EEG-based BMIs have suYcient throughput and acceptable latency to be used for demonstrating BMI-based environmental control. A second demonstrator is a hands-free control of two DOFs. Practical scenarios include steering of a camera (e.g., a rover-mounted camera, the astronautequivalent head) while the user’s hands are controlling robotic manipulators, by means of joysticks or exoskeletons, for ground exploration or spaceship maintenance. This application shares many aspects with interfaces allowing an impaired user to scroll the screen and reach icons and widgets on a computer desktop (Citi et al., 2008). A related application, namely 2-DOF cursor control and map navigation on a computer display by means of a dependent BMI that requires change of the gaze fixation point, has been recently investigated at NASA by Trejo et al. (2006). If, while using the BMI to control two DOFs, the user was able to use his hands to control additional interfaces, this would be an augmenting application, that is, an application that could not be performed in the same way by one person alone. However, further investigation is required to rule out that the use of the BMI concurrently with traditional HMIs is made impossible by an excessive cognitive load or by interferences between the mental activity related to the BMI and the one related to the task at hand. A third demonstrator is a direct porting of an existing rehabilitation device, namely a BMI-driven wheelchair, to a space application, by substituting the wheelchair with a space rover. BMIs may not be the best choice for driving a
MATCHING BMI PERFORMANCE TO SPACE APPLICATIONS
209
rover: conventional interfaces, such as a joystick, yield better results with almost no training and user fatigue at all. Nevertheless, brain activity recorded noninvasively is suYcient to control a robot moving on a surface, especially if the devices embodies some smart high-level controller (Gala´n et al., 2008; Tanaka et al., 2005). In this regard, concerning complete HBSs in which the interface part has lower performance than required by the application, it is possible to overcome limitations of the interface by improving the eVectiveness of the commands sent to the device, that is, by developing smart high-level controllers, which are able to perform parts of the tasks autonomously (Sheridan, 1993). HBSs with low-level controllers and no autonomous behavior will leave all decisions to the users and will require many simple commands to be driven interactively. The commands will be simple (few bit/command) but frequent (many commands/s). On the other hand, an embedded high-level controller with a high degree of autonomy will accept complex commands from the user and then act autonomously, typically in a closed feedback loop based on data read from internal sensors. Such a controller will require complex commands from the user (many bit/command) but less often (few commands/s). Controllers with a modular degree of autonomy allow the user to switch between lower and higher levels of control, ensuring that the user is always in control of the device, but freeing them from the burden of controlling it continuously. Modulating degrees of autonomy could also be a means to overcome gaps between interface performance and application needs, by developing more deeply integrated HBS.
VI. Conclusions
In this chapter, a method to match interfaces and devices to form HBSs has been presented and possible space applications have been pointed out. Though the main focus is on BMI applications, the method is applicable to all kinds of HMIs and can be used in general to determine, for a given application, what interface is best suited to control it. It can also be used conversely, to find the applications that are most suited for a newly developed interface. Throughput and latency were selected as measures, since they are defined on all kinds of devices and interfaces and can easily be computed or estimated. Besides them, other variables aVect performance of HBSs and need to be taken into account for the development of a complete system. Especially in the case of space applications, the diVerent performance of the human component of the HBS cannot be neglected. Results show that devices requiring simpler control are suitable to be driven by means of BMI technology. Devices with many DOF can be controlled at the cost of suboptimal interactivity. More demanding applications require conventional interfaces, though they could be controlled by BMIs once the
210
CITI et al.
same levels of performance as currently recorded in animal experiments are attained. Integrating smart controllers in HBSs could boost the use of BMI technology in space applications. In conclusion, it appears as the future of research in HBSs will have many facets: there is room for not only improvement in all their individual components (user, device, interface), but also developing more eYcient strategies to make those components interact (control).
References
Acharya, S., Tenore, F., Aggarwal, V., Etienne-Cummings, R., Schieber, M. H., and Thakor, N. V. (2008). Decoding individuated finger movements using volume-constrained neuronal ensembles in the m1 hand area. IEEE Trans. Neural Syst. Rehabil. Eng. 16, 15–23. Achtman, N., Afshar, A., Santhanam, G., Yu, B. M., Ryu, S. I., and Shenoy, K. V. (2007). Free-paced high-performance brain-computer interfaces. J. Neural Eng. 4, 336–347. Bai, O., Lin, P., Vorbach, S., Floeter, M. K., Hattori, N., and Hallett, M. (2008). A high performance sensorimotor beta rhythm-based brain-computer interface associated with human natural motor behavior. J. Neural Eng. 5, 24–35. Bell, C. J., Shenoy, P., Chalodhorn, R., and Rao, R. P. N. (2008). Control of a humanoid robot by a noninvasive brain-computer interface in humans. J. Neural Eng. 5, 214–220. Brychta, R. J., Shiavi, R., Robertson, D., and Diedrich, A. (2007). Spike detection in human muscle sympathetic nerve activity using the kurtosis of stationary wavelet transform coeYcients. J. Neurosci. Methods 160, 359–367. Cincotti, F., Aloise, F., Babiloni, F., Marciani, M., Morelli, D., Paolucci, S., Oriolo, G., Cherubini, A., Bruscino, S., Sciarra, F., Mangiola, F., Melpignano, A., et al. (2006). Brain-operated assistive devices: The ASPICE project. In ‘‘The First IEEE/RAS-EMBS International Conference Biomedical Robotics Biomechatronics,’’ pp. 817–822. Citi, L., Poli, R., Cinel, C., and Sepulveda, F. (2008). P300-based BCI mouse with geneticallyoptimized analogue control. IEEE Trans. Neural Syst. Rehabil. Eng. 16, 51–61. Donoghue, J. P. (2002). Connecting cortex to machines: Recent advances in brain interfaces. Nat. Neurosci. 5, 1085–1088. Farina, D., do Nascimento, O. F., Lucas, M.-F., and Doncarli, C. (2007). Optimization of wavelets for classification of movement-related cortical potentials generated by variation of force-related parameters. J. Neurosci. Methods 162, 357–363. Gala´n, F., Nuttin, M., Lew, E., Ferrez, P. W., Vanacker, G., Philips, J., and Milla´n, J. D. R. (2008). A brain-actuated wheelchair: Asynchronous and non-invasive brain-computer interfaces for continuous control of robots. Clin. Neurophysiol. 119, 2159–2169. Gao, X., Xu, D., Cheng, M., and Gao, S. (2003). A BCI-based environmental controller for the motion-disabled. IEEE Trans. Neural Syst. Rehabil. Eng. 11, 137–140. Hirzinger, G., Brunner, B., and Lampariello, R. (2000). Advances in orbital robotics. In ‘‘Proceedings of 2000 IEEE International Conference Robotics Automation.’’ Vol. 1, pp. 898–907. Hochberg, L. R., Serruya, M. D., Friehs, G. M., Mukand, J. A., Saleh, M., Caplan, A. H., Branner, A., Chen, D., Penn, R. D., and Donoghue, J. P. (2006). Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442, 164–171.
MATCHING BMI PERFORMANCE TO SPACE APPLICATIONS
211
HoVmann, U., Vesin, J. M., Ebrahimi, T., and Diserens, K. (2008). An eYcient P300-based braincomputer interface for disabled subjects. J. Neurosci. Methods 167, 115–125. Karim, A. A., Hinterberger, T., Richter, J., Mellinger, J., Neumann, N., Flor, H., Ku¨bler, A., and Birbaumer, N. (2006). Neural internet: Web surfing with brain potentials for the completely paralyzed. Neurorehabil. Neural Repair 20, 508–515. Kim, W. S., Hannaford, B., and Bejczy, A. K. (1992). Force reflection and shared compliance control in operating telemanipulators with time delay. IEEE Trans. Robot. Automat. 8, 176–185. Krepki, R., Curio, G., Blankertz, B., and Mu¨ller, K. R. (2007). Berlin brain-computer interface – The HCI communication channel for discovery. Int. J. Hum. Comput. Stud. 65, 460–477. Kronegg, J., Voloshynovskiy, S., and Pun, T. (2005). Analysis of bit-rate definitions for brain-computer interfaces. In ‘‘Proceedings of the 2005 International Conference on Human-Computer Interaction,’’ Las Vegas, Nevada, June 20–23, 2005, Vol. 1, pp. 40–46. Miller, D. P., and Machulis, K. (2005). Visual aids for lunar rover tele-operation. In ‘‘Proceedings of the 8th International Symposium Artificial Intelligence Robotics Automation Space – iSAIRAS’’. Momen, K., Krishnan, S., and Chau, T. (2007). Real-time classification of forearm electromyographic signals corresponding to user-selected intentional movements for multifunction prosthesis control. IEEE Trans. Neural Syst. Rehabil. Eng. 15, 535–542. Mu¨ller-Putz, G. R., and Pfurtscheller, G. (2008). Control of an electrical prosthesis with an SSVEPbased BCI. IEEE Trans. Biomed. Eng. 55, 361–364. Mu¨ller-Putz, G. R., Eder, E., Wriessnegger, S. C., and Pfurtscheller, G. (2008). Comparison of DFT and lock-in amplifier features and search for optimal electrode positions in SSVEP-based BCI. J. Neurosci. Methods 168, 174–181. Mussa-Ivaldi, F. A., and Miller, L. E. (2003). Brain-machine interfaces: Computational demands and clinical needs meet basic neuroscience. Trends Neurosci. 26, 329–334. Nijboer, F., Furdea, A., Gunst, I., Mellinger, J., McFarland, D. J., Birbaumer, N., and Ku¨bler, A. (2008). An auditory brain-computer interface (BCI). J. Neurosci. Methods 167, 43–50. Pen˜ı´n, L., Matsumoto, K., and Wakabayashi, S. (2000). Force reflection for time-delayed teleoperation of space robots. In ‘‘Proceedings of the 2000 IEEE International Conference Robotics Automation,’’ Vol. 4 pp. 3120–3125. Pham, M., Hinterberger, T., Neumann, N., Ku¨bler, A., Hofmayer, N., Grether, A., Wilhelm, B., Vatine, J. J., and Birbaumer, N. (2005). An auditory brain-computer interface based on the selfregulation of slow cortical potentials. Neurorehabil. Neural Repair 19, 206–218. Shannon, C. (1948). A mathematical theory of communication. Bell Syst. Tech. J. 27, 379–423, and see also 623–656. Shenoy, P., Miller, K. J., Crawford, B., and Rao, R. N. (2008). Online electromyographic control of a robotic prosthesis. IEEE Trans. Biomed. Eng. 55, 1128–1135. Sheridan, T. (1993). Space teleoperation through time delay: Review and prognosis. IEEE Trans. Robot Automat. 9, 592–606. Tanaka, K., Matsunaga, K., and Wang, H. (2005). Electroencephalogram-based control of an electric wheelchair. IEEE Trans Robot. 21, 762–766. Tonet, O., Tecchio, F., Sepulveda, F., Citi, L., Rossini, P. M., Marinelli, M., Tombini, M., Laschi, C., and Dario, P. (2006). Critical review and future perspectives of non-invasive brain-machine interfaces, Technical report 05/6402. European Space Agency, Noordwijk. Tonet, O., Marinelli, M., Citi, L., Rossini, P. M., Rossini, L., Megali, G., and Dario, P. (2008). Defining brain-machine interface applications by matching interface performance with device requirements. J. Neurosci. Methods 167, 91–104. Trejo, L. J., Rosipal, R., and Matthews, B. (2006). Brain-computer interfaces for 1-D and 2-D cursor control: Designs using volitional control of the EEG spectrum or steady-state visual evoked potentials. IEEE Trans. Neural Syst. Rehabil. Eng. 14, 225–229.
212
CITI et al.
Truccolo, W., Friehs, G. M., Donoghue, J. P., and Hochberg, L. R. (2008). Primary motor cortex tuning to intended movement kinematics in humans with tetraplegia. J. Neurosci. 8, 1163–1178. Wilcox, B., Ambrose, R., and Kumar, V. (2006). Space robotics. WTEC Panel Report on International Assessment of Research and Development in Robotics, Vol. 1, pp. 25–39. World Technology Evaluation Center, College Park. Wolpaw, J. R., Ramoser, H., McFarland, D. J., and Pfurtscheller, G. (1998). EEG-based communication: Improved accuracy by response verification. IEEE Trans. Rehabil. Eng. 6, 326–333. Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., and Vaughan, T. M. (2002). Braincomputer interfaces for communication and control. Clin. Neurophysiol. 113, 767–791.
BRAIN–MACHINE INTERFACES FOR SPACE APPLICATIONS—RESEARCH, TECHNOLOGICAL DEVELOPMENT, AND OPPORTUNITIES
Leopold Summerer,* Dario Izzo,* and Luca Rossini*,y *Advanced Concepts Team, European Space Agency, Keplerlaan 1, 2201 AZ Noordwijk, The Netherlands y Biomedical Robotics and Biomicrosystems Laboratory, Universita` Campus Biomedico di Roma, Via Alvaro del Portillo 21, 00128 Roma, Italy
I. II. III. IV. V.
Introduction An Outlook on BMI Research Trends Future Manned Space Programs—Planned or Envisioned Next Steps Toward BMIs for Space Applications Conclusion References
Recent advances in brain research and brain–machine interfaces suggest these devices could play a central role in future generation computer interfaces. Successes in the use of brain machine interfaces for patients aVected by motor paralysis, as well as first developments of games and gadgets based on this technology have matured the field and brought brain–machine interfaces to the brink of more general usability and eventually of opening new markets. In human space flight, astronauts are the most precious ‘‘payload’’ and astronaut time is extremely valuable. Astronauts operate under diYcult and unusual conditions since the absence of gravity renders some of the very simple tasks tedious and cumbersome. Therefore, computer interfaces are generally designed for safety and functionality. All improvements and technical aids to enhance their functionality and eYciency, while not compromising safety or overall mass requirements, are therefore of great interest. Brain machine interfaces show some interesting properties in this respect. It is however not obvious that devices developed for functioning on-ground can be used as hands-free interfaces for astronauts. This chapter intends to highlight the research directions of brain machine interfaces with the perceived highest potential impact on future space applications, and to present an overview of the long-term plans with respect to human space flight. We conclude by suggesting research and development steps considered necessary to include brain–machine interface technology in future architectures for human space flight.
INTERNATIONAL REVIEW OF NEUROBIOLOGY, VOL. 86 DOI: 10.1016/S0074-7742(09)86016-9
213
Copyright 2009, Elsevier Inc. All rights reserved. 0074-7742/09 $35.00
214
SUMMERER et al.
I. Introduction
Brain–machine interfaces (BMIs) theoretically allow restoring or augmenting human communication and control skills (Wolpaw et al., 2002) by directly interfacing the brain activity with the controlled devices. Some of the potential uses of BMIs for space applications (Citi et al., 2009; Menon et al., 2009) have already been demonstrated in ground experimental labs. The most advanced systems currently conceived are able, in ground experimentation, to serve as general purpose computer interfaces, replacing the use of mouse and keyboards in dedicated environments (Krusienski and Wolpaw, 2009), potentially oVering a solution to free the hands of astronauts, for example, during glove-box operations or maintenance tasks which require a continuous attention to check complex procedures typically displayed on (laptop) screens. Systems have been proposed for use with oV-the-shelve software, and in particular in connection to virtual navigation systems or 3D virtual object manipulation software (Scherer et al., 2009). BMI systems have been optimized to interface with specific tools in order to reach the best possible performance for controlling domotic environments (Babiloni et al., 2009). It may be argued that BMI devices considered in these works have been originally conceived and implemented to restore part of the lost abilities of people seriously aVected by diVerent kinds of motor paralysis (for clinical perspectives on BMIs see Birbaumer et al., 2009) and not in the framework of a general eVort to augment human interactions with its environment nor tailored to facilitate astronaut operations (Rossini et al., 2007). To understand the applicability of the current concepts, it is necessary to understand up to what extent one can make use of devices designed to be the main (if not only) source of communication or the only mean to control the external environment. When the brain–motor frequencies, the logic and the associative areas, and the attention markers are all included into multidegrees of freedom interface and constantly monitored, the user brain activity has to be focused on the operation of the interface. Any other thought, in fact, could produce noise or could be translated into undesired commands. The inevitable outcome is that using a BMI usually brings almost to zero any possible level of multitasking, intended as any other action diVerent from operating the BMI itself (Neumann et al., 2003). As a consequence, in those cases where some of the potential applications for BMIs in space have been demonstrated on ground, it is not obvious that the same BMI devices can be used as an hands-free interfaces for human space flight: the astronaut would in fact need to avoid any postural movement, even those necessary to keep a stable position in microgravity conditions, otherwise producing brain activity which would interfere with the signals used by the BMI. This introduces a need to rethink BMIs in order to optimize the multitasking capabilities of the subject and adapt therefore its functioning to the
BMIs FOR SPACE APPLICATIONS
215
requirements that are typical of healthy subjects and in particular of astronauts operating in a microgravity environment. This short communication contributes to the discussion on the use of BMIs for space applications by highlighting the research and development steps necessary to include this technology in future architectures for human space flight. The suggestions are rooted in the analysis of the state of the art of the current BMI research for Earth applications and on long-term plans of diVerent space agencies. We thus start, in Section II, by discussing the role of invasive and noninvasive systems in the most recent research and developments eVorts of BMI scientists, highlighting the research directions with the highest potential impact on future applications. We then present in Section III an overview of the long-term plans of diVerent agencies with respect to human space flight to conclude, in Section IV, by suggesting some development steps for BMI for human space flight.
II. An Outlook on BMI Research Trends
In an exploratory phase, the first generation of BMIs in use for human space flight is likely to be based entirely on noninvasive brain-imaging techniques. This is primarily because of the high risks still associated with neurosurgery, which make implementation on healthy subjects unacceptable, but also to the strong cultural feelings against what is sometimes referred to as ‘‘cyborgization,’’ literally the enhancement of a biological being with mechanical devices or capabilities. On the other hand, the results in the field of invasive BMIs have made possible substantial scientific advances, and invasive systems are expected to keep playing a crucial role in the future of BMIs. With current technology, only invasive interfaces, especially those using cortical electrodes, can interact with neurons as intimately as to be able to listen to the single spikes produced by the surrounding neurons, and potentially to answer with a similar electrical activity (Lebedev and Nicolelis, 2006). This currently unique spatial and temporal resolution therefore makes them the most suitable research tool to understand the full potentials of BMIs in general. The importance of cortical electrodes in BMI research leads scientists to address the many technical diYculties related with performing research on animal models (Fagg et al., 2007; Fetz, 2007; Nicolelis, 2003; Schwartz, 2007; Serruya et al., 2002; Taylor et al., 2002; Velliste et al., 2008). Typically such experimental work requires a prolonged training period in which the animal, often a monkey, learns to perform a control task with a normal interface (be it to correctly position a mouse pointer on a screen or to move a robotic hand in a three-dimensional environment using a joystick). Only after the monkey reaches good performance in these ‘‘simple’’ tasks, the implanted electrodes can start
216
SUMMERER et al.
recording and decoding the electrical activity produced by the brain during the control of the system and, eventually, the BMI can replace entirely the normal interface during the execution of the whole task. On the other hand, noninvasive systems have so far not been reported to allow for the execution of a complex task that requires comparable brain activity. These, together with the observation that task learning would be much faster in humans, are arguments that make many scientists optimistic on the possible performances achievable by invasive systems on humans. Along these lines, the American Food and Drug Administration (FDA) exceptionally authorized few human experiments with invasive cortical electrodes specifically designed for invasive BMIs on humans (Hochberg et al., 2006; Kennedy and Bakay, 1998; Truccolo et al., 2008). The selected subjects were aVected by complete paralysis and hence were able to communicate with the external environment only via BMIs. The results, contrary to the expectations of the involved scientific community, showed that this setup was not able to achieve better performances than noninvasive systems. The fact that human experiments with noninvasive and invasive BMIs provide similarly defective control suggests that the problem is independent from the recording system (Wolpaw, 2007). One explanation suggests that these results could be due, in part, to the training protocols adopted, and specifically to the feedback protocol. In general, while complete and coherent feedback helps speeding up and maintaining long lasting skill achievements, a feedback only based on visual signals makes it hard for the user to understand how to systematically gain good results and hence keeps the execution of any motor activity far from reaching its best performance (Gordon et al., 1995). Invasive BMIs’ monkey subjects are trained to control the system via classical ‘‘limb controlled’’ interfaces, that is, joysticks. During joystick control, the monkey receives tactile and proprioceptive information on the position of all its upper limb’s joints, as well as information regarding the force applied to any muscle, the mechanical resistance of the joystick and any other mechanical parameter that plays a role in the task’s outcome. This multimodal information is then proficiently integrated by and within the monkeys’ brain, and coupled with the associated task’s outcome. Thus, the neural networks devoted to the most eYcient approach are quickly selected and reinforced with respect to all the others, almost hard-wiring the neural pathway associated with the whole task execution. At the end of this training phase, when the cortical interface is finally inserted, it finds a stable and repeatable neural signal associated with task execution, easily recognizable and decipherable. From then on, the monkey has to rely only on its visual feedback, which proves to be suYcient for an already acquired and reinforced skill. On the other hand, the human subjects were already severely paralyzed and deprived of any tactile or proprioceptive sensory information about most of their body (Kennedy and Bakay, 1998; Hochberg et al., 2006). The training phase was
BMIs FOR SPACE APPLICATIONS
217
based on the imagination of limb movements, that is, the right hand closure or disclosure, and could not be associated with any other feedback than visual ones, that is, a virtual hand performing the same movement. The interface was intended to bypass the damage which prevented the patients to normally control their muscles, by connecting few tens of neurons of cortical areas not directly aVected by the pathology or the condition causing the paralysis with the external ‘‘eVectors,’’ that is, a mouse pointer, a prosthetic hand, or a speller. Without any tactile or proprioceptive feedback provided during neither the training nor the execution phases, the neural networks related with the task of movement imaging received very little reinforcements. This probably made the neural signal much less stable and repeatable, which in turn made the mathematical classifiers perform worse than expected. Since it is currently inconceivable to implant BMIs’ cortical electrodes in healthy subjects, and it is still very diYcult to obtain permissions for implanting them in severely paralyzed subjects (in Europe this has never been done as far as known to the authors), the only other possible approach to pursue invasive interfaces research on humans in the near future is to exploit other kind of electrodes which are, for clinical reasons, already implanted in patients’ brain. Even though those electrodes, like the electrocortical (ECoG) electrodes used in the therapy of the most severe cases of epilepsy (Kahane et al., 2004), lack the specificity and selectivity of BMI cortical electrodes, they can be tested with subjects still provided with the natural feedback pathways ( Jerbi et al., 2009), and the first results on BMI systems are indeed holding promising results. Possibly, the joint research on noninvasive techniques and ECoG invasive experiments tailored toward understanding the deep and specific events which underlay the neural activity related to BMI operations will lead to new design solutions able to overcome many of the limits of current BMIs. On the other hand, in the long term, should neurosurgery become less risky and complex, and some level of cyborgization accepted by society (e.g., robotic prosthesis neurally controlled) the expected superior performances of invasive BMIs could eventually justify their implementation on humans, firstly on severely disabled persons, and subsequently even on healthy persons. In such a scenario the performances of the resulting BMIs would be substantially enhanced, as well with their potential for use in space missions.
III. Future Manned Space Programs—Planned or Envisioned
This section intends to provide a rough overview of current space mission plans for which the use of BMI might be of potential interest. So far, with the exception of the US Apollo program, all human space exploration activities have
218
SUMMERER et al.
been in low Earth orbit, below 500 km altitude. The traditional space faring nations with human space exploration programs, the United States (US) and Russia, have been joined by China that has recently shown to be the third state with an independent capability to launch humans into low Earth orbit. A similar program has been announced by India so that by 2020, there could be four independent, possibly competing human space exploration programs. Japan and Europe have substantial space programs, but have been relying so far on international cooperation for their respective human exploration activities, especially via the international space station (ISS) program. Private enterprises are also taking the first steps into human space flight. Private suborbital space flight has been successfully demonstrated and several commercial companies based on business plans centered on what is generally called ‘‘space tourism’’ are advancing the field. One private company has already arranged for the ‘‘visit’’ of space tourists in the ISS and has announced an ambitious lunar mission (Space Adventures, 2009). Almost 40 years after the last humans have ventured beyond low Earth orbit and landed on the lunar surface in 1972, renewed interest in such mission has led to the announcement of human space exploration plans for returning to the Moon and eventually reaching Mars before the mid of this century (Bodeen, 2008; CNSA, 2006; NASA, 2008; People’s Daily Online, 2006). The main European objectives with respect to human space exploration, expressed essentially via strategic planning and missions of the European Space Agency (ESA), are currently to make full use of the ISS and to prepare for following exploration missions. These include the definition of new transportation system derived from the European Automatic Transfer Vehicle (ATV), which is currently providing cargo return capabilities to low Earth orbit, specifically to the ISS, but which might be modified and upgraded to provide an essential element for an independent European human launch capability. In parallel, ESA has been studying options and technologies for Moon exploration missions and is currently in the process of recruiting new astronauts. Furthermore, ESA has expressed at diVerent occasions the objective to become a leading participant in the robotic exploration of Mars, together with NASA on the basis of a long-term agreement, starting with preparatory robotic exploration missions as the enhanced ExoMars mission (to be launched in 2016 in order to search for past and present signs of life on Mars) and a Mars sample return mission roughly around 2020. Naturally, planning for space applications and missions beyond 2020 is of more speculative nature and given the limited budgetary implications of the preparation phase during early mission phases subject to substantial changes and reorientations. While priorities change, the general lines of reflection as outlined by Creola (1994) are probably still valid. Such a longer term vision was formulated for ESA and Europe by a long-term space policy committee in the late 1990s (Naja, 2000). A more formal interaction between the European Union and
BMIs FOR SPACE APPLICATIONS
219
ESA has led recently to a European space policy resolution adopted by the 5th European Space Council in September 2008 and outlining Europe’s strategic priorities for space (Council of the European Union, 2008). On a more general long-term outlook and following the reasoning of Naja et al., the combination of general research trends and educated expectations, three scenarios for space application can be described (Naja et al., 2008). While scientific space missions are expected to remain part of the core activities of governmental space agencies, the role of human space flight will depend on the general orientation of all space activities. These range from a more inward-looking ‘‘space for protection and security’’ focus to an optimistic, outward oriented ‘‘outreach and expansion’’ focus, which is by nature more favorable toward human space flight. Almost 50 years after the first human in space, the role of governmental human space exploration will also be influenced by the evolution and success of the private sector which has just entered this domain. In parallel, there are observable, stable research trends that favor the introduction of more human-centered research in space and thus at least implicitly human space flight. Governmental and private research funding ratios between physical and life sciences have gradually shifted over the past 20–30 years toward life science research and especially medical and biological research. Brain research in general is considered as one of the most vibrant and active research areas advances will aVect also space activities. Most of the current life sciences research related to human spaceflight is performed within the European Programme for Life and Physical Sciences and Applications (ELIPS) program of ESA (ELIPS, 2004), dedicated to life and physical research on the ISS and especially to take full advantage of the unique Columbus laboratory. The clear objectives in human space flight activities are to maximize the return on investment in the ISS, in particular its innovative scientific and technological/industrial utilization. For this purpose, the subject of human adaptation and performance is one of the core disciplines where research cornerstones have been identified to help reaching the defined goals. These include the area of integrative gravitational physiology, muscle and bone physiology, neuroscience, and nongravitational physiology of space flight. While the details of future human space exploration missions are still being drafted and agreed upon, the number of astronauts remains one of the key sizing factors for all nonterrestrial human habitat designs. It is therefore important to maximize the eYciency of human activities. As a consequence, all types of stateof-the-art technologies and medical research, from virtual reality to degrees of human augmentation are likely to be considered, while, on the other hand, assuring that any of such additions to enhance eYciency would not jeopardize the overall astronaut and mission safety. Given their nature and potential, BMIs constitute therefore an attractive candidate for such applications.
220
SUMMERER et al.
IV. Next Steps Toward BMIs for Space Applications
We have argued that noninvasive systems have already reached a maturity that makes their consideration to support astronaut activities a concrete possibility and that much research is ongoing that is expected to refine further the current performances of these devices. These foreseen developments, and also the current state of the art devices, are not directly applicable to human space exploration and exploitation activities and need a parallel research eVort driven by space applications. The main requirements of space operations are in fact not taken into account in the current international research and development mainly driven by funds coming from entities interested in the therapeutic possibilities of these devices, their use for defense-related purposes and on applications on ground in general. A BMI can be useful for astronauts only if able to operate continuously in a reduced gravity condition (Cheron et al., 2009) and if allowing for multiple tasks to be performed concurrently with the BMI protocol execution. A first eVort in this direction has been recently made by Milla´n et al.(2009) who monitored the brain activity of subjects in the process of operating a BMI in microgravity conditions during a parabolic flight. Furthermore, designing general purpose BMI systems is a much more diYcult task than focusing on BMIs specifically conceived for a restricted number of applications. Hence, space research on BMIs should be driven by a prior identification of the main applications which could be addressed by brain interfaces. Because of the singular, and diYcult to simulate, conditions in which astronauts have to operate, it is crucial to involve astronauts from the earliest discussion and development stages as to identify in detail the possible situations where astronauts may benefit from using a BMI. Possible BMI systems for such applications will have to be tested and refined in short-time zero gravity conditions experiments, like in parabolic flights campaigns, in order to refine and adapt the protocols to microgravity conditions. Some kind of benchmark tests (Tonet et al., 2008; Citi et al., 2009) should be implemented to assess the fitness level between systems and their target applications, and once the best BMI systems are identified, these would undergo longterm microgravity conditions tests, like on-board the ISS. A part from evaluating the eVective usefulness of such systems in the accomplishment of specific astronauts’ tasks, it is still very important to assess the eVects of microgravity conditions on the use of BMIs once long-term neural adaptations have happened in astronauts’ brains (Rossini, 2009). It is predictable that the eVects of brain adaptation will diVerently aVect the performance of diVerent signal-processing techniques, as well as of diVerent interface technologies, and it is crucial for the future of BMIs in space that those diVerences will emerge.
BMIs FOR SPACE APPLICATIONS
221
These steps are necessary to construct that critical set of data which will eventually allows for a complete matching between BMI systems performances and their related applications’ requirements, paving the road toward the very first complete design of BMI for space applications.
V. Conclusion
Developments in the technologies and knowledge required to design BMIs are happening at a great pace, mainly driven by terrestrial applications and in particular biomedical researches. Such achievements are not directly applicable to microgravity operations where the specific tasks and the unique environment are not considered by the mainstream research eVorts of BMI experts. As a consequence, a number of important research steps are needed to prove the potential of BMI for human space flight and have to be established in a parallel research eVort requiring the involvement of the final users (astronauts), BMI researchers and space experts.
References
Babiloni, F., Cincotti, F., Marciani, M., Salinari, S., Astolfi, L., Aloise, F., De Vico Fallani, F., and Mattia, D. (2009). On the use of brain computer interfaces outside scientific laboratories: Toward an application in domotic environments. In ‘‘Brain Machine Interfaces for space applications: Enhancing astronauts’ capabilities’’ (L. Rossini, D. Izzo, and L. Summerer, Eds.), pp. 133–146. International Review of Neurobiology. Birbaumer, N., Murguialday, A. R., Weber, C., and Montoya, P. (2009). Neurofeedback and braincomputer-interface (BCI): Clinical applications. In ‘‘Brain Machine Interfaces for space applications: Enhancing astronauts’ capabilities’’ (L. Rossini, D. Izzo, and L. Summerer, Eds.), pp. 107–187. International Review of Neurobiology. Bodeen, Ch. (2008). China space program reaches turning point with third manned mission featuring spacewalk. China Post, http://www.chinapost.com.tw/china/national%20news/2008/09/23/ 175925/China-space.htm. Accessed on March 8, 2009. Cheron, G., Cebolla, A. M., Petieau, M., Bengoetxea, A., Palmero-Soler, E., Leroy, A., and Dan, B. (2009). Adaptive changes of rhythmic EEG oscillations in space: Implications for brain machine interface applications. In ‘‘Brain Machine Interfaces for space applications: Enhancing astronauts’ capabilities’’ (L. Rossini, D. Izzo, and L. Summerer, Eds.), pp. 171–187. International Review of Neurobiology. Citi, L., Tonet, O., and Marinelli, M. (2009). Matching brain-machine interface performance to space applications. In ‘‘Brain Machine Interfaces for space applications: Enhancing astronauts’ capabilities’’ (L. Rossini, D. Izzo, and L. Summerer, Eds.), pp. 199–212. International Review of Neurobiology.
222
SUMMERER et al.
CNSA (2006). China National Space Administration, China’s Space Activities in 2006, http://www. cnsa.gov.cn/n615709/n620681/n771967/79970.html. Accessed on March 8, 2009. Council of the European Union (2008). 5th European Space Council, Council Resolution. Taking forward the European Space Policy, http://register.consilium.europa.eu/pdf/en/08/st13/ st13569.en08.pdf. Accessed on March 8, 2009. Creola, P. (1994). Has space a future? In ‘‘Address to the Annual Meeting of the Swiss Academy of Technical Sciences’’ Bern, September 22/23, 1994. ESA Bulletin Nr. 82. ELIPS (2004). European programme for Life and Physical sciences and Applications Utilising the International Space Station (ELIPS). European Space Agency, http://www.spaceflight.esa.int/ users/index.cfm?act=default.page&level=16&page=453. Accessed on March 8, 2009. Fagg, A. H., Hatsopoulos, N. G., de Lafuente, V., Moxon, K. A., Niemati, S., Rebesco, J. M., Romo, R., Solla, S. A., Reimer, J., Tkach, D., Pohlmeyer, E. A., and Miller, L. E. (2007). Biomimetic brain machine interface for the control of movement. J. Neurosci. 27(44), 11842–11846. Fetz, E. E. (2007). Volitional control of neural activity: Implications for brain-computer interfaces. J. Physiol. 579(3), 571–579. Gordon, J., Ghilardi, M. F., and Ghez, C. (1995). Impairments of reaching movements in patients without proprioception I. Spatial errors. J. Neurophysiol. 73(1), 347. Hochberg, L. R., Serruya, M. D., Friehs, G. M., Mukand, J. A., Saleh, M., Caplan, A. H., Branner, A., Chen, D., Penn, R. D., and Donoghue, J. P. (2006). Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442, 164–171. Jerbi, K., Freyermuth, S., Minotti, L., Kahane, P., Berthoz, A., and Lachaux, J. P. (2009). Watching brain TV and playing brain ball: Exploring novel BCI strategies using real-time analysis of human intracranial data. In ‘‘Brain Machine Interfaces for space applications: Enhancing astronauts’ capabilities’’ (L. Rossini, D. Izzo, and L. Summerer, Eds.), pp. 159–168. International Review of Neurobiology. Kahane, P., Minotti, L., HoVmann, D., Lachaux, J.-P., and Ryvlin, P. (2004). Invasive EEG in the definition of the seizure onset zone: Depth electrodes. In ‘‘Handbook of Clinical Neurophysiology. Pre-surgical assessment of the Epilepsies with Clinical Neurophysiology and Functional Neuroimaging’’ (F. Rosenow and H. O. Lu¨ders, Eds.), pp. 109–133. Elsevier Science, Amsterdam. Kennedy, P. R., and Bakay, R. A. (1998). Restoration of neural output from a paralysed patient by a direct brain connection. Neuroreport 9(8), 1707–1711. Krusienski, D. J., and Wolpaw, J. R. (2009). Brain-computer interface research at the Wadsworth center: Developments in noninvasive communication and control. In ‘‘Brain Machine Interfaces for space applications: Enhancing astronauts’ capabilities’’ (L. Rossini, D. Izzo, and L. Summerer, Eds.), pp. 147–157. International Review of Neurobiology. Lebedev, M. A., and Nicolelis, M. A. L. (2006). Brain-machine interfaces: Past, present and future. Trends Neurosci. 29(9), 537–546. Menon, C., de Negueruela, C., Millan, J. d. R., Tonet, O., Carpi, F., Broschart, M., Ferrez, P., Buttfield, A., Tecchio, F., Sepulveda, F., Citi, L., Laschi, C., et al. (2009). Prospects of brain machine interfaces for space system control. Acta Astronaut. 64(4), 448–456. Milla´n, J. R., Ferrez, P. W., and Seidl, T. (2009). Validation of brain-machine interfaces during parabolic flight. In ‘‘Brain Machine Interfaces for space applications: Enhancing astronauts’ capabilities’’ (L. Rossini, D. Izzo, and L. Summerer, Eds.), pp. 189–197. International Review of Neurobiology. Naja, G. (2000). The second report by ESA’s long-term space policy committee (LSPC). ESA Bull. 102. Naja, G., Summerer, L., and Johannsen, M. (2008). Space 2030—Research trends as input for longterm planning. In ‘‘59th International Astronautical Congress’’ Glasgow. NASA (2008). NASA 101—From Vision to Reality, NP-2008-02-498-HQ , http://www.nasa.gov/ externalflash/nasa101/index.html. Accessed on March 8, 2009.
BMIs FOR SPACE APPLICATIONS
223
Neumann, N., Kubler, A., Kaiser, J., Hinterberger, T., and Birbaumer, N. (2003). Conscious perception of brain states: Mental strategies for brain-computer communication. Neurophysiologia 41, 1028–1036. Nicolelis, M. A. L. (2003). Brain-machine interfaces to restore motor function and probe neural circuits. Nature 4, 417–422. People’s Daily Online (2006). Roundup: China to Develop Deep Space Exploration in Five Years, http:// english.peopledaily.com.cn/200607/20/eng20060720_284801.html. Accessed on March 8, 2009. Rossini, P. M. (2009). Implications of brain plasticity to brain machine interfaces operation: A potential paradox? In ‘‘Brain Machine Interfaces for space applications: Enhancing astronauts’ capabilities’’ (L. Rossini, D. Izzo, and L. Summerer, Eds.), pp. 81–90. International Review of Neurobiology. Rossini, L., Seidl, T., Izzo, D., and Summerer, L. (2007). Beyond astronaut’s capabilities: A critical review. ‘‘58th International Astronautical Congress’’ Hyderabad. Scherer, R., Mu¨ller-Putz, G. R., and Pfurtscheller, G. (2009). Flexibility and practicality: Graz braincomputer interface approach. In ‘‘Brain Machine Interfaces for space applications: Enhancing astronauts’ capabilities’’ (L. Rossini, D. Izzo, and L. Summerer, Eds.), pp. 119–131. International Review of Neurobiology. Schwartz, A. B. (2007). Useful signal from motor cortex, 2007. J. Physiol. 579(3), 581–601. Serruya, M. D., Hatsopoulos, N. G., Paminski, L., Fellows, M. R., and Donoghue, J. P. (2002). Instant neural control of a movement signal. Nature 416, 141–142. Space Adventures (2009). Home Page, Space Adventures Ltd., http://www.spaceadventures.com/ index.cfm. Accessed on March 8, 2009. Taylor, D. A., Hels Tillery, S. I., and Schwartz, A. B. (2002). Direct cortical control of 3D neuroprosthetic devices. Science 296, 1829–1832. Tonet, O., Marinelli, M., Citi, L., Rossini, P. M., Rossini, L., Megali, G., and Dario, P. (2008). Defining brain–machine interface applications by matching interface performance with device requirements. J. Neurosci. Methods 167, 91–104. Truccolo, W., Friehs, G. M., Donoghue, J. P., and Hochberg, L. R. (2008). Primary motor cortex tuning to intended movement kinematics in humans with tetraplegia. J. Neurosci. 28(5), 1163–1178. Velliste, M., Perel, S., Spalding, M. C., Whittford, A. S., and Schwartz, A. B. (2008). Cortical control of a prosthetic arm for self-feeding. Nature 453, 1098–1101. Wolpaw, J. (2007). Brain–computer interfaces as new brain output pathways. J. Phys. 579, 613–619. Wolpaw, J., Birbaumer, N., McFarland, D., Pfurtscheller, G., and Vaughan, T. (2002). Brain-computer interfaces for communication and control. J. Clin. Neurophysiol. 113, 767–791.
INDEX
A Action potential, 54 Adaptive plasticity, 82–83 Alpha and Mu oscillations in space, management, 173–174. See also Electroencephalography ALS. See Amyotrophic lateral sclerosis Amyotrophic lateral sclerosis, 110–111, 120, 154 Astrocytes, 52 Asynchronous interfaces, 102 Attention deficit disorder and hyperactivity (ADHD), 108–109 Attentive interfaces, 18 Automatic Transfer Vehicle (ATV), 218 Autoregressive (AR) spectrum, 151 B Barthel index score (BI), 135 BCI. See Brain-computer interface BCI2000 software system, 137, 141, 148 BCI system control of domestic environment, methodology, 135, 144–145 domotic system prototype features, 139 estimation of cortical activity, 139–140 experimental task, 136–137 experimental training, 137–139 high-resolution EEG methods, use, 134–135 off-line analysis, 141–142 online processing, 141 feature selection and combination, 141 normalization, 141 spatial filter, 141 spectral feature extraction, 141 patients’ preparation and training, 136 results obtained experimentation with healthy subjects, 142–143 experimentation with patients, 143–144 subjects, in BCI system training, 135 Betz cells, 58, 60
Blood oxygen level-dependent (BOLD) contrast, 75–76 regulation of, BOLD response, 109–110 Blurring effect, and EEG data, 134 Bradykinesia, 63 Brain ball, 163. See also Brain-computer interface Brain-computer interface, 4, 100, 120, 134. See also Graz BCI activity control, EEG role alpha and Mu oscillations in space, 173–174 brain oscillations in space, 174–176 EEG fluctuations, 172–173 gamma EEG oscillations, 179 influence of top-down dynamics on, 176–178 SEP in, 179–183 for disabled individuals, 147–149 current and future directions, 154–155 P300 matrix class of BCI systems, 152–154 sensorimotor rhythm-based BCI control, 149–152 ESA parabolic flight, validation, 189–190 methodology, 190–193 outcomes, 193–196 for human intracranial data analysis, 160 BCI control via higher cognitive processes, 163–164 brain ball, 163 brain TV, 161–162 decoding oculomotor intentions, 164–165 human intracerebral recordings, 161 subjects and experimental paradigms, 161 for motor restoration, in chronic stroke, 114 presentation to space applications, matching, 200 BMI, 202–203 demonstrators, 208–209 HBS, measures, 201–202
225
226 Brain-computer interface (cont.) matching interfaces and devices, 205–208 space applications, robotics and automation, 203–205 for space applications, 214–215 BMI for, 220–221 BMI research development, 215–217 future manned space programs, 217–219 Brain-machine interfaces (BMIs), 4, 94 brain monitoring system, 94–95 definition, 94 elements, 95 feature selection, 96–97 mental task selection, 96 output and feedback methods, 99 translation algorithm, 97–99 f MRI-BMI effects, 109–110 in locked-in syndrome, 110–112 in stroke and spinal cord injury, 112–114 types, 99–100 dependent vs. independent BMIs, 100–101 invasive vs. non invasive, 100 spontaneous vs. evoked vs. event related, 101–102 synchronous vs. asynchronous, 102 and user’s ability, 102–103 operant conditioning approach, 103 operational protocol, 103 Brain plasticity, 83 associative learning, 86 and BMI systems, 87–88 different plastic events, 87 habituation, 83–84 monitoring during BMI control, 88–89 plastic changes, effects, 87 sensitization, 83, 86, 88 Brain power modulations, visualization, 161–162. See also Brain-computer interface Brain-switch, 129 C Central nervous system (CNS), 200 cell types glia, 52 neurons, 52–53 Cerebral cortex, motor areas, 57–58 anatomy cingulate motor areas, 59–60
INDEX
premotor areas, 58–59 primary motor cortex, location, 58 supplementary motor area, 59 effect of lesions, 62–63 inputs to, 61 neurons activity, 61–62 output, 60–61 corticospinal and noncorticospinal pathways, 60–61 primary motor cortex, plasticity, 63–64 Cingulate motor areas (CMA), 59 C-Is. See Cortical invasive interfaces 3-class self-paced ERD–BCI, 124–125, 128 C-NIs. See Cortical noninvasive interfaces Computer cursor control, mu and beta oscillations in, 173. See also Brain-computer interface Computer interfaces, 18 Computer mouse, SMR control, 151. See also Brain-computer interface Corneal reflection techniques, 14, 16–18 Cortical invasive interfaces, 202–203 Cortical noninvasive interfaces, 202 Cortical pyramidal neurons, 69 Corticospinal tract, 60 Crosstalk phenomenon, 9 Cuff electrodes, 29, 31 Cybernetic hands, 11 Cyborg beetle microsystem, 43–44 D Degrees of freedom (DoF), 10, 204 in human hand, 11 in prosthetic hands, 11 DLPFC. See Dorsolateral prefrontal cortex Dorsolateral prefrontal cortex, 164 Duchenne muscular dystrophy, 135 E Echo-planar imaging (EPI) method, 75 ECoG. See Electrocorticographic EEG. See Electroencephalogram EEG and MEG signals, 100 brain rhythms, study, 73–74 EEG technique, 68 generation current density, in macrocolumn, 70 examples of, high-resolution techniques, 73–75
INDEX
Maxwell’s equations use, for differentiating techniques, 70–72 from postsynaptic ionic currents, of pyramidal cortical neurons, 69 MEG technique, 69 use, in BMI research, 94–95 EEG-based interfaces, 4 Electrocorticograms, 94 Electrocorticographic, 148, 160 Electroencephalogram, 190 Electroencephalography, 4, 68, 134 in BCI activity control, 172 alpha and Mu oscillations in space, 173–174 brain oscillations in space, 174–176 EEG fluctuations, 172–173 gamma EEG oscillations, 179 influence of top-down dynamics on, 176–178 SEP in, 179–183 and blurring effect, 134 cortical estimation methodology, 135, 144–145 inverse procedures, 134 limitations, 68 use in BMIs, 94–95, 100 Electromyography, 4, 5, 149 EMG signals, 34 Electroneurographic (ENG) signals, 32–33 Electrooculography, 14–15, 149 limitations, 17 nystagmography, 15 saccadic response, 15 ELIPS. See European Programme for Life and Physical Sciences and Applications EMG. See Electromyography EMG-based interfaces applications, examples EMG controlled cybernetic hands, 10–11 EMG controlled exoskeletons, 11–12 characteristic features, 9 fundamental aspects EMG recording, by skin electrodes, 5–6 EMG signal, as-detected and after elementary preprocessing, 8 EMG signals, bioelectric genesis, 5–6 frequency spectrum, of EMG signal, 5, 7 NAL, estimation, 7 neural network-based algorithms, use, 8–9
227
use of EMG signal, for control purposes, 7–8 issues and difficulties involved, 9–10 EOG. See Electrooculography ERD–BCI-training period, 124 ERP. See Event-related potentials Error potentials (ErrP), 190 ESA. See European Space Agency ESA parabolic flight campaign, validation of BMI, 189–190. See also Brain-computer interface methodology, 190–193 outcomes, 193–196 European Programme for Life and Physical Sciences and Applications, 219 European Space Agency, 218 European space policy resolution, 219 EVAs. See Extravehicular activities Event-related desynchronization (ERD), 120–121 Event-related (de)synchronization (ERD/ERS), 202 Event-related potentials, 102, 202 Evoked potentials (EPs), in brain, 101 Evolutionary plasticity, 82 Exoskeleton, application, 11 Extraneural electrode, 29 Extravehicular activities, 203 F Feature selection, in BMIs, 96–97 joint time–frequency ( JTF) domain, 97 FES systems, closed-loop control, 32 Flat interface nerve electrode (FINE), 29, 32 Food and Drug Administration (FDA), 216 Forearm, EMG signal from, 8 Fovea, 12 Free floating robots, role, 204 FreehandÒ system, 124 Functional electrical stimulations (FES), 4, 24 Functional magnetic resonance imaging (f MRI), 74–75, 100 applications, 77–78 BOLD signal, generation, 75–76 example of, presurgical f MRI mapping, 78 experimental designs block designs, 77 event-related design, 77
228
INDEX
Functional magnetic resonance imaging (f MRI) (cont.) f MRI-BCI system, 110 use in BMIs, 94 G GABA, 55 Gamma EEG oscillations, 179. See also Brain-computer interface Gaze, 12 direction, 12 eye structure and, 12–13 fixation point, 13 tracking techniques, 13–14 corneal reflection, 14 electrooculography, 14–16 search coil in scleral contact lens, 16 Gaze-tracking-based interfaces applications, examples, 17–18 attention tracking systems, 18 task selection systems, 18 vigilance monitoring systems, 18 characteristics and limitations, 16–17 fundamental aspects, 12–16 EOG signals, bioelectric genesis, 15 gaze information, 12–13 gaze tracking techniques, 13–16 Genetic plasticity, 82 Glia, 52 Glutamate, 55–56 GNU Public License (GPL), 121 Google Earth, 127 Graz BCI, 120–121, 127–129 applications haptic stimulation, 122–124 off-the-shelf software, operation, 127 operating (neuro)prosthetic hand, 122–123 operation of neuroprosthesis, in SCI patients, 123–124 virtual environments, navigation, 125 virtual keyboard spelling device, 124–126 BCI training, 121 software package, 121 H Habituation, in simple reflex in snail, 84 Haptic device, exoskeleton as, 11 HBS. See Hybrid bionic system
HMI. See Human–machine interface Human intracranial data real-time analysis, BCI, 160. See also Brain-computer interface BCI control via higher cognitive processes, 163–164 brain ball, 163 brain TV, 161–162 decoding oculomotor intentions, 164–165 human intracerebral recordings, 161 subjects and experimental paradigms, 161 Human–machine interface, 24, 200 classification, 203 Hybrid bionic system, 24–25, 201–202 Hybrid control architecture, model for, 45–46 I Interfaces levels of interfacing, 41 natural interfaces, 41–42 in vivo neuronal bidirectional interfaces, in insects, 42–44 International Space Station (ISS), 218 EEG recording, 174 Interneuron, 83 Intraneural electrode, 29–30, 33 Invasive technique, 100 Inverse procedures, 134 L Laplacian spatial filters, usage, 149. See also Brain-computer interface LFPs. See Local field potentials Local field potentials, 160 Locked-in syndrome (LIS), 110–112 Longitudinal intrafascicular electrodes (LIFEs), 29–30, 32, 35 thin-film LIFEs (tf LIFE), 30 Long-term potentiation (LTP) mechanism, 82, 86 M Machine learning techniques, usage, 193. See also Brain-computer interface Macrocolumns, 69 Magnetoencephalography (MEG) technique, 69 Manipulator robots, usage, 204 Man–machine interfaces, 4 types
INDEX
electromyography (EMG) recordings, based on (see EMG-based interfaces) gaze-tracking techniques, based on (see Gaze-tracking-based interfaces) Master/slave system, 11 Matlab/Simulink-based rapid prototyping, 121 MEAs. See Multielectrode arrays Mental task selection, in BMIs, 96 Mirror neurons, 62 Moon exploration missions, ESA, 218 Moth–robot, 43–44 Motor epilepsies, 57 Motor imagery, 121 Motor unit action potential (MUAP), 5 Motor unit action potential train (MUAPT), 5 MUA. See Multiunit activity Multichannel video-EEG system, for human intracerebral recordings, 161. See also Brain-computer interface Multielectrode arrays, 30, 33, 35 Multiunit activity, 160 Mu-rhythm. See Sensori-motor-rhythm (SMR) Mutual adaptation loop, in BMIs, 99 N NC-Is. See Noncortical invasive interfaces Near-infrared-spectroscopy, 94, 100, 109 Neural plasticity, 87 Neural prostheses, and BMI, 99 Neurofeedback, 108 Neuromodulatory transmitters, 57 Neuron-chip, implanted in Manduca sexta, 43–44 Neurons, 52 motor neurons, 52–53 sensory neurons, 52–53 structure of, 52–53 synapses transmission, 55–57 transmission of information, process, 53–55 Neurotransmitters, role, 55–56 NIRS. See Near-infrared-spectroscopy Noncortical invasive interfaces, 203 Normalized muscle activation level (NAL), 7 Nystagmography, 15–16 O Oculomotor signals, 97 Oligodendrocytes, 52 Optokinetic nystagmus, 16 Orthosis functions, BMI control, 113
229
P Peripheral nerve interfaces. See also Peripheral nerves applications, 33 cybernetic prostheses, 34–35 neuroprostheses for, CNS injured patients, 33–34 nerve electrodes, 28–29 extraneural electrodes, 28–29 intraneural electrodes, 29–30 regenerative electrodes, 30–31 peripheral nervous system (PNS), 24 organization and function, 25–28 recording and processing neural signals, 32–33 stimulation of PNS, 31–32 Peripheral nerves, 25 afferent sensory fibers, 25–26 efferent motor fibers, 26–27 endoneurium, 27 epineurium, 27 perineurium, 27 structure, 28 P300-evoked potential-based paradigms, role, 148–149. See also Brain-computer interface Plasticity, 82. See also Brain plasticity history of development, 82 P300 matrix class, of BCI systems, 152–154. See also Brain-computer interface Positron emission tomography (PET), 75, 94, 100 Powered exoskeleton, 11 Primary motor cortex, 58–59 Prosthetic devices, myoprocessors, 10–11 Prosthetic hands, 7, 11 P300 wave, 101 R Realistic head model, for estimation of cortical activity, 139–140 Real-time f MRI (rt-f MRI), 109 Regenerative electrodes, 30–31 Region of interest (ROI), 135, 140–142 Roachbot, 42 Robotic devices, use of EMG technology, 9 Robotic missions, 40, 45 Robotics and automation (R&A), 203 Rover robots, usage, 204
230
INDEX
S Saccades, 12 Schwann cells, 52 SCPs. See Slow cortical potentials Search coil-based technology, 16–17 SEEG. See Stereotactic-electroencephalography Sensitization, in simple reflex in snail, 85 Sensorimotor rhythm-based BCI control, 149–152. See also Brain-computer interface Sensorimotor rhythms, 148 Sensori-motor-rhythm (SMR), 108 Serotonine (5HT), 83 Shape memory alloy (SMA) electrode, 43 Signal-to-noise ratio (SNR), 162 Skin conductance responses (GSR), biofeedback, 108 Slow cortical potentials, 202 control, 108 SMA lesions in humans, 62–63 SMRs. See Sensorimotor rhythms Somatosensory-evoked potential (SEP) tool and BCI, 179–183 Space applications. See also Brain-computer interface BCI, 147–148 BMI performance, 200, 202–203 demonstrators, 208–209 HBS, measures, 201–202 matching interfaces and devices, 205–208 space applications, robotics and automation, 203–205 of Graz-BCI system, 128–129 insect/machine hybrid controllers for (see Hybrid control architecture, model for; Interfaces) research and technological development, 214–215 BMI for, 220–221 future manned space programs, 217–219 research development, 215–217 Space, brain oscillations, 174–176. See also Braincomputer interface Spinal cord injury (SCI) patients, 124 SSVEPs. See Steady-state visual-evoked potentials
Steady-state evoked potentials (SSEPs), 120–121 SSEP-based systems, 128 Steady-state visual-evoked potentials, 122, 202 Stepwise linear discriminant analysis, 152 Stereotactic-electroencephalography, 162 Stroke, and BMI training, 112–114 Superconducting quantum interference devices (SQUIDs), 69 Supplementary motor area (SMA), 59 Surface electrodes (skin electrodes), 5 Surface electromyography (SEMG), 5 and muscle activation, 9 SWLDA. See Stepwise linear discriminant analysis Synapses, 55–57 Synaptic plasticity, 82 Synchronous interfaces, 102 T Telemetric system, Periplaneta americana, 43 Telemonitoring, 121 Tetraplegic patient, and FES device, 113 Throughput of interfaces (TPi), 201 Total locked-in syndrome (TLIS), 111–112 U Utah Slanted Electrode Array (USEA), 30, 32 V Vestibular nystagmus, 16 Virtual environment, 125 Virtual Keyboard (VK) spelling device, 124–126 Visual evoked potential (VEP), 101 W Wadsworth Center, BCI research at, 147–149. See also Brain-computer interface current and future directions, 154–155 P300 matrix class of BCI systems, 152–154 sensorimotor rhythm-based BCI control, 149–152 Word generation task, in f MRI, 78
CONTENTS OF RECENT VOLUMES
Volume 37
Memory and Forgetting: Long-Term and Gradual Changes in Memory Storage Larry R. Squire
Section I: Selectionist Ideas and Neurobiology in
Implicit Knowledge: New Perspectives on Unconscious Processes Daniel L. Schacter
Population Thinking and Neuronal Selection: Metaphors or Concepts? Ernst Mayr
Section V: Psychophysics, Psychoanalysis, and Neuropsychology
Selectionist and Neuroscience Olaf Sporns
Instructionist
Ideas
Selection and the Origin of Information Manfred Eigen
Phantom Limbs, Neglect Syndromes, Repressed Memories, and Freudian Psychology V. S. Ramachandran
Section II: Populations
Neural Darwinism and a Conceptual Crisis in Psychoanalysis Arnold H. Modell
Development
and
Neuronal
Morphoregulatory Molecules and Selectional Dynamics during Development Kathryn L. Crossin
A New Vision of the Mind Oliver Sacks
Exploration and Selection in the Early Acquisition of Skill Esther Thelen and Daniela Corbetta
index
Population Activity in the Control of Movement Apostolos P. Georgopoulos Section III: Functional Integration in the Brain
Segregation
and
Reentry and the Problem of Cortical Integration Giulio Tononi Coherence as an Organizing Principle of Cortical Functions Wolf Singerl
Volume 38 Regulation of GABAA Receptor Function and Gene Expression in the Central Nervous System A. Leslie Morrow Genetics and the Organization of the Basal Ganglia Robert Hitzemann, Yeang Olan, Stephen Kanes, Katherine Dains, and Barbara Hitzemann
Section IV: Memory and Models
Structure and Pharmacology of Vertebrate GABAA Receptor Subtypes Paul J. Whiting, Ruth M. McKernan, and Keith A. Wafford
Selection versus Instruction: Use of Computer Models to Compare Brain Theories George N. Reeke, Jr.
Neurotransmitter Transporters: Biology, Function, and Regulation Beth Borowsky and Beth J. Hoffman
Temporal Mechanisms in Perception Ernst Po¨ppel
231
Molecular
232
CONTENTS OF RECENT VOLUMES
Presynaptic Excitability Meyer B. Jackson
Volume 40
Monoamine Neurotransmitters in Invertebrates and Vertebrates: An Examination of the Diverse Enzymatic Pathways Utilized to Synthesize and Inactivate Biogenic Amines B. D. Sloley and A. V. Juorio
Mechanisms of Nerve Cell Death: Apoptosis or Necrosis after Cerebral Ischemia R. M. E. Chalmers-Redman, A. D. Fraser, W. Y. H. Ju, J. Wadia, N. A. Tatton, and W. G. Tatton
Neurotransmitter Systems in Schizophrenia Gavin P. Reynolds
Changes in Ionic Fluxes during Cerebral Ischemia Tibor Kristian and Bo K. Siesjo
Physiology of Bergmann Glial Cells Thomas Mu¨ller and Helmut Kettenmann
Techniques for Examining Neuroprotective Drugs in Vitro A. Richard Green and Alan J. Cross
index
Volume 39
Techniques for Examining Neuroprotective Drugs in Vivo Mark P. Goldberg, Uta Strasser, and Laura L. Dugan
Modulation of Amino Acid-Gated Ion Channels by Protein Phosphorylation Stephen J. Moss and Trevor G. Smart
Calcium Antagonists: Their Role in Neuroprotection A. Jacqueline Hunter
Use-Dependent Regulation Receptors Eugene M. Barnes, Jr.
GABAA
Sodium and Potassium Channel Modulators: Their Role in Neuroprotection Tihomir P. Obrenovich
Synaptic Transmission and Modulation in the Neostriatum David M. Lovinger and Elizabeth Tyler
NMDA Antagonists: Their Role in Neuroprotection Danial L. Small
of
The Cytoskeleton and Neurotransmitter Receptors Valerie J. Whatley and R. Adron Harris
Development of the NMDA Ion-Channel Blocker, Aptiganel Hydrochloride, as a Neuroprotective Agent for Acute CNS Injury Robert N. McBurney
Endogenous Opioid Regulation of Hippocampal Function Michele L. Simmons and Charles Chavkin
The Pharmacology of AMPA Antagonists and Their Role in Neuroprotection Rammy Gill and David Lodge
Molecular Neurobiology of the Cannabinoid Receptor Mary E. Abood and Billy R. Martin
GABA and Neuroprotection Patrick D. Lyden
Genetic Models in the Study of Anesthetic Drug Action Victoria J. Simpson and Thomas E. Johnson Neurochemical Bases of Locomotion and Ethanol Stimulant Effects Tamara J. Phillips and Elaine H. Shen Effects of Ethanol on Ion Channels Fulton T. Crews, A. Leslie Morrow, Hugh Criswell, and George Breese index
Adenosine and Neuroprotection Bertil B. Fredholm Interleukins and Cerebral Ischemia Nancy J. Rothwell, Sarah A. Loddick, and Paul Stroemer Nitrone-Based Free Radical Traps as Neuroprotective Agents in Cerebral Ischemia and Other Pathologies Kenneth Hensley, John M. Carney, Charles A. Stewart, Tahera Tabatabaie, Quentin Pye, and Robert A. Floyd
CONTENTS OF RECENT VOLUMES
Neurotoxic and Neuroprotective Roles of Nitric Oxide in Cerebral Ischemia Turgay Dalkara and Michael A. Moskowitz
Sensory and Cognitive Functions Lawrence M. Parsons and Peter T. Fox
A Review of Earlier Clinical Studies on Neuroprotective Agents and Current Approaches Nils-Gunnar Wahlgren
Skill Learning Julien Doyon
index
Volume 41
Section V: Clinical and Neuropsychological Observations Executive Function and Motor Skill Learning Mark Hallett and Jordon Grafman
Section I: Historical Overview
Verbal Fluency and Agrammatism Marco Molinari, Maria G. Leggio, and Maria C. Silveri
Rediscovery of an Early Concept Jeremy D. Schmahmann
Classical Conditioning Diana S. Woodruff-Pak
Section II: Anatomic Substrates
Early Infantile Autism Margaret L. Bauman, Pauline A. Filipek, and Thomas L. Kemper
The Cerebrocerebellar System Jeremy D. Schmahmann and Deepak N. Pandya Cerebellar Output Channels Frank A. Middleton and Peter L. Strick Cerebellar-Hypothalamic Axis: Basic Circuits and Clinical Observations Duane E. Haines, Espen Dietrichs, Gregory A. Mihailoff, and E. Frank McDonald Section III. Physiological Observations Amelioration of Aggression: Response to Selective Cerebellar Lesions in the Rhesus Monkey Aaron J. Berman Autonomic and Vasomotor Regulation Donald J. Reis and Eugene V. Golanov Associative Learning Richard F. Thompson, Shaowen Bao, Lu Chen, Benjamin D. Cipriano, Jeffrey S. Grethe, Jeansok J. Kim, Judith K. Thompson, Jo Anne Tracy, Martha S. Weninger, and David J. Krupa
Olivopontocerebellar Atrophy and Friedreich’s Ataxia: Neuropsychological Consequences of Bilateral versus Unilateral Cerebellar Lesions The´re`se Botez-Marquard and Mihai I. Botez Posterior Fossa Syndrome Ian F. Pollack Cerebellar Cognitive Affective Syndrome Jeremy D. Schmahmann and Janet C. Sherman Inherited Cerebellar Diseases Claus W. Wallesch and Claudius Bartels Neuropsychological Abnormalities in Cerebellar Syndromes—Fact or Fiction? Irene Daum and Hermann Ackermann Section VI: Theoretical Considerations Cerebellar Microcomplexes Masao Ito
Visuospatial Abilities Robert Lalonde
Control of Sensory Data Acquisition James M. Bower
Spatial Event Processing Marco Molinari, Laura Petrosini, and Liliana G. Grammaldo
Neural Representations of Moving Systems Michael Paulin
Section IV: Functional Neuroimaging Studies Linguistic Processing Julie A. Fiez and Marcus E. Raichle
233
How Fibers Subserve Computing Capabilities: Similarities between Brains and Machines Henrietta C. Leiner and Alan L. Leiner
234
CONTENTS OF RECENT VOLUMES
Cerebellar Timing Systems Richard Ivry
Volume 43
Attention Coordination and Anticipatory Control Natacha A. Akshoomoff, Eric Courchesne, and Jeanne Townsend
Early Development of the Drosophila Neuromuscular Junction: A Model for Studying Neuronal Networks in Development Akira Chiba
Context-Response Linkage W. Thomas Thach
Development of Larval Body Wall Muscles Michael Bate, Matthias Landgraf, and Mar Ruiz Gmez Bate
Duality of Cerebellar Motor and Cognitive Functions James R. Bloedel and Vlastislav Bracha Section VII: Future Directions Therapeutic and Research Implications Jeremy D. Schmahmann
Volume 42 Alzheimer Disease Mark A. Smith Neurobiology of Stroke W. Dalton Dietrich Free Radicals, Calcium, and the Synaptic Plasticity-Cell Death Continuum: Emerging Roles of the Trascription Factor NFB Mark P. Mattson AP-I Transcription Factors: Short- and LongTerm Modulators of Gene Expression in the Brain Keith Pennypacker
Development of Electrical Properties and Synaptic Transmission at the Embryonic Neuromuscular Junction Kendal S. Broadie Ultrastructural Correlates of Neuromuscular Junction Development Mary B. Rheuben, Motojiro Yoshihara, and Yoshiaki Kidokoro Assembly and Maturation of the Drosophila Larval Neuromuscular Junction L. Sian Gramates and Vivian Budnik Second Messenger Systems Underlying Plasticity at the Neuromuscular Junction Frances Hannan and Yi Zhong Mechanisms of Neurotransmitter Release J. Troy Littleton, Leo Pallanck, and Barry Ganetzky Vesicle Recycling at the Drosophila Neuromuscular Junction Daniel T. Stimson and Mani Ramaswami Ionic Currents in Larval Muscles of Drosophila Satpal Singh and Chun-Fang Wu
Ion Channels in Epilepsy Istvan Mody
Development of the Adult Neuromuscular System Joyce J. Fernandes and Haig Keshishian
Posttranslational Regulation of Ionotropic Glutamate Receptors and Synaptic Plasticity Xiaoning Bi, Steve Standley, and Michel Baudry
Controlling the Motor Neuron James R. Trimarchi, Ping Jin, and Rodney K. Murphey
Heritable Mutations in the Glycine, GABAA, and Nicotinic Acetylcholine Receptors Provide New Insights into the Ligand-Gated Ion Channel Receptor Superfamily Behnaz Vafa and Peter R. Schofield
Volume 44
index
Human Ego-Motion Perception A. V. van den Berg Optic Flow and Eye Movements M. Lappe and K.-P. Hoffman
CONTENTS OF RECENT VOLUMES
The Role of MST Neurons during Ocular Tracking in 3D Space K. Kawano, U. Inoue, A. Takemura, Y. Kodaka, and F. A. Miles Visual Navigation in Flying Insects M. V. Srinivasan and S.-W. Zhang Neuronal Matched Filters for Optic Flow Processing in Flying Insects H. G. Krapp A Common Frame of Reference for the Analysis of Optic Flow and Vestibular Information B. J. Frost and D. R. W. Wylie Optic Flow and the Visual Guidance of Locomotion in the Cat H. Sherk and G. A. Fowler Stages of Self-Motion Processing in Primate Posterior Parietal Cortex F. Bremmer, J.-R. Duhamel, S. B. Hamed, and W. Graf Optic Flow Perception C. J. Duffy
Analysis
for
Self-Movement
Neural Mechanisms for Self-Motion Perception in Area MST R. A. Andersen, K. V. Shenoy, J. A. Crowell, and D. C. Bradley Computational Mechanisms for Optic Flow Analysis in Primate Cortex M. Lappe Human Cortical Areas Underlying the Perception of Optic Flow: Brain Imaging Studies M. W. Greenlee
235
Brain Development and Generation of Brain Pathologies Gregory L. Holmes and Bridget McCabe Maturation of Channels and Receptors: Consequences for Excitability David F. Owens and Arnold R. Kriegstein Neuronal Activity and the Establishment of Normal and Epileptic Circuits during Brain Development John W. Swann, Karen L. Smith, and Chong L. Lee The Effects of Seizures of the Hippocampus of the Immature Brain Ellen F. Sperber and Solomon L. Moshe Abnormal Development and Catastrophic Epilepsies: The Clinical Picture and Relation to Neuroimaging Harry T. Chugani and Diane C. Chugani Cortical Reorganization and Seizure Generation in Dysplastic Cortex G. Avanzini, R. Preafico, S. Franceschetti, G. Sancini, G. Battaglia, and V. Scaioli Rasmussen’s Syndrome with Particular Reference to Cerebral Plasticity: A Tribute to Frank Morrell Fredrick Andermann and Yuonne Hart Structural Reorganization of Hippocampal Networks Caused by Seizure Activity Daniel H. Lowenstein Epilepsy-Associated Plasticity in gammaAmniobutyric Acid Receptor Expression, Function and Inhibitory Synaptic Properties Douglas A. Coulter
What Neurological Patients Tell Us about the Use of Optic Flow L. M. Vaina and S. K. Rushton
Synaptic Plasticity and Secondary Epileptogenesis Timothy J. Teyler, Steven L. Morgan, Rebecca N. Russell, and Brian L. Woodside
index
Synaptic Plasticity in Epileptogenesis: Cellular Mechanisms Underlying Long-Lasting Synaptic Modifications that Require New Gene Expression Oswald Steward, Christopher S. Wallace, and Paul F. Worley
Volume 45 Mechanisms of Brain Plasticity: From Normal Brain Function to Pathology Philip. A. Schwartzkroin
Cellular Correlates of Behavior Emma R. Wood, Paul A. Dudchenko, and Howard Eichenbaum
236
CONTENTS OF RECENT VOLUMES
Mechanisms of Neuronal Conditioning David A. T. King, David J. Krupa, Michael R. Foy, and Richard F. Thompson
Biosynthesis of Neurosteroids and Regulation of Their Synthesis Synthia H. Mellon and Hubert Vaudry
Plasticity in the Aging Central Nervous System C. A. Barnes
Neurosteroid 7-Hydroxylation Products in the Brain Robert Morfin and Luboslav Sta´rka
Secondary Epileptogenesis, Kindling, and Intractable Epilepsy: A Reappraisal from the Perspective of Neuronal Plasticity Thomas P. Sutula Kindling and the Mirror Focus Dan C. McIntyre and Michael O. Poulter Partial Kindling and Behavioral Pathologies Robert E. Adamec The Mirror Focus and Secondary Epileptogenesis B. J. Wilder Hippocampal Lesions in Epilepsy: A Historical Review Robert Naquet Clinical Evidence for Secondary Epileptogensis Hans O. Luders Epilepsy as a Progressive (or Nonprogressive ‘‘Benign’’) Disorder John A. Wada Pathophysiological Aspects of Landau-Kleffner Syndrome: From the Active Epileptic Phase to Recovery Marie-Noelle Metz-Lutz, Pierre Maquet, Annd De Saint Martin, Gabrielle Rudolf, Norma Wioland, Edouard Hirsch, and Chriatian Marescaux
Neurosteroid Analysis Ahmed A. Alomary, Robert L. Fitzgerald, and Robert H. Purdy Role of the Peripheral-Type Benzodiazepine Receptor in Adrenal and Brain Steroidogenesis Rachel C. Brown and Vassilios Papadopoulos Formation and Effects of Neuroactive Steroids in the Central and Peripheral Nervous System Roberto Cosimo Melcangi, Valerio Magnaghi, Mariarita Galbiati, and Luciano Martini Neurosteroid Modulation of Recombinant and Synaptic GABAA Receptors Jeremy J. Lambert, Sarah C. Harney, Delia Belelli, and John A. Peters GABAA-Receptor Plasticity during LongTerm Exposure to and Withdrawal from Progesterone Giovanni Biggio, Paolo Follesa, Enrico Sanna, Robert H. Purdy, and Alessandra Concas Stress and Neuroactive Steroids Maria Luisa Barbaccia, Mariangela Serra, Robert H. Purdy, and Giovanni Biggio
Local Pathways of Seizure Propagation in Neocortex Barry W. Connors, David J. Pinto, and Albert E. Telefeian
Neurosteroids in Learning and Processes Monique Valle´e, Willy Mayo, George F. Koob, and Michel Le Moal
Multiple Subpial Assessment C. E. Polkey
Neurosteroids and Behavior Sharon R. Engel and Kathleen A. Grant
Transection:
A
Clinical
The Legacy of Frank Morrell Jerome Engel, Jr. Volume 46 Neurosteroids: Beginning of the Story Etienne E. Baulieu, P. Robel, and M. Schumacher
Memory
Ethanol and Neurosteroid Interactions in the Brain A. Leslie Morrow, Margaret J. VanDoren, Rebekah Fleming, and Shannon Penland Preclinical Development of Neurosteroids as Neuroprotective Agents for the Treatment of Neurodegenerative Diseases Paul A. Lapchak and Dalia M. Araujo
CONTENTS OF RECENT VOLUMES
Clinical Implications of Circulating Neurosteroids Andrea R. Genazzani, Patrizia Monteleone, Massimo Stomati, Francesca Bernardi, Luigi Cobellis, Elena Casarosa, Michele Luisi, Stefano Luisi, and Felice Petraglia Neuroactive Steroids and Central Nervous System Disorders Mingde Wang, Torbjo¨rn Ba¨ckstro¨m, Inger Sundstro¨m, Go¨ran Wahlstro¨m, Tommy Olsson, Di Zhu, Inga-Maj Johansson, Inger Bjo¨rn, and Marie Bixo Neuroactive Steroids in Neuropsychopharmacology Rainer Rupprecht and Florian Holsboer Current Perspectives on the Role of Neurosteroids in PMS and Depression Lisa D. Griffin, Susan C. Conrad, and Synthia H. Mellon index
237
Processing Human Brain Tissue for in Situ Hybridization with Radiolabelled Oligonucleotides Louise F. B. Nicholson In Situ Hybridization of Astrocytes and Neurons Cultured in Vitro L. A. Arizza-McNaughton, C. De Felipe, and S. P. Hunt In Situ Hybridization on Organotypic Slice Cultures A. Gerfin-Moser and H. Monyer Quantitative Analysis of in Situ Hybridization Histochemistry Andrew L. Gundlach and Ross D. O’Shea Part II: Nonradioactive in Situ hybridization Nonradioactive in Situ Hybridization Using Alkaline Phosphatase-Labelled Oligonucleotides S. J. Augood, E. M. McGowan, B. R. Finsen, B. Heppelmann, and P. C. Emson
Volume 47
Combining Nonradioactive in Situ Hybridization with Immunohistological and Anatomical Techniques Petra Wahle
Introduction: Studying Gene Expression in Neural Tissues by in Situ Hybridization W. Wisden and B. J. Morris
Nonradioactive in Situ Hybridization: Simplified Procedures for Use in Whole Mounts of Mouse and Chick Embryos Linda Ariza-McNaughton and Robb Krumlauf
Part I: In Situ Hybridization with Radiolabelled Oligonucleotides In Situ Hybridization with Oligonucleotide Probes Wl. Wisden and B. J. Morris
index
Cryostat Sectioning of Brains Victoria Revilla and Alison Jones
Volume 48
Processing Rodent Embryonic and Early Postnatal Tissue for in Situ Hybridization with Radiolabelled Oligonucleotides David J. Laurie, Petra C. U. Schrotz, Hannah Monyer, and Ulla Amtmann
Assembly and Intracellular GABAA Receptors Eugene Barnes
Trafficking
of
Processing of Retinal Tissue for in Situ Hybridization Frank Mu¨ller
Subcellular Localization and Regulation of GABAA Receptors and Associated Proteins Bernhard Lu¨scher and Jean-Marc Fritschy D1 Dopamine Receptors Richard Mailman
Processing the Spinal Cord for in Situ Hybridization with Radiolabelled Oligonucleotides A. Berthele and T. R. To¨lle
Molecular Modeling of Ligand-Gated Ion Channels: Progress and Challenges Ed Bertaccini and James R. Trudel
238
CONTENTS OF RECENT VOLUMES
Alzheimer’s Disease: Its Diagnosis and Pathogenesis Jillian J. Kril and Glenda M. Halliday DNA Arrays and Functional Genomics in Neurobiology Christelle Thibault, Long Wang, Li Zhang, and Michael F. Miles
The Treatment of Infantile Spasms: An Evidence-Based Approach Mark Mackay, Shelly Weiss, and O. Carter Snead III
index
ACTH Treatment of Infantile Spasms: Mechanisms of Its Effects in Modulation of Neuronal Excitability K. L. Brunson, S. Avishai-Eliner, and T. Z. Baram
Volume 49
Neurosteroids and Infantile Spasms: The Deoxycorticosterone Hypothesis Michael A. Rogawski and Doodipala S. Reddy
What Is West Syndrome? Olivier Dulac, Christine Soufflet, Catherine Chiron, and Anna Kaminski
Are there Specific Anatomical and/or Transmitter Systems (Cortical or Subcortical) That Should Be Targeted? Phillip C. Jobe
The Relationship between encephalopathy and Abnormal Neuronal Activity in the Developing Brain Frances E. Jensen
Medical versus Surgical Treatment: Which Treatment When W. Donald Shields
Hypotheses from Functional Neuroimaging Studies Csaba Juha´sz, Harry T. Chugani, Ouo Muzik, and Diane C. Chugani Infantile Spasms: Unique Sydrome or General Age-Dependent Manifestation of a Diffuse Encephalopathy? M. A. Koehn and M. Duchowny
Developmental Outcome with and without Successful Intervention Rochelle Caplan, Prabha Siddarth, Gary Mathern, Harry Vinters, Susan Curtiss, Jennifer Levitt, Robert Asarnow, and W. Donald Shields Infantile Spasms versus Myoclonus: Is There a Connection? Michael R. Pranzatelli
Histopathology of Brain Tissue from Patients with Infantile Spasms Harry V. Vinters
Tuberous Sclerosis as an Underlying Basis for Infantile Spasm Raymond S. Yeung
Generators of Ictal and Interictal Electroencephalograms Associated with Infantile Spasms: Intracellular Studies of Cortical and Thalamic Neurons M. Steriade and I. Timofeev
Brain Malformation, Epilepsy, and Infantile Spasms M. Elizabeth Ross
Cortical and Subcortical Generators of Normal and Abnormal Rhythmicity David A. McCormick Role of Subcortical Structures in the Pathogenesis of Infantile Spasms: What Are Possible Subcortical Mediators? F. A. Lado and S. L. Moshe´ What Must We Know to Develop Better Therapies? Jean Aicardi
Brain Maturational Aspects Relevant to Pathophysiology of Infantile Spasms G. Auanzini, F. Panzica, and S. Franceschetti Gene Expression Analysis as a Strategy to Understand the Molecular Pathogenesis of Infantile Spasms Peter B. Crino Infantile Spasms: Criteria for an Animal Model Carl E. Stafstrom and Gregory L. Holmes index
CONTENTS OF RECENT VOLUMES
Volume 50 Part I: Primary Mechanisms How Does Glucose Generate Oxidative Stress In Peripheral Nerve? Irina G. Obrosova Glycation in Diabetic Neuropathy: Characteristics, Consequences, Causes, and Therapeutic Options Paul J. Thornalley Part II: Secondary Changes
Nerve Growth Factor for the Treatment of Diabetic Neuropathy: What Went Wrong, What Went Right, and What Does the Future Hold? Stuart C. Apfel Angiotensin-Converting Enzyme Inhibitors: Are there Credible Mechanisms for Beneficial Effects in Diabetic Neuropathy? Rayaz A. Malik and David R. Tomlinson Clinical Trials for Drugs Against Diabetic Neuropathy: Can We Combine Scientific Needs With Clinical Practicalities? Dan Ziegler and Dieter Luft
Protein Kinase C Changes in Diabetes: Is the Concept Relevant to Neuropathy? Joseph Eichberg
index
Are Mitogen-Activated Protein Kinases Glucose Transducers for Diabetic Neuropathies? Tertia D. Purves and David R. Tomlinson
Volume 51
Neurofilaments in Diabetic Neuropathy Paul Fernyhough and Robert E. Schmidt Apoptosis in Diabetic Neuropathy Aviva Tolkovsky Nerve and Ganglion Blood Flow in Diabetes: An Appraisal Douglas W. Zochodne Part III: Manifestations Potential Mechanisms of Neuropathic Pain in Diabetes Nigel A. Calcutt Electrophysiologic Measures of Diabetic Neuropathy: Mechanism and Meaning Joseph C. Arezzo and Elena Zotova Neuropathology and Pathogenesis of Diabetic Autonomic Neuropathy Robert E. Schmidt Role of the Schwann Cell in Diabetic Neuropathy Luke Eckersley
239
Energy Metabolism in the Brain Leif Hertz and Gerald A. Dienel The Cerebral Glucose-Fatty Acid Cycle: Evolutionary Roots, Regulation, and (Patho) physiological Importance Kurt Heininger Expression, Regulation, and Functional Role of Glucose Transporters (GLUTs) in Brain Donard S. Dwyer, Susan J. Vannucci, and Ian A. Simpson Insulin-Like Growth Factor-1 Promotes Neuronal Glucose Utilization During Brain Development and Repair Processes Carolyn A. Bondy and Clara M. Cheng CNS Sensing and Regulation of Peripheral Glucose Levels Barry E. Levin, Ambrose A. Dunn-Meynell, and Vanessa H. Routh
Part IV: Potential Treatment
Glucose Transporter Protein Syndromes Darryl C. De Vivo, Dong Wang, Juan M. Pascual, and Yuan Yuan Ho
Polyol Pathway and Diabetic Peripheral Neuropathy Peter J. Oates
Glucose, Stress, and Hippocampal Neuronal Vulnerability Lawrence P. Reagan
240
CONTENTS OF RECENT VOLUMES
Glucose/Mitochondria in Neurological Conditions John P. Blass Energy Utilization in the Ischemic/Reperfused Brain John W. Phillis and Michael H. O’Regan
Stress and Secretory Immunity Jos A. Bosch, Christopher Ring, Eco J. C. de Geus, Enno C. I. Veerman, and Arie V. Nieuw Amerongen Cytokines and Depression Angela Clow
Diabetes Mellitus and the Central Nervous System Anthony L. McCall
Immunity and Schizophrenia: Autoimmunity, Cytokines, and Immune Responses Fiona Gaughran
Diabetes, the Brain, and Behavior: Is There a Biological Mechanism Underlying the Association between Diabetes and Depression? A. M. Jacobson, J. A. Samson, K. Weinger, and C. M. Ryan
Cerebral Lateralization and the Immune System Pierre J. Neveu
Schizophrenia and Diabetes David C. Henderson and Elissa R. Ettinger Psychoactive Drugs Affect Glucose Transport and the Regulation of Glucose Metabolism Donard S. Dwyer, Timothy D. Ardizzone, and Ronald J. Bradley index
Behavioral Conditioning of the Immune System Frank Hucklebridge Psychological and Neuroendocrine Correlates of Disease Progression Julie M. Turner-Cobb The Role of Psychological Intervention in Modulating Aspects of Immune Function in Relation to Health and Well-Being J. H. Gruzelier index
Volume 52 Volume 53 Neuroimmune Relationships in Perspective Frank Hucklebridge and Angela Clow Sympathetic Nervous System Interaction with the Immune System Virginia M. Sanders and Adam P. Kohm Mechanisms by Which Cytokines Signal the Brain Adrian J. Dunn Neuropeptides: Modulators of Responses in Health and Disease David S. Jessop
Immune
Brain–Immune Interactions in Sleep Lisa Marshall and Jan Born Neuroendocrinology of Autoimmunity Michael Harbuz Systemic Stress-Induced Th2 Shift and Its Clinical Implications Ibia J. Elenkov Neural Control of Salivary S-IgA Secretion Gordon B. Proctor and Guy H. Carpenter
Section I: Mitochondrial Structure and Function Mitochondrial DNA Structure and Function Carlos T. Moraes, Sarika Srivastava, Ilias Kirkinezos, Jose Oca-Cossio, Corina van Waveren, Markus Woischnick, and Francisca Diaz Oxidative Phosphorylation: Structure, Function, and Intermediary Metabolism Simon J. R. Heales, Matthew E. Gegg, and John B. Clark Import of Mitochondrial Proteins Matthias F. Bauer, Sabine Hofmann, and Walter Neupert Section II: Primary Respiratory Chain Disorders Mitochondrial Disorders of the Nervous System: Clinical, Biochemical, and Molecular Genetic Features Dominic Thyagarajan and Edward Byrne
CONTENTS OF RECENT VOLUMES
Section III: Secondary Respiratory Chain Disorders Friedreich’s Ataxia J. M. Cooper and J. L. Bradley Wilson Disease C. A. Davie and A. H. V. Schapira
241
The Mitochondrial Theory of Aging: Involvement of Mitochondrial DNA Damage and Repair Nadja C. de Souza-Pinto and Vilhelm A. Bohr index
Hereditary Spastic Paraplegia Christopher J. McDermott and Pamela J. Shaw Cytochrome c Oxidase Deficiency Giacomo P. Comi, Sandra Strazzer, Sara Galbiati, and Nereo Bresolin Section IV: Toxin Induced Mitochondrial Dysfunction Toxin-Induced Mitochondrial Dysfunction Susan E. Browne and M. Flint Beal Section V: Neurodegenerative Disorders Parkinson’s Disease L. V. P. Korlipara and A. H. V. Schapira Huntington’s Disease: The Mystery Unfolds? A˚sa Peterse´n and Patrik Brundin Mitochondria in Alzheimer’s Disease Russell H. Swerdlow and Stephen J. Kish Contributions of Mitochondrial Alterations, Resulting from Bad Genes and a Hostile Environment, to the Pathogenesis of Alzheimer’s Disease Mark P. Mattson Mitochondria and Amyotrophic Lateral Sclerosis Richard W. Orrell and Anthony H. V. Schapira
Volume 54 Unique General Anesthetic Binding Sites Within Distinct Conformational States of the Nicotinic Acetylcholine Receptor Hugo R. Ariaas, William, R. Kem, James R. Truddell, and Michael P. Blanton Signaling Molecules and Receptor Transduction Cascades That Regulate NMDA ReceptorMediated Synaptic Transmission Suhas. A. Kotecha and John F. MacDonald Behavioral Measures of Alcohol Self-Administration and Intake Control: Rodent Models Herman H. Samson and Cristine L. Czachowski Dopaminergic Mouse Mutants: Investigating the Roles of the Different Dopamine Receptor Subtypes and the Dopamine Transporter Shirlee Tan, Bettina Hermann, and Emiliana Borrelli Drosophila melanogaster, A Genetic Model System for Alcohol Research Douglas J. Guarnieri and Ulrike Heberlein index
Section VI: Models of Mitochondrial Disease Models of Mitochondrial Disease Danae Liolitsa and Michael G. Hanna
Volume 55
Section VII: Defects of Oxidation Including Carnitine Deficiency
Section I: Virsu Vectors For Use in the Nervous System
Defects of Oxidation Including Carnitine Deficiency K. Bartlett and M. Pourfarzam
Non-Neurotropic Adenovirus: a Vector for Gene Transfer to the Brain and Gene Therapy of Neurological Disorders P. R. Lowenstein, D. Suwelack, J. Hu, X. Yuan, M. Jimenez-Dalmaroni, S. Goverdhama, and M.G. Castro
Section VIII: Mitochondrial Involvement in Aging
242
CONTENTS OF RECENT VOLUMES
Adeno-Associated Virus Vectors E. Lehtonen and L. Tenenbaum Problems in the Use of Herpes Simplex Virus as a Vector L. T. Feldman Lentiviral Vectors J. Jakobsson, C. Ericson, N. Rosenquist, and C. Lundberg Retroviral Vectors for Gene Delivery to Neural Precursor Cells K. Kageyama, H. Hirata, and J. Hatakeyama
Processing and Representation of SpeciesSpecific Communication Calls in the Auditory System of Bats George D. Pollak, Achim Klug, and Eric E. Bauer Central Nervous System Control of Micturition Gert Holstege and Leonora J. Mouton The Structure and Physiology of the Rat Auditory System: An Overview Manuel Malmierca Neurobiology of Cat and Human Sexual Behavior Gert Holstege and J. R. Georgiadis
Section II: Gene Therapy with Virus Vectors for Specific Disease of the Nervous System
index
The Principles of Molecular Therapies for Glioblastoma G. Karpati and J. Nalbatonglu
Volume 57
Oncolytic Herpes Simplex Virus J. C. C. Hu and R. S. Coffin
Cumulative Subject Index of Volumes 1–25
Recombinant Retrovirus Vectors for Treatment of Brain Tumors N. G. Rainov and C. M. Kramm
Volume 58
Adeno-Associated Viral Vectors for Parkinson’s Disease I. Muramatsu, L. Wang, K. Ikeguchi, K-i Fujimoto, T. Okada, H. Mizukami, Y. Hanazono, A. Kume, I. Nakano, and K. Ozawa HSV Vectors for Parkinson’s Disease D. S. Latchman Gene Therapy for Stroke K. Abe and W. R. Zhang Gene Therapy for Mucopolysaccharidosis A. Bosch and J. M. Heard index
Volume 56 Behavioral Mechanisms and the Neurobiology of Conditioned Sexual Responding Mark Krause NMDA Receptors in Alcoholism Paula L. Hoffman
Cumulative Subject Index of Volumes 26–50
Volume 59 Loss of Spines and Neuropil Liesl B. Jones Schizophrenia as a Disorder of Neuroplasticity Robert E. McCullumsmith, Sarah M. Clinton, and James H. Meador-Woodruff The Synaptic Pathology of Schizophrenia: Is Aberrant Neurodevelopment and Plasticity to Blame? Sharon L. Eastwood Neurochemical Basis for an Epigenetic Vision of Synaptic Organization E. Costa, D. R. Grayson, M. Veldic, and A. Guidotti Muscarinic Receptors in Schizophrenia: Is There a Role for Synaptic Plasticity? Thomas J. Raedler
CONTENTS OF RECENT VOLUMES
243
Serotonin and Brain Development Monsheel S. K. Sodhi and Elaine Sanders-Bush
Volume 60
Presynaptic Proteins and Schizophrenia William G. Honer and Clint E. Young
Microarray Platforms: Introduction and Application to Neurobiology Stanislav L. Karsten, Lili C. Kudo, and Daniel H. Geschwind
Mitogen-Activated Protein Kinase Signaling Svetlana V. Kyosseva Postsynaptic Density Scaffolding Proteins at Excitatory Synapse and Disorders of Synaptic Plasticity: Implications for Human Behavior Pathologies Andrea de Bartolomeis and Germano Fiore Prostaglandin-Mediated Signaling in Schizophrenia S. Smesny Mitochondria, Synaptic Plasticity, and Schizophrenia Dorit Ben-Shachar and Daphna Laifenfeld Membrane Phospholipids and Cytokine Interaction in Schizophrenia Jeffrey K. Yao and Daniel P. van Kammen Neurotensin, Schizophrenia, and Antipsychotic Drug Action Becky Kinkead and Charles B. Nemeroff Schizophrenia, Vitamin D, and Brain Development Alan Mackay-Sim, Franc¸ois Fe´ron, Darryl Eyles, Thomas Burne, and John McGrath Possible Contributions of Myelin and Oligodendrocyte Dysfunction to Schizophrenia Daniel G. Stewart and Kenneth L. Davis Brain-Derived Neurotrophic Factor and the Plasticity of the Mesolimbic Dopamine Pathway Oliver Guillin, Nathalie Griffon, Jorge Diaz, Bernard Le Foll, Erwan Bezard, Christian Gross, Chris Lammers, Holger Stark, Patrick Carroll, Jean-Charles Schwartz, and Pierre Sokoloff S100B in Schizophrenic Psychosis Matthias Rothermundt, Gerald Ponath, and Volker Arolt Oct-6 Transcription Factor Maria Ilia NMDA Receptor Function, Neuroplasticity, and the Pathophysiology of Schizophrenia Joseph T. Coyle and Guochuan Tsai index
Experimental Design and Low-Level Analysis of Microarray Data B. M. Bolstad, F. Collin, K. M. Simpson, R. A. Irizarry, and T. P. Speed Brain Gene Expression: Genomics and Genetics Elissa J. Chesler and Robert W. Williams DNA Microarrays and Animal Models of Learning and Memory Sebastiano Cavallaro Microarray Analysis of Human Nervous System Gene Expression in Neurological Disease Steven A. Greenberg DNA Microarray Analysis of Postmortem Brain Tissue Ka´roly Mirnics, Pat Levitt, and David A. Lewis index Volume 61 Section I: High-Throughput Technologies Biomarker Discovery Using Molecular Profiling Approaches Stephen J. Walker and Arron Xu Proteomic Analysis of Mitochondrial Proteins Mary F. Lopez, Simon Melov, Felicity Johnson, Nicole Nagulko, Eva Golenko, Scott Kuzdzal, Suzanne Ackloo, and Alvydas Mikulskis Section II: Proteomic Applications NMDA Receptors, Neural Pathways, and Protein Interaction Databases Holger Husi Dopamine Transporter Network and Pathways Rajani Maiya and R. Dayne Mayfield Proteomic Approaches in Drug Discovery and Development Holly D. Soares, Stephen A. Williams,
244
CONTENTS OF RECENT VOLUMES
Peter J. Snyder, Feng Gao, Tom Stiger, Christian Rohlff, Athula Herath, Trey Sunderland, Karen Putnam, and W. Frost White Section III: Informatics Proteomic Informatics Steven Russell, William Old, Katheryn Resing, and Lawrence Hunter Section IV: Changes in the Proteome by Disease Proteomics Analysis in Alzheimer’s Disease: New Insights into Mechanisms of Neurodegeneration D. Allan Butterfield and Debra Boyd-Kimball Proteomics and Alcoholism Frank A. Witzmann and Wendy N. Strother Proteomics Studies of Traumatic Brain Injury Kevin K. W. Wang, Andrew Ottens, William Haskins, Ming Cheng Liu, Firas Kobeissy, Nancy Denslow, SuShing Chen, and Ronald L. Hayes Influence of Huntington’s Disease on the Human and Mouse Proteome Claus Zabel and Joachim Klose Section V: Overview of the Neuroproteome Proteomics—Application to the Brain Katrin Marcus, Oliver Schmidt, Heike Schaefer, Michael Hamacher, AndrA˚ van Hall, and Helmut E. Meyer index
Volume 62 GABAA Receptor Structure–Function Studies: A Reexamination in Light of New Acetylcholine Receptor Structures Myles H. Akabas Dopamine Mechanisms and Cocaine Reward Aiko Ikegami and Christine L. Duvauchelle Proteolytic Dysfunction in Neurodegenerative Disorders Kevin St. P. McNaught Neuroimaging Studies in Bipolar Children and Adolescents
Rene L. Olvera, David C. Glahn, Sheila C. Caetano, Steven R. Pliszka, and Jair C. Soares Chemosensory G-Protein-Coupled Receptor Signaling in the Brain Geoffrey E. Woodard Disturbances of Emotion Regulation after Focal Brain Lesions Antoine Bechara The Use of Caenorhabditis elegans in Molecular Neuropharmacology Jill C. Bettinger, Lucinda Carnell, Andrew G. Davies, and Steven L. McIntire index Volume 63 Mapping Neuroreceptors at work: On the Definition and Interpretation of Binding Potentials after 20 years of Progress Albert Gjedde, Dean F. Wong, Pedro Rosa-Neto, and Paul Cumming Mitochondrial Dysfunction in Bipolar Disorder: From 31P-Magnetic Resonance Spectroscopic Findings to Their Molecular Mechanisms Tadafumi Kato Large-Scale Microarray Studies of Gene Expression in Multiple Regions of the Brain in Schizophrenia and Alzeimer’s Disease Pavel L. Katsel, Kenneth L. Davis, and Vahram Haroutunian Regulation of Serotonin 2C Receptor PREmRNA Editing By Serotonin Claudia Schmauss The Dopamine Hypothesis of Drug Addiction: Hypodopaminergic State Miriam Melis, Saturnino Spiga, and Marco Diana Human and Animal Spongiform Encephalopathies are Autoimmune Diseases: A Novel Theory and Its supporting Evidence Bao Ting Zhu Adenosine and Brain Function Bertil B. Fredholm, Jiang-Fan Chen, Rodrigo A. Cunha, Per Svenningsson, and Jean-Marie Vaugeois index
CONTENTS OF RECENT VOLUMES
245
Volume 64
G-Protein–Coupled Receptor Deorphanizations Yumiko Saito and Olivier Civelli
Section I. The Cholinergic System John Smythies
Mechanistic Connections Between Glucose/ Lipid Disturbances and Weight Gain Induced by Antipsychotic Drugs Donard S. Dwyer, Dallas Donohoe, Xiao-Hong Lu, and Eric J. Aamodt
Section II. The Dopamine System John Symythies Section III. The Norepinephrine System John Smythies Section IV. The Adrenaline System John Smythies
Serotonin Firing Activity as a Marker for Mood Disorders: Lessons from Knockout Mice Gabriella Gobbi
Section V. Serotonin System John Smythies
index
index
Volume 66
Volume 65
Brain Atlases of Normal and Diseased Populations Arthur W. Toga and Paul M. Thompson
Insulin Resistance: Causes and Consequences Zachary T. Bloomgarden
Neuroimaging Databases as a Resource for Scientific Discovery John Darrell Van Horn, John Wolfe, Autumn Agnoli, Jeffrey Woodward, Michael Schmitt, James Dobson, Sarene Schumacher, and Bennet Vance
Antidepressant-Induced Manic Conversion: A Developmentally Informed Synthesis of the Literature Christine J. Lim, James F. Leckman, Christopher Young, and Andre´s Martin Sites of Alcohol and Volatile Anesthetic Action on Glycine Receptors Ingrid A. Lobo and R. Adron Harris Role of the Orbitofrontal Cortex in Reinforcement Processing and Inhibitory Control: Evidence from Functional Magnetic Resonance Imaging Studies in Healthy Human Subjects Rebecca Elliott and Bill Deakin
Modeling Brain Responses Karl J. Friston, William Penny, and Olivier David Voxel-Based Morphometric Analysis Using Shape Transformations Christos Davatzikos The Cutting Edge of f MRI and High-Field f MRI Dae-Shik Kim Quantification of White Matter Using DiffusionTensor Imaging Hae-Jeong Park
Common Substrates of Dysphoria in Stimulant Drug Abuse and Primary Depression: Therapeutic Targets Kate Baicy, Carrie E. Bearden, John Monterosso, Arthur L. Brody, Andrew J. Isaacson, and Edythe D. London
Perfusion f MRI for Functional Neuroimaging Geoffrey K. Aguirre, John A. Detre, and Jiongjiong Wang
The Role of cAMP Response Element–Binding Proteins in Mediating Stress-Induced Vulnerability to Drug Abuse Arati Sadalge Kreibich and Julie A. Blendy
Neural Modeling and Functional Brain Imaging: The Interplay Between the Data-Fitting and Simulation Approaches Barry Horwitz and Michael F. Glabus
Functional Near-Infrared Spectroscopy: Potential and Limitations in Neuroimaging Studies Yoko Hoshi
246
CONTENTS OF RECENT VOLUMES
Combined EEG and fMRI Studies of Human Brain Function V. Menon and S. Crottaz-Herbette
W. Gordon Frankle, Mark Slifstein, Peter S. Talbot, and Marc Laruelle index
index
Volume 68 Volume 67 Distinguishing Neural Substrates of Heterogeneity Among Anxiety Disorders Jack B. Nitschke and Wendy Heller Neuroimaging in Dementia K. P. Ebmeier, C. Donaghey, and N. J. Dougall Prefrontal and Anterior Cingulate Contributions to Volition in Depression Jack B. Nitschke and Kristen L. Mackiewicz Functional Imaging Research in Schizophrenia H. Tost, G. Ende, M. Ruf, F. A. Henn, and A. Meyer-Lindenberg Neuroimaging in Functional Somatic Syndromes Patrick B. Wood Neuroimaging in Multiple Sclerosis Alireza Minagar, Eduardo Gonzalez-Toledo, James Pinkston, and Stephen L. Jaffe Stroke Roger E. Kelley and Eduardo Gonzalez-Toledo Functional MRI in Pediatric Neurobehavioral Disorders Michael Seyffert and F. Xavier Castellanos Structural MRI and Brain Development Paul M. Thompson, Elizabeth R. Sowell, Nitin Gogtay, Jay N. Giedd, Christine N. Vidal, Kiralee M. Hayashi, Alex Leow, Rob Nicolson, Judith L. Rapoport, and Arthur W. Toga Neuroimaging and Human Genetics Georg Winterer, Ahmad R. Hariri, David Goldman, and Daniel R. Weinberger Neuroreceptor Imaging in Psychiatry: Theory and Applications
Fetal Magnetoencephalography: Viewing the Developing Brain In Utero Hubert Preissl, Curtis L. Lowery, and Hari Eswaran Magnetoencephalography in Studies of Infants and Children Minna Huotilainen Let’s Talk Together: Memory Traces Revealed by Cooperative Activation in the Cerebral Cortex Jochen Kaiser, Susanne Leiberg, and Werner Lutzenberger Human Communication Investigated With Magnetoencephalography: Speech, Music, and Gestures Thomas R. Kno¨sche, Burkhard Maess, Akinori Nakamura, and Angela D. Friederici Combining Magnetoencephalography and Functional Magnetic Resonance Imaging Klaus Mathiak and Andreas J. Fallgatter Beamformer Analysis of MEG Data Arjan Hillebrand and Gareth R. Barnes Functional Connectivity Analysis in Magnetoencephalography Alfons Schnitzler and Joachim Gross Human Visual Processing as Revealed by Magnetoencephalographys Yoshiki Kaneoke, Shoko Watanabe, and Ryusuke Kakigi A Review of Clinical Applications of Magnetoencephalography Andrew C. Papanicolaou, Eduardo M. Castillo, Rebecca Billingsley-Marshall, Ekaterina Pataraia, and Panagiotis G. Simos index
CONTENTS OF RECENT VOLUMES
247
Volume 69
Spectral Processing in the Auditory Cortex Mitchell L. Sutter
Nematode Neurons: Anatomy and Anatomical Methods in Caenorhabditis elegans David H. Hall, Robyn Lints, and Zeynep Altun
Processing of Dynamic Spectral Properties of Sounds Adrian Rees and Manuel S. Malmierca
Investigations of Learning and Memory in Caenorhabditis elegans Andrew C. Giles, Jacqueline K. Rose, and Catharine H. Rankin
Representations of Spectral Coding in the Human Brain Deborah A. Hall, PhD
Neural Specification and Differentiation Eric Aamodt and Stephanie Aamodt Sexual Behavior of the Caenorhabditis elegans Male Scott W. Emmons The Motor Circuit Stephen E. Von Stetina, Millet Treinin, and David M. Miller III Mechanosensation in Caenorhabditis elegans Robert O’Hagan and Martin Chalfie
Volume 70 Spectral Processing by the Peripheral Auditory System Facts and Models Enrique A. Lopez-Poveda Basic Psychophysics of Human Spectral Processing Brian C. J. Moore Across-Channel Spectral Processing John H. Grose, Joseph W. Hall III, and Emily Buss Speech and Music Have Different Requirements for Spectral Resolution Robert V. Shannon Non-Linearities and the Representation of Auditory Spectra Eric D. Young, Jane J. Yu, and Lina A. J. Reiss Spectral Processing in the Inferior Colliculus Kevin A. Davis Neural Mechanisms for Spectral Analysis in the Auditory Midbrain, Thalamus, and Cortex Monty A. Escab and Heather L. Read
Spectral Processing and Sound Source Determination Donal G. Sinex Spectral Information in Sound Localization Simon Carlile, Russell Martin, and Ken McAnally Plasticity of Spectral Processing Dexter R. F. Irvine and Beverly A. Wright Spectral Processing In Cochlear Implants Colette M. McKay index
Volume 71 Autism: Neuropathology, Alterations of the GABAergic System, and Animal Models Christoph Schmitz, Imke A. J. van Kooten, Patrick R. Hof, Herman van Engeland, Paul H. Patterson, and Harry W. M. Steinbusch The Role of GABA in the Early Neuronal Development Marta Jelitai and Emı´lia Madarasz GABAergic Signaling in the Developing Cerebellum Chitoshi Takayama Insights into GABA Functions in the Developing Cerebellum Mo´nica L. Fiszman Role of GABA in the Mechanism of the Onset of Puberty in Non-Human Primates Ei Terasawa Rett Syndrome: A Rosetta Stone for Understanding the Molecular Pathogenesis of Autism Janine M. LaSalle, Amber Hogart, and Karen N. Thatcher
248
CONTENTS OF RECENT VOLUMES
GABAergic Cerebellar System in Autism: A Neuropathological and Developmental Perspective Gene J. Blatt
A Systematic Examination of Catatonia-Like Clinical Pictures in Autism Spectrum Disorders Lorna Wing and Amitta Shah
Reelin Glycoprotein in Autism and Schizophrenia S. Hossein Fatemi
Catatonia in Individuals with Autism Spectrum Disorders in Adolescence and Early Adulthood: A Long-Term Prospective Study Masataka Ohta, Yukiko Kano, and Yoko Nagai
Is There A Connection Between Autism, Prader-Willi Syndrome, Catatonia, and GABA? Dirk M. Dhossche, Yaru Song, and Yiming Liu Alcohol, GABA Receptors, and Neurodevelopmental Disorders Ujjwal K. Rout Effects of Secretin on Extracellular GABA and Other Amino Acid Concentrations in the Rat Hippocampus Hans-Willi Clement, Alexander Pschibul, and Eberhard Schulz Predicted Role of Secretin and Oxytocin in the Treatment of Behavioral and Developmental Disorders: Implications for Autism Martha G. Welch and David A. Ruggiero Immunological Findings in Autism Hari Har Parshad Cohly and Asit Panja Correlates of Psychomotor Symptoms in Autism Laura Stoppelbein, Sara Sytsma-Jordan, and Leilani Greening GABRB3 Gene Deficient Mice: A Potential Model of Autism Spectrum Disorder Timothy M. DeLorey The Reeler Mouse: Anatomy of a Mutant Gabriella D’Arcangelo Shared Chromosomal Susceptibility Regions Between Autism and Other Mental Disorders Yvon C. Chagnon index
Are Autistic and Catatonic Regression Related? A Few Working Hypotheses Involving GABA, Purkinje Cell Survival, Neurogenesis, and ECT Dirk Marcel Dhossche and Ujjwal Rout Psychomotor Development and Psychopathology in Childhood Dirk M. J. De Raeymaecker The Importance of Catatonia and Stereotypies in Autistic Spectrum Disorders Laura Stoppelbein, Leilani Greening, and Angelina Kakooza Prader–Willi Syndrome: Atypical Psychoses and Motor Dysfunctions Willem M. A. Verhoeven and Siegfried Tuinier Towards a Valid Nosography and Psychopathology of Catatonia in Children and Adolescents David Cohen Is There a Common Neuronal Basis for Autism and Catatonia? Dirk Marcel Dhossche, Brendan T. Carroll, and Tressa D. Carroll Shared Susceptibility Region on Chromosome 15 Between Autism and Catatonia Yvon C. Chagnon Current Trends in Behavioral Interventions for Children with Autism Dorothy Scattone and Kimberly R. Knight
index
Case Reports with a Child Psychiatric Exploration of Catatonia, Autism, and Delirium Jan N. M. Schieveld
Volume 72
ECT and the Youth: Catatonia in Context Frank K. M. Zaw
Classification Matters for Catatonia and Autism in Children Klaus-Ju¨rgen Neuma¨rker
Catatonia in Autistic Spectrum Disorders: A Medical Treatment Algorithm Max Fink, Michael A. Taylor, and Neera Ghaziuddin
CONTENTS OF RECENT VOLUMES
Psychological Approaches to Chronic Catatonia-Like Deterioration in Autism Spectrum Disorders Amitta Shah and Lorna Wing
Volume 74
Section V: Blueprints Blueprints for the Assessment, Treatment, and Future Study of Catatonia in Autism Spectrum Disorders Dirk Marcel, Dhossche, Amitta Shah, and Lorna Wing
Section I: Visual Aspects
index
Volume 73 Chromosome 22 Deletion Syndrome and Schizophrenia Nigel M. Williams, Michael C. O’Donovan, and Michael J. Owen Characterization of Proteome of Human Cerebrospinal Fluid Jing Xu, Jinzhi Chen, Elaine R. Peskind, Jinghua Jin, Jimmy Eng, Catherine Pan, Thomas J. Montine, David R. Goodlett, and Jing Zhang Hormonal Pathways Regulating Intermale and Interfemale Aggression Neal G. Simon, Qianxing Mo, Shan Hu, Carrie Garippa, and Shi-Fang Lu Neuronal GAP Junctions: Expression, Function, and Implications for Behavior Clinton B. McCracken and David C. S. Roberts Effects of Genes and Stress on the Neurobiology of Depression J. John Mann and Dianne Currier Quantitative Imaging with the Micropet SmallAnimal Pet Tomograph Paul Vaska, Daniel J. Rubins, David L. Alexoff, and Wynne K. Schiffer Understanding Myelination through Studying its Evolution Ru¨diger Schweigreiter, Betty I. Roots, Christine Bandtlow, and Robert M. Gould index
249
Evolutionary Neurobiology and Art C. U. M. Smith
Perceptual Portraits Nicholas Wade The Neuropsychology of Visual Art: Conferring Capacity Anjan Chatterjee Vision, Illusions, and Reality Christopher Kennard Localization in the Visual Brain George K. York Section II: Episodic Disorders Neurology, Synaesthesia, and Painting Amy Ione Fainting in Classical Art Philip Smith Migraine Art in the Internet: A Study of 450 Contemporary Artists Klaus Podoll Sarah Raphael’s Migraine with Aura as Inspiration for the Foray of Her Work into Abstraction Klaus Podoll and Debbie Ayles The Visual Art of Contemporary Artists with Epilepsy Steven C. Schachter Section III: Brain Damage Creativity in Painting and Style in BrainDamaged Artists Julien Bogousslavsky Artistic Changes in Alzheimer’s Disease Sebastian J. Crutch and Martin N. Rossor Section IV: Cerebrovascular Disease Stroke in Painters H. Ba¨zner and M. Hennerici Visuospatial Neglect in Lovis Corinth’s SelfPortraits Olaf Blanke
250
CONTENTS OF RECENT VOLUMES
Art, Constructional Apraxia, and the Brain Louis Caplan Section V: Genetic Diseases Neurogenetics in Art Alan E. H. Emery A Naı¨ve Artist of St Ives F. Clifford Rose Van Gogh’s Madness F. Clifford Rose Absinthe, The Nervous System and Painting Tiina Rekand Section VI: Neurologists as Artists Sir Charles Bell, KGH, FRS, FRSE (1774–1842) Christopher Gardner-Thorpe Section VII: Miscellaneous Peg Leg Frieda Espen Dietrichs The Deafness of Goya (1746–1828) F. Clifford Rose
Transmitter Release at the Neuromuscular Junction Thomas L. Schwarz Vesicle Trafficking and Recycling at the Neuromuscular Junction: Two Pathways for Endocytosis Yoshiaki Kidokoro Glutamate Receptors at the Drosophila Neuromuscular Junction Aaron DiAntonio Scaffolding Proteins at the Drosophila Neuromuscular Junction Bulent Ataman, Vivian Budnik, and Ulrich Thomas Synaptic Cytoskeleton at the Neuromuscular Junction Catalina Ruiz-Can˜ada and Vivian Budnik Plasticity and Second Messengers During Synapse Development Leslie C. Griffith and Vivian Budnik Retrograde Signaling that Regulates Synaptic Development and Function at the Drosophila Neuromuscular Junction Guillermo Marque´s and Bing Zhang
index Volume 75 Introduction on the Use of the Drosophila Embryonic/Larval Neuromuscular Junction as a Model System to Study Synapse Development and Function, and a Brief Summary of Pathfinding and Target Recognition Catalina Ruiz-Can˜ada and Vivian Budnik Development and Structure of Motoneurons Matthias Landgraf and Stefan Thor
Activity-Dependent Regulation of Transcription During Development of Synapses Subhabrata Sanyal and Mani Ramaswami Experience-Dependent Potentiation of Larval Neuromuscular Synapses Christoph M. Schuster Selected Methods for the Anatomical Study of Drosophila Embryonic and Larval Neuromuscular Junctions Vivian Budnik, Michael Gorczyca, and Andreas Prokop index
The Development of the Drosophila Larval Body Wall Muscles Karen Beckett and Mary K. Baylies Organization of the Efferent System and Structure of Neuromuscular Junctions in Drosophila Andreas Prokop Development of Motoneuron Electrical Properties and Motor Output Richard A. Baines
Volume 76 Section I: Physiological Correlates of Freud’s Theories The ID, the Ego, and the Temporal Lobe Shirley M. Ferguson and Mark Rayport
CONTENTS OF RECENT VOLUMES
ID, Ego, and Temporal Lobe Revisited Shirley M. Ferguson and Mark Rayport Section II: Stereotaxic Studies Olfactory Gustatory Responses Evoked by Electrical Stimulation of Amygdalar Region in Man Are Qualitatively Modifiable by Interview Content: Case Report and Review Mark Rayport, Sepehr Sani, and Shirley M. Ferguson Section III: Controversy in Definition of Behavioral Disturbance Pathogenesis of Psychosis in Epilepsy. The ‘‘Seesaw’’ Theory: Myth or Reality? Shirley M. Ferguson and Mark Rayport Section IV: Outcome of Temporal Lobectomy Memory Function After Temporal Lobectomy for Seizure Control: A Comparative Neuropsy chiatric and Neuropsychological Study Shirley M. Ferguson, A. John McSweeny, and Mark Rayport Life After Surgery for Temporolimbic Seizures Shirley M. Ferguson, Mark Rayport, and Carolyn A. Schell
251
Evidence for Neuroprotective Effects of Antipsychotic Drugs: Implications for the Pathophysiology and Treatment of Schizophrenia Xin-Min Li and Haiyun Xu Neurogenesis and Neuroenhancement in the Pathophysiology and Treatment of Bipolar Disorder Robert J. Schloesser, Guang Chen, and Husseini K. Manji Neuroreplacement, Growth Factor, and Small Molecule Neurotrophic Approaches for Treating Parkinson’s Disease Michael J. O’Neill, Marcus J. Messenger, Viktor Lakics, Tracey K. Murray, Eric H. Karran, Philip G. Szekeres, Eric S. Nisenbaum, and Kalpana M. Merchant Using Caenorhabditis elegans Models of Neurodegenerative Disease to Identify Neuroprotective Strategies Brian Kraemer and Gerard D. Schellenberg Neuroprotection and Enhancement of Neurite Outgrowth With Small Molecular Weight Compounds From Screens of Chemical Libraries Donard S. Dwyer and Addie Dickson index
Appendix I Mark Rayport Appendix II: Conceptual Foundations of Studies of Patients Undergoing Temporal Lobe Surgery for Seizure Control Mark Rayport index Volume 77 Regenerating the Brain David A. Greenberg and Kunlin Jin Serotonin and Brain: Evolution, Neuroplasticity, and Homeostasis Efrain C. Azmitia Therapeutic Approaches to Promoting Axonal Regeneration in the Adult Mammalian Spinal Cord Sari S. Hannila, Mustafa M. Siddiq, and Marie T. Filbin
Volume 78 Neurobiology of Dopamine in Schizophrenia Olivier Guillin, Anissa Abi-Dargham, and Marc Laruelle The Dopamine System and the Pathophysiology of Schizophrenia: A Basic Science Perspective Yukiori Goto and Anthony A. Grace Glutamate and Schizophrenia: Phencyclidine, N-methyl-D-aspartate Receptors, and Dopamine–Glutamate Interactions Daniel C. Javitt Deciphering the Disease Process of Schizophrenia: The Contribution of Cortical GABA Neurons David A. Lewis and Takanori Hashimoto Alterations of Serotonin Transmission in Schizophrenia Anissa Abi-Dargham
252
CONTENTS OF RECENT VOLUMES
Serotonin and Dopamine Interactions in Rodents and Primates: Implications for Psychosis and Antipsychotic Drug Development Gerard J. Marek
The CD8 T Cell in Multiple Sclerosis: Suppressor Cell or Mediator of Neuropathology? Aaron J. Johnson, Georgette L. Suidan, Jeremiah McDole, and Istvan Pirko
Cholinergic Circuits and Signaling in the Pathophysiology of Schizophrenia Joshua A. Berman, David A. Talmage, and Lorna W. Role
Immunopathogenesis of Multiple Sclerosis Smriti M. Agrawal and V. Wee Yong
Schizophrenia and the 7 Nicotinic Acetylcholine Receptor Laura F. Martin and Robert Freedman Histamine and Schizophrenia Jean-Michel Arrang Cannabinoids and Psychosis Deepak Cyril D’Souza Involvement of Neuropeptide Systems in Schizophrenia: Human Studies Ricardo Ca´ceda, Becky Kinkead, and Charles B. Nemeroff Brain-Derived Neurotrophic Factor in Schizophrenia and Its Relation with Dopamine Olivier Guillin, Caroline Demily, and Florence Thibaut Schizophrenia Susceptibility Genes: In Search of a Molecular Logic and Novel Drug Targets for a Devastating Disorder Joseph A. Gogos
Molecular Mimicry in Multiple Sclerosis Jane E. Libbey, Lori L. McCoy, and Robert S. Fujinami Molecular ‘‘Negativity’’ May Underlie Multiple Sclerosis: Role of the Myelin Basic Protein Family in the Pathogenesis of MS Abdiwahab A. Musse and George Harauz Microchimerism and Stem Cell Transplantation in Multiple Sclerosis Behrouz Nikbin, Mandana Mohyeddin Bonab, and Fatemeh Talebian The Insulin-Like Growth Factor System in Multiple Sclerosis Daniel Chesik, Nadine Wilczak, and Jacques De Keyser Cell-Derived Microparticles and Exosomes in Neuroinflammatory Disorders Lawrence L. Horstman, Wenche Jy, Alireza Minagar, Carlos J. Bidot, Joaquin J. Jimenez, J. Steven Alexander, and Yeon S. Ahn
index
Multiple Sclerosis in Children: Clinical, Diagnostic, and Therapeutic Aspects Kevin Rosta´sy
Volume 79
Migraine in Multiple Sclerosis Debra G. Elliott
The Destructive Alliance: Interactions of Leukocytes, Cerebral Endothelial Cells, and the Immune Cascade in Pathogenesis of Multiple Sclerosis Alireza Minagar, April Carpenter, and J. Steven Alexander Role of B Cells in Pathogenesis of Multiple Sclerosis Behrouz Nikbin, Mandana Mohyeddin Bonab, Farideh Khosravi, and Fatemeh Talebian The Role of CD4 T Cells in the Pathogenesis of Multiple Sclerosis Tanuja Chitnis
Multiple Sclerosis as a Painful Disease Meghan Kenner, Uma Menon, and Debra Elliott Multiple Sclerosis and Behavior James B. Pinkston, Anita Kablinger, and Nadejda Alekseeva Cerebrospinal Fluid Analysis in Multiple Sclerosis Francisco A. Luque and Stephen L. Jaffe Multiple Sclerosis in Isfahan, Iran Mohammad Saadatnia, Masoud Etemadifar, and Amir Hadi Maghzi Gender Issues in Multiple Sclerosis Robert N. Schwendimann and Nadejda Alekseeva
253
CONTENTS OF RECENT VOLUMES
Differential Diagnosis of Multiple Sclerosis Halim Fadil, Roger E. Kelley, and Eduardo Gonzalez-Toledo
Optic Neuritis and the Neuro-Ophthalmology of Multiple Sclerosis Paramjit Kaur and Jeffrey L. Bennett
Prognostic Factors in Multiple Sclerosis Roberto Bergamaschi
Neuromyelitis Optica: Pathogenesis Dean M. Wingerchuk
Neuroimaging in Multiple Sclerosis Robert Zivadinov and Jennifer L. Cox Detection of Cortical Lesions Is Dependent on Choice of Slice Thickness in Patients with Multiple Sclerosis Ondrej Dolezal, Michael G. Dwyer, Dana Horakova, Eva Havrdova, Alireza Minagar, Srivats Balachandran, Niels Bergsland, Zdenek Seidl, Manuela Vaneckova, David Fritz, Jan Krasensky, and Robert Zivadinov The Role of Quantitative Neuroimaging Indices in the Differentiation of Ischemia from Demyelination: An Analytical Study with Case Presentation Romy Hoque, Christina Ledbetter, Eduardo GonzalezToledo, Vivek Misra, Uma Menon, Meghan Kenner, Alejandro A. Rabinstein, Roger E. Kelley, Robert Zivadinov, and Alireza Minagar
New
Findings
on
index Volume 79 Epilepsy in the Elderly: Scope of the Problem Ilo E. Leppik Animal Models in Gerontology Research Nancy L. Nadon Animal Models of Geriatric Epilepsy Lauren J. Murphree, Lynn M. Rundhaugen, and Kevin M. Kelly Life and Death of Neurons in the Aging Cerebral Cortex John H. Morrison and Patrick R. Hof
HLA-DRB1*1501, -DQB1*0301, -DQB1*0302, -DQB1*0602, and -DQB1*0603 Alleles Are Associated with More Severe Disease Outcome on MRI in Patients with Multiple Sclerosis Robert Zivadinov, Laura Uxa, Alessio Bratina, Antonio Bosco, Bhooma Srinivasaraghavan, Alireza Minagar, Maja Ukmar, Su yen Benedetto, and Marino Zorzon
An In Vitro Model of Stroke-Induced Epilepsy: Elucidation of the Roles of Glutamate and Calcium in the Induction and Maintenance of Stroke-Induced Epileptogenesis Robert J. DeLorenzo, David A. Sun, Robert E. Blair, and Sompong Sambati
Glatiramer Acetate: Mechanisms of Action in Multiple Sclerosis Tjalf Ziemssen and Wiebke Schrempf
Epidemiology and Outcomes of Status Epilepticus in the Elderly Alan R. Towne
Evolving Therapies for Multiple Sclerosis Elena Korniychuk, John M. Dempster, Eileen O’Connor, J. Steven Alexander, Roger E. Kelley, Meghan Kenner, Uma Menon, Vivek Misra, Romy Hoque, Eduardo C. GonzalezToledo, Robert N. Schwendimann, Stacy Smith, and Alireza Minagar
Diagnosing Epilepsy in the Elderly R. Eugene Ramsay, Flavia M. Macias, and A. James Rowan
Remyelination in Multiple Sclerosis Divya M. Chari
Use of Antiepileptic Medications in Nursing Homes Judith Garrard, Susan L. Harms, Lynn E. Eberly, and Ilo E. Leppik
Trigeminal Neuralgia: A Modern-Day Review Kelly Hunt and Ravish Patwardhan
Mechanisms of Action of Antiepileptic Drugs H. Steve White, Misty D. Smith, and Karen S. Wilcox
Pharmacoepidemiology in Community-Dwelling Elderly Taking Antiepileptic Drugs Dan R. Berlowitz and Mary Jo V. Pugh
254
CONTENTS OF RECENT VOLUMES
Differential Diagnosis of Multiple Sclerosis Halim Fadil, Roger E. Kelley, and Eduardo Gonzalez-Toledo
Optic Neuritis and the Neuro-Ophthalmology of Multiple Sclerosis Paramjit Kaur and Jeffrey L. Bennett
Prognostic Factors in Multiple Sclerosis Roberto Bergamaschi
Neuromyelitis Optica: Pathogenesis Dean M. Wingerchuk
Neuroimaging in Multiple Sclerosis Robert Zivadinov and Jennifer L. Cox Detection of Cortical Lesions Is Dependent on Choice of Slice Thickness in Patients with Multiple Sclerosis Ondrej Dolezal, Michael G. Dwyer, Dana Horakova, Eva Havrdova, Alireza Minagar, Srivats Balachandran, Niels Bergsland, Zdenek Seidl, Manuela Vaneckova, David Fritz, Jan Krasensky, and Robert Zivadinov TheRole ofQuantitativeNeuroimaging Indices in the Differentiation of Ischemia from Demyelination: An Analytical Study with Case Presentation Romy Hoque, Christina Ledbetter, Eduardo GonzalezToledo, Vivek Misra, Uma Menon, Meghan Kenner, Alejandro A. Rabinstein, Roger E. Kelley, Robert Zivadinov, and Alireza Minagar HLA-DRB1*1501, -DQB1*0301,-DQB1*0302,DQB1*0602, and -DQB1*0603 Alleles Are Associated with More Severe Disease Outcome on MRI in Patients with Multiple Sclerosis Robert Zivadinov, Laura Uxa, Alessio Bratina, Antonio Bosco, Bhooma Srinivasaraghavan, Alireza Minagar, Maja Ukmar, Su yen Benedetto, and Marino Zorzon Glatiramer Acetate: Mechanisms of Action in Multiple Sclerosis Tjalf Ziemssen and Wiebke Schrempf Evolving Therapies for Multiple Sclerosis Elena Korniychuk, John M. Dempster, Eileen O’Connor, J. Steven Alexander, Roger E. Kelley, Meghan Kenner, Uma Menon, Vivek Misra, Romy Hoque, Eduardo C. GonzalezToledo, Robert N. Schwendimann, Stacy Smith, and Alireza Minagar Remyelination in Multiple Sclerosis Divya M. Chari Trigeminal Neuralgia: A Modern-Day Review Kelly Hunt and Ravish Patwardhan
New
Findings
on
index Volume 81 Epilepsy in the Elderly: Scope of the Problem Ilo E. Leppik Animal Models in Gerontology Research Nancy L. Nadon Animal Models of Geriatric Epilepsy Lauren J. Murphree, Lynn M. Rundhaugen, and Kevin M. Kelly Life and Death of Neurons in the Aging Cerebral Cortex John H. Morrison and Patrick R. Hof An In Vitro Model of Stroke-Induced Epilepsy: Elucidation of the Roles of Glutamate and Calcium in the Induction and Maintenance of Stroke-Induced Epileptogenesis Robert J. DeLorenzo, David A. Sun, Robert E. Blair, and Sompong Sambati Mechanisms of Action of Antiepileptic Drugs H. Steve White, Misty D. Smith, and Karen S. Wilcox Epidemiology and Outcomes of Status Epilepticus in the Elderly Alan R. Towne Diagnosing Epilepsy in the Elderly R. Eugene Ramsay, Flavia M. Macias, and A. James Rowan Pharmacoepidemiology in Community-Dwelling Elderly Taking Antiepileptic Drugs Dan R. Berlowitz and Mary Jo V. Pugh Use of Antiepileptic Medications in Nursing Homes Judith Garrard, Susan L. Harms, Lynn E. Eberly, and Ilo E. Leppik
CONTENTS OF RECENT VOLUMES
Age-Related Changes in Pharmacokinetics: Predictability and Assessment Methods Emilio Perucca Factors Affecting Antiepileptic Drug Pharmacokinetics in Community-Dwelling Elderly James C. Cloyd, Susan Marino, and Angela K. Birnbaum Pharmacokinetics of Antiepileptic Drugs in Elderly Nursing Home Residents Angela K. Birnbaum The Impact of Epilepsy on Older Veterans Mary Jo V. Pugh, Dan R. Berlowitz, and Lewis Kazis Risk and Predictability of Drug Interactions in the Elderly Rene´ H. Levy and Carol Collins Outcomes in Elderly Patients With Newly Diagnosed and Treated Epilepsy Martin J. Brodie and Linda J. Stephen Recruitment and Retention in Clinical Trials of the Elderly Flavia M. Macias, R. Eugene Ramsay, and A. James Rowan Treatment of Convulsive Status Epilepticus David M. Treiman Treatment of Nonconvulsive Status Epilepticus Matthew C. Walker Antiepileptic Drug Formulation and Treatment in the Elderly: Biopharmaceutical Considerations Barry E. Gidal index Volume 82 Inflammatory Mediators Leading to Protein Misfolding and Uncompetitive/Fast Off-Rate Drug Therapy for Neurodegenerative Disorders Stuart A. Lipton, Zezong Gu, and Tomohiro Nakamura Innate Immunity and Protective Neuroinflammation: New Emphasis on the Role of Neuroimmune Regulatory Proteins M. Griffiths, J. W. Neal, and P. Gasque
255
Glutamate Release from Astrocytes in Physiological Conditions and in Neurodegenerative Disorders Characterized by Neuroinflammation Sabino Vesce, Daniela Rossi, Liliana Brambilla, and Andrea Volterra The High-Mobility Group Box 1 Cytokine Induces Transporter-Mediated Release of Glutamate from Glial Subcellular Particles (Gliosomes) Prepared from In Situ-Matured Astrocytes Giambattista Bonanno, Luca Raiteri, Marco Milanese, Simona Zappettini, Edon Melloni, Marco Pedrazzi, Mario Passalacqua, Carlo Tacchetti, Cesare Usai, and Bianca Sparatore The Role of Astrocytes and Complement System in Neural Plasticity Milos Pekny, Ulrika Wilhelmsson, Yalda Rahpeymai Bogesta˚l, and Marcela Pekna New Insights into the Roles of Metalloproteinases in Neurodegeneration and Neuroprotection A. J. Turner and N. N. Nalivaeva Relevance of High-Mobility Group Protein Box 1 to Neurodegeneration Silvia Fossati and Alberto Chiarugi Early Upregulation of Matrix Metalloproteinases Following Reperfusion Triggers Neuroinflammatory Mediators in Brain Ischemia in Rat Diana Amantea, Rossella Russo, Micaela Gliozzi, Vincenza Fratto, Laura Berliocchi, G. Bagetta, G. Bernardi, and M. Tiziana Corasaniti The (Endo)Cannabinoid System in Multiple Sclerosis and Amyotrophic Lateral Sclerosis Diego Centonze, Silvia Rossi, Alessandro FinazziAgro`, Giorgio Bernardi, and Mauro Maccarrone Chemokines and Chemokine Receptors: Multipurpose Players in Neuroinflammation Richard M. Ransohoff, LiPing Liu, and Astrid E. Cardona Systemic and Acquired Immune Responses in Alzheimer’s Disease Markus Britschgi and Tony Wyss-Coray Neuroinflammation in Alzheimer’s Disease and Parkinson’s Disease: Are Microglia Pathogenic in Either Disorder? Joseph Rogers, Diego Mastroeni, Brian Leonard, Jeffrey Joyce, and Andrew Grover
256
CONTENTS OF RECENT VOLUMES
Cytokines and Neuronal Ion Channels in Health and Disease Barbara Viviani, Fabrizio Gardoni, and Marina Marinovich Cyclooxygenase-2, Prostaglandin E2, and Microglial Activation in Prion Diseases Luisa Minghetti and Maurizio Pocchiari Glia Proinflammatory Cytokine Upregulation as a Therapeutic Target for Neurodegenerative Diseases: Function-Based and Target-Based Discovery Approaches Linda J. Van Eldik, Wendy L. Thompson, Hantamalala Ralay Ranaivo, Heather A. Behanna, and D. Martin Watterson Oxidative Stress and the Pathogenesis of Neurodegenerative Disorders Ashley Reynolds, Chad Laurie, R. Lee Mosley, and Howard E. Gendelman Differential Modulation of Type 1 and Type 2 Cannabinoid Receptors Along the Neuroimmune Axis Sergio Oddi, Paola Spagnuolo, Monica Bari, Antonella D’Agostino, and Mauro Maccarrone Effects of the HIV-1 Viral Protein Tat on Central Neurotransmission: Role of Group I Metabotropic Glutamate Receptors Elisa Neri, Veronica Musante, and Anna Pittaluga Evidence to Implicate Early Modulation of Interleukin-1 Expression in the Neuroprotection Afforded by 17 -Estradiol in Male Rats Undergone Transient Middle Cerebral Artery Occlusion Olga Chiappetta, Micaela Gliozzi, Elisa Siviglia, Diana Amantea, Luigi A. Morrone, Laura Berliocchi, G. Bagetta, and M. Tiziana Corasaniti A Role for Brain Cyclooxygenase-2 and Prostaglandin-E2 in Migraine: Effects of Nitroglycerin Cristina Tassorelli, Rosaria Greco, Marie There`se Armentero, Fabio Blandini, Giorgio Sandrini, and Giuseppe Nappi The Blockade of K+-ATP Channels has Neuroprotective Effects in an In Vitro Model of Brain Ischemia Robert Nistico`, Silvia Piccirilli, L. Sebastianelli, Giuseppe Nistico`, G. Bernardi, and N. B. Mercuri
Retinal Damage Caused by High Intraocular Pressure-Induced Transient Ischemia is Prevented by Coenzyme Q10 in Rat Carlo Nucci, Rosanna Tartaglione, Angelica Cerulli, R. Mancino, A. Spano`, Federica Cavaliere, Laura Rombol, G. Bagetta, M. Tiziana Corasaniti, and Luigi A. Morrone Evidence Implicating Matrix Metalloproteinases in the Mechanism Underlying Accumulation of IL-1 and Neuronal Apoptosis in the Neocortex of HIV/gp120-Exposed Rats Rossella Russo, Elisa Siviglia, Micaela Gliozzi, Diana Amantea, Annamaria Paoletti, Laura Berliocchi, G. Bagetta, and M. Tiziana Corasaniti Neuroprotective Effect of Nitroglycerin in a Rodent Model of Ischemic Stroke: Evaluation of Bcl-2 Expression Rosaria Greco, Diana Amantea, Fabio Blandini, Giuseppe Nappi, Giacinto Bagetta, M. Tiziana Corasaniti, and Cristina Tassorelli index Volume 83 Gender Differences in Pharmacological Response Gail D. Anderson Epidemiology and Classification of Epilepsy: Gender Comparisons John C. McHugh and Norman Delanty Hormonal Influences on Seizures: Basic Neurobiology Cheryl A. Frye Catamenial Epilepsy Patricia E. Penovich and Sandra Helmers Epilepsy in Women: Special Considerations for Adolescents Mary L. Zupanc and Sheryl Haut Contraception in Women with Epilepsy: Pharmacokinetic Interactions, Contraceptive Options, and Management Caryn Dutton and Nancy Foldvary-Schaefer
CONTENTS OF RECENT VOLUMES
Reproductive Dysfunction in Women with Epilepsy: Menstrual Cycle Abnormalities, Fertility, and Polycystic Ovary Syndrome Ju¨rgen Bauer and De´irdre Cooper-Mahkorn Sexual Dysfunction in Women with Epilepsy: Role of Antiepileptic Drugs and Psychotropic Medications Mary A. Gutierrez, Romila Mushtaq, and Glen Stimmel Pregnancy in Epilepsy: Issues of Concern John DeToledo Teratogenicity and Antiepileptic Drugs: Potential Mechanisms Mark S. Yerby Antiepileptic Drug Teratogenesis: What are the Risks for Congenital Malformations and Adverse Cognitive Outcomes? Cynthia L. Harden Teratogenicity of Antiepileptic Drugs: Role of Pharmacogenomics Raman Sankar and Jason T. Lerner Antiepileptic Drug Therapy in Pregnancy I: Gestation-Induced Effects on AED Pharmacokinetics Page B. Pennell and Collin A. Hovinga
257
Metabolic Effects of AEDs: Impact on Body Weight, Lipids and Glucose Metabolism Raj D. Sheth and Georgia Montouris Psychiatric Comorbidities in Epilepsy W. Curt Lafrance, Jr., Andres M. Kanner, and Bruce Hermann Issues for Mature Women with Epilepsy Cynthia L. Harden Pharmacodynamic and Pharmacokinetic Interactions of Psychotropic Drugs with Antiepileptic Drugs Andres M. Kanner and Barry E. Gidal Health Disparities in Epilepsy: How Patient-Oriented Outcomes in Women Differ from Men Frank Gilliam index Volume 84 Normal Brain Aging: Clinical, Immunological, Neuropsychological, and Neuroimaging Features Maria T. Caserta, Yvonne Bannon, Francisco Fernandez, Brian Giunta, Mike R. Schoenberg, and Jun Tan Subcortical Ischemic Cerebrovascular Dementia Uma Menon and Roger E. Kelley
Antiepileptic Drug Therapy in Pregnancy II: Fetal and Neonatal Exposure Collin A. Hovinga and Page B. Pennell
Cerebrovascular and Cardiovascular Pathology in Alzheimer’s Disease Jack C. de la Torre
Seizures in Pregnancy: Diagnosis and Management Robert L. Beach and Peter W. Kaplan
Neuroimaging of Cognitive Impairments in Vascular Disease Carol Di Perri, Turi O. Dalaker, Mona K. Beyer, and Robert Zivadinov
Management of Epilepsy and Pregnancy: An Obstetrical Perspective Julian N. Robinson and Jane Cleary-Goldman Pregnancy Registries: Strengths, Weaknesses, and Bias Interpretation of Pregnancy Registry Data Marianne Cunnington and John Messenheimer Bone Health in Women With Epilepsy: Clinical Features and Potential Mechanisms Alison M. Pack and Thaddeus S. Walczak
Contributions of Neuropsychology and Neuroimaging to Understanding Clinical Subtypes of Mild Cognitive Impairment Amy J. Jak, Katherine J. Bangen, Christina E. Wierenga, Lisa Delano-Wood, Jody Corey-Bloom, and Mark W. Bondi Proton Magnetic Resonance Spectroscopy in Dementias and Mild Cognitive Impairment H. Randall Griffith, Christopher C. Stewart, and Jan A. den Hollander
258
CONTENTS OF RECENT VOLUMES
Application of PET Imaging to Diagnosis of Alzheimer’s Disease and Mild Cognitive Impairment James M. Noble and Nikolaos Scarmeas
Targeted Lipidomics as a Tool to Investigate Endocannabinoid Function Giuseppe Astarita, Jennifer Geaga, Faizy Ahmed, and Daniele Piomelli
The Molecular and Cellular Pathogenesis of Dementia of the Alzheimer’s Type: An Overview Francisco A. Luque and Stephen L. Jaffe
The Endocannabinoid System as a Target for Novel Anxiolytic and Antidepressant Drugs Silvana Gaetani, Pasqua Dipasquale, Adele Romano, Laura Righetti, Tommaso Cassano, Daniele Piomelli, and Vincenzo Cuomo
Alzheimer’s Disease Genetics: Current Status and Future Perspectives Lars Bertram Frontotemporal Lobar Degeneration: Insights from Neuropsychology and Neuroimaging Andrea C. Bozoki and Muhammad U. Farooq Lewy Body Dementia Jennifer C. Hanson and Carol F. Lippa Dementia in Parkinson’s Disease Bradley J. Robottom and William J. Weiner Early Onset Dementia Halim Fadil, Aimee Borazanci, Elhachmia Ait Ben Haddou, Mohamed Yahyaoui, Elena Korniychuk, Stephen L. Jaffe, and Alireza Minagar Normal Pressure Hydrocephalus Glen R. Finney Reversible Dementias Anahid Kabasakalian and Glen R. Finney index Volume 85 Involvement of the Prefrontal Cortex in Problem Solving Hajime Mushiake, Kazuhiro Sakamoto, Naohiro Saito, Toshiro Inui, Kazuyuki Aihara, and Jun Tanji GluK1 Receptor Antagonists and Hippocampal Mossy Fiber Function Robert Nistico`, Sheila Dargan, Stephen M. Fitzjohn, David Lodge, David E. Jane, Graham L. Collingridge, and Zuner A. Bortolotto Monoamine Transporter as a Target Molecule for Psychostimulants Ichiro Sora, BingJin Li, Setsu Fumushima, Asami Fukui, Yosefu Arime, Yoshiyuki Kasahara, Hiroaki Tomita, and Kazutaka Ikeda
GABAA Receptor Function and Gene Expression During Pregnancy and Postpartum Giovanni Biggio, Maria Cristina Mostallino, Paolo Follesa, Alessandra Concas, and Enrico Sanna Early Postnatal Stress and Neural Circuit Underlying Emotional Regulation Machiko Matsumoto, Mitsuhiro Yoshioka, and Hiroko Togashi Roles of the Histaminergic Neurotransmission on Methamphetamine-Induced Locomotor Sensitization and Reward: A Study of Receptors Gene Knockout Mice Naoko Takino, Eiko Sakurai, Atsuo Kuramasu, Nobuyuki Okamura, and Kazuhiko Yanai Developmental Exposure to Cannabinoids Causes Subtle and Enduring Neurofunctional Alterations Patrizia Campolongo, Viviana Trezza, Maura Palmery, Luigia Trabace, and Vincenzo Cuomo Neuronal Mechanisms for Pain-Induced Aversion: Behavioral Studies Using a Conditioned Place Aversion Test Masabumi Minami Bv8/Prokineticins and their Receptors: A New Pronociceptive System Lucia Negri, Roberta Lattanzi, Elisa Giannini, Michela Canestrelli, Annalisa Nicotra, and Pietro Melchiorri P2Y6-Evoked Microglial Phagocytosis Kazuhide Inoue, Schuichi Koizumi, Ayako Kataoka, Hidetoshi Tozaki-Saitoh, and Makoto Tsuda PPAR and Pain Takehiko Maeda and Shiroh Kishioka Involvement of Inflammatory Mediators in Neuropathic Pain Caused by Vincristine Norikazu Kiguchi, Takehiko Maeda, Yuka Kobayashi, Fumihiro Saika, and Shiroh Kishioka
CONTENTS OF RECENT VOLUMES
Nociceptive Behavior Induced by the Endogenous Opioid Peptides Dynorphins in Uninjured Mice: Evidence with Intrathecal N-ethylmaleimide Inhibiting Dynorphin Degradation Koichi Tan-No, Hiroaki Takahashi, Osamu Nakagawasai, Fukie Niijima, Shinobu Sakurada, Georgy Bakalkin, Lars Terenius, and Takeshi Tadano Mechanism of Allodynia Evoked by Intrathecal Morphine-3-Glucuronide in Mice Takaaki Komatsu, Shinobu Sakurada, Sou Katsuyama, Kengo Sanai, and Tsukasa Sakurada (–)-Linalool Attenuates Allodynia in Neuropathic Pain Induced by Spinal Nerve Ligation in C57/ Bl6 Mice Laura Berliocchi, Rossella Russo, Alessandra Levato, Vincenza Fratto, Giacinto Bagetta, Shinobu Sakurada, Tsukasa Sakurada, Nicola Biagio Mercuri, and Maria Tiziana Corasaniti Intraplantar Injection of Bergamot Essential Oil into the Mouse Hindpaw: Effects on CapsaicinInduced Nociceptive Behaviors Tsukasa Sakurada, Hikari Kuwahata, Soh Katsuyama, Takaaki Komatsu, Luigi A. Morrone, M. Tiziana Corasaniti, Giacinto Bagetta, and Shinobu Sakurada New Therapy for Neuropathic Pain Hirokazu Mizoguchi, Chizuko Watanabe, Akihiko Yonezawa, and Shinobu Sakurada Regulated Exocytosis from Astrocytes: Physiological and Pathological Related Aspects Corrado Calı`, Julie Marchaland, Paola Spagnuolo, Julien Gremion, and Paola Bezzi Glutamate Release from Astrocytic Gliosomes Under Physiological and Pathological Conditions Marco Milanese, Tiziana Bonifacino, Simona Zappettini, Cesare Usai, Carlo Tacchetti, Mario Nobile, and Giambattista Bonanno Neurotrophic and Neuroprotective Actions of an Enhancer of Ganglioside Biosynthesis Jin-ichi Inokuchi
259
Involvement of Endocannabinoid Signaling in the Neuroprotective Effects of Subtype 1 Metabotropic Glutamate Receptor Antagonists in Models of Cerebral Ischemia Elisa Landucci, Francesca Boscia, Elisabetta Gerace, Tania Scartabelli, Andrea Cozzi, Flavio Moroni, Guido Mannaioni, and Domenico E. PellegriniGiampietro NF-kappaB Dimers in the Regulation of Neuronal Survival Ilenia Sarnico, Annamaria Lanzillotta, Marina Benarese, Manuela Alghisi, Cristina Baiguera, Leontino Battistin, PierFranco Spano, and Marina Pizzi Oxidative Stress in Stroke Pathophysiology: Validation of Hydrogen Peroxide Metabolism as a Pharmacological Target to Afford Neuroprotection Diana Amantea, Maria Cristina Marrone, Robert Nistico`, Mauro Federici, Giacinto Bagetta, Giorgio Bernardi, and Nicola Biagio Mercuri Role of Akt and ERK Signaling in the Neurogenesis following Brain Ischemia Norifumi Shioda, Feng Han, and Kohji Fukunaga Prevention of Glutamate Accumulation and Upregulation of Phospho-Akt may Account for Neuroprotection Afforded by Bergamot Essential Oil against Brain Injury Induced by Focal Cerebral Ischemia in Rat Diana Amantea, Vincenza Fratto, Simona Maida, Domenicantonio Rotiroti, Salvatore Ragusa, Giuseppe Nappi, Giacinto Bagetta, and Maria Tiziana Corasaniti Identification of Novel Pharmacological Targets to Minimize Excitotoxic Retinal Damage Rossella Russo, Domenicantonio Rotiroti, Cristina Tassorelli, Carlo Nucci, Giacinto Bagetta, Massimo Gilberto Bucci, Maria Tiziana Corasaniti, and Luigi Antonio Morrone index
