Lecture Notes in Control and Information Sciences Editors: M. Thoma · M. Morari
306
Springer Berlin Heidelberg NewYork Hong Kong London Milan Paris Tokyo
Z. Zenn Bien Dimitar Stefanov (Eds.)
Advances in Rehabilitation Robotics Human-friendly Technologies on Movement Assistance and Restoration for People with Disabilities With 249 Figures
13
Series Advisory Board A. Bensoussan · P. Fleming · M.J. Grimble · P. Kokotovic · A.B. Kurzhanski · H. Kwakernaak · J.N. Tsitsiklis
Editors Prof. Z. Zenn Bien Human-Friendly Welfare Robot System Research Center Department of Electrical Engineering and Computer Science Korea Advanced Institute of Science and Technology 373-1 Guseong-dong, Yuseong-gu Daejeon, 305-701 Korea PhD Dimitar Stefanov, SRCS Cardiff & Vale NHS Trust Rehabilitation Engineering Unit Cardiff, CF5 2 YN UK and Institute of Mechanics Bulgarian Academy of Sciences Acad. G. Bonchev Street Block 4, 1113 Sofia Bulgaria
ISSN 0170-8643 ISBN 3-540-21986-2
Springer-Verlag Berlin Heidelberg New York
Library of Congress Control Number: 2004106092 This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in other ways, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer-Verlag. Violations are liable to prosecution under German Copyright Law. Springer-Verlag is a part of Springer Science+Business Media springeronline.com © Springer-Verlag Berlin Heidelberg 2004 Printed in Germany The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typesetting: Data conversion by the author. Final processing by PTP-Berlin Protago-TeX-Production GmbH, Berlin Cover-Design: design & production GmbH, Heidelberg Printed on acid-free paper 62/3020Yu - 5 4 3 2 1 0
Preface
It is now evident that one of the major application targets of the service robots is to use them as assistive devices for rehabilitation of the physically disabled and for the elderly people. Rehabilitation robotics (RR) is a relatively young but dynamically developing area of research. Some rehabilitation robots have already got out of the research laboratories and have become important members in everyday lives of growing users from many developed countries. It is expected that, in the near future, the rehabilitation robots (RR) will become a significant component of the futuristic welfare service systems in the world. Primarily limited to a small number of relatively simple movement tasks such as object replacement and eating, the application areas of the rehabilitation robotics, along with various intelligent technologies for movement assistance of people with disabilities, are continuously expanding to new dimensions that aim at improved assistance in different kinds of activities and entertainment of people with disabilities and aged people as well. We are witnessing that such intensive development of novel human-machine interfaces, intelligent control algorithms, new materials and efficient actuators have made it possible to invent and test various advanced design ideas. Common understanding and main tendency in the rehabilitation robotics design is that the robots should be human-friendly in the sense that the robotic machine and peripheral devices must be designed for the user to feel more comfortable, safer, and more convenient. Recent intelligent robotic devices for movement assistance are often designed to be equipped with the control strategies that do not cause high cognitive load to the users with various severe movement disorders. The idea for organizing this volume was inspired from the 8th International Conference on Rehabilitation Robotics (ICORR’2003) that was held during April 22-25, 2003 at Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Korea. The papers presented at the conference represent of course most recent tendencies of R&D in the rehabilitation robotics and intelligent assistive technology. With confidence, however, we like to declare that the current book is not just a variant of the conference proceedings! Different from the papers reported at the ICORR 2003 where specific problems and solutions of various subjects were discussed, the chapters of this book include original, reworked, and generalized materials that match
VI
Preface
to the style and objectives of the book. The book contains not only review articles on some advanced theoretical ideas in the rehabilitation robotics and results from some of the latest projects under development but also details on new advanced rehabilitation devices which have been recently transferred to the industry. A significant part of the book is devoted to the assessment of new rehabilitation technologies and evaluation of prototype devices with endusers. Safety of rehabilitation robots, historical remarks and perspective of rehabilitation robotics are also commented in the book. Also, different from many other books on rehabilitation engineering, the present volume includes a long chapter on robot-assisted neuro-rehabilitation that is considered as one of the latest trends in that area. One of the principal aims of this book is to promote dissemination of the information on the recent status of the rehabilitation robotics (RR). Our intention was to arrange the book in such a way that it is not just a simple collection of papers that would be of some interest to the specialists in a particular area, but rather, a book that contains some basics on the rehabilitation research and can help beginners to start their work in the same area, such as students and young researchers, or can help lecturers who want to introduce their students basics of the modern rehabilitation technology. In order to achieve this objective, most of the articles contain a detailed introduction to the problem to be discussed and an extended overview on the particular subject matter. The chapters that are contained in this book are authored by leading researchers in the field of rehabilitation robotics and represent a large part of the international research community. The book contains 27 chapters, which are grouped into 7 parts. The book begins with an introductory Part 1 devoted to description of the role of the rehabilitation robotics and some important issues of its development. The same part represents also some important milestones of the development of rehabilitation robotics. The chapters included in Part 2 cover three important issues on rehabilitation robotics for assistance of human movements: conceptions and experimental design, safety issues of the rehabilitation robots, and rehabilitation-robot evaluation. Some recent issues of the prosthetics and orthotics design are discussed in Part 3. Part 4 is concerned with the intelligent wheelchairs that can be considered as special mobile robots, designed to accomplish indoor user transportation. A recent trend in the design of assistive devices for mobility is the mechatronic devices for assistance in walking that sense the user’s movement intentions and provide gentle gait support, giving independency and safety to the user. Some examples of such devices are given in Part 5. Part 6 is dedicated to robotassisted neurorehabilitation. Examples of both upper limb robot mediated therapy and lower limb robot mediated therapy are commented in that part of the book. The final part of the book (Part7) talks about the perspectives and trends of the rehabilitation robotics.
Preface
VII
We are assured that the book would provide with a comprehensive overview of the field of rehabilitation robotics, and would satisfy a large group of readers, including researchers in the field, graduate and postgraduate students, and designers that use the RR technology. We believe that the book will become a representative selection of the latest trends and achievements on the rehabilitation robotics area. We would be extremely happy if such an important goal would be achieved. Finally, as the Editors of present volume, we like to take this opportunity to express our heartiest appreciation for all the authors who have worked their chapters with dedication and integrity, and contributed to the highest standard of this volume. We would like to thank also to Dr. Thomas Ditzinger and Ms. Heather King from Springer-Verlag for encouraging us in editing the present book and for their help in arranging this volume.
Daejeon March 2004
Z. Zenn Bien Dimitar Stefanov
Contents
List of Contributors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
XIX
List of Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . XXVII Part I Introduction 1 Advances in Human-Friendly Robotic Technologies for Movement Assistance/Movement Restoration for People with Disabilities Dimitar Stefanov, Z. Zenn Bien . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Areas of the RR Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.1 Robotic Systems for Movement Assistance . . . . . . . . . . . . . . 1.2.2 Robots for Physical Support and Indoor Navigation . . . . . . 1.2.3 Robots for Physical Rehabilitation . . . . . . . . . . . . . . . . . . . . . 1.2.4 Vocational RR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.5 Emotional Interactive Entertainment Robots . . . . . . . . . . . . 1.3 Specialized Human-Machine Interface . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Rehabilitation Robots in the Smart House Design . . . . . . . . . . . . . . 1.5 Functional Integration of the Robotic Environment . . . . . . . . . . . . . 1.6 Commercialization of RR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.7 Some Issues for Futuristic Intelligent Robotic House Model . . . . . . 1.8 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3 3 5 5 7 7 8 9 10 10 12 13 16 18
2 Rehabilitation Robotics from Past to Present – A Historical Perspective Michael Hillman . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Earliest Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Assistive Robotics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.1 Fixed Site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.2 Mobile Robots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.3 Wheelchair Mounted Manipulators . . . . . . . . . . . . . . . . . . . . . 2.3.4 Human Machine Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
25 25 26 27 28 31 33 35
X
2.4 2.5 2.6 2.7 2.8 2.9 2.10 2.11 2.12
Contents
Mobility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Prosthetics and Orthotics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Robot Mediated Therapy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Robotics in Special Needs Education . . . . . . . . . . . . . . . . . . . . . . . . . Robotics in Communications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Historical Perspective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Commercialisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Alternatives to Robotics in Rehabilitation . . . . . . . . . . . . . . . . . . . . . Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
35 36 37 38 39 40 40 41 42
Part II Rehabilitation Robots for Assistance of Human Movements II.1 Conceptions and Experimental Design 3 Toward a Human-Friendly User Interface to Control an Assistive Robot in the Context of Smart Homes Mounir Mokhtari, Mohamed Ali Feki, Bessam Abdulrazak, Bernard Grandjean . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 MANUS Assistive Robot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Networking Technologies and Developments . . . . . . . . . . . . . . . . . . . 3.4 General Software Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 User Interface Adaptation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 Implementation of a Path Planner . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6.1 Gesture Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6.2 Obstacles Avoidance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7 Towards the Co-autonomy Concept . . . . . . . . . . . . . . . . . . . . . . . . . . 3.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
47 47 48 49 50 51 52 52 53 54 55
4 Welfare-Oriented Service Robotic Systems: Intelligent Sweet Home & KARES II Z. Zenn Bien, Kwang-Hyun Park, Dae-Jin Kim, Jin-Woo Jung . . . . . . . 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Intelligent Sweet Home . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.1 Questionnaire Survey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.2 Assistive Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.3 Intelligent Man-Machine Interfaces . . . . . . . . . . . . . . . . . . . . . 4.3 KARES II System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.1 Questionnaire Survey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.2 Overall Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.3 Soft Robotic Arm with Visual Servoing . . . . . . . . . . . . . . . . . 4.3.4 Intelligent Human-Robot Interfaces . . . . . . . . . . . . . . . . . . . . 4.3.5 User Trials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
57 57 59 59 62 67 72 73 74 77 79 82 89
Contents
XI
5 “FRIEND” – An Intelligent Assistant in Daily Life O. Kouzmitcheva, C. Martens, A. Pape, H. She, I. Volosyak, A. Gr¨ aser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Basic Concepts and Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.1 The FRIEND Project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.2 Hardware Structure of FRIEND . . . . . . . . . . . . . . . . . . . . . . . 5.1.3 Multi-layered Control Architecture of FRIEND . . . . . . . . . . 5.2 Application and Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.1 The “Beverage Serving” Task . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.2 Obstacle Avoidance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.3 Task Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.4 Demonstration-Based Programming . . . . . . . . . . . . . . . . . . . . 5.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
95 95 95 96 97 100 101 109 111 119 124
6 GIVING-A-HAND System: The Development of a Task-Specific Robot Appliance M.J. Johnson, E. Guglielmelli, G.A. Di Lauro, C. Laschi, M.C. Carrozza, P. Dario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.1 Domotic-Robotic Integrated System . . . . . . . . . . . . . . . . . . . . 6.2.2 Localized System of Appliances . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Design Concept for the Giving-A-Hand System . . . . . . . . . . . . . . . . 6.4 Domotic/Telematic and Robotic Assistance . . . . . . . . . . . . . . . . . . . 6.5 The Fetch and Carry Robot Appliance Development . . . . . . . . . . . . 6.6 User-Centered Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.7 Prototype of a Local Network with the Robot Appliance . . . . . . . . 6.8 Summary and Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
127 127 128 129 130 132 133 133 135 138 140
7 Cooperative Welfare Robot System Using Hand Gesture Instructions Noriyuki Kawarazaki, Ichiro Hoya, Kazue Nishihara, Tadashi Yoshidome . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Cooperative Robot System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Measurement of Distance Using Stereo Images . . . . . . . . . . . . . . . . . 7.4 Detection of the Hand and the Target Object . . . . . . . . . . . . . . . . . . 7.4.1 Detection of the Hand Area Using Color Image . . . . . . . . . . 7.4.2 Tracking of the Hand Using CP . . . . . . . . . . . . . . . . . . . . . . . . 7.4.3 Detection of the Object Using Gesture Instruction . . . . . . . 7.5 Recognition of the Hand Gesture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
143 143 144 145 146 146 147 148 149 150 152
XII
Contents
8 Selectable Operating Interfaces of the Meal-Assistance Device “My Spoon” Ryoji Soyama, Sumio Ishii, Azuma Fukase . . . . . . . . . . . . . . . . . . . . . . . . . 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2 Meal-Assistance Device “My Spoon” . . . . . . . . . . . . . . . . . . . . . . . . . 8.3 Operating Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4 Basic Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4.1 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4.2 Compartment Selection Command Set . . . . . . . . . . . . . . . . . . 8.4.3 Position Adjustment Command Set . . . . . . . . . . . . . . . . . . . . 8.5 Control Modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5.1 Manual Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5.2 Semi-automatic Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5.3 Automatic Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.6 Future Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.6.1 Food Recognition by Using Color Image Processing . . . . . . 8.6.2 Improvements in Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
155 155 155 156 156 157 158 158 159 159 159 160 161 161 162 163
9 Enhancing the Usability of the MANUS Manipulator by Using Visual Servoing A.H.G. Versluis, B.J.F. Driessen, J.A. van Woerden . . . . . . . . . . . . . . . . 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2 Visual Servoing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2.1 Vision Aspects of the Visual Servoing . . . . . . . . . . . . . . . . . . 9.3 Control Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4 Vision System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4.1 Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4.2 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5 Stability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.6 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.7 Conclusions and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
165 165 167 168 168 169 169 170 171 171 174
Part II Rehabilitation Robots for Assistance of Human Movements II.2 Safety Issues of the Rehabilitation Robots 10 A Safety Strategy for Rehabilitation Robots Makoto Nokata, Noriyuki Tejima . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2 Principles of Safety Standards for Robots . . . . . . . . . . . . . . . . . . . . . 10.2.1 Framework of New Safety Standards for Robots . . . . . . . . . 10.2.2 Safety Standard for Machinery . . . . . . . . . . . . . . . . . . . . . . . . 10.2.3 Risk Assessment Process and Risk Reduction . . . . . . . . . . . . 10.2.4 Tolerable Risks for Robots . . . . . . . . . . . . . . . . . . . . . . . . . . . .
177 177 177 177 178 179 180
Contents
XIII
10.3 Case Study on Safety of Rehabilitation Robots . . . . . . . . . . . . . . . . 10.3.1 Risk Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3.2 Safety Measures of Risk Reduction . . . . . . . . . . . . . . . . . . . . . 10.3.3 Benefit Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.4 Proposal of Risk Assessment Guideline for Rehabilitation Robots 10.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
181 182 183 183 184 185
11 Safety Evaluation Method of Rehabilitation Robots Makoto Nokata, Koji Ikuta, Hideki Ishii . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Safety Strategy for Human-Care Robots . . . . . . . . . . . . . . . . . . . . . . 11.2.1 Injury to Humans from Human-Care Robots . . . . . . . . . . . . 11.2.2 Classification of Safety Strategies . . . . . . . . . . . . . . . . . . . . . . 11.3 Proposing Evaluation Measures of Safety . . . . . . . . . . . . . . . . . . . . . . 11.3.1 Necessity of Safe Quantitative Evaluation . . . . . . . . . . . . . . . 11.3.2 Selection of Evaluation Measures . . . . . . . . . . . . . . . . . . . . . . . 11.4 General Evaluation Method Using Evaluation Measures . . . . . . . . . 11.5 Deriving Danger-Indexes of Safety Strategy . . . . . . . . . . . . . . . . . . . 11.5.1 Safety Design Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.5.2 Safety Control Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.6 Proposal of Design Optimization and Practical Examples . . . . . . . 11.6.1 Formulating the Design Optimization Method . . . . . . . . . . . 11.6.2 Maximizing Safety Under Fixed Cost . . . . . . . . . . . . . . . . . . . 11.6.3 A New Method of Calculate a Safe Approach Motion . . . . . 11.7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
187 187 187 187 188 189 189 189 190 192 192 193 194 194 195 196 197
12 Risk Reduction Mechanisms for Safe Rehabilitation Robots Noriyuki Tejima . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2 Tolerable Risk and Surface Injury . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3 Force Limitation Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.4 A Straight Movement-Type Force Limitation Mechanism . . . . . . . . 12.5 A Three-Dimensional Force Limitation Mechanism . . . . . . . . . . . . . 12.6 Reflex Mechanism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
199 199 199 201 202 204 206 207
Part II Rehabilitation Robots for Assistance of Human Movements II.3 Rehabilitation-Robot Evaluation 13 Usability of an Assistive Robot Manipulator: Toward a Quantitative User Evaluation Bessam Abdulrazak, Mounir Mokhtari, Bernard Grandjean . . . . . . . . . . . 211 13.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 13.2 Users Needs Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
XIV
Contents
13.3 Hardware and Software Organization . . . . . . . . . . . . . . . . . . . . . . . . . 13.3.1 Hardware Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3.2 Software Command Architecture . . . . . . . . . . . . . . . . . . . . . . . 13.4 Quantitative Evaluation Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.5 Preliminary Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.5.1 Modes and Time of Use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.5.2 Actions Number . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
212 213 214 215 215 216 217 219 219
14 Processes for Obtaining a “Manus” (ARM) Robot within The Netherlands GertWillem R¨ omer, Harry Stuyt, Geer Peters, Koos van Woerden . . . . 14.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.2 Wheelchair Mounted Service Manipulator ARM . . . . . . . . . . . . . . . 14.3 The Current Process of Providing an ARM to a User . . . . . . . . . . . 14.3.1 Informing Users about the Benefits of the ARM . . . . . . . . . 14.3.2 Indication Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.3.3 Stand-Alone Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.3.4 Formal Application and Funding of an ARM . . . . . . . . . . . . 14.3.5 Mounting the ARM on the Wheelchair . . . . . . . . . . . . . . . . . 14.3.6 Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.3.7 Service and Maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.4 The Future Process of Prescribing the ARM . . . . . . . . . . . . . . . . . . . 14.5 Summary of Two Recent Dutch Arm-User Evaluations . . . . . . . . . 14.5.1 User Study Conducted by iRV . . . . . . . . . . . . . . . . . . . . . . . . . 14.5.2 User Study Conducted by hetDorp . . . . . . . . . . . . . . . . . . . . .
221 221 221 223 224 224 224 226 226 226 227 227 228 228 229
Part III Prostheses and Orthoses 15 Experimental Analysis of the Proprioceptive and Exteroceptive Sensors of an Underactuated Prosthetic Hand M. Zecca, G. Cappiello, F. Sebastiani, S. Roccella, F. Vecchi, M.C. Carrozza, P. Dario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.2 Mechanical Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.3 Sensory System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.4 Materials and Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.4.1 Slider Position Sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.4.2 Tendon Tensiometer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.4.3 Thumb Position Sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.4.4 Force Sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
233 233 234 235 236 236 237 239 240 241
Contents
16 Design and Testing of WREX Tariq Rahman, Whitney Sample, Rahamim Seliktar . . . . . . . . . . . . . . . . . 16.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16.2 Design of WREX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16.3 Gravity Balancing With x "= 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16.4 Clinical Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
XV
243 243 244 245 248 248
Part IV Intelligent Wheelchairs 17 A Concept for Control of Indoor-Operated Autonomous Wheelchair Dimitar Stefanov, Alexander Avtanski, Z. Zenn Bien . . . . . . . . . . . . . . . . 17.1 Introduction and Related Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.1.1 Methods for Navigation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.1.2 Path Planning and Navigation to the Goal . . . . . . . . . . . . . . 17.2 Conception of Wheelchair Navigation . . . . . . . . . . . . . . . . . . . . . . . . . 17.2.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.2.2 Initial Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.3 Localization of the Wheelchair Position . . . . . . . . . . . . . . . . . . . . . . . 17.4 Scenario of the Wheelchair Control . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.5 Navigation System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.6 Computer Simulation of the Control Algorithm . . . . . . . . . . . . . . . . 17.6.1 Wheelchair Kinematics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.6.2 Modeling of the Sensors and Their Arrangement on the Wheelchair Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.6.3 Navigation Algorithm of the Simulator . . . . . . . . . . . . . . . . . 17.7 Evaluation of the Control Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . 17.7.1 Navigation to Multiple Goals . . . . . . . . . . . . . . . . . . . . . . . . . . 17.7.2 Obstacle Avoidance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.7.3 Avoiding a “Trap” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.7.4 Navigation in a Complex Environment . . . . . . . . . . . . . . . . . . 17.7.5 Route Generation in Partially Known Environment . . . . . . 17.8 Future Plans and Concluding Remark . . . . . . . . . . . . . . . . . . . . . . . . 18 Design of an Intelligent Wheelchair for the Motor Disabled Chong Hui Kim, Jik Han Jung, Byung Kook Kim . . . . . . . . . . . . . . . . . . . 18.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.2 Related Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.3 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.4 System Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.4.1 Hardware Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.4.2 Software Design for Real-Time System . . . . . . . . . . . . . . . . . .
253 253 254 256 258 258 258 260 262 264 267 267 269 272 283 283 284 286 288 292 294
299 299 300 301 302 302 303
XVI
Contents
18.5 Navigation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.5.1 Localization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.5.2 Hierarchical Control Architecture . . . . . . . . . . . . . . . . . . . . . . 18.6 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
304 304 306 307 309
Part V Mechatronics Devices for Assistance in Walking 19 Electrically Assisted Walker with Supporter-Embedded Force-Sensing Device Saku Egawa, Ikuo Takeuchi, Atsushi Koseki, Takeshi Ishii . . . . . . . . . . . . 19.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.2 Electrically Assisted Walker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.3 Supporter-Embedded Force Sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.3.1 Requirements for the Force Sensor . . . . . . . . . . . . . . . . . . . . . 19.3.2 Sensor Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.3.3 Sensing Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.3.4 Advantages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
313 313 314 315 315 316 317 318 319 321 321
20 Human-Friendly Care Robot System for the Elderly Dong Hyun Yoo, Hyun Seok Hong, Han Jo Kwon, Myung Jin Chung . . 20.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.1.1 The Functions of Do-u-mi Robot . . . . . . . . . . . . . . . . . . . . . . . 20.2 Overall System of Do-u-mi Robot . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.3 Sound Localization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.4 Face Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.4.1 Face Candidate Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.5 Autonomous Navigation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
323 323 323 325 326 327 328 330 331
21 Newly Designed Rehabilitation Robot System for Walking-Aid Choon-Young Lee, Kap-Ho Seo, Changmok Oh, Ju-Jang Lee . . . . . . . . . . 21.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.2 Electric Motor Based Gait Rehabilitation System . . . . . . . . . . . . . . 21.2.1 System Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.2.2 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.3 Newly Developed Gait Rehabilitation System . . . . . . . . . . . . . . . . . . 21.3.1 System Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.3.2 Control Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
333 333 334 334 336 338 338 340 342
Contents
XVII
Part VI Robot-Assisted Neurorehabilitation 22 A Gentle/S Approach to Robot Assisted Neuro-Rehabilitation Rui Loureiro, Farshid Amirabdollahian, William Harwin . . . . . . . . . . . . . 22.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.2 Background to Stroke . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.3 Gentle/S . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.3.1 Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.4 Clinical Prototype for Machine Mediated Neurorehabilitation . . . . 22.4.1 Antigravity Mechanism for the Shoulder and Elbow . . . . . . 22.4.2 Exercises & Movement Guidance . . . . . . . . . . . . . . . . . . . . . . . 22.4.3 Different Therapy Modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.5 Clinical Trials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.5.1 Outcome Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.5.2 Data Analysis and Statistical Methodology . . . . . . . . . . . . . . 22.5.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
347 347 348 349 350 351 353 354 357 357 358 359 360 361
23 Wire Driven Robots for Rehabilitation Paolo Gallina . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23.1.1 Advantages of Wire Driven Robots . . . . . . . . . . . . . . . . . . . . . 23.1.2 Problems Related to Wire Driven Robots . . . . . . . . . . . . . . . 23.2 Manipulability and Wire Tension Computation . . . . . . . . . . . . . . . . 23.3 NeRebot: An Example of Wire Driven Robot for Rehabilitation . . 23.3.1 Software and Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23.3.2 Treatment Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23.4 Conclusions and Future Research . . . . . . . . . . . . . . . . . . . . . . . . . . . .
365 365 366 367 367 369 372 373 374
24 A Wrist Extension for MIT-MANUS Hermano Igo Krebs, James Celestino, Dustin Williams, Mark Ferraro, Bruce Volpe, Neville Hogan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24.2 Specification for a New Wrist Device . . . . . . . . . . . . . . . . . . . . . . . . . 24.2.1 Kinematic Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24.2.2 Actuator Placement and Transmission Selection . . . . . . . . . 24.2.3 Actuator Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24.2.4 Sensor Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24.3 Alpha-Prototype Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24.4 Robotic Therapy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
377 377 380 381 382 382 383 383 386 388
XVIII Contents
25 Post Stroke Shoulder–Elbow Physiotherapy with Industrial Robots Andr´ as T´ oth, Guszt´ av Arz, G´ abor Fazekas, Daniel Bratanov, Nikolay Zlatov . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25.2 Analysis of Spastic Upper Limb Physiotherapy . . . . . . . . . . . . . . . . 25.3 System Design and Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25.3.1 Mechanical Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25.3.2 The Instrumented Orthoses . . . . . . . . . . . . . . . . . . . . . . . . . . . 25.3.3 Control Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25.3.4 User Interface and Programming . . . . . . . . . . . . . . . . . . . . . . . 25.3.5 Safety Measures and Devices . . . . . . . . . . . . . . . . . . . . . . . . . . 25.4 Testing and Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25.5 Clinical Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25.5.1 Subjects of the Clinical Trial . . . . . . . . . . . . . . . . . . . . . . . . . . 25.5.2 Assessment Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25.5.3 Analysis of Assessment Results . . . . . . . . . . . . . . . . . . . . . . . . 25.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
391 391 392 394 394 399 399 401 402 403 405 405 406 408 409
26 STRING-MAN: A Novel Wire-Robot for Gait Rehabilitation Dragoljub Surdilovic, Rolf Bernhardt, Tobias Schmidt and Jinyu Zhang 26.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26.2 Development Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26.3 Robotic Mechanisms Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26.4 Human/Robot Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26.5 Sensory Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26.6 Control Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
413 413 414 414 419 420 421 424
Part VII Perspectives and Trends of the Rehabilitation Robotics 27 Great Expectations for Rehabilitation Mechatronics in the Coming Decade H.F. Machiel Van der Loos, Richard Mahoney, Chantal Ammi . . . . . . . . 27.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27.2 Emerging Demographics and Healthcare Trends . . . . . . . . . . . . . . . . 27.3 Emerging Technologies Relevant to Robotics . . . . . . . . . . . . . . . . . . 27.4 RoadBlocks and Enablers of Robotic Applications in Rehabilitation 27.5 Mechatronic/Robotic Applications to Rehabilitation . . . . . . . . . . . . 27.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
427 427 428 429 431 432 432
Subject Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 435 Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 439 About the Editors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441
List of Contributors
Abdulrazak, Bessam GET/Institut National des T´el´ecomunications–INSERM U.483, Handicom Lab. Evry, France
[email protected] Amirabdollahian, Farshid The University of Newcastle, C.R.E.S.T., Stephenson Building, Claremont Road, Newcastle Upon Tyne, NE1 7RU, UK,
[email protected] Ammi, Chantal Dept. of Business Administration, National Telecommunications Institute, 9, rue Charles Fournier, 91011 Evry, France,
[email protected] av Arz, Guszt´ Budapest University of Technology and Economics, Department of Manufacturing Engineering, Egry J. u. 1. Budapest 1111, Hungary
[email protected] Avtanski, Alexander Savvion Inc., 5104 Old Ironsides Dr, Santa Clara, CA 95054, USA
[email protected] Bernhardt, Rolf Fraunhofer Institute IPK-Berlin, Department Robotics and Automation, Pascalstraße 8-9, 10587 Berlin, Germany
Bien, Z. Zenn Department of Electrical Engineering and Computer Science, KAIST, 373-1 Guseong-dong, Yuseong-gu, Daejeon 305-701, Republic of Korea
[email protected]
Bratanov, Daniel University of Rousse, Department of Manufacturing Engineering, Automation & Robotics Laboratory, 8 Studentska str. 7017, Rousse, Bulgaria
[email protected]
Cappiello, Giovanni Centro INAIL RTR, Via Vetraia 7, 55049 Viareggio (LU), Italy
[email protected]
Carrozza, Maria Chiara ARTS Lab, Scuola Superiore Sant’Anna, Polo Sant’Anna Valdera, viale Rinaldo Piaggio, 34-56025 Pontedera (PI), Italy; Research Centre on Rehabilitation Bioengineering of the Centro Protesi INAIL, Vigorso di Budrio (Bologna), via della Vetraia 7, 55049 Viareggio (Lucca), Italy
[email protected]
XX
List of Contributors
Celestino, James Department of Mechanical Engineering, Massachusetts Institute of Technology, Room 3-173, 77 Massachusetts Ave, Cambridge, MA 02139, USA
[email protected] Chung, Myung Jin Dept. of Electrical Engineering and Computer Science, KAIST, 373-1, Guseong-dong, Yuseong-gu, Daejeon, 305-701, Korea
[email protected] Dario, Paolo ARTS Lab, Scuola Superiore Sant’Anna, Polo Sant’Anna Valdera, viale Rinaldo Piaggio, 34-56025 Pontedera (PI), Italy; Research Centre on Rehabilitation Bioengineering of the Centro Protesi INAIL, Vigorso di Budrio (Bologna), via della Vetraia 7, 55049 Viareggio (Lucca), Italy
[email protected] Di Lauro, G.A. Scuola Superiore Sant’Anna, ARTS (Advanced Robotics Technology Systems) Lab, c/o Polo Sant’Anna Valdera, Viale Rinaldo Piaggio 34, 56025 Pontedera (Pisa), Italy; Research Centre on Rehabilitation Bioengineering of the Centro Protesi INAIL, Vigorso di Budrio (Bologna), via della Vetraia 7, 55049 Viareggio (Lucca), Italy
[email protected] Driessen, B.J.F. TNO TPD, PO-BOX 155, 2600 AD Delft, The Netherlands,
[email protected] Egawa, Saku Mechanical Engineering Research
Laboratory, Hitachi, Ltd., 502 Kandatsu, Tsuchiura, Ibaraki 300–0013, Japan
[email protected] Fazekas, G´ abor National Institute for Medical Rehabilitation, Szanatorium u. 19 Budapest 1528 Hungary
[email protected] Ferraro, Mark Burke Rehabilitation Hospital, 785 Mamaroneck Avenue, White Plains, NY 10605, USA
[email protected] Feki, M.A. GET/Institut National des T´el´ecomunications–INSERM U.483, Handicom Lab. Evry, France
[email protected] Fukase, Azuma Intelligent Systems Laboratory, SECOM Co., Ltd., R& D Center, 8-10-16, Shimorenjaku, Mitaka, Tokyo 181–8528, Japan
[email protected] Gallina, Paolo Department of Energetics, University of Trieste, Trieste, via A. Valerio 10, 34127 Trieste (Italy)
[email protected] Gr¨ aser, Axel Institute of Automation, University of Bremen, Otto Hahn Allee NW1, 28359 Bremen, Germany,
[email protected] Grandjean, Bernard GET/Institut National des T´el´ecomunications–INSERM U.483, Handicom Lab. Evry, France
[email protected]
List of Contributors
XXI
Guglielmelli, E. Scuola Superiore Sant’Anna, ARTS (Advanced Robotics Technology Systems) Lab, c/o Polo Sant’Anna Valdera, Viale Rinaldo Piaggio 34, 56025 Pontedera (Pisa), Italy; Research Centre on Rehabilitation Bioengineering of the Centro Protesi INAIL, Vigorso di Budrio (Bologna), via della Vetraia 7, 55049 Viareggio (Lucca), Italy
[email protected]
of Technology, 1030 Shimo-ogino, Atsugi-shi, Kanagawa, 243-0292, Japan
Harwin, William The University of Reading, School of Systems, Engineering, Department of Cybernetics, Whiteknights, Reading, RG6 6AY, UK
[email protected]
Ishii, Sumio Intelligent Systems Laboratory, SECOM Co., Ltd., R& D Center, 8-10-16, Shimorenjaku, Mitaka, Tokyo 181–8528, Japan
[email protected]
Hillman, Michael Bath Institute of Medical Engineering, Wolfson Centre, Royal United Hospital, Bath BA1 3NG, UK
[email protected]
Ishii, Takeshi Mechanical Engineering Research Laboratory, Hitachi, Ltd., 502 Kandatsu, Tsuchiura, Ibaraki 300–0013, Japan
[email protected]
Hogan, Neville Department of Mechanical Engineering, Massachusetts Institute of Technology, Room 3-173, 77 Massachusetts Ave, Cambridge, MA 02139, USA; Brain and Cognitive Sciences Dept., Massachusetts Institute of Technology, 77 Massachusetts Ave, Cambridge, MA 02139, USA
[email protected] Hong, Hyun Seok Dept. of Electrical Engineering and Computer Science, KAIST, 373-1, Guseong-dong, Yuseong-gu, Daejeon, 305-701, Korea
[email protected] Hoya, Ichiro Department of Welfare Systems Engineering, Kanagawa Institute
Ikuta, Koji Department of Micro System Engineering, Graduate School of Engineering, Nagoya University
[email protected] Ishii, Hideki Department of Micro System Engineering, Graduate School of Engineering, Nagoya University
[email protected]
Johnson, M.J. Scuola Superiore Sant’Anna, ARTS (Advanced Robotics Technology Systems) Lab, c/o Polo Sant’Anna Valdera, Viale Rinaldo Piaggio 34, 56025 Pontedera (Pisa), Italy; Research Centre on Rehabilitation Bioengineering of the INAIL Centro Protesi, Vigorso di Budrio (Bologna), via della Vetraia 7, 55049 Viareggio (Lucca), Italy
[email protected] Jung, Jik Hang Department of Electrical Engineering & Computer Science, KAIST. 373-1 Guseong-dong, Yuseong-gu, Daejeon, 305-701, Korea
[email protected]
XXII
List of Contributors
Jung, Jin-Woo Department of Electrical Engineering and Computer Science, KAIST, 373-1 Guseong-dong, Yuseong-gu, Daejeon 305-701, Republic of Korea
[email protected] Kawarazaki, Noriyuki Department of Welfare Systems Engineering, Kanagawa Institute of Technology, 1030 Shimo-ogino, Atsugi-shi, Kanagawa, 243-0292, Japan
[email protected] Kim, Byung Kook Department of Electrical Engineering & Computer Science, KAIST, 373-1 Guseong-dong, Yuseong-gu, Daejeon, 305-701, Korea
[email protected] Kim, Chong Hui Department of Electrical Engineering & Computer Science, KAIST, 373-1 Guseong-dong, Yuseong-gu, Daejeon, 305-701, Korea,
[email protected] Kim, Dae-Jin Department of Electrical Engineering and Computer Science, KAIST, 373-1 Guseong-dong, Yuseong-gu, Daejeon 305-701, Republic of Korea
[email protected] Koseki, Atsushi Mechanical Engineering Research Laboratory, Hitachi, Ltd., 502 Kandatsu, Tsuchiura, Ibaraki 300–0013, Japan
[email protected] Kouzmitcheva, Olena Institute of Automation, University of Bremen, Otto-Hahn-Allee, NW1, 28359 Bremen, Germany,
[email protected]
Krebs, Hermano Igo Massachusetts Institute of Technology, Department of Mechanical Engineering, 77 Massachusetts Ave, 3-137, Cambridge, MA 02139 USA; Weill Medical College of Cornell University, The Winifred Masterson Burke Medical Research Institute, Department of Neurology and Neuroscience
[email protected] Kwon, Han Jo Dept. of Electrical Engineering and Computer Science, KAIST, 373-1, Guseong-dong, Yuseong-gu, Daejeon, 305-701, Korea
[email protected] Laschi, C. Scuola Superiore Sant’Anna, ARTS (Advanced Robotics Technology Systems) Lab, c/o Polo Sant’Anna Valdera, Viale Rinaldo Piaggio 34, 56025 Pontedera (Pisa), Italy
[email protected] Lee, Choon-Young Digital Content Research Division, ETRI, 161 Gajeong-dong Yuseong-gu, Daejeon, 305-350 Korea
[email protected] Lee, Ju-Jang Department of Electrical Engineering and Computer Science, KAIST, 373-1 Guseong-dong, Yuseong-gu, Daejeon 305-701, Republic of Korea
[email protected] Loureiro, Rui The University of Reading, School of Systems, Engineering, Department of Cybernetics, Whiteknights,Reading, RG6 6AY, UK
[email protected]
List of Contributors XXIII Mahoney, Richard Rehabilitation Technology Division, Applied Resources Inc., 1275 Bloomfield Ave., Fairfield, NJ, 07004 USA
[email protected] Martens, Christian RHEINMETALL-DEFENCEELECTRONICS, Br¨ uggeweg 54, 28309 Bremen, Germany,
[email protected] Mokhtari, Mounir GET/Institut National des T´el´ecomunications–INSERM U.483, Handicom Lab. Evry, France
[email protected] Nishihara, Kazue Department of Welfare Systems Engineering, Kanagawa Institute of Technology, 1030 Shimo-ogino, Atsugi-shi, Kanagawa, 243-0292, Japan
[email protected]
Guseong-dong, Yuseong-gu, Daejeon 305-701, Republic of Korea
[email protected] Peters, Geer RTD hetDorp, Heijenoordseweg 130, NL-6813 GC, Arnhem, The Netherlands,
[email protected] Rahman, Tariq A.I. duPont Hospital for Children, 1600 Rockland Rd, Wilmington, DE 19899
[email protected] Roccella, Stefano Centro INAIL RTR, Via Vetraia 7, 55049 Viareggio (LU), Italy
[email protected] R¨ omer, Gert Willem Exact Dynamics, Edisonstraat 96, NL-6942 PZ, Didam, The Netherlands,
[email protected]
Nokata, Makoto Department of Robotics, Faculty of Science and Engineering, Ritsumeikan University
[email protected]
Sample, Whitney A.I. duPont Hospital for Children, 1600 Rockland Rd, Wilmington, DE 19899
[email protected]
Oh, Changmok Department of Electrical Engineering and Computer Science, KAIST, 373-1 Guseong-dong, Yuseong-gu, Daejeon 305-701, Republic of Korea
[email protected]
Schmidt, Tobias Fraunhofer Institute IPK-Berlin, Department Robotics and Automation, Pascalstraße 8-9, 10587 Berlin, Germany
Pape, Andreas Robert BOSCH GmbH, Gasoline Systems, GS/EFA3, Postfach 30 02 40, 70442 Stuttgart,
[email protected] Park, Kwang-Hyun Department of Electrical Engineering and Computer Science, KAIST, 373-1
Sebastiani, Francesco Centro INAIL RTR, Via Vetraia 7, 55049 Viareggio (LU), Italy
[email protected] Seliktar, Rahamim School of Biomedical Engineering, Drexel University, 32 and Chestnut Sts, Philadelphia PA 19104
[email protected]
XXIV List of Contributors Seo, Kap-Ho Department of Electrical Engineering and Computer Science, KAIST, 373-1 Guseong-dong, Yuseong-gu, Daejeon 305-701, Republic of Korea
[email protected]
Tejima, Noriyuki Department of Robotics, Faculty of Science and Engineering, Ritsumeikan University
[email protected]
She, Haiying Institute of Automation, University of Bremen, Otto-Hahn-Allee, NW1, 28359 Bremen, Germany,
[email protected]
T´ oth, Andr´ as Budapest University of Technology and Economics, Department of Manufacturing Engineering, Egry J. u. 1. Budapest 1111 Hungary
[email protected]
Soyama, Ryoji SECOM Co., Ltd., Intelligent Systems Laboratory, Medical Welfare Division, 8-10-16 Shimorenjaku, Mitaka, Tokyo 181–8528, Japan
[email protected] Stefanov, Dimitar Cardiff & Vale NHS Trust, Rehabilitation Engineering Unit, Cardiff, CF5 2YN, UK; Institute of Mechanics, Bulgarian Academy of Sciences, Acad. G. Bonchev Street, Block 4, 1113 Sofia, Bulgaria D
[email protected] Stuyt, Harry Exact Dynamics BV, Edisonstraat 96, NL-6942 PZ, Didam, The Netherlands
[email protected] Surdilovic, Dragoljub Fraunhofer Institute IPK-Berlin, Department Robotics and Automation, Pascalstraße 8-9, 10587 Berlin, Germany
[email protected] Takeuchi, Ikuo Mechanical Engineering Research Laboratory, Hitachi, Ltd., 502 Kandatsu, Tsuchiura, Ibaraki 300–0013, Japan
[email protected]
Van der Loos, H.F. Machiel, Rehabilitation R& D Center, VA Palo Alto Health Care System, 3801 Miranda Ave. # 153, Palo Alto, CA, 94304 USA
[email protected] van Woerden, J.A. TNO TPD, PO-BOX 155, 2600 AD Delft, The Netherlands,
[email protected] van Woerden, Koos TNO TPD, Stieltjesweg 1, NL-2628 CK, Delft, The Netherlands,
[email protected] Vecchi, Fabrizio Centro INAIL RTR, Via Vetraia 7, 55049 Viareggio (LU), Italy
[email protected] Versluis, A.H.G. TNO TPD, PO-BOX 155, 2600 AD Delft, The Netherlands,
[email protected] Volosyak, Ivan Institute of Automation, University of Bremen, Otto-Hahn-Allee, NW1, 28359 Bremen, Germany,
[email protected]
List of Contributors Volpe, Bruce Department of Neurology and Neuroscience, Weill Medical College Cornell University, Burke Medical Research Institute, 785 Mamaroneck Avenue, White Plains, NY 10605, USA; Burke Rehabilitation Hospital, 785 Mamaroneck Avenue, White Plains, NY 10605, USA
[email protected] Yoo, Dong Hyun Dept. of Electrical Engineering and Computer Science, KAIST, 373-1, Guseong-dong, Yuseong-gu, Daejeon, 305-701, Korea
[email protected] Yoshidome, Tadashi Department of Welfare Systems Engineering, Kanagawa Institute of Technology, 1030 Shimo-ogino, Atsugi-shi, Kanagawa, 243-0292, Japan
[email protected]
XXV
Williams, Dustin Interactive Motion Technologies, Inc., 56 Highland Ave, Cambridge, MA 02139, USA
[email protected] Zecca, Massimiliano Scuola Superiore Sant’Anna, ARTS Labs, Polo Sant’Anna Valdera, Viale Rinaldo Piaggio 34, 56025 Pontedera (Pisa), Italy
[email protected] Zhang, Jinyu Fraunhofer Institute IPK-Berlin, Department Robotics and Automation, Pascalstraße 8-9, 10587 Berlin, Germany Zlatov, Nikolay Cardiff University, Manufacturing Engineering Centre, The Parade P O Box 925 Cardiff CF24 0YF Wales, UK
[email protected]
List of Abbreviations
ACC ADL AGW ARM BWS COP DLS DoF DoFs EMG FMMNN FSR HMM ICORR LNA LRF LPM QOL ROL RR RRs RTAI sls TOD
active compliance control activities of daily living automatically guided wheelchairs assistive robotic manipulator body weight support center of pressure double limb support degree of freedom degrees of freedom electromyography, electromyographic Fuzzy Min-Max Neural Networks force sensing resistors Hidden Markov Model International Conference on Rehabilitation Robotics low noise amplifier laser range finder log-polar mapping quality of life respect of living rehabilitation robot rehabilitation robots Real-Time Application Interface single limb support task-oriented design
1 Advances in Human-Friendly Robotic Technologies for Movement Assistance/Movement Restoration for People with Disabilities Dimitar Stefanov and Z. Zenn Bien
Abstract Rehabilitation robots (RR) are expected to play an important role toward the independent life of older persons and persons with disabilities. Such intelligent devices, embedded into the home environment, can provide the resident with 24-hour movement assistance. Modern home-installed robots tend to be not only physically versatile in functionality but also emotionally human-friendly, i.e. they may be able to perform their functions without disturbing the user and without causing him/her any pain, inconvenience, or movement restriction, instead possibly providing him/her with comfort and pleasure. This paper analyzes the main categories of RR. The paper will then discuss some important issues of the future development of an intelligent residential space with a human-friendly rehabilitation robots integrated with it.
1.1 Introduction Recent statistics show a trend of rapid growth in the number of persons with physical disabilities and aged people who need external help in their everyday movement tasks [1]. The problem of caring for older persons and persons with physical disabilities will become more serious in the near future when a significant part of the increasing global population is in the group of 65 years of age or over, and the existing welfare model is not capable to meet the increased needs. It is obvious that such a problem cannot be solved by increasing the number of care-givers. On the other hand, an optimistic view is growing in that the quality of life for the people with movement limitations can be significantly improved by means of various modern technologies, in particular, through rehabilitation robots [2, 3]. It is expected that rehabilitation robots (RR) will have a strong positive emotional impact on persons with physical disabilities and older persons, improving their quality of life, increasing their movement independence, and giving them privacy. The same notion will reduce to some extend the medical care costs per person. Z.Z. Bien and D. Stefanov (Eds.): Advances in Rehabilitation Robotics, LNCIS 306, pp. 3-23, 2004. © Springer-Verlag Berlin Heidelberg 2004
4
D. Stefanov and Z.Z. Bien
The idea of rehabilitation robots evolves with the recent development of relevant technology. While the initial conceptions for RR design were mainly concerned with a manipulator usually controlled in a direct mode, the latest design tendencies target to multi-agent robotic solutions where more than one home installed robots are integrated with other home installed devices. Whereas many service robots for ordinary non-impaired people easily meet requirements that are standardized to some extend, the problems and needs of older persons and persons with physical disabilities significantly vary from person to person, which add many serious requirements to the RR design. The robots for people with special needs should apply control algorithms based on a small number of commands relevant to the specifics of the user’s own motions. The technology solutions should also consider the individual’s habits and should meet all safety regulations. The development of sophisticated RR was strongly influenced by the recent fast progress of various intelligent technologies such as fuzzy logic, artificial neural net, and evolutionary algorithms. Thus, those RR’s are sometimes termed “intelligent RR”. The concept of RR for older persons and persons with physical disabilities has been considered important by many social and economic institutions in the advanced countries. In fact, numerous research and demonstration projects on RR have been already completed or are in the phase of development all over the world. Many of these projects are often funded by international R&D organizations and involve participants from different countries. Research activity on RR is relatively higher in Japan, Europe, and the USA, where there is a strong growth (by far) of the aged population along with the availability of broad high technology achievements. Some RRs are available on the market as commercial products and become an important helper for increasing number of patients, improving significantly their quality of life. The objective of this paper is to provide a summary of some recent works related to the notion of the RR for service of older persons and persons with physical disabilities. The subject matter lies in the interdisciplinary area of many different branches of science and technology, and its successful design can be possible only from the result of joint efforts of many specialists in different areas. We find that the technologies and solutions for RR should be human friendly, i.e. the RR should be designed according to the notion of human-centeredness and should possess a high level of intelligence in their control, actions, and interactions with the users, offering them a high level of comfort and functionality. In that chapter we have included some ideas and tendencies that have been proposed and developed recently. Comments on the RR research projects, products, and conceptions are done with a focus on technology innovations that have been taking place. This chapter does not treat non-technical components of the RR design such as interior design, organization of medical care and care-giver servicing, maintenance, etc. This chapter is organized as follows. In Sect. 1.2, we give a brief classification of the RR regarding their application and introduce some recent projects and commercially-available RR products. Next, an overview on the place of the RR in the arrangement of some smart home projects is given. Our vision on the futuristic robotic smart house is also presented.
1 Advances in Human-Friendly Robotic Technologies
5
1.2 Areas of the RR Application Rehabilitation robots are intended to meet different aspects of user’s needs. Although the area of the rehabilitation robotics is relatively new, its intensive development during the last 20 years formed several different domains. 1.2.1 Robotic Systems for Movement Assistance Most of the rehabilitation robots are designed to assist disabled individuals in their everyday movement needs, such as eating, drinking, objects replacement, etc. [4]. Currently, three main types of robot schemes are in operation: desktop-mounted robots (workstations), wheelchair-mounted robots, and mobile autonomous robots. In some simple applications, the robot is fixed to a desk or to the floor. The operator is located in a suitable position near the worktable and controls the robot that performs unaided pick-and-place ADL tasks [5]. Wheelchair-mounted robots [6, 7] can be used both for indoor and for outdoor assistance for the user. The attachment of a robot to the wheelchair significantly increases the movement independence of the user because one can move freely to different location of the house and can perform manipulative tasks in each position with the help of the rehabilitation robot. The drawbacks of such a solution are inclination of the wheelchair due to the weight of the robot, enlargement of the wheelchair width (which is critical for narrow door passage), and changes of the dynamic characteristics of the wheelchair. The mobile robots are remotely controlled devices that navigate autonomously through the home environment and serve the user who is located at a certain position (bed, chair, etc.). The cognitive load of a user can be reduced considerably if the robot automatically performs repetitive movements (in a pre-programmed mode of control). Programs can be successfully executed if the robot, the user, and the manipulated objects remain in the same initial position every time when the concrete task is performed. In the case of a wheelchair-mounted manipulator, the relative position of the user with respect to the manipulator remains the same, but the relative position between the manipulator and the objects may depend on the wheelchair position. In order to avoid this problem, a technique of vision-based automatic navigation of the gripper is adopted to handle the grasped object (KARES I, KARES II, TAURO, etc.) [8–10], or the user performs the end-point control in which the trajectory, orientation, and velocity of the gripper are directly adjusted (HOPE) [5]. Automatically guided wheelchairs (AGW) (known also as “go-to-goal wheelchairs”) are intended to facilitate transportation of individuals with severe dexterity limitations in their hands. Because of their automatic guidance in the home environment, these wheelchairs are often considered a special class of mobile rehabilitation robots for the transportation of a user. Most of the devices of that category are oriented to application in indoor environment but recently some results from design of outdoor autonomous wheelchairs were reported [11]. Different from the standard wheelchairs where the user directly controls the wheelchair movement, AGW
6
D. Stefanov and Z.Z. Bien
autonomously navigate toward the goal. After receiving the user’s instruction about the destination point of the wheelchair, the navigation system first generates the travel routine and then independently steers the wheelchair to the desired position. The automatic control of go-to-goal wheelchairs dramatically reduces the cognitive load of the user and makes possible the safe movement through narrow doors and corridors. In the most wheelchair projects, wheelchair movement is often assumed to occur in a structured or semi-structured home environment. Localization of the current wheelchair position is based either on fixed-location beacons strategically placed at pre-defined locations within the operating home environment [12, 13] or on natural landmarks of the structured environment [14]. Beacon-based systems can be further grouped into systems that refer to active beacons (most often fixed activebeacon emitters installed on the walls) and systems which get navigational parameters from passive targets [12]. Usually, passive beacons are of low cost and can be easily attached to the walls, but the procedures of detecting such markers (typically CCD sensors) and extracting the coded information are rather complicated. The localization systems that are based on active-beacon emitters typically involve simple sensors to identify the beacon positions and apply a simple information decoding procedure. However, such a solution does not allow flexible change of the beacon positions because each sensor should be separately powered and controlled. The guidepath systems for wheelchair navigation can be considered as a special class of beacon-based systems where the guide tracks are embedded in the floor. The magnetic-tape guidepath scheme [15] involves a strip of flexible magnetic material attached to the floor surface. The magnetic stripe follower is based on an array of fluxgate or Hall effect sensors, mounted on the board of the vehicle. Although the approach is widely used in many material-handling applications, its use for wheelchair guidance in the home environment is limited due to the complexity of the movement routines and the need for frequent reconfiguration of the path. An additional limitation comes from the requirement of embedment of the guidepath in the floor. The natural landmark navigation does not require installation of special beacons and the algorithms allow faster adaptation of the wheelchair to the unknown home environment. The proposed navigation solutions vary from detection of the location of ceiling mounted lamps [16] to detection of doorframes and furniture edges [14, 17]. New travel routines can be easily added to the computer memory without assistance of a specialized staff. On the other hand, the solutions based on natural landmarks require much complicated vision sensors, involve complex algorithms for analysis of the visual scene, and entail detection of artifacts from ambient lights. In order to achieve successful wheelchair navigation in case of absence or malfunctioning of some beacons/landmarks, most of the control algorithms identify the current wheelchair location not only by the information from the beacon navigation system but also by the information on the angular position of driving wheels (dead-reckoning procedure). Apart from the automatic navigation to the goal, most of the automatically guided wheelchairs can automatically perform obstacle avoidance maneuvers, referring to the information from range sensors (mostly sonar or optical retroreflective sensors) [18]. In many wheelchair solutions, the same range sensors are used for running the wheelchair in a semi-autonomous mode in which
1 Advances in Human-Friendly Robotic Technologies
7
the user’s instructions can be modified in conjunction with the sensor information on nearly located objects. For semi-autonomous control, several schemes are proposed/tested, such as wall following, people following/avoidance, and narrow corridor passage [14]. In [19] is considered the design of a robotic device that combines autonomous navigated wheelchair with a manipulator installed on it. It is noted that such a wheelchair can be used in three modes: (1) mode for autonomous indoor transportation of the user; (2) mode as a mobile robot that is remotely-controlled to deliver different objects to the user when he/she is placed in the bed; (3) mode as an home inspection robot that is remotely controlled and sends TV images from different places from the user’s home. Powered feeders are rehabilitation robots especially designed for feeding patients with severe movement limitations of their upper limbs. Controlling the robot by themselves, such users can eat a normal meal at their own pace. To this class belong Handy 1 [20, 21], My Spoon [22, 23], and ISAC [24–26]. Handy 1 also has optional functions for brushing teeth, shaving and applying make up. The ISAC implements image recognition in order to determine the exact position of the user’s mouth. 1.2.2 Robots for Physical Support and Indoor Navigation Devices from that group are intended to assist users with both movement weakness and visual impairments. Potential users of these RR are aged people and people with multiple impairments. Such machines are usually designed as motorized base that gives physical support to the user. Forces, caused by the user’s body are monitored and used for the control of the platform. Robot control system determines the current location of the platform and outputs voice-synthesized navigation instructions or warnings about potential obstacles on the intended route. Good examples of that conception are the HITOMI project [27] and the PAM-AID project [28–31], [W1]. The system WHERE (Walking and moving HElper Robot system) developed at KAIST [32, 33] aims to assist the user in walking and gait rehabilitation and provides body weight support to the user. The system automatically detects the user intention regarding walking speed and movement direction. A picture of the WHERE system is shown in Fig. 1.1. 1.2.3 Robots for Physical Rehabilitation Different from the electromechanical devices for passive motion rehabilitation (such as the Artromot system of the ORTHOMED Medizintechnik Ltd) [W2], robotassisted systems for movement rehabilitation can perform various movement programs and can sense the user’s force reactions. In relation to this application, we may mention here the robotic device for stroke rehabilitation in the GENTLE/S Project [34], the robotic therapy system, called Stanford Driver’s SEAT [35], the MIME project [36] of the VA Palo Alto Rehabilitation R&D Center, the robotic
8
D. Stefanov and Z.Z. Bien
system for neuro-rehabilitation of the Newman Laboratory at MIT [37], the system of the University of California, Irvine [38], REHAROB project [39, 40], etc. These robotic systems can be easily programmed to implement different rehabilitation exercises that fit to the concrete needs of the particular users and offer flexible adjustment of various movement parameters such as range of flexion and extension, pause between the sequential motions, force, speed, etc.
Fig. 1.1. Walking and moving helper robot system (WHERE) developed at KAIST (Courtesy of Ju-Jang Lee). The system is used for gait rehabilitation and assists users in walking by providing body weight support. The robot automatically detects user intention regarding walking speed and movement direction
1.2.4 Vocational RR Different from the robots that are primarily designed to help those paralyzed users in their everyday movement activities, the vocational rehabilitation robots perform assistance for the paralyzed user in concrete occupational activities, such as office work [41, 42], programming (RAID and Epi-RAID) [43], work at chemical and biology laboratories (Walky) [44], visual inspection of hybrid integrated circuits (IRVIS) in a real manufacturing environment [45, 46], and operation of a commercial lathe [47].
1 Advances in Human-Friendly Robotic Technologies
9
1.2.5 Emotional Interactive Entertainment Robots Similar to the virtual reality systems for communication with virtual subjects or game play, the emotional interactive entertainment robots (EIAR) [48] are intended to increase the emotional comfort and give some emotional relief of people who live alone. Different from some virtual reality software products where the virtual creatures appear only as images on the computer screen, the entertainment robots are mechatronics devices that exhibit animal-like behavior. Pet robots are one of the latest tendencies in the development of entertainment home robots. Within the project HII House (Home Information Infrastructure House), National Panasonic has demonstrated a conceptual idea for the user-friendly interface for older people in their homes. National Panasonic has developed an electronic home interface and memory jogger designed as cuddly toys – Tama the robocat and Kuma the robot bear. A speech synthesis device can reproduce a number of phrases as a voice response to particular user voice-activated inputs. The device can also be programmed to remind the user to take his/her medication at a particular time. Failure to respond to the device can activate a direct-dial call to the care-giving staff that can then check whether or not the user’s condition is normal. The interactive robot BECKY [49], developed at KAIST, Korea, demonstrates different behaviour in accordance with the emotional status of the user. BECKY recognizes the current emotion of the user by observing the user’s response to its actions and considering the environmental information that can affect the user’s emotional status. BECKY can adapt to the user’s preferences by a learning mechanism based on neural networks. In addition, BECKY can choose and play some music to resume human spirits. The seal robot and the cat robot, developed in Japan [50, 51], are recent results of the research study on the physical interaction between the human and pet robots. The robots were tested for aged people in the hospital in Japan. To the same category we may classify the AuRoRA Project [52, 53], [W3] that studies application of the robots to the education and therapy for children with autism helping them to develop and increase their communication and social interaction skills. Robot is used as an interactive "toy" whose behaviour can change, depending on the child response. We may note that applications of rehabilitation robots are not limited to the listed above. Contemporary technology achievements would be a premise for the design of much sophisticated robots that meet new sides of user’s needs. As a result, we may expect in the near future some new categories of RR that refer to much sophisticated tasks, such as helping the user in bathing, linen change, assisting in cloth change, helping for recovering after falling on the floor, lifting the patient from the bed to the wheelchair, cooking, etc.
10
D. Stefanov and Z.Z. Bien
1.3 Specialized Human-Machine Interface Human-Machine Interface (HMI) can translate the user’s commands for proper operation of the rehabilitation robots, wheelchair, or other home equipment such as lamps, TV sets, telephones, doors, home security systems, etc. on an easy and efficient way. Movement limitations of users with severe paralysis cause a serious problem for reliable control of the home-installed assistive systems. As an HMI, the head-tracking devices are widely used because of their ability to produce up to three independent proportional signals that correspond to the forward-backward head tilting, left-right head rotation, and lateral head tilting. The manner of control is very natural for the user. Recently, some new head tracking techniques involving facial detection [54] and optoelectronic detection of light-reflective head-attached markers (Tracker2000, Head Mouse) have been proposed [55, 56]. Some new technologies such as eye-movement control, brain control [57], as well as gesture recognition [58] and facial expression recognition give new opportunities for human – friendly interaction between the user and the home-installed devices and become a base for new interface solutions in the near future. Voice control is also considered as a natural and easy way of operation of home-installed devices, but its application is still limited because of the high dependence of the recognition ratio on the voice specific and the ambient noise level. Recently, some new voice recognition algorithms based on neural networks have provided optimism that those drawbacks can be overcome soon, and the voice control could be applied in a noisy environment [59]. The soft computing technique1 offers new perspectives on the application of the EMG signals in the control of the home environment, allowing successful extraction of informative signal features in case that a strong noise signal interferes with useful EMG signals [60].
1.4 Rehabilitation Robots in the Smart House Design Intelligent houses for persons with physical disabilities and for older persons should provide their residents with better environmental conditions for independent indoor life. Numerous smart devices and systems, installed in the house, should be capable of being linked with each other, to process information from the inhabitant or from his/her environment, and to make independent decisions and actions for the survival of the resident in cases of emergency. During the last decade, different concepts of a smart house have been developed and tested under various projects. These developments are oriented to different groups of people with special needs (PSN) and refer to different social infrastructure
1
Soft computing differs from conventional (hard) computing in that it is tolerant of imprecision, uncertainty, and partial truth. It includes neural networks, fuzzy logic, evolutionary computation, rough set theory, probabilistic reasoning, and expert systems.
1 Advances in Human-Friendly Robotic Technologies
11
(that can vary from single house to village, nursing home, hospital, etc.). Rehabilitation robots become an important part of the recently developed smart-house models. The “Robotic Room” is a project developed at the Sato Laboratory of the Research Center for Advanced Science and Technology in the University of Tokyo [61, 62]. The Robotic Room can be considered as a step ahead toward the realization of intelligent houses for persons with physical disabilities and as a further development of the idea of the Techno Houses. The environment of the Robotic Room consists of a ceiling-mounted robot arm (called the “long reach manipulator”) and an intelligent bed with pressure sensors for monitoring of the inhabitant’s posture. Modules can monitor person’s respiration without attachment of any additional sensors to the user’s body. The robot arm is intended to bring various objects to the user. The developed life-support infrastructure is to comply with the needs of the rapidly aging society. In the “Intelligent Sweet Home” project at KAIST [63], gestures are adopted for the control of home-installed devices. Parts of the scenario are two rehabilitation robots. First of them is mounted to the user’s bed and helps the user in activities such as book reading, object replacement, quilt adjustment, massage, scratching, etc. The experimental design employs a Manus robot. A supplementary mechanical module provides translation of the robot along the bed. The second robot is a mobile one and it role is to perform transportation tasks. (A Pioneer robot from ActivMedia Robotics, LLC was used in that design.) Ceiling-mounted cameras are used for localization of the current robot position. The same cameras detect the positions of the quilt ends. The interface, called “soft remocon”, consists of three ceiling-mounted video cameras that detect the orientation of the user’s hand. By pointing at the robots, television, or curtains, the user can choose the device that will be controlled. Special light signals confirm user’s selection. After choosing the device, the user sets his/her instruction to it by pre-defined hand gestures. A voice-generated message confirms the recognized gesture command before its execution. A picture of the KAIST Intelligent robotic room is presented in Fig. 1.2.
Fig. 1.2. The intelligent robotic room at KAIST, Korea
12
D. Stefanov and Z.Z. Bien
Figure 1.3 shows the main idea of the “soft remocon”.
a
b Fig. 1.3. Gesture-based human-machine interface. a Soft remocon – The user’s hand gesture is automatically recognized by the TV-based image recognition system and the desired action (“Turn the TV on!”) is executed; b Pointing recognition system – By pointing to the concrete object, the user specified the object that should be replaced, and the robot performs the preliminary defined task
1.5 Functional Integration of the Robotic Environment In some new developments, RR’s are components of the intelligent home environment and act in conjunction with other home-installed devices. In such arrangement, RRs need to be controlled in a coordinated manner. The M3S (Multiple Master Multiple Slave) is a communication strategy especially designed for functional integration of home-installed rehabilitation devices. The M3S specification started from TIDE Project [64, 65], and later it was developed as an open standard (available for free). M3S allows users to assemble a specific complete modular system and to extend or modify the system later in a plug-and-play manner. In case of emergency, the user can halt the operation of the whole M3S system by a Dead Man Switch. The efficiency of that integration strategy has been demonstrated by several evaluations with users from some European countries. The ICAN Project (Integrated Control for All Needs) developed further the idea of functional integration of the home-installed devices [66], [W4]. The main objec-
1 Advances in Human-Friendly Robotic Technologies
13
tive of the project is to propose an optimal control over all home-installed devices by a single interface device, such as a joystick or switch input. A portable device, named function carrier, distributes the commands of a single interface device to various output devices. For example, depending on the setting of the functional carrier, the user will be able to control the wheelchair or the rehabilitation robot using one joystick. The function carrier itself can be designed as a portable computer or palmtop PC that is connected to the interface modules of the separate rehabilitation devices. The overall integration method applies the M3S architecture and communication protocols. ICAN is a collaborative project supported by the Telematics Applications Programme of the European Commission [67].
1.6 Commercialization of RR Commercialization of the rehabilitation robots is often impeded by several factors, such as high cost, low efficiency, and existing welfare regulation. Despite of these problems, some rehabilitation robots, such as Manus (Netherlands) [68, 69], Handy1 (UK) [20, 21], Raptor (USA) [70], and My Spoon (Japan) [22, 23] have already become commercially available products that can be used daily by an increasing number of end-users, offering them enhanced movement assistance and comfort in operation. Although the research in RR continuously explores new service areas for people with disabilities, it seems that people with disabilities have not taken enough advantage of this technology so far. When the MRI was introduced for musculoskeletal imaging, clinicians were resistant to learn the technology because it did not offer anything over CT scans. However, it is now been accepted as a vital part of diagnostic imaging. Similarly, it can be expected that after certain period of improvement and advertising, RR will become a vital part of the treatment of people with disabilities. In general, the decision for buying a rehabilitation robot (DB) can be expressed as:
efficiency, appearance, safety, easiness in control DB = f price
(1.1)
Efficiency of the RR can be found if good answers can be found for the following important questions: 1. What is the list of activities that can be assisted with that robot? 2. What is the importance of each performed tasks for the everyday life and user’s movement freedom? - For example, scratching is a quite natural human action that is important for the user but other activities, such as feeding, bathing, handling with fine objects have dominant priority for the everyday life. The impact of the RR to the quality of life of individuals with movement disabilities can be enhanced by careful choice of activities where the RR can help. However, the gap between academic laboratory research and clinical practice continues to
14
D. Stefanov and Z.Z. Bien
3. 4. 5.
6.
7.
8. 9. 10. 11. 12. 13. 14. 15.
1. 2.
exist. In order to significantly advance the application of RR to rehabilitation practice, the link between scientific investigations rehabilitation robotics and the clinical application of results must be strengthened. Does the robot contribute to the user’s privacy? – Can the robot help activities that are strictly private? - For instance, cloth change and bathing are strict private activities and it will be important if the robot can be helpful for them. For what kind of movement impairments the robot can be applied? –Larger user’s group means better chances for manufacturing of the robot in large series. What is the robot contribution for completion of each task? - For instance, when we command robot assistance in eating, we must clarify the details on whether the robot can prepare the food by itself or whether it is used to serve food that has been prepared in advance by the human-helper. Knowing the robot characteristics, we should answer also the question how long time in a day the robot is expected to be used, i.e. how many hours in a day the robot will replace the human-helper and how many hours movement independence will be provided with such robot. What is the speed of execution of certain task with the robot if we compare it with the speed of natural motions? For example, if we assume that nonimpaired person who uses his/her hands usually completes certain task within 20 seconds, what is the expected speed of execution of a similar task by the user that uses the robot to perform the task, i.e. what is the intensity of the help? On the other hand, speed should be relatively low because of safety considerations and it is highly influenced on the skills of a particular user to interact with the robot. What kind of HMI is used? Does it require any sensor attachments to the user? What is the robot reliability? What are the results from the test evaluation and the user’s feedback about the robot? What is the noise level when the robot is in operation? What kind of energy is applied for robot operation and how long time the robot can operate with batteries that are initially completely charged? What space can be accessed with the robot? Can the robot be used to pick up distant objects (for instance, objects that are located on the floor)? Is the robot gripper precise enough to handle with thin objects and objects with sophisticated shapes? What are the weight and dimensions of the robot? In case of a wheelchair-mounted robot, it should be calculated how the robot attachment affects some wheelchair characteristics, such as ability for narrow door passage and shifting of the centre of gravity. What are the lifting power of the robot and its maximal speed?
Easiness in control relates to the following questions: How long time is needed for a user to learn how to operate the robot? Does the robot control demand some user’s instructions in each moment of the task execution? What is the level of automatic task performance?
1 Advances in Human-Friendly Robotic Technologies
3.
15
Does the robot refer to the exact user’s commands only or its control system can respond on a correct way to tasks that are set with certain level of “fuzziness”?
For instance, if the user wants to drop a letter at the mail box, usually that requires many adjustment maneuvers controlled by the user until the gripper becomes oriented in the correct position regarding the mail slot. However, the user’s task will become much easier if the gripper adjustment can be done automatically based on sensor information. Another aspects of automation of some tasks are: the dosage of the force applied to the object; sliding prevention of the grasped object; maintaining of the initial object orientation during pick-and place tasks with liquid containers (cup, spoon, etc.). Appearance of the rehabilitation robot is very important issue from the aesthetical and psychological point of view. Appropriate robot design should make the robot naturally looking and will not attract others’ attention. Safety is an item of major consideration in the robot design. Different from other robots, the RR’s are intended to work in close proximity to the face and head of the user with limited own motions. RR’s are complicated mechatronic devices, produced in very small numbers and currently the price of such robots is relatively high. User’s readiness to meet the asked price depends on two main factors: 1. The amount, provided by insurance companies, government agencies, charity organizations, etc. 2. Own financial contribution of the end-user. In addition to the price for purchase of the robot, we should take into account also the maintenance costs of the robot during the period of its usage. The maintenance costs include labor from highly qualified personnel and will increase much if user’s house is very far from the technical service center. When we comment commercialization of the rehabilitation robots, we should take into account that the design of robots for service of people with movement disabilities is a relatively new area and currently existing applications cover only a small number of user’s needs. Due to the small number of the users, the average cost of a RR is quite high.–Currently, RRs are only a tiny part of the service robots, which are small portion of the manufactured robots. Probably the situation will be dramatically changed when home robots are developed. Currently, many successful research projects reveal optimism that we are not far from the era when personal robots will become just a part of the home environment, similar to the refrigerator, video, computer, etc. Perhaps in a changed situation, the RR will differ from the ordinary home robots only by some special functions that they will have added. Apart from their primary functions, such as objects replacement, cooking, serving meal, house guarding, partnering in playing games, interlocutors, etc., the new generation of RR will possess advanced human-friendly interface and will be able to help persons with disabilities and aged people in a much efficient way.
16
D. Stefanov and Z.Z. Bien
1.7 Some Issues for Futuristic Intelligent Robotic House Model A research trend in the smart house design is concerned with the futuristic residential space equipped with advanced monitoring system that observes not only the physical status of the inhabitants but also their behavior and emotional condition. It is also expected that one of the biggest reforms in the home arrangement will be the implementation of various service robots that help the inhabitants in many ways. Actions of the RR will be matched with the operation of other home-installed devices. In this sense, the residential space can be considered as a multi-agents robotic system. In this paper, we note that new concept as “Intelligent Robotic House (IRH)”. One example of such intelligent structure is shown in Fig. 1.4.
Fig. 1.4. A Futuristic Intelligent Robotic House Model. The intelligent robotic house integrates advanced technology for non-invasive health monitoring, movement assistance, and leisure activities, and offers easy human-machine interaction
Each subsystem in the residential space will be capable of solving any local task, while a central control unit will coordinate the work of all subsystems and will build the general control strategy. Performed actions in the intelligent residential space will be initiated either by the user’s command or from analysis of the information from home installed sensors. The human-friendly HMI will be based on general user’s instructions, addressed to the common control system of the IRH. Complex tasks may involve many home installed devices. For instance, user’s command for bringing some food will activate first some special automatic cooking devices to prepare the food; next, the mobile robots will serve the food and another robot will feed the paralyzed user. In this example, the user does not control the robot directly.
1 Advances in Human-Friendly Robotic Technologies
17
Instead, a special computer plans the task and distributes separate subtasks to the different devices. A block diagram of the Intelligent Robotic Home is given in Fig. 1.5. Here, some of the sensor information is used by several subsystems. For example, the information on the user’s temperature and heart rate is used by the health monitoring system, the system for emotional status recognition, and the home environment controller; while the visual information of the home-installed TV cameras is utilized for gesture recognition, face expression recognition, walk pattern recognition, posture recognition, and for home security. The same visual information is also applied for monitoring the health and emotional status of the user.
Fig. 1.5. Functional Architecture of a model Intelligent Robotic House. All home-installed devices and robotic systems are controlled by a common control system. The user can use different modalities in order to communicate with his/her house. Some information is used from several subsystems
The example structure includes three service robots: kitchen service robot, robot for movement assistance, and entertainment robot. The entertainment robot is controlled not only upon the information from its local vision sensors but also on the information from the home-installed vision sensors. Audio and video programs in the intelligent home can be selected automatically upon the recognized current state of emotion of the inhabitant. By monitoring of the user’s behavior during listening to the music or watching the video program, the system learns the user’s preferences and includes favorite programs in the future play lists.
18
D. Stefanov and Z.Z. Bien
It is expected that the intelligent robotic house will be based on advanced techniques for sensing and health monitoring. Some recent vision-based health monitoring systems refer to recognition of the facial expression and the facial color (paleness) as an indication of the current health status of the patient. Emotion monitoring is another challenging subject for providing information on the human state of emotion. An initial result exploring this idea can be found in [71]. New interface design will offer more convenient, human-friendly, natural, and autonomous, or simply, “intelligent” ways of human-machine interaction. Facial expressions and gestures are indicative of human intention and emotion. According to some statistics [72], 93% of messages in the face-to-face communication situation are transmitted via facial expression, gestures, and voice tone, while only 7% are via linguistic words. Home-installed RR should be highly efficient and capable of responding to the user’s exact commands and also to the user’s intentions with a high level of “fuzziness” as well, treating and executing them in a proper way. It is expected that the new generation interface devices will be able to adapt to the user’s specifics and will be highly resistant to various artifacts. We anticipate that a major reform of the future residential space will take place by the advent of the use of various service robots. Low cost robots with increased functionality and high reliability will extend the range of activities in which the persons with disabilities will be supported. The IRH will implement strong fusion among different sources of sensor information. Collected data from the wearable sensors will be used not only for monitoring of the inhabitant’s health status but the same information can be also considered in the control of the home environment. For example, the room temperature can be adjusted in consideration of the current health condition of the user.
1.8 Concluding Remarks The rehabilitation robots have become an important goal of development with strong social and economic motivations. Its realization will render a powerful solution for many existing problems in the society with increasing numbers of aged/physically weak and disabled people and will make human life more pleasant and easier. In this paper, we have tried to formulate and classify some recent tendencies of the research in the area so as to obtain our vision about the future tendencies in the development of the technology and its organization. Through examples, we have shown that the development strategy of the intelligent home-installed technology has changed from the design of separate devices (at the beginning) to a form of integrated system arrangement where many homeinstalled devices communicate with each other and synchronously serve/monitor different parameters of the house. Because the problem for the aging society becomes quite common for many countries, the research problems on the smart-home technology with service robots have already become an important subject of international research. Diverse forms of international cooperation, such as international
1 Advances in Human-Friendly Robotic Technologies
19
conferences, international joint projects for development, and result evaluation, are in progress. From the design point of view, we observe that the development strategy is also experiencing a rapid evolution. Although the first designs of home-installed devices were hardware-oriented, the recent strategies are mainly oriented toward intelligent algorithms, where the software solution takes the main part of the whole design. In addition, we have stressed the fact that the future RR’s will include humancentered technologies where important technological components should provide human-friendly interaction with the user. The home-installed technology will further be oriented toward a custom-tailored design where the modular-type components of the smart house will meet the individual user’s needs, emotional characteristics, and preferences.
References 1.
Saito M (2000) Expanding welfare concept and assistive technology. In: Proc. IEEK Annual Fall Conference, Ansan, Korea 2. Warren S, Craft R (1999) Designing smart health care technology into the home of the future. In: Proc. 1st Joint BMES/EMBS Conf., Atlanta, GA, USA, p 677 3. Lindström JI (2001) From R&D to market products – the TIDE Bridge phase. In: Marinček C, Bühler C, Knops H, Andrich R (eds) Assistive Technology – Added Value to the Quality of Life. Amsterdam, The Netherlands: IOS Press, pp 688–692 4. Harwin W, Rahman T, Foulds R (1995) A review of design issues in rehabilitation robotics with reference to North American research. IEEE Trans. Rehab. Eng 3(1): 3–13 5. Stefanov D (1994) Model of a special orthotic manipulator. Mechatronics 4(4): 401–415 6. Kwee H (2001) Integrating control of MANUS and wheelchair. In: Proc. 7th Int. Conf. Rehabilitation Robotics, Evry, France, pp 107–112 7. Rosier JC, van Woerden JA, van der Kolk LW, Driessen BJF, Kwee HH, Duimel JJ, Smits JJ, Tuinhof de Moed AA, Honderd G, Bruyn PM (1991) Rehabilitation robotics: the MANUS concept. In: Proc. 5th Int. Conf. Advanced Robotics, Pisa, Italy, pp 893 – 898 8. Song WK, Lee H, Bien Z (1999) KARES: Intelligent wheelchair-mounted robotic arm system using vision and force sensor. Robotics and Autonomous Systems 28(1): 83–94 9. Bien Z, Kim DJ, Stefanov DH, Han JS, Park HS, Chang PH (2002) Development of a novel type rehabilitation robotic system KARES II. In: Keates S, Langdon P, Clarkson PJ, Robinson P (eds) Universal Access and Assistive Technology. London, UK, Springer, pp 201–212 10. Pauly M (1995) TAURO – Teilautonomer Serviceroboter für Überwachungsaufgaben. In: Dillmann R, Rembold U, Lüth T (eds) Autonome Mobile Systeme. Berlin, Germany: Springer, pp 30–39 11. Prassler E, Scholz J, Fiorini P (2001) A robotic wheelchair for crowded public environments. IEEE Robotics and Automation Magazine 7(1): 38– 45 12. Baumgartner E, Skaar S (1994) An autonomous vision – based mobile robot. IEEE Trans. Automat. Control 39(3): 493–502
20
D. Stefanov and Z.Z. Bien
13. Yoder JD, Baumgartner E, Skaar S (1996) Initial results in the development of a guidance system for a powered wheelchair. IEEE Trans. Rehab. Eng 4(3): 143 –302 14. Gomi T, Griffith A (1998) Developing intelligent wheelchairs for the handicapped. In: Proc. Evolutionary Robotics Symp., Tokyo, Japan, pp 461–478 15. Wakuami H, Nakamura K, Matsumara T (1992) Development of an automated wheelchair guided by a magnetic ferrite marker lane. J. of Rehab. Research and Development 29(1): 27–34 16. Wang H, Kang CU, Ishimatsu T, Ochiai T (1996) Auto navigation on a wheelchair. In: Proc. 1st Int. Symp. Artificial Life and Robotics, Beppu, Oita, Japan 17. Kreutner M, Horn O (2001) Contribution to rehabilitation mobile robotics: Localization of an autonomous wheelchair. In: Proc. 7th Int. Conf. Rehabilitation Robotics, Evry, Paris, pp 207–214 18. Yanco H (1998) Wheelesley: A robotic wheelchair system: indoor navigation and user interface. In: Mittal VO, Yanco HA, Aronis J, Simpson R (eds) Assistive Technology and Artificial Intelligence – Application in Robotics, User Interfaces and Natural Language Processing. Heidelberg, Germany: Springer, pp 256–286 19. Stefanov D (1999) Integrated control of a desktop mounted manipulator and a wheelchair. In: Proc. of the Sixth International Conference on Rehabilitation Robotics (ICORR'99), Stanford University, USA, July 1-2, 1999, pp 207 - 214 20. Finney R, Topping M (1997) After sales care provision for the Handy 1 robotic aid to independence. In: Proc. 7th Int. Conf. Rehabilitation Robotics, Bath, UK 21. Topping M, Smith J (1999) The development of Handy 1, a robotic system to assist the severely disabled. International Conference on Rehabilitation Robotics (ICORR’99), 1999, Stanford, CA, pp 244 – 249 22. Ishii S, Tanaka S, Hiramatsu F (1995) Meal assistance robot for severely handicapped people. Proceedings IEEE Robotics and Automation Conference, pp 1308-1313 23. Soyama R, Ishii S, Fukase A (2003) The development of meal-assistance robot 'MySpoon'. Proceedings of the 8th International Conference on Rehabilitation Robotics, pp 88–91 24. Bagchi S, Kawamura K (1994) ISAC: A robotic aid system for feeding the disabled. AAAI Spring Symposium on Physical Interaction and Manipulation, March 1994 25. Kawamura K, Bagchi S, Iskarous M, Bishay M (1995) Intelligent robotic systems in service of the disabled. IEEE Transactions on Rehabilitation Engineering 3(1): 14 –21 26. Kawamura K, Peters II RA, Wilkes MW, Alford WA, Rogers TE (2000) ISAC: foundations in human-humanoid interaction. IEEE J Intelligent Systems 15: 38–45 27. Mori H, Kotani S, Kiyohiro N (1998) HITOMI: Design and development of a robotic travel aid. In: Mittal VO, Yanco HA, Aronis J, Simpson R (eds) Assistive Technology and Artificial Intelligence – Application in Robotics, User Interfaces and Natural Language Processing. Heidelberg, Germany: Springer, pp 221 – 234 28. Lacey G, Mac Namara S, Dawson-Howe KM (1998) Personal adaptive mobility aid for the infirm and elderly blind. In: Mittal VO, Yanco HA, Aronis J, Simpson R (eds) Assistive Technology and Artificial Intelligence – Application in Robotics, User Interfaces and Natural Language Processing. Heidelberg, Germany: Springer, pp 211–220 29. Lacey G, Dawson-Howe KM (1997) Evaluation of robot mobility aid for older persons blind. In Proc. Symp. Intelligent Robot Systems, Stockholm, Sweden 30. MacNamara S, Lacey G (1999) A robotic walking aid for frail visually impaired people. In: Proc. 6th Intl. Conf. Rehabilitation Robotics, Stanford, CA, USA, pp 163–169
1 Advances in Human-Friendly Robotic Technologies
21
31. MacNamara S, Lacey G (1999) PAMAID: a passive robot for frail visually impaired people. In: Proc. RESNA Annual Conf., Long Beach, CA, USA, pp 358–361 32. Lee CY, Seo KH, Oh C, Lee JJ (2000) A system for gait rehabilitation with body weight support: Mobile manipulator approach. Journal of HWRS-ERC 2(3): 16–21 33. Lee CY, Seo KH, Kim CH, Oh SK, Lee JJ (2002) A system for gait rehabilitation: Mobile manipulator approach. Proc. of the 2002 IEEE International Conference on Robotics&Automation, Washington DC, May 2002, pp 3254–3259 34. Harwin W, Loureiro R, Amirabdollahian F, Taylor M, Johnson G, Stokes E, Coote S, Topping M, Collin C, Tamparis S, Kontoulis J, Munih M, Hawkins P, Driessen B (2001) The GENTLE/S project: A new method of delivering neuro-rehabilitation. In: Marinček C, Bühler C, Knops H, Andrich R (eds) Assistive Technology–Added Value to the Quality of Life. Amsterdam, The Netherlands: IOS Press, pp 36–41 35. Johnson MJ, Van der Loos HFM, Burgar CG, Shor P, Leifer LJ (1999) Designing a robotic stroke therapy device to motivate use of the impaired limb. In: Proc. 7th Int. Conf. Rehabilitation Robotics, Evry, France, pp 123–132 36. Lum PS, Burgar CG, Shor PC, Majmundar M, Van der Loos HFM (2002) Robotassisted movement training compared with conventional therapy techniques for the rehabilitation of upper limb motor function after stroke. Archives of PM&R, 83: 952 – 959 37. Krebs HI, Hogan N, Aisen ML, Volpe BT (1998) Robot-Aided Neurorehabilitation. IEEE Trans. Rehab. Eng 6(1): 75–86 38. Lum PS, Reinkensmeyer DJ, Lehman SL (1993) Robotic assist devices for bimanual physical therapy: Preliminary experiments. IEEE Trans. Rehab. Eng 1(3): 185–191 39. Arz G, Toth A (2001) REHAROB: A project and a system for motion diagnosis and robotized physiotherapy delivery. In: Mokhtari M (ed) Integration of Assistive Technology in the Information Age, ICORR’2001, 7th International Conference on Rehabilitation Robotics. IOS Press, Amsterdam, Evry, France, pp 93–100 40. Toth A, Arz G, Varga Z, Varga P (2001) Conceptual design of an upper limb physiotherapy system with industrial robots. In: Mokhtari M (ed) Integration of Assistive Technology in the Information Age, ICORR’2001, 7th International Conference on Rehabilitation Robotics. IOS Press, Amsterdam, Evry, France, pp 109–116 41. Van der Loos HFM, Hammel J, Lees DS, Chang D, Perkash I (1990) Field evaluation of a robot workstation for quadriplegic office workers. Eur. Rev. Biomed. Tech 5(12): 317– 319 42. Van der Loos HFM (1995) VA/Stanford rehabilitation robotics research and development program: Lessons learned in the application of robotics technology to the field of rehabilitation. IEEE Trans. Rehab. Eng 3(1): 46–55 43. Eftring H (1994) Robot control methods and results from user trials on the RAID workstation. In: Proc. 4th Int. Conf. Rehabilitation Robotics, Wilmington, DE, USA, pp 97– 101 44. Neveryd H, Bolmsjö G (1995) WALKY, an ultrasonic navigating mobile robot for persons with physical disabilities. In: Proc. 2nd TIDE Congress, Paris, France, pp 366–370 45. Keates S, Clarkson PJ, Robinson P (2001) Designing a usable interface for an interactive robot. In: Proc. 7th Int. Conf. Rehabilitation Robotics, Evry, France, pp 156–162 46. Clarkson PJ, Keates S, Dowland R (1999) The design and control of assistive devices. In: Proc. Int. Conf. Engineering Design, pp 425–428
22
D. Stefanov and Z.Z. Bien
47. Oderud T, Tyrmi G (2001) One touch is enough… In: Marinček C, Bühler C, Knops H, Andrich R (eds) Assistive Technology–Added Value to the Quality of Life. Amsterdam, The Netherlands: IOS Press, pp 144–147 48. Shibata T, Tanie K (2001) Emergence of affective behaviours through physical interaction between human and mental commit robot. Journal of Robotics and Mechatronics 13(5): 505–516 49. Do JH, Park KH, Bien Z, Park KC, Oh YT, Kang DH (2001) A development of emotional interactive robot. In: Proc. 32nd Int. Symp. Robotics, Seoul, Korea, pp. 544–549 50. Shibata T, Tashima T, Tanie K (1999) Emergence of emotional behaviour through physical interaction between human and robot. In: Proc. IEEE Int. Conf. Robotics and Automation, Detroit, MI, USA, pp 2868–2873 51. Shibata T, Tashima T, Tanie K (1998) Emergence of emotional behaviour through physical interaction between human and pet robot. In: Proc. IARP 1st Int. Workshop Humanoid and Human Friendly Robotics, Tsukuba, Japan, pp 1–6 52. Dautenhahn K (2003) Roles and functions of robots in human society-implications from research in autism therapy. Robotica 21: 443–452 53. Werry I, Dautenhahn K, Harwin W (2000) Challenges in rehabilitation robotics: A mobile robot as a teaching tool for children with autism. Proc. International Workshop Recent Advances in Mobile Robots, June 29th 2000, De Montfort University, Leicester, UK 54. Xu G, Sugimoto T (1998) Rits eye: a software-based system for realtime face detection and tracing using pan-tilt-zoom controllable camera. In: Proc. 14th Int. Conf. Pattern Recognition, Brisbane, Australia, pp 1194–1197 55. MacBride C, Fleming B, Tanberg BJ (2001) Interdisciplinary team approach to AAC assessment and intervention. In: Proc. 16th Annual CSUN’s Conf. Technology and Persons with Disabilities, Los Angeles, CA 56. Evans DG, Drew R, Blenkhorn P (2000) Controlling mouse pointer position using an infrared head-operated joystick. IEEE Trans. Rehab. Eng 8(1): 107–117 57. Levine SP, Huggins JE, BeMent SL, Kushwaha RK, Schuh LA, Rohde MM, Passaro EA, Ross DA, Elisevich KV, Smith BJ (2000) A direct brain interface based on eventrelated potentials. IEEE Trans on Rehab. Eng 8(2): 180–185 58. Bien Z, Kim JB, Jung JW, Park KH, Bang WC (2000) Issues of human-friendly manmachine interface for intelligent residential system. In: Proc. 1st Int. Workshop Humanfriendly Welfare Robotic Systems, Taejon, Korea, pp 10–14 59. Kasabov N, Kozma R, Kilgour R, laws M, Taylor J, Watts M, Gray A (1997) A methodology for speech data analysis and a framework for adaptive speech recognition using fuzzy neural networks. In: Proc. 4th Int. Conf. Neural Information Processing, Dunedin, New Zealand, pp 1055–1060 60. Han JS, Stefanov DH, Park KH, Lee HB, Kim DJ, Song WK, Kim JS, Bien Z (2001) Development of an EMG-based powered wheelchair controller for users with high-level spinal cord injury. In: Proc. Int. Conf. Control, Automation and System, Jeju Island, Korea, pp 503–506 61. Nakata T, Sato T, Mizoguchi H, Mori T (1996) Synthesis of robot-to-human expressive behaviour for human-robot symbiosis. In: Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems, Minneapolis, MN, USA, pp 1608–1613
1 Advances in Human-Friendly Robotic Technologies
23
62. Eguchi I, Sato T, Mori T (1996) Visual behaviour understanding as a core function of computerized description of medical care. In: Proc. IEEE/RSJ Int. Con. Intelligent Robots and Systems, Minneapolis, MN, USA, pp 1573–1578 63. Bien Z, Park KH, Bang WC, Stefanov DH (2002) LARES: An intelligent sweet home for assisting older persons and the handicapped. In: Proc. 1st Cambridge Workshop Universal Access and Assistive Technology, Cambridge, UK, pp 43–46 64. Dallaway J, Jackson R, Timmers P (1995) Rehabilitation robotics in Europe. IEEE Trans. Rehab. Eng 3(1): 35–45 65. Nelisse M (1995) M3S: A general-purpose integrated and modular architecture for the rehabilitation environment. In: Proc. 2nd Int. CAN Conf., London, UK, pp 10.2–10.9 66. Willems C (1999) ICAN Integrated Communication and Control for All Needs; Assistive technology on the threshold of the new millennium. In: Proc. 5th AAATE Conf., Dusseldorf, Germany 67. Allen B (1999) Bus systems in a three tiered world, experiences from the ICAN project. In: Proc. Int. Conf. Smart Homes and Telematics, Eindhoven, Netherlands 68. Eftring H, Boschian K (1999) Technical results from MANUS user trials. In: Proc. 6th Int. Conf. Rehabilitation Robotics, Stanford, CA, USA, pp 136–141 69. Gelderblom GJ, de Witte L, van Soest K, Wessels R Dijcks B, van’t Hoofd W, Goossens M, Tilli D, and van der Pijl D (2001) Evaluation of the MANUS robot manipulator. In: Marinček C, Bühler C, Knops H, Andrich R (eds) Assistive Technology – Added Value to the Quality of Life. Amsterdam, The Netherlands: IOS Press, pp 268–273 70. Mahoney R (2001) The Raptor wheelchair robot system. In: Proc. 7th Int. Conf. Rehabilitation Robotics, Evry, Paris, pp 135–141 71. Bien Z, Do JH (2000) Interactive robot for emotion monitoring. In: Proc. Korea-Japan Joint Workshop on Network based Human Friendly Mechatronics and Systems, Seoul, Korea, pp 62–65 72. Mehrabian A (1968) Communication without words. Psychology Today 2(9): 52–55.
2 Rehabilitation Robotics from Past to Present – A Historical Perspective Michael Hillman
2.1 Introduction Popular culture presents the image of a robot as a mechanical, humanoid device, often evil in intent. Those with a slightly more informed technical knowledge would point to the industrial robot arm of the automotive factory, Even professional engineers, are affected to some extent or other by these popular images. In attempting to survey four decades of the development of rehabilitation robotics it is wise to start from an official definition. The Robot Institute of America defined a robot as “A re-programmable, multifunctional manipulator designed to move material, parts, tools or specialized devices through variable programmed motions for the performance of a variety of tasks.” Although this definition was obviously intended for industrial robots, it identifies the key features of programmability, flexibility and movement. Robotics has obviously moved on from that early definition. While robots were initially employed as handling machines in factories, their application is now much wider. In 1987 the Department of Trade and Industry in the UK had an Advanced Robotics initiative to encourage the wider use of robotics in areas other than factories. They used this definition of Advanced Robotics. “The integration of enabling technologies and attributes embracing manipulators, mobility, sensors, computing (IKBS, AI) and hierarchical control to result ultimately in a robot capable of autonomously complementing man’s endeavours in unstructured and hostile environments.” This is a very wide-ranging definition, but one of the key features is the “integration of technologies”. The aim of this integration of advanced technologies is to produce a device that can both operate autonomously and in an environment which may be unstructured and/or hostile. More recently the term “Mechatronics” has been coined as “the synergistic combination of precision mechanical engineering, electronic control and systems thinking in the design of products and processes.” (Festo Didactic GmbH Co 1998). In many ways, mechatronics and robotics cover much of the same territory and it is the application of this level of technology to “the restoration of a person to an optimal level of physical, mental, and social function and well being”, that we are concerned with here. Not just in the UK, but across the world there are now many examples of “advanced robotics”. What is key is not just the level of technology, but that the roZ.Z. Bien and D. Stefanov (Eds.): Advances in Rehabilitation Robotics, LNCIS 306, pp. 25-44, 2004. © Springer-Verlag Berlin Heidelberg 2004
26
M. Hillman
bots have moved out of the factory and into the wider, unstructured and sometimes hostile world. The examples cover a wide area and include devices for the exploration both of Mars and earth’s oceans as well as more domestic applications such as filling a car with fuel, mowing the lawn and vacuuming the floor. The use of robotics in rehabilitation is another major area in which robots are coming “out of the factory”. The use of robots in rehabilitation is often associated with assistive devices, some used in a vocational environment, some as aids to daily living or more specifically as feeding devices. There are however other areas either where robotic technology has been applied or might be. One such area is in mobility, increasing the usefulness of more traditional powered wheelchairs. Prosthetics and orthotics is another area where robotics has already been applied. A major area at present is in robot mediated therapy. Education is an area where robotic technology has been applied in some instances and where modern robotic toys might have an impact as well as equipment more specifically aimed at the disabled. Finally communication is an area which has used advanced computer technology, but where there is the scope of incorporating more mechanical technologies as well. This survey covers 40 years of the development of rehabilitation robotics. Rather than attempting to mention every research group and project, this survey is deliberately selective. The emphasis is on those projects that represent the first in a line of development, those that seem particularly innovative, those that have a commercial significance and those which have been used by the greatest number of real life users. The choice of projects is also unashamedly personal in that these are the projects that have most influenced the author. The term “robotics” should be interpreted as widely as possible in an inclusive rather than an exclusive way. The definition above of mechatronics is the one that most simply describes the scope of our survey.
2.2 Earliest Work Most reviews of rehabilitation robotics cite the work at the CASE Institute of Technology in the early 1960’s [24] as the first application of robotics technology to a rehabilitative manipulator. This was a powered orthosis with four degrees of freedom. The exoskeletal structure supported the user’s paralysed arm while performing pre-recorded manipulative tasks, these sequences being taught by an able bodied assistant during training. Interestingly, when so much current work uses electrical actuators, this used pneumatic actuators with closed loop position control achieved using incremental encoders. Another early project was the Rancho Los Amigos “Golden Arm” (Fig. 2.1) [24]. This was a seven degree of freedom battery powered electric orthosis. Several versions were built, and at least one was wheelchair mounted. It was controlled using a form of joint-by-joint control, which was found during evaluation to be not very intuitive.
2 Rehabilitation Robotics from Past to Present – A Historical Perspective
27
Fig. 2.1. Rancho Golden Arm
In considering these two early devices it is instructive to put them in the context of the technology of the day. The early 1960’s was a time when the integrated circuit had just been invented, and ten years before the microprocessor. Computers were beginning to come down in size from room size to a more compact cabinet. Neil Armstrong and Buzz Aldrin weren’t to set foot on the moon until the end of the decade, in July 1969.
2.3 Assistive Robotics Assistive robotics can be divided into three main areas based on the mobility of the device. Firstly we consider those that operate in a fixed site. Secondly will be those that may be moved around from one location to another. Thirdly those devices that are attached to a wheelchair. Many people [30] have surveyed the potential uses of robotics to assist people with physical disabilities, and the following areas have been identified. • Eating & drinking • Personal hygiene – washing, shaving, applying make up • Work & leisure – particularly computer use, equipment such as hi-fi and video systems, also games • Mobility – opening doors, windows • General reaching – up to shelves, down to the floor. These are all valuable areas. There are many devices dedicated to specific tasks, some designed for people with disabilities others readily available on the general market. In this case the choice and installation of such devices (not without consultation with the user) predetermines what tasks he or she can carry out. Most assistive robots however are designed as a general-purpose tool and intended to be used as the user desires, rather than for any predetermined task. Independence only comes when the user can decide at any time what activity they would
28
M. Hillman
like to do. A good example of this from our own work is the user who gained great satisfaction from using it to open his Christmas presents. 2.3.1 Fixed Site Apart from the two orthotic devices mentioned above, work in the more specific area of assistive rehabilitation robotics started in the mid 1970's. One of the earliest projects was the workstation based system designed by Roesler [32] in Heidelberg, West Germany. The purpose designed, five degree of freedom manipulator was placed in a specially adapted desktop environment, using rotating shelf units. Another early workstation system was that of Seamone and Schmeisser [33] at the Johns Hopkins University, supported by the Veterans Administration in the United States from 1974. The arm of this system was based around an electrically powered prosthetic arm, mounted on a horizontal track. Various items of equipment (e.g. telephone, book rest, computer discs) were laid out on the simple but cleverly designed workstation table and could be manipulated by the arm using pre-programmed commands. The system thus required that items be in precisely known positions as there were no sensors on the arm. User input was by simple scanning switch selection of routines on a simple LED display. In France, an early project was the Spartacus robot [21], based around a large high power manipulator from the nuclear industry. The table-mounted arm was able to reach down to the floor or up to a shelf. User control was by an analogue input, particularly a head position operated joystick. Safety was of particular importance with this relatively high power device, and early training of users was done with the arm behind a clear screen. This project is of particular significance in that it led to the Manus project in Holland and the Master project in France. In any review of work in rehabilitation robotics there must be recognition of the continuing work at Stanford University, initiated by Larry Leifer in the Department of Mechanical Engineering, with Machiel van der Loos at the Palo Alto VA Center. They built 4 generations of DeVAR (Desktop Vocational Assistive Robot) systems [11, 38]. DeVAR III was a tabletop system laid out for daily living tasks, while DeVAR IV was used in a vocational environment. The DeVAR IV system (Fig. 2.2) used the Puma 260 arm, a standard industrial manipulator, mounted upside down on an overhead track, thus making much better use of the available space. This highlights the problem of using a commercially available robot in that the work environment has to be tailored around the arm. How this is achieved can be crucial to how successful the system is. The usefulness of the system depends on how many “tasks” can be laid out within the work environment. Whenever a certain task is not available to the robot or the user, the usefulness of the system comes to a halt and a colleague has to be called in to intervene. Obviously in an office environment this can be allowed for, but the ability to work independently has been compromised.
2 Rehabilitation Robotics from Past to Present – A Historical Perspective
29
Fig. 2.2. DeVAR IV workstation
Besides the engineering work this project is notable for its use in a real working environment on an 8 hours a day basis at the Pacific Gas and Electric Company by Bob Yee. The Stanford group has also done a lot of work to justify the high cost of such a system ($50 - $100,000) in terms of financial saving relative to the cost of employing a human assistant. While the Stanford system used the Puma arm, initially designed for an industrial application, many other projects have been based around the RT series robot. The RTX robot was designed by Tim Jones of UMI (Universal Machine Intelligence, UK) in 1985. The arm was of what is known as a modified SCARA configuration. This has one vertical degree of freedom and two rotational joints with vertical axes that allow the main arm to move in a horizontal plane. This configuration has proved particularly appropriate for rehabilitation applications. One of the early application areas promoted by the manufacturers was its use in rehabilitation (another application area was in the laboratory). The original RTX robot was followed by higher powered versions, the RT100 and the RT200. A mobile version was also investigated, R-Theta but never commercialised One early use of the RTX was a workstation system developed by Caroline Fu [9] of Boeing in Seattle for one of their employees. Interestingly in a number of cases the impetus for using robotics has been from employers, to enable them to continue with valuable staff after accidents, and to keep their expertise within the company. Another significant use of the RT series robot was by the Master project [5] in France. Their approach to maximising the workspace was to mount the arm at the back of the workstation. This gave good visibility of the whole work area. In setting up any workstation it is vital that the arm does not obscure visibility of the environment, in this case either the large vertical column or the bulk of the arm or gripper in different orientations. One significant area of work using the RT series robot has been not just in designing workstations but also in developing and modifying the software and hardware for rehabilitation applications. The use of interchangeable or alternative grippers is one area and this has been a part of the Master system. Several reha-
30
M. Hillman
bilitation friendly programming languages have been developed for the RT robots, although different drivers could allow the software to be applied to other robots. One example is the CURL Cambridge University Robot Language [4]. The RT robots were used as the basis for the RAID project [17], funded by the European Commission. The RAID project, as with all projects under the TIDE initiative (Telematics for the Integration of Disabled and Elderly people), was collaborative and multinational. Amongst the partners were Oxford Intelligent Machines (OxIM, UK) who were then the manufacturers of the RT robots, and the Master project team. The RT series robot was set up in an extended workstation. Another example of the work space of the basic robot being extended by, in this case, mounting the robot on a horizontal track enabling it to retrieve paper work etc from shelving units. The outcome of the RAID project was commercialised by OxIM (who have now ceased trading), and Afma Robots in France So far all the systems we have looked at have used commercially available robots, whether the Puma, or the RT robot, with the problem of integrating such robots into a work environment. The other approach is to design and build a manipulator to best suit the environment. The main advantage is that the robot is designed to be best suited to the likely tasks and working environment, rather than modifying the tasks and environment to suit the robot. Commercially there is the advantage of not being reliant on the continuing availability of a commercial device. Against this there are several possible disadvantages. The resulting device, produced in small numbers, is likely to be more expensive than one made for a wider market in higher volumes. This disadvantage may be countered by making compromises to the design appropriate to the specific system requirements. There is likely to be a longer development time and cost. Any finished device needs to be of a quality and reliability comparable with the best commercial standards.
Fig. 2.3. Regenesis workstation
This was the approach taken at the Neil Squire Foundation in designing their Regenesis manipulator (Fig. 2.3) [3] to best suit the environment and tasks. Their manipulator was based on a horizontal beam, around which an extending arm could translate and rotate. This gave access to a large working volume and could be mounted at the back of a desk, or across a bed for example. This device was, for a while, made commercially available.
2 Rehabilitation Robotics from Past to Present – A Historical Perspective
31
Fig. 2.4. Handy 1
In terms of numbers sold, probably the most successful rehabilitation robot is the Handy 1 (Fig. 2.4) [36], sold by Rehab Robotics (UK). The project originated from the Masters Degree project of Mike Topping at the University of Keele (UK). Topping had a neighbour, a young boy named Peter, who had severe problems eating. He used a cheap educational robot to produce a device that allowed Peter to eat independently, at the speed he chose, for the first time. The company has sold at least 250 systems with very positive feedback of its effectiveness. One of the strengths of this project, and reasons for its success, is that it stemmed from the real problems of an individual client. Another feature is that originally it made no attempt to be multi functional. However more recently extensions of the system have been developed for it to be used for applying make up, painting, washing and shaving. The Handy 1 is essentially a feeding aid based around a robot. By comparison the Winsford feeder (RTD-ARC, New Jersey, US) has been available for a long while as a feeding aid, but only recently has been promoted as being a robotic device. With around 2000 units having been sold its commercial impact is obviously greater than the Handy 1. The MySpoon (Secom Co. Ltd, Tokyo, Japan) is available in Japan and is similar in concept. In the UK the Neater Eater (Buxton, UK) was initially designed as a purely manual device to assist those with a tremor eat, utilising a damped arm. More recently a powered, programmable device has emerged which may be considered to be a robotic device. 2.3.2 Mobile Robots Compared with the workstation based devices, the number of mobile assistive robots is very small and the commercial impact has been negligible. One of the best known is probably the MoVAR system [39] from Stanford University, which is essentially a DeVAR on wheels. The mobile base had sophisticated omni-directional “mechanum” wheels. The MoVAR was controlled
32
M. Hillman
from a console with several monitor screens giving feedback to the user from an on board camera as well as a map of the environment and a control environment. A very capable system but at the time it was not packaged in a way which would appeal to a general user. With advances in computer technology the idea could be revisited and a more marketable product achieved. As the founder of the Unimation Company, Joe Engelberger has been called the father of robotics. He has had an interest in service robotics and particularly in medical applications. He proposed [6] the use of his Helpmate robot as a fetch and carry robot for a disabled person. However, while the Helpmate has been successfully used for moving supplies around a hospital, it has not been used successfully in a rehabilitation application. The cluttered environment of a home is not appropriate for such a mobile robot. More recently the KARES II robot system [2] has been developed at KAIST in Korea. This is a wide-ranging project investigating various modes of user control including the use of visual servoing, an eye mouse and a haptic suit as well as the design of the robot arm itself. The arm has been mounted in a number of different configurations, but primarily on a remote controlled mobile base.
Fig. 2.5. Wessex robot
A different approach was investigated at the Bath Institute of Medical Engineering with their Wessex robot (Fig. 2.5) [13]. Having identified the shortcomings of a fixed site robotic workstation in a domestic environment, a non-powered mobile base was designed. While a workstation system works well in a vocational environment it may be very restricting in a domestic environment where different tasks are normally carried out in different rooms of the home. For example to wash in the bathroom, to listen to music in the living room and to eat in the kitchen or dining room. The mobile base was intended to be moved from room to room by a carer or might be clipped to the front of a wheelchair.
2 Rehabilitation Robotics from Past to Present – A Historical Perspective
33
2.3.3 Wheelchair Mounted Manipulators If a mobile robot can be seen as a mechanical servant or slave, the concept of mounting a manipulator onto a wheelchair provides what may be termed a third arm. One very early wheelchair mounted robot was designed by Carl Mason [27] at the VA Prosthetics Center in New York. Mechanically it was a well engineered system, able to reach from the floor to ceiling. Apparently though it was too springy, and the hook end effector (as used by many prosthetic hand wearer’s) was not very successful. The control was quite basic, being operated on a joint by joint basis that has been found to be unintuitive and time consuming to control. By comparison placing a simple educational robot on a wheelchair is the most basic of arrangements. Zeelenberg’s son was diagnosed as having Muscular Dystrophy. His parents wanted him to make the fullest development of his abilities and skills. His father obtained an educational robot and simply mounted it on the wheelchair tray [41]. With Muscular Dystrophy it is possible to use a small push button controller and this was chosen for the input device. This wouldn’t pretend to be a technically sophisticated device, but arose out of a real need and close collaboration between the developer and user. Amongst the uses of the robot are for opening the door, moving chess pieces and using the telephone. One reason why this is a very significant part of our historical survey is that out of this work came the Manus project.
Fig. 2.6. Manus
Manus (Fig. 2.6) [22] is extremely well known and although it hasn’t sold as many units as the Handy 1 it is at least as well respected. The work started as far back as 1984 involving collaboration between IRV, (Institute for Rehabilitation Research) in Hoensbroek, led by Hok Kwee, the Institute for Applied Physics and the TNO Product Centre in Delft and the Netherlands Institute of Preventive Health Care. It is a sophisticated robotic manipulator able to be mounted to a number of different wheelchairs. It has seven degrees of freedom, as well as a simple gripper.
34
M. Hillman
The extra degree of freedom extends its vertical range, while allowing it to fold compactly at the side of the wheelchair. The mounting of the arm, protruding from the side of the chair, raises the crucial issue for all wheelchair mounted robots of integrating the arm with the wheelchair, not least to ensure there is no unacceptable increase in overall width, or compromise of the stability of the wheelchair. Many have been sold to rehabilitation centres with much development going on around it, but more importantly with significant sales to end-users. Manus is seen as the standard against which other rehabilitation robotic systems are measured and has been commercialised by Exact Dynamics. Further development of Manus is being carried out by both the manufacturers and also under the European Commanus project [7].
Fig. 2.7. Raptor
The other wheelchair-mounted manipulator that is available commercially is the Raptor (Fig. 2.7) [26], which is being produced by the Rehabilitation Technologies Division of the Applied Resources Corporation. It makes an interesting comparison with Manus. While Manus is a relatively high cost, sophisticated device, the Raptor has introduced compromises to bring the cost down to about a third of that of Manus. In particular it has only four degrees of freedom. It will be very interesting to see how these two devices perform commercially and in terms of their effectiveness, with their differences in cost and functionality. It will also be interesting to see how the larger American market affects the viability. A slightly off-beat approach to wheelchair-mounted manipulators came from Jim Hennequin in the UK. He was best known for his Spitting Image satirical puppets, which appeared on UK television. The puppets used a pneumatic air muscle and these were used for the drive motors of the Inventaid [15] wheelchair mounted robot. He claimed a high power to rate ratio for the air muscles. He also claimed that it was simple enough to be maintained by a back-street fitter.
2 Rehabilitation Robotics from Past to Present – A Historical Perspective
35
2.3.4 Human Machine Interface A mobile assistive robot may be envisaged as a “slave” – it can be instructed to fetch an item from the kitchen or to place a book on the shelf. If the technology is adequate for it to operate autonomously, the instructions can be in what is virtually a natural language. A similar approach can be taken to a workstation environment. Since the environment is more structured it is easier for the robot to operate autonomously. By comparison a wheelchair-mounted manipulator may be seen as a “third arm” and would normally be controlled in some direct way, for example to move the arm forward or to grip an object. It is one of the great challenges to come up with a control system that can even begin to compete with the way in which those without disabilities freely move their arms and hands. This distinction need not be hard and fast. It is obviously possible to drive a mobile robot around the home in the same way one would drive a remote controlled car, perhaps with the benefit of an on-board camera. Similarly task type commands can be given to a wheelchair-mounted manipulator. For example “Grip the red mug in front of you”, or “simply reach down to the floor”. Different media can be used to communicate such commands to the controlling computer. For natural language commands speech recognition may well be used, while for direct control a joystick might be seen as more appropriate. For those not able to control a joystick some form of scanning system can be used. The chosen solution in either case may well be a combination of different media and will also depend on the abilities of the user. While in earlier days the interface was a fixed part of the overall system, more recently the user is able to choose the most appropriate interface device.
2.4 Mobility Within mainstream robotics a major area of both research and commercial application is that of “Automatic Guided Vehicles” (AGV). This technology obviously has potential in addressing the mobility needs of people with a wide range of disabilities. All powered wheelchairs operate in two degrees of freedom, conventionally under the direct control of the user and so are not that much different from a two degree of freedom telemanipulator. Modern powered wheelchairs will often add other powered functions such as leg raisers, and seat tilt. Most modern powered wheelchairs also have programmable controllers. Such chairs may be technically sophisticated but would not make the claim of being robotic. As soon as sensors are added, and a control system which can react to the output of those sensors, the wheelchair is becoming an AGV. Such devices are normally referred to as smart wheelchairs. These use sensors to detect objects in the environment. On board processing of the constantly changing and moving relative environment can allow the user such functions as to track a wall, go through a door or dock at a table or desk.
36
M. Hillman
One approach is to adapt a standard commercial base. For example the CALL Centre in Edinburgh, UK have many years experience in this area. In their initial work [28] they used a standard electric wheelchair to produce a smart wheelchair for children and teenagers. In their latest smart wheelchair the Smart Controller acts as if it was a second joystick plugged into the DX (Dynamic, New Zealand) wheelchair bus system. Various Smart Wheelchair ‘tools’ can be easily selected in different combinations to suit the pilot and environment Alternatively it is possible to build a wheelchair that is “smart” from the outset. Such an approach was used by the CEC TIDE funded OMNI project [14]. This was an omni-directional wheelchair integrated with autonomous control features The big difference between an AGV and a smart wheelchair is that a powered wheelchair is not normally required to be completely autonomous. The issue is how to handle the conflicts between when the user should have an appropriate level of control of the chair and when the smart processor will take over, and vice versa. A different approach to mobility comes from Dean Kamen who invented the iBOT wheelchair. While the chair may be driven in a conventional way, gyroscopic sensors allow the chair to balance on two wheels or to climb stairs. The safety issues of relying on gyroscopes and processors to provide the basic stability of the device are paramount. In common with other safety critical “fly by wire” systems multiple redundancy is used. A commercial company Independence Technology (US) has recently received FDA approval for the iBOT in the US and hopes to start making them available to selected clinics and rehabilitation centres towards the end of 2003. Not all mobility implies that the person needs to be transported by the device. In 1977 Meldog [35] was developed at the Mechanical Engineering Labs at Tsukuba Science City in Japan. It provided mobility for a blind person by guiding them around city streets, downloading a basic map and using landmark sensors. It would function in much the same way as a guide dog would be used in other cultures. With the increasing miniaturisation of electronics and GPS positioning it is possible that the same functionality could be obtained today on a body worn device without the problems of kerbs and the steps. Many people have investigated simple electronic white sticks [8] which have met with limited success, but there may be scope for a much more sophisticated device.
2.5 Prosthetics and Orthotics It is clear from the early work mentioned above that prosthetics and orthotics have been closely associated with rehabilitation robotics. It is useful at this stage to define a prosthesis as an artificial limb (although the term can also be used for an internal organ or joint) and an orthosis as a device to support or control part of the body. The early work at CASE and Rancho Los Amigos (mentioned above) were orthotic systems. More recently the Mulos project, funded under the EU TIDE
2 Rehabilitation Robotics from Past to Present – A Historical Perspective
37
funding initiative [40] was a powered upper limb orthotic system. Rahman and colleagues at the AI duPont Institute (Wilmington, US) have been intimately involved in the rehabilitation robotics field. They have been involved in the design of both powered and non-powered arm orthoses. In particular their anti gravity arm orthosis [31] is noteworthy although, being a balanced system, with no external power supply may not come within our definition of a robotic system. Although there is a lot of commercial work in prosthetic arms and hands very little of this has used robotic technology, but rather has been a development from existing technologies. However, one computer controlled upper arm prosthesis is the Utah/MIT artificial arm and dextrous hand developed by Jacobsen [16]. Another longstanding project is the work that originated with the Southampton Hand [23] at Southampton University and progressed at the Nuffield Orthopaedic Centre in Oxford. Initially a complex five fingered hand, the mechanism has been simplified, while retaining the capabilities of forming the hand into several functional configurations both for precision and power. As a continuation of this work, the ToMPAW project [29] combines the earlier Leverhulme hand prosthesis with the prosthetic arm developed at Edinburgh. There are two main issues in powered prosthetics and orthotics. One is the issue of miniaturisation and the other of power. The problem of miniaturisation is particularly critical for a hand prosthesis, where the complete system has to fit within the outline of a human hand. We have already noted the problems of integrating a robotic system onto a wheelchair. The problems of prosthetics are obviously at a far greater level of magnitude. Although a hand prosthesis and an upper arm orthosis require different levels of power both present problems in how to store the energy to give a day’s use of the device before recharging is required. With a hand prosthesis the requirement is to fit batteries with sufficient capacity within the hand. For an upper arm prosthesis the energy requirement is far greater. Although the volume/mass constraints are not so difficult, for a truly portable system this is still a major problem area. Besides electrical batteries, compressed CO2 has also been used as a power supply. Blatchford’s Intelligent Knee prosthesis (Basingstoke, UK) however is a nonpowered device. It uses sensors to regulate the swing of the knee, dependent on the rate of walking and other programmable values. Although it is a passive, rather than an active, device it is truly a robot, although nowhere in Blatchford’s publicity is the word “robot” used.
2.6 Robot Mediated Therapy The use of robotics to provide movement therapy for the rehabilitation of patients following stroke has been an area of major growth within rehabilitation robotics over the past few years. It is interesting to see the growth in the number of papers presented at the International Conference on Rehabilitation Robotics (ICORR) in this area. Before 1999 there was at the most a couple of papers. Since 1999 the number of papers has increased from to 15 in 2003.
38
M. Hillman
The reason for the increase is obviously the potential both for more effective therapy and more cost effective therapy. Such potential exists for all areas of rehabilitation robotics, but is perhaps most easily quantifiable in this area. A robot could be used to replicate the exercise regime used by physiotherapist, but has potential for other regimes not easily carried out by a human. For rehabilitation following stroke there are three main ways [25] in which robots have been applied. • Passive is where the movement is externally imposed by the robot while the patient remains relaxed. This movement can maintain the range of motion at the joints • Active assisted is where the patient initiates the movement, but the robot assists along a predefined path • Active resisted is the opposite case where the patient must move against a resistance generated by the robot. At Palo Alto the MIME system [34] can be used both in a passive or active mode or a bilateral mode in which the patient attempts to move both the affected and unaffected limbs. While MIME uses two six degree of freedom Puma robot arms, the ARM Guide developed by Reinkensmeyer and colleagues uses a one degree of freedom robotic device working in a similar fashion to a trombone slide [18]. The MIT-Manus system [20] is a two degree of freedom system, similarly designed for stroke rehabilitation, and is now available as a commercial product. Another stroke rehabilitation project is the GENTLE/S project [1], which encourages the patient to move against a resisted haptic arm in a computer, generated virtual 3D room. With several different approaches it is important to be able to demonstrate the clinical effectiveness. A lot of useful information has been accumulated for the MIME project. After one and two months the group receiving therapy from the robot showed a faster improvement, although after 6 months there was little difference from those who had received more conventional therapy. While the robot seems to be at least as effective as conventional therapy, it seems to be working in a different fashion, and it may be that different approaches will be needed for different levels of impairment. Most of the current work is involved with stroke rehabilitation, but in the past robots have also been used with patients with cerebral palsy and following orthopaedic surgery. At Santa Clara University two planar robot arms have been used for the rehabilitation of joints following surgery [19]. The two arms, each with force sensors at base and gripper, hold firmly two adjacent limb segments (e.g. upper and lower leg). Using the two robots, the leg is manipulated, with the joint kept under compression for effective rehabilitation.
2.7 Robotics in Special Needs Education The use of robots in education for those with physical and learning disabilities has received attention over the years. For example a young child learns much from
2 Rehabilitation Robotics from Past to Present – A Historical Perspective
39
simple play activities. However if he has impaired mobility these opportunities are greatly restricted. The Cambridge/CUED robot [12], based on a UMI RTX robot with a vision recognition system, allowed the child to interact with his environment in various ways ranging from simply dropping a toy brick onto a drum, to painting or playing board games.
Fig. 2.8. Cambridge University Educational Robot
Many electronic toys are now cheaply available on the mainstream market, which may provide especial benefit for the disabled. However AnthroTronix (Maryland, US) are developing what they describe as telerehabilitation tools to motivate and integrate therapy, learning, and play
2.8 Robotics in Communications We have already identified that most communication aids are not robots. This certainly applies when communication is through an acoustic medium. Communication can however use other senses – especially visual, but also tactile The Dexter hand [10] is intended to act as a finger spelling hand for those who are deaf/blind. Such people use finger spelling for communication but this language is not widely known by the general population. Dexter enables someone to input text, for example at a keyboard which is then converted to finger spelling by Dexter, to communicate with a person who uses finger spelling. One difficulty which many physically disabled people encounter is the desire to read a book, magazine or newspaper. Page turning is one of the tasks that comes
40
M. Hillman
highest on the list of priorities for an assistive robot. It is also one of the most difficult, although there are ways to achieve it. There are several page-turners on the market, most of them bulky, expensive. They are often not very effective and limited in what they can achieve. Surely this is a prime area for the application of robotic technologies. For what is essentially a single function device, cost may be a constraint, but if it was reliable and effective there would be a market for such a device.
2.9 Historical Perspective Looking back at the past 40 years it is sobering to consider what has been achieved, although challenging to imagine what might be achieved in the next 40. This survey has outlined the progress in rehabilitation robotics from the early orthotic devices, workstation based systems, and the start of the commercial products Handy 1 and Manus in the 1980’s. The first International Conference on Rehabilitation Robotics (ICORR) was held in 1990 at AI duPont Institute, Delaware where 12 papers were presented. For several years it alternated between the US and Europe, and most recently was held in Korea in 2003 with 66 oral presentations, and 27 demonstrations and posters. The term “robot” was first used in Capek’s “Rossums Universal Robots” in the 1920’s, from the Czech word for a slave. The film Metropolis in 1927 represented the familiar humanoid robot loved by science fiction. In practical terms robotics can be dated back to the DeVilbiss paint sprayer in the 1930’s. In the 1940’s the fiction author Asimov formulated the 3 laws of robotics. In the 1950’s the Unimation robot company was formed. There are now 750,000 robots in industrial use. The progress in surgical/medical robotics has been very impressive. From the early work in the 1980’s they are now used in neurosurgery, orthopaedic surgery and assistance during endoscopic operations. In rehabilitation, wheelchairs are a much more traditional product – easily defined in what they can and should achieve. Everest & Jennings launched the first modern wheelchair in the 1930’s and the first powered wheelchair in the 1950’s. Since then they have sold their 1 millionth wheelchair and there are many other companies worldwide. It is an industry in which many of the products follow a very traditional design and perhaps the first truly novel product is the iBOT wheelchair.
2.10 Commercialisation The real benefit of rehabilitation robotics is in devices being readily available on the open market, so I believe the number of systems sold commercially is of paramount importance. Table 2.1 is a summary of those systems that are currently commercially available. In a number of cases it has not been possible to quote
2 Rehabilitation Robotics from Past to Present – A Historical Perspective
41
numbers of systems sold, as the manufacturers hold this as commercially sensitive information.
Table 2.1. Current (as at April 2003) commercially available rehabilitation robots No. sold
Cost (USD)
AfMaster
?
$50,000
Manus Raptor
>150 13
$35,000 $12,500
Handy 1 Winsford Neater MySpoon
>250 2000 100 ?
$6300 $2499 $3600 $3200
MIT Manus – Planar MIT Manus – Wrist
20 2
$70,000 $65,000
Workstation Wheelchair mounted Feeder
Therapy
Although a number of workstation systems have been available in the past, the AfMaster is the only one currently available. For wheelchair mounted robots there is the interesting comparison between Manus and Raptor. It will be interesting to see which will win out – the technically and functionally superior Manus or the lower cost Raptor with the benefit of the huge US market? Feeders are an interesting lower cost area, and the Winsford feeder has been available for many years. Handy 1 is the device that is most obviously marketed as being robotic, but its cost is highest. Therapy based robots are beginning to become available. Besides the MIT/Manus, the Arm Guide and MIME projects are being commercialise by the Rehabilitation Technologies Division of ARC as the ARC-MIME system.
2.11 Alternatives to Robotics in Rehabilitation Before concluding this survey the alternatives to using robots in rehabilitation should be considered. These alternatives should be seen not as competition, but complementary. Most assistive robots aim to be multifunctional, but stand alone devices, whether sold specifically for the disabled market or mainstream market can have elements of the same functionality. One area of growth at the moment is the integration of different technological devices and approaches, particularly within a smart house environment. Nowadays many activities can be carried out on a computer without the need to interact with the real world. Examples are computer art and music, computer chess and the whole area of 3D computer games and virtual reality.
42
M. Hillman
Animals are often used in rehabilitation. Particularly dogs for the blind, but also for those with mobility and hearing impairment. Monkeys have also been used although more difficult to train. Human carers will always be important. Against the issues of independence must be balanced the need for human interaction and companionship. However sophisticated our technical systems, they are unlikely to match the abilities of a human. In parallel with the development of robotic devices for rehabilitation, much research is continuing into the origins and treatment of debilitating diseases and conditions.
2.12 Conclusions However good the research may be, success is ultimately measured in assistance being given to patients and disabled people in real life. Research must be seen as a stepping stone to commercial products. This will be through devices being sold and bought commercially (whether through private, institutional or state funding). For devices to succeed commercially the correct balance between function and cost must be achieved. One way in which this field will expand is with a move away from the traditional idea of a robot “arm” to the concept of using robotic technologies – sometimes as a multifunctional device, or sometimes as a single function tool – using robotic technologies in the most appropriate fashion. Finally the question – does it matter whether it’s a robot - should we use the “R” word? Commercially the word robot may be used positively to give an impression of technical sophistication. On the other hand many consumers are frightened by the word robot. But ultimately and finally the most important aspect is the benefit of disabled people and their carers.
References 1.
2.
3.
4.
Amirabdollahian F, et al. (2001) Error correction movement for machine assisted stroke rehabilitation In: Mokhtari M (ed) Integration Of Assistive Technology In The Information Age. IOS, Netherlands, pp 60–65 Bien Z, et al. (2002) Development of a novel type rehabilitation robotic system KARES II, In: Keates S, et al. (eds) Universal Access and Assistive Technology, Springer, London, pp 201–212 Cameron WM. (1986) Manipulative appliance development in Canada, In: Foulds R (ed) Interactive Robotic Aids. World Rehabilitation Fund Monograph #37 New York, World Rehabilitation Fund pp 24–28 Dallaway JL, et al. (1993) An interactive robot control environment for rehabilitation applications, Robotica, 11: 541–551
2 Rehabilitation Robotics from Past to Present – A Historical Perspective 5. 6. 7. 8. 9.
10. 11. 12.
13. 14. 15. 16. 17. 18.
19.
20. 21. 22.
23. 24. 25.
43
Detriche JM et al. (1991) Development of a Workstation for Handicapped People Including the Robotized System Master. Proc. ICORR, Atlanta Engelberger J. (1989) Robotics in Service. Cambridge, Massachussets, The MIT Press Evers HG, et al. (2001) MANUS towards a new decade, In: Mokhtari M (ed) Integration Of Assistive Technology In The Information Age. IOS, Netherlands, pp 155–161 Farcy R, Belik Y (2002) Locomotion assistance for the blind, In: Keates S, et al. (eds) Universal Access and Assistive Technology, Springer, London, pp 277–284 Fu C (1986) An Independent Vocational Workstation for a Quadriplegic. In: Foulds R (ed) Interactive Robotic Aids. World Rehabilitation Fund Monograph #37 New York, World Rehabilitation Fund, pp 42–44 Gilden D, Jaffe D. Dexter, (1988) A Robotic Hand Communication Aid for DeafBlind, Int’l Jnl. of Rehabilitation Research 11(2): 188–189 Hammel J, et al. (1989) Clinical evaluation of a desktop robotic assistant, J. of Rehabilitation Research and Development 26(3): 1–16 Harwin.WS, Ginige A, Jackson RD. (1986) A Potential Application in Early Education and a Possible Role for a Vision System in a Workstation Based Robotic Aid for Physically Disabled Persons. In: Foulds R (ed) Interactive Robotic Aids. World Rehabilitation Fund Monograph #37 New York, pp 18–23 Hillman M, Gammie A (1994) The Bath Institute of Medical Engineering assistive robot, In: Proc ICORR ’94 Wilmington, US, pp 211–212 Hoyer H, et al. (1997) An Omnidirectional wheelchair with enhanced comfort features. In: Proc ICORR 97, Bath, UK, pp 31–34 Jackson RD, (1993) Robotics and its role in helping disabled people, Engineering Science and Education Journal 2(6): 267–272 Jacobsen SC, et al. (1982) Development of the Utah Artificial Arm, IEEE Transactions on Biomedical Engineering 29(4): 249–269 Jones T. (1999) RAID – Towards greater independence in the office and home environment. In: Proc ICORR 99, Stanford, US, pp 201–206 Kahn LE, et al. (2001) Comparison of robot assisted reaching to free reaching in promoting recovery from chronic stroke. In: Mokhtari M (ed) Integration Of Assistive Technology In The Information Age. IOS, Netherlands, pp 39–44 Khalili D, Zomlefer M. (1988) An Intelligent Robotic System for Rehabilitation of Joints and Estimation of Body Segment Parameters. IEEE Transactions on Biomedical Engineering 35: 138 Krebs HI, et al. (2003) Robotic applications in neuromotor rehabilitation. Robotica 21(1): 3–12 Kwee HH et al. (1983) First Experimentation of the Spartacus Telethesis in a Clinical Environment. Paraplegia 21: 275 Kwee HH, et al. (1989) The MANUS Wheelchair-Borne Manipulator: System Review and First Results. In: Proc. IARP Workshop on Domestic and Medical & Healthcare Robotics, Newcastle Kyberd PJ, et al. (2001) The design of anthropomorphic prosthetic hands: A study of the Southampton Hand, Robotica 16(6): 593–600 Leifer L (1981) Rehabilitative Robotics, Robotics Age May/June 1981, pp 4–15 Lum P, et al. (2002) Robotic devices for movement therapy after stroke: Current status and challenges to clinical acceptance. Topics in stroke rehabilitation 8(4): 40–53
44
M. Hillman
26. Mahoney RM. (2001) The Raptor Wheelchair Robot System. In: Mokhtari M (ed) Integration Of Assistive Technology In The Information Age. IOS, Netherlands, pp 135– 141 27. Mason CP, Peizer E. (1978) Medical Manipulator for Quadriplegic. In: Proc Int’l Conf. on Telemanipulators for the Physically Handicapped. IRIA 28. Nisbet P, et al. (1988) The CALL Centre Smart Wheelchair. In: Proc. First Int’l Workshop on Robotic Applications in Medical and Healthcare, Ottawa, Canada 29. Poulton AS, et al. (2002) Progress of a modular prosthetic arm, In: Keates S, et al. (eds) Universal Access and Assistive Technology, Springer, London, pp 193–200 30. Prior SD, (1990) An electric wheelchair mounted arm – a survey of potential users, Jnl of Medical Engineering and Technology 14(4): 143–154 31. Rahman T, et al. (2001) An anti-gravity arm orthosis for people with muscular weakness. In: Mokhtari M (ed) Integration of Assistive Technology In The Information Age. IOS, Netherlands, pp 31–36 32. Roesler H et al. (1978) The Medical Manipulator and its Adapted Environment: A System for the Rehabilitation of Severely Handicapped. In: Proc Int’l Conf. on Telemanipulators for the Physically Handicapped. IRIA 33. Seamone W, Schmeisser G. (1986) Evaluation of the JHU/APL Robot Arm Workstation. In: Foulds R (ed) Interactive Robotic Aids. World Rehabilitation Fund Monograph #37 New York, pp 51–53 34. Shor PC, et al. (2001) The effect of Robotic-Aided therapy on upper extremity joint passive range of motion and pain, In: Mokhtari M (ed) Integration Of Assistive Technology In The Information Age. IOS, Netherlands, pp79–83 35. Tachi S, et al. (1985) Electrocutaneous Communication in a Guide Dog Robot (MELDOG). IEEE Transactions on Biomedical Engineering 32: 461 36. Topping M. Handy 1, (2001) A robotic aid to independence for severely disabled people. In: Mokhtari M (ed) Integration Of Assistive Technology In The Information Age. IOS, Netherlands, pp 142–147 37. Topping M, Smith J (1999) The development of Handy 1, a robotic system to assist the severely disabled, In: Proc ICORR 99, Stanford, US, pp 244–249 38. Van der Loos M. (1995) VA/Stanford rehabilitation robotics research and development program: Lessons learned in the application of robotics technology to the field of rehabilitation. IEEE Transactions on Rehabilitation Engineering, 3(1): 46–55 39. Van der Loos M, Michalowski S, Leifer L. (1986) Design of an Omnidirectional Mobile Robot as a Manipulation Aid for the Severely Disabled. In: Foulds R (ed) Interactive Robotic Aids. World Rehabilitation Fund Monograph #37 New York pp 61–63 40. Yardley A et al. (1997) Development of an upper limb orthotic exercise system In: Proc ICORR 97, Bath, UK, pp 59–62 41. Zeelenberg AP. (1986) Domestic use of a training robot-manipulator by children with muscular dystrophy. In: Foulds R (ed) Interactive Robotic Aids. World Rehabilitation Fund Monograph #37 New York, pp 29–33.
3 Toward a Human-Friendly User Interface to Control an Assistive Robot in the Context of Smart Homes Mounir Mokhtari, Mohamed Ali Feki, Bessam Abdulrazak, and Bernard Grandjean
Abstract The design of robot dedicated to person with disabilities necessitates users implication in all steps of product development: design solution, prototyping the system, choice of users interfaces, and testing it with users in real conditions. However, before any design of any system, it is necessary to understand and meet the needs of the disabled users. In this chapter, we describe our research activity on the integration of a robotic arm in the environment of disabled people who have lost the abilities to use their proper arms to perform daily living tasks and who are able to use an adapted robot to compensate, even partly, the problems of manipulation in their environments. To develop a human friendly interface it is necessary to act on the system itself to make it more flexible and easy to use. Improvement of assistive robot’s functionalities must comply with users environment which is composed of several assistive aids complementary to the robot. To meet this target we should have a consistent knowledge on users’ needs and taking into account their specific types of disability, their restricted possibilities, and also their acceptation level of technologies. This requires multidisciplinary competencies on several research areas, such as computer sciences, networking, robotics, home automation, and also on ergonomic to provide standardized functionalities which allow efficient usability of assistive technological aids. This chapter describes the adaptation software architecture developed for an assistive robotic arm, called Manus manipulator, in the context of smart homes where the robot is considered as an object among the others.
3.1 Introduction Usually our environment became not adapted for people having lost the ability to use their proper lower limbs to walk or their proper arms to perform daily living task, as such opening a door, eating, or even having access to a computer. To compensate their incapabilities people with disabilities have often recourse to assistive technological aids such as, electrical wheelchair to compensate moving function, a robot manipulator to move objects in their environment, environmental control systems to control the home environment, and communication Z.Z. Bien and D. Stefanov (Eds.): Advances in Rehabilitation Robotics, LNCIS 306, pp. 47-56, 2004. © Springer-Verlag Berlin Heidelberg 2004
48
M. Mokhtari et al.
systems to improve their ability to communicate with people or to get access to information by the use of a computer. Consequently the user is confronted to several heterogeneous systems, imposing several user interfaces, providing multiple and complementary functionalities, and forming a whole complex environment that we describe as being a smart environment. This situation is usually described in the literature and by some industrials as the smart homes concept that is not necessary limited to home environment, but also to hospital environment and outside (school, train station, leisure places …). Assistive robotics consists mainly on developing systems aiming to compensate motor capabilities of people having lost the ability to use their proper arm, usually due to spinal cord injuries or muscular dystrophies. Several systems appeared these last two decades, with the goal to perform daily living tasks, such as the Manus manipulator we are using in this research work. Smart homes, or as defined in Europe as domotics, consist on acting on the user environment to make it more accessible by adding automated controlled systems used through a common user interface which is defined as an environmental control system. The smart homes context is not only limited to home environment, but also to hospital, school, outside and so on. In term of tasks we could say that smart homes are dedicated to control systems in the environment, such as doors, windows, lights, TV, VCR and so on, and that assistive robotics is mainly dedicated to manipulate objects, such as gripping object from the floor, drinking, eating and so on. The aim of our work is to develop human-friendly user interface, independent from the controlled system, or from the communication protocols, which must be flexible and personalized for each end user. The objective is to develop a generic and unified user interface able to control, not only the robot manipulator, but also any available equipments in user’s environments, such as electrical wheelchair, telephone, TV, Doors and so on. This implies to experiment existing and emerging technologies to fits the needs of people with disabilities.
3.2 MANUS Assistive Robot The MANUS tele-manipulator is a robot mounted on an electric wheelchair (Fig. 3.1). Its objective is to favour the independence of severally handicapped people who have lost their upper and lower limbs mobility, by increasing the potential activity and by compensating the prehension motor incapabilities. Manus is a robot with six degrees of freedom, with a gripper in the extremity of the arm which permits the capturing of objects (payload of 1,5 kg) in all directions, and a display. All are controlled by a 4x4 buttons keypad or by a joystick with the latest prototype version and soon, a mouse or a touch screen. The 4x4 keypad gives the user the possibility of handling MANUS, and the display unit gives the current state of the MANUS.
3 Toward a Human-Friendly User Interface to Control an Assistive Robot
49
Fig. 3.1. Manus robot used in a supermarket
3.3 Networking Technologies and Developments To design a mobile system optimized to offer mobility for the handicapped people, we need to determine the main parameters and technological means allowing supporting solutions in residential and outside environments. Human-Machine interaction in this system appears not only by the user interface, but it is also dependent on wireless and wired networks protocols which control the devices. Indeed, human-machine interface makes possible to control various environment equipment and taking into account their current status “feedback”, whereas, wireless networks ensure the desired mobility of the users [2]. The Figure 3.2 shows the hardware architecture of our concept of smart home and attempts to outline the networking problems of heterogeneous systems in user’s environment. The remote controller integrates the human-machine interface and is accessible by suitable input to each end user (joystick, touch screen, microphones...). The user interface has as a crucial managing role of various equipments functionalities. Among equipment we distinguish several types of devices: electrical devices (White goods), household equipment (Brown goods), data-processing equipment (Gray goods), and also mobile devices (mobile phones, pocket PCs…). The diversity of these products brings a wide range of networking protocols necessary to manage the whole smart environment, such as radio (ex. Bluetooth and 802.11), Infra-red (ex. IrDA), Ethernet, and Power lines communications. The solution consists on the design of a generic user interface independent of the communication protocols.
50
M. Mokhtari et al.
This approach permits to obtain a rather acceptable time response without weighing down the task of the supervisor. Indeed, supervisor plays the central role by processing various interconnections between protocols to allow control communication to corresponding specific devices [5].
Fig. 3.2. Smart homes concept
3.4 General Software Architecture Based on the Commanus software architecture we have developed a new software architecture which consider the Manus robot as an object of the environment, at the same level as an electrical wheelchair, or a domotic system as a TV or VCR (Fig. 3.3). This software architecture is decomposed into three main layers: − User Interface layer: manage user interface events according to any input device (keypad, joystick, voice recognition…) selected and configured with the OT software according to each end user. Remote maintenance of the system is also planned through the Tele-Maintenance Unit (TMU) [6]. − Human-Machine Interface (HMI) layer: convert user events into actions according to selected output devices (Manus, TV, VCR…) − Low Level controller: deals with the specific characteristics of any output device and according to its communication protocol (infra red, radio…). Full control of the Manus robot has been implemented through this software architecture. Actually we are implementing control of home devices through a unified user interface.
3 Toward a Human-Friendly User Interface to Control an Assistive Robot
51
Fig. 3.3. General software architecture
3.5 User Interface Adaptation Re-design of software control architecture is not sufficient to allow access to smart environment by people with severe disabilities. The problem is that each end user, with his deficiencies and his individual needs, is considered as a particular case who requires a typical configuration of any assistive system. Selecting the most adapted input device is the first step and the objective is to allow the adaptation of available functionalities according to his needs. For this purpose we have developed a software configuration tools, called OT (Occupational Therapist interface), which allows a non expert in computer science to configure easily any selected input device with the help of different menus containing activities associated to action commands of any system including the Manus robot. The idea is to describe each device using XML, to generate automatically corresponding functionalities, and to display actions in an interactive graphical user interface. According to user needs, and to the selected input devices, the OT offers the mean to associate graphically the selected actions to the input device events (buttons, joystick movements…). The OT software is actually running and fully compatible with the Manus robot. The extension to other home equipment is under development.
52
M. Mokhtari et al.
Below we will focus on the improvements performed on the Manus controller to facilitate the use of the robot in real condition, in a non deterministic environment, as the home environment.
3.6 Implementation of a Path Planner To favour the integration of Manus in home environment it was necessary to facilitate its control by providing natural movements of the arm taking into account obstacles. 3.6.1 Gesture Library In the human physiology, any complete natural gesture is described as being twophased: an initial phase that transports the limb quickly towards the target location and a second long phase of controlled adjustment that allow reaching the target accurately. These two phases are defined respectively as a transport component and a grasp component [4]. Each component is a spatio-temporal transformation between an initial state, and a final state of the arm (Fig. 3.4).
Oi (xi , yi , zi , yawi , pitchi ,rolli )
Of (xf , yf , zf , yawf , pitchf ,rollf )
Gripper trajectory Fig. 3.4. Robot configurations characterising a gesture
In our approach, we are interested in automating the first phase. The second one requires complex sensors (such as cameras and effort sensors) that are, from a usability point of view, not suitable to be integrated on the Manus. The gesture library contains a set of generic global gestures corresponding to transport component. Each gesture (Gi) is characterised by an initial operational variable of the robot workspace (Oii) corresponding to the initial robot arm configuration, and a final operational variable (Oif) corresponding to the final robot arm configuration. Each variable (Oi) is defined in the Cartesian space by the gripper position (xi, yi, zi) and orientation (yawi, pitchi, rolli). The gestures generated by our system are linked only to the final operational variables. The path planner is able, from any initial arm configuration, to generate the appropriate trajectory to reach the final configurations.
3 Toward a Human-Friendly User Interface to Control an Assistive Robot
53
3.6.2 Obstacles Avoidance To improve the point-to-point mode which performs movement in blind way without taking into account environmental obstacles, we have decided to design a new strategy based on the dynamic generation of 3D obstacle. One example of commonly performed task with the Manus is gripping a glass from a table as shown in Fig. 3.5. The path planner takes into account obstacles located inside the working space of the robot between the initial and final configuration of the gripper. Physical obstacles are virtually encapsulated in boxes playing the role of forbidden areas. In this case a first box represents the arm column which could not be crossed during the movement, and a second box representing the table obstacle. Intermediates points defining the robot trajectory are generated by the use of a developed avoidance algorithm based 3D geometrical calculation [2, 3, 10, 11]. Actually, the 3D virtual boxes are defined statically to validate path planner functionalities. A dynamic definition of forbidden area is currently under design to allow the user defining obstacles according to his own changing environment. The path planner integrates the intermediate points when calculating an automatic gesture as defined above. Consequently, the control of the arm should be simplified for the user and this will offer gain in term of time and of control efficiency.
Fig. 3.5. Path planning process avoiding obstacle
This concept is complementary to the co-autonomy concept, described below, where, in case of a non defined obstacle, the user should always have the ability to modify the trajectory generated by the planner trajectory.
54
M. Mokhtari et al.
3.7 Towards the Co-autonomy Concept The co-autonomy concept was recently introduced as a promising way to design assistive robots intended to meet the needs of disabled people [1]. This concept is based on the control charring between the human and the assistive robot. This approach was also proposed for obstacle avoidance in telerobotic systems applications in hazardous environments [3]. Three types of situations were mentioned to define the co-autonomy concept. 1. User is in total control. 2. Machine is in total control. 3. User and the machine share the control. The software command architecture is designed to fit this co-autonomy concept. In the first version of the command architecture, the first and the third type of the situations cited above can occur. Users are in total control when they are using the Cartesian Mode and share the control with an autonomous controller when they are using the Point-to-Point Mode. As describe in [4] the gesture in the Point-to-Point mode is controlled by the user by pressing, for example, the keypad button continuously until completion. The gesture stops if the button is released or continues otherwise (We can qualify such control as a pseudo-sharing control). This was designed to prevent collisions with the user, other persons, or obstacles. Pressing a button of a keypad or pushing a stick of a joystick until completion of the gesture may sometimes be exhausting for some users with severe disability. To prevent from this fatigue, we thought to include the second type of situation of the co-autonomy concept in the command architecture and integrate the user in the autonomous control loop, i.e. allowing him/her to intervene during the automated gestures. The user may then, during the progression of the arm towards the target, make gripper position adjustments. For example, it could occur that the path planner generates a trajectory that would go throw an obstacle. In this case, a collision of the arm with the obstacle will happen. The user may then, act on the input device to ovoid this collision. Such a situation may be done, as shown in Fig. 3.6, in three phases. An automatic phase, where the end-effector follows the trajectory processed by the path planner, a semi-automatic phase where the user intervenes to avoid the obstacle, and finally, another automatic phase, when the user stops intervening, and where a new trajectory towards the target is generated. We have called such control mode Pointing-and-Doing Mode which is complementary to Point-to-Point Mode. As shown in Fig. 3.6, the task is performed following different phases: st − 1 phase: autonomous phase − The end-effector follows the trajectory processed by the path planner nd − 2 phase: semi-autonomous phase − The user intervenes during the autonomous phase to avoid the obstacle rd − 3 phase: autonomous phase. The user stops intervening: A new trajectory towards the target is generated.
3 Toward a Human-Friendly User Interface to Control an Assistive Robot
55
Fig. 3.6. Pointing-and-Doing mode
3.8 Conclusion In this chapter we have outlined the developments done on the Manus robot, mainly on the software architecture, through the Commanus project, and the evolution of the control system to be adapted to the smart environment. This system is designed on one hand, to reduce manipulation problems for user when using the robot in their environment, and on the other hand, to solve the problems linked to the user-interface. With its new functions, we plan to reduce the task time and the number of commands necessary for complex task. The application of results obtained in assistive robotics in the context of smart homes is considering the Manus robot as a standard object in an environment composed of several objects. The main advantage of this approach is, in one hand, to promote the use of assistive robotics as a function of a whole system, and on the other hand, to favour the integration of a unified and adaptable user interface.
56
M. Mokhtari et al.
This work is currently supported by the GET1 through a national project on smart homes for dependant people. The continuation of this research work to improve the Manus robot and facilitating its integration in daily users environment is insured through the AMOR2 project which is starting with the support of the European Commission.
Acknowledgment The authors would like to thank the people who have participated actively in this presented research work, in particular C. Rose from AFM, and J.P. Souteyrand from INSERM U.483 for graphical design. Funds for this project are provided by GET and Foundation Louis Leprince Ringuet through the Smart Homes project, in association with ENST Bretagne and ENST Paris; and by European Commission through AMOR project.
References 1. 2. 3.
4.
5.
6.
Chatila R, Moutarlier P, Vigouroux N (1996) Robotics for the impaired and elderly persons, IARP Workshop on Medical Robots. Vienna, Austria Feki MA (2002) Communication development system in case of smart homes. Engineering Handicam Lab report, ENIS-Sfax, Tunisia Guo C, Tarn TJ, Xi N, Bejczy AK (1995) Fusion of Human and Machine Intelligence for Telerobotic systems", IEEE International Conference on Robotics and Automation, Nagoya, Japan, pp 3110–3115 Jannerod M (1981) Intersegmental coordination during reaching at natural visual object. In: Long J and Baddeley A (eds.) Attention and Performance IX, Hillsdale, NJ: Lawrence Erlbaum Associates, pp 153–169 Mokhtari M, Abdulrazak B, Fki MA (2003) Human-smart environment interaction in th case of severe disability. Proc. 10 International Conference on Human Computer Interaction (HCI’2003). Greece Truche C, Mokhtari M, Vallet C (1999) Telediagnosis and remote maintenance system on Internet network for the Manus robot. Fifth European Conference for the Advancement of Assistive Technology (AAATE’99), IOS Press, Düsseldorf, Germany.
1 GET: Groupe des Ecoles des Télécommunications, which federates several telecommunication Engineering Schools, including INT. 2 AMOR project EEC Growth program: Mechatronic upgrade & wheelchair integration of the Manus Arm manipulator. Partners involved: Exact dynamics, TNO-TPD and Koningh in the Netherlands, Ideasis and ExpertCam in Greece, Lund University in Sweden, HMC in Belgium, and INT and AFM in France
4 Welfare-Oriented Service Robotic Systems: Intelligent Sweet Home & KARES II Z. Zenn Bien, Kwang-Hyun Park, Dae-Jin Kim, and Jin-Woo Jung
4.1 Introduction Utopia – it would be a society where the welfare of the people is properly guaranteed. In the society, each constituent would live his/her life with the blessed feeling of equality. In particular, it would be a society where the elderly and even the handicapped would live well, independently, and comfortably along with people without disabilities. It is instructive to note that the number of the elderly is drastically increasing along with the number of the handicapped caused by a variety of accidents in the complicated and diversified society [64]. In order to realize a welfare-driven society, it is essential to build an infrastructure with a variety of convenient facilities, high-tech equipments, and systems based on advanced technologies with human-friendliness. Rehabilitation robotics is mostly concerned with application of the robotic technology to the rehabilitative needs of people with disabilities as well as growing population of the elderly [28]. Rehabilitation robotic systems aim to solve daily living problems in individual activities. One may say that the primary role of rehabilitation robotic systems is to endow as much independence as possible so as to improve human life quality. Typically, rehabilitation robotic systems are classified as a kind of service robots in the area of robotic technology, while they are also considered as some form of assistive devices in rehabilitation engineering. In fact, major functions of such robotic systems are two types: one for replacement (or rehabilitation therapy) of the user’s handicapped function [46, 52, 60] and the other for assisting the user to carry out necessary tasks. In this chapter, we shall concentrate on developing the second type of robotic systems with assistive functions, and, to this end, shall consider the development of a welfare-driven smart home for high quality of daily life and an indispensable assistive robotic system for the severely-handicapped. The importance of smart home for the elderly and the handicapped may be well understood from the various existing studies such as AID project [7], Smart House project at University of Sussex, HS-ADEPT [25], HERMES, the Smart Home project at Brandenburg Technical University, the Gloucester Smart House, the SmartBO Project [21], the smart house at the Colorado University [55], Welfare Techno Houses in Japan, Robotic Room at the University of Tokyo [56], etc. Considering the existing smart homes, we comment some of the concepts that are implemented in our Intelligent Sweet Home project. In the proposed smart home, some recent technology innovations are considered as well as some specifics in Z.Z. Bien and D. Stefanov (Eds.): Advances in Rehabilitation Robotics, LNCIS 306, pp. 57-94, 2004. © Springer-Verlag Berlin Heidelberg 2004
58
Z. Z. Bien et al.
the lifestyle and traditions in Korea. Since R&D of smart home is urgently in demand to comply with the needs of human being to lead more convenient and safe living and to deal with the increase in the number of the elderly and the handicapped, Intelligent Sweet Home for assisting the elderly and the handicapped, developed at the Human-friendly Welfare Robot System Engineering Research Center in KAIST, aims at development and testing of new ideas for the future smart home and their control. Our work is based on the idea that the technologies and solutions for such a smart home should be human-friendly, i.e. smart homes should possess high level of intelligence in their control, actions and interactions with the users, offering them high level of comfort and functionality. According to their implemented form, the assistive robotic systems are divided into three kinds: 1) Workstation-based system, 2) Mobile robot-based system, and 3) Wheelchair-based system. Some of the workstation-based systems assist the user by using the voice command. DeVAR (Desktop Vocational Assistant Robot) [22], TIDE-RAID (Robot for Assisting for Integration for the Disabled) [27], ISAC (Intelligent Soft Arm Control) [39], IST-MATS (Mechatronic Assistive Technology System) [32] and AFMASTER [1] are well-known examples. Basically, the workstation-based system performs various delicate tasks in a stable mode but its operation is confined to a predefined limited workspace due to lack of mobility. The mobile robot-based system consists of robotic arm and mobile platform. This system is used for transportation of small baggage, guidance for the user, and so on. Walky [18], MoVAR (Mobile Vocational Assistant Robot) [66], TIDE-MOVAID (MObility and actiVity AssIstance system for the Disabled) [16], Care-O-bot I/II [9] and Helpmate [31] are good examples in this category. The wheelchair-based system is currently focused on assistance of the daily living activities of the elderly and the physically handicapped. This type of assistive robotic systems adopts various user interfaces. MANUS [47], FRIEND [53] and RAPTOR [15] are well-known examples of this category. Since, in assistive robotic systems, the human-robot interaction technology becomes increasingly important for user’s convenience and safety in addition to autonomous function of the robotic operations, we report some important results in designing and evaluating KARES II, which is newly developed in KAIST, considering the various human-friendly interfaces and the adaptability to the user. The following items are considered as very important factors for future direction of R&D on the rehabilitation robotics [18, 33]: − − − −
Intelligent interaction/interface that is adaptable to the levels of disability Human-friendly design that assumes the user’s comfort Development of the technology for the user’s safety Increase of the system’s autonomy to compensate the user’s laborious direct control.
Considering these factors, this chapter introduces the development of welfareoriented service robotic systems – Intelligent Sweet Home and KARES II.
4 Welfare-Oriented Service Robotic Systems: Intelligent Sweet Home & KARES II
59
4.2 Intelligent Sweet Home Intelligent Sweet Home consists of four main parts based on the results of statistical survey [40,57] and questionnaire survey [45] for the demand of the users: intelligent bed robot system, intelligent wheelchair system, transferring system, and human-friendly interfaces including soft remote control system, intention reading mechanism and health monitoring system. The information exchange between subsystems is performed by the home network using wired and wireless communications. An overall view of the current scenario of the Intelligent Sweet Home is shown in Fig. 4.1. The target users of the system will be preferably the elderly and the physically handicapped based on our statistical survey. And the functionalities of the system are derived from the demand of the users.
Fig. 4.1. Overall view of Intelligent Sweet Home
4.2.1 Questionnaire Survey We conducted a questionnaire survey for the people with disability in their limbs. More than 70% of them are elderly people and most of them in the hospital or in the rehabilitation center for a long time. The total number of participants is 70. We first asked a number of questions to understand inconveniences of the participants in daily life, and Table 4.1 shows the results of the survey about what are they feel mostly inconvenient in their basic living. Their responses were concerned with eating meal, easing nature and small activities such as handling nearby things and slight moving of one’s body. Table 4.1. Causes of inconveniences in daily life Function Eating meal Moving on the bed Bedsore Transferring between bed & wheelchair Easing nature Using home appliances Putting on/Taking off one’s clothe
Percentage of respondents 51% 58% 55% 76% 96% 76% 90%
60
Z. Z. Bien et al.
Our system is developed to eliminate or ease those inconveniences by assisting the activities of the users. In the following, such systems for this purpose are described in more detail. 4.2.1.1 Questionnaire Survey on Intelligent Bed Robot System We have asked the potential users several questions for designing an efficient bed robot system. These questions are related to the life style of users and the function that they hope the bed to do. In the survey, the users who stay in bed for more than 10 hours are about two times as many as the users who stay for less than 10 hours. Table 4.2 shows that the main inconveniences of a current bed system which the handicapped feel are movement on bed or movement between bed and wheelchair, since they cannot use one’s arms and legs freely and they tumble down and slip on bed. To surmount this type of difficulties, we have designed the robotic structure to have a supporting bar. Table 4.2. Inconveniences in bed Function Movement between bed and wheelchair Movement on bed Management of feces and urine Reading book or newspaper Avoidance of decubitus Eating Hobbies
# of respondents 10 10 7 5 5 4 2
4.2.1.2 Questionnaire Survey on Intelligent Wheelchair System The powered wheelchair is an important rehabilitation device for the handicapped and the elderly. We carried out a survey to identify various requirements of potential wheelchair users. Surveys were conducted with 62 handicapped persons, among whom 12 are powered wheelchair users and 50 manual wheelchair users. Most of them had a spinal cord injury (SCI) and, especially, powered wheelchair users had SCI at C4 ~ C6. We found that about 51% of respondent has spent over 9 hours daily in the wheelchair. Thus, the wheelchair can be a major important rehabilitation device for the handicapped to maintain daily life. In spite of indispensability, many potential users do not employ the powered wheelchair because of cost, difficulty of transferring between bed and wheelchair, safety, unfriendliness appearance, maintenance, and so on. In response to a question regarding preferable control input device for operation, 55% of the surveyed preferred to voice recognition, while 19% wanted touch screen and 26% wanted joystick. Most of the participants wished that the wheelchair would support automatic battery charging, prevention of decubitus which was frequently occurred in hips, door passage, and autonomous navigation, as shown in Table 4.3. Some participants suggest that driving a
4 Welfare-Oriented Service Robotic Systems: Intelligent Sweet Home & KARES II
61
car in wheelchair, standing/lying function for the circulation of the blood, smooth motion control, and adjustment of height of chair are also desirable. Table 4.3. Most necessary function in wheelchair Function Automatic battery charging Prevention of decubitus Door passage Autonomous navigation Home appliance control Others
Percentage of response 19.3% 18.4% 18.4% 15.8% 14.0% 14.1%
4.2.1.3 Questionnaire Survey on Transferring System between Bed and Wheelchair The purpose of this survey for the handicapped is to find out what kind of help in moving between bed and wheelchair can make them feel convenient and necessary. Totally 45 people answered questions. Among them, 35 people can move between bed and wheelchair by themselves, while the other 10 people need some assistants. Table 4.4 shows the result of the survey. Table 4.4. Transfer form between wheelchair and bed
Transfer process
Necessity of some system Reason
By oneself (can move by own force) 1. Put away armrest in wheelchair 2. Put down leg on the floor between bed and wheelchair 3. Use two arms and move hips to bed − Necessary: 13 persons − Unnecessary: 10 persons − No response: 6 persons Comfort
1. 2. 3. − − −
By help of family (difficult to move by oneself) Put away armrest in wheelchair Put down leg on the floor between bed and wheelchair Hold arms to bed and transfer body by protector’s help Necessary: 14 persons Unnecessary: 1 person No response: 1 person Safety
Among those who responded in the survey, we found that about 70 % says that the current way of transfer is very inconvenient or uncomfortable, and that most of them can not take shower nor go to a stool without assistant. 4.2.1.4 Questionnaire Survey on Home Appliance Control by Intelligent Man-Machine Interface We have asked to choose a reason why conventional remote controllers are inconvenient to the handicapped among the following examples: 1. It is difficult to bring it to him 2. The size of its button is too small to push
62
Z. Z. Bien et al.
3. It is a heavy task to push the button 4. It is cumbersome to need individual one for each appliance 5. No problem. Among the examples, most of respondent checked no. 1 and no. 4 as shown in Fig. 4.2. From this result, we have confirmed that it is necessary to develop a system that can control most of home appliances in a natural and easy way.
Fig. 4.2. The reason for inconvenience of conventional remote controllers
4.2.2 Assistive Systems 4.2.2.1 Intelligent Bed Robot System [37] Based on the survey that we conducted, we have developed an intelligent bed robot system composed of a pressure sensor-laid bed and a manipulator as shown in Fig. 4.3. Most previous researches have focused on the system which can monitors the patient's posture and motion on bed [56]. We, however, propose a robotic system which can actively help the patient using a robotic manipulator. While the patient is on the bed, the pressure sensors monitor his posture and motions. When he moves on the bed, the robotic manipulator can support his body.
Fig. 4.3. Intelligent bed robot system
In developing the pressure sensor-laid mattress, a set of Force Sensing Resistors (FSR) is used as a pressure sensor as shown in Fig. 4.4. The resistance value
4 Welfare-Oriented Service Robotic Systems: Intelligent Sweet Home & KARES II
63
of the FSR decreases in proportion to applied force on the active surface. Meas2 urement range of force is up to 10 kg/cm and the size of pressure mattress is 1900 mm×800 mm. The spatial interval is 70 mm in vertical direction and 50 mm in horizontal direction. This sensor pad is divided into three modules in order to correspond to reclining beds. The sampling frequency of the images is 10 Hz with its resolution being 10 bits. By using these pressure distribution images, posture and gross movement estimation are realized.
Fig. 4.4. Arrangement of pressure sensors in bed mattress
In designing the robotic manipulator, the parallel manipulator was proposed for supporting the patient’s body. The mechanism consisted of three actuated arms, which were attached to the mobile base via the driven revolute joints as shown in Fig. 4.5. A 50 W DC motor was used as the actuator. Three passive arms were also attached to an upper platform via passive revolute joints. This parallel manipulator is built on the moving platform that is actuated by a 50 W DC motor. The parallel manipulator is responsible for subtle motion on the fixed position of the mobile platform, and the mobile platform can move between the ends of the bed so that the manipulator can reach any position on the bed.
Fig. 4.5. Linkage mechanism in bed robot
64
Z. Z. Bien et al.
4.2.2.2 Intelligent Wheelchair System [41] We have developed an intelligent wheelchair that can help daily life of the elderly and the physically handicapped. We have first analyzed commercial power wheelchair and developed interface board between wheelchair and PC controller based on real-time Linux, RTAI. The system has two incremental encoders and laser range finder to sense environment, localize its position, and detect obstacle as shown in Fig. 4.6. Fig. 4.7 shows the result of laser range finder in intelligent wheelchair and Fig. 4.8 shows the consecutive localization result that was carried out in Intelligent Sweet Home in Fig. 4.1.
Fig. 4.6. Intelligent wheelchair
Fig. 4.7. Result of laser range finder
Fig. 4.8. Localization result
Since charging of a wheelchair battery is one of the tiresome burdens for the handicapped, we concurrently developed a battery charging station and plug for autonomous charging. Fig. 4.9 shows the prototype system. In order to ensure mechanical and electrical security, we have used several micro-switches and relays.
4 Welfare-Oriented Service Robotic Systems: Intelligent Sweet Home & KARES II
65
In an undocked state, all electrodes are electrically disconnected to the charging station and plug. During docking process, charging begins by relay control of PC.
Fig. 4.9. Battery charging station and plug
4.2.2.3 Transferring System We have developed the robotic system that can transfer the handicapped between bed and wheelchair. This system consists of mobile part, man-machine interface part and charging system. Fig. 4.10 and Table 4.5 show the mobile part of the system and its specification, respectively. Sling part was specially designed for safety and comfortability. We have developed the man-machine interface part as shown in Fig. 4.11, to be applicable for both left and right handler, and even the handicapped. We have also designed the charging system that is applicable to the other system and can guarantee its safety as shown in Fig. 4.12. Since charging battery is one of the tiresome burdens for the handicapped, the battery charging station and plug will be very helpful to the handicapped.
Fig. 4.10. Transferring system
66
Z. Z. Bien et al. Table 4.5. The specification of the mobile part Part Overall length Narrowest external base width Docking foot inside width Docking foot height Unit net weight Max speed
Length 1330 mm 650 mm 650 mm 250 mm 100 kg 0.4 m/s
Fig. 4.11. Man–machine interface of transferring system
Fig. 4.12. Automatic charging system in transferring system
4.2.2.4 Home Network and Management System [62] Information exchange between subsystems is performed by the home network using wired and wireless communications based on Ethernet. The configuration of the network adopts both server-client and peer-to-peer methods. When a new subsystem is added or an existing subsystem is removed, the server updates address map and send the map to each subsystem. The server collects the state of subsystems and shows the information using a graphic user interface. On the other hand, some operations are achieved using peer-to-peer communication. For example, when the user wants to move from the bed to the wheelchair, the corresponding subsystems directly communicate each other so that they can move to the specified place according to some predefined procedure. For effective management of Intelligent Sweet Home, an outdoor management is also investigated. Even when the elderly or the handicapped is left alone at
4 Welfare-Oriented Service Robotic Systems: Intelligent Sweet Home & KARES II
67
home, the conditions of the home environment can be controlled and checked by caregivers by using internet. 4.2.3 Intelligent Man-Machine Interfaces 4.2.3.1 Soft Remote Control System [5, 19, 58] Significant part of Intelligent Sweet Home is devoted to development of innovative solutions for more natural human-oriented interface for control of homeinstalled devices. From the survey result, we have learned that frail handicapped would feel much comfortable if the applied HMI does not require any attachment of sensors to the user. In our study, we propose soft remote control system and voice recognition system. Soft remote control system is a concept for control of the robot and home appliances by predefined hand gestures remotely sensed by ceiling-mounted CCD cameras. There are two modes of control of the home environment [5]: − Simple mode: User selects devices by pointing to them, and then dwelling or voice is used to confirm the selection and activation of the selected device. Next pointing to the same device turns it off. This mode is applied to “on-off” control of home appliances such as TV, lamps, curtains (opening/closing), etc. − Extended mode: User first activates a mode where a list of tasks and services appears on the TV screen by using a hand gesture. Then, pointing to the TV and moving one’s hand, the user selects from the menu a command that should be executed. Last, taking certain hand posture or using voice commands, the user confirms selected command and initiates its execution. Such mode can be used for changing of the TV channels, setting of home environmental parameters such as indoor temperature, light intensity, sound loudness of audio devices, as well as for selection of pre-programmed tasks that will be automatically executed by the robot or other home-installed devices. Three ceiling-mounted CCD color cameras with pan/tilt motions are used to acquire the image of the room. For the simple identification of the commanding hand in the complex background, it is assumed that the user should wear a color (red & blue) hand band. The color hand band is tracked by means of the condensation algorithm [30]. Then, image segmentation is applied to extract the hand color region from the neighborhood of the color hand band region. For representation of raw data, a feature extraction procedure is also included. It is followed by pointing recognition procedure that recognizes the pointing gesture and calculates the orientation angle and pointing direction of the hand. The control procedures end with sending appropriate IR signal for controlling home appliances. We have tested pointing gestures in Intelligent Sweet Home as shown in Fig. 4.13. When the user points at the TV, as the result of recognizing the pointing gesture, the TV is turned on/off. As an extended version, we are testing pointing gestures with 3×3 menu matrix on TV (Fig. 4.14). With the menu, we can control
68
Z. Z. Bien et al.
appliances including home robots. By taking hand posture and orientation into consideration, this system enables supplementary extension of possible number of commands and enhances freedom of user’s movement.
Fig. 4.13. Soft remote controller used in Intelligent Sweet Home: hand pointing recognition
Fig. 4.14. Extended mode of Soft Remote Controller with 3×3 menu matrix
4.2.3.2 Intention Reading in Bed [37] Until now, there have been many intelligent bed systems but most of them were focused on the monitoring of user’s behavior on bed based on temperature, pressure, or vision sensor. So, it can tell what he is doing but cannot tell what he wants to do. Compared with other systems, our research is also focused on finding what he likes to do, i.e. user’s intention as an input of the system as shown in Fig. 4.15. This function becomes very important for the elderly and the handicapped since they are not good at mobility and at manipulation of devices. To recognize the user’s intention, we use pressure sensors on bed as mentioned in 4.2.2.1. When the user intends to lift his/her body up, center of pressure (COP) moves toward his/her hip and total contacting area (TA) becomes bigger. When the user intends to lower his/her body down, COP moves toward his/her head and
4 Welfare-Oriented Service Robotic Systems: Intelligent Sweet Home & KARES II
69
TA becomes smaller. By these observations, we use COP and TA as the features for recognizer and use Hidden Markov Model (HMM) as recognizer since the intention feature data is a sequential form as shown in Fig. 4.16. From 6 person’s experiment, we could find that intention recognition rate is directly proportional to the ratio of intention interval and motion interval. Fig. 4.17 shows the recognition results of body lowering HMM and body lifting HMM during its motion. In Fig. 4.17, y-value means recognition rate and x-value means time. Motion was started at x = 60 and ended at x = 140 . By setting proper intention interval, we could get user’s moving intention. We are now studying the effect of mattress on pressure distribution and classification of various intentions including turning or moving.
Fig. 4.15. Interaction between user and intelligent bed robot
Fig. 4.16. Intention recognition algorithm
70
Z. Z. Bien et al.
a
b Fig. 4.17. Recognition rates: (a) body lowering, (b) body lifting
4.2.3.3 Health Monitoring System [4, 49] Human health monitoring system is one of the intelligent man-machine interfaces in the sense that it can detect the unconscious human intention [67]. For example, when someone catches a cold, his or her body will generate heat during the sleep. In this example, human health monitoring system can detect the heat from the user and give him/her a proper task such as raising the room temperature. Since the elderly or the handicapped are always apt to become sick suddenly, the health monitoring system is much needed and important [38]. Thus, the health monitoring system must be able to detect the basic human signs by telemetries, diagnose the emergent level of some disease and tell the doctors or the caregivers by network as an example in Fig. 4.18. If this system can know the pre-information about the user’s health, then the reliability of this system will be bigger. Finally, the elderly and the handicapped can have a conviction about his or her health using this health monitoring system.
Fig. 4.18. Diagram for human health monitoring system
For Intelligent Sweet Home, we have developed telemetry for various biosignals with 4-ch wireless bio-signal monitoring system as shown in Fig. 4.19.
4 Welfare-Oriented Service Robotic Systems: Intelligent Sweet Home & KARES II
71
Test results for various bio-signals are shown as examples in Figs. 4.20, 4.21 and 4.22.
Fig. 4.19. 4-ch wireless biosignal monitoring system
Fig. 4.20. EMG test result during human walking
Fig. 4.21. Human ECG test result
Fig. 4.22. EEG test result during human sleep
72
Z. Z. Bien et al.
4.3 KARES II System In 1996, we developed KARES I system as a wheelchair-based rehabilitation robot system [64]. KARES I can provide four basic tasks (pick a cup on the table, pick an object on the floor, approaching a cup to the user and switch on/off on the wall) to help severely physically handicapped person. KARES I consists of a 6DOF robotic arm with a mono-vision system for visual servoing, voice recogTM nizer, 6D force-torque sensor and 3D input device (SpaceBall ). All the subsystems are integrated into a powered wheelchair platform. Although KARES I contains many factors as pointed out by [18, 33], human-friendliness and adaptability to the levels of disability have to be considered for the better system. Furthermore, due to flexibility of the rubber tire of powered wheelchair, the vibration of the robotic base causes very unstable operation of the robotic arm during the task. The lessons learned from KARES I and analysis of many conventional rehabilitation robotic systems enable us to address two technical issues as follows:
− A new kind of rehabilitation robot system is required for comfortable and robust operation. Although KARES I is a compact version of various sensors and technologies, exact operation is hard to achieve due to vibration of the robotic base caused by rubber tire of the powered wheelchair. One may say that a workstation-based system (e.g. ISAC of Vanderbilt University [39]) is one solution for resolving vibration problems since such a system enables the robot to operate in the stable mode. However, the workstation-based system cannot provide enough workspace due to its limited mobile capability. Thus, a novel combination of a wheelchair-based system and a workstation-based system can be adopted to realize a futuristic rehabilitation robotic system. In this case, a mobile base plays a key role due to its stable operation during stoppage as well as its free-to-move capability. − To cope with variety of handicaps, man-machine interaction/interface should be realized in a modular form. On the contrary to conventional rehabilitation robot systems, a futuristic rehabilitation robot system has to provide wide range of services for variety of handicaps. The modularized approach of dealing with different degrees of disability will give the user a benefit of minimum redundancy regarding the components/subsystems, and thus enhance the user affordability about cost. In KARES II system, various kinds of interaction/interface are implemented so that the user can choose according to the degree of disability. Comparing with conventional rehabilitation robots, KARES II shows several unique features as follows:
− KARES II, a product based on “Task-oriented Design” (TOD), has adaptability to the user according to his/her level of disability. From the able-bodied to the severely spinal cord injured (lesion level C4 or C5), KARES II can support twelve predefined tasks. The twelve tasks are collected from six month’s sur-
4 Welfare-Oriented Service Robotic Systems: Intelligent Sweet Home & KARES II
73
veys for the handicapped in the factories and clinics. Based on the survey, those specific twelve tasks were defined according to the usability, feasibility and relationship with the rehabilitation purpose. − KARES II possesses high-level autonomy along with various human-robot interaction interfaces. − KARES II is a successful combination of the fixed workstation-type system and the wheelchair-based system. Through previous experiences on KARES I, we have ensured that this hybrid type system is operation-wise effective in many cases. − Opinions by user trials have been used for refining not only the technical aspects but also aesthetic (or human-friendly) design of KARES II system. This section describes some important requirements and philosophy of design as well as the overall system structure of KARES II system which we have developed for assisting the daily living activities of the physically handicapped. 4.3.1 Questionnaire Survey KARES II system is implemented according to the principle of Task-Oriented Design (TOD) [10] which confirms realization of the predefined tasks, and as a byproduct, it may further attain additional tasks owing to the flexible nature of a robotic system. Specifically, we surveyed as a first step basic activities of end-users (i.e. the spinal cord injured people and the other physically handicapped) and caregivers for six months (see Table 4.6). Table 4.6. Information for samples of survey Period Locations Type(numbers) Living situations
1999.1 – 1999.6 hospital(1), industrial workplace(3), asylum(6) quadriplegia(21), poliomyelitis(9), mental disorder(6), others(4) inpatient(24), outpatient(or dwelling at home)(16)
After having examined those activities collected through surveys, about 150 items were related as possible tasks in a brainstorming process. Then, these items were categorized according to their usability, feasibility and suitability for assistive purposes. Finally, twelve basic tasks were determined as described in Table 4.7. These tasks are the ultimate target for TOD which has guided our development of the subsystems such as a robotic arm, necessary user interfaces, and other hardware modules. Along with the notion TOD, we have taken the concept of “humanfriendliness” into consideration as design philosophy. Since the robotic arm is very likely to interact with human users, safety should be guaranteed when the robotic arm makes contact with the users. For the user interfaces, easier accessibility to the system is required since most of the handicapped people are not good at operating robotic systems. For human-friendliness of the robotic arm, we have
74
Z. Z. Bien et al.
adopted active compliance control for the safety of a user in contact with the robotic arm. For easier accessibility to the system, all the user interfaces have a capability of faster execution of each task. In addition, the appearance of every subsystem of KARES II is paid attention so that the subsystems look human-friendly and comfortable. Another important design philosophy considered in KARES II is “modularization of subsystems”. In consideration of the variety of the level of handicaps, accessible interfaces should differ for cost effectiveness and simplicity. The modularized subsystems can make it possible to construct a personally optimized system. Table 4.7. Twelve tasks for KARES II Task no. 1 2 3 4 5 6 7 8 9 10 11 12
Task name Serving a meal Serving a beverage Wiping/Scratching face Shaving Picking up objects Turning switches on/off Opening/Closing doors Making tea Pulling a drawer Playing games Changing CD/tapes Removing papers from printer/fax
Distance between user & robot hand Near Near Near Near Far Far Far Far Far Near/Far Near/Far Near/Far
4.3.2 Overall Structure 4.3.2.1 H/W Structure of KARES II System As shown in Fig. 4.23, KARES II system consists of the wheelchair platform with various user interfaces (specified as positions 3–6) and the mobile platform with a robotic arm for compliance control and for visual servoing (positions 1 and 2 in Fig. 4.23). Here, the mobile platform is essential to perform tasks that consist of operations far away from the user. In the mobile platform, the mobile base gives mobility and extends the workspace of KARES II system. Considering twelve tasks in Table 4.7, we have found that the mobile platform is very effective to perform some tasks which should be done in the spots far from the user: the tasks include picking an object, turning switches on/off and opening/closing the door. We have also concluded that the mobile base needs not to be omni-directional but autonomous. For the robotic arm manipulation, a six DOF robotic arm with all revolute joints is developed to perform the twelve predefined tasks. It has the PUMA type Denavit-Hartenberg parameters [17] and the lengths of links are optimized for the predefined tasks. The design procedure begins with task points of the predefined twelve tasks and as shown in Fig. 4.24, the optimized arm is obtained [10].
4 Welfare-Oriented Service Robotic Systems: Intelligent Sweet Home & KARES II
75
Fig. 4.23. KARES II system: conceptual view
Fig. 4.24. Design procedure of KARES II
As the wheelchair platform for human-robot interfaces, a powered wheelchair with programmable capability is adopted [29], so that the interface developed by us can control the wheelchair itself. 4.3.2.2 S/W Structure of KARES II System: I/O Relations and Control Architecture Fig. 4.25 shows the I/O relations among all the subsystems of KARES II. As we can see from Fig. 4.25, KARES II system includes various human-robot interfaces for smooth communication and comfortable interaction between the user (the elderly and the people with spinal cord injury) and the robotic arm (position 1). Each interface commands velocity and position in response to the inputs from the user. The user can control the robotic arm as well as the wheelchair itself using various interfaces such as EMG interface (position 6), Eye-mouse (position 3), head interface (position 4) and shoulder interface (position 5). Choice may be dictated according to the levels of disability. In the case of shoulder interface, the user can acquire status information of the system through a haptic feedback function. Also, the visual servoing subsystem (position 2) can provide with two kinds of services:
76
Z. Z. Bien et al.
(i) autonomous service in which the user’s intervention is not necessary and (ii) human-friendly service in which the user’s intention can be acquired via the user’s facial expressions. The latter one is considered as a form of feedback from the user to robot.
Fig. 4.25. KARES II system: I/O relations among each subsystem
Realization of KARES II system as a whole is possible only if the system is constructed based on some efficient control architecture. Fig. 4.26 shows the control architecture of KARES II. If the user selects a certain task through GUI, an overall task sequencer (OTS) decides the necessary interfaces and the sequence for the task. Based on the arrangement of the overall task sequencer (OTS), a submodule task sequencer (STS) decides the corresponding module’s action-like request information to a sub-module, the command to the actuators, notification of the result (if any), and so forth. The sequencer acts as the central coordination unit within the control architecture [54]. Even though we have confirmed that the integrated system based on the above control architecture works reasonably well, there are rooms for optimization. For improvement, we find that a top-down approach of architecture is preferred to the evolutionary design approach in consideration of the following aspects [54]:
− Enhancement and reuse of current software − Modularity of subsystems to support easy exchange of components − Implementation of distributed system concept in order to achieve scalable computing performance.
4 Welfare-Oriented Service Robotic Systems: Intelligent Sweet Home & KARES II
77
Fig. 4.26. Control architecture for KARES II system
4.3.3 Soft Robotic Arm with Visual Servoing In this section, we describe two human-robot interaction technologies employed for the robotic arm of KARES II. First, we report that the robot system is designed to have the compliance function. Notably, as shown in Table 4.7, Task 3 and Task 4 require it in the course of task execution. The compliance function can increase the safety level when there occurs an unexpected collision with the user. Moreover, it can provide more comfortable services to the user. Second, it is noted that the robotic arm is equipped with a visual servoing function. This function is required not only for detecting and locating an object autonomously but also for basic intention reading by analyzing facial expressions of the user. 4.3.3.1 Active Compliance Control of the Robotic Arm [10, 11, 12] Since our robotic arm should have various values of compliance in accordance with the twelve tasks, the active compliance control (ACC) is implemented on our robotic arm. In our approach, we have implemented ACC without any force or torque sensors. For ACC, the controller requires not only the position measurement but also the force which is usually measured from the force sensors or the torque sensors. However, we have adopted a method for sensorless measurement for simple and low-cost design. Provided below is the method for sensorless torque measurement. At a static contact situation, the output torque of a motor ( τ motor ) is equal to the external torque ( τ ext ) due to the contact. By using this fact, we can easily know
τ ext while the motor of the robotic arm is controlled to exert τ motor . But, unfortunately, this easy method requires some hard constraints such as negligible back-
78
Z. Z. Bien et al.
lash and friction. These constraints can hardly be satisfied for those robotic arms with speed reducing gears since most gears, by nature, have frictions and backlashes. As a remedy to this problem, the cable-driven mechanism is adopted for speed reduction. The cable-driven mechanism is known to have negligible friction and backlash [65]. Accordingly, it enables the sensorless torque sensing. To realize the desired compliance, Time-Delay Control Based Compliance Control is proposed [12]. This concept is easily implemented while providing with efficiency in control performance. This compliance increases the level of safety for unexpected collision, and furthermore, can make the user feel more comfortable when he/she is performing contact-type tasks such as shaving and wiping of his/her face. To confirm the compliance control algorithm proposed above, a simple experiment is conducted. For a static configuration of the robotic arm, an external force is given as shown in Fig. 4.27. The desired compliances are given by Cd 1 = 2.21 deg/Nm, Cd 2 = 1.17 deg/Nm, and Cd 3 = 0.39 deg/Nm. The results show that the desired compliances are realized at the first three joints, which verifies that the torque-senseless compliance control works well.
Fig. 4.27. Schematic diagram for the experiment
4.3.3.2 Visual Servoing [43, 63] Visual servoing (Fig. 4.25, position 2) module is adopted to provide a visionbased control for autonomy of the robotic arm [13, 59] and to implement a humanfriendly interface for face recognition and intention reading. In the first version of our wheelchair-based rehabilitation robotic systems called KARES I [64], we had found that visual servoing is not an easy task due to requirements of real-time control and robustness to varying illumination, and in particular, the performance is deteriorated due to vibration of the robotic base supported by flexible rubber tires of the wheelchair. In KARES II system, we have separated the robotic arm from the wheelchair platform and have used a vision technique called “space variant vision” for real-time control and robustness to varying illumination. For effective execution of the predefined tasks, we have used a novel stereo camera head in an eye-in-hand configuration [63]. The developed smallsized/light-weighted stereo camera head is installed on the robotic arm in eye-in-
4 Welfare-Oriented Service Robotic Systems: Intelligent Sweet Home & KARES II
79
hand configuration [14]. For fast image processing, the log-polar mapping (LPM) is adopted which is a kind of space variant vision technique [6]. Since LPM image shows invariance to the scaling and the rotation, in addition to its high image reduction ratio (of 22:1 in our system), it is very suitable for visual servoing with eye-in-hand camera configuration [36]. Here, we report that we have performed an experiment of “intention reading” by utilizing the visual images obtained through visual servoing. We assumed that one can show his/her intention to drink or not to drink by opening or closing one’s mouth. Thus, we implemented an intention reading skill based on the information about the user’s mouth [43]. Fig. 4.28 shows sequential images of the user’s face with different degree of mouth openness and the result of intention reading. According to the extracted features about the user’s mouth, we can easily estimate the positive/negative level of the user’s intention to drink or not to drink. With 110 facial images, this method results in a reasonable classification rate (92.7%) [43].
a
b
Fig. 4.28. Intention reading from sequential images: (a) face images, (b) extracted intention
4.3.4 Intelligent Human-Robot Interfaces For the user of KARES II, there are four types of human-robot interfaces as will be described shortly, and it is proposed that the user choose a proper combination of interfaces according to his/her level of disability. Such combination has an advantage of guaranteeing better reliability of the system. Table 4.8 shows a guide to select appropriate interfaces according to the level of disability. In fact, selection of an interface is determined not only by the level of disability but also by the residual functional ability of the user. Table 4.8. Appropriate human–robot interfaces according to the level of disability and the residual functional ability Interface Eye-mouse Head interface Shoulder interface EMG interface
Head/neck Ο Ο × Ο
C4 shoulder(partial) Ο Ο ¨ Ο
C5 shoulder Ο Ο Ο Ο
arms(partial) Ο Ο Ο Ο
80
Z. Z. Bien et al.
4.3.4.1 Eye-Mouse [42, 68] In order for the people with severe motor disability such as C4 lesion to use the KARES II system, it is recommended to use the Eye-mouse system as an input device. The users can indicate the position of the object that they want to grab and give commands to the robot to do something about the object by using the Eyemouse on a computer that is mounted on the wheelchair. So far, there have been many techniques reported to obtain the eye-gaze direction [23]. Those methods can be divided into two types: the contact method and the non-contact method. In the non-contact method, no device is attached on a user’s head while some sensor around the user estimates the user’s eye-gaze direction. CCD camera [3, 20] has been popularly used since it needs no attached device that may cause inconvenience for a user. However, this method has lower accuracy than the contact method and need a head-pursuing system under the free head condition. In the contact method, the head pose and eye movement is obtained using some device that is attached on the user’s head [2, 61]. This method is more accurate than the non-contact method and has no need for a head-pursuing system. However, a disadvantage of this method is that it is inconvenient due to the attached device. In spite of the inconvenience of the contact method, we adopt the contact method because an interface system for the handicapped should be accurate and reliable under the free head condition. Some commercial systems are available adopting the contact method (e.g., [2, 61]), but the design of the systems is not suitable for supporting the handicapped in daily life, so we have developed our own system. We have developed a human-friendly Eye-mouse system based on the opinion of the handicapped (Fig. 4.29). Here, head pose is measured by using ‘Magnetic Sensor Receiver’ on the cap. Eye-gaze direction is acquired by image-based method using CCD camera, IR LED and mirror [20, 44, 48].
Fig. 4.29. Proposed Eye-mouse system
4 Welfare-Oriented Service Robotic Systems: Intelligent Sweet Home & KARES II
81
4.3.4.2 Biosignal-Based Interface [26] EMG (electromyogram) signal is a form of electric manifestation of neuromuscular activation associated with contracting muscle. KARES II adopts EMG interface for the user with disability who can move one’s shoulder or head for controlling a robotic arm or a powered wheelchair. We have developed a small sized LNA (low noise amp)-type EMG AMP with differential AMP to remove common mode component noise, two biquad (2nd order) notch filters to reduce hum noise [34], and a band-pass filter to remove high frequency band. To extract the user’s intentions from muscle movement of shoulders, we defined basic motions as shown in Fig. 4.30. To tackle the user dependency problem exhibited in many previous works, we propose an algorithm capable of classifying the bio-signals obtained from different subjects into the predefined classes using a Fuzzy C-means algorithm and a rough set-based technique selecting a necessary and sufficient set of features out of all feature combinations extracted [26]. The overall signal processing procedure is briefly explained as follows. EMG signals of the predefined motions are measured from four predetermined muscles (channels) with electrodes attached to each subject. A second order high pass butterworth filter with 30Hz cut-off frequency is used for reducing low frequency noise such as motion artifacts. Well-known features such as integral absolute value (IAV), variance (VAR), zero crossing (ZC), frequency ratio (FR) are extracted to classify the predefined motions from noiseless EMG signals. By applying a well-established feature extraction algorithm [26] to these numerous extracted features, we can obtain the minimized feature combination sets which have enough information for complete classification. The minimized feature combination sets extracted by the proposed algorithm are used as input-output pairs to make Fuzzy Min-Max Neural Networks (FMMNN) learn the motions. After learning, FMMNN gives the classified results and actuates the robotic arm based on the user’s movement. In the experiment1, the basic motions are recognized with success rates of approximately 90 % for four untrained users.
Fig. 4.30. Basic 8 motions
1
Each user has 10 trials of each motion for validation.
82
Z. Z. Bien et al.
4.3.4.3 Head&Shoulder Interface [50, 51] Head interface is a two DOF interface for people with C4 lesion. It is used for body-operated control of a wheelchair and a robotic arm. Force sensitive resistor (FSR) is a suitable element for developing a human-robot interface satisfying the guidelines because of its characteristics: low price, ease to measure force, arbitrary shape, and thinness. Human head motion is analyzed in order to determine the motion detection range of the head interface. Average maximum tilt angles are 41° for the front, 73° for the rear, and 60° for right and left side. A head interface valid in the analyzed range (73°) has been developed as shown in Fig. 4.31(a). Shoulder interface is a wearable sensor suit converting the human body motion into a useful command [50, 51]. Humans shoulder motion is also analyzed in the same method as in the previous subsection. The average maximum ranges of shoulder motion are 7.5 cm for the front, 7 cm for the rear, 10.1cm for the upper direction, and 2.5 cm for the downward direction. We have decided that the lift motion of shoulder is most useful for human-robot interaction. A tension sensor measuring the lift motion of shoulder has been developed as shown in Fig. 4.31 (b).
(a)
(b)
Fig. 4.31. Main components of head/shoulder interface: (a) angle sensor, (b) tension sensor
4.3.5 User Trials We have performed user trials by letting six subjects with spinal cord injury use the various functions of KARES II system. Six subjects were rehabilitants of Korean NRC (National Rehabilitation Center) in Seoul who are currently taking appropriate rehabilitation programs (see Table 4.9). The order of user trials is basically composed of following steps. At first, we showed to the subjects how to execute the scenario of “Task 2” in Table 4.7 by using the integrated system. We helped then the subjects to understand the objective of KARES II system. Next, we let the subjects experience by actually letting them operate and observe the various interface subsystem/modules of the system. In every case, some short questionnaires are given to the users with detailed ex-
4 Welfare-Oriented Service Robotic Systems: Intelligent Sweet Home & KARES II
83
planations by us. Next, based on the collected qualitative answers and quantitative measurements for each subsystem, we have drawn the figure which depicts a distribution of the satisfaction degrees (0-100 %) according to the predefined evaluation aspects of each subsystem2. From these procedures, we have obtained the results of user trials. Table 4.9. Information for subjects subject ID A B C
sex
age
M M M
33 36 35
lesion level C4 C5 C4
residual motor ability Head/neck shoulder arms ¨ Ο × Ο Ο × ¨ Ο ×
D
M
51
C5
Ο
Ο
Ο
E
M
21
C4
Ο
¨
×
F
M
31
C5
Ο
Ο
¨
technical aids none none none powered wheelchair none powered wheelchair
4.3.5.1 Robotic Arm When the task of shaving or cleaning-face is conducted by the robotic arm, there exists some form of contact between the trial subject and the robot hand. For such a task, the robot must have some level of compliance for safety of the user. In these cases, the magnitude of compliance may be dependent on each individual and on the kind of tasks. In order to implement comfortable shaving and cleaningface tasks, we investigated a preferred compliance level in each task by performing the task, for the users followed by a set of questionnaires. Three levels of compliance are used in the user trial for both tasks as shown in Table 4.10. Table 4.10. Three levels of compliance [deg/Nm] 1st axis 2nd axis 3rd axis
Level 1 1.232 0.7815 0.1309
Level 2 2.053 1.303 0.218
Level 3 4.107 2.61 0.436
Compliance is realized to each of the 1st, 2nd and 3rd axes in the form of joint compliance in consideration of link length. In Table 4.10, Level 1 is the lowest compliance level (or the highest stiffness case), and Level 3 is the highest compliance level (or the lowest stiffness case). Based on the investigation of human arm compliance done in [24], we have designed three levels; the first one is smaller than that of human arm, second one is nearly the same as that of human arm and the third one is bigger than that of human arm. 2
Predefined evaluation aspects are diverse due to each subsystem's unique characteristics.
84
Z. Z. Bien et al.
For the shaving task, we applied three levels of compliance to six subjects and found that the preference of the handicapped is distributed as shown in Fig. 4.323. Fig. 4.32 (a) and (b) show the degree of safety and the degree of satisfaction of each compliance level that was obtained from the interviews for shaving task case. As shown in Fig. 4.32, Level 2 and Level 3 mark higher preference in regard to safety and comfortable feeling. This result shows that in case of shaving task, the trial subject prefers the level 2 of compliance which corresponds to a weak nurse arm’s compliance [24]. In the task of cleaning-face, we have found that all three levels of compliance in Table 4.10 fail to render a satisfactory result: the degree of satisfaction is very low. We learned that the six subjects preferred to compliance that is lower than the presented values in Table 4.10, which means that the task of cleaning-face needs a more strong contact force with high stiffness to perform the task in a satisfactory way. Also, we have found that it is desirable to position a towel in a desirable position and let the trial subject move their head to clean up the face.
a
b Fig. 4.32. Evaluation of shaving task by the subjects: (a) safety, (b) ease of use
4.3.5.2 Visual Servoing The visual servoing subsystem consists mainly of a newly-developed stereo camera head in an eye-in-hand configuration, a function for object recognition, a function for face recognition and a function for intention reading based on facial expression recognition. In the user trials, we have asked all the subjects about their opinions on physical appearance of the visual servoing mechanism. Fig. 4.33 shows the evaluation summary of visual servoing subsystem. As shown in Fig. 4.33, the stereo camera head should be redesigned to provide more human-friendly appearance. Also, most of the subjects pointed out some difficulty of the language of our GUI due to its small-sized font and English-expression. For easy use and higher satisfaction degree, GUI should be organized using our mother tongue language with bigger character size. In the aspect of function for object recognition, the number of recognizable objects should be predefined by the user candidates and should be increased so as to be capable of handling various 3
In every figure, each degree of satisfaction is represented by ‘Boxplot’ (Boxplot 2002).
4 Welfare-Oriented Service Robotic Systems: Intelligent Sweet Home & KARES II
85
objects in our real life. Finally, we report that all the users are satisfied with the function for intention reading. For more general usage of intention reading, recognition of some parts of face (other than mouth) is recommended.
Fig. 4.33. Results from use trials for visual servoing
4.3.5.3 Eye-Mouse We have carried out user trials with the proposed Eye-mouse system. The accuracy result of the estimated eye-gaze direction in Table 4.11 was obtained when the distance between the user and the monitor is about 500 mm. The two numbers in the each cell of the first and second rows of the table are the horizontal error and the vertical error, respectively. With these errors, a suitable size of the button of the interface program was computed as shown in the third row of the table. The fourth row includes the maximum resolution in case of a 15" monitor. It is noted that the result is worse than the resolution (14×12) obtained in the laboratory [48] and the results are different for different users. The possible reasons for these results are as follows: 1. To extract eye-gaze direction using the Eye-mouse, the center coordinate of eye should be obtained anatomically with respect to the receiver of the magnetic sensor 2. Since the motions of eyes include saccadic motion, concentration is needed for the users to fix gaze on one point 3. Because the proposed method to track the pupil is a vision-based method, it can be affected by illumination of surroundings. From the survey results shown in Fig. 4.34, we may say that most of the users were satisfied with the structure of the interface program. Especially, the convenience of the additional ‘OK’ button for click operation was acclaimed. The users also said that it is easy to control the pan/tilt unit due to the function of ‘automatic pushing’ when the mouse pointer moves into the button. The satisfaction degree
86
Z. Z. Bien et al.
about the design and easiness for wearing the system was turned out to be acceptable. The users wanted the system to be light and not to be tight. They also pointed out the problem of perspiration on the surface of contact. Since the proposed method is a ‘contact’ method, tiredness will be a critical problem when using the device for a long period of time, and thus, the level of tiredness needs to be observed for a longer period of user trials. Table 4.11. Experimental results of accuracy of Eye-mouse Subject Error Mean (pixel) Error STD (pixel) Possible Button Size (mm) Possible Monitor Resolution (15")
1
2
3
4
5
6
(-36.1,-6.7)
(6.3,-7.1)
(1.3,20.0)
(-44.5,-8.9)
(-49.9,-4.8)
(66.4,-5.7)
(50.4,45.1)
(98.4,88.4)
(32.6,15.4)
(63.9,79.8)
(46.1,34.6)
(31.3,43.1)
51.5 × 31.1
62.3 × 57.2
20.2 × 21.2
64.6 × 53.2
57.2 × 23.6
58.2 × 29.2
5.9 × 7.4
4.9 × 4.0
15.1 × 10.8
4.7 × 4.3
5.3 × 9.6
5.2 × 7.9
Fig. 4.34. Results of inquiries about Eye-mouse
4.3.5.4 Head&Shoulder Interface Two experiments were performed to evaluate the performance of the shoulder interface and head interfaces which can make 2 DOF signals. The experimental process is shown as follows: 1. Wear our new interface and try to make four direction signals, which are forward, backward, right and left direction. This list of actions is for familiarizing the user with the new interfaces. 2. Operate the real wheelchair while adapting oneself to the moving mechanism with the new interface.
4 Welfare-Oriented Service Robotic Systems: Intelligent Sweet Home & KARES II
87
3. While driving the wheelchair along the predefined path shown in Fig. 4.35, the total elapsed time from start to end point, and the number of collision and recognition rate of interface are measured. 4. Evaluate the performance of new interface.
Fig. 4.35. Predefined path for wheelchair control
The results for wheelchair control according to the path are shown in Fig. 4.354. Table 4.12 is the results about head interface control and shoulder interface control. We conducted another experiment in which the same procedures are conducted by some ordinary person who uses hand to operate joystick. This latter result shows that the elapsed time is 21 seconds while the average number of collision is 0.5 times. In comparison with this case of ordinary person, the experimental result of using joystick shows that the traveling time by the handicapped is twice longer than that of ordinary person and the collision occurs more frequently as many as five times. But people with spinal cord injury do not necessary to spend much adapting time to control the powered wheelchair by our new interfaces. They control the wheelchair along the path with 2 times collision on the average. This result of user trial presents big potential for applicability in that even the severelyhandicapped can control the power wheelchair without any assistance. Also, the results show that our interfaces may not perfectly work with all the handicapped. In the test of shoulder interface, the first and the fifth subject could not use each shoulder independently and therefore the experiment was not carried out. And in the head interface test, the first subject had difficulty in tilting the head for roll motion so he could not turn to appropriate direction. From the survey results in Fig. 4.36 (a) and (b), we may say that the subjects were satisfied with overall impression about both interfaces. Most of the people who used the shoulder interface put in the overcoat said it is a very attractive device because it does not attract other people’s attention. 4
Based on Jones et.al (Jones et al. 1998), we modify the path.
88
Z. Z. Bien et al. Table 4.12. Experimental results of two interfaces Interface Type
subject Elapsed Time (s) Number of Collision Recognition rate (%)
1
Head interface 2 3 4 5
6
1
2
Shoulder interface 3 4 5
6
45
42
58
92
120
34
30
113
28
2
0
2
2
5
2
3
2
1
80
80
65
80
60
80
65
55
80
a
b
Fig. 4.36. Survey results about two interfaces: (a) head interfaces, (b) shoulder interface
4.3.5.5 EMG Interface Wheelchair control was performed also by an EMG interface. To assess the performance of EMG interface objectively, we have measured the total elapsed times, and the number of collisions while driving along a predefined path shown in Fig. 4.35. After experiment, we interviewed the users with questionnaire for subjective evaluation. We tested two control modes as shown in Table 4.13. The difference between Mode 1 and Mode 2 is a command for forward movement. In Mode 1 case, the wheelchair will go forward until a user keeps up-forward command motion such as both shoulders up. In Mode 2 case, the same command was executed by using a toggle switch which makes the wheelchair go straight or stop according to its current state. We attached four electrodes (two channels, bipolar type) for measuring EMG signals in both Trapezius muscles. Table 4.13. Motion command for controlling of wheelchair Motion Initial state Both shoulders up Right shoulder up Left shoulder up
Wheelchair motion in Mode 1 Stop Forward movement Right movement Left movement
Wheelchair motion in Mode 2 Current state hold (forward/stop) Forward/Stop (toggle) Right movement Left movement
4 Welfare-Oriented Service Robotic Systems: Intelligent Sweet Home & KARES II
89
After experiment, we have found that the forward command in Mode 1 has made the users to get tired because the users had to maintain forward motions till the wheelchair reached the target position ahead. The users, however, gave forward command easily in Mode 2 by both shoulders up. The difficulty of forward command usage in Mode 2 is the wheelchair response delay time. The wheelchair response delay time makes the user confuse whether the controller accepts the command correctly. After using this controller for a while, however, the user adapted to this delay time, and felt more comfortable than the forward command in Mode 1. The users also commented about electrode attachment and its outlook appearance. Some users disliked the procedure of electrode attachment (including skin preparation) to skins. Overall wheelchair control performance of the subjects is not as good as ordinary persons, but we have ascertained that EMG interface based on the head movement or shoulder movement would be applied as a wheelchair controller to the users with spinal cord injury with lesion level C4 and C5 (see Fig. 4.37).
Fig. 4.37. Results from user trial for EMG interface
4.4 Concluding Remarks As the preliminary result of developing welfare-oriented robotic systems, we have suggested and implemented several subsystems that help the elderly and the handicapped have independent daily living. We have further collected feedbacks from the design stage according to evaluation by the handicapped. We believe that one of the most important things in welfare-oriented robotic systems is to adopt various service robots to help the habitants in many ways. The idea of Intelligent Sweet Home is treated at present not as a science fiction but as important goal with strong social and economics aspect. Its realization will give a solution of many existing problems of the welfare society and will make the handicapped/elderly’s lives as well as human life much pleasant and easier. We have also shown that, by hybridizing a workstation-based frame and a wheelchair-based one, a novel type of rehabilitation robotic system can be realized
90
Z. Z. Bien et al.
to have advantages of the two types, and that modularized man-machine interface/interaction is realized to cope with the variety of handicaps. In realizing the user’s input commands and interaction mechanism, various human-robot interfaces including Eye-mouse, head/shoulder interfaces, and EMG signal interface are developed so as to cope with different levels of disability. Based on our experiences of developing Intelligent Sweet Home and KARES II, and the feedback from the users, we have concluded that the proposed systems need further improvement in several aspects as follows:
− Further study is needed to design a convenient operation methodology of the system on behalf of novice users and long-term handling. More sensitive and wide intention reading capability of various kinds is desirable for humanfriendly interaction. − Although each subsystem performs its own functions well, we find that a central decision maker is desirable as a means of communication between subsystems for exchanging necessary information. If this kind of decision maker is installed, each subsystem could have worked fully without additional software programming. In this case, what the subsystems need to do is only to send information that is requested from the decision maker. And also, if it is necessary to add a new task, the system operator may modify the decision maker properly only. In this way, we can make the system more simple and flexible. − It is necessary for each subsystem to check whether the information communication is well under way or not. This capability is necessary in preventing safety reduction because of communication failure.
Acknowledgement This research is supported by Human-friendly Welfare Robot System Engineering Research Center (Sponsored by KOSEF) of KAIST and the Ministry of Science and Technology of Korea as a part of Critical Technology 21 Program on “Development of Intelligent Human-Robot Interaction Technology”. We like to acknowledge various forms of support from Prof. Ju-Jang Lee, Prof. Byung Kook Kim, Prof. Jin-Oh Kim, Prof. Jong-Tae Lim, Prof. Heyoung Lee and their student staffs in developing Intelligent Sweet Home, and acknowledge helpful comments on extended mode of soft remote control system from Dr. Dimitar Stefanov. We also like to acknowledge a variety of aid from Prof. Pyung-Hun Chang, Prof. Myung Jin Chung, Prof. Dong-Soo Kwon, and their student staffs in developing KARES II system, and acknowledge assistance from Dr. Byung-Sik Kim and his staff of National Rehabilitation Center, Korea in user trials.
4 Welfare-Oriented Service Robotic Systems: Intelligent Sweet Home & KARES II
91
References 1. 2. 3. 4.
5.
6.
7.
8. 9. 10. 11.
12.
13. 14. 15. 16. 17. 18. 19.
20.
Afma-robots (2003) AFMASTER, http://www.afma-robots.com ASL501 (2003) Model 501. http://www.a-s-l.com/501_home.htm ASL504 (2003) Model 504. http://www.a-s-l.com/504_home.htm Bang W, Stefanov D, Jung J, Kim M, Lee J, Lee H, Bien Z (2001) Human-friendly th health monitoring system for service to the elderly and disabled. Proceedings of the 8 International Conference on Rehabilitation Robotics (ICORR2001), Evry Cedex, France, pp 333–339 Bien Z, Park KH, Kim JB, Do JH, Stefanov D (2003) User-friendly interaction/interface control of intelligent home for movement-disabled people. Proceedings th of the 10 International Conference on Human-Computer Interaction, Crete, Greece, Vol. 4, pp 304–308 Bolduc M, Levine MD (1998) A review of biologically motivated space-variant data reduction models for robotic vision. Computer Vision and Image Understanding 69(2):170–184 Bonner S (1998) AID HOUSE: Edinvar housing association smart technology demonstrator and evaluation site. Proceedings of the 3rd TIDE Congress, Helsinki, Finland, pp 396–400 Boxplot (2002) http://www.shodor.org/interactivate/activities/boxplot/ Care-O-bot (2003) http://www.care-o-bot.de/english/Care-O-bot_2.php Chang PH, Park HS (2003) Development of a robotic arm for handicapped people: a task-oriented design approach. Autonomous Robots 15(1): 81–92 Chang PH, Park HS, Park J, Jung JH, Jeon BK (2001) Development of a robotic arm th for handicapped people: a target-oriented design approach. Proceedings of the 7 International Conference on Rehabilitation Robotics (ICORR2001), pp 84–92 Chang PH, Kang SH, Park HS, Kim ST, Kim JH (2003) Active compliance control for th the disabled with cable transmission. Proceedings of the 8 International Conference on Rehabilitation Robotics (ICORR2003), Dajeon, Korea, pp 84–87 Chen N, Parker GA (1994) Inverse kinematic solution to a calibrated puma 560 industrial robot. Control Engineering Practice 2: 239–245 Choi J (2001) Design of a behavior-based controller using a novel camera head and its application to service robots (in Korean). MS Thesis, KAIST, Korea Colello MS, Mahoney RM (2002) Commercializing assistive and therapy robotics. Universal Access and Assistive Technogy, Keates S et al. (eds), pp 223–234 Conte G, Longhi S, Zulli R (1996) Motion planning for unicycle and car-like robots. International Journal of Systems Science 27(8): 791–798 Craig JJ (1989), Introduction to robotics: mechanics and control. Addison-Wesley Publishing Co. Dallaway JL, Jackson RD, Timmers PHA (1995) rehabilitation robotics in Europe. IEEE Transactions on Rehabilitation Engineering 3: 35–45 Do JH, Kim JB, Park KH, Bang WC, Bien ZZ (2002) Soft remote control system using hand pointing gesture. International Journal of Human-friendly Welfare Robotic Systems 3(1): 27–30 Ebisawa Y (1998) Improved video-based eye-gaze detection method. IEEE Transactions on Instrument and Measurement 47(4): 948–955
92
Z. Z. Bien et al.
21. Elger G, Furugren B (1998) SmartBO – an ICT and computer-based demonstration home for disabled people. Proceedings of the 3rd TIDE Congress, Helsinki, Finland, pp 392–395 22. Erlandson RF (1995) Applications of robotic/mechatronic systems in special education, rehabilitation therapy, and vocational training: a paradigm shift. IEEE Transactions on Rehabilitation Engineering 3: 22–32 23. Glenstrup AJ, Engell-Nielsen T (1995) Eye controlled media: present and future state. B.S. Dissertation, Copenhagen University 24. Gomi H, Kawato M (1997) Human arm stiffness and equilibrium-point trajectory during multi-joint movement. Biological Cybernetics 76: 163–171 25. Hammond J, Sharkey P, Foster G (1996) Integrating augmented reality with home systems. Proceedings of the 1st International Conference on Disability, Virtual Reality and Associated Technologies ECDVRAT '96, pp 57–66 26. Han JS, Bang WC, Bien ZZ (2002) Feature set extraction algorithm based on soft computing techniques and its application to EMG pattern classification. Journal of Fuzzy Optimization and Decision Making 1: 269–286 27. Harwin WS, Rahman T, Foulds RA (1995) A review of design issues in rehabilitation robotics with reference to north American research. IEEE Transactions on Rehabilitation Engineering 3: 3–13 28. Hillman M (1998) Introduction to the special issue on rehabilitation robotics. Robotica 16: 485 29. Hillman M, Hagan K, Hagan S, Jepson J, Orpwood R (2002) The Weston wheelchair mounted assistive robot – the design story. Robotica 20: 125–132 30. Isard M, Blake A (1998) Condensation-conditional density propagation for visual tracking. International Journal of Computer Vision 29(1): 5–28 31. ISRA (1995) The service robot market, an in-depth study from the international service association. ISRA 32. IST-MATS (2003) http://www.bcdi.be/en/projects/data.html 33. Iwata H, Hoshino H, Morita T, Sugeno S (1999) A physical interference adapting hardware system using MIA arm and humanoid surface covers. Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, pp 1216–1221 34. Johnson DE (1975) Rapid practical designs of active filters, John Wiley&Sons 35. Jones DK, Cooper RA, Albright S, DiGiovine M (1998) Powered wheelchair driving performance using force- and position-sensing joysticks. Proceedings of the IEEE 24th Annual Northeast Bioengineering Conference, pp 130–132 36. Jruger V (1995) Optical flow computation in the complex logarithmic plane. DiplomaThesis, University of Kiel, Germany 37. Jung JW, Lee CY, Lee JJ, Bien ZZ (2003) User intention recognition for intelligent th bed robot system. Proceedings of the 8 International Conference on Rehabilitation Robotics (ICORR2003), Daejeon, Korea, pp 100–103 38. Kawarada A, Takagi T, Tsukada A, Sasaki K (1998) Evaluation of automated health th monitoring system at the ‘welfare techno house’. Proceedings of 20 IEEE/EMBS, pp 1984–1987 39. Kawamura K, Isakarous M (1994) Trends in service robots for the disabled and the elderly. Proceedings of IROS’94, pp 1647–1654 40. KIHASA (2000) National survey of the disabled persons. Korea Institute for Health and Social Affairs
4 Welfare-Oriented Service Robotic Systems: Intelligent Sweet Home & KARES II
93
41. Kim CH, Jung JH, Kim BK (2003) Design of intelligent wheelchair for the motor disth abled. Proceedings of the 8 International Conference on Rehabilitation Robotics (ICORR2003), Daejeon, Korea, pp 92–95 42. Kim DH, Kim JH, Chung MJ (2001) A computer interface for the disabled using eyegaze information. International Journal of Human-friendly Welfare Robotic Systems 2(3): 22–27 43. Kim DJ, Song WK, Han JS, Bien Z (2003) Soft computing based intention reading techniques as a means of human-robot interaction for human centered system. Journal of Soft Computing 7: 160–166 44. Kim JH, Lee BR, Kim DH, Chung MJ (2003) Eye-mouse system for people with moth tor disabilities. Proceedings of the 8 International Conference on Rehabilitation Robotics (ICORR2003), Dajeon, Korea, pp 159–163 45. Kim Y, Park KH, Seo KH, Kim CH, Lee WJ, Song WG, Do JH, Lee JJ, Kim BK, Kim JO, Lim JT, Bien ZZ (2003) A report on questionnaire for developing Intelligent Sweet Home for the disabled and the elderly in Korea living conditions. Proceedings th of the 8 International Conference on Rehabilitation Robotics (ICORR2003), Dajeon, Korea, pp 171–174 46. Krebs HI, Hogan N, Volpe BT, Aisen ML, Edelstein L, Diels C (1999) Robot-aided neuro–rehabilitation in stroke: three-year follow–up. Proceedings of ICORR1999, pp 34–41 47. Kwee HH (1998) Integrated control of MANUS manipulator and wheelchair enhanced by environmental docking. Robotica 16(5): 491–498 48. Lee BR (2002) A real-time eye-gaze tracking system using infrared rays and vision sensor (in Korean). M.S. Dissertation, Korea Advanced Institute of Science and Technology 49. Lee H, Bien Z (2002) Variable bandwidth filter for reconstruction of bio-medical signd nals with time-varying instantaneous bandwidth. Proceedings of the 2 Joint EMBS/BMES Conference, Houston, USA, pp 141–142 50. Lee K, Kwon DS (2000) Sensors and actuators of wearable haptic master device for the disabled. Proceedings of the 2000 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 371-376 51. Lee K, Kwon DS (2001) Wearable master device for spinal injured persons as a control device of motorized wheelchairs. Journal of Artificial Life and robot 4(4):182-187 52. Lum PS, Burgar CG, Shor PC, Majmundar M, Van der Loos HFM (2002) Robotassisted movement training compared with conventional therapy techniques for the rehabilitation of upper limb motor function after stroke. Archives of Physical Medicine and Rehabilitation 83: 952–959 53. Martens C, Ivlev O, Graser A, Lang O, Ruchel N (2001) A FRIEND for assisting handicapped people. IEEE Robotics and Automation Magazine 8(1): 57–65 54. Martens C, Kim DJ, Han JS, Graeser A, Bien Z (2002) Concept for a modified hybrid multi-layer control architecture for rehabilitation robots. Proceedings of the 3rd International Workshop on Human-friendly Welfare Robotic Systems, Daejeon, Korea, pp 49–54 55. Mozer MC (1999) An intelligent environment must be adaptive. IEEE Intelligent Systems and Their Applications 14(2): 11–13
94
Z. Z. Bien et al.
56. Nakata T, Sato T, Mizoguchi H, Mori T (1999) Synthesis of robot-to-human expressive behavior for human-robot symbiosis. Proceedings of the 1996 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp 1608–1613 57. NSO (2001) The future estimated population. National Statistical Office 58. Park KH, Bien ZZ (2003) Intelligent Sweet Home for assisting the elderly and the handicapped. Proceedings of the 1st International Conference on Smart Homes and Health Telematics (ICOST200), Paris, France, pp 151–158 59. Peters II RA, Bishay M, Cambron ME, Negishi K (1996) Visual servoing for service robot. Robotics and Autonomous Systems 18: 213–224 60. Rao R, Agrawal SK, Scholz JP (2000) A Robot test-bed for assistance and assessment in physical therapy. Advanced Robotics 14(7): 565–578 61. SMI (2003) 3D VOG Video-oculography. http://www.smi.de/3d/index.htm 62. Song WG, Lim JT (2003) Design and management in smart home. Proceedings of the 1st International Conference on Smart Homes and Health Telematics (ICOST2003), Paris, France, pp 33–37 63. Song WK, Bien Z (2003) Blend of soft computing techniques for effective humanmachine interaction in service robotic systems. Fuzzy Sets and Systems 134:5–25 64. Song WK, Lee H, Bien Z (1999) KARES: Intelligent wheelchair-mounted robotic arm system using vision and force sensor. Robotics and Autonomous Systems 28(1): 83– 94 65. Townsend WT (1988) The Effect of Transmission Design on Force-Controlled Manipulator Performance. PhD Thesis, MIT 66. Van der Loos HFM (1995) VA/Stanford rehabilitation robotics research and development program: lessons learned in the application of robotics technology to the field of rehabilitation. IEEE Transactions on Rehabilitation Engineering 3: 46–55 67. Yamaguchi A, Ogawa M, Tamura T, Togawa T (1998) Monitoring behavior in the th home using positioning sensors. Proceedings of 20 IEEE/EMBS, pp 1977–1979 68. Yoo DH, Chung MJ (2002) Vision-based eye gaze estimation system using robust pupil detection and corneal reflections. International Journal of Human-friendly Welfare Robotic Systems 3(4): 2–8.
5 „FRIEND“ – An Intelligent Assistant in Daily Life O. Kouzmitcheva, C. Martens, A. Pape, H. She, I. Volosyak, and A. Gräser
Abstract The research and development in the field of rehabilitation robots produced a multiplicity of rehabilitation robots available as off-the- shelf products or laboratory prototypes (e.g. [1]). The application scope of these systems is large and covers ranges such as support for everyday tasks, assistance in the vocational surroundings or support in health care. An in-depth analysis unveils that rehabilitation robots, which are intended for a flexible use but not for individual special applications, offer services only on a relatively low level of abstraction, i.e. the direct low level control of the system [2] remains by the user. This leads to a high cognitive load with accompanying concentration loss, especially for persons depending on interfaces like speech control or eye movement trackers. In order to relieve the users from this kind of tiresome control the treatment of tasks on higher abstraction level becomes desirable [3]. The system should be able to perform chains of actions, which are repeated during daily life tasks, autonomously and/or with minimum necessary user interaction.
5.1 Basic Concepts and Hardware 5.1.1 The FRIEND Project With the motivation to overcome the situation explained above, the Institute of Automation (IAT), University of Bremen, has been developing the rehabilitation robotic system FRIEND since 1997. The system belongs to the category “intelligent wheelchair mounted manipulators”. It focuses on users with high spinal cord injury that are unable to control the manipulator by means of a keyboard or joystick. The system shall offer support during life activities, so that its users will become independent from care personnel. The strategic objective of the FRIEND project is to offer life autonomy for approximately 2 hours per day. Beside the aspect, that this is one of the main requirements mentioned by the users, the fulfilment of the objective would have a strong impact on the economical acceptance of the rehabilitation robotic system. In order to reach this strategic objective the IAT has chosen the so-called “pour a drink task” as the first daily life task to be offered by the FRIEND system. The realization of a robust and (semi-) autonomous execution of this task unveils a number of challenging technical problems to be solved, that are also representative Z.Z. Bien and D. Stefanov (Eds.): Advances in Rehabilitation Robotics, LNCIS 306, pp. 95-126, 2004. © Springer-Verlag Berlin Heidelberg 2004
96
O. Kouzmitcheva et al.
for other tasks. Therefore, the investigation and realization of the “pour a drink task” have the potential to develop a general method for robust high-level task execution in rehabilitation robotic systems. Within this chapter, the hardware and software structure of the FRIEND are presented. Afterwards, chapter 2 focuses on the realization of the “pour a beverage task” with the help of FRIEND. 5.1.2 Hardware Structure of FRIEND FRIEND (see Fig. 5.1) consists of an electric wheelchair (SPRINT, Meyra, Germany) and a 6 DOF robot arm (MANUS, Exact Dynamics, Netherlands). The arm is connected to a PC, which is mounted on the backside of the wheelchair, via a CAN bus interface. As a user interface an off-the-shelf speech recognition system is used. Other devices may replace speech recognition, as an input device, which may be better adapted to the users needs.
Fig. 5.1. Front view of FRIEND
FRIEND operates in a flexible human centred environment. It must be able to react to environmental changes dynamically during robotic arm movements. For this purpose, the system possesses a number of sensors for environmental perception. For visual perception, the system is equipped with an adjustable stereocamera system, which is mounted behind the users neck, and a gripper mounted camera. Using visual sensors for environmental perception is twofold: On the one hand, it offers the highest amount of information about the current environmental state.
5 „FRIEND“ – An Intelligent Assistant in Daily Life
97
On the other hand, the complexity of signal processing and pattern recognition algorithms based on visual information is tremendous. Additionally, in many cases the reliability of the perceived information is insufficient for a robust task execution. Therefore, the IAT developed a “smart” tray that is mounted at the front side of the wheelchair. The “smart” tray (see Fig. 5.2) offers information about weights of objects placed upon the tray through a scale. The location of objects relative to a coordinate system fixed to the tray is measured through a touchpad. As shown within the succeeding chapters, the reliable information offered by the tray in combination with the visual information perceived by the cameras facilitates robust task execution. Beside the sensor systems mentioned so far, the gripper of the robotic arm is equipped with a force sensitive foil to perform sensitive grasping actions.
a
b
Fig. 5.2. Smart Tray. a top-view on scale surface, b matrix foil of position sensor
5.1.3 Multi-layered Control Architecture of FRIEND The integration of sensors, actuators and human machine interfaces into a computer program is a challenging task from the viewpoint of software engineering. A control-architecture is required, that offers an infrastructure for task planning, user interaction, resource administration as well as for dynamic activation and deactivation of closed loop control processes for the realisation of reactive operations. A dominating design principle for this kind of problem is called hybrid multi-layer architecture. This type of architecture combines the main design-principles for autonomous physical agents i.e. reactive and deliberative agent. Example architectures based on this principle are 3T [4], TCA [5] or SmartSoft [6]. Hybrid multi-layer architectures consist of three layers: The bottom layer realizes the reactive part of the robotic system (autonomous physical agent), i.e. the direct coupling of sensorial input to actuator control. Within this layer, concur-
98
O. Kouzmitcheva et al.
rently running and interacting behaviours determine the interaction of the system and the environment. The top layer −called the deliberator− plans operators on the highest level of abstraction. Here, methods from the field of classical artificial intelligence are used. Between these two layers resides the sequencer. The sequencer creates task schemes based on reactive behaviours that are offered as higher level abstraction operators to the deliberator. Beside this “gluing functionality” this layer is responsible for deadlock free execution of reactive behaviours that have to share limited system resources. Even though the latter aspect is crucial for secure execution, the avoidance of such situations cannot be guaranteed by some implemented architectures [5]. Here, the formal specification and analysis of all possible action sequences on sequencer level, like presented in section 5.2.5, is necessary. As shown in [7], traditional three layered control architectures do not meet the needs within the field of rehabilitation robotics systems. For this purpose a modified layered control architecture was designed that enables autonomous execution interrupted by man-machine interactions (see Fig. 5.3). The main design principle is still based on the hybrid multi-layer concepts and that the integration of reactive behaviours as well as deliberative capabilities becomes possible. Within the modified architecture, a human-machine-interface (HMI) replaces the deliberator. Additionally, it offers a direct control path from the HMI to the actuators and sensors. The system can rely on the user’s cognitive capabilities whenever necessary. The sequencer plays the role of a discrete event controller (DEC) that is responsible for a proper generation of action sequences related to high-level commands. The arrow from the HMI to the sequencer indicates such a command, like pouring and offering a drink. After the receipt of a command, the sequencer processes the following steps: First, the sequencer has to load command-related task knowledge to fix the information necessary for the determination of the current internal and environmental state. The state information is required for the succeeding generation of the command related action sequence and stored within the symbolic part of the world model. On this level of abstraction, a state is defined as a binary vector where each vector element represents a single fact about the environment or the system itself. The determination of the facts that are necessary for a complete state description takes place during the process of task knowledge modelling. In order to acquire the current state at the beginning of the task execution the sequencer queries the world-model about the desired facts. If no information is available, monitoring commands or user interactions are activated by the sequencer in order to update the state description within the world-model. After the initial state has been determined successfully, the sequencer generates sequences of executable elementary operations (EEOPs) that will be passed over to the reactive layer or the human-machine interface. From the sequencers point of view EEOPs are split up into four categories: • Reactive operators (direct sensor and actuator coupling, like visually controlled gripping of an object) • Monitoring commands (e.g. identification of objects) • User commands (e.g. moving camera into a direction of an object)
5 „FRIEND“ – An Intelligent Assistant in Daily Life
99
• Calculation operators (e.g. trajectory planning) • Direct actuator control (e.g. gripper movements). Task
User Interactions
Human-Machine Interface
Direct Actuator Control
Execute Monitoring
Task
User Interactions
User Interaction
Worldmodel
Set Facts
Squencer
Execute Monitoring
Closed Loop Control
Read Facts
Symbolic Represenation of System and Environmental State
Closed Loop Control
Monitoring Operation
Direct Actuator Control
Write subsymbolic information Read and write subsymbolic information
Subsymbolic Represenation of System and Environmental State
Reactive Layer
Sensors
Actuators
Fig. 5.3. Modified hybrid multi-layer control architecture
In order to indicate its execution result, the EEOP returns a discrete value, like Success or NotFound, to the sequencer. Depending on this value the sequencer decides whether to proceed with the execution or to modify the generated sequence, i.e. to perform a re-planning step. The reactive layer is located on the lowest level of the control architecture. This layer consists of software servers that encapsulate the hardware of sensors (e.g. cameras) and actuators (e.g. robot arm). Each server offers its services i.e. monitoring operations or actuator control via client-objects. Each client-object can be
100
O. Kouzmitcheva et al.
instantiated within the context of another process i.e. the sequencer or the humanmachine interface. By means of the clients-objects EEOPs for monitoring, direct control, reactive operation or user interaction are constructed. For instance, if a reactive operation has to be performed, e.g. visually guided grasping of objects, the necessary clients for sensor input and actuator control are instantiated and connected dynamically within the context of the EEOP. The exchange of information between EEOPs is performed via the sub-symbolic part of the world model. Here, beside the administration of the required resources, the sequencer controls the related flow of information. For instance, if visual position information about an object is required by a reactive operation, a monitoring operation that produces this information has to be executed first. If the monitoring operation fails the sequencer is informed via an appropriate return value and it can react to this unforeseen situation. The following chapter describes the development of executable elementary operators that are necessary for the execution of a “beverage serving task”. First, each operator and its integration into the FRIEND system are described separately by means of the stepwise explanation of the task scenario. Afterwards, a task planning approach that makes use of these operators and that is used within the control architecture is presented.
5.2 Application and Control This chapter explains the realization of the “beverage serving task” by means of the FRIEND system. First, the scenario is described on an abstract and intuitive level. This sketchy explanation is followed by a detailed description of the technical realisation of executable elementary operators for object detection, object manipulation, like visually guided object grasping, obstacle avoidance and the pour process itself. To get a robust behaviour this operation is executed under closed loop control that exploits weight information from the “smart” tray and adapts a trajectory obtained via the method of demonstration-based programming. A conclusion of the task explanation is that the ability of task planning becomes a necessary requisite if autonomous task execution is required. Therefore, a real-time suitable task planning approach based on enhanced assembly planning methods is introduced. The resulting task planning component is integrated into the sequencer of FRIEND’s control architecture and makes use of its executable elementary operators. Finally, the demonstration based programming method for the creation of operators is presented in more detail. The programming method helps to create executable elementary operations more easily than traditional engineering approaches. This helps to enhance the systems flexibility and robustness during task execution.
5 „FRIEND“ – An Intelligent Assistant in Daily Life
101
5.2.1 The „Beverage Serving“ Task The scenario can be described as follows: a person (e.g. care personnel) arbitrarily places a glass and an already opened bottle on top of the tray. Afterwards, the user enters the single command “Serve Drink” and the system shall fill the glass with the beverage inside the bottle and move it to the user’s mouth autonomously. After the user finishes drinking, the glass will be put back on the tray, ready for the next serving drinking process. Even though some hard restrictions like empty glass or already opened bottle have been introduced, the realization of the autonomous execution of the task is still a great technical challenge. First, the system has to detect and grasp the bottle. Second, it has to locate the neck of the bottle relatively to the glass to perform a succeeding pour-in action. Third, the pouring process has to be performed and observed to fill the glass sufficiently. Afterwards, the bottle has to be placed back on the tray and the glass has to be grasped. Finally, the glass has to be moved into the vicinity of the user’s mouth. In the following, the realization of these steps is described in detail. 5.2.1.1 Object Detection The ability of object detection is a fundamental requirement in autonomous task execution. Within this scenario object detection based on natural characteristics is required in order to avoid artificial object markings. It turned out that the colour of an object is suitable for this purpose. Unfortunately, using colour information introduces some additional problems compared to grey value image processing. Different external influences like variable lighting conditions can be the reason of large fluctuations in objects’ colours. It might happen that differently coloured objects possess the same colour within the captured image. In order to overcome this problem our object detection algorithm makes use of the HSV (huesaturation-value). The main advantage of this colour space is that the HS-channels for objects of the same colour are almost independent of illumination condition changes. On the basis of this colour space a new method for automatic object recognition based on fuzzy decision system is used for object classification [8]. The result of the method is shown in Fig. 5.4 (a), (b). The system detects the bottle as well as the glass in both images of the stereo-camera system. This is symbolized by the bounding ellipses within the pictures to the right. The detection process itself works with a frame rate of 10fps achieved on a Pentium IV 2.8 GHz platform with an image size of 512 × 256 pixels. This frame rate is necessary for smooth movements of the robot arm if it is operating in visual servoing control mode. The main disadvantage of the presented object identification and detection method is that it does not reason about the detected objects. Within a clustered and unstructured environment, it might happen that wrong objects are identified, because of changing illumination conditions. By using the ‘smart’ tray, we can cope with this problem. With the information from the tray, unreasonable information can be excluded. Fig. 5.4 depicts the identified objects necessary for the pouring action by using image information in combination with the touchpad.
102
O. Kouzmitcheva et al.
a
b
c
d
Fig. 5.4. Object identification. a stereo camera view, b detected objects, c raw sensor information, d touchpad view
5.2.1.2 Object Approaching, Grasping, and Crossover Based on the image information obtained by object detection the pouring action can be performed. In order to implement this part of the scenario the gripper of the robot arm has to grasp the identified bottle and move it to the vicinity of the glass to execute pouring action in closed loop mode. For this purpose, image based visual servoing as well as the “look-and-move” approach are used. Vision Based Object Manipulation A human being grasp objects almost invariably with the aid of vision. He or she uses visual information to locate and identify objects, and to decide how to grasp them. Additionally, they use visual information for obstacle avoidance during the movement of their hand towards the desired object as well as for the correct alignment of the hand. The latter aspect, i.e. the hand-eye coordination, is the basic principle of the visual servoing method for robust robot arm control. The method uses visual information as feedback value in a closed control loop, so that approaching objects becomes independent from calibration errors of the vision system, like camera or position calibration. In visual servoing, it is necessary to generate gripper motions from visual observation. The visual controller (see Fig. 5.5) uses the location of features on the image plane directly for feedback.
5 „FRIEND“ – An Intelligent Assistant in Daily Life
103
Fig. 5.5. Conventional visual controller
The controller includes an image jacobian matrix, which describes the relationship between changes in image plane and in world coordinates [9]. During the control process, the system simultaneously tracks the robot arm and the target, e.g. I bottle. Within this context, the image error e (i.e. control error) is defined as the I distance in both images between the reference point r actual, e.g. gripper, and the tarI get point r desired, e.g. bottle. Driving this error to zero in both images is equivalent to a 3D movement of the gripper into the vicinity of the object to be gripped (see W W Fig. 5.6: u - output of the controller, r – position of the robot gripper in world coordinates). This conventional visual servoing is a well-known method and realised for many applications worldwide. The disadvantage of the method is that fixed cameras with constant, usually small, focal length are used. It reduces the complexity of the control algorithm, but at the same time it restricts the workspace of the system.
Fig. 5.6. Visual servoing based grasping of a glass
In order to enhance the flexibility of the system, it must be possible to manipulate objects that are not placed on the tray. Even in the “gripping from the tray” scenario, it might happen that the robot gripper leaves the common field of view of the fixed cameras, so that vision based control becomes impossible. To overcome this problem adjustable cameras are used. Both cameras are mounted on pan-tilt heads and have variable focal lengths. In this case, the common field of view covers the whole workspace of the robot arm [10]. To use the additional degrees of freedom two additional control loops (Pan-Tilt and Zoom) are included for each camera (see Fig. 5.7).
104
O. Kouzmitcheva et al.
Fig. 5.7. Extended visual controller
The pan-tilt control loop keeps the object in the centre of the image and the zoom loop controls the image size so that the objects of interest always have a sufficient resolution (see Fig. 5.8).
Fig. 5.8. Adaptation of image resolution and camera position during pan-tilt head (PTH) control
The application of these enhanced control loops enlarges the robustness of the visual servoing based grasping process. To enhance the robustness of the system visual information and information from the ‘smart’ tray are merged. With this additional information, the gripper position can be adjusted with the necessary accuracy. “Look-and-move” Based Object Manipulation First, we remind briefly the classical ‘Look-and-move’ paradigm. This approach uses visual sensor information to compute the 3D position and orientation of the target object in order to report a 3D pose for the robot to achieve. The ability to compute relative 3D position and orientation of the object in Cartesian space implies that: 1. a 3D model of the object is available 2. visual features used to recognize and locate the object are represented in the model 3. the visual sensor is calibrated in order to be able to estimate Cartesian positions and orientations.
5 „FRIEND“ – An Intelligent Assistant in Daily Life
105
In order to overcome time-consuming calibration and to exclude the necessity of image models for objects the ‘smart’ tray is used as an additional sensor. This tray provides the information about the placement of the objects relative to the tray coordinate frame. By means of the combination of this information with the results of the colour image processing described in 2.1.1, the 3D position of identified objects can be calculated easily with sufficient accuracy. At first glance, this modified ‘Look-and-move’ method seems to be a suitable technique to control the robot arm in order to manipulate different objects. However, it still has some disadvantages: 1. No real-time correction of the robot path is possible 2. Errors in calibration directly affect the accuracy with which the desired position is determined. 3. The information about 3D object characteristics, e.g. height, is required in order to compute the target position of the robot arm; 4. The provided sensor information is restricted to X and Y directions. 5. The number of executable tasks is restricted to a priori known objects that have to be placed on the tray. In order to overcome the problem of storing object related data and to be able to grasp and to manipulate different objects anywhere in the workspace of the robot arm, a combination of visual servoing and ‘look-and-move’ is used. This avoids the disadvantages of the look-and-move approach and the inaccuracy of visual servoing due to small focal length. For grasping and manipulation of the objects on the tray the result is as follows: 1. After the first object identification the robot gripper will be moved into the vicinity of the object to be gripped by means of visual servoing. 2. Due to the information provided by the ‘smart’ tray the gripper position is adjusted with sufficient accuracy. 3. After the gripper has reached its target position relatively to the desired object, a pre-programmed grasp action is executed. In order to execute the pouring task the bottle has to be moved into predetermined relative position close to the glass. For the execution of this motion, both techniques described above can be used. The difference to the ‘gripping task’ is the definition of the desired position and the reference point for control. In this case, the gripped bottle determines the reference point instead of the gripper. The desired position is calculated with respect to desired relative position of the bottle and the glass determined by the pouring trajectory. After the desired position is reached, the beverage pouring can start. 5.2.1.3 Beverage Pouring To guarantee that no beverage splashes out of the glass, the pouring trajectory as well as the flow have to be controlled. The pouring trajectory is determined by the movement of the bottle tip with respect to the glass and the slope of the bottle. While it is difficult to model this trajectory, it is much easier to obtain it through
106
O. Kouzmitcheva et al.
demonstration [11]. That is, a human instructor demonstrates how the bottle moves during a pouring process. The demonstrated movement will be measured and recorded with a position sensor providing 6 DOF information. This information is transformed to get the bottle’s tip with respect to the centre of glass opening. Hence, the acquired pouring trajectory is independent from specific bottles and glasses, and can be easily applied in a pouring task for various objects. In addition, this demonstrated trajectory is chosen as a general one, which covers most pouring processes regardless different filling levels. For the utilization in the system, the trajectory is stored in form of a list data structure. Figure 5.9 shows the whole procedure. As a result, a human being’s experience on beverage pouring has been observed and stored in the robot system.
a
b
c Fig. 5.9. Generation of the pouring trajectory. a Demonstration, b List of trajectory, c Transformation
When a pouring process begins, the acquired trajectory is used as a reference of the movement of the bottle, while the actual pose of the bottle is controlled by the beverage flow, i.e., the actual trajectory is a modification of the original observed trajectory. Since the initial filling level of the bottle is unknown, the observed trajectory may be repeated partly or wholly till the desired filling weight is achieved. If the whole trajectory has been executed and the filling set point is still not reached, the system will stop the pour-in task and offers a warning message because the bottle is empty.
5 „FRIEND“ – An Intelligent Assistant in Daily Life
107
To eliminate the influence of the initial conditions on the actual pouring task, such as the shape of bottle and glass and the initial liquid level, closed control loop is applied. The ‘smart’ tray measures the filling weight and derives the beverage flow. Figure 5.10 shows the set-up when beverage pouring begins.
Fig. 5.10. System set-up for pouring beverage
The control loop is shown in Fig. 5.11, where Win, Fref and Factual stand for the desired weight to be filled, desired flow range and actual flow respectively. As a result, the pouring process is accomplished without an interaction of the user. Due to closed loop control a very simple structure is able to handle different initial and boundary conditions.
Fig. 5.11. Schema of pouring control loop
5.2.1.4 Put an Object Down After the pouring process is finished, the bottle has to be placed back on the tray. Visual servoing, as used by human, would imply that: 1. Image processing is able to recognize the tray shape and all objects, which are still placed on it; 2. A free position on the tray has been calculated in both camera images;
108
O. Kouzmitcheva et al.
3. The contact of two planes in stereo images (lay at bottom of tray) must be determined. The visual servoing procedure is very robust and accurate as long as the image error between two objects has to be reduced. In the case, that the image error is defined as the difference between two virtual points in the image, the inaccuracy in cameras and correspondence problem in image processing become noticeable. To realize put down action accurately and collision free, it is needed that the image
