Karl-Friedrich Kraiss (Ed.) Advanced Man-Machine Interaction
Karl-Friedrich Kraiss (Ed.)
Advanced Man-Machine Interaction Fundamentals and Implementation
With 280 Figures
123
Editor
Professor Dr.-Ing. Karl-Friedrich Kraiss RWTH Aachen University Chair of Technical Computer Science Ahornstraße 55 52074 Aachen Germany
[email protected]
Library of Congress Control Number: 2005938888
ISBN-10 3-540-30618-8 Springer Berlin Heidelberg New York ISBN-13 978-3-540-30618-4 Springer Berlin Heidelberg New York This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable for prosecution under the German Copyright Law. Springer is a part of Springer Science+Business Media springer.com © Springer-Verlag Berlin Heidelberg 2006 Printed in Germany The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typesetting: Digital data supplied by editor Cover Design: design & production GmbH, Heidelberg Production: LE-TEX Jelonek, Schmidt & Vöckler GbR, Leipzig Printed on acid-free paper
7/3100/YL
543210
To: Eva, Gregor, Lucas, Martin, and Robert.
Preface
The way in which humans interact with machines has changed dramatically during the last fifty years. Driven by developments in software and hardware technologies new appliances never thought of some years ago appear on the market. There are apparently no limits to the functionality of personal computers or mobile telecommunication appliances. Intelligent traffic systems promise formerly unknown levels of safety. Cooperation between factory workers and smart automation is under way in industrial production. Telepresence and teleoperations in space, under water or in micro worlds is made possible. Mobile robots provide services on duty and at home. As, unfortunately, human evolution does not keep pace with technology, we face problems in dealing with this brave new world. Many individuals have experienced minor or major calamities when using computers for business or in private. The elderly and people with special needs often refrain altogether from capitalizing on modern technology. Statistics indicate that in traffic systems at least two of three accidents are caused by human error. In this situation man-machine interface design is widely recognized as a major challenge. It represents a key technology, which promises excellent pay-offs. As a major marketing factor it also secures a competitive edge over contenders. The purpose of this book is to provide deep insight into novel enabling technologies of man-machine interfaces. This includes multimodal interaction by mimics, gesture, and speech, video based biometrics, interaction in virtual reality and with robots, and the provision of user assistance. All these are innovative and still dynamically developing research areas. The concept of this book is based on my lectures on man-machine systems held regularly for students in electrical engineering and computer science at RWTH Aachen University and on research performed during the last years at this chair. The compilation of the materials would not have been possible without contributions of numerous people who worked with me over the years on various topics of man-machine interaction. Several chapters are authored by former or current doctoral students. Some chapters build in part on earlier work of former coworkers as, e.g., Suat Akyol, Pablo Alvarado, Ingo Elsen, Kirsti Grobel, Hermann Hienz, Thomas Kr¨uger, Dirk Krumbiegel, Roland Steffan, Peter Walter, and Jochen Wickel. Therefore their work deserves mentioning as well. To provide a well balanced and complete view on the topic the volume further contains two chapters by Professor Gerhard Rigoll from Technical University of Munich and by Professor R¨udiger Dillmann from Karlsruhe University. I’m very much indebted to both for their contributions. I also wish to express my gratitude to Eva Hestermann-Beyerle and to Monika Lempe from Springer Verlag: to the former for triggering the writing of this book, and to both for provided helpful support during its preparation.
VIII
Preface
I am deeply indebted to J¨org Zieren for administrating a central repository for the various manuscripts and for making sure that the guidelines were accurately followed. Also Lars Libuda deserves sincere thanks for undertaking the tedious work of composing and testing the books CD. The assistance of both was a fundamental prerequisite for meeting the given deadline. Finally special thanks go to my wife Cornelia for unwavering support, and, more general, for sharing life with me.
Aachen, December 2005
Karl-Friedrich Kraiss
Table of Contents
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1
2 Non-Intrusive Acquisition of Human Action . . . . . . . . . . . . . . . . . . . . . . . 2.1 Hand Gesture Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.1 Image Acquisition and Input Data . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.1.1 Vocabulary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.1.2 Recording Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.1.3 Image Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.1.4 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.2 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.2.1 Hand Localization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.2.2 Region Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.2.3 Geometric Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.2.4 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.3 Feature Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.3.1 Classification Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.3.2 Classification Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.3.3 Feature Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.3.4 Rule-based Classification . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.3.5 Maximum Likelihood Classification . . . . . . . . . . . . . . . . 2.1.3.6 Classification Using Hidden Markov Models . . . . . . . . . 2.1.3.7 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.4 Static Gesture Recognition Application . . . . . . . . . . . . . . . . . . . . 2.1.5 Dynamic Gesture Recognition Application . . . . . . . . . . . . . . . . . 2.1.6 Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Facial Expression Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.1 Image Acquisition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.2 Image Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.2.1 Face Localization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.2.2 Face Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.3 Feature Extraction with Active Appearance Models . . . . . . . . . . 2.2.3.1 Appearance Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.3.2 AAM Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.4 Feature Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.4.1 Head Pose Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.4.2 Determination of Line of Sight . . . . . . . . . . . . . . . . . . . . . 2.2.4.3 Circle Hough Transformation . . . . . . . . . . . . . . . . . . . . . . 2.2.4.4 Determination of Lip Outline . . . . . . . . . . . . . . . . . . . . . . 2.2.4.5 Lip modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.5 Facial Feature Recognition – Eye Localization Application . . . . 2.2.6 Facial Feature Recognition – Mouth Localization Application .
7 7 8 8 9 10 11 12 14 20 22 27 29 29 31 32 33 34 37 47 48 50 56 56 59 60 61 67 68 73 75 78 78 83 83 87 88 90 90
1
X
Table of Contents
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
92
3 Sign Language Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 3.1 Recognition of Isolated Signs in Real-World Scenarios . . . . . . . . . . . . . . . . 99 3.1.1 Image Acquisition and Input Data . . . . . . . . . . . . . . . . . . . . . . . . . 101 3.1.2 Image Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 3.1.2.1 Background Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 3.1.3 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 3.1.3.1 Overlap Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 3.1.3.2 Hand Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 3.1.4 Feature Normalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 3.1.5 Feature Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 3.1.6 Test and Training Samples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 3.2 Sign Recognition Using Nonmanual Features . . . . . . . . . . . . . . . . . . . . . . . . 111 3.2.1 Nonmanual Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 3.2.2 Extraction of Nonmanual Features . . . . . . . . . . . . . . . . . . . . . . . . 113 3.3 Recognition of continuous Sign Language using Subunits . . . . . . . . . . . . . 116 3.3.1 Subunit Models for Signs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 3.3.2 Transcription of Sign Language . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 3.3.2.1 Linguistics-orientated Transcription of Sign Language . 118 3.3.2.2 Visually-orientated Transcription of Sign Language . . . 121 3.3.3 Sequential and Parallel Breakdown of Signs . . . . . . . . . . . . . . . . 122 3.3.4 Modification of HMMs to Parallel Hidden Markov Models . . . . 122 3.3.4.1 Modeling Sign Language by means of PaHMMs . . . . . . 124 3.3.5 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 3.3.5.1 Classification of Single Signs by Means of Subunits and PaHMMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 3.3.5.2 Classification of Continuous Sign Language by Means of Subunits and PaHMMs . . . . . . . . . . . . . . . . . . . . . . . . . 126 3.3.5.3 Stochastic Language Modeling . . . . . . . . . . . . . . . . . . . . 127 3.3.6 Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 3.3.6.1 Initial Transcription . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 3.3.6.2 Estimation of Model Parameters for Subunits . . . . . . . . 131 3.3.6.3 Classification of Single Signs . . . . . . . . . . . . . . . . . . . . . . 132 3.3.7 Concatenation of Subunit Models to Word Models for Signs. . . 133 3.3.8 Enlargement of Vocabulary Size by New Signs . . . . . . . . . . . . . . 133 3.4 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 3.4.1 Video-based Isolated Sign Recognition . . . . . . . . . . . . . . . . . . . . 135 3.4.2 Subunit Based Recognition of Signs and Continuous Sign Language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 3.4.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
Table of Contents
XI
4 Speech Communication and Multimodal Interfaces . . . . . . . . . . . . . . . . . 141 4.1 Speech Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 4.1.1 Fundamentals of Hidden Markov Model-based Speech Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 4.1.2 Training of Speech Recognition Systems . . . . . . . . . . . . . . . . . . . 144 4.1.3 Recognition Phase for HMM-based ASR Systems . . . . . . . . . . . 145 4.1.4 Information Theory Interpretation of Automatic Speech Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 4.1.5 Summary of the Automatic Speech Recognition Procedure . . . . 149 4.1.6 Speech Recognition Technology . . . . . . . . . . . . . . . . . . . . . . . . . . 150 4.1.7 Applications of ASR Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 4.2 Speech Dialogs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 4.2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 4.2.2 Initiative Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 4.2.3 Models of Dialog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 4.2.3.1 Finite State Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 4.2.3.2 Slot Filling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 4.2.3.3 Stochastic Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 4.2.3.4 Goal Directed Processing . . . . . . . . . . . . . . . . . . . . . . . . . 159 4.2.3.5 Rational Conversational Agents . . . . . . . . . . . . . . . . . . . . 160 4.2.4 Dialog Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 4.2.5 Scripting and Tagging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 4.3 Multimodal Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 4.3.1 In- and Output Channels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 4.3.2 Basics of Multimodal Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . 166 4.3.2.1 Advantages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 4.3.2.2 Taxonomy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 4.3.3 Multimodal Fusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 4.3.3.1 Integration Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 4.3.4 Errors in Multimodal Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 4.3.4.1 Error Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 4.3.4.2 User Specific Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 4.3.4.3 System Specific Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 4.3.4.4 Error Avoidance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 4.3.4.5 Error Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 4.4 Emotions from Speech and Facial Expressions . . . . . . . . . . . . . . . . . . . . . . . 175 4.4.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 4.4.1.1 Application Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 4.4.1.2 Modalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176 4.4.1.3 Emotion Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176 4.4.1.4 Emotional Databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 4.4.2 Acoustic Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 4.4.2.1 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178 4.4.2.2 Feature Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 4.4.2.3 Classification Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
XII
Table of Contents
4.4.3 Linguistic Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.3.1 N-Grams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.3.2 Bag-of-Words . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.3.3 Phrase Spotting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.4 Visual Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.4.1 Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.4.2 Holistic Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.4.3 Analytic Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.5 Information Fusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
180 181 182 182 183 184 184 186 186 187 187
5 Person Recognition and Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 5.1 Face Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 5.1.1 Challenges in Automatic Face Recognition . . . . . . . . . . . . . . . . . 193 5.1.2 Structure of Face Recognition Systems . . . . . . . . . . . . . . . . . . . . . 196 5.1.3 Categorization of Face Recognition Algorithms . . . . . . . . . . . . . 200 5.1.4 Global Face Recognition using Eigenfaces . . . . . . . . . . . . . . . . . . 201 5.1.5 Local Face Recognition based on Face Components or Template Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206 5.1.6 Face Databases for Development and Evaluation . . . . . . . . . . . . 212 5.1.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214 5.1.7.1 Provided Images and Image Sequences . . . . . . . . . . . . . . 214 5.1.7.2 Face Recognition Using the Eigenface Approach . . . . . 215 5.1.7.3 Face Recognition Using Face Components . . . . . . . . . . . 221 5.2 Full-body Person Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224 5.2.1 State of the Art in Full-body Person Recognition . . . . . . . . . . . . 224 5.2.2 Color Features for Person Recognition . . . . . . . . . . . . . . . . . . . . . 226 5.2.2.1 Color Histograms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226 5.2.2.2 Color Structure Descriptor . . . . . . . . . . . . . . . . . . . . . . . . 227 5.2.3 Texture Features for Person Recognition . . . . . . . . . . . . . . . . . . . 228 5.2.3.1 Oriented Gaussian Derivatives . . . . . . . . . . . . . . . . . . . . . 228 5.2.3.2 Homogeneous Texture Descriptor . . . . . . . . . . . . . . . . . . 229 5.2.4 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231 5.2.4.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231 5.2.4.2 Feature Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231 5.3 Camera-based People Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232 5.3.1 Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234 5.3.1.1 Foreground Segmentation by Background Subtraction . 234 5.3.1.2 Morphological Operations . . . . . . . . . . . . . . . . . . . . . . . . 236 5.3.2 Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 5.3.2.1 Tracking Following Detection . . . . . . . . . . . . . . . . . . . . . 238 5.3.2.2 Combined Tracking and Detection . . . . . . . . . . . . . . . . . . 243 5.3.3 Occlusion Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248 5.3.3.1 Occlusion Handling without Separation . . . . . . . . . . . . . 248
Table of Contents
5.3.3.2 Separation by Shape . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.3.3 Separation by Color . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.3.4 Separation by 3D Information . . . . . . . . . . . . . . . . . . . . . 5.3.4 Localization and Tracking on the Ground Plane . . . . . . . . . . . . . 5.3.5 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
XIII
248 249 250 250 253 259
6 Interacting in Virtual Reality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263 6.1 Visual Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265 6.1.1 Spatial Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265 6.1.2 Immersive Visual Displays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266 6.1.3 Viewer Centered Projection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268 6.2 Acoustic Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272 6.2.1 Spatial Hearing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272 6.2.2 Auditory Displays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273 6.2.3 Wave Field Synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274 6.2.4 Binaural Synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275 6.2.5 Cross Talk Cancellation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276 6.3 Haptic Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279 6.3.1 Rendering a Wall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281 6.3.2 Rendering Solid Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 6.4 Modeling the Behavior of Virtual Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 6.4.1 Particle Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 6.4.2 Rigid Body Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288 6.5 Interacting with Rigid Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292 6.5.1 Guiding Sleeves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294 6.5.2 Sensitive Polygons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296 6.5.3 Virtual Magnetism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296 6.5.4 Snap-In . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297 6.6 Implementing Virtual Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298 6.6.1 Scene Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298 6.6.2 Toolkits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301 6.6.3 Cluster Rendering in Virtual Environments . . . . . . . . . . . . . . . . . 303 6.7 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305 6.7.1 A Simple Scene Graph Example . . . . . . . . . . . . . . . . . . . . . . . . . . 307 6.7.2 More Complex Examples: SolarSystem and Robot . . . . . . 308 6.7.3 An Interactive Environment, Tetris3D . . . . . . . . . . . . . . . . . . . . . . 310 6.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310 6.9 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
XIV
Table of Contents
7 Interactive and Cooperative Robot Assistants . . . . . . . . . . . . . . . . . . . . . 315 7.1 Mobile and Humanoid Robots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316 7.2 Interaction with Robot Assistants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321 7.2.1 Classification of Human-Robot Interaction Activities . . . . . . . . . 323 7.2.2 Performative Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323 7.2.3 Commanding and Commenting Actions . . . . . . . . . . . . . . . . . . . . 325 7.3 Learning and Teaching Robot Assistants . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325 7.3.1 Classification of Learning through Demonstration . . . . . . . . . . . 326 7.3.2 Skill Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326 7.3.3 Task Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327 7.3.4 Interactive Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328 7.3.5 Programming Robots via Observation: An Overview . . . . . . . . . 329 7.4 Interactive Task Learning from Human Demonstration . . . . . . . . . . . . . . . . 330 7.4.1 Classification of Task Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . 331 7.4.2 The PbD Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333 7.4.3 System Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335 7.4.4 Sensors for Learning through Demonstration . . . . . . . . . . . . . . . . 336 7.5 Models and Representation of Manipulation Tasks . . . . . . . . . . . . . . . . . . . 338 7.5.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338 7.5.2 Task Classes of Manipulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338 7.5.3 Hierarchical Representation of Tasks . . . . . . . . . . . . . . . . . . . . . . 340 7.6 Subgoal Extraction from Human Demonstration . . . . . . . . . . . . . . . . . . . . . 342 7.6.1 Signal Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343 7.6.2 Segmentation Step . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343 7.6.3 Grasp Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344 7.6.3.1 Static Grasp Classification . . . . . . . . . . . . . . . . . . . . . . . . 344 7.6.3.2 Dynamic Grasp Classification . . . . . . . . . . . . . . . . . . . . . 346 7.7 Task Mapping and Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 348 7.7.1 Event-Driven Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 348 7.7.2 Mapping Tasks onto Robot Manipulation Systems . . . . . . . . . . . 350 7.7.3 Human Comments and Advice . . . . . . . . . . . . . . . . . . . . . . . . . . . 354 7.8 Examples of Interactive Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354 7.9 Telepresence and Telerobotics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356 7.9.1 The Concept of Telepresence and Telerobotic Systems . . . . . . . . 356 7.9.2 Components of a Telerobotic System . . . . . . . . . . . . . . . . . . . . . . 359 7.9.3 Telemanipulation for Robot Assistants in Human-Centered Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 360 7.9.4 Exemplary Teleoperation Applications . . . . . . . . . . . . . . . . . . . . . 361 7.9.4.1 Using Teleoperated Robots as Remote Sensor Systems 361 7.9.4.2 Controlling and Instructing Robot Assistants via Teleoperation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364
Table of Contents
XV
8 Assisted Man-Machine Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369 8.1 The Concept of User Assistance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 370 8.2 Assistance in Manual Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372 8.2.1 Needs for Assistance in Manual Control . . . . . . . . . . . . . . . . . . . . 373 8.2.2 Context of Use in Manual Control . . . . . . . . . . . . . . . . . . . . . . . . . 374 8.2.3 Support of Manual Control Tasks . . . . . . . . . . . . . . . . . . . . . . . . . 379 8.3 Assistance in Man-Machine Dialogs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386 8.3.1 Needs for Assistance in Dialogs . . . . . . . . . . . . . . . . . . . . . . . . . . 386 8.3.2 Context of Dialogs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387 8.3.3 Support of Dialog Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 390 8.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395 A LTI-L IB — A C++ Open Source Computer Vision Library . . . . . . . . . . 399 A.1 Installation and Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 400 A.2 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402 A.2.1 Duplication and Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402 A.2.2 Serialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403 A.2.3 Encapsulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403 A.2.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403 A.3 Data Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404 A.3.1 Points and Pixels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405 A.3.2 Vectors and Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 406 A.3.3 Image Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 408 A.4 Functional Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409 A.4.1 Functor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410 A.4.2 Classifier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411 A.5 Handling Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412 A.5.1 Convolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412 A.5.2 Image IO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413 A.5.3 Color Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414 A.5.4 Drawing and Viewing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 416 A.6 Where to Go Next . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 418 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 421 B I MPRESARIO — A GUI for Rapid Prototyping of Image Processing Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423 B.1 Requirements, Installation, and Deinstallation . . . . . . . . . . . . . . . . . . . . . . . 424 B.2 Basic Components and Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424 B.2.1 Document and Processing Graph . . . . . . . . . . . . . . . . . . . . . . . . . . 426 B.2.2 Macros . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 426 B.2.3 Processing Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 429 B.2.4 Viewer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 430 B.2.5 Summary and further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . 431 B.3 Adding Functionality to a Document . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 432
XVI
Table of Contents
B.3.1 Using the Macro Browser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.3.2 Rearranging and Deleting Macros . . . . . . . . . . . . . . . . . . . . . . . . . B.4 Defining the Data Flow with Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.4.1 Creating a Link . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.4.2 Removing a Link . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.5 Configuring Macros . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.5.1 Image Sequence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.5.2 Video Capture Device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.5.3 Video Stream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.5.4 Standard Macro Property Window . . . . . . . . . . . . . . . . . . . . . . . . B.6 Advanced Features and Internals of I MPRESARIO . . . . . . . . . . . . . . . . . . . . B.6.1 Customizing the Graphical User Interface . . . . . . . . . . . . . . . . . . B.6.2 Macros and Dynamic Link Libraries . . . . . . . . . . . . . . . . . . . . . . . B.6.3 Settings Dialog and Directories . . . . . . . . . . . . . . . . . . . . . . . . . . . B.7 I MPRESARIO Examples in this Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.8 Extending I MPRESARIO with New Macros . . . . . . . . . . . . . . . . . . . . . . . . . . B.8.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.8.2 API Installation and further Reading . . . . . . . . . . . . . . . . . . . . . . . B.8.3 I MPRESARIO Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
432 434 434 434 436 436 437 440 442 444 446 446 448 449 450 450 450 451 451 451
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453
Contributors
Dipl.-Ing. Markus Ablaßmeier Institute for Human-Machine Communication Technische Universit¨at M¨unchen, Munich, Germany Email:
[email protected] Dr.-Ing. Jos´e Pablo Alvarado Moya Escuela de Electr´onica Instituto Tecnol´ogico de Costa Rica Email:
[email protected] Dipl.-Inform. Ingo Assenmacher Center for Computing and Communication, RWTH Aachen University, Aachen, Germany Email:
[email protected] Dr. rer. nat. Britta Bauer Gothaer Versicherungen, K¨oln, Germany Email:
[email protected] Dr.-Ing. Ulrich Canzler CanControls, Aachen Email:
[email protected] Prof. Dr.-Ing. R¨udiger Dillmann Institut f¨ur Technische Informatik (ITEC) Karlsruhe University, Germany Email:
[email protected] Dipl.- Ing. Peter D¨orfler Chair of Technical Computer Science, RWTH Aachen University, Aachen, Germany Email: ammi@pdoerfler.com Dipl.- Ing. Holger Fillbrandt Chair of Technical Computer Science, RWTH Aachen University, Aachen, Germany Email: fi
[email protected] Dipl.- Ing. Michael H¨ahnel Chair of Technical Computer Science, RWTH Aachen University, Aachen, Germany Email:
[email protected]
XVIII Dipl.- Ing. Lenka Jer´abkov´a Center for Computing and Communication, RWTH Aachen University, Aachen, Germany Email:
[email protected] Prof. Dr.-Ing. Karl-Friedrich Kraiss Chair of Technical Computer Science, RWTH Aachen University, Aachen, Germany Email:
[email protected] Dr. rer. nat. Torsten Kuhlen Center for Computing and Communication, RWTH Aachen University, Aachen, Germany Email:
[email protected] Dipl.- Ing. Lars Libuda Chair of Technical Computer Science, RWTH Aachen University, Aachen, Germany Email:
[email protected] Dipl.-Ing. Ronald M¨uller Institute for Human-Machine Communication Technische Universit¨at M¨unchen, Munich, Germany Email:
[email protected] Dipl.-Ing. Tony Poitschke Institute for Human-Machine Communication Technische Universit¨at M¨unchen, Munich, Germany Email:
[email protected] Dipl.-Ing. Stefan Reifinger Institute for Human-Machine Communication Technische Universit¨at M¨unchen, Munich, Germany Email: reifi
[email protected] Prof. Dr.-Ing. Gerhard Rigoll Institute for Human-Machine Communication Technische Universit¨at M¨unchen, Munich, Germany Email:
[email protected] Dipl.-Ing. Bj¨orn Schuller Institute for Human-Machine Communication Technische Universit¨at M¨unchen, Munich, Germany Email:
[email protected]
Contributors
Contributors Dipl.- Ing. J¨org Zieren Chair of Technical Computer Science, RWTH Aachen University, Aachen, Germany Email:
[email protected] Dipl.-Inform. Raoul D. Z¨ollner Institut f¨ur Technische Informatik (ITEC) Karlsruhe University, Germany Email:
[email protected]
XIX
Chapter 1
Introduction Karl-Friedrich Kraiss Living in modern societies relies on smart systems and services that support our collaborative and communicative needs as social beings. Developments in software and hardware technologies as e.g. in microelectronics, mechatronics, speech technology, computer linguistics, computer vision, and artificial intelligence are continuously driving new applications for work, leisure, and mobility. Man-machine interaction, which is ubiquitous in human life, is driven by technology as well. It is the gateway providing access to functions and services which, due to increasing complexity, threatens to turn into a bottleneck. Since for the user the interface is the product, interface design is a key technology which accordingly is subject to significant national and international research efforts. Interfaces to smart systems exploit the same advanced technologies as the said systems themselves. This textbook therefore introduces to researchers, students and practitioners not only advanced concepts of interface design, but also gives insight into the related interface technologies. This introduction first outlines the broad spectrum of applications in which manmachine interaction takes place. Then, following an overview of interface design goals, major approaches to interaction augmentation are identified. Finally the single chapters of this textbook are shortly characterized.
Application Areas of Man-Machine Interaction According to their properties man-machine systems may be classified in dialog systems and dynamic systems (Fig. 1.1). The former comprise small single user appliances like mobile phones, personal digital assistants and personal computers, but also multi-user systems like plant control stations or call centers. All of these require discrete user actions by, e.g., keyboard or mouse in either stationary or mobile mode. Representatives of the latter are land, air, or sea vehicles, robots, and masterslave systems. Following the needs of an aging society also rehabilitation robots, which provide movement assistance and restoration for people with disabilities, increase steadily in importance. All these systems put the operator in a control loop and require continuous manual control inputs. As dynamic systems are becoming more complex, functions tend to be automated in parts, which takes the operator out of the control loop and assigns supervisory control tasks to him or suggests cooperation with machines by task sharing. This approach has a long tradition in civil and military aviation and industrial plants. Currently smart automation embedded in the real world makes rapid progress in road traffic, where a multitude of driver assistance functions are under development.
2
Introduction
Many of these require interaction with the driver. In the near future also mobile service robots are expected on the marketplace as companions or home helpers. While these will have built-in intelligence for largely autonomous operation, they nevertheless will need human supervision and occasional intervention. The characterization proposed in (Fig. 1.1) very seldom occurs in pure form. Many systems are blended in one way or other, requiring discrete dialog actions, manual control, and supervisory control at the same time. A prototypical example is a commercial airliner, where the cockpit crew performs all these action in sequence or, at times, in parallel. Man machine systems
Dynamic systems
Dialog systems
Stationary Electronic appliances in home and office, Industrial plants Control stations
Manual control
Mobile Mobile phones Laptops PDAs Blackberries
Land, air, sea vehicles, Master-slave systems rehabilitation robotics
Supervisory control Automated land, air, sea vehicles, Servicerobots Industrial plants
Fig. 1.1. Classification of man-machine systems and associated application areas.
Design Goals for Man-Machine Systems For an interaction between man and machine to run smoothly, the interface between both must be designed appropriately. Unfortunately, much different from technical transmission channels, no crisp evaluation measures are at hand to characterize quality of service. Looking from the user’s perspective, the general goals to be achieved are usability, safety of use, and freedom of error. Usability describes to which extend a product can be used effectively, efficiently, and with satisfaction as defined by DIN EN ISO 9241. In this definition effectivity describes the accuracy and completeness with which a user can reach his goals. Efficiency means the effort needed to work effectively, while satisfaction is a measure of user acceptance. More recently it has been realized that user satisfaction also correlates with fun of use. The design of hedonistic interfaces that grant joy of use is already apparent in game boxes and music players, but also attracts the automotive industry, which advertises products with pertinent slogans that promise, e.g. ”driving pleasure”. The concept of safety of use is complementary to usability. It describes to which extend a product is understandable, predictable, controllable, and robust. Understandability implies that the workings of system functions are understood by a user.
3 Predictability describes to which extend a user is aware of system limits. Controllability indicates if system function can be switched off or overruled at the user’s discretion. Robustness, finally, describes how a system reacts to unpredictable performance of certain risk groups, due to complacency, habituation processes, reflexes in case of malfunctioning, or from testing the limits (e.g. when driving too fast). Freedom of error is a third objective of interface design, since a product free of errors complies with product liability requirements so that no warranty related user claims must be apprehended. Freedom of errors requires that an interface satisfies a user’s security expectations, which may be based on manuals, instructions, warnings or advertising. Bugs in this sense are construction errors due to inadequate consideration of all possible modes of use and abuse. Other sources of malfunction are instruction errors resulting from incomplete or baffling manuals or mistakable marketing information. It is interesting to note here, that a product is legally considered defective, if it fails to meet the expectations of the least informed user. Since such a requirement is extremely difficult to meet, this legal situation prevents the introduction of more advanced functions with possibly dangerous consequences. The next section will address the question, how advanced interaction techniques can contribute to match the stated design goals. Advanced Interaction Concepts Starting point for the discussion of interface augmentation is the basic interaction scheme represented by the white blocks in (Fig. 1.2), where the term machine stands for appliance, vehicle, robot, simulation or process. Man and machine are interlinked via two unidirectional channels, one of which transmits machine generated information to the user, while the other one feeds user inputs in opposite direction. Interaction takes place with a real machine in the real world.
Context of use
User assistance Information display augmentation
User input augmentation
Automation
Fission Speech Gesture Mimics ...
Man
Fusion Speech Gesture Mimics ...
Real machine in real world
Operation in Virtual reality
Multimodality
Fig. 1.2. Interaction augmented by multimodality, user assistance, and virtual reality.
4
Introduction
There are three major approaches, which promise to significantly augment interface quality. These are multimodal interaction, operation in virtual reality, and user assistance, which have been integrated into the basic MMI scheme of (Fig. 1.2) as grey rectangles. Multimodality is ubiquitous in human communication. We gesture and mimic while talking, even at the phone, when the addressee can not see it. We nod or shake the head or change head pose to indicate agreement or disagreement. We also signal attentiveness by suitable body language, e.g. by turning towards a dialog partner. In so doing conversation becomes comfortable, intuitive, and robust. It is exactly this lack of multimodality why current interfaces often fail. It therefore is a goal of advanced interface design, to make use of various modalities. Modalities applicable to interfaces are mimics, gesture, body posture, movement, speech, and haptics. They serve either information display, or user input, sometimes even both purposes. The combination of modalities for input is sometimes called multimodal fusion while the combination of different modalities for information display is labeled multimodal fission. For fission modalities must be generated e.g. by synthesizing speech or by providing gesturing avatars. This book focuses on fusion, which requires automatic recognition of multimodal human actions. Speech recognition has been around for almost fifty years and is available for practical use. Further improvements of speech recognizers with respect to context sensitive continuous speech recognition and robustness against noise are under way. Observation of lip movements during talking contributes to speech recognition under noise conditions. Advanced speech recognizers will also cope with incomplete grammar and spontaneous uttering. Interest in gesture and mimic recognition is more recent. In fact the first related papers appeared only in the nineties. First efforts to record mimics and gesture in real time in laboratory setups and in movie studios involved intrusive methods with calibrated markers and multiple cameras, which are of no use for the applications at hand. Only recently video based recognition has achieved an acceptable performance level in out-of-the-laboratory settings. Gesture, mimics, head pose, line of sight and body posture can now be recognized based on video recordings in real time, even under adverse real world conditions. People may even be optically tracked or located in public places. Emotions derived from a fusion of speech, gestures and mimics now open the door for yet scarcely exploited emotional interaction. The second column on which interface augmentation rests is virtual reality, which provides comfortable means of interactive 3D-simulation, hereby enabling various novel approaches for interaction. Virtual reality allows, e.g., operating virtual machines in virtual environments such as flight simulators. It may also be used to generate virtual interfaces. By providing suitable multimodal information channels, e.g., realistic teleprecence is achieved that permits teleoperating in worlds which are physically not accessible because of distance or scale (e.g. space applications or micro- and nano-worlds). In master-slave systems the operator usually acts on a local real machine, and the actions are repeated by a remote machine. With virtual reality the local machine can be substituted by simulation. Virtual reality also has a major impact for robot explo-
5 ration, testing, and training, since robot actions can be taught-in and tested within a local system simulation before being transferred and executed in a remote and possibly dangerous and costly setting. Interaction may be further improved by making use of augmented or mixed reality, where information is presented non-intrusively on head mounted displays as an overlay to virtual or real worlds. Apart from multimodality and virtual reality, the concept of user assistance is a third ingredient of interpersonal communication lacking so far in man-machine interaction. Envision, e.g., how business executives often call upon a staff of personal assistants to get their work done. Based on familiarity with the context of the given task, i.e. the purpose and constraints of a business transaction, a qualified assistant is expected to provide correct information just in time, hereby reducing the workload of the executive and improving the quality of the decisions made. In man-machine interaction context of task transfers to context of use, which may be characterized by the state of the user, the state of the system, and the situation. In conventional man-machine interaction little of this kind is yet known. Neither user characteristics nor the circumstances of operation are made explicit for support of interaction; the knowledge about context resides with the user alone. However, if context of use was made explicit, assistance could be provided to the user, similar to that offered to the executive by his personal assistant. Hence knowledge of context of use is the key to exploiting interface adaptivity. As indicated by vertical arrows in (Fig. 1.2) assistance can augment interaction in three ways. First information display may be improved, e.g., by providing information just in time and coded in the most suitable modality. Secondly user inputs may be augmented, modified, or even suppressed as appropriate. A third method for providing assistance exists for dynamic systems, where system functions may be automated either permanently or temporarily. An appliance assisted in this way is supposed to appear to the user simpler than it actually is. It also will be easier to learn, more pleasant to handle, and less errorprone. The Scope of This Book Most publications in the field of human factors engineering focus on interface design, while leaving the technical implementation to engineers. While this book addresses design aspects as well, it puts in addition special emphasis on advanced interface implementation. A significant part of the book is concerned with the realization of multimodal interaction via gesture, mimics, and speech. The remaining parts deal with the identification and tracking of people, the interaction in virtual reality and with robots, and with the implementation of user assistance. The second chapter treats picture processing algorithms for the non-intrusive acquisition of human action. In the first section techniques for identifying and classifying hand gesture commands from video are described. These techniques are extended to facial expression commands in the second section. Sign language recognition, the topic of chapter 3, is the ultimate challenge in multimodal interaction. It builds on the basic techniques presented in the chapter 2, but considers specialized requirements for the recognition of isolated and continuous
6
Introduction
signs using manual as well as facial expression features. A method for the automatic transcription of signs into subunits is presented, which, comparable to phonemes in speech recognition, opens the door to large vocabularies. The role of speech communication in multimodal interfaces is the subject of chapter 4, which in its first section presents basics of advanced speech recognition. Speech dialogs are treated next, followed by a comprehensive description of multimodal dialogs. A final section is concerned with the extraction of emotions from speech and gesture. The 5th chapter deals with video-based person recognition and tracking. A first section is concerned with face recognition using global and local features. Then fullbody person recognition is dealt with based on the color and texture of the clothing a subject wears. Finally various approaches for camera-based people tracking are presented. Chapter 6 is devoted to problems arising when interacting with virtual objects in virtual reality. First technological aspects of implementing visual, acoustic, and haptic virtual worlds are presented. Then procedures for virtual object modeling are outlined. The third section is devoted to various methods to augment interacting with virtual objects. Finally the chapter closes with a description of software tools applicable for the implementation of virtual environments. The role of service robots in the future society is the subject of chapter 7. First a description of available mobile and humanoid robots is given. Then actual challenges arising from interacting with robot assistants are systematically tackled like, e.g., learning, teaching, programming by demonstration, commanding procedures, task planning, and exception handling. As far as manipulation tasks are considered, relevant models and representations are presented. Finally telepresence and telerobotics are treated as special cases of man-robot interaction. The 8th chapter addresses the concept of assistance in man-machine interaction, which is systematically elaborated for manual control and dialog tasks. In both cases various methods of providing assistance are identified based on user capabilities and deficiencies. Since context of use is identified as the key element in this undertaking, a variety of algorithms are introduced and discussed for real-time non-intrusive context identification. For readers with a background in engineering and computer science chapters 2 and 5 presents a collection of programming examples, which are implemented with ’Impresario’. This rapid prototyping tool offers an application programming interface for comfortable access to the LTI-Lib, a large public domain software library for computer vision and classification. A description of the LTI-Lib can be found in Appendix A, a manual of Impresario in Appendix B. For the examples a selection of head, face, eye, and lip templates, hand gestures, sign language sequences, and people tracking clips are provided. The reader is encouraged to use these materials to start experimentation on his own. Chapter 6 finally, provides the Java3D code of some simple VR application. All software, programming code, and templates come with the book CD.
Chapter 2
Non-Intrusive Acquisition of Human Action J¨org Zieren1 , Ulrich Canzler2 This chapter discusses control by gestures and facial expression using the visual modality. The basic concepts of vision-based pattern recognition are introduced together with several fundamental image processing algorithms. The text puts low-level methods in context with actual applications and enables the reader to create a fully functional implementation. Rather than provide a complete survey of existing techniques, the focus lies on the presentation of selected procedures supported by concrete examples.
2.1 Hand Gesture Commands The use of gestures as a means to convey information is an important part of human communication. Since hand gestures are a convenient and natural way of interaction, man-machine interfaces can benefit from their use as an input modality. Gesture control allows silent, remote, and contactless interaction without the need to locate knobs or dials. Example applications include publicly accessible devices such as information terminals [1], where problems of hygiene and strong wear of keys are solved, or automotive environments [2], where visual and mental distraction are reduced and mechanical input devices can be saved.
Fig. 2.1. The processing chain found in many pattern recognition systems.
Many pattern recognition systems have a common processing chain that can roughly be divided into three consecutive stages, each of which forwards its output to the next (Fig. 2.1). First, a single frame is acquired from a camera, a file system, a network stream, etc. The feature extraction stage then computes a fixed number of scalar values that each describe an individual feature of the observed scene, such as the coordinates of a target object (e.g. the hand), its size, or its color. The image is 1 2
Sect. 2.1 Sect. 2.2
8
Non-Intrusive Acquisition of Human Action
thereby reduced to a numerical representation called a feature vector. This step is often the most complex and computationally expensive in the processing chain. The accuracy and reliability of the feature extraction process can have significant influence on the system’s overall performance. Finally, the extracted features of a single frame (static gesture) or a sequence of frames (dynamic gesture) are classified either by a set of rules or on the basis of a training phase that occurs prior to the actual application phase. During the training the correct classification of the current features is always known and the features are stored or learned in a way that later allows to classify features computed in the application phase, even in the presence of noise and minor variations that inevitably occur in the real world. Moving along the processing chain, the data volume being processed continuously decreases (from an image of several hundred kB to a feature vector of several bytes to a single event description) while the level of abstraction increases accordingly (from a set of pixels to a set of features to a semantic interpretation of the observed scene). The following sections 2.1.1, 2.1.2, and 2.1.3 explain the three processing stages in detail. En route, we will create example applications that recognize various static and dynamic hand gestures. These examples show the practical application of the processing concept and algorithms presented in this chapter. Since the C++ source code is provided on the accompanying CD, they can also serve as a basis for own implementations. 2.1.1 Image Acquisition and Input Data Before looking into algorithms, it is important to examine those aspects of the planned gesture recognition system that directly affect the interaction with the user. Three important properties can be defined that greatly influence the subsequent design process: •
•
•
The vocabulary, i.e. the set of gestures that are to be recognized. The vocabulary size is usually identical with the number of possible system responses (unless multiple gestures trigger the same response). The recording conditions under which the system operates. To simplify system design, restrictions are commonly imposed on image background, lighting, the target’s minimum and maximum distance to the camera, etc. The system’s performance in terms of latency (the processing time between the end of the gesture and the availability of the recognition result) and recognition accuracy (the percentage or probability of correct recognition results).
The first two items are explained in detail below. The third involves performance measurement and code optimization for a concrete implementation and application scenario, which is outside the scope of this chapter. 2.1.1.1 Vocabulary Constellation and characteristics of the vocabulary have a strong influence on all processing stages. If the vocabulary is known before the development of the recog-
Hand Gesture Commands
9
nition system starts, both design and implementation are usually optimized accordingly. This may later lead to problems if the vocabulary is to be changed or extended. In case the system is completely operational before the constellation of the vocabulary is decided upon, recognition accuracy can be optimized by simply avoiding gestures that are frequently misclassified. In practice, a mixture of both approaches is often applicable. The vocabulary affects the difficulty of the recognition task in two ways. First, with growing vocabulary size the number of possible misclassifications rises, rendering recognition increasingly difficult. Second, the degree of similarity between the vocabulary’s elements needs to be considered. This cannot be quantified, and even a qualitative estimate based on human perception cannot necessarily be transferred to an automatic system. Thus, for evaluating a system’s performance, vocabulary size and recognition accuracy alone are insufficient. Though frequently omitted, the exact constellation of the vocabulary (or at least the degree of similarity between its elements) needs to be known as well. 2.1.1.2 Recording Conditions The conditions under which the input images are recorded are crucial for the design of any recognition system. It is therefore important to specify them as precisely and completely as possible before starting the actual implementation. However, unless known algorithms and hardware are to be used, it is not always foreseeable how the subsequent processing stages will react to certain properties of the input data. For example, a small change in lighting direction may cause an object’s appearance to change in a way that greatly affects a certain feature (even though this change may not be immediately apparent to the human eye), resulting in poor recognition accuracy. In this case, either the input data specification has to be revised not to allow changes in lighting, or more robust algorithms have to be deployed for feature extraction and/or classification that are not affected by such variations. Problems like this can be avoided by keeping any kind of variance or noise in the input images to a minimum. This leads to what is commonly known as “laboratory recording conditions”, as exemplified by Tab. 2.1. While these conditions can usually be met in the training phase (if the classification stage requires training at all), they render many systems useless for any kind of practical application. On the other hand, they greatly simplify system design and might be the only environment in which certain demanding recognition tasks are viable. In contrast to the above enumeration of laboratory conditions, one or more of the “real world recording conditions” shown in Tab. 2.2 often apply when a system is to be put to practical use. Designing a system that is not significantly affected by any of the above conditions is, at the current state of research, impossible and should thus not be aimed for. General statements regarding the feasibility of certain tasks cannot be made, but from experience one will be able to judge what can be achieved in a specific application scenario.
10
Non-Intrusive Acquisition of Human Action Table 2.1. List of common “laboratory recording conditions”. Domain Condition(s) Image Content The image contains only the target object, usually in front of a unicolored untextured background. No other objects are visible. The user wears long-sleeved, non-skin colored clothing, so that the hand can be separated from the arm by color. Lighting Strong diffuse lighting provides an even illumination of the target with no shadows or reflections. Setup The placement of camera, background, and target remains constant. The target’s distance to the camera and its position in the image do not change. (If the target is moving, this applies only to the starting point of the motion.) Camera The camera hardware and camera parameters (resolution, gain, white balance, brightness etc.) are never changed. They are also optimally adjusted to prevent overexposure or color cast. In case of a moving target, shutter speed and frame rate are sufficiently high to prevent both motion blur and discontinuities. A professional quality camera with a high resolution is used.
Table 2.2. List of common “real world recording conditions”. Domain Condition(s) Image Content The image may contain other objects (distractors) besides, or even in place of, the target object. The background is unknown and may even be moving (e.g. trees, clouds). A target may become partially or completely occluded by distractors or other targets. The user may be wearing short-sleeved or skin-colored clothing that prevents color-based separation of arm and hand. Lighting Lighting may not be diffuse, resulting in shadows and uneven illumination within the image region. It may even vary over time. Setup The position of the target with respect to the camera is not fixed. The target may be located anywhere in the image, and its size may vary with its distance to the camera. Camera Camera hardware and/or parameters may change from take to take, or even during a single take in case of automatic dynamic adaptation. Overexposure and color cast may occur. In case of a moving target, low shutter speeds may cause motion blur, and a low frame rate (or high target velocity) may result in discontinuities between successive frames. A consumer quality camera is used.
2.1.1.3 Image Representation The description and implementation of image processing algorithms requires a suitable mathematical representation of various types of images. A common representation of a rectangular image with a resolution of M rows and N columns is a discrete function
Hand Gesture Commands
11
I(x, y)
with
x ∈ {0, 1, . . . , N − 1} y ∈ {0, 1, . . . , M − 1}
(2.1)
The 2-tuple (x, y) denotes pixel coordinates with the origin (0, 0) in the upperleft corner. The value of I(x, y) describes a property of the corresponding pixel. This property may be the pixel’s color, its brightness, the probability of it representing human skin, etc. Color is described as an n-tuple of scalar values, depending on the chosen color model. The most common color model in computer graphics is RGB, which uses a 3-tuple (r, g, b) specifying a color’s red, green, and blue components: ⎛ ⎞ r(x, y) I(x, y) = ⎝ g(x, y) ⎠ (2.2) b(x, y) Other frequently used color models are HSI (hue, saturation, intensity) or YCbCr (luminance and chrominance). For color images, I(x, y) is thus a vector function. For many other properties, such as brightness and probabilities, I(x, y) is a scalar function and can be visualized as a gray value image (also called intensity image). Scalar integers can be used to encode a pixel’s classification. For gesture recognition an important classification is the discrimination between foreground (target) and background. This yields a binary valued image, commonly called a mask, Imask (x, y) ∈ {0, 1}
(2.3)
Binary values are usually visualized as black and white. An image sequence (i.e. a video clip) of T frames can be described using a time index t as shown in (2.4). The amount of real time elapsed between two successive frames is the inverse of the frame rate used when recording the images. I(x, y, t),
t = 1, 2, . . . , T
(2.4)
The C++ computer vision library LTI-L IB (described in Annex A) uses the types channel and channel8 for gray value images, image for RGB color images, and rgbPixel for colors. 2.1.1.4 Example We will now specify the input data to be processed by the two example recognition applications that will be developed in the course of this chapter, as well as recording conditions for static and dynamic gestures. Static Gestures The system shall recognize three different gestures – “left”, “right”, and “stop” – as shown in Fig. 2.2 (a-c). In addition, the event that the hand is not visible in the image or rests in an idle position (such as Fig. 2.2d) and does not perform any of these gestures shall be detected. Since this vocabulary is of course very small and its elements are sufficiently dissimilar (this is proven in Sect. 2.1.3.7), high recognition accuracy can be expected.
12
Non-Intrusive Acquisition of Human Action
(a) “left”
(b) “right”
(c) “stop”
(d) none (idle)
Fig. 2.2. Static hand gestures for “left”, “right”, and “stop” (a-c), as well as example for an idle position (d).
Dynamic Gestures The dynamic vocabulary will be comprised of six gestures, where three are simply backwards executions of the other three. Tab. 2.3 describes these gestures, and Fig. 2.3 shows several sample executions. Since texture will not be considered, it does not matter whether the palm or the back of the hand faces the camera. Therefore, the camera may be mounted atop a desk facing downwards (e.g. on a tripod or affixed to the computer screen) to avoid background clutter. Table 2.3. Description of the six dynamic gestures to be recognized. Gesture “clockwise”
Description circular clockwise motion of closed hand, starting and ending at the bottom of an imaginary circle “counterclockwise” backwards execution of “clockwise” “open” opening motion of flat hand by rotation around the arm’s axis by 90°(fingers extended and touching) “close” backwards execution of “open” “grab” starting with all fingers extended and not touching, a fist is made “drop” backwards execution of “grab”
Recording Conditions As will be described in the following sections, skin color serves as the primary image cue for both example systems. Assuming an office or other indoor environment, Tab. 2.4 lists the recording conditions to be complied with for both static and dynamic gestures. The enclosed CD contains several accordant examples that can also be used as input data in case a webcam is not available. 2.1.2 Feature Extraction The transition from low-level image data to some higher-level description thereof, represented as a vector of scalar values, is called feature extraction. In this process,
Hand Gesture Commands
13
(a) “clockwise” (left to right) and “counterclockwise” (right to left)
(b) “open” (left to right) and “close” (right to left)
(c) “grab” (left to right) and “drop” (right to left) Fig. 2.3. The six dynamic gestures to be recognized. In each of the three image sequences, the second gesture is a backwards execution of the first, i.e. corresponds to reading the image sequence from right to left.
irrelevant information (background) is discarded, while relevant information (foreground or target) is isolated. Most pattern recognition systems perform this step, because processing the complete image is computationally too demanding and introduces an unacceptably high amount of noise (images often contain significantly more background pixels than foreground pixels). Using two-dimensional images of the unmarked hand, it is at present not possible to create a three-dimensional model of a deformable object as complex and flexible as the hand in real-time. Since the hand has 27 degrees of freedom (DOF), this would require an amount of information that cannot be extracted with sufficient accuracy and speed. Model-based features such as the bending of individual fingers are therefore not available in video-based marker-free real-time gesture recognition. Instead, appearance-based features that describe the two-dimensional view of the hand are exploited. Frequently used appearance-based shape features include geometric properties of the shape’s border, such as location, orientation, size etc. Texture features shall not be considered here because resolution and contrast that can be achieved with consumer cameras are usually insufficient for their reliable and accurate computation. This section presents algorithms and data structures to detect the user’s hand in the image, describe its shape, and calculate several geometric features that allow the distinction of different static hand configurations and different dynamic gestures,
14
Non-Intrusive Acquisition of Human Action Table 2.4. Recording conditions for the example recognition applications. Domain Condition(s) Image Content Ideally, the only skin colored object in the image is the user’s hand. Other skin colored objects may be visible, but they must be small compared to the hand. This also applies to the arm; it should be covered by a long-sleeved shirt if it is visible. In dynamic gestures the hand should have completely entered the image before recording starts, and not leave the image before recording ends, so that it is entirely visible in every recorded frame. Lighting Lighting is sufficiently diffuse so that no significant shadows are visible on the hand. Slight shadows, such as in Fig. 2.2 and Fig. 2.3, are acceptable. Setup The distance between hand and camera is chosen so that the hand fills approximately 10–25% of the image. The hand’s exact position in the image is arbitrary, but no parts of the hand should be cropped. The camera is not rotated, i.e. its x axis is horizontal. Camera Resolution may vary, but should be at least 320 × 240. Aspect ratio remains constant. The camera is adjusted so that no overexposure and only minor color cast occurs. (Optimal performance is usually achieved when gain, brightness and shutter speed are set so that the image appears somewhat underexposed to the human observer.) For dynamic gestures, a frame rate of 25 per second is to be used, and shutter speed should be high enough to prevent motion blur. A consumer quality camera is sufficient.
such as those shown in Fig. 2.2 and 2.3. This process, which occurs within the feature extraction module in Fig. 2.1, is visualized in Fig. 2.4.
Fig. 2.4. Visualization of the feature extraction stage. The top row indicates processing steps, the bottom row shows examples for corresponding data.
2.1.2.1 Hand Localization The identification of foreground or target regions constitutes an interpretation of the image based on knowledge which is usually specific to the application scenario. This
Hand Gesture Commands
15
knowledge can be encoded explicitly (as a set of rules) or implicitly (in a histogram, a neural network, etc.). Known properties of the target object, such as shape, size, or color, can be exploited. In gesture recognition, color is the most frequently used feature for hand localization since shape and size of the hand’s projection in the twodimensional image plane vary greatly. It is also the only feature explicitly stored in the image. Using the color attribute to localize an object in the image requires a definition of the object’s color or colors. In the RGB color model (and most others), even objects that one would call unicolored usually occupy a range of numerical values. This range can be described statistically using a three-dimensional discrete histogram hobject(r, g, b), with the dimensions corresponding to the red, green, and blue components. hobject is computed from a sufficiently large number of object pixels that are usually marked manually in a set of source images that cover all setups in which the system is intended to be used (e.g. multiple users, varying lighting conditions, etc.). Its value at (r, g, b) indicates the number of pixels with the corresponding color. The total sum of hobject over all colors is therefore equal to the number of considered object pixels nobject, i.e. hobject(r, g, b) = nobject (2.5) r
g
b
Because the encoded knowledge is central to the localization task, the creation of hobject is an important part of the system’s development. It is exemplified below for a single frame. Fig. 2.5 shows the source image of a hand (a) and a corresponding, manually generated binary mask (b) that indicates object pixels (white) and background pixels (black). In (c) the histogram computed from (a), considering only object pixels (those that are white in (b)), is shown. The three-dimensional view uses points to indicate colors that occurred with a certain minimum frequency. The three one-dimensional graphs to the right show the projection onto the red, green, and blue axis. Not surprisingly, red is the dominant skin color component.
(a) Source image
(b) Object mask
(c) Object color histogram
Fig. 2.5. From a source image (a) and a corresponding, manually generated object mask (b), an object color histogram (c) is computed.
16
Non-Intrusive Acquisition of Human Action
On the basis of hobject, color-based object detection can now be performed in newly acquired images of the object. The aim is to compute, from a pixel’s color, a probability or belief value indicating its likeliness of representing a part of the target object. This value is obtained for every pixel (x, y) and stored in a probability image Iobject(x, y) with the same dimensions as the analyzed image. The following paragraphs derive the required stochastic equations. Given an object pixel, the probability of it having a certain color (r, g, b) can be computed from hobject as P (r, g, b|object) =
hobject(r, g, b) nobject
(2.6)
By creating a complementary histogram hbg of the background colors from a total number of nbg background pixels the accordant probability for a background pixel is obtained in the same way: P (r, g, b|bg) =
hbg (r, g, b) nbg
(2.7)
Applying Bayes’ rule, the probability of any pixel representing a part of the object can be computed from its color (r, g, b) using (2.6) and (2.7):
P (object|r, g, b) =
P (r, g, b|object) · P (object) P (r, g, b|object) · P (object) + P (r, g, b|bg) · P (bg)
(2.8)
P (object) and P (bg) denote the a priori object and background probabilities, respectively, with P (object) + P (bg) = 1. Using (2.8), the object probability image Iobj,prob is created from I as Iobj,prob(x, y) = P (object|I(x, y))
(2.9)
To classify each pixel as either background or target, an object probability threshold Θ is defined. Probabilities equal to or above Θ are considered target, while all others constitute the background. A data structure suitable for representing this classification is a binary mask Iobj,mask (x, y) =
1 0
if Iobj,prob(x, y) ≥ Θ otherwise
(target) (background)
(2.10)
Iobj,mask constitutes a threshold segmentation of the source image because it partitions I into target and background regions. Rewriting (2.8) as (2.11), it can be seen that varying P (object) and P (bg) is equivalent to a non-linear scaling of P (object|r, g, b) and does not affect Iobj,mask when Θ is scaled accordingly (though it may improve contrast in the visualization of Iobj,prob ). In other words, for any prior probabilities P (object), P (bg) = 0, all possible segmentations can be obtained by varying Θ from 0 to 1. It is therefore
Hand Gesture Commands
17
practical to choose arbitrary values, e.g. P (object) = P (bg) = 0.5. Obviously, the value computed for P (object|r, g, b) will then no longer reflect an actual probability, but a belief value.3 This is usually easier to handle than the computation of exact values for P (object) and P (bg). P (object|r, g, b) = 1 +
P (bg) P (r, g, b|bg) · P (r, g, b|object) P (object)
−1 (2.11)
A suitable choice of Θ is crucial for an accurate discrimination between background and target. In case the recording conditions remain constant and are known in advance, Θ can be set manually, but for uncontrolled environments an automatic determination is desirable. Two different strategies are presented below. If no knowledge of the target object’s shape and/or position is available, the following iterative low-level algorithm presented in [24] produces good results in a variety of conditions. It assumes the histogram of Iobj,prob to be bi-modal but can also be applied if this is not the case. 1. Arbitrarily define a set of background pixels (some usually have a high a priori background probability, e.g. the four corners of the image). All other pixels are defined as foreground. This constitutes an initial classification. 2. Compute the mean values for background and foreground, μobject and μbg , based on the most recent classification. If the mean values are identical to those computed in the previous iteration, halt. 3. Compute a new threshold Θ = 12 (μobject + μbg ) and perform another classification of all pixels, then goto step 2. Fig. 2.6. Automatic computation of the object probability threshold Θ without the use of highlevel knowledge (presented in [24]).
In case the target’s approximate location and geometry are known, humans can usually identify a suitable threshold just by observation of the corresponding object mask. This approach can be implemented by defining an expected target shape, creating several object masks using different thresholds, and choosing the one which yields the most similar shape. This requires a feature extraction as described in Sect. 2.1.2 and a metric to quantify the deviation of a candidate shape’s features from those of the expected target shape. An example for hand localization can be seen in Fig. 2.7. Using the skin color histogram shown in Fig. 2.5 and a generic background histogram hbg , a skin probability image (b) was computed from a source image (a) of the same hand (and some clutter) as shown in Fig. 2.5, using P (object) = P (bg) = 0.5. For the purpose of visualization, three different thresholds Θ1 < Θ2 < Θ3 were then applied, resulting in three different binary masks (c, d, e). In this example none of the binary masks 3
For simplicity, the term “probability” will still be used in either case.
18
Non-Intrusive Acquisition of Human Action
is entirely correct: For Θ1 , numerous background regions (such as the bottle cap in the bottom right corner) are classified as foreground, while Θ3 leads to holes in the object, especially at its borders. Θ2 , which was computed automatically using the algorithm described in Fig. 2.6, is a compromise that might be considered optimal for many applications, such as the example systems to be developed in this section.
(a) Source Image
(c) Skin/Background classification using Θ1
(b) Skin Probability Image
(d) Skin/Background classification using Θ2
(e) Skin/Background classification using Θ3
Fig. 2.7. A source image (a) and the corresponding skin probability image (b). An object/background classification was performed for three different thresholds Θ1 < Θ2 < Θ3 (c-e).
While the human perception of an object’s color is largely independent of the current illumination (an effect called color constancy), colors recorded by a camera are strongly influenced by illumination and hardware characteristics. This restricts the use of histograms to the recording conditions under which their source data was created, or necessitates the application of color constancy algorithms [7, 9].4 In [12], a skin and a non-skin color histogram are presented that were created from several thousand images found on the WWW, covering a multitude of skin 4
In many digital cameras, color constancy can be achieved by performing a white balance: The camera is pointed to a white object (such as a blank sheet of paper), and a transformation is performed so that this object appears colorless (r = g = b) in the image as well. The transformation parameters are then stored for further exposures under the same illumination conditions.
Hand Gesture Commands
19
hues, lighting conditions, and camera hardware. Thereby, these histograms implicitly provide user, illumination, and camera independence, at the cost of a comparably high probability for non-skin objects being classified as skin (false alarm). Fig. 2.8 shows the skin probability image (a) and binary masks (b, c, d) for the source image shown in Fig. 2.7a, computed using the histograms from [12] and three different thresholds Θ4 < Θ5 < Θ6 . As described above, P (object) = P (bg) = 0.5 was chosen. Compared to 2.7b, the coins, the pens, the bottle cap, and the shadow around the text marker have significantly higher skin probability, with the pens even exceeding parts of the fingers and the thumb (d). This is a fundamental problem that cannot be solved by a simple threshold classification because there is no threshold Θ that would achieve a correct result. Unless the histograms can be modified to reduce the number of false alarms, the subsequent processing stages must therefore be designed to handle this problem.
(a) Skin Probability Image
(b) Skin/Background classification using Θ4
(c) Skin/Background classification using Θ5
(d) Skin/Background classification using Θ6
Fig. 2.8. Skin probability image (b) for Fig. 2.8a and object/background classification for three thresholds Θ4 < Θ5 < Θ6 (b, c, d).
Implementational Aspects In practical applications the colors r, g, and b commonly have a resolution of 8 bit each, i.e. r, g, b ∈ {0, 1, . . . , 255}
(2.12)
A corresponding histogram would have to have 2563 = 16, 777, 216 cells, each storing a floating point number of typically 4 bytes in size, totaling 64 MB. To reduce these memory requirements, the color values are downsampled to e.g. 5 bit through a simple integer division by 23 = 8:
g
r b g = b = (2.13) r = 8 8 8 This results in a histogram h (r , g , b ) with 323 = 32, 768 cells that requires only 128 kB of memory. The entailed loss in accuracy is negligible in most appli-
20
Non-Intrusive Acquisition of Human Action
cations, especially when consumer cameras with high noise levels are employed. Usually the downsampling is even advantageous because it constitutes a generalization and therefore reduces the amount of data required to create a representative histogram. The LTI-L IB class skinProbabilityMap comes with the histograms presented in [12]. The algorithm described in Fig. 2.6 is implemented in the class optimalThresholding. 2.1.2.2 Region Description On the basis of Iobj,mask the source image I can be partitioned into regions. A region R is a contiguous set of pixels p for which Iobj,mask has the same value. The concept of contiguity requires a definition of pixel adjacency. As depicted in Fig. 2.9, adjacency may be based on either a 4-neighborhood or an 8-neighborhood. In this section the 8-neighborhood will be used. In general, regions may contain other regions and/or holes, but this shall not be considered here because it is of minor importance in most gesture recognition applications.
Fig. 2.9. Adjacent pixels (gray) in the 4-neighborhood (a) and 8-neighborhood (b) of a reference pixel (black).
Under laboratory recording conditions, Iobj,mask will contain exactly one target region. However, in many real world applications, other skin colored objects may be visible as well, so that Iobj,mask typically contains multiple target regions (see e.g. Fig. 2.8). Unless advanced reasoning strategies (such as described in [23, 28]) are used, the feature extraction stage is responsible for identifying the region that represents the user’s hand, possibly among a multitude of candidates. This is done on the basis of the regions’ geometric features. The calculation of these features is performed by first creating an explicit description of each region contained in Iobj,mask. Various algorithms exist that generate, for every region, a list of all of its pixels. A more compact and computationally more efficient representation of a region can be obtained by storing only its border points (which are, in general, significantly fewer). A border point is conveniently defined as a pixel p ∈ R that has at least one pixel q ∈ / R within its 4-neighborhood. An example region (a) and its border points (b) are shown in Fig. 2.10. In a counterclockwise traversal of the object’s border, every border point has a predecessor and a successor within its 8-neighborhood. An efficient data structure for the representation of a region is a sorted list of its border points. This can be
Hand Gesture Commands
(a) Region pixels
21
(b) Border points
(c) Polygon
Fig. 2.10. Description of an image region by the set of all of its pixels (a), a list of its border pixels (b), and a closed polygon (c). Light gray pixels in (b) and (c) are not part of the respective representation, but are shown for comparison with (a).
interpreted as a closed polygon whose vertices are the centers of the border points (Fig. 2.10c). In the following, the object’s border is defined to be this polygon. This definition has the advantages of being sub-pixel accurate and facilitating efficient computation of various shape-based geometric features (as described in the next section).5 Finding the border points of a region is not as straightforward as identifying its pixels. Fig. 2.11 shows an algorithm that processes Iobj,mask and computes, for every region R, a list of its border points BR = {(x0 , y0 ), (x1 , y1 ), . . . , (xn−1 , yn−1 )}
(2.14)
Fig. 2.12 (a, b, c) shows the borders computed by this algorithm from the masks shown in Fig. 2.8 (b, c, d). In areas where the skin probability approaches the threshold Θ, the borders become jagged due to the random nature of the input data, as can be seen especially in (b) and (c). This effect causes an increase in border length that is random as well, which is undesirable because it reduces the information content of the computed border length value (similar shapes may have substantially different border lengths, rendering this feature less useful for recognition). Preceding the segmentation (2.10) by a convolution of Iobj,prob with a Gaussian kernel alleviates this problem by dampening high frequencies in the input data, providing smooth borders as in Fig. 2.12 (d, e, f). The LTI-L IB uses the classes borderPoints and polygonPoints (among others) to represent image regions. The class objectsFromMask contains an algorithm that is comparable to the one presented in Fig. 2.11, but can also detect holes within regions, and further regions within these holes. 5
It should be noted that the polygon border definition leads to the border pixels no longer being considered completely part of the object. Compared to other, non-sub-pixel accurate interpretations, this may lead to slight differences in object area that are noticeable only for very small objects. 6 m(x, y) = 1 indicates touching the border of a region that was already processed. 7 m(x, y) = 2 indicates crossing the border of a region that was already processed.
22
Non-Intrusive Acquisition of Human Action 1. Create a helper matrix m with the same dimensions as Iobj,mask and initialize all entries to 0. Define (x, y) as the current coordinates and initialize them to (0, 0). Define (x , y ) and (x , y ) as temporary coordinates. 2. Iterate from left to right through all image rows successively, starting at y = 0. If m(x, y) = 0 ∧ Iobj,mask (x, y) = 1, goto step 3. If m(x, y) = 1, ignore this pixel and continue with the next pixel.6 If m(x, y) = 2, increase x until m(x, y) = 2 again, ignoring the whole range of pixels.7 3. Create a list B of border points and store (x, y) as the first element. 4. Set (x , y ) = (x, y). 5. Scan the 8-neighborhood of (x , y ) using (x , y ), starting at the pixel that follows the butlast pixel stored in B in a counterclockwise orientation, or at (x − 1, y − 1) if B contains only one pixel. Proceed counterclockwise, skipping coordinates that lie outside of the image, until Iobj,mask (x , y ) = 1. If (x , y ) is identical with the first element of B, goto step 6. Else store (x , y ) in B. Set (x , y ) = (x , y ) and goto step 5. 6. Iterate through B, considering, for every element (xi , yi ) its predecessor (xi−1 , yi−1 ) and its successor (xi+1 , yi+1 ). The first element is the successor of the last, which is the predecessor of the first. If yi−1 = yi+1 = yi , set m(xi , yi ) = 1 to indicate that the border touches the line y = yi at xi . Otherwise, if yi−1 = yi ∨ yi = yi+1 , set m(xi , yi ) = 2 to indicates that the border intersects with the line y = yi at xi . 7. Add B to the list of computed borders and proceed with step 2. Fig. 2.11. Algorithm to find the border points of all regions in an image.
2.1.2.3 Geometric Features From the multitude of features that can be computed for a closed polygon, only a subset is suitable for a concrete recognition task. A feature is suitable if it has a high inter-gesture variance (varies significantly between different gestures) and a low intra-gesture variance (varies little between multiple productions of the same gesture). The first property means that the feature carries much information, while the second indicates that it is not significantly affected by noise or unintentional variations that will inevitably occur. Additionally, every feature should be stable, meaning that small changes in input data never result in large changes in the feature. Finally the feature must be computable with sufficient accuracy and speed. A certain feature’s suitability thus depends on the constellation of the vocabulary (because this affects inter- and intra-gesture variance) and on the actual application scenario in terms of recording conditions, hardware etc. It is therefore a common approach to first compute as many features as possible and then examine each one with respect to its suitability for the specific recognition task (see also Sect. 2.1.3.3). The choice of features is of central importance since it affects system design throughout the processing chain. The final system’s performance depends significantly on the chosen features and their properties. This section presents a selection of frequently used features and the equations to calculate them. Additionally, the way each feature is affected by the camera’s perspective, resolution, and distance to the object is discussed, for these are often
Hand Gesture Commands
23
(a)
(b)
(c)
(d)
(e)
(f)
Fig. 2.12. a, b, c: Borders of the regions shown in Fig. 2.8 (d, e, f), computed by the algorithm described in Fig. 2.11. d, e, f: Additional smoothing with a Gaussian kernel before segmentation.
important factors in practical applications. In the image domain, changing the camera’s perspective results in rotation and/or translation of the object, while resolution and object distance affect the object’s scale (in terms of pixels). Since the error introduced in the shape representation by the discretization of the image (the discretization noise) is itself affected by image resolution, rotation, translation, and scaling of the shape, features can, in a strict sense, be invariant of these transformations only in continuous space. All statements regarding transformation invariance therefore refer to continuous space. In discretized images, features declared “invariant” may still show small variance. This can usually be ignored unless the shape’s size is on the order of several pixels. Border Length The border length l can trivially be computed from B, considering that the distance between √ two successive border points is 1 if either their x or y coordinates are equal, and 2 otherwise. l depends on scale/resolution, and is translation and rotation invariant. Area, Center of Gravity, and Second Order Moments In [25] an efficient algorithm for the computation of arbitrary moments νp,q of polygons is presented. The area a = ν0,0 , as well as the normalized moments αp,q
24
Non-Intrusive Acquisition of Human Action
and central moments μp,q up to order 2, including the center of gravity (COG) (xcog , ycog ) = (α1,0 , α0,1 ), are obtained as shown in (2.15) to (2.23). xi and yi with i = 0, 1, . . . , n − 1 refer to the elements of BR as defined in (2.14). Since these equations require the polygon to be closed, xn = x0 and yn = y0 is defined for i = n. 1 xi−1 yi − xi yi−1 2 i=1 n
ν0,0 = a =
(2.15)
α1,0 = xcog =
1 (xi−1 yi − xi yi−1 )(xi−1 + xi ) 6a i=1
(2.16)
α0,1 = ycog =
1 (xi−1 yi − xi yi−1 )(yi−1 + yi ) 6a i=1
(2.17)
n
n
α2,0 =
1 (xi−1 yi − xi yi−1 )(x2i−1 + xi−1 xi + x2i ) 12a i=1
α1,1 =
1 (xi−1 yi − xi yi−1 )(2xi−1 yi−1 + xi−1 yi + xi yi−1 + 2xi yi ) 24a i=1
n
(2.18)
n
(2.19) α2,0
n 1 2 (xi−1 yi − xi yi−1 )(yi−1 + yi−1 yi + yi2 ) = 12a i=1
(2.20)
μ2,0 = α2,0 − α21,0
(2.21)
μ1,1 = α1,1 − α1,0 α0,1 μ0,2 = α0,2 − α20,1
(2.22) (2.23)
The area a depends on scale/resolution, and is independent of translation and rotation. The center of gravity (xcog , ycog ) is obviously translation variant and depends on resolution. It is rotation invariant only if the rotation occurs around (xcog , ycog ) itself, and affected by scaling (i.e. changing the object’s distance to the camera) unless its angle to the optical axis remains constant. The second order moments (p+q = 2) are primarily used to compute other shape descriptors, as described below. Eccentricity One possible measure for eccentricity e that is based on central moments is given in [10]: (μ2,0 − μ0,2 )2 + 4μ1,1 (2.24) a (2.24) is zero for circular shapes and increases for elongated shapes. It is rotation and translation invariant, but variant with respect to scale/resolution. e=
Hand Gesture Commands
25
Another intuitive measure is based on the object’s inertia along and perpendicular to its main axis, j1 and j2 , respectively: e=
j1 j2
(2.25)
with
μ2,0 + μ0,2 + j1 = 2a μ2,0 + μ0,2 − j2 = 2a
μ2,0 − μ0,2 2a μ2,0 − μ0,2 2a
2 + 2 +
μ
1,1
2 (2.26)
a μ
1,1
a
2 (2.27)
Here e starts at 1 for circular shapes, increases for elongated shapes, and is invariant of translation, rotation, and scale/resolution. Orientation The region’s main axis is defined as the axis of the least moment of inertia. Its orientation α is given by 2μ1,1 1 (2.28) α = arctan 2 μ2,0 − μ0,2 and corresponds to what one would intuitively call “the object’s orientation”. Orientation is translation and scale/resolution invariant. α carries most information for elongated shapes and becomes increasingly random for circular shapes. It is 180◦ periodic, which necessitates special handling in most cases. For example, the angle by which an object with α1 = 10◦ has to be rotated to align with an object with α2 = 170◦ is 20◦ rather than |α1 − α2 | = 160◦ . When using orientation as a feature for classification, a simple workaround to this problem is to ensure 0 ≤ α < 180◦ and, rather than use α directly as a feature, compute two new features f1 (α) = sin α and f2 (α) = cos 2α instead, so that f1 (0) = f1 (180◦ ) and f2 (0) = f2 (180◦ ). Compactness A shape’s compactness c ∈ [0, 1] is defined as 4πa (2.29) l2 Compact shapes (c → 1) have short borders l that contain a large area a. The most compact shape is a circle (c = 1), while for elongated or frayed shapes, c → 0. Compactness is rotation, translation, and scale/resolution invariant. c=
26
Non-Intrusive Acquisition of Human Action
Border Features In addition to the border length l, the minimum and maximum pixel coordinates of the object, xmin , xmax , ymin and ymax , as well as the minimum and maximum distance from the center of gravity to the border, rmin and rmax , can be calculated from B. Minimum and maximum coordinates are not invariant to any transformation. rmin and rmax are invariant to translation and rotation, and variant to scale/resolution. Normalization Some of the above features can only be used in a real-life application if a suitable normalization is performed to eliminate translation and scale variance. For example, the user might perform gestures at different locations in the image and at different distances from the camera. Which normalization strategy yields the best results depends on the actual application and recording conditions, and is commonly found empirically. Normalization is frequently neglected, even though it is an essential part of feature extraction. The following paragraphs present several suggestions. In the remainder of this section, the result of the normalization of a feature f is designated by f . For both static and dynamic gestures, the user’s face can serve as a reference for hand size and location if it is visible and a sufficiently reliable face detection is available. In case different resolutions with identical aspect ratio are to be supported, resolution independence can be achieved by specifying lengths and coordinates relative to the image width N (x and y must be normalized by the same value to preserve image geometry) and area relative to N 2 . If a direct normalization is not feasible, invariance can also be achieved by computing a new feature from two or more unnormalized features. For example, from xcog , xmin , and xmax , a resolution and translation invariant feature xp can be computed as xp =
xmax − xcog xcog − xmin
(2.30)
xp specifies the ratio of the longest horizontal protrusion, measured from the center of gravity, to the right and to the left. For dynamic gestures, several additional methods can be used to compute a normalized feature f (t) from an unnormalized feature f (t) without any additional information: •
Subtracting f (1) so that f (1) = 0: f (t) = f (t) − f (1)
•
(2.31)
Subtracting the arithmetic mean f or median fmedian of f : f (t) = f (t) − f
f (t) = f (t) − fmedian
(2.32) (2.33)
Hand Gesture Commands •
27
Mapping f to a fixed interval, e.g. [0, 1]: f (t) =
•
f (t) − min f (t) max f (t) − min f (t)
(2.34)
Dividing f by max |f (t)| so that |f (t)| ≤ 1: f (t) =
f (t) max |f (t)|
(2.35)
Derivatives In features computed for dynamic gestures, invariance of a constant offset may also be achieved by derivation. Computing the derivative f˙(t) of a feature f (t) and using it as an additional element in the feature vector to emphasize changes in f (t) can sometimes be a simple yet effective method to improve classification performance. 2.1.2.4 Example To illustrate the use of the algorithms described above, the static example gestures introduced in Fig. 2.2 were segmented as shown in Fig. 2.13, using the automatic algorithms presented in Fig. 2.6 and 2.11, and features were computed as shown in Tab. 2.5, using the LTI-L IB class geometricFeatures. For resolution independence, lengths and coordinates are normalized by the image width N , and area is normalized by N 2 . α specifies the angle by which the object would have to be rotated to align with the x axis.
(a) “left”
(b) “right”
(c) “stop”
(d) none
Fig. 2.13. Boundaries of the static hand gestures shown in Fig. 2.2. Corresponding features are shown in Tab. 2.5.
For the dynamic example gestures “clockwise”, “open”, and “grab”, the features area a, compactness c, and x coordinate of the center of gravity xcog were computed. Since a depends on the hand’s anatomy and its distance to the camera, it was normalized by its maximum (see (2.35)) to eliminate this dependency. xcog was divided by N to eliminate resolution dependency, and additionally normalized according to (2.32) so that x cog = 0. This allows the gestures to be performed anywhere in the image. The resulting plots for normalized area a , compactness c (which does not require normalization), and normalized x coordinate of the center of gravity xcog are visualized in Fig. 2.14. The plots are scaled equally for each feature to allow a direct comparison. Since “counterclockwise”, “close”, and “drop” are backwards executions of the above gestures, their feature plots are simply temporally mirrored versions of these plots.
28
Non-Intrusive Acquisition of Human Action Table 2.5. Features computed from the boundaries shown in Fig. 2.13. “left” Feature Symbol (2.13a) Normalized Border Length l 1.958 Normalized Area a 0.138 Normalized Center of Gravity xcog 0.491 ycog 0.486 Eccentricity e 1.758 Orientation α 57.4° Compactness c 0.451 Normalized Min./Max. Coordinates xmin 0.128 xmax 0.691 ymin 0.256 ymax 0.747 Protrusion Ratio xp 0.550
Gesture “right” “stop” (2.13b) (2.13c) 1.705 3.306 0.115 0.156 0.560 0.544 0.527 0.487 1.434 1.722 147.4° 61.7° 0.498 0.180 0.359 0.241 0.894 0.881 0.341 0.153 0.747 0.747 1.664 1.110
none (2.13d) 1.405 0.114 0.479 0.537 2.908 58.7° 0.724 0.284 0.681 0.325 0.747 1.036
Fig. 2.14. Normalized area a , compactness c, and normalized x coordinate of the center of gravity xcog plotted over time t = 1, 2, . . . , 60 for the dynamic gestures “clockwise” (left column), “open” (middle column), and “grab” (right column) from the example vocabulary shown in Fig. 2.3.
Hand Gesture Commands
29
2.1.3 Feature Classification The task of feature classification occurs in all pattern recognition systems and has been subject of considerable research effort. A large number of algorithms is available to build classifiers for various requirements. The classifiers considered in this section operate in two phases: In the training phase the classifier “learns” the vocabulary from a sufficiently large number of representative examples (the training samples). This “knowledge” is then applied in the following classification phase. Classifiers that continue to learn even in the classification phase, e.g. to automatically adapt to the current user, shall not be considered here. 2.1.3.1 Classification Concepts This section introduces several basic concepts and terminology commonly used in pattern classification, with the particular application to gesture recognition. Mathematical Description of the Classification Task From a mathematical point of view, the task of classification is that of identifying an unknown event ω, given a finite set Ω of n mutually exclusive possible events ω ∈ Ω = {ω1 , ω2 , . . . , ωn }
(2.36)
The elements of Ω are called classes. In gesture recognition Ω is the vocabulary, and each class ωi (i = 1, 2, . . . , n) represents a gesture. Note that (2.36) restricts the allowed inputs to elements of Ω. This means that the system is never presented with an unknown gesture, which influences the classifier’s design and algorithms. The classifier receives, from the feature extraction stage, an observation O, which, for example, might be a single feature vector for static gestures, or a sequence of feature vectors for dynamic gestures. Based on this observation it outputs a result ω ˆ ∈Ω
where ω ˆ = ωk ,
k ∈ {1, 2, . . . , n}
(2.37)
k denotes the index of the class of the event ω assumed to be the source of the observation O. If ω ˆ = ω then the classification result is correct, otherwise it is wrong. To account for the case that O does not bear sufficient similarity to any element in Ω, one may wish to allow a rejection of O. This is accomplished by introducing a ˆ as pseudo-class ω0 and defining a set of classifier outputs Ω ˆ = Ω ∪ {ω0 } Ω
(2.38)
Thus, (2.37) becomes ˆ ω ˆ ∈Ω
where ω ˆ = ωk ,
k ∈ {0, 1, . . . , n}
(2.39)
30
Non-Intrusive Acquisition of Human Action
Modeling Classification Effects Several types of classifiers can be designed to consider a cost or loss value L for classification errors, including rejection. In general, the loss value depends on the actual input class ω and the classifier’s output ω ˆ: L(ω, ω ˆ ) = cost incurred for classifying event of class ω as ω ˆ
(2.40)
The cost of a correct result is set to zero: L(ω, ω) = 0
(2.41)
This allows to model applications where certain misclassifications are more serious than others. For example, let us consider a gesture recognition application that performs navigation of a menu structure. Misclassifying the gesture for “return to main menu” as “move cursor down” requires the user to perform “return to main menu” again. Misclassifying “move cursor down” as “return to main menu”, however, discards the currently selected menu item, which is a more serious error because the user will have to navigate to this menu item all over again (assuming that there is no “undo” or “back” functionality). Simplifications Classifiers that consider rejection and loss values are discussed in [22]. For the purpose of this introduction, we will assume that the classifier never rejects its input (i.e. (2.37) holds) and that the loss incurred by a classification error is a constant value L: L(ω, ω ˆ) = L
for ω = ω ˆ
(2.42)
This simplifies the following equations and allows us to focus on the basic classification principles. Garbage Classes An alternative to rejecting the input is to explicitly include garbage classes in Ω that represent events to which the system should not react. The classifier treats these classes just like regular classes, but the subsequent stages do not perform any action when the classification result ω ˆ is a garbage class. For example, a gesture recognition system may be constantly observing the user’s hand holding a steering wheel, but react only to specific gestures different from steering motions. Garbage classes for this application would contain movements of the hand along the circular shape of the steering wheel.
Hand Gesture Commands
31
Modes of Classification We distinguish between supervised classification and unsupervised classification: •
•
In supervised classification the training samples are labeled, i.e. the class of each training sample is known. Classification of a new observation is performed by a comparison with these samples (or models created from them) and yields the class that best matches the observation, according to a matching criterion. In unsupervised classification the training samples are unlabeled. Clustering algorithms are used to group similar samples before classification is performed. Parameters such as the number of clusters to create or the average cluster size can be specified to influence the clustering process. The task of labeling the samples is thus performed by the classifier, which of course may introduce errors. Classification itself is then performed as above, but returns a cluster index instead of a label.
For gesture recognition, supervised classification is by far the most common method since the labeling of the training samples can easily be done in the recording process. Overfitting With many classification algorithms, optimizing performance for the training data at hand bears the risk of reducing performance for other (generic) input data. This is called overfitting and presents a common problem in machine learning, especially for small sets of training samples. A classifier with a set of parameters p is said to overfit the training samples T if there exists another set of parameters p that yields lower performance on T , but higher performance in the actual “real-world” application [15]. A strategy for avoiding overfitting is to use disjunct sets of samples for training and testing. This explicitly measures the classifier’s ability of generalization and allows to include it in the optimization process. Obviously, the test samples need to be sufficiently distinct from the training samples for this approach to be effective. 2.1.3.2 Classification Algorithms From the numerous strategies that exist for finding the best-matching class for an observation, three will be presented here: • • •
A simple, easy-to-implement rule-based approach suitable for small vocabularies of static gestures. The concept of maximum likelihood classification, which can be applied to a multitude of problems. Hidden Markov Models (HMMs) for dynamic gestures. HMMs are frequently used for the classification of various dynamic processes, including speech and sign language.
Further algorithms, such as artificial neural networks, can be found in the literature on pattern recognition and machine learning. A good starting point is [15].
32
Non-Intrusive Acquisition of Human Action
2.1.3.3 Feature Selection The feature classification stage receives, for every frame of input data, one feature vector from the feature extraction stage. For static gestures this constitutes the complete observation, while dynamic gestures are represented by a sequence of feature vectors. The constellation of the feature vector (which remains constant throughout the training and classification phases) is an important design decision that can significantly affect the system’s performance. Most recognition systems do not use all features that, theoretically, could be computed from the input data. Instead, a subset of suitable features is selected using criteria and algorithms described below. Reducing the number of features usually reduces computational cost (processing time, memory requirements) and system complexity in both the feature extraction and classification stage. In some cases it may also increase recognition accuracy by eliminating erratic features and thereby emphasizing the remaining features. In the feature selection process one or more features are chosen that allow the reliable identification of each element in the vocabulary. This is provided if the selected features are approximately constant for multiple productions of the same gesture (low intra-gesture variance) but vary significantly between different gestures (high inter-gesture variance). It is not required that different gestures differ in every element of the feature vector, but there must be at least one element in the feature vector that differs beyond the natural (unintentional) intra-gesture variance, segmentation inaccuracy, and noise. It is obvious that, in order to determine intra- and inter-gesture variance, a feature has to be computed for numerous productions of every gesture in the vocabulary. A representative mixture of permitted recording conditions should be used to find out how each feature is affected e.g. by lighting conditions, image noise etc. For person-independent gesture recognition this should include multiple persons. Based on these samples features are then selected either by manual inspection according to the criteria mentioned above, or by an automatic algorithm that finds a feature vector which optimizes the performance of the chosen classifier on the available data. Three simple feature selection algorithms (stepwise forward selection, stepwise backward elimination, and exhaustive search) that can be used for various classifiers, including Hidden Markov Models and maximum likelihood classification (see Sect. 2.1.3.5 and 2.1.3.6), are presented in Fig. 2.15. It is important to realize that although the feature selection resulting from either manual or automatic analysis may be optimal for the data upon which it is based, this is not necessarily the case for the actual practical application. The recorded samples often cannot cover all recording conditions under which the system may be used, and features that work well on this data may not work under different recording conditions or may even turn out to be disadvantageous. It may therefore be necessary to reconsider the feature selection if the actual recognition accuracy falls short of the expectation.
Hand Gesture Commands
33
Stepwise Forward Selection 1. Start with an empty feature selection and a recognition accuracy of zero. 2. For every feature that is not in the current feature selection, measure recognition accuracy on the current feature selection extended by this feature. Add the feature for which the resulting recognition accuracy is highest to the current selection. If no feature increases recognition accuracy, halt. 3. If computational cost or system complexity exceed the acceptable maximum, remove the last added feature from the selection and halt. 4. If recognition accuracy is sufficient or all features are included in the current selection, halt. 5. Goto step 2.
Stepwise backward elimination 1. Start with a feature selection containing all available features, and measure recognition accuracy. 2. For every feature in the current feature selection, measure recognition accuracy on the current feature selection reduced by this feature. Remove the feature for which the resulting recognition accuracy is highest from the current selection. 3. If computational cost or system complexity exceed the acceptable maximum and the current selection contains more than one feature, goto step 2. Otherwise, halt.
Exhaustive search 1. Let n be the total number of features and m the number of selected features. n possible 2. For every m ∈ {1, 2, . . . , n}, measure recognition accuracy for all m n combinations). Discard feature combinations (this equals a total of n i=1 i all cases where computational cost or system complexity exceed the acceptable maximum, then choose the selection that yields the highest recognition accuracy. Fig. 2.15. Three algorithms for automatic feature selection. Stepwise forward selection and stepwise backward elimination are fast heuristic approaches that are not guaranteed to find the global optimum. This is possible with exhaustive search, albeit at a significantly higher computational cost.
2.1.3.4 Rule-based Classification A simple heuristic approach to classification is a set of explicit IF-THEN rules that refer to the target’s features and require them to lie within a certain range that is typical of a specific gesture. The fact that an IF-THEN rule usually results in just a single line of source code makes this method straightforward to implement. For example, to identify the static “stop” gesture (Fig. 2.2c) among various other gestures, a simple solution would be to use the shape’s compactness c: IF
c < 0.25
THEN
the observed gesture is ”stop”.
(2.43)
34
Non-Intrusive Acquisition of Human Action
The threshold of 0.25 for c was determined empirically from a data set of multiple productions of “stop” and all other gestures, performed by different users. On this data set, max c ≈ 0.23 and c ≈ 0.18 was observed for “stop”, while c 0.23 for all other gestures. Obviously, with increasing vocabulary size, number and complexity of the rules grow and they can quickly become difficult to handle. The manual creation of rules, as described in this section, is therefore only suitable for very small vocabularies such as the static example vocabulary, where appropriate thresholds are obvious. Algorithms for automatic learning of rules are described in [15]. For larger vocabularies, rules are often used as a pre-stage to another classifier. For instance, a rule like IF
a < Θa N 2
THEN
the object is not the hand.
(2.44)
discards noise and distractors in the background that are too small to represent the hand. The threshold Θa specifies the required minimum size relative to the squared image width. Another example is to determine whether an object in an image sequence is moving or idle. The following rule checks for a certain minimum amount of motion:
IF
max i=j
(xcog (i) − xcog (j))2 + (ycog (i) − ycog (j))2 ) < Θmotion N THEN
(2.45)
the hand is idle.
where i and j are frame indices, and Θmotion specifies the minimum dislocation for the hand to qualify as moving, relative to the image width. 2.1.3.5 Maximum Likelihood Classification Due to its simplicity and general applicability, the concept of maximum likelihood classification is widely used for all kinds of pattern recognition problems. An extensive discussion can be found in [22]. This section provides a short introduction. We assume the observation O to be a vector of K scalar features fi : ⎛ ⎞ f1 ⎜ f2 ⎟ ⎟ ⎜ O=⎜ . ⎟ (2.46) ⎝ .. ⎠ fK Stochastic Framework The maximum likelihood classifier identifies the class ω ∈ Ω that is most likely to have caused the observation O by maximizing the a posteriori probability P (ω|O): ω ˆ = argmax P (ω|O) ω∈Ω
To solve this maximization problem we first apply Bayes’ rule:
(2.47)
Hand Gesture Commands
35
P (ω|O) =
P (O|ω)P (ω) P (O)
(2.48)
For simplification it is commonly assumed that all n classes are equiprobable: 1 (2.49) n Therefore, since neither P (ω) nor P (O) depend on ω, we can omit them when substituting (2.48) in (2.47), leaving only the observation probability P (O|ω): P (ω) =
ω ˆ = argmax P (O|ω)
(2.50)
ω∈Ω
Estimation of Observation Probabilities For reasons of efficiency and simplicity, the observation probability P (O|ω) is commonly computed from a class-specific parametric distribution of O. This distribution is estimated from a set of training samples. Since the number of training samples is often insufficient to identify an appropriate model, a normal distribution is assumed in most cases. This is motivated by practical considerations and empirical success rather that by statistical analysis. Usually good results can be achieved even if the actual distribution significantly violates this assumption. To increase modeling accuracy one may decide to use multimodal distributions. This decision should be backed by sufficient data and motivated by the nature of the regarded feature in order to avoid overfitting (see Sect. 2.1.3.1). Consider, for example, the task of modeling the compactness c for the “stop” gesture shown in Fig. 2.2c. Computing c for e.g. 20 different productions of this gesture would not provide sufficient motivation for modeling it with a multimodal distribution. Furthermore, there is no reason to assume that c actually obeys a multimodal distribution. Even if measurements do suggest this, it is not backed by the biological properties of the human hand; adding more measurements is likely to change the distribution to a single mode. In this section only unimodal distributions will be used. The multidimensional normal distribution is given by 1 1 exp − (O − μ)T C−1 (O − μ) N (O, μ, C) = 2 (2π)K det C
(2.51)
where μ and C are the mean and covariance of the stochastic process {O}: μ = E{O} C = E{(O − μ)(O − μ) } = cov{O} T
mean of {O}
(2.52)
covariance matrix of {O}
(2.53)
For the one-dimensional case with a single feature f this reduces to the familiar equation
36
Non-Intrusive Acquisition of Human Action 2 1 f −μ 1 exp − N (f, μ, σ ) = 2 σ (2π)σ 2
(2.54)
Computing μi and Ci from the training samples for each class ωi in Ω yields the desired n models that allow to solve (2.50): P (O|ωi ) = N (O, μi , Ci ),
i = 1, 2, . . . , n
(2.55)
Evaluation of Observation Probabilities Many applications require the classification stage to output only the class ωi for which P (O|ω) is maximal, or possibly a list of all classes in Ω, sorted by P (O|ω). The actual values of the conditional probabilities themselves are not of interest. In this case, a strictly monotonic mapping can be applied to (2.55) to yield a score S that imposes the same order on the elements of Ω, but is faster to compute. S(O|ωi ) is obtained by multiplying P (O|ωi ) by (2π)K and taking the logarithm: 1 1 S(O|ωi ) = − ln(det Ci ) − (O − μi )T C−1 i (O − μi ) 2 2
(2.56)
ω ˆ = argmax S(O|ω)
(2.57)
Thus,
ω∈Ω
It is instructive to examine this equation in feature space. The surfaces of constant score S(O|ωi ) form hyperellipsoids with center μi . This is caused by the fact that, in general, the elements of O show different variances. In gesture recognition, variance is commonly caused by the inevitable “natural” variations in execution, by differences in anatomy (e.g. hand size), segmentation inaccuracy, and by noise introduced in feature computation. Features are affected by this in different ways: For instance, scale invariant features are not affected by variations in hand size. If the covariance matrices Ci can be considered equal for i = 1, 2, . . . , n (which is often assumed for simplicity), i.e. Ci = C i = 1, 2, . . . , n
(2.58)
we can further simplify (2.56) by omitting the constant term − 21 ln(det C) and multiplying by −2: S(O|ωi ) = (O − μi )T C−1 (O − μi )
(2.59)
Changing the score’s sign requires to replace the argmax operation in (2.57) with argmin. The right side of (2.59) is known as the Mahalanobis distance between O and μ, and the classifier itself is called Mahalanobis classifier. Expanding (2.59) yields
Hand Gesture Commands
37
S(O|ωi ) = OT C−1 O − 2μTi C−1 O + μTi C−1 μi
(2.60)
For efficient computation this can be further simplified by omitting the constant OT C−1 O: S(O|ωi ) = −2μTi C−1 O + μTi C−1 μi
(2.61)
2
If all elements of O have equal variance σ , C can be written as the product of σ 2 and the identity matrix I: C = σ2 I
(2.62)
Replacing this in (2.59) and omitting the constant σ 2 leads to the Euclidian classifier: S(O|ωi ) = (O − μi )T (O − μi ) = |O − μi |2
(2.63)
2.1.3.6 Classification Using Hidden Markov Models The classification of time-dependent processes such as gestures, sign language, or speech, suggests to explicitly model not only the samples’ distribution in the feature space, but also the dynamics of the process from which the samples originate. In other words, a classifier for dynamic events should consider variations in execution speed, which means that we need to identify and model these variations. This requirement is met by Hidden Markov Models (HMMs), a concept derived from Markov chains.8 A complete discussion of HMMs is out of the scope of this chapter. We will therefore focus on basic algorithms and accept several simplifications, yet enough information will be provided for an implementation and practical application. Further details, as well as other types and variants of HMMs, can be found e.g. in [8, 11, 13, 19–21]. The LTI-L IB contains multiple classes for HMMs and HMMbased classifiers that offer several advanced features not discussed here. In the following the observation O denotes a sequence of T individual observations ot : O = (o1 o2 · · · oT )
(2.64)
Each individual observation corresponds to a K-dimensional feature vector: ⎛ ⎞ f1 (t) ⎜ f2 (t) ⎟ ⎜ ⎟ ot = ⎜ . ⎟ (2.65) ⎝ .. ⎠ fK (t) For the purpose of this introduction we will at first assume K = 1 so that ot is scalar: 8
Markov chains describe stochastic processes in which the conditional probability distribution of future states, given the present state, depends only on the present state.
38
Non-Intrusive Acquisition of Human Action
ot = ot
(2.66)
This restriction will be removed later. Concept This section introduces the basic concepts of HMMs on the basis of an example. In Fig. 2.16 the feature x˙ cog (t) has been plotted for three samples of the gesture “clockwise” (Fig. 2.3a). It is obvious that, while the three plots have similar shape, they are temporally displaced. We will ignore this for now and divide the time axis into segments of 5 observations each (one observation corresponds to one frame). Since the samples have 60 frames this yields 12 segments, visualized by straight vertical lines.9 For each segment i, i = 1, 2, . . . , 12, mean μi and variance σi2 were computed from the corresponding observations of all three samples (i.e. from 15 values each), as visualized in Fig. 2.16 by gray bars of length 2σi , centered at μi .
Fig. 2.16. Overlay plot of the feature x˙ cog (t) for three productions of the gesture “clockwise”, divided into 12 segments of 5 observations (frames) each. The gray bars visualize mean and variance for each segment.
This represents a first crude model (though not an HMM) of the time dependent stochastic process x˙ cog (t), as shown in Fig. 2.17. The model consists of 12 states 9
The segment size of 5 observations was empirically found to be a suitable temporal resolution.
Hand Gesture Commands
39
si , each characterized by μi and σi2 . As in the previous section we assume unimodal normal distributions, again motivated by empirical success. The model is rigid with respect to time, i.e. each state corresponds to a fixed set of observations. For example, state s2 models x˙ cog (t) for t = 6, 7, 8, 9, 10.
Fig. 2.17. A simple model for the time dependent process x˙ cog (t) that rigidly maps 60 observations to 12 states.
This model allows to compute the probability of a given observation O originating from the modeled process (i.e., x˙ cog (t) for the gesture “clockwise”) as
P (O|“clockwise”) =
60
t−1 + 1 (2.67) with i(t) = 5
2 N (ot , μi(t) , σi(t) )
t=1
where N is the normal distribution (see (2.54)). Thus far we have modeled the first stochastic process, the amplitude of x˙ cog (t), for fixed time segments. To extend the model to the second stochastic process inherent in x˙ cog (t), the variation in execution speed and temporal displacement apparent in Fig. 2.16, we abandon the rigid mapping of observations to states. As before, s1 and s12 are defined as the initial and final state, respectively, but the model is extended by allowing stochastic transitions between the states. At each time step the model may remain in its state si or change its state to si+1 or to si+2 . A transition from si to sj is assigned a probability ai,j , with ai,j = 1 (2.68) j
and ai,j = 0 for j ∈ / {i, i + 1, i + 2}
(2.69)
Fig. 2.18 shows the modified model for x˙ cog (t), which now constitutes a complete Hidden Markov Model. Transitions from si to si allow to model a “slower-thannormal” execution part, while those from si to ss+2 model a “faster-than-normal” execution part. Equation (2.69) is not a general requirement for HMMs; it is a property of the specific Bakis topology used in this introduction. One may choose other topologies, allowing e.g. a transition from si to si+3 , in order to optimize classification performance. In this section we will stick to the Bakis topology because it is widely used
40
Non-Intrusive Acquisition of Human Action
Fig. 2.18. A Hidden Markov Model in Bakis topology. Observations are mapped dynamically to states.
in speech and sign language recognition, and because it is the simplest topology that models both delay and acceleration.10 Even though we have not changed the number of states compared to the rigid model, the stochastic transitions affect the distribution parameters μi and σi2 of each state i. In the rigid model, the lack of temporal alignment among the training samples leads to μi and σi2 not accurately representing the input data. Especially for the first six states σ is high and μ has a lower amplitude than the training samples (cf. Fig. 2.16). Since the HMM provides temporal alignment by means of the transition probabilities ai,j we expect σ to decrease and μ to better match the sample plots (this is proven in Fig. 2.21). The computation of ai,j , μi and σi2 will be discussed later. For the moment we will assume these values to be given. In the following a general HMM with N states in Bakis topology is considered. The sequence of states adopted by this model is denoted by the sequence of their indices q = (q1 q2 . . . qT ) with
qt ∈ {1, 2, . . . , N },
t = 1, 2, . . . , T
(2.70)
where qt indicates the state index at time t and T equals the number of observations. This notion allows to specify the probabilities ai,j as ai,j = P (qt = j|qt−1 = i)
(2.71)
For brevity, the complete set of transition probabilities is denoted by a square matrix of N rows and columns A = [ai,j ]N ×N
(2.72)
One may want to loosen the restriction that q1 = s1 by introducing an initial state probability vector π: 10
If the number of states N is significantly smaller than the number of observations T , acceleration can also be modeled solely by the transition from si to si+1 .
Hand Gesture Commands π = (π1 π2 . . . πN )
41 with πi = P (q1 = si ) and
πi = 1
(2.73)
i
This can be advantageous when recognizing a continuous sequence of gestures (such as a sentence of sign language) and coarticulation effects might lead to the initial states of some models being skipped. In this section, however, we stick to π1 = 1 and πi = 0 for i = 1. To achieve a compact notation for a complete HMM we introduce a vector containing the states’ distribution functions: B = (b1 b2 . . . bN )
(2.74)
where bi specifies the distribution of the observation o in state si . Since we use unimodal normal distributions, bi = N (o, μi , σi2 )
(2.75)
A Hidden Markov Model λ is thus completely specified by λ = (π, A, B)
(2.76)
Obviously, to use HMMs for classification, we again need to compute the probability that a given observation O originates from a model λ. In contrast to (2.67) this is no longer simply a product of probabilities. Instead, we have to consider all possible state sequences of length T , denoted by the set QT , and sum up the accordant probabilities:
P (O|λ) =
q∈QT
P (O, q|λ) =
b1 (o1 )
q∈QT
T
aqt−1 ,qt bqt (ot )
(2.77)
t=2
with q1 = 1 ∧ qT = N
(2.78)
Since, in general, the distributions bi will overlap, it is not possible to identify a single state sequence q associated with O. This property is reflected in the name Hidden Markov Model. ˆ that best matches the observation O is found by maximizing (2.77) The model λ on the set Λ of all HMMs: ˆ = argmax P (O|λ) λ
(2.79)
λ∈Λ
Even though efficient algorithms exist to solve this equation, (2.79) is commonly approximated by ˆ = argmax P (O|λ, q∗ ) λ λ∈Λ
(2.80)
42
Non-Intrusive Acquisition of Human Action
where q∗ represents the single most likely state sequence: q∗ = argmax P (q|O, λ) q∈QT
(2.81)
This is equivalent to approximating (2.77) by its largest summand, which may appear drastic, but (2.79) and (2.80) have been found to be strongly correlated [21], and practical success in terms of accuracy and processing speed warrant this approximation. Application of Bayes’ rule to P (q|O, λ) yields P (q|O, λ) =
P (O, q|λ) P (O|λ)
(2.82)
Since P (O|λ) does not depend on q we can simplify (2.81) to q∗ = argmax P (O, q|λ) q∈QT
(2.83)
In order to solve this maximization problem it is helpful to consider a graphical representation of the elements of QT . Fig. 2.19 shows the Trellis diagram in which all possible state sequences q are shown superimposed over the time index t.
Fig. 2.19. Trellis diagram for an HMM with Bakis topology.
The optimal state sequence q∗ can efficiently be computed using the Viterbi algorithm (see Fig. 2.20): Instead of performing a brute force search of QT , this method
Hand Gesture Commands
43
exploits the fact that from all state sequences q that arrive at the same state si at time index t, only the one for which the probability up to si is highest, q∗i , can possibly solve (2.83): q∗i = argmax b1 (o1 ) q∈{q|qt =i}
t
aqt −1 ,qt bqt (ot )
(2.84)
t =2
All others need not be considered, because regardless of the subsequence of states following si , the resulting total probability would be lower than that achieved by following q∗i by the same subsequence. (A common application of this algorithm in everyday life is the search for the shortest path from a location A to another location B in a city: From all paths that have a common intermediate point C, only the one for which the distance from A to C is shortest has a chance of being the shortest path between A and B.) Viterbi Search Algorithm Define t as a loop index and Qt as a set of state sequences of length t. 1. Start with t = 1, Q1 = {(s1 )} and Qt = ∅ for t = 2, 3, . . . , T . 2. For each state si (i = 1, 2, . . . , N ) that is accessible from the last state of at least one of the sequences in Qt do: Find the state sequence q with t
q = argmax b1 (o1 ) q∈Qt
t =2
aqt −1 ,qt bqt (ot ) aqt ,i bi (ot+1 )
(2.85)
Append i to q then add q to Qt+1 . 3. Increment t. 4. If t < T , goto step 2. 5. The state sequence q ∈ QT with qT = N solves (2.83). Fig. 2.20. The Viterbi algorithm for an efficient solution of (2.83).
In an implementation of the Viterbi algorithm the value of (2.85) is efficiently stored to avoid repeated computation. In case multiple state sequence maximize (2.85) only one of them needs to be considered. Implementational Aspects The product of probabilities in (2.77) and (2.85) may become very small, causing numerical problems. This can be avoided by using scores instead of probabilities, with Score(P ) = ln(P )
(2.86)
Since ln ab = ln a + ln b this allows to replace the product of probabilities with a sum of scores. Because the logarithm is a strictly monotonic mapping, the inverse
44
Non-Intrusive Acquisition of Human Action
exp(·) operation may be omitted without affecting the argmax operation. In addition to increased numerical stability the use of scores also reduces computational cost.11 Estimation of HMM Parameters We will now discuss the iterative estimation of the transition probabilities A and the distribution functions B for a given number of states N and a set of M training samples O1 , O2 , . . . , OM . Tm denotes the length of Om (m ∈ {1, 2, . . . , M }). An initial estimate of B is obtained in the same way as shown in Fig. 2.16, i.e. by equally distributing the elements of all training samples to the N states.12 A is initialized to ⎞ ⎛ 0.3 0.3 0.3 0 0 · · · 0 ⎜ 0 0.3 0.3 0.3 0 · · · 0 ⎟ ⎟ ⎜ ⎜ .. .. ⎟ .. ⎜ . . . ⎟ (2.87) A=⎜ ⎟ ⎜ 0 ··· 0 0.3 0.3 0.3 ⎟ ⎟ ⎜ ⎝ 0 ··· 0 0 0.5 0.5 ⎠ 0 ··· 0 0 0 1 We now apply the Viterbi algorithm described above to find, for each training sample Om = (o1,m o2,m . . . oTm ,m ), the optimal state sequence q∗m = (q1,m q2,m . . . qT,m ). Each q∗m constitutes a mapping of observations to states, and the set of all q∗m , m = 1, 2, . . . , M , is used to update the model’s distributions B and transition probabilities A accordingly: Mean μi and variance σi2 for each state si are recomputed from all sample values ot,m for which qt,m = i holds. The transition probabilities ai,j are set to the number of times that the subsequence {. . . , qi , qj , . . . } is contained in the set of all q∗m and normalized to fulfill (2.68). Formally, μi = E{ot,m |qt,m = i} σi2 = var{ot,m |qt,m = i} a ˜i,j ai,j = ˜i,j ja
(2.88) (2.89)
a ˜i,j = |{qt,m |qt = i ∧ qt+1 = j}|
(2.91)
(2.90)
with
This process is called segmental k-means or Viterbi training. Each iteration improves the model’s match with the training data, i.e. P (O, q∗ |λ(n) ) ≤ P (O, q∗ |λ(n−1) )
(2.92)
where λ(n) denotes the model at iteration n. 11 12
To avoid negative values, the score is sometimes defined as the negative logarithm of P . If N is not a divisor of Tm , an equal distribution is approximated.
Hand Gesture Commands
45
The training is continued on the same training samples until λ remains constant. This is the case when the optimal state sequences q∗m remain constant for all M samples.13 A proof of convergence can be found in [13]. It should be noted that the number of states N is not part of the parameter estimation process and has to be specified manually. Lower values for N reduce computational cost at the risk of inadequate generalization, while higher values may lead to overspecialization. Also, since the longest forward transition in the Bakis topology is two states per observation, N ≤ 2 min{T1 , T2 , . . . , TM } − 1
(2.93)
is required. Revisiting the example introduced at the beginning of this section, we will now create an HMM for x˙ cog from the three productions of “clockwise” presented in Fig. 2.16. To allow a direct comparison with the rigid state assignment shown in Fig. 2.17 the number of states N is set to 12 as well. Fig. 2.21 overlays the parameters of the resulting model’s distribution functions with the same three plots of x˙ cog .
Fig. 2.21. Overlay plot of the feature x˙ cog (t) for three productions of the gesture “clockwise”, and mean μ and variance σ of the 12 states of an HMM created therefrom. The gray bars are centered at μ and have a length of 2σ. Each represents a single state.
Alternatively, the training may also be aborted when the number of changes in all q∗m falls below a certain threshold. 13
46
Non-Intrusive Acquisition of Human Action
Compared to Fig. 2.16, the mean values μ now represent the shape of the plots significantly better. This is especially noticeable for 10 ≤ t ≤ 30, where the differences in temporal offset previously led to μ having a much smaller amplitude than x˙ cog . Accordingly, the variances σ are considerably lower. This demonstrates the effectiveness of the approach. Multidimensional Observations So far we have assumed scalar observations ot (see (2.66)). In practice, however, we will rarely encounter this special case. The above concepts can easily be extended to multidimensional observations, removing the initial restriction of K = 1 in (2.65). (2.75) becomes bi = N (o, μi , Ci )
(2.94)
where μi and Ci are mean and covariance matrix of state si estimated in the Viterbi training: μi = E{ot,m |qt,m = i} Ci = cov{ot,m |qt,m = i}
(2.95) (2.96)
These equations replace (2.88) and (2.89). In practical applications the elements of o are frequently assumed uncorrelated so that C becomes a diagonal matrix, which simplifies computation. Extensions A multitude of extensions and modifications exist to improve the performance of the simple HMM variant presented here, two of which shall be briefly mentioned here. Firstly, the distributions b may be chosen differently. This includes other types of distributions, such as Laplace, and multimodal models.14 The latter modification brings about a considerable increase in complexity. Furthermore, in the case of multidimensional observations one may want the temporal alignment performed by the HMM to be independent for certain parts of the feature vector. This can be achieved by simply using multiple HMMs in parallel, one for each set of temporally coupled features. For instance, a feature vector in a gesture recognition system typically consist of features that describe the hand’s position, and others that characterize its posture. Using a separate HMM for each set of features facilitates independent temporal alignment (see Sect. 3.3.4 for a discussion of such Parallel Hidden Markov Models (PaHMMs)). Whether this is actually desirable depends on the concrete application.
14
For multimodal models, the considerations stated in Sect. 2.1.3.5 apply.
Hand Gesture Commands
47
2.1.3.7 Example Based on the features of the static and the dynamic vocabulary computed in Sect. 2.1.2.4, this section describes the selection of suitable features to be used in the two example applications. We will manually analyze the values in Tab. 2.5 and the plots in 2.14 to identify features with high inter-gesture variance. Since all values and plots describe only a single gesture, they first have to be computed for a number of productions to get an idea of the intra-gesture variance. For brevity, this process is not described explicitly. The choice of classification algorithm is straightforward for both the static and the dynamic case. Static Gesture Recognition The data in Tab. 2.5 shows that a characteristic feature of the “stop” gesture (Fig. 2.2a) is its low compactness: c is smaller than for any other shape. This was already mentioned in Sect. 2.1.3.4, and found to be true for any production of “stop” by any user. Pointing gestures result in an roughly circular, compact shape with a single protrusion from the pointing finger. For the “left” and “right” gestures (Fig. 2.2b,c), this protrusion is along the x axis. Assuming that the protruding area does not significantly affect xcog , the protrusion ratio xp introduced in (2.30) can be used: Shapes with xp 1 are pointing to the right, while x1p 1 indicates pointing to the left. Gestures for which xp ≈ 1 are not pointing along the x axis. This holds regardless of whether the thumb or the index finger is used for pointing. Empirically, xp > 1.2 and x1p > 1.2 was found for pointing right and left, respectively. Since the hand may not be the only skin-colored object in the image (see Tab. 2.4), the segmentation might yield multiple skin colored regions. In our application scenario it is safe to assume that the hand is the largest object; however, it need not always be visible. The recording conditions in Tab. 2.4 demand a minimum shape size amin of 10% of the image size, or amin = 0.1N M
(2.97)
Therefore, all shapes with a < amin may be discarded, and from the remainder, we simply choose the one with the largest area a. If no shape exceeds the minimum area, the image is considered empty. The feature vector for static gesture recognition is therefore ⎛ ⎞ a O=⎝ c ⎠ (2.98) xp The above observations regarding a, c, and xp suggest to use a set of rules for classification. The required thresholds can easily be identified since different gestures (including typical “idle” gestures) scarcely overlap in the feature space. A suitable threshold Θc for identifying “stop” was already found in (2.43).
48
Non-Intrusive Acquisition of Human Action Rephrasing the above paragraphs as rules yields 1. Discard all objects with a < 0.1N M. 2. If no objects remain then the image is empty.
(2.99) (2.100)
3. Consider the object for which a is maximum. 4. If c < 0.25 then the observed gesture is “stop”.
(2.101) (2.102)
5. If xp > 1.2 then the observed gesture is “right”.
(2.103)
6. If
1 xp
> 1.2 then the observed gesture is “left”.
7. Otherwise the hand is idle and does not perform any gesture.
(2.104) (2.105)
These rules conclude the design of the static gesture recognition example application. Sect. 2.1.4 presents the corresponding I MPRESARIO process graph. Dynamic Gesture Recognition The feature plots in Fig. 2.14 show that normalized area a , compactness c, and normalized x coordinate of the center of gravity xcog should be sufficient to discriminate among the six dynamic gestures. “clockwise” and “counterclockwise” can be identified by their characteristic sinusoidal xcog feature. “open” and “grab” both exhibit a drop in a around T2 but can be distinguished by compactness, which simultaneously drops for “open” but rises for “grab”. “close” and “drop” are discerned correspondingly. We thus arrive at a feature vector of ⎛ ⎞ a O=⎝ c ⎠ (2.106) xcog An HMM-based classifier is the obvious choice for processing a time series of such feature vectors. The resulting example application and its I MPRESARIO process graph is presented in Sect. 2.1.5. 2.1.4 Static Gesture Recognition Application To run the example application for static gesture recognition, please install the rapid prototyping system I MPRESARIO from the accompanying CD as described in Sect. B.1. After starting I MPRESARIO, choose “File”→“Open...” to load the process graph named StaticGestureClassifier.ipg (see Fig. 2.22). The data flow is straightforward and corresponds to the structure of the sections in this chapter. All six macros are described below. Video Capture Device Acquires a stream of images from a camera connected e.g. via USB. In the following, each image is processed individually and independently. Other sources may be used by replacing this macro with the appropriate source macro (for instance, Image Sequence (see Sect. B.5.1) for a set of image files on disk).
Hand Gesture Commands
49
Fig. 2.22. The process graph of the example application for static gesture recognition (file StaticGestureClassifier.ipg). The image shows the classification result.
ProbabilityMap Computes a skin probability image as described in Sect. 2.1.2.1. The parameters Object color histogram file and Non-object color histogram file point to the files skin-32-32-32.hist and nonskin-32-32-32.hist (these are the histograms presented in [12], with a resolution of 32 bins per dimension as described in (2.13)). All parameters should be set to their default values. GaussFilter Smoothes the skin probability image to prevent jagged borders (see Sect. 2.1.2.2). The parameter Kernel size should be set to around 15 for a resolution of 320×240. Higher resolutions may require larger kernels. Setting Variance to zero computes a suitable Gauss kernel variance automatically. ConvertChannel The output of the preceding stage is a floating point gray value image I(x, y) ∈ [0, 1]. Since the next macro requires a fixed point gray value image I(x, y) ∈ {0, 1, . . . , 255}, a conversion is performed here. ObjectsFromMask Extracts contiguous regions in the skin probability image using an algorithm similar to that presented in Fig. 2.11. This macro outputs a list of all detected regions sorted descending by area (which may be empty if no regions were found). A value of zero for the parameter Threshold uses the algorithm shown in Fig. 2.6 for automatic threshold computation. If this fails, the threshold
50
Non-Intrusive Acquisition of Human Action
may have to be set manually. The remaining parameters control further aspects of the segmentation process and should be left at their defaults. StaticGestureClassifier Receives the list of regions from the preceding stage and discards all but the largest one. The macro then computes the feature vector described in (2.98), using the equations discussed in Sect. 2.1.2.3, and performs the actual classification according to the rules derived in Sect. 2.1.3.7 (see (2.99) to (2.105)). The thresholds used in the rules can be changed to modify the system’s behavior. For the visualization of the classification result, the macro also receives the original input image from Video Capture Device (if the output image is not shown, double-click on the macro’s red output port). It displays the border of the hand and a symbol indicating the classification result (“left”, “right”, or “stop”). If the hand is idle or not visible, the word IDLE is shown. Press the “Play” button or the F1 key to start the example. By double-clicking on the output port of GaussFilter the smoothed skin probability image can be displayed. The hand should be clearly distinguishable from the background. Make sure that the camera’s white balance is set correctly (otherwise the skin color detection will fail), and that the image is never overexposed, but rather slightly underexposed. If required, change the device properties as described in Sect. B.5.2. The application recognizes the three example gestures “left”, “right”, and “stop” shown in Fig. 2.2. If the classification result is always IDLE or behaves erratic, see Sect. 2.1.6 for troubleshooting help. 2.1.5 Dynamic Gesture Recognition Application The process graph of the example application for dynamic gesture recognition is stored in the file DynamicGestureClassifier.ipg (see Fig. 2.23). It is identical to the previous example (cf. Fig. 2.22), except for the last macro DynamicGestureClassifier. From the list of regions received from ObjectsFromMask, this macro considers only the largest one (just as StaticGestureClassifier) and computes the feature vector shown in (2.106). The feature vectors of subsequent frames are buffered, and when the gesture is finished, the resulting feature vector sequence is forwarded to an HMM-based classifier integrated in the macro. The following explains the parameters and operation of DynamicGestureClassifier. When starting the example, the macro is in “idle” state: It performs a segmentation and visualizes the resulting hand border in the color specified by Visualization Color but does not compute feature vectors. This allows you to position the camera, set the white balance etc. Setting the State parameter to “running” activates the recording and recognition process. Whenever the macro’s state is switched from “idle” to “running” it enters a preparation phase in which no features are computed yet, in order to give you some time to position your hand (which is probably on the mouse at that point) in the image. During this period a green progress bar is shown at the bottom of the image so that you know when the preparation phase is over and the actual recording starts. The duration of the preparation phase (in frames) is specified by the parameter Delay Samples.
Hand Gesture Commands
51
Fig. 2.23. Process graph of the example application for dynamic gesture recognition (file DynamicGestureClassifier.ipg). The image was recorded with a webcam mounted atop the screen facing down upon the keyboard.
After the preparation phase, the recording phase starts automatically. Its duration can be configured by Gesture Samples. A red progress bar is shown during recording. The complete gesture must be made while this progress bar is running. The hand motion should be smooth and take up the entire recording phase. The parameter Hidden Markov Model specifies a file containing suitable Hidden Markov Models.15 You can either use gestures.hmm, which is provided on the CD and contains HMMs for the six example gestures shown in Fig. 2.3, or generate your own HMMs using the external tool generateHMM.exe as described below. Since the HMMs are identified by numerical IDs (0, 1, . . . ), the parameter Gesture Names allows to set names for each model that are used when reporting classification results. The names are specified as a list of space-separated words, where the first word corresponds to ID 0, the second to ID 1, etc. Since the HMMs in gestures.hmm are ordered alphabetically this parameter defaults to “clockwise close counterclockwise drop grab open”.
15
The file actually contains an instance of lti::hmmClassifier, which includes instances of lti::hiddenMarkovModel.
52
Non-Intrusive Acquisition of Human Action
The parameter Last Classified Gesture is actually not a parameter, but is set by the macro itself to reflect the last classification result (which is also dumped to the output pane). Start the example application by pressing the “Play” button or the F1 key. Adjust white balance, shutter speed etc. as for StaticGestureRecognition (Sect. 2.1.4). Additionally, make sure that the camera is set to the same frame rate used when recording the training samples from which the HMMs were created (25 fps for gestures.hmm). When the segmentation of the hand is accurate, set State to “running” to start the example. Perform one of the six example gestures (Fig. 2.3) while the red progress bar is showing and check Last Classified Gesture for the result. If the example does not work as expected, see Sect. 2.1.6 for help. Test/Training Samples Provided on CD The subdirectory imageinput\Dynamic_Gestures in your I MPRESARIO installation contains three productions of each of the six example gestures. From these samples, gestures.hmm was created (this process is described in detail below). For testing purposes you can use them as input to DynamicGestureClassifier. However, conclusions regarding classification performance cannot be made by feeding training data back to the system, but only by testing with previously unseen input. The process graph DynamicGestureClassifierFromDisk.ipg shown in Fig. 2.24 uses Image Sequence as the source macro. It allows to select one of six image sequences from a drop-down list, where each sequence corresponds to one production of the gesture of the same name. A detailed description of Image Sequence is provided in B.5.1. In contrast to acquiring images from a camera, reading them from disk does not require a preparation phase, so the parameter Delay Samples of DynamicGestureClassifier is set to 0. Gesture Samples should exactly match the number of files to read (60 for the example gestures). Make sure that State is set to “running” every time before starting the example, otherwise one or more images would be ignored. Generation of Hidden Markov Models The command line utility generateHMM.exe creates Hidden Markov Models from a set of training samples and stores them in an .hmm file16 for use in DynamicGestureClassifier. These training samples may either be the above described samples provided on the CD, your own productions of the example gestures, or any other gestures that can be classified based on the feature vector introduced in (2.106). (To use a different feature vector, modify the class DynamicGR accordingly and recompile both the affected I MPRESARIO files and generateHMM.) The training samples must be arranged on disk as shown in Fig. 2.25. The macro RecordGestures facilitates easy recording of gestures and creates this directory 16
As above, this file actually contains a complete lti::hmmClassifier.
Hand Gesture Commands
53
Fig. 2.24. Recognition of dynamic gestures read from disk frame by frame (process graph file DynamicGestureClassifierFromDisk.ipg). Each frame corresponds to a single file.
Fig. 2.25. Directory layout for training samples to be read by the generateHMM.exe command line utility.
structure automatically. Its process graph is presented in Fig. 2.26. Like DynamicGestureRecognition, it has an “idle” and a “running” state, and precedes the ac-
54
Non-Intrusive Acquisition of Human Action
tual recording with a preparation phase. The parameters State, Delay Samples, and Gesture Samples work as described above.
Fig. 2.26. Process graph RecordGestures.ipg for recording gestures to be processed by generateHMM.exe.
To record a set of training gestures, first set the parameter Destination Directory to specify where the above described directory structure shall be created. Ensure that Save Images to Disk is activated and State is set to “idle”, then press “Play” or F1. For each gesture you wish to include in the vocabulary, perform the following steps: 1. Set the parameter Gesture Name (since the name is used as a directory name, it should not contain special characters). 2. Set Sequence Index to 0. 3. Set State to “running”, position your hand appropriately in the preparation phase (green progress bar) and perform the gesture during the recording phase (red progress bar). After recording has finished, Sequence Index is automatically increased by 1, and State is reset to “idle”. 4. Repeat step 3 several times to record multiple productions of the gesture. 5. Examine the results using an image viewer such as the free IrfanView17 or the macro Image Sequence (see Sect. B.5.1). If a production was incorrect or oth17
http://www.irfanview.com/
Hand Gesture Commands
55
erwise not representative, set Sequence Index to its index and repeat step 3 to overwrite it. Each time step 3 is performed, a directory with the name of Sequence Index is created in a subdirectory of Destination Directory named according to Gesture Name, containing as many image files (frames) as specified in Gesture Samples. This results in a structure as shown in Fig. 2.25. All images are stored in BMP format. The files created by RecordGestures can be used for both training and testing. For instance, in order to observe the effects of changes to a certain parameter while keeping the input data constant, replace the source macro Video Capture Device in the process graphs StaticGestureRecognition.ipg or DynamicGestureRecognition.ipg with Image Sequence and use the recorded gestures as image source. To run generateHMM.exe on the recorded gestures, change to the directory where it resides (the I MPRESARIO data directory), ensure that the files skin-32-32-32.hist and non-skin-32-32-32.hist are also present there, and type
generateHMM.exe -d
-e <extension> -f where the options -d, -e, and -f specify the input directory, the input file extension, and the output filename, respectively. For use with Record Gestures, replace with the directory specified as Destination Directory, <extension> with bmp18 , and with the name of the .hmm file to be created. The directories are then scanned and each presentation is processed (this may take several seconds). The resulting .hmm file can now be used in DynamicGestureClassifier. To control the segmentation process, additional options as listed in Tab. 2.6 may be used. It is important that these values match the ones of the corresponding parameters of the ObjectsFromMask macro in DynamicGestureClassifier.ipg to ensure that the process of feature computation is consistent. The actual HMM creation can be configured as shown in Tab. 2.7. Table 2.6. Options to control the skin color segmentation in generateHMM.exe. -k -t
Size of the Gauss kernel used for smoothing the skin probability images. Segmentation threshold Θ.
Calling generateHMM.exe with no arguments shows a short help as well as the default values and allowed ranges for all options. For the provided file gestures.hmm the option -s 15 was used, and all other parameters were left at their defaults.
18
For the samples provided on CD, specify jpg instead.
56
Non-Intrusive Acquisition of Human Action Table 2.7. Options to control the HMM creation in generateHMM.exe.
-c -s -m, -M
Type of the distribution functions B to be used. Number of states N . Minimum and maximum number of states to jump forward per observation. These values specify a range, so -m 0 -M 2 allows jumps of zero, one, or two states (Bakis topology).
2.1.6 Troubleshooting In case the classification performance of either example is poor, verify that the recording conditions in Tab. 2.4 are met. Wooden furniture may cause problems for the skin color segmentation, so none should be visible in the image. Point the camera to a white wall to be on the safe side. If the segmentation fails (i.e. the hand’s border is inaccurate) even though no other objects are visible, recheck white balance, shutter speed, and gain/brightness. The camera may perform an automatic calibration that yields results unsuitable for the skin color histogram. Disable this feature and change the corresponding device parameters manually. The hand should be clearly visible in the output of the GaussFilter macro (double-click on its red output port if no output image is shown).
2.2 Facial Expression Commands The human face is, besides speech and gesture, a primary channel of information exchange during natural communication. By facial expressions emotions can be expressed faster, more subtle, and more efficiently than with words [3]. Facial expressions are used actively by a speaker to support communication, but also intuitively for mimic feedback by a dialog partner. In medicine, symptoms and progress in therapy may be read from the face. Sign languages make use of facial expressions as additional communication channel supplementing syntax and semantics of a conversation. Babies focus already after few minutes after birth on faces and differentiate hereby between persons. While the recognition of facial expressions is easy for humans, it is extremely difficult for computers. Many attempts are documented in the literature which mostly concentrate on subparts of the face [17]. Eye blinks are, e.g., applied in the automotive industry to estimate driver fatigue, lip outline and motion contribute to the robustness of speech recognizers, and face recognition is used as a biometric method for identification [4]. Parallel registration of facial features currently takes place only in “motion capturing” for the realistic animation of computer-generated characters. Here tiniest movements in the face are recorded with intricate optical measurement devices. Fig. 2.27 shows a selection of current hardware for facial expression recording. The use of non-intrusive video-based registration techniques poses serious problems due to possible variations in user appearance, which may have intrinsic and ex-
Facial Expression Commands
57
Fig. 2.27. Hardware for facial expression recording. Sometimes the reflections of Styrofoam balls are exploited (as in a. and b.) Other solutions use a head cam together with markers on the face (c) or a mechanical devise (d). In medical diagnostics line of sight is measured with piezo elements or with cameras fixed to the spectacles to measure skin tension.
trinsic causes. Intrinsic variations result from individual face shape and from movements while speaking, extrinsic effects result from camera position, illumination, and possibly occlusions. Here we propose a generic four stage facial feature analysis system that can cope with such influence factors. The processing chain starts with video stream acquisition and continues over image preprocessing and feature extraction to classification. During image acquisition, a camera image is recorded while camera adjustments are optimized continuously. During preprocessing, the face is localized and its appearance is improved in various respects. Subsequently feature extraction results in
58
Non-Intrusive Acquisition of Human Action
Fig. 2.28. Information processing stages of the facial expression recognition system.
the segmentation of characteristic facial regions among which significant attributes are identified. A final classification stage assigns features to predefined facial expressions. An overview of the complete system is depicted in Fig. 2.28. The single processing stages are described in more detail in the following sections.
Facial Expression Commands
59
2.2.1 Image Acquisition During image acquisition, the constancy of color and luminance even under variable illumination conditions must be guaranteed. Commercial web cams which are used here, usually permit the adjustment of various parameters. Mostly there exists a feature for partial or full self configuration. The hardware implemented algorithms take care of white balance, shutter speed, and back light compensation and yield, to the average, satisfactory results. Camera parameter adjustment can however be further optimized if context knowledge is available, as e.g. shadows from which the position of a light source can be estimated, or the color of a facial region. Since the factors lighting and reflection cannot be affected due to the objective, only the adjustment of the camera parameters remains. The most important parameters of a digital camera for the optimization of the representation quality are the shutter speed, the white-balance, and the gain. In order to optimize these factors in parallel, the extended simplex algorithm is utilized, similar to the approach given in [16]. With a simplex in n-dimensional space (here n=3 for the three parameters) it acts around a geometrical object with n+1 position vectors which possess an identical distance to each other in each case. The position vector is defined as follows: ⎛ ⎞ s s = shutter w = white − balance (2.107) P = ⎝w⎠ g g = gain P is a scalar value that the number of skin-colored pixel in the face region represents, dependent on the three parameters s, w and g. The worst (smallest) vector is mirrored at the line, that by the remaining vectors is given and results in a new simplex. The simplex algorithm tries to iteratively find an optimal position vector which comes the desired targets next. In the available case it applies to approximate the average color of the face region with an empirically determined reference value. For the optimal adjustment of the parameters, however, a suitable initialization is necessary which can be achieved for example by a pre-use of the automatic whitebalance of the camera. Fig. 2.29 (left) shows an example of the two-dimensional case. To the position vector two additional position vectors U and V are added. Since |P| produces too few skin-colored areas, it is mirrored at the straight line UV on P . It results the new simplex S . The procedure is repeated until it converges towards a local maximum Fig. 2.29 (right). A disadvantage of the simplex algorithm is its rigid gradual and linear performance. Therefore the approach was extended, as an upsetting and/or an aspect ratio was made possible for the mirroring by introduction of a set of rules. Fig. 2.30 represents the extended simplex algorithm for a two-dimensional parameter space. The rules are as follows:
60
Non-Intrusive Acquisition of Human Action
Fig. 2.29. Simple simplex similar to the approach described in [16].
1. If P > |U| ∧ P > |V| then repeat the mirroring with stretching factor 2 from P to P2 2. If P > |U| ∧ P < |V| ∨ P < |U| ∧ P < |V| then use P 3. If P < |U| ∧ P < |V| ∧ P > |P| then repeat mirroring with stretching factor 0.5 from P to P0,5 4. If P < |U| ∧ P < |V| ∧ P < |P| then repeat mirroring with stretching factor -0.5 from P to P−0,5
Fig. 2.30. Advanced simplex with dynamic stretching factor.
2.2.2 Image Preprocessing The goal of image preprocessing is the robust localization of the facial region which corresponds to the rectangle bounded by bottom lip and brows. With regard to processing speed only skin colored regions are analyzed in an image. Additionally a general skin color model is adapted to each individual. Face finding relies on a holistic approach which takes the orientation of facial regions into consideration. Brows, e.g., are characterized by vertically alternating
Facial Expression Commands
61
bright and dark horizontal regions. A speed-up of processing is achieved by a socalled integral-image approach and an Ada-Boosting classification. Variations in head pose and occlusion by, e.g., hands are compensated by tracing characteristic face regions in parallel. Furthermore, a reduction of shadow and glare effects is performed as soon as the face has been found [5]. 2.2.2.1 Face Localization Face localization is simplified by exploiting a-priori knowledge, either with respect to the whole face or to parts of it. Analytical or feature-based approaches make use of local features like edges, intensity, color, movement, contours and symmetry apart or in combination, to localize facial regions. Holistic approaches consider regions as a whole. The approach described here is holistic by finding bright and dark facial regions and their geometrical relations. A search mask is devised to find skin colored regions with suitable movement patterns only. For the exploitation of skin color information the approach is utilized which is already described in Sect. 2.1.2.1. A movement analysis makes use of the fact that neighboring pixels of contiguous regions move in similar fashion. There are several options how to calculate optical flow. Here a Motion History Image (MHI), i.e. the weighted sum of subsequent binary difference images is used, where a motion image is defined as: 1.0 if || I(x, y, tk ) − I(x, y, tk−1 )||2 > Θ Imotion (x, y, tk ) = max(0.0, IB (x, y, tk−1 ) − T1 otherwise
(2.108)
Imotion is incremented, if the intensity difference of a pixel in two following images is, according to the L2-norm, larger than a given threshold Θ. Otherwise the contribution of elapsed differences will be linearly decreased by the constant T . Fig. 2.31 shows the result of this computation for head and background movements. The intensity of the visualized movement pattern indicates contiguous regions.
Fig. 2.31. Visualized movement patterns for movement of head and background (left), movement of the background alone (middle), and movement of the head alone (right).
62
Non-Intrusive Acquisition of Human Action
For a combined skin color and movement search the largest skin colored object is selected and subsequently limited by contiguous, non skin colored regions (Fig. 2.32).
Fig. 2.32. Search mask composed of skin color (left) and motion filter (right).
Holistic face finding makes use of three separate modules. The first transforms the input image in an integral image for the efficient calculation of features. The second module supports the automatic selection of suitable features which describe variations within a face, using Ada-Boosting. The third module is a cascade classifier that sorts out insignificant regions and analyzes in detail the remaining regions. All three modules are subsequently described in more detail. Integral Image For finding features as easily as possible, Haar-features are used which are presented by Papgeorgiou et al. [18]. A Haar-feature results from calculating the luminance difference of equally sized neighboring regions. These combine to the five constellations depicted in Fig. 2.33, where the feature M is determined by: M (x1 , y1 , x2 , y2 ) = abs(Intensity
− Intensity )
(2.109)
Fig. 2.33. The five feature constellations used.
With these features, regions showing maximum luminance difference can be found. E.g., the brow region mostly shows different intensity from the region above the brow. Hence, the luminance difference is large and the resulting feature is rated significant. However, in a 24x24 pixel region there are already 45396 options to form the constellations depicted in Fig. 2.33. This enormous number results from the fact that
Facial Expression Commands
63
the side ratio of the regions may vary. To speed-up search in such a large search space an integral image is introduced which provides the luminance integral in an economic way. The integral image Iint at the coordinate (x, y) is defined as the sum over all intensities I(x, y): Iint = I(x , y ) (2.110) x ≤x,y ≤y,
By introducing the cumulative column sum Srow (x, y) two recursive calculations can be performed: Srow (x, y) = Srow (x, y − 1) + Iint (x, y)
(2.111)
Iint (x, y) = Iint (x − 1, y) + Srow (x, y)
(2.112)
The application of these two equations enables the computation of a complete integral image by iterating just once through the original image. The integral image has to be calculated only once. Subsequently, the intensity integrals Fi of any rectangle are determined by executing a simple addition and two subtractions.
Fig. 2.34. Calculation of the luminance of a region with the integral image. The luminance of region FD is I(xD , yD ) + I(xA , yA ) − I(xB , yB ) − I(xD , yD ).
In Fig. 2.34 we find:
64
Non-Intrusive Acquisition of Human Action FA =Iint (xA , yA )
(2.113)
FB =Iint (xB , yB ) − Iint (xA , yA ) FC =Iint (xC , yC ) − Iint (xA , yA )
(2.114) (2.115)
FD =Iint (xD , yD ) + Iint (xA , yA ) − Iint (xB , yB ) − Iint (xC , yC ) (2.116) (2.117) FE =Iint (xE , yE ) − Iint (xC , yC ) FF =Iint (xF , yF ) + Iint (xC , yC ) − Iint (xD , yD ) − Iint (xE , yE ) (2.118) (2.119) FF − FD =Iint (xF , yF ) + Iint (xA , yA ) − Iint (xB , yB ) − Iint (xE , yE ) + 2Iint (xC , yC ) − 2Iint (xC , yC ) The approach is further exemplified when regarding Fig. 2.35. The dark/gray constellation is the target feature. In a first step the integral image is constructed. This, in a second step, enables the economic calculation of a difference image. Each entry in this image represents the luminance difference of both regions with the upper left corner being the reference point. In the example there is a maximum at position (2, 3).
Fig. 2.35. Localizing a single feature in an integral image, here consisting of the luminance difference of the dark and gray region.
Ada-Boosting Now the n most significant features are selected with the Ada-Boosting-algorithm, a special kind of ensemble learning. The basic idea here is to construct a diverse ensemble of “weak” learners and combine these to a “strong” classifier.
Facial Expression Commands
65
Starting point is a learning algorithm A, that generates a classifier h from training data N , where: A:D→h
(2.120)
N = (xi , yi ) ∈ X × Y i = 1 . . . m h:X→Y
(2.121) (2.122)
X is the set of all features; Y is the perfect classification result. The number of training examples is m. Here A was selected to be a perceptron: 1 if pj fj (x) < pj Θj (2.123) hj (x) = 0 otherwise The perceptron hj (x) depends on the feature fj (x) and a given threshold Θj whereby pj serves for the correct interpretation of the inequality. During search for a weak classifier a number of variants Dt with t = 1, 2, . . . K is generated from the training data D. This results in an ensemble of classifiers ht = A(Dt ). The global classifier H(x) is then implemented as weighted sum: H(x) =
K
αk hk (x)
(2.124)
k=1
Ada-Boosting iteratively generates classifiers. During iteration each training example is weighted according to its contribution to the classification error. Since the weight of “difficult” examples is steadily increased, the training of later generated classifiers concentrates on the remaining problematic cases. In the following the algorithm for the selection of suitable features by Ada-Boosting is presented. Step 1: Initialization • • • •
let m1 . . . mn be the set of training images let bi be a binary variable, with: bi = 0, 1∀mi ∈ Mfaces , mi ∈ / Mfaces let m be the number of all faces and l the number of non-faces 1 1 , 2l ∀bi = 0, 1 w1,i are initialization weights with: w1,i = 2m Step 2: Iterative search for the T most significant features For t = 1, . . . , T
• • • •
w
Normalize the weights wt,i = n t,iwt,j such that wt forms a probability distrij=1 bution For each feature j train a classifier hj . The error depending on wt is: j = i wi · |hj (xi − yi | Select the classifier ht with the smallest error t Actualize the weights wt+1,i = wt,i βt1−ei , where ei = 0, 1 for correct, uncorrect t classification and βt = 1− t Step 3: Estimation of the Resulting-Classifier
66 •
Non-Intrusive Acquisition of Human Action The resulting classifier finally is: h(x) =
1 0
for Tt=1 αt ht (x) ≥ otherwise
1 2
T t=1
αt with αt = log β1t
(2.125)
Cascaded Classifier This type of classifier is applied to reduce classification errors and simultaneously shorten processing time. To this end small efficient classifiers are built, that reject numerous parts of a image as non-face (false positives), without incorrectly discarding faces (false negatives). Subsequently more complex classifiers are applied to further reduce the false positives-rate. A simple classifier for a two-region feature hereby reaches, e.g., in a first step a false-positive rate of only 40 % while at the same time recognizing 100 % of the faces correctly. The total system is equivalent to a truncated decision tree, with several classifiers arranged in layers called cascades (Fig. 2.36).
Fig. 2.36. Schematic diagram of the Cascaded Classifier: Several Classifiers in line are applied for each image-region. Each classifier can reject that region. This approach strongly reduces computational and at the same provides very accurate results.
The false-positive rate F of a cascade is: F =
K
fi
(2.126)
i=1
where fi is the false-positive rate of one of the K layers. The recognition rate D is respectively: D=
K
di
(2.127)
i=1
The final goal for each layer is to compose a classifier from several features, until a required maximum recognition rate D is reached and simultaneously the number F of misclassifications is minimal. This is achieved by the following algorithm:
Facial Expression Commands
67
Step 1: Initialization • • • •
The user defines f and FTarget Let Mfaces be the set of all faces Let M¬faces be the set of all non-faces F0 = 1.0; D0 = 1.0; i = 0
Step 2: Iterative Determination of all Classifiers of a Layer • •
•
i = i + 1; ni = 0; Fi = Fi−1 while Fi > FTarget – ni = ni+1 – Use Mfaces and M¬faces to generate with Ada-Boosting a classifier with ni features. – Test the classifier of the actual layer on a separate test set to determine Fi and Di . – Reduce the threshold of the i-th classifier, until the classifier of the actual layer reaches a recognition rate of d × Di−1 . If Fi > FTarget , test the classifier of the actual layer on a set of non-faces and add each misclassified object to N .
2.2.2.2 Face Tracking For side view images of faces the described localization procedure yields only uncertain results. Therefore an additional module for tracking suitable points in the facial area is applied, using the algorithm of Tomasi and Kanade [26]. The tracking of point pairs in subsequent images can be partitioned in two subtasks. First suitable features are extracted which are then tracked in the following frames of a sequence. In case features are lost, new ones must be identified. If in subsequent images only small motions of the features to be tracked occur, the intensity of immediately following frames is: I (x, y) = I(x + ζ, y + η)
(2.128)
Here d := (ζ, η) describes the shift of point (x, y). Now the assumption holds that points whose intensity varies strongly in either vertical or horizontal direction in the first frame show similar characteristics in the second frame. Therefore, a feature is described by a small rectangular region w with center (x, y) that shows a large variance with respect to its pixel intensities. The region W is called feature window. The intensity variance is determined by the gradient g = (gx gy ) at the position (x, y): 2 gx gx gy (2.129) G= gx gy gy2 In case matrix G has large eigenvalues λ1 and λ2 , the region establishes a good feature with sufficient variance. Consequently those windows w are selected, for which the eigenvalue
68
Non-Intrusive Acquisition of Human Action
λ = min(λ1 , λ2 )
(2.130)
is highest. In addition, further restrictions like the minimal distance between features or the minimal distance from image border determine the selection. For the selected features, the shift in the next frame is determined based on translation models. If the intensity function is approximated linearly by I(x + ζ, y + η) = I(x, y) + g T d
(2.131)
then the following equations have to be solved: G·d =e with e = (I(x, y) − I (x, y))g T · ω dW
(2.132) (2.133)
w
Here the feature region W is integrated, with ω being a weighting function. Since neither the translation model nor the linear approximation is exact, the equations are solved iteratively and the admittable shift d is assigned an upper limit. In addition a motion model is applied which enables to define a measure of feature unsimiliarity by estimating the affine transform. In case this unsimiliarity gets too large it is substituted by a new one. 2.2.3 Feature Extraction with Active Appearance Models For the analysis of single facial features the face is modeled by dynamic face graphs. Most common are Elastic Bunch Graphs (EBG) [27] and Active Appearance Models (AAM), the latter are treated here. An active Appearance Model (AAM) is a statistic model combining textural and geometrical information into an appearance. The combination of outline and texture models are based on an eigenvalue approach which reduces the amount of data needed, hereby enabling real-time calculation. Outline Model The contour of an object is represented by a set of points placed at characteristic positions of the object. The point configuration must be stable with respect shift, rotation and scaling. Cootes et al. [6] describe how marking points, i.e. landmarks, are selected consistently on each image of the training set. Hence, the procedure is restricted to object classes with a common basic structure. In a training set of triangles, e.g., typically the corner points are selected since they describe the essential property of a triangle and allow easy distinction of differences in the various images of a set. In general, however, objects have more complicated outlines so that further points are needed to adequately represent the object in question as, e.g., the landmarks for a face depicted in Fig. 2.37. As soon as n points are selected in d dimensions, the outline (shape) is represented by a vector x which is composed of the single landmarks.
Facial Expression Commands
69
Fig. 2.37. 70 landmarks are used for the description of a face.
x = [x11 , . . . , x1n , x21 , . . . , x2n , . . . , xd1 , . . . , xdn ]T
(2.134)
If there are s examples in the training set, then s vectors x are generated. For statistical analysis images must be transformed to the same pose, i.e. the same position, scaling, and rotation. This is performed by a procrustes analysis which considers the outlines in a training set and minimizes the sum of distances with respect to the average outline (Fig. 2.38). For making adjustments, only outline-preserving operations must be applied in order to keep the set of points compact and to avoid unnecessary nonlinearities.
Fig. 2.38. Adjustment of a shape with respect to a reference shape by a procrustes analysis using affine transformation. Shape B is translated such that its center of gravity matches that of shape A. Subsequently B is rotated and scaled until the average distance of corresponding points is minimized (procrustes distance).
Now the s point sets are adjusted in a common coordinate system. They describe the distribution of the outline points of an object. This property is now used to construct new examples which are similar to the original training set. This means that a new example x can be generated from a parameterized model M in the form
70
Non-Intrusive Acquisition of Human Action x = M (p)
(2.135)
where p is the model parameter vector. By a suitable selection of p new examples are kept similar to the trainings set. The Principal Component Analysis (PCA) is a means for dimensionality reduction by first identifying the main axes of a cluster. Therefore it involves a mathematical procedure that transforms a number of possibly correlated parameters into a smaller number of uncorrelated parameters called principal components. The first principal component accounts for as much of the variability in the data as possible, and each succeeding component accounts for as much of the remaining variability as possible. For performing the PCA, first the mean vector x of the s trainings set vectors xi is determined by: 1 x s i=1 i s
x=
(2.136)
Let the outline points of the model be described by n parameter xi . Then the variance of a parameter xi describes how far the values x2ij deviate from the expectation value E(Xi ). The covariance Cov then describes the linear coherence of the various xi parameters: Cov(x1 , x2 , . . . , xn ) = E(x1 − E(x1 )) · E(x2 − E(x2 )) · . . . · E(xn − E(xn )) (2.137) The difference between each outline vector xi and the average xi is ∂xi . Arranging the differences as columns of a matrix yields: D = (∂x1 . . . ∂xs )
(2.138)
From this the covariance matrix follows 1 DDT (2.139) s from which the eigenvalues λi and Eigenvectors ϕi which form a orthogonal basis, are calculated. Each eigenvalue represents the variance with respect to the mean value in the direction of the corresponding Eigenvector. Let the deformation matrix Φk contain the t largest Eigenvectors of S, i.e. those influencing the deviation from the average value most: S=
Φk = (ϕ1 , ϕ2 , . . . ϕt )
(2.140)
Then these t vectors form a subspace basis which again is orthogonal. Each point in the training set can now be approximated by x ≈ x + Φk · pk
(2.141)
Facial Expression Commands
71
where pk corresponds to the model parameter vector mentioned above, now with a reduced number of parameters. The number t of Eigenvectors may be selected so that the model explains the total variance to a certain degree, e.g. 98%, or that each example in the training set is approximated to a desired accuracy. Thus an approximation for the n marked contour points with dimension d each is found for all s examples in the training set, as related to the deviations from the average outline. With this approximation, and considering the orthogonality of the eigenvalue matrix, the parameter vector pk results in pk = ΦT (x − x)
(2.142)
By varying the elements of pk the outline x may be varied as well. The average outline and exemplary landmark variances are depicted in (Fig. 2.39).
Fig. 2.39. Average outline and exemplary landmark variances.
The eigenvalue λi is the variance of the i-th parameter pki , over all examples in the training set. To make sure that a newly generated outline is similar to the training patterns, limits are set. Empirically it was√found that a maximum deviation for the parameter pki should be no more than ±3 λi (Fig. 2.40). Texture Model With the calculated mean outline and the principal components it is possible to reconstruct each example of the training data. This removes texture variances resulting from the various contours. Afterwards the RGB values of the normalized contour are extracted in a texture-vector tim . To minimize illumination effects, the texture vectors are normalized once more using a scaling factor and an offset a resulting in:
72
Non-Intrusive Acquisition of Human Action
Fig. √ models for variations of the first three eigenvalues φ1 , φ2 and φ3 between √ 2.40. Outline 3 λ, 0 and −3 λ.
(tim − β · 1) (2.143) α For the identification of t the average of all texture vectors has to be determined first. Additionally a scaling and an offset have to be estimated, because the sum of the elements must be zero and the variance should be 1. The values for α and β for normalizing tim are then t=
α = tim · tβ =
(tim · 1) n
(2.144)
where n is the number of elements in a vector. As soon as all texture vectors have been assigned scaling and offset, the average is calculated again. This procedure is iterated for the new average until the average converges to a stable value. From here the further processing is identical to that described in the outline section. Again a PCA is applied to normalized data which results in the following approximation for a texture vector in the training set t = t + Φt · pt
(2.145)
where Φt is again the Eigenvector matrix of the covariance matrix, now for texture vectors. The texture parameters are represented by pt .
Facial Expression Commands
73
For texture description here an artificial 3-band-image is used which is composed by the intensity channel K1 , the I3 channel K2 and the Y gradient K3 (Fig. 2.41). It boosts edges and color information, facilitating the separation of lips from the face. The color space with its three channels K1 , K2 and K3 is defined as: R+G+B 3 2G − R − B K2 = 4 2G − R − B K3 = 4 K1 =
(2.146) (2.147) (2.148)
Fig. 2.41. An artificial 3-band image (a) composed of intensity channel (b), I3 channel (c), and Y gradient (d).
2.2.3.1 Appearance Model The parameter vectors pk and pt permit to describe each example in the training set by outline and texture. To this end a concatenated vector pa is generated: Wk pk (2.149) pa = pt where Wk is a diagonal matrix of weights for each outline parameter. This measure is needed, since the elements of the outline parameter vectors pk are pixel distances while those of the texture parameter vectors are pixel intensities. Hence both can not be compared directly. The weights are identified by systematically varying the values of the outline parameter vector p and controlling the effect of this measure on texture. Since the Eigenvector matrix is orthogonal, its inverse is the transposed. Therefore: pk = ΦTk (x − x) pt =
ΦTt (t
− t)
(2.150) (2.151)
74
Non-Intrusive Acquisition of Human Action By insertion we then get: pa =
Wk ΦTk (x − x) ΦTt (t − t)
(2.152)
A further PCA is applied to exploit possible correlations between the outline and texture parameters for data compression. Since the parameter vectors pk and pt describe variances, their mean is the zero vector. The PCA then results in the following approximation for an arbitrary appearance vector from the training set: pa ≈ Φa c
(2.153)
Here Φa denotes the appearance Eigenvector matrix and c its parameter vector. The latter controls the parameters of outline as well as of texture. The linearity of the model now permits to apply linear algebra to formulate outline and texture as a function of c. Setting:
Φpk Φp = Φpt cpk c= cpt
(2.154) (2.155)
results in Φpk cpk Wk ΦTk x − Wk ΦTk x Wk ΦTk (x − x) = = P = ΦTt t − ΦTt t Φpt cpt ΦTt (t − t) Φpk cpk + Wk ΦTk x Wk ΦTk x = ⇔ ΦTt t Φpt cpt + ΦTt t Φk Wk−1 Φpk cpk + x x = ⇔ t Φt Φpt cpt + t
(2.156) (2.157) (2.158)
From this the following functions for the determination of an outline x and a texture t can be read: x = x + Φk Wk−1 Φpk cpk
(2.159)
t = t + Φt Φpt cpt
(2.160)
or, in shortened form x = x + Qk cpk
(2.161)
t = t + Qt cpt
(2.162)
Facial Expression Commands
75
with Qk = Φk Wk−1 Φpk Qt = Φt Φpt
(2.163) (2.164)
Using these equations an example model can be generated by • • •
defining a parameter vector c, generating the outline free texture image with vector t, deforming this by using the control points in x.
In Fig. 2.42√example √ appearance models for variations of the first five Eigenvectors between 3 λ, 0, −3 λ are presented. 2.2.3.2 AAM Search Image interpretation in general tries to recognize a known object in an a priori unknown image, even if the object is changed e.g. by a variation in camera position. When Active Appearance Models are used for image interpretation, the model parameters have to be identified such that the difference between the model and the object in the image is minimized. Image interpretation is thus transformed into an optimization problem. By performing a PCA the amount of data is considerably reduced. Nevertheless the optimization process is rather time consuming if the search-space is not limited. There are although possibilities to optimize the search-algorithm. Each trial to match the face-graph on an actual image proceeds similar, that means on changing one parameter of the configuration a specific pattern changes in the output-image. One can prove the existence of linear dependencies in this process that can be utilized for speeding-up the system. The general approach is as follows: The distance vector ∂I between a given image and the image generated by an AAM is: ∂I = Ii − Im
(2.165) 2
For minimizing this distance, the error Δ = |∂I| is considered. Optimization then concerns • •
Learning the relation between image difference ∂I and deviations in the model parameters. Minimizing the distance error Δ by applying the learned relation iteratively.
Experience shows that a linear relation between ∂I and ∂c is a good approximation: ∂c = A · ∂I
(2.166)
To identify A a multiple multivariate linear regression is performed on a selection of model variations and corresponding difference images ∂I. The selection could be
76
Non-Intrusive Acquisition of Human Action
Fig. for variations of the first five Eigenvectors c1 , c2 , c3 , c4 and c5 between √ √ 2.42. Appearance 3 λ, 0, −3 λ.
emphasized by adding carefully, randomly changing the model parameters for the given examples. For the determination of error Δ the reference points have to be identical. Therefore the image outlines to be compared are made identical and their texture is compared. In addition to variations in the model parameters, small deviations in pose, i.e. translation, scaling and orientation are introduced. These additional parameters are made part of the regression by simply adding them to the model parameter deviation vector ∂c. To maintain model linearity, pose
Facial Expression Commands
77
is varied only by scaling s, rotation Θ and translation in x-/y-direction (tx , ty ). The corresponding parameters are sx , sy , tx , ty and sx = s·cos(Θ)−1 and sy = s·sin(Θ) perform scaling and rotation in combination. To identify the relation between ∂c, ∂I, and ∂t the following steps are performed for each image in the training set: • • •
• • •
Initialize the starting vector c0 with the fixed appearance parameters c of the actually selected texture tim . Define arbitrary deviations for the parameters in |∂c| and compute new model parameters c = ∂c + c0 With this new parameter vector c, generate the corresponding target contour which is then used to normalize ti , such that its outline matches that of the modeled object. Then generate a texture image tm free of outline by applying c. Now the difference to the current image induced by ∂c can be calculated as ∂t = tn − tm Note ∂c and the corresponding ∂t and iterate several times for the same example image. Then iterate over all images in the training set.
The selected ∂c should be made as large as possible. Empirical evaluations have shown that the optimal region for the model parameters is 50% of the training set standard deviation, 10% scaling, and 2 pixel translation. For larger values the assumption of a linear relation is no longer warranted. The identified value pairs ∂c and ∂t are now submitted to a multiple multivariate regression to find A. In general a regression analysis the effect of influence variables on some target variable. Here, more specifically, the linear dependency of the single induced image differences of texture ∂t are examined which in turn depend on several selected parameter deviations ∂c. So A encodes a-priori knowledge which is gained during the iteration described above. By establishing the relation between model parameter deviations and their effect on generated images the first part of the optimization problem has been solved. What follows now in a second step is error minimization. This is achieved by iterations, whereby the linear model described earlier predicts for each iteration the change of model-parameters which lets the model converge to an optimum. Convergence is assumed as soon as the error reaches a given threshold. If the estimated model parameters c0 are initialized with those of the standard model, i.e. with the average values, once again the normalized image tn is involved. The algorithm proceeds as follows: • • • • •
Identify image difference ∂t0 = tn − tm and den error E0 = |∂t0 |2 Compute the estimated deviation ∂c = A · ∂t0 Set k = 1 and c1 = c0 − k∂c Generate the image to estimate the parameter vector c1 which results in a new tm . Execute for the given normalized image a further normalization, this time for the outline given by the new parameter-vector c1 .
78 • •
Non-Intrusive Acquisition of Human Action Compute the new error vector ∂t1 . Repeat the procedure until the error Ei converges. In case error Ei does not converge, retry with smaller values, e.g., 0.5, 0.25 etc. for k.
This results in a unique configuration of model parameters for describing an object, here the face, in a given image. Hence the object can be assigned to a particular object class, i.e. to the one for which the model was constructed. In case the configuration coincides with a normalized example of the trainingset, the object is not only localized but also identified. To give an example: If the algorithm for optimization determines a graph with a certain parameter-configuration very similar to parameters of a particular view in the training-set, then the actually localized face could match that of the trained person. 2.2.4 Feature Classification To localize areas of interest, a face graph is iteratively matched to the face using a modified active appearance model (AAM) which exploits geometrical and textural Information within a triangulated face. The AAM must be trained with data of many subjects. Facial features of interest which have to be identified in this processing stage, are head pose, line of sight, and lip outline. 2.2.4.1 Head Pose Estimation For the identification of the posture of the head two approaches are prosecuted in parallel. The first method is an analytical method based on the trapezoid described by the outer corners of the eyes and the corners of the mouth. The four required points are taken from a matched face graph. The second, holistic method also makes use of a face graph. Here the face region is derived from the convex hull that contains all nodes of the face-graph (Fig. 2.43). Then a transformation of the face region into the eigenspace takes place. The base of the eigenspace is created by the PCA transformation of 60 reference views which represent different head poses. The calculation of the Euclidian Distance of the actual view into the eigenspace allows the estimation of the head pose. Finally the results of the analytic and the holistic approach are compared. In case of significant differences the actual head pose is estimated by utilizing the last correct result combined with a prediction involving the optical flow. Analytic Approach In a frontal view the area between eye and mouth corners appears to be symmetrical. If, however, the view is not frontal, the face plane will be distorted. To identify roll σ and pitch angle τ the distorted face plan is transformed into a frontal view (Fig. 2.44). A point x on the distorted plane is transformed into an undistorted point x by x = U x + b
(2.167)
where U is a 2x2 transformation matrix and b the translation. It can be shown that U can be decomposed to:
Facial Expression Commands
79
Fig. 2.43. Head pose is derived by a combined analytical and holistic approach.
U = λ1 R(Θ)P (λ, τ )
(2.168)
Here R(T heta) is a rotation around Θ and P (λ, τ ) is the following symmetric matrix: λ0 P (λ, τ ) = R(τ ) R(−τ ) (2.169) 01 with R(τ ) =
cos τ − sin τ sin τ cos τ
(2.170)
The decomposition of U shows the ingredients of the linear back transformation consisting of an isotropic scaling by λ1 , a rotation around the optical axis of the virtual cam corresponding to R(Θ), and a scaling by factor λ in the direction of τ . With two orthogonal vectors a (COG eyes – outer corner of the left eye) and b (COG eyes – COG mouth corners) the transformation matrix U is: U (ab) = ⇔U =
0 21 Re −1 0 0 12 Re −1 0
(2.171)
(ab)−1
(2.172)
Where Re is the ratio of a and b Since a and b are parallel and orthogonal with respect to the symmetry axes it follows:
80
Non-Intrusive Acquisition of Human Action
Fig. 2.44. Back transformation of a distorted face plane (right) on an undistorted frontal view of the face (left) which yields roll and pitch angle.
aT · U T · U · b = 0
(2.173)
Hence U must be a symmetrical matrix: αβ UT · U = βγ
(2.174)
Also it follows that:
tan(2τ ) =
2β α−γ
and
1 √ cos(σ) = √ μ+ μ−1
with
μ=
(α + γ)2 4(α · γ − β 2 )
Finally the normal vector for head pose is: ⎡ ⎤ sin(σ) · cos(τ ) n = ⎣ sin(σ) · sin(τ ) ⎦ − cos(σ)
(2.175) (2.176)
(2.177)
Caused by the implied trigonometric functions, roll and pitch angles are always ambiguous, i.e. the trapezoid described by mouth and eye corners is identical for different head poses. This problem can be solved by considering an additional point such as, e.g., the nose tip, to fully determine the system.
Facial Expression Commands
81
Fig. 2.45. Pose reconstruction from facial geometry distortion is ambiguous as may be seen by comparing the left and right head pose.
Holistic Approach A second method for head pose estimation makes use of the Principal Component Analysis (PCA). It transforms the face region into a data space of Eigenvectors that results from different poses. The monochrome images used for reference are first normalized by subtracting the average intensity from individual pixels and then dividing the result by the standard deviation. Hereby variations resulting from different illuminations are averaged. For n views of a rotating head a pose eigenspace (PES) can be generated by performing the PCA. A reference image sequence distributed equally between -60 and 60 degrees yaw angle is collected, where each image has size m = w × h and is represented as m-dimensional row vector x. This results in a set of such vectors (x1 , x2 , . . . , xn ). The average μ and the covariance matrix S is calculated by:
μ=
1 x n i=1 i
(2.178)
S=
1 [x − μ][xi − μ]T n i=1 i
(2.179)
n
n
Let φj with j = 1 . . . n be the Eigenvectors of S with the largest corresponding eigenvalues λj : (2.180) ϕj = λj ϕj
82
Non-Intrusive Acquisition of Human Action
The n Eigenvectors establish the basis for the PES. For each image of size w × h a pattern vector Ω(x) = [ω1 , ω2 , . . . , ωn ] may be found by projection with the Eigenvectors φj . ωj = ϕTj (x − μ) with j = 1, . . . , n
(2.181)
After normalizing the pattern vector with the eigenvalues the image can be projected in the PES: # " ωn ω1 ω2 (2.182) , ,··· , Ωnorm (x) = λ1 λ2 λn The reference image sequence then corresponds to a curve in the PES. This is illustrated by Fig. 2.46 where only the first three Eigenvectors are depicted.
Fig. 2.46. top: Five of sixty synthesized views. middle: masked faces. bottom: cropped faces. right: Projection into the PES.
In spite of color normalization the sequence projection is influenced significantly by changes in illumination as illustrated by Fig. 2.47. Here three image sequences with different illumination are presented together with the resulting projection curves in the PES.
Fig. 2.47. Variations in the PES due to different illumination of the faces.
Facial Expression Commands
83
To compensate this influence a Gabor Wavelet Transformation (GWT) is applied which is the convolution with a Gabor Wavelet in Fourier space. Here a combination of the four GWT responses corresponding to 0°, 45°, 90° and 145° is used. As a result the curves describing head pose for different illumination are closer together in PES (Fig. 2.48).
Fig. 2.48. Here four gabor wavelet transforms are used instead of image intensity. The influence of illumination variations is much smaller than in Fig. 2.47
Now, if the pose of a new view is to be determined, this image is projected into the PES as well and subsequently the reference point with the smallest Euclidean distance is identified. 2.2.4.2 Determination of Line of Sight For iris localization the circle Hough transformation is used which supports the reliable detection of circular objects in an image. Since the iris contains little red hue, only the red channel is extracted from the eye region which contains a high contrast between skin and iris. By the computation of directed X and Y gradients in the red channel image the amplitude and phase between iris and its environment is emphasized. The Hough space is created using an extended Hough transform. It is applied on search mask which is generated based on a threshold segmentation of the red channel image. Local maxima then point to circular objects in the original image. The iris by its very nature represents a homogeneous area. Hence the local extremes in the Hough space are being validated by verifying the filling degree of the circular area with a expected radius in the red-channel Finally the line of sight is determined by comparing the intensity distributions around the iris with trained reference distributions using a Maximum-Likelihood-Classifier. In Fig. 2.49 the entire concept for line of sight analysis is illustrated. 2.2.4.3 Circle Hough Transformation The Hough transformation (HT) represents digital images in a special parameter space called Hough Space. For a line, e.g., this space is two dimensional and has the coordinates slope and shift. For representing circles and ellipses a Hough space of higher dimensions is required.
84
Non-Intrusive Acquisition of Human Action
Fig. 2.49. Line of sight is identified based on amplitude and phase information in the red channel image. An extended Hough transformation applied to a search mask finds circular objects in the eye region.
For the recognition of circles, the circle hough transformation (CHT) is applied. Here the Hough space H(x, y, r) corresponds to a three dimensional accumulator that indicates the probability that there is a circle with radius r at position (x, y) in the input image. Let Cr be a circle with radius r: Cr = {(i, j)|i2 + j 2 = r2 } Then the Hough Space is described by: I(x + i, y + j) H(x, y, r) =
(2.183)
(2.184)
(i,j)∈Cr
This means that for each position P of the input image where the gradient is greater than a certain threshold, a circle with a given radius is created in the Hough space. The contributions of the single circles are accumulated in Hough space. A local maximum at position P then points to a circle center in the input image as illustrated in Fig. 2.50 for 4 and 16 nodes, if a particular radius is searched for. For the application at hand the procedure is modified such that the accumulator in Hough Space is no longer incremented by circles but by circular arc segments (Fig. 2.51). The sections of the circular arc are incremented orthogonally in respect of the phase to both sides.
Facial Expression Commands
85
Fig. 2.50. In the Hough transformation all points in an edge-coded image are interpreted as centers of circles. In Hough space the contributions of single circles are accumulated. This results in local maxima which correspond to circle centers in the input image. This is exemplarily shown for 4 and 16 nodes.
Fig. 2.51. Hough transformation for circular arc segments which facilitates the localization of circles by reducing noise. This is exemplarily shown for 4 and 16 nodes.
For line of sight identification the eyes’ sclera is analyzed. Following iris localization, a concentric circle is described around it with the double of its diameter. The resulting annulus is divided in to eight segments. The intensity distribution of these segments then indicates line of sight. This is illustrated by Fig. 2.52 where distributions for two different lines of sight are presented which are similar for both eyes but dissimilar for different lines of sight. A particular line of sight associated with a distribution as shown in Fig. 2.52 is then identified with a maximum likelihood classifier. The maximum likelihood classifier assigns a feature vector g = m(x, y) to one of t object classes. Here g consists of the eight intensities of the circular arc segments mentioned above. For classifier training a sample of 15 (5 x 3) different lines of sights were collected from ten subjects. From these data the average vector μi (i = 0 . . . 14), the covariance matrix Si , and the inverse covariance matrix Si−1 were calculated. Based on this the statistical spread vector si could be calculated.
86
Non-Intrusive Acquisition of Human Action
Fig. 2.52. Line of sight identification by analyzing intensities around the pupil.
From these instances the rejection threshold dzi = sTi · Si−1 · si for each object class is determined. For classification the Mahalanobis distances di = (g − μi )T Si−1 (g − μi ) of the actual feature vector to all classes is calculated. Afterwards a comparison with pretrained thresholds takes place. The feature vector g then is assigned to that class kj for which the Mahalanobis distance is minimal and for which the distance is lower than the rejection threshold of that class(Fig. 2.53). .
Fig. 2.53. A maximum likelihood classifier for line of sight identification. A feature vector is assigned to that class to which the Mahalanobis distance is smallest.
Facial Expression Commands
87
2.2.4.4 Determination of Lip Outline A point distribution model (PDM) in connection with an active shape model (ASM) is used for lip outline identification. For ASM initialization the lip borders must be segmented from the image as accurately as possible. Lip Region Segmentation Segmentation is based on four different feature maps which all emphasize the lip region from the surrounding by color and gradient information Fig. 2.54.
Fig. 2.54. Identification of lip outline with an active shape model. Four different features contribute to this process.
The first feature represents the probability that a pixel belongs to the lips. The required ground truth, i.e. lip color histograms, has been derived from 800 images segmented by hand. The a posteriori probabilities for lips and background are then calculated as already described in Sect. 2.1.2.1. In the second feature the nonlinear LUX color space (Logarithmic hUe eXtention) is exploited to emphasize the contrast between lips and surrounding [14], where L is the luminance, and U and X incorporate chromaticity. Here the U channel is used as a feature.
88
Non-Intrusive Acquisition of Human Action L = (R + 1)0.3 (G + 1)0.6 (B + 1)0.1 − 1 256 R+1 if R < L 2 ( L+1 ) U= L+1 ( ) else 256 − 256 2 R+1 256 B+1 if B < L 2 ( L+1 ) X= 256 L+1 256 − 2 ( B+1 ) else
(2.185) (2.186) (2.187)
A third feature makes use of the I1 I2 I3 color space which also is suited to emphasize the contrast between lips and surrounding. The three channels are defined as follows: R+G+B (2.188) 3 R−B (2.189) I2 = 2 2G − R − B I3 = (2.190) 4 The fourth feature utilizes a Sobel operator that emphasizes the edges between the lips and skin- or beard-region. ⎡ ⎤ ⎡ ⎤ −1 0 1 −1 −2 −1 ⎣ −2 0 2 ⎦ or rather ⎣ 0 0 0 ⎦ (2.191) −1 0 1 1 2 1 The filter mask is convoluted with the corresponding image region. The four different features need to be combined for establishing an initialization mask. For fusion, a logical OR without individual weighting was selected, as weighting did not improve the results. While the first three features support the segmentation between lips and surrounding, the gradient map serves the union of single regions in cases where upper and lower lips are segmented separately due to dark corners of the mouth or to teeth. Nevertheless, segmentation errors still occur when the color changes between lips and surrounding is floating. This effect is observed if, e.g., shadow falls below the lips, making that region dark. Experiments show that morphological convolution operators like erosion or dilatation are inappropriate to smooth out such distortions. Therefore here an iterative procedure is used which relates the center of gravity of the rectangular surrounding the lips Cog Lip to the center of gravity of the segmented area Cog Area . If Cog Lip lies above Cog Area , this most likely is due to line-formed artifacts. These then are erased linewise until a recalculation results in approximately identical centers of gravity (Fig. 2.55). 2.2.4.5 Lip modeling I1 =
For the collection of ground truth data, mouth images were taken from 24 subjects with the head filling the full image format. Each subject had to perform 15 visually distinguishable mouth forms under two different illumination conditions. Subsequently upper and lower lip, mouth opening, and teeth were segmented manually
Facial Expression Commands
89
Fig. 2.55. The mask resulting from fused features (left) is iteratively curtailed until the distance of the center of gravity from the bounding box to the center of gravity of the area is minimized (right).
in these images. Then 44 points were equally assigned on the outline with point 1 and point 23 being the mouth corners. It is possible to use vertically symmetrical and asymmetrical PDMs. Research into this shows that most lip outlines are asymmetrical, so asymmetrical models fit better. Since the segmented lip outlines vary in size, orientation, and position in the image, all points have to be normalized accordingly. The average form of training lip outlines, their Eigenvector matrix and variance vector, then form the basis for the PDMs which are similar to the outline models of the Active Appearance Models. The resulting PDMs are depicted√ in Fig. 2.56, √ where the first four Eigenvectors have been varied in a range between 3 λ, 0, −3 λ.
Fig. 2.56. √ Point distribution models for variations of the first four Eigenvectors φ1 , φ2 , φ3 , φ4 √ between 3 λ, 0, −3 λ.
90
Non-Intrusive Acquisition of Human Action
2.2.5 Facial Feature Recognition – Eye Localization Application To run the example application for eye localization, please install the rapid prototyping system I MPRESARIO from the accompanying CD as described in Sect. B.1. After starting I MPRESARIO, choose File → Open to load the process graph named FaceAnalysis_Eye.ipg (Fig. 2.57). The data flow is straightforward and corresponds to the structure of the sections in this chapter. All six macros are described below. Image Sequence Acquires a sequence of images from hard disk or other storage media. In the following, each image is processed individually and independently. Other sources may be used by replacing this macro with the appropriate source macro (for instance, Video Capture Device for a video stream from an USB cam). SplitImage Split image in three basic channels Red – Green – Blue. It is possible to split all the channels at the same time or to extract just one channel (here the Red channel is used). Gradient This is a simple wrapper for the convolution functor with some convenience parameterization to choose between different common gradient kernels. Not only the classical simple difference computation (right minus left for the x direction or bottom minus top for the y direction) and the classical Sobel, Prewitt, Robinson, Roberts and Kirsch kernels can be used, but more sophisticated optimal kernels (see lti::gradientKernelX) and the approximation using oriented Gaussian derivatives can be used. (output here: An absolute channel computed from X and Y gradient channels) ConvertChannel The output of the preceding stage is a floating point gray value image I(x, y) ∈ [0, 1]. Since the next macro requires a fixed point gray value image I(x, y) ∈ {0, 1, . . . , 255} a conversion is performed here. EyeFinder creates the Hough space by utilizing the gradient-information of the red input-channel. Local maxima in the Hough space are taken as the center of the eyes. There are two possibilities to visualize the results: First the Hough space itself could be shown, second the contours of the iris could be high-lighted green. 2.2.6 Facial Feature Recognition – Mouth Localization Application To run the example application for mouth localization, please install the rapid prototyping system I MPRESARIO from the accompanying CD as described in Sect. B.1. After starting I MPRESARIO, choose File → Open to load the process graph named FaceAnalysis_Mouth.ipg (Fig. 2.58). The data flow is straightforward and corresponds to the structure of the sections in this chapter. All six macros are described below.
Facial Expression Commands
91
Fig. 2.57. Process graph of the example application for eye-localization.
Image Sequence Acquires a sequence of images from hard disk or other storage media. In the following, each image is processed individually and independently. Other sources may be used by replacing this macro with the appropriate source macro (for instance, Video Capture Device for a video stream from an USB cam). SplitImage Split image in three basic-channels Red – Green – Blue. It is possible to split all the channels at the same time or to get just one channel (Here the Red channel is used). Gradient This is a simple wrapper for the convolution functor with some convenience parameterization to choose between different common gradient kernels. Not only the classical simple difference computation (right minus left for the x direction or bottom minus top for the y direction) and the classical Sobel, Prewitt, Robinson, Roberts and Kirsch kernels can be used, but more sophisticated optimal kernels (see lti::gradientKernelX) and the approximation using oriented Gaussian derivatives can be used. (output here: An absolute channel computed from X and Y gradient channels) GrayWorldConstancy Performs a color normalization on an image using a gray world approach, in order to eliminate effects of illumination color. Gray world methods and white world methods assume that some low order statistics on the
92
Non-Intrusive Acquisition of Human Action
colors of a real world scene lie on the gray line of a color space. The most simple approach uses only the mean color and maps it linearly into one specific point on the gray line, assuming that a diagonal transform (a simple channel scaling) is valid. Normalization is done under the assumption of mondrian worlds, constant point light sources and delta function camera sensors. ASMInitializer creates an initialization mask for the MouthFinder. Therefore three color-based maps are generated and together with the gradient-information combined. The result is a gray image that represent the probability for each pixel to belong to background or to the lips. ObjectsFromMask outputs a list of all detected regions sorted descending by area (which may be empty if no regions were found). A value of zero for the parameter Threshold uses the algorithm shown in Fig. 2.6 for automatic threshold computation. If this fails, the threshold can be set manually. The remaining parameters control further aspects of the segmentation process and should be left at their defaults. MouthFinder utilizes point distribution models to extract the mouth contour. Therefore, first a selectable number of points are extracted from the initialization mask given by module ASMInitializer. Afterwards the points are interpolated by a spline. The won contour serves for the initialization of the Active Shape models.
References 1. Akyol, S. and Canzler, U. An Information Terminal using Vision Based Sign Language Recognition. In B¨uker, U., Eikerling, H.-J., and M¨uller, W., editors, ITEA Workshop on Virtual Home Environments, VHE Middleware Consortium, volume 12, pages 61–68. 2002. 2. Akyol, S., Canzler, U., Bengler, K., and Hahn, W. Gesture Control for Use in Automobiles. In Proceedings of the IAPR MVA 2000 Workshop on Machine Vision Applications. 2000. 3. Bley, F., Rous, M., Canzler, U., and Kraiss, K.-F. Supervised Navigation and Manipulation for Impaired Wheelchair Users. In Thissen, W., Wierings, P., Pantic, M., and Ludema, M., editors, Proceedings of the IEEE International Conference on Systems, Man and Cybernetics. Impacts of Emerging Cybernetics and Huamn-Machine Systems, pages 2790–2796. 2004. 4. Canzler, U. and Kraiss, K.-F. Person-Adaptive Facial Feature Analysis for an Advanced Wheelchair User-Interface. In Drews, P., editor, Conference on Mechatronics & Robotics, volume Part III, pages 871 – 876. Sascha Eysoldt Verlag, September 2004. 5. Canzler, U. and Wegener, B. Person-adaptive Facial Feature Analysis. The Faculty of Electrical Engineering, page 62, May 2004. 6. Cootes, T., Edwards, G., and Taylor, C. Active Appearance Models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(6):681 – 685, 2001. 7. Csink, L., Paulus, D., Ahlrichs, U., and Heigl, B. Color Normalization and Object Localization. In 4. Workshop Farbbildverarbeitung, Koblenz, pages 49–55. 1998.
References
93
Fig. 2.58. Process graph of the example application for mouth localization. 8. Dugad, R. and Desai, U. B. A Tutorial on Hidden Markov Models. Technical Report SPANN-96.1, Department of Electrical Engineering, Indian Institute of Technology, May 1996. 9. Finlayson, G. D., Schiele, B., and Crowley, J. L. Comprehensive Colour Image Normalization. Lecture Notes in Computer Science, 1406(1406), 1998. 10. Jain, A. K. Fundamentals of Digital Image Processing. Prentice-Hall, 1989. 11. Jelinek, F. Statistical Methods for Speech Recognition. MIT Press, 1998. ISBN 0-26210066-5. 12. Jones, M. and Rehg, J. Statistical Color Models with Application to Skin Detection. Technical Report CRL 98/11, Compaq Cambridge Research Lab, December 1998. 13. Juang, B. H. and Rabiner, L. R. The Segmental k-means Algorithm for Estimating the Parameters of Hidden Markov Models. In IEEE Trans. Acoust., Speech, Signal Processing, volume ASSP-38, pages 1639–1641. 1990. 14. Lievin, M. and Luthon, F. Nonlinear Colour Space and Spatiotemporal MRF for Hierarchical Segmentation of Face Features in Video. IEEE Transactions on Image Processing, 13:63 – 71, 2004. 15. Mitchell, T. M. Machine Learning. McGraw-Hill, 1997. 16. Nelder, J. and Mead, R. A Simplex Method for Function Minimization. Computer Journal, 7(4):308 – 313, 1965. 17. Pantic, M. and Rothkrantz, L. J. M. Expert Systems for Automatic Analysis of Facial Expressions. In Image and Vision Computing Journal, volume 18, pages 881–905. 2000.
94
Non-Intrusive Acquisition of Human Action
18. Papageorgiou, C., Oren, M., and Poggio, T. A General Framework for Object Detection. pages 555 – 562. 1998. 19. Rabiner, L. and Juang, B.-H. An Introduction to Hidden Markov Models. IEEE ASSP Magazine, 3(1):4–16, 1986. 20. Rabiner, L. R. A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition. Readings in Speech Recognition, 1990. 21. Schukat-Talamazzini, E. Automatische Spracherkennung. Vieweg Verlag, Braunschweig/Wiesbaden, 1995. 22. Sch¨urmann, J. Pattern Classification. John Wiley & Sons, 1996. 23. Sherrah, J. and Gong, S. Tracking Discontinuous Motion Using Bayesian Inference. In Proceedings of the European Conference on Computer Vision, pages 150–166. Dublin, Ireland, 2000. 24. Sonka, M., Hlavac, V., and Boyle, R. Image Processing, Analysis and Machine Vision. Brooks Cole, 1998. 25. Steger, C. On the Calculation of Arbitrary Moments of Polygons. Technical Report FGBV–96–05, Forschungsgruppe Bildverstehen (FG BV), Informatik IX, Technische Universit¨at M¨unchen, October 1996. 26. Tomasi, C. and Kanade, T. Detection and Tracking of Point Features. Technical Report CS-91-132, CMU, 1991. 27. Wiskott, L., Fellous, J., Kr¨uger, N., and von der Malsburg, C. Face Recognition by Elastic Bunch Graph Matching. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7):775–779, 1997. 28. Zieren, J. and Kraiss, K.-F. Robust Person-Independent Visual Sign Language Recognition. In Proceedings of the 2nd Iberian Conference on Pattern RecognitionandImage Analysis IbPRIA 2005, volume Lecture Notes in Computer Science. 2005.
Chapter 3
Sign Language Recognition J¨org Zieren1 , Ulrich Canzler2 , Britta Bauer3 , Karl-Friedrich Kraiss Sign languages are naturally developed and fully fledged languages used by the Deaf and Hard-of-Hearing for everyday communication among themselves. Information is conveyed visually, using a combination of manual and nonmanual means of expression (hand shape, hand posture, hand location, hand motion; head and body posture, gaze, mimic). The latter encode e.g. adjectives and adverbials, or provide specialization of general items. Some signs can be distinguished by manual parameters alone, while others remain ambiguous unless additional nonmanual information is made available. Unlike pantomime, sign language does not include its environment. Signing takes place in a 3D space close to the trunk and the head, called signing space. The grammar of sign language is fundamentally different from spoken language. The structure of a sentence in spoken language is linear, one word followed by another, whereas in sign language, a simultaneous structure exists with a parallel temporal and spatial configuration. The configuration of a sign language sentence carries rich information about time, location, person, or predicate. Thus, even though the duration of a sign is approximately twice as long as the duration of a spoken word, the duration of a signed sentence is about the same. Unfortunately, very few hearing people speak sign language. The use of interpreters is often prohibited by limited availability and high cost. This leads to problems in the integration of deaf people into society and conflicts with an independent and self-determined lifestyle. For example, most deaf people are unable to use the World Wide Web and communicate by SMS and email in the way hearing people do, since they commonly do not surpass the level of literacy of a 10-year-old. The reason for this is that hearing people learn and perceive written language as a visual representation of spoken language. For deaf people, however, this correspondence does not exist, and letters – which encode phonemes – are just symbols without any meaning. In order to improve communication between Deaf and Hearing research in automatic sign language recognition is needed. This work shall provide the technical requirements for translation systems and user interfaces that support the integration of deaf people into the hearing society. The aim is the development of a mobile system, consisting of a laptop equipped with a webcam that visually reads a signer’s gestures and mimic, and performs a translation into spoken language. This device is intended as an interpreter in everyday life (e.g. at the post office or in a bank). Fur1
Sect. 3.1 Sect. 3.2 3 Sect. 3.3 2
96
Sign Language Recognition
thermore, it allows deaf people intuitive access to electronic media such as computers or the internet (Fig. 3.1).
Fig. 3.1. laptop and webcam based sign language recognition system.
State of the Art in Automatic Sign Language Recognition Since a two-dimensional video signal is significantly more complex than a onedimensional audio signal, the current state in sign language recognition is roughly 30 years behind speech recognition. In addition, sign language is by far not fully explored yet. Little is known about syntax and semantics, and no dictionary exists. The lack of national media – such as radio, TV, and telephone for the hearing – leads to strong regional variation. For a large number of signs, there is not even a common definition. Sign language recognition has become the subject of scientific publications only in the beginning of the 90s. Most presented systems operate near real-time and require 1 to 10 seconds of processing time after completion of the sign. For videobased approaches, details on camera hardware and resolution are rarely published, suggesting that professional equipment, high resolution, low noise, and optimal camera placement was used. The method of data acquisition defines a user interface’s quality and constitutes the primary feature for classification of different works. The most reliable, exact, and at the same time the simplest techniques are intrusive: Data gloves measure the
97 flexion of the finger joints, optical or magnetic markers placed on face and hands facilitate a straightforward determination of mimic and manual configuration. For the user, however, this is unnatural and restrictive. Furthermore, data gloves are unsuitable for practical applications due to high cost. Also most existing systems exploit manual features only; so far facial features were rarely used [17]. A recognition system’s usability is greatly influenced by the robustness of its image processing stage, i.e. its ability to handle inhomogeneous, dynamic, or generally uncontrolled backgrounds and suboptimal illumination. Many publications do not explicitly address this issue, which – in connection with accordant illustration – suggests homogeneous backgrounds and strong diffuse lighting. Another common assumption is that the signer wears long-sleeved clothing that differs in color from his skin, allowing color-based detection of hands and face. The majority of systems only support person-dependent operation, i.e. every user is required to train the system himself before being able to use it. Person-independent operation requires a suitable normalization of features early in the processing chain to eliminate dependencies of the features on the signer’s position in the image, his distance from the camera, and the camera’s resolution. This is rarely described in publications; instead, areas and distances are measured in pixels, which even for person-dependent operation would require an exact reproduction of the conditions under which the training material was recorded. Similar to the early days of speech recognition, most researchers focus on isolated signs. While several systems exist that process continuous signing, their vocabulary is very small. Recognition rates of 90% and higher are reported, but the exploitation of context and grammar – which is sometimes rigidly fixed to a certain sentence structure – aid considerably in classification. As in speech recognition, coarticulation effects and the resulting ambiguities form the primary problem when using large vocabularies. Tab. 3.1 lists several important publications and the described systems’ features. When comparing the indicated performances, it must be kept in mind that, in contrast to speech recognition, there is no standardized benchmark for sign language recognition. Thus, recognition rates cannot be compared directly. The compilation also shows that most systems’ vocabularies are in the range of 50 signs. Larger vocabularies have only been realized with the use of data gloves. All systems are person-dependent. All recognition rates are valid only for the actual test scenario. Information about robustness in real-life settings is not available for any of the systems. Furthermore, the exact constellation of the vocabularies is unknown, despite its significant influence on the difficulty of the recognition task. In summary it can be stated that none of the systems currently found in literature meets the requirements for a robust real world application. Visual Non-Intrusive Sign Language Recognition This chapter describes the implementation of a video-based non-intrusive sign language classifier for real world applications. Fig. 3.2 shows a schematic of the concept that can be divided into a feature extraction stage and a classification stage.
98
Sign Language Recognition Table 3.1. Classifiers characteristics for user dependent sign language recognition. Author, Year Vamplew 1996 [24] Holden 2001 [10] Yang 2002 [28] Murakami 1991 [15] Liang 1997 [14] Fang 2002 [8] Starner 1998 [20] Vogler 1999 [25] Parashar 2003 [17]
Features manual manual manual manual manual manual manual manual manual and facial
Interface
Vocabulary Language Recognition Level Rate in % data glove 52 word 94 optical markers 22 word 95,5 video 40 word 98,1 data glove 10 sentence 96 data glove 250 sentence 89,4 data glove 203 sentence 92,1 video 40 sentence 97,8 video 22 sentence 91,8 video 39 sentence 92
Fig. 3.2. Schematic of the sign language recognition system.
The input image sequence is forwarded to two parallel processing chains that extract manual and facial features using a priori knowledge of the signing process. Before the final sign classification is performed, a pre-classification module restricts the active vocabulary to reduce processing time. Manual and facial features are then classified separately, and both results are merged to yield a single recognition result. In the following section of this chapter the person-independent recognition of isolated signs in real world scenarios is dealt with. The second section explains how facial features can be integrated in sign language recognition process. Both parts build on related basic concepts presented already in chapter 2. The third section describes how signs can be divided automatically in subunits by a data driven procedure and how these subunits enable coding of large vocabularies and accelerate training times. Finally some performance measures for the described system are provided.
Recognition of Isolated Signs in Real-World Scenarios
99
3.1 Recognition of Isolated Signs in Real-World Scenarios Sign language recognition constitutes a challenging field of research in computer vision. Compared to gesture recognition in controlled environments as described in Sect. 2.1, recognition of sign language in real-world scenarios places significantly higher demands on feature extraction and processing algorithms. First, sign language itself provides a framework that restricts the developer’s freedom in the choice of vocabulary. In contrast to an arbitrary selection of gestures, common computer vision problems like overlap of both hands and the face (Fig. 3.3a4 ), ambiguities (Fig. 3.3b), and similarities between different signs (Fig. 3.3c, d) occur frequently and cannot be avoided.
(a) “digital”
(b) “conference”
(c) “food”
(d) “date”
Fig. 3.3. Difficulties in sign language recognition: Overlap of hands and face (a), tracking of both hands in ambiguous situations (b), and similar signs with different meaning (c, d).
Just like in spoken language, there are dialects in sign language. In contrast to the pronunciation of words, however, there is no standard for signs, and people may use an altogether different sign for the same word (Fig. 3.4). Even when performing identical signs, the variations between different signers are considerable (Fig. 3.5). Finally, when using color and/or motion to detect the user’s hands and face, uncontrolled backgrounds as shown in Fig. 3.6 may give rise to numerous false alarms 4
The video material shown in this section was kindly provided by the British Deaf Association (http://www.signcommunity.org.uk). All signs are British Sign Language (BSL).
100
Sign Language Recognition
(a)
(b)
Fig. 3.4. Two different signs for “actor”: Circular motion of hands in front of upper body (a) vs. dominant hand (here: right) atop non-dominant hand (b).
(a) “autumn”
(b) “recruitment”
(c) “tennis”
(d) “distance”
(e) “takeover”
Fig. 3.5. Traces of (xcog , ycog ) for both hands from different persons (white vs. black plots) performing the same sign. The background shows one of the signers for reference.
that require high-level reasoning to tell targets from distractors. This process is computationally expensive and error-prone, because the required knowledge and experience that comes natural to humans is difficult to encode in machine-readable form.
Fig. 3.6. Examples for uncontrolled backgrounds that may interfere with the tracking of hands and face.
Recognition of Isolated Signs in Real-World Scenarios
101
A multitude of algorithms has been published that aim at the solution of the above problems. This section describes several concepts that were successfully combined in a working sign language recognition system at the RWTH Aachen university [29]. A comprehensive overview of recent approaches can be found in [6, 16]. This section focuses on the recognition of isolated signs based on manual parameters. The combination of manual and non-manual parameters such as facial expression and mouth shape, which constitute an integral part of sign language, is discussed in Sect. 3.2. Methods for the recognition of sign language sentences are described in Sect. 3.3. The following text builds upon Sect. 2.1. It follows a similar structure since it is based on the same processing chain. However, the feature extraction stage is extended as shown in Fig. 3.7. An image preprocessing stage is added that applies low-level algorithms for image enhancement, such as background modeling described in Sect. 3.1.2.1. High-level knowledge is applied for the resolution of overlaps (Sect. 3.1.3.1) and to support hand localization and tracking (Sect. 3.1.3.2). Compared to Sect. 2.1, the presentation is less concerned with concrete algorithms. Due to the complexity and diversity of the topic, the focus lies on general considerations that should apply to most, though certainly not all, applications.
Fig. 3.7. Feature extraction extended by an image preprocessing stage. High-level knowledge of the signing process is used for overlap resolution and hand localization/tracking.
3.1.1 Image Acquisition and Input Data With regard to the input data, switching from gestures to sign language input brings about the following additional challenges:
102
Sign Language Recognition
1. While most gestures are one-handed, signs may be one- or two-handed. The system must therefore not only be able to handle two moving objects, but also to detect whether the sign is one- or two-handed, i.e. whether one of the hands is idle. 2. Since the hands’ position relative to the face carries a significant amount of information in sign language, the face must be included in the image as a reference point. This reduces image resolution available for the hands and poses an additional localization task. 3. Hands may occlude one another and/or the face. For occluded objects, the extraction of features through a threshold segmentation is no longer possible, resulting in a loss of information. 4. Signs may be extremely similar (or even identical) in their manual features and differ mainly (or exclusively) in non-manual features. This makes automatic recognition based on manual features difficult or, for manually identical signs, even unfeasible. 5. Due to dialects and lack of standardization, it may not be possible to map a given word to a single sign. As a result, either the vocabulary has to be extended to include multiple variations of the affected signs, or the user needs to know the signs in the vocabulary. The latter is problematic because native signers may find it unnatural to be forced to use a certain sign where they would normally use a different one.5 The last two problems cannot be solved by the approach described here. The following text presents solution strategies for the former three. 3.1.2 Image Preprocessing The preprocessing stage improves the quality of the input data to increase the performance (in terms of processing speed and/or accuracy) of the subsequent stages. High-level information computed in those stages may be exploited, but this bears the usual risks of a feedback loop, such as instability or the reinforcement of errors. Low-level pixel-wise algorithms, on the other hand, do not have this problem and can often be used in a wide range of applications. A prominent example is the modeling of the image background. 3.1.2.1 Background Modeling The detection of background (i.e. static) areas in dynamic scenes is an important step in many pattern recognition systems. Background subtraction, the exclusion of these areas from further processing, can significantly reduce clutter in the input images and decrease the amount of data to be processed in subsequent stages. If a view of the background without any foreground objects is available, a statistical model can be created in a calibration phase. This approach is described in Sect. 5.3.1.1. In the following, however, we assume that the only available input data is the signing user, and that the signing process takes about 1–10 seconds. Thus the 5
This problem also exists in speech recognition, e.g. for the words “zero” and “oh”.
Recognition of Isolated Signs in Real-World Scenarios
103
background model is to be created directly from the corresponding 25–250 image frames (assuming 25 fps) containing both background and foreground, without prior calibration. Median Background Model A simple yet effective method to create a background model in the form of an RGB image Ibg (x, y) is to compute, for every pixel (x, y), the median color over all frames: Ibg (x, y) = median{I(x, y, t)|1 ≤ t ≤ T }
(3.1)
where the median of a set of vectors V = {v1 , v2 , . . . , vn } is the element for which the sum of Euclidian distances to all other elements is minimal: median V = argmin v∈V
n
|vi − v|
(3.2)
i=1
For the one-dimensional case this is equivalent to the 50th percentile, which is significantly faster to compute, e.g. by simply sorting the elements of V . Therefore, (3.2) is in practice often approximated by the channel-wise median ⎛ ⎞ median{r(x, y, t)|1 ≤ t ≤ T } Ibg (x, y) = ⎝ median{g(x, y, t)|1 ≤ t ≤ T } ⎠ (3.3) median{b(x, y, t)|1 ≤ t ≤ T } The median background model has the convenient property of not requiring any parameters and is consequently very robust. Its only requirement is that the image background must visible at the considered pixel in more than 50% of the input frames, which is a reasonable assumption in most scenarios. A slight drawback might be that all input frames need to be buffered in order to compute (3.1). To apply the background model, i.e. classify a given pixel as background or foreground, we need to define a suitable metric Δ to quantify the difference between a background color vector (rbg gbg bbg )T and a given color vector (r g b)T : Δ (r g b)T , (rbg gbg bbg )T ≥ 0
(3.4)
Computing Δ for every pixel in an input image I(x, y, t) and comparison with a motion sensitivity threshold Θmotion yields a foreground mask Ifg,mask : 1 if Δ (I(x, y, t), Ibg (x, y)) ≥ Θmotion Ifg,mask (x, y, t) = (3.5) 0 otherwise Θmotion is chosen just large enough so that the camera noise is not classified as foreground. A straightforward implementation of Δ is the Euclidian distance: Δ (r g b)T , (rbg gbg bbg )T = (r − rbg )2 + (g − gbg )2 + (b − bbg )2
(3.6)
104
Sign Language Recognition
However, (3.6) does not distinguish between differences in brightness and differences in hue. Separating the two is desirable since it permits to ignore shadows (which mainly affect brightness) cast on the background by moving (foreground) objects. Shadows can be modeled as a multiplication of Ibg with a scalar. We define Δ = I − Ibg ϕ = (I, Ibg )
(3.7) (3.8)
This allows to split the color difference vector Δ in orthogonal components Δbrightness and Δhue and weight the contribution of shadows in the total difference with a factor w ∈ [0, 1]: Δbrightness = |Δ| cos ϕ Δhue = |Δ| sin ϕ Δ = (wΔbrightness )2 + Δ2hue
(3.9) (3.10) (3.11)
A value of w = 1 disables shadow detection and is equivalent to (3.6), while w = 0 yields maximum shadow tolerance. No distinction is made between shadows that appear on the background and shadows that vanish from it. Obviously, the choice of w is a tradeoff between errors (foreground classified as background) and false alarms (background classified as foreground) and thus depends on which is handled better in the subsequent processing stages. Similar approaches to shadow detection are described e.g. in [13, 18]. Thus, while the computation of the median background model is itself parameterfree, its application involves the two parameters Θmotion and w. Fig. 3.8 shows four example images from an input video of 54 frames in total. Each of the 54 frames contains at least one person in the background. The resulting background model, as well as an exemplary application to a single frame, is visualized in Fig. 3.9. Gaussian Mixture Models Most background models, including the one presented above, assume a static background. However, backgrounds may also be dynamic, especially in outdoor scenarios. For example, a tree with green leaves that move in front of a white wall results in each background pixel alternating between green and white. Approaches that model dynamic backgrounds are described in [13, 18, 21]. Instead of computing a single background color as in (3.1), background and foreground are modeled together for each pixel as a mixture of Gaussian distributions, usually in RGB color space. The parameters of these distributions are updated online with every new frame. Simple rules are used to classify each individual Gaussian as background or foreground. Handling of shadows can be incorporated as described above.
Recognition of Isolated Signs in Real-World Scenarios
105
Fig. 3.8. Four example frames from the sign “peel”. The input video consists of 54 frames, all featuring at least one person in the background.
Compared to simpler methods this is computationally more demanding and requires more input frames to compute an accurate model, because the number of parameters to be estimated is higher. Also, several parameters have to be specified manually to control creation and updates of the individual Gaussians and their background/foreground classification. Performance may degrade if these parameters are inappropriate for the input video. If a sufficient amount of input data is available, the resulting model performs well under a wide variety of conditions. For single signs of approximately two seconds duration this is usually a critical factor, while one or more sign language sentences may provide enough data. For the recognition of isolated signs one might therefore consider to never stop capturing images and continuously update the distribution parameters. 3.1.3 Feature Extraction Systems that operate in real-world conditions require sophisticated feature extraction approaches. Rather than localize a single hand, both hands and the face (which is needed as a reference point) have to be segmented. Since a sign may be one- or two-handed, it is impossible to know in advance whether the non-dominant hand will remain idle or move together with the dominant hand. Also, the resolution available for each target object is significantly smaller than in most gesture recognition appli-
106
Sign Language Recognition
(a) Background Model Ibg
(b) Example Frame
(c) Foreground Mask Ifg,mask Fig. 3.9. Background model computed for the input shown in Fig. 3.8 (a), its application to an example frame (b), and the resulting foreground mask (c).
cations, where the hand typically fills the complete image. Background subtraction alone cannot be expected to isolate foreground objects without errors or false alarms. Also, the face – which is mostly static – has to be localized before background subtraction can be applied. Since the hands’ extremely variable appearance prevents the use of shape or texture cues, it is common to exploit only color for hand localization. This leads to the restriction that the signer must wear long-sleeved, non-skin-colored clothing to facilitate the separation of the hand from the arm by means of color. By using a generic color model such as the one presented in [12], user and illumination independence can be achieved at the cost of a high number of false alarms. This method therefore requires high-level reasoning algorithms to handle these ambiguities. Alternatively, one may choose to either explicitly or automatically calibrate the system to the current user and illumination. This chapter describes the former approach. 3.1.3.1 Overlap Resolution In numerous signs both hands overlap with each other and/or with the face. When two or more objects overlap in the image, the skin color segmentation yields only a single blob for all of them, rendering a direct extraction of meaningful features impossible. Low contrast, low resolution, and the hands’ variable appearance usually
Recognition of Isolated Signs in Real-World Scenarios
107
do not allow a separation of the overlapping objects by an edge-based segmentation either. Most of the geometric features available for unoverlapped objects can therefore not be computed for overlapping objects and have to be interpolated. However, a hand’s appearance is sufficiently constant over several frames for template matching [19] to be applied. Using the last unoverlapped view of each overlapping object as a template, at least position features – which carry much information – can be reliably computed during overlap. The accuracy of this method decreases with increasing template age. Fortunately, the same technique can be used twice for a single period of overlap, the second time starting with the first unoverlapped view after the cessation of the overlap and proceeding temporally backwards. This effectively halves the maximum template age and increases precision considerably. 3.1.3.2 Hand Tracking Performing a simple hand localization in every frame, as described in Sect. 2.1.2.1, is not sufficiently reliable in complex scenarios. The relation between temporally adjacent frames has to be exploited in order to increase performance. Localization is thus replaced by tracking, using information from previous frames as a basis for finding the hand in the current frame. A common problem in tracking is the handling of ambiguous situations where more than one interpretation is plausible. For instance, Fig. 3.10a shows the skin color segmentation of a typical scene (Fig. 3.9b). This observation does not allow a direct conclusion as to the actual hand configuration. Instead, there are multiple interpretations, or hypotheses, as visualized in Fig. 3.10 (b, c, d). Simple approaches weigh all hypotheses against each other and choose the most likely one one in every frame on the basis of information gathered in preceding frames, such as, for instance, a position prediction. All other hypotheses are discarded. This concept is error-prone because ambiguous situations that cannot reliably be interpreted occur frequently in sign language. Robustness can be increased significantly by evaluating not only preceding, but also subsequent frames, before any hypothesis is discarded. It is therefore desirable to delay the final decision on the hands’ position in each frame until all available data has been analyzed and the maximum amount of information is available. These considerations lead to a multiple hypotheses tracking (MHT) approach [29]. In a first pass of the input data all conceivable hypotheses are created for every frame. Transitions are possible from each hypothesis at time t to all hypotheses at time t + 1, resulting in a state space as shown exemplarily in Fig. 3.11. The total $ number of paths through this state space equals t H(t), where H(t) denotes the number of hypotheses at time t. Provided that the skin color segmentation detected both hands and the face in every frame, one of these paths represents the correct tracking result. In order to find this path (or one as close as possible to it), probabilities are computed that indicate the likeliness of each hypothesized configuration, pstate , and the likeliness of each transition, ptransition (see Fig. 3.11).
108
Sign Language Recognition
(a)
(b)
(c)
(d)
Fig. 3.10. Skin color segmentation of Fig. 3.9b (a) and a subset of the corresponding hypotheses (b, c, d; correct: d).
The computation of pstate and ptransition is based on high-level knowledge, encoded as a set of rules or learned in a training phase. The following aspects, possibly extended by application-dependent items, provide a general basis: •
•
•
•
The physical configuration of the signer’s body can be deduced from position and size of face and hands. Configurations that are anatomically unlikely or do not occur in sign language reduce pstate . The three phases of a sign (preparation, stroke, retraction), in connection with the signer’s handedness (which has to be known in order to correctly interpret the resulting feature vector), should also be reflected in the computation of pstate . Even in fast motion, the hand’s shape changes little between successive frames at 25 fps. As long as no overlap occurs, the shape at time t can therefore serve as an estimate for time t + 1. With increasing deviation of the actual from the expected shape, ptransition is reduced. Abrupt shape changes due to overlap require special handling. Similarly, hand position changes little from frame to frame (at 25 fps) so that coordinates at time t may serve as a prediction for time t + 1. Kalman filters [27] may increase prediction accuracy by extrapolating on the basis of all past measurements.
Recognition of Isolated Signs in Real-World Scenarios
109
Fig. 3.11. Hypothesis space and probabilities for states and transitions.
•
Keeping track of the hand’s mean or median color can prevent confusion of the hand with nearby distractors of similar size but different color. This criterion affects ptransition .
To search the hypothesis space, the Viterbi algorithm (see Sect. 2.1.3.6) is applied in conjunction with pruning of unlikely paths at each step. The MHT approach ensures that all available information is evaluated before the final tracking result is determined. The tracking stage can thus exploit, at time t, information that becomes available only at time t1 > t. Errors are corrected retrospectively as soon as they become apparent. 3.1.4 Feature Normalization After successful extraction, the features need to be normalized. Assuming that resolution dependencies have been eliminated as described in Sect. 2.1.2.3, the area a still depends on the signer’s distance to the camera. Hand coordinates (xcog , ycog ) additionally depend on the signer’s position in the image. Using the face position and size as a reference for normalization can eliminate both dependencies. If a simple threshold segmentation is used for the face, the neck is usually included. Different necklines may therefore affect the detected face area and position (COG). In this case one may use the topmost point of the face and its width instead to avoid this problem. If the shoulder positions can be detected or estimated, they may also provide a suitable reference.
110
Sign Language Recognition
3.1.5 Feature Classification The classification of the extracted feature vector sequence is usually performed using HMMs as discussed in Sect. 2.1.3.6. However, due to the nature of sign language, the following additional processing steps are advisable. Firstly, leading and/or trailing idle frames in the input data can be detected by a simple rule (e.g. (2.45)). This is done for each hands individually. Cropping frames in which both hands are idle speeds up classification and prevents the classifier from processing input data that carries no information. Obviously, this process must be applied in the training phase as well. If the sign is single-handed, i.e. one hand remains idle in all frames, all classes in the vocabulary that represent two-handed signs can be disabled (or vice versa). This further reduces computational cost in the classification stage. 3.1.6 Test and Training Samples The accompanying CD contains a database of test and training samples that consists of 80 signs in British Sign Language (BSL) performed five times each, all by the same signer.6 The signs were recorded under laboratory conditions, with a unicolored blue background that allows to fade-in arbitrary other backgrounds using the common bluescreen method. Image resolution is 384 × 288, the frame rate is 25 fps, and the average sign duration is 73.4 frames.
Fig. 3.12. Example frame from the test and training samples provided on CD.
As shown in Fig. 3.12 the signer is standing. The hands are always visible, and their start/end positions are constant and identical, which simplifies tracking. The directory slrdatabase has five subdirectories named var00 to var04, each of which contains one production of all 80 signs. Each sign production is stored in the form of individual JPG files in a directory named accordingly. 6
We sincerely thank the British Deaf Association (http://www.signcommunity.org.uk) for these recordings.
Sign Recognition Using Nonmanual Features
111
This database can be used to test tracking and feature extraction algorithms, and to train and test a sign language classifier based e.g. on Hidden Markov Models.
3.2 Sign Recognition Using Nonmanual Features Since sign languages are multimodal languages, several channels for transferring information are used at the same time. One basically differentiates between the manual/gestical channels and the nonmanual/ facial channels and their respective parameters [4]. In the following first the nonmanual parameters are described in more detail and afterwards the extraction of facial features for sign language recognition is presented. 3.2.1 Nonmanual Parameters Nonmanual parameters are indispensable in signs language. They code adjectives and contribute to grammar. In particular some signs are identical with respect to gesturing and can only be differentiated by making reference to nonmanual parameters [5]. This is, e.g., the case for the signs ”not” and ”to” in German Sign Language, which can only be distinguished by making reference to head motion (Fig. 3.13). Similarly in British Sign Language (BSL) the signs ”now” and ”offer” need lip outline for disambiguation (Fig. 3.14). In the following the most important nonmanual parameters will be described in more detail.
Fig. 3.13. In German Sign Language the signs ”not”(A) and ”to”(B) are identical with respect to manual gesturing but vary in head movement.
Upper Body Posture The torso generally serves as reference of the signing space. Spatial distances and textual aspects can be communicated by the posture of the torso. The signs ”reject-
112
Sign Language Recognition
Fig. 3.14. In British Sign Language the signs ”now” (A) and ”offer” (B) are identical with respect to manual gesturing but vary in lip outline.
ing” or ”enticing”, e.g., show a slight inclination of the torso towards the rear and in forward direction. Likewise grammatical aspects as, e.g., indirect speech can be coded by torso posture. Head pose The head pose also supports the semantics of sign language. Questions, affirmations, denials, and conditional clauses are, e.g., communicated with the help of head pose. In addition, information concerning time can be coded. Signs, which refer to a short time lapse are, e.g., characterized by a minimal change of head pose while signs referring to a long lapse are performed by turning the head clearly into the direction opposite to the gesture. Line of sight Two communicating deaf persons will usually establish a close visual contact. However, a brief change of line of sight can be used to refer to the spatial meaning of a gesture. Likewise line of sight can be used in combination with torso posture to express indirect speech as, e.g., as representation between two absent persons is virtual placed behind the signer. Facial expression Facial expressions essentially serve the transmission of feelings (lexical mimics). In addition grammatical aspects may be mediated as well. A change of head pose combined with the lifting of the eye brows corresponds, e.g., to a subjunctive.
Sign Recognition Using Nonmanual Features
113
Lip outline Lip outline represents the most pronounced nonmanual characteristic. Often it differs from voicelessly expressed words in that part of a word is shortened. Lip outline solves ambiguities between signs (brother vs. sister), and specifies expressions (meat vs. Hamburger). It also provides information redundant to gesturing to support differentiation of similar signs. 3.2.2 Extraction of Nonmanual Features In Fig. 3.15 a general concept for nonmanual feature extraction is depicted. At the beginning of the processing chain images of the camera are acquired. In the next step the Face Finder (Sect. 2.2.2.1) is included, in order to locate the face region of the signer in the image, which covers the entire sign space. Afterwards this region is cropped and upscaled.
Fig. 3.15. Processing chain of nonmanual feature extraction.
The next step is the fitting of a user-adapted face-graph by utilizing Active Appearance Models (Sect. 2.2.3). The face graph serves the extraction of the nonmanual parameters like lip outline, eyes and brows (Fig. 3.16). The used nonmanual features (Fig. 3.17) can be divided in three groups. At first the lip outline is described by width, height and form-features like invariant moments, eccentricity and orientation (Sect. 2.1.2.3). The distances between eyes and mouth corners are in the second group and the third group contains the distances between eyes and eye brows. For each image of the sequence the extracted parameters are merged into a feature vector, which in the next step is used for classification (Fig. 3.18). Overlap Resolution In case of partially overlapping of hands and face an accurate fitting of the Active Appearance Models is usually no longer possible. Furthermore it is problematic that
114
Sign Language Recognition
Fig. 3.16. Processing scheme of the face region cropping and the matching of an adaptive face-graph.
sometimes the face graph is computed astounding precisely, if there is not enough face texture visible. In this case a determination of the features in the affected regions is no longer possible. For compensation of this effect an additional module is involved, that evaluates all specific regions separately with regard to overlappings by hands. In Fig. 3.19 two cases are presented. In the first case one eye and the mouth are hidden, so the feature-vector of the facial parameters is not used for classification. In the second case the overlapping is not critical for classification so the facial features are thus consulted for the classification process. The hand tracker indicates a crossing of one or both hands with the face, once the skin colored surfaces touch each other. In these cases it is necessary to decide, whether the hands affect the shape substantially. Therefore, an ellipse for the hand is computed by using the manual parameters. In addition an oriented bounding box is drawn around the lip contour of the active appearance shape. If the hand ellipse touches the bounding box, the Mahalanobis-distance of the shape fitting is deciding. If this is too large, the shape is marked as invalid. Since the Mahalanobis-distance of the shapes depends substantially on the trained model, not an absolute value is
Sign Recognition Using Nonmanual Features
115
Fig. 3.17. Representation of nonmanual parameters suited for sign language recognition.
Fig. 3.18. Scheme of the extraction and classification of nonmanual features.
used here, but a proportional worsening. In experiments it showed itself that a good overlapping recognition can be achieved, if 25% of the face is hidden.
116
Sign Language Recognition
Fig. 3.19. During the overlapping of the hands and face several regions of the face are evaluated separately. If e.g. mouth and one eye could be hidden (left), no features of the face are considered. However, if eyes and mouth are located sufficiently the won features could be used for the classification (right).
3.3 Recognition of continuous Sign Language using Subunits Sign language recognition is a rather young research area compared to speech recognition, where research is already done for about 50 years. The headstart in the field of speech recognition makes a comparison between these two fields worthwhile, especially to draw conclusion in regards of how to model sign language. While phoneme based speech recognition systems represent today’s state of the art, the early speech recognizer dealt with words as a model unity. A similar development is observed for sign language recognition. Current systems are still based on word models. First steps towards subunit based recognition systems have been undertaken only recently [2,26]. The following section outlines a sign language recognition system, which is based on automatic generated subunits of signs. 3.3.1 Subunit Models for Signs It is yet unclear in sign language recognition, which part of a sign sentence serves as a good underlying model. The theoretical best solution is a whole sign language sentence as one model. Due to their long duration sentences do not lead to ill classification. However, the main disadvantage is obvious: there are far too many combinations to train all possible sentences in sign language. Consequently most sign language recognition systems are based on word models where one sign represents one model in the model database. Hence, the recognition of sign language sentences is more flexible. Nevertheless, even this method contains some drawbacks: • •
The training complexity increases with vocabulary size. A future enlargement of the vocabulary is problematic as new signs are usually signed in context of other signs for training (embedded training).
Recognition of continuous Sign Language using Subunits
117
As an alternative approach signs can also be modeled with subunits of signs, which is similar to modeling speech by means of phonemes. Subunits are segments of signs, which emerge from the subdivision of signs. Figure 3.20 shows an example of the above mentioned different possibilities for model unities in the model database. Subunits should represent a finite quantity of models. Their number should be chosen I
Position [Pixel]
TODAY
COOK
300 250 200 150 100 50 0 0
10
SU1
SU2
SU3
TODAY
20
SU4
SU5 I
30
SU6
40
SU7
SU8
50 Time [Frames]
SU9
SU10
COOK
Fig. 3.20. The signed sentence TODAY I COOK (HEUTE ICH KOCHEN) in German Sign Language (GSL). Top: the recorded sign sequence. Center: Vertical position of right hand during signing. Bottom: Different possibilities to divide the sentence. For a better visualization the signer wears simple colored cotton gloves.
in such a way that any sign can be composed with subunits. The advantages are: •
• •
The training period terminates when all subunit models are trained. An enlargement of the vocabulary is achieved by composing a new sign through concatenation of convenient subunit models. The general vocabulary size can be enlarged. Eventually, the recognition will be more robust (depending on the vocabulary size).
Modification of Training and Classification for Recognition with Subunits A subunit based recognition system needs an additional knowledge source, where the coding (also called transcription) of the sign into subunits is itemized. This knowledge source is called sign-lexicon and contains all transcriptions of signs and their coherence with subunits. Both, training and classification processes are based on a sign-lexicon.
118
Sign Language Recognition
Modification for Training Aim of the training process is the estimation of subunit model parameters. The example in figure 3.21 shows that sign 1 consists of subunits (SU) SU4 , SU7 and SU3 . The accordant model parameter of these subunits are trained on the recorded data of sign 1 by means of the Viterbi algorithm (see section 2.1.3.6). Segmented Images
Recorded images
VIDEOCAMERA
FEATURE EXTRACTION
IMAGE PROCESSING
CLASSIFIKATION
recognised Signal
Sequence of Feature Vectors
TRAINING
Model Database Subunits Sign 1 Sign 2
. . .
Sign M
SU 4 SU 7 SU 3 SU 6 SU 1 SU 5
. . .
SU 2 SU 7 SU 5
Sign-Lexicon
Fig. 3.21. Components of the training process for subunit identification.
Modification for Classification After completion of the training process, a database is filled with all subunit models which serve now as a base for the classification process. However, the aim of the classification is not the recognition of subunits but of signs. Hence, again the information which sign consists of which subunit is needed, which is contained in the sign-lexicon. The accordant modification for classification is depicted in figure 3.22. 3.3.2 Transcription of Sign Language So far we have assumed that a sign-lexicon is available, i.e. it is already known which sign is composed of which subunits. This however is not the case. The subdivision of a sign into suitable subunits still poses difficult problems. In addition, the semantics of the subunits have yet to be determined. The following section gives an overview of possible approaches to linguistic subunit formation. 3.3.2.1 Linguistics-orientated Transcription of Sign Language Subunits for speech recognition are mostly linguistically motivated and are typically syllables, half-syllables or phonemes. The base of this breakdown of speech is similar to the speech’s notation system: a written text with the accordant orthography is the standard notation system for speech. Nowadays, huge speech-lexica exist consisting
Recognition of continuous Sign Language using Subunits Segmented Images
Recorded images
VIDEOCAMERA
119
FEATURE EXTRACTION
IMAGE PROCESSING
CLASSIFIKATION
recognised Signal
Sequence of Feature Vectors
TRAINING
Model Database Subunits Sign 1 Sign 2
. . .
Sign M
SU 4 SU 7 SU 3 SU 6 SU 1 SU 5
. . .
SU 2 SU 7 SU 5
Sign-Lexicon
Fig. 3.22. Components of the classification process based on subunits.
of the transcription of speech into subunits. These lexica are usually the base for today’s speech recognizer. Transferring this concept of linguistic breakdown to sign language recognition, one is confronted with a variety of options for notation, which all are unfortunately not yet standardized as is the case for speech. An equally accepted notation system as written text does not exist for sign language. However, some known notation systems are examined in the following section, especially with respect to its applicability in a recognition system. Corresponding to phonemes, the term cheremes (derived from the Greek term for ’manual’) is used for subunits in sign language. Notation System by Stokoe Stokoe was one of the first, who did research in the area of sign language linguistic in the sixties [22]. He defined three different types of cheremes. The first type describes the configuration of handshape and is called dez for designator. The second type is sig for signation and describes the kind of movement of the performed sign. The third type is the location of the performed sign and is called tab for tabula. Stokoe developed a lexicon for American Sign Language by means of the above mentioned types of cheremes. The lexicon consists of nearly 2500 entries, where signs are coded in altogether 55 different cheremes (12 ’tab’, 19 ’dez’ and 24 different ’sig’). An example of a sign coded in the Stokoe system is depicted in figure 3.23 [22]. The employed cheremes seem to qualify as subunits for a recognition system, their practical employment in a recognition system however turns out to be difficult. Even though Stokoe’s lexicon is still in use today and consists of many entries, not all signs, especially no German Sign Language signs, are included in this lexicon. Also most of Stokoe’s cheremes are performed in parallel, whereas a recognition system expects subunits in subsequent order. Furthermore, none of the signations cheremes (encoding the movement of a performed sign) are necessary for a recognition system, as movements are modeled by Hidden Markov Models (HMMs). Hence, Stokoe’s
120
Sign Language Recognition
Fig. 3.23. Left: The sign THREE in American Sign Language (ASL). Right: notation of this sign after Stokoe [23].
lexicon is a very good linguistic breakdown of signs into cheremes, however without manual alterations, it is not useful as a base for a recognition system. Notation system by Liddell and Johnson Another notation system was invented by Liddell and Johnson. They break signs into cheremes by a so called Movement-Hold-Model, which was introduced in 1984 and further developed since then. Here, the signs are divided in sequential order into segments. Two different kinds of segments are possible: ’movements’ are segments, where the configuration of a hand is still in move, whereas for ’hold’-segments no movement takes place, i.e. the configuration of the hands is fixed. Each sign can now be modeled as a sequence of movement and hold-segments. In addition, each hold-segment consists of articulary features [3]. These describe the handshape, the position and orientation of the hand, movements of fingers, and rotation and orientation of the wrist. Figure 3.24 depicts an example of a notation of a sign by means of movement- and hold-segements.
Fig. 3.24. Notation of the ASL sign FATHER by means of the Movement-Hold model (from [26]).
While Stokoe’s notation system is based on a mostly parallel breakdown of signs, here a sequence of segments is produced, which is better suited for a recognition systems. However, similar to Stokoe’s notation system, there exists no comprehensive lexicon where all signs are encoded. Moreover, the detailed coding of the articulary features might cause additional problems. The video-based feature extraction of the recognition system might not be able to reach such a high level of detail. Hence, the
Recognition of continuous Sign Language using Subunits
121
movement-hold-notation system is not suitable for an employment as a sign-lexicon within a recognition system, without manual modifications or even manual transcription of signs. 3.3.2.2 Visually-orientated Transcription of Sign Language The visually7 approach of a notation (or transcription) system for sign language recognition does not rely on any linguistic knowledge about sign language – unlike the two approaches described in the previous sections. Here, the breakdown of signs into subunits is based on an data-driven process. In a first step each sign of the vocabulary is divided sequentially into different segments, which have no semantic meaning. A subsequent process determines similarities between the identified segments. Similar segments are then pooled and labeled. They are deemed to be one subunit. This process requires no other knowledge source except the data itself. Each sign can now be described as a sequence of the contained subunits, which are distinguished by their labels. This notation is also called fenonic baseform [11]. Figure 3.25 depicts as an example the temporal horizontal progression (right hand) of two different signs.
Position [Pixel]
Sign 2
SU3
SU4 SU6
SU2
1600
Sign 2
1400 1200 1000 800 600 400
Sign 1
200 0 0
Sign 1
5
SU3
SU7
10
SU1
15 Time [Frames]
SU9
SU5
Transcription of Sign Fig. 3.25. Example for different transcriptions of two signs.
7
For speech-recognition the accordant name is acoustic subunits. For sign language recognitions the name is adapted.
122
Sign Language Recognition
The performed signs are similar in the beginning. Consequently, both signs are assigned to the same subunit (SU3 ). However the further progression differs significantly. While the gradient of sign 2 is going upwards, the slope of sign 1 decreases. Hence, the further transcription of both signs differs. 3.3.3 Sequential and Parallel Breakdown of Signs The example in Fig. 3.25 illustrates merely one aspect of the performed sign: the horizontal progression of the right hand. Regarding sign language recognition and their feature extraction, this aspect would correspond to the y-coordinate of the right hand’s location. For a complete description of a sign, one feature is however not sufficient. In fact a recognition system must handle many more features which are merged in so called feature groups. The composition of these feature groups must take the lingustical sign language parameters into account, which are location, handshape and -orientation. The further details in this section refer to an example separation of a feature vector into a feature group ’pos’, where all features regarding the position (pos) of the both hands are grouped. Another group represents all features describing the ’size’ of the visible part of the hands (size), whereas the third group comprises all features regarding distances between all fingers (dist). The latter two groups stand for the sign parameter handshape and -orientation. Note, that these are only examples of how to model a feature vector and its accordant feature groups. Many other ways of modeling sign language are conceivable, also the number of feature groups may vary. To demonstrate the general approach, this example makes use of the three feature groups ’pos’, ’size’ and ’dist’ mentioned above. Following the parallel breakdown of a sign, each resulting feature group is segmented in sequential order into subunits. The identification of similar segments is not carried out on the whole feature vector, but only within each of the three feature groups. Similar segments finally stand for the subunits of one feature group. Pooled segments of the feature group ’pos’ e.g., now represent a certain location independent of any specific handshape and -orientation. The parallel and sequential breakdown of the signs finally yields three different sign lexica, which are combined to one (see also figure 3.26). Figure 3.27 shows examples of similar segments of different signs according to specific feature groups. 3.3.4 Modification of HMMs to Parallel Hidden Markov Models Conventional HMMs are suited to handle sequential signals (see section 2.1.3.6). However, after the breakdown into feature groups, we now deal with parallel signals which can be handled by Parallel Hidden Markov Models (PaHMMs) [9]. Parallel Hidden Markov Models are conventional HMMs used in parallel. Each of the parallel combined HMMs is called a channel. Each channel is independent from each other, the state probabilities of one channel do not influence any of the other channels.
Recognition of continuous Sign Language using Subunits
SignLexicon ‘Pos’
Sign 1 Sign 2. . .
Sign M
123
SU4 SU7 SU3 SU6 SU1 SU5 . . .
SU2 SU7 SU5 Sign 1
SignLexicon Size
SignLexicon Dist
Sign 1 Sign 2.
SU1 SU9 SU7 SU5 SU8 SU2
Sign 2
Sign M
SU5 SU1 SU3
Sign M
Sign 1 Sign 2.
SU3 SU1 SU2 SU1 SU6 SU3
. .
. .
Sign M
. . .
SU4 SU 7 SU 3 Pos SU1 SU 9 SU 7 Size SU3 SU 1 SU 2 Dist SU6 SU 1 SU 5 Pos SU5 SU . 8 SU 2 Size SU1 SU . 6 SU 3 Dist
. . .
.
SU2 SU 7 SU 5 Pos SU5 SU 1 SU 3 Size SU2 SU 4 SU 1 Dist
SignLexicon
. . .
SU2 SU4 SU1
Fig. 3.26. Each feature group leads to an own sign-lexicon, which are finally combined to one sign-lexicon.
ESSEN
Same Position
RED
EAT
Same Distance
HOW MUCH
SALAT
Same Size
LIKE
TASTE
Fig. 3.27. Different examples for the assignment to all different feature groups.
An accordant PaHMM is depicted in figure 3.28. The last state, a so called confluent state, combines the probabilities of the different channels to one probability, valid for the whole sign. The combination of probabilities is determined by the following equation: J P (O j |λj ) . (3.12) P (O|λ) = j=1
O j stands here for the relevant observation sequence of one channel, which is evaluated for the accordant segment. All parallel calculated probabilities of a sign finally
124
Sign Language Recognition s11
s12
s13
s41
s12
s22
s32
s42
s13
s32
s33
s34
Fig. 3.28. Example of a PaHMMs with Bakis-Topology.
result in the following equation: P (O|λ) =
K J
P (O kj |λkj ) .
(3.13)
j=1 k=1
3.3.4.1 Modeling Sign Language by means of PaHMMs For recognition based on subunit models each of the feature groups is modeled by one channel of the PaHMMs (see also 3.3.3). The sequential subdivision into subunits is then conducted in each feature group separately. Figure 3.29 depicts the modeling of the GSL sign HOW MUCH (WIEVIEL) with its three feature groups. The figure shows the word model of the sign, i.e. the sign and all its contained subunits in all three feature groups. Note that it is possible and highly probable that the different feature groups of a sign contain different numbers of subunit models. This is the case if, as in this example, the position changes during the execution of the sign, whereas the handshape and -orientation remains the same.
s0
s1
s2
s3
s1
s2
s3
s1
s2
s3
s4
s5
s4
s5
s6
s7
s8
s9
Pos s0
Size s0
Dist Gebärde WIEVIEL Sign HOW MUCH
Fig. 3.29. Modeling the GSL sign HOW MUCH (WIEVIEL) by means of PaHMMs.
Figure 3.29 also illustrates the specific topology for a subunit based recognition system. As the duration of one subunit is quite short, a subunit HMM consists merely
Recognition of continuous Sign Language using Subunits
125
of two states. The connection of several subunits inside one sign depend however on Bakis topology (see also equation 2.69 in section 2.1.3.6). 3.3.5 Classification ˆ which fits the feature vector sequence Determining the most likely sign sequence S best X results in a demanding search process. The recognition decision is carried out by jointly considering the visual and linguistic knowledge sources. Following [1], where the most likely sign sequence is approximated by the most likely state sequence, a dynamic programming search algorithm can be used to compute the probabilities P (X|S) · P (S). Simultaneously, the optimization over the unknown sign sequence is applied. Sign language recognition is then solved by matching the input feature vector sequence to all the sequences of possible state sequences and finding the most likely sequence of signs using the visual and linguistic knowledge sources. The different steps are described in more detail in the following subsections. 3.3.5.1 Classification of Single Signs by Means of Subunits and PaHMMs Before dealing with continuous sign language recognition based on subunit models, the general approach will be demonstrated by a simplified example of single sign recognition with subunit models. The extension to continuous sign language recognition is then given in the next section. The general approach is the same in both cases. The classification example is depicted in figure 3.30. Here, the sign consists of three subunits for feature group ’size’, four subunits for ’pos’ and eventually two for feature group ’dist’. It is important to note, that the depicted HMM is not a random sequence of subunits in each feature group, nor is it a random parallel combination of subunits. The combination of subunits – in parallel as well as in sequence – depends on a trained sign, i.e. the sequence of subunits is transcribed in the sign-lexicon of the accordant feature-group. Furthermore, the parallel combination, i.e. the transcription in all three sign-lexica codes the same sign. Hence, the recognition process does not search ’any’ best sequence of subunits independently. The signal of the example sign of figure 3.30 has a total length of 8 feature-vectors. In each channel an assignment of feature-vectors (the part of the feature vector of the accordant feature group) to the different states happens entirely independent from each other by time alignment (Viterbi algorithm). Only at the end of the sign, i.e. after the 8th feature vector is assigned, the so far calculated probabilities of each channel are combined. Here, the first and last states are confluent states. They are not emitting any probability, as they serve as a common beginning- and end-state for the three channels. The confluent end-state can only be reached by the accordant end-states of all channels. In the depicted example this is the case only after feature vector 8, even though the end-state in channel ’dist’ is already reached after 7 times steps. The corresponding equation for calculating the combined probability for one model is: (3.14) P (X|λ) = P (X|λP os ) · P (X|λSize ) · P (X|λDist ). The decision on the best model is reached by a maximisation over all models of the signs of the vocabulary:
Sign Language Recognition
Size
Pos
Dist
126
Sequence of MerkmalsFeature Vectors vektorfolge
t=1
t=2
t=3
t=4
t=5
t=6
t=7
t=8
x1
x2
x3
x4
x5
x6
x7
x8
Time t Zeit
t
Fig. 3.30. Classification of a single sign by means of subunits and PaHMMs.
Sˆ = argmax P (X|λi ).
(3.15)
i
After completion of the training process, a word model λi exists for each sign S i , which consists of the Hidden Markov Models of the accordant subunits. This word model will serve as reference for recognition. 3.3.5.2 Classification of Continuous Sign Language by Means of Subunits and PaHMMs In principle the classification process for continuous and isolated sign language recognition is identical. However, in contrast to the recognition of isolated signs, continuous sign language recognition is concerned with a number of further difficulties, such as: • • • •
A sign may begin or end anywhere in a given feature vector sequence. It is ambiguous, how many signs are contained in each sentence. There is no specific order of given signs. All transitions between subsequent signs are to be detected automatically.
All these difficulties are strongly linked to the main problem: The detection of sign boundaries. Since these can not be detected accurately, all possible beginning and
Recognition of continuous Sign Language using Subunits
127
Sign N N Gebärde
Size
Pos Dist Size
. . .
Gebärde i Sign i
Size Pos Dist
Gebärde i Sign i
Sign 1 1 Gebärde
Gebärde N
Size Pos Dist
Size Pos Dist
. . .
. . .
Size Pos Dist
Size Pos Dist
Sign 1 1 Gebärde
Size Pos Dist
Size
Pos Dist
Pos Dist
end points have to be accounted for. As introduced in the last section, the first and last state of a word model is a confluent common state for all three channels. Starting from the first state, the feature vector is divided into feature groups for the different channels of the PaHMM. From the last joint confluent state of a model exists a transition to first confluent states of all other models. This scheme is depicted in figure 3.31.
. . .
. . .
Gebärde 1 Sign 1
Gebärde i Sign i
Sign N Gebärde N
Fig. 3.31. A Network of PaHMMs for Continuous Sign Language Recognition with Subunits.
Detection of Sign Boundaries At the time of classification, the number of signs in the sentence as well as the transitions between these signs are unknown. To find the correct sign transitions all models of signs are combined as depicted in figure 3.31. The generated model now constitutes one comprehensive HMM. Inside a sign-model there are still three transitions (Bakis-Topology) between states. The last confluent state of a sign model has transitions to all other sign models. The Viterbi algorithm is employed to determine the best state sequence of this three-channel PaHMM. Hereby the assignment of feature vectors to different sign models becomes obvious and with it the detection of sign boundaries. 3.3.5.3 Stochastic Language Modeling The classification of sign language depends on two knowledge sources: the visual model and the language model. Visual modeling is carried out by using HMMs as described in the previous section. Language modeling will be discussed in this section. Without any language model technology, the transition probabilities between two successive signs are equal. Knowledge about a specific order of the signs in the training corpus is not utilised during recognition. In contrast, a statistical language model takes advantage of the knowledge, that pairs of signs, i.e. two successive signs, occur more often than others. The following equation gives the probability of so called bigramm-models.
128
Sign Language Recognition P (w) = P (w1 ) ·
m
P (wi |wi−1 )
(3.16)
i=2
The equation describes the estimation of sign pairs wi−1 and wi . During the classification process the probability of a subsequent sign changes, depending on the classification result of the preceding sign. By this method, typical sign pairs receive a higher probability. The estimation of these probabilities requires however a huge training corpus. Unfortunately, since training corpora do not exist for sign language, a simple but efficient enhancement of the usual statistical language model is introduced in the next subsection. Enhancement of language models for sign language recognition The approach of an enhanced statistical language model for sign language recognition is based on the idea of dividing all signs of the vocabulary into different sign-groups (SG) [2]. The determination of occurance probabilities is then calculated between these sign-groups and not between specific signs. If a combination of different sign-groups is not seen in the training corpus, this is a hint, that signs of these specific sign-groups do not follow each other. This approach does not require that all combinations of signs occur in the data base. If the sequence of two signs of two different sign-groups SGi and SGj is observed in the training corpus, any sign of sign-group SGi followed by any other sign of sign-group SGj is allowed for recognition. For instance, if the sign succession ’I EAT’ is contained in the training corpus, the probability that a sign of group ’verb’ (SGverb ) occurs when a sign of sign-group ’personal pronoun’ (SGpersonalpronoun ) was already seen, is increased. Hereby, the occurrence of the signs ’YOU DRINK’ receives a high probability even though this succession does not occur in the training corpus. On the other hand, if the training corpus does not contain a sample of two succeeding ’personal pronoun’ signs (e.g. ’YOU WE’), it is a hint that this sequence is not possible in sign language. Therefore, the recognition of these two succeeding signs is excluded from the recognition process. By this modification, a good compromise between statistical and linguistic language modeling is found. The assignment to specific sign-groups is mainly motivated by word categories known from German Speech Grammar. Sign-groups are ’nouns’, ’personal pronouns’, ’verbs’, ’adjectives’, ’adverbs’, ’conjunctions’, ’modal verbs’, ’prepositions’ and two additional groups, which take the specific sign language characteristics into account. 3.3.6 Training This section finally details the training process of subunit models. The aim of the training phase is the estimation of all subunit model parameters λ = (Π, A, B) for each sign of the vocabulary. It is important to note, that the subunit-model parameters are based on data recorded by single signs. The signs are not signed in context of
Recognition of continuous Sign Language using Subunits
129
a whole sign language sentences, which is usually the case for continuous sign language recognition and is then called embedded training. As only single signs serve as input for the training, this approach leads to a decreased training complexity. The training for subunits of signs proceeds in four iterative different steps. An overview is depicted in figure 3.32. In a first step a transcription for all signs has to be determined for the sign-lexicon. This step is called initial transcription (see section 3.3.6.1). Hereafter, the model parameter are estimated (estimation step, see section 3.3.6.2), which leads to to preliminary models for subunits. A classification step follows (see section 3.3.6.3). Afterwards, the accordant results has to be reviewed in an evaluation step. The result of the whole process is also known as singleton fenonic base form respectively synthetic base forms [11]. However, before the training process starts, the feature vector has to be split to consider the different feature groups. Split of feature-vector The partition of the feature vector into three different feature groups (pos, size, dist) has to be considered during the training process. Hence, as a first step before the actual training process starts, the feature vector is split into the three feature groups. Each feature group can be considered as a complete feature vector and is trained separately, since they are (mostly) independent from each other. Each training step is performed for all three feature groups. The training process eventually leads to subunit-models λpos , λsize and λdist . The following training steps are described for one of the three feature groups. The same procedure applies to the two remaining feature groups. The training process eventually leads to three different sign-lexica, where the transcription for all signs in the vocabulary are listed with respect to the corresponding feature-group. 3.3.6.1 Initial Transcription To start the training process, a sign-lexicon is needed which does not yet exist. Hence, an initial transcription of the signs for the sign-lexicon is needed before the training can start. This choice of an initial transcription is arbitrary and of little importance for the success of the recognition system, as the transcription is updated iteratively during training. Even with a poor initial transcription, the training process may lead to good models for subunits of signs [11]. Two possible approaches for the initial transcription are given in the following. 1. Clustering of Feature Vectors Again, this step is undertaken for each of the feature group data separately. The existing data is divided into K different cluster. Each vector is assigned to one of the K cluster. Eventually, all feature vectors belonging to the same cluster are similar, which is one of the goals of the initial transcription. A widespread cluster method is the K-Means-Algorithm [7]. After processing of the K-means algorithm, a assignment of the feature vectors to the clusters is fixed. Each cluster is also associated
130
Sign Language Recognition
Initial Transcription
Estimation for Model-parmeters for Subunits
Trainingsdata (Feature Vectors of single signs)
Sign 1 Sign 2
SU SU
. . .
. . .
Sign M
SU
6
SU 7 SU SU 1 SU
2
SU 7 SU
4
3 5
5
Sign-Lexicon Model-parameter of Subunits
Classification of single signs based on the Models for Subunits
Sign 1
Recognition Result
SU SU SU
6
SU 7 SU 5 SU 5 SU . 1 SU 5
6
SU
1 6
Sign M
SU
2
SU
. .
SU
1
SU
7
SU
5
7
SU
5
. . .
. . . SU
5
9
SU
Evaluation of preliminary Recognition Results
next Iteration
Sign 1 Sign 2
Training Result
Model-parameter of Subunits
. . . Sign M
SU SU
4
. . . SU
6
SU SU
2
SU
7 1
SU SU
7
SU
3 5
5
Final Sign-Lexicon
Fig. 3.32. The different processing steps for the identification of subunit models.
with a unique index number. For an initial transcription in the sign-lexicon, these index numbers have to be coded for each feature vector into the sign-lexicon. An example is depicted in figure 3.33. Experiments show, that an inventory of K = 200 is reasonable for sign recognition.
Recognition of continuous Sign Language using Subunits
131
THIS x.
x.
x
x
.. ... .. .
11
1D
X1
.. ... ...
21
HOW MUCH 2D
x.
x.
x.
x.
x
x
x
x
X3
X4
X5
X6
... ... ..
X2
31
3D
.. ... .. .
41
.. ... .. .
4D
51
.. ... .. .
5D
61
COST x.
x.
x.
x
x
x
X7
X8
X9
6D
... ... ..
71
... ... ..
7D
x. ... ... ..
11
x
1D
X1
x. .. ... .. .
81
8D
... ... ..
91
9D
41
x
4D
X4
x. .. ... ...
x.
91
.. ... .. .
9D
X9
C1 (SU1)
... ... ..
x.
x
.. ... .. .
. ... ... .
x
2D
X2
. ... ... .
31
X7
x. . ... ... .
8D
x.
5D
X5
7D
81
x
21
X8
51
x
x
x.
x.
71
C2 (SU2)
61
x
6D
x
3D
X6
X3 C3(SU3)
THIS
SU1
SU2
HOW MUCH
SU2
SU1
SU3 SU2
COST
SU3
SU2
SU1
Sign-Lexicon
Fig. 3.33. Example for clustering of feature vectors for a GSL sign language sentence THIS HOW MUCH COST (DAS KOSTEN WIEVIEL).
2. Evenly Division of Signs into Subunits For this approach, each sign is divided in a fixed number of subunits, e.g. six, per sign and feature group. Accordingly, sign 1 is associated with subunit 1 - 6, sign 2 with subunit 7 - 12 and so forth. However, as it is not desirable to have 6 times more subunits than signs in the vocabulary, not all signs of the vocabulary are chosen. Instead only a subgroup, i.e. the first approximately 30 signs are chosen at random for the initial transcription, which finally leads to approximately 200 subunits. 3.3.6.2 Estimation of Model Parameters for Subunits The estimation of model parameters is actually the core of this training process, as it leads eventually to the parameters of an HMM λi for each subunit. However, to start with, the data of single signs (each sign with m repetitions) serve here as input. In addition, the transcription of signs coded in the sign-lexicon (which is in the first iteration the result of the former initial transcription step) is needed. In all following steps, the transcription of the prior iteration step is the input for the actual iteration. Before the training starts, the number of states per sign has to be determined. The total number of states for a sign is equal to the number of contained subunits of the sign times two, as each subunit consists of two states. Viterbi training, which has been already introduced in section 2.1.3.6 is employed to estimate the model parameters for subunits. Thus, only the basic approach will here be introduced, for further details and equations see section 2.1.3.6. The final result
132
Sign Language Recognition
UE SU5
UE SU19
UE SU1
UE7 SU
of this step are HMMs for subunits. However, as data for single signs are recorded, we are again concerned with the problem of finding boundaries, here the boundaries of subunits within signs. Figure 3.34 illustrates how these boundaries are found.
Feature-
MerkmalsVectors vektorfolge
t=1
t=2
t=3
t=4
t=5
t=6
t=7
Zeit t t=8 Time
x1
x2
x3
x4
x5
x6
x7
x8
UE SU5
UE SU19
UE SU1
UE SU7
Fig. 3.34. Example for the detection of subunit boundaries. After time alignment the assignment of feature vectors to states is obvious.
The figure details the situation after time alignment by the Viterbi algorithm performed on a single sign. Here, the abscissa presents the time and thereby the feature vector sequence, whereas the concatenated models for each subunit contained in the sign are depicted at the ordinate. After the alignment of each feature vector sequence to the states, the detection of subunit boundaries is obvious. In this example the first two feature vectors are assigned to the states of the model SU5 , the following two are assigned to subunit SU1 9, one to SU1 and finally the last three to subunit SU7 . Now the Viterbi algorithm has to be performed again, this time on the feature vectors assigned to each subunit, in order to identify the parameters of an HMM λi for one subunit. 3.3.6.3 Classification of Single Signs The same data of single signs, which already served as input to the training process is now classified to proof the quality of the estimated subunit models (of the previous step). It is expected that most recognition results are similar to the transcription coded in the sign-lexicon. However, not all results will comply with the original transcription, which is especially true if data of several, e.g. m, repetitions of one sign exist. As the recognition results of all repetitions of a single sign are most probable not identical, the best representation for all repetitions of that specific sign has to be
Recognition of continuous Sign Language using Subunits
133
found. To reach this goal, an HMM for each of the m repetition is constructed. The Hidden Markov Models is based on the recognition result of the last mentioned step. The HMMs of the accordant subunit models are concatenated to a word model for that repetition of the sign. For instance, if the recognition result of a repetition of a sign1 is SU4 , SU1 , SU7 and SU9 , the resulting word model for that repetition is the concatenation of SU4 ⊕ SU1 ⊕ SU7 ⊕ SU9 . λsi refers here to the ith repetition of a sign s. We start with the evaluation of the word model λs1 of the first repetition. For this purpose, a probability is determined, that the feature vector sequence of a specific repetition, say s2 , is produced by the model (λs1 ). This procedure leads eventually to m different probabilities (one for each of the m repetitions) for the model λs1 . The multiplication of all these probabilities finally results in an indication of how ’good’ this model fits to all m repetitions. For the remaining repetitions s2 , s3 , . . . , sm of that sign the proceeding is analogous. Finally, the model of that repetition si is chosen, which results in the highest probability. This probability is determined by: M
M
j=1
i=1
C ∗ (w) = argmax
PAj (w) (Ai (w)).
(3.17)
Eventually, the transcription of the most probable repetition si is coded in the signlexicon. If the transcription of this iteration step differs from the previous iteration step, another iteration of the training will follow. 3.3.7 Concatenation of Subunit Models to Word Models for Signs. By determination of subunit models the basic training process is complete. For continuous sign language recognition ’legal’ sequences of subunits must be composed to signs. It may however happen, that an ’illegal’ sequence of subunits, i.e. a sequence of subunits, which is not coded in the sign-lexicon, is recognised. To avoid this misclassification, the subunits are concatenated according to their transcription in the sign-lexicon. For example the model λSi for sign S with a transcription of SU6 , SU1 , SU4 is then defined by λSi = λUE6 ◦ λUE1 ◦ λUE4 . 3.3.8 Enlargement of Vocabulary Size by New Signs A data-driven approach of subunit-models gives the opportunity to increase the vocabulary size by non-trained signs, as simply a new entry in the sign-lexicon has to be determined. An additional training step is not needed for the new signs. The approach is similar to the approach described in section 3.3.6, the training of subunit models [11]. For new signs, i.e. signs which have so far not been included in the vocabulary, it has to be determined, which sequence of already trained subunit models is able to represent the specific sign. Figure 3.35 illustrates this process. To determine a new entry to the sign-lexicon at least one performance of the sign is needed. However, for a better modeling, more repetitions of that sign are desirable. In a first step the new sign is recognised. For this recognition process, the
134
Sign Language Recognition New Signs (Sequence of Feature Vectors of Single Signs)
Model-Parameter of Subunits
Classification of Single Signs Reference: Models of Subunits
Sign 1
Recognition Result
SU SU SU
1 6 6
SU 7 SU 5 SU 5 SU . 1 SU 5
Sign M
SU
2
6
7
SU
5
. . .
. . .
. .
SU
SU
SU 1 SU 5
SU
9
SU
7
SU
5
Evaluation of preliminary Recognition Results
Sign 1 Sign 2
SU SU
4
. . .
. . .
Sign M
SU
6
SU 7 SU SU 1 SU
2
SU
7
SU
3 5
5
Sign-Lexicon including new Signs
Fig. 3.35. Steps to enlarge the vocabulary by new, non-trained signs.
trained subunit models serve as reference for classification. However, if more than one repetition of the sign is available it is most unlikely, that all repetitions lead to the same recognition result. The worst case represents n different results for n different sign repetitions. To solve this problem, a further evaluation step, which is the same as the one in the training process, is required. Here, for all repetitions of the sign, the ’best’ recognition result (in terms of all repetitions) is chosen and added to the sign-lexicon.
3.4 Performance Evaluation In this section some performance data that has been achieved with the video based sign language recognizer described above are presented. Person dependent as well
Performance Evaluation
135
as person independent recognition rates are given for single signs under controlled laboratory condition as well as under real world condition. Recognition results are also presented as to the applicability of subunits, which were automatically derived from signs. 3.4.1 Video-based Isolated Sign Recognition As a first result Tab. 3.2 shows the person-dependent recognition rates from a leaving-one-out test for four signers and various video resolutions under controlled laboratory conditions. The training resolution was always 384 × 288. Since the number of recorded signs varies slightly the vocabulary size is specified separately for each signer. Only manual features were considered. Interestingly COG coordinates alone accounted for approx. 95% recognition of a vocabulary of around 230 signs. On a 2 GHz PC, processing took an average of 11.79s/4.15s/3.08s/2.92s per sign, depending on resolution. Low resolutions caused only a slight decrease in recognition rate but reduced processing time considerably. So far a comparably high performance has only been reported for intrusive systems. Table 3.2. Person-dependent isolated sign recognition with manual features in controlled environments. Test Signer, Vocabulary Size Video Features Ben Michael Paula Sanchu Resolution 235 signs 232 signs 219 signs 230 signs 229 signs 384 × 288 all 98.7% 99.3% 98.5% 99.1% 98.9% 192 × 144 all 98.5% 97.4% 98.5% 99.1% 98.4% 128 × 96 all 97.7% 96.5% 98.3% 98.6% 97.8% 96 × 72 all 93.1% 93.7% 97.1% 95.9% 94.1% 384 × 288 x, x, ˙ y, y˙ 93.8% 93.9% 95.5% 96.1% 94.8%
Under the same conditions head pose, eyebrow position, and lip outline were employed as nonmanual features. The recognition rates achieved on the vocabularies presented in Tab. 3.2 varied between 49,3% and 72,4% among the four signers, with an average of 63,6%. Hence roughly two of three signs were recognized just from nonmanual features, a result that emphasizes the importance of mimics for sign language recognition. Tab. 3.3 shows results for person-independent recognition. Since the signers used different signs for some words, the vocabulary has been chosen as the intersection of the test signs with the union of all training signs. No selection has been performed otherwise, and no minimal pairs have been removed. As expected, performance drops significantly. This is caused by strong interpersonal variance in signing, as visualized in Fig. 3.5 by COG traces for identical signs done by different signers. Recognition rates are also affected by the exact constellation of training/test signers.
136
Sign Language Recognition
Table 3.3. Person-independent isolated sign recognition with manual features in controlled environments. The n-best rate indicates the percentage of results for which the correct sign was among the n signs deemed most similar to the input sign by the classifier. For n = 1 this corresponds to what is commonly specified as the recognition rate. Training Signer(s) Michael Paula, Sanchu Ben, Paula, Sanchu Ben, Michael, Paula Ben, Michael, Sanchu Michael, Sanchu
Test Vocabulary n-Best Rate Signer Size 1 5 10 Sanchu 205 36.0% 58.0% 64.9% Michael 218 30.5% 53.6% 63.2% Michael 224 44.5% 69.3% 77.1% Sanchu 221 54.2% 79.4% 84.5% Paula 212 37.0% 63.6% 72.8% Ben 206 48.1% 70.0% 77.4%
Person-independent performance in uncontrolled environments is difficult to measure since it depends on multiple parameters (signer, vocabulary, background, lighting, camera). Furthermore, noise and outliers are inevitably introduced in the features when operating in real-world settings. Despite the large interpersonal variance, person-independent operation is feasible for small vocabularies, as can be seen in Tab. 3.4. Each test signer was recorded in a different real-life environment, and the selection of signs is representative of the complete vocabulary (it contains onehanded and two-handed signs, both with and without overlap). Table 3.4. Person-independent recognition rates in real life environments. Vocabulary Test Signer Size Christian Claudia Holger J¨org Markus Ulrich 6 96.7% 83.3% 96.7% 100% 100% 93.3% 95.0% 18 90.0% 70.0% 90.0% 93.3% 96.7% 86.7% 87.8%
3.4.2 Subunit Based Recognition of Signs and Continuous Sign Language Recognition of continuous sign language requires a prohibitive amount of training material if signed sentences are used. Therefore the automated transcription of signs to subunits described in Sect. 3.3 was used here, which is expected to reduce the effort required for vocabulary extension to a minimum. For 52 signs taken arbitrarily from the domain ”shopping” an automatic transcription was performed, which resulted in 184 (Pos), 184 (Size) and 187 (Dist) subunits respectively. Subsequently performed classification tests with signs synthesized from the identified subunits yielded ¿ 90% correct recognition. This result shows, the the automatic segmentation of signs produced reasonable subunits. Additionally, 100 continuous sign language sentences were created using only signs synthesized from subunits. Each sentence
References
137
consisted of two to nine signs. After eliminating coarticulation effects and applying bigrams according to the statistical probability of sign order, a recognition rate of 87% was achieved. This finding is very essential as it solves the problem of vocabulary extension without additional training. 3.4.3 Discussion In this chapter a concept for video based sign language recognition was described. Manual and non-manual features have been integrated into the recognizer to cover all aspects of sign language expressions. Excellent recognition performance has been achieved for person-dependent classification and medium sized vocabularies. The presented system is also suitable for person-independent real world applications where small vocabularies suffice, as e.g. for controlling interactive devices. The results also show that a reasonable automatic transcription of signs to subunits is feasible. Hence the extension of an existing vocabulary is made possible without the need for large amounts of training data. This constitutes a key feature in the development of sign language recognition systems supporting large vocabularies.
References 1. Bahl, L., Jelinek, F., and Mercer, R. A Maximum Likelihood Approach to Continuous Speech Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 5(2):179–190, March 1983. 2. Bauer, B. Erkennung kontinuierlicher Geb¨ardensprache mit Untereinheiten-Modellen. Shaker Verlag, Aachen, 2003. 3. Becker, C. Zur Struktur der deutschen Geb¨ardensprache. WVT Wissenschaftlicher Verlag, Trier (Germany), 1997. 4. Canzler, U. and Dziurzyk, T. Extraction of Non Manual Features for Videobased Sign Language Recognition. In The University of Tokyo, I. o. I. S., editor, Proceedings of IAPR Workshop on Machine Vision Applications, pages 318–321. Nara, Japan, December 2002. 5. Canzler, U. and Ersayar, T. Manual and Facial Features Combination for Videobased Sign Language Recognition. In 7th International Student Conference on Electrical Engineering, page IC8. Prague, May 2003. 6. Derpanis, K. G. A Review of Vision-Based Hand Gestures. Technical report, Department of Computer Science, York University, 2004. 7. Duda, R., Hart, P., and Stork, D. Pattern Classifikation. Wiley-Interscience, New York, 2000. 8. Fang, G., Gao, W., Chen, X., Wang, C., and Ma, J. Signer-Independent Continuous Sign Language Recognition Based on SRN/HMM. In Revised Papers from the International Gesture Workshop on Gestures and Sign Languages in Human-Computer Interaction, pages 76–85. Springer, 2002. 9. Hermansky, H., Timberwala, S., and Pavel, M. Towards ASR on Partially Corrupted Speech. In Proc. ICSLP ’96, volume 1, pages 462–465. Philadelphia, PA, 1996. 10. Holden, E. J. and Owens, R. A. Visual Sign Language Recognition. In Proceedings of the 10th International Workshop on Theoretical Foundationsof Computer Vision, pages 270–288. Springer, 2001.
138
Sign Language Recognition
11. Jelinek, F. Statistical Methods for Speech Recognition. MIT Press, 1998. ISBN 0-26210066-5. 12. Jones, M. and Rehg, J. Statistical Color Models with Application to Skin Detection. Technical Report CRL 98/11, Compaq Cambridge Research Lab, December 1998. 13. KaewTraKulPong, P. and Bowden, R. An Improved Adaptive Background Mixture Model for Realtime Tracking with Shadow Detection. In AVBS01. 2001. 14. Liang, R. H. and Ouhyoung, M. A Real-time Continuous Gesture Interface for Taiwanese Sign Language. In UIST ‘97 Proceedings of the 10th Annual ACM Symposium on User Interface Software and Technology. ACM, Banff, Alberta, Canada, October 14-17 1997. 15. Murakami, K. and Taguchi, H. Gesture Recognition Using Recurrent Neural Networks. In Proceedings of the SIGCHI conference on Human factors in computingsystems, pages 237–242. ACM Press, 1991. 16. Ong, S. C. W. and Ranganath, S. Automatic Sign Language Analysis: A Survey and the Future Beyond Lexical Meaning. IEEE TPAMI, 27(6):873–8914, June 2005. 17. Parashar, A. S. Representation And Interpretation Of Manual And Non-manual InformationFor Automated American Sign Language Recognition. Phd thesis, Department of Computer Science and Engineering, College of Engineering,University of South Florida, 2003. 18. Porikli, F. and Tuzel, O. Human Body Tracking by Adaptive Background Models and Mean-Shift Analysis. Technical Report TR-2003-36, Mitsubishi Electric Research Laboratory, July 2003. 19. Sonka, M., Hlavac, V., and Boyle, R. Image Processing, Analysis and Machine Vision. Brooks Cole, 1998. 20. Starner, T., Weaver, J., and Pentland, A. Real-Time American Sign Language Recognition Using Desk and WearableComputer Based Video. IEEE Transactions on Pattern Analysis and Machine Intelligence,, 20(12):1371–1375, December 1998. 21. Stauffer, C. and Grimson, W. E. L. Adaptive Background Mixture Models for Real-time Tracking. In Computer Vision and Pattern Recognition 1999, volume 2. 1999. 22. Stokoe, W. Sign language structure: An Outline of the Visual Communication Systems of the American Deaf. (Studies in Linguistics. Occasional Paper; 8) University of Buffalo, 1960. 23. Sutton, V. http://www.signwriting.org/, 2003. 24. Vamplew, P. and Adams, A. Recognition of Sign Language Gestures Using Neural Networks. In European Conference on Disabilities, Virtual Reality and AssociatedTechnologies. 1996. 25. Vogler, C. and Metaxas, D. Parallel Hidden Markov Models for American Sign Language Recognition. In Proceedings of the International Conference on Computer Vision. 1999. 26. Vogler, C. and Metaxas, D. Toward Scalability in ASL Recognition: Breaking Down Signs into Phonemes. In Braffort, A., Gherbi, R., Gibet, S., Richardson, J., and Teil, D., editors, The Third Gesture Workshop: Towards a Gesture-Based Communication in Human-Computer Interaction, pages 193–204. Springer-Verlag Berlin, Gif-sur-Yvette (France), 2000. 27. Welch, G. and Bishop, G. An Introduction to the Kalman Filter. Technical Report TR 95-041, Department of Computer Science, University of North Carolina at Chapel Hill, 2004. 28. Yang, M., Ahuja, N., and Tabb, M. Extraction of 2D Motion Trajectories and Its Application to HandGesture Recognition. IEEE Transactions On Pattern Analysis And Machine Intelligence, 24:1061–1074, 2002.
References
139
29. Zieren, J. and Kraiss, K.-F. Robust Person-Independent Visual Sign Language Recognition. In Proceedings of the 2nd Iberian Conference on Pattern RecognitionandImage Analysis IbPRIA 2005, volume Lecture Notes in Computer Science. 2005.
Chapter 4
Speech Communication and Multimodal Interfaces Bj¨orn Schuller, Markus Ablaßmeier, Ronald M¨uller, Stefan Reifinger, Tony Poitschke, Gerhard Rigoll Within the area of advanced man-machine interaction, speech communication has always played a major role for several decades. The idea of replacing the conventional input devices such as buttons and keyboard by voice control and thus increasing the comfort and the input speed considerably, seems that much attractive, that even the quite slow progress of speech technology during those decades could not discourage people from pursuing that goal. However, nowadays this area is in a different situation than in those earlier times, and these facts shall be also considered in this book section: First of all, speech technology has reached a much higher degree of maturity, mainly through the technique of stochastic modeling which shall be briefly introduced in this chapter. Secondly, other interaction techniques became more mature, too, and in the framework of that development, speech became one of the preferred modalities of multimodal interaction, e.g. as ideal complementary mode to pointing or gesture. This shall be also reflected in the subsection on multimodal interaction. Another relatively recent development is the fact that speech is not only a carrier of linguistic information, but also one of emotional information, and emotions became another important aspect in today’s advanced man-machine interaction. This will be considered in a subsection on affective computing, where this topic is also consequently investigated from a multimodal point of view, taking into account the possibilities for extracting emotional cues from the speech signal as well as from visual information. We believe that such an integrated approach to all the above mentioned different aspects is appropriate in order to reflect the newest developments in that field.
4.1 Speech Recognition This section is concerned with the basic principles of Automatic Speech Recognition (ASR). This research area has had a dynamic development during the last decades, and has been considered in its early stage as an almost unsolvable problem, then went through several evolution steps during the 1970s and 1980s, and eventually became more mature during the 1990s, when extensive databases and evaluation schemes became available that clearly demonstrated the superiority of stochastic machine learning techniques for this task. Although the existing technology is still far from being perfect, today there is a speech recognition market with a number of existing products that are almost all based on the before mentioned technology.
142
Speech Communication and Multimodal Interfaces
Our goal here is neither to describe the entire history of this development, nor to provide the reader with a detailed presentation of the complete state-of-the-art of the fundamental principles of speech recognition (which would probably require a separate book). Instead, the aim of this section is to present a relatively compact overview on the currently most actual and successful method for speech recognition, which is based on a probabilistic approach for modeling the production of speech using the technique of Hidden-Markov-Models (HMMs). Today, almost every speech recognition system, including laboratory as well as commercial systems, is based on this technology and therefore it is useful to concentrate in this book exclusively on this approach. Although this method makes use of complicated mathematical foundations in probability theory, machine learning and information theory, it is possible to describe the basic functionality of this approach using only a moderate amount of mathematical expressions which is the approach pursued in this presentation. 4.1.1 Fundamentals of Hidden Markov Model-based Speech Recognition The fundamental assumption of this approach is the fact that each speech sound for a certain language (more commonly called phoneme) is represented as a HiddenMarkov-Model, which is nothing else but a stochastic finite state machine. Similarly to classical finite state machines, an HMM consists of a finite number of states, with possible transitions between those states. The speech production process is considered as a sequence of discrete acoustic events, where typically each event is characterized by the production of a vector of features that basically describe the produced speech signal at the equivalent discrete time step of the speech production process. At each of these discrete time steps, the HMM is assumed to be in one of its states, where the next time step will let the HMM perform a transition into a state that can be reached from its current state according to its topology (including possible transitions back to its current state or formerly visited states).
Fig. 4.1. Example for a Hidden Markov Model.
Fig. 4.1 shows such an HMM, which has two major sets of parameters: The first set is the matrix of transition probabilities describing the probability p(s(k − 1) →
Speech Recognition
143
s(k)), where k is the discrete time index and s the notation for a state. These probabilities are represented by the parameters a in Fig. 4.1. The second set represents the so-called emission probabilities p(x|s(k)), which is the probability that a certain feature vector can occur while the HMM is currently in state s at time k. This probability is usually expressed by a continuous distribution function, denoted as function b in Fig. 4.1, which is in many cases a mixture of Gaussian distributions. If a transition into another state is assumed at discrete time k with the observed acoustic vector x(k), it is very likely that this transition will be into a state with a high emission probability for this observed vector, i.e. into a state that represents well the characteristics of the observed feature vector.
Fig. 4.2. Example for HMM-based speech recognition and training.
With these basic assumptions, it is already possible to formulate the major principles of HMM-based speech recognition, which can be best done by having a closer look at Fig. 4.2: In the lower part of this figure, one can see the speech signal of an utterance representing the word sequence ”this was”, also displayed in the upper part of this figure. This signal has been subdivided into time windows of constant length (typically around 10 ms length) and for each window a vector of features has been generated, e.g. by applying a frequency transformation to the signal, such as a Fourier-transformation or similar. This results in a sequence of vectors displayed just above the speech signal. This vector sequence is now ”observed” by the MarkovModel in Fig. 4.2, which has been generated by representing each phoneme of the underlying word sequence by a 2-state-HMM and concatenating the separate phoneme HMMs into one larger single HMM. An important issue in Fig. 4.2 is represented by the black lines that visualize the assignment of each feature vector to one specific state of the HMM. We have thus as many assignments as there are vectors in the feature vector sequence X = [x(1), x(2), ..., x(K)], where K is the number of vectors,
144
Speech Communication and Multimodal Interfaces
i.e. the length of the feature vector sequence. This so-called state-alignment of the feature vectors is one of the essential capabilities of HMMs and there are different algorithms for computation of this alignment, which shall not be presented here in detail. However, the basic principle of this alignment procedure can be made clear by considering one single transition of the HMM at discrete time k from state s(k−1) to state s(k), where at the same time the occurrence of feature vector x(k) is observed. Obviously, the joint probability of this event, consisting of the mentioned transition and the occurrence of vector x(k) can be expressed according to Bayes law as: p(x(k), s(k − 1) → s(k)) = p(x(k)|s(k − 1) → s(k)) · p(s(k − 1) → s(k)) = p(x(k)|s(k)) · p(s(k − 1) → s(k))
(4.1)
and thus is exactly composed out of the two parameter sets a and b mentioned before, that describe the HMM and are shown in Fig. 4.1 As already stated before, a transition into one of the next states will be likely, that results into a high joint probability as expressed in the above formula. One can thus imagine that an algorithm for computing the most probable state sequence as alignment to the observed feature vector sequence must be based on an optimal selection of a state sequence that eventually leads to the maximization of the product of all probabilities according to the above equation for k = 1 to K. A few more details on this algorithm will be provided later. For now, let us assume that the parameters of the HMMs are all known and that the optimal state sequence has been determined as indicated in Fig. 4.2 by the black lines that assign each vector to one most probable state. If that is the case, then this approach has produced two major results: The first one is a segmentation result that assigns each vector to one state. With this, it is for instance possible to determine which vectors – and thus which section of the speech signal – has been assigned to a specific phoneme, e.g. to the sound /i/ in Fig. 4.2 (namely the vectors nr. 6-9). The second result will be the before mentioned overall probability, that the feature vector sequence representing the speech signal has been assigned to the optimal state sequence. Since this probability will be the maximum possible probability and no other state sequence will lead to a larger value, this probability can be considered as the overall production probability that the speech signal has been produced by the underlying hidden Markov model shown in Fig. 4.2. Thus, an HMM is capable of processing a speech signal and producing two important results, namely a segmentation of the signal into subordinate units (such as e.g. phonemes) and the overall probability that such a signal can have been produced at all by the underlying probabilistic model. 4.1.2 Training of Speech Recognition Systems These results can be exploited in HMM-based speech recognition in the following manner: In the training phase, the HMM parameters in Fig. 4.2 are not known, but the transcription of the corresponding word sequence (here: ”what is” will be known). If some initial parameters for the large concatenated HMM of Fig. 4.2 are assumed
Speech Recognition
145
for the start of an iterative procedure, then it will be of course possible to compute with these parameters the optimal corresponding state sequence, as outlined before. This will be certainly not an optimal sequence, since the initial HMM parameters might not have been selected very well. However, it is possible to derive from that state sequence a new estimation for the HMM parameters by simply exploiting its statistics. This is quite obvious for the transition probabilities, since one only needs to count the occurring transitions between the states of the calculated state sequence and divide that by the total number of transitions. The same is possible for the probabilities p(x(k)|s(k)) which can basically be (without details) derived by observing which kind of vectors have been assigned to the different states and calculating statistics of these vectors, e.g. their mean values and variances. It can be shown that this procedure can be repeated iteratively: With the HMM parameters updated in the way as just described, one can now compute again a new improved alignment of the vectors to the states and from there again exploit the state sequence statistics for updating the HMM parameters. Typically, this leads indeed to a useful estimation of the HMM parameters after a few iterations. Moreover, at the end of this procedure, another important and final step can be carried out: By ”cutting” the large concatenated HMM in Fig. 4.2 again into the smaller phoneme-based HMMs, one obtains a single HMM for each phoneme which has now estimated and assigned parameters that represent well the characteristics of the different sounds. Thus, the single HMM for the phoneme /i/ in Fig. 4.2 will have certainly different parameters than the HMM for the phoneme /a/ in this figure and serves well as probabilistic model for this sound. This is basically the training procedure of HMMs and it becomes obvious, that the HMM technology allows the training of probabilistic models for each speech unit (typically the unit ”phoneme”) from the processing of entire long sentences without the necessity to phoneme-label or to pre-segment these sentences manually, which is an enormous advantage. 4.1.3 Recognition Phase for HMM-based ASR Systems Furthermore, Fig. 4.2 can also serve as suitable visualization for demonstrating the recognition procedure in HMM-based speech recognition. In this case – contrary to the training phase – now the HMM parameters are given (from the training phase) and the speech signal represents an unknown utterance, therefore the transcription in Fig. 4.2 is now unknown and shall be reconstructed by the recognition procedure. From the previous description, we have to recall once again that very efficient algorithms exist for the computation of the state-alignment between the feature vector sequence and the HMM-states, as depicted by the black lines in the lower part of Fig. 4.2. So far we have not looked at the details of such an algorithm, but shall go into a somewhat more detailed analysis of one of these algorithms now by looking at Fig. 4.3, which displays in the horizontal direction the time axis with the feature vectors x appearing at discrete time steps and the states of a 3-state HMM on the vertical axis. Assuming that the model starts in the first state, it is obvious that the first feature vector will be assigned to that initial state. Looking at the second feature vector,
146
Speech Communication and Multimodal Interfaces
Fig. 4.3. Trellis diagram for the Viterbi algorithm.
according to the topology of the HMM, it is possible that the model stays in state 1 (i.e. makes a transition from state 1 to state1) or moves with a transition from state 1 into state 2. For both options, the probability can be computed according to Eqn. 4.1 and both options are shown as possible path in Fig. 4.3. It is then obvious that from both path end points for time step 2, the path can be extended, in order to compute if the model has made a transition from state 1 (into either state 1 or state 2) or a transition from state 2 (into either state 2 or state 3). Thus, for the 3rd time step, the model can be in state 1, 2 or 3, and all these options can be computed with a certain probability, by multiplying the probabilities obtained for time step 2 by the appropriate transition and emission probabilities for the occurrence of the third feature vector. In this way it should be easily visible that the possible state sequence can be displayed in a grid (also called trellis) as shown in Fig. 4.3, which displays all possible paths that can be taken from state 1 to the final state 3 in this figure, by assuming the observation of five different feature vectors. The bold line in this grid shows one possible path through this grid and it is clear that for each path, a probability can be computed that this path has been taken, according to the before described procedure of multiplying the appropriate production probabilities of each feature vector according to Eqn. 4.1. Thus, the optimal path can be computed with the help of the principle of dynamic programming, which is well-known from the theory of optimization. The above procedure describes the Viterbi algorithm, which can be considered as one of the major algorithms for HMM-based speech recognition. As already mentioned several times, the major outcome of this algorithm is the optimal state sequence as well as the ”production probability” for this sequence, which is as well the probability that the given HMM has produced the associated feature vector sequence X. Let’s assume that the 3-state HMM in Fig. 4.3 represents a phoneme, as mentioned in the previous section on the training procedure, resulting in trained parameters for HMMs which typically represent all the phonemes in a given language. Typically, an unknown speech signal will either represent a spoken word or an entire spoken sentence. How can such a sentence then be recognized by the Viterbi algorithm as described above for the state-alignment procedure of an HMM representing a single
Speech Recognition
147
phoneme? This can be achieved by simply extending the algorithm so that it computes the most likely sequence of phoneme-HMMs that maximize the probability of emitting the observed feature vector sequence. That means that after the final state of the HMM in Fig. 4.3 has been reached (assuming a rather long feature vector sequence of which the end is not yet reached) another HMM (and possibly more) will be appended and the algorithm is continued until all feature vectors are processed and a final state (representing the end of a word) is obtained. Since it is not known which HMMs have to be appended and what will be the optimal HMM sequence, it becomes obvious that this procedure implies a considerable search problem and this process is therefore also called ”decoding”. There are however several ways to support this decoding procedure, for instance by considering the fact that the phoneme order within words is rather fixed and variation can basically only happen between word boundaries. Thus the phoneme search procedure is more a word search procedure which will be furthermore assisted by the so-called language model, that will be discussed later and assigns probabilities for extending the search path into a new word model if the search procedure has reached the final state of a preceding word model. Therefore, finally the algorithm can compute the most likely word sequence and the final result of the recognition procedure is then the recognized sentence. It should be noted that due to the special capabilities of the HMM approach, this decoding result can be obtained additionally with the production probability of that sentence, and with the segmentation result indicating which part of the speech signal can be assigned to the word and even to the phoneme boundaries of the recognized utterance. Some extensions of the algorithms make it even possible to compute the N most likely sentences for the given utterance (for instance for further processing of that result in a semantic module), indicating once again the power and elegance of this approach. 4.1.4 Information Theory Interpretation of Automatic Speech Recognition With this background, it is now possible to consider the approach to HMM-based speech recognition from an information theoretic point of view, by considering Fig. 4.4.
Fig. 4.4. Information theory interpretation of automatic speech recognition.
Fig. 4.4 can be interpreted as follows: A speaker formulates a sentence as a sequence of words denoted as W = [w(1), w(2), ..., w(N )]. He speaks that sentence into the microphone that captures the speech signal which is actually seen by the automatic speech recognizer. Obviously, this recognizer does not see the speaker’s
148
Speech Communication and Multimodal Interfaces
originally uttered word sequence, but instead sees the ”encoded” version of that in form of the feature vector sequence X that has been derived from the acoustic waveform, as output of the acoustic channel as displayed in Fig. 4.4. There is a probabilistic relation between the word sequence W and the observed feature vector sequence X, and indeed this probabilistic relation is modeled by the Hidden Markov Models that represent the phoneme sequence implied by the word sequence W . In fact, this model yields the already mentioned ”production probability” that the word sequence W , represented by the appropriate sequence of phoneme HMMs has generated the observed feature vector sequence and this probability can be denoted as p(X|W ). The second part of Fig. 4.4 shows the so-called ”linguistic decoder”, a module that is responsible for decoding the original information W from the observed acoustic sequence X, by taking into account the knowledge about the model provided by the HMMs expressed in p(X|W ). The decoding strategy of this module is to find the best possible word sequence W from observing the acoustic feature string X and thus to maximize p(W |X) according to Bayes’ rule as follows: max p(W |X) = max[p(X|W ) · W
p(W ) ] p(X)
(4.2)
And because finding the optimal word sequence W is independent of the probability p(X), the final maximization rule is: max[p(X|W ) · p(W )] W
(4.3)
Exactly this product of probabilities has to be maximized during the search procedure described before in the framework of the Viterbi algorithm. In this case, it should be noted that p(X|W ) is nothing else but the probability of the feature vector sequence X under the assumption that it has been generated by the underlying word sequence W , and exactly this probability is expressed by the HMMs resulting from the concatenation of the phoneme-based HMMs into a model that represents the resulting word string. In this way, the above formula expresses indeed the before mentioned decoding strategy, namely to find the combination of phoneme-based HMMs that maximize the corresponding emission probability. However, the above mentioned maximization rule contains an extra term p(W ) that has to be taken into account additionally to the so far known maximization procedure: This term p(W ) is in fact the ”sentence probability” that the word sequence W will occur at all, independent of the acoustic observation X. Therefore, this probability is described by the so-called language model, that simply expresses the occurrence probability of a word in a sentence given its predecessors, which can be expressed as: p(w(n)|w(n − 1), w(n − 2), ..., w(n − m))
(4.4)
In this case, the variable m denotes the ”word history”, i.e. the number of predecessor words that are considered to be relevant for the computation of the current word’s appearance probability. Then, the overall sentence probability can be expressed as the product of all single word probabilities in a sentence according to
Speech Recognition
p(W ) =
149 N
p(w(n)|w(n − 1), ..., w(n − m))
(4.5)
n=1
where N is the length of the sentence and m is the considered length of the word history. As already mentioned, the above word probabilities are completely independent of any acoustic observation and can be derived from statistics obtained e.g. from the analysis of large written text corpora by counting the occurrence of all words given a certain history of word predecessors. 4.1.5 Summary of the Automatic Speech Recognition Procedure
Fig. 4.5. Block diagram for HMM-based speech recognition.
Finally, to summarize the functioning of HMM-based speech recognition, the block diagram in Fig. 4.5 can be interpreted as follows: The speech signal is captured by a microphone, sampled and digitized. Preprocessing of the speech signal includes some filtering process and possible noise compensation. The next step in Fig. 4.5 is feature extraction, where the signal is split into windows of roughly 10 msec length and for each window, a feature vector is computed that typically represents the windowed signal in the frequency domain. Then, in recognition mode, for each vector of the resulting feature vector sequence, state conditional probabilities are computed, basically by inserting the feature vector x(k) into the right hand side of Eqn. 4.1 for each state which will be considered in the decoding procedure. According to Fig. 4.5, the gray-shaded table labeled as ”acoustic phoneme models”
150
Speech Communication and Multimodal Interfaces
contains the parameters of the distribution functions that are used to compute these state conditional probabilities. This computation is integrated into the already mentioned search procedure, that attempts to find the best possible string of concatenated HMM phoneme models that maximize the emission probability of the feature vector sequence. This search procedure is controlled by the gray-shaded table containing the phonological word models (i.e. how a word is composed of phonemes) and the table containing the language model that delivers probabilities for examining the next likely word if the search procedure has reached the final HMM-state of a previous word model. In this way, the above mentioned computation of state conditional probabilities does not need to be carried out for all possible states, but only for the states that are considered to be likely by the search procedure. The result of the search procedure is the most probable word model sequence that is displayed as transcription representing the recognized sentence to the user of the speech recognition system. This brief description of the basic functionality of HMM-based speech recognition can of course not cover this complicated subject in sufficient detail and it is therefore not amazing that many interesting sub-topics in ASR have not been covered in this introduction. These include e.g. the area of discriminative training techniques for HMMs, different HMM architectures such as discrete, hybrid and tied-mixture approaches, the field of context-dependent modeling where acoustic models are created that model the coarticulation effects of phonemes in the context of neighboring phonemes, as well as clustering techniques that are required for representing the acoustic parameters of these extended models. Other examples include the use of HMM multi-stream techniques to handle different acoustic feature streams or the entire area of efficient decoding techniques, e.g. the inclusion of higher level semantics in decoding, fast search techniques, efficient dictionary structures or different decoder architectures such as e.g. stack decoders. The interested reader is advised to study the available literature on these topics and the large number of conference papers describing those approaches in more detail. 4.1.6 Speech Recognition Technology As already mentioned, the HMM-technology has become the major technique for Automatic Speech Recognition and has nowadays reached a level of maturity that has led to the fact that this is not only the dominating technology in laboratory systems but in commercial systems as well. The HMM technology is also that much flexible that it can be deployed for almost every specialization in ASR. Basically, one can distinguish the following different technology lines: Small-vocabulary ASR systems with 10-50 word vocabulary, in speaker-independent mode, mainly used for telephone applications. Here, the capability of HMMs to capture the statistics of large training corpora obtained from many different speakers is exploited. The second line is represented by speaker-independent systems with medium size vocabulary, often used in automotive or multimedia application environments. Here, noise reduction technologies are often combined with the HMM framework and the efficiency of HMMs for decoding entire sequences of phonemes and words are exploited for the recognition of continuously spoken utterances in adverse environments. The last ma-
Speech Recognition
151
jor line are dictation systems with very large vocabulary (up to 100,000 words) which often operate in speaker-dependent and/or speaker-adaptive mode. In this case, the special capabilities of HMMs are in the area of efficient decoding techniques for searching very large trellis spaces, and especially in the field of context-dependent acoustic modeling, where coarticulation effects in continuous speech can be efficiently modeled by so-called triphones and clustering techniques for their acoustic parameters. One of the most recent trends is the developments of so-called embedded systems, where medium to large vocabulary size ASR systems are implemented on systems such as mobile phones, thin clients or other electronic devices. This has become possible with the availability of appropriate memory cards with sufficiently large storage capacity, so that the acoustic parameters and especially the memoryintensive language model with millions of word sequence probabilities can be stored directly on such devices. Due to these mentioned memory problems, another recent trend is so-called Distributed Speech Recognition (DSR), where only feature extraction is computed on the local device and the features are then transferred by wireless transmission to a large server where all remaining recognition steps are carried out, i.e. computation of emission probabilities and the decoding into the recognized word sequence. For these mentioned steps, an arbitrarily large server can be employed, with sufficient computation power and memory for large language models and a large number of Gaussian parameters for acoustic modeling. 4.1.7 Applications of ASR Systems It is not completely amazing that the above mentioned algorithms and available technologies have led to a large variety of interesting applications. Although ASR technology is far from being perfect, the currently achievable performance is in many cases satisfactory enough in order to create novel ideas for application scenarios or revisit already established application areas with now improved technology. Although current speech recognition applications are manifold, a certain structure can be established by identifying several major application areas as follows: Telecommunications: This is still the probably most important application area due to the fact that telecommunications is very much concerned with telephony applications involving naturally speech technology and in this case, speech recognition is a natural bridge between telecommunications and information technology by providing a natural interface in order to enter data via a communication channel into information systems. Thus, most application scenarios are in the area of speech recognition involving the telephone. Prominent scenarios include inquiry systems, where the user will inquire information by calling an automated system, e.g. for banking information or querying train and flight schedules. Dialing assistance, such as speaking the telephone number instead of dialing or typing it belongs to this application area. More advanced applications include telephony interpretation, i.e. the automatic translation of phone calls between partners from different countries, and all techniques involving mobile communications, such as embedded speech recognition on mobile clients and distributed speech recognition.
152
Speech Communication and Multimodal Interfaces
Office Automation: Similarly to telecommunications, office automation has been a traditional application area of ASR for several decades and has been one of the driving forces for very large vocabulary ASR. This area has been revived by the latest emerging commercial large vocabulary speech recognition systems that really provided the required performance in terms of vocabulary size and appropriate recognition performance. Typical concrete applications in that area include the classical dictation task where a secretary creates a letter directly via voice input and other scenarios, such as e.g. the use of ASR in CAD applications or the direct command input for PCs. Medical Applications: This area has also a quite long tradition and has been a favorite experimental scenario for ASR since at least 20 years. The most prominent field has been radiology, where the major idea has been to create a medical report directly from the visual analysis of an x-ray image by dictating directly into a microphone connected to an ASR system. Other scenarios include the control of microscopes via voice and another major application area of speech technology in general, namely the area of handicapped users, where the malfunction of hands or arms can be compensated by voice control of devices and interfaces. The size of this potential user group and the variety of different applications for handicapped users make this one of the most important ASR application scenarios. Production and Manufacturing: This area may be less popular than the previously mentioned application areas, but has been also investigated for a long time as potentially very interesting application with a large industrial impact. Popular applications include data distribution via spoken ID codes, programming of NC machines via voice, or spoken commands for the control of large plants, such as power or chemical plants. Multimedia Applications: Naturally, the rise of multimedia technology has also increased the demand for speech-based interfaces, as one major modality of multimodal interfaces. Most of those systems are still in the experimental phase, but some industrial applications are already underway, such as e.g. in smart environments or information kiosks, that require e.g. user input via voice and pointing. Another interesting area in this field are voice-enabled web applications, where speech recognition is used for the access to web documents. Private Sector: This field contains several very popular speech application fields with huge potential, such as the automotive sector, electronic devices and games. Without any doubt, those represent some of the most important application fields, where consumers are ready to make some extra investment in order to add more functionality to their user interface, e.g. in case of luxury automobiles or expensive specialized electronic devices. This overview demonstrates the huge potential of speech recognition technology for a large variety of interesting applications that can be well subdivided into the above mentioned major application areas which will represent also in the future the most relevant domains for even further improved ASR systems to come.
Speech Dialogs
153
4.2 Speech Dialogs 4.2.1 Introduction It is strongly believed that, following command line interfaces being popular in the years 1960-80 and graphical user interfaces in the years 1980-2000, the future lies in speech and multimodal user interfaces. A lot of factors thereby clearly speak for speech as interaction form: • • • •
Speech is the most natural communication form between humans. Only limited space is required: a microphone and a loudspeaker can be placed even in wearable devices. However, this does not respect computational effort. Hands and eyes are left free, which makes speech interaction the number one modality in many controlling situations as driving a car. Approximately 1.3 billion telephones exist worldwide, which resembles more than five times the number of computers connected to the Internet at the time. This provides a big market for automatic dialog systems in the future.
Using speech as an input and output form leads us to dialog systems: in general these are, in hierarchical order by complexity, systems that allow for control e.g. of functions in the car, information retrieval as flight data, structured transactions by voice, for example stock control, combined tasks of information retrieval and transactions as booking a hotel according to flight abilities and finally complex tasks as care of elder persons. More specifically a dialog may be defined as an exchange of different aspects in a reciprocal conversation between at least two instances which may either be human or machine with at least one change of the speaker. Within this chapter we focus on spoken dialog, still it may also exist in other forms as textual or combined manners. More concretely we will deal with so called Spoken Language Dialog Systems, abbreviated SLDS, which in general are a combination of speech recognition, natural language understanding, dialog act generation, and speech synthesis. While it is not exactly defined which parts are mandatory for a SLDS, this contributes to their interdisciplinary nature uniting the fields of speech processing, dialog design, usability engineering and process analysis within the target area. As enhancement of Question and Answer (Q&A) systems, which allow natural language access to data by direct answers without reference to past context (e.g. by pronominal allusions), dialog systems also allow for anaphoric expressions. This makes them significantly more natural as anaphora are frequently used as linguistic elements in human communication. Furthermore SLDS may also take over initiative. What finally distinguishes them from sheer Q&A and Command and Control (C&C) systems in a more complex way is that user modeling may be included. However, both Q&A and SLDS already provide an important improvement over today’s predominant systems where users are asked to adhere to a given complex and artificial syntax. The following figure gives an overview of a typical circular pipeline architecture of a SLDS [23]. On top the user can be found who controls an application
154
Speech Communication and Multimodal Interfaces
found at the bottom by use of a SLDS as interaction medium. While the components are mostly the same, other architectures exist, as organization around a central process [15]. In detail the single components are:
Fig. 4.6. Overview of a SLDS.
•
•
•
Automatic Speech Recognition (ASR): Spoken input analysis leading to hypotheses of linguistic units as phonemes or words and often some form of confidence measurement of the certainty of these (see Sect. 4.1. The output is mostly provided in so called lattices resembling tree structures or n-best lists. Key factors of an ASR module or engine, as it is mostly referred to, are speaker(in)dependence, the vocabulary size of known words and its general robustness. On the acoustic level also a user’s underlying affect may be analyzed in order to include emotional aspects (see Sect. 4.4 [48]. Natural Language Understanding (NLU): Interpretation of the intention or meaning of the spoken content. Here again several hypotheses may be forwarded to the dialog management combined with certainty. Dialog Management (DM): The DM is arguably the central module of a voice interface as it functions as an intermediate agent between user and application and is responsible for the interaction between them. In general it operates on an intention representation provided by the NLU which models what the user (presumably) said. On the basis of this information, the DM has several options as to change the state of an underlying application in the case of voice control or retrieve a piece of data from a database of interest in the case of an information service. Furthermore the DM decides, when and which type of system voice output is performed. Shortly summarized the DM’s primary tasks are storage
Speech Dialogs
• •
•
155
and analysis of context and dialog history, flow control e.g. for active initiative or barge-in handling, direction of the course of a conversation, answer production in an abstract way, and database access or application control. Database (DB): Storage of information in respect of dialog content. Natural Language Generation (NLG): Formulation of the abstract answer provided by the DM. A variety of approaches exists for this task reaching from probabilistic approaches with grammatical post-processing to pre-formulated utterances. Speech Synthesis (TTS): Audio-production for the naturally formulated system answer. Such modules are in general called Text-to-Speech engines. The two major types of such are once formant synthesizers resembling a genuinely artificial production of audio by formant tract modeling and concatenative synthesis. Within the latter audio clips at diverse lengths reaching from phonemes to whole words of recorded speech are concatenated to produce new words or sentences. At the moment these synthesizers tend to sound more natural depending on the type of modeling and post-processing as pitch and loudness correction. More sophisticated approaches use bi- or trigrams to model phonemes in the context of their neighboring ones. Recently furthermore prosodic cues as emotional speech gain interest in the field of synthesis. However, the most natural form still remains prerecorded speech, while providing less flexibility or more cost at recording and storage space.
Still, as it is disputed which parts besides the DM belong to a strict definition of a SLDS, we will focus hereon in the ongoing. 4.2.2 Initiative Strategies Before getting into dialog modeling, we want to make a classification in view of the initiative: • • •
system-driven: the system keeps the initiative throughout the whole dialog user-driven: the user keeps the initiative mixed initiative: the initiative changes throughout the dialog
Usually, the kind of application, and thus the kind of voice interface, codetermines this general dialog strategy. For example systems with limited vocabulary tend to employ rigid, system initiative dialogs. The system thereby asks very specific questions, and the user can do nothing else but answer them. Such a strategy is required due to the highly limited set of inputs the system can cope with. C&C applications tend to have rigid, user initiative dialogs: the system has to wait for input from the user before it can do anything. However, the ideal of many researchers and developers is a natural, mixed initiative dialog: both counterparts have the possibility of taking the initiative when this is opportune given the current state of the dialog, and the user can converse with the system as (s)he would with another human. This is difficult to obtain in general, for at least two reasons: Firstly, it is technically demanding, as the user should have the freedom to basically say anything at any moment,
156
Speech Communication and Multimodal Interfaces
which is a severe complication from a speech recognition and language processing point of view. Secondly, apart from such technical hurdles, it is also difficult from a dialog point of view, as initiative has to be tracked and reactions have to be flexible. 4.2.3 Models of Dialog Within the DM an incorporated dialog model is responsible for the structure of the communication. We want to introduce the most important such models in the ongoing. They can be mainly divided into structural models, which will be introduced firstly, and nonstructural ones [23, 31]. Thereby a more or less predefined path is given within structural ones. While these are quite practicable in the first order, their methodology is not very principled, and the quality of dialogs based on them is arguable. Trying to overcome this, the non-structural approaches rely rather on general principles of dialog. 4.2.3.1 Finite State Model The dialog structure represents a sequence of limited predetermined steps or states in form of a condition transition graph which models all legal dialogs within the very basic graph-based-, or Finite State Automaton (FSA) model. This graph’s nodes represent system questions, system outputs or system actions, while its edges show all possible paths in the network which are labeled with the according user expressions. As all ways in a deterministic finite automaton are fixed, it cannot be spoken of an explicit dialog control. The dialog therefore tends to be rigid, and all reactions for each user-input are determined. We can denote such a model as the quintuple {S, s0 , sf , A, τ }. Thereby S represents a set of states besides s0 , the initial state, and sf , the final state, and A a set of actions including an empty action . τ finally is a transition function with τ : S × A → S. By τ it is specified to which state an action given the actual state leads. Likewise a dialog is defined as the path from s0 to sf within the space of possible states. Generally, deterministic FSA are the most natural kind to represent system-driven dialogs. Furthermore they are well suitable for completely predictable information exchange due to the fact that the entire dialog model is defined at the time of the development. Often it is possible to represent the dialog flow graphically, which resembles a very natural development process. However, there is no natural order of the conditions, as each one can be followed by any other resulting in a combinatorial explosion. Furthermore the model is unsuited for different abstraction levels of the exchanged information, and for complex dependencies between information units. Systems basing on FSA models are also inflexible, as they are completely system-led, and no path deviations are possible. The user’s input is restricted to single words or phrases that provide responses to carefully designed system prompts. Tasks, which require negotiations, cannot be implemented with deterministic automata due to the uncertainty of the result. Grounding and repair is very rigid and must be applied after each turn. Later corrections by the user are hardly possible, and the context of the statements cannot be used. Summed up, such
Speech Dialogs
157
systems are appropriate for simple tasks with flat menu structure and short option lists. 4.2.3.2 Slot Filling In so called frame-based systems the user is asked questions that enable to fill slots in a template in order to perform a task [41]. These systems are more or less today’s standard for database retrieval systems as flight or cinema information. In difference to automaton techniques the dialog flow is not predetermined but depends on the content of the user’s input and the information that the system has to elicit. However, if the user provides more input than is requested at a time, the system can accept this information and check if any additional item is still required. A necessary component therefore is a frame that controls over already expressed information fragments. If multiple slots are yet to fill, the question of the optimal order in respect of system questioning remains. The idea thereby is to constrain the originally large set of a-priori possibilities of actions in order to speed up the dialog. Likewise, one reasonable approach to establish a hierarchy of slots is to stick to the order of highest information based on Shannon’s entropy measure, as proposed in [24]. Such an item with high information might e.g. be the one leaving most alternatives open. Let therefore ck be a random variable of items within the set C, and vj a value of attribute a. The attribute a that minimizes the following entropy measure H(c|a) will be accordingly selected: P (ck , a = vj ) log2 P (ck |a = vj ) (4.6) H(c|a) = − vj ∈a ck ∈C
Thereby the probability P (ck , a = vj ) is set to 0 unless ck matches the revised partial description. In this case it resembles P (ck ). The further missing probability P (ck |a = vj ) is calculated by the following equation, where C is the set of items that match the revised description that includes a = vj , and P (ck ) can be approximated basing on the assumption that items are equally likely: 1 P (ck ) , P (ck ) = P (ck |a = vj ) = |C| P (cj )
(4.7)
cj ∈C
An advantage of dialog control with frames is higher flexibility for the user and multiple slot filling. The duration or dialogs are shorter, and a mixed initiative control is possible. However, an extended recognition grammar is required for more flexible user expressions, and the dialog control algorithm must specify the next system actions on basis of the available frames. The context that decides on the next action (last user input, status of the slots, simple priority ranking) is limited, and the knowledge level of the user, negotiating or collaborative planning cannot – or only be modeled to a very limited degree. No handling for communication problems exists, and the form of system questions is not specified. Still, there is an extension [53], whereby complex questions are asked if communication works well. In case of problems a system can switch to lower-level questions splitting up the high-level ones. Finally,
158
Speech Communication and Multimodal Interfaces
an overview over the system is not possible due to partially complex rules: which rule fires when? 4.2.3.3 Stochastic Model In order to find a less hand-crafted approach which mainly bases on a designer’s input, recently data-driven methods successfully applied in machine learning are used within the field of dialog modeling. They aim at overcoming the shortcomings of low portability to new domains and the limited predictability of potential problems within a design process. As the aim is to optimize a dialog, let us first define a cost function C. Thereby the numbers Nt of turns , Ne of errors, and Nm of missing values are therefore summed up by introduction of individual according weights wt , we , and wm : C = wt (Nt ) + we (Ne ) + wm (Nm )
(4.8)
Now if we want to find an ideal solution minimizing this target function, we face |A||S| possible strategies. A variety of suited approaches for a machine based solution to this problem exists [57]. However, we chose the predominant Markov Decision Processes (MDP) as a probabilistic extension to the introduced FSA following [28] herein. MDP differ from FSA, as they introduce transition probabilities instead of the functions τ . We therefore denote at a time t the state st , and the action at . Given st and at we change to state st+1 with the conditional probability P (st+1 |st , at ). Thereby we respect only events one time step behind, known as the limited horizon Markov property. Next, we combine this with the introduced costs, whereby ct shall be the cost if in state st the action at is performed, with the probability P (ct |st , at ). The cost of a complete dialog can likewise be denoted as: C=
tf
ct ,
(4.9)
t=0
where tf resembles the instant when the final state sf is reached. In the consequence the best action V ∗ (s) to take within a state s is the action that minimizes over incurred and expected costs for the succeeding state with the best action within there, too: % & ∗ ∗ Ptf (s |s, a)V (s ) (4.10) V (s) = arga min c(s, a) + s
Given all model parameters and a finite state space, the unique function V ∗ (s) can be computed by value interaction techniques. Finally, the optimal strategy ψ ∗ resembles the chain of actions that minimizes overall costs. In our case of a SLDS the parameters are not known in advance, but have to be learned by data. Normally such data might be acquired by test users operating on a simulated system known as Wizard-of-Oz experiments [33]. However, as loads of data are needed, and there is no data like more data, the possibility of simulating a user by another stochastic process as a solution to sparse data handling exists [27]. Basing on annotated dialogs with
Speech Dialogs
159
real users, probabilities are derived with which a simulated user acts responding to the system. Next reinforced learning e.g. by Monte Carlo with exploring starts is used to obtain an optimal state-action value function Q∗ (s, a). It resembles the costs to be expected of a dialog starting in state s, moving to s by a, and optimally continuing to the end: Ptf (s |s, a) arga min Q∗ (s , a ), (4.11) Q∗ (s, a) = c(s, a) + s
Note that V ∗ (s) = arga min Q∗ (s, a). Starting in an arbitrary set of Q∗ (s, a) the algorithm iteratively finds the costs for a dialog session. In [27] it is shown that it converges to an optimal solution, and that it is expensive to immediately shortcut a dialog by ”bye!”. The goals set at the beginning of this subsection can be accomplished by stochastic dialog modeling. The necessity of data seems one drawback, while it often suffices to have very sparse data, as low accuracies of the initial transition probabilities may already lead to satisfying results. Still, the state space is hand-crafted, which highly influences the learning process. Likewise the state space itself should be learned. Furthermore the cost function determination is not trivial, especially assigning the right weights, while also highly influencing the result. In the suggested cost function no account is taken for subjective costs as user satisfaction. 4.2.3.4 Goal Directed Processing The more or less finite state and action based approaches with predefined paths introduced so far are well suited for C&C and information retrieval, but less suited for task-oriented dialogs where the user and the system have to cooperate to solve a problem. Consider therefore a plan-based dialog with a user that has only sparse knowledge about the problem to solve, and an expert system. The actual task thereby mostly consists of sub-tasks, and influences the structure of the dialog [16]. A system has now to be able to reason about the problem at hand, the application, and the user to solve the task [50]. This leads to a theorem prover that derives conclusions by the laws of logic from a set of true considered axioms contained in a knowledge-base. The idea is to proof whether a subgoal or goal is accomplished, as defined by a theorem, yet. User modeling is thereby also done in form of stored axioms that consist of his competence and knowledge. Whenever an axiom is missing, interaction with the user becomes necessary, which claims for an interruptible theorem prover that is able to initiate a user directed question. This is known as the missing axiom theory [50]. As soon as a subgoal is fulfilled, the selection of a new one can be made according to the highest probability of success given the actual dialog state. However, the prover should be flexible enough to switch to different sub-goals, if the user actively provides new information. This process is iteratively repeated until the main goal is reached. In the framework of Artificial Intelligence (AI) this can be described by the Beliefs, Desires, and Intentions (BDI) model [23], as shown in the following figure 4.7.
160
Speech Communication and Multimodal Interfaces
This model can be applied to conversational agents that have beliefs about the current state of the world, and desires, how they want it to be. Basing on these they determine their intention, respectively goal, and build a plan consisting of actions to satisfy their desires. Note that utterances are thereby treated as (speech) actions.
Fig. 4.7. Beliefs, Desires, and Intentions Model.
To conclude, the advantages of goal directed processing are the ability to also deal with task-oriented communication and informational dialogs, a more principled approach to dialog based on a general theory of communication, and less domaindependency at least for the general model. Yet, by applying AI methods, several problems are inherited: high computational cost, the frame problem dealing with the specification of non-influenced parts of the world by actions, and the lack of proper belief and desire formalizations and independently motivated rationality principles in view of agent behavior. 4.2.3.5 Rational Conversational Agents Rational Conversational Agents directly aim at imitation of natural conversation behavior by trying to overcome plan-based approaches’ lack in human-like rationality. The latter is therefore reformulated in a formal framework establishing an intelligent system that relies on a more general competence basis [43]. The beliefs, desires and intentions are logically formulated, and a rational unit is constructed basing thereon that decides upon the actions to be taken. Let us denote an agent i believing in proposition p as B(i, p), and exemplary propositions as ϕ, and ψ. We can next construct a set of logical rules known as NKD45 or Weak S5 that apply for the B operator: (N ) : always ϕ → always B(i, ϕ)
(4.12)
(K) : B(i, ϕ) ∧ B(i, ϕ → ψ) → B(i, ψ)
(4.13)
(D) : B(i, ϕ) → ¬B(i, ¬ϕ)
(4.14)
(4) : B(i, ϕ) → B(i, B(i, ϕ))
(4.15)
(5) : ¬B(i, ϕ) → B(i, ¬B(i, ϕ)
(4.16)
Speech Dialogs
161
Let us furthermore denote desires, respectively goals, as G(i, p). Likewise an agent indexed i has the desire that p comes true. As an exercise, one can reflect why only (K) and (4) apply for goals: (K) : G(i, ϕ) ∧ G(i, ϕ → ψ) → G(i, ψ)
(4.17)
(4) : G(i, ϕ) → G(i, B(i, ϕ))
(4.18)
We next connect goals and beliefs by a realism constraint which states that an agent cannot have a goal he believes to be false: B(i, ϕ) → G(i, ϕ)
(4.19)
Expected consequences of a goal furthermore also have to be a goal of an agent, which is known as the expected consequences constraint: G(i, ϕ) ∧ B(i, ϕ → ψ) → G(i, ϕ)
(4.20)
Finally, let us introduce the persistent goal constraint with the persistent goal P G(i, ϕ): an agent i should not give up a goal ϕ to be true in the future (1) that he believes not true presently (2) until either its fulfillment or necessarily non-availability (3). (1) : P G(i, ϕ) if and only if G(i, (Future ϕ))∧ (4.21) (2) : B(i, ¬ϕ)∧ (3) : [Before ((B(i, ϕ) ∨ B(i, (Necessary ¬ϕ)))¬G(i, (Future ϕ)))]
(4.22) (4.23)
The rational unit of a SLDS fed with the formalized notions of belief, desires and intentions has now a selection of communication actions to choose from that are associated with feasibility preconditions and rational effects [43]. Such an action is e.g. that an agent i informs an agent j about ϕ, whereby the user is also understood as an agent. Now, if an agent has the intention to achieve a goal, it selects an appropriate action and thereby inherits the intention to fulfill according preconditions. This approach defines a planning algorithm demanding for a theorem prover as in the previous section. However, user directed questions do not directly concern missing axioms – it is rather only checked whether preconditions are fulfilled and effects of system-actions help in view of the current goal. To sum up, the characteristics of rational conversational agents are full mixed initiative, the ability to implement theoretically complex systems which solve dynamic and (only) cooperative complex tasks. They are also much more oriented on the linguistic theory. However, there is no general definition of the agent term besides that they should be reactive, autonomous, social, rational and antropomorph. Furthermore way more resources are needed both quantitatively (computer speed, programming expenditure) and qualitatively (complexity of the problems which can be solved). Also, the formalization of all task relevant domain-knowledge is not trivial. So far there are only academic systems realized.
162
Speech Communication and Multimodal Interfaces
4.2.4 Dialog Design Let us now turn our attention to the design of a dialog, as a lot of factors besides recognition performance of the ASR unit have significant influence on the utmost design goals naturalness, efficiency, and effectiveness. As we learned so far, initiative can e.g. be chosen mixed or by one side only. Furthermore confirmations can be given or spared, and suggestions made in case of failure, etc. In the following three main steps in the design of a dialog will be outlined: •
•
•
Script writing: In this part, also known as call flow layout, the interaction between user and system is laid out step-wisely. Focus should be given to the naturalness of the dialog, which is also significantly influenced by the quality of the dialog flow. Prompt Design: Similar to a prompt in console based interfaces a sound or announcement of the system signals the user when and depending on the situation what to speak. This acoustic prompt has to be well designed in order to be informative, well heard, but not disturbing. A frequent prompt might therefore be chosen short. On the other hand in the case of error handling, tutorial information or help provision, prompts might be chosen more complex, as the quality of a user’s answer depends strongly on the quality of the prompt. Finally, only by appropriate prompt crafting the initiative throughout a dialog may be controlled effectively. Grammar Writing: Within the grammar the possible user statements given a dialog-state are defined. A compromise between coverage and recognition accuracy has to be found, as too broad a coverage often leads to decreased performance due to a large sphere of possible hypotheses. Also, phrases of multiple words should be handled. This is often realized by finite state grammars, and triggering by keywords.
For an automatic system it seems crucial to understand the actual meaning of a user statement, which highly depends on the context. Furthermore it is important to design system announcements clear and understandable for the user. We therefore also want to take a brief linguistic view on dialog within this section. Three different aspects are important considering the meaning of a verbally uttered phrase in the chain from sender to receiver: Firstly, we have the locution, which represents the semantic or literal significance of the utterance; secondly there is the illocution, the actual intention of the speaker, and finally the perlocution stands for how an utterance is received by the counterpart. Likewise to speak is to perform a locution, but to speak with an intent (ask, promise, request, assert, demand, apologize, warn, etc.) is to perform an illocution. The purpose, the illocutionary intent, is meaningful and will ordinarily be recognized by hearers. Within this context we want to have a look at four well known Greek conversational maxims: •
Maxim of relevance: be relevant. Consider hereon: ’He kicked the bucket’ (we assume that someone died because that’s what’s relevant), or ’Do you know what time it is?’ (we assume that the speaker wants to know the time because the ”real” question is irrelevant).
Speech Dialogs •
• •
163
Maxim of quality: be truthful. E.g. ’If I hear that song again I’ll kill myself’ (we accept this as a hyperbole and do not immediately turn the radio off), or ’The boss has lost his marbles’ (we imagine a mental problem and not actual marbles). Maxim of quantity: be informative, say neither too much not too little. (Asked the date, we do not include the year). Maxim of manner: be clear and orderly.
In order to obtain high overall dialog quality, some aspects shall be further outlined: Firstly, consistency and transparency are important to enable the user to picture a model of the system. Secondly, social competence in view of user behavior modeling and providing a personality to the system seems very important. Thirdly, error handling plays a key role, as speech recognition is prone to errors. The precondition thereby is clarification, demanding that problems at the user-input (no input, cut input, word-recognition errors, wrong interpretations, etc.) must become aware to the system. In general it is said that users accept a maximum of five percent errors or less. Therefore trade-offs have to be made considering the vocabulary size, and naturalness of the input. However, there is a chance to raise the accuracy by expert prompt modeling, allusion to the problem nature, or at least cover errors and reduce annoyance of the users by a careful design. Also, the danger of over-modeling a dialog in view of world knowledge, social experience or general complexity of manlike communication shall be mentioned. Finally, the main characteristic of a good voice interface is probably that is usable. Usability thereby is a widely discussed concept in the field of interfaces, and various operationalizations have been proposed. In [33] it is stated that usability is a multidimensional concept comprising learnability, efficiency, memorability, errors and satisfaction, and ways are described, in which these can be measured. Defining when a voice interface is usable is one thing, developing one is quite another. It is by now received wisdom that usability design is an iterative process which should be integrated in the general development process. 4.2.5 Scripting and Tagging We want to conclude this section with a short introduction of the two most important dialog mark-up languages in view of scripting and tagging: VoiceXML and DAMSL. VoiceXML (VXML) is the W3C’s standard XML format for specifying interactive voice dialogs between a human and a computer. VXML is fully analogous to HTML, and just as HTML documents are interpreted by a visual web browser, VXML documents are interpreted by a voice browser. A common architecture is to deploy banks of voice browsers attached to the public switched telephone network so that users can simply pick up a phone to interact with voice applications. VXML has tags that instruct the voice browser to provide speech synthesis, automatic speech recognition, dialog management, and soundfile playback. Considering dialog management, a form in VXML consists of fields and control units: fields collect information from the user via speech input or DMTF (dual tone multi-frequency). Control units are sequences of procedural statements, and the control of the dialog is made by the following form interpretation algorithm, which consists of at least one major loop with three phases:
164 • •
•
Speech Communication and Multimodal Interfaces Select: The first form not yet filled is selected in a top down manner in the active VXML document that has an open guard condition. Collect: The selected form is visited, and the following prompt algorithm is applied to the prompts of the form. Next the input grammar of the form is activated, and the algorithm waits for user input. Process: Input evaluation of a form’s fields in accordance to the active grammar. Filled elements are called for example to the input validation. The process phase ends, if no more items can be selected or a jump point is reached.
Typically, HTTP is used as the transport protocol for fetching VXML pages. While simpler applications may use static VXML pages, nearly all rely on dynamic VXML page generation using an application server. In a well-architected web application, the voice interface and the visual interface share the same back-end business logic. Tagging of dialogs for machine learning algorithms on the other hand is mostly done using Dialogue Act Mark-up in Several Layers (DAMSL) - an annotation scheme for communicative acts in dialog [9]. While different dialogs being analyzed with different aims in mind will lead to diverse acts, it seems reasonable to agree on a common basis for annotation in order to enable database enlargement by integration of other ones. The scheme has three layers: Forward Communicative Functions, Backward Communicative Functions, and Information Level. Each layer allows multiple communicative functions of an utterance to be labeled. The Forward Communicative Functions consist of a taxonomy in a similar style as the actions of traditional speech act theory. The most important thereby are statement (assert, reassert, other statement), influencing addressee future action (open-option, directive (inforequest, action-directive), and committing speaker future action (offer, commit). The Backward Communicative Functions indicate how the current utterance relates to the previous dialog: agreement (accept, accept-part, maybe, reject-part, reject, hold), understanding (signal non-understanding, signal understanding (acknowledge, repeat phrase, completion), correct misspeaking), and answer. Finally, the Information Level annotation encodes whether an utterance is occupied with the dialog task, the communication process or meta-level discussion about the task.
4.3 Multimodal Interaction The research field of Human-Computer Interaction (HCI) focuses on arranging the interaction with computers easier, safer, more effective and to a high degree seamless for the user. ”Human-Computer Interaction is a discipline concerned with the design, evaluation and implementation of interactive computing systems for human use and with the study of major phenomena surrounding them.” [18] As explained, HCI is an interdisciplinary field of research where many different subjects are involved to reach the long-term objective of a natural and intuitive
Multimodal Interaction
165
way of interaction with computers. In general, the term interaction describes the mutual influence of several participants to exchange information that is transported by most diverse means in a bilateral fashion. Therefore, a major goal of HCI is to converge the interface increasingly towards a familiar and ordinary interpersonal way of interaction. For natural human communication several in- and output channels are combined in a multimodal manner. 4.3.1 In- and Output Channels The human being is able to gather, process and express information through a number of channels. The input channel can be described as sensor function or perception, the processing of information as cognition, and the output channel as motor function (see figure 4.8).
Fig. 4.8. Human information input and output channels.
Humans are equipped with six senses respectively sense organs to gathers stimuli. These senses are defined by physiology [14]: sense of sight sense of hearing sense of smell sense of taste sense of balance sense of touch
visual channel auditive channel olfactory channel gustatory channel vestibular channel tactile channel
The human being is also provided with broad range of abilities for information output. The output channel can process less amount of information than the input channel. Outgoing information is transmitted by the auditive, the visual and the haptic channel (tactile as perception modality is distinguished from haptic as output manner). By this means, the human is enabled to communicate with others in a simple, effective, and error robust fashion. With an integrated and synchronized use of
166
Speech Communication and Multimodal Interfaces
different channels he can flexibly adapt to the specific abilities of the conversational partner and the current surroundings. However, HCI is far from information transmission of this intuitive and natural manner. This is due to the limitations of the technical systems in regard of their number and performance of the single in- and output modalities. The term modality refers to the type of communication channel used to convey or acquire information. One of the core difficulties of HCI is the divergent boundary conditions between computers and human. Nowadays, the user can transfer his commands to the computer only by standard input devices – e.g. mouse or keyboard. The computer feedback is carried out by visual or acoustic channel – e.g. monitor or speakers. Thus, todays’ technology uses only a few interaction modalities of the humans. Whenever two or more of these modalities are involved you speak from multimodality. 4.3.2 Basics of Multimodal Interaction The term ”multimodal” is derived from ”multi” (lat.: several, numerous) and ”mode” (lat.: naming the method). The word ”modal” may cover the notion of ”modality” as well as that of ”mode”. Scientific research focuses on two central characteristics of multimodal systems: • •
the user is able to communicate with the machine by several input and output modes the different information channels can interact in a sensible fashion
According to S. Oviatt [35] multimodal interfaces combine natural input modes - such as speech, pen, touch, manual gestures, gaze and head and body movements - in a coordinated manner with multimedia system output. They are a new class of interfaces which aims to recognize naturally occurring forms of human language and behavior, and which incorporate one or more recognition-based technologies (e.g., speech, pen, vision). Benoit [5] expanded the