Advances in Agent-Based Complex Automated Negotiations (Studies in Computational Intelligence)

Aruna Chakraborty and Amit Konar Emotional Intelligence Studies in Computational Intelligence, Volume 234 Editor-in-Ch...

Author: Aruna Chakraborty | Amit Konar

34 downloads 680 Views 5MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!

Report copyright / DMCA form

DOWNLOAD PDF

Aruna Chakraborty and Amit Konar Emotional Intelligence

Studies in Computational Intelligence, Volume 234 Editor-in-Chief Prof. Janusz Kacprzyk Systems Research Institute Polish Academy of Sciences ul. Newelska 6 01-447 Warsaw Poland E-mail: [email protected] Further volumes of this series can be found on our homepage: springer.com Vol. 212. Viktor M. Kureychik, Sergey P. Malyukov, Vladimir V. Kureychik, and Alexander S. Malyoukov Genetic Algorithms for Applied CAD Problems, 2009 ISBN 978-3-540-85280-3 Vol. 213. Stefano Cagnoni (Ed.) Evolutionary Image Analysis and Signal Processing, 2009 ISBN 978-3-642-01635-6 Vol. 214. Been-Chian Chien and Tzung-Pei Hong (Eds.) Opportunities and Challenges for Next-Generation Applied Intelligence, 2009 ISBN 978-3-540-92813-3 Vol. 215. Habib M. Ammari Opportunities and Challenges of Connected k-Covered Wireless Sensor Networks, 2009 ISBN 978-3-642-01876-3 Vol. 216. Matthew Taylor Transfer in Reinforcement Learning Domains, 2009 ISBN 978-3-642-01881-7 Vol. 217. Horia-Nicolai Teodorescu, Junzo Watada, and Lakhmi C. Jain (Eds.) Intelligent Systems and Technologies, 2009 ISBN 978-3-642-01884-8 Vol. 218. Maria do Carmo Nicoletti and Lakhmi C. Jain (Eds.) Computational Intelligence Techniques for Bioprocess Modelling, Supervision and Control, 2009 ISBN 978-3-642-01887-9 Vol. 219. Maja Hadzic, Elizabeth Chang, Pornpit Wongthongtham, and Tharam Dillon Ontology-Based Multi-Agent Systems, 2009 ISBN 978-3-642-01903-6 Vol. 220. Bettina Berendt, Dunja Mladenic, Marco de de Gemmis, Giovanni Semeraro, Myra Spiliopoulou, Gerd Stumme, Vojtech Svatek, and Filip Zelezny (Eds.) Knowledge Discovery Enhanced with Semantic and Social Information, 2009 ISBN 978-3-642-01890-9 Vol. 221. Tassilo Pellegrini, S¨oren Auer, Klaus Tochtermann, and Sebastian Schaffert (Eds.) Networked Knowledge - Networked Media, 2009 ISBN 978-3-642-02183-1 Vol. 222. Elisabeth Rakus-Andersson, Ronald R. Yager, Nikhil Ichalkaranje, and Lakhmi C. Jain (Eds.) Recent Advances in Decision Making, 2009 ISBN 978-3-642-02186-2

Vol. 223. Zbigniew W. Ras and Agnieszka Dardzinska (Eds.) Advances in Data Management, 2009 ISBN 978-3-642-02189-3 Vol. 224. Amandeep S. Sidhu and Tharam S. Dillon (Eds.) Biomedical Data and Applications, 2009 ISBN 978-3-642-02192-3 Vol. 225. Danuta Zakrzewska, Ernestina Menasalvas, and Liliana Byczkowska-Lipinska (Eds.) Methods and Supporting Technologies for Data Analysis, 2009 ISBN 978-3-642-02195-4 Vol. 226. Ernesto Damiani, Jechang Jeong, Robert J. Howlett, and Lakhmi C. Jain (Eds.) New Directions in Intelligent Interactive Multimedia Systems and Services - 2, 2009 ISBN 978-3-642-02936-3 Vol. 227. Jeng-Shyang Pan, Hsiang-Cheh Huang, and Lakhmi C. Jain (Eds.) Information Hiding and Applications, 2009 ISBN 978-3-642-02334-7 Vol. 228. Lidia Ogiela and Marek R. Ogiela Cognitive Techniques in Visual Data Interpretation, 2009 ISBN 978-3-642-02692-8 Vol. 229. Giovanna Castellano, Lakhmi C. Jain, and Anna Maria Fanelli (Eds.) Web Personalization in Intelligent Environments, 2009 ISBN 978-3-642-02793-2 Vol. 230. Uday K. Chakraborty (Ed.) Computational Intelligence in Flow Shop and Job Shop Scheduling, 2009 ISBN 978-3-642-02835-9 Vol. 231. Mislav Grgic, Kresimir Delac, and Mohammed Ghanbari (Eds.) Recent Advances in Multimedia Signal Processing and Communications, 2009 ISBN 978-3-642-02899-1 Vol. 232. Feng-Hsing Wang, Jeng-Shyang Pan, and Lakhmi C. Jain Innovations in Digital Watermarking Techniques, 2009 ISBN 978-3-642-03186-1 Vol. 233. Takayuki Ito, Minjie Zhang, Valentin Robu, Shaheen Fatima, and Tokuro Matsuo (Eds.) Advances in Agent-Based Complex Automated Negotiations, 2009 ISBN 978-3-642-03189-2 Vol. 234. Aruna Chakraborty and Amit Konar Emotional Intelligence, 2009 ISBN 978-3-540-68606-4

Aruna Chakraborty and Amit Konar

Emotional Intelligence A Cybernetic Approach

123

Dr. Aruna Chakraborty Associate Professor, St. Thomas’ College of Engineering and Technology, 4 Diamond Harbour Road, Calcutta- 700 023. Visiting Faculty for the M.Tech. Course on Intelligent Automation and Robotics, Department of Electronics and Tele-Communication Engineering, Jadavpur University, Calcutta- 700 032 India E-mail: aruna [email protected]

Dr. Amit Konar Professor, Artificial Intelligence Research Lab. Department of Electronics and Tele-Communication Engineering, Jadavpur University, Calcutta- 700 032 India E-mail: [email protected]

ISBN 978-3-540-68606-4

e-ISBN 978-3-540-68609-5

DOI 10.1007/978-3-540-68609-5 Studies in Computational Intelligence

ISSN 1860-949X

Library of Congress Control Number: Applied for c 2009 Springer-Verlag Berlin Heidelberg This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typeset & Cover Design: Scientific Publishing Services Pvt. Ltd., Chennai, India. Printed in acid-free paper 987654321 springer.com

Preface

Emotional Intelligence is a new discipline of knowledge, dealing with modeling, recognition and control of human emotions. The book Emotional Intelligence: A Cybernetic Approach, to the best of the authors’ knowledge is a first comprehensive text of its kind that provides a clear introduction to the subject in a precise and insightful writing style. It begins with a philosophical introduction to Emotional Intelligence, and gradually explores the mathematical models for emotional dynamics to study the artificial control of emotion using music and videos, and also to determine the interactions between emotion and logic from the points of view of reasoning. The later part of the book covers the chaotic behavior of coexisting emotions under certain conditions of emotional dynamics. Finally, the book attempts to cluster emotions using electroencephalogram signals, and demonstrates the scope of application of emotional intelligence in several engineering systems, such as human-machine interfaces, psychotherapy, user assistance systems, and many others. The book includes ten chapters. Chapter 1 provides an introduction to the subject from a philosophical and psychological standpoint. It outlines the fundamental causes of emotion arousal, and typical characteristics of the phenomenon of an emotive experience. The relation between emotion and rationality of thoughts is also introduced here. Principles of natural regulation of emotions are discussed in brief, and the biological basis of emotion arousal using an affective neuroscientific model is introduced next. Scope of mathematical modeling to study the dynamics of emotion is also discussed. Principles of controlling emotion by artificial means, and the effect of emotion modeling on human-machine interactions are also outlined. Chapter 2 overviews the mathematical foundations required to understand the rest of the book. It introduces the principles of modeling of a static and dynamic system, and compares the classical first order logic with a static system, and a temporal logic with a dynamic system. General method of stability analysis of dynamical systems using Lyapunov functions is introduced next. The chapter also explores the stability analysis of fuzzy systems, and introduces the principles of Lyapunov exponents to study the existence of chaos in a neuro-dynamic system. Finally, the chapter presents the methodology of modeling emotional dynamics,

VI

Preface

and reviews the scope of both stability analysis and Lyapunov exponents, to study the stable and chaotic behavior of the dynamics. Chapter 3 on Image Processing is introduced as a prerequisite to the rest of the book. It begins with frequency domain transforms, such as Fourier and Cosine transforms, and then outlines the techniques for pre-processing and noise filtering from images using neighborhood averaging, median filtering and thresholding. It then explores image segmentation algorithm, including boundary detection, region detection, and fuzzy clustering algorithm. Principles of boundary description algorithm are then outlined with emphasis on chain codes, Fourier descriptors, and regional descriptors. An introduction to object recognition technique is reviewed through unsupervised clustering, supervised classification and intelligent matching algorithm. The chapter ends with a brief discussion on scene interpretation. Chapter 4 outlines brain imaging and psycho-pathological studies on selfregulation of emotion. It begins with the mechanism of emotion processing by the human brain, and provides the latest research outcome on the role of the amygdala, the Orbitofrontal Cortex, the Insula, and the Anterior Cingulated Cortex on emotion arousal and its self-regulation. Experimental research undertaken on both humans and animals are given in brief. Principles of voluntary self-regulation of emotion are overviewed through f-MRI studies and neural models. The EEG conditioning and analysis for depression and pre-menstural disphoric disorder are also presented. Studies with both clinical and non-clinical samples for emotion dysregulation in childhood and adulthood are discussed in a nutshell. Chapter 5 is an original contribution by the authors on human emotion recognition from facial expressions, and its control by audio-visual means using fuzzy logic. The chapter begins with segmentation and localization of important facial components, such as the mouth region, the eye region, and the eyebrow region. Next a scheme for determining facial attributes, such as mouth-opening, eyeopening, and the length of eyebrow-constriction is briefly outlined. Principles of fuzzy relational system have been employed for the detection of emotion from the facial attributes. Finally, the chapter introduces a novel dynamics for emotion transition, and provides an architecture for the proposed emotional dynamics to control the emotion of subjects using the logic of fuzzy sets. Stabilization of the human mind, and its emotion-logic encounter remained an unsolved problem until this date. Chapter 6 provides a novel scheme to handle this problem by representing formalisms for emotional dynamics and logical reasoning, when both of them share common information resources of the subject. Stability analysis of the emotional dynamics and the temporal logic are then undertaken in order to activate the control of emotion over logic and vice-versa under stabilized condition of the mental system. Chapter 7 deals with multiple emotions and chaos. A model of competitive/cooperative emotional dynamics is proposed, and the variations of different parameters on the temporal response of the dynamics are studied through computer simulations. A Lyapunov-based stability analysis of the emotional dynamics is undertaken to determine the parametric condition for stability. A system identification

Preface

VII

approach is then overviewed to determine the suitable parameters of the dynamics to minimize an objective function, indicating error between sampled response of the amygdala and that of the model. This error function is minimized by three alternative algorithms, namely genetic algorithm, particle swarm optimization algorithm, and differential evolution algorithm. A stabilization scheme for mixed emotional dynamics is then studied through a closed loop feedback system, where the control signal attempts to improve the damping level of the dynamics in order to stabilize its behavior. Chapter 8 provides an overview of EEG signal processing for detection and prediction of emotion. It begins with LMS, NLMS, RLS and Kalman filter algorithms, and demonstrates their scope in EEG signal prediction. Next, it employs wavelet techniques for prediction of EEG signal. The later part of the chapter proposes a scheme for emotion clustering from bio-potential signals using neural networks. Chapter 9 is the concluding chapter of the book. It outlines possible application of the proposed scheme for emotion modeling, detection and control in humanmachine interactive systems, multi-agent robotics, psycho-therapy, digital movie making, matrimonial counseling, and also in personality building of artificial creatures. Future research directions on voice and multi-modal emotion recognition are also indicated. Chapter 10 outlines open-ended research problems in Emotional Intelligence, and also cites a number of important references to pursue research in this young discipline of knowledge.

June 2009

Aruna Chakraborty Amit Konar

Acknowledgement

The authors gratefully acknowledge the support they received from St. Thomas’ College of Engineering and Technology, and Jadavpur University during the preparatory phase of the book. They would like to thank Prof. P. N. Ghosh, ViceChancellor of Jadavpur University, and Prof. M. K. Mitra, Dean of the Faculty of Engineering and Technology, Jadavpur University, Dr. S. Mukhopadhyay, ExPrincipal, St. Thomas’ College of Engineering and Technology (STCET), and Prof. S. Sen, Principal (Acting), STCET, Mr. Goutam Banerjea, Registrar, STCET for providing all the necessary support to complete the book in the present form. The authors also wish to thank Prof. S. Bhattacharya, Head, department of Computer Science and Engineering of St. Thomas’ College of Engineering and Technology and Prof. S. K. Sanyal, Head, department of Electronics and TeleCommunication Engineering, Jadavpur University for creating an ambient environment for the successful completion of the book. Special thanks are due to Prof. Amit Siromoni of St. Thomas’ College of Engineering and Technology, who always stood by the authors, and provided whatever support is needed to complete the book. The first author wishes to express her deep gratitude to her parents: Mrs. Sheela Chakraborty and Mr. Amalendu Chakraborty, who always stood by her throughout her life, and guided her in her time of crisis. She is equally grateful to her elder sister Chandana, and brother-in-law Saspo-da. The presence of Kuchai and Pupu acted as an inspiration to continue writing with pleasure. Finally, she acknowledges the inspiration she received from her teacher Prof. Amit Konar of Jadavpur University, who nurtured her academic career since her post-graduation. The second author wishes to thank all his family members, and particularly his son Dipanjan for sharing his emotions during the period of writing this book.

June 2009

Aruna Chakraborty Amit Konar

Contents Contents

1

Introduction to Emotional Intelligence……………………………….. 1.1 What Is Emotional Intelligence?........................................................ 1.2 Causes of Emotions........................................................................... 1.3 Typical Characteristics of Emotion................................................... 1.4 Basic Components of Emotion.......................................................... 1.4.1 The Cognitive Component...................................................... 1.4.2 The Evaluative Component.................................................... 1.4.3 The Motivational Component................................................. 1.4.4 The Feeling Component.......................................................... 1.5 Rationality of Emotion....................................................................... 1.6 Regulation and Control of Emotion................................................... 1.7 The Biological Basis of Emotion....................................................... 1.7.1 An Affective Neuro Scientific Model..................................... 1.8 Self Regulation Models of Emotion.................................................. 1.9 Emotional Learning........................................................................... 1.10 Mathematical Modeling of Emotional Dynamics............................ 1.11 Controlling Emotion by Artificial Means........................................ 1.12 Effect of Emotion Modeling on Human Machine Interactions….... 1.13 Scope of the Book............................................................................. 1.14 Summary........................................................................................... References..................................................................................................

1 1 4 4 5 6 6 6 6 7 8 10 11 13 17 18 21 22 23 25 31

2

Mathematical Modeling and Analysis of Dynamical Systems………. 2.1 Introduction………………………………………………………… 2.2 System Modeling and Stability.......................................................... 2.3 Stability Analysis of Dynamics by Lyapunov Energy Functions….. 2.3.1 Stability Analysis for Continuous Dynamics……………….. 2.4 Stability Analysis of Fuzzy Systems………………………………. 2.4.1 Mamdani Type Fuzzy Systems……………………………... 2.4.2 Takagi-Sugeno Type Fuzzy Systems………………………. 2.4.3 Stability Analysis of T-S Fuzzy Systems…………………... 2.5 Chaotic Neuro Dynamics and Lyapunov Exponents………………. 2.6 Emotional Dynamics and Stability Analysis………………………. 2.7 The Lyapunov Exponents and the Chaotic Emotional Dynamics…. References……………………………………………………………….

35 35 36 40 42 45 45 46 48 52 54 55 60

XII

Contents

3 Preliminaries on Image Processing……………...................................... 3.1 Introduction…………………………………………………………. 3.2 Discrete Fourier and Cosine Transforms…………………………… 3.3 Preprocessing and Noise Filtering………………………………….. 3.3.1 Neighborhood Averaging…………………………………… 3.3.2 Median Filtering…………………………………………….. 3.3.3 Thresholding………………………………………………… 3.4 Image Segmentation Algorithms……………………………………. 3.4.1 Boundary Detection Algorithms…………………………….. 3.4.2 Region Oriented Segmentation Algorithm………………….. 3.4.2.1 Region Growing by Pixel Aggregation…………….. 3.4.2.2 Regions Splitting and Merging…………………….. 3.4.2.3 Image Segmentation by Fuzzy Clustering…………. 3.5 Boundary Description………............................................................. 3.5.1 Chain Codes…………………………………………………. 3.5.2 Fourier Descriptors………………………………………….. 3.5.3 Regional Descriptors………………………………………… 3.6 Object Recognition from an Image…………………………………. 3.6.1 Unsupervised Clustering…………………………………….. 3.6.2 Supervised Classification……………..................................... 3.6.3 Image Matching……………………………………………… 3.6.4 Template Matching………………………………………….. 3.7 Scene Interpretation………………………………………………… 3.8 Conclusions…………………………………………………………. References………………………………………………………………... 4

Brain Imaging and Psycho-pathological Studies on Self-regulation of Emotion………………........................................................................ 4.1 Introduction………………………………………………………… 4.2 Emotion Processing by the Human Brain………………………...... 4.2.1 The Amygdale………………………..................................... 4.2.2 Animal Studies on Amygdale………………………............. 4.2.3 Fear and Threat Perception of the Amygdale………………. 4.2.4 The Orbitofrontal Cortex (OFC) ………………………........ 4.2.5 Animal Lesions to Prefrontal OFC………………………..... 4.2.6 Neuro-psychology and Functional Neuro-imaging Studies on OFC Behavior………………………................................ 4.2.7 The Insula………………………........................................... 4.2.8 Experiment of Selective Lesion of Insula Cortex…………... 4.2.9 The Anterior Cingulated…………......................................... 4.2.10 Emotion Monitoring by the Cingulated Cortex………….... 4.3 The Role of Medial Frontal Cortex in Self-regulation of Emotion... 4.4 The Anterior Cingulate Cortex as a Self-regulatory Agent………... 4.5 Voluntary Self-regulation of Emotion……………........................... 4.5.1 fMRI Studies on Voluntary Regulation of Sexual Arousals... 4.5.2 Voluntary Regulation of Sadness in Adults…………………

63 63 64 66 66 66 67 69 69 75 76 76 77 84 84 86 86 87 87 88 88 89 90 91 91

93 93 94 94 94 95 96 96 96 97 97 97 98 99 99 101 102 103

Contents

4.5.3 Neural Circuitry Underlying Emotional Self-regulation…… 4.6 EEG Conditioning and Affective Disorders…………...................... 4.6.1 Pain Conditioning in Rats…………....................................... 4.6.2 Clinical Study of Depression Using EEG…………............... 4.6.3 EEG Analysis for Premenstrual Dysphoric Disorder………. 4.7 Emotion Dysregulation and Psycho-pathological Issues…………... 4.7.1 Emotinal Dysregulation in Childhood from Non-clinical Samples..…………...…………...…………...…………........ 4.7.2 Clinical Samples for Emotional Dysregulation for Children..…………...…………...………….......................... 4.7.3 Emotion Regulation in Adulthood………….......................... 4.7.3.1 Non-clinical Studies…………...…………............... 4.7.3.2 Clinical Studies…………......…………................... 4.8 Conclusions…………...…………...…………...…………............... References…………...…………...…………...…………...…………...... 5

6

XIII

103 104 105 106 106 107 107 108 109 109 110 110 124

Fuzzy Models for Facial Expression-Based Emotion Recognition and Control............................................................................................... 5.1 Introduction………………………………………………………… 5.2 Filtering, Segmentation and Localization of Facial Components….. 5.2.1 Segmentation of the Mouth Region………………………… 5.2.2 Segmentation of the Eye-Region…………............................ 5.2.3 Segmentation of the Eyebrow Constriction………………… 5.3 Determination of Facial Attributes…………..…………………….. 5.3.1 Determination of the Mouth-Opening…………………........ 5.3.2 Determination of the Eye-Opening…………………………. 5.3.3 Determination of the Length of Eyebrow-Constriction…….. 5.4 Fuzzy Relational Model for Emotion Detection…………..……….. 5.4.1 Fuzzification of Facial Attributes…………..………………. 5.4.2 The Fuzzy Relational Model for Emotion Detection……….. 5.5 Experiments and Results…………..……………………………….. 5.6 Validation of the System Performance…………..………………… 5.7 Proposed Model of Emotion Transition and Its Control…………… 5.7.1 The Model…………..………………………………………. 5.7.2 Properties of the Model…………..…………………………. 5.7.3 Emotion Control by Mamdani’s Model…………..………… 5.7.4 Architecture of the Proposed Emotion Control Scheme……. 5.7.5 Experiments and Results…………..………………………... 5.8 Conclusions…………..…………………………………………….. References…………..……………………..…………..…………………

133 133 135 136 137 139 139 139 139 139 141 141 141 144 152 153 153 155 157 163 163 166 171

Control of Mental Stability in Emotion-Logic Interactive Dynamics………………………………….….......................................... 6.1 Introduction……………..……..…………………………………… 6.2 Stable Points of Non-temporal Logic………………………………

175 175 176

XIV

Contents

6.2.1 Finding Common Interpretations of Propositional Statements…………………………………………………... 6.2.2 Determining Stable Points of Logical Statements………….. 6.3 Stable Points in Propositional Temporal Logic……………………. 6.4 Stability of Propositional Temporal System Using Lyapunov Energy Function………..................................................................... 6.4.1 The Lyapunov Energy Function……………………………. 6.4.2 Stability Analysis of Propositional Temporal System.……... 6.5 Human Emotion Modeling and Stability Analysis………………… 6.5.1 Stability Analysis of the Emotional Dynamics……………... 6.5.2 Weight Adaptation in Emotion Dynamics by Hebbian Learning……………...……………...……………................ 6.5.3 Improving Relative Stability of the Learning Dynamics…… 6.6 The Fuzzy Temporal Representation of Phenomena Involving Emotional States……………............................................................ 6.7 Stabilization of Emotional Dynamics……………............................ 6.8 Psychological Stability in Emotion-Logic Counter-Actions………. 6.9 Conclusions…………….................................................................... References…………….............................................................................. 7

8

Multiple Emotions and Their Chaotic Dynamics……………………. 7.1 Introduction ………………………………………………………... 7.2 Proposed Model for Chaotic Emotional Dynamics ………….......... 7.3 Effect of Variation in Parameters of the Emotional Dynamics…..... 7.3.1 Variation in aii …………………………………….………... 7.3.2 Variation in cij …………………………………….………... 7.3.3 Variation of bij …………………………………….………... 7.4 Chaotic Fluctuation in Emotional State …………………………… 7.5 Stability Analysis of the Proposed Emotional Dynamics by Lyapunov Energy Function ….......................................................... 7.6 Parameter Selection of the Emotional Dynamics by Experiments with Audio-Visual Stimulus……………………………………….. 7.7 A Stabilization Scheme for the Mixed Emotional Dynamics……… 7.8 Conclusions………………………………………………………… References……………………………………………………………….. Electroencephalographic Signal Processing for Detection and Prediction of Emotion………………………………………………….. 8.1 Introduction………………………………………………………… 8.2 EEG Prediction by Adaptive Filtering……………………………... 8.2.1 LMS Filter…………………………………………………... 8.2.2 EEG Prediction by NLMS Algorithm……………………… 8.2.3 The RLS Filter for EEG Prediction………………………… 8.2.4 The Kalman Filter for EEG Prediction……………………... 8.2.5 Implication of the Results…………………………………... 8.3 EEG Signal Prediction by Wavelet Coefficients…………………...

177 179 180 183 183 183 185 186 189 190 190 194 195 197 207 209 209 210 212 212 214 215 217 219 220 226 228 232

235 235 236 237 238 240 242 245 247

Contents

9

XV

8.4 Bio-potential Signals in Emotion Prediction………………………. 8.4.1 Principles in SVM…………………………………………... 8.5 Emotion Clustering by Neural Networks…………………………... 8.6 Conclusions………………………………………………………… References………………………………………………………………..

252 253 256 259 259

Applications and Future Directions of Emotional Intelligence……... 9.1 Introductions……………………………………………………….. 9.2 Application in Human-Machine Interactive Systems……………… 9.2.1 Input Interfaces……………………………………………... 9.2.2 Output Interfaces……………………………………………. 9.2.3 Embodiment of Artificial Characters……………………….. 9.3 Application in Multi-agent Co-operation of Mobile Robotics…….. 9.4 Emotional Intelligence in Psycho-therapy…………………………. 9.5 Detection of Anti-social Motives from Emotional Expressions…… 9.6 Applications in Video Photography/Movie Making……………….. 9.7 Applications in Personality Matching of People for Matrimonial Counseling…………………………………………………………. 9.8 Synthesizing Emotions in Voice…………………………………… 9.9 Application in User Assistance Systems…………………………... 9.10 Emotion Recognition from Voice Samples……………………….. 9.10.1 Speech Articulatory Features……………………………... 9.11 Personality Building of Artificial Creatures………………………. 9.12 Multimodal Emotion Recognition………………………………… 9.12.1 Current Status…………………………………………….. 9.12.2 Research Initiative at Jadavpur University……………….. 9.13 Parameter Identification of Emotional Dynamics Using EEG and fMRI Brain Imaging………………………………………………. 9.13.1 System Identification Approach to EEG Dynamics Modeling by Evolutionary Algorithms…………………… 9.13.2 Genetic Algorithm in Emotional System Identification….. 9.13.3 Particle Swarm Optimization in Emotional System Identification……………………………………………… 9.13.4 Differential Evolution Algorithm in Emotional System Identification……………………………………………… 9.14 Conclusions………………………………………………………... References………………………………………………………………..

261 261 263 263 263 264 265 266 267 269

10 Open Research Problems……………………………………………..... 10.1 Introduction……………….............................................................. 10.2 Reasoning with Emotions…………………………………………. 10.3 Uncertainty Management in Emotion-Based Reasoning………….. 10.4 Selected Open Problems................................................................... 10.4.1 Multi-modal Emotion Recognition…………………….......

270 271 272 273 273 275 277 277 278 283 283 285 286 289 290 291 295 295 297 298 299 299

XVI

Contents

10.4.2 Artificial Control of Emotion by Takagi-Sugeno Method………………………………………………...….. 10.4.3 Determining Manifestation of Emotion on Facial Expression from EEG Signals……………………………. 10.5 Further Readings for Researchers……………………………….... References…………………………………………………………….....

299 299 300 301

Appendix………………………………………………………………..

305

Author’s Biography …………………………………………………...

319

Subject Index…………………………………………………………...

321

1 Introduction to Emotional Intelligence

This chapter provides an introduction to emotional intelligence. It attempts to define emotion from different perspectives, and explores possible causes and varieties. Typical characteristics of emotion, such as great intensity, instability, partial perspectives and brevity are outlined next. The evolution of emotion arousal through four primitive phases such as cognition, evaluation, motivation and feeling is briefly introduced. The latter part of the chapter emphasizes the relationship between emotion and rational reasoning. The biological basis of emotion and the cognitive model of its self-regulation are discussed at the end of the chapter.

1.1 What Is Emotional Intelligence? There exist quite a large number of contemporary theories on emotional intelligence [5], [21], [31]. These theories have been developed from different angles of understanding emotions. Naturally, researchers of different domains attempted to interpret the phenomena of emotions [36], its arousal and control from the point of views of respective subject domains. For example, physiologists co-relate emotions with the changes in the neurological and hormonal activity of the humans, which are caused by the various physiological conditions of the human body, including blood pressure, blood circulation, respiration, body temperature, gastrointestinal activity and many others. Psychologists, on the other hand, consider emotion to have four main evolvable phases, such as cognition, evaluation, motivation and feeling. Philosophers are mainly concerned with the issues of emotion and rationality. The book provides a detailed discussion on emotions from different angles with an ultimate aim to formalize a unified theory of emotional intelligence from a cognitive standpoint. The true explanation of the causes of emotion-arousals remained a mystery until this date. However, most of the researchers are of the opinion that there is a strong correlation between perceiving significant changes in one’s personal situations and arousal of emotions. According to them, humans compare their current situations with previous situations, and when the level of current stimulation far exceeds the level they have experienced long enough to get accustomed to it, A. Chakraborty and A. Konar: Emotional Intelligence, SCI 234, pp. 1–33. © Springer-Verlag Berlin Heidelberg 2009 springerlink.com

2

1 Introduction to Emotional Intelligence

arousal of emotion takes place. Experimental observations reveal that arousal of common/simple emotions, such as sadness, happiness, disgust, fear, love, hatred and even sexual desire supports the above phenomenon. It may further be added that the principles of comparisons that humans adopt depend on an individual’s own basis of judgement, and therefore is not free from personal bias. It may be noted that the perceived changes in situations, that cause arousal of emotions, need not be always genuine. In fact, humans on occasions consider imaginary situations causing generation of emotions. It is a well-known phenomenon that people sometimes are unnecessarily tensed and afraid of unknown situations. Emotional intelligence (EI) is a new discipline of knowledge. Philosophically, it refers to the competence to identify and express emotions, understand emotions, assimilate emotions in thought and regulate emotions in the self and in others [19]. Apparently, the definition follows from our commonsense understanding about emotions. The word competence in the present context, perhaps, indicates the degree or relative power of judgment of persons to recognize/understand emotions. The power of representing emotions in thoughts, according to the definition, also is a measure of EI. Control/regulation of emotion too is considered as a measure to qualify the term competence in this context. There exists a vast literature on EI [5], [14], [20], [21], [27], [34] considering one or more aspects of the above emotional attributes toward intelligence. Most of the literature considers emotions from philosophical and/or psychological standpoint, and ignores the scope of possible automations to detect and regulate human emotions. The book provides computational models for detection and regulation of human emotions. The subject of emotional intelligence, which was originated in early 1980’s, could draw attention of a limited group of people within academic circles and philosophers. In the last decade, emotional intelligence has earned widespread publicity because of significant progress in experimental psychology. The most promising best-selling title on Emotional Intelligence is due to Daniel Goleman [14]. According to Goleman, emotional intelligence was believed to have significant impact on individuals from the point of view of cognitive ability. Naturally, people with a high level of emotional intelligence may not have high intelligent quotient (IQ). Apparently the hybrid term “Emotional Intelligence” seems to be contradictory as it includes emotions, conveying the idea of irrational passion and intelligence, which is characterized by rationality of thoughts. In fact, since the beginning of the era of emotional intelligence, the conflict between emotion and rationality was given much importance, and no legitimate solution to this problem could be traced until the beginning of this decade. Most of the researchers in the last few decades emphasized the importance of intellect at the expense of emotion. This trend, however, is being changed as the experimental research on emotions demonstrated many promising results, citing the need of the emotional component over the intellect component of EI. Formalization of emotional intelligence from the classical definition of emotion and intelligence, though apparent, is hard to conceive. Traditionally, researchers in

1.1 What Is Emotional Intelligence?

3

Artificial Intelligence [28] define intelligence from different perspectives, including reasoning, learning, planning and perception [17]. Naturally, the art of carrying emotions with intelligence may not be always consistent. The following examples highlight some aspects of this issue. Consider a reasoning problem, comprising a set of rules and facts, where all the facts are not free from emotional bias. The inferences under this circumstance may contradict its premise or other information, similar to reasoning in nonmonotonic systems. Consequently, the effect of our emotional component on reasoning cannot be ignored. While reasoning refers to deriving new inferences from available facts and rules, planning refers to determining the sequence of rules that leads to a given goal state from a pre-defined starting state. Naturally, like reasoning, planning too may suffer from emotional contamination. It is also noted that humans suffering from anxiety/fear are unable to learn new ideas. Apparently, it reveals that emotion has a great role to play in the learning process. The interpretation of this phenomenon can be given as follows. Humans under strenuous conditions cannot select the right learning strategy, perhaps because of malfunctioning of proper hormonal/neural coordination. Perception too is highly influenced by our emotional counterpart. For example, the act of perceiving by humans may be disturbed, when they suffer from anxiety/anger/fear. Unfortunately, there is hardly any literature on the effect of emotional disturbance on our perception. A little thinking, however, reveals that humans perceive objects at two levels: sensory level and interpretation level. Hopefully, percepts at sensory level are not disturbed by emotional bias, but interpretation of the percepts at higher thought processing level suffers because of emotional disturbances. Emotional Intelligence (EI) thus can alternatively be defined as a subject that emphasizes emotion as a primary component of intellectual activities, including reasoning, learning, planning and perception. We are afraid that unfortunately, there is hardly any literature covering these aspects of Emotional Intelligence. The book, however, explores EI from a completely different perspective. It provides different computational models of emotional intelligence, so as to bridge the gap between philosophical foundations and psychological findings on emotions. These models will serve as the basic building blocks toward automation for the next generation human/machine interface. For example, emotions can be detected from facial expressions or other external manifestations like gesture and posture. Automatic extraction of emotions from the images containing facial expression, gesture and posture thus calls for computational models. Emotions can alternatively be detected from the psychological experiments undertaken by brain imaging techniques. The book provides various methods for detection of emotions by image analysis. Regulation of emotion by artificial means also is an important issue under EI. Experiments on emotion regulation envisage that music, videos and ultrasonic signals have a good effect on the regulation/control of emotion. The book undertakes experiments and suggests new techniques for emotion control by artificial means.

4

1 Introduction to Emotional Intelligence

1.2 Causes of Emotions The psychological interpretation on the cause/arousal of emotion remained a mystery until this date. From a philosophical angle, the cause in arousal of emotion can be interpreted as the substantial positive/negative changes in our personal situations [5]. The above issue raises two fundamental questions. The first question is: how do we define/interpret positive/negative changes? The second question is: how do we label changes in personal situations as substantial? The answer to the first question lies in realistic interpretation of changes in our personal situations. The normative state of human mind is perturbed, when we come across relatively positive (hopeful) or negative (full of despair) changes in personal situations. The degree or extent of these perceived changes, however, depends on our “focus of attention” [5]. People may have differences in the focus of attention of their (selected) attributes of interest. Thus, when the changes in the selected attributes go beyond a threshold, the perceived changes seem to be significant, and cause people to arouse emotions. The following example demonstrates the role of focus of attention. Suppose, a person Y is envious of person X’s achievements. Under this circumstance the focus of concern in Y’s undeserved inferiority imbibes the evaluative theme of the person to compare his present status with that of X. This causes a change in the emotional status of the person making him/her unhappy. Sometimes the cause and the object causing emotion are same. Mice, for example, are both cause and object of our fear. There are however examples, demonstrating that the cause and the object causing emotion are different. For example, a man may be angry at his wife, although the real cause is having been insulted by his boss. This example shows that the object (anger on his wife) is hardly related to the cause. Briefly speaking, the emotional cause always precedes the emotion and the focus of concern and the emotional object are the two possible causes of our emotional experience. Ben-Ze’ev and the contemporary philosophers differentiate focus of concern with focus of attention. According to them, the focus of concern usually refers to our personal situations, whereas focus of attention may be influenced by our visual sensory perception. On occasions, both of them may go together, but there are examples where focus of concern is given priority in arousal of emotion. BenZe’ev [5] cites an interesting example to illustrate the difference of the above two issues in causing arousal of emotion. It is a common experience that we feel resentment because of improper activity/behavior of a person. Here, the focus of attention is the improper behavior, but the focus of concern is its implication on our own states and self-image. Thus, the focus of concern triggers our emotional systems, making us anxious and unhappy.

1.3 Typical Characteristics of Emotion According to Ben-Ze’ev [5], emotion may be characterized with respect to four common attributes: intensity, brevity, partiality and instability.

1.4 Basic Components of Emotion

5

Intensity: Emotions usually are of relatively great intensity. This is because of a common phenomenon that our mental system, giving rise to emotion, usually takes longer time to adapt itself with the change of the environment. Naturally, a large change in mental state causes intense reactions, forcing aggravation in emotion. Brevity: Typically emotions are transient states of our psychological processes. Naturally, the durations of emotions are short in time of the order of few minutes to limited few hours. The philosophical interpretation of brevity in emotion can be given as follows. In the transient phase, mobilization of physiological, neuronal and hormonal resources takes place, and such mobilization is a cause of systems instability in the transient phase. The instability of a psychological system cannot prolong as it has its own control to prohibit the mobilization process. This explains the brevity of emotions. Partiality: Emotions usually express personal and interested perspectives found on a narrow target, such as a person or an object. It usually directs our attention by selecting what attracts and holds our attention. It makes us preoccupied with some objects and oblivious to others. Generally, all objects do not have equal emotional significance to us. Depending on our focus of attention to an object, the intensity of emotion is determined. Naturally, objects of higher preferences have higher emotional value to us. Instability: The last important characteristic of emotion is the instability of our psychological/physiological processes. Emotions usually indicate a transient, in which the preceding context has changed, but no new context has been stabilized. The instability associated with intense emotion is revealed by their interferences with activities that require a higher degree of co-ordination or control. For instance, someone trembling with fear or anger cannot easily thread a needle.

1.4 Basic Components of Emotion The emotion generating process can be described as a sequence of four psychological phases. These phases are usually referred to as components, describing a conceptual partitioning about the elements of an emotional experience. BenZe’ev considered emotion to have 2 main components: i) intentionality and ii) feeling. Intentionality refers to a subject-object relationship, whereas feeling indicates the subject’s own state of mind. For example, a person in love has a particular feeling (thrill) that he experiences when the lovers are together. The intentional component in the present context is the person’s knowledge of his beloved, his evaluation about her attributes and his motivation/desire to get involved in the love affair. The intentional component again is subdivided into 3 basic components/modules: i) cognition ii) evaluation and ii) motivation (Fig. 1.1).

6

1 Introduction to Emotional Intelligence

1.4.1 The Cognitive Component The cognitive component usually provides necessary information about a given situation. From common sense reasoning, it can be revealed that without any information from our cognitive counterpart, emotion cannot be aroused. For example, without the knowledge of dangers on riding motorcycles, one does not have fear of riding a motorcycle. The cognitive component is responsible to match a given situation to already known situations, and commands the evaluative component to evaluate the emotion from the suggested features of the new experience. 1.4.2 The Evaluative Component The most important module in emotion arousal perhaps is the evaluative component. Every emotion entails a certain evaluation. For example, hatred implies the negative evaluation of a person, pride indicates the positive evaluation of oneself, whereas regret involves an evaluation of a wrong commitment by the person. A highly positive evaluation of a person’s action causes generation of love, affection and admiration. In absence of any evaluative component, people usually are indifferent to the behaviors and actions of other. Naturally, without evaluative component, there would be no feedback to the emotion generating system, and consequently no emotions would be aroused under this circumstance. The positive feedback from the evaluative component to the emotion generating system enhances our expectation, as our action seems to be favorable to the desired environment. On the other hand, when a negative feedback is generated by the evaluative component, we lose our hopes and desires, causing self-arousal of anger, distress and fear. 1.4.3 The Motivational Component The motivational component promotes our desire or readiness to maintain or change our emotional states. For example, the anger and sexual desire can be revealed by the overt behavior of a person. On the other hand, the envy and hopes, which can be regarded as “dispassionate” emotions, are expressed merely as a desire and naturally it proves the lack of sufficient role of motivational component. Many neuro-physiologists are of the opinion that the hormonal system plays a significant role in the arousal of emotion, and consequently enhance our readiness to show overt behavior. 1.4.4 The Feeling Component “Feeling” in an emotional system, has several meaning. It includes bodily sensation, awareness of tactile quality, moods and in general awareness. In the present discussion, we confine the meaning of feeling in modes of awareness, which express our own state of the mind. The feeling usually depends on our intentional capacities. In the literature [4] feelings are synonymously used as emotions, and

1.5 Rationality of Emotion

7

emotions are often defined with reference to feelings. For example, in our daily dialogue, we often use to say that we feel ashamed or embarrassed to describe our respective emotion. Despite the importance of feelings in emotion, equating the two is incorrect as emotions have intentional components in addition to the feeling components.

1.5 Rationality of Emotion Most people regard emotion as an impediment to rational reasoning and hence as an obstacle to normal functioning. Such belief presumes many assumptions, all of which are not necessarily consistent with each other. The assumptions are listed below. a) Intellectual thinking is the essence of mental faculty. b) Emotions are not the outcome of intellectual thinking and consequently they are non-rational c) Since emotions lead us to distorted conclusions, they are irrational in a normative sense. Intentional Component

Experiencing new situations

Cognitive Component

Selects features to evaluate emotion

Evaluative Component

Motivational Component

Motivation/de-motivation of emotion arousal depending on positive/negative evaluation of emotion

Feeling Component

Emotion

Fig. 1.1. The modular structure of the emotion-generating process.

8

1 Introduction to Emotional Intelligence

According to Ben-Ze’ev [5], the assumption (a) is incorrect. Ben Ze’ev also remarked that accepting (b) refutes (a) and (c). In this book, we shall consider the stabilization of our behavior when emotions contradict our reasoning counter-part. We propose two alternative schemes to stabilize our behavior amidst the contradictory response of our emotional reasoning models. The first scheme attempts to stabilize our behavior by over-empowering our rational reasoning to emotions. In the second scheme, we consider the significance of the emotional response over our reasoning mechanism. The details of this will be discussed in the later part of this book. Some philosophers are of the opinion that both emotional reasoning and intellectual reasoning follow the basic rules of formal logic, such as the rules of contradiction and identity. But the main difference appears because of their difference in origin, as they refer to two distinct contexts. The above distinction has a sense of similarity to Kant’s distinction [16] between formal and transcendental logic. The following example illustrates the distinction between emotional and intellectual reasoning. Our emotional counterpart always prefers people who are close to us most of the time. In other words, we give more weightage to our near and dear ones. On the contrary, intellectual thinking reveals that our distance from an object does not change its value or weightage. Naturally, on many occasions giving more preference to the emotional faculty of the mind than the intellectual counterpart is appropriate. A question that naturally arises: should we then call emotional intelligence irrational? A commonsense thinking reveals that the answer is situation dependent. However, it may be stressed that reasoning by taking into consideration both emotion and logic undoubtedly is not an easy task.

1.6 Regulation and Control of Emotion Emotion regulation or control refers to any initiative that we need to take to control the over-arousal of emotion. For example, when someone is too sad, we can make him/her cheerful or at least normal by distracting him/her from his/her current mental engagements. On many occasions, regulation of emotion becomes almost mandatory to prohibit our over-excessive display of emotional feelings. For instance a policeman on duty should control his anger/arrogance to handle situations that needs a display of sympathy. A hair-dresser should be able to read the emotion of his customer and should display emotional feelings to satisfy the customer. In fact, many real world trades demand control of emotions and we are habituated to do the needful. The power of regulating emotion increases with age. Children readily learn diverse strategies to regulate their emotion, such as self -gratification, cognitive distraction, nurturing social interaction and taking corrective action. It has been noticed that younger children under 5 years old, usually face no difficulty in finding means to regulate their emotion than older ones (> 8 years old). But they prefer behavioral means, whereas the older ones opt for cognitive intervention. Regulation of emotion and regulation of manifestation of emotion are two important issues to be considered in the present context. Controlling the manifestation of

1.6 Regulation and Control of Emotion

9

emotion is easier to achieve, as it requires useful pretence only. For instance, calm people, who want themselves to be taken seriously by others sometimes pretend that they are angry. Naturally, pretension is the regulation of the manifestation of emotion. Pretension can be of two basic types. In the last example, the anger of the calm people was pretense, whereas there are examples such as exhibition of anger while collecting revenues for payments by bill-collectors. Sometimes, it is difficult to separate the regulation of an emotion from the regulation of its manifestation, because pretense can easily be transformed into real emotion. In other words, when people are induced to behave as if they feel an emotion, they sometimes report that they really feel it. There exist two alternative methods of regulating the manifestation of emotion. The first method refers to changing the structural components of emotion arousal, whereas the second method is concerned with changing emotional sensitivity of people. A brief outline to these is given below. As already discussed earlier, an emotional system consists of cognition, evaluation, motivation and feeling components. Naturally, regulation of emotion can be accomplished by cognitive, evaluative and motivational means. Motivational means facilitate or prevent certain type of changes in the personality, thereby regulating emotional strength (intensity). The evaluative means can be modified by changing our evaluative structure, such as the method of comparing our own situation with others. In fact, people adopt various learning strategies from their own experiences and employ them to modify their evaluative means of regulating emotion. The cognitive means of regulating emotion attempts to segregate some facts or phenomenon of our memory from the rest. Which information or interpretations are to be selected from the cognitive storage is mainly decided by our focus of attention and interest. It is indeed important to mention here that both focus of attention and interest change depending on our own desire and feeling. Naturally, the first step to select appropriate information from the cognitive storage is to regulate our own desire and feelings. Sometimes combination of one or more of the above means may be adopted in regulating arousal of emotional strength. The alternative method to regulate emotion is by making an attempt to adapt our emotional sensitivity. Emotional sensitivity refers to sensitivity in emotion to external influence. Emotional sensitivity is an intrinsic parameter of our emotional system. Through practicing, people can gradually change this parameter, though it may have side effect on their personality. By changing emotional sensitivity, we can ignore many odd situations that we dislike, and can control our feelings and subsequently emotion arousal. Emotional sensitivity naturally increases with aging of people. For instance, it has been observed that aged people do not react on silly activities of others, whereas younger people exhibit temporal unpleasant behavior as a reaction to such situations. How can we reduce our emotional sensitivity? The most important consideration in reducing emotional sensitivity lies in the evaluative measure of the emotional structure. Naturally, a less evaluative feedback does not cause sufficient excitation to our emotional system, thereby regulating our emotional sensitivity.

10

1 Introduction to Emotional Intelligence

1.7 The Biological Basis of Emotion A portion of the human cortical and sub-cortical brain system has been found to be liable for our emotional expression. Psychologists are still catering for a suitable interpretation of emotional intelligence. There remains a controversy on the origin of emotion even today. Some researchers are of the opinion that emotion originates because of our hormonal influence, whereas the other group favors neuropsychological views in the origin of emotion. However, medical physiologist are of the opinion that hormonal regulation mainly depends on the stimulation of a part of human brain called amygdala. Experimental results reveal that secretion of various anterior pituitary hormones specially the Gonedo Trophins and Adrenocoerticotropic hormones are controlled by external simulation by the amygdale. Naturally many of our emotion such as love and despair are due to the effects of external stimulation of the amygdala. It has also been noticed that various postures and gestures exhibiting our arousal of emotion have co-relation with the stimulation of the amygdala. For example, tonic movements, such as bending our body and movements associated with olfaction, such as licking, sucking, chewing and swallowing are directly dependent on the stimulating condition of the amygdala. In addition, the external manifestation on our face and body movements due to fear and pleasure also greatly depend on the amygdala. It is clear from the above discussion that the behavioral response of a person to external stimulation is mainly controlled by the amygdala. The common sense belief of antagonistic behavior between emotion and reasoning is also supported by the neurological behavior of the human brain. The high level reasoning capability of the cerebral cortex and the power of emotional exposition by the sub-cortical brain system, especially by the amygdala are already established in the literature of medical psychology [15], [21]. A recent review by Goleman [14] on the comparative performance of the cerebral cortex and the sub-cortical brain reveals that during an excessive outburst of emotion (especially in passion), the reasoning ability of the cerebral cortex is stolen by the subcortical structure like the amygdala. Damasio [12] however noted that a damage to certain areas of the brain associated with emotion also impairs the cognitive ability to take decisions. This indicates that emotion is essential to rational reasoning. To reconcile the above two conflicting views on the functional role of emotion, Goleman later extended his views to handle the situation [14]. A number of pharmacological studies have been used to investigate the controlling power of neurotransmitters in behavioral response of the people. Experimental studies undertaking recording of single neuron activity was supplemented by ethological observation of behavior in the animal’s normal environment. Experiments on patients, having their brains been damaged by trauma or disease, exhibit capability of patients to localize emotional experience and behavior. Drug studies in humans also provide useful evidence, but the pharmacology of drugs generally is too complex to ensure which neural path- ways are mediating on pharmacological effects.

1.7 The Biological Basis of Emotion

11

Psycho-physiological recording of autonomic and central nervous system activity during emotional encounters, are taken to describe the levels of metabolic activity within specific brain regions. Current recording techniques include brain imaging by Positron Emission Tomography (PET) and functional Magnetic Resonance Imaging (fMRI). The fMRI image visualizes the precise location of the area of the brain, such as the amygdala in Fig. 1.2, while the PET scan image reveals activity in the said brain region, such as the amygdala. The psychological functioning as evident from the behavior and the physiological functioning of the brain obtained by the said imaging techniques have a good co-relation. Such co-relation indicates that many psychological disorders can to some extend be predicted from the behavioral response of the subject. On the other hand, from the similarity of brain images, the behavioral aspect of the subject can also be analyzed. Unfortunately, the origin of control commands over our emotional behavior could not be traced from the physiological models of brain imaging.

Fig. 1.2. The activity in amygdala obtained from the fMRI images (top row) and the PET scan images (bottom row) of two individuals.

1.7.1 An Affective Neuro Scientific Model In late 1990’s Panksepp [25], [26] proposed a general biological model for emotion-arousals. He assumed that mammalian brain supports several distinct (central) neural-control systems to motivate external stimuli and generate subjective emotion. The general criteria for arousal of emotions are listed below following Panksepp [25]. a)

The neural circuit usually are pre-wired and designed to respond to stimuli arising from major external changes.

12

1 Introduction to Emotional Intelligence

b) These circuits can organize various activities, behaviors of the creatures such as changing hormonal levels and controlling motor activities. c) The sensitivity of emotional system can also be controlled by the emotion generating neural circuits. d) Neural activity of causing arousal of emotion sustains longer, even when the external stimuli are absent. e) The emotive circuit comes to an equilibrium condition after a sufficient time of the arousal of emotion. The emotive circuits have reciprocal interactions with brain mechanisms that activates higher level decisionmaking processes and conciousness. Neurotransmitters play a vital role in the arousal of emotions. Experimental investigation reveals that some neurotransmitters such as the catecholamines usually participates in the arousal of most emotions, whereas there exist evidence of specific neurotransmitters such as neuro-peptides to be more emotion- specific. It has also been experimentally established that the co-relation between emotion and behavior is due to hardwired motor outputs of the neural emotive circuits. For instance, Panksepp noticed a close co-relation between i) joyfulness and rough & tumble play, ii) struggling and energetic attacking iii) feeling and fearfulness iv) crying and separation distress, and v) activated exploration and anticipatory excitation. One important aspect of Pansepp’s theory lies in the structural organization of the emotion building in different parts of our brain systems. Most biologists, however, refuted the proposal of Panksepp, claiming that arousal of human emotions are mainly due to the cortical and the sub-cortical structure of the human brain. Further, how the functional behaviors of the individual modules of the brain systems together synthesize the human behaviors is not clearly known until this date. Crudely, we can differentiate the levels of control signals in the cortical and the sub-cortical regions. Brain structure in the neo-cortex are especially concerned with high-level human functions including reasoning, conscious awareness, voluntary choice of actions and linguistic comments. Beneath the cortex, there is a group of structures well-known as “limbic system”, which is presumed to function in humans in a similar manner as it does in lower class mammals to providing a higher level of control of emotion. The famous Goleman’s quote “ Emotional Highjacking” by the limbic system and emotional control by the frontal lobes of the cortex thus makes sense. In late 1990’s Le Doux [20] proposed an interesting concept on sub-cortical control of emotion, especially by the amygdala. It has experimentally been noted that lesions of rat brain disrupt conditioning top fear-stimuli. Experiments also reveal that the key structure in the sub-cortical control of emotion is because of amygdala only. As discussed earlier, the role of amygdala in the generation and control of emotion is an established phenomenon. Both lesions and electric stimulation of the amygdala provoke emotional responses. Le Doux proposed two different pathways in the amygdala for generating “fear responses”. Fierce stimuli receive early sensory analysis at pathway arriving at the thalamus. The road is then bifurcated like a fork at this structure. Thus there exist two pathways to cause

1.8 Self Regulation Models of Emotion

13

emotional response from emotional stimuli. A direct thalamo-amygdala pathway provides a short root to cause prompt emotional responses to coarse-grained information, such as fear by a rope, presuming it to be a snake. There exist an alternative lower root via sensory cortex that permits more detailed analysis of the cause of fear on the basis of its cortical representation. Most people use the latter pathways for handling threat stimuli. A schematic diagram describing the two alternative pathways to amygdala to cause emotional response to fierce stimuli is presented in Fig 1.3. The frontal cortex too appears to be a critical region for causing human emotions. A damage in the frontal cortex may cause a profound change in personality, such as becoming hostile, impulsive and unreliable. The frontal lobes occupy a substantial part of the cortex. In very recent times, it has been found that the prefrontal cortex has the power of more emotional control than the frontal counter part. Rolls [27] noticed that lesions in the orbito frontal cortex is responsible for both anger and irritability, and also impairment in decision making. Such patients appear to lack awareness of the possible consequences of his/her actions. The control of endocrinic system, which some researchers still believe to be an important module in the arousal of emotion, also greatly depends on the structure and the functions of the orbito-frontal cortex.

1.8 Self Regulation Models of Emotion Richard Lazarus [18], [19] in the 1990’s proposed a new model for self-regulation of emotion. According to him, we require an ecological view of the person and environment as an interlinked system. He also stressed the need for interaction between dynamic nature of a person and his environment. When a person acts on the environment, he receives a feedback from the environment, which on some occasions may enhance the person’s own believe about his actions, or teaches him to disregard his current actions. How the learning process helps us adapting and regulating our emotion is an interesting issue to be discussed in this section. The acts of a person, describing certain emotional behavior, may be considered to be targeted towards his personal goal realized with his self-believes. Carver and Scheier [8] proposed an interesting architecture for self-regulation of emotion. A detailed report on this architecture is available in subsequent books and papers [9], [11] by the authors. The starting point for this model is an equation of goal-directed behavior with the operation of a homeostatic feedback control system. For the convenience of the readers, not familiar with the cybernetic system, we briefly introduce here the principles of a feedback control system. In a feedback control system, we sense the response of a process c (t) and compare it with a given (predetermined) reference signal r(t), called set-point. An error detector takes the difference of the measured process response called process value from the set-point, and generates an error signal e(t). The error is then passed through a controller, which generates a control signal u(t) to provide the necessary actuation to the plant or the processes, so as to control the response of

14

1 Introduction to Emotional Intelligence

Sensory cortex

High Road

Low road

Sensory thalamus

Emotional stimulus

Amygdala

Emotional responses

Fig. 1.3. LeDoux’s two amygdalar pathways for fear response.

the plant. An indication in the control of the plant response is obtained by a gradual reduction in the amplitude of the error signal. At one point of time, the error diminishes to zero, signifying that the plant response is same as we wanted, because the process response has become equal to the supplied set-point. Fig 1.4 provides a schematic view of a typical process control system. In the light of the above cybernetic model of control systems, we now present the Carver–Scheier model of self-regulation. Carver-Scheier considers the external disturbance to our cognitive mind as the input to our cybernetic system. This can be regarded as the reference input, like the set-point in a process control system. A logical comparator compares a feedback signal, representing change in our mental status, with the disturbance input, generating an error signal of suitable sign and magnitude. The error signal is generated by the following logical basis. The error detector goes on checking whether the received feedback signal is in tune with the disturbance input. If it is so, the error is zero. If not, a positive (or negative) error is generated by considering whether the feedback signal is less (greater) than the disturbance input. It is worthwhile to mention here that the error signal, in the present context represents whether the subject is achieving his/her personal goals, maintaining security and self-worth, and meeting social standards. The error signal then is passed on to our brain that computes the appraisal or self-discrepancies of ours and generates control commands for corrective actions by provoking us to initiate efforts at coping in order to reduce the discrepancies. This initiative for corrective action changes our mental and behavioral status that acts as a feedback signal to the error detector. Fig. 1.5 presents a schematic representation of the cybernetic model of self-regulation of our emotional states.

1.8 Self Regulation Models of Emotion

+

e(t)

∑

Controller

u(t)

Actuator

Process

15

c (t)

error

Setpoint r (t)

Measurement Unit/Sensor

Fig. 1.4. A schematic view of a closed loop (feedback) control system.

Disturbance input (stressful event)

+

∑ error

Initiate efforts at coping to reduce discrepancy

Change system states

Behavior/ emotional status

−

Fig. 1.5. A schematic view of the Carver-Scheier model to control self-behavior/emotional status by changing internal system states.

How the feedback system actually controls our emotional status is an interesting issue for the researchers of emotional intelligence. As an example, let us consider that a person fails to behave properly in her environment because of the difference in her mental/psychological status in comparison to her fellow beings. If the person values the behavior of her neighbors, an anxiety develops, acting as a disturbance input to her emotional mind pondering her to fight against herself to doubly check her own mental states in comparison with the current social states. If the person is rational then she generates an error signal to remove the discrepancy, she has in comparison with others, and attempts to act and behave to reduce the discrepancy. Consequently, her emotional states of anxiety gets reduced. Thus a disturbance input of anxiety causes self-regulating actions to ultimately control the level of anxiety by changing her behaviors. Wells and Matthews [35] also proposed an alternative model for self-regulation of emotional dysfunction. The term dysfunction is related to our negative emotion. Wells and Matthews considered cognition as an important module in the process of controlling emotional dysfunction (Fig.1.6). Their scheme can be briefly outlined as follows. The architecture of emotional dysfunction consists of three main levels: i) a set of low level processing network, ii) an executive system ii) a self-knowledge level

16

1 Introduction to Emotional Intelligence

Accessing of belief

Selection of generic plan

S-REF Control of action

Controlled Processing

Finish

Appraisal (including active worry)

Intrusion

Intensification / suppression of activity

Monitoring

Cognitive Info.

External Info.

Body State info

Automatic Processing

Fig. 1.6. Wells & Matthew’s dysfunction model.

that represents self believes and generic plan for coping. The low level-processing network generates the necessary cognitive information and our mental and body states information from the environmental input. This cognitive and body states information are then passed on to self-regulative executive function (S-REF) model. This module receives low-level information from the low level processing

1.9 Emotional Learning

17

network and generic plans based on our believes from a high level self-believe network. It also updates our believes in the high-level network and information in the low level network. The information updating in the low level network corresponds to intensification or suppression of activity in the low level site. The adaptive process of the S-REF network continues until changes in the low and the high-level network ceases [21]. The S-REF network in the present context has similarity with the “interrupt model of affect” proposed by Simon in early nineteen sixty’s [33]. The S-REF model is activated by external events that generate self-discrepancy, such as threat or somatic signal or spontaneously arising thought or image. After being activated, the S-REF, network supports processing towards reducing discrepancy by initiating and supervising coping responses, which are implemented by biasing information in the low level networks. The influence of personality on self-regulation is mediated by individual differences of knowledge and believes of the people. The most important aspect of the cognitive models of emotion lies in the belief that cognition has a major role in originating emotions. The dynamics of emotions attempt to update the emotional state based on the belief of a phenomena, some events of which are obtained from information of low level processing unit. The process of updating belief in the cognitive counterpart of emotion helps in selfregulating our emotion. Thus emotional dynamics can be regarded as a homoeostasis system, which has its own self-regulatory action to balance the arousal of emotion by the phenomenological interpretation of the cognitive counterpart. A question then naturally arises what exactly is emotional intelligence in the light of cognitive metaphor? Though there are a number of controversy on the definition of emotional intelligence, a convincing definition that supports both neuro-scientific and cognitive science is given as follows. “Emotional Intelligence may be defined as a quality of executive control system for emotion regulation supported by sites in frontal cortex [27]. Lesions to orbito frontal cortex leads to substantial deficit in social problem solving [4]. More general control system for attention and decision making have also been found in the frontal cortex [21], [32]. From the cognitive point of view, emotional intelligence relates to some superordinate executive system, which is capable to handle the “problem of modularity”. [21]. This idea has compatibility with the current theory of linking emotional intelligence to behavioral self-regulation [22], [23] or to affecting copying [3] [29]. Taking into consideration of all the above issues, emotional intelligence may be defined as an executive system capable of adaptive selections of evaluative and action oriented processing routines. Wells and Mathews [35] stressed the need for supportive learning environment to cause substantial changes to long-term memory as a consequence of emotional self-regulation.

1.9 Emotional Learning Bichel et al., in a recent paper [7] presents a novel approach to study the associative learning mechanism using classical delay conditioning paradigms. They employed functional neuro-imaging techniques to characterize neuronal responses

18

1 Introduction to Emotional Intelligence

mediating aversive trace conditioning. During conditioning, neutral auditory tones were paired with an aversive sound (unconditioned stimulus (US)). The neuronal responses evoked by conditioned and non-conditioned stimuli in which 50% pairing of conditioned stimuli (CS) and the US enabled them to limit their analysis to responses evoked by the CS alone. Differential responses of conditioned and nonconditioned stimuli were observed in anterior cingulate and anterior insula region, already implicated with delay fear conditioning. A further activation of the anterior hippocampus supports a view that its role in trace conditioning is to maintain memory trace between the offset of the conditioned stimulus and the delayed onset of the unconditioned stimulus to evoke associative learning in trace conditioning. Fig 1.7 illustrates the detailed description of an evoked response for CS and US.

cs-

CS+ unpaired

CS-

unpaired

CS+ paired

CS-

cs+ unpaired cs+ paired

0

4

11

24

35

46

time (s) Onset CS Onset US

CS

0

US

Trace 3

4

4.5

Fig. 1.7. The evoked response of conditioned and unconditioned stimuli.

The rapid decrease in amygdala-response ensures a scope of emotional learning by the amygdale through enabling of associative changes in synaptic efficacy by neuro-modulatory mechanisms [13], [30]. The hypothesis that we now propose is that brain systems mediate learning and the amygdala plays a vital role to do so by enabling or permitting associative plasticity of the neurons. After acquisition of sensory information, the learned association is expressed as a cortical level reflecting changes in synaptic connection strengths [2], [24]. Once the association is learned there is no need for further modulation of plasticity of the neurons.

1.10 Mathematical Modeling of Emotional Dynamics Philosophical basis for the arousal of emotion has been narrated in section 1.4. On the other hand, the experimental foundation on the arousal of emotion in the amygdala of the human brain has also been presented in sections 1.7 and 1.8. However, there remains a gap between the philosophical overview of emotion

1.10 Mathematical Modeling of Emotional Dynamics

19

arousal and its psychological counterpart. A complete understanding of the emotion arousal process requires filling this gap by mathematical models on emotional dynamics. This section provides an introduction to mathematical modeling of emotion generation, its sustenance and natural decay. Our common experience dealing with people reveals that humans have arousals of a single or mixed (multiple) emotion. When multiple emotions come into play, experimental investigation by PET or fMRI scanning becomes too difficult to comment on the existence of the types of emotion. Emotion modeling can primarily be accomplished by considering either a natural cause of emotion, or ignoring its natural cause but considering its self-growth and natural decay. The first type of model is difficult to realize, as the cause of arousal of a particular emotion may not be unique. The second type of model employs the classical Lotka-Volterra type of dynamics. It may be added here that the Lotka-Volterra model basically describes a predator-prey type relations. So, when there exist only a single emotion, a modified form of Lotka-Volterra type model is needed to ignore the possibility of competition among the multiple emotions. To be specific let x (t) denote the arousal of an emotion at time t. Then the growth equation of x (t) can be written in a simplified form as follows: dx/dt = αx (1—x/k) - βx

(1.1)

In equation (1.1), α, β and k respectively denote the intrinsic growth rate of x, mortality (decay) rate of x and the carrying capacity of x. While the first two terms are self-explanatory, the third needs a little explanation. Carrying capacity, here, means the maximum level of emotion x that our psychological system should arouse. As long as x < k, x is supposed to increase, whereas when x exceeds k, the growth rate should be negative. When x = k, the first term in the right hand side of Equation (1.1) should be zero, signifying that we do not want any more growth in x. In case of mixed emotion, i.e., when a number of emotions co-exist, we model the dynamics of emotion by the following manner. Let xi (t) denote the ith emotion at time t. The general dynamics of the emotion xi now can be written as dxi/dt = αi xi ( 1- xi/k) + ∑ βj xi xj - ∑ χk xk xi, ∃j ∃k

(1.2)

where we presume that the jth emotion with strength xj co-operates with xi, whereas emotion k with strength xk competes with xi. Here, αi denotes the natural growth rate of xi. The co-efficient βj and χk are rate constants and have a dimension 1/secs. The quantifier ∃j (for some j) and ∃k (for some k) denote that there exist at least one j and k. In case there does not exist any xj (i.e., co-operative emotions) we drop the second term in the R.H.S. of equation 1.2. Similarly, if there is no competitive emotion xk, we drop the third term from R.H.S. of equation (1.2). Selection of αi, βj and χk require experimental considerations. We can measure the growth of emotion from external manifestation, such as constriction of our eyebrows, our mouth-opening and eye-opening. Neglecting the time delays for the appearance of the manifestation of emotion on our face from the time of emotion generation, we

1 Introduction to Emotional Intelligence

Mouth-opening (x) in pixels

Stimulated happiness of a subject

Stimulator for exciting happiness

20

0

1

2

3

Frame- no (time) → Fig. 1.8. Mouth-opening of a subject in different frames of a recorded video, obtained through stimulation by a movie clip (top) causing happiness of the subject in different frames.

can easily determine the parameters αi,, βj and χk. We for the sake of completeness, briefly outline the principles of measurement of these parameters. To experimentally determine the parameters of emotional dynamics, we first need to identify specific stimulus, responsible for arousal of different emotions. Stimulus may be short-term (impulsive) or long-term. For long-term stimulus, we also need to know the time samples, where the stimulus has significant changes in its strength of excitation. When audio-visual movie clips are used as stimulus, the strength of individual frames in the clip responsible for exciting an emotion is also determined. Details of this go beyond the scope of the present discussion. Chapter 5 provides experimental procedure for the assessment of audio-visual movies by a team of expert observers. The emotion-transition of a subject, stimulated by a selected audio-visual movie is determined next from the video recording of his/her facial expression. Suppose the movie clip used for excitation contains four frames. If the frames correspond to significant changes in the stimulus, we should expect four frames in the

1.11 Controlling Emotion by Artificial Means

21

video of facial expression, having four major transitions in the gesture. The facial extract, such as mouth-opening, eye-opening and eyebrows’ constriction are all determined from the individual frames. Suppose to measure the degree of happiness, we use only mouth-opening. Fig. 1.8 shows a pattern of mouth-opening of a subject, stimulated by a movie clip causing happiness. Assuming the strength of emotion x(t), synonymous with the measure of mouth-opening, we can easily determine the parameters of equation (1.1) by the following transformation. x(t+1) –x(t) dx/dt= ---------------- = x(t + 1) – x(t) = αx(t) (1- x(t)/k) - βx(t) (t+1) – t Or, x(t =1) x(t) [1 + α - x(t)/k] - βx(t) Since the above equations include three unknown α, β and k, we need to form three equations and solve them. At t=0, x(1) = x(0) [1 + α - x(0)/k] - βx(0). At t=1, x(2) = x(1) [1 + α - x(1)/k] - βx(1). At t=2, x(3) = x(2) [1 + α - x(2)/k] - βx(2). Since x(0), x(1), x(2) and x(3) are all known, solving the above set of equations we can easily obtain α, β and k. For mixed emotions, the parameters like αi, βj and γk are determined in a similar manner. This, however, is an open research problem until this date.

1.11 Controlling Emotion by Artificial Means Experimental investigations undertaken by cognitive psychologists [7] reveal that music, videos, ultrasonic and voice messages can be judiciously selected and used to control the emotion of a subject. Unfortunately, very little progress in this area has been noticed to compare our work presented in this text. A complete scheme for emotion control using the logic of fuzzy sets will be given in Chapter 5. Music, videos and voice messages usually have non-uniform impact on different subjects. However, a judicious selection and ranking of music, videos in their respective library is possible to have more or less similar input on different persons. For instance, a high pitch in music causes happiness in most people. However, to what extent the pitch is increased sometimes becomes can important consideration, as many users may not like very high pitch. Similarly, a fall-off in frequencies of music may sometimes cause anxiety and fear in some people. Before to use the above items for theraputic use, experiments need to be conducted on large population to identify their suitability to cause similar emotions in most people.

22

1 Introduction to Emotional Intelligence

Because of the non-uniform impacts of the same music/video/message on different people, a fuzzy estimator may be employed to judge the quality of the item to cause transition of emotions from one state to another. Why do we select fuzzy logic as an estimator and controller for emotion-transition is an interesting story. The logic of fuzzy sets is rich with sophisticated tools to perform partial matching of events and to reason with partially matched data items. In this book, we consider partial matching of a given music with the necessary requirements to cause transition of emotion from one state to a desired state. Similarly, appropriate visual input/voice message that can transform the emotion to a desired state can also be identified. The details of the scheme of emotion control will be undertaken in a separate chapter.

1.12 Effect of Emotion Modeling on Human Machine Interactions Emotion plays an important role in human machine interactive systems. It can be synthesized and generated as the response of the machine. It can also be read from the facial expression of the users. Emotion synthesis can be of various forms. First, when a machine translates a text into voice, emotion can be added to the voice depending on the context of the subject under reference. This enhances human-machine interactions to a much more intricate level. Emotion can also be synthesized on the facial expression of a humanoid model of the machine, capable of changing facial structures and attributes, describing different emotions. In brief, a multi-media presentation of a theme through oral, voice, structural (face modeling) and animation models of the theme may be constructed. For example, consider the role of a robot commentator. Suppose, a football tournament is broadcasted over TV or radio by robots. Under this circumstance, the robots should be equipped with video cameras as the input device, and micro-phone as the output device. If the commentary is presented over TV, representation of the feeling of the robot in its gesture and posture becomes almost mandatory. Besides synthesis of emotion, determining the emotion of the subject is also of great concern in the next generation human- machine systems, detection of emotion of the subjects can be accomplished by 2 alternative methods. The first method is concerned with grabbing the facial image of each subject, segmenting the important regions of interest on the image, and finally to interpret the emotion involved in the facial expressions by a local analysis on the selected regions of interest. The alternative method to detect human emotion, perhaps is to obtain visual images of the human brain by sophisticated brain imaging technologies, such as magnetic resonance imaging and electroencephalography. These images may be analyzed and interpreted by experienced professionals in experimental psychology and medicine. Several engineering techniques need to be adopted to eliminate noise from facial images and brain images. A discussion on noise filtering algorithm is given in chapter 3. Besides filtering, segmentation too is of important concern for interpretation of emotion of the subjects.

1.13 Scope of the Book

23

1.13 Scope of the Book The book presents a novel approach to modeling emotional dynamics, so as to understand human behaviors and interpret human emotions from its external manifestations on human faces. It also provides a new direction to control human emotions using music, video and voice messages. The book consists of ten chapters. Chapter 2 presents mathematical preliminaries related to modeling of dynamical systems. Different methods of stability analysis for non-linear dynamical systems are introduced. A discussion on fuzzy non-linear dynamics and its stability is given at the end of the chapter. Principles of the Lyapunov exponents in the analysis of chaos are also outlined. Chapter 3 provides an introduction to image processing. The basic image transforms are presented to familiarize the readers with frequency domain information about an image. Principles of image filtering are then introduced. Both classical and fuzzy clustering methods of image segmentation are then presented to give readers an idea in selecting appropriate algorithms. Principles of localization on segmented components in an image are then introduced. Later sections introduce various intelligent tools and techniques used in object recognition and scene interpretation. The section on interpretation of an image is included in the chapter for its completeness. Knowledge based interpretation of a scene is briefly presented. The chapter ends with a discussion on the scope of image processing in the rest of the book. Chapter 4 provides a review on psychopathological and brain imaging studies on emotion regulation. Special emphasis is given on neuro-psychology and functional neuro-imaging studies of brain structures, especially the orbitofrontal cortex, the amygdale and the cingulated cortex. Neural basis of voluntary selfregulation is considered next. The regulation of sexual arousals through fMRI is also undertaken. EEG conditioning and its role in emotion detection is considered next. Experimental studies on operant conditioning neural activity are also discussed here. Special considerations are given on pain conditioning in rats and depression in humans. Chapter 5 outlines a new approach to detection of human emotions from three common facial attributes, such as eye-opening, mouth-opening and the length of eyebrow-constriction [5]. These attributes are detected and measured in a human face, and the fuzzified measures of these items are used to map them onto a emotion space. The mapping from the facial feature space to the emotion space is accomplished by a set of fuzzy production rules. The latter part of the chapter deals with controlling human emotions by providing music, videos, and consolation messages of suitable strength to the subject under reference. The strength of the above items is determined by a specialized fuzzy controller. Experimental results, supporting the proposed scheme for controlling human emotions, are encouraging. An agent program running on the desktop machine can automatically detect the emotion of the user and provide him/her the necessary control signals such as music/video/consolation messages.

24

1 Introduction to Emotional Intelligence

In chapter 6, we propose a dynamic model of multiple emotions using timedifferential equations. Stability analysis of emotional dynamics, which is an important issue in human-machine interactive systems, is also outlined in this chapter using Lyapunov energy function. The analysis of stability envisages that to ensure emotional stability, a condition involving parameters of the emotional dynamics is to be maintained permanently. A control algorithm used for adaptation of the necessary parameters of the emotional dynamics to ensure its stability has also been presented in this chapter. The concluding part of the chapter deals with a very important aspect of cognitive cybernetic system. It has been noted that humans on occasions suffer from the conflict between their emotional and logical states of the mind. The book attempts to provide a solution to this problem by comparing emotional states with the logical states and generating a negative feedback to adapt the parameters of the emotional dynamics, so as to maintain harmony between the logical and the emotional states. Chapter 7 examines the scope of chaotic emotional states of the human mind. Possible causes of chaotic emotion and general characteristics of the chaotic emotion are explored in this chapter. A mathematical model of chaotic emotional dynamics is constructed by taking into account the psycho-physiological processes involved in dys-regulation of human emotions. Stabilization of such chaotic dynamic is undertaken through adaptation of emotional parameters. The condition that bifurcates the chaotic emotion from its stable and limit cyclic counterparts is also ascertained. The concluding section compares the stabilization of chaotic dynamics with the stabilization in the arousals of emotions. Chapter 8 provides a novel approach to emotion clustering using electroencephalogram (EEG). EEG samples from visually stimulated persons are taken, and the selected features from the EEG samples are used to cluster emotions. The experiments performed for feature selection include determining digital filter coefficients, wavelet coefficients, peak and average power in the alpha (low and high), beta, gamma and theta frequency bands. Classical neural nets, such as backpropagation algorithm and support vector machines (SVM) have been used to cluster emotions from different feature vectors. Chapter 9 introduces the possible applications of emotional intelligence in different scientific and engineering problems. The response of a computer in human-computer interactive systems can be made more human-like by attaching emotional components. The models of emotional dynamics presented in this book can be employed to improve the interactions of the next generation human – machine interfaces. The arousals of fear, happiness and anxiety can be artificially stimulated in a mobile robot for better communication and co-operation among its teammates in a multi-agent system. Emotional intelligence models can be used for psychotherapy applications, especially for mentally retarded patients. The reflection of anti-social motives on emotional expressions can be detected to prevent nuisance. Generation of emotional expressions has much scope in commercial movie-making. The chapter explores the possible methods to synthesize emotional expressions to improve creation of the next generation digital movies. The chapter ends with a discussion on the application of emotional intelligence models in personality matching of people. The commercial software for matrimonial counseling

Exercises

25

will be greatly benefited with the proposed personality matching methodology. Chapter 10 outlines some open problems on emotional intelligence, and provides a pointer to references as recommended readings for researchers.

1.14 Summary This chapter presented a philosophical review of emotion. Various causes of arousal of emotion have been clearly explained in this chapter. The psychological approach to study emotional dynamics through fMRI and PET scan techniques has also been introduced. The need for mathematical modeling on emotional dynamics to bridge the gap between philosophical interpretation of emotion and its psychological counterpart has been narrated in this chapter. The scope of the book has also been outlined at the end of this chapter.

Exercises 1. Suppose a person P evaluates his emotion using three psychological state variables x, y and z. Let the positive support of P on x, y and z be F1, and the strong positive support of P on x, y and Z be F2, where F1(x, y, z) = x2+ y2+ z2 , and F2(x, y, z) = (x + y + z)2. Show that F2 is unconditionally greater than F1. [Hints: F2(x, y, z) = (x + y + z)2 = x2 + y2 + z2 + 2 (xy + yz + zx) ≥ (x2 + y2 + z2) for x, y and z to be real numbers. Except at x=y=z=0, we can always write(x+ y+ z)2 > x2+ y2+ z2 Thus, F2(x, y, z) > F1(x, y, z).] 2.

A person X attempts to measure the support of Y he may receive in executing a task jointly. Suppose the evaluation is based on a state variable x. Let the evaluation be given by f(x) = −x2+6x−9. Prove that X builds up a feeling of hatred over Y. [Hints: f(x) = −x2+6x−9 = −(x2−6x+9) = −(x2−2.x.3+32) = −(x−3) 2 < 0, for all real x Therefore, X builds up a feeling of hatred over Y.]

26

3.

1 Introduction to Emotional Intelligence

A, B and C are three coworkers, sharing common psychological resources x and y. Given that with increasing interaction between A, B and C, the evaluative feedback of A from B and C increase following f1 and f2 respectively, where

and

f1 (x, y, z) = x3y2+x2y3+xy2 f2(x, y, z) = xy+x2y.

Show that f1 and f2 monotonically increase with x and y, but the growth rate of f1 is more than that of f2. Given x, y, z >1. [Hints: It can be shown that ∂f1/∂x, ∂f1/∂y, ∂f2/∂x and ∂f2/∂y are all positive, proving that both the functions f1 and f2 are monotonically increasing. Further, since ∂f1/∂x > ∂f2/∂x and ∂f1/∂y > ∂f2/∂y, therefore the growth rate in f1 is greater than growth rate in f2.]. 4. The evaluative component of a person underestimates himself by a margin of the square-root of its original measure. Show that the minimum value of the actual measure of his positive feedback should be 1/4 units. [Hints: Let the actual measure be x. Then the person measures it by y, where y=x−√x. The minimum value of y is computed by setting dy/dx = 0 ⇒ dy/dx = 1 – 1/2√x = 0 which yields x=1/4]. 5. Let the evaluative component of a person gives his self-measure as the square of its actual value. The motivational component however degrades it by a margin of the square root of the evaluative feedback. Show that the person will be prompted to exhibit his emotional willingness as long as actual measure of emotion is greater than 1. [Hints: Let x be the actual measure of the self assessment, and y=x2 be the evaluative feedback. The motivational component transforms it to z, where z =x2 −x Now, z = x2 −x > 0 if x2>x, or x>1. Hence, the result follows.]

Exercises

27

6. Given the law of identity: q ∨ ¬q= true, for any Boolean variable q, show that the following propositional clause: (p∧q)→(p∨q) is a identity/tautology (i.e. always true). [Hints: Let z = (p∧q)→(p∨q) ≡ ¬ (p∧q) ∨ (p∨q) ≡ (¬p ∨¬q) ∨ (p∨q) ≡ (¬p ∨ p) ∨ (q ∨¬q) ≡ true ∨ true ≡ true. So, z is true for any truth values of p and q, Thus it is a identity.] 7.

Show that: ¬q ∧ (p→q) ≡ ¬p follows from the law of contradiction: p ∧ (¬p)=false, [Hints: ¬ q ∧ (p→q) ≡ ¬ q∧ (¬ p∨q) ≡ (¬q∧ ¬ p) ∨ (¬ q ∧ q) ≡ (¬q∧ ¬ p) ∨ false ≡ (¬q∧ ¬ p) ≡ ¬p, as ¬q is true (given).]

8.

Both emotion and (logical) reasoning follow the basic rules of formal logic, such as the laws of contradiction and identity. But the main difference occurs because of their referencing to different contexts. For instance, suppose A loves B because B is dependent on A, but this does not refer to the standard context of love, that may include A’s mental attachment to B. Construct two predicate logic based rules, showing that A loves B for the two causes cited above. Argue that when B is not dependent on A, logical reasoning still supports that A loves B. [Hints: The logical basis of A loving B can be defined by the clause: Loves(A, B) ← Has−mental−attachment−to(A, B) The emotional justification of A loving B is given by Loves(A, B) ← Is−dependent−on(B, A) When ¬Is−dependent−on(B, A) is true, still Loves(A, B) may be true by the first clause, as Has−mental−attachment−to(A,B) is true, and by modus ponens Loves(A,B) follows.] 9. Suppose δ(t) is a disturbance input, causing arrogance in a person, and the person uses a Sigmoid type non-linearity to control the external manifestation of his arrogance. Fig 1.9 provides a schematic diagram.

28

1 Introduction to Emotional Intelligence 2 S3

δ(t)

1 1+e−Net

Net

out

Sigmoid type Non linearity to block external manifestation

Arrogance generating system

Fig. 1.9. An arrogance generating system with external manifestation.

Determine the amplitude of the arousal of emotion at time t=2 second, and the magnitude of the external manifestation at the same time. [Hints: Net=2∫∫∫δ(t)dt.dt.dt=t2. At t=2 second, Net=22=4 units, out=1/(1+e−Net)≈1/18, which is insignificantly small compared to Net = 4 units.]

10. Sensitivity of a system is defined as

∂Out

Out

∂In

In

,

where In and Out

denote the input and the output of the system. Fig. 1.10 describes an emotional system excited by a ramp input: In= a.t and a parabolic output of the form at2/2 is obtained. Determine the sensitivity of the system. Emotional System

In

Out

Fig. 1.10. An emotional system.

[Hints: Out= at2/2, In = a.t ∂Out

In

∂In

Out

=

=

∂Out

∂In

∂t

∂t

(at/a)

at at2/2

t.t

= =

t2/2 2.]

In

. Out

Exercises

11.

29

The emotional dynamics of a person is given by Out = 5In2 − 0.25,

where Out and In denotes its input and output respectively, compute the sensitivity of the emotional system, when In=0.5

[Hints: Sensitivity =

δOut

In

δIn

Out In = 0.5

=

In

(10 In)

Out In = 0.5

=

10In2 Out

In = 0.5 In2

= 10

5 In2 − 0.25 In = 0.5

=

=

10 × (0.5)

2

5 (0.5)2 − 0.25

2.5 ].

12. The evaluation of emotion of a person has the following form g(x) = 3x3 − 5x. A person employs an approximate evaluation function f(x) to measure the same emotion. Given that f(x) = ax2 − bx + 2 Design a performance index ‘J’ that measures the error2 of the two evaluations. Determine the locus of a & b that satisfy the minimization of J with respect to x.

30

1 Introduction to Emotional Intelligence

[Hints: We define J = [(3x3 − 5x) − (ax2 − bx + 2)]2 = [3x3 − ax2 + (b − 5)x − 2]2

δJ = 0 yields, δx ⇒ 2{ 9x2 − 2ax + (b − 5) }= 0 ⇒ 9x2 − 2ax + b − 5 = 0 ⇒ (3x)2 − 2(3x) (a/3)+ a2/9 - a2/9 + b − 5 = 0 ⇒ (3x – a/3)2 = a2/9 –b+5 Since (3x – a/3)2 is always positive for equality constraints we find, a2/9 – b + 5 ≥ 0 or, a2 – 9b + 45 ≥ 0, which is the required locus.] 13. Fig 1.11 presents a schematic view of controlling emotion, where U(t) and M(t) denote the control input vector and the emotion vector respectively, A is the system matrix and B is the controller matrix. Show that n M (n) = An M (0) + ∑ A n-j B U (j-1). j=1

Unit Delay

A

M(t)

+ B

U(t)

+ M(t+1)

Control Vector Fig. 1.11. A feedback system.

References

31

[Hints: From Fig.1.11 we obtain M(t+1) = A M(t) + B U (t)

(1.3)

Substituting t=0 in (1.3), we have M(1) = A M (0) + B U (0). With t=1 in (1.3), we have, M (2) = A M (1) + B U (1) = A [AM (0) + B U (0)] + B U (1) = A2 M (0) + A B U (0) + B U (1) With t = 2 in (1.3), we have: M (3) = A M(2) + B U (2) = A [A2 M (0) + A B U (0) + B U (1)] + B U (2) = A3 M (0) + A2 B U (0) + A B U (1) + B U (2) M (4) = A M (3) + B U (3) = A4 M (0)+A3 B U (0) + A2 B U (1) + A B U (2) + BU(3). Proceeding in the same manner, we obtain: M(n) = An M (0) + An-1 B U (0) + An-2B U (1) +…….+ B U (n-1) n = An M (0) + ∑ An-j B U(j-1). j=1

References 1. Albus, J.A.: Meystel, Engineering of the Mind: An Introduction to the Science of Intelligent Systems. Wiley, New York (2001) 2. Alvarez, P., Squire, L.R.: Memory consolidation and the medial temporal lobe; a simple network model. Proc. Nat. Acad. Sci. 91, 4045–7041 (1994) 3. Bar-On, R.: Emotional and Social Intelligence: Insights from the Emotional Quotient Inventory. In: Bar-On, R., Parker, J.D.A. (eds.) The handbook of Emotional Intelligence, pp. 363–388. Jossey-Bass, San Francisco (2000)

32

1 Introduction to Emotional Intelligence

4. Bechara, A., Tranel, D., Damasio, A.R.: Poor Judgement in spite of High Intellect: neurological Evidence for Emotional Intelligence. In: Bar-On, R., Parker, J.D.A. (eds.) Handbook of Emotional Intelligence. Jossey-Bass, San Francisco (2000) 5. Ben-Zéev, A.: The Subtlety of Emotions. MIT Press, Cambridge (2000) 6. Biehl, M., Matsumoto, D., Ekman, P., Hearn, V., Heider, K., Kudoh, T., Ton, K.: Matsumoto and Ekman’s Japanese and Caucasian Facial Expressions of Emotion (JACFEE): Reliability data and cross-national difference. Journal of Nonverbal Behavior 21, 3–21 (1997) 7. Buchel, C., Dolan, R.J., Armony, J.L., Friston, K.J.: Amygdalae-Hippocampal involvement in human aversive trace conditioning revealed through event-related functional Magnetic Resonance Imaging. The Journal of Neuroscience 19 (December 15, 1999) 8. Carver, C.S., Scheier, M.F.: Attention and Self-Regulation: A Control-Theory Approach to Human Behavior. Springer, Berlin (1981) 9. Carver, C.S., Scheier, M.F.: Autonomy and Self-Regulation. Psychological Inquiry 11, 284–291 (2000) 10. Carver, C.S., Scheier, M.F.: On the Self-regulation of Behavior. Cambridge University Press, New York (1998) 11. Carver, C.S., Scheier, M.F.: On the Structure of Behavioral Self-Regulation. In: Boekaerts, M., Pintrich, D., Zeidher, M. (eds.) Handbook of Self-Regulation, pp. 41– 84. Academic Press, San Diego (2000) 12. Damasio, A.R.: The brain binds entities and events by multiregional activation from convergence zones. In: Gtfreund, H., Toulouse, G., et al. (eds.) Biology and Computation: A Physicist’s Choice, pp. 749–758. World Scientific, Singapore (1994) 13. Friston, K.J., Toroni, G., Reeke Jr., G.N., Sporns, O., Edelman, G.M.: Valuedependent selection in the brain: Simulation in a synthetic neural model. Neuroscience 59, 229–243 (1994) 14. Goleman, D.: Emotional Intelligence. Bantam Books, New York (1995) 15. Guyton, A.C., Hall, J.E.: Text Book of Medical Physiology. Harcourt Brace & Company Asia PTE LTD, Saunders (1998) 16. Kant, I.: Groundwork of the metaphysics of morals. Cambridge University Press, Cambridge (1998) 17. Konar, A.: Artificial Intelligence and Soft Computing: Behavioral and Cognitive Modeling of the Human Brain. CRC Press, Boca Raton (1999) 18. Lazarus, R.S.: Stress and Emotions. A New Synthesis. Springer, New York (1999) 19. Lazarus, R.: Emotion and Adaptation. Oxford University Press, New York (1991) 20. Le Doux, J.E.: Emotion: Clues from the Brain. Annual Review of Psychology 46, 209– 235 (1995) 21. Mattthews, G., Zeidner, M., Roberts, R.D.: Emotional Intelligence: Science and Myth, Bradford Book. MIT Press, Cambridge (2004) 22. Mayer, J.D., Salovey, P., Caruso, D.R.: Emotional Intelligence as Zeitgeist, as personality, and as a mental ability. In: Bar-On, R., Parker, J.D.A. (eds.) The handbook of emotional intelligence. Jossey-Bass, San Francisco (2000) 23. Mayer, J.D., Salovey, P., Caruso, D.R.: Competing models of emotional intelligence. In: Sternberg, R.J. (ed.) Handbook of human intelligence. Cambridge University Press, New York (2000) 24. McGaugh, J.L., Cahill, L., Roozendaal, B.: Involvement of the amygdala in memory storage: Interaction with other brain systems. Proc. Natl. Acad. Sci. 93, 13508–13514 (1996)

References

33

25. Panksepp, J.: Affective Neuroscience: The Foundation of Human and Animal Emotions. Oxford University Press, New York (1998) 26. Panksepp, J.: Towards a general psychological theory of emotions. Behavioral and Brain Sciences 5, 407–467 (1982) 27. Rolls, E.T.: The Brain and Emotion. Oxford University Press, New York (1999) 28. Russell, S., Norvig, P.: Artificial Intelligence A Modern Approach. Prentice-Hall, Englewood Cliffs (1995) 29. Salovey, P., Bedell, B.T., Detweiler, J.B., Mayer, J.D.: Coping Intelligently: Emotional Intelligence and the Coping process. In: Snyder, C.R. (ed.) Coping: The Psychology of what works, pp. 141–164. Oxford University, New York (1999) 30. Schultz, W., Dayan, P., Montague, P.R.: A neural substrate of prediction and reward. Science 275, 1593–1599 (1997) 31. Senior, C., Russell, T., Gazzaniga, M.S.: Methods in Mind. The MIT Press, Cambridge (2006) 32. Shallice, T., Burgers, P.: The domain of supervisory processes and the temporal organization of behavior. In: Roberts, A.C., Robbins, T.W. (eds.) The prefrontal cortex: Executive and Cognitive functions, pp. 22–35. Oxford University Press, New York (1996) 33. Simon, H.A.: Motivational and Emotional Controls of Cognition. Psychological Review 74, 29–39 (1967) 34. Solomon, R.C.: The Philosophy of Emotions. In: Lewis, M., Haviland, J.M. (eds.) Handbook of Emotions. Guilford Press, New York (1993) 35. Wells, A., Matthews, G.: Attention and Emotion: A Clinical Perspective. Lawrence Erlbaum Associates, Hove (1994) 36. Zanden, J.W.V., Crandell, T.L., Crandell, C.H.: Human Development. McGraw-Hill, New York (2006)

2 Mathematical Modeling and Analysis of Dynamical Systems

This chapter provides mathematical preliminaries on dynamical systems to enable the readers to visualize the dynamic behavior of physical systems, with an ultimate aim to construct models of emotional dynamics and analyze its stability. It begins with various representational models of dynamical systems, and presents general methods of stability analysis, including phase trajectories and the general method of Lyapunov. The later part of the chapter provides discussion on nonlinear fuzzy systems and its stability analysis. Examples from different domains of problems have been undertaken to develop expertise of the readers in modeling and analysis of emotional dynamics.

2.1 Introduction A physical system consists of several elements that together represent the system. Such systems are usually modeled with differential/difference equation. This chapter proposes modeling and analysis of physical system with special reference to their dynamic behaviors and stability. Psychological systems can also be modeled by differential/difference equation. For instance a temporal logical system, which can be regarded by a discrete system can be modeled by difference equation. On the other hand, an emotion building processes, which too is a psychological system, usually is modeled by differential equation. This chapter presents an analysis of stability for both logical and emotional system. Dynamic behavior of a system can take different terms: stable, unstable, limit cyclic and chaotic. The time response of a stable system attains steady state value after a finite time, well known as settling time for the system. A limit cyclic system describes sustained oscillation in its temporal responses. Phase-trajectories of such system describe a closed curve, signifying repeated traversal of the same trajectory by the system. A chaotic system too has sustained oscillation but there is no definite period of oscillation. Usually the amplitude of chaotic response varies within bounds. An unstable system on the other hand, exhibits ever increasing/decreasing behavior in its temporal responses. In this chapter, we provide Lyapunov analysis for detecting asymptotic stability and chaos in the behavior of dynamical systems. A. Chakraborty and A. Konar: Emotional Intelligence, SCI 234, pp. 35–61. © Springer-Verlag Berlin Heidelberg 2009 springerlink.com

36

2 Mathematical Modeling and Analysis of Dynamical Systems

Different types of dynamical systems, such as temporal logical system, fuzzy systems, neural systems and emotional system have been considered in this chapter to study the methodologies and principles of their stability analysis. The principle of Lyapunov exponent has also been briefly outlined to study the chaotic behavior of physical systems. The analysis undertaken in this chapter will be used throughout the book to study different aspects of emotional systems. The chapter is classified into seven main sections. Section 2.2 provides modeling aspect of simple static and dynamical systems. In section 2.3, we provide the principles of stability analysis by Lyapunov energy function. The principles have been employed to temporal logical system and neuronal dynamics. In section 2.4, stability analysis of fuzzy system has been undertaken. The analysis is extended to both Mamdani and Takagi-Sugeno type fuzzy systems. The Lyapunov exponent is introduced in section 2.5 to analyze chaotic dynamics, and stability behavior of a simple emotional dynamics is introduced in section 2.6. The chaotic behavior of emotional dynamics is undertaken in section 2.7.

2.2 System Modeling and Stability Physical systems are of two common types: static and dynamic. A static body/ system maintains the condition of equilibrium in a given reference frame, where the conditions prevent the body or each of its components form being dynamic. A house, building or a bridge are examples of static body, and as long as they do not collapse, they maintain the equilibrium condition. To study the equilibrium of two-dimensional static bodies, we usually consider three conditions: (i) sum of the projected components of forces (acting on the body) along the X-direction (reference direction) is zero, ii) sum of the projected component of forces along the Y-direction (perpendicular to the reference direction) is zero, and iii) sum of the moments of all the forces with respect to any reference point on the X-Y plane is zero. For higher dimensional systems, the conditions of equilibrium are slightly different. It is thus apparent that the condition for equilibrium of a physical system can be derived from the basic laws of statics introduced above. A logical system, which too is free from the notion of time, is a static system, but the laws that govern the equilibrium of the interpretation of the logical statements are different from the laws of mechanics (statics). The examples below provide an insight to the equilibrium of a mechanical and a logical system. Example 2.1: Consider the mechanical system shown in Fig. 2.1. We determine the condition of equilibrium of the block on the inclined plane. To solve this problem, we project the forces along the inclined plane (Xdirection) and its perpendicular direction (Y-direction). Then the rules of statics yield: F = W sin α

(2.1)

2.2 System Modeling and Stability

R

37

Y X

F

α

W Fig. 2.1. A block on an inclined plane is under equilibrium. W is the weight of the block, F is the frictional force and R is the normal reaction. α is the angle of inclination.

R = W cos α.

and

(2.2)

Further, by the rule of static friction, we have F = μ R,

(2.3)

μ being the co-efficient of friction. Thus, from expression (2.3), F μ= R W sin α =

[From (2.1) and (2.2)] W cos α

= tan α or,

α = tan-1 ( μ).

(2.4) (2.5)

Expression (2.5) describes the condition for static equilibrium of the block on the inclined plane. It is important to note: when the angel α increases, cos α decreases, but sin α increases. This indicates that the reactive force R (=W cos α) decreases, and so is the F (= μR = μ W cos α) but W sin α that acts to balance F increases, and after a threshold value of α= αmax, the block will lose equilibrium □ and start moving down the plane. Like static bodies, logical systems too have stability [2]. In this section, we would like to examine the stability of time-invariant logical systems. Logicians usually give emphasis to interpretation of logical statements. Unfortunately all the interpretations of a logical system are not stable. A question then naturally arises: what

38

2 Mathematical Modeling and Analysis of Dynamical Systems

do we mean by stability of a logical system? It may be maintained here that like the laws of equilibrium in statics, the logical constraints maintain the equilibrium of one or more interpretation of a logical system. These interpretations are called stable points. To determine the stable points from all possible interpretations of a logical system, we need to perturb the logical system by a small amount around each interpretation. In case, the perturbed system maintains logical consistency with the unperturbed one, we call the interpretation “stable points”. The following example gives an outline to this problem. Example 2.2: Let us now consider a logical system, described by propositions and logical connectives: p→q q → p.

and

where p and q are two propositions, having truth value: true/false, and → is a ifthen operator. For example, p → q means: ‘if p is true then q is true’ holds. The rule p → q has interpretations: (p, q) ∈{(1,1), (0,0), (0,1)}, and rule q→ p has interpretations: (p, q) ∈{(1,1), (0,0), (1,0)}. The common interpretation of p → q and q → p are (p, q) ∈{(1,1), (0,0)}. The given logical system is stated to be in equilibrium at (p, q) ∈{(0,0), (1,1)}. The points (p, q) = (0,0) and (1,1) may be called stable points of the static logical system. □ When the notion of time is attached to a physical system, we call it dynamic. We now construct models of mechanical and, logical dynamic systems. First, let us consider a simple spring-mass-load system in Example 2.3. Example 2.3: In a spring-mass-load system (Fig. 2.2), a force F is applied on to a mass m. Let the dynamic frictional co-efficient of the mass be f and, spring constant be k. The dynamics representing the above spring-mass-load system is formally given by d 2x F= m

dx +f

dt2

+ kx,

(2. 6)

dt

k

f m

x Fig. 2.2. A spring-mass load system, illustrating a dynamical system.

F

2.2 System Modeling and Stability

39

where x denotes the position of the mass m at time t. A simple free body diagram of the above system envisages that the physically applied force F, here, balances three other forces: dx i) The dynamic frictional force of magnitude: f , dt d2x ii) the inertial force M , dt2 and iii) the spring force kx. Any time-varying system usually has a dynamics, However, representation of the dynamics in a differential/difference equation is not always very easy. In the next section, we consider some formalism of temporal logic. In a temporal logic, time is the most pertinent information that plays a decisive role in the inference generating process of the logic. Some progress of representing temporal behaviors of logic by predicates is already prevalent in the exiting literature of the symbolic artificial intelligence [8], [1]. Unfortunately, determining stability of such logical system is model-specific and therefore no general treatment for stability analysis is amenable for such system. In this chapter, we provide a general framework for representing, dynamic logical systems using differential/difference equations. Example 2.4: In this example, we consider the following 2 logical clauses involving temporal variation of the proposition p and q. We presume that p and q to be fuzzy proposition and their membership value lie in the range [0, 1]. p (t+1) = p(t) ∧ q(t)

(2.7)

q(t+1) = q(t) ∨ ¬p(t)

(2.8)

Given the initial value of q(0) and p(0), we can evaluate the temporal variation of the logical transition. For instance given p (0) = 0.7 and q(0) = 0.2, we can evaluate [p(1), q(1)], [p(2), q(2)], [p(3), q(3)] and so on by using the given dynamics . Here, p(1) = p(0) ∧ q(0) = 0.7 ∧ 0.2 = 0.2 q (1) = q (0) ∨ ¬p(0) = 0.2 ∨ 0.3 = 0.3 p(2) = p(1) ∧ q(1) = 0.2 ∧ 0.3 = 0.2

40

2 Mathematical Modeling and Analysis of Dynamical Systems

and

q(2) = q(1) ∨ ¬p(1) = 0.3 ∨ 0.8 = 0.8 p(3) = p(2) ∧ q(2) = 0.2 ∧0.8 = 0.2 q(3) = q(2) ∨ ¬p(2) = 0.8 ∨ 0.8 = 0.8.

It is indeed important to note that p(3) = p(2) and q(3) = q(2) and thus the dy□ namics has a stable point at (p(2),q(2)) = (0.2, 0.8). How can we determine stability of a generalized dynamics for temporal logic? To discuss these issues, we shall use Lyapunov energy functions.

2.3 Stability Analysis of Dynamics by Lyapunov Energy Functions In this section, we briefly outlined Lyapunov Energy Function [10], [12] and its use in stability analysis of a dynamical system. A Lyapunov energy function for n variables: x1, x2, ….., xn is given by L (x1, x2, ….., xn) where: 1) L (x1, x2, ….., xn) = 0 at the origin, 2) L (x1, x2, ….., xn) > 0 anywhere except at the origin, 3) ∂L/∂xi exist for i = 1 to n. It is now clear from the above definition that a function that satisfy the above 3 criteria is called a Lyapunov energy function. To study the stability of a discrete dynamical system using Lyapunov energy function we need to evaluate: ΔL (X (k)) =

ΔL (x1(k), x2(k),……, xn(k))

=

L (x1(k+1), x2(k+1),……, xn(k+1)) - L (x1(k), x2(k),……, xn(k)).

A given dynamics is said to be stable in an asymptotic sense if ΔL (X (k)) < 0.

(2.9)

Example 2.5: Consider a discrete data system consisting of 2 states x1(t) and x2(t). x1(t+1) = - 0.5x1 (t) x2(t+1) = - 0.5x2 (t)

(2.10) (2.11)

2.3 Stability Analysis of Dynamics by Lyapunov Energy Functions

41

We here consider a Lyapunov energy function L(X) = x12(t) + x22(t). It is easy to note that the above function L (X) satisfies all the 3 conditions for Lyapunov energy function, and therefore its selection is justified. How can one test the stability of a given system by the proposed Lyaponov energy function? To determine the stability of the given dynamics we evaluate: ΔL (X (k)) = L (x1(k+1), x2(k+1)) - L (x1(k), x2(k)) = {x12 (t+1) + x22(t+1)} – {x12 (t) +x22(t)} = - 0.75 x12(t) – 0.75x22(t) < 0 for any real values of x1(k) and x2(k). Since ΔL < 0, for X (t) ≠ [0, 0], the system is stable in the Lyapunov sense. □ Example 2.6: Consider the dynamics given by x1(t+1) = -1.5x1(t) and x2(t+1) = - 0.5x2(t). Taking L(X) = a1x12 (t) + 2a2x1(t) x2(t) + a3x22(t)

(2.12)

We compute ΔL (X) = L (x1(t+1), x2(t+1)) - L (x1(t), x2(t)) = 1.25 a1 x12(t) – 0.5a2x1(t) x2(t) – 0.75 a3x22(t).

(2.13)

Since ΔL (X) is not negative for all values of x1 and x2, no conclusion about stability can be inferred. In the next section, we consider stability analysis of a logical system. □ A discrete temporal system is described by state equation: p (t+1) = F1 ( p(t), q(t)) q (t+1) = F2 ( p(t), q(t))

(2.14) (2.15)

where F1 and F2 are two non-linear logical functions of p(t) and q(t) respectively. To illustrate the stability analysis of the dynamics for logical systems, we consider the following example. Example 2.7: Consider the following dynamic system of two propositions p and q p (t+1) = q (t) (1-p(t)) q (t+1) = p (t) (1- q (t)) Let L(p(t), q(t)) be the Lyapunov function defined as L(p(t), q(t)) = p2(t) + q2(t). We evaluate: ΔL= L(p(t+1), q(t+1)) – L (p(t), q(t))

(2.16) (2.17)

42

2 Mathematical Modeling and Analysis of Dynamical Systems

= (p2(t +1) + q2(t + 1)) – (p2 (t) + q2(t)) = q2(t) (1- p(t))2 + p2(t) ( 1- q(t))2 – p2(t) – q2(t) = q2(t) [(1- p(t))2 – 1] + p2 (t) [(1- q(t))2 – 1]

(2.18)

< 0, ∀ p(t) , q(t). Therefore, the given dynamics is stable in the Lyapunov sense. 2.3.1 Stability Analysis for Continuous Dynamics So far we restricted our discussions on stability analysis for discrete dynamical systems. In this section, we would like to undertake stability analysis for continuous systems. Most of the real world systems are essentially continuous. For example, the birth and death process of a species, neurological systems, and even socio-economic situations are all continuous, and therefore are modeled by differential equations. This section provides stability analysis for continuous systems using Lyapunov approach. For convenience of the readers, we briefly outline the rule of stability analysis. Given a dynamics: dxi/dt = F(x1, x2, …., xn), i =1 to n.

(2.19)

We construct a Lyapunov energy function L(x1, x2, …, xn), and check whether dL/dt <0 for the given dynamics. If it is so, the dynamics is said to be stable in the Lyapunov sense. We now present an illustrative neural system called Hopfield dynamics to study the Lyapunov approach for stability analysis for continuous systems. The Hopfield dynamics is given by dui C

m = ∑ wij xj – βiui + θi, dt j=1

(2.20)

for i=1 to n. Here, C, βi and wij are system parameters. θi is the input, xj is the system state, and ui is the stored signal [9]. Such systems require an integral type Lyapunov energy function for its stability analysis. The typical Lyapunov energy function used in the present context is given by n n n xi n E (t) = - ½ ∑ ∑ wij xj xi + ∑ βi ∫0 f –1 (ξ i) dξi - ∑ θi xi i=1 j=1 i =1 i=1

(2.21)

where f(ui) = xi and thus ui = f –1 (xi) dE dt

n = -1/2 ∑ i=1

n dxi ∑ ( wij xj + wij xi j=1 dt

dxj

n dxi ) + ∑ βi ui dt i=1 dt

2.3 Stability Analysis of Dynamics by Lyapunov Energy Functions

n dxi - ∑ θi i=1 dt

43

(2.22)

.

With symmetric weights, i.e. wij = wji , we obtain: dE dt

n dxi = - ∑ i=1 dt

n [( ∑ wij xj ) - βi ui + θi] j=1

n dxi = - C ∑ i=1 dt

dui

n = -C∑ i=1

d

n =-C∑ i=1

dxi

dt

( f -1 ( xi ) ) dt

dt

∂f –1 (xi)

dxi

2 (2.23)

∂ xi

dt

< 0, ∂f- -1 if ∂ xi

>0 , i.e., f -1(xi) is a monotonically increasing function of xi.

The following examples give a detailed idea about the selection of the integral type Lyapunov energy function to determine the stability of the dynamics. Example 2.8: In this example, we consider the well known Cohen-Grossberg theorem [9] stated as follows A non-linear dynamical system is given by dxi

N = ai(xi)[bi(xi) - ∑wik dk(xk)], dt k=1

(2.24)

for i = 1, 2,…, N. The global stability of the system can be determined by the following Lyapunov energy function:

44

2 Mathematical Modeling and Analysis of Dynamical Systems

N xi

ddi(ξi)

V(x)= - ∑ ∫0 i=1

bi(ξi)

N N

dξi

+ (1/2)

∑ ∑wi k dI(xi) dk(xi).

(2.25)

i=1 k=1

Proof: We compute dV

dV(x(t)) =

dt

dt N =∑ i=1

∂V

dxi

∂xi

dt

N

ddi(xi)

= - ∑ bi(xi) i=1

dxi

N N +

N =-∑ i=1

∑ ∑wi k i=1 k=1

dt

ddi(xi) dxi

ddi(xi)

dt

dxi

= - ∑ ai(xi) i=1

ddi(xi) dxi

ddi(xi) dxi

(2.26) dt

N bi(xi) - ∑ wik dk(xk) k=1

N bi(xi) - ∑ wik dk(xk) i=1

< 0, if ai(xi) ≥ 0 and

dxi dk(xk)

-

dxi

N

dxi

≥0.

2 (2.27)

2.4 Stability Analysis of Fuzzy Systems

45

Thus, the condition for stability of the given system is i) ii)

ai(xi) is non-negative, di(xi) is monotonically non-decreasing with respect to xi

2.4 Stability Analysis of Fuzzy Systems Humans sometimes reason with approximate data/facts. Classical logic that deals with binary truth functionality has undergone several extensions to handle inexactness of real world problems. Fuzzy logic is one such extension of classical logic that offers membership values in the range [0, 1] for each element x in a given set A. This is usually denoted by μA(x). For discrete fuzzy systems, we take sample values x1, x2, ……, xn, where for each value of x = xi in set A = {x1, x2, ……, xn}, we have a μA(x). Fuzzy systems are usually of two types Mamdani type [3] and Takagi-Sugeno type [13]. We now present these two systems and their stability analysis. 2.4.1 Mamdani Type Fuzzy Systems A Mamdani type fuzzy system, employs fuzzy rules of the following format: If x1 is A1 then y is B1. Given a fuzzy proposition: x ix A/, where A/ is approximately equal to A1, we can infer y is B/ by using a fuzzy implication relation between x is A 1 and y is B1, and the membership distribution of x is A/. Let R (x, y) be a relational matrix for the implication rule: if x is Ai then y is Bi. Usually R(x, y) is denoted by a matrix, as indicated below. Suppose R(x, y) is given and μA/(x) is given, we now present a scheme for evaluation of μB/(y). y

y1

y2 ……. ym

x x1 R(x, y)= x2 . . xn

(2.28)

Given x1, x2 ….., xn as distinct values of x and y1, y2,….., ym as distinct values of y respectively. Let the membership distribution of x is A/ be symbolically represented by x1 x2…….. xn μA/(x) = [a1 a2 …….. an], where a1 = μA/(x1) , a2 = μA/(x2), …. , and an = μA/(xn)

46

2 Mathematical Modeling and Analysis of Dynamical Systems

The membership distribution of y is B/ is represented by y1 μB/(y) = [b1

y2…….. ym b2 …….. bm],

(2.29)

where b1 = μB/(y1) , b2 = μB/(y2), ….. , and bn = μB/(ym). Now, according to fuzzy implication methods, we evaluate μB/(y) = μA/(x) ° R (x, y),

(2.30)

where “ ° ” is a max-min composition operator. When there exist a number of fuzzy rules of the following form: If x1 is A1 then x2 is A2, If x2 is A2 then x3 is A3, If xn-1 is An-1 then xn is An, then we can evaluate the transitive relationship between x1 is A1 and xn is An by taking composition of the following relations. R (x1, xn) = R1 (x1, x2) ° R2 (x2, x3) ° ……° Rn-1 (xn-1, xn)

(2.31)

Here, Ri (xi, xi+1) denotes the implication relation for the rule: If xi is Ai Then xi+1 is Ai+1. This is valid for i=1 to (n-1). If the relation R1 (x1, x2) = R2 (x2, x3) =….= Rn-1(xn-1, xn) = R, then we can write the above result as follows: R(x1, xn)=Rn .

(2.32)

The above transitive relationship may demonstrate limit cyclic behavior if Rn+k.=Rn , where k denotes the period of limit cycles. This can happen, if Rk =I, the identity matrix. 2.4.2 Takagi-Sugeno Type Fuzzy Systems Takagi-Sugeno (T-S) type fuzzy system includes the following production rules: Rule1 If x1 is A1 and x2 is A2 then x1(k+1)=A1x(k), Rule2 If x1 is A3 and x2 is A4 then x2(k+1)=A2x(k), where x(k)=[x1(k) x2(k)]T.

2.4 Stability Analysis of Fuzzy Systems

47

Given μA1(x1)=0.3, μA2(x2)=0.4, μA3(x1)=0.7 and μA4(x2)=0.8. Suppose, measured values of x1 and x2 at k=1 are x1(1)=40 units, x2(1)=20 units. 0.5

0.4

A1 =

and 2.0

0.6

0.8

1.0

2.0

A2 =

1.0

We want to compute x(k+1) at k=1. Now, 2

x(k+1) = Σ hi (k) Ai x(k) i=1

w1(1) = μA1(x1(1)) . μA2(x2(1)) = 0.3 × 0.4 = 0.12.

where

w2(1) = μA3(x1(1)) . μA4(x2(1)) = 0.7 × 0.8 = 0.56. w1(1)

h1(1) =

h2(1) =

w1(1) + w2(1)

w2(1) w1(1) + w2(1)

0.12 =

0.56 =

0.12 + 0.56

=

12 68

.

=

0.12 + 0.56 56

.

68

we can evaluate 2 x(2) = Σ hi (1) Ai x(1) i=1 = h1(1) A1 x(1) + h2(1) A2 x(1)

=

12 68

= 37.88

0.5

0.4

40

2.0

1.0

20

56 +

83.52

T .

68

0.6

0.8

40

1.0

2.0

20

2 Mathematical Modeling and Analysis of Dynamical Systems

48

For stability analysis of a dynamical system, we should select the Lyapunov energy function V in a manner, so that the dynamics can be mapped on the energy surface, and asymptotic stability of the system in the sense of Lyapunov can be ascertained by noting a negative change in ΔV for a state transition of the system from XT(k) to xT(k +1). Suppose, given a discrete system x(k+1) = A x(k),

(2.33)

and we want to analyze the stability of the system. Let V(x1, x2,…., xn) = xT(k) P x(k),

(2.34)

where P is a positive definite matrix. To illustrate how the above V satisfies characteristics of the Lyapunov energy function, suppose we take P = I, the identity matrix. Then V= x2(k) =x12(k) + x22(k)+…..+xn2 (k),

(2.35)

which satisfies all the three characteristics of a Lyapunov function. Now, ΔV= Δ[ xT(k) P x(k)] = xT(k+1) P x(k+1) - xT(k) P x(k) =xT(k) AT P A x(k) - xT(k) P x(k)

[by (2.39)]

= xT(k)[ATPA- P] x(k) = - xT(k) Q x(k), say

(2.36)

ATPA- P= - Q.

(2.37)

where

The system x(k+1) = Ax(k) is said to be asymptotically stable if Q is positive definite. 2.4.3 Stability Analysis of T-S Fuzzy Systems Tanaka and Sugeno [14] suggested an important criterion for the analysis of stability of the T-S fuzzy system. Property 1: The equilibrium state x=0 of the fuzzy system (2.33) is globally asymptotically stable, if there exists a common positive definite matrix P such that Ai P Ai - P <0, for all i=1,2,….,N.

2.4 Stability Analysis of Fuzzy Systems

49

First, let us consider N=2 for simplicity. Then for stability analysis of the system (2.21), we break it into 2 sub-systems. x(k+1) = x1(k+1) + x2(k+1) where

x1(k+1) = h1(k) A1 x(k)

(2.38) (2.39)

= A1 x1(k),say x2(k+1) = h2(k) A2 x(k)

and

(2.40)

= A2 x2(k), say In general, xi (k+1) = Ai xi (k)

for i= 1,2,…,N.

(2.41)

Narendra and Balakrishnan [11] suggested a systematic way of finding the common P matrix for N-simultaneous continuous-time linear system satisfying pairwise commutative characteristics. The results are extended below for discrete–time systems following [6]. Property 2: Suppose matrices A1 and A2 are Schur and commutative, i.e., A 1 A 2 = A2 A 1

(2.42)

Now, for each discrete system given by (2.39) and (2.40) we employ the following Lyapunov criteria for stability analysis. A1T P1 A1 – P1 = -Q A2TP2A2 – P2 = -P1

(2.43) (2.44)

where Q>0 and P1 and P2 are the unique positive definite solutions of the above equations respectively. Then we always have AiT P2 Ai - P2 < 0 for i=1,2,….. Proof: Details of the proof is available in [6]. We here give an outline, so that interested readers can prove it themselves with the given outline. ♦ Substitute the value of P1 from (2.44) in (2.43). ♦ Replace (A2A1)T = A1T A2T and (A1A2)T = A2T A1T in the resulting expression, so that the result can finally be expressed as -Q = A2T ψ1 A2 - ψ1

(2.45)

2 Mathematical Modeling and Analysis of Dynamical Systems

50

ψ1 = - A1T P2 A1 + P2

where

(2.46)

Since Q>0 and A2 is Schur, the solution ψ1>0 is unique. ♦Assuming the Lyapunov function V (x (k)) = xT(k) P2 x(k)

(2.47)

for both systems, we now compute ΔV for A2 system as follows: ΔV = xT(k) (A2T P2A2 – P2) x(k) = -xT(k) P1x(k) < 0 for all x(k)

(2.48)

since P1>0 in (2.44). ♦Further for the A1 system, ΔV = xT(k) (A1 T P2 A1 – P2) x(k) = -xT(k) ψ1(k) x(k) < 0 for x(k)≠0

(2.49)

since ψ1 > 0. ♦From expressions (2.29) and (2.30) we find that AiT P2 Ai – P2 < 0 for i=1, 2 and P2 thus can be used as common P matrix of the T-S fuzzy model. For N-plant rules, we can easily extend the above concept as summarized in property 4.3. Property 3: Suppose Ai for i=1,2,…..,N is Schur, i.e., Aj Aj+1 = Aj+1 Aj for j=1,2,……,N-1 Considering N Lyapunov equations: A 1T P 1 A 1 – P 1 = - Q A2 P2 A2 – P2 = -P1 . . . ANT PN AN – PN = -PN-1

(2.50)

where Q >0 and Pi for i=1,2,….,N is the unique positive definite symmetric solution for each equation. Then we always have AiT PN Ai – PN < 0

for i=1,2,…..,N

Proof of the above property is similar to previous proof.

(2.51)

2.4 Stability Analysis of Fuzzy Systems

51

Example 2.9: For 3 plant rules and 2 state variables, let the corresponding Ai’s be A1 =

1.0 -0.3

0.5

0.2 0.5

0.12

and

A2 = - 0.18 0.2 0.8 A3=

0.16

-0.24

It can easily be verified that A1A2 = A2A1 =

0.464

0.6

-0.24

0.064

0.3712

0.128

and A2A3 = A3A2 =

-0.192

0.051

With Q=I, we can show that A1TP1A1 – P1 = -Q yields P1=

7.30

2.00

2.00

2.25

9.21

2.67

T

A2 P2A2 – P2 = -P1 yields P2=

2.67

2.62

19.374

6.53

6.543

4.711

T

A3 P3A3 – P3 = -P2 yields P3=

,

,

.

52

2 Mathematical Modeling and Analysis of Dynamical Systems

Thus P3 which is symmetric and positive definite is the common matrix that should satisfy AiT P3 Ai – P3 < 0 for i=1, 2, 3. In fact, a computation of AiT P3 Ai – P3 for i=1 to 3 can be shown to satisfy the inequality (2.57). Thus a common P matrix which guarantees the stability of T-S fuzzy model with N plant rules can be found systematically by using property 2.4.3.

2.5 Chaotic Neuro Dynamics and Lyapunov Exponents Mesoscopic level neuronal dynamics deals with collective dynamical behavior of neural population. The brain processes can be interpreted with the mesocopic neural models. It has been noted that the human (mammalian brain) demonstrates non-periodic oscillations than fixed-point and limit cyclic behaviors. Freeman proposed a K-set meososcopic level population model, and Harter et al. [5] employed this model for studying chaotic behavior of autonomous agents in real time. In particular, Harter et al. studied a discrete recurrent version of the Freeman’s mesoscopic model. The basic differential equation of neural population proposed by Freeman is given by αβ

d2ai (t) d t2

d ai (t) + (α + β)

+ ai (t) = neti (t)

dt

(2.52)

where ai (t) is the activity level, i.e., average amplitude of the ith neural population; α and β are time constants. The output of the jth neuron is obtained by a non-linear function given by

Oj (t) =

and

∈ 1 – exp

-(eaj(t)- 1) ∈

neti(t-1) = ∑ wij Oj (t) j

A discrete version of the model is given by ai (t) = ai (t+1) + deci(t-1) + moni (t-1) + neti (t-1) deci(t-1) = -ai(t) × α moni (t-1) = ri(t) ×β neti (t-1) = ∑ wij Oj (t)

(2. 59)

(2.60)

2.5 Chaotic Neuro Dynamics and Lyapunov Exponents

53

1

↑ t+12

- 0.5

0 t→

0.5

1.0

Fig. 2.3a. The chaotic behavior of the continuous Freeman dynamics.

Power

100

10-5 10-6 10 100 Frequency in Hz Fig. 2.3b. EEG response of a rat olfactory bulb (OB).

Oj (t) = ∈ 1 – exp

-(ea j(t) - 1) ∈

It has been noted that the discrete model when plotted at time t and t + 12 along the x and y-axis provides a chaotic behavior. Such chaotic behavior is noted in the

2 Mathematical Modeling and Analysis of Dynamical Systems

54

Calculated largest Lyapunov exponent

0.15

.05

0

0

0.3

0.6

0.9

1.2

1.3

Weight scaling Fig. 2.4. The Lyapunov exponent for the discrete model, indicating the effect of scaling of excitatory weights. The average plot for ten experiments is included, describing variation in exponent by bars.

EEG spectrum of rats. The chaotic behavior of the EEG spectrum of rat is given in Fig. 2.3a and Fig. 2.3b respectively. Lyapunov exponent is normally calculated for difficult weight setting of the discrete dynamics. When Lyapunov exponent is found to be positive, the dynamics is said to have strong chaotic behavior. Harter et al. considered different parameter sets of the discrete dynamics, and plotted the Lyapunov exponents against various weight scaling in the range 0 to 1. Their graphical demonstration envisages that for most weight scaling, the Lyapunov exponent is positive indicating the positive behavior of the dynamics (Fig. 2.4).

2.6 Emotional Dynamics and Stability Analysis Researchers on brain sciences proposed different models on emotional dynamics of humans. These models take into account various emotional states and their interactions through temporal differential equations. Most of these models represent emotional dynamics by a special type of recurrent neural network. Such networks include both positive and negative weights, representing activation and inhibition of neural activities. An illustrative neural dynamics consisting of two neural states is given below dx1 dt

=

α x1 x2 − β x2 + I1

(2.55)

2.7 The Lyapunov Exponents and the Chaotic Emotional Dynamics

55

αx2

I1

dx1

x1

dt −β −δ x2 I2

dx2 dt

γx1−δ Fig. 2.5. Neural dynamics of emotional states x1 and x2.

dx2 dt

=

γ x1 x2 − δ x2 + I2

(2.56)

Fig 2.5 provides a schematic representation of the given neuronal dynamics. Here, x1 and x2 denote two emotional states in a mixed emotional system. The weights - β and -δ provide negative feedback to the neuronal states, whereas the self-loops through positive weights denote positive feedback/growth of the neurodynamical system. I1 and I2 in Fig. 2.5 denote instantaneous inputs. Determining asymptotic stability of the given system can be performed by Lyapunov approach. Depending on the range of the parameters, an emotional system may give rise to chaotic behavior. We now briefly outline one method of analyzing chaos, and demonstrate the same for a given emotional dynamics.

2.7 The Lyapunov Exponents and the Chaotic Emotional Dynamics Lyapunov exponents are commonly used to study the existence of chaos in a nonlinear dynamical system. It is formally defined as the average rate of divergence (or convergence) of two neighboring trajectories of the non-linear dynamical system. To obtain the Lyapunov spectra, let us consider a infinitesimally small ball with radius dr, sitting on the initial state of a trajectory. The flow will deform this ball

2 Mathematical Modeling and Analysis of Dynamical Systems

56

into an ellipsoid, i.e., after a finite time t, all orbits which have started in the ball will be in the ellipsoid. The ith Lyapunov exponent is defined by 1 Xi = Lt t→∝

t

dli(t) ln

(2.57)

dr

where dli(t) is the radius of the ellipsoid along its principal axis. It is well-known as three possible types of dynamic behavior are apparent from the following three conditions of the Lyapunov exponent lambda. a)

b)

c)

When λ<0, the orbit attracts to a stable fixed point or stable periodic orbit. Negative Lyapunov exponents represent dissipative or nonconservative systems. Such systems exhibit asymptotic stability. The more is the negative exponents, the greater is the stability. When λ=0, the orbit is a neural fixed point (or eventually fixed point). The zero value of the Lyapunov exponents indicates that the system is in some form of steady-state. A physical system with a zero value of λ indicates a conservative system. Such system also exhibits Lyapunov stability. For instance consider two identical simple harmonic oscillators at different amplitudes. Here, as the frequency is independent of the amplitude, a phase portrait of the two oscillators would be a pair of concentric circles. When λ> 0, the orbit is unstable and chaotic. Nearly points, known close they are, will diverge to any arbitrary separation. All neighborhoods in the phase space will eventually be visited.

Some typical orbits for different values of the Lyapunov exponent are presented in Fig. 2.6. The limit form of the Lyapunov exponent is given by

(a) (b)

(c) Fig. 2.6. Behavior of non-linear dynamics: a) a stable focus, b) a limit cycles, and c) chaos.

2.7 The Lyapunov Exponents and the Chaotic Emotional Dynamics

λ=

1

57

N

Σ log 2 dxn+1 N dxn N→∝ n =1 Lt

(2.58)

Example 2.10: Given the logistic equation, xn+1= r xn(1-xn), the Lyapunov exponent of which is given by λ≈

Lt

1

N

Σ log 2 (r-2rx0)

(2.59)

N n =1 when r=2, x0=1/2, χ=0, i.e. the dynamics is stable. On the other hand, when r=3, the system extend a chaotic zone. N→∝

Example 2.11: Consider the discrete model of a simple emotional dynamics with parameters a, bjk and cik. x xk+1 = axk (1− dxk+1 dxk

= a (1 −

k

2xk k

dxk+1

Now,

) + bjk (1− exp (− βjxj)) − Cik (1− exp (− λixi)) >1

dxk

a (1−

) + bjkxk (1− exp (− βjxj)) − Cikxk (1− exp (−λ ixi))

2xk k

(2.60)

(2.61)

implies:

) + bjk (1− exp (− βjxj)) − Cik (1− exp (− χixi)) > 1

(2.62)

Expanding exp [- (βi xi)] and exp [-(λi xi)] for the first two terms, we obtain: a (1 − or, a (1 −

2xk k 2xk k

) + bjk (1− 1+ βjxj) − Cik (1− 1 + λixi) >1

(2.63)

) + bjk . βjxj − Cik . λi xi >1.

(2.64)

The existence of chaos in the given emotional system, now, can easily be ascertained, when the values of xk, xj, a, k, bjk and cik are supplied, and the parameters a, k, bjk and ci k are given. So, we can test the stability, instability and chaotic behavior of the dynamics by the Lyapunov exponent approach.

58

2 Mathematical Modeling and Analysis of Dynamical Systems

Exercises 1. The Lorenz attractor dynamics is given by dx dt

= a (y – x)

dy dt

= (r-z)x-y

dz dt

= xy –bz.

Show that the phase-portrait of the Lorenz attractor for a =10, r = 28, and b = 8/3 with an initial point of (x, y, z) = (5, 5, 5) describes two foci in the plots x- y, x-z and y-z. [Hints: Use 4th order Runge-Kutta numerical method to solve the dynamics over time t, and plot x-y, x-z, y-z in 3 graph papers for 3000 iterations. Use graphics to denote the current point and the next point in the plot by suitable notations]. 2. The Henon map is given by the following dynamics xi+1 = 1- α xi2 + yi yi+1 = β xi a) Given α = 1.4 and β = 0.3, plot xi versus i and yi versus i for i = 0, 1, …, 30 with initial values x0 = y0= 0. b) Plot xi versus yi for the same and different initial conditions, and cheek whether the dynamics yields limit cyclic and/or chaotic behavior. 3. A discrete dynamical system is given by X (k+1) = A X (k) + B u (k) where

u(k) = -G X (k).

Given that the above dynamics is stable, if for any positive definite real symmetric matrix P we get a positive definite matrix Q such that AT PA –P = - Q

Given A =

0.5 0

0 0.2

and

B=

1 1

2.7 The Lyapunov Exponents and the Chaotic Emotional Dynamics

Determine the optimal control law U(k) in order to stabilize the given system. [Hints: Let V (X (k)) = XT (k) P X (k). Then ∆V (X (k)) = XT (k +1) P X (k + 1) – XT (k) P X (k) = XT (k) ( A-BG)T P (A –BG) X (k) – XT (k) P X (k) =XT(k)ATPAX(k) - XT(k)GTBTPAX(k) - XT(k)ATPBGX(k) + XT(k)GTBTPBGX(k) - XT(k)PX(k) = XT(k)ATPAX(k) + uT(k)BTPAX(k) + XT(k)ATPBu(k) + uT(k) BTPBu(k) - XT(k)PX(k)

[Since u(k) = -GX (k)]

For minimum ∆V (X (k)), we set ∂ ∆V(X (k))

=0

∂ U(k) which yields 2 BT PAX(k) + 2 BTPBu(K) = 0 ⇒

u(k) = - ( BT PB)-1 BT PAX(k) = - G X (k)

So, G = ( BT P B )-1 B T PA. With – Q = AT PA – P, we evaluate

1.333

0

P=

. 0

1.042

Next we evaluate G = ( B T PB)-1 BTPA = [0.28 So, u(k) = -0.28 x1 (k) – 0.087 x2 (k)].

0.0876]

59

60

2 Mathematical Modeling and Analysis of Dynamical Systems

4. Select a Lyapunov energy function to study the asymptotic stability of the following dynamics: dxi dt

= α xixj - βxi + Ii

for i = 1 to n, and hence determine the condition for stability of the dynamics, if any. 5. Given a = 1.2, bjk = 0.001, cik = 0.001, K= 1000, x0 = 200 and xk = 50, βj= 0.2, λi =0.3. Check whether the emotional dynamics given by xk+1 = a xk (1 – xk /K) + bjk xk (1-exp(-βj xj)) – cik xk ( 1- exp(- λixi)) is chaotic. [Hints: Determine Lyapunov exponent for the dynamics, and show that the exponent is positive. Thus claim that the dynamics is chaotic.]

References 1. Bolc, L., Szatas, A. (eds.): Time and Logic-A Computational Approach. UCL Press, London (1995) 2. Chakraborty, A., Sanyal, S., Konar, A.: Fuzzy temporal extension of classical logic and its stability analysis. IE(I) Journal-ET 86, 50–53 (2006) 3. Drainkov, D., Hellendoorn, H., Reinfrank, M.: An Introduction to Fuzzy Control, pp. 132–140. Springer, Berlin (1993) 4. Dubois, D., Prade, H.: Fuzzy Sets and Systems: Theory and Applications. Academic Press, New York (1980) 5. Harter, D., Kozma, R.: Chaotic neurodynamics for autonomous agents. IEEE Trans. on Neural Networks 16(3) (May 2005) 6. Joh, J., Chen, Y.-H., Langari, R.: On the stability issues of linear takagi-Sugeno fuzzy models. IEEE Trans. on Fuzzy Systems 6(3) (August 1998) 7. Klir, G.J., Yuan, B.: Fuzzy Sets and Fuzzy Logic: Theory and Applications. PrenticeHall, New Jersey (1995) 8. Konar, A.: Artificial Intelligence and Soft Computing: Behavioral and Cognitive Modeling of the Human Brain. CRC Press, Boca Raton (1999) 9. Konar, A.: Computational Intelligence: Principles, Techniques and Applications. Springer, Heidelberg (2005) 10. Kuo, B.C.: Digital Control Systems. Hault-Sounders, Tokyo (1983) 11. Narendra, K.S., Balakrishnan, J.: A common Lyapunov function for stable LIT system with commuting A-matrices. IEEE Trans. on Automatic Control 39, 2569–2571 (1994) 12. Ogata, K.: Modern Control Engineering. Prentice-Hall, New Jersey (1990)

References

61

13. Takagi, T., Sugeno, M.: Fuzzy identification of systems and its application to modeling and control. IEEE Trans. on Systems, Man and Cybernetics SMC-15, 116–132 (1985) 14. Tanaka, K.: Stability and stabilizability of fuzzy-neural linear control systems. IEEE Trans. on Fuzzy Systems 3, 438–447 (1995) 15. Wang, L.-X.: A Course in Fuzzy Systems and Control. Prentice-Hall, New Jersey (1997) 16. Zimmerman, H.J.: Fuzzy Set Theory and Its Applications. Kluwer Academic Publishers, Dordrecht (1991)

3 Preliminaries on Image Processing

This chapter provides an introduction to digital image processing. With a brief review on file formatting of images, it addresses the well-known frequency domain transforms such as discrete Fourier and discrete cosine transforms, and applies these transforms in pre-processing/filtering a given image. The principles of image segmentation are then outlined. Various algorithms for labeling of segmented images are then presented for subsequent application in object recognition. Some algorithms on high-level vision, including object recognition and image interpretation are also introduced in this chapter.

3.1 Introduction Analysis of human emotions from facial expression is an important issue of discussion in this book. Various image processing tools and techniques are often used for reading human faces. This chapter serves as a pre-requisite to human emotion detection and control to be introduced in the latter part of the book. A digital image is a 2-dimensinal array of pixels that contains intensity information of pixels in the image. To be specific, let i and j be 2 array indices, that describe the i-th row and j-th column in an image. Under this case f (i, j) denotes the intensity of pixel (i, j) in the image. A digital image can be colored or monochrome. A monochrome image is represented by gray intensity values between 0 and a positive integer, usually 255. When f (i, j) at pixel (i, j) is close to zero, that pixel looks dark. On the other hand, when f (i, j) is close to 255, it looks bright. Thus, a variation of intensity between 0 to 255 can produce different shades of grayness in the image. A colored image, on the other hand, includes color information of the pixel. The color information is represented by red (R), green (G), and blue (B) intensity values of the pixels in the image. Both monochromes and colored images are represented by different file formats. Typical file formats are .bmp, .tiff, .pgm, .jpeg and .mpeg. Each type of file has a header and a body, containing image data. The header includes many characteristic information about the data, such as number of gray levels in the image, the width and height of the image and the like. In this A. Chakraborty and A. Konar: Emotional Intelligence, SCI 234, pp. 63–92. © Springer-Verlag Berlin Heidelberg 2009 springerlink.com

64

3 Preliminaries on Image Processing

chapter, we introduce a simple type of image file called .pgm files. A .pgm file contains a small header containing 4 information as follows: “P6” “Image-width” “image Height” ……..“ No of Gray Levels”. The body of the image contains red, green and blue pixel intensity one after another for every pixel in the image in ASCII codes. An image processing system usually consists of image sensors/transducers, such as digital video cameras, ultrasonic sensors and infrared range detectors. Sensory data obtained from the transducers usually contain noise because of variations in environmental conditions and/or poor illuminating conditions. To extract necessary information from an image, we need to preprocess the image for i) elimination of the effect of such noise and ii) enhancement of details. The next important step in image processing is segmentation, which partitions an image into objects of interest. After segmentation of an image into important components, we need to label each of them. This calls for labeling algorithms that label individual components with reference to their size, shape and similarity of objects. The most important task of image processing is recognition of the image or its components. The interpretation of a scene consisting of several objects already recognized can be performed using intelligent reasoning tools and techniques. This chapter provides a review on the well-known works in image processing as a prerequisite knowledge for the subsequent chapters. Frequency domain discrete transforms such as Fourier transforms and cosine transforms are introduced in section 3.2. Noise filtering algorithms are presented in section 3.3. Segmentation algorithms are covered in section 3.4. An overview of labeling through boundary detection algorithms is given in section 3.5. Various techniques of object recognition by image analysis are discussed in section 3.6. High-level interpretation of a scene by intelligent tools and techniques are given in section 3.7. In section 3.8, we present concluding remarks and possible application of the proposed methodology in human emotion detection.

3.2 Discrete Fourier and Cosine Transforms Analysis of signal is usually performed in both time and frequency domain. The time- domain information describes the variation of the signal over time, whereas in frequency domain, we study the variation of the signal over frequency. In general, a complex signal consists of a number of sinusoidal components. Naturally, there exists a variation of the signal over a certain frequency band. Unlike time varying signals, an image may be considered as an aggregation of pixel intensities over a 2-dimensional space. Naturally, we need spatial domain representation for analysis of an image. Further, an image can also be represented in 2-dimensional frequency domain, where frequencies u and v denote the count of pixels having equal intensities along the x and the y directions respectively of the image.

3.2 Discrete Fourier and Cosine Transforms

65

Frequency domain transforms are widely used for transformation of time domain signal to frequency domain. Similar transforms can equally be used for transformation of spatial domain images to frequency domain. Among the wellknown frequency domain transforms, Fourier transform [4] is most common. The discrete Fourier transform for one-dimensional spatial signal is given by 1

F (u) =

N-1 ∑ f(x) e-j2π ux/N N x=0

(3.1)

where x = 0, 1, 2, …., N-1 and u is an integer frequency variable given by u = 0, 1, 2, …, N-1. The 2-dimensioal extension of the discrete Fourier transform is given by 1 F (u, v) = N

M-1 N-1 ∑ ∑ f (x, y) e –j2π x=0 y=0

(ux + vy) / N

(3.2a)

where u = 0, 1, 2, …., M-1, and v= 0, 1, 2, …,N-1. Fourier transforms can be used in a vision system by a number of ways. For example, one-dimensional Fourier transform is used as a powerful tool for detecting motion of an object. The application of 2-dimensional Fourier transform in image enhancement, restoration and image reconstruction is well- known, but the usefulness of this approach is restricted in individual vision system. Because of the need of extensive computational aspects of Fourier transform, a fast Fourier Transform (FFT) algorithm has been developed. When someone wants to do modifications on frequency domain transforms and likes to see the changes in the spatial representation in an image, we need to take inverse Fourier Transform. The inverse Fourier Transform f(x, y) is computed by using the following formula 1

F (x, y) =

M-1 N-1 ∑ ∑ f (u, v) e j2π N u=0 v=0

(ux + vy) / N

(3.2b)

where x = 0, 1, 2, …, M-1 and y= 0, 1, 2, ……, N-1. Discrete Cosine Transformation Discrete cosine transforms are currently being used in JPEG and MPEG compressed image files. DCT-powered compression have delivered a magnificent computing environment to inter-networked world rich with multimedia contents. DCT is normally taken block-wise over a given image. The (8 × 8) forward and inverse two-dimensional DCTs used in JPEG and MPEG images are given by

66

3 Preliminaries on Image Processing

f(u, v)= (1/4) CuCv∑u∑v s(x, y) cos [(2x+1)uπ/16] cos[(2y+1)vπ/16] and s(x, y)= (1/4) ∑u∑v Cu Cv f(u, v) cos [(2x+1)uπ/16] cos[(2y+1)vπ/16]

(3.3a) (3.3b)

where Cu or Cv =1/√2 for u=0 or v=0, and Cu or Cv =1, otherwise. The f(u, v) are the DCT coefficients and s(x, y) are the values of the (x, y) input samples.

3.3 Preprocessing and Noise Filtering After acquisition of an image is complete, we need to preprocess the image to make it free from the contamination of noise. This is well known as image filtering. Various techniques of image filtering are prevalent in the current literature of image processing [10]. The most common among them is neighborhood averaging. 3.3.1 Neighborhood Averaging In “neighborhood averaging” [11], the average intensity in the neighborhood of a pixel is computed for each non-overlap regions on the image. This can be implemented by rolling a window usually of (3 × 3) dimension over the image, and for each position of the window the average intensity of all the 9 pixels inside the window is computed. The values are stored in a 2-dimensional array of suitable size. For displaying the image, the information from the array is transferred to a data file, and a header is attached with the data file for visualization of the image using a suitable image driver. Example 3.1: The smoothing effect produced by neighborhood averaging of an image, corrupted by noise is illustrated in this example. Fig. 3.1 presents the original noisy image, and Fig. 3.2 is obtained after neighborhood averaging on Fig. 3.1. It is noted that in the process of neighborhood averaging, the resulting image in Fig. 3.2 is blurred. This can be overcome by an alternative filtering technique introduced below. 3.3.2 Median Filtering Median filtering [6] is an alternative method for smoothing an image. Unlike averaging, this filtering technique computes the median of intensity in the neighborhood

Fig. 3.1. The original image.

Fig. 3.2. Image obtained by neighborhood averaging on Fig. 3.1.

3.3 Preprocessing and Noise Filtering

67

of a pixel. Usually, a (3 × 3) window is used here to reduce the size of the filtered image by a factor of 1/9. The rolling of the window on the image is similar to neighborhood averaging methods. Example 3.2: Consider the image of a dark robot in a relatively light background (Fig. 3.3). The median filtering algorithm is employed on Fig. 3.3, and the resulting image is shown in Fig. 3.4. From Fig. 3.4, it is clear that blurring can often be reduced significantly by the use of the so called median filters.

Fig. 3.3. The original Image.

Fig. 3.4. The Median Filtering view of Fig.3.3.

3.3.3 Thresholding A gray image usually consists of a number of gray levels between 0 to 2n, where n denotes number of bits used to represent pixel intensity. Typically, in most of the applications, n is set to 8, i.e., there exist 28 (=256) gray levels. Thresholding [5] is required to convert a gray image into a binary image. This is needed to identify important regions of interest in an image. The principle of thresholding is briefly outlined below. Let f (x, y) be the intensity at pixel (x, y) on an image. Let T be a threshold, representing a intensity value such that 0 < T < 2n, where n is the number of bits used to represent pixel intensity. Let g (x, y) be the intensity of the threshold image at pixel (x, y). The g (x, y) may be obtained by using the following rule: g (x, y) = 255, if f (x, y) > T, (3.4) = 0, if f (x, y) ≤ T. In order to select the threshold, we plot the intensity histogram [2] of the input image. Histogram is a plot of frequency in the image against intensity values. This plot usually has a number of peaks representing local maximum in the intensity profile. Generally, a threshold is selected in between two peaks. Such selection helps in automatically identifying a region form the rest. Fig. 3.5 (a) and (b) describe selection of threshold in 2 histogram. In Fig. 3.5 (a), there is a single threshold T, whereas in Fig. 3.5(b) we have 2 thresholds T1 and T2.

3 Preliminaries on Image Processing

Frequency count →

68

T Fig. 3.5(a). Histogram with a single threshold. Fig. 3.5(a): Histogram with a single threshold.

Fig. 3.6(a). The image before thresholding.

T1

T2

Fig. 3.5(b). Histogram with multiple (here 2) thresholds.

Fig. 3.6(b). The image after thresholding.

The importance of thresholding lies in automatic identification of an object from its background. For instance consider a dark object on a light background. Naturally, the histogram will have 2 peaks, one corresponding to the object and the other corresponding to the background. If a threshold T is selected in between 2 peaks, then the dark object can be represented by a darker region, while the light background may be described by a lighter region (Fig. 3.6(a) & (b)). When the histogram contains more than one peak, we need to apply thresholding in multiple steps. For example, consider the histogram shown in Fig. 3.5(b). Here, first we select the threshold T1 to binarize the image into 2 clusters. This helps in identifying a region whose pixel intensities are less than T1. Next, we select a threshold T2, and again binarize the image with respect to this threshold. Naturally, intensity profiles of the pixels that fall below threshold T2 will look darker and pixels having intensity greater than T2 look lighter. There exist novel algorithms to automatically select a unique threshold in an image to identify the regions of interest. This process is generally referred to as image segmentation. When there is single large object on the image against a background of sufficient contrast, segmentation of the region containing the object can be performed by thresholding. The segmentation of Fig. 3.7(a) is given in Fig. 3.7(b) to isolate the dark robot from the rest.

3.4 Image Segmentation Algorithms

69

Fig. 3.7(a). The original image. Fig. 3.7(b). Segmentation of the image Fig. 3.7(a).

3.4 Image Segmentation Algorithms As already introduced, segmentation deals with automatically partitioning an image into regions of interest. Segmentation is one of the most important steps in an automated vision system, because at this stage of processing, objects are extracted from a scene for subsequent recognition and analysis. Segmentation algorithms are designed using any one of the following 2 principles: similarity and discontinuity. Similarity among pixels is determined using thresholding and region growing. The discontinuity among pixels is determined by edge detection. 3.4.1 Boundary Detection Algorithms Boundaries of an object isolate the object from its background. Boundary detection [3], therefore, can be used for identifying the region of an object against its background. Main difficulties in boundary detection arise because of nonuniformity in illumination and spurious intensity discontinuity in the image. One traditional method is to link edge points in background to analyze the characteristics of pixels in a small neighborhood. In other words, pixels that are similar in some imaging characteristics/attributes can be linked to form a boundary of pixels. To determine the similarity of edge pixels on the boundary, we define a gradient operator. Let G (f(x, y)) be the gradient of pixel intensity at point (x, y). Let G (f(x/, y/)) be the gradient of another pixel (x/, y/) in the neighborhood of pixel (x, y). We now check whether: |G (f(x, y)) − G (f (x/, y/))| ≤ T,

(3.5)

where T denotes a threshold, and | . | denotes the modulus operation over its argument. When the inequality (3.5) is satisfied, we say that the gradient of pixel (x, y) is close enough to that of (x/, y/). Consequently we can consider the pixels (x, y) and (x/, y/) to lie on the same edge contour. In the process of edge linking, the above inequality is checked for all pixels on a given image, and contours of edges are constructed following the similarity measurement of edges for the pixels. On occasions, the magnitude of gradients is not adequate, and we need to consider the direction of gradient. The direction of gradient is evaluated by

70

3 Preliminaries on Image Processing

θ = tan-1 (Gy/Gx)

(3.6)

where Gy and Gx be the gradient of pixels along y and x directions respectively. The angle is computed with respect to the x-axis of the image frame. Let θ and θ/ be the directions of gradient of pixels (x, y) and (x/, y/), where (x/, / y ) lies in the neighborhood of pixel (x, y). To check whether a pixel (x, y) and (x/, y/) lie on an edge contour, we use the following in equality. | θ − θ/ | < A

(3.7)

where A is a threshold. Usually, we satisfy both the magnitude and the angle criteria to identify the pixels that lie on the edge contour. Edge detection attempts to detecting significant local changes in an image. Significant changes in the gray value in an image can be detected by using a discrete approximation to the gradient. It is the two-dimensional equivalent of the first derivative and defined by the following vector: df G [ f(x, y)]

=

Gx

dx

=

(3.8)

df

Gy

dy

The vector G [f(x, y)] points to the direction of the maximum rate of increase of the function f(x, y) and the magnitude of the gradient. G [ f(x, y)] = √ (Gx2 + Gy2) ≈ |Gx| + |Gy| ≈ max (|Gx|, |Gy|)

(3.9)

From vector analysis, the direction of the gradient is defined as α (x, y) = tan-1 (Gy / Gx),

(3.10)

where α is measured with respect to x. An edge physically signifies a boundary between two regions with relatively distinct gray level properties. The technique of edge-based segmentation signifies isolation of desired objects from a scene using different types of gradient operators. Sobel Mask: The sobel mask (3 × 3) is given in the following. -1

-2

-1

-1

0

1

0

0

0

-2

0

2

1

2

1

-1

0

1

3.4 Image Segmentation Algorithms

71

Assume that the mask is given by z1

z2

z3

z4

z5

z6

z7

z8

z9

Sobel mask is used to compute derivative of the image in the x direction, and is given by Gx=(z7 + 2z8 + z9) – (z1 + 2z2 + z3). (3.11) Further, Sobel mask is used to compute derivative of the image in the y direction, and is given by Gy=(z3 + 2z6 + z9) – (z1 + 2z4 + z7)

(3.12)

Combining Gx and Gy, we get | ∇f | = (Gx + Gy)1/2 ≈ | Gx | + |Gy | where ∇f denotes the magnitude of the gradient vector. α(x, y) denotes the direction of the gradient vector. The 4 neighbor of a pixel (x, y) is given by (x,y+1)

(x-1,y)

(x,y)

(x+1,y)

(x, y-1) Algorithm of edge detection (using Sobel operator) Begin For (i= 1 to image-height) For (j= 1 to image-width)

(3.13)

3 Preliminaries on Image Processing

72

1. Place a proper window of size (3 × 3) on an input image; 2. At each (i, j) pixel compute the weighed sum wkl zlj; (where wk1 are the window coefficients and z1j denotes the pixel value at position (i, j)); 3. Store the weighted sum at the centre co-ordinate in a separate image file computing the edge of image; Shift the window by 1 unit horizontally; End for; Shift the window by 1 unit vertically; End for; End. Example 3.3: In this example, we outline the concept of boundary direction using edge linking. The gradient-based edge-detection is applied on Fig. 3.8(a), and the resulting image after edge detection by Sobel mask is given in Fig. 3.8(b).

Fig. 3.8(a). The image before applying Sobel mask.

Fig. 3.8(b). The image obtained after

applying Sobel mask on Fig. 3.8(a).

The second derivative of a smoothed step edge is a function that crosses zero at the location of the edge. It is a two-dimensional equivalent of the second derivative. The Laplacian of 2-D function f(x, y) is the second order derivative defined by ∇2f = d2f/dx2 + d2f/dx2 .

(3.14)

A (3 × 3) mask for 4-neighborhood implementation of Laplacian is given by the following expression: ∇2f = 4z5-(z2+z4+z6+z8) .

(3.15)

The Laplacian operator incidentally gives gradient magnitude only. The Laplacian in discrete form looks like the following:

.

0

-1

0

-1

4

-1

0

-1

0

3.4 Image Segmentation Algorithms

73

Algorithm of edge detection (using Laplacian operator) Begin For (i= 1 to image height) For (j= 1 to image width) 1. Place a proper window of size (3 × 3) on an input image; 2. At each (i, j) pixel compute the weighed sum ∑wk1z1j; (where wk1 are the window coefficients and z1j denotes the pixel value at position (i, j)); 3. Store the weighted sum at the centre co-ordinate in a separate image file comprising the edge of image; Shift the window by 1 unit horizontally. End for; Shift the window by 1 unit vertically; End for; End. The Laplacian operator has been applied on Fig. 3.9(a) to obtain edges (Fig. 3.9(b)).

Fig. 3.9(a). The original image.

Fig. 3.9(b). The image obtained after applying Laplacian on Fig. 3.9(a).

Laplace of Gaussian: The 2-D Gaussian smoothing operator G(x, y) is given by the following formula G(x, y) = exp(-(x2+y2)/2σ2),

(3.16)

where (x, y) is a image coordinate. σ is the standard deviation of the associated probability distribution. Mathematically, Laplace of Gaussian (LoG) is represented by ∇2[G(x, y, σ ) * f(x, y)] =[∇2G(x, y, σ )] * f(x, y). writing r2 for x2+y2, we get

(3.17)

3 Preliminaries on Image Processing

74

G(r) = exp(-r2 / 2σ2).

(3.18)

G/(r) = -(ϒ/σ2) (exp(-r2 / 2σ2))

(3.19)

G//(r) = (1/σ2) (r2/σ2 – 1) (exp(-r2 / 2σ2)).

(3.20)

Differentiating once, we get

Differentiating once again

Now, substituting r2 = x2+y2, and incorporating a normalization constant c, we can re-write (3.20) as h(x, y)=c (-(x2+y2-σ2) / σ4 ) exp (-(x2+y2)/2σ2 ).

(3.21)

A discrete approximation of (5 × 5) LoG operator ∇2G is given by 0

0

-1

0

0

0

-1

-2

-1

0

-1

-2

16

-2

-1

0

-1

-2

-1

0

0

0

-1

0

0

The above discrete matrix representation of LoG is not unique. However, one pertinent feature of the above mask is that the centermost coefficient has the largest value 16, and there is a decay from that value on all the 4 sides. Algorithm of edge detection (Laplacian of Gaussian operator)

Begin For (i= 1 to image height) For (j= 1 to image-width) 1. Place a proper window of size (3×3) on an input image; 2. At each (i, j) pixel compute the weighed sun ∑wklz1j; (where wk1 is the window coefficients and z1j denotes the pixel value at position (i, j)); 3. Store the weighted sum at the centre co-ordinate in the separate image file comprising the edge of image; Shift the window by 1 unit horizontally;

3.4 Image Segmentation Algorithms

75

End for; Shift the window by 1 unit vertically; End for; End.

Fig. 3.10(a). The original image.

Fig. 3.10(b). The image after applying LOG on Fig. 3.10(a).

The LOG operator has been applied on Fig.3.10 (a) to obtain edges Fig. 3.10(b).

3.4.2 Region Oriented Segmentation Algorithm As already discussed, the objective of segmentation is to partition an image of region R into n different regions R1 , R2 , …..,Rn such that n 1. ∪ Ri = R i=1 2. Ri is a connected region i = 1, 2, …, n; 3.

Ri ∩ Rj = φ for all i, j, and i ≠ j;

4.

P (Ri) = True for i =1, 2, …, n;

5.

P (Ri ∪ Rj) = False for i ≠ j.

Condition 1 indicates that every pixel in the image must be present in one of the regions. Condition 2 implies that there must be 0 or more regions between every two regions on the image. Condition 3 states that the regions need to be distinct. Condition 4 states that some properties must hold for all the pixels on the regions Ri. For example, if all pixel in a region are of same intensity, the P (Ri) = True. It is important to note that instead of intensity profile, any other imaging parameters/attributes such as moments, distance metric etc. can also be used to justify the P (Ri) to be true. Finally, condition 5 indicates that 2 regions together cannot satisfy a common imaging attribute. There are 2 common methods for region-oriented segmentation as introduced below.

76

3 Preliminaries on Image Processing

3.4.2.1 Region Growing by Pixel Aggregation Under this method, pixels are grouped to form region, and sub-regions are grouped to form larger region, depending on the common feature of the pixels or sub-regions. Usually, the algorithm starts with a set of “seed” points, which grow into regions by appending neighborhood pixels to the seed point by judging similarities in some imaging attributes, such as intensity, texture and color. The regions then gradually expand through augmentation of neighborhood pixels with respect to the same imaging characteristics. The accommodation of pixels to a region is continued, until no further pixel can be appended because of difference in imaging characteristics. 3.4.2.2 Regions Splitting and Merging Under this context, we sub-divide an image into a set of disjoint regions, and then merge and/or split the regions, so as to obtain uniformity in some characteristics/attributes of the regions. The splitting and merging of regions is continued, until no further splitting or merging is necessary. One simple method for region splitting and merging is to subdivide an image into quadrants and then sub-divide

R1

R2

R41

R42

R43

R44

R3

Fig. 3.11. Partitioning of a region R into four quadrants: R1, R2, R3 and R4, and repartitioning of R4 into sub-regions: R41, R42, R43, R44.

R

R1

R2

R41

R3

R42

R4

R43

Fig. 3.12. The quadtree corresponding to Fig. 3.11.

R44

3.4 Image Segmentation Algorithms

77

a quadrant into sub-quadrants, and so on until no further division is meaningful. A structure that ultimately results in is called a quadtree. Figure 3.11 describes partitioning of a region R into 4 quadrants and then re-partitioning of the region R4 into sub-quadrants R41, R42, R43, R44. Fig. 3.12 illustrates the corresponding quadtree as obtained from Fig. 3.11 3.4.2.3 Image Segmentation by Fuzzy Clustering Fuzzy logic has proved its excellence in clustering of noisy data. In recent times, Raghukrishnapuram took an initiative to use fuzzy clustering algorithm in image segmentation. In this section, we briefly outline fuzzy clustering and demonstrate the scope of its application in image segmentation. 3.4.2.3.1 The Fuzzy C-means Clustering Algorithm The objective of the fuzzy c-means clustering algorithm [8] is to classify a given set of p dimensional data points X = [x1 x2 x3 …xn ] into a set of c-fuzzy classes or partitions Ai [6], represented by clusters, such that the sum of the memberships of any component of X, say xk, in all the c classes is 1. Mathematically, we can represent this by c ∑ μAi (xk) =1, for all k = 1 to n. i=1

(3.22)

Further, all elements of X should not belong to the same class with membership 1. This is so because otherwise there is no need of the other classes. Thus mathematically, we state this by n 0 < ∑ μAi (xk) < n. k=1

(3.23)

For example, if we have 2 fuzzy partitions A1 and A2, then for a given X = [x1 x2 x3] say, we can take μAi = [ 0.6/ x1 0.8/ x2 0/x3] and μA2 = [0.4/ x1 0.2/ x2 1/x3 ]. It is to be noted that the conditions described by expressions (3.22) and (3.23) are valid in the present classification. Given c classes A1, A2, …, Ac we can determine their cluster centers Vi for i = 1 to c by using the following expression. n n Vi = [ ∑ [μAi(xk) ]m xk] / ∑ [μAi(xk) ]m . k=1 k=1

(3.24)

78

3 Preliminaries on Image Processing

Here, m (> 1) is any real number that influences the membership grade. It is to be noted from expression (3.24) that the cluster center Vi basically is the weighted average of the memberships μAi(xk). A common question now naturally arises: can we design the fuzzy clusters A1, A2, …., Ac in a manner so that the data point (feature) vector xk for any k is close to one or more cluster centers Vi.. This can be formulated by a performance criterion given by Minimize Jm over Vi (for fixed partitions U) and μAi (for fixed Vi) n c Jm (U, Vi) = ∑ ∑ [μAi(xk) ]m ⎢⎢xk - Vi ⎢⎢2 k=1 i=1

(3.25)

c subject to ∑ μAi(xk) =1 i=1

(3.26)

where ⎢⎢. ⎢⎢ is a inner product induced norm in p dimension. First, let us consider the above constrained optimization problem for fixed Vi. Then we need to satisfy (3.25) and (3.26) together. Using Lagrange’s multiplier method, we obtain that the problem is equivalent to minimizing n c n c L(U, λ) = ∑ ∑ (μAi (xk))m || xk – Vi||2) - ∑ λk (∑μAi (xk) – 1) k=1 i=1 k=1 i=1

(3.27)

without constraints. The necessary condition for this problem is ∂L ∂μAi ∂L and ∂λk

= m (μAi (xk)) m – 1 || xk – Vi|| 2 - λk = 0

c = ∑μAi (xk) – 1 = 0. i=1

(3.28)

(3.29)

From (3.28) we have 1 / (m – 1) μAi (xk) =

λk (3.30) m || xk – Vi||2

.

3.4 Image Segmentation Algorithms

79

Substituting (3.30) in (3.29) we find: 1/ (m – 1) λk

1 =

m

,

(3.31)

c ∑ ( 1/ ||xk – Vi || 2) 1 / (m – 1) i=1

Substituting (3.31) in (3.30) we have: μAi (xk)

c = [∑ ( || xk – Vi ||2 / ||xk – Vj||2 ) 1/ (m – 1) ] –1 j=1

(3.32)

for 1 ≤ i ≤c and 1≤ k ≤n. Now, suppose μAi (xk)’s are fixed. Then this is an unconstrained minimization problem, and the necessary condition is ∂Jm

n

= - ∑ 2 ( μAi(xk)) m (xk – Vi) = 0 ∂Vi k=1

(3.33)

which yields (3.24). This is a great development, which led to the foundation of the fuzzy c-means clustering algorithm. The algorithm is formally presented below. Procedure Fuzzy c-means clustering Input: Initial pseudo-partitions μAi(xk) for i= 1 to c and k= 1 to n Output: Final cluster centers Begin Repeat For i:= 1 to c Evaluate Vi by expression (3.24); End For; For k:= 1 to n For i:= 1 to c Call μAi(xk) OLD_μAi(xk); If ⎢⎢xk - Vi ⎢⎢2 > 0 Then evaluate μAi(xk) by (3.32) and call it CURRENT_μAi(xk) Else do Begin set μAi(xk):=1 and call it (them) CURRENT_μAi(xk) and set μAj(xk):=0 for j≠i and call it (them) CURRENT_μAj(xk); End If;

3 Preliminaries on Image Processing

80

End For; End For; Until ⎢CURRENT_μAi(xk) − OLD_μAi(xk) ⎢ ≤ ∈ = 0.01(say) End. The procedure fuzzy c-means clustering inputs the xk’s and the initial values of μAi(xk)’s supplied by the user for i=1 to c and k=1 to n. It then computes Vi for i= 1 to c. If ⎢⎢xk - Vi ⎢⎢2 > 0 then the procedure evaluates μAj(xk) by (3.32) and defines μAj(xk) as its current value. If ⎢⎢xk - Vi ⎢⎢2 = 0 then the point xk has been correctly located at the clusters and thus its current value of μAi(xk)’s is set to 1, and consequently the current membership values of μAj(xk) for j≠i are set to 0. The outer repeat-until loop keeps track of the terminating condition, ensuring that the

algorithm will terminate only when ⎢CURRENT_μAi(xk) − OLD_μAi(xk)| ≤ ∈ = 0.01(say). On execution of the procedure, it returns the location of the cluster centers. Table 3.1. A set of fifteen 2-dimensional data points

k xk1 xk2

1 0 0

2 0 2

3 0 4

4 1 1

5 1 2

6 1 3

7 2 2

8 3 2

9 4 2

10 11 12 13 14 15 5 5 5 6 6 6 2 1 2 3 0 4

Example 3.3: Consider a set of fifteen 2-dimensional data points x1 to x15 (Table 3.1), we want to classify them into 2 pseudo-partitions A1 and A2, say. Let the components of each point xk be xk1 and xk2. Thus, we can plot these 15 points with respect to xk1 and xk2 axes. Let the initial pseudo-partitions be A1 and A2, such that μA1 = {0.854/ x1

0 .854/x2 …….. 1/x15} and

μA2 ={0.146/ x1

0.146/x2

…….. 0/x15}.

We now following the procedure evaluate the cluster centers for the 2partitions. Here, the cluster center for partition A1 will have 2-coordinates one for xk1 and the other for xk2. Let the xk1 coordinate for partition A1 be V11 and the xk2 coordinate for the same partition be V12. Then with m= 1.25 we find: {(0.854)1.25 (0+0+0+1+1+1+2+3+4+5+5+5+6+6) + (1)1.25 (6)} V11 = =3.042, and

{14 (0.854)1.25 + (1)1.25}

3.4 Image Segmentation Algorithms

81

{(0.854)1.25 (0+2+4+1+2+3+2+2+2+2+1+2+3+0) + (1)1.25 (4)} V12 =

{14 (0.854)1.25 + (1)1.25}

= 2.028. The cluster centers V21 and V22 for the partition A2 can also easily be evaluated by replacing 0.854 by 0.146 and 1.0 by 0 in the last 2 expressions. The computation of the new membership of each point xk can now also be evaluated with the computed values of V11, V12, V21 and V22 and last coordinate of the points. After 6 iterations of execution of the algorithm, the cluster centers are found as V1 = (V11, V12)= (0.88, 2.0) and

V2 = (V21, V22)= (5.14, 2.0).

The membership values of the points to belong to partition A1 and A2 after 6 iterations are found as μA1 = {0.99/ x1 1./x2 0.99/x3 1 /x4 1/x5 1/x6 0.99/x7 0.47/x8 0.01/x9 0/x10

0/x11 0/x12 0.01/x13 0/x14 0.01/x15} and

μA2 ={0.01/ x1

0/x2

1/x10

1/x11

0.01/x3 0/x4 0/x5 0/x6 0.01/x7 0.53/x8 0.99/x9 1/x12

0.99/x13 1/x14 0.99/ x15}.

It may indeed be noted that sum of the memberships of any point xk to belong to the 2 partitions A1 and A2 to be 1 is always maintained irrespective of the iterations. 3.4.2.3.2 Image Segmentation Using Fuzzy C-Means Clustering Algorithm Quite a number of well-known algorithms of image segmentation is available in a textbook of image processing [4]-[10]. In this section, we following Raghukrishnapuram present a study of image segmentation [8] using fuzzy c-means algorithm. The fuzzy c-means clustering algorithm for image segmentation has 2 parameters, namely, the number of clusters, denoted by ‘c’, and the exponential weighting factor ‘m’ over the membership functions. The experiments are performed by gradually varying these two parameters and their effects on clustering are noted. The number of clusters needed is usually determined by the problem in hand. For segregating a dark object from a light background or vice- versa, we should select c=2, and thus obtain 2 clusters, one corresponding to the dark region and the other

82

3 Preliminaries on Image Processing

(a)

(b)

(c)

(d)

Fig. 3.13. Fuzzy C -means clustering algorithm applied to segment the given image (a) into 2 clusters with m=1.01 in (b), m=1.2 in (c), and m=2.5 in (d).

(a)

(b)

(c) Fig. 3.14. Fuzzy C-means clustering algorithm applied to segment the image in 3.13(a) into 3 Clusters with m=1.01 in (a), m=1.2 in (b) and m=2.5 in (c).

to the lighter region. Fig 3.13 shows the results of clustering with c=2. Further, the value of exponential weighting factor m has been increased in steps from m slightly greater than 1, followed by m=1.2 and m=2.5. The variation of m clearly indicates the difference between a hard cluster and a soft cluster.

3.4 Image Segmentation Algorithms

83

For the purpose of illustration, we have constructed the figures using the membership value of each pixel mapped to gray value levels. The contrasting shades in Fig. 3.13(b) indicate that each pixel belongs to either of the classes with large membership value for one class, and with a very small membership value for the other class. This is typical of a hard cluster, where the pixels are assigned to either of the two classes. In the subsequent Figures 3.13(c) and 3.13(d), the 2 shades become less contrasting. As the shades are representing the membership values, it means that each pixel now have intermediate membership values of belonging to the 2 classes. This is fuzzy clustering where each pixel has finite memberships of belonging to the 2 classes. Thus, we have greater latitude of deciding which pixels to select based on their membership values. In Fig. 3.14, the number of clusters is 3 and they are represented by 3 shades: dark, intermediate gray and light. Here, too, we observe that how the value of m alters the final membership values of the pixels. As explained for Fig. 3.13, a

(a)

(b)

(c) Fig. 3.15. Fuzzy C-means clustering algorithm applied to isolate very black objects from the image in (a) with grayscale as the feature vector in (b), and logarithm of the grayscale as the feature vector in (c ).

84

3 Preliminaries on Image Processing

value of m close to 1 (m=1.01) generates crisp clusters, while m=1.2 or larger makes the clusters fuzzy. In all the above experiments, we used gray level of the pixels as the feature of interest and accordingly constructed the feature vector. In Fig 3.15(b), it is observed that although the clustering is done, we do not get the cluster as expected intuitively. This may be attributed to the fact that the general illumination of the image is so low that the black objects cannot be distinguished from the surrounding floor based on gray level alone. We propose that in such circumstances, some transformation of the gray level values be taken to construct the feature vector. The choice of this transformation is based on some prior knowledge of the object of interest. To illustrate the above point, let us consider the case where we are interested in segregating some dark object from a poorly illuminated workspace. The suitable transformation in such a circumstance is to take the logarithm of the gray level values of the pixels. As we know, logarithmic transform expands and emphasizes the lower values and cramps the larger value in a smaller range, thus de-emphasizing them. Fig. 3.15(c) illustrates the process of clustering with gray levels as the elements of the feature vector. It is now observed that this transformation has successfully clustered the ‘very’ dark objects from the dark environment. Thus, we have a third controlling factor, namely, the transformation on the gray level whereby we may classify the image according to the requirements of the problem in hand.

3.5 Boundary Description After an image is segmented into components/regions of interest, a labeling algorithm is invoked to describe the boundary of the image. There exist a large number of boundary descriptor algorithms in the current literature on image processing [4], [6]. Some well-known algorithms that need special mention in the present content are: chain codes, Fourier descriptors, shape numbers and moments. We briefly outline the algorithms below. 3.5.1

Chain Codes

One simple way to describe the chain code is to consider 8 vectors representing 8 directions at an interval of 45o each. Let the vectors be denoted by 0, 1, 2, …,7 where the integers denote the co-efficient of 45o, describing angles suspended by the vectors with respect to the x-axis. Fig. 3.16 describes the vectors. After an image is segmented into regions, we can describe the boundary of the region by the 8-dimensional Chain code. For representation of the boundary, we consider a straight line of small length. The smaller is the length of the line, the better the accuracy. We now appropriately construct a polygon to describe the boundary using the said straight line. The representation index of the line is used to describe the boundary as a string of digits.

3.5 Boundary Description

85

6 5 7

4

0

1

3 2

Fig. 3.16. The eight directional chain code.

The principle outlined above is presented below using a given region (Fig. 3.17) 0 2

6

2

0 6

2

6

2 2

0

0

6

B 4 Fig. 3.17(a). A segmented image.

4

4

4

Fig. 3.17(b). Boundary descriptor of Fig. 3.17(a).

Fig. 3.17(a) represents the boundary of a segmented image, and Fig. 3.17(b) represents the corresponding boundary descriptor: 4 – 4 – 4 – 4 –2 – 2 – 2 – 2 –2- 0 – 6 – 0 – 6 – 6 – 0 – 0 – 6 – 4. The boundary descriptors thus obtained can take different forms, if we start labeling from point B instead of point A. One way to overcome this problem is to consider the circularity of the boundary descriptor. We re-write the code in a manner, so that the resulting string generates the smallest possible number. In the given example, the smallest resulting number is given by 0 0 6 4 4 4 4 2 2 2 2 2 0 6 0 6 6. The smallest number being unique may be used as a chain code descriptor of the given polygon.

86

3 Preliminaries on Image Processing

3.5.2 Fourier Descriptors Consider the boundary of an image in a complex plane as points xk + jyk for k = 1 to n. The sequence of points along the boundary forms a function, whose Fourier transform F (u) for u = 0, 1, 2, …., (M − 1) can be used a descriptor. Computation of F(u) can be done by Fast Fourier Transform, (FFT) algorithm. The fundamental advantage of Fourier Descriptor lies in the fact that the first few Fourier coefficients contain sufficient information to discriminate two dissimilar objects. The time needed to match object-boundary by Fourier descriptor thus is very efficient. 3.5.3 Regional Descriptors A region can be described in terms of its major and minor axis. This axis is useful for establishing the orientation of an object. Sometimes the ratio of the length of these 2 axes, called eccentricity of the region, is used as a global descriptor of shape of the region. On occasions, the perimeter of a region also is used to describe a boundary of an object. In many image-processing applications, compactness of a region, defined as perimeter2/area is used to describe the region. A region in which all pairs of point can be connected by a curved line on the region is called a connected region. In a set of connected regions, a few of which may include holes, we normally use Euler number as a region descriptor. The Euler number is defined as the difference of the number of connected regions and the number of holes. For example, the letter B that contains 2 holes and one connected region has an Euler number 1 − 2 = −1. On the contrary the letter D that contains 1 hole and 1 connected region has an Euler number 0. The texture of a region [7] that describes smoothness, coarseness and regularity/repeatability of granules in a region, too is a important parameter to describe a region. Usually, statistical approaches are employed to characterize the texture in an image. One of the simplest methods to describe texture is to employ moments of intensity histogram of an image. Consider an image consisting of L number of gray levels. Let x be a random variable describing image intensity, and P(xi) denotes the probability of occurrence of xi in the image. The mean (average) intensity of the image now can be computed by using expression (3.34). L M = ∑ xi P (xi) i =1

(3. 34)

The nth moment about the mean μn(x) is not defined as L μn (x) = ∑ ( xi − m)n P (xi) i =1

( 3.35)

3.6 Object Recognition from an Image

87

When n = 2 in (3. 35), we compute μ2, the variance of the image. The descriptor of relative smoothness R, now can be computed from the expression of variance σ2 (x). 1 R = 1−

(3.36) 1 + σ2 (x)

It is indeed important to note that for a region of uniform intensity, variance = 0 and thus R = 1 − 1 = 0.

3.6 Object Recognition from an Image After the segmentation of an image is over, sometimes we need to recognize the object in a scene. In other words, the segmented objects in an image are to be labeled. The recognition process can be accomplished by various techniques including decision-theoretic models, classification techniques and automated clustering techniques. In this book, we present an overview of supervised classification and clustering techniques. 3.6.1 Unsupervised Clustering The process of unsupervised clustering involves extraction of features from a given image or segmented region, and representation of the segmented region in some form in the feature space. Usually, the intensity values in a monochrome image or the RGB values in a color image carry sufficient information for clustering. For instance, pixels having similar intensity levels automatically form a class, defining an object’s contour. Naturally, when the image contains n objects, all the objects maintain their distinction in the feature space and thus are automatically clustered. In supervised learning on the other hand, a table comprising extracted features of the object and the possible class to which the object may be categorized are first determined. The feature class tuples listed in the table are called training instances. Various neural network based algorithms are prevalent in the current literature on unsupervised clustering. Hopefield network [8] for instance is typical neural system that can classify a given pattern into one of the stable points stored by the network. It may be noted that a stable point has the same dimension as that of the given patterns and thus a given pattern can always be mapped to the nearest stable point. Usually, the feature space of an object is represented by a vector, and the object described by its features is mapped to its stable point. In other words, stable points correspond to different objects. Naturally, noisy pattern representing an object is automatically mapped to a stable point, representing a specific (known) object. Besides the Hopefield net, Self-Organizing Feature Map (SOFM) neural networks and Principal Component Analysis (PCA) are equally useful for automatic

88

3 Preliminaries on Image Processing

clustering of unknown high dimensional data points, describing the object. An SOFM neural network maps similar data points in the geographically close neighborhood of a set of 2D neurons. Thus objects of similar types are automatically mapped on to the neurons in a 2D plane at close proximity. 3.6.2 Supervised Classification Supervised learning can also be used for recognition of objects. Unlike unsupervised clustering, a supervised learning algorithm should be provided with all the necessary input-output training instances. The input-output components of a training instance here correspond to stimuli-response pair of a cognitive system, where the stimulus denotes a feature vector describing the object, and the response denotes the object under reference. For example, we can define 2D geometric objects by its area, perimeter and number of line segments. These features may be used as the stimulus-input of the neural networks, whereas the output vector provides an object-description in linguistic, phonetic or other forms. The components of output vector usually have binary values. When the features of a particular object are applied as the stimulus input of the neural net, one component of the output is set high and the rest are low. The training instances for the objects thus are prepared in a tabular form. Supervised learning algorithms [7], ]8] can store the input-output instances of an object in its memory. When a pattern, sufficiently close to one of the known patterns is submitted to a supervised learning algorithm, it yields an average of the similar responses stored in the memory. Some of the well known supervised learning algorithm includes back-propagation neural learning, Widrow-Hoff’s least min-square algorithm and the Radial Basis Function (RBF) neural learning algorithm. In this book, we used back-propagation learning algorithm in human emotion detection. Given the facial features of a person representing different emotion, we can recognize the emotions from the facial attributes of the person. A set of training instances may be prepared to describe the mapping from facial attributes to emotion. The training may be used to train a feed-forward neural network by back propagation type supervised learning. During the recognition phase, the facial features of the person is supplied to the network as the input, and the network yields the average of the nearest stored patterns as the output. 3.6.3 Image Matching Another form of recognition of objects from images is performed through image matching. Matching of images can be done in different levels. It may be at pixellevel, sub-pixel level or feature level. Pixel-level matching algorithms usually are very slow as it involves n2 comparisons to match images of (n × n) pixels. Most pixel-level matching algorithms employ a distance metric over the pixels at same location of the two images. One such metric is the Euclidean distance [2] that takes the sum of the difference square of the intensities at pixel (x, y) of the two images for x=1to n and y = 1 to n.

3.6 Object Recognition from an Image

89

Formally, n n D = Σ Σ (f(x,y) – g(x, y))2 y =1 x=1

(3.37)

where f(x, y) and g (x, y) denote the intensities of two images at pixel (x, y). Given a set of N images and a reference image R- all of same (n × n) pixels. We need to match R with the images available in the set. Here, we use the Euclidean distance metric to measure the distance between R and any image i, where 1 ≤ i ≤ N. Suppose, the kth image is found to have the smallest distance Dmin where Dmin ≤ Di , for i = 1 to N. We under this circumstance, consider the kth image to be the nearest matched image with respect to the reference image R. The main drawback of the above image-matching scheme is the computational overhead of pixel-wise matching, and therefore is not recommended for real time applications. To circumvent the above problem, we consider feature-based image matching. In feature based image matching, we attempt to extract some features from the image. Let the features be f1, f2, …, fm. The number of features, m usually is much smaller than the image size (n × n). So, any distance metric used to compare two images, will take small time when feature-based matching is performed. 3.6.4 Template Matching Sometimes, we need to match a template [2] over a large image. Usually, the templates are of smaller dimensions in comparison to an image. In template matching scheme, the template is first placed at the left-top of the image, and some features of the template are compared with the part of the image, covered by the template using a pre-defined distance metric. The template is rolled over the image, and for each position of the template, we evaluate the “matching score” by the same distance metric. In the rollover process, suppose we evaluate k-times the distance metric D. Then the i-th position of the template, where Di ≤ Dl, for 1 ≤ l ≤ k is called the best position of the template in the image. One important aspect in template matching is to determine a good policy for successful positioning of the template. If the two successive locations of the template have a maximum overlap, then matching yields good accuracy. But a large overlap between two successive positions of the template increases the timecomplexity of the matching algorithm. A moderate choice is to shift the template half of its width, and continue positioning it k-times along the x-width of the image. For the template of (m × m) and image of (n × n), k = (2n/m – 1). After k xshifts, we need to position the template by shifting it down by half of its width from its initial position, and continue shifting it along its x-width direction ktimes. Such process of shifting the template over the image requires (2n/m – 1)2 comparison. The complexity of matching is thus significantly reduced.

90

3 Preliminaries on Image Processing

(a)

(b) Fig. 3.18. The template (b) is searched in the image (a) by a template-matching algorithm.

One illustrative template and a group photograph containing the template are presented in Fig. 3.18(b) and 3.18(a) respectively. A pixel-wise template matching algorithm with interleave of half of the width of the template has been used in the present context to correctly position the template in the group photograph. The Euclidean norm has been employed here to measure the matching score of the template with the part the image covered by the template. It may be noted that in the present template-matching scheme both the template and image are of unequal height and width. This however, does not make any serious problem as the height of the image is an integer multiple of the height of the template. Similarly, the width of the image is also an integer multiple of the width of the template.

3.7 Scene Interpretation After the objects in a scene are recognized, automated interpretation [9] of the scene is needed in some applications. The interpreter program attempts to determine spatial/structural relationships among the recognized objects, and then employs classical logic or rule–based techniques to interpret the scene. Sometimes temporal/spatio-temporal propositions/predicates are constructed to describe the

References

91

object geometry, size, shape, color and other attributes of the object. These predicates are then used to instantiate rules, expressed in first order logic. The reasoning methodology adopted for such systems are available in any standard textbook on Artificial Intelligence [7]. The detailed discussion on these is beyond the scope of the book.

3.8 Conclusions This chapter introduced some fundamental algorithms in image processing. Computer simulations are given to illustrate the algorithms. Main focus of concern of the chapter is given on image segmentation. The image segmentation using fuzzy C-means clustering will be used in subsequent sections in this book. A brief outline to image filtering, object recognition and scene interpretation are included for the sake of completeness of the book.

Exercise 1. 2.

3. 4. 5.

Write a program for median filtering in C/Pascal or any programming language you are familiar with. Implement fuzzy C-means (FCM) clustering algorithm in your favorite language. Use r-g-b features of an image to cluster it using FCM clustering algorithm. Write a program for obtaining chain code of a given 2D object boundary in your favorite language. Design a program for template matching by using the algorithm outlined in the text. Read any textbook on neural nets [8], and design programs for a) training, and b) recognition of patterns using back-propagation algorithm.

References 1. Ahmed, N., Nataranjan, T., Rao, K.R.: Discrete Cosine Transforms. IEEE Trans. Comp. c-23, 90–93 (1974) 2. Biswas, B.: Building Intelligent Decision-Support System for Criminal Investigation, Ph.D. Thesis, Jadavpur University (2002) 3. Davis, L.S.: A Survey of Edge detection Techniques. Computer Graphics Image Processing 4, 248–270 (1975) 4. Gonzalez, R.C., Woods, R.E.: Digital Image processing. Addison-Wesley, Reading (2000) 5. Harlick, R.M., Shapiro, L.G.: Survey: Image Segmentation. Computer Vision, Graphics, Image Processing 29, 100–132 (1985) 6. Jain, A.K.: Fundamentals of Digital Image Processing. Prentice-Hall, Englewood Cliffs (1989)

92

3 Preliminaries on Image Processing

7. Konar, A.: Artificial Intelligence and Soft Computing: Behavioral and Cognitive Modeling of the Human Brain. CRC Press, Boca Raton (1999) 8. Konar, A.: Computational Intelligence: Principles, Techniques and Applications. Springer, Heidelberg (2006) 9. Marr, D.: Vision. Freeman, San Francisco (1982) 10. Pratt, W.K.: Digital Image Processing. John Wiley & Sons, New York (1978) 11. Schalkoff, R.J.: Digital Image Processing and Computer Vision. John Wiley & Sons, New York (1989)

5 Fuzzy Models for Facial Expression-Based Emotion Recognition and Control

The chapter examines the scope of fuzzy relational approach to human emotion recognition from facial expressions, and its control. Commercial audio-visual movies pre-selected for exciting specific emotions have been presented before subjects to arouse their emotions. The video clips of their facial expressions describing the emotions are recorded and analyzed by segmenting and localizing the individual frames into regions of interest. Selected facial features such as eyeopening, mouth-opening and the length of eyebrow-constriction are next extracted from the localized regions. These features are then fuzzified, and mapped on to an emotion space by employing Mamdani type relational model. A scheme for the validation of the system parameters is also presented. The later part of the chapter provides a fuzzy scheme for controlling the transition of emotion dynamics toward a desired state using suitable audio-visual movies. Experimental results and computer simulations indicate that the proposed scheme for emotion recognition and control is simple and robust with a good level of experimental accuracy.

5.1 Introduction Humans often use non-verbal cues, such as hand gestures, facial expressions and tone of the voice to express their feeling in interpersonal communications. Unfortunately, currently available human-computer interfaces do not take complete advantage of these valuable communicative media, and thus are unable to provide the full benefits of natural interaction to the users. Human-computer interactions could be improved significantly, if computers could recognize the emotion of the users from their facial expressions and hand gestures, and react in a friendly manner according to the users’ needs and preferences [4]. The phrase affective computing [30] is currently gaining popularity in the literature of human computer interfaces [35], [46]. The primary role of affective computing is to monitor the affective states of the people, engaged in critical/accident-prone environment, in order to provide assistance in terms of appropriate alerts to prevent accidents. Li and Ji [25] proposed a probabilistic framework to dynamically model and recognize users’ affective states, so as to provide them corrective assistance in a A. Chakraborty and A. Konar: Emotional Intelligence, SCI 234, pp. 133–173. © Springer-Verlag Berlin Heidelberg 2009 springerlink.com

134

5 Fuzzy Models for Facial Expression-Based Emotion Recognition and Control

timely and efficient manner. Picard et al.[31] stressed the significance of human emotions on their affective psychological states. Rani et al. [32] presented a novel scheme for fusion of multiple psychological indices for real-time detection of a specific affective state (anxiety) of people using fuzzy logic and regression trees, and compared the relative merits of the two schemes. Among the other interesting applications on affective computing, the works by Scheirer et al. [35], Contai and Zhou [7], Kramer et al. [22], and Rani et al. [32], [33] need special mention. Apart from human-computer interfaces, emotion recognition by computers has interesting applications also in computerized psychological counseling and therapy, and detection of criminal and anti-social motives. Identification of human emotions from facial expressions [27] by a machine is a complex problem for the following reasons. First identification of the exact facial expression from a blurred facial image is not easily amenable. Secondly, segmentation of a facial image into regions of interest sometimes is difficult, when the regions do not have significant difference in their imaging attributes. Thirdly, unlike humans, machines usually do not have visual perception to map facial expressions into emotions. Very few works on human emotion detection have so far been reported in the current literature on machine intelligence. Ekman and Friesen [9] proposed a scheme for recognition of facial expressions from the movements of cheek, chin and wrinkles. They have reported that there exist many basic movements of human eyes, eyebrows and mouth, which have direct co-relation with facial expressions. Kobayashi and Hara [18]-[20] designed a scheme for recognition of human facial expressions using the well-known back-propagation neural algorithms [16], [39]-[41]. Their scheme is capable of recognizing six common facial expressions depicting happiness, sadness, fear, anger, surprise and disgust. Among the well-known methods of determining human emotions, Fourier descriptor [40], template-matching [2], neural network models [11], [34], [40] and fuzzy integral [15] techniques need special mention. Yamada in one of his recent papers [45] proposed a new method for recognizing emotions through classification of visual information. Fernandez–Dols et al. proposed a scheme for decoding emotions from facial expression and content [12]. Carroll and Russell [16] in a recent book chapter analyzed in detail the scope of emotion modeling from facial expressions. Busso et al. compared the scope of facial expressions, speech and multimodal information in emotion recognition [4]. Cohen in her papers [5], [6] considered temporal variation in facial expressions, displayed in live video to recognize emotions. She proposed a new architecture of hidden Markov models to automatically segment and recognize facial expressions. Gao et al. [13] presented a methodology for facial expression recognition from a single facial image using line-based caricatures. Lanities et al. [23] proposed a novel technique for automatic interpretation and coding of face images using flexible models. Among other interesting works on recognition of facial expression, conveying emotions [3], [8], [10]. [24], [26], [29], [34], [37], [38], [44] need special mention. The chapter provides an alternative scheme for human emotion recognition from facial images, and its control using fuzzy logic. Audio-visual stimulus is

5.2 Filtering, Segmentation and Localization of Facial Components

135

used to excite emotions of the subjects, and their facial expressions are recorded as video movie clips. The individual video frames are analyzed to segment the facial images into regions of interest. Fuzzy C-means clustering algorithm [1] has been used for segmentation of the facial images into three important regions containing mouth, eyes and eyebrows. Next a fuzzy reasoning algorithm is invoked to map fuzzified attributes of the facial expressions into fuzzy emotions. The exact emotion is extracted from fuzzified emotions by a de-normalization procedure, similar to defuzzification. The proposed scheme for emotion recognition is both robust and insensitive to noise, because of the non-linear mapping of image-attributes to emotions in the fuzzy domain. Experimental results reveal that the detection accuracy of emotions for adult male, adult female and children between (8-12) years are as high as 88%, 92% and 96% respectively, outperforming the percentage accuracies of the existing techniques [14], [20]. The later part of the chapter proposes a scheme for controlling emotion by judiciously selecting appropriate audio-visual stimulus for presentation before the subject. The selection of the audio-visual stimulus was undertaken using fuzzy logic. Experimental results indicate that the proposed control scheme has good experimental accuracy, and repeatability. The chapter is organized into eight sections. Section 5.2 provides new techniques for segmentation and localization of important components in a human facial image. In section 5.3, a set of image attributes, including eye-opening, mouth-opening, and the length of eyebrow-constriction is determined on-line in the segmented images. In section 5.4, we fuzzify the measurements of imaging attributes into 3 distinct fuzzy sets: HIGH, MEDIUM and LOW. Principles of fuzzy relational scheme for human emotion recognition are also stated in this section. Experimental issues pertaining to emotion recognition are covered in section 5.5. Validation of the proposed scheme has been undertaken in section 5.6. The scheme attempts to tune the membership distributions so as to improve the performance of the overall system to a great extent. A complete scheme for emotion control with detailed experimental issues is covered in section 5.7. Conclusions are listed in section 5.8.

5.2 Filtering, Segmentation and Localization of Facial Components Identification of facial expressions by pixel-wise analysis of images is both tedious and time consuming. The chapter attempts to extract significant components of facial expressions through segmentation of the image. Because of the difference in the regional profiles on an image, simple segmentation algorithms, such as histogram based thresholding technique, do not always yield good results. After several experiments, it has been observed that for the segmentation of the mouthregion, a color-sensitive segmentation algorithm is most appropriate. Further, because of apparent non-uniformity in the lip color profile, a fuzzy segmentation

136

5 Fuzzy Models for Facial Expression-Based Emotion Recognition and Control

algorithm is preferred. Taking into account of the above viewpoints, a colorsensitive Fuzzy C-Means clustering algorithm [9] has been selected for the segmentation of the mouth region. Segmentation of eye-regions, however, in most images has been performed successfully by the traditional thresholding method. The hair region in human face can also be easily segmented by thresholding technique. Segmentation of the mouth and the eye regions is required for subsequent determination of mouthopening and eye-opening respectively. Segmentation of the eyebrow region is equally useful to determine the length of eyebrow-constriction. Details of the segmentation techniques of different regions are presented below. 5.2.1 Segmentation of the Mouth Region Before segmenting the mouth region, we first represent the image in the L*a*b space from its conventional RGB space. The L*a*b system has the additional benefit of representing a perceptually uniform color space. It defines a uniform matrix space representation of color so that a perceptual color difference is represented by the Euclidean distance. The color information, however, is not adequate to identify the lip region. The position information of pixels, along with their color together is a good feature to segment the lip region from the face. The fuzzy-C Means (FCM) Clustering algorithm that we would employ to detect human lip region is supplied with both color and pixel-position information of the image. The FCM clustering algorithm is a well-known technique for pattern classification. But its use in image segmentation in general and lip region segmentation in particular is a virgin area of research until this date. The FCM clustering algorithm is available in any books on fuzzy pattern recognition [1], [9], [21]. In this chapter, we just demonstrate how to use the FCM clustering algorithm in the present application. A pixel in this chapter is denoted by 5 attributes: 3 color information and 2pixel position information. 3 color information are L*a*b and 2 pixel position information are (x, y). The objective of this clustering algorithm is to classify the above set of 5-dimensional data points into 2 classes/partitions namely the lip region and non-lip region. Initial membership values are assigned to each 5 dimensional pixel data, such that sum of membership in the lip and the non-lip region is equal to one. Mathematically for the kth pixel xk , μL (xk) + μNL (xk) = 1,

(5.1)

where μL(xk) and μNL(xk) denote the membership of xk to fall in the lip and the non-lip regions respectively. Given the initial membership value of μL (xk) and μNL (xk) for k = 1 to n2 (assuming that the image is of n × n size). With these initial memberships for all n2 pixels, we following the FCM algorithm first determine the cluster center of the lip and the non-lip regions VL and VNL respectively by the following formulae.

5.2 Filtering, Segmentation and Localization of Facial Components

and

137

n2 n2 m VL = Σ [ μL (xK)] xK / Σ [μL (xk)]m k=1 k=1

(5.2)

n2 n2 m VNL = Σ [ μNL (xK)] xK ] / Σ [μNL (xk)]m k=1 k=1

(5.3)

Expressions (5.2) and (5.3) given above provide centroidal measure of the lip and non-lip clusters, evaluated over all data points xk for k=1 to n2. The parameter m (>1) is any real number that influences the membership grade. The membership value of pixel xk in the image to fall both in the lip and the non-lip region is evaluated by the following formulae:

and

μL (xK) =

2 -1 2 2 1/ (m-1) , Σ {||xk – vL || / || xk – vj || } j=1

(5.4)

μNL (xK) =

2 Σ {||xk – vNL ||2 / || xk – vj ||2} 1/ (m-1) j=1

(5.5)

-1 ,

where Vj denotes the j-th cluster center for j∈{L, NL}. Determination of the cluster centers by expression (5.2) and (5.3) and membership evaluation by (5.4) and (5.5) are repeated several times following FCM algorithm until the position of the cluster centers do not change further. Fig. 5.1 presents a section of a facial image with a large mouth opening. This image is passed through a median filter and the resulting image is shown in Fig. 5.2. Application of FCM algorithm on Fig. 5.2 yields Fig. 5.3. In Fig. 5.4, we demonstrate the computation of mouth opening, details of which is given in Section 5.3. 5.2.2 Segmentation of the Eye-Region The eye-region in a monochrome image has a sharp contrast with respect to the rest of the face. Consequently, the thresholding method can be employed to segment the eye-region from the image. Images grabbed at poor illumination conditions have a very low average intensity value. Segmentation of eye region in these images is difficult because of the presence of dark eyebrows in the neighborhood of the eye region. To overcome this problem, we consider the images grabbed under good illuminating condition. After segmentation in the image is over, we need to localize the left and the right eyes on the image. In this chapter, we use template-matching scheme to localize the eyes. The eye template we used looks like Fig.5.5. The template-matching scheme we used here is taken from our

138

5 Fuzzy Models for Facial Expression-Based Emotion Recognition and Control

Fig. 5.1. The original face image.

Fig. 5.2. The median-filtered image.

previous works [2], [13]. It attempts to minimize the Euclidean distance between a fuzzy descriptor of the template with the fuzzy descriptor of the part of the image where the template is located. It needs mention here that even when the template is not a part of the image, the nearest matched location of the template in the image can be traced.

Fig. 5.3. The image after applying Fuzzy C-Means Clustering algorithm.

Fig. 5.4. Measurement of mouth opening from the dips in average intensity plot.

Y Fig. 5.5. A synthetic eye template.

5.3 Determination of Facial Attributes

139

5.2.3 Segmentation of the Eyebrow Constriction In a facial image, eyebrows are the 2nd dark region after the hair region. The hair region is easily segmented by setting a very low threshold in the histogram-based thresholding algorithm. The eye regions are also segmented by the method discussed earlier. Naturally, a search for a dark narrow template can easily localize the eyebrows. It is indeed important to note that localization of eyebrows is essential for determining its length. This will be undertaken in the next section.

5.3 Determination of Facial Attributes In this section, we present a scheme for measurements of facial extracts such as mouth-opening (MO), eye-opening (EO) and the length of eyebrow-constriction (EBC). 5.3.1 Determination of the Mouth-Opening After segmentation of the mouth region, we plot the average intensity profile against the mouth opening. The dark region in the segmented image represents the lip profile, whereas the white regions embedded in the dark region denote the teeth region. Noisy images, however, may include false white patches. Fig. 5.4, for instance includes a white patch on the lip region. Determination of MO in a black and white image becomes easier because of the presence of the white teeth. A plot of average intensity profile against the MO reveals that the curve may have several minima, out of which the first and the third correspond to the inner region of the top lip and the inner region of the bottom lip respectively. The difference of the above two measurements along the Y-axis gives a measure of the MO. An experimental instance of MO is shown in Fig. 5.4. In this Fig., the pixel count between the thick horizontal lines gives a measure of MO. When no white band is detected in the mouth region, MO is set to zero. When only two minima are observed in the plot of average intensity, the gap between the two minima is the measure of MO. 5.3.2 Determination of the Eye-Opening After the localization of the eyes, the count of dark pixels (having intensity <30) plus the count of the white pixels (having intensity> 225) is plotted against the xposition. Suppose that the peak of this plot occurs at x= a. Then the ordinate at x= a gives a measure of the eye-opening (Fig.5.6). 5.3.3 Determination of the Length of Eyebrow-Constriction Constriction in the forehead region can be explained as a collection of white and dark patches called hilly and valley regions respectively. The valley regions usually are darker than the hilly regions. Usually the width of the patches is around 10-15 pixels for a given facial image of (512 × 512) pixels.

5 Fuzzy Models for Facial Expression-Based Emotion Recognition and Control

White pixel count + dark pixel count →

White pixel count having intensity > 225 →

Dark pixel count having intensity < 30 →

140

x-position → Fig. 5.6. Determination of the eye-opening.

Let Iav be the average intensity in a selected rectangular profile on the forehead and Iij be the intensity of pixel (i, j). To determine the length of eyebrow constriction on the forehead region, we scan for variation in intensity along the x-axis of the selected rectangular region. The maximum x-width that includes variation in intensity is defined as the length of eyebrow-constriction. The length of the eyebrow-constriction has been measured in Fig. 5.7 by using the above principle

5.4 Fuzzy Relational Model for Emotion Detection

141

An algorithm for eyebrow-constriction is presented below. Main steps: 1.

Take a narrow strip over the eyebrow region with thickness two-third of the width of the forehead, determined by the maximum count of pixels along the length of projections from the hairline edge to the top edges of the eyebrows. 2. The length l of the strip is determined by identifying its intersection with the hair regions at both ends. Determine the center of the strip, and select a window of x-length 2l/ 3 symmetric with respect to the center. 3. For x-positions central to window-right-end do a) Select 9 vertical lines in the window and compute average intensity on each line; b) Take variance of the 9 average intensity values; c) If the variance is below a threshold, stop; Else shift one pixel right; 4. Determine the total right shift. 5. Similar to step 3, determine the total left shift. 6. Compute length of eyebrow-constriction = total left shift + total right shift.

5.4 Fuzzy Relational Model for Emotion Detection Fuzzification of facial attributes and their mapping to the emotion space is discussed here using Mamdani-type implication relation. 5.4.1 Fuzzification of Facial Attributes The measurement we obtain about MO, EO and EBC are fuzzified into 3 distinct fuzzy sets: HIGH, LOW, MODERATE. Typical membership functions [16] that we have used in our simulation are presented below. For any real feature x μHIGH(x) = 1- exp (- a x), a>0, μLOW(x) = exp(- b x), b>0, μMODERATE (x) = exp [- (x – xmean)2 / 2σ 2] where xmean and σ2 are the mean and variance of the parameter x. For the best performance we need to determine the optimal values of a, b and σ. Details of these will be discussed in Section 5.4. 5.4.2 The Fuzzy Relational Model for Emotion Detection Examination of a large facial database reveals that the degree of a specific human emotion, such as happiness or anger depends greatly on the degree of mouthopening, eye-opening and the length of eyebrow-constriction. The following two

142

5 Fuzzy Models for Facial Expression-Based Emotion Recognition and Control

y

x

Fig. 5.7. Determination of eyebrow constriction in the selected rectangular patch, identified by image segmentation.

sample rules give an insight to the problem of mapping from the fuzzified measurement space of facial extracts to the fuzzified emotion space. Rule 1: IF (eye-opening is MODERATE) & (mouth-opening is SMALL) & (eyebrow-constriction is LARGE) THEN emotion is VERY-MUCH-DISGUSTED. Rule 2: IF (eye-opening is LARGE) & (mouth-opening is SMALL/ MODERATE) & (eyebrow-constriction is SMALL) THEN emotion is VERY-MUCH-HAPPY. Since each rule contains antecedent clauses of 3 fuzzy variables, their conjunctive effect is taken into account to determine the fuzzy relational matrix. The general formulation of a production rule with an antecedent clause of three linguistic variables and one consequent clause of a single linguistic variable is discussed below. Consider for instance a fuzzy rule: If x is A and y is B and z is C Then w is D. Let μA(x), μB(y), μC (z) and μD (w) be the membership distribution of linguistic variables x, y, z, w belonging to A, B, C and D respectively. Then the membership distribution of the clause “x is A and y is B and z is C” is given by t(μA(x), μB(y), μC(z)) where t denotes the fuzzy t-norm operator [16]. Using Mamdani type implication operator, the relation between the antecedent and consequent clauses for the given rule is described by μR (x, y, z; w) = Min [t (μA (x), μB (y), μC (z)), μD (w)] (5.6) Taking Min as the t-norm, the above expression can be rewritten as μR (x, y, z; w) = Min [Min (μA (x), μB(y), μC (z)), μD (w)] = Min [μA (x), μB(y), μC (z), μD (w)].

(5.7)

5.4 Fuzzy Relational Model for Emotion Detection

143

Now given an unknown distribution of (μA/(x), μB/(y), μC/ (z)) where A/ ≈A, B ≈B and C/ ≈C, we can evaluate μD/ (w) by the following fuzzy relational equation: /

μD/ (w) = Min [(μA/ (x), μB/(y), μC/ (z)] o μR (x, y, z; w).

(5.8)

For discrete systems, the relation μR (x, y, z; w) is represented by a matrix (Fig. 5.8), where xi, yi and zi and wi denote the specific instances of the variables x, y, z and w respectively. In our proposed application, the row index of the relational matrix is represented by conjunctive sets of values of mouth-opening, eye-opening and eyebrowconstriction. The column index of the relational matrix denotes the possible values of six emotions: anxiety, disgusting, fear, happiness, sadness, and surprised. For determining the emotion of a person, we define two vectors namely fuzzy descriptor vector F and emotion vector M. The structural forms of the 2 vectors are given below. F=

[μS (eo) μM (eo) μL(eo) μS (mo) μM (mo) μL (mo) μL(ebc) ]

μS(ebc)

μM(ebc) (5.9)

where suffices S, M and L stand for SMALL, MEDIUM and LARGE respectively. j

w→

w1

μR (x, y, z; w) =

x1,y1,z1 x2,y2,z2 i . . .

wn

w2

rij

xn, yn,zn Fig. 5.8. Structure of a fuzzy relational matrix.

Emotion vector M = [μVA (emotion) μMA (emotion) μN-So-A (emotion) μVD( emotion) μMD (emotion) μN-So-D (emotion) μVaf(emotion) μMaf(emotion) μN-So-af(emotion) μVH(emotion) μMH (emotion) μN-So-H(emotion) μVS(emotion) μMS(mod) μN-So-S(emotion) μVSr(emotion) μMSr(emotion) μN-So-Sr (emotion)] (5.10) where

144

5 Fuzzy Models for Facial Expression-Based Emotion Recognition and Control

V, M, N-SO in suffices denote VERY, MODERATELY and NOT-SO, and A, D, Af, H, S and Sr in suffices denote ANXIOUS, DISGUSTED, AFRAID, HAPPY SAD, and SURPRISED respectively. The relational equation used for the proposed system is given by M = F/ o RFM,

(5.11)

where RFM is the fuzzy relational matrix with the row and column indices as described above, and the i-th component of F/ vector is given by Min {μA/ (xi), μB/ (yi), μC/(zi)}, where the variables xi, yi, zi ∈{eo, mo, ebc}, and the fuzzy sets A/, B/,C/ ∈{S, M, L}are determined by the premise of the i-th fuzzy rule. Given an F/ vector and the relational matrix RFM, we can easily compute the fuzzified emotion vector M using the above relational equation. Finally, to determine the membership of an emotion from their three graded fuzzy memberships, we need to employ a pseudo-defuzzification scheme like (5.12). μHAPPY (emotion) = μVH (emotion).w1 + μMH (emotion).w2 + μN-SO-H (emotion).w3 (5.12) w1 + w2 + w3 where w1, w2, and w3 denote weights of the respective graded memberships, which in the present context have been set equal to 0.33 arbitrarily.

5.5 Experiments and Results The experiment is conducted in a laboratory environment, where illumination, sounds, and temperature are controlled to maintain uniformity in experimental conditions. Most of the subjects of the experiments are students, young faculties, and family members of the faculties. The experiment includes two sessions: a) presentation session, followed by a b) face-monitoring session. In the presentation session, audio-visual movie clips taken from commercial films are projected on a screen before individual subjects as a stimulus to excite their brain for arousal of emotion. A computer-controlled pan-tilt type high-resolution camera is used for online monitoring of facial expressions of the subjects in the next phase. The grabbed images about facial expressions are stored on the desktop computer for feature analysis in the subsequent phase. Prior experiments were undertaken over the last two years to identify the appropriate audio-visual movie clips that cause arousal of six different emotions: anxiety, disgusting, fear, happiness, sadness, and surprised.

5.5 Experiments and Results

145

A questionnaire was prepared to determine the consensus of the observers about the arousal of the first five emotions using a given set of audio-visual clips. It includes questions to a given observer on the percentage level of excitation of different emotions by a set of two hundred fifty audio-visual movie clips. The independent response of 50 observers from different students, faculties, nonteaching staff and family members of the staff of Jadavpur University were taken, and the results obtained from the questionnaire is summarized in the format of Table 5.1. It is apparent from Table 5.1 that row- sum assigned to the emotions by a subject should be hundred. Table 5.1. Assessment of the arousal potential of selected audio-visual movie clips in exciting different emotions Subjects used to assess the emotion aroused by the audio-visual clips

Title of audiovisual clip

Subject 1 Subject 2 ………

Clip 1 Clip1 …… …. Clip1

Subject 50 ……….. Subject 1 Subject 2 ………..

Subject 50 ………

Subject 1 Subject 2

Percentage of arousal of different emotions by a clip Anxiety

Happin-ess

Sadne -ss

Fea -r

Relaxation

0 0

0 0

80 75

0 0

0 0

20 25

0

0

78

0

0

22

Clip 2 Clip 2

0 0

82 80

0 0

9 12

9 8

0 0

…… …. Clip 2

0

84

0

10

6

0

78

10

0

0

12

0

80

16

0

0

4

0

84

8

0

0

8

0

…… …. Clip 250 Clip 250

Anger

………

Subject 50

…… …. Clip 250

146

5 Fuzzy Models for Facial Expression-Based Emotion Recognition and Control

Fig. 5.9. Movie clips containing four frames (row-wise) used to excite anxiety, disgust, fear, happiness, and sadness.

The arousal of surprise, however, needs prior background of the subjects about an object or scene, and arousal starts when the object/scene significantly differs from the expected one. The main difficulty to get someone surprised with a movie clip is that the clip should be long enough to prepare the background knowledge of the subject before to presenting a strange scene, causing his arousal. To eliminate possible error in the selection of stimulus due to background difference of the subjects, it is suggestive to employ alternative means, rather than selection of audiovisual movie clips, to cause arousal of surprise. Experiments undertaken reveal that an attempt to recognizing lost friends (usually schoolmates) from their current photographs causes arousal of surprise.

5.5 Experiments and Results

147

Fig. 5.10. Movie clips of a lady indicating arousal of anxiety, disgust, fear, happiness, and sadness using stimulator of Fig. 5.9.

To identify the right movie clip capable of exciting specific emotion, we now need to define a few parameters with respect to Table 5.1, which would clearly indicate a consensus of the observers about an arousal of the emotion. Let Oji, k=

percentage level of excitation of emotion j by an observer k using audio-visual clip i, Eji = Average percentage score of excitation assigned to emotion j by n-no. of observers using clip i, σji = standard deviation of the percentage score assigned to emotion j by all the subjects using clip i, n = total no. of observers, assessing emotion arousal.

148

5 Fuzzy Models for Facial Expression-Based Emotion Recognition and Control

Fig. 5.11. Movie clips of a man indicating arousal of anxiety, disgust, fear, happiness, and sadness using stimulator of Fig. 5.9.

Then Eji and σji are evaluated using the following expressions. n Eji= ∑Oji, k / n k=1 n σji = √[∑ (Oji, k - Eji)2/n] k=1

(5.13)

(5.14)

Now, to determine Ewi, i.e. the largest percentage average assigned to an emotion using audio-visual clip i, we compare Ejis, and identify Ewi, such that Ewi>= Eji for all j. Consequently, the emotion w, for which Ewi>= Eji for all j, is the most likely

5.5 Experiments and Results

149

Table 5.2. Theme of the movie clips causing stimulation of specific emotions Emotion stimulated by the movie clip Anxiety

Movie title

Theme of the movie clip

Mr. and Mrs. Iyer (Indian movie translated in English) Razz (Indian Hindi movie)

A scene of possible murder

Disgust

Volcano (American movie)

Fear

Chaddabeshi (Indian Bengali movie) Sarfarosh (Indian Hindi movie)

Happiness

A sudden shouting of a lady in the jungle at high pitch A girl surrounded by lava erupted from a volcano Funny actions and words

Sadness

Sudden killing of tourists

aroused emotion due to excitation by audio-visual clip i. The above process is then repeated for all the 250 audio-visual clips, used as stimulators for the proposed experiments. We next select only six audio-visual movies from the pool of 250 movie samples, such that the selected movies best excite six specific emotions. The selection was performed from the measure of average to standard deviation ratio for competitive audio-visual clips used for exciting the same emotion. The audio-visual clip for which the average to standard deviation ratio is the largest is considered to be the most significant sample to excite a desired emotion. We formalize this as follows. For a specific emotion m, we determine Emi/σmi where the audio-visual clip i best excites emotion m, i.e., Emi>= Eji, for any emotion j. Now, to identify the clip k that receives the best consensus from all the subjects, we evaluate Emi/σmi for all possible clips i that best excites emotion m. Let S be the set of possible samples, all of which best excite emotion m. Let k be a sample in S, such that Eki/σki >= Emi/σmi for i∈S. Then we consider the k-th audio-visual movie sample to be the right choice to excite emotion m. This process is repeated to identify the most significant audio-visual movie sample for the stimulation of individual emotions. Table 5.3. Summary of results of membership evaluation for the aroused the emotion of Madhumala, stimulated by the audio-visual clips given in Fig. 5.10 Audio

Emotion

Feature

visual

expected

extracted

clip

to

title

aroused

eo

mo

ebc

Low

Mod

Hi

Low

Mod

Hi

Low

Mod

Hi

Clip 1

Anxiety

6

2

20

0.8

0.6

0.2

0.9

0.6

0.1

0.1

0.5

0.7

Clip 2

Disgust

5

1

26

0.9

0.5

0.1

0.95

0.5

0.8

0.1

0.4

0.9

Clip 3

Fear

10

6

5

0.1

0.3

0.9

0.2

0.8

0.6

0.8

0.2

0.1

Clip 4

Happi-

8

11

1

0.1

0.4

0.8

0.1

0.7

0.9

0.9

0.3

0.09

3

4

2

0.9

0.4

0.1

0.7

0.8

0.2

0.9

0.4

0.09

be

Memb. for eo

Memb. for mo

Memb. for ebc

(in pixels)

ness Clip 5

Sadness

150

5 Fuzzy Models for Facial Expression-Based Emotion Recognition and Control

Table 5.4. Results of percentage classification of aroused emotions in six classes for adult males Aroused emotion obtained from facial expression Anger

Surprise

Happiness

Sadness

4.0

0.0

0.0

2.0

Anxiety Fear

Anger

88.5

5.5

Anxiety

4.0

88.2

3.8

0.0

0.0

4.0

Fear

0.0

1.0

88.4

0.0

0.0

10.6

Surprise

0.0

0.0

0.0

91.2

8.8

0.0

Happiness

0.0

0.0

0.0

9.8

90.2

0.0

Sadness

0.0

3.8

8.0

0.0

0.0

88.2

Table 5.5. Results of percentage classification of aroused emotions in six classes for adult females Aroused emotion obtained from facial expression Desired Emotion

Anger

Anxiety

Fear

Surprise

Happiness

Sadness

Anger

92.2

3.8

2.0

0.0

0.0

2.0

Anxiety

1.4

92.4

4.2

0.0

0.0

2.0

Fear

0.0

2.0

92.6

0.0

0.0

5.4

Surprise

0.0

0.0

0.0

92.2

7.8

0.0

Happiness

0.0

0.0

0.0

7.2

92.4

0.0

Sadness

0.0

2.0

5.4

0.0

0.0

92.6

Table 5.6. Results of percentage classification of aroused emotions in six classes for children in age group 8-12 years Aroused emotion obtained from facial expression Desired Emotion

Anger

Anxiety

Fear

Surprise

Happiness

Sadness

Anger

96.0

1.2

2.2

0.0

0.0

0.6

Anxiety

0.6

96.2

2.2

0.0

0.0

1.0

Fear

0.0

1.6

96.4

0.0

0.0

2.0

Surprise

0.0

0.0

0.0

96.5

3.5

0.0

Happiness

0.0

0.0

0.0

3.4

96.6

0.0

Sadness

0.0

1.8

2.2

0.0

0.0

96.0

5.5 Experiments and Results

eo

mo

a b m σ

Fuzzifier

a b m σ

Fuzzifier

151

ebc

a b m σ

Fuzzifier

The F/-vector

M ← F o RFM /

_ Learning Algorithm

∑∑ error

+

Desired emotion vector Fig. 5.12. Validation of the proposed system by tuning the parameters a, b, m, σ of the fuzzifiers.

Fig. 5.9 presents the five most significant audio-visual movie clips selected from the pool of 250 movies, where each clip obtained the highest consensus to excite one of five different emotions. Table 5.2 explains the theme of the selected audiovisual movies. The selected clips are presented before eight hundred people, and their facial expressions are recorded for the entire movie. The facial expressions of the subjects recorded for the respective stimulus of Fig. 5.9 are given in Fig. 5.10 and 5.11 for two people, one female and one male in the age group 22-25 years. Image segmentation algorithm is then invoked to segment the mouth region, eye region and eyebrow region of individual frames for each clip. Mouth-opening (mo), eye-opening (eo), and the length of eyebrow constriction (ebc) are then determined for individual frames of each clip. The average of eo, mo and ebc over all the frames under a recorded emotion clip is then evaluated. The membership

152

5 Fuzzy Models for Facial Expression-Based Emotion Recognition and Control

of eo, mo and ebc in three fuzzy sets: LOW, MEDIUM AND HIGH are then evaluated using the membership functions given in section IV. The results of membership evaluation for the emotion clips given in Fig.5.10 are presented in Table 5.3. After evaluation of memberships, as indicated in Table 5.3, we next determine emotion vector M by using (5.11), and then employ defuzzification rule (5.12) to determine the membership of different emotions for all the five clips in Fig. 5.10. The emotion that comes up with the highest value is regarded as the emotion of the individual clips. The above analysis is repeated for one thousand people including 200 children, 400 adult males and 400 adult females, and the results of emotion classification are given in Tables 5.4, 5.5 and 5.6. In these Tables, desired emotion refers to the emotion tag of the most significant audio-visual sample used to excite emotion of a subject. Aroused emotion obtained from facial expression in these Tables is divided into six columns to visualize the percentage of people excited with common emotion by individual audio-visual movies, responsible for arousal of the desired emotion. Experimental results obtained from Tables 5.4 - 5.6 reveal that that the accuracy in the classification of emotion for adult male, adult female and children between (8-12) years are 88.2%, 92.2% and 96% respectively. The classification accuracies obtained in this chapter are better in comparison to the existing results on accuracy reported elsewhere [11], [12], [14], [20].

5.6 Validation of the System Performance After a prototype design of an intelligent system is complete, we need to validate its performance. The term validation here refers to building the right system that truly resembles the system intended to build. In other words, validation refers to relative performance of the system, and suggests reformulation of the problem characteristic and concepts based on the deviation of its performance from that of the desired (ideal) system [10]. It has been observed experimentally that performance of the proposed system greatly depends on the parameters of the fuzzifiers. To determine optimal settings of the parameters, a scheme for validation of the system’s performance is proposed in Fig. 5.12. In Fig. 5.12, we tune parameter a, b, xmean and σ of the fuzzifiers by a supervised learning algorithm, so as to generate the desired emotion from the given measurements of the facial extract. The back-propagation algorithm has been employed to experimentally determine the parameters a, b, m, σ. The feed-forward neural network used for realization of the back-propagation algorithm has three layers with 26 neurons in the hidden layer. The number of neurons in the input and the output layers are determined by the dimension of F/ vector, and the M vector respectively. The root mean square error accuracy of the algorithm was set 0.001. For a sample space of approximately 100 known emotions of persons, the experiment is conducted and the following values of the parameters a = 2.2, b = 1.9, xmean = 2.0 and σ = 0.17 are found to yield the best results.

5.7 Proposed Model of Emotion Transition and Its Control

153

5.7 Proposed Model of Emotion Transition and Its Control A commonsense thinking reveals that the emotion of a person at a given time is determined by the current state of his/her mind. The current state of the human mind is mainly controlled by the positive and the negative influences of the input sensory data, including voice, video clips, ultrasonic signals and music on the human mind. We now propose a model of emotion transition dynamics, where the strength of an emotion at time t + 1 depends on the strength of all emotions of a subject at time t, and the positive/ negative influences that have been applied as stimulation of the subject at time t. 5.7.1 The Model Let mi (t) be a positive un-normalized singleton membership of the i-th emotion at time t ; [wij] be a weight matrix of dimension (n × n), where wij denotes a cognitive (memory-based) degree of transition from the i-th to the j-th emotional state, and is a signed finite real number; μPOS-IN (strengthk, t) denotes a fuzzy membership distribution of a input stimulus with strength k to act as a positive influence at time t; μNEG-IN (inputl, t) denotes a fuzzy membership distribution of a input stimulus with strength l to act as a negative influence at time t; bik denotes the weight representing the influence of a input with strength k on the i-th emotional state; and cil denotes the weight representing the influence of a input with strength l on the i-th emotional state. The un-normalized membership value of an emotional state i at time (t+1) now can be expressed as a function of un-normalized membership values of all possible emotional states j, and the membership distribution of the input positive and negative influences at time t. mi (t+1)=∑ wi,j . mj (t) + ∑ bi,k . μPOS-IN (strengthk, t) -∑ci,l . μNEG-IN (strengthl, t) ∀j ∀k ∀l (5.15) The first term in the right hand side of expression (5.15) is due to cognitive transition of emotional states, concerning human memory, while the second and the third term indicate the effect of external influence on the membership of the i-th emotional state. The weights wij of the cognitive memory here is considered timeinvariant. Since wij are time-invariants, controlling transition of emotional states can only be accomplished by μPOS-IN (strengthk, t), and μNEG-IN (strengthl, t). It is apparent from (5.15) that the first 2 terms in its R.H.S. have a positive sign, while the 3rd term has a negative sign. The positive sign indicates that with a

154

5 Fuzzy Models for Facial Expression-Based Emotion Recognition and Control

growth in mj and μPOS-IN (strengthk, t), mi (t+1) also increases. The negative sign in the 3rd term signifies that with a growth in μNEG-IN(strengthl, t), mi (t +1) decreases. A psychological analysis of human emotion transition dynamics reveals that people change their emotional state from happy to anxious, when they smell chances of losing something. Further, an anxious person becomes sad, when he discovers a loss. A sad person becomes disgusted, when he realizes that he is not responsible for his loss/failure. In other words, with increasing negative influence (neg), human emotion undergoes a transition in the following order: neg disgusted

neg sad

neg anxious

happy.

Alternatively, with increasing positive influence (pos), the human emotion has a gradual transition from disgusted state to happy state as follows: pos

pos

disgusted

sad

pos anxious

happy,

Combining the above two state transition schemes, we can generalize the emotion transitions by a graph (Fig.5.13). neg

neg

Sad

disgusted

pos

neg Happy

Anxious

pos

pos

Fig. 5.13. The proposed emotion transition graph.

Fig. 5.13 provides a clear picture of transition of emotions from a given state j to a next state i, where each state denotes a specific emotion. It is clear from this figure that with application of positive influence there is a transition of emotion from the disgusted state to the sad state or from the sad state to the anxious state or from the anxious state towards the happy state. An application of negative influence on the other hand attempts to control the emotion in the reverse direction. Let M = [mi] be the un-normalized membership vector of dimension (n × 1), whose i-th element denotes the un-normalized singleton membership of emotion i at time t; μ = μPOS-IN (strengthk, t) be the positive influence membership vector of dimension (m × 1) whose k-th component denotes the fuzzy membership of strength k of the input stimulus;

5.7 Proposed Model of Emotion Transition and Its Control

155

be the negative influence membership vector of μ/ = μNEG-IN (inputl, t) dimension (m × 1), whose l-th component denotes the fuzzy membership of strength l of the input stimulus; B = [bij] be a (n × m) companion matrix to μ vector; and C = [cij] be a (n × m) companion matrix to μ/ vector. Considering the emotion transition dynamics (5.15) for i=1 to n, we can represent the complete system of emotion transition in vector-matrix form: M (t+1) = W. M (t) + B μ - Cμ/.

(5.16a)

The weight matrix W in equation (5.16a) is given by From To

Happy

Anxious

Sad

Angry

Happy

W1, 1

W1, 2

W1, 3

W1, 4

Anxious

W2, 1

W2,2

W2, 3

W2, 4

Sad

W3, 1

W3, 2

W3, 3

W3, 4

Angry

W4, 1

W4, 2

W4, 3

W4, 4

W=

In order to keep the membership vector M normalized, we use the following scaling operation and thus obtain: n MS(t +1)= M(t +1)/ Max{mi(t+1)}. (5.16b) i=1 Normalization of memberships in [0, 1] is needed for convenience of interpretation, but is not directly related to the emotion control problem undertaken here. A control theoretic representation of emotion transition dynamics (5.16a) and (5.16b) is given in Fig. 5.14, where the system has a inherent delayed positive feedback along with provision for control with external input vectors μ and μ/. 5.7.2 Properties of the Model In an autonomous system, the system states change without application of any control inputs. The emotion transition dynamics can be compared with an autonomous system with a setting of μ = μ/ = 0. The limit cyclic behavior of an autonomous emotion transition dynamics is given in Theorem 1 below

156

5 Fuzzy Models for Facial Expression-Based Emotion Recognition and Control

Unit delay

+

∑

W M (t)

+

∑

+

M (t+1)

− B

μ

Scaling

MS(t+1)

C

Control inputs

μ/

Fig. 5.14. The emotion transition dynamics.

Theorem 5.1: The vector M (t+1) in an autonomous emotion transition dynamics with μ=μ / = 0, exhibits limit cyclic behavior after every k-iterations, if Wk =I . Proof: Since μ = μ/ = 0, we can rewrite expression (16a) by M(t+1) =W.M(t) Iterating t = 0 to (k – 1), we have M(1) =W.M(0) M(2) =W.M(1)= W2 M(0) : : M(k) = Wk M(0) Since the membership vector exhibits limit cyclic behavior after every kiterations, then M(k) = M (0) holds, which in turn requires Wk= I. Hence, the theorem holds. Theorem 5.1 indicates that without external perturbation, the cognitive memory helps in maintaining a recurrent relation in emotional states under a restrictive selection of memory weights, satisfying WK=I. For controlling system states in Fig. 5.13 towards happy state, we need to provide positive influences in the state diagram at any state. Similarly, for controlling state

5.7 Proposed Model of Emotion Transition and Its Control

157

transition towards disgusted state from any state s, we submit negative influence at state s. This, however, demands a pre-requisite of controllability of the membership state vector M. The controllability of a given state vector M to a desired state can be examined by Theorem 5.2. Theorem 5.2: The necessary and sufficient condition for the state-transition system to be controllable is that the controllability matrices: P= B WB W2B …..Wn-1B and Q= C WC W2C …..Wn-1C should have a rank equal to n. Proof: Proof of the theorem directly follows from the test criterion of controllability of linear systems [21]. It is apparent from Theorem 5.2 that for controlling emotion using positive external influence, we need to satisfy the first condition, i.e., P should have rank n, while to control emotion using negative external influence, we need to satisfy the second condition, i.e., Q should have rank n. In other words, we need to select W, B and C matrices judiciously, such that P and Q have rank n. 5.7.3 Emotion Control by Mamdani’s Model In a closed loop process control system [21], error is defined as the difference of the set-point (reference input) and the process response, and the task of a controller is to gradually reduce the error toward zero. When emotional state transition is regarded as a process, we define error as the qualitative difference of the desired and the current emotional state. In order to represent error in a quantitative sense, we attach a index to individual emotional states in Fig.5.13, such that when error is positive (negative), we can reduce error toward zero by applying positive (negative) influence. One possible indexing of emotional states that satisfies the above principle is given in Table 5.7. Table 5.7. The Emotion Index

Emotion Happy Anxious Sad Disgusted

Index 4 3 2 1

158

5 Fuzzy Models for Facial Expression-Based Emotion Recognition and Control

We now can quantify error by taking the difference of desired emotional index (DEI) and current emotional index (CEI). For example, if the desired emotion is happiness, and the current emotion is sadness, the error is e = DEI – CEI = 4 – 2 =2. Similarly, when desired emotion is disgust, and the current emotion is happiness, error e= DEI- CEI= 1- 4= -3. Naturally, for the generation of control signals, we need to consider both sign and magnitude of error. To eliminate the scope of entry of noise in controlling emotion, we, instead of directly using the signed errors, fuzzify magnitude of errors in four fuzzy sets: SMALL, MODERATE, LARGE, VERY-LARGE, and sign of errors in two fuzzy sets: POSITIVE and NEGATIVE, using nonlinear (Gaussian type) membership functions. The non-linearity of the membership functions eliminates small Gaussian noise with zero mean and small variance over the actual measurement of error. Further, to generate fuzzy control signals μ and μ /, we need to represent the strength of positive and negative influence in four fuzzy sets: SMALL, MODERATE, LARGE and VERY-LARGE as well. Tables 5.8 and 5.9 provide a complete list of membership functions used for fuzzy control, the corresponding plots of membership functions are given in Fig. 5.15 (a) through (c). Table 5.8. Membership functions of magnitude of error and unsigned strength of positive/negative influence

Signal SMALL Magnitude of error (|e|) Unsigned strength of influence (|s|)

exp(- k1. |e|) exp (-k1/ . |s|)

Membership Functions MODERATE LARGE exp(-k2.(|e| 1.5)2) exp(-k2/. (|s| 1.5)2)

-

1- exp(-k3. |e|) 1-exp(-k3/. |s|)

VERYLARGE 1-exp (-k4 .|e|) 1- exp(-k4/. |s|)

Table 5.9. Membership Functions of Sign of Error

Signal

Membership Functions POSITIVE

Error (e)

1 - exp(-k5. e), 0<e<3

NEGATIVE 1 - exp(-k5. |e|), -3<e<0

5.7 Proposed Model of Emotion Transition and Its Control

159

Table 5.10. Selected Parameters of the Membership Functions Parameter k1 0.15 Selected value

k2

k3

k4

k1/

k2/

k3/

k4/

k5

0.12

0.16

0.008

0.16

0.14

0.18

0.008

0.10

1.0

MOERATE

LARGE

SMALL VERY LARGE

Membership

0.5

0 | error | →

1.5

3

Fig. 5.15(a). Membership distribution of error in fuzzy set SMALL, MODERATE, LARGE and VERY LARGE.

1.0

MOERATE

LARGE

Membership

SMALL VERY LARGE 0.5

0

± 1.5

±3

Strength of positive/ negative influence

Fig. 5.15 (b). Membership distribution of the strength of positive/negative influence to be SMALL, MODERATE, LARE and VERY-LARGE.

160

5 Fuzzy Models for Facial Expression-Based Emotion Recognition and Control

μPos (error), μNeg

1.0

NEGATIVE

POSITIVE

↑ 0.5

-3

-2

-1

0

1

2

3

Error →

Fig. 5.15 (c). Membership distribution of error to be POSITIVE and NEAGTIVE.

Parameter selection of the membership functions here has been considered by trial and error method. To attain the best performance of the controller, we considered fifty single step control instances, and found that the following settings listed in Table 5.10 give the best performance for all the fifty instances Let AMPLITUDE and SIGN be two fuzzy universes of error, and STRENGTH be a fuzzy universe of positive/negative influences. Here, SMALL, MODERATE, LARGE and VERY_LARGE are fuzzy subsets of AMPLITUDE and STRENGTH, while POSITIVE and NEGATIVE are fuzzy subsets of the universe SIGN. Let x, y and z be fuzzy linguistic variables, where x and y denote error, while z denotes positive/negative influence. Let Ai and Ci be any singleton fuzzy subset of {SMALL, MODERATE, LARGE, VERY-LARGE} and Bi be a singleton subset of {POSITIVE, NEGATIVE}. The general form of the i-th fuzzy control rule then is written as follows. Rule Ri: If x is Ai and y is Bi Then z is Ci. Typical examples of the i-th rule are presented below for convenience. Rule 1: If error is SMALL & POSITIVE Then apply positive influence of SMALL STRENGTH. Rule 2: If error is MOERATE & POSITIVE Then apply positive influence of MOERATE STRENGTH.

5.7 Proposed Model of Emotion Transition and Its Control

161

Rule 3: If error is LARGE & POSITIVE Then apply positive influence of LARGE STRENGTH. Rule 4: If error is VERY LARGE & POSITIVE Then apply positive influence of VERY LARGE STRENGTH. Rule 5: If error is SMALL & NEGATIVE Then apply negative influence of SMALL STRENGTH. Rule 6: If error is MOERATE & NEGATIVE Then apply negative influence of MOERATE STRENGTH. Rule 7: If error is LARGE & NEGATIVE Then apply negative influence of LARGE STRENGTH Rule 8: If error is VERY LARGE & NEGATIVE Then apply negative influence of VERY LARGE STRENGTH. Mamdani type implication relation for Rulei is now given by μRi(x, y; z)= Min [Min ( μAi(x), μBi(y)), μCi(z)

(5.17)

Now, for a given distribution of μA/(x) and μB/(y), where A/≈Ai and B/≈Bi, we can evaluate μCi/(z)= Min ( μA/(x), μB/(y)) o μRi (x, y; z).

(5.18)

To take the aggregation of all the n number of rules, we determine: n μC/(z)= Max [μCi/(z)]

(5.19)

i=1

Example 1: In this example, we illustrate the construction of μRi (x, y; z) and then evaluation of μCi/(z). Given μPOSITIVE (error) = {1/0.2, 2/0.5, 3/0.6} μSMALL (error) = {0/0.9, 1/0.1, 2/0.01, 3/0.005} and μSMALL (pos-in) = {10/0.9, 30/0.6, 60/0.21, 90/0.01}. The relational matrix R(error, error; pos-in) now can be evaluated by Mamdani’s implication function as follows: Min [μPOSITIVE (error), μSMALL (error)] = {(1, 0)/ 0.2, (1, 1)/ 0.1, (1, 2)/ 0.01, (1, 3)/ 0.005; (2, 0)/ 0.5, (2, 1)/ 0.1, (2, 2)/ 0.01, (2, 3)/ 0.005; (3, 0)/ 0.6, (3, 1)/ 0.1, (3,2)/ 0.01, (3,3)/ 0.005}

162

5 Fuzzy Models for Facial Expression-Based Emotion Recognition and Control

Using expression (5.17), the R (error, error; pos-in) matrix is now obtained as pos- influence

e, e

10

30

60

90

1, 0

0.2

0.2

0.2

0.01

1, 1

0.1

0.1

0.1

0.01

1, 2

0.01

0.01

0.01

0.01

1,3

0.005

0.005

0.005

0.005

2, 0

0.5

0.5

0.2

0.01

2,1

0.1

0.1

0.1

0.01

2,2

0.01

0.01

0.01

0.01

2,3

0.005

0.005

0.005

0.005

3, 0

0.6

0.6

0.2

0.01

3, 1

0.1

0.1

0.1

0.01

3,2

0.01

0.01

0.01

0.01

3, 3

0.005

0.005

0.005

0.005

Let us now consider the observed membership distribution of error to be positive and small as follows. μPOSITIVE/ (error) = {1/0.1, 2/0.5, 3/0.7} and μSMALL/ (error) = {0/0.2, l/ 0.1, 2/0.4, 3/0.5} We can evaluate: μSMALL/ (pos-in) = Min [μPOSITIVE/ (error), μSMALL/ (error)] ο R (error, error; pos-in) =[0.1 0.1 0.1 0.1 0.2 0.1 0.4 0.5 0.2 0.1 0.4 0.5] o R (error, error; pos-in) =[10/0.2 30/0.2 60/ 0.2 90/ 0.01]. To determine the strength of audio-visual movies to be selected for presentation before the subject, we need to defuzzify the control signal μCi(pos-in) or μCi(negin). Let μC(pos-in)= (x1/ μC(x1), x2/μC(x2),…., xn/μC(xn)}. Then centroidal type defuzzification [21] yields the value of the control signal n n xdeffzy= ∑μC(xi). xi / ∑μC(xi). i=1 i=1 Example 2: Let μC(pos-in)= [10/0.2, 30/0.2, 60/0.2, 90/0.1]

5.7 Proposed Model of Emotion Transition and Its Control

163

The defuzzification of the control signal by the center of gravity method is obtained as xdeffzy = 10 × 0.2 + 30 × 0.2 + 60 × 0.2 + 90 × 0.1 0.2 + 0.2+ 0.2 + 0.1 =72.5. 5.7.4 Architecture of the Proposed Emotion Control Scheme A complete scheme of Mamdani-type emotion control is presented in Fig.5.16. Here, the scheme includes an emotion transition dynamics with provision for control input µ and µ/, and a fuzzy controller to generate the necessary control signals. The fuzzy controller compares the DEI and the CEI, and their difference is passed through AMPLITUDE and SIGN fuzzifier. A fuzzy relational approach to automatic reasoning introduced in section C is then employed to determine the membership distribution of µCi(pos-in) and µ/Ci(neg-in) using the i-th fired rule. The MAX units determine the maximum of µCi(pos-in) and µ/Ci(neg-in) corresponding to all the fired rules. These µ and µ/ are supplied as control input to the emotion transition dynamics. The generation of control signals µ and µ/ are continued until the next emotional index (NEI) is equal to the DEI. When NEI=DEI, both control switches S and S/, which were closed since start-up, are now opened, as further transition of emotion is no longer required. The defuzzification units are required to defuzzify the control signals µ and µ/. The defuzzification process yields the absolute strength of audio-visual movies in the range [-100, 100]. The defuzzification process thus selects the audio-visual movie of appropriate strength for presentation before real subjects. It is indeed important to note that there are two defuzzifiers in Fig.5.16. When error is positive, only defuuzifier1 is used. On the other hand, when error is negative, defuzzifier2 is used. 5.7.5 Experiments and Results The proposed architecture (Fig. 5.16) was studied on around eight hundred students/faculty and their family members of Jadavpur University. The experiment began with two hundred fifty audio-visual movies labeled with a positive/negative number in [-100, 100], representing strength of positive/negative external stimulus. The labeling was done by a group of 50 volunteers, who assigned a score between [-100, 100]. The average of the scores assigned to an audio-visual movie stimulus is used as its label. When the variance of the assigned scores for a movie is above a selected small threshold (≈1.8), we drop it from the list. In this manner, we selected only 201 movies, dropping 250 - 201= 49 movie samples. This signifies that scores obtained from 50 volunteers are close enough for the selected 201 audio-visual movies, indicating good accuracy of the stimulus.

164

5 Fuzzy Models for Facial Expression-Based Emotion Recognition and Control

Unit Delay

M(t)

+

+

∑ ∑

∑

W

M(t+1) MS(t+1)

−

Scaling

S/

S

Select audio-visual movie of computed strength

Defuzzifier2

Defuzzifier1

No

Close Sw.

e=0

Yes DEI

+

MAX

MAX

∑ −

μ (pos-in)

μ (neg-in)

Fuzzy Relational System

Amplitude and Sign Fuzzifier

∑ e

CEI

N NEI

+ −

Fig. 5.16. The complete scheme of Mamdani type emotion control.

Open Sw.

5.7 Proposed Model of Emotion Transition and Its Control

165

72 64

38

Time (in seconds) →

Fig. 5.17. Submission of audio-visual stimulus of strength 72, 64 and 38 for controlling emotion of a subject from the disgusted state (leftmost) to finally happy state through sad and anxious states in order.

Current emotion of a subject is detected by the emotion recognition scheme presented in previous sections. Desired emotion is selected randomly for a given subject. The control scheme outlined in section D is then invoked. When defuzzifier 1 or 2 (Fig. 5.16) generates a signed score, the nearest available average scored audio-visual movie is selected for presentation before the subject. It is indeed important to note that when error e is equal to ±m, (m<=3), we need to select a sequence of at least m number of audio-visual movies, until NEI becomes equal to DEI. An experimental instance of emotion control is given in Fig. 5.17, where the current emotion of the subject at time t=0 is disgusted, and the desired emotion is happiness. This requires three state transitions in emotions, which can be undertaken by presenting three audio-visual movies of strength +72 units, +64 units and +38 units respectively in succession before the subject. It is indeed important to note that here DEI=3 and CEI=0, so error e= DEI – CEI = 3 > 0. Since error is positive, we apply positive instance of suitable strength as decided by the fuzzy controller. In case the error was negative, the fuzzy controller would have selected audio-visual stimulus of negative strength. Two interesting and noteworthy points of the experiment include: i) good experimental accuracy, ii) repeatability. Experimental accuracy ensures that we could always control the error to zero for all the eight hundred subjects. The repeatability ensures that for the same subject, and same current and desired emotional state, the audio-visual movies selected are unique. Robustness of the control algorithm thus automatically is established because of its good accuracy and repeatability. In Fig. 5.17, respective widths of the control pulses are 3sec., 2 sec. and 2 sec. The justification of the results is given as follows. At time t=0, error was large positive. So, the control signal generated had large positive strength of long duration. Then with gradual decrease in error, the strength of the control signal and its duration decreased.

166

5 Fuzzy Models for Facial Expression-Based Emotion Recognition and Control

5.8 Conclusions The merits of the proposed scheme for emotion detection lie in segmentation of the mouth region by the FCM clustering and determination of the mouth-opening from the minima in average-pixel intensity plot of the mouth region. The proposed eye-opening determination also adds flavor to the emotion detection system. The fuzzy relational approach to emotion detection from facial feature space to emotion space has high classification accuracy of around 90%, which is better than any other reported results. Because of its good classification accuracy, the proposed emotion detection scheme is expected to have applications in the next generation human-machine interactive systems. Ranking of audio-visual stimulus considered in this chapter also gives a new direction to determine the best movies to excite specific emotions. Since benchmark for arousal of specific emotion remains a far cry until this date, the proposed ranking scheme of stimulators would help future researchers to select stimulus from a given audio-visual movie pool. Existing emotion detection methods hardly consider the effect of near-past stimulation on the current arousal of emotion. To overcome this problem, we submitted an audio-visual stimulus of relaxation before submission of any other movie clip to excite emotions. State transition in emotion by an audio-visual movie thus always occurs at relaxed state of mind, giving full effect of the current stimulus on the excitatory sub-system of the brain, causing arousal of the desired emotion with its full manifestation on the facial expression. Feature extraction from the face thus becomes easy when manifestation of facial expression truly resembles the aroused emotion. The most important aspect of the present work is the design of an emotion control scheme. The significance of the proposed control scheme lies in its good accuracy and repeatability. While accuracy ensures convergence of the control algorithm with a zero error, repeatability ensures right selection of the audiovisual stimulus. A composite feature of accuracy and repeatability indicates robustness of the control algorithm. The proposed scheme of emotion recognition and control can be applied in system design for two different problem domains. First, it can serve as an intelligent layer in the next generation human-machine interactive system. Such system would have extensive applications in the frontier technology of pervasive and ubiquitous computing [42]. Secondly, the emotion monitoring and control scheme would be useful for psychological counseling and therapeutic applications

Exercises 1. Table 5.10 presents a set of 15 2-dimensional points in the reference frame Xk1, Xk2. k

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

XK1

0

0

0

1

1

1

2

3

4

5

5

5

6

6

6

XK2

0

2

4

1

2

3

2

2

2

2

1

2

3

0

4

Exercises

167

Assuming that we have to partitions A1 and A2 where μA1 = { 0.854/ x1 μA2 = {0.146/x1

and a)

0.854/ x2 … .. 0.854/x14 0.146/x2 0.146/x14

1.0/ x15} 0/x15}

Determine the cluster centers for partitions A1 and A2 using Fuzzy CMeans Algorithm with M = 01.25

b) Also evaluate the membership of the data points to fall in class A1 and A2 respectively. 2. Would you expect any change in results in the computation of cluster centers when m =1 instead of 1. 25? How do you ensure fuzziness in the clustering when M >1? 3. In a median filter, the neighboring 8 pixels around a given pixel (x, y) in a digital image is placed in one dimensional array and sorted in ascending order. If the array has a length of 2M+1 units, then the (M + 1)th value is regarded as the median and the intensity of point (x, y) is replaced by the median. The process is continued for all pixels in the image excluding the boundary points. Use median filtering on any image of your known format based on the above algorithm. 4. A colour image is expressed in RGB, L* a* b* and HSV space co-ordinate systems. The quantitative expression for conversion from RGB space to XYZ, L*a*b* and HSV space are given below:

X Y

=

0.490

0.310

0.177

0.813

0.011

G

0.000

0.010

0.990

B

Z 1/3 L* = 25

100y Y0

1/3 *

a = 500

x x0

−

1/3 y y0

0.200

R

168

5 Fuzzy Models for Facial Expression-Based Emotion Recognition and Control

1/3 y

b* = 200

1/3 z

−

y0

z0

where x0 ,y0, z0 triplet denotes the reference white

−1

H1 = Cos

½ (R − G) + (R − B ) √ (R − G ) 2 + (R − B ) (G − B)

H = H1 if B ≤ G and H = 3600 − H1 if B > G. Max (R, G, B) − Min ( R, G, B) S= Max (R, G, B) Max (R, G, B) V= 255 a) Write a program to convert a colour facial image in XYZ L*a*b* and HSV space. You can use any file format you know BMP or TIFF. b) Show that human skin colour usually has a hue (h) around 30. Use this information to segment the skin part of a ace from a face with the head covered with hair. 5. In a template matching algorithm a given template of k × l is moved over a given image of m × n where m = c × k and n = c × l for a integer constant c. usually certain feature of the template are compared with the same feature of the part of the image over which the template resides. Given below the expressions or computing mean, variance, skewness, kurtosis and energy. Use these items as the feature of the template and Euclidean distance as an estimator o the error between the template and the part of the image covered by the template. L-1 Mean, Mi = ∑ bP(b) b=0 L-1

Variance, Vi = ∑ (b-Mi)2 P(b) b=0 L-1

Skewness, Ski = (1/Vi3) ∑ (b-Mi)3 P(b) b=0

Exercises

169

L-1

Kurtosis, Kui =(1/Vi4 ) ∑ (b-Mi)4 P(b) -3 b=0 L-1

Energy, Ei

= ∑ (P(b))2 b=0

a) Shift the template half of its width and for each such position determine the Euclidean distance between the template features and the feature of the image covered by the template. Repeat this process until the template reaches the last colour pointer. Then reposition the template in its initial position and shift it down by half of its width and again repeat the process discussed above. Repeat the last 2 steps until the template reaches the last right most corner position in the image. Construct an algorithm for template matching using the above idea. b) Design a program in C or any other language you know to identify an eye template in a facial image of a person. c) How many comparison do you require to determine the position of the t template in the image. 6. Table 5.2 presents a synthesis data of a human forehead with several constriction. Apply the algorithm presented in the text to determine the total length in number of pixels of eyebrow constriction on the image. Mouth opening of a person is determined and found to be 1.5 cm. Compute using the following expression the membership of the mouth opening to be HIGH, MODERATE and LOW. μHIGH (x) = 1- exp ( − 0.5x) μLOW(x) = exp ( 0.05x) μMODERATE (x) = exp

− (x- 1.2)2 2 (0.1)2 Table 5.11. Synthetic image data

222 222 222 224 222

222 222 222 224 222

160 162 163 159 160

160 162 163 159 160

158 158 158 157 158

158 158 158 157 158

160 162 163 159 160

160 162 163 159 160

222 222 222 224 222

222 222 222 224 222

170

5 Fuzzy Models for Facial Expression-Based Emotion Recognition and Control

7. Given the following 2 rules for determining the emotion of a person to be VERY-MUCH-SURPRISED and VEY-MUCH-HAPPY. Rule 1: IF (eye-opening is LARGE) & (mouth-opening is LARGE) & (eyebrow-constriction is SMALL) THEN emotion is VERY-MUCH-SURPRISED. Rule 2: IF (eye-opening is LARGE) & (mouth-opening is SMALL/ MODERATE) & (eyebrow-constriction is SMALL) THEN emotion is VERY-MUCH-HAPPY Using the membership function listed in problem 6 and the following related matrix, determine the membership of emotion to be very surprised and very happy. Emotion →

Facial features ↓ eo, mo, ebc

VERY MUCH SURPRISED

L, L,

S

L, S,

S

VERY MUCH HAPPY

0.9

0.1

0.1

0.9

Assume that the mouth opening, eye-opening and large eyebrow contrition obtained from facial image are 2cm, 1.2cm and 1.5cm respectively 8. Determine the rank of the following matrix A. 1 2 4 1

2 3 4 2

6 9 12 6

1 8 2 0

9. For the given differential equation dX/dt= AX + Bu where A and B given by 0 A = 4.4537 0

and B =

1 0 0

0 − 0.3947 0 0.9211

0 0 0

0 0 0

References

171

Determine the controllability of the system. [Hints: Compute W where W = B AB

A2B

A3B

and show that W has

a full rank of 4. Thus the system is completely controllable.

References 1. Bezdek, J.C.: Fuzzy Mathematics in Pattern Classification, Ph.D. Thesis, Applied Mathematics Center. Cornell University, Ithaca (1973) 2. Biswas, B., Mukherjee, A.K.: Template Matching with Fuzzy Descriptors. J. of Inst. of Engineers (1997) 3. Black, M.T., Yacoob, Y.: Recognizing facial expressions in image sequences using local parameterized models of image motion. Int. J. Com. Vis. 25, 23–48 (1997) 4. Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C.M., Kazemzadeh, A., Lee, S., Neumann, U.: Analysis of emotion recognition using facial expressions, speech and multimodal information. In: Proc. of ICMI 2004, Pennsylvania (October 2004) 5. Cohen, I., Garg, A., Huang, T.S.: Emotion recognition using multilevel HMM. In: Proc. of the NIPS Workshop on Affective Computing, Colorado (2000) 6. Cohen, I.: Facial Expression Recognition from Video Sequences, MS Thesis, Univ. of Illinois at Urbana-Champaign, Dept. of Electrical Engg. (2000) 7. Conati, C., Zhou, X.: Modeling students’ emotions from cognitive appraisal in educational games. In: Proc. of the Sixth Int. Conf. On Intelligent Tutoring System, France (2002) 8. Donato, G., Bartlett, M.S., Hager, J.C., Ekman, P., Sejnowski, T.J.: Classifying facial actions. IEEE Trans. Pattern Anal. Machine Intell. 21, 974–989 (1999) 9. Ekman, P., Friesen, W.V.: Unmasking the Face: A Guide to Recognizing Emotions from Facial Clues. Prentice-Hall, New Jersey (1975) 10. Essa, I.A., Pentland, A.P.: Coding, analysis, interpretation and recognition of facial expressions. IEEE Trans. Pattern Anal. Machine Intell. 19, 757–763 (1997) 11. Fellenz, W.A., Taylor, J.G., Cowie, R., Douglas-Cowie, E., Piat, F., Kollias, S., Orovas, C., Apolloni, B.: On emotion recognition of faces and of speech using neural networks, fuzzy logic and the ASSESS Systems. In: Proc. of the IEEE -INNS-ENNS Int. Joint Conf. Neural Networks, p. 2093 (2000) 12. Fernandez-Dols, J.M., Wallbotl, H., Sanchez, F.: Emotion Category Accessibility and the Decoding of Emotion from Facial Expression and Context. Journal of Nonverbal Behavior 15 (1991) 13. Gao, Y., Leung, M.K.H., Hui, S.C., Tananda, M.W.: Facial expression recognition from line-based caricatures. IEEE Trans. Systems, Man and Cybernetics- Part A: Systems and Humans 33(3) (May 2003) 14. Gordon, R.N.: The Structure of Emotions: Investigations in Cognitive Philosophy. Cambridge Studies in Philosophy. Cambridge University Press, Cambridge (1990) 15. Izumitani, K., Mikami, T., Inoue, K.: A Model of Expression Grade for Face Graphs Using Fuzzy Integral. System and Control 28(10), 590–596 (1984) 16. Kawakami, F., Morishima, S., Yamada, H., Harashima, H.: Construction of 3-D Emotion Space Using Neural Network. In: Proc. of the 3rd International Conference on Fuzzy Logic, Neural Nets and Soft Computing, Iizuka, pp. 309–310 (1994)

172

5 Fuzzy Models for Facial Expression-Based Emotion Recognition and Control

17. Klir, G.J., Yuan, B.: Fuzzy Sets and Fuzzy Logic: Theory and Applications. PrenticeHall, New Jersey (1995) 18. Kobayashi, H., Hara, F.: The Recognition of basic Facial Expressions by neural network. Trans. on the society of Instrument and Control Engineers 29(1), 112–118 (1993) 19. Kobayashi, H., Hara, F.: Measurement of the Strength of Six Basic Facial Expressions by Neural Network. Trans. of the Japan Society of Mechanical Engineers (C) 59(567), 177–183 (1993) 20. Kobayashi, H., Hara, F.: Recognition of Mixed Facial Expressions by Neural Network, ibid., pp. 184–189 (1993) 21. Konar, A.: Computational Intelligence: Principles, Techniques and Applications. Springer, Heidelberg (2005) 22. Krammer, A.F., Sirevaag, E.J., Braune, R.: A psycho-physiological assessment of operator workload during simulated flight missions. Human Factors 29(2), 145–160 (1987) 23. Lanitis, A., Taylor, C.J., Cootes, T.F.: Automatic interpretation and coding of face images using flexible models. IEEE Trans. Pattern Anal. Machine Intell. 19, 743–756 (1997) 24. Li, H., Roivainan, P., Forchheimer, R.: 3D Motion estimation in model-based facial image coding. IEEE Trans. Pattern Anal. Machine Intell. 15, 545–555 (1993) 25. Li, X., Ji, Q.: Active affective state detection and user assistance with dynamic Bayesian networks. IEEE Trans. Systems, Man and Cybernetics-Part A: Systems and Humans 35(1) (January 2005) 26. Mase, K.: Recognition of facial expression from optical flow. Proc. IEICE Trans., Special Issue Coput. Vis. And Its Applications 74(10), 3474–3483 (1991) 27. Pantic, M., Rothkrantz, L.: Automatic analysis of facial expressions: State of the Art. IEEE Trans. Pattern Anal. Machine Intell. 22(2), 1424–1445 (2000) 28. Pedrycz, W., Gomide, F.: An Introduction to Fuzzy Sets: Analysis and Design. MIT Press, Massachusetts (1998) 29. Pentland, E.A.P.: Coding, analysis, interpretation and recognition of facial expressions. IEEE Trans. on Pattern Analysis and Machine Intelligence 19(7), 757–763 (1997) 30. Picard, R.: Affective Computing. MIT Press, Cambridge (1997) 31. Picard, R.W., Vyzas, E., Healy, J.: Toward machine emotional intelligence: analysis of affective psychological states. IEEE Trans. on Pattern Analysis and Machine Intelligence 23(10), 1175–1191 (2001) 32. Rani, P., Sarkar, N., Adams, J.: Anxiety-based affective communication for implicit human-machine interaction. Advanced Engineering, Informatics 21(3), 323–334 (2007) 33. Rani, P., Sarkar, N., Smith, C., Kirby, L.: Anxiety detecting robotic systems-towards implicit human-robot collaboration. Robotica 22(1), 83–93 (2004) 34. Rosenblum, M., Yacoob, Y., Davis, L.: Human expression recognition from motion using a radial basis function network architecture. IEEE Trans. Neural Networks 7, 1121–1138 (1996) 35. Scheirer, J., Fernadez, R., Klein, J., Picard, R.: Frustrating the user on purpose: a step toward building an affective computer. Interacting with Computers 14(2), 93–118 (2002) 36. Simon, H.: Motivational and Emotional Control of Cognition. In: Models of Thought, pp. 29–38. Yale University Press, New Haven (1979)

References

173

37. Terzopoulus, D., Waters, K.: Analysis and Synthesis of facial image sequences using physical and anatomical models. IEEE Trans. Pattern Anal. Machine Intell. 15, 569– 579 (1993) 38. Tian, Y., Kanade, T., Cohn, J.: Recognizing action units for facial expression analysis. IEEE Trans. Pattern Anal. Machine Intell. 23(2), 97–116 (2001) 39. Ueki, N., Morishima, S., Harashima, H.: Expression Analysis/Synthesis System Based on Emotion Space Constructed by Multilayered Neural Network. Systems and Computers in Japan 25(13) (1994) 40. Uwechue, O.A., Pandya, S.A.: Human Face Recognition Using Third-Order Synthetic Neural Networks. Kluwer Academic publishers, Boston (1997) 41. Vanger, P., Honlinger, R., Haykin, H.: Applications of Synergetic in Decoding Facial Expressions of Emotions. In: Proc. of Int. Workshop on Automatic face and Gesture recognition, Zurich, pp. 24–29 (1995) 42. Vasilakos, A., Pedrycz, W.: Ambient Intelligence, Wireless Networking and Ubiquitous Computing. Artech House, Norwood (June 2006) 43. Yacoob, Y., Davis, L.: Computing spatio-temporal representations of human faces. In: Proc. of Computer Vision and Pattern Recognition, pp. 70–75. IEEE Computer Society Conference, Los Alamitos (1994) 44. Yacoob, Y., Davis, L.: Recognizing human facial expression from long image sequences using optical flow. IEEE Trans. Pattern Anal. Machine Intell. 16, 636–642 (1996) 45. Yamada, H.: Visual Information for categorizing Facial expression of Emotion. Applied Cognitive Psychology 7, 257–270 (1993) 46. Zeng, Z., Fu, Y., Roisman, G.I., Wen, Z., Hu, Y., Huang, T.S.: Spontaneous emotional facial expression detection. J. of Multimedia 1(5) (August 2006)

4 Brain Imaging and Psycho-pathological Studies on Self-regulation of Emotion

Emotion regulation shares knowledge from several disciplines including psychology, neuroscience, philosophy and cybernetics. Plato, Aristotle and the Stoich emphasized relative importance of reasoning to emotion. Although there exist many differences among the contemporary philosophers, many of them had the beliefs that negative emotions are one of the primary cause of suffering and dysfunction of the people. Naturally, since the days of Aristotle, the importance of negative emotions was given a primary consideration. In recent times, various neurological and psychological findings opened up a new era on emotion regulation and control. This chapter provides the cutting edge information with regard to neurological basis of emotion regulation. Healthy individuals usually are capable of consciously and voluntarily changing their neural activity describing their emotional state and process. This chapter will emphasize the capability of human-consciousness to volitionally influence the electrical activity of our brain and modulate the impact of emotion on the neuro-endocrine immune network.

4.1 Introduction Chapter 1 outlined various causes for emotion arousal and its regulation. This chapter provides a detailed treatment on the control and regulation of emotion. It begins with anatomy and physiology of different brain structures and their role in arousal and regulation of emotions. Special emphasis is given to neuro-psychology and functional neuro-imaging studies of brain structures, especially the orbitofrontal cortex, the amygdale and the cingulated cortex. The role of individual structures in emotion regulation is undertaken next. In this study, we focus the readers’ attention in cingulated cortex, medial frontal cortex and the anterior cingulated cortex. Neural basis of voluntary self-regulation is considered next. The regulation of sexual arousals through fMRI studies is also undertaken. A. Chakraborty and A. Konar: Emotional Intelligence, SCI 234, pp. 93–132. © Springer-Verlag Berlin Heidelberg 2009 springerlink.com

94

4 Brain Imaging and Psycho-pathological Studies on Self-regulation of Emotion

The chapter further takes into account EEG conditioning and its role in emotion detection. It includes experimental studies for EEG conditioning to demonstrate that voluntary control of brain waves is not trivially mediated. Several experiments on operant conditioning neural activity are discussed, including pain conditioning in rats and depression of humans. The chapter has been divided into 8 sections. Section 4.2 provides a discussion on anatomy and physiology of brain structures in emotion modeling. We briefly outline the role of amygdale, orbitofrontal cortex and cingulated cortex in emotion regulation in the same section. The detailed discussion on the role of medial frontal cortex and anterior cingulated cortex in emotion regulation are introduced in sections 4.3 and 4.4 respectively. Experimental studies on voluntary selfregulation of emotion are included in section 4.5. The EEG conditioning in rats and humans are studied in section 4.6. The psychopathological issues of emotion dysregulation are considered in section 4.7. Conclusions are listed in section 4.8.

4.2 Emotion Processing by the Human Brain Human brain contains different structures responsible for arousal of different emotions. In this section, we briefly outline anatomy and psychology of different brain structures and their functional characteristics in processing and regulating emotions. 4.2.1 The Amygdale The amygdale is a heterogeneous structure consisting of at least 13 anatomically and functionally distinct sub-nuclei [2]. In addition to its complex internal structure, the amygdale has extensive external anatomical connections. These extrinsic connectivity provides the amygdale a potential to integrate sensory information and to influence autonomic and motor output systems. It has also evidence for direct sub-cortical projections from auditory and visual thalamic nuclei [70], [82], [85], [86]. Electrophysiological studies in animals indicate modality specific and multi modal cells in amygdale supporting that the amygdale is involved in cross modal integration of the biologically salient stimuli, such as the sight and taste of food [104], [105]. Evidence of amygdale in processing fear conditioned stimuli is apparent as abolishment of amygdale by lesions shows failure in handling fear [34], [65]. Amygdale also receives input from early sensory processing regions including occipito temporal area, insula and cortices. These feedback connections enable amygdale to modulate early sensory processing and thus enhancing neural representation of biologically significant stimuli [98], [100], [120]. 4.2.2 Animal Studies on Amygdale The fear conditioning characteristic of the amygdale has been reported by various researchers following lesions on monkies, rats and human. [73], [143]. Electrical

4.2 Emotion Processing by the Human Brain

95

stimulation of the amygdale consistency evokes fear responses on the above animals [57], [81]. The experimental research also demonstrates that in addition to fearful stimuli, the amygdale is also involved in learning threats and dangers in the environment [9], [83]. In fact, Le Doux has shown that destruction of the amygdale prevents the capability of learning temporal association between neutral and aversive stimuli [83]. The visual pathway from retina to amygdale has also been experimentally established through lesions study on animal [67], [82]. The different pathways from retina to amygdale are shown in Fig 4.1. Amygdale can generate automatic reflexive responses from visual and auditory stimulus [99], [120]. Inferior Temporal Cortex Visual cortical areas

Amygdale

Posterior Thalamus

Autonomic Reflexive Responses

Striate Cortex

Superior Colliculus

Retina

Visual stimulus Fig. 4.1. Different visual pathways from retina to Amygdala.

4.2.3 Fear and Threat Perception of the Amygdale Fear Conditioning and auditory tones have significant changes in the behavioral responses such as heart pulse rate and freezing. It has also been noted that amygdale plays a vital role in the manifestation of behavior on facial expressions. [41]. One interesting demonstration on emotional sensory processing by amygdale comes from neuro-physiological studies of patients with restricted bi-lateral amygdale damage. These patients exhibit a profound deficiency in the negotiation of fearful facial expression, but are capable to do other psychological task very well [1], [17], [18]. Several neuro-imaging studies reveal that the amygdale generates response to fearful faces, and the activation takes place in the left side of the amygdale [101], [12]. Functional neuro-imaging also reveals that amygdale responses to fearful

96

4 Brain Imaging and Psycho-pathological Studies on Self-regulation of Emotion

faces diminish with repeated presentation [12], [113], [114], [115], [146]. Studies in both monkies and humans confirm that the fear in facial expression is principally conveyed by eyes. Fearful eyes can generate responses in the right amygdale, superior collisculus and posterior thalamus. However, no additional responses have so far been revealed of fearful mouth, although the combination of fearful eyes and mouth has a significant effect in the left amygdale. Early functional imaging studies in fear conditioning reveal that learning regulated responses are observed in thalamus, anterior cingulated cortex, Orbitofrontal and sensory cortical areas but not in amygdale [47], [66]. Amygdale also has shown response to appetite related learning [69]. The fMRI studies reveal amygdale involvement in the modulation of memory by appetitive stimuli [57], [97]. Greenberg noted the role of amygdale in the reproductive and aggressive behavior of lizard [53]. 4.2.4 The Orbitofrontal Cortex (OFC) The cortex on the ventral (orbital) surface of the fontal lobe is highly developed in primates, and is comprised of several areas in the brain. The areas include area 11 rostrally, area 12 laterally, 13 caudally and area 14 medially [21], [27]. The OFC receives input from the magnocellular (medial) part of the mediodorsal nucleus, which in turn receives input from temporal lobe structures including the amygdale. Medial and lateral OFC receives input from anterior cingulated [20], [21]. OFC has outputs to basal ganglia, hypothalamus and brain stem. It also sends projections back to interior temporal cortex, amygdale and anterior cingulated [26]. The anatomical connectivity of OFC provides it the potential to integrate sensory information from diverse sources, modulate sensory and other cognitive processing, and finally influence motor and autonomic output responses. 4.2.5 Animal Lesions to Prefrontal OFC Animals with lesions to prefrontal OFC shows changed behavior towards aversive and appetitive stimuli. For example, lesioned monkies demonstrate less aggressive behavior towards humans. Lesioned snakes are less likely to refuse food [15], [16]. OFC lesions in animals have also proved disruption in learning behavior in the animals. Electrophysiological recording in monkies reveal that the cells in the OFC respond to reinforcing stimuli such as the sight, taste, and smell of food. [21], [138]. 4.2.6 Neuro-psychology and Functional Neuro-imaging Studies on OFC Behavior Experimental results confirm that damage to human OFC produces a clinical syndrome of irresponsibility, dis-inhibited behavior and childish fatuous effect [42]. The role of human OFC in emotional learning reversal has been investigated in an event-related fMRI study [100]. In an experiment conducted by them, they have shown pictures of 2 faces before the subjects. One of the faces [A, CS+] was

4.2 Emotion Processing by the Human Brain

97

followed by a loud aversive noise for 33% of the presentations, while the other face (B, CS-) was not followed by any noise for the rest of the presentations. This is termed as the initial phase. The initial phase was followed by three other phases. The first of these three phases includes a repeat phase, in which face A was paired with a noise, and B was free from noise (i.e., A, CS+, B, CS-). The second phase includes a reversal, in which B was followed by a noise, and A is free from noise (i.e., B, CS+, A, CS-). In the third phased two other faces C and D were shown, where C was paired with noise for 33% of the trials, and D was free from noise (i.e., C, CS+, D, CS-). During acquisition of conditioning, both in the initial and the repeat phase, face A (CS+) evoked increased responses in amygdale, whereas in the reversal phase, face A continued to elicit increased responses in the amygdale, while face B (new CS+, old CS-) elicited increased activity in orbito frontal cortex. The above experiment reveals that amygdale and orbito frontal cortex have distinct roles in emotional learning. Amygdale is crucial for acquisition and retention of emotional association [83], whereas OFC is involved in rapid and flexible modification of emotional responses. 4.2.7 The Insula Insula cortex is a multimodal sensory region with somatosensory, visual, gustatory and auditory afferents and reciprocal connections to amygdale, hypothalamus, cingulated gyrus and orbitofrontal cortex (OFC). Besides playing a significant role in interceptive representation and autonomic control [110], [132], the insula also acts as an agent in avoiding inhibitory behavior [7]. It has been experimentally shown that insula functions as a integration cortex that co-ordinates sensorimotor responses to unexpected stimuli [3]. 4.2.8 Experiment of Selective Lesion of Insula Cortex Patients with selective lesion of insula cortex exhibit impaired recognition and a disgustful experience to both facial and vocal expressions [17]. Functional neuroimaging studies on healthy subjects have reported increased insula responses to facial expressions of disgust [113], [114], [134]. Increased insula activity has also reported to develop sadness [79], fear [101], fear conditioning [13], [30], instructed fear [112], the experience of phobic symptoms [119], and the power of facial emotion categorization [50], [51]. It is indeed important to note that insula although has a functional specialization for disgust [18], it can participate in other emotion processing as well. According to Critchlay et al. [31], insula plays an important role in mediating the influence of peripheral autonomic arousal on consciously experienced emotional states. 4.2.9 The Anterior Cingulated The cingulate cortex is functionally heterogeneous, comprised of different subregions [14], [36], [140]. Based on its functional architecture, it can be divided

98

4 Brain Imaging and Psycho-pathological Studies on Self-regulation of Emotion

into two basic zones: i) cognitive zone, and ii) affective zone. The affective subdivision of anterior cingulate has connections to amygdale, orbitofrontal cortex, anterior insula and autonomic brainstem region [36]. The dorsal cognitive subdivision has anatomical connections with parietal cortex, posterior cingulated, supplementary motor area, and dorsolateral prefrontal cortex [36]. The anterior cingulated has a good anatomical connectivity to evaluate the behavioral relevance of stimuli and influence to autonomic and motor responses. 4.2.10 Emotion Monitoring by the Cingulated Cortex The cortex of the cingulated gyrus may be synonymously considered as the receptive organ for the experiencing of emotion [111]. Electrical stimulation of the anterior cingulated cortex in macaque monkeys changes the heart rate, blood pressure and respiration, and also participates in eliciting vocalizations and facial expressions [133]. Functional neuro-imaging data also reveal that the anterior cingulated cortex has a wide role in emotional processing. Anterior cingulate responses appear in the facial expressions of emotion including fear and anger [8], [51], [101]. On the other hand, anterior cingulated cortex responds to fear-conditioned stimuli [13], [74] [144], cardiovascular arousal [31], painful stimuli, pain intensity [13], pain affect [118] and pain expectation [128]. Anterior cingulated has a proven evidence in regulating mood states as well as emotional and social behavior. In mammals, anterior cingulated lesions are reported to disrupt social behaviors. Fig. 4.2 provides a simple scheme for controlling emotion and emotional expression. The error signal is evaluated to trigger a stimulation regulating system that fixes the stimulation input of the emotion regulatory system. The emotion regulatory system generates the control signal for the emotion –growing system, which finally generates the driving signal for the emotional expression generating system. The inner feedback adjusts the driving input of the emotion regulatory system, whereas the outer feedback controls the emotional expression.

+ Desired emotion

∑ -

+

Determine desired emotional expression

G1

∑ -

Change facial expression

Desired facial expression

G2

H1 Sense emotion from facial expression

Fig. 4.2. A composite feedback system to regulate emotion and emotional expression.

4.4 The Anterior Cingulate Cortex as a Self-regulatory Agent

99

4.3 The Role of Medial Frontal Cortex in Self-regulation of Emotion Luu and Tucker in a recent Survey [90], discussed the role of medial frontal cortex in the self-regulation of emotion. According to them, self-regulation refers to adjustment or adaptability of the parameters of an organism to meet the challenges of existence. In other words, they compared the self-regulation process of emotion with a cybernetic feedback system that autonomously adjusts its internal states to satisfy a predetermined control-objective. Self-regulation at a neuro-physiological level refers to maintaining our bodily equilibrium (homeostasis) of different processes, such as temperature, pH and oxygen tension [94]. In a homeostatic system, the response of the system is compared with a given set-point, and the error thus generated may be used to do the necessary corrective action, so as to return the response to its given set-point. Some researchers are of the opinion that homeostatic mechanism is unable to explain the observed phenomenon of self-regulation [145]. Recently, systems with varying set-points have been identified to have changing motivational states in the normal glands including hypothalamic-pituitary-adrenal system. McEwen calls the above system allostatic [94]. In general, allostasis refers to regulation of several variables including behavioral and psychological ones over a given duration in order to meet motive set-points, which are fixed dynamically. Allostasis can be compared with regulation of anticipatory type that allows an organism to provide the necessary regulatory actions in advance to overcome the predictive failures. Thus, allostasis includes learning and adaptation in the process of homoeostasis control. In this section, we review electro-physiological and hemodynamic models of self-regulation that involves both monitoring and generating corrective actions to prohibit unwanted growth in the emotional states. In the human brain, anterior cingulated cortex (ACC) plays an important role in the regulatory actions of our emotion. Electro-physiological responses provide evidence of automatically generating corrective actions of our brain, when it fails to meet up the targeted goal [117]. It has been noted that in human action monitoring, one particular electrophysiological signal called the error related negativity (ERN) is observed on a regular basis when one interpretation of the occurrence of the ERN at regular intervals is set to autonomously generate the control command to take necessary corrective actions. The ERN and related medial frontal negativities (MFNs) provide important clues to the cortico limbic mechanisms of allostatic cybernetic systems. Although ERN is localized to the anterior cingulate cortex, we can argue that this signal drives multiple components of a limbic network.

4.4 The Anterior Cingulate Cortex as a Self-regulatory Agent Patients with frontal lobe lesions are usually characterized as being changed, and they are often concerned about their mistakes in their daily life [127]. Apparently, frontal lobe patients appear unimpaired because lesions to frontal lobe do not affect the performance of the people in traditional examinations or intellectual

100

4 Brain Imaging and Psycho-pathological Studies on Self-regulation of Emotion

functions [43]. However, when these patients have to make their own choices and learn from their mistakes, they show striking failures. These people because of impairment in the frontal lobe cannot generate the error information for subsequent generation of regulatory actions. [75], [87]. Recent research on cingulate gyrus, which lies in each hemisphere, has the capability of controlling executive processes in the human brain. Early studies on ACC reveal that patients can generate verbs for a given noun or color of a word in the stroop interference task [46], [50]. The above phenomenon indicates that ACC is involved in executive attention, sometimes referred to as attention for action [116]. The findings on neuro-imaging studies are complemented from the experimental results showing that the patients with ACC lesions perform poorly on task requiring executive control. [68]. Recent theories of the ACC, however, have shifted away from the view that the ACC participates in strategic control. In fact, current researchers are of the view that ACC plays an evaluative role to detect when control signal is required to be generated [22], [92]. Ochseher et al. [109], McDonald and his colleagues noted through an fMRI study using the stroop interference task that the patients when instructed to have a set of works or the color of the words could easily differentiate the two types of instruction using their dorsolateral prefrontal cortex (DLPFC). The researchers found greater activity in the DLPFC in response to the instruction addressing the subjects to name the color of the word, which is apparent as recognizing color is more complex than recognizing words. Based on the results cited above, [92] they pointed out that ACC is involved in the evaluation when cognitive control is required, whereas the DLPFC is involved in the strategic control of executing the task. Falkenstein [44] and Gehring et al. [49] identified a response-locked component of an EEG signal, capable of differentiating error responses form the correct responses. Response-locked component is defined as the error negativity or error related negativity. Fig. 4.3 describes the waveforms for the correct and the error response from an electrode positioned at FCz. Here, the ERN is defined as the peak of the differences approximately 80 ms after the button press. In early brain imaging studies, ERN was presumed to result from the automatic detection of a mismatch between the overt response and the outcome of the response selection response [44]. It is noteworthy that the detection of an error does not depend on sensory information as its onset is coincident with movement. On the contrary, the detection occurs through corollary discharge mechanism related to the execution involved with movement. Experimental studies reveal that medial frontal negativity was located to determine the evaluative feedback (i.e., feedback indicating correctness of responses) to compare with the ERN on the same subject. In this study, subjects were presented with a feedback signal (measured through better grading) about their performance. The feedback provided the subjects’ information about the speed of their response and its correctness. The process was repeated 5 times, and the delay of 5 trials was used to separate the immediate control signal generated from the

4.5 Voluntary Self-regulation of Emotion

Fcz

101

Correct

8

Error

4

-

-104

88

184

-4 -8

Fig. 4.3. The waveforms for correct and error responses at electrode position FCz. The peak of the difference occurs approximately 80ms after the button press.

feedback from its affective value. It was noted that a MFN with a peak around 300 ms post-stimulus differentiated the feedbacks among the letters A, C and F. The MFN was found to be more negative for C and F feedback in comparison to that of A. Luu, et al. [91] took an attempt to compare the sources of the ERN and the MFN on the same group of subjects. It was noted that the stimulus, locked ERN (MFN) and the response locked ERN are not same, but they share an overlapping in the dorsal ACC/SMA (supplementary motor area) generation.

4.5 Voluntary Self-regulation of Emotion Beauregard et al. in a recent review [6] presented the neural basis of voluntary self-regulation of emotion. Various evidence from experimental lesions studies in animals and humans lead the researchers to postulate that prefrontal cortex, (PFC) and limbic and paralimbic structures play a vital role in regulating emotion in the fronto limbic circuit [88], [89], [102], [103], [139]. Davidson and his associates also concluded in the same direction, signifying that emotional regulation is normally accomplished by the prefrontal cortex, including the orbito frontal, the dorosolateral, the anterior cingulated cortex and the subcortical limbic structure, including the hypothalamus and the amygdala. Very recently Mario Beauregard took an attempt to experimentally verify the hypothesis proposed by Nauta [103], Tucker et al. [139] and Davidson [33]. They use fMRI to investigate the voluntary self-regulation of sexual arousal in male subjects and voluntary regulation of sadness in adults as well as in children. In this

102

4 Brain Imaging and Psycho-pathological Studies on Self-regulation of Emotion

section, we would discuss the details of these studies by Mario Beauregard and her research team. According to Beauregard et al. [6], emotional self-regulation refers to the heterogeneous set of cognitive processes by which emotions are regulated. The instance of self-regulation involves one-or-more components of emotion [55]. Such Changes have relevance with emotional dynamics [137], which describes the magnitude, rise time, duration and offset of responses in the various components of emotion [55]. Emotional self-regulation involves generating, maintaining, exciting or prohibiting negative/positive emotions [29], [80]. Davidson et al. [33] studied the effect of controlling negative emotion, such as sadness and fear, and concluded that the genesis of clinical depression and anxiety disorders are due to regulation of negative emotion. Impressive aggression and violence arise as a consequence of defective regulation of anger. The ability to self-regulate negative emotion can be practiced through modern psychotherapeutic approaches. 4.5.1 fMRI Studies on Voluntary Regulation of Sexual Arousals Mario Beauregard studied the voluntary regulation of sexual arousals of healthy male volunteers in the age group 20-42. These people were scanned during 2 experimental conditions i) sexual arousal condition ii) suppression condition. During the sexual arousal condition, the subjects were shown a series of erotic film excerpts and emotionally neutral film excerpts. They were instructed to react normally to both types of stimuli. In the suppression condition, the same groups were instructed to suppress their feelings aroused due to the erotic film excerpts. The 2 excerpts of film of a given duration lasting for 39 seconds and separated by a period of 15 seconds were presented before the audience. After the demonstration is over, subjects were asked to rate verbally their emotional reaction between 0 to 8, where 0 denotes absence of any emotional reaction and 8 denotes the strongest emotion ever felt in his lifetime. In suppression condition, most subjects reported to maintain sufficient distance from their induced sexual arousal by their erotic film. It was noted that the mean of the numerical value during sexual arousal condition was found to be 5, while the same in the suppression condition was obtained to be 2. The fMRI results reveal that during the sexual arousal condition significant conditioning took place in the right amygdale, right anterior temporal lobe and the hypothalamus. Naturally, these findings indicate that the amygdale plays a key role in the evolution of the emotional significance of stimuli. [78]. The hypothalamus is a pivotal brain structure that implicates the endocrine and automatic expression of emotion [63]. The anterior temporal lobe adds emotional color to subjective experience [95]. The suppression activity due to the volitional inhibitory action exerted by the subject to decrease their intensity of their sexual arousal was mainly associated with the right lateral prefrontal cortex (LPFC). It is important to note that the amygdale did not participate during the suppression condition of the subject.

4.5 Voluntary Self-regulation of Emotion

103

4.5.2 Voluntary Regulation of Sadness in Adults Levesque et al. [84] studied the scope of voluntary suppression of sadness with the help of an fMRI instrument. Levesque considered 20 healthy female volunteers in the age group between 20-30 to perform the experiment. The experiment was similar with the one discussed in the previous section, except films on sexual arousal. Film excerpts on sadness and emotionally neutral conditions were presented for 48 seconds instead of 39 seconds. It was noted that during the sad stimulation period, the subjects scored a sad mean condition of 5.15 with a standard deviation of 1.30, when the sadness was between 2 to 7. During the suppression phase, the mean obtained was 1.85 with a standard deviation of 1.42 for a range of 4 to 0. The fMRI results reveal that during the sad condition, significant bilateral loci of deviation were measured in the anterior temporal pole and the midbrain. Significant activation of loci was also noted in the right pre-frontal cortex and left amygdale and left insula. On the other hand, during the suppression condition, significant loci of activation were observed in the right OFC and the right LPFC. Clinical neuro-physiogical studies reveal that OFC provides an inhibitory control to protect goal-directed behavior from interference [23]. For instance, a damage to the OFC leads to a frontal lobe syndrome [131] or pseudo-pathetic syndrome, [135] which is characterized with impulsivity, emotional outburst, argumentativeness, physical and verbal aggressiveness, hyper-sexuality and distractibility, People with OFC lesions display inappropriate and childish humor. These individuals thus demonstrate abnormal autonomic responses to emotional elicitors. People with early OFC lesions manifest high level of aggression and anti-social behaviors. People with less-OFC lesions usually attempt to display moral reasoning and verbal responses to social situation. Unlike OFC, the ACC does not demonstrate any capability of exhibiting voluntary suppression of sad feelings. The LPFC, however, demonstrates a major role in regulating sad emotion and suppression. Such suppression may sometimes cause depression in patients. It is further noted that the pre-frontal cortex is significantly correlated with severity of depression in patients [93]. 4.5.3 Neural Circuitry Underlying Emotional Self-regulation The detailed pathway of signal flow in the brain to process emotional information is discussed in this section. It may be recapitulated that the subjects are initially given some instruction prior to experimentation with emotional self-regulation. The fMRI study reveals that the instructions are held up in the LPFC unit to select the appropriate cognitive operation to generate the desired outcome, such as suppression of emotion induced by film excerpts. The executive control command is usually sent to OFC, which attempts to suppress various dimension of the emotion. The OFC then transfers a message to the amygdale to note the changes in the interpretation of the emotional significance. The amygdale in return sends back a message about "cognitive reframing by virtue of the anatomic projection" [5] to

104

4 Brain Imaging and Psycho-pathological Studies on Self-regulation of Emotion

the OFC. The OFC then commands the ACC to generate the error signal by taking the difference of the desired goal and regulated feedback produced by the OFC. The ACC generates the ERN (error negativity) signal to control the emotional state of the organism with the help of the OFC. The OFC then commands the anterior temporal pole (ATP) to modify the emotional state based on the subject's experience. The OFC also informs the MPFC the changes related to individual’s emotional state based on his personality. The MPFC performs a meta-cognitive analysis about the subjective experience of emotion and provides a feedback to the OFC, to inform the LPFC whether the current emotional state has to be further modulated. A complete feedback circuit among the different modules of the neural structures is given in Fig. 4.4. Desired Goal

Change Emotional State

+

_ ∑

ATP

ACC

+

+ LPFC

ERN

∑

∑ _

+

OFC

Amygdala

Meta Cognitive Analysis MPFC Unit Gain System

Fig. 4.4. The control loop describing the feedbacks among the module within the neural structures.

The manifestation of the emotion regulation appears because of the transfer of the control signal from the ACC to hypothalamus, (Hypo), mid-brain (MID) and brain stem nuclei (BSN).

4.6 EEG Conditioning and Affective Disorders Functional magnetic resonance imaging has already been undertaken as an experimental means of emotion detection. Electroencephalogram (EEG) is an alternative form of brain imaging technique, which too is equally useful in the detection of emotion-arousals. In the last few sections, we introduced different

4.6 EEG Conditioning and Affective Disorders

105

methods to regulate emotions by our forceful will-power, biologically controlled by orbitofrontal cortex (OFC). But there are situations, when emotion regulation becomes impossible, even with several attempts to control it manually. This chapter undertakes experimental studies by EEG conditioning to demonstrate that voluntary control of brain waves are not trivially mediated. The term neurofeedback has been coined very recently to describe operant conditioning of neural activity. Currently, it is widely used in clinical contents. Operant-conditioning of action potentials from single cortical neuron [45] or conditioning of spontaneous EEG [72], are some of the common methods to study the emotional issues of people from neuro-feedback. But what exactly we mean by operant conditioning of neural events? It means that a subject is rewarded for voluntarily changing the future probability of the specific event. Usually, the time-locked derivatives, called event-related potentials (ERPs) are taken from animal brains [46] and/or human-scalps [126]. It may be mentioned here that the history of operant conditioning started with Skinner's work (1930), which is concerned with training a rat to press a bar. Anytime the rat presses the bar, the rat is positively reinforced or rewarded with a food pellet. Naturally, when the rat is hungry, it will learn to associate between response and reward, and it will eagerly press the bar. The operant conditioning experiments have also been undertaken on human subjects. To study this, we first need to determine the evoked potential under normal condition, and then determine the evoked potential for rewarding condition, i.e., the subject is told that he will be rewarded if he executes a specific activity. In some clinical situations, however, operant training may not come up with sufficient changes in evoked potential. A series of papers, summarized by Rosenfield [123], reports that trivial mediatation of operant controlled ERPs may not be successful on different occasions. 4.6.1 Pain Conditioning in Rats The evoked response of visual cortex is well known to the brain-researchers for quite a long time. But evoked response on pain conditioning at somatic sensory cortex (a terminus of major pain systems) is of recent relevance [124]. The neural pathway, which transmits orofacial and dental pain to the central nervous system, is well known to the brain-researchers. A high-level shock to this tract provides us pain. Rosenfield and Baehr [124] stimulated this path several hundred times, and noted the ERPs on the pain perception at the somatic sensory cortex. The stimulus was produced by a heater attached to the rat's face. When the heat is excessive, the rat started rubbing at the heater, and the heater was made off. The recording time from heat onset to the first face-rubbing is defined as the pain-tolerance index. One interesting observation that follows from this experiment was that all rats with histologically confirmed correct electrode placements were readily able to increase as well as decrease the amplitude of the selected ERP component.

106

4 Brain Imaging and Psycho-pathological Studies on Self-regulation of Emotion

4.6.2 Clinical Study of Depression Using EEG Henriques and Davidson [61], [62] demonstrated that expression was characterized by frontal-cortex asymmetry with reference to the EEG from normal (non–depressed) people. It has been experimentally observed that frontal cortex asymmetry attempts to produce either equal or more left cortical activation than the right cortical activation. Depression has correlation with reduced left cortical activation. Davidson’s model [32] confirmed that for positive emotion left cortical activation grows, whereas for negative withdrawals, right frontal cortex activation grows. With this knowledge, we can easily check whether or not the EEG asymmetry is operant- conditioned, and the emotional consequence, if any, is a clinical situation. Davidson in a recent paper [32] studied activation asymmetry between the left and the right frontal cortex by EEG analysis using alpha (8-12) Hz power. The asymmetry is described by two expressions A[1] and A[2], where, A[1] and A[2] denote asymmetry score correlate [122]. Defining R and L as the right and the left Alpha power or magnitude, we following Davidson, [32] define A[1] and A[2] as follows: A[1] = log (R) – log (L) R-L and

A[2] =

.

R+L Both the formulae may be used inter-changeably for assessment of Alpha asymmetry. The higher is the score, the greater is the relative ratio of right to left alpha. Naturally, the larger is the value in the right Alpha, the higher is the withdrawal of the emotional system from right frontal cortex. On the other hand, the lower the ratio of right to left activation, the greater is the activation in the left frontal cortex. Depressed people usually have negative emotions causing withdrawal of emotion from the right frontal cortex. This can be visualized in the Alpha asymmetry with a high score describing greater relative ratio of right to left alpha. Depression in a human patient over a long time may cause long-enduring emotional changes, causing specific change in the EEG pattern representing a psychological change in attitude. There is a controversy in the current literature on the above issue and no full proof psychological changes have been cited until this date [124], [125]. 4.6.3 EEG Analysis for Premenstrual Dysphoric Disorder Rosenfeld and Bachr [124] reported to have studied depression in women suffering from premenstrual dysphoric disorder (PMDD). This syndrome was first officially announced in DSM-IV (Diagnostic and Statistical manual of Mental Disorders, Fourth Edition, 1999). The PMDD symptom begins (7-10) days prior to

4.7 Emotion Dysregulation and Psycho-pathological Issues

107

menstruation, well known as the luteal phase of the menses. Women suffering from PMDD syndrome, exhibit alpha asymmetry, indicating that they are depressed. Naturally, by alpha-asymmetry test, we can discriminate women suffering from PMDD from normal women. All these experimental results confirm that the cortical activation asymmetry is not hardwired, but is a reasonable state index and may be altered under operant-control.

4.7 Emotion Dysregulation and Psycho-pathological Issues Thompson in early 90’s identified several dimensions as central to the emotion regulation process. He stressed the importance of neuro-physiological factors, attentional and attributional processes, copying resources and emotional demand of the environment. The neuro-psychological factors include both inhibitory and excitatory process with our nervous system that regulate emotion arousal and physiological reactivity. Experimental research reveals that a difference in the above items relates to social and personality functioning. For instance, children having a lower threshold in reactivity of their limbic system mediating fear and defense are more likely to exhibit behavioral inhibition and anxiety [71], [136]. Attentional processes can serve regulatory functions by managing emotionally arousing information through processes changing in developmental sophistication from visual disengagement to behavioral distraction and cognitive redirection. The visual disengagement means turning away our vision from external stimulation, whereas behavioral distraction refers to engaging ourselves in another activity to distract us from our current activity. Attributional processes sometimes involve alerting one’s interpretation of a situation in a way to facilitate emotional regulation. As an example, a child when left out of a game may interpret this as an oversight instead of an indication that the other children are not interested in his company. Access to copying resources also facilitates effective emotion regulation. Children and adult alike enhance interpersonal support and pragmatic assistance from family and friends to manage their emotional arousal. The management of emotional arousal in the present context refers to enhancement of positive emotion, decrease in distress and discomfort as well as maintenance of the strength of arousal. Finally, regulation of emotion depends on the emotional demand of familiar situation. It is important to note that regulatory processes serve important adaptive functions. But in occasions they may be disregulated, i.e., regulation is done at the cost of psychological adjustment. Shipman, Schndier and Brown in [5] consider the dysregulation of emotion and psychopathology of humans. They analyzed both non-clinical and clinical samples and ultimately came to similar conclusions. In this section, we briefly outline the dysregulation of emotion in childhood and clinical cases. 4.7.1 Emotional Dysregulation in Childhood from Non-clinical Samples The functionalist theory proposed by Barrett and Campos [4] and also by Thompson [136] provides a formal definition of emotion regulation. This definition

108

4 Brain Imaging and Psycho-pathological Studies on Self-regulation of Emotion

includes modulation of emotional expressivity, attention, copying behavior and physiological arousal. The difference in one or many of the above issues distinguishes children with behavioral problem from their peers. The current researchers on emotion of children reveals that children with internalizing or externalizing behavioral problems differ from non-disordered children in the frequency and intensity of their emotional arousal. Research findings indicate that children with internalizing problems are more likely to express sadness, [37], [38], [39] but less likely to express anger in comparison to other children [148]. In addition, children with internalizing problems report a higher incidence of emotional dysregulation, such as uncontrolled sobbing and slamming doors [148]. On the other hand, children with externalizing problem are related as more emotionally intense to express their angers inappropriately [28], [38]. Some researchers are of the opinion that children with externalizing problems express negative emotion, particularly anger very often than fear. The research with nonclinical samples also reveals that a low level attentional control is predictive of behavioral difficulties in early and middle childhood [35], [37]. It has been noted that attentional control can regulate both positive and negative emotions by increasing the focus of attention of the child to positive stimuli and by distracting the child from negative or threatening stimuli. Current research findings support the existence of negative relationship between attentional control and both externalizing and internalizing behavioral problems. Eisenberg [40] clearly pointed out that attentional control predicts behavioral problems primarily for children, who exhibit high levels of negative emotionality. Calkins et al. [19] in late 1990’s emphasized that children who focus on the source of their distress or use physical or emotional avoidance in an effort to escape distress, are more likely to manifest behavioral problems relating to their peers. In contrast, children who rely on distraction to regulate negative emotions are less likely to manifest behavioral problems [52]. Deficits in skills to emotional understanding also relate to children’s behavioral problems. Research findings suggest that children with externalizing behavioral problems are less accurate than non-disordered children at interpreting facial expression of others [77], [108], [147]. 4.7.2 Clinical Samples for Emotional Dysregulation for Children The psychopathological research on emotional dysregulation suggests that dysregulation is the central feature of many childhood disorders. Preliminary research investigation reports that children usually demonstrate a bias for negative emotion, particularly anger and hostility. This problem is well known as oppositional defiant disorder (ODD) [24], [25]. Casey experimentally found that children having ODD are found to be less attentive to positive social cues and express more hostility and negativity in inter-personal situations, when compared to non-disordered children. He also noted that children with ODD usually lack in emotional understanding, and they are unable to understand the cause of their emotional experience and fail to recognize their own and other’s emotional expression.

4.7 Emotion Dysregulation and Psycho-pathological Issues

109

Children with Attentional Deficiency with Hyper Activity Disorder (ADHD) also exhibit regulatory difficulties, especially in social information processing and also in encoding and decoding emotional expression [96]. Casey experimented with a group of students working in a laboratory session. He noted that the students with ADHD continued observing their partners’ behavior but fail to recognize their emotional expressions. Apparently, it seems that because of hyper activity, the children cannot focus their attention to a given issue, and consequently they fail to recognize fast changing emotions. Experimental investigation also reveals that depressed children are more sad and angry relating to their peers. [10], [11]. Children with depression are also less emotionally expressive than non-disordered children and may often fail to respond contingently during inter-personal interactions. It has also been observed that depressed children and adolescents use different strategies for regulating emotion, which are unusual to their non-depressed peers. Garber, and Brafladt [48] and Zeman [148] took a serious attempt to compare the emotion regulation strategies of depressed children with their non-depressed medically ill youths. They confirm that medically ill youths use distractions and problem solving strategies more often than their depressed partners. The depressed children on the other hand use avoidance and engage themselves in problematic behaviors to regulate their emotions. Research findings also indicated that in inter-personal situations, depressed children, as compared to their non-depressed peers, usually seek less support in achievement-oriented situations. Depressed boys usually respond with high levels of anger, whereas girls are reported to have less reliance on problem solving strategies. 4.7.3 Emotion Regulation in Adulthood This section presents both clinical and non-clinical methods of emotion regulation in adulthood. 4.7.3.1 Non-clinical Studies Research on emotional dysregulation in adulthood stresses the importance to copying styles and psychological adjustment. Two distinct types of coping are focused in the adult literature. These are approach coping and disengagement coping [64]. Approach coping refers to acknowledging and accepting negative thoughts and emotional experience, and involves problem-solving strategies to solve the problem. On the other hand, disengagement coping involves intentional efforts to suppress negative thoughts and emotions. Disengagement coping is usually of two different types: distraction and avoidance of coping. These two types have shown significant differences in the regulatory action of emotion. Distraction in particular helps in decreasing the frequency of intrinsic thoughts and thus is associated with a lower level of emotional distress [106] [107]. Avoidance to coping on the other hand helps in suppressing negative

110

4 Brain Imaging and Psycho-pathological Studies on Self-regulation of Emotion

thoughts, and thus it has an impact on the physical and psychological health of the subject. Naturally, people who want to avoid or suppress undesirable thoughts may cause a paradoxical increase in emotional distress [16]. 4.7.3.2 Clinical Studies Clinical studies show that emotion dysregulation to adult psychopathology mainly arises because of depression of the subjects. Such depression may sometimes cause personality disorders. Depressed adults demonstrate high levels of negative affect, particularly sadness, and low levels of positive affect, suggestive of regulation deficits in emotion [130], [141]. Current research is aimed at identifying emotion regulation strategies for depressed adults. This, however, requires a clear understanding of these strategies and their contribution to the maintenance of depressive symptoms [54], [56], [129]. Current researchers are of the views that rumination (i.e., dwelling on various aspects of emotional distress) and avoidance (suppression of negative thought) enhance the kind of depression leading to continued negative thoughts, depressed mood and overall distress [54]. On the other hand, distraction appears to decrease negative emotion and symptoms of depression [106]. Research result also reveals that depressed individuals use fewer effortful coping strategies, making it difficult for them to disrupt negative thoughts and enhance positive emotional experience. The efficiency in controlling regulation in instinctive responses to stress is not feasible by the depressed individual [59], [142]. Borderline Personality Disorder (BPD), which has the characteristic of intense level of emotion, causing social and behavioral instability is considered as a disorder of emotional dysregulation. Adults suffering from BPD demonstrate automatic or involuntary responses to emotional events and have a lower threshold for emotional reaction, when compared to normal individual. According to Linehan (vide [5]), the environment during the childhood of a person is responsible to the regulation disorder in their adulthood. An invalidating environment fails to develop tolerance level to emotional distress. Naturally, these people in adulthood cannot regulate their emotional expression because of a mild positive or negative feedback from the environment.

4.8 Conclusions The chapter presented a detailed overview on psycho-pathological studies on emotion regulation using fMRI and EEG brain imaging techniques. Both clinical and non-clinical studies undertaken by several research laboratories over the globe have been summarized with special emphasis in determining the role of different modules of humans/mammalian brains in regulating typical emotions. The origin of several psychological disorders due to malfunctioning of one or more modules of the brain is also narrated in detail. A control theoretic feedback mechanism that persists among the modules of the brain is re-discovered from the experimental results obtained by different research groups.

Exercises

111

Exercises 1.

Let G1 and G2 be two blocks in cascade (Fig. 4.5), the overall gain of the system is given by G = C (S)/R(S) = G1G2 .

G1

R(S)

G1

C(S)

(a)

G

R(S)

C(S)

(b) Fig. 4.5. (a) and (b) are equivalent when G = C(S) /R(S) = G1G2 . Also note that when there is a negative feedback as shown in Fig. 4.6, the overall gain C(S)/R(S) is given by

C(S)

R(S)

G(S) =

1 + G(S) H(S)

+

∑

R(S)

G(S)

C(S)

H(S) (a)

R(S)

G(S)

C(S)

1+ G(S) H (S) (b) Fig. 4.6. (a) and (b) are equivalent, when G/ = C(S)/R(S) = G(S) /[1+ G(S) H(S)].

112

4 Brain Imaging and Psycho-pathological Studies on Self-regulation of Emotion

Using the above formulas, determine the overall gain for the system, given in Fig. 4.7. +

+ R(s)

-

Σ

Σ

G2

G1

G3

C(s)

-

H3

Fig. 4.7. An illustrative system for computing overall gain.

[Hints: The small inner loop has the gain: G3 G/ =

1 + G3H3

The overall system has gain equal to G1 G2 G/

C(S) R(S)

=

1+ G1 G2 G/

The final result can be obtained by substituting the value of G/ in C(S)/R(S)]. 2.

A signal flow graph is a simple but elegant method for evaluating gain of a closed loop system. In this exercise, we attempt to draw signal flow graphs, corresponding to block diagrams using the following equivalence. (Fig. 4.8). R(S)

G1

G2

C(S)

(a)

R(S)

G1

G1 C(S) (b)

Fig. 4.8. The signal flow graphs (b) and (d) are equivalent to block diagrams (a) and (c) respectively.

Exercises

+

∑

R(S)

-

G1

113

C(S)

H1 (c) G1 R(S)

C(S)

- H1 (d) Fig. 4.8. (continued)

Construct a signal flow graph for the block diagram given in Fig. 4.7. [Hints: The signal flow graph corresponding to the block diagram is given in Fig. 4.9.]

G1

G2

G3

1

R(S)

C(S) -H3

-1 Fig. 4.9. The signal flow graph corresponding to Fig. 4.7.

3.

Masson in [76] proposed a gain formula by evaluating forward path gains, individual loop gains, sum of gain product of two non-touching loops, 3 nontouching loops etc., and the gain of the loops that do not touch the forward path to evaluate C(S)/R(S) is obtained by identifying the path from node R(S) to node C(S), without considering a node more than once. Thus forward path gain for C(S)/R(S) computation in Fig. 4.9 is G1 G2 G3 . Individual loop gains here are evaluated by identifying a closed directed path, taking each node once only. For example, loop gains here are

114

4 Brain Imaging and Psycho-pathological Studies on Self-regulation of Emotion

-G3H3 and

G1 G2 G3 (1) (-1) = -G1 G2 G3.

Since here two loops are touching at arc G3, there does not exist non-touching loops. Thus sum of gain-product of non-touching loops = 0. Further, there is no loop that does not touch the forward path. The general form of the gain formula is given by (Forward path gain) × (1 – gain of a loop not touching the forward path)

C(S) R(S)

= delta

where

delta = 1− ∑ individual loop gains + ∑ gain product of two non- touching loops − ∑ gain-product of three non-touching loops + ……. Now, with respect to Fig. 4.9, we obtain: Forward path gain = G1 G2 G3, Individual loop gains: - G3H3 and - G1 G2 G3. Gain product of two or more non-touching loops = 0, Gain of a loop, not touching any forward path = 0. Thus, according to the gain formula G1 G2 G3

C(S) R(S)

= 1- (- G3H3 - G1 G2 G3) G1 G2 G3

=

.

1 + G3H3 + G1 G2 G3 Consider the simplified scheme of regulation of emotion and emotional expression outlined in Fig. 4.2. Assume that desired emotional expression (D) be a step function u(t). Also assume that 1 G1 =

1+S 1

G2 = and

S

H1 = 1.

Exercises

115

Determine the overall closed loop gain of the complete system. What does the overall loop gain implicate to evaluate the emotional expression when the system is driven by a step input? [Hints: First, we compute the interior loop game, which is found to be G2 1 + G2

.

The overall loop game is then evaluated as G1 G2

. 1 + G2 + G1 G 2 H 1

]

The remaining computation is left for the students themselves. The overall loop gain, here, indicates how much changes in emotional expression can take place when the desired input has unit amplitude. For evaluation of the emotional expression, where the system is driven by a step input, we evaluate the overall loop gain C(S)/ D(S) and substitute 1 D(S) =

S and then evaluate the response C(t) by taking Laplace inverse of C(S)]. 4.

A person attempts to control his emotional expression by looking at the reaction of his fellow beings. Assuming that the reaction of his fellow being appears as an external negative simulation in Fig 4.10. Determine the transfer function C(s)/E(s), where C(s) denotes the emotional expression of the person and E(s) denotes the external stimulation. Assume that the desired emotion D(s) = 0. [Hints: For simplicity in representation, we draw a signal flow graph for the given system. We use Masson’s gain formula to solve the problem. Forward path gain = -G2G3G4 Individual loop gains are -G2G3H1 and – G1G2G3G4 Sum of gain products of two or more non-touching loops = 0.

116

4 Brain Imaging and Psycho-pathological Studies on Self-regulation of Emotion

E(s)

-1 1

G1

1

G2

G3

G4

C(s)

1

D(S) = 0 -H1 -1 Fig. 4.10. A signal flow graph indicating control of emotion.

Now evaluate, C(s) /E(s) = Forward path gain (1-sum of non-touching loop gain)/Delta, where sum of two non-touching loop gain= 0. Delta= 1 − G2G3H1 − G1 G2 G3 G4. The rest of the evaluation follows from substitution of the results.] 5. Express the emotional expression of a person as a function of desired expression and external stimulation following Fig.4.10. [Hints: The above computation can be performed on the equivalent signal flow graph for the given system. The signal flow graph for the given system is presented below: E(s)

-1 1

G1

1

G3

G2

G4

C(S)

D(S) = 0

1 C(S)

-H1 -1 We now evaluate C(s)/D(s) with E(s) = 0 and C(s)/E(s) with D(s) = 0

Exercises

117

Let C(S)

=

G5

D(S) E(S) =0 and C(S)

=

G6

E(S) D(S) =0 ∴C(s) = G5 D(s) + G6 E(s) where Delta = 1-sum of individual loop gain + sum of gain product of 2 nontouching loops = 1- (-G1G2G3G4 – G2G3H1) + 0 = 1+ G1G2G3G4 + G2G3H1. Since the toward path here touches both the loops, the non-touching loop gain in the numerator of C(s)/H(s) is zero. Thus,

C(s) E(s)

=

-G2G3G4 1+ G1G2G3G4 + G2G3H1

The negative result indicates that for unit increase in the response of one’s fellow beings, i.e. E(s), there is a decrease in emotional expression C(s). -G2G3G4 G6 =

1+ G1G2G3G4 + G2G3H1

and G1G2G3G4 G5 = 1+ G1G2G3G4 + G2G3H1 G1G2G3G4 D(s) – G2G3G4E(s) So,

C(s) =

. 1+ G1G2G3G4 + G2G3H1

118

6.

4 Brain Imaging and Psycho-pathological Studies on Self-regulation of Emotion

Davidson studied activation asymmetry between the left and right frontal cortex by EGG analysis using alpha (8-12 Hz power). The asymmetry is described by A[1] and A[2] as given below: A[1] = log(R) – log(L), and A[2] = (R − L)/(R + L) where R and L denote the right and the left alpha power. Show that if left and right alpha power are equal, the asymmetry A[1] and A[2] are both zero. Also show that when R >> L, A[1] and A[2] have large magnitude. [Hints: When R = L, A[1] and A[2] following the given expressions are evaluated to be 0. When R >> L A[1] = log(R/L), as R>>L and log (.) is a monotonically increasing function, therefore, A[1] increases with an increase in R/L. Similarly, when R>>L, A[2] also increases.

7.

Under what condition ∂A[1]

≥

∂(R/L)

∂A[2] ∂(R/L)

What does this imply? [Hints: A[1] = log (R) –log(L) = log(R/L) Therefore, ∂A[1] ∂(R/L) A[2] ∂A[2]

≥

L

1 (R/L)

=

R

= ( (R/L) –1)/((R/L) +1) ((R/L) +1) (1) - ((R/L) –1) (1) =

((R/L) +1)2

∂(R/L) 22 =

(R/L+1)2

Exercises

∂A[2]

∂A[1]

Now,

≥

∂(R/L) L

⇒

119

≥

∂(R/L) 22 (R/L+1)2

R L

Let x =

R 1

Then,

≥

x

22 (1+x)2

or, (1 + x)2 – 2x ≥ 0 or, (1 − x)2 – 2x ≥ 0 or, |1+x| ≥ 0 R where x =

.

L 8. Experimental evidences reveal that an amygdale response to fear conditioning and the effect of fear conditioning appears in the changes of heart pulse rate and facial expressions. Fig 4.11(a) & (b) describe the amygdale effect on fear conditioning. Assume that the gain of the amygdale in Fig 4.11(a) & (b) respectively are

R1 Fear conditioning

Amygdale M1(s)

C1 Heart pulse rate

(a) Left Amygdale M2(s)

R2 Fear conditioning

C2 Fearful facial expression (say, contraction of check Fearful facial expression region) (say, contraction of check region)

(b) Fig. 4.11. The fear-conditioning of the amygdale.

120

4 Brain Imaging and Psycho-pathological Studies on Self-regulation of Emotion

1 M1(s) = S + 0.1

1 and

M2(s) = S(S +0.1)

a) Given that fear threat appears as a unit impulse, determine the changes in heart pulse rate and facial expression vide Fig. 4.11(a) & (b). b) Also evaluate the response of amygdale when unit step threat is applied as input to Fig. 4.11(a) and (b). [Hints: The Laplace transform of an impulse δ(t) = 1.

1

∴C1(s) =

.R1(s) S + 0.1

1 =

. (as R1(s)= 1) S + 0.1

C1(t) = e-0.1t Similarly when R2 (s) = 1, C2(s) =

1

.

s(s +0.1) = 1/0.1[1/s – 1/(s+0.1)] C2(t) = 10L-1 (1/s) – 10L-1 (1/(s +0.1)) = 10(1-e-0.1t). The response of the amygdala to impulsive fear stimuli (Fig. 4.12 (a)) for the two systems given in Fig. 4.11 (a) and (b) are indicated in Fig. 4.12(b) and (c) respectively.

121

Fear Conditioning

Exercises

Time →

Heart pulse rate

(a)

Time →

Fearful facial expression

(b)

Time →

(c) Fig. 4.12. The input impulsive stimulus and the response of the amygdale to the fierce stimuli.

Part (a) of the answers ends here. For past (b), we need to consider R1(s) = R2(s) = 1/s and repeat the above procedure.] 9. Suppose we need to model repeated threats of unit amplitude and short duration τ. Let the threats occur at an interval of T seconds. Construct a mathematical function, considering threats as pulses. [Hints: The threats here has been modeled by a periodic train of pulses having duration τ and time period T, Fig. 4.13 presents a schematic view of the threat.

122

4 Brain Imaging and Psycho-pathological Studies on Self-regulation of Emotion

Threat ↑

T

→| τ

|← Time →

Fig. 4.13. The threats as a periodic train of pulses.

[Hints: First we consider a single occurrence of threat. This can be modeled by subtracting u(t-τ) from u(t). Defining the first threat by Th(1), we say Th(1) = u(t) – u(t−τ). Similarly, Th(2) = u(t-T) – u(t−T− τ) . . . Th(n) = u(t – (n −1)T) − u(t− (n −1)T – τ). Thus adding Th(1), Th(2),......., Th(n) we have temporal representation of threat, Th (say) Th = [u(t) – u(t−τ)] + [u(t−T) –u(t−T−τ)] +.......+ [u(t−(n−1)T)– u(t−(n−1)T−τ)] = [u(t) + u(t−T) + u (t−2T) + …… + u (t−(n−1)T)] – [u(t−τ) + u(t−T−τ) + u(t −2T−τ) + ……. + u(t − (n−1)T −τ)] ]. 10. Represent the train of pulses, mentioned in problem 8 in Laplace domain. [Hints: First we evaluate the Laplace transform of u(t) – u(t-τ), which is (1/s) – e-sτ/s = (1/s) (1- e-sτ). For the second pulse, the Laplace transform of u(t-T) – u(t-T-τ) is given by (1/s) [ e-sT –e-s(T+τ)] = (1/s) e-sT ( 1- e-s τ ). For the third pulse, the Laplace transform of u(t -2T)-U(t-2T-τ) is given by (1/s) e-2Ts (1 – e-sτ)

Exercises

123

Thus for n-pulses the complete representation of the pulses in Laplace domain is (1/s) (1- e-sτ) e-sT+ (1/s) (1- e-sτ) e-2Ts + ......+ (1/s) (1- e-sτ) e-(n-1)sT = (1/s) (1- e-sτ) [ e-sT + e-2Ts+ ...... + e-(n-1) sT] = (1/s) e-sT (1- e-s(n-2)T). ]

11. Assume that threat occurs as a periodic train of infinite number of pulses of duration τ and time period T. Show that the effect of these train of threats does not have significant manifestation to a subject, when amygdale gain to threat perception is unity. The amygdala gain, here, refers to its response to unit threat. [Hints: First we consider n → α in the results of the expression obtained in problem 3. Lt (1/s)e-sT (1- e-s(n-2)T) n →α = (1/s) e-sτ . Since amygdala gain is unity, therefore, the output is (1/s) e-sτ, the inverse transform of which is u(t- τ), i.e. a delayed unit step function.] 12. Assuming R and L as the right and the left Alpha power, suppose we use A[2] = (R-L)/(R+L). For assessment of alpha asymmetry, show that the higher is the ratio of right to left alpha, the greater is the score. Also prove that A[2] is a monotonically increasing function of (R/L). [Hints: A[2] =

R-L R+L

=

(R/L) -1 (R/L) +1

Naturally, when R/L increases, A[2] also increases.

124

4 Brain Imaging and Psycho-pathological Studies on Self-regulation of Emotion

To prove that A[2] is monotonically increasing function of (R/L), we evaluate (R/L +1). 1 - ( R | L – 1). 1

dA[2] = d(R/L)

(R/L +1)2 2

= (R/L +1)2

>0.

Consequently. A[2] is a monotonically increasing function of (R/L). 13. Show that when A[1] = 0.5, the right alpha power is more than 3 times the left alpha power. [Hints: A[1] = log10R - log10L = log10 (R/L) ∴ R/L = 10A[1] or, R = 10A[1] .L when A[1] = 0.5, R= √10 . L ≈ 3.3 L]

References 1. Adolphs, R., Tranel, D., Damasio, H., Damasio, A.R.: Impaired recognition of emotion in facial expression following bilateral damage to the human amygdala. Nature 372, 669–672 (1994) 2. Amaral, D.G., Price, J.L., Pitkanen, A., Carmichael, S.T.: Anatomical organization of the primate amygdaloid complex. In: Aggleton, J.P. (ed.) The Amygdala: Neurobiological Aspects of Emotion, Memory and Mental Dysfunction, pp. 1–66. Wiley-Liss, New York (1992) 3. Augustine, A.R.: Circuitry and functional aspects of the insular lobe in primates including humans. Brain Resa. Rev. 22, 229–244 (1996) 4. Barrett, K., Campos, J.: Perspectives on emotional development: II. A functionalist approach to emotion. In: Olsofsky, J. (ed.) Handbook of infant development, 2nd edn., pp. 555–578. Wiley-Interscience, New York (1987) 5. Beauregard, M.: Consciousness, Emotional Self-Regulation and the Brain. John Benjamins Publishing Company, Amsterdam (2003) 6. Beauregard, M., Levesque, J., Paquette, V.: Neural basis of conscious and voluntary self-regulation of emotion. In: Beauregard, M. (ed.) Consciousness, Emotional SelfRegulation and the Brain. John Benjamins Publishing Company, Amsterdam (2003)

References

125

7. Bermudez-Rattoni, F., Introini-Collison, I.B., McGaugh, J.L.: Reversible lesions of the insular cortex by tetrodotoxin produce retrograde and anterograde amnesia for inhibitory avoidance and spatial learning. Proc. Natl. Acad. Sci. 88, 5379–5382 (1991) 8. Blair, R.J., Morris, J.S., Frith, C.D., Perrett, D.I., Dolan, R.J.: Dissociable neural responses to facial expressions of sadness and anger. Brain 122, 883–893 (1999) 9. Blanhard, D.C., Blanchard, R.J.: Innate and conditioned reactions to threat in rats with amygdaloid lesions. J. Comp. Physiol. Psychol. 81, 281–290 (1972) 10. Blumberg, S., Izard, C.: Affect and cognitive characteristics of depression in 10-and 11 year old children. Journal of Personality and Social Psychology 49, 194–202 (1985) 11. Blumberg, S., Izard, C.: Discriminating patterns of emotion in 10-and 11-year old children’s anxiety and depression. Journal of Personality and Social Psychology 51, 852–857 (1986) 12. Breiter, H.C., Etcoff, N.L., Whalen, P.J., Kennedy, D.N., Rauch, S.L., Buckner, R.L., Strauss, M.M., Hyman, S.E., Rosen, B.R.: Response and habituation of the human amygdala during visual processing of facial expression. Neuron 2, 875–887 (1996) 13. Buchel, C., Morris, J., Dolan, R.J., Friston, K.J.: Brain systems mediating aversive conditioning: an event-related fMRI study. Neuron 20, 947–957 (1998) 14. Bussey, T.J., Everitt, B.J., Robbins, T.W.: Dissociable effects of cingulated and medial frontal cortex lesions on stimulus-reward learning using a novel Pavlovian autoshaping procedure for the rat: implications for the neurobiology of emotion. Behav. Neurosci. 111, 908–919 (1997) 15. Butter, C.M., Snyder, D.R., McDonald, J.A.: Effects of orbitofrontal lesions on aversive and aggressive behaviors in rhesus monkeys. J. Comp. Physiol. Psychol. 72, 132–144 (1969) 16. Butter, C.M., McDonald, J.A., Snyder, D.R.: Orality preference behavior and reinforcement value of non-food object in monkeys with orbital frontal lesions. Science 164, 1306–1307 (1969) 17. Calder, A.J., Keane, J., Manes, F., Antoun, N., Young, A.W.: Impaired recognition and experience of disgust following brain injury. Nat. Neurosci. 3, 1077–1078 (2000) 18. Calder, A.J., Lawrence, A.D., Young, A.W.: Neuropsychology of fear and loathing. Nat. Rev. Neurosci. 2, 352–363 (2001) 19. Calkins, S.D., Gill, K.L., Johnson, M.C., Smith, C.L.: Emotional reactivity and emotional regulation strategies as predictors of social behavior with peers during toddlerhood. Social Development 8, 310–334 (1999) 20. Carmichael, S.T., Price, J.L.: Architecture subdivision of the orbital and medial prefrontal cortex in the macaque monkey. J. Comp. Neurol. 346, 366–402 (1994) 21. Carmihael, S.T., Price, J.L.: Sensory and premotor connections of the orbital and medial prefrontal cortex of macaque monkeys. J. Comp. Neurol. 363, 642–664 (1995) 22. Carter, C.S., Botvinick, M.M., Cohen, J.D.: The contribution of the anterior cingulated cortex to executive processes in cognition. Reviews in Neuroscience 10, 49–57 (1999) 23. Casey, B.J., Cohen, J.D., Jezzard, P., Turner, R., Noll, D.C., Trainor, R.J., Giedd, J., Kaysen, D., Hertz-Pannier, L., Rapoport, J.L.: Activation of prefrontal cortex in children during a nonspatial working memory task with functional MR. Neuroimage 2, 221–229 (1995)

126

4 Brain Imaging and Psycho-pathological Studies on Self-regulation of Emotion

24. Casey, R.: Emotional competence in children with externalizing and internalizing disorders. In: Lewis, M., Sullivan, M.W. (eds.) Emotional Development in atypical children, pp. 161–183. Lawrence Erlbaum, Mahwah (1996) 25. Casey, R., Schlosser, S.: Emotional responses to peer praise in children with and without a diagnosed externalizing disorder. Merrill-Palmer Quarterly 40, 60–81 (1994) 26. Cavada, C., Company, T., Tejedor, J., Cruz-Rizzolo, R.J., Reinoso-Suarez, F.: The anatomical connections of the macaque monkey orbitofrontal cortex: a review. Cereb. Cortex. 10, 220–242 (2000) 27. Chiavaras, M.M., Petrides, M.J.: Orbitofrontal sulci of the human and the macaque monkey brain. J. Comp. Neurology 422, 35–54 (2000) 28. Cole, P.M., Zahn-Waxler, C., Smith, K.D.: Expressive control during a disappointment: Variations related to preschooler’s behavior problems. Developmental psychology 30, 835–846 (1994) 29. Cole, P.M., Michel, M.K., Teti, L.O.: The Development of Emotion Regulation and Dysregulation: A Clinical Perspective. Monograph of the society for Research on Child Development 59, 73–100 (1994) 30. Critchley, H.D.: Fear conditioning in Humans: the influence of awareness and automatic arousal on functional neuroanatomy. Neuron 33, 653–663 (2002) 31. Critchley, H.D., Daly, E., Philips, M., Brammer, M., Bullmore, E., Williams, S., Van Amelsvoort, T., Robertson, D., David, A., Murphy, D.: Explicit and implicit neural mechanisms for processing of social information from facial expressions: a functional magnetic resonance imaging study. Hum. Brain Mapping 9, 93–105 (2000) 32. Davidson, R.J.: Cerebral asymmetry, emotion, and affective style. In: Davidson, R.J., Hugdahl, K. (eds.) Brain asymmetry. MIT Press, Cambridge (1995) 33. Davidson, R.J., Putnam, K.M., Larson, C.L.: Dysfunction in the neural circuitry of emotion regulation – A possible prelude to violence. Science 289, 591–594 (2000) 34. Davis, M.: The amygdala and conditioned fear. In: Aggleton, J.P. (ed.) The Amygdala: Neurobiological Aspects of Emotion, Memory and Mental Dysfunction, pp. 255–306. Wiley-Liss, New York (1992) 35. Derryberry, D., Rothbart, M.K.: Arousal, affect and attention as components of temperament. Journal of Personality and Social Psychology 55, 958–966 (1988) 36. Devnisky, O., Morell, M.J., Vogt, B.A.: Contributions of anterior cingulated cortex to behavior. Brain 118, 279–306 (1995) 37. Eisenberg, N., Cumberland, A., Spinrad, T.L., Fabes, R.A., Shepard, S.A., Reiser, M., et al.: The relations of regulation and emotionality to children’s externalizing and internalizing problem behavior. Child Development 72, 1112–1134 (2001) 38. Eisenberg, N., Fabes, R.A., Murphy, B.C., Shepard, S., Guthrie, I.K., Mazsk, P., et al.: Prediction of elementary school children’s socially appropriate and problem behavior from anger reactions at age 4-6 years. Journal of Applied Developmental Psychology 20, 119–142 (1999) 39. Eisenberg, N., Fabes, R.A., Shepard, S.A., Guthrie, I.K., Murphy, B.C., Reiser, M.: Parental reactions to children’s negative emotions: Longitudinal relations to quality of children’s social functioning. Child Development 70, 513–534 (1999) 40. Eisenberg, N., Guthrie, I.K., Fabes, R.A., Shepard, S.A., Losoya, S., Murphy, B.C., et al.: Prediction of elementary school children’s externalizing problem behaviors from attentional and behavioral regulation and negative emotionality. Child Development 71, 1367–1382 (2000)

References

127

41. Ekman, P., Frisen, W.V.: Pictures of Facial Affect. Consulting Psychologists Press, Palo Alto (1976) 42. Eslinger, P., Damasio, A.: Severe disturbance of higher cognition after bilateral frontal lobe ablation: patient EVR. Neurology 35, 1731–1741 (1985) 43. Eslinger, P.J., Damasio, A.R.: Severe disturbance of higher cognition after bilateral frontal lobe ablation. Neurology 35, 1731–1741 (1985) 44. Falkenstein, M., Hohnsbein, J., Blanke, L.: Effects of crossmodal divided attention on late ERP components II. Error processing in choice reaction tasks. Electroencephalography and Clinical Neurophysiology 78, 447–455 (1991) 45. Fetz, E.E., Finnochio, D.V.: Operant conditioning of specific patterns of neural and muscular activity. Science 174, 431–435 (1971) 46. Fox, S.S., Rudell, A.P.: Operant controlled neural event: Formal and systematic approach to electrical coding of behavior in brain. Science 162, 1299–1302 (1968) 47. Fredrikson, M., Wik, G., Fischer, H., Anderson, J.: Affective and attentive neural networks in humans: a PET study of Pavlovian conditioning. NeuroReport 7, 97–101 (1995) 48. Garber, J., Braadfladt, N., Weiss, B.: The regulation of sad effect: An information processing perspective. In: Garber, J., Dodge, K.A. (eds.) The development of emotion regulation and dysregulation, pp. 208–240. Cambridge University Press, New York (1991) 49. Gehring, W.J., Goss, B., Coles, M.G.H., Meyer, D.E., Donchin, E.: A neural system for error detection and compensation. Psychological Science 4, 385–390 (1993) 50. George, M.S., Ketter, P.I., Rosinsky, N., Ring, H., Casey, B.J., Trimle, M.R., et al.: Region brain activity when selecting a response despite interference: an H2150 Pet study of the stroop and emotional stroop. Human Brain Mapping 1, 194–209 (1994) 51. George, M.S., Ketter, T.A., Gill, D.S., Haxby, J.V., Ungerleider, L.G., Herscovitch, P., Post, R.M.: Brain regions involved in recognizing facial emotion or identity: an oxygen-15 PET study. J. Neuropsychiatry Clin. Neurosci. 5, 384–394 (1993) 52. Gilliom, M., Shaw, D.S., Beck, J.E., Schonberg, M.A., Lukon, J.L.: Anger regulation in disadvantaged preschool boys: Strategies, antecedents, and the development of self-control. Developmental Psychology 38, 222–235 (2002) 53. Greenberg, N., Scott, M., Crews, D.: Role of the amygdala in the reproductive and aggressive behavior of the lizart. Physiology and behavior 32, 147–151 (1984) 54. Gross, J.J., Leveson, R.W.: Hiding Feelings: The acute affects of inhibiting negative and positive emotion. Journal of Abnormal Psychology 106, 95–103 (1997) 55. Gross, J.J.: Emotion Regulation past, present and future. Cognition and Emotion 13, 551–573 (1999) 56. Gross, J.J., Munoz, R.F.: Emotion regulation and mental health. Clinical Psychology: Science and Practice 2, 151–164 (1995) 57. Halgren, E., Babb, T.L., Rausch, R., Crandall, P.H.: Mental phenomena evoked by electrical stimulation of the human hippocampal formation and amygdala. Brain 101, 83–117 (1978) 58. Hamann, S.B., Ely, T.D., Grafton, S.T., Kilts, C.D.: Amygdala activity related to enhanced memory for pleasant and aversive stimuli. Nat. Neuroscience 2, 289–293 (1999) 59. Hartlage, S., Alloy, L.B., Vazquez, C., Dykman, B.: Automatic and effortful processing in depression. Psychological Bulletin 113, 247–278 (1993)

128

4 Brain Imaging and Psycho-pathological Studies on Self-regulation of Emotion

60. Hayes, S.C., Wilson, K.G., Gifford, E.V., Follete, V.M., Strosahl, K.: Experiential avoidance and behavioral disorders: A functional dimensional approach to diagnosis and treatment. Journal of Consulting and Clinical Psychology 64, 1152–1168 (1906) 61. Henriques, J.B., Davidson, R.J.: Left frontal hypoactivation in depression. Journal of Abnormal Psychology 100, 535–545 (1991) 62. Henriques, J.B., Davidson, R.J.: Regional brain electrical activities discriminate between previously depressed and healthy control subjects. Journal of Abnormal Psychology 99, 22–31 (1990) 63. Henry, J.P.: Neuroendocrine patterns of emotional response. In: Plutchik, R., Kellerman, H. (eds.) Biological foundations of emotion. Academic Press, New York (1986) 64. Herman-Stahl, M.A., Stemmler, M., Petersen, A.C.: Approach and avoidant coping: Implications for adolescent mental health. Journal of Youth and Adolescent 24, 649– 665 (1995) 65. Hitchcock, J.M., Davis, M.: Efferent pathway of the amygdala involved in conditioned fear as measured with the fear-potentiated startle paradigm. Behav. Neurosci. 105, 826–842 (1991) 66. Hugdahl, K., Berardi, A., Thompson, W.L., Kosslyn, S.M., Macy, R., Baker, D.P., Alpert, N.M., Ledoux, J.E.: Brain mechanisms in human classical conditioning: a PET blood flow study. NeuroReport 6, 1723–1728 (1995) 67. Iwata, J., LeDoux, J.E., Meeley, M.P., Arrneric, S., Reis, D.J.: Intrinsic neurons in the amygdoloid field projected to by the medial geniculate body mediate emotional responses conditioned to acoustic stimuli. Brain Research 383, 195–214 (1986) 68. Janer, K.W., Pardo, J.V.: Deficits in selective attention following bilateral anterior cingulotomy. Journal of Cognitive Neuroscience 3, 231–241 (1991) 69. Johnsrude, I.S., Owen, A.M., White, N.M., Zhao, W.V., Bohbot, V.: Impaired preference conditioning after anterior temporal lobe resection in humans. J. Neuroscience 20, 2649–2656 (2000) 70. Jones, B., Mishkin, M.: Limbic relations and the problem of stimulus reinforcement associations. Experimental Neurology 36, 362–377 (1972) 71. Kagan, J., Snidman, N., Zenter, M., Peterson, E.: Infant temperament and anxious symptoms in school age children. Development and Psychopathology 11, 209–224 (1999) 72. Kamiya, J.: Operant control of the EEG Alpha rhythm and some of its reported effects on consciousness. In: Tart, C. (ed.) Altered states of Consciouness, pp. 507– 517. John Wiley and Sons, New York (1969) 73. Kluwer, H., Bucy, P.C.: Preliminary analysis of the functions of the temporal lobes in monkies. Arch. Neurol. Psychiat. 42, 979–1000 (1939) 74. Knight, D.C., Smith, C.N., Stein, E.A., Helmstetter, F.J.: Functional MRI of Human Pavlovian fear conditioning: Patterns of Activation as a function of learning. NeuroReport 10, 3665–3670 (1999) 75. Konow, A., Prilbram, K.H.: Error recognition and utilization produced by injury to the frontal cortex in man. Neuropsychologia 8, 489–491 (1970) 76. Kuo, B.C.: Automatic Control Systems. Prentice-Hall, Englewood Cliffs (1991) 77. Lancelot, C., Nowicki Jr., S.: The association between receptive nonverbal processing abilities and internalizing problems in girls and boys. Journal of Genetic Psychology 158, 297–302 (1997) 78. Lane, R.D., Nadel, L.: Cognitive neuroscience of emotion. Oxford University Press, New York (2000)

References

129

79. Lane, R.D., Reiman, E.M., Ahern, G.N., Schwartz, G.E., Davidson, R.D.: Neuroanatomical correlates of happiness, sadness and disgust. American J. Psychiatry 154, 926–933 (1997) 80. Langston, C.A.: Capitalizing on and coping with daily-life events: Expressive responses to positive events. Journal of Personality and Social Psychology 67, 1112– 1125 (1994) 81. LeDoux, J.E.: Emotion. In: Plum, F. (ed.) Handbook of Physiology, Section 1: The Nervous System. Higher Function of the Brain, vol. 5, pp. 419–460. American Physiological Society, Bethesda (1987) 82. LeDoux, J.E., Sakaguchi, A., Reis, D.J.: Sub-cortical efferent projections of the medial geniculate nucleus mediate emotional responses conditioned to acoustic stimuli. J. Neurosci. 4, 683–693 (1984) 83. LeDoux, J.E.: The Emotional Brain. Simon & Scuster, New York (1996) 84. Levesque, J., Eugene, F., Joanette, Y., Paqueete, V., Mensour, B., Beaudoin, G., Leroux, J.M., Bourgouin, P., Beauregard, M.: Neural circuitry underlying voluntary self-regulation of sadness. Biological psychiatry 53, 502–510 (2003); Linehan, M.M. (ed.): Cognitive-behavioral treatment of borderline personality disorder. Guilford, New York (1993) 85. Linke, R., Braune, G., Schwegler, H.: Differential projections of the posterior paralaminar thalamic nuclei to the amygdaloid complex in the rat. J. Exp. Brain Rese. 134, 520–532 (2000) 86. Linke, R., de Lima, A.D., Schwegler, H., Pape, H.-C.: Direct synaptic connections of axons from superior colliculus with identified thalamo-amygdaloid projection neurons in the rat: possible substrates of a subcortical visual pathway to the amygdala. J. Comp. Neurol. 403, 158–170 (1999) 87. Luria, A.R., Pribram, K.H.: An experimental analysis of the behavioral disturbance produced by a left frontal arachnoidal endotheliomo (meningioma). Neuropsychologia 2, 257–281 (1964) 88. Luu, P., Tucker, D.M.: Regulating action: alternating activation of human prefrontal and motor cortical networks. Clinical Neurophysiology 112, 1295–1306 (2001) 89. Luu, P., Tucker, D.M.: Self-regulation and the executive functions: electrophysiological clues. In: Zani, A., Preverbio, A.M. (eds.) The cognitive electrophysiology of mind and brain. Academic Press, San Diego (2002) 90. Luu, P., Tucker, D.M.: Self-regulation by the medial frontal cortex: Limbic representation of motive set-points. In: Beauregard, M. (ed.) Consciousness, Emotional SelfRegulation and the Brain. John Benjamins Publishing Co., Philadelphia (2003) 91. Luu, R., Tucker, D.M., Derryberrry, D., Reed, M., Poulsen, C.: Activity in human medial frontal cortex in emotional evaluation and error monitoring. Psychological Science 14, 47–53 (2003) 92. MacDonald, A.W., Cohen, J.D., Stenger, V.A., Carter, C.S.: Dissociating the role of dorosolateral prefrontal and anterior cingulated cortex in cognitive control. Science 288, 1835–1838 (2000) 93. MacFall, J.R., Payne, M.E., Provenzale, J.E., Krishnan, K.R.R.: Medial orbiral frontal lesions in late-onset depression. Biological Psychiatry 49, 803–806 (2001) 94. McEwen, B.S.: The neurobiology of stress: from screendipty to clinical relevance. Brain Research 886, 172–189 (2000) 95. Mesulam, M.M.: Patterns in behavioural neuroanatomy: Association areas, the limbic system, and hemispheric specialization. In: Mesulam, M.M. (ed.) Principles of behavioral Neurology, pp. 1–70. F.A. Davis Co, Philadelphia (1985)

130

4 Brain Imaging and Psycho-pathological Studies on Self-regulation of Emotion

96. Moore, L.A., Hughes, J.N., Robinson, M.: A comparison of the social informationprocessing abilities of rejected and accepted hyperactive children. Journal of Clinical Child Psychology 21, 123–131 (1992) 97. Morris, J.S., Buechel, C., Dolan, R.J.: Parallel neural responses in amygdala subregions and sensory cortex during implicit aversive conditioning. Neuroimage 13, 1044–1052 (2001) 98. Morris, J.S., Friston, K.J., Buechal, C., Frith, C.D., Young, A.W., Calder, A.J., Dolan, R.J.: A neuromodulatory role for the human amygdala in processing emotional fear expression. Brain 121, 47–57 (1998) 99. Morris, J.S., Friston, K.J., Dolan, R.J.: Experience-dependent modulation of tomotopic neural responses in human auditory cortex. Proc. Royal Society, London, Ser. B. Biol. Sci. 265, 649–657 (1988) 100. Morris, J.S., Friston, K.J., Dolan, R.J.: Neural responses to salient visual stimuli. Proc. Royal Society, London, Ser. B. 264, 760–775 101. Morris, J.S., Frith, C.D., Perrett, D.I., Rowland, D., Young, A.W., Calder, A.J., Dolan, R.J.: A differential neural response in the human amygdala to fearful and happy facial expressions. Nature 383, 812–815 (1996) 102. Nauta, W.J.H.: The problem of the frontal lobe: A reinterpretation. Journal of Psychiatric Research 8, 167–187 (1971) 103. Nishijo, H., Ono, T., Nishino, H.: Topographic distribution of modality-specific amygdalar neurons in alert monkey. J. Neurosci. 8, 3556–3569 (1988) 104. Nishijo, H., Uwano, T., Tamura, R., Ono, T.: Gustatory and multimodal neuronal responses in the amygdala during licking and discrimination of sensory stimuli in aeake rats. J. Neurophysiol. 79, 21–36 (1998) 105. Nolen-Hoeksema, S.: Responses to depression and their effects on the duration of depressive episodes. Journal of Abnormal Psychology 100, 569–582 (1991) 106. Nolen-Hoeksema, S., Morrow, J., Fredrickson, B.L.: Response styles and the duration of episodes of depressed mood. Journal of Abnormal Psychology 102, 20–28 (1993) 107. Nowicki Jr., S.: The relation of non-verbal processing ability of faces and voices and children’s feelings of depression and competence. Journal of Genetic Psychology 158, 357–363 (1997) 108. Ochsner, K.N., Kosslyn, S.M., Cosgrove, G.R., Cassem, E.H., Price, B.H., et al.: Deficits in visual cognition and attention following bilateral anterior cingulotomy. Neuropsychologia 39, 219–230 (2001) 109. Oppenheimer, S.M., Gelb, A., Grivin, J.P., Hachinski, V.: Cardiovascular effects of human insula cortex stimulation. Neurology 42, 1727–1732 (1992) 110. Papez, J.W.: A proposed mechanism for emotion. Arch. Neurol. Psychiatry 38, 725– 743 (1937) 111. Phelps, E.A., O’Connor, K.J., Gatenby, J.C., Gore, J.C., Grillon, C., Davis, M.: Activation of the left amygdala to a cognitive representation of fear. Nat. Neurosci. 4, 437–441 (2001) 112. Phillips, M.L., Young, A.W., Scott, S.K., Calder, A.J., Andrew, C., Giampietro, V., Williams, S.C., Bullmore, E.T., Brammer, M., Gray, J.a.: Neural responses to facial and vocal expressions of fear and disgust. Proc. Royal Society, London, Ser. B. Biol. Sci. 265, 1809–1817 (1998) 113. Phillips, M.L., Young, A.W., Senior, C., Brammer, M., Andrew, C., Calder, A.J., Sprengelmeyer, E.T.R., Rausch, M., Eysel, U.T., Przuntek, H.: Neural structures associated with recognition of facial expressions of basic emotions. Proc. Royal Society, London Ser. B. Biol. Sci. 265, 1809–1817 (1988)

References

131

114. Phillips, M.L., Young, A.W., Senior, C., Brammer, M., Andrew, C., Calder, A.J., Bullmore, E.T., Perrett, D.I., Rowland, D., Williams, S.C., Gray, J.A., David, A.S.: A specific neural substrate for perceiving facial expressions of disgust. Nature 389, 495–498 (1998) 115. Posner, M.I., Dehaene, S.: Attentional networks. Trends in Neuroscience 17, 75–79 (1994) 116. Pribram, K.H.: A review of theory in physiological psychology. Annual Review of Psychology 11, 1–40 (1960) 117. Rainville, P., Duncan, G.H., Price, D.D., Carrier, B., Bushnell, M.C.: Pain affect encoded in human anterior cingulated but not somatosensory cortex. Science 277, 968– 971 (1997) 118. Rauch, S.L., et al.: A positron emission tomographic study of simple phobic symptom provocation. Arch. Gen. Pschiatry 52, 20–28 (1995) 119. Rolls, E.T., Treves, A.: Neural Networks and Brain Function. Oxford University Press, London (1998) 120. Rolls, E.T., Sienkiewicz, Z.J., Yaxley, S.: Hunger modulates the responses to gustatory stimuli of single neurons in the orbitofrontal cortex of the macaque monkey. Eur. J. of Neurosci. 1, 53–60 (1989) 121. Rosenfield, J.P.: An EEG biofeedback protocol for affective disorders. Clinical Electroencephalography 31, 7–12 (2000) 122. Rosenfield, J.P.: Applied psychophysiology and biofeedback of eventrelated potentials (brain waves):Historical perspective, review, future directions. Biofeedback and Self-Regulation 15, 99–120 (1990) 123. Rosenfield, J.P., Bachr, E.: EEG biofeedback (“Neorufeedback”) and affective disorders. In: Beauregard, M. (ed.) Consciousness, Emotional Self-Regulation and the Brain. John Benjamins Publishing Company. Amsterdam (2003) 124. Rosenfield, J.P., Bachr, E., Baehr, R., Gotlib, I., Ranganath, C.: Daily changes in frontal alpha asymmetry correlate with changes in affect in therapy sessions. Int. J. Psychophysiology 23, 37–41 (1996) 125. Rosenfield, J.P., Rudell, A.P., Fox, S.S.: Operant control of neural events in humans. Science 165, 821–823 (1969) 126. Rylander, G.: Personality analysis before and after frontal lobotomy. In: Fulton, J.F., Aring, C.D., Wortis, B.S. (eds.) Research publications association for research in nervous and mental disease: The frontal lobes, pp. 691–705. Williams and Wilkins Publishers, Baltimore (1947) 127. Sawamoto, N., Honda, M., Okada, T., Hanakawa, T., Kanda, M., Fukuyama, H., Konishi, J., Shibasaki, H.: Expectation of pain enhances responses to nonpainful somatosensory stimulation in the anterior cingulated cortex and parietal operculum/posterior insula: an event related functional magnetic resonance imaging studies. J. Neurosci., 20, 7438–7445 (2000) 128. Scott, W.D., Winters, R.W., Beevers, C.G.: Affective distress as a central and organizing symptom in depression: Psychological mechanisms. In: Johnson, S.L., Hayes, A.M., Field, T.M., Schneiderman, N., McCabe, P.M. (eds.) Stress, coping, and depression, pp. 145–175. Lawrence Erlbaum Associates, New Jersey (2000) 129. Shaver, P., Schwartz, J., Kirson, D., O’Connor, C.: Emotion Knowledge: Further explorations of a prototype approach. Journal of Personality and social Psychology 52, 1061–1086 (1987) 130. Silver, J.M., Yudofsky, S.C.: Aggressive behavior in patients with neuropsychiatric disorders. Psychiatric Annals 17, 367–370 (1987)

132

4 Brain Imaging and Psycho-pathological Studies on Self-regulation of Emotion

131. Small, D.A., Zald, D.H., Jones-Gottman, M., Zatorre, R.J., Pardo, J.V., Frey, S., Petrides, M.: Human cortical gustatory areas: a review of functional neuroimaging data. NeuroReport 10, 7–14 (1999) 132. Smith, W.: The functional significance of the rostral cingular cortex as revealed by its responses to electrical stimulation. J. Neurophysiol. 8, 241–255 (1945) 133. Sprengelmeyer, R., Rausch, M., Eysel, U.T., Przuntek, H.: Neural structures associated with recognition of facial expressions of basic emotions. Proc. Royal Society, London, Ser. B. Biol. Sci. 265, 1927–1931 (1998) 134. Stuss, D.T., Benson, D.F.: Neuropsychological studies of the frontal lobes. Psychological Bulletin 95, 3–28 (1984) 135. Thompson, R.: Emotion regulation: A theme in search of a definition. In: Fox, N. (ed.) The development of emotion regulation. Monographs of the society of Research in Child Development, vol. 59, pp. 25–52 (1994) 136. Thompson, R.A.: Emotion and self-regulation. In: Thompson, R.A. (ed.) Socioemotional development: Nebraska symposium on motivation, pp. 367–467. University of Nebraska Press, Lincoln (1990) 137. Thorpe, S.J., Rolls, E.t., Maddison, S.: Neuronal activity in the orbitofrontal cortex of the behaving monkey. Exp. Brain Res. 49, 93–115 (1983) 138. Tucker, D.M., Luu, P., Pribram, K.H.: Social and emotional self-regulation. Annals of New York Academy of Sciences 769, 213–239 (1995) 139. Vogt, B.A., Finch, D.M., Olson, C.R.: Functional heterogeneity in cingulated cortex: the anterior executive and posterior evaluative regions. Cereb. Cortex 2, 435–443 (1992) 140. Watson, D., Clark, L.A., Weber, K., Assenheimer, J.S., Strauss, M.E., McCormick, R.A.: Testing a Tripartite Model: II. Exploring the symptom structure of Anxiety and Depression in Student, Adult, and Patient Samples. Journal of Abnormal Psychology 104, 15–25 (1995) 141. Wegner, D.M.: Ironic processes of mental control. Psychological Review 101, 34–52 (1994) 142. Weiskrantz, L.: Behavioral changes associated with ablation of the amygdale complex in monkeys. J. Comp. Physiol. Psychology 49, 381–391 (1956) 143. Whalen, P.J., Rauch, S.L., Etcoff, N.L., McInerney, S.C., Lee, M.B., Jenike, M.A.: Masked presentations of emotional facial expressions modulate amygdale activity without explicit knowledge. J. Neurosci. 18, 411–418 (1998) 144. Winn, P.: The lateral hypothalamus and motivated behavior: An old syndrome reassessed and a new perspective gained. Current Directions in Psychological Science 4, 182–187 (1995) 145. Wright, C.L., Fischer, H., Whalen, P.J., McInerney, S.C., Shin, L.M., Rauch, S.L.: Differential prefrontal cortex and amygdale habituation to repeatedly presented emotional stimuli. NeuroReport 12, 379–383 (2001) 146. Zabel, R.: Recognition of emotions in facial expressions by emotionally disturbed and nondisturbed children. Psychology in the schools 16, 119–126 (1979) 147. Zeman, J., Shipman, K., Suveg, C.: Anger and sadness regulation: Predictions to internalizing and externalizing symptoms in children. J. of Clinical Child and Adolescent Psychology 31, 393–398 (2002)

6 Control of Mental Stability in Emotion-Logic Interactive Dynamics

Stability of physical system is usually analyzed from the static and dynamic considerations of the system. Classical logic of propositions can be compared with a static physical system from the point of view of its stability. When a notion of time is attached to the classical logic, its dynamic behavior becomes comparable to the behavior of a dynamic physical system. The chapter addresses the issues of stability analysis of both (non-temporal) static and temporal logic systems. It extends propositional temporal logic by fuzzy sets and provides a general method to determine the condition for stability of a dynamic fuzzy reasoning system. The latter part of the chapter provides a framework to represent emotional dynamics by differential equations and determines the condition for stability of the dynamics. Next a control scheme to stabilize the emotional dynamics by adapting its parameters is discussed. Finally, the chapter proposes a new scheme to control the state of mind/actions, by reducing the difference between emotional states and temporal states of fuzzy reasoning system.

6.1 Introduction This chapter proposes a new model of emotional dynamics, where the emotions are represented by states and the transition of states takes place by external stimulus, indicating changes in the environmental conditions. The conditions liable for state-transition are labeled against the directed arcs, representing transition from a state to the same/other states. A positive/negative arc from state i to state j is used to describe the positive/negative influence of the external stimulus for state transition. Analysis of equilibrium behavior of emotional dynamics, and setting the parametric conditions in the dynamics for its stabilization is a virgin area of research in machine intelligence. Stability analysis of a dynamical system is usually performed with the help of a suitable Lyapunov energy function, capable of supporting the trajectory of motion of the dynamics. The Lyapunov function, which represents an energy surface, is judiciously selected to confine the dynamics of a physical system. The stability of the dynamics is ascertained by testing the negative sign in the time derivative of A. Chakraborty and A. Konar: Emotional Intelligence, SCI 234, pp. 175–207. © Springer-Verlag Berlin Heidelberg 2009 springerlink.com

176

6 Control of Mental Stability in Emotion-Logic Interactive Dynamics

the Lyapunov energy function. This in other words signifies that a dynamical system is stable when it looses energy in the process of movements on a trajectory over a Lyapunov energy surface. The chapter emphasizes the interaction of emotion and logic. Emotional states and logical premises/interpretations both share common phenomena of the real world, but the decisions derived by these two counterparts of the human mind may not always be unique. The chapter introduces a novel scheme to represent temporal behavior of a logical system by difference equation. It also proposes a new means to represent emotional state transitions by a dynamical system. The main difficulty in emotional reasoning is faced when the emotional states and the logical states contradict each other. A solution to emotion-logic encounter is given here by over-powering logical states over emotional states. An alternative situation may also take place when emotional states are strong enough to control the excessive growth in logical states. However, one fundamental hindrance in controlling emotion by logic or vice-versa is to ensure the stability of the dynamics. A general scheme for stability analysis of dynamical systems using Lyapunov energy function is presented in this chapter. Emotion arousal being a continuous process, emotional dynamics is represented by differential equations. On the other hand, the model of temporal logic we used here being a discrete system, we represented the temporal dynamics by difference equation. Since Lyapunov analysis is applicable to both differential/difference equation-based systems, we can employ Lyapunov method to determine the parametric condition for stability of the dynamical systems. Ensuring the parametric conditions by external perturbations to the dynamics would ensure stability of the dynamical systems. The chapter is divided into 10 sections. Section 6.2 presents a method to determine the stable points of (non-temporal) propositional logic. A time differential representation of propositional temporal logic and its local stability analysis is included in section 6.3. The Lyapunov approach for stability analysis of dynamical propositional temporal system is included in section 6.4. A causal model to represent human emotional dynamics is given in section 6.5 along with its Lyapunov stability analysis. The weight adaptation of the emotional dynamics is also included in the same section. Fuzzy extension of the propositional temporal logic representing emotional states and their transition is discussed in section 6.6 along with its stability analysis. A control policy to stabilize emotional dynamics using machine learning technique is presented in section 6.7. A scheme for controlling mental stability under the dynamic interaction of emotion and logic is presented in section 6.8. Conclusions are listed in section 6.9.

6.2 Stable Points of Non-temporal Logic The laws of statics as proposed by the physicist aim at obtaining the equilibrium of a physical system by analyzing the force distributions in a physical body. For example, consider the frame truss used in holding a bridge connecting banks of a river. When the law of statics is applied on a frame-truss, it presumes that every junction (or every member/arm) of the truss remains static without any external force on the body. In other words, the frame truss is in static equilibrium. To

6.2 Stable Points of Non-temporal Logic

177

understand the stable points of a static body, let us consider a rectangular box of 6 surfaces, where the 2 areas of surfaces are appreciably larger than the remaining 4 surfaces. The center of gravity (CG) of the box, when it stands on any of its surface, may be regarded as stable point of the body. Thus if the box can stand on any of its six surfaces, it will have six stable points. But how we compare the relative stability of the stable points? Suppose the box stands on any of its 2 larger surfaces. Under this condition, if a small force is applied to the box, then the box restores the stable point by returning to the same equilibrium position within a moment. But what happens when a small force is applied to the box standing on a smaller surface area? Definitely we cannot always say that the box will return to the same equilibrium position. A non-temporal Logic that does not include the notion of time to represent logical states can be compared with a static body from the point of view of its stability. Until now the literature on propositional logic and its extensions determine the interpretations of the statements, but did not take into account the stability of the logical statements with respect to their interpretations. In this chapter, we present a method for determining stable points of a logical statement. Informally speaking, stable points are selective interpretations of logical statement, causing no changes in the functional behavior of the statement. A dynamical system dealing with motion of an object in a continuous framework is usually represented by one or more time differential equations. In a complex system consisting of several variables: x1, x2, ……, xn the equations are sometimes coupled representing a positive or negative feedback from the attributes of one system-component to another. A logical reasoning system, where the notion of time is included to represent a dynamic behavior of the logic, can similarly be compared with the dynamical system of physical bodies. In the analysis of stability of dynamical frameworks, researchers pay more attention usually in two types of stability analysis: i) local stability ii) stability in the asymptotic sense (Lyapunov Stability). In this section, we would like to undertake the stability issues of propositional temporal and fuzzy-temporal logic, and interpret the result of stability analysis from the point of view of the stable interpretations of these dynamical systems. 6.2.1 Finding Common Interpretations of Propositional Statements In this section, we illustrate a simple method to computing common interpretations of propositional logical statements. The method involves determining interpretations of individual logical statement, and finally taking intersection of the interpretations obtained from the individual statement. Example 6.1: Given (p → q) → p and (q → p) → p, where p and q are 2 atomic propositions and the ‘→’ denote the if-then operator, we in this example illustrate the computation of common propositional interpretations of the given statements. ≡ ≡

(p → q) → p (p → q) → p ¬ (p → q) ∨ p

178

6 Control of Mental Stability in Emotion-Logic Interactive Dynamics

≡ ≡ ≡ ≡

¬ (¬p ∨ q) ∨ p (p ∧ ¬q) ∨ p (p∨ p) ∧ (p ∨ ¬q) p∧(p∨¬q),

(6.1)

the interpretations of which are given by (p, q)∈{(1, 0), (1,1)}.

(6.2)

Again, (q → p) → q q∧(q∨¬p),

≡

(6.3)

the interpretation of which are given by (p,q)∈{(0,1), (1,1)}.

(6.4)

The common interpretation of (6.2) and (6.4) is (p, q) = (1,1). Example 6.2: Given p → q and q → p, find the common interpretations. p→q ≡

¬ p ∨ q, the interpretations of which are (p, q)∈{(0,0), (0,1),(1,1)}.

(6.5)

q→p ≡

¬ q ∨ p, the interpretations of which are (p, q)∈{(0,0),(1,1),(1,0)}.

(6.6)

The common interpretations are (p, q) ∈ {(0, 0), (1, 1)}. The general method for determining the common interpretations of a set of logical statements is to construct a truth-table for the given statements by considering all possible combinations of the atomic proposition: p, q, r, ….., and finally determining the common interpretations of the given expression. Example 6.3 below illustrates the problem. Example 6.3: Given the propositional statement p ∧ q and p →q, we determine the common interpretations by constructing truth tables. From the Table 6.1 it is clear that if p →q and p ∧ q have a single common interpretation: (p, q) = (1, 1).

6.2 Stable Points of Non-temporal Logic

179

Table 6.1. Truth-table

p 0 0 1 1

q 0 1 0 1

p∧q 0 0 0 1

p →q 1 1 0 1

6.2.2 Determining Stable Points of Logical Statements In this section, we illustrate a simple method to determine stable points of a prepositional logical statement. Suppose, we need to determine the stable interpretation of the clause ‘q ←p.’. Replacing ‘q ←p.’ by ‘¬p∨ q’ and then further replacing ‘OR (∨)’ by s-norm [5], where for any two propositions a and b, a s b = a + b – ab, we can construct a relation F(p, q) as follows: F(p, q) = (1 − p) sq = (1 − p) + q − (1 − p) q = 1 − p + pq. It can be verified that F(p, q) = 1 − p + pq satisfies all the three interpretations of ‘q ←p.’ To determine the stable points, if any, on the constructed surface of F(p, q), let us presume that there exists at least one stable point (p*, q*). We now perturb (p*, q*) by (h, δ), i.e., p = p* + h q = q* + δ Thus, we obtain: F(p* + h, q* + δ) = 1 − p* + p*q* − h + hq* + δp* + hδ = F(p*, q*) − h + hq* + δp* + hδ Now for stability of the given clause at (p*, q*), we need to satisfy the following condition: F(p* + h, q* + δ) = F(p*, q*), which ultimately demands (−h + hq* + δp* + hδ) = 0. It can be verified that the aforementioned condition is satisfied only at (p*, q*) = (0, 1), irrespective of any small value of h and δ. However, if we put other two interpretations of ‘q ←p.’, such as (1, 1), (0, 0) in condition (1.21), we note that it imposes restriction on h and δ, which are not feasible. Thus the interpretation (p*, q*) = (0, 1) is a stable point. Determining stable points from common interpretations of logical statements can be obtained easily by extending the above idea.

180

6 Control of Mental Stability in Emotion-Logic Interactive Dynamics

6.3 Stable Points in Propositional Temporal Logic In a propositional temporal logic, the truth-value of propositions changes over time. The dynamics of a propositional temporal system on a continuous framework can be represented by time-differential equation. The solution of time differential equation in the steady state yields the stable points of the propositional temporal statements. Example 6.4 below gives an insight to this problem. Example 6.4: Given the following 2 differential equations, we determine the local stability of the dynamics. dp =

(p(t)→q(t))→p(t)

(6.7)

=

(q(t)→p(t))→q(t)

(6.8)

dt and

dq dt

where p (t) and q (t) denote the truth values of the propositions p and q at time t. The local stability of a time differential equation denotes a condition for temporal stability where the time derivative becomes zero. Simplifying (6.7) and (6.8) we have: dp =

p(t)∧(p(t)∨¬q(t))

by (6.1)

(6.9)

dt dq = q(t) ∧ (q(t) ∨ ¬ p(t)) by (6. 3)

and

(6.10)

dt dp Determination of the local stable points demands

dq = 0,

dt

= 0, dt

p(t)∧(p(t)∨¬q(t))=0

(6.11)

q(t)∧(q(t)∨¬p(t))=0

(6.12)

It is clear from (6.11) that either p(t) = 0 or (p(t) ∨ ¬ q(t)) = 0 Further (p(t) ∨ ¬ q(t)) = 0 yields Max (p(t), ¬ q(t)) = 0

6.3 Stable Points in Propositional Temporal Logic

⇒

p (t) = 0, ¬q (t) = 0

⇒

p (t) = 0, q (t) = 1.

181

So, the dynamical stable point of (6.11) is (p(t),q(t))∈{(0,1)}.

(6.13)

Similarly, the dynamic local stable point of (6.10) is (p(t),q(t))∈{(1,0)}.

(6.14)

Interpretation of (6.13) and (6.14) being a null set, the overall system does not have a single stable point. For the sake of implementation on digital computers, people nowadays prefer time-difference equations to time differential equation. A time difference equation has the advantage of evaluating the next value of a proposition, if its current value and its previous values are known. A recurrent model of time difference equation is generally used in the literature of dynamic system to model many practical problems on a discrete framework. Example 6.5 below provides the construction process of a time difference model of propositional temporal statements. Example 6.5: Given p and q be two atomic propositions and the argument t of p and q denotes the time of their occurrences. Suppose, p (t +1) is true if (p (t) → q (t)) → p (t) is true, and if

q (t+1) is true (q (t) → p (t)) → q(t) is true.

The above statements can be restructured into the following form: p(t+1)=(p(t)→q(t))→p(t) and

q(t+1)=(q(t)→p(t))→q(t)

(6.15) (6.16)

The logical basis of the above expression directly follows from the given statement. Since p (t+1) = 1 when (p (t) → q (t)) → p (t) is true; automatically, p (t+1) = 0 when the given condition fails. Thus expression (6.15) follows. The explanation of the construction process of expression (6.16) is analogous to that of expression (6.15). Determination of the stable points in discrete system requires setting of p ( t+1) = p (t) = p* and, q (t+1) = q (t) = q* say at time = t*

182

6 Control of Mental Stability in Emotion-Logic Interactive Dynamics

where t* denotes equilibrium time. It is indeed important to note that the equilibrium time refers to the minimum time, following which the equality of proposition at time t and (t+1) is maintained. Example 6.6: In this example, we determine the local stable points of the discrete propositional temporal statements: p (t+1) = p(t) ∧ (p(t) ∨ ¬ q(t))

(6.17)

q (t+1) = q(t) ∧ (q(t) ∨ ¬ p(t))

(6.18)

Now, let at time t = t*, we have p (t+1) = p(t) = p* say q (t+1) = q(t) = q* say. Then from (6.17) and (6.18) we have: p* = p* ∧ ( p* ∨ ¬q*) q* = q* ∧ ( q* ∨ ¬p*)

and

(6.19) (6.20)

Let us now analyze (6.19): p* = p* ∧ ( p* ∨ ¬q*). Since p* in the R.H.S. will be equal to p* in the L.H.S. counterpart, (6.15) holds only if p*≤ p*∨¬q* (6.21) or, p*≤ Max (p*,¬q*)

(6.22)

Therefore, p* = Max (p*,¬q*) if p* > ¬q* and (p*, q*) = (1, 1), and p* < Max(p*,¬q*) if ¬q* >p* and (p*,q*) = (0,0).

(6.23) (6.24)

Rewriting (6.20) we have, q* = q* ∧ (q*∨ ¬p*),

(6.25)

the stable points of which are (p*, q*) ∈{(1, 1), (0, 0)}. ∴ the stable points of the dynamic system is (0, 0) & (1, 1).

(6.26)

6.4 Stability of Propositional Temporal System

183

6.4 Sta bility of Propositio na l Te mporal System

6.4 Stability of Propositional Temporal System Using Lyapunov Energy Function The local stability analysis we introduced in the last section attempts to determine all the stable points of a dynamical system. Among the stable points, some are relatively more stable than the rest. What does relative stability mean? It means that for a small variation in the stable point, the dynamics may lose its stability and may have a transition to another state. This is similar to changing the stable point of a box standing on a very small surface. Dynamical systems that we are taking care of also have similar stable points with a narrow zone of stability. For determining the stability in the asymptotic sense, we need to identify stable points that allow larger disturbances without violating the stability of the system. In this system, we briefly outline the Lyapunov energy function and demonstrate as to how to use this function for determining the asymptotic stability of a dynamical system. 6.4.1 The Lyapunov Energy Function A Lyapunov energy function represents the energy contour of a particle when subjected to a set of forcing functions. Usually, the differential or the difference equation for which the stable points need to be investigated, act as forcing functions on a point mass particle residing over the Lyapunov energy surface. To analyze the stability of the dynamical systems, we have to determine a suitable energy surface that includes the trajectory due to motion of the particle by the forcing functions. The dynamics is said to be stable if the particle loses energy over time. Let L(p(t), q(t)) be a Lyapunov energy function, then it should satisfy the following criteria: i) L (0, 0) = 0 (zero at origin) ii) (p, q) > 0 for (p, q) ≠ (0,0)

(positive definition) (6.27)

iii)

∂L ∂L ------ & ------- should exist. (smooth surface) ∂q ∂p

For a selected Lyapunov energy function L, the given dynamics is asymptotically stable if dL/dt < 0 for continuous system dynamics and ΔL < 0 for discrete system dynamics. 6.4.2 Stability Analysis of Propositional Temporal System The smoothness criteria in (6.27), i.e. ∂L/∂p and ∂L/∂q should exist, require that L must be continuous function. Since ∧ and ∨ are discontinuous functions, we consider t and s norm instead of ∧ and ∨. Thus let

184

6 Control of Mental Stability in Emotion-Logic Interactive Dynamics

p t q = p. q p s q = p + q – pq.

and

Example 6.7: Consider the following 2 difference equations. Let us determine the stability of the dynamics, if it exists, by Lyapunov energy function. p (t+1) = q(t) → p(t)

(6.28)

and q (t+1) = p(t) → q(t)

(6.29)

Simplifying, we have:

and

p (t+1) = ¬ q (t) s p (t) = ¬ q(t) + p(t) – (¬ q(t)) (p(t))

(6.30)

q(t+1) = ¬ p(t) + q (t) – (¬p (t)) (q(t))

(6.31)

For determining the global stable points of the above dynamics, let L (p, q) = p + q

(6.32)

be the Lyapunov energy function. It is to be noted that the above function satisfies the Lyapunov energy criteria, i.e. L (0, 0) = 0 L (p, q) > 0 for (p, q) ≠ (0, 0). ∂L

∂L and

∂p

should exist. ∂q

Since the given system is discrete, we compute: ∆L = L (p (t +1), q (t+1)) – L (p(t), q(t)) ={p(t+1) + q(t+1)}–{p(t) + q(t)}

(6.33)

From (6.30), (6.31) and (6.33) we have ∆L = {(1- q (t)) + p (t) – (1- q (t)). p (t)} + {(1-p (t)) + q (t) – (1-p (t)). q (t)} - p (t) – q (t) = {(1- q (t)) – (1- q (t)). p (t)} + {(1-p (t)) - (1-p (t)). q (t) = (1- q (t)) (1-p (t)) + (1-p (t)) (1- q (t))

6.5 Human Emotion Modeling and Stability Analysis

185

(6.34)

= 2 (1- p(t)) (1- q(t)) > 0, for 0 < p(t) < 1, 0 < q(t) < 1. ∴ the given dynamics have no global stable points.

In section 6.5, we address the issues of emotion modeling by differential equations and analyze the stability of the emotional dynamics. Section 6.6 provides a fuzzy temporal model of dynamics involving emotional states.

6.5 Human Emotion Modeling and Stability Analysis Each human emotion can be regarded as a collection of mental (psychological) states and their relationship. For example, a strong anger that sustains for a long duration is aroused because of certain facts and their positive/negative causal relationships. Let x1, x2, ……., xn be n state variables that describe the arousal of emotions. Usually every state variable has a re-generating self-feedback with a positive causal dependence. Let xj for some j be states that excite state xi through a positive causal relationship. Also let xk for some k be an internal psychological state that has a negative causal impact on state xi. Fig 6.1 (a) presents a schematic view of a causal dependence relationship of xi, xj and xk on xi. Principles of emotion transition by causal events on a state diagram are illustrated in Fig. 6.1(b). We can represent the above causal dependence relationship by the following time differential equation + aii (1-xi/ k) dxi

xi

dt

- cki

xk

+ bji xi

Fig. 6.1(a). A state diagram depicting causal relationship among xi, xj and xk, the signed arcs denote causal temporal dependence.

186

6 Control of Mental Stability in Emotion-Logic Interactive Dynamics

dxi dt

= aii xi (1- xi/k) + ∑ bji xi xj - ∑ cki xi xk ∃j ∃k

(6.35)

Here, the first term in the R.H.S. of (6.35) indicates that as long as xi > k, the term will have a negative contribution to dxi/dt. When xi = k, the first term in the R. H. S. offers no contribution to dxi/dt. The second term in the right hand side of equation (6.35) correspond to cooperation between xi and xj for some j. The third term corresponds to competition of xi and xk for some k. Here, aii, bjk and cki are important system parameters that control the growth rate of xi. The personality of the subject is an important consideration in the modeling of human emotions. The softer the personality, the less is the competition and cooperation among the states. On the other hand, the stronger the personality the larger is the competition and the co-operation among the states. One simple way to represent the effect of personality on emotion modeling is to use an index p in the power of the product xixj and xixk. The dynamical system we presented in equation (6.35) is now remodeled as follows: dxi dt

= aii xi ( 1- xi/k) + ∑ bji (xi xj)p - ∑ cki (xi xk)p , ∃j ∃k

(6.36)

where p < 1 for softer personality and p > 1 for stronger personality and p = 1 for normal personality. 6.5.1 Stability Analysis of the Emotional Dynamics In this section we present an asymptotic analysis to stability of human emotions by the Lyapunov Approach. Let L(xi, xj, xk) be a Lyapunov energy function of the following form xi L(xi, xj, xk) = [ - ( aii k / 2 ) xi (1-xi/k)2 + ∫ (Σ bji xj - Σ ckixk) xi dxi] (6.37) 0 ∃j ∃k It is important to note that (6.37) satisfies the basic criteria to be a Lyapunov energy function. For instance, i) ii) iii)

L (0, 0, 0) = 0 L (xi, xj, xk)> 0 for non-zero xi,xj, xk ∂L/∂xi ; ∂L/∂xj and ∂L/∂xk exist.

The condition ii) L (xi, xj, xk)> 0 xi ∫ (Σbji xj - Σ ckixk) xidxi – (aii k/ 2) xi (1-xi/k)2 > 0 0 ∃j ∃k

(6.38)

6.5 Human Emotion Modeling and Stability Analysis

187

Mother Happy

Mother Angry

Treatment failure

Mother Anxious

Child under treatment Child under treatment

Child victim of thalasamia

Discovers wrong treatment of doctor

Treatment successful

Mother Sad

Fig. 6.1(b). Emotion transition graph of a mother, whose child is suffering from thalasamia disease.

and xi xi ∫ Σbji xj xidxi > ∫ Σ ckixkxidxi + (aii k/ 2) xi (1-xi/k)2 . (6.39) 0 ∃j 0 ∃k Fortunately, all these partial derivatives exist, satisfying criteria (iii). Now, to determine the condition for stability of the dynamics we set dL < 0

(6.40)

dt dL

∂L

dxi

∂xi

dt

= dt

(6.41) = [{- (aiik/ 2) (-2xi/ k) (1-xi/ k) + (∑ bji xj - ∑ cki xk) xi} ∃j ∃k - (aii k/ 2).1. (1- xi/ k)2] . (dxi/ dt) = [dxi/dt – (aii k/ 2) (1-xi/ k)2] (dxi/ dt) = (dxi/ dt)2 – (aiik/ 2) . (1-xi/ k)2 (dxi/ dt)

which is < 0

188

6 Control of Mental Stability in Emotion-Logic Interactive Dynamics

oscillatory

↑ (aiik/ 2) (1-xi/ k)2

unstable

dxi/ dt stable xi → Fig. 6.2. The zone of stability, instability and limit cycles.

if (aii k/ 2) (1-xi/ k )2 > dxi/ dt > 0

(6.42)

The zone of stability, satisfying condition (6.42) is visualized in Fig. 6.2. It is evident from condition that the dynamics is i) ii) iii)

stable when dxi/dt is below the curve oscillatory when dxi/dt lies on the curve unstable when dxi/dt is above the given curve.

Re-writing inequalities (6.39) and (6.42) and then computing them we have, xi xi ∫ Σbji xj xidxi - ∫ Σ ckixkxidxi 0 ∃j 0 ∃k

> xi

(dxi/dt + ∈)

(6.43)

xi xi ∫ Σbji xj xidxi - ∫ Σ ckixkxidxi > aii xi2 ( 1 – xi/k) + ∈xi + (Σbji xj xi2 0 ∃j 0 ∃k ∃j - (Σcki xk xi2 [by (6.36)] ∃j Differentiating with respect to xi on both sides of the above inequality: Σbji xj xi - Σcki xk xi > ∈ + 2Σbji xj xi - 2Σcki xk xi + ∃j ∃k ∃j ∃k aii (1-xi/k) (2xi) + aii xi2 (-1/k) ⇒ 2aii xi (1- xi/k) + Σbji xj xi - Σcki xj xi + ∈ - (aii/k) xi2 < 0 ∃j ∃k

(6.43a)

6.5 Human Emotion Modeling and Stability Analysis

189

⇒ (aii/k) xi2 > 2aiixi (1-xi/k) + Σbji xj xi - Σcki xj xi + ∈ ∃j ∃k 2 ⇒ (aii/k) xi - aiixi (1-xi/k) > dxi/dt + ε ⇒ 2aii xi2/k - aiixi > dxi/ dt + ε ⇒ aiixi (2xi/k – 1) > dxi/dt + ε ⇒ dxi/dt < aii xi (2xi/k-1)

(6.44)

The inequality (6.44) states the condition for stability of the emotional state xi. The condition for stability of the other emotional states xj and xk can be derived similarly. 6.5.2 Weight Adaptation in Emotion Dynamics by Hebbian Learning This case study illustrates the phenomenological transition of emotion by a causal network (Fig 6.1 (a)). The emotion in this diagram is denoted by states (circles), and the causal relationship for state transition is denoted by directed arc with a linguistic label describing the cause for emotion transition. Such diagrams have much resemblance with cognitive maps proposed by Bart Kosko [6] and later extended by Pal and Konar [8]. Given an initial fuzzy strength (membership) of the emotional states, we can determine the dynamic behavior of state transition by solving the coupled differential equation for each emotion. The emotion that comes up with the highest membership value is regarded as the emotion possessed by the person. Like classical cognitive maps [8], the directed arcs representing transition of emotion has a time-varying weight, the adaptation of which is usually maintained by Hebbian learning along with a self-decay in weight. For instance, the weight bij between emotion i and emotion j is given by sbij = -αbij + f (xi) f (xj),

(6.45)

dt where α is the mortality rate, xi and xj denote the signal strength of emotions i and j respectively, and f (.) is a sigmoid type nonlinear inhibiting function given by 1 f (xi) =

(6.46) 1 + e -xi

Konar and Pal [5] have already shown that the above learning dynamics of weight bij converges to stable state if and only if 0< α <1.

(6.47)

190

6 Control of Mental Stability in Emotion-Logic Interactive Dynamics

6.5.3 Improving Relative Stability of the Learning Dynamics In discrete domain, stability of an open loop system is ensured by identifying its poles within | z | = 1 circle. The closer the pole towards the center (0, 0) of the circle, the better is the stability of a system. On the other hand, when a pole is located on the periphery (but within) the | z | = 1 circle, the system looses its stability. Thus the notion of relative stability is important in the analysis of a dynamical system. To determine the relative stability of the given learning dynamics, we first discretize it in the form of a difference equation and then represent the resulting dynamics in the following form: | z – ( 1 - α) | = A

(6.48)

1

1

1- e-xi

1 + e-xj

where A =

.

(6.49)

The transient behavior of the modified learning dynamics (6.48) is given by | z – ( 1 - α) | = 0,

(6.50)

which indicates that larger is the value of α in 0 < α < 1, the faster is the fall–off of the transients towards zero. Naturally, the relative stability of the system is improved when α approaches 1 but is < 1. The relative stability of the dynamics can also be ascertained from the solution of the differential equation (6.45). The solution of 6.45 is given by A* bij (t) = {bij ( 0) -

α

A* } (1 - α)t +

,

(6.51)

α

where A* denotes the steady-state value of A. As α approaches 1, the first term in the right hand side of (6.51) approaches 0, indicating that the transient vanishes irrespective of time t. Naturally, this is possible when the relative stability of the system is very high.

6.6 The Fuzzy Temporal Representation of Phenomena Involving Emotional States 6.6 The Fuzzy Temporal Re presentation

To illustrate the scope of fuzzy temporal encoding of phenomena using emotional states, let us consider the following 2 dynamical system. dx1 = a x1 ( 1- x1/k) –b (x1 x2)p dt

(6.52)

6.6 The Fuzzy Temporal Representation

dx2

= -c (x1 x2)p + dx2

191

(6.53)

dt where x1 and x2 are 2 emotional states, a, b, c, d and k are parameters of the dynamics, whose explanation is given earlier. In equation (6.52) and (6.53), p is the personality index. Here, p > 1 for strong personality, p < 1 for weak (soft) personality and p =1 for normal personality. We briefly analyze the stability of the dynamics using the following Lyapunov energy function: L (x1, x2) = (kax1/ 2) (1-x1/k)2 + [b/ (p+1)] x2p . x1p+1 + [c/ (p+1)] x1p . x2p+1 – (d/2) x22

(6.54)

∂L/ ∂x1 = (ka/2) (1-x1/k)2 + (kax1/ 2) ( 1-x1/k) ( -2/k) + b(x1 . x2)p + [c . p/ (p+1)] x1p-1. x2p+1

(6.55)

∂L/ ∂x2 = [b . p/ (p+1)] x2p-1 . x1p+1 + c ( x1x2)p – dx2 dL/ dt = (∂L/∂x1) . (dx1/ dt) + (∂L/∂x2) . (dx2/ dt) = - (dx1/ dt)2 + {(ka/2) (1-x1/k)2 – [cp/(p+1)] x1p-1. x2p+1}(dx1/dt). - (dx2/ dt)2 + [b . p/ (p+1)] . x2p-1 . x1p+1. (dx2/dt) = - (dx1/dt)2 – (dx2/dt)2 + {(ka/2) ( 1-x1/k)2 – [cp/ (p+1)] x1p-1. x2p+1 . (dx1/dt)} + [b.p/ (p+1)]. x1p+1 . x2p-1. (dx2/ dt)

(6.56)

which will be negative definite if dx1/dt > 0 and dx2/dt < 0 (6.57) and i.e.,

p-1

{cp/ (p+1)}x1 x2

p+1

2

> ka/ 2 (1-x1/k) ,

x1 (t) is a monotonically increasing function

x2 (t) is a monotonically decreasing function, and the above inequality has to be satisfied. The equation (6.52) and (6.53) can be represented in fuzzy domain by a set of fuzzy rules. Two sample rules are discussed below for convenience.

192

6 Control of Mental Stability in Emotion-Logic Interactive Dynamics

Rule 1: If x1 (t) is LARGE and x1(t) is SMALLER than k and the contact between x1(t) and x2(t) is SMALL Then x1(t+1)is LARGE Rule 2: If x2(t) is LARGE and contact between x1(t) and x2(t) is SMALL Then x2(t+1) is LARGE. The fuzzy qualification of SMALL and LARGE may be changed to consider different fuzzy sets to take into account of the entire possibility space of all sets in the universe of x1 and x2. Let μLARGE(x1, t) denote the membership of x1 to be large at time t, μNEGATIVE ((x1(t) – k), t) denote the membership of x1(t) to be smaller than k at time t μSMALL ((x1(t). x2(t)), t) denote the membership of the product x1, x2 to be small at time t. The Rule1 thus can be stated in the following form of fuzzy temporal equation: μLARGE (x1, t+1) = aμLARGE(x1, t) × μNEGATIVE ((x1(t) – k), t) – bμSMALL ((x1 (t). x2 (t)), t).

(6.58)

We represent μSMALL ((x1(t). x2(t)), t) by μSMALL (x1(t), t) . μSMALL (x2(t), t). Finally, we have: μLARGE (x1, t+1) = aμLARGE(x1, t) × μNEGATIVE (x1 – k, t) -bμSMALL (x1, t). μSMALL(x2,t).

(6.59)

The Rule2 can be represented in the following form of fuzzy temporal equation: μLARGE (x2,t+1) = aμLARGE (x2,t) – cμSMALL (x1,t) . μSMALL(x2,t)

(6.60)

The intrinsic growth rate of μL (x1, t) and μL (x2, t) in equations (6.58) and (6.60) have been presented to be equal to ‘a’ to keep the results bias-free. Stability Analysis Abbreviating L for LARGE, N for NEGATIVE, along with S for SMALL we consider the following Lyapunov energy function for stability analysis of the above dynamics

6.6 The Fuzzy Temporal Representation

193

Let V be a Lyapunov energy function given by V(μL(x1,t+1),μL (x2, t+1)) =(bμL(x2, t+1) − cμL (x1, t+1))2

(6.61) (6.62)

= (-ac μL ( x1, t) × μNEG ( x1−k, t) + a b μL (x2, t))2 ΔV = V(μL ( x1, t+1) , μL (x2, t+1)) − V(μL ( x1, t) , μL (x2, t)) = (ab μL (x2, t) − ac μL ( x1, t) × μNEG ( x1−k, t))2 − (b μL (x2,t) − cμL(x1,t))2

(6.63)

< 0 if, ab μL (x2, t) − ac μL ( x1, t) × μNEG ( x1−k, t) < b μL ( x2, t) – cμL (x1, t) ⇒

( a − 1) b μL ( x2, t) < cμL (x1, t) (aμNEG ( x1− k, t) – 1)

⇒

μL (x2, t)/ μL ( x1, t) < (c/ b (a−1)) (aμNEG(x1–k, t) –1)

(6.64)

If μNEG (x1 - k, t) = 1 ( i.e., x1 (t) – k < 0 is certain) Then μL (x2, t)/ μL ( x1, t) < c (a−1)/ b (a−1) ⇒ μL(x2,t)/μL(x1, t)< c/b

(6.65)

To take into account the effect of personality, we can re-structure the dynamics represented by equations (6.59) and (6.60). Let p be a personality index, where p > 1 corresponds to strong personality, p < 1 corresponds to weak personality, and p = 1 corresponds to normal personality. The effect of personality appears in the dynamics to represent the contact between x1 (t) and x2 (t). One simple way of modeling the influence of personality in the dynamics is to attach p as a power of x1(t). x2(t). Thus equations (6.59) and (6.60) can be re-structured as μL (x1,t +1) = aμL (x1,t) × μNEG (x1−k, t) − b (μS (x1, t) .μS (x2, t))p

(6.66)

μL(x2,t+1) = aμL(x2, t) −c(μS(x1, t) . μS (x2, t))p

(6.67)

194

6 Control of Mental Stability in Emotion-Logic Interactive Dynamics

The stability analysis of (6.66) and (6.67) can be done using the same Lyapunov function introduced above.

6.7 Stabilization of Emotional Dynamics The differential equation (model 6.35) representing the emotional dynamics is not always stable. This is also apparent from the condition of stability (6.44), obtained from Lyapunov stability analysis of the emotional dynamics. The condition envisages: dxi 0<

< aii xi ( 2xi/ k – 1)

(6.44 re-written)

dt Naturally, when the dynamics fails to satisfy the above inequality, we need to adapt the parameters of the dynamics. It is clear from the emotional dynamics (6.35) that growth rate in xi (i.e., dxi/dt) can be controlled by increasing the negative feedback (Fig 6.1(a)) cki to state xi, or by decreasing positive feedback (bji). “cki” can be adapted by brute-force method as humans have a natural tendency to block negative forces. The positive feedbacks to the emotional state xi through weights bji can also be reduced to control the growth rate in xi. Since humans have a self-bias to positive feedback inputs, instead of brute-force method, we allow an unsupervised learning cycle to adapt the positive feedback co-efficient bji. To control dxi/dt in Fig. 6.3, we first use a comparator to determine the error signal e. Here error is defined as e=aiixi (2xi/k–1) − dxi/dt.

(6.68)

When this error is negative, we increase cki by cki ← cki + β |e|, β > 0.

(6.69)

This is termed as brute-force adaptation of cki. Further, to adjust bji, we decrease α as a linear function of error by (6.70), and employ Hebbian learning as in (6.71): α ← α - k | e |, k > 0

(6.70)

δbji = − α bji + f (xi). f (xj)

(6.71)

bji ← bji − δbji .

(6.72)

This control policy prohibits the growth rate of dxi/dt within a prescribed limit of aii xi (2xi/k – 1).

6.8 Psychological Stability in Emotion-Logic Counter-Actions

195

aiixi (2xi/k – 1) Xi (0)

−

Xi (t)

+

D Xj (0)

Emotional dynamics

Xk (0)

Xj (t)

Xk (t)

cki

error e comparator error e

bji If e < 0 then

cki ← cki + β | e | yes

α←α-k|e|

δbji = - α bji + f (xi). f (xj) bji ← bji - δbji

Fig. 6.3. Stabilization of Emotional Dynamics by a Learning Controller.

6.8 Psychological Stability in Emotion-Logic Counter-Actions Let xi(0), xj(0), …., xk(0) be the emotional states supplied to the input of a controlled emotional dynamics and fuzzy temporal reasoning system (Fig. 6.4). It is preferred that human should perform actions based on their controlled emotional states. Let y1, y2, …., yn be a set of actions to be executed by a person. Actions can also be planned by the human beings from the emotional states: x1(0), x2(0), …, xk(0) using fuzzy temporal reasoning. Let the actions generated by fuzzy temporal

196

6 Control of Mental Stability in Emotion-Logic Interactive Dynamics y/ 1

Fuzzy temporal

y/ 2

y/ k

………

x1(0)

− Controlled Emotional System

G

xk(0)

+

+

+

−

y

−

1 y 2 y k

e1

e2 ek

Controller

Fig. 6.4. Stabilization of emotion by fuzzy temporal reasoning.

reasoning unit be y1/, y2/, …, yn/. In order to match yj/ with yj for j= 1 to n, we first take their difference and call it an error ej. Formally, ej=yj/−yj .

(6.73)

In order to attain psychological equilibrium, humans attempt to reduce the sum of squared errors, called Min Square Error (MSE). Em2 = e12 + e22 + …. + en2.

(6.74)

One simple way to minimize Em2 is to adapt the gain matrix G, where G= [gji](k × n) . [y1y2…..yn] =[xi(t) xj (t) …. xk (t)] [gij](k×n).

(6.75) (6.76)

Appropriate selection of the gain matrix G can reduce the Min square error without violating the stability condition of the controlled emotional dynamics. The controller in the present context could be a proportional controller where,

6.9 Conclusions

197

gii = kii × ei

(6.74)

gij = 0 for i≠j.

(6.75)

and, Assuming that the [y1 y2 …. yn] is generated from state [xi(t) xj(t) ….xk(t)] by the following formula, we find the above proportional control scheme sufficient to adjust the amplitude, and the sign of the state variables xi, xj and xk to reduce the mean square error Em towards zero. Naturally, it is clear from expression (6.76) that [y1 y2 …yn] are scaled values of [xi xj xk]. In case the [y1 y2 ….yn] are different from state variables [xi xj xk], we may consider a linear contribution of xi, xj, xk to compute yr for r=1, 2, …, n. It may be noted that in the second configuration, we can employ Genetic Algorithm (GA) to reduce the Min Square Error towards zero. For realization of the GA, we restructure the matrix G by the chromosome where the fields of the chromosome carry the value of gij. The Min Square norm may be used as the fitness function of GA, so as to select better offspring from the population in each genetic iteration. The typical crossover operation and the 9’s complement mutation operator may be invoked to execute the GA in the present application. The typical Roulette Wheel algorithm may also be utilized to select offspring for the crossover operation.

6.9 Conclusions The chapter introduced a new technique for controlling emotional stability by Hebbian type machine learning. It also presented a new scheme for controlling psychological stability of the human mind in presence of emotion-logic dynamic interactions. The condition for stability of the emotional dynamics and its maintenance through autonomous learning provides an insight to the emotion-causing and controlling mechanism of the human mind. A judicious selection of the parameters in the emotional dynamics with reference to a real human mind is under investigation. Typical model reference approach used in cybernetic theory may be adapted to determine the parameters of the emotional dynamics. A series of experiments is needed to determine the learning rates in the Hebbian type machine learning paradigm. This too is under investigation and within the coming few years, we expect to identify the system parameters for both emotion and learning dynamics. Though perceptions and actions reigned for more than a few decades in the dynasty of Robotics and machine intelligence, there is hardly any reference to generate control action in presence of emotion-logic counter actions. The chapter attempted a humble approach to minimize the differences between the emotional and reasoning states, so as to arrive at a definite conclusion or generate suitable control actions. The scheme undertaken in the chapter attempts to control the parameters of the G matrix using a proportional type controller. Several improvements in the basic control scheme can be attained by suitable modification in the design of the controller.

198

6 Control of Mental Stability in Emotion-Logic Interactive Dynamics

The most important achievement of the present work lies in representing logic by time differential/difference equations. Secondly, the chapter demonstrates a novel approach to determine the stability of temporal logic using the general theory of Lyapunov energy function. Thirdly, the dynamics of the emotion has been modeled with coupled differential equations. Finally, the proposed scheme for controlling stabilized emotional dynamics by stabilized fuzzy temporal system is also unique.

Exercises 1.

Find the stable interpretations for the following 2 logical statements ¬q→¬p p∨ q [Hints: The interpretations of the 1st statement are given by (p, q) ∈ {(1, 1), (0, 0), (0, 1)} The interpretations of the 2nd statement is given by (p, q) ∈ {(1, 1), (1, 0 ), (0, 1)}. The common interpretation of the two statements thus is (p, q) ∈ (1, 1), (0,0).]

2.

Consider the following 2 time differential equations dp/dt = dq/dt =

p (t) → q (t) q (t) → p (t)

where p and q are 2 propositions and t denotes the time. Test whether the above dynamic system has any stable points. [Hints: The stable points can be determined by setting dp/dt = 0 and dq/dt = 0. dp/dt = 0 reveals p (t) → q (t) = 0 which ultimately yields ¬ p (t) ∨ q (t) = 0 ⇒ p (t) = 1 and q (t) = 0 dq/ dt = 0 implies q (t) → p (t) = 0 which finally yields

Exercises

199

¬ q (t) ∨ p (t) = 0 ⇒ q (t) = 1 and p (t) = 0 Since the 2 conditions yield no common solution, the dynamic system does not have any stable points.] 3. The dynamics of a discrete propositional temporal system is given below: p (t+1) = p(t) ∧ (p (t) → q(t)) and q (t+1) = p (t)∨ q (t) Find the stable point if any of the above dynamical system. [Hints: Let p (t) = p* and q (t) = q* be the stable point. Then substituting p(t+1) = p(t) = p* and q (t+1) = q (t) =q* in the above dynamical system, we obtain p* = p* ∧ (p* → q*) (1) (2) and q* = p* ∨ q* From (1) we have: p* = p* ∧ (¬ p* ∨ q* ) = (p* ∧ ¬ p* ) ∨ (p* ∧ q* ) = 0 ∨ (p* ∧ q* ) = p* ∧ q* Now, p* = p* ∧ q* yields q* = 1. From (2) we obtain p* = 0. Therefore the given dynamics has a single stable point: (p*, q*) = (0, 1)]. 4. Given dx/dt = − ax, and dy/dt = − by for a > 0 and b > 0. Considering V (x, y ) =x2 + y 2 as the Lyapunov energy function that the given dynamical system is stable. [Hints: dv/dt = (dv/dx) . (dx/dt) + (dv/dy) . (dy/dt) = 2x (− ax) + 2y (− by) = −2ax2 – 2by2 = − 2 ( ax2 + by2) <0, since a > 0 and b > 0. Since dv/dt is negative, the given dynamics is stable]. 5. x (t+1) = − ax (t) and y (t+1) = − by (t) for a >0 and b > 0. Assuming L (x, y) = x2 + y2, show that the above discrete dynamics is conditionally stable, and hence find the parametric range of a and b for stability.

200

6 Control of Mental Stability in Emotion-Logic Interactive Dynamics

[Hints:

ΔL = L (t+1) − L (t) = L(x (t+1), y (t+1)) − L (x (t), y (t)) = {x2 (t+1) + y 2 (t+1)} − {x2(t) + y2(t)} = a2x2 (t) + b2y2(t) − x2(t) − y2(t) = x2 (t) (a2 − 1) + y2 (t) (b2 −1) = − {x2(t) ( 1- a2) + y2(t) (1− b2)} < 0, if, 1- a2 > 0 and 1- b2 > 0 ⇒

a < 1 and b < 1

(1)

Given that a> 0 and b> 0

(2)

Combining 1 and 2 we obtain: 0 < a< 1 and 0< b<1]. 6. Given p (t+1) = q (t) → p (t) and q (t+1) = p (t) → q(t). Assuming p → q = ¬ p s q = ¬ p + q − (¬p)q and considering L (p, q) = p + q as the Lyapunov Energy Function, test the stability of the system. [Hints: p (t+1) = q (t) → p (t) = ¬ q (t) + p (t) − (¬ q ( t)) p (t) ΔL = L [p (t+1), q (t+1)] − L [p (t) + q (t)] = [p (t+1) + q (t+1)] − [p (t) + q( t)] = [{q (t) → p (t)} + {p (t) →q (t)}] − [p (t) + q (t)] = [{¬ q (t) + p (t) − (¬ q (t)) . p (t)} + {¬ p (t) + q (t) − (¬ p (t)) . q (t)}] − p (t) − q (t) = ¬ q (t) − (¬ q (t) p (t)) + ( ¬ p (t)) − (¬ p (t) q (t)) = ¬ q (t) (1 – p (t)) + ¬ p (t) (1− q (t)) = (1 − q (t)). (1 − p (t)) + (1− p (t)) (1 − q (t)) = 2 (1 − p (t)) (1− q (t))

Exercises

201

which is always greater than 0 as 0 < p (t) < 1 and 0 < q (t) < 1. Therefore, given dynamics is unstable.] 7. Given that dxi/dt = aii xi ( 1- xi/k) + bji xi xj − cki xi xk, for i = 1 to n be a set of n differential equations, representing the emotional dynamics of a person. Select a suitable Lyapunov energy function to test the stability of the dynamical system. xi xi [Hints: Let L = (− aii k/2) xi (1 − xi/k)2 + ∫ bji xj xi dxi − ∫ cki xi xk dxi 0 0 be the Lyapunov Energy Function. Now, dL/dt = (∂L/∂xi ) (dxi/dt) = {(−aiik/2) (1− xi/k)2 + (−aiik/2) xi . ( 2 ) (1− xi/k) (− 1/k) + bji xi xj − cki xi xk} dxi/dt = [ dxi/dt − aiik/2 (1− xi/k)2] (dxi /dt) < 0. If dxi/dt − aiik/2 (1− xi/k)2 < 0 and dxi /dt > 0. Combining the above two conditions, we obtain: 0 < dxi /dt < aiik/2 (1− xi/k)2. This is the condition for stability of the dynamics.] 8. In the fuzzy learning systems, the weights have the following dynamics: dωij /dt = − αωij + f (xi) f (xj) where f (xi) = 1/(1+ exp (-xi)), 0 ≤ xi ≤ 1 show that at steady-state, ω*ij lies between (1/α) (1/ (1+ e−1))2 ≤ ω*ij ≤ 1/4α. [Hints: At steady-state, dωij /dt = 0. ∴

α ωij = f (xi) f (xj)

(1)

Denoting steady-state value of ωij , xi and xj by ω*ij xi* and xj*, we obtain: α ω*ij = f (xi*) f (xj*) When xi* = xj* = 0 , f (xi*) = f (xj*) = 1/ 4,

(2)

202

6 Control of Mental Stability in Emotion-Logic Interactive Dynamics

When xi* = xj* = 1, f (xi*) f (xj*) = {(e/(1+ e)}2 ∴ (1/ α) {e/(1+ e)}2 ≤ ω*ij ≤ 1/4α.]

9. Given the dynamical system involving two emotional states x1 and x2: dx1/dt = ax1(1− x1/k) − b(x1 x2) dx2/dt = − c(x1x2) + dx2 Determine the equilibrium value of x1 and x2 from the given emotional dynamics. [Hints: At equilibrium dx1/dt = dx2/dt = 0. Denoting steady-state values of x1 and x2 by x1* an x2* respectively, we obtain: a.x1*(1− x1*/k) = b x1* x2* and cx1*x2*=dx2*

(1) (2)

From (2) we obtain, x1* = d/c ,

(3)

and from (1) we obtain: a ( 1− x1*/k) = b x2* .

(4)

Substituting x1* = d/c in (4) we obtain: x2* = (a/b) (1− d/ck). Thus the steady state value of x1 and x2 are: (a/b) ( 1− d/ck) and (d/c) respectively.] 10. Assuming the steady-state value of x1 and x2 be x1*= d/c, and x2*= (a/b)(1−d/ck) in the dynamics given in problem 9, linearize the dynamics around the equilibrium level of x1= x1*, x2 = x2*, and k = k*, and represent the linearized system by a vector-matrix equation. [Hints: Let n1 = x1 − x1*, n2 = x2 − x2*, n3 = k − k*, the linearized dynamics is then given by:

Exercises

dn1/dt =

A11

n1(t) + x1*, x2*, k*

dn2/dt =

A21

n1(t) + x1*, x2*, k*

where, A11 = ∂ G1/ ∂x1 = ∂/∂x1 [ax1 ( 1− x1/k) − b x1 x2] = ∂/∂x1 [ax1_− ax12/k− b x1 x2] = [a − 2ax1/k − bx2] A12 = ∂ G1/ ∂x2 = ∂/∂x2 [ax1_− ax12/k− b x1 x2] =[0 − 0 − b x1] = − b x1 . A13 = ∂ G1/ ∂k = ∂/∂k [ax1_− ax12/k −b x1 x2] = [0 + ax12/k2] = ax12/k 2. A21 = ∂ G2/ ∂x1 = ∂/∂x1 [− c x1 x2 + x2] = − c x2. A22 = ∂ G2/ ∂x2 = ∂/∂x2 [− c x1 x2 + dx2] = − c x1 + d.

A12

A22

n2 (t) + A13 x1*, x2*, k*

203

n3(t) x1*, x2*, k*

n2 (t) + A23 n3(t) x1*, x2*, k* x1*, x2*, k*

204

6 Control of Mental Stability in Emotion-Logic Interactive Dynamics

A23 = ∂G3/∂k = ∂/∂k [− c x1 x2 + dx2] = 0. A11 =

x1*, x2*, k* ( a – 2ax1/k − bx2)

= a – 2ax1 / k − bx2 *

x1*, x2*, k*

*

= a − (2a/k) (d/c) – b. (a/b) ( 1− d/ck) = a − 2ad/kc − a + ad/ck = − ad/kc . A12

x1*, x2*, k*

= [(− b x1)] x1* = − b x1* = − bd/c . A13

x1*, x2*, k*

= (a x12/k2)

x1*

= ax1*2/ k 2 = a/k2 ( d/ c)2 = ac2/ d2k2 A21

x1*, x2*, k*

= (−c x2)

x2*

Exercises

205

= ( −c x2*) = − ac/b ( 1− d/ck) . A22

x1*, x2*, x3*

= ( −c x1* + d)

x1*, x2*, k*

= −c x1* + d = −c (d /c) + d =0. A23

x1*, x2*, x3*

=0 The linearized dynamics is now given by dx1/ dt

A11

dx2/ dt

A12

A13

n1 (t)

=

dk/ dt

n2 (t)

A21

A22

A23

n3 (t)

x* 1 x* 2 x* 3

− (ad/ kc) − ( bd/ c)

ad2/c2k2

=

n1 (t) n2 (t)

−(ac/b) (1− d/ ck)

0

0

n3 (t) ]

206

6 Control of Mental Stability in Emotion-Logic Interactive Dynamics

11. Let x3(t) be the control input in the emotional dynamics that regulates the shift of emotional variables x1 and x2 from their equilibrium level x1* and x*2 respectively. Let the shift: x1 − x*1 = n1, x2 − x2* = n2., and x3 –x3*=n3. Given the following dynamics that may be used for the above control purpose. dn1/dt = a11n1 (t) + a12 n2 (t) + a13 n3(t) dn2/dt = a21n1 (t) + a22 n2 (t) + a23 n3(t) where, aij for i = 1 to 3 and j = 1 to 3 are evaluated following problem (9) and (10). Determine the transfer function | n1 (jω)/n3 (jω)| and | n2 (jω)/ n3(jω)|. [Hints: Taking Laplace transform on both sides of the given dynamics: s n1 (s) = a11 n1 ( s) + a12 n2 (s) + a13 n3 ( s)

(1)

s n2 (s) = a21 n1 (s) + a22 n2 (s) + a23 n3 (s)

(2)

Simplifying (1) and (2) (s − a11) n1(s) − a12 n2 (s) − a13 n3 (s) = 0

(3)

− a21 n1 (s) + ( s− a22) n2 (s) − a23 n3 (s) = 0

(4)

Cross-multiplying (3) and (4) n1 (s)

n2 (s)

= (−a12) (− a23) − (− a13) (s− a22)

(s-a11) ( −a23) − (− a13) (− a21)

(s− a11) (s −a22) − (−a12) (−a21)

a12 . a23 + a13 (s − a22)

n1 (s) =

(5) (s − a11) (s − a22) − a12. a21

n3 (s)

− a23 (s − a11)− a13 . a21

n2 (s) =

(6) (s − a11) (s − a22) − a12. a21

n3 (s)

Substituting s = j ω in (5) and (6) we have: (a12 . a23 − a13 a22) + jω a13

n1 (jω) = n3 (jω)

n3(s)

=

(jω− a11) (jω − a22) − a12. a21

References

207

(a12 a23 − a13 a22) + jω a13 = −ω2 + a11.a12 − jω (a11 + a22) − a12.a21 (a12 a23 − a13 a22) + jω a13 = (a11.a22 − a12 a21 −ω2) − jω (a11. a22)

√

n1 (jω) = n3 (jω)

√

(a12. a23 − a13 . a22)2 + ω2 a213 (a11.a22 − a12 a21 −ω2)2 + ω2 (a11 + a22 )2

Similarly, we can evaluate | n2 (jω) / n3 (jω) |. ]

References 1. Ben- Ari, M.: Mathematical Logic for Computer Science, pp. 200–241. Prentice–Hall, Englewood Cliffs (1993) 2. Besnard, P.: An Introduction to Default Logic, pp. 27–35, 163–177. Springer, Berlin (1989) 3. Bolc, L., Szalas, A. (eds.): Time and Logic: A Computational Approach, pp. 1–50. UCL Press, London (1995) 4. Konar, A.: Artificial Intelligence and Soft Computing: Behavioral Cognitive Modeling of the Human Brain. RC Press, Boca Raton (1999) 5. Konar, A., Pal, S.: Modeling cognition with fuzzy neural nets. In: Leondes, C.T. (ed.) Fuzzy Logic Theory: Techniques and Applications. Academic Press, New York (1999) 6. Kosko, B.: Fuzzy Cognitive Maps. Int. J. of Man-Machine Studies 24, 65–75 (1986) 7. Kuo, B.C.: Digital Control Systems. In: Holt-Saunders International Editions, Japan, ch. 5, pp. 267–302 (1981) 8. Pal, S., Konar, A.: Cognitive Reasoning with fuzzy neural nets. IEEE Trans on Systems, Man and Cybernetics, Part B (August 1996) 9. Soanes, C. (ed.): The Compact Oxford Reference Dictionary. Oxford University Press, New York (2004)

7 Multiple Emotions and Their Chaotic Dynamics

The chapter addresses the issues of co-operation/competition of multiple emotional states in the human brain, and realizes the same using a recurrent neural dynamics. It also deals with stability analysis of dynamic emotional systems using Lyapunov energy function. Computer simulation of the proposed model reveals that the dynamic behavior of an emotional system exhibits limit cycles, chaos or steady-state, depending on the selection of appropriate parameters in the dynamics. Phase-portraits have been used, whenever necessary, for analysis of the dynamical behavior of mixed emotional systems. The chapter ends with a study on parameter selection of emotional dynamics from the facial expressions of subjects, stimulated with audio-visual movies. Finally a scheme for stabilization of emotional dynamics is proposed with a proportional plus integral controller.

7.1 Introduction In Chapter 2, we presented the fundamentals of dynamical systems, with special references to stability, instability and chaotic behavior of a physical system. Special emphasis was given to Lyapunov energy function to analyze the behavioral dynamics of both discrete and continuous systems. An emotional system is also a dynamical system, consisting of several emotional states, such as sadness, anger, anxiety, fear and happiness. One approach to emulate the emotional system of a person is to construct a generic model of mixed emotional dynamics [5], [6], and then employ system identification techniques to determine the parameters of the dynamics with reference to the emotion-arousal process of the given subject. Researchers [10], [11] are keen to use neuronal dynamics to model complex emotional systems [3]. These neuronal dynamics [14], [15] include positive feedback, indicating self-growth and negative feedback indicating decay in a neuronal state through counter-interactions with other neuronal states. Lee oscillator, for example, is one such oscillatory system that includes both the above types of feedbacks. In this chapter, one such neuronal model of mixed emotion will be proposed, and dynamics of such system will be analyzed using Lyapunov energy function. The chapter proposes a new approach to determine the parameters of emotional dynamics from the facial expressions of the subjects, stimulated by audio-visual movies. Selected audio-visual movies are used to excite stimulation for arousal of specific emotions, and the facial expression changes are recorded to determine A. Chakraborty and A. Konar: Emotional Intelligence, SCI 234, pp. 209–233. © Springer-Verlag Berlin Heidelberg 2009 springerlink.com

210

7 Multiple Emotions and Their Chaotic Dynamics

primary features of emotions, including mouth-opening, eye-opening and eyebrows’ constriction. The measure of the primary features is used to finally determine the parameters of the emotional dynamics. The chapter also provides a control policy to block chaos and oscillations in the dynamics of a nonlinear emotional system. The control algorithm attempts to stabilize the dynamic behavior by adjusting the intrinsic growth rate of the emotional dynamics. Computer simulations indicate that the said control policy blocks oscillations/chaos by changing the damping level of the dynamic system. An underdamped dynamics yields oscillations, whereas a damped dynamics can block oscillations [12]. The selection of the damping level of the dynamics has been realized with a proportional plus integral type controller. The chapter is divided into 6 sections. Section 7.2 introduces a model for chaotic emotional dynamics. In section 7.3, we studied the effect of variation in parameters of the emotional dynamics. Chaotic fluctuation in emotional states based on the dynamics presented earlier is outlined in section 7.4. A stability analysis of the proposed emotional dynamics by the Lyapunov energy function is presented in section 7.5. An experimental set up to determine the parameters of emotional dynamics for a given subject is presented in section 7.6. A novel scheme for stabilization of the mixed emotional dynamics is undertaken in section 7.7. Conclusions are listed in section 7.8.

7.2 Proposed Model for Chaotic Emotional Dynamics There exist two types of emotions: simple and complex. A simple emotion includes only one emotional state, while a complex emotion includes arousal of more than one concurrent emotional state. The latter has been described in this chapter by a competitive/co-operative emotional dynamics. Main emphasis of the chapter is given on the growth of an emotional state in the influence of co-operation and competition of other emotional states [5]. There exists work on feedback models of emotional dynamics, cited in [2], but the approach undertaken here is new and unknown to the domain of emotional intelligence community. Stimulated brain imaging [6] by electroencephalography (EEG) [7], undertaken in [8] supports the response of the mathematical model of emotional dynamics presented in chapter. Details of this are given in Chapter 9. Let xi, xj and xk be three representative emotional states that describe the concentration of individual emotions. Suppose xj for some j co-operates with xi, while xk, for some k, competes with xi. In other words, the growth rate of xi will be accelerated with an increase in xj, but will be decelerated with an increase in xk. Assuming that there exist m number of co-operative emotional states like xj and n number of competitive emotional states like xk, we can represent the dynamic behavior of emotional state xi by the following differential equation. dxi/dt = aii xi (1- xi/K) + Σbji xi (1- exp(-βji xj)) – Σcki xi (1-exp(-λki xk)) (7.1) ∃j

∃k

7.2 Proposed Model for Chaotic Emotional Dynamics

211

Fig. 7.1. The state transition diagram of emotional dynamics of a thalasemia patient.

The 1st term in the R.H.S. of the above equation corresponds to self-growth of emotional state xi. Here, aii denotes the inertial co-efficient that regulates the selfgrowth of xi. The factor (1- xi/K) is a controlling term that selects the sign of intrinsic growth rate aii. For instance, when xi< K, the first term in the right hand side is positive, when xi =K it becomes zero, and when xi> K, it becomes negative. In other words, xi is allowed to increase up to a level of K, and a fall-off in the growth rate in xi starts once it exceeds K. The second term represents the cooperation between emotion xj and xi for some j. It is indeed important to note that the second term takes into account the co-operation of xi with a growing xj. The third term on the other hand represents competition of xi with growing xk for some k. The parameters: βji and λki control the growth of xj and xk respectively in the growth dynamics of xi. To illustrate the use of the proposed dynamics in emotion modeling, we consider the emotional states of a thalasemia patient’s mother (Fig. 7.1). The dynamics of the system shown in Fig.1.is presented here by equations (7.2) – (7.5). dx1/dt= a11x1(1-x1/K) + b31x1(1-exp(-β31x3)) – c21 x1(1-exp(-λ21x2)

(7.2)

dx2/dt = a22 x2 (1- x2/K) + b12 x2 (1- exp (-β12 x1)) + b32 x2 (1-exp(-β32 x3)) + b42 x2 (1- exp (-β42x4)) – c32 x2(1-exp(-λ32x3) (7.3) dx3/dt = a33 x3 (1- x3/K) + b23 x3 (1- exp (-β23 x2)) – c23 x3 (1-exp(-λ23 x2)) – c13x3 (7.4) (1- exp (-λ13 x1)) – c43 x3 (1-exp(λ5x4)) dx4/dt = a44 x4 (1- x4/K) + b34 x4 (1- exp (-β34 x3)) – c24 x4 (1-exp(-λ24 x2)) (7.5) where the parameters have their usual meaning as discussed above.

212

7 Multiple Emotions and Their Chaotic Dynamics

7.3 Effect of Variation in Parameters of the Emotional Dynamics In this section, we study the effect of variation in the parameters of the dynamics in the emotional response. We fix the parameters as listed in Tables 7.1 through 7.5, and report the effect of changing the parameters in the appropriate sections. Table 7.1. List of parameters of equation (7.2)

a11 0.33

b31 0.5

β1 0.006

c12 0.55

λ1 0.0033

Table 7.2. List of parameters of equation (7.3)

a22 0.002

b12 0.2

b32 0.2

b42 0.3

β12 0.001

c32 0.4

β32 0.001

β42 0.001

λ32 0.005

Table 7.3. List of parameters of equation (7.4)

a33 0.2

b23 0.4

c23 0.5

c13 0.45

β23 0.005

c43 0.3

λ23 0.005

λ14 0.005

λ15 0.005

Table 7.4. List of parameters of equation (7.5)

a44 0.1

b34 0.4

c24 0.35

β34 0.001

λ24 0.001

Table 7.5. Initial values of the parameters

X1(0) 500

X2(0) 800

X3(0) 900

X4(0) 700

7.3.1 Variation in aii The aii plays an important role in the stabilization of the said emotional dynamics. It has been noted that keeping the other parameters of the dynamics constant, a large variation in a11 forces the system to a limit cyclic or chaotic situations. This observation, of course, is apparent from the dynamics. It is to be noted that an increase in a11 causes a positive feedback in x1 through the first dynamics (7.2). The rapid growth in x1 induces a growth in x2 following equation (7.3). When c13 in

7.3 Effect of Variation in Parameters of the Emotional Dynamics

213

Eqn. (7.4) is small, the growth in x1 cannot influence x3 significantly. Thus a large a11 pushes the dynamics towards chaos/limit cycles. On the other hand, when a22 or a33 increases, stabilization of the system is hampered. Further an increase in a44 increases the stability of the overall dynamics. The effect of changes in a22, a33 and a44 on the dynamics, as discussed above, can be explained in a manner similar to the explanation of a11. Fig. 7.2(a), (b), and, (c) illustrate the phenomena of chaotic, limit cyclic, stable behavior of the dynamics due to variation in a11.

(a) 1400 x1 x2 x3 x4

1200

x(i)----------------------->

1000

800

600

400

200

0

0

100

200

300

400

500 t----------------->

600

700

800

900

1000

(b)

Fig. 7.2. Chaotic, limit cyclic and stable behavior in the emotional dynamics (7.2−7.5) for varying a11.

214

7 Multiple Emotions and Their Chaotic Dynamics 1200 x1 x2 x3 x4 1000

800

600

400

200

0

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

(c)

Fig. 7.2. (continued) 1200 data 1 data 2 data 3 data 4

1000

800

600

400

200

0

-200

0

100

200

300

400

500

600

700

800

900

1000

Fig. 7.3. Chaotic variation of all the emotional states due to increase in c12..

7.3.2 Variation in cij Decreasing the value of c21 from 0.55 reduces oscillatory nature of the dynamics. When c21 attains 0.35, there exists an overshoot in the under-damped case, but oscillations die out. Apparently, this seems to be counter-intuitive, as a decrease in c21 causes a relative increase in the growth rate of x1, causing over-arousal of emotion x1. In fact, the same thing happens in the transient phase, but in the steadystate phase, the increased growth rate of x1 suppresses the arousal of x2, x3 and x4 states. Consequently, decreasing c21 stabilizes the behavior of all the emotional

7.3 Effect of Variation in Parameters of the Emotional Dynamics

215

states. It has been noted that when c21 is decreased to 0.01, all the states return to the equilibrium state. An increase in c21 pushes the dynamics towards limit cyclic behavior. An overincrease in c21 causes a large negative feedback to the growth rate in x1, causing x1 to gradually diminishing towards zero. It is, however, noteworthy that even when x1 stabilizes to zero level, x2, x3, x3 and x4 continue to demonstrate limit cyclic behavior (Fig. 7.3). 7.3.3 Variation of bij Computer simulation reveals that a decrease in b31 below 0.5, keeping all other parameters fixed, retains the oscillatory nature of the dynamics. When b31 is increased up to 0.9, the oscillation in all the emotional states, x1, x2, x3, x4 are sustained. When b31 is increased further to 1.0, the oscillation die out, and all the emotional states attain equilibrium. The phenomena for sustaining oscillation at small values of bij and maintaining equilibrium at higher value than bij are explained as follows. When b31 is small (≤ 0.9), the positive feedback provided by the emotional states x3 to x1 is inadequate to maintain large amplitude in x1 because of a large negative feedback from other emotional states. It is apparent that with a large initial value in x2, the emotion level of x1 has a significant phase delay of around 180o, causing x1 to oscillate with low amplitude (around 500 unit). Naturally, the dynamics of x3 and x4, which are strongly influenced by x2 have similar behavior like x2. When b31 approaches 1, the dynamics of x2 settles down with small amplitude of oscillation around 830 units, causing x1, x3 and x4 to maintain low equilibrium levels depending on their feedback co-efficient from x2 to x1, x3 and x4 (Fig. 7.4). The effect of variations of aii, bij, cij for other values of i and j can be envisaged in a similar manner as discussed above.

(a) b31 =0.5 Fig. 7.4. The effect of variation of b31 on the dynamics.

216

7 Multiple Emotions and Their Chaotic Dynamics

(b) b31=0.9

(c ) b31=0.9

(d) b31=1.1 Fig. 7.4. (continued)

7.4 Chaotic Fluctuation in Emotional State

217

(e) b31=1.1 Fig. 7.4. (continued)

7.4 Chaotic Fluctuation in Emotional State The dynamics of emotion exhibit stable, limit cyclic and chaotic behavior depending on the parameter set of the dynamics. When the intrinsic growth rate for state x2, i.e. a22, is decreased significantly, the dynamics describes a chaotic behavior in all the emotional states. Fig. 7.5 (a), (b), (c), (d), describe the chaotic variations of individual emotional states. One simple way to represent chaotic dynamics is to draw a phase trajectory of 2 or 3 variables, when all of which have a chaotic variation over time. Fig. 7.6 describes a phase trajectory of 2 emotional states. The phase portrait in the present context exhibits several annular rings of gradually changing dimensions. Such diagram usually provides a clear indication of chaos.

(a)x1

Fig. 7.5. Chaotic temporal variation of all the emotional states.

218

7 Multiple Emotions and Their Chaotic Dynamics 1800

1600

1400

x2------------->

1200

1000

800

600

400

200

0

4

4.1

4.2

4.3

4.4

4.5 t---------->

4.6

(b) x2

(c) x3

(d) x4

Fig. 7.5. (continued)

4.7

4.8

4.9

5 4

x 10

7.5 Stability Analysis of the Proposed Emotional Dynamics

219

1800 1600 1400 1200 1000 800 600 400 200 0 0

200

400

600

800

1000

1200

1400

1600

Fig. 7.6. Phase trajectory for chaotic variation of states: Sad (x2) versus Anxious (x2). 7.5 Sta bility A nalys is of the Pro pose d Emotiona l Dy na mics

7.5 Stability Analysis of the Proposed Emotional Dynamics by Lyapunov Energy Function Theorem 7.1: The following dynamics is unconditionally stable and L (xi, xj) is the Lyapunov function to prove its asymptotic stability. n dxi = a x (1− x /K) − Σ b x (1− e −βji xj ) ij i j ij i dt j=1

for i = 1 to n.

(7.6)

aii x 3) − Σ ∫ ib x (1− e−βjixj)dx x2 i ij i j − (aii 2i − 3K j=1 0

(7.7)

Proof: Let n

L (xi, xj) =

x

be the Lyapunov energy function for the given dynamics. To prove that L (xi, xj) is a Lyapunov energy function, we verify that 1.

L(0, 0)=0,

2.

L (xi, xj)

3.

∂L ∂L and ∂x j ∂xi

> 0

xi ≠ 0 xj ≠ 0

both exist,

220

7 Multiple Emotions and Their Chaotic Dynamics

Consequently, L (xi, xj) is a Lyapunov energy function. Now, to show that dL/dt is unconditionally negative we evaluate: n ∂L dxi Σ dx j=1 i dt n 2 n = -Σ aijxi (1− xj/k) − Σ bijxi (1− e−βjxj ) j=1 j=1

dL = dt

<0

(7.8)

Since dL/dt is negative, the dynamics is unconditionally stable.

□

When dxi/dt conatins co-operative terms like Cki xi (1- exp (- λki xk)), for some n, the modified dynamics given by (7.1) can also be proved unconditionally stable by Theorem 7.2. Thorem 7.2: L (xi, xj, xk) given below be the Lyapunov function for the dynamics given by equation (7.1). L (xi, xj, xk)

m xi -αkxk -βjxj n xi ) dxj + ∑ ∫ cki xi ( 1- e )dxk (7.9) = -[ ( aii x2 – (aii/3k) xi3) -∑ ∫bij xi ( 1- e j=1 0 k=1 0 Dynamics (7.1) is unconditionally stable using the above Lyapunov function. Proof: The proof can be completed analogous to the proof of Theorem 7.1.

□

7.6 Parameter Selection of the Emotional Dynamics by Experiments with Audio-Visual Stimulus 7.6 Para meter Selection of the Emotional Dy na mics

The experiment deals with excitation of subjects by audio-visual stimulus for arousal of emotions [9]. In order to arouse mixed emotions, we need to submit time-staggered audio-visual stimulus causing arousal of different emotions. One way to realize this is to excite subjects with one base emotion, super-imposed with other relatively short-duration emotions. For instance, a base emotion of happiness, when perturbed by a burst of disgust and fear in time-succession, represents a co-existence of multiple emotions for a certain time frame. After the mixed emotions are synthesized, three important facial attributes, such as mouth opening (MO), eye-opening (EO), and eyebrow’s constriction (EBC) are determined from the facial expression of the subject. This needs segmentation of eye, mouth and eyebrow regions and their localization, the details of which are discussed in the previous section.

7.6 Parameter Selection of the Emotional Dynamics

221

Happiness Disgust Fear

1

2

4 ----

6

7

8

9

Fig. 7.7. Facial expressions of two subjects corresponding to different stimulus causing arousal of happiness, disgust and fear. Table 7.6. Features and primary features of emotions

Emotion

Eye-open

Mouth–open

Eyebrow-const.

DISGUST

SMALL

MEDIUM

LARGE

FEAR

LARGE

LARGE

SMALL

HAPPY

MEDIUM

LARGE

SMALL

ANXIOUS

MEDIUM

SMALL

LARGE

SAD

MEDIUM

SMALL

SMALL

ANGER

LARGE

LARGE

LARGE

222

7 Multiple Emotions and Their Chaotic Dynamics

Usually, the feature space of emotion includes all the three parameters MO, EBC, and EO [5]. In this work, we, however, considered only one primary feature, which essentially describes an emotion. Selection of the primary feature was obtained by taking a consensus of 50 observers, the average of the score assigned by them to individual features for a given emotion was evaluated, and the feature having the largest average score is declared as the winner. Table7.6 provides the features for six emotions, where the primary feature is set in bold. Since MO, EO and EBC all are useful features for an emotion, we categorized the necessary and primary features for an emotion by adjectives like Small, Medium and Large. The exact range of these three adjectives depends on individuals’ personality, culture and upbringing, and country/state where he/she belongs to. In order to keep the measurements error-free, we normalized the measurements by dividing the actual measurement of MO, EO and EBC by its largest experimental value for the given subject. The stimulator movie clips, and the aroused emotions of two people at different instances are given in Fig. 7.7. For convenience of measurement, we assumed that the degree/strength of an emotion is proportional to its primary feature. Thus when we note a large MO in a facial expression, we consider him/her to be happy. The degree of happiness, here is proportional to mouth-opening, and for convenience we set this proportional constant to be one. Similarly, a large eyebrows’ constriction indicates disgust, and the measure of eyebrows’ constriction determines the degree of disgust. Similarly, a large EO refers to fear, and the measure of EO determines the degree of fear. In this work, we represent EBC, MO and EO by state variables X1, X2 and X3 respectively. Consequently, the degree of disgust, happiness and fear are represented respectively by X1, X2 and X3. Since these primary features are all measurable, we can evaluate the parameters of the emotional dynamics: aii , bji, cki in (1) for stable and chaotic cases by respectively setting

dX i dt

=0

and

dX i (t + 1) >1 dX i (t)

for all i. The parameters λ ik , β ji

and K are set empirically from computer simulations, and are not intervened, while determining the dynamic behavior of emotions from experiments. Principles of parameter selection for the dynamics are illustrated here using Example 7.1, 7.2 and 7.3. The examples consider excitation of one or more dynamics of emotion with different stimulus at different time frames, as indicated in Fig. 7.7. Table 7.7 Table 7.7. Parameters of facial extract of subject 2 (lady) (in pixels) Features

T0

T1

T2

T3

T4

EO

14

17

13

12

15

19

6

18

7

16

MO

10

14

13

5

3

6

5

8

12

5

EBC

4

5

10

24

24

28

27

10

13

6

T5

T6

T7

T8

T9

7.6 Parameter Selection of the Emotional Dynamics

223

Table 7.8. Parameters of facial extract of subject 1 (man) (in pixels) Features

T0

T1

T2

T3

T4

T5

T6

T7

T8

T9

EO MO EBC

12 8 3

16 14 6

14 13 10

15 5 23

15 4 24

17 4 27

7 5 27

16 8 8

17 12 15

18 4 3

and 7.8 below provide the measurements of MO, EO and EBC respectively for the lady and the man in Fig. 7.7. These measurements will be required for parameter evaluation of the dynamics to be undertaken in the following examples. Example 7.1: Analysis at Time-Slot 1 The arousal of happiness at time slot 1 in Fig. 7.7, is described by dX 2 dt

= a 22 X 2 (1 −

X2

(7.10)

)

K

Expressing, dX 2 = X 2 (t + 1) − X 2 (t) , dt

(7.11)

we have X 2 (t + 1) = X 2 (t)[1 + a 22 (1 −

X 2 (t) K

)] .

(7.12)

Now, empirically, we have K=1000 [1]; and X2(1)=14 pixels, X2(0)= 10 pixels as obtained from measurements in MO for the lady in Fig. 7.7. A solution from the last equation yields a22 =0.404. Example 7.2: Analysis at Time-Slot 2 When stimulus for disgust appears at time slot 2, the disgust dynamics grows and the happiness dynamics faces competition with the growth in disgust, given by dX1 X = a11X1(1− 1 ) dt K

(7.13)

and dX2 X = a 22X2 (1 − 2 ) − c21X2 (1 − exp(−λ12X1)) dt K

(7.14)

Representing derivatives by difference of the state variables at two successive instances, we re-organize (7.13) and (7.14) by X2 (t + 1) = X2 (t)[1+ a 22(1 −

X2 (t) )] − c21X2 (1 − exp(−λ12X1)) K

X1 (t + 1) = X1 (t)[1 + a11 (1 −

X1 (t) )] . K

(7.15)

(7.16)

224

7 Multiple Emotions and Their Chaotic Dynamics

With λ12 =0.0045 and K=1000 obtained empirically and X 2 (T 3) = 5, X 2 (T 4) = 3 obtained from measurements of the Lady images in Fig. 7.7, we finally obtain from equations (7.15) and (7.16): X2 (T4) X (T3) = 1 + a 22(1 − 2 ) − c21(1 − exp(−0.0045.X1(T3)) X2 (T3) K X1 (T4) = X1 (T3) + a11X1 (T3)(1 −

X1 (T3) . ) K

(7.17) (7.18)

Solving the above equations and setting a22 =0.404 as obtained from Example1, we finally have c21 =0.0342,c23=11.315 and a11=0.253. Example 7.3: Analysis at Time-Slot 4 When disgust and fear both appear concurrently at time slot 4, the above dynamics is modified to dX1 X = a11X1(1 − 1 ) − c13X1(1 − exp( − λ 31X3 )) − c12 X1(1 − exp( − λ 21X 2 )) dt K dX 2 dt dX 3 dt

X2

= a 22 X 2 (1 − = a 33 X 3 (1 −

K X3 K

(7.19)

) − c 21X 2 (1 − exp( − λ12 X1 )) − c 23 X 2 (1 − exp( − λ 32 X 3 ))

(7.20)

) − c 32 X 3 (1 − exp( − λ 23 X 2 )) − c 31 X 3 (1 − exp( − λ13 X 1 ))

(7.21)

In similar way, by solving the above equations, and setting a22 =0.404 as obtained from Example 7.1, c21 =0.0342 and a11=0.253 as obtained from Example 7.2, we finally obtained c32=10.01,c31=0.3355 and a33=0.505. We now need to test whether the parameters of the dynamics evaluated above refers to chaos or stable emotional dynamics. This is tested by checking the conditions for chaos or stability of the dynamics. Table 7.9 provides the results of computation of the three time-derivates at different time-slots for the lady in Fig. 7.7. Plots of the time-derivates given in Fig. 7.8 below indicate that dX3/dt is positive for all time t, and, thus the subject 2 (the lady in Fig. 7.7) exhibits a chaotic phobia. Similar analysis was carried out for the man images in Fig. 7.7, and the results of the analysis confirm that the man’s emotional dynamics is stable. Table 7.9. Obtained parameters from the analysis undertaken

Cij

Cik

λkj

λki

K

a11=0.217

c12=1.61

c13=13.29

λ21=0.0045

λ31=0.007

k=1000

a22=0.404

c21=0.0342

c23=11.315

Λ12=0.0045

λ32=0.007

k=1000

a33=0.505

c32=10.01

c31=0.3355

λ23=0.009

λ13=0.005

k=1000

aii

7.6 Parameter Selection of the Emotional Dynamics

225

Table 7.10. Evaluation of time derivatives at different time slots

dX 1 (t + 1) dX 1 (t) dX 2 (t + 1) dX 2 (t) dX 3 (t + 1) dX 3 (t)

t1/t0

t2/t1

t3/t2

t4/t3

t5/t4

t6/t5

t7/t6

t8/t7

t9/t8

t10/t9

0.77

0.74

0.22

-0.8

-0.8

-1.1

-1.1

0.25

-.02

0.62

1.08

1.00

0.59

-0.3

-0.3

-0.6

-.05

0.63

0.41

0.93

0.61

0.28

0.36

1.02

1.18

0.92

1.02

0.77

0.45

1.03

dX1 (t + 1) dX1 (t) 1 0.5 dX1 (t + 1) dX1 (t)

0 1

2

3

4

5

6

7

8

9

10

-0.5 -1 -1.5 tim e

Fig. 7.8 (a). The value of

dX 1 (t + 1) dX 1 (t)

with respect to time.

dX 2 (t + 1) dX 2 (t) 1.2 1 0.8 0.6 dX 2 (t + 1) 0.4 0.2 dX 2 (t) 0 -0.2 -0.4 -0.6 -0.8

1

2

3

4

5

6

7

8

9

time

Fig. 7.8 (b). The value of

dX 2 (t + 1) dX 2 (t)

with respect to time.

10

226

7 Multiple Emotions and Their Chaotic Dynamics

dX 3 (t + 1) dX 3 (t)

1.4 1.2 1 dX 3 (t + 1) dX 3 (t)

0.8 0.6 0.4 0.2 0 1

2

3

4

5

6

7

8

9

10

tim e

Fig. 7.8 (c). The value of

dX 3 (t + 1) dX 3 (t)

with respect to time.

7.7 A Stabilization Scheme for the Mixed Emotional Dynamics The mixed emotional dynamics model presented by Equation 7.1 exhibits chaotic, limit cyclic and stable behavior for different parameter sets of the dynamics. In this section, we present a scheme for controlling damping (Fig. 7.11) for the proposed non-linear model of the emotional dynamics. Let x1(t), x2(t), x3(t) and x4(t) be 4 emotional states, exhibiting chaotic or limit cyclic behavior. Under this circumstance, we determine the damping level of the dynamics from its transient response. If the predicted damping level is low, we call it under-damped, and we attempt to increase the intrinsic growth rate aii of the dynamics. On the other hand, if the dynamics is found over-damped, we need not generate any control signal.

Fig. 7.9. The un-stabilized (chaotic) response of x2.

7.7 A Stabilization Scheme for the Mixed Emotional Dynamics

227

Fig. 7.10. The stabilized response of the emotion controller.

Prediction of the damping level from the response of the dynamics, indeed, is an open research problem until this date. A classical model for damping level prediction proposed by Konar and Roy [12] in population control of a predator-prey system, however, seems to be useful in the present context. The damping level predicted here by examining the sign constraint is presented next. Sign constraint: if sign of xi(t) = sign of xi(t= 0), ∀ t ≥ 0 then the dynamics is said to be under-damped, else it is over-damped. The above damping level prediction rule, as proposed by Konar and Roy, has been employed here to determine the damping level of the non-linear emotional dynamics. In case it is found under-damped, a proportional plus integral (PI) type controller is employed to control the parameter aii of the emotional dynamics. In the process of generating control signal, special emphasis is given to keep the dynamics minimally over-damped. The phrase minimally over-damped refers to the minimum level of damping that causes an over-damped response of the dynamics. In Fig. 7.11 we have 3 major modules: i) an emotional dynamics, ii) a damping level prediction, and iii) a PI type controller used for enrichment of the intrinsic growth rate aii, so as to control the damping level of the dynamics.

228

7 Multiple Emotions and Their Chaotic Dynamics Desired slope

x1(t) D

Emotional Dynamics

∑

x1,x2, x3, x4

e(t)=0

a

If sign of D2 x1(t) = sign of D2 x1(t) at t=0+ for all t >0+

Count= Count -1

c(t)= kp |e(t)| + kI ∫ |e(t) | dt

Count =0 ?

Set natural a

Δa = c(t)

a= a + Δa

Δa =Δamax

Δa = Δa Δamax?

Fig. 7.11. A scheme of controlling damping in emotional dynamics for stabilization.

A computer simulation of the proposed damping level prediction and stabilization scheme of emotional system is undertaken. Simulation results reveal that an under-damped chaotic emotional dynamics (Fig. 7.9) can be stabilized by artificially increasing the intrinsic growth rate a11 of the proposed system. The resulting response (Fig. 7.10) exhibits a significant decrease in percentage of peak overshoot.

7.8 Conclusions This chapter presented a new model of mixed emotional dynamics, and demonstrated a scheme for parameter identification of the dynamics by evolutionary algorithm. An analysis of the dynamics reveals that it may demonstrate chaos, limit cycles and stable behavior, depending on the range of parameters of the dynamics. A PI type control scheme has been proposed for the prediction of the damping level of the dynamics. In case, the damping level is found under-damped, the PI

Exercises

229

type controller enhances the intrinsic growth rate of the dynamics to make it overdamped. Thus the relative stability of the emotional dynamics is improved. Detailed realization of the increase in intrinsic growth rate by presenting selected music, videos and voice message has been presented in [5].

Exercises 1.

Consider the following difference equation: 2 (xn- 1/8) + ¾ + r, 0 ≤ xn ≤ ¼ Xn+1 =

2 (1/2 – xn) + ½ , ¼ ≤ xn ≤ ¾ 2 (xn –7/8 ) + ¼ + r, ¾ ≤ xn≤ 1

where r ∈ ( 0, 1/2 ), given that x0 ∈ [0, 1]. The Lyapunov exponent for studying chaotic situation of he dynamics is given by λ = Lt N→∝

1

N−1

Σ ln

N n=0

dxn+1 dxn

.

Show the plot of xn+1 verses xn. Also compute the Lyapunov exponent, and verify that λ = ln (2) for the linear region of the xn+1 verses xn plot. Hence, conclude that the system is chaotic. 2.

A discrete system is modeled by a difference equation, given by xn+1= a xn (1- xn).

a) Draw the phase trajectory (xn+1 vs. xn) with x0 =0.91 and a=4. b) Plot the phase trajectory for the given equation with a<1, and show that for any initial choice of x0, xn+1 vanishes (i.e. becomes zero). 3.

Consider the following discrete dynamical system given by p(n+1) = (1 − p(n) * (1 − q(n))) (1-q(n)) and q(n+1) = (1 − q(n)),

where pn and qn lie in the range [0, 1]. Also given that p0 = 0.2, and q0 = 0.6. Show the plot using the following sequence of co-ordinates

230

7 Multiple Emotions and Their Chaotic Dynamics

(p(0), 0.0), (p(0), p(1)), (p(1), p(1)), (p(1), p(2)), (p(2), p(2)), (p(2), p(3)), (p(3), p(3)), (p(3), p(4)). Similarly, repeat the plot for q(t) by replacing p by q in the above co-ordinate set. Verify from the plot that the given dynamic represent limit cycle behavior for both p and q. Compute dpn dpn

and

dqn dqn

.

Check that both are negative. 4. Repeat the above problem for the following dynamics. p(n+1) = a11 p(n) + a22 [1 − q(n) (1 − p(n)) (1 − q(n))] and q(n+1) = [1- p(n) (1 − q(n))] where a11=0.2 and a22= 0.5. Assume p(0)= 0.2 and q(0)= 0.3. 5. The van der Pol dynamics is given by d2x/dt2 − ∈(1 – x2) dx/dt + x = 0, ∈>0. a)

Using Runga-Kutta method of numerical techniques, find the solution of the van der Pol equation, and hence draw the phase trajectory of the system. Show that the phase trajectory describes a cycloidal path, converging at a stable focus for small ∈. b) Also show that the phase trajectory describes a limit cyclic behavior for large value of ∈. 6.

The Lee-Oscillator consists of 4 neural elements u, v, w, z. The dynamics of Lee-Oscillator is given by u(t+1) = f [ a1 . u(t) + a2 . v(t) + I(t) − θu ] v(t+1) = f [ b1 . u(t) − b2 . v(t) − θv ] w(t+1) = f [ I(t)] −kI2(t) Z(t) = [ u(t) − v(t) ] . e + w(t) where f( .) denotes the sinusoid function a1, a2, b1, and b2 are weight parameters of those neurons;

Exercises

231

S

z

I(t)

exp(-kI2)

exp(-kI2)

a a U

I(t) b1

V

b2 Fig. 7.12. The Lee Oscillator.

θu and θv are the threshold of exhibitory and inhibitory neurons. I(t) is the external input stimulus and k is the decay constant. The diagram of Lee-Oscillator is given in Fig 7.12. Show the bifurcation of the Lee Oscillator with k = 500, a1=5, a2=5, b1=1, b2=1, and assume external stimulus X = I + e * sign (i). 7.

The Lyapunov exponent, given by N L= (1/N) Lt ∑ dxi(n+1)/dxi(n) n → ∞ n=1 for i=1 to 4, where dx1/dt = a11x1(1-x1/K) + b31x1(1-exp(-β31x3)) – c21 x1(1-exp(-λ21x2) dx2/dt = a22 x2 (1- x2/K) + b12 x2 (1- exp (-β12 x1)) + b32 x2 (1-exp(-β32 x3)) + b42 x2 (1- exp (-β42x4)) – c32 x2(1-exp(-λ32x3) )

232

7 Multiple Emotions and Their Chaotic Dynamics

dx3/dt = a33 x3 (1- x3/K) + b23 x3 (1- exp (-β23 x2)) – c23 x3 (1-exp(-λ23 x2)) – c13x3 (1- exp (-λ13 x1)) – c43 x3 (1-exp(-λ43x4)) dx4/dt = a44 x4 (1- x4/K) + b34 x4 (1- exp (-β34 x3)) – c24 x4 (1-exp(-λ24 x2)). Verify that L>1 for i=1 to 4, using the parameters listed in the Table below: a11 = 0.33, b31 = 0.9, c21 = 0.55, β31= 0.006, λ21 = 0.0033; a22 = 0.002, b12 = 0.2, b32 = 0.2, b42 = 0.3, c23= 0.4, β12 = 0.001, β32= 0.001, β42= 0.001, λ32= 0.005; a33 = 0.2, b23 = 0.4, c32 = 0.5, c13 = 0.45, c43 = 0.3, β23= 0.005, λ23 = 0.005,λ13= 0.005, λ43 = 0.005; a44 = 0.1, b34 = 0.4, c2\4 = 0.35, β34= 0.001, λ24 = 0.001; and K= 1000. Hence comment on the chaotic behavior of the dynamics for the given parameter settings. [Hints: A discrete approximation of the above model can be reconstructed by replacing dxi

xi(t+1) – x(t) =

dt

(t+1) – t = xi(t+1) – xi(t).

Replace dxi/dt by xi(t+1) – xi(t) in the given dynamics, and hence evaluate dxi(t+1)/dxi(t), and show that for i= 1 to 4 dxi(t+1)/dxi(t)>1. Hence, the dynamics is chaotic.]

References 1. Beauregard, M., Levesque, J., Paquette, V.: Neural Basis of Conscious and Voluntary Self-Regualtion of Emotion. In: Beauregard, M. (ed.) Consciousness, Emotional Selfregulation and the Brain. Benjamins Pub. (2004) 2. Buchel, C., Dolan, R.J., Armony, J.L., Friston, K.J.: Amygdala-Hippocampal Involvement in human Aversive Trace Conditioning Revealed through Event-Related fMRI. The J. of Neuroscience 19(24) (1999) 3. Guyton, B.C., Hall, J.E.: Textbook of Medical Physiology 4. Cambel, A.B.: Applied Chaos Theory: A Paradigm for Complexity. Academic Press, San Diego (1923) 5. Chakraborty, A., Konar, A.: Chaotic Emotional Dynamics Realized with Recurrent Neural Network. Organized by Computer Society of India, Calcutta (2005)

References

233

6. Chakraborty, A.: Cognitive Cybernetics: Behavioral Models of Human-Machine Interactions, Ph.D. Thesis, Jadavpur University (2005) 7. Durka, P.: Matching Pursuit and Unification in EEG Analysis. Artech House, Norwood (2007) 8. Frackowiak, R.S.J., Friston, K.J., Frith, C.D., Dolan, R.J., Price, C.J., Zeki, S., Ashburner, J., Penny, W.: Human Brain Function. Elsevier, Amsterdam (2005) 9. Ghosh, M., Chakraborty, A., Konar, A., Nagar, A.: Detection of Chaos and Limit Cycles in Emotional Dynamics from the Facial Expressions of the Stimulated Subjects. In: Subudhi, B. (ed.) Computational Intelligence, Control and Computer Vision in Robotics and Automation. Narosa Publisher, New Delhi (2009) 10. Grossberg, S., Seidman, D.: Neural dynamics of autistic behaviors: Cognitive, emotional and timing substrates. Psychological Review 113(3), 483–525 (2006) 11. Harter, D., Kozma, R.: Chaotic neurodynamics for autonomous agents. IEEE Trans. Neural Networks 16(3) (May 2005) 12. Konar, A., Roy, A.B.: Population drift and control of damping in a predator-prey system. IMA J. of Mathematics Applied in Medicine and Biology 7, 245–259 (1990) 13. Landini, L., Positano, V., Santarelli, M.F. (eds.): Advanced Magnetic Image in Resonance Processing Imaging. Taylor and Francis, Abington (2005) 14. Lee, R.S.T.: Advanced Paradigms in Artificial Intelligence: From Neural Oscillators, Chaos Theory to Chaotic Neural Networks, January 2006, vol. 5. Advanced Knowledge International Publishers, Adelaide (2006) 15. Xu, D., Principe, J.C.: Dynamical analysis of neural oscillators in an alfactory cortex model. Neural Networks 23, 46–55 (2000)

9 Applications and Future Directions of Emotional Intelligence

This chapter provides various applications of emotional intelligence, and possible extension of the research methodologies presented in previous chapters. It begins with human-machine interactive systems, emphasizing the design aspects of interactive agents. The co-operation of hardwired agents such as mobile robots, capable of exchanging of emotion is also undertaken in this chapter. Psychotheraputic use of emotional intelligence models is considered next. The methodology of detecting anti-social motives from emotional expressions is briefly introduced. Application of emotional intelligence in video photography/movie-making is also considered. Matrimonial counseling through personality matching is appended at the end of the chapter.

9.1 Introductions The last three chapters in the book introduced different models of emotional intelligence [18], concerning control and stabilization of emotions. This chapter provides various engineering applications of emotional intelligence [33], such as human-machine interactions, multi-agent co-operation, psychotherapy, detection of anti-social motives, digital movie-making and matrimonial counseling. Emotion modeling has an interesting role in the next generation humanmachine interactive systems. It can be realized by modeling both input and output parameters of the interactive system. For instance, the visual expression of the input-user-interface, such as laughter, cry and other forms of facial manifestation can directly be regarded as input to the computers. On the other hand, the emotional expression of computer can be realized by synthesizing emotions either on the video display unit or on an artificial emotional creature, depicting the machine response. Details of discussion on emotional models, synthesizing next generation human-machine interactive systems, will be discussed in details in a separate section. The essence of emotional reaction of agents to execute a complex task through co-operation among the agents will be discussed in this chapter. Agents in a team can share their emotional experience through manifestation of the emotion in their behavioral response. The former agents can be noticed by its teammates, thereby A. Chakraborty and A. Konar: Emotional Intelligence, SCI 234, pp. 261–293. © Springer-Verlag Berlin Heidelberg 2009 springerlink.com

262

9 Applications and Future Directions of Emotional Intelligence

instantiating the emotional part of the global plan determined by the agents. The advantage of sharing emotional expression lies in minimizing communication among the agents. Very little work has so far been undertaken in this area, and there is ample scope to design new formalisms and methodology of multi-agent planning and co-operation. A brief discussion on multi-agent co-operation taking into consideration of the emotion arousal of the agent will be undertaken in this chapter. Models of emotional intelligence are currently given much importance for their increasing application in psychotherapy. The behavior of a person can to some extent be visualized from his/her emotional expressions. Sometimes a number of emotions co-exist in the psychological brain of the subject, and various forms of emotional reactions can appear under this circumstance. For example, when the individual emotion, such as anger or anxiety, has a random non-periodic fluctuation of non-uniform amplitude, we call it chaotic. Chaotic response has correlation with temporal psychological disorder, such as epilepsy. An early diagnosis of epilepsy from the observed chaotic behavior of emotional dynamics helps in taking necessary corrective actions, protecting patients from dangers. This chapter focuses various issues of emotional intelligence models for applications in psychotherapy. Detection of antisocial motives is currently given special emphasis for increased terrorist activity in our society. Emotional expression of the subject can be used to determine their possible anti-social motives. A further detailed enquiry of the suspects then may be undertaken to identify the anti-socials. In this chapter, we provide a discussion on the detection of anti-social motives from emotional expressions. The emotion of the actor-actress can be synthesized in digital movies to reduce both the time- and the money- cost of preparing the movies. A brief discussion on emotion synthesis in digital movie-making is outlined in this chapter. Matrimonial counseling also is an important area, which has a great role to play in personality matching of people. Personality of two people can be matched by generating queries to the possible partners and observing their emotional reactions. When the tune of responses of both the people is in harmony, it is expected that they will understand each other easily. Naturally, emotion modeling and synthesis finds an interesting application in matrimonial counseling. A discussion on personality matching of people ready for matrimonial engagement is undertaken in this chapter. The chapter consists of 13 sections. The scope of emotional intelligence in human-machine interactive system is discussed in 9.2. In section 9.3, we provide application of emotional intelligence in multi-agent co-operation of robots. Section 9.4 is devoted to applications of emotional intelligence in psychotherapy. Section 9.5 outlines the detection of antisocial motives from emotional expression of the possible culprits. Application of emotional intelligence in video photography/movie making is discussed in section 9.6. The scope of emotional intelligence in matrimonial counseling of people is discussed in section 9.7. In section 9.8, we provide a discussion on synthesizing emotions in voice. An application of emotional intelligence in user-assistance system is introduced in section 9.9. Principles of emotion recognition from voice samples are outlined in section 9.10. Scope of emotional intelligence in personality building of artificial creatures is given in

9.2 Application in Human-Machine Interactive Systems

263

section 9.11. An outline to multimodal emotion recognition is presented in section 9.12. Principles of parameter identification of emotional dynamics using EEG signals are outlined in section 9.13.

9.2 Application in Human-Machine Interactive Systems Emotion plays an important role in human-machine-interactive systems [9]. Users nowadays are not only satisfied with visual and audio responses of machines, but they also want to employ all their sensory organs to receive the details of a multimedia presentation. Consequently, they require tactile, olfactory and other modes of communications as well. In turn, users also want to provide information to the computer in different ways, such as gestures, postures, languages, graphics and emotion as well. How emotion can exactly be embedded in voice, gesture, posture and presentation style is an important consideration in design of the next generation human-machine interactive systems. In this chapter, we discuss the scope of emotion modeling in both input and output interfaces from the computers point of view. 9.2.1

Input Interfaces

Users can provide visual, auditory, tactile, olfactory and taste information about an object to a machine. Some of these inputs carry emotional information. For example the change of voice of a person can be modeled to represent different emotions he/she possess. When a person is shocked or bewildered, his/her voice becomes grave and the pitch in the voice is less prominent. When someone is surprised, he normally uses special tones to express his surprise. The change of pitch under this context is remarkable. When someone is angry, he usually talks much faster than his normal speed of conversation. All the above items indicate that emotions are embedded within voice. Gestures and postures are also indicative of emotion arousals. For example, a person under stress usually shakes his hands/legs or other parts of his body very frequently. The eye-movement of a stressed person is also very frequent. When a person expresses an emotion of liking an objects/loving a person, the excessive secretion of saliva is indicative of his emotional exposition. 9.2.2 Output Interfaces The machines’ reaction can be embedded in its emotional counterpart. The question then naturally arises as to how to create emotion on virtual characters? In fact, this is a hot topic of current research in the domain of emotional intelligence. Synthesizing emotion on virtual characters requires understanding the syntax and semantics of natural languages that the machine is going to deliver to the user. Emotion then can be expressed as part of voice or graphical user interface with external manifestation of facial deformations on the graphical interface. How the behavior of the emotion generating creature changes on the graphics is an

264

9 Applications and Future Directions of Emotional Intelligence

interesting issue of discussion. Currently researchers are taking keen interest on 2dimensional and 3-dimensional modeling of the artificial creature such as pets on the screen. For instance a pet cat may go on mewing until the user serves her milk with a soft tone. Once these are provided, the artificial creature may go on shaking its tail and change its tone as a token of its emotion. Sometimes user can directly interact with their artificial creatures by touching them and holding them with their mouse-controlled hand curser. The pet can describe its own affectionate experience by licking and nuzzling the hand curser. Also when the pet perceives a lack of affection, it can generate ‘howl’ and ‘yowl’ sounds. Artificial human babies can represent their lack of perceiving affect by crying and continue saying, “mumma” or “dad” in a sad tone of voice. Many researchers of prestigious research laboratories around the world have successfully implemented the feelings of emotional creatures, such as love, warmth, happiness and loneliness of the characters. 9.2.3 Embodiment of Artificial Characters Significant research works have already been undertaken on virtual modeling of the artificial characters [35] called “avatars”. An avatar is a virtual actor i.e., solely controlled by the computer and is therefore autonomous. The visual and behavioral features are the important characteristics of the virtual avatars. In modeling avatars, special consideration is given in designing its face, structure animation and its ability in communicating with the real world. The emotional component of the avatar is realized both on its face and poses. Body movements of the avatar are also important consideration to represent their emotional expressions. Recently Magnete-Thalmann et al. studied the scope of avatars in a network virtual park [30]. They considered predefined facial expressions applied to a virtual actor to update its emotion based on the current human interactions. These virtual actors can control their hand and leg movements, bend their bodies in different orientations, and get excited with external information from the user. Avatars are generally modeled by using a time-chart, where the timer is a system clock or counter, whose time slot are divided into uniform intervals to represent facial, body animations and speech animation of the avatars in a time-multiplexed manner. The sequence of actions are prescribed in the time chart, and then can be played on a movie player or any other software to visualize the effect of the avatars on the users. Break points can be added in the time chart to control or stop the progression of the avatar on the time line. There are different options for triggering the timeline of the avatars. The instance, when an avatar enters the range of proximity sensors of an object in a virtual park, its time line is triggered to do necessary actions on the object, such as movements or changing visual expressions of the avatars. When an avatar touches an object its timeline is also triggered to do the necessary actions. The necessary actions include picking up the object or transferring the object to a distant place, or eating or smelling the object. Appropriate emotions are generated in the facial expression, gestures and postures of the avatars, when the avatar touches an object. When a user wants to intervene the action of an avatar, he hits the avatar or its timeline by his mouse and thus can command the avatar to change its course of action or to pay attention to a different job.

9.3 Application in Multi-agent Co-operation of Mobile Robotics

265

9.3 Application in Multi-agent Co-operation of Mobile Robotics In multi-agent robotics, a number of agents work together to execute a complex task, such as transfer of large objects from one location to another or playing in a soccer tournament. The co-ordination in multi-agent robotics can be of two common varieties: co-operation and competition. When two or more robots jointly work for successful completion of a common goal, we call them a co-operative team. On the contrary, when the robots have different goals, and an implementation of a goal by a robot corresponds to failure in implementation of goals by other robots, the team is said to have competitive members. For example, in a football tournament, the teammates work in a co-operative manner, while the opponent team works in a competitive sense. Emotion in multi-agent robotics plays a vital role in representing and characterizing robots’ own behavior. For instance, suppose an agent fails to grip an object because the object-size, measured by its diameter, is too large to be gripped by the robot. Alternatively, suppose a robot fails to grip an object because of slip, the agent under these contexts can represent its features by generating appropriate tone indicative of its failures. Different instrumental tones indicating various situations faced by the robots can be used to describe its failures or successes or to get rescued when it generates an alarming tone to its teammates. The oscillation of the robot’s gripper when sustained for a long time, indicates a failure of the robot in correctly positioning its arm. A teammate of the robot may notice this failure by its own camera and can provide necessary supports to rescue the partner in trouble. In mining applications because of blockades in different regions of the mines, communication among the robots through visually synthesized emotions becomes impossible. Alarming tone, however, is a good choice, in this kind of application. In battlefields, a robot troop may have a commander, and a number of supporters who jointly participate in the battle to defeat the opponents. The commander prepares a job schedule for the individual participants, and the participants in turn execute the job as advised by the commander. Both the commander and the participants usually carry visual sensors to understand the emotion of each other. Jerking of the robots structure, continued oscillation of the gripper or the arms, generating specially coded tones or waving robot’s hands are some of the possible representation of the robots’ emotion for exchange among two peers. Undersea oil exploration or nuclear waste explorations are some of the areas where robot teams jointly participate in the execution of the job. Emotion can sometimes be coded to exchange significant hidden messages among the agents. For instance a sequence of laughter and cry or oscillation of robot’s grippers/arms may represent a specific code related to the goal of the job. Suppose a small laughter, a small cry and small oscillation of the arm may be indexed as 1, 2 and 3 respectively. Thus a consecutive occurrence of 1, 2 and 3 represents a job of 123 units. This 123 unit may correspond to movement of an object by say 123 inches. Such coding of emotion representing actions of appropriate metric is an important consideration in modern agent robotics system.

266

9 Applications and Future Directions of Emotional Intelligence

9.4 Emotional Intelligence in Psycho-therapy Emotional intelligence models presented in chapter 5 and 6 can readily be used in psycho-therapy related applications. It needs mention here that the models outlined in chapter 5 and 6 contain various parameters. These parameters denote different time constants of our emotional expressions. These time constants can be measured by fMRI/PET scan experiments. It is indeed important to note that the emotional parameters are highly person-dependent. Naturally, a priori determination of these parameters is mandatory before to use the emotional models in psycho-theraputic applications. A common question then may be raised: why to use emotional models when an fMRI/PET scan machine is available? The answer to this depends on whether a patient can afford an fMRI/PET scan machine. In presence of such a machine, the emotion arousal can be easily determined from a prior knowledge of the clusters on the imageries. In other words, for a happy person and an anxious person the fMRI or the PET scan yields different responses indicating the emotional states of the patient. In absence of such machines, the emotional model that describe a patient’s emotional counterpart can be readily used to determine the changes when excited with external inputs, such as happiness, anger, shock etc. The response of the emotional models may then be noted and controlled-excitation of appropriate magnitude may be applied to the patients’ emotional systems to activate his/her emotional brain. In the coming future, emotional intelligence models will play a significant role in the psycho-theraputic applications. According to the medical physiologist [19], the amygdale is the centre of biological emotion arousal. Recently, a medical group at the university of London, speculated the behavioral response of the amygdale with and without external stimulation [15]. Their observation reports that the amygdale is the major controlling and regulating unit of many human emotions, including anxiety, anger, fear and love. The amygdale responses can be visualized on an fMRI instrument. To determine and control, the growth of emotion arousal, the parameters of emotional models, if determined online during the arousal of a specific emotion, gives a clear visualization of the psychological aspect of the human brain. It is apparent from Chapter 6 that the parameters aii for any emotional state i provides a good measure of the psychological behavior of the emotional brain. In other words, for a certain range of aii, emotional state exhibits limit cyclic behavior. For a different range of aii, we obtain chaotic and stable behavior of emotional dynamics. One interesting issue that needs mention here: whether we are able to determine the emotional behavior of the patients, if we can measure the parameters aii online (with an fMRI instrument)? Apparently, it seems that we can easily narrate the emotional behavior of aii. This naturally is appealing for its possible applications in controlling emotional dynamics of the patients. A simple scheme of controlling aii to change the emotional states has already been demonstrated in Chapter 7. Similar schemes can be extended for psycho-therapeutic use of emotional intelligence models.

9.5 Detection of Anti-social Motives from Emotional Expressions

267

9.5 Detection of Anti-social Motives from Emotional Expressions Emotional expressions sometimes carry information about anti-social motives of the people. Anti-social motives refer to execution of tasks, causing pains and anxiety to others. Humans as rational agents, even when performing anti-social activities, usually suffer from a consciousness about negative feelings, reproducing self-anxiety in their emotional exposure. Naturally, with a slightest external perturbation to the emotional brain of the anti-social people, the psychological motives become apparent in their emotional expression. The emotional expression for these people engaged in anti-social activities may come up in different form depending on situations, their personality and the external influence that causes them to regenerate their emotional expression. Sometimes, they express these emotions in the sudden change of their voice or their facial expression, gesture and posture. Fumbling is also a good indication of people engaged in activities against their will. Until now there exist no instrument to detect anti-socials and criminals from their emotional expression. A low cost fMRI or PET scan instrument is a good choice to detect the psychological oscillation/chaos in the emotional brain, when these people are excited with external perturbation. Currently non-psycho-metric methods are being adapted to detect anti-social motives of the suspects. For example, (hidden) video cameras placed in banks and airports are normally used in many countries to determine the motives of the people from the emotional expression. Such systems usually include frame grabbers and computers connected in networks. The photographs of suspected anti-socials, identified from their facial expressions, gestures and postures, are recorded at the local computer and communicated to a central pool for possible enquiry by the investigation departments. The most important aspect in the detection of antisocials by video cameras, perhaps is tracking the people and their facial expressions. Very little progress on tracking of human poses and facial expressions has so far been reported in the current literature on multimedia/image processing [34]. Currently, face tracking is an active area of research for its application in other problem domains, including video cell phones. Details of face/pose tracking go beyond the scope of the book. However, the principles of tracking may be narrated briefly. In visual tracking, few image pixels change within a short span of time. So, if the sampling time between successive frames is small (≈ 1/30 seconds), minute changes in the intensities may be noticed. Usually, tracking is performed in two steps, first by determining the current direction of maximum intensity changes on successive image frames, and then by predicting the possible changes in the close vicinity around the current direction of maximum change. It is apparent to note that the current direction of change can be any one of the neighborhood eight pixels (Fig. 9.1). Naturally, when the current direction of maximum intensity change is determined, the predicted direction and possible search space becomes narrower (Fig. 9.2).

268

9 Applications and Future Directions of Emotional Intelligence

a

b

c

d

X

e

f

g

h

Fig. 9.1. Eight pixel around the central (X) pixel.

Predicted direction of maximum change in intensity

i n p r t

j a d f u

k b X g v

Current direction of maximum change in intensity

l c e h w

m d q s x

Fig. 9.2. The predicted direction of maximum change in intensity, computed from the current direction of maximum change in intensity around the central (X) pixel.

Besides tracking of faces and postures, image analysis on static frames also give many pertinent information, such as eye-opening, the length of eyebrow constriction etc. on the faces of the captured images. Naturally, determining mood/emotion of the people from their facial expression becomes apparent. However, one interesting issue of detecting anti-socials from their facial expressions/emotions is to consider the phenomenological issue, which includes the psychological states of the human mind involving anti-social motives. Human develop emotions based on phenomena, and thus the effect of the phenomenological considerations are reflected in the manifestation of the emotions. The manifestation may be surprisingly excessive, when stimulated with external perturbations. The suspects during the interrogation phase, may be stimulated with emotioncausing input, and their reactions may be recorded by video camera. The detection of anti-social motives then can be extracted from the emotional expressions.

9.6 Applications in Video Photography/Movie Making

269

9.6 Applications in Video Photography/Movie Making Since the beginning of movie making in early 1950’s, natural scenes displaying the activities of the real actor/actress is photographed for audience. Only in last decade, people are thinking of virtual movies, which is fully prepared on a computer for presentation. Unfortunately, people are habituated to see their favorite actors or actresses on the screen, and dislike artificial characters on movies. The emotional models presented in chapter 5 and chapter 6 however, can be fruitfully utilized in synthesizing emotions of the artificial agents. In order to attract the viewers, the initial characters can be created by using the video photographs of the real actor or actress. The photographs, as and when needed, may be modified to display the natural emotion of the artificial agents, symbolically represented by the real actor/actress. The control input of the emotional models may be adapted to cause the arousal of necessary emotions following the thematic sketch of the story, describing the movie. Now, the control inputs automatically receive information from a given linguistic representation of this story. Of course, it is an open problem until this date. This in fact requires a syntax and semantic analysis of natural languages, where the syntax represents the grammar based on which the story is described, and the semantics denote the conceptual meaning of the theme of the individual sentences in the story. Specific data/knowledge structures such as semantic nets may be employed to represent both the syntax and semantics together on a common framework. One important aspect of extraction of emotional information from a story is to identify appropriate words or phase or a clause that represents specific emotion related to the theme under reference. The notion of time also is an important consideration to represent the progress of the story. A special type of data/knowledge structure, from which the emotion generation becomes apparent, can be used to represent each sequential theme in a story. Thus the synthesis of emotion on the characters becomes trivial from the semantic description of the individual theme. Some common clues of synthesizing emotion from natural language description of a story is to prepare a dictionary of emotion against an index of synonymous emotion-causing words or phrases. The individual emotion corresponding to a word or phrase may be digitally recorded in the computerized dictionary. Whenever a specific emotion has to be generated for a given role (real) character, similar facial changes as given in the dictionary, has to be realized on the role character’s face. On many occasions, gesture and posture also play an important role in modeling and synthesizing emotion in a movie. These too have to be realized for the “role model”, following the semantics of the theme in the story and the visual interactions of the words/phrases, as given in the dictionary. One important aspect in emotion synthesis from a given story is to encounter contradictory emotion-causing phrases in the close vicinity within the story. If the story interpreter software is unable to extract the appropriate emotion causing words or phrases, a contradiction may appear, and in such cases it is better to synthesize any emotion. The phrase “close vicinity” however needs to be emphasized because the closeness is relative to the speed of progress in the story. A fast

270

9 Applications and Future Directions of Emotional Intelligence

changing story may have a fast changing emotion and the close vicinity in that case may be just a single sentence. On the contrary a slow story reflects slow changes of emotion, and the vicinity here may refer to one paragraph.

9.7 Applications in Personality Matching of People for Matrimonial Counseling 9.7 Applications in Personality Matching Personality matching especially in matrimonial issues is an important consideration for the stability of the married couples. A number of software is currently available for automatic matching of personality of the pre-married couples. Unfortunately, the existing software does not take into account the emotional aspect of the couples. In order to match the emotion of pre-married couples, we need to evaluate a weight matrix, describing the transition of emotion from one state to another. The weight matrix we presented in chapter 5 is reproduced below for the convenience of the readers. From Happy

Anxious

Sad

Angry

W1, 1

W1, 2

W1, 3

W1, 4

W2, 1

W2,2

W2, 3

W2, 4

Sad

W3, 1

W3, 2

W3, 3

W3, 4

Afraid

W4, 1

W4, 2

W4, 3

W4, 4

To

Happy Anxious W=

In the above matrix, wij denotes the necessary weight representing the transition of emotion from the ith state to the jth state. An fMRI instrument can be employed to evaluate the weight matrix. It may indeed be noted from the emotion transition equation of Chapter 5 that the weight matrix W(t) is an intrinsic system matrix that depends on the personality of a human being. Measurement of weight matrix W(t) can be undertaken even in absence of any external stimulation. By noting the changes in fMRI images and determining the changes of state of emotion from these images gives us a relative measure of the weight matrix. It is indeed important to note that without any external perturbation, the emotion transition naturally takes place because of the cognitive impact on the emotional brain. The B and C matrices in the emotion transition equation reproduced below: M(t+1) = W(t) M(t) + Bμ - Cμ/,

(9.1)

9.8 Synthesizing Emotions in Voice

271

M1j (t + 1)

External stimulation μj or μj/

Emotional dynamics of a person

+

Emotional dynamics of his/her partner

M2j (t + 1)

Scoring index

n ∑ Jj j=1

||Ej(t + 1)|| Jj

Fig. 9.3. Computing scoring index for matching emotion of two people.

however, can be evaluated by external stimulation to emotional brain. Assuming that the W matrix remains unchanged over time, we can evaluate B and C from the measurement of M(t+1) vector by fMRI instrument. Once W, B and C are known, we can compare the emotion of any 2 people by external stimulation. Fig. 9.3 presents a schematic view for matching emotion of two people. When excited with a common stimulus, the emotion of the individuals changes following the emotional dynamics (9.1). The emotion vector M(t+1) of the individuals now can be compared by taking the component-wise difference of the 2 vectors. An Euclidean norm is then applied on the difference vector E(t+1). Now, for different stimulations j of n-types, we compute the sum of the Euclidean norms Ej(t+1) and define it as the scoring index. It is now apparent from Fig. 9.3 that smaller is the scoring index the better is the matching between the emotional dynamics of the 2 people.

9.8 Synthesizing Emotions in Voice Very little has been reported so far on synthesizing emotion in voice. For instance when, someone says “hello” in a telephonic conversation, the emotion of the person can easily be detected at the other end. In fact, our vocal system is influenced by the prefrontal cortex, where emotion are aroused by the external influences including music, videos and messages. There is ample scope of future research in understanding and synthesizing emotional voice by changing prosody (TD-PSOLA) and the LPC spectrum. Recognizing emotion from speech dialogues is of great concern to study emotional system.

272

9 Applications and Future Directions of Emotional Intelligence

Bulut et al. [2] studied the capability of human-subject in recognizing emotion from speech dialogue by mixing prosodic information and diphones belonging to distinct emotional states. They produced 80 synthetic sentences by combining both prosodic information for anger, happiness, sadness and neutral mental state with the same states representing diphones. The sentences were then presented to audience through a web-based listening system. The listeners were allowed to play the test files as many times as they wished, and were asked to choose the most suitable emotion they encounter in the synthesized voice. They also rated the success of expressing the emotion along a 5 point scale, indicating the degree of expressing power of emotion with 5 for excellent, 4 for good, 3 for fair, 2 for poor and 1 for bad. Hundred files consisting of 8 synthetic sentences and 20 original target sentences were presented in random and the presentation order was also different for each listener. Recognition result of their experiment reveals that anger, sadness and happiness can be synthesized fairly successfully by applying concatenate synthesis techniques. It is also noted that when a mixture of neutral prosody and anger inventory was recognized mostly as anger, the combination of sad prosody and neutral inventory is recognized as sadness.

9.9 Application in User Assistance Systems With the rapid development in computer networks and mobile computing, researchers are keen to connect any two persons in the network anywhere at anytime. These give rise to a new computational paradigm, called pervasive and ubiquitous computing. These new computational challenge has extensive applications in hospitals, banking services, army, and navy and even in factory environment. For example, out stationed doctors/ nurses can serve the patients in the hospital by monitoring their behavior on these cell phones using visual inspection systems. A team leader in a factory can serve his team form a far end by noticing them failure in executing a complex task. In recent times pervasive and ubiquitous computing is employed in intelligent user assistance systems. User assistance is a crying need in many safety-critical applications, such as driving a heavily loaded truck on a risky highway. Various human-machine interactive systems have been developed to handle this problem, but unfortunately no pertinent solution has been found until this time. Recently affective states of the worker are taken into consideration to determine the psychological changes of their mind while working in an accident-prone environment. Necessary control signals/alarms are generated to prevent accidents. Emotional Intelligence has a great role to measure the affective states of the workers form the facial expression. Usually multiple cameras are employed to measure the changes in facial expression of the workers form different angles, in order to derive specific psychological changes form their facial expressions. An autonomous alarming system is then used to generate alarm synchronously with the abrupt changes in the psychological states.

9.10 Emotion Recognition from Voice Samples

273

Various inferential machineries have been developed to effectively derive the control signals/alarms in modern user assistance system. Li and Ji [29] in a recent work introduced a probabilistic framework based on dynamic Bayesian network and information theory to design an efficient user assistance system. They use a generic hierarchical probabilistic framework for user modeling to obtain sensory data and contextual information related to users affective states. This framework is then dynamically evolved to take into account the temporal changes in sensory data as a result of changes in users affective states. User states are then determined by affective states and then selecting the “most informative” sensory strategy to confirm or refute the hypothesized user states.

9.10 Emotion Recognition from Voice Samples Emotion recognition form voice [27], [28] has many interesting applications. For example, recognizing the speaker from telephonic threat is well known problem to identify anti-socials. Prosodic features to speech are of important concern to determine emotion from voice samples. Prosody concerns with variations in syllable stresses and timing to form rhythmic patterns. [40]. The prosodic variations are used to determine emphasis attitude and intentions of speaker form an acoustic point of view, the features that play important role in prosody include duration of speech, amplitude of sound and the variation in pitch characteristics. Typically pitch is defined as the fundamental frequency F0 at which the vocal cords vibrate during speech. Actually, pitch is difficult to measure accurately. We can have a perception of pitch form the fundamental frequency F0, as it serves as the primary cue to intonation and stress. The nature of the F0 contour over an utterance characterizes the emotional state, the speaking style and the head motion of the speaker. Several algorithms on pitch detection are available in the current literature [40]. Typically pitch detection can be performed using time domain, frequency domain and time plus frequency domain analysis of the speech signal. Frequency domain pitch detectors rely on the frequency spectrum of the speech signal, which includes a series of impulses at the fundamental frequency and its harmonics. Consequently, simple measurement of the frequency spectrum provides an estimate of the period of the signals and hence the pitch. Time domain pitch detectors directly operate on the speech waveform to estimate the pitch period. These detectors often make measurements of peaks, valleys, zero crossing and auto correlation on speech signal to evaluate the pitch. Hybrid pitch detectors incorporate both time and frequency- domain features for pitch detection. For example, a hybrid pitch detector might use frequency domain technique to generate a spectrally flattened time waveform, and then employ auto-correction measurement to estimate pitch period. 9.10.1 Speech Articulatory Features Speech has a great role to play on the dynamics of the face. The sound provided by a speaker is mainly determined by the shape of the vocal tract. Consequently, articulatory features, which measures an approximate vocal tract envelope,

274

9 Applications and Future Directions of Emotional Intelligence

provides a good estimate of facial dynamics. Among the articulatory speech parameters, formants are most popular. A formant represents a peak in an acoustic frequency spectrum, resulting from the resonant frequencies of the acoustical system. Usually, formants are considered as the resonant frequencies of the vocal tract. The formants with the lowest frequencies are called F1. The second, third and subsequent resonant frequencies are called F2, F3, F4 and the like. The first two formants are sufficient to disambiguate the vowels. To recognize vowels we need 4 formants. Linear prediction coding (LPC) is one of the fundamental methods for analysis of formants. It has become popular as it gets round typical difficulties of measuring them directly in the sound wave. An advantage of LPC lies in the estimation of the resonance models of the speaker’s vocal tract. The vocal tract consists of 3 sections, namely the pharynx, the mouth and the nasal cavity. Shape of these components can be controlled by the movement of the vocal organs, such as the tongue, the lips and the soft palate. Phonemes in a speech are distinguished from one another by resonance of the vocal tract and hence by the formants. Hence by the first two or three formants. Spectral power of the speech signal which also has co-relation with facial movement, such as the lips and the jaw, is also an important feature for emotion recognition from voice. Usually the spectral power is evaluated by the computing the average energy per speech frame. Mathematically, it is defined as N –1

Ps = (1/N) ∑ xn2

(9.2)

n =0

where N is the total no of samples in a speech frame, and xn is the n-th sample of the speech signal. Tripathi in her Ph.D. thesis [38] considered pitch 1st 4 formants, F1, F2, F3, F4 and peak power at the 4 formants (measured in dB) for recognition of emotion for voice samples. It has been noted from experiments that for anger and fear pitch are normally very high whereas for sadness and neutral emotion pitch is significantly low. Further it is noted that pitch is moderate for happiness. Formants F1, F2, are relatively higher for disgust compared to all other emotions. It is also noted that the peak power at F1 and F4 is significantly smaller for fear compared to the peak powers of other emotions. Datta in her M.E. thesis [10] employed learning vector quantization algorithm for classification of emotion using speech samples. The vector quantization scheme is designed using a set of code book vectors. Each class of object is represented by a code book vector and a data vector is assigned to the class to which the closest code book vector belongs. The code book vectors in the present context are called “neurons”, of vector quantization (VQ) network. VQ networks are trained with supervised learning algorithms. A typical algorithm for learning vector quantization (LVQ) is given below.

9.11 Personality Building of Artificial Creatures

275

Main steps of the used LVQ algorithm Begin 1. Randomly initialize weight vectors, and also initialize the learning vector. 2. While stopping criteria not met do Begin a. For each training vector xi find j for weight wij to minimize D(j) = ∑ (wij –xi)2 ∀i b.

Update Wj = [ W1j, W2j, ……,Wnj] by the following rule. If target T = Cj then adopt wj wj + α[x – wj] Wj ← Else Wj ← wj - α[x – wj]

Reduce learning rate α by α =α -δ End while; End. After the LVQ algorithm converges, we can submit any input vector X and identify the weight vector of the neuron closest enough of the input vector X with respect to Euclidian distance norm. The unknown input vector X thus is clustered into a single neuron with matched weight vector. The classification accuracy has been found to be correct to a level of 76%.

9.11 Personality Building of Artificial Creatures Artificial creatures in particular robots are now considered a new species, whose behavior can be controlled by their emulated genes [11]. The artificial creature usually is a man-made organism, the structure and fundamentals of which are designed based on the mechanism of real creature. Researchers are keen to develop personalities in such creatures [37]. In order to maintain similarity in behavior of these artificial agents, human genetic codes are employed to represent personality of the creatures. The genetic code determines the artificial creatures inherent feeling of happiness, sadness anger, fear etc and inherent manner of response. Kim and Lee [24] proposed a novel approach to multi-objective optimization to personality building, where the dimension of the personality model is defined as the optimization objective. In their multi-objective formulation, the objectives are set as agreeable antagonistic, extroverted, introverted, conscientious and negligent models by considering 5 personality dimensions [8], [31]. By applying Strength Pareto Evolutionary Algorithm (SPEA) [40] they considered a set of genomes,

276

9 Applications and Future Directions of Emotional Intelligence

which characterize various personality of the artificial creatures. When a stimulus appears to the artificial creatures, the perception module of the agent classifies the stimulus, and the attention module selects one attentional stimulus from the incoming stimuli. The attentional module holds the attention of the artificial creature towards the selected stimulus, until bigger (more important) stimulus is received. A context module is employed to gather information “when”, “where”, “what”. The architecture of artificial creature includes an internal state module, which consists of excited state of body and mind to prepare itself for a complex task (behavior) before its execution. The internal state module activates the internal states on motivation, homeostasis and emotion, the response of which goes to the behavior selection module through the memory module. The memory module continuously adapts its short-term, long-term and task oriented working memory through an autonomous learning module. The behavior selection module activates the necessary motor modules for execution of a complex task. The schematic internal architecture of an artificial creature is given in Fig. 9.4 following Kim and Li [24]. The artificial creature designed by Kim and Li considered a genomic pool of 14 chromosome C1 through C14, which has the ability of perceiving 47 different types of percepts and 77 different behavior as response. The 1st 6 chromosomes C1 –C6

Sensor Module (Virtual or Real Sensors)

Perception Module

Context Module

Internal State Module

External Environment

Motivation

Homeostasis

Emotion

Memory Module Learning Rule

Reflexive Behavior Module

Short-Term

Long-Term

Behavior Selection Module

TaskOriented

Attention Module

Motor Module

Fig. 9.4. Schematic Architecture of an artificial creature.

9.12 Multimodal Emotion Recognition

277

are related to motivation: curiosity (C1), intimacy (C2), monotony (C3), avoidance (C4), greed (C5) and desire to control (C6). The chromosome C7-C9 is related to homeostasis: fatigue (C7), drowsiness (C8) and hunger (C9). The last 5 chromosomes C10 – C14 correspond to emotion: happiness (C10), sadness (C11), anger (C12), fear (C13) and neutral (C14). Each chromosome Ck for k=1 to 16 consists of 3 gene vectors: the fundamental vector (F-gene vector), the internal state related gene vector (I- gene vector) and the behavior gene vector (B- gene vector). Genes are represented by real numbers. F – genes have a range 0 to 1, I – genes vary between -0.5 to +0.5, whereas B – genes vary between 0 to 1. In artificial creature design, Kim and Lee [24], considered several groups of percepts including posture, obstacle-avoidance, object-detected etc. For each desired percept, the creature has to trigger its internal state and actions, such that an error norm, defined by the unsigned difference between desired percept and corresponding action is minimized. When multiple objectives are desired simultaneously, we have more than one objective functions, which needs to be addressed by a multi-objective optimization algorithm like non-dominated sorting genetic algorithm, details of which is beyond the scope of the book.

9.12 Multimodal Emotion Recognition The interaction between humans and computers can be made more natural if computers can recognize non-verbal communications of the users, such as emotions. Several methods to recognize emotion from facial expression [4], [7], [39] or speech [13], [29] have been invented. However, there is a scarcity of works on multimodal emotion recognition that takes care of facial expression and/or electroencephalogram to accurately determine the emotion of the users. 9.12.1 Current Status In recent times, researchers are motivated to fuse multimodal information for improving accuracy and robustness in emotion recognition. One of such works undertaken by Busso, et al. [3] considers changes in marker positions on the facial expression of a user along with her voice to determine four emotion classes: sadness, anger, happiness and neutral state. The markers on the face help in the motion of facial muscles, thereby clearly visualizing the changes in facial expression due to activation by external stimulus. For recognition of emotion from speech samples, researchers are keen to use global supra-segmental/prosodic features as their acoustic cues. For instance, mean, standard deviation, maximum and minimum of pitch contour and energy in the utterances are widely being used in this regard. Dellart et al. [13] classified human emotions into four classes by employing pitch-related features. They realized three different emotion classifiers by using the well-known K-nearest neighbors (KNN), Kernel Regression (KR) and Maximum Likelihood Bayes Classifier (MLB) algorithm. Roy and Pentland [32] used Fisher linear classifier to classify emotions into distinct classes. They conducted experiments with features extracted from the measure of pitch and energy, and obtained an accuracy ranging from 65% to 88%. The main drawback of the

278

9 Applications and Future Directions of Emotional Intelligence

global level acoustic features lies in the inability of describing dynamic variations along an utterance. Dynamic variation in emotion from speech can be traced in spectral changes at a local segmental level using short-term spectral features. For example, 13 Mel Frequencies Cepstral Coefficients (MFCC) were used in [25] to recognize four emotions using Hidden Markov Model (HMM). Very little effort has so far been given on designing emotion recognition system that includes both facial expression and acoustic information as input. De Silva et al. [8] presented a rule based audio-visual emotion recognition system. In their system, the outputs of the unimodal classifiers are fused at the decision level. They extracted prosodic features form the audio and maximum distance and velocities between six specific facial points. Chen et al. [6] presented a similar approach to determine the dominant modality to resolve discrepancies between the outputs of unimodal system. Both the researchers [6], [12] arrived at same conclusion that the performance of the system increases significantly when both modalities were used together. In [5] and [20], a bimodal emotion recognition system was proposed to recognize six emotions, where the audio-visual information was fused at feature level. Here, the researchers used prosodic features from audio and movement of facial organs on video. Finally, the best features from both unimodal systems were submitted to a bimodal classifier for emotion recognition. The experimental results obtained by them demonstrate that the bimodal recognition system can accurately recognize emotion to 97.2% in comparison to only audio or video based recognition, which yield an accuracy of 75% and 69.4%. Busso [3] considered two approaches for fusing speech and visual data for emotion recognition, and compared the relative merits of both techniques. Their approaches include feature-level and decision-level data fusion. In the feature-level bimodal classifier, the anger and the relaxed state were accurately recognized, compared to the facial expression based unimodal classifier. In decision-level bi-modal classifier, sadness and happiness were classified with good accuracy. Therefore the choice of feature-level/decisionlevel fusion depends on the application. 9.12.2 Research Initiative at Jadavpur University A small research team at Artificial Intelligence Laboratory of Jadavpur University also started working on multi-modal emotion recognition from voice and facial expression of the subjects1. The experiment was conducted with 10 subjects, where each subject was asked to utter a given sentence with emotions in both his/her voice and facial expression. The voice and facial expressions are recorded for subsequent feature extraction and classification of emotions. In Fig. 9.5, we present the speech waveform of a subject, conveying 7 different emotions in the utterance of a given sentence: “what are you doing here?”. The features extracted from the voice samples include: i)time required to complete the sentence, ii) pitch, iii) first three formants, and iv) power at the three formants. Similarly, from the facial expressions (Fig. 9.6), we determine mouth-opening, eye-opening and 1

This work was part of the M.E. Thesis by Anisha Halder, who initiated the early research on multi-modal emotion recognition at Jadavpur University.

9.12 Multimodal Emotion Recognition

Neutral:

Happy:

Sad:

Fear:

Disgust:

Anger:

Surprise:

Fig. 9.5. Speech waveform for different emotions.

279

280

9 Applications and Future Directions of Emotional Intelligence

Relaxed:

[--------What----------are---------you---------doing--------here---]

Happy:

[--What--------are--------you----------doing----------here---]

Sad:

[------What-------------are-------you-------doing------here--]

Fear:

[What--------are--------you------------doing------------here---] Anger:

[What---------are--------you------------doing-----------here----]

Disgust:

[-------What--------------are--------you-------doing------here---]

Surprise:

[What--------are---------you------------doing-----------here----] Fig. 9.6. Recorded facial expressions while pronouncing the given sentence.

9.12 Multimodal Emotion Recognition

281

Table 9.1. Features extracted for relaxed emotion Features

1st

2nd

3rd

4th

5th

6th

Eye opening in pixels

7

7

7

6

7

7

Mouth opening in pixels

7

8

11

11

10

7

Eye-brow constriction in pixels

1

2

1

1

1

2

Table 9.2. Features extracted for happy emotion Features

1st

2nd

3rd

4th

5th

6th

Eye opening in pixels

7

9

11

11

5

7

Mouth opening in pixels

12

12

13

14

13

10

Eye-brow constriction in pixels

1

3

2

2

1

2

Table 9.3. Features extracted for sad emotion Features

1st

2nd

3rd

4th

5th

6th

Eye opening in pixels

7

7

6

5

5

7

Mouth opening in pixels

7

8

9

9

10

8

Eye-brow constriction in pixels

1

1

2

1

1

2

Table 9.4. Features extracted for emotion anger Features

1st

2nd

3rd

4th

5th

6th

7th

Eye opening in pixels Mouth opening in pixels Eye-brow constriction in pixels

7 9 2

10 11 3

12 14 9

13 13 6

13 14 5

13 15 5

8 13 3

Table 9.5. Features extracted for emotion disgust Features

1st

2nd

3rd

4th

5th

6th

7th

Eye opening in pixels Mouth opening in pixels Eye-brow constriction in pixels

6 8 9

5 10 19

4 11 20

4 9 21

5 9 19

6 10 15

6 8 8

Table 9.6. Features extracted for emotion fear Features

1st

2nd

3rd

4th

5th

6th

Eye opening in pixels Mouth opening in pixels Eye-brow constriction in pixels

9 14 2

11 13 3

12 11 9

13 10 8

12 9 7

9 11 6

282

9 Applications and Future Directions of Emotional Intelligence Table 9.7. Features extracted for emotion surprise Features Eye opening in pixels Mouth opening in pixels Eye-brow constriction in pixels

1st 10 10 1

2nd 12 12 2

3rd 12 11 4

4th 13 10 3

5th 11 10 2

6th 12 8 2

Table 9.8. Features extracted from voice data Emotion

Pitch Time duration of (Hz) speech (secs)

Formnt 1 (Hz)

Formnt 2 Formant 3 Power at (Hz) (Hz) Formant 1

Power at Formant 2

Power at formant 3

neutral

3.93

182.944

265.625

1263.7

2043.0

0.000841

0.0000339

0.0000140

happy

2.85

196.721

265.625

1310.5

2105.5

0.000482

0.0000581

0.0000125

sad

3.87

175.862

109.375

1169.9

2699.2

0.0018

0.00002854

0.0000207

fear

3.57

142.433

281.25

1482.4

2043.0

0.0010

0.0006623

0.0000200

disgust

3.21

204

171.875

1388.7

2449.2

0.0024

0.0000182

0.00002097

anger

3.21

298.763

109.375

1044.9

2308.6

0.0075

0.0000687

0.0000278

surprise

3.21

257.07

140.625

1888.7

2683.6

0.0006339

0.00002779

0.00000479

the length of eyebrows’ constriction. Table 9.1-9.7 and Table 9.8 provide experimentally obtained measurements of the facial and voice features respectively. These features are fuzzified in three scales: LOW, MEDIUM and HIGH, and the fuzzified features are used to map to the emotion space using pattern recognition technique. Various pattern recognition techniques, including neural nets, fuzzy logic, and others can be employed for mapping of fuzzified features onto emotion space. Experiments undertaken so far with 50 volunteers reveal that the recognition accuracy for multi-modal emotion recognition system is around 97%, which is higher than facial expression or voice based recognition. It is to be noted that recognition accuracy for voice-based and facial expression-based emotion is low around 68% and 92% respectively. The experiment was undertaken using Mamdani type fuzzy reasoning as introduced in Chapter 5. In our future research, we would examine the scope of other pattern recognition techniques for multi-modal emotion recognition. We also have plans to fuse other biopotential signals like electroencephalography (EEG), electrocardiogram (ECG), pulse rate, and body temperature with facial expression and voice to determine emotion of a subject. Such scheme would help in improving the recognition accuracy further. The effect of noise on the biopotential signal has an important role on the recognition accuracy of emotions. In this regard, we already performed some studies on misclassification of emotions in presence of Gaussian noise with zero mean and small variance. The study reveals that for small changes in noise variance over a zero mean, the recognition scheme is much robust. However, when the noise variance goes beyond a threshold, recognition accuracy falls off for certain emotions like anxiety, anger, and fear as they are misclassified into other emotions.

9.13 Parameter Identification of Emotional Dynamics

283

9.13 Para meter Identificatio n of Emotio nal Dy na mics

9.13 Parameter Identification of Emotional Dynamics Using EEG and fMRI Brain Imaging 9.13 Para meter Identificatio n of Emotio nal Dy na mics

In this chapter, we propose a general scheme for parameter identification of emotional dynamics using EEG and fMRI instruments. One typical set-up could be NeuroScan’s 64-channel electroencephalography (EEG), and functional Magnetic Resonance Imaging (fMRI) instruments. A computerized stimulator STIM2 of Neuroscan is also needed to stimulate the emotional counterpart of the human brain with audio-visual stimulus. The EEG machine would receive the stimulated response of the brain at different locations and amplify it, and transfer it to a desktop computer for displaying the EEG response. The fMRI image of the brain needs to be pre-recorded to determine the geographical location of various lobes in the brain. A neuro-scan software may be employed to determine the locations of origin of the stimulated responses. For instance, the left amygdale may respond to a particular type of simulation, whereas the anterior cingulated cortex may be excited with a different stimulation. The chaotic behavior of the emotional dynamics has similarity with the chaotic emotional states, originated at the frontal lobe of the human brain especially at the amygdale, anterior cingulated cortex (ACC) and the medial frontal cortex (MFC) regions. It is already known that the amygdale region [2] plays a vital role in actuating different involuntary movements associated with olfaction and licking [11], originating feelings for love/hatred, and generating reaction to fear stimuli. The ACC is mainly responsible for the manifestation of fear and anger stimuli on human facial expressions [8]. Since the parameters of an emotional dynamics (vide equation (7.1)) represent functions of time-constants, we can determine them from the time-delay analysis of stimulated emotional reactions. Freeman [16] and Kay et al. [22] made an interesting study to compare EEG time series data from rat olfactory system with a model composed of nonlinear coupled oscillators. They used an Ordinary Differential Equation (ODE) model to represent the olfaction dynamics, and determined parameters of the dynamics by Hebbian type unsupervised learning. Since both the EEG dynamics and the model exhibit chaotic and limit cyclic behavior, determination of the suitable range of the parameters that yield limit cycles and chaos is of practical interest for this type system identification problem. Inspired by the work of Freeman [16], we in this chapter propose a model of system identification, which might help in autonomously determining the parameters of the dynamics by machine learning/evolutionary computing approach. Once the parameters of the dynamics are determined, we can employ the model of the dynamics in psychotherapy and human-machine interactive systems of the next generation. 9.13.1 System Identification Approach to EEG Dynamics Modeling by Evolutionary Algorithms In this section, we present a technique to match the stimulated EEG response of the human forebrain with the response of our proposed model. This has similarity with the system identification problem [26] in control engineering. A question

284

9 Applications and Future Directions of Emotional Intelligence

Amygdale (forebrain) Visually inspired external stimuli

+ ∑

Model of Emotional Dynamics

−

Evolutionary Algorithm

Fig. 9.7. Emotional system identification by evolutionary algorithm.

0.6 0.4 0.2 0.0 - 0.2 - 0.4

100

200

300

Fig. 9.8. Stimulated response of the left amygdale to visually inspired stimuli paired with food reward, where x-axis denotes time in ms and y-axis denotes voltage in mv.

then naturally arises why to identify the system parameters of our emotional dynamics. This, in fact, is needed to use the emotional model for determining the emotion of a person at any time t from his known emotional states at time t = t0. A schematic diagram for identification of the system parameter is presented in Fig. 9.7. In this figure, the stimulated emotion of a person (at time t ≥ t0) in a certain region in the human forebrain is first recorded on an EEG machine. Fig. 9.8 is such an instance obtained from the left amygdale, indicating the appetitive conditioning of a patient to visual stimuli paired with a food reward [21]. How do we adapt the parameters of the emotional dynamics? Since there is no guidance to handle this problem, we use a random search in prescribed ranges of

9.13 Parameter Identification of Emotional Dynamics

285

individual parameters. Any classical evolutionary algorithm may be employed here to identify the parameters of the emotional dynamics. It is indeed important to note that classical evolutionary algorithm, such as genetic algorithm [17], particle swarm optimization (PSO) algorithm and differential evolution (DE) algorithm may easily be employed here to minimize the error square norm [26] to whatever extend possible. In this section, we discuss the scope of these algorithms in error minimization and system identification. 9.13.2 Genetic Algorithm in Emotional System Identification A look at the emotional dynamics (Equation 7.1) reveals that for n emotional states: x1, x2, …., xn, we have a set of n number of differential equations with parameters aii ( i=1 to n), bji ( j= 1 to n, i = 1 to n) and cki ( k = 1 to n, i = 1 to n) respectively. The other parameters of the dynamics, such as K, βji and λki are presumed to be constant in our emotional system. System identification thus refers to determination of aii, bji, cki respectively. Since aii, bji, cki all have 2-dimensional indices, we can represent them by matrix of dimension (n × n) respectively. The chromosomes in the present context are thus represented by matrices, and a matrix type chromosome operator is used to allow variation in the population at each generation of the genetic cycle. The matrix type crossover operation is introduced in Fig. 9.9.

Parents:

Offspring:

Fig. 9.9. Crossover of two parent matrices, denoted by two shades, generates 2 offspring matrices.

The crossover operation, which in this case is described by a small window, is performed around a crossover point, i.e. the left top most point of the window. The mutation operation here has been defined non-conventionally. Let (aii)max be the maximum element in a matrix A= [aii]. The mutation operation transforms aii by ((aii)max – aii) for any 1 ≤ i ≤ n. Similarly, any matrix B and C can be mutated by the above processes. The mutation point and the crossover point are determined randomly, i.e., by generating two random numbers between 1 to n to identify the location of the crossover and mutation point in a given matrix of (n × n) dimension.

286

9 Applications and Future Directions of Emotional Intelligence

The fitness function in the present context evaluates the error between the time samples of the amygdale response, and the corresponding samples of the response for the given model. Let Aj be the jth –sample of the amygdale response and Mj be the jth – sample of the model response. We in the present context define fitness function E as l E = ∑ ( Aj – Mj)2 j =1

(9.3)

where l denotes the number of samples taken on the 2 responses at a uniform interval of T seconds. The objective in the present context is to determine the right candidate matrices A, B and C so as to minimize error |E| as far as possible. The typical steps used in a genetic algorithm, such as selection, crossover and mutation are followed here to perform the system identification process. One important and noteworthy issue is to select the ranges of the parameters aii, bji and cki. This in the present context has been fixed from our background in computer simulation of the emotional dynamics. Computer simulation undertaken here reveals that aii, bji and cki lie in the range: 0.001≤ aii, bji, cki ≤ 0.3. For the best response, the crossover and mutation probabilities have been selected 0.6 and 0.2 respectively. 9.13.3 Particle Swarm Optimization in Emotional System Identification Particle Swarm Optimization algorithm (PSO) [14], [23] is inspired by the behavior of the biological creatures, such as eagles to identify food in a given environment. It has been noted that an eagle joins a team with the largest eagle population in a forest with the help of having a good share of food. James Kennedy, Russell and Eberhart in 1995 employed this bio-inspired phenomena to model the dynamics of stochastic particles with an ultimate aim to optimize non-linear functions of significant complexity. PSO in principle is a parallel multi-agent search technique. Particles are conceptual entities which fly through the multi-dimensional search space. At any particular instant, each particle has a position and a velocity. The position vector of a particle with respect to the origin of the search space represents a trial solution of the search problem. At the beginning, ar population of particles is rinitialized with random positions marked by vectors X i and random velocities Vi . The population of such particles is called a ‘swarm’ S. A neighborhood relation N is defined in the swarm. N determines for any two particles Pi and Pj whether they are neighbors or not. Thus for any particle P, a neighborhood can be assigned as N(P), containing all the neighbors of that particle. Different neighborhood topologies and their effect on the swarm performance have been discussed in [23]. The

9.13 Parameter Identification of Emotional Dynamics

287

PSO used in this work, implicitly uses a so-called fully connected neighborhood topology (or gbest). Every particle is a neighbor of every other particle. Each particle P has two state variables: r ¾ Its current position x (t ) r ¾ Its current velocity v (t ) . And also a small memory comprising, r ¾ its previous best position p(t ) i.e. personal best experience in terms of

r

the objective function value f ( p (t )) . r r The best p(t ) of all P ∈ N (P ) : g (t ) i.e. the best position found so far in the neighborhood of the particle. The PSO scheme has the following algorithmic parameters: r ¾ V max or maximum velocity which restricts Vi (t ) within the inter¾ ¾ ¾

val [−v max , v max ] . An inertial weight factor ω. Two uniformly distributed random numbers φ1 and φ2 which respectively r r determine the influence of p(t ) and g (t ) on the velocity update formula. Two constant multiplier terms C1 and C2 known as “self confidence” and “swarm confidence” respectively.

r r r r r Initially the settings for p(t ) and g (t ) are p(0) = g (0) = x (0) for all particles. Once the particles are initialized, the iterative optimization process begins where the positions and velocities of all the particles are altered by the following recursive equations. The equations are presented for the d-th dimension of the position and velocity of the i-th particle. Vid (t+1) = ωVid (t) + C1φ1. (Pd(t) -Xid (t)) + C2φ2. (gd(t)-X id(t)) Xid (t+1) = Xid (t) + Vid (t+1)

(9.4)

The first term in the velocity updating formula represents the inertial velocity r of the particle. The second term involving P(t ) represents the personal experience of each particle and is referred to as “cognitive part”. The last term of the same relation is interpreted as the “social term” which represents how an individual particle is influenced by the other members of its society. After having calculated the velocities and position for the next time step t+1, the first iteration of the algorithm is completed. Typically, this process is iterated for a certain number of time steps, or until some acceptable solution has been found by the algorithm or until an upper limit of CPU usage has been reached. The algorithm can be summarized in the following pseudo code.

288

9 Applications and Future Directions of Emotional Intelligence

Procedure Particle_swarm_optimization

set t = 0; initialize φ1, φ2 , Vmax and define N; while (termination_condition = FALSE) { r r ∀p ∈ S : calculate v (t + 1) and x (t + 1) using equations (9.4); r r r r ∀p ∈ S : update p(t + 1) with x (t + 1) if f ( x (t + 1)) is better than f ( x (t )) r r ∀p ∈ S : update g (t + 1) with the best p(t + 1) in N(p); } Once the iterations are terminated, most of the particles are expected to converge to a small radius surrounding the global optima of the search space. In the present system identification problem, we describe 3 matrices [aii]n×n, [bij]n×n and [cki]n×n by a single vector as evident from Fig. 9.10.

……. aii ………

……..bji ……..

………cki …….

(3n × 1) where

a11 a12 …a1n a21 a22…a2n …an1an2….ann ≡

……aii……..

…..bji….

≡

b11 b12 ….b1n b21 b22…b2n …bn1bn2….bnn

……cki…

≡

c11 c12 …c1n c21 c22…c2n …cn1cn2….cnn

Fig. 9.10. Representation of matrices [aii], [bji] and [cki] by a single vector.

The matrixes [aii], [bij] and [cki] are represented by a single vector for convenience of implementation of PSO based emotional system identification. The PSO based identification scheme initiates with a random set of 20 chromosomes described by position vectors of dimension (3n2 × 1), where n2 denotes the number of element in the matrix A, B and C respectively. Fitness of the individual particle is then evaluated and velocity and position updating equation are recurrently executed over the iterations of the PSO algorithm. The algorithm is then invoked for 2000 iterations and the position of the best-fit particle is noted. The position of best-fit particle is declared as the parameter set of the proposed emotional system.

9.13 Parameter Identification of Emotional Dynamics

289

9.13.4 Differential Evolution Algorithm in Emotional System Identification

Like the PSO, Differential Evolution, (DE) [36] is also a novel algorithm of current interest, which has received attention by the researchers in optimization theory. In DE algorithm we represent a n-dimensional point by a position vector of n-components. DE starts with a population of N, D-dimensional parameter vectors. The i-th vector of the population at iteration (time) t is r X i ( t ) = [ x i ,1 ( t ), x i , 2 ( t ), x i , 3 ( t )..... x i , D ( t )]

(9.5)

r In each iteration of the algorithm, for each population member X i (t ) , a ‘do-

r nor’ vector Vi (t + 1) is created. It is the method of creating this donor vector that distiguishes various DE schemes from one another. In “scheme DE1” r (DE/rand/1), to create Vi (t + 1) , three vectors (say r1, r2, and r3) are randomly chosen from the current population. Next the difference of any two of these three vectors is scaled by a scalar F and the scaled difference is added to the third vector whence we obtain the donor vector:

v i , j (t + 1) = x r1, j (t ) + F .( x r 2 , j (t ) − x r 3 , j (t ))

(9.6)

The DE family of algorithms use two kinds of crossover, namely ‘exponential’ and ‘binary’. In ‘exponential’ crossover, we first choose an integer n randomly from [0, D-1]. We choose another integer L from the interval [1, D]. An offspring vector:

r U i (t + 1) = [u i,1 (t + 1), u i , 2 (t + 1), u i, 3 (t + 1),...,ui , D (t + 1)]

(9.7)

is then formed: u i,j (t+1) = Vi,j (t+1) for j =< n >D , < n +1 >D,...< n − L +1 >D j ∈[0, D − 1] = X i,j (t+1) for all other (9.8) where the angular brackets < >D denote a modulo function with modulus D. The integer L is drawn from [1, D] according to the following policy: L = 0; Do L=L+1; while (rand (0, 1) < CR) AND (Lm) = (CR) m-1 for any m>0. CR, the crossover con-

r

stant, is a control parameter of DE. For each donor vector Vi , a new set of n and L must be chosen randomly as shown above. ‘Binary’ crossover is implemented as follows:

ui , j (t + 1) = =

vi, j (t +1)

if rand(0, 1) < CR

xi , j (t ) otherwise

(9.9)

290

9 Applications and Future Directions of Emotional Intelligence

Selection favors the better between the parent and the child: r r r r f ( U X i(t + 1) = U i (t + 1) i (t +1)) < f ( X i (t)) if r r r X i (t ) f ( X i (t )) < f (U i (t + 1)) if

(9.10 )

where f( ) is the function to be minimized. DE/best/1 follows the same procedure except that the donor vector is created using two randomly selected members of the population as well as the best vector of the current time step: r r r r r r (9.11) V (t + 1) = X (t ) + λ.(X (t ) − X (t )) + β.(X (t ) − X (t )) i

i

best

i

r2

r3

where λ is another control parameter of DE in [0, 2]. To reduce the number of control parameters a usual choice is to put λ = F. Storn and Price [8] suggested a total of ten different working strategies for DE. In this chapter, we have used DE/rand/1/bin for comparison with the proposed method.

Average fitness→

GA

PSO DE No. of function evaluations →

Fig. 9.11. Performance (speed of convergence) study of DE with PSO and GA in the emotional system identification problem.

The DE-algorithm presented above is executed for the system identification problem undertaken here. We considered a population –size of 20 chromosomes, and obtained the parameter vector after 1200 genetic iterations. The estimated parameters obtained by PSO, DE and GA is then compared, and is found comparable. But convergence-time of the three algorithms differs significantly. Fig. 9.11 reveals that for obtaining a given average fitness, the DE evaluates less functions than PSO, and PSO evaluates less functions than GA. So, the DE converges earlier than the PSO, and PSO converges earlier than GA.

9.14 Conclusions The chapter introduced a wide variety of applications of emotional intelligence, and also outlined possible direction of future research. Emotional intelligence is an

References

291

open-ended discipline of knowledge, and naturally putting an end to this diverse discipline in a single chapter is merely impossible. The objective of this chapter, however, was different. The authors wanted to motivate the young researchers to develop their own models/methods to fruitfully capture one or more interesting problems of emotional intelligence, and design an effective system to successfully handle the problem.

References 1. Blairy, S., Herrera, P., Hess, U.: Mimicry and the Judgement of Emotional facial Expressions. J. of Nonverbal Behavior 23, 5–41 (1999) 2. Bulut, M., Narayanan, S.J., Syrdal, A.K.: Expressive speech synthesis using a concatenative synthesizer, http://sail.usc.edu/publications/BulutNarayananSyrdal.pdf 3. Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C.M., Kazemzadeh, A., Lee, S., Neumann, U., Narayanan, S.: Analysis of Emotion Recognition using Facial Expressions, Speech and Multimodal information. In: ICMI 2004 (October 13-15, 2004) 4. Carroll, J.M., Russell, J.A.: Do facial expressions signal specific emotions? Judging Emotions from the face in Context. J. of Personality and Social Psychology 70, 205– 218 (1996) 5. Chen, L.S., Huang, T.S.: Emotional expressions in Audiovisual Human Computer Interactions. IEEE Int. Conf. on Multimedia and Expo. 1, 423–426 (2000) 6. Chen, L.S., Huang, T.S., Miyasato, T., Nakatsu, R.: Multimodal human emotion/expression recognition. In: Proc. of Int. Conf. on Automatic Face and Gesture Recognition, Nara, Japan. IEEE Compu. Soc., Los Alamitos (1988) 7. Cooley, E.L., Nowicki, S.: Discrimination of facial expressions of emotion by depressed subjects. Genetic, Social and General Psychology Monographs 115, 449–465 (1989) 8. Costa, P.T., McCrae, R.R.: The NEO personality inventory manual, Psychological Assessment Resources, Odessa, FL (1985) 9. Curron, E., Sykacek, P., Stokes, M., Roberts, S.J., Penny, W., Johnsrude, I., Owen, A.M.: Cognitive tasks for driving a brain-computer interfacing system: a pilot study. IEEE Trans. on Neural Systems and Rehabilitation Engineering 12(1), 48–54 (2004) 10. Datta, S.: Emotion Detection and Control: A Computational Intelligence Approach, M.E. Thesis. Jadavpur University, India (2007) 11. Dawkins, R.: The Selfish Gene. Oxford University Press, Oxford (1976) 12. De Silva, L.C., Ng, P.C.: Bimodal emotion recognition. In: Proc. of the Fourth IEEE Int. Conf. on Automatic Face and Gesture Recognition, pp. 332–335 (March 2000) 13. Dellaert, F., Polzin, T., Waibel, A.: Recognizing emotion in speech. Fourth Int. Conf. on Spoken Language 3(3-6), 1970–1973 (1996) 14. Eberhart, R.C., Kennedy, J.: A new optimizer using particle swarm theory. In: Proc. of the Sixth Int. Symp. on Micro Machine and Human Science, Nayoga, Japan (1995) 15. Frackowiak, R.S.J., Ashburner, J.T., Penny, W.D., Zeki, S., Friston, K.J., Frith, C.D., Dolan, R.J., Price, C.J.: Human Brain Function, 2nd edn. Academic Press, CA (2004) 16. Freeman, W.J.: Simulation of Chaotic EEG patterns with a dynamic model of the alfactory system. Biol. Cybern. 56, 139–150 (1987)

292

9 Applications and Future Directions of Emotional Intelligence

17. Goldberg, D.E.: Genetic Algorithm in Search, Optimization and Machine Learning. Addison-Wesley, Reading (1989) 18. Goleman, D.: Emotional Intelligence. Bantam Books, New York (1995) 19. Guyton, A.C., Hall, J.E.: Textbook of Medical Physiology. Harcourt Brace and Co., Singapore (1998) 20. Huang, T.S., Chen, L.S., Tao, H., Miyasato, T., Nakatsu, R.: Bimodal emotion recognition by Man and Machine. In: Proc. of ATR Workshop on Virtual Communication Environments, Japan (April 1998) 21. Johnsrude, I.S., Owen, A.M., White, N.M., Zhao, W.V., Bohbot, V.: Impaired preference conditioning after anterior temporal lobe resection in humans. J. of Neuroscience 20(7), 2649–2656 (2000) 22. Kay, L., Shimoide, K., Freeman, W.J.: Comparison of EEG time series from rat alfactory system with model composed of nonlinear coupled oscillator. Int. J. Bifurcation and Chaos 5(3), 849–858 (1995) 23. Kennedy, J., Eberhart, R.C.: Swarm Intelligence. Morgan-Kaufma, San Francisco (2001) 24. Kim, J.-H., Lee, C.-H.: Multi-objective Evolutionary Process for Specific Personalities of artificial Creature. IEEE Computational Intelligence Magazine 3(1) (February 2008) 25. Kim, J.-H., Lee, K.-H., Kim, Y.-D., Park, I.-W.: Genetic Representation for Evolving Artificial Creature. In: Proc. of the IEEE Congress Evolutionary Computation, pp. 6838–6843 (2006) 26. Kuo, B.C.: Digital Control Systems. Holt Sounders, New York (1981) 27. Kwon, O.-W., Chan, K., Hao, J., Lee, T.-W.: Emotion Recognition by Speech Signals. In: EUROSPEECH, Geneva (2008) 28. Lee, C.M., Yildirim, S., Dulut, N., Kazemzedeh, K., Busso, C., Deng, Z., Lee, S., Narayanan, S.S.: Emotion recognition based on phoneme classes. In: Proc. ICSLP 2004 (2004) 29. Li, X., Gi, Q.: Active Affective State Detection and User-Assistance with Dynamic Bayesian Networks. IEEE Trans. on Systems, Man and Cybernetics, Part-A: Systems and Humans 35(1) (January 2005) 30. Magnenat-Thalmann, N., Joslin, C., Berner, U.: Networked Virtual Park. In: Jain, L., Wilde, P.D. (eds.) Practical Applications of Computational Intelligence Techniques. Kluwer Academic, Dordrecht (2001) 31. McCrae, R.R., Costa, P.T.: Validation of a Five-factor model of personality across instruments and observers. Journal of personality and social psychology 52, 81–90 (1987) 32. Roy, D., Pentland, A.: Automatic spoken affect classification and analysis. In: Proc. of the Second International Conference on Automatic Face and Gesture Recognition, pp. 363–367 (1996) 33. Ryback, D., Sweeney, L.: Application of Emotional Intelligence to Schools and Workspace. In: Luciano, L. (ed.) Low Cost Approach to Promote Physical and Mental Health, pp. 485–502 (June 2007) 34. Shi, Y.Q., Sun, H.: Image and Video Compression for Multimedia Engineering: Fundamentals, Algorithms and Standards, ch. 11, pp. 221–248. CRC Press, Boca Raton (1999) 35. Stern, A.: Creating Emotional Relationships with Virtual Characters. In: Trappl, R., Petta, P., Payer, S. (eds.) Emotions in Human Artifacts. MIT Press, Cambridge (2002)

References

293

36. Storn, R., Price, K.: Differential Evolution- A Simple and Efficient Adaptive Scheme for Global Optimization over Continuous Spaces, Technical Report TR-95-012, Berkeley, CA (1995) 37. Trappl, R., Petta, P., Payr, S.: Emotions in Humans and Artifacts, Bradford Book. MIT Press, Cambridge (2002) 38. Tripathy, N.: Study of Face and speech parameters and Identification of their Relationship for Emotional Status Recognition, Ph.D. Thesis, National Institute of Technology, Raipur, India (2006) 39. Wright, B., Clark, N., Jordan, J., Young, A.W., Clarke, P., Miles, J., Nation, K., Clarke, L., Williams, C.: Emotion recognition in faces and the use of visual context V0 in Young People with High Functioning Autism Spectrum Disorders. AUTISM 12(6), 607–626 (2008) 40. Zitzler, E., Thiele, L.: Multiobjective evolutionary algorithms: A comparative case study and the strength Pareto approach. IEEE Trans. on Evolutionary Computation 3(4), 257–271 (1999)

8 Electroencephalographic Signal Processing for Detection and Prediction of Emotion 8 Electroencephalographic Signal Processing

This chapter deals with processing of electroencephalographic signals for detection and prediction of emotion. Prediction of electroencephalographic signal form past samples are needed for early diagnosis of patients, suffering from frequent epileptic seizure and/or psychotherapeutic treatment. The chapter begins with comparing the performance of various digital filter algorithms to identify the right candidate for application in electroencephalographic signal prediction. It then considers clustering/classification of emotion from filter co-efficient, wavelet co-efficient and Fast Fourier Transform co-efficient. It also considers the scope of bio-potential signals, such as skin conductance and pulse count along with electroencephalographic signal on emotion detection of personals. Several algorithms of pattern classification have been examined to determine the best algorithm for the emotion classification problem.

8.1

Introduction

Electroencephalography (EEG) is one of the well-known (and perhaps the oldest among all) [10] brain imaging techniques that provides cognitive underpinnings of various brain processes: reasoning, learning, perception-building and emotion arousals. An EEG system usually includes non-metallic electrodes such as carbon and carbon fiber. These electrodes are placed on the scalp at specific locations, determined by internationally agreed system. In a typical “10/20 system” [11], the distance from the patients’ nasion to inion and between the preauricular points is measured. Each electrode is placed at 10/20 percent intersections along these distances. Fig.1 [4] describes a multi-channel EEG electrode array. Signals obtained by the EEG electrodes are amplified and then fed through a low-pass-filter (<100Hz). The EEG signals represent the electric potential differences in neuronal dendrites from the transmembrane currents. The primary source of EEG signals is presumed to be the current flow in the apical dendrites of pyramidal cells in the cerebral cortex [1]. These cells are located solely in gray matters and are oriented perpendicularly in the cortex. The values of the post-synaptic potentials continually vary, even when the brain is “at rest”. The fluctuation in post-synaptic potentials is described as different frequencies in the EEG-waveform. A. Chakraborty and A. Konar: Emotional Intelligence, SCI 234, pp. 235–260. © Springer-Verlag Berlin Heidelberg 2009 springerlink.com

236

8 Electroencephalographic Signal Processing

Fig. 8.1. A multi-channel EEG electrode array.

In order to localize the source of EEG samples, current research is moving towards an increasing density of electrodes. This is well-known as topographic mapping of source localization in the EEG literature [1], [7], where the smaller the distance between electrodes, the more accurate the mapping. Localization of the source is important for its practical utility. The anatomy and physiology of human brain reveals that specific lobes in the frontal brain are responsible for arousal of specific emotions. For instance, the amygdale is responsible for fear conditioning, threat perception, sadness and disgust, and also for the chaotic behavior in EEG under epileptic seizure [2]. Prediction of EEG from past samples thus has significance to detect possible abnormalities and remedial measure for psychotherapeutic treatment of mental patients. In this chapter, we present different digital filter algorithm to predict EEG signal from past samples and identify the best algorithm from the performance of the filters. Section 8.2 deals with EEG prediction by adaptive filters. The predictive capability of several filters have been studied and relative performance of the filters have been compared. In section 8.3, we discuss the issues of EEG prediction by wavelet coefficients. Section 8.4 provides a discussion on emotion recognition from bio-potential signals by using support vector machines. Principles of emotion clustering by classical neural nets are introduced in section 8.5. Conclusions are listed in section 8.6.

8.2 EEG Prediction by Adaptive Filtering Finite impulse response (FIR) filters are widely being used for prediction of timeseries data. EEG being a time varying signal, thus can be predicted by adaptive FIR filter. FIR filters autonomously adapt their weights to match their response with user-provided desired signal in order to reduce the root means square error. Several filter algorithms namely, LMS, NLMS, RLS and Kalman are prevalent in the existing literature of digital signal processing [9]. In this study, we attempt to predict EEG signal from its previous time samples using adaptive filters, and compare the relative performance of different filter algorithms in order to identify the most appropriate one that yields minimum root means square (RMS) error.

8.2 EEG Prediction by Adaptive Filtering

237

d(k) +

error

+ y(k)

W0

W1

x(k)

x(k-1) Z-1

x(k) W0

Wp-1

Z-1

W1

x(k-p-1) Z-1

Wp-1

W(k+1) = W(k) + α . Error . X(k)

Fig. 8.2. An LMS adaptive FIR filter.

Fig. 8.3. A sample EEG signal.

8.2.1 LMS Filter A Least Min Square (LMS) [5], [8], [16] finite impulse response (FIR) digital filter can be represented by Fig. 8.2. The experiment was conducted by sampling an EEG signal and submitting the discrete samples as x (1), …, x(p-1) to the input of an LMS filter. The original EEG and the predicted (estimated) signal for a filter of order 30 (i.e. p=30) are given in Fig. 8.3 and 8.4 below. The error plot given in Fig. 8.5 demonstrates that the average error is around 0.5 units with large variance. It is noteworthy from

238

8 Electroencephalographic Signal Processing

Fig. 8.4. EEG prediction by LMS filter algorithm.

Fig. 8.5. The error in EEG prediction by LMS filter.

Fig. 8.5 that % RMS error falls off significantly with increasing filter order until p=45. In the above computer simulation, we took α = 0.0002, and the initial weight vector was presumed to be zero. 8.2.2 EEG Prediction by NLMS Algorithm The Normalized Least Min Square (NLMS) [7] filter has similar structure with the LMS filter with a modified weight adaptation rule, given by

8.2 EEG Prediction by Adaptive Filtering

239

Fig. 8.6. The percentage RMS error versus Filter order plot for LMS filter.

Fig. 8.7. EEG prediction by NLMS filter algorithm.

W(k +1) = W(k) + α . error. X(k) / (mean-square (X(k) + offset).

(8.1)

The offset is kept to ensure no division by zero. We used an offset settings of 50, and α = 0.005 for the computer simulation. The original EEG and the predicted EEG by NLMS filter is given in Fig. 8.7, and the error in each sampling instant is plotted in Fig. 8.8. Here, the average value of error is around 0.2, but the variance is large. It is important to note that variance here is less than that of LMS filter. An illustrative weight variation by

240

8 Electroencephalographic Signal Processing

Fig. 8.8. The error in EEG prediction by NLMS filter.

Fig. 8.9. Sample weight variation for NLMS filter prediction.

NLMS algorithm is given in Fig. 8.9. The error versus sample size and order of the filter given in Fig. 8.10 reveals that with both increase in filter order and sample size, the prediction error diminishes to as low as 0.05. 8.2.3

The RLS Filter for EEG Prediction

The Recursive Least Square (RLS) [5] filter employs the following algorithm for prediction.

8.2 EEG Prediction by Adaptive Filtering

241

Fig. 8.10. Error versus sample size and order of the filter.

Here [X(n−1) x(n-2) ….x(n-p)]T for a p order adaptive filter and X(k) = X(n) at n = k. while |error| not less than a predefined ε > 0 do begin evaluate i) error e(n)= d(n) – WTn X(n) ii) g(n) = P(n-1)X*(n){λ + XT(n)P(n-1)X*(n)}-1 iii) P(n) = λ-1 P(n-1) – g(n) XT(n) λ-1 P(n-1) iv) Wn = Wn-1 + e(n) g(n) end While; Here P(n) is the inverse of the weighted auto-correlation matrix of X(k) weighted by the forgetting factor λ. i.e. P(n) = Rx-1(n) where Rx(n) = ∑ λ(n-i) X*(i) XT(i)

(8.2)

d(n) = desired signal, e(n)= error, and Wn = weight vector all at time sample t = n. With W0 = 0, P(0)= δ-1I, where I is a (p+1) × (p +1) identity matrix, and λ = forgetting factor = 0.99. We run the above algorithm until ε = 10-3. It is indeed interesting to note from Fig. 8.11 that the original and estimated signal have very negligible difference. The error plot in Fig. 8.12 also ensures the same point. A sample weight adaptation is shown in Fig. 8.13.

242

8 Electroencephalographic Signal Processing

Fig. 8.11. EEG prediction by RLS filter algorithm.

Fig. 8.12. The error in EEG prediction by RLS filter.

8.2.4 The Kalman Filter for EEG Prediction A Kalman filter [5] is a recursive p-order adaptive digital filter with filter input X = [ x(n-1) x(n-2) ...........x(n-p)] With a measurement equation fi = WnT X – x(n) = 0,

(8.3)

8.2 EEG Prediction by Adaptive Filtering

243

Fig. 8.13. Weight variation for RLS filter prediction.

where Wn is the weight vector at the nth time instant. Wn = [w(1) w(2) ........... w(p)]

(8.4)

We estimate the signal x(n+1) as y(n) = WnTX. Also let Ri be the expected value of WiWiT i.e. Ri = E[WiWiT].

(8.5)

The estimator vector in the present context is given by A = [w(1) w(2) ........... w(p)]T = WnT

(8.6)

Let Si be the expected value of the estimation noise, i.e., Si = E[(Ai – Ai*)(Ai – Ai*)T]

(8.7)

where Ai* is the updated value of the estimator vector. For the signal estimation by the filter we need to determine the following derivatives: dfi/dx= [w(1) w(2) ........w(p)] at ith instant, dfi/dA = [x(n-1) x(n-2) ........x(n-p)] at ith instant

(8.8) (8.9)

244

8 Electroencephalographic Signal Processing

Now the algorithm is as follows: Begin 1. Initiatize: a) R0 = (df0/dx)(df0/dx)T; in this case R0 = a matrix of dimension (p × p) with all its entries zero. b) S0 = a diagonal matrix with large positive value of the diagonal terms, which for this case is taken to be 50*I where I is a (p × p) identity matrix. c) W0 = [0 0 0 0 ……. p terms] 2. Repeat: a) Input new signal sample x(n) and evaluate y(n) = x(n+1)* = WnTX. b) Update Ki as Ki = Si-1 MiT (Wi + MiSi-1MiT) where Mi = dfi/dA. c) Update Ai as Ai* = Ai-1* + Ki [x(n+1) – Mi-1Ai-1*] d) Update Si as Si = [I – Ki Mi] Si-1 e) Update dfi/dx and dfi/dA as indicated in previous equations. Until prediction error is less than a preset value. End. The EEG input and the prediction error using Kalman filter are plotted in Fig. 8.14 and 8.15 respectively. One sample weight adaptation is included in Fig. 8.16. For all the results filter order is taken to be 30 while the number of samples is taken to be 1360.

Fig. 8.14. EEG prediction by Kalman filter algorithm.

8.2 EEG Prediction by Adaptive Filtering

245

Fig. 8.15. The error in EEG prediction by Kalman filter.

Fig. 8.16. Weight variation for Kalman filter prediction.

8.2.5 Implication of the Results A comparison of the RLS and Kalman filter in EEG prediction is undertaken in this research. The computer simulation (Fig. 8.17) envisages that the Kalman filter yields less % RMS error in comparison to RLS filters, irrespective of filter order and sample size. The LMS and the NLMS Filters provide poorer performance than the Kalman and RLS filters, which can be understood from the following Table 8.1 In this

246

8 Electroencephalographic Signal Processing

Fig. 8.17. Performance comparison of Kalman and RLS filters. Table 8.1. Performance comparison of all the 4 filters

Filter Name LMS NLMS RLS Kalman

Percentage RMS Error in Prediction M = 30 M = 60 M = 90 M = 120 S = 1360 S = 2720 S = 4080 S = 5440 10.687 5.775 3.023 2.430 7.615 4.972 3.782 3.060 3.577 2.433 1.754 1.481 0.375 0.167 0.109 0.082

M = 150 S = 6800 1.711 2.566 1.256 0.063

table, M is the filter order while S is the number of samples. It is clearly evident that Kalman filter performs the best followed by RLS Filter. It can be also observed from the Table that in some cases LMS filter performs better than the NLMS Algorithm. This happens at a high filter order and for a high number of signal samples. However, if we examine the RMS error vs. number of samples plot of the LMS and the NLMS filters as shown in Fig 8.18, we see that for a small range of sample numbers, the performance of the LMS filter betters that of the NLMS filter. However, after that range when the number of samples becomes 8704 and the filter order is 96 then the performance of the LMS filter deteriorates rapidly, while that of the NLMS filter remains steady. The chapter thus re-emphasizes the utility of Kalman filter for EEG prediction. Such prediction will help early diagnosis and prognosis of epileptic seizure and emotional outbreak/discharge for psychiatric patients.

8.3 EEG Signal Prediction by Wavelet Coefficients

247

Fig. 8.18. Performance comparison of LMS and NLMS filters.

8.3 EEG Signal Prediction by Wavelet Coefficients In this section, we would like to examine the scope of emotion detection using EEG signals. A complex signal is usually represented by a collection of sinusoids and co-sinusoids of a large number of frequency components along with a DC signal. This is the basis of typical Fourier series. The amplitudes of the sinusoids and co-sinusoids at a given frequency can be evaluated by taking Fourier transform of the complex signal. Unfortunately, transient signals can hardly be represented by collection of sine/cosine waves. Wavelet transform is one possible solution to this problem. Unlike sinusoids/co-sinusoids, a wavelet is usually denoted by a small non-sinusoid type signal, called wavelet base, which when shifted in time and amplitude-scaled may help in representing signal transient. The fundamental signal used for wavelet representation is called wavelet base. The most common wavelet perhaps is the Harr wavelet, defined by ψ(t) = 1 if 0 ≤ t < ½ = -1 if ½ ≤ t < 1 = 0, otherwise

(8.10)

The representation of the Harr wavelet is given in Fig.8.19. The time-shifting and amplitude scaling of the wavelet base is usually referred to as translation and dilation respectively in the literature. Formally speaking, a wavelet function ψ(t) usually has a zero average when dilated with a scale parameter s and translated in time axis by u, and is represented by

248

8 Electroencephalographic Signal Processing ↑ 1 ψ(t)

1

½

t→ -1

Fig. 8.19. The Harr wavelet.

ψu, s = (1/√s) ψ ((t - u)/s).

(8.11)

The wavelet transform of f at the scale s and position u is evaluated by correlating f with a wavelet atom +∝

∫ f(t) (1/√s) ψ ((t – u)/s) dt, *

Wf(u, s) =

(8.12)

∝

where ψ* denotes complex conjugate of ψ. The wavelet transform represented by (8.12) can be written as +∝ Wf(u, s)=

∫ f(t) ψ -

∝

*

u, s(t)

dt = < f, ψu, s> .

(8.13)

Any finite energy signal f can be decomposed over the wavelet orthogonal basis {ψj, n}, where f(n) belonging to Z2 is given by +∝

f=∑ -∝

+∝

∑
-∝

ψj, n> ψj, n ,

(8.14)

and ψj, n= (1/ √2) ψ(( t – 2j n)/ 2j) for (j, n) ∈Z2.

(8.15)

For a discrete signal a0[n], we define the wavelet transform by +∝

f(t) =

∑ao[n] ∅(t – n). n= - ∝

(8.16)

8.3 EEG Signal Prediction by Wavelet Coefficients

249

Since {∅(t – n)}n∈Z is orthogonal, ao[n] = = f *∅(n).

(8.17)

Each ao[n] is a weighted average of f in the neighborhood of n. Usually, the discrete wavelet coefficient a0 are defined as the wavelet coefficient of f. An EEG signal having significant rise and fall within small durations can be considered as transients of a system. One simple way to represent an EEG signal is to evaluate its wavelet coefficients and then sort them in descending order to identify the first few large coefficients to approximately represent the signal. We have noted that stimulated EEG response for joy, anger, sadness, fear and relaxation have significant difference in temporal variations in the EEGs. Fig. 8.20-8.24 demonstrates samples of EEG signals for the above five possible stimulated situations. We note from this figure that Harr wavelet coefficients for the EEG signal corresponding to different stimulations for joy anger, sadness, fear and relaxation have significant differences. The difference can be measured in terms of the locations in the peaks of the wavelet coefficients or average and variance or any other statistical measures. We prefer the location of the first thirty peaks in the representation of an EEG signal. These peak positions in the wavelet coefficients may be defined as EEG descriptors. Given a large number of EEG samples (say 10,000) for stimulated emotion, we can cluster EEGs using their wavelet coefficients. We have verified using Self-Organizing Feature Map (SOFM) neural network that stimulated EEG patterns for similar emotions are automatically grouped into one cluster. This experimental finding ensures that EEG and emotion have a good correlation. Original signal 10 5 0 -5

0

10

20

30

40

50

60

Approx. coef. for haar 5

10

0

0

-5

0

20

40

60

-10

reconstructed signal 10

2

0

0

-10

0

50

70

80

90

100

Detail coef. for haar

100

-2

0

20

40

60

-15 x 10 error in reconstructed signal

0

50

100

Fig. 8.20. The original EEG, its wavelet coefficients and the reconstructed signal for anger.

250

8 Electroencephalographic Signal Processing Original s ignal 20

0

-20

0

10

20

30

40

50

60

70

A pprox . coef. for haar 10

20

0

0

-10

0

20

40

60

-20

rec ons truc ted s ignal 20

2

0

0

-20

0

50

80

90

100

Detail c oef. for haar

-2

100

0

20

x 10

40

60

-15

error in rec ons truc ted signal

0

50

100

Fig. 8.21. The original EEG, its wavelet coefficients and the reconstructed signal for fear.

Original signal 20 10 0 -10

0

10

20

30

40

50

60

Approx. coef. for haar 20

10

0

0

-20

0

20

40

60

-10

reconstructed signal 20

2

0

0

-20

0

50

70

80

90

100

Detail coef. for haar

100

-2

0

20

40

60

-15 x 10 error in reconstructed signal

0

50

100

Fig. 8.22. The original EEG, its wavelet coefficients and the reconstructed signal for joy.

8.3 EEG Signal Prediction by Wavelet Coefficients

251

Original signal 20 10 0 -10

0

10

20

30

40

50

60

70

Approx. coef. for haar 20

10

0

0

-20

0

20

40

-10

60

0

20

5

0

0

0

50

90

100

-5

100

20

40

60

-15 x 10 error in reconstructed signal

reconstructed signal

-20

80

Detail coef. for haar

0

50

100

Fig. 8.23. The original EEG, its wavelet coefficients and the reconstructed signal for relaxation.

O riginal s ignal 10

0

-10

0

10

20

30

40

50

60

A pprox . c oef. for haar 10

10

0

0

-10

0

20

40

60

-10

rec ons truc ted s ignal 10

5

0

0

-10

0

50

70

80

90

100

Detail c oef. for haar

100

-5

0 x 10

0

20 -15error

40

60

in rec ons truc ted s ignal

50

100

Fig. 8.24. The original EEG, its wavelet coefficients and the reconstructed signal for sadness.

252

8 Electroencephalographic Signal Processing

8.4 Bio-potential Signals in Emotion Prediction In a recent study of emotion detection by bio-potential signals, Takahasi [15] stressed the needs of EEG, skin conductance and pulse count as three important items. In this study, he considered various external stimulation to cause emotion arousal, and obtained EEG, pulse count and skin conductance during the period of EEG arousal. Takahasi then evaluated six parameters, hereafter called features to determine emotions uniquely from estimated features. Let X(t) be any one of the above three measurements of the sampled biopotential signal at time t. The six features in the present context are: T μx= (1/T) ∑ X(t) t=1

(8.18)

T σx =√ {(1/T) ∑ (X(t) - μx)2} t=1

(8.19)

T-1 δx = (1/T-1)∑ | X(t + 1) – X (t)| t=1

(8.20)

T-1 δx = (1/(T-1)) ∑ |X(t + 1) – X (t)| = δx/σx t=1

(8.21)

T-2 γx = (1/(T-2)) ∑ |X(t + 2) – X (t)| t=1

(8.22)

T-2 γx = (1/(T-2)) ∑ |X(t + 2) – X (t)| = γx /σx, t=1

(8.23)

where T denotes the total number of samples, and t denotes the sample number. Since there are three bio-potential signals and six features for each signal, we altogether have eighteen signals put together in the form of vector X as follows: μe σe δe δe γe γe μp σp δp X= δp γp γp μs σs δs δs γs γs

.

8.4 Bio-potential Signals in Emotion Prediction

253

α

f

X ∼

Y = +1 or -1 depending on X

∼

y=1

X2 ↑

y = -1

X1 →

Fig. 8.25. A schematic view of a linear classifier, classifying the pattern X into two regions, denoted by y= +1 or y= -1.

Takahasi [15] used Support Vector Machine (SVM) Learning algorithm for training several feature vector in order to ultimately recognize the emotion. We now briefly outline the basics of SVM technique. 8.4.1 Principles in SVM An SVM has successfully been used in the literature for both linear and non-linear classification. To understand the basic operation of SVM, let us consider Fig. 8.26, where X is the input vector and y is the desired scalar output that can take +1 or -1 values, indicating linear separation of the pattern vector X. One way to represent the function f (.) is as follows: f (X, W, b) = sign (WX+ b) where W= [w1 w2 …… wn] is the weight vector, X= [x1 x2……….xn]T is the input vector, b= [b1 b2…………bn] is the bias vector.

(8.24)

254

8 Electroencephalographic Signal Processing

↑ x2

Support vectors x1 → Fig. 8.26. Defining support vector for a linear SVM system.

Naturally, this is evident that the function f classifies the input vector X into two classes denoted by +1 or -1. The straight line that segregates the two pattern classes is usually called the linear classifier. Further, the set of data points that marginally support the boundaries of the linear classifier are called support vectors. Fig. 8.26 describes a support vector for a linear SVM. Let us now select two points X+ and X- as two support vectors as indicated in Fig. 8.26. Thus by definition and

WX+ + b = + 1 WX− + b = - 1

which jointly yields W(X+ − X− )= 2. Now, the separation between the two support vectors lying in the class +1 and class -1, called marginal width is given by M ={(WX+ + b) – (WX− + b)}/ |W| =W(X+ − X−)/ |W| =2/ |W|.

8.4 Bio-potential Signals in Emotion Prediction

255

The main objective in linear Support Vector Machine is to minimize the margin M =2/ |W|, which is same as (1/2) W+W. Consequently, the linear SVM can be mathematically described by Minimize ∅(W) = (1/2) W+ W subject to yi =( WXi + b) >= 1 for all i, where yi =1 when WXi +b > 1,

(8.25)

and yi= -1 when WXi + b<1. Here, our objective is to solve W and b that satisfies (8.25). In this book, we are not presenting the solution to the optimization problem, referred to above. This is available in standard text in neural network [6]. One important aspect of SVM is the kernel function selection. For linear SVM, the kernel K for two data points Xi and Xj is defined by K(Xi, Xj)= XiT Xj. Besides the above kernel function, researchers also employ a Gaussian Radial Basis Function as the kernel [6]. For two data points Xi and Xj, the Gaussian type kernel is defined by K(Xi, Xj)= exp(- ||Xi - Xj || /2σ2), Recognized emotion

Decision Logic

SVM

….

….. …. …… …..

… … … … …

Fig. 8.27. Emotion classification by Support Vector Machines.

256

8 Electroencephalographic Signal Processing

where σ is a parameter used for controlling the kernel measure with the distance norm between Xi and Xj. In his basic scheme, Takahasi employed five SVM networks, one each for joy, anger, sadness, fear and relaxation. The i-th SVM network is trained with all of the training instances of the i-th class with positive levels, and all other training instances with negative levels. The decision logic box driven by the five SVM networks ultimately recognize the emotion corresponding to the supplied feature vector X. Fig. 8.27 provides the SVM-based emotion recognition scheme, as outlined in [15].

8.5 Emotion Clustering by Neural Networks Several methods for emotion clustering have been tested to examine the scope of the most appropriate model for emotion detection from EEG sample. The most important factor detrimental to emotion clustering perhaps is the feature set. In our recent study, we consider the average peak for the delta waves (<4Hz), theta waves (4-7 Hz) lower alpha (8-10 Hz) and upper alpha band (11-13 Hz) and the beta 1 band (13-16 Hz) and beta 2 band (16-20 Hz) along with the location of the first few large peaks in the EEG spectrum. Fig. 8.28 below provides the fast Fourier Transform (FFT) to determine the average of the lower and upper alpha band and the beta band. We use first 20 large wavelet coefficients along with the previous information as the feature set. Any classical wavelet transform method may be employed to determine the first few large wavelet coefficients. However, for its widespread publicity, we have selected the Haar wavelet to determine the wavelet co-efficient of the given EEG signals for joy, anger, fear sadness and relaxation. To ensure that the first few large wavelet co-efficient are sufficient to describe an 1000 900 800 700 600 500 400 300 200 100 0

0

2

4

6

8

10

Fig. 8.28. FFT for the stimulated EEG pattern for anger.

12

8.5 Emotion Clustering by Neural Networks

257

1800 1600 1400 1200 1000 800 600 400 200 0

0

2

4

6

8

10

12

Fig. 8.29. FFT for the stimulated EEG for fear. 6000

5000

4000

3000

2000

1000

0

0

2

4

6

8

10

12

Fig. 8.30. FFT for the stimulated EEG pattern for joy.

EEG signal, we have taken inverse wavelet transformation on the amplitude spectrum for selected wavelet coefficients. Fig. 8.20 − 8.24 describe the visually stimulated EEG for appropriate emotion, its corresponding Haar wavelet coefficients and the inverse wavelet transform for the selected co-efficient. The RMS error plots indicate that the error at the sampling points of the original EEG and the inverse wavelet transform is significantly low. Naturally the selected wavelet co-efficient can be used for emotion prediction for EEG.

258

8 Electroencephalographic Signal Processing 1500

1000

500

0

0

2

4

6

8

10

12

Fig. 8.31. FFT for the stimulated EEG pattern for relaxation. 1000 900 800 700 600 500 400 300 200 100 0

0

2

4

6

8

10

12

Fig. 8.32. FFT for the stimulated EEG pattern for sadness.

Different neural networks have been used to cluster EEG for different emotions. In this section, we compare the performance of radial basis function (RBF) neural net, and the classical back-propagation algorithm from the points of view of emotion classification. Experiments with emotion recognition form EEG samples reveals that proper selection of the radial basis function gives high classification rate in comparison with back propagation algorithm. Learning steps required in

8.6 Conclusions

259

the back propagation algorithm is also exceedingly very high, when number of data samples (i.e., EEG patterns and emotion classes) are very high (>2000). For small number of data samples below 50, a 3-layered back propagation algorithm yields good classification accuracy of the order of 92%. In our study, we increase number of hidden layers form 1 to 3, and found a slight increase in accuracy form 92.2% to 94.6%. A Gaussian type radial basis function (RBF) with mean and variance values calculated from 2000 training instances yields a clustering accuracy of around 96.6%. Because of simplicity of the perceptron-like learning mechanism of the RBF network, and small training time even with large data set (around 2000), we prefer RBF-based classification to back propagation neural net-based classification of emotion. Although the problem considered in the present context was a classification problem, we could solve the same problem as a clustering problem by using different clustering algorithm, such as K-means, Fuzzy C-means, Hopfield neural network and many others. Selection of the right clustering/classification algorithm that yields best accuracy with minimum training time remains an open problem for our future study.

8.6 Conclusions The chapter proposed various methods for prediction of EEG signals, such as wavelet transform, FFT and statistical methods. The wavelet and FFT filter coefficient are then used with other statistical features as an input to an emotion recognition system. Several methods of pattern classification such as SVM, RBF and BPNN and LVQ algorithms have been used to classify emotion from the measured bio-potential signals. Experimental results demonstrate that emotion clustering by RBF neural network outperforms the classical BPNN. It is further noted that support vector machine (SVM) too yields good accuracies in emotion classification from bio-potential signals.

References 1. Arnold, S.H., Karrass, J., Edward, G.C., Tedra, A.W., Susan, M.W., Alexandra, F.K.: Emotional Reactivity and Regulation in Young Children who Stutter: Preliminary Behavioral and Brain Activity Data. Vanderbilt University, Nashville 2. Datta, S.: Emotion Detection and Control: A Computational Intelligence Approach, M. Tech. Thesis. Jadavpur University (2007) 3. Durka, P.: Matching Pursuit and Unification in EEG Analysis. Artech House, Norwood (2007) 4. Frackowiak, R.S.J., Ashburner, J.T., Penny, W.D., Zeki, S., Friston, K.J., Frith, C.D., Dolan, R.J., Price, C.J. (eds.): Human Brain Function, ch. 19. Elsevier Publisher, North Holand (2005) 5. Greg, W., Bishop, G.: An Introduction to the Kalman Filter TR 95 – 041, Department of Computer Science, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599-3175, Updated Monday (July 24, 2006)

260

8 Electroencephalographic Signal Processing

6. Haykin, S.: Neural Networks: A Comprehensive Foundation. Prentice-Hall, New Jersey (1999) 7. Michel, C.M., Lehmann, B., Hemggler, B., Brandeis, D.: Localization of sources of EEG delta, theta, alpha and beta frequency bands using the FFT dipole approximation. Electroencephalography and clinical Neurophysiology 82, 38–44 (1992) 8. Monson, H.H.: Statistical Digital Signal Processing and Modeling. Wiley, Chichester (1996) 9. Proakis, J.G., Manolakis, D.G.: Digital Signal Processing: Principles, Algorithms and Applications. Prentice-Hall, Englewood-Cliffs (1996) 10. Rippon, G.: Electroencephalography. In: Senior, C., Russell, T., Gazzaniga, M.S. (eds.) Methods in Mind. MIT Press, Cambridge (2006) 11. Senior, C., Russell, T., Gazzaniga, N.S. (eds.): Methods in Mind. MIT Press, Cambridge (2006) 12. Simon, H.: Adaptive Filter Theory. Prentice Hall, Englewood Cliffs (2002) 13. Solo, V., Kong, X.: Adaptive Signal Processing Algorithms: Stability and Performance. Prentice-Hall, Englewood-Cliffs (1986) 14. Takahashi, K.: Remarks on Emotion Recognition from Biopotential Signals. In: Second Int. Conf. on Autonomous Robots and Agents, Japan, pp. 186–191 (2004) 15. Widrow, B., Starns, S.D.: Adaptive Signal Processing. Prentice-Hall, EnglewoodCliffs (1985)

10 Open Research Problems

This chapter outlines some open research problems in Emotional Intelligence. It begins with reasoning with and without emotions, and justifies that reasoning with emotions may introduce contradictions, and thus requires a formalism of nonmonotonic reasoning to handle the problem. The chapter also addresses multi-modal emotion recognition from voice, facial expression and electroencephalographic data. One problem on emotion control by Takagi-Sugeno approach, and the other on prediction of manifestation of facial expression of a person from EEG data are briefly outlined in the chapter.

10.1 Introduction The last nine chapters of the book cover various methods of emotion recognition from facial expression, voice, and electroencephalographic (EEG) signals. In fact there exist a vast literature on emotion recognition from single or multimodal information. Previous chapters also provide principles and algorithms for controlling emotion of a subject by using multimedia presentations. However, the main thrust of emotional intelligence from the context of artificial intelligence (AI), perhaps, is to consider emotion as a vehicle to promote the basic cognitive ability of humans and/or machines. As a matter of fact, the AI research until date is concerned with building intelligent models and algorithms to enable machines perform complex cognitive tasks, including reasoning, learning, planning and perception. Naturally, a systematic synergy of emotion arousals with the intelligent cognitive tasks would yield better solutions to many real world problems. Let us for instance consider the impact of emotion in an intelligent reasoning problem. We presume that the database or facts, based on which inferences are derived, are available to two distinct reasoning systems, one of which is free from emotional expositions, while the other considers emotion as additional phenomenon to its normal counterpart. The emotions may appear in the second system as additional facts and/or knowledge, which individually or together may reinforce or contradict the inferences derived by the first reasoning system. In case the inferences are reinforced, it is not feasible to derive logically new information in A. Chakraborty and A. Konar: Emotional Intelligence, SCI 234, pp. 295–304. © Springer-Verlag Berlin Heidelberg 2009 springerlink.com

296

10 Open Research Problems

presence of emotion. But when contractions are discovered in presence of emotion, a non-monotonism is detected, and suitable logic and/or framework needs to be designed to handle the situation. The non-monotonism enters into an emotion-assisted reasoning system, when the reasoning is accomplished with binary logic like the logic of propositions/predicates. When multi-valued logic is used for reasoning in an emotionally induced reasoning system, the system may encounter a problem of stability, as the inferences are pushed back and forth by emotion-logic counteractions. In chapter 6, we introduced the issues of emotion-logic interactions, and proposed one scheme to match the response of emotional component with that of logical counterpart. However, the notion of automatic reasoning in presence of emotion, to the best of the authors’ knowledge, is unknown in the literature of Artificial Intelligence. Learning emotional pattern from voice or text also is an interesting issue. For example, the mewing pattern of an angry cat can easily be distinguished from that of a hungry cat. Humans too occasionally modulate their tones to express their emotions. Given a set of utterances of a given sentence padded with different emotions. We can easily represent the problem as a pattern classification problem with pitch, formants and peak power at formants as selective features, and emotion carried by the voice as different classes. We can then employ typical pattern classification algorithms, such as K-means, supervised neural algorithms and support vector machines to solve the classification problem. It may be mentioned here that we already undertook one such problem in chapter 9. A common natural problem that remains unsolved is to cluster emotion of people from their behavior and voice patterns. For example, people when gets angry talk at a higher speed than usual with a speedy movement of their body parts and/or rapid changes in their gesture. Clustering anger from other emotions using the behavioral pattern of speedy body part movement is an interesting issue for the next generation human-machine interface. The computers would be able to recognize anger of the subject from his/her behavioral patterns. Similarly, learning sadness, fear, anxiety etc. of people in work would be of immense importance to protect them from accident-prone environments. Multi-sensory coordination also is an important issue for building perception of intelligent agents. An agent senses its neighborhood using audio, video, tactile and other sensors. Arousal of certain emotions in an agent calls for fusion of multisensory data. Naturally, perception about an object can be developed in an intelligent agent by extracting different features about the object using different sensors, and then integrating them by multi-sensory data fusion algorithms. There is a vast literature on multi-sensory data fusion algorithms. The most well known among them include Dempester-Shafer theory [39], Kalman filter [17], Bayesian technique [5], and supervised neural algorithms [18]. This chapter briefly addresses the issues of reasoning in presence of emotions in section 10.2. In section 10.3, we provide a brief overview of uncertainty management in emotion–based reasoning. Section 10.4 presents some open-ended research problems for the researchers on emotional intelligence. A list of further readings for researchers is given in section 10.5.

10.2 Reasoning with Emotions

297

10.2 Reasoning with Emotions Reasoning deals with generating inferences from a given set of facts (data) and knowledge. There are different techniques of knowledge representation [18], including rule-based, structured approach, predicate logic, and many others. Naturally, the methodology of reasoning differs from one type of representation to the other. When emotion comes into reasoning, it adds new facts and/or rules, which may support or refute the normal reasoning process without emotion. Consider, for instance the following rules and facts used in a typical reasoning system. Suppose, a woman Mary loves her friend Joseph, and wants to marry him. But because of social reasons, her friends and relatives advise her not to marry her beloved. We now represent this problem by first order predicate logic, and demonstrate how non-monotonism enters the reasoning system. Example 1 discusses the problem in detail. Example 1: The first order logic representation of facts and rules are given below. Fact set 1: Woman (mary), Man (joseph), Loves (mary, joseph). Rule 1: Woman (X), Man (Y), Loves (X, Y) → Can-Marry (X, Y). Here, → denotes an implication sign, and “, (comma)” between each two predicates in the left hand side of → represents AND operator. Suppose, Y is cousin brother of X, and therefore, for social reasons, X should not marry Y. When we add the society’s reaction to the woman, we add new facts and rules as follows. Fact set 2: Cousin-Brother (joseph, mary). Rule 2: Cousin-Brother (Y, X) → ¬ Can-Marry (X, Y). It is apparent from fact set 1 and rule 1 that Can-Marry (mary, joseph) is true. But once the new fact: Cousin-Brother (joseph, mary) is added to the system, nonmonotonicity starts, as the fact set 2 and rule 2 together infer: ¬Can-Marry (mary, joseph), i.e. Mary cannot marry Joseph. Consequently, the inference ¬Can-Marry (mary, joseph) contradicts the previous inference: Can-Marry (mary, joseph). The reasoning system considered in Example 1, now has to select only one of two alternative inferences: ¬ Can-Marry (mary, joseph) and Can-Marry (mary, joseph). If Mary disregards rule 2, i.e. she gives higher preference to her emotional counterpart, she can marry Joseph, i.e., Can-Marry (mary, joseph) is true. But if Mary regards rule 2, and withdraws rule 1, then she cannot marry Joseph, i.e. ¬Can-Marry (mary, joseph) is true.

298

10 Open Research Problems

It is apparent from the above example that when emotion is taken into account as part of a reasoning process, the system may encounter non-monotonicity. There are many approaches to handle non-monotonic reasoning. The most popular among them include Nonmonotonic Logic I and II [22], and Truth-Maintenance [1] like reasoning. The main problem in NML I and II is to determine fixed points (also called stable points) of logic. In the last example, we could have one or two stable points, Can-Marry (mary, joseph) and ¬ Can-Marry (mary, joseph), but unfortunately, none of them are stable, as existence of one contradicts the presence of the other. Consequently, the above reasoning system has no stable points. One way to handle non-monotonism is to discover contradictions, and then dynamically partition the facts and inferences into two partitions, so that the information in each partition is consistent. In Truth-Maintenance System (TMS), Doyle attempted to achieve consistency among inferences by adopting a similar policy.

10.3 Uncertainty Management in Emotion-Based Reasoning In the last section, we noted non-monotonism in presence of emotion in the reasoning space. An alternative formulation of reasoning with emotion is to attach a degree of truth with the facts and premises, and then design a methodology to determine the degree of truth of the concluding inferences. In case contradictory pairs of information are present in the reasoning system, we may consider a weighted voting scheme of individual information, so as to select a single conclusion from a set of competitive inferences. There exist quite a large number of models on uncertainty management in the current literature on Artificial Intelligence. Among the well-known models, Bayesian model for distributed belief revision by Pearl [29] and fuzzy reasoning model realized with Petri like nets [19] need special mention. In Pearl’s belief revision model, the user has to provide the a priori probability of the axioms (starting propositions) and conditional probability of the rules. On the other hand, the fuzzy Petri net model introduced by Konar [19], we need to assign the fuzzy memberships (beliefs) of the starting propositions (axioms), and the certainty factors [40] of the rules to derive the belief of the concluding inferences. Both the methods have relative merits and demerits. For instance, Pearl’s belief revision scheme is computationally intensive, and gives good accuracy. When we do not have sufficient measurements about the degree of precision of facts and certainty about rules, we can go for fuzzy Petri net based reasoning, where the initial beliefs are assigned to propositions/predicates based on users’ guess. Konar et al. in [18] proposed an algorithm to handle contradictions in a fuzzy Petri net. The principle of this contradiction management is to allow all information, except the contradictory pairs, to participate in the voting scheme, so that the cumulative favor to one exceeds that of the other. After the voting is completed, we identify the information with the highest belief from the contradictory pair. The process is repeated for each set of contradictory pair of information.

10.4 Selected Open Problems

299

10.4 Selected Open Problems In this section, we present some open problems for interested researchers, including graduate students keen in emotion recognition and modeling. 10.4.1 Multi-modal Emotion Recognition In multi-modal emotion recognition problem, we need to determine the emotion of a subject from multiple physiological signals, such as facial expression, voice, EEG, heart pulse rate, body temperature, and many others. Design a multi-modal emotion recognition scheme that inputs three physiological signals, say EEG, facial expression and voice. Before to measure these signals, collect a set of audio-visual stimulus from commercial movies, so that they are capable of arousing independent emotions. Validate these by at least 10 people. You can use the scoring scheme introduced in Chapter 5 for validation. Excite emotion of 20 people, and record their facial expression, voice and at different time-slots. Identify the features of EEG, facial expression and voice, and check how you can classify the emotions from these features using SelfOrganizing Feature Map (SOFM) Neural Net, Radial Basis Function neural net, Support Vector Machine, and Learning Vector Quantization techniques. Compare the relative performance of the algorithms in recognition of emotion, and comment on the selection of the right technique. 10.4.2 Artificial Control of Emotion by Takagi-Sugeno Method In Chapter 6, we proposed a Mamdani-type model for controlling emotion using music and video. Replace the Mamdani’s model by Takagi-Sugeno model for emotion control. In your design, consider construction of rules to evaluate strength of audio-visual signals from the measured value of signed error and time rate of change of error. In computing time rate of change, you can use discretized values of error at successive time intervals. Note that you need to design two fuzzy controllers, one for positively signed error, and the other for negatively signed error. Further, you need to consider two fuzzifiers in order to encode (fuzzify) the computed signals by the controllers for submission to the input of the emotional dynamics. Develop a program to realize the Takagi-Sugeno controller, and test the performance of the proposed emotion controller with classical Mamdani-type controller. Also perform a stability analysis of the controller using the Lyapunov approach, introduced in Chapter 2, and hence derive the conditions for stability. Also determine the system matrices Ai,, for all i, satisfying the conditions for stability. 10.4.3 Determining Manifestation of Emotion on Facial Expression from EEG Signals The problem deals with determining (nonlinear) relationships between EEG signals and manifestation of emotion on the facial expression of the subjects. The

300

10 Open Research Problems

subjects are to be excited with audio-visual stimulus, and their facial expression and EEG are to be recorded during the phase of arousal of specific emotions, in particular anger, anxiety, and disgust. The features of EEG, such as power at specific frequency bands and wavelet coefficients, and features of facial expression, such as eye-opening, mouth-opening and eyebrow-constriction are first determined, and a dispersion in parameters for different emotions from relaxed emotion is determined. The dispersal in parameters of EEG, and facial expression may be considered as input and output of a multi-layered feed-forward neural network, and a supervised learning algorithm may be employed to train the network with these input-output instances. After the training is complete, we can use this network for synthesis of facial expression from the on-line recordings of EEG signals of the subjects. Suppose we synthesize the facial expression on an artificial model human face, imitating the real human. Then for a given audio-visual stimulus, if a discrepancy is noted between the facial expression of the real human and the model human face, we would be able to say that the manifestation of the facial expression is not normal. This could be a good approach to liar detection, provided one has the recordings of EEG and facial expression of the same person for different emotions.

10.5 Further Readings for Researchers Cohen et al. presented an interesting approach to facial expression recognition from multiple video sequences, considering both temporal and single image information [7]. Chen et al. in [6] considered audio-visual information for emotion recognition. Busso et al. analyzed facial expression and voice for multimodal emotion recognition [4]. Kim considered speech and physiological changes for bimodal emotion recognition [16]. Sebe et al. [38] proposed a new technique for emotion recognition using Cauchy Naive based classifier. Lee et al. [20] studied recognition of negative emotion from speech signal. Chen studied joint processing of audio-visual information for the recognition of emotional expression in human-computer interaction [6]. Garrett et al. employed functional magnetic resonance imaging technique to determine neuro-physiological correlates in the comprehension of emotional prosody [11]. Greene et al. used fMRI experiments to study emotional engagement in moral judgments [12]. Horlings et al. emphasized the importance of brain activity in emotion recognition [14]. Murugappan et al. designed an EEG-based scheme for recognition of emotion by exciting the subjects with audio-visual stimulus [23]. Johnstone et al. studied angry and happy vocal expression using fMRI experiments [15]. Lin et al. employed perceptron classifier to classify emotions of a subject during the period of music-listening [21]. Streit considered EEG correlates of facial affect recognition and categorization of blurred faces in schizophrenic patients [42]. Streit et al. considered EEG brain mapping in both schizophrenic patients and healthy subjects during facial emotion recognition [43]. Zhang et al. analyzed positive and negative emotions, stimulated by natural scenes using EEG, fMRI images and GIST (extract of relevant visual low level features used for classification) [48]. Sabatinelli et al. studied emotional perception by determining

References

301

correlation of functional MRI and event related potential [35]. Baumgartner et al. [3] studied arousal of emotion evoked by classical music. Neth et al. [24] presented a novel approach to describe emotionless faces by a norm-based representation. Zeelenberg et al. [47] studied the impact of emotion on perception. Atkinson et al. [1] studied emotion-perception from dynamic and static body expression in point-light and full-light displays. Phelps et al. [30] argues that emotion facilitates perception, and potentiates the perceptual benefits of attentions. Ayesh [2] studied perception and emotion based reasoning by a connectionist approach. Phillips [31] studied the effect of neurobiology of emotionperception to determine their implications in psychiatry. Paul et al. examined the contribution of emotion and reasoning in the prediction of hallucinatory pre-disposition in non-clinical individuals [28]. Steinbeis et al. [41] compared the processing of music and language meaning using EEG and fMRI to provide evidence of similar and distinct neural representations. Haag et al. [13] present a novel design of biosensors for emotion recognition. Saiwaki et al. [36] gave an analysis of EEG based on information flow with SD method. Picard et al. [33] gave an interesting view of machine emotional intelligence with emphasis on affective physical states. Picard also gave an overview of computational models, capable of recognizing and responding to users’ emotions [32]. Park et al. [27] studied affective interaction between emotional expression and environment. Cowie et al. [8] presented a new scheme of emotion recognition for the next generation human-computer interface. Xu et al. [45] presented an interesting approach to modeling of artificial emotion recognition. Custodial et al. [9] proposed a novel scheme for emotion based control system. Nomura et al. [25] discussed the problems of artificial emotions in mental therapy. Richard [34] outlines an overview of cognitive neuroscience of emotion. Salovey et al. [37] gave an overview of emotional intelligence with special emphasis to imagination, cognition, and personality. Ortoney et al. in [26] discussed in details the cognitive structure of emotions. Vuilleumier [44] discussed the brain mechanism during emotion face perception. Xu proposed a dynamic circuit to model emotions [46].

References 1. Atkinson, A.P., Dittrich, W.H., Gemmell, A.J., Young, A.W.: Emotion perception from dynamic and static body expressions in point-light and full-light displays. Perception 33(6), 717–746 (2004) 2. Ayesh, A.: Perception and Emotion Based Reasoning: A Connectionist Approach. Informatica 27, 119–126 (2003) 3. Baumgartner, T., Esslen, M., Jancke, L.: From emotion perception to emotion experience: Emotions evoked by pictures and classical music. International Journal of Psychophysiology, Science Direct 60, 34–43 (2006) 4. Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C.M., Kazemzadeh, A., Lee, S., Neumann, U., Narayanan, S.: Analysis of Emotion Recognition using Facial Expressions, Speech and Multimodal Information. In: Proc. of the 6th international conference on Multimodal interfaces, State College, PA, USA, pp. 205–211 (2004)

302

10 Open Research Problems

5. Castillo, E., Gutierrez, J.M., Hadi, A.S.: Expert Systems and Probabilistic Network Models. Springer, Heidelberg (1997) 6. Chen, L.S., Tao, H., Huang, T.S., Miyasato, T., Nakatsu, R.: Emotion recognition from audiovisual information. In: Proc. IEEE Workshop on Multimedia Signal Processing, pp. 83–88 (1998) 7. Cohen, I., Sebe, N., Chen, L., Garg, A., Huang, T.S.: Facial expression recognition from video sequences: Temporal and static modeling. Computer Vision and Image Understanding, 160–187 (2003) 8. Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Fellenz, W., Taylor, J.G.: Emotion Recognition in human computer interactions. IEEE Signal Processing Magazine 18(1), 33–80 (2001) 9. Custodial, V.R., Pinto-Ferreira, C.: Artificial Emotions and Emotion-Based Control System. In: 7-th IEEE Int. Conf. Emerging Technologies and Factory Automation, Spain, Bercelona, vol. 3, pp. 1415–1420 (1999) 10. Doyle, J.: A Truth Maintenance System. Artificial Intelligence 12, 231–272 (1979) 11. Garrett, K.D., Koenig, P., Moore, P., Devita, C., Listerud, J., Liberman, M., Grossman, M.: Neurophysiological correlates in the comprehension of emotional prosody: An fMRI study. Journal of the International Neuropsychology Society 8(2), 139 (2002) 12. Greene, J.D., Sommerville, R.B., Nystrom, L.E., Darley, J.M., Cohen, J.D.: An fMRI Investigation of Emotional Engagement in Moral Judgment. Science 293(5537), 2105– 2108 (2001) 13. Haag, A., Goronzy, S., Schaich, P., Williams, J.: Emotion recognition Using Biosensors: First Steps towards an Automatic System. In: André, E., Dybkjær, L., Minker, W., Heisterkamp, P. (eds.) ADS 2004. LNCS, vol. 3068, pp. 36–48. Springer, Heidelberg (2004) 14. Horlings, R., Datcu, D., Othkrantz, I.J.M.: Emotion Recognition using Brain Activity. In: Proc. of Int. Conf. on Computer Systems and Technologies - CompSysTech, pp. 1– 6 (2008) 15. Johnstone, T., Reekum, C.M.V., Oakes, T.R., Davidson, R.J.: The voice of emotion: an FMRI study of neural responses to angry and happy vocal expressions. Soc. Cogn. Affect Neurosci. 1(3), 242–249 (2006) 16. Kim, J.: Bimodal Emotion Recognition using Speech and Physiological Changes. In: Grimm, M., Kroschel, K. (eds.) Robust Speech Recognition and Understanding, Vienna, Austria, p. 460 (June 2007) 17. Kosko, B.: Neural Networks and Fuzzy Systems: A Dynamical Systems Approach to Machine Intelligence. Prentice-Hall, Inc., Englewood Cliffs (1992) 18. Konar, A.: Artificial Intelligence and Soft Computing: Behavioral Cognitive Modeling of the Human Brain. CRC Press, Boca Raton (2000) 19. Konar, A., Jain, L.C.: Cognitive Engineering: A Distributed Approach to Machine Intelligence. Springer, Heidelberg (2006) 20. Lee, C.M., Narayanan, S., Pieraccini, R.: Recognition of Negative Emotions from the Speech Signal. In: Proc. Automatic Speech Recognition and Understanding, Trento, Italy (December 2001) 21. Lin, Y.P., Wang, C.-H., Wu, T.-L., Jeng, S.-K., Chen, J.-H.: Multilayer Perceptron for EEG Signal Classification during Listening to Emotional Music. In: IEEE TENCON Int. Conf., pp. 1–3 (2007) 22. McDermott, D., Doyle, J.: Nonmonotonic Logic I. Artificial Intelligence 13(1-2) (1980)

References

303

23. Murugappan, M., Juhari, M.R.B.M., Nagarajan, R., Yaacob, S.: An investigation on visual and audiovisual stimulus based emotion recognition using EEG. Int. J. of Medical Engineering and Informatics 1(3), 342–356 (2009) 24. Neth, D., Martinez, A.M.: Emotion perception in emotionless face images suggests norm-based representation. Journal of Vision 9(5), 1–11 (2009) 25. Nomura, T.: Problems of Artificial Emotion in Mental Therapy. In: Proc. of the 2003 IEEE Int. Symp. on Computational Intelligence in Robotics and Automation, Kobe, July 16-20, 2003, pp. 567–570 (2003) 26. Ortony, A., Clore, G.L., Collins, A.: The Cognitive Structure of Emotions. Cambridge University Press, Cambridge (1988) 27. Park, C., Ryu, J.W., Kim, J., Kang, S., Sohn, J., Cho, Y.-J.: Emotion Expression and Environment through Affective Interaction. In: Proc. of the 17th World Congress, The International Federation of Automatic Control, Seoul, Korea (July 6-11, 2008) 28. Paul, A., Daniel, F., Philip, M.G., Philippa, G., Elizabeth, K., David, F., Paul, B., Catherine, G., Graham, D., Katarzyna, R.: The Prediction of Hallucinatory Predisposition in non-clinical individuals: Examining the Contribution of Emotion and Reasoning. British Journal of Clinical Psychology 44(1), 127–132 (2005) 29. Pearl, J.: Distributed Revision of Composite Belief. Artificial Intelligence 33, 173–213 (1987) 30. Phelps, E.A., Ling, S., Carrasco, M.: Emotion Facilitates Perception and Potentiates the Perceptual Benefits of Attention. Psychological Science 17(4), 292–299 (2006) 31. Philips, M.L.: Understanding the neurobiology of emotion perception: implications for psychiatry. British Journal of Psychiatry 182, 190–192 (2003) 32. Picard, R.W.: Toward computers that recognize and respond to user emotion. IBM Systems Journal 39(3-4), 705–719 (2000) 33. Picard, R.W., Vyzas, E., Healey, J.: Towards machine emotional intelligence: analysis of affective physical state. IEEE Trans. Pattern Analysis and Machine Intelligence 23(10), 1175–1191 (2001) 34. Richard, L.D. (ed.): Cognitive Neuroscience of Emotion. Oxford University Press, New York (2002) 35. Sabatinelli, D., Lang, P.J., Keil, A., Bradley, M.M.: Emotional Perception: Correlation of Functional MRI and Event-Related Potentials. Cerebral Cortex 17(5), 1085–1091 (2007) 36. Saiwaki, N., Kimura, N., Nishida, S.: An analysis of EEG based on information flow with SD method. In: IEEE Int. Conf. on Systems, Man, Cybernetics, vol. 4, pp. 4115– 4119 (October 1998) 37. Salovey, P., Mayer, J.D.: Emotional Intelligence. Imagination, Cognition and Personality 9(3), 185–211 (1990) 38. Sebe, N., Lew, M.S.: Emotion recognition using a cauchy naive bayes classifier. In: Proc. Int. conf. Pattern Recog., pp. 17–20 (2002) 39. Shafer, G.: A Mathematical Theory of Evidence. Princeton University Press, Princeton (1976) 40. Shortliffe, E.H.: Computer Based Medical Consultations: MYCIN. American Elsevier, New York (1976) 41. Steinbeis, N., Koelsch, S.: Comparing the Processing of Music and Language Meaning Using EEG and fMRI Provides Evidence for Similar and Distinct Neural Representations. PLoS ONE 3(5), 2226 (2008) 42. Streit, M.: EEG- correlates of facial affect recognition and categorization of blurred faces. Schizophrenic Research 49(1), 145–155

304

10 Open Research Problems

43. Streit, M., Brinkmever, J., Wolwer, W., Gaebel, W.: EEG brain mapping in Schizophrenic patients and healthy subjects during facial emotion recognition. Schizophrenic Research 61(1), 121 (2003) 44. Vuilleumier, P., Pourtois, G.: Distributed and interactive brain mechanisms during emotion face perception: Evidence from functional neuroimaging. Neuropsychologia 45, 174–194 (2007) 45. Xu, L., Zhuge, Z., Yang, J.: Artificial Emotion and Its Recognition, Modeling and Applications: An Overview. In: Proc. of the 5th World Congress on Intelligent Control and Automation, Hangzhou. P.R. China (June 15-19, 2004) 46. Xu, K.-F., Tu, X.-Y., Wang, C.: The Artificial Emotional Modeling Based on Dynamic Circuit. In: Proc. of the 2007 Int. Conf. on Wavelet Analysis and Pattern Recognition, Beijing, China (November 2-4, 2007) 47. Zeelenberg, R., Wagenmakers, E.-J., Rotteveel, M.: Impact of Emotion on Perception. Psychological Science 17(4), 287–291 (2006) 48. Zhang, Q., Lee, M.: Analysis of positive and negative emotions in natural scene using brain activity and GIST. Neurocomputing 72, 1302–1306 (2009)

Appendix In this Appendix, we provide some of the experimental instances used for recognition of emotion from facial expression and voice of the subjects as introduced in Chapter 9. The subjects were asked to utter a given sentence (What are you doing here?), by adding emotion in the utterance and also expressing emotions in their facial expressions. The voice waveforms and facial expressions of the subjects during utterance of the different fragments of the sentence are given in Fig. A.1 - A.12.

306

Appendix

Neutral: [….what……..are………you…………..doing………..….here..]

Happy: […what……are……..you……………doing………..…here...]

Sad: [...what……are…..…you……………doing………….here..]

Anger: [..what………are……...you…………...doing…………..here...]

Disgust: […what….. ...are …… you………….doing……….…...here..]

Fear: [....what………are…...….you…………...doing……………here..]

Surprise: [...what……...are……...you………….doing……………..here..] Fig. A.1. Facial expression of Kasturi while uttering the given sentence.

Appendix

Normal

307

Happy

0.6

1 0.8

0.4 0.6 0.4

0

Amplitude

Amplitude

0.2

-0.2

0.2 0 -0.2 -0.4

-0.4

-0.6 -0.6 -0.8 -0.8

0

0.2

0.4

0.6

0.8 1 1.2 Time in seconds

1.4

1.6

1.8

-1

2

0

0.5

1 1.5 Time in seconds

Anger

2

2.5

Disgust

0.6

0.8

0.4

0.6 0.4

0.2

Amplitude

Amplitude

0.2

0

-0.2

0 -0.2

-0.4 -0.4

-0.6

-0.8

-0.6

0

0.5

1 1.5 Time in seconds

2

-0.8

2.5

0

0.5

1 1.5 Time in seconds

Fear

2

2.5

Sad 0.3

0.8 0.6

0.2

0.4

0.1

Amplitude

0 -0.2

0

-0.1

-0.4

-0.2 -0.6

-0.3 -0.8 -1

-0.4 0

0.5

1 1.5 Time in seconds

2

2.5

0

0.5

1 1.5 Time in seconds

Surprise 1 0.8 0.6 0.4

Amplitude

Amplitude

0.2

0.2 0 -0.2 -0.4 -0.6 -0.8 -1

0

0.5

1 1.5 Time in seconds

2

2.5

Fig. A.2. Speech waveform of Kasturi.

2

2.5

308

Appendix

Normal:

[--what--------are-----------you---------------doing--------------here----]

Happy: [--what---------are----------you--------------doing----------------here—]

Anger: [--what-------are---------you--------------doing---------------here--]

Dsigust: [--what--------are---------you---------------doing--------------here----]

Fear: [--what--------are-----------you---------------doing--------------here----]

Sad: [-----what-------are---------you-------------doing--------------here----]

Surprise: [----what-------are----------you-------------doing---------------here----] Fig. A.3. Facial expression of Rupsa while uttering the given sentence.

Appendix

Normal

309

Happy

0.6 0.6

0.4

0.4

0.2

0

0

Amplitude

Amplitude

0.2

-0.2

-0.2

-0.4

-0.4

-0.6

-0.8

-0.6

0

0.5

1 1.5 Time in seconds

2

-0.8

2.5

0

0.5

1 1.5 Time in seconds

Anger

2

2.5

Disgust

1 0.4

0.8

0.3

0.6

0.2

0.4

0.1 Amplitude

Amplitude

0.2 0 -0.2

0 -0.1

-0.4

-0.2

-0.6

-0.3

-0.8 -1

-0.4

0

0.5

1 1.5 Time in seconds

2

-0.5

2.5

0

0.2

0.4

0.6

Fear

0.8 1 1.2 Time in seconds

1.4

1.6

1.8

2

Sad 0.2

0.6

0.15 0.4

0.1

Amplitude

0.05

0

-0.2

0 -0.05 -0.1

-0.4

-0.15 -0.6

-0.8

-0.2 -0.25 0

0.5

1 1.5 Time in seconds

2

2.5

0

0.2

0.4

0.6

0.8 1 1.2 Time in seconds

Surprise 0.6 0.4 0.2 0 Amplitude

Amplitude

0.2

-0.2 -0.4 -0.6 -0.8 -1

0

0.5

1 1.5 Time in seconds

2

2.5

Fig. A.4. Speech waveform of Rupsa.

1.4

1.6

1.8

2

310

Appendix

Neutral: [….what……...are…..…..you……………doing…………..….here..]

Happy: […what…….are………you……………doing…………here…..]

Sad:

[..what……...are…...…you………….…doing……..……here..] Fear:

[..what……are……….you……….…doing………..here..] Anger:

[..what……….are……...you………….…doing………....here…]

Disgust:

[..what………are……....you……..……doing………….…here..]

Surprise:

[..what………are……...you…………………doing……..…here..] Fig. A.5. Facial expression of Archana while uttering the given sentence.

Appendix

Normal:

Happy: 0.4

0.2 0.15

r!

0.3 0.2

0.1

Amplitude 0.1

Amplitude 0.05

0

0

-0.1

-0.05

-0.2

-0.1

-0.3

-0.15

-0.4

-0.2 -0.25 0

311

0.5

1 1.5 Time in seconds

2

-0.5 0

2.5

0.5

1 1.5 Time in seconds

Anger:

2

2.5

2

2.5

Disgust:

0.8

0.2

0.6

0.15 0.1

0.4

Amplitude 0.05

Amplitude 0.2

0 0 -0.05 -0.2 -0.1 -0.4 -0.15 -0.6 -0.8 0

-0.2 0.5

1 1.5 Time in seconds

2

-0.25 0

2.5

0.5

1 1.5 Time in seconds

Fear:

Sad 0.08

0.25

0.06

0.2 0.15

0.04

0.1 Amplitude 0.02 Amplitude 0.05 0

0 -0.05

-0.02

-0.1

-0.04

-0.15 -0.06 -0.2 -0.25 0

0.5

1 1.5 Time in seconds

2

-0.08 0

2.5

0.5

1 1.5 Time in seconds

Surprise:

Amplitud e

0. 5 0. 4 0. 3 0. 2 0. 1 0 0.1 0.2 0.3 0.4 0.5 0

0. 5

1 Time in seconds

1. 5

2

2. 5

Fig. A.6. Speech waveform of Archana.

2

2.5

312

Appendix

Normal: [---what-------are--------you--------------doing------------here---]

Happy:

[---what-------are--------you--------------doing------------here---]

Anger:

[---what-------are--------you--------------doing------------here---]

Disgust:

[---what-------are--------you--------------doing------------here---]

Fear:

[---what-------are--------you--------------doing------------here---]

Sad:

[---what-------are---------you----------------doing------------here---]

Surprise:

[--what-------are---------you----------------doing------------here---] Fig. A.7. Facial expression of Mohua while uttering the given sentence.

Appendix

Normal

313

Happy

0.1

0.5

0.08

0.4

0.06

0.3 0.04

0.2 Amplitude

A m plitud e

0.02 0 -0.02

0.1 0 -0.1

-0.04 -0.06

-0.2

-0.08

-0.3

-0.1

0

0.5

1

-0.4

1.5

0

0.5

1 1.5 Time in seconds

Time in seconds

Anger

2

2.5

2

2.5

Disgust

0.8

0.4

0.6

0.3

0.4

0.2

0.2 Amplitude

-0.2

0 -0.1

-0.4 -0.2

-0.6

-0.3

-0.8 -1

0

0.2

0.4

0.6

0.8 1 1.2 Time in seconds

1.4

1.6

1.8

-0.4

2

0

0.5

1 1.5 Time in seconds

Fear

Sad

0.15 0.4

0.1

0.3

0.05

0.2 0.1 Amplitude

0

-0.05

0 -0.1

-0.1 -0.2

-0.15

-0.2

-0.3

0

0.2

0.4

0.6 0.8 1 Time in seconds

1.2

1.4

1.6

-0.4

0

0.5

1 1.5 Time in seconds

Surprise 0.4

0.3

0.2

Amplitude

Amplitude

Amplitude

0.1

0

0.1

0

-0.1

-0.2

-0.3

0

0.2

0.4

0.6

0.8 1 1.2 Time in seconds

1.4

1.6

1.8

2

Fig. A.8. Speech waveform of Mohua.

2

2.5

314

Appendix

Normal: [---what------are--------you-------------doing-----------here---]

Happy: [---what------are--------you-------------doing-----------here---]

Anger:

[---what------are--------you-------------doing-----------here---]

Disgust: [---what------are--------you-------------doing-----------here---]

Fear: [-----what------are--------you-------------doing-----------here---]

Sad: [---what------are--------you-------------doing-----------here---]

Surprise: [---what------are--------you-------------doing-----------here---] Fig. A.9. Facial expression of Prativa while uttering the given sentence.

Appendix

Normal

315

Happy 0.15

0.15 0.1

0.1

0.05

0.05

Amplitude

Amplitude 0

0

-0.05

-0.05

-0.1

-0.1

-0.15

-0.15

-0.2 0

0.5

1 1.5 Time in seconds

2

-0.2 0

2.5

0.2

0.4

0.6

Anger

0.8 1 1.2 Time in seconds

1.4

1.6

1.8

2

Disgust

0.8

0.15

0.6 0.1 0.4 0.05

0.2

Amplitude

Amplitude 0

0

-0.2 -0.05 -0.4 -0.1

-0.6 -0.8 -1 0

-0.15 0.2

0.4

0.6

0.8 1 1.2 Time in seconds

1.4

1.6

1.8

2

-0.2 0

0.5

1 1.5 Time in seconds

Fear

2

2.5

Sad

0.15

0.03

0.1

0.02

0.05

0.01 Amplitude

Amplitude 0

0

-0.05

-0.01

-0.1

-0.02

-0.15

-0.03

-0.2 0

0.5

1 1.5 Time in seconds

2

-0.04 0

2.5

0.2

0.4

0.6

Surprise 0.2 0.15 0.1 0.05 Amplitude 0 -0.05 -0.1 -0.15 -0.2 0

0.5

1 1.5 Time in seconds

2

2.5

Fig. A.10. Speech waveform of Prativa.

0.8 1 1.2 Time in seconds

1.4

1.6

1.8

2

316

Appendix

Normal:

[---what-------are---------you----------------doing-------------here---]

Happy:

[---what-------are---------you--------------doing----------------here--]

Anger: [---what-------are--------you----------------doing-------------here--]

Disgust:

[---what-------are---------you----------------doing-------------here--]

Fear:

[---what-------are--------you----------------doing-------------here----]

Sad:

[---what--------are----------you----------------doing-------------here----]

Surprise: [---what--------are----------you----------------doing-------------here----] Fig. A.11. Facial expression of Souvik while uttering the given sentence.

Appendix

Normal

317

Happy

0.25

0.5

0.2

0.4

0.15

0.3

0.1 Amplitude 0.05

0.2 Amplitude 0.1

0

0

-0.05

-0.1

-0.1

-0.2

-0.15

-0.3

-0.2

-0.4

-0.25 0

0.2

0.4

0.6

0.8 1 1.2 Time in seconds

1.4

1.6

1.8

-0.5 0

2

0.5

Anger 0.4

0.6

0.3

0.4

0.2

0.2

0.1 Amplitude

Amplitude 0

0

-0.2

-0.1

-0.4

-0.2

-0.6

-0.3

0.5

2

2.5

2

2.5

Disgust

0.8

-0.8 0

1 1.5 Time in seconds

1 1.5 Time in seconds

2

2.5

-0.4 0

0.5

1 1.5 Time in seconds

Fear

Sad 0.08

0.4

0.06

0.3

0.04 0.2 0.02 Amplitude 0

0.1 Amplitude 0

-0.02 -0.1 -0.04 -0.2

-0.06

-0.3 -0.4 0

-0.08 0.5

1 1.5 Time in seconds

2

2.5

-0.1 0

0.5

1 1.5 Time in seconds

Surprise 0.4 0.3 0.2 0.1 Amplitude 0 -0.1 -0.2 -0.3 -0.4 -0.5 0

0.5

1 1.5 Time in seconds

2

2.5

Fig. A.12. Speech waveform of Souvik.

2

2.5

Author’s Biography

Aruna Chakraborty is currently an Assistant Professor in the department of Computer Science and Engineering, St. Thomas’ College of Engineering and Technology, Calcutta. She is also a Visiting Faculty of Jadavpur University, where she offers graduate level courses in Intelligent Automation and Robotics, and Cognitive Science. Aruna received her M.A. degree in Cognitive Science, and Ph. D. degree on Emotional Intelligence and Human-Computer Interactions from Jadavpur University in 2000 and 2005 respectively. Her current research interest includes artificial intelligence, emotion modeling, and their applications in the next generation human-machine interactive systems and psychotherapy. She has published extensively on emotional intelligence in top ranking journals and international conferences. Dr. Chakraborty serves as an editor to the International Journal of Artificial Intelligence and Soft Computing, Inderscience, UK. She is a nature-lover, and loves music and painting. She is co-editing a book on Emotion Modeling, Recognition and Control with her teacher Amit Konar. Amit Konar is currently a Professor in the dept. of Electronics and Tele-communication Engineering (ETCE), Jadavpur University. He is founding coordinator of the M.Tech. program on Intelligent Automation and Robotics, offered by ETCE department, Jadavpur University. He received his B. E degree from Bengal Engineering and Science University (B. E. College), Shibpur in 1983 and his M. E. Tel E, M. Phil. and Ph.D. (Engineering) degrees from Jadavpur University in 1985, 1988, and 1994 respectively. Dr. Konar’s research areas include the study of computational intellig ence algorithms and their applications

320

Author’s Biography

to the entire domain of electrical engineering and computer science. Specifically he worked on sets and logic, neuro-computing, evolutionary algorithms, DempsterShafer theory and Kalman filtering, and applied the principles of computational intelligence in image understanding, VLSI design, mobile robotics and pattern recognition. Dr. Konar has supervised 10 Ph.D. theses. He has around 200 publications in international journal and conferences. He is an author of 6 books, including two popular texts: Artificial Intelligence and Soft Computing, from CRC Press in 2000 and Computational Intelligence: Principles, Techniques and Applications from Springer in 2005. Dr. Konar serves as the Editor-in-Chief of the International Journal of Artificial Intelligence and Soft Computing from Inderscience, U.K., and he is also the member of the editorial board of 5 other international journals. He was a recipient of AICTE-accredited 1997-2000 Career Award for Young Teachers for his significant contribution in teaching and research. He was a Visiting Professor for the Summer Courses in University of Missouri, St. Louis, USA in 2006. Dr. Konar is a Principal Investigator or Co-Principal Investigator of four external projects funded by University Grants Commission (the UGC is one of the main federal funding agencies in India) and two projects funded by the All India Council of Technical Education, Government of India. The research areas of these projects include decision support system for criminal investigation, navigational planning for mobile robots, AI and image processing, neural net based dynamic channel allocation, human mood detection from facial expressions and DNAstring matching algorithms. Dr. Konar served as a member of Program Committee of several International Conferences and workshops, such as Intl. Conf. on Hybrid Intelligent Systems (HIS 2003), held in Adelaide, Australia and Int. Workshop on Distributed Computing (IWDC 2002), held in Calcutta.

Index

Activation asymmetry 106, 107, 118 Adaptive filters 236 Affective computing 133, 134 Affective Disorders 104 Affective Neuro Scientific Model 11 Amplitude and Sign Fuzzifier 197 Amygdala 10, 11, 12, 13, 14, 18 Analysis of Dynamical System 35 Anterior cingulated cortex (ACC) 93, 94, 96, 98, 99, 101, 283 Anterior Cingulated 18, 97 Anterior insula region 18 Antisocial motives 262 Anti-social motives 261, 262, 267 Artificial Control of Emotion by Takagi-Sugeno Method 299 Artificial creature design 276, 277 Artificial emotional creature 261 Artificial Intelligence 3 Associative learning 17, 18 Asymptotic stability 219 Asymptotically stable 48 Attentional control 108 Attentional Deficiency with Hyper Activity Disorder 109 Audio-visual movies 133, 149, 152, 163, 165, 209 Audio-Visual Stimulus 220 Autonomic Reflexive Responses 95

Back-propagation neural learning 88 Bayesian technique 296 Biological Basis of Emotion 10 Biopotential signals 260 Bio-potential Signals 235, 236, 252, 259 Borderline Personality Disorder (BPD) 110 Boundary descriptor algorithms 84 Boundary Detection Algorithms 64, 69 BPNN 259 Brain Imaging 93, 100, 104, 110, 283 Break points 264 Brevity 1, 4, 5 Carver–Scheier model of self-regulation 14 Chain Codes 84 chaotic behavior 209, 217, 232 Chaotic Dynamics 209, 217 Chaotic Emotional Dynamics 55, 210, 228 chaotic emotional states 24 Chaotic Fluctuation in Emotional State 217 Chaotic Neuro Dynamics 52 chaotic system 35 Chaotic temporal variation 217 Clinical Study of Depression 106 Cognitive Component 6 Cohen-Grossberg theorem 43 competitive emotional dynamics 210 Components of Emotion 5, 9 Concurrent emotional state 210

322

Index

Connected region 86 Continuous Dynamics 42 Control of emotion 116 Control of endocrinic system 13 Control policy 210 Control Vector 30 Controlling Emotion by Artificial Means 21 Co-operative emotional dynamics 210 Damping level 210, 226, 227, 228 Defuzzification 135, 152, 162, 163 Dempester-Shafer theory 296 Detection of Anti-social Motives from Emotional Expressions 289 Differential Evolution Algorithm in Emotional System Identification 289 Digital filter coefficients 24 Digital filter 235, 236, 237, 242 Digital image processing 63 Digital movie-making 261, 261 Discontinuity 69 Discrete dynamical system 229 Discrete fuzzy systems 45 Dynamic Bayesian network 273 Dynamic behavior 35, 56 Dynamical system 175, 176, 177, 183, 186, 190, 199, 201, 202 EEG Analysis for Premenstrual Dysphoric Disorder 106 EEG Conditioning 104 EEG Dynamics Modeling by Evolutionary Algorithms 283 EEG electrodes 235 EEG Signal Prediction 247 EEG spectrum 54 EEG system 235 EEG- waveform 235 Electroencephalographic signals 235 Electroencephalography (EEG) 235, 282, 283 Embodiment of Artificial Characters 264 Emotion Clustering by Neural Networks 256

Emotion Control by Mamdani’s Model 157 Emotion controller 227 Emotion Dysregulation 94, 107, 110 Emotion Modeling 185 Emotion Recognition from Voice Samples 273 Emotion Regulation in Adulthood 109 Emotion transition graph 154, 187 Emotion transition 154, 156 Emotional dynamics 34, 35, 54, 55, 102, 209, 210, 212, 219, 220, 226, 227 Emotional dysfunction 15 Emotional expressions 267 Emotional Intelligence in Psychotherapy 262 Emotional intelligence 1, 2, 3, 10, 17, 266, 272 Emotional Learning 17 Emotional sensitivity 9 Emotion-Based Reasoning 298 Emotion-Logic Counter-Actions 195 Epileptic seizure 235, 236, 246 Equilibrium 215 Error related negativity (ERN) 99 Error square norm 285 Euler number 86 Evaluative Component 6, 26 Evaluative feedback 9, 26 Evoked response of conditioned and unconditioned stimuli 18 Experimental accuracy 133, 135, 165 Face tracking 267 Facial Attributes 139, 141 FCM clustering algorithm 136 Fear and Threat Perception 95 Features extracted from voice data 282 Feeling Component 6, 7, 9 Filtering 135, 167 Finite impulse response (FIR) 236, 237 Fitness function 286 fMRI instrument 266, 270, 271, 283 Focus of attention 4, 5, 9

Index Formants 274, 278 Forward path gain 114, 115, 116 Fourier and discrete cosine transforms 63 Fourier descriptor 134 Fourier Descriptors 84, 86 Fourier Transform (FFT) 256 Functional Magnetic Resonance Imaging (fMRI) 11, 283 Functional Neuro-imaging 96, 98 Functionalist theory 107 Fuzzification of Facial Attributes 141 Fuzzy C-means Clustering Algorithm 77, 81 Fuzzy C-means clustering 135, 136 Fuzzy estimator 22 Fuzzy Petri net 298 Fuzzy relational approach 133, 163, 166 Fuzzy relational matrix 142, 144 Fuzzy Relational Model for Emotion Detection 141 Fuzzy Temporal Representation of Phenomena Involving Emotional States 190 Gain formula 113, 114, 115 Gain product of two non- touching loops 113, 114 Gaussian noise 282 Gaussian type kernel 255 Genetic Algorithm in Emotional System Identification 285 Harr wavelet 247, 249 Hebbian type unsupervised learning 283 Henon map 58 Hidden Markov Model (HMM) 278 High pitch in music 21 Histogram based thresholding 135, 139 Histogram 67, 68, 86 Hopfield dynamics 42 Human emotion recognition 133, 134, 135

323

Human machine interactive systems 22, 24 Human-machine interactive systems 261, 263, 272, 283 Hybrid pitch detectors 273 Image Matching 88, 89 Image segmentation Algorithms 69, 77, 81 Implication relation 45, 46 Impulsive fear stimuli 120 Individual loop gains 113, 114, 115 Inferior Temporal Cortex 95 Instability 1, 4, 5 Intelligent quotient (IQ) 2 Intensity 1, 4, 5, 9 Intentional Component 5, 7 Interpretation 36, 37, 38 intrinsic growth rate 210, 211, 217, 226, 227, 228, 229 Kalman Filter 242, 244 Kernel function 255 Kernel Regression (KR) 277 K-nearest neighbors (KNN) 277 Lagrange’s multiplier 78 Laplace of Gaussian 73 Laplace transform 206 Laplacian operator 72, 73 Lateral prefrontal cortex (LPFC) 102 Laws of statics 176 Learning vector quantization 274 Learning 3, 9, 13, 17, 18 Lee-Oscillator 209, 230, 231 Limit cycles 188, 209, 213, 228 Limit cyclic system 35 Linearized system 202 LMS Filter 237, 245 Localization of Facial Components 135 Logic of fuzzy sets 21, 22 Lotka-Volterra model 19 Low-pass-filter 235

324

Index

LVQ algorithms 259 Lyapunov analysis 35 Lyapunov approach 299 Lyapunov energy function 183, 184, 193, 199, 200, 201, 209, 210, 219, 220 Lyapunov exponent 36, 52, 55, 57, 229, 231 Mamdani Type Fuzzy Systems 45 Manifestation of Emotion on Facial Expression from EEG Signals 299 Marginal width 254 Markov models 134 Masson’s gain formula 115 Mathematical Modeling of Emotional Dynamics 18 Matrimonial counseling 270 Maximum Likelihood Bayes Classifier (MLB) 277 Max-min composition operator 46 Medial frontal cortex (MFC) 283 Medial Frontal Cortex 99 Median Filtering 66, 67 Mel Frequencies Cepstral Coefficients (MFCC) 278 Membership distribution 45, 46 Mesocopic neural models 52 Meta Cognitive Analysis 104 Minimally over-damped 227 Mixed emotional systems 209 Mobile Robotics 265 Monochrome image 63, 87 Motivational Component 6, 26 Movie Making 269 Multi-Agent Co-operation 265 Multi-agent planning 262 Multimodal Emotion Recognition 277 Multi-modal Emotion Recognition 299 Multimodal information 277, 295 Multiple Emotions 209, 220 Negative feedback 215 Network virtual park 264

Neural Circuitry Underlying Emotional Self-Regulation 103 Neural networks 87, 88 Neurofeedback 105 Neuro-modulatory mechanisms 18 Neuronal dynamics 209 NLMS Algorithm 238, 246 Non-dominated sorting genetic algorithm 277 Nonlinear coupled oscillators 283 Nonlinear emotional system 210 Nonmonotonic Logic 298 Non-monotonism 296, 297, 298 Non-psycho-metric methods 267 Non-Temporal Logic 176 Nuclear waste explorations 265 Object Recognition 87 Oil exploration 265 Operant-conditioning 105 Orbitofrontal Cortex (OFC) 96, 97, 105 Orbitofrontal cortex 23 Ordinary Differential Equation (ODE) 283 Overall gain 111 Over-damped 227 Pansepp’s theory 12 Parameter Identification of Emotional Dynamics 283 Parameter Selection of the Emotional Dynamics 220 Partiality 4, 5 Particle swarm optimization (PSO) 285 Particle Swarm Optimization in Emotional System Identification 286 Perception 3, 4 Performance index 29 Personality Building of Artificial Creatures 275 Personality Matching of People for Matrimonial Counseling 270 Phase trajectory 229, 230 PI type 227

Index Pitch detection 273 Planning 3 Positron Emission Tomography (PET) 11 Posterior Thalamus 96 Post-synaptic potentials 235 Principal Component Analysis 87 Problem of modularity 17 Proposed Emotion Control Scheme 163 Propositional logic 176, 177 Propositional temporal system 183, 199 Prosodic information 272 Pseudo-defuzzification 144 Psycho-pathological Issues 107 Psychotherapeutic treatment 235, 236 Radial Basis Function 88, 255 Rationality of Emotion 7 Reasoning with Emotions 297 Reasoning 3 Reconstructed signal 249, 250, 251 Recurrent neural dynamics 209 Recurrent neural network 54 Regional Descriptors 86 Regulation and Control of Emotion 8 Relational matrix 45 Relative Stability of the Learning Dynamics 190 Relative stability 229 Repeatability 165 RLS Filter for EEG Prediction 240 Role model 269 Rules of contradiction and Identity 8 Runge-Kutta numerical method 58 Sampling time 267 Scene Interpretation 90, 91 Segmentation 134, 135, 137, 139 Selective Lesion of Insula Cortex 97 Self Regulation Models of Emotion 13, 15 Self-Organizing Feature Map (SOFM) neural network 249 Self-Organizing Feature Map 87

325

Self-Regulation of Emotion 93, 99, 100, 102, 126, 128 Semantic nets 269 Sensitivity 9, 12, 28 Sensory cortex 13, 14 Sensory thalamus 14 Sigmoid type non-linearity 27 Sign constraint 227 Signal flow graph 112, 115 Similarity 64, 69 Sobel Mask 70, 71, 72 Superordinate executive system 17 Speech Articulatory Features 273 Speech waveform for different emotions 279 Stability Analysis of the Emotional Dynamics 186, 194 Stability analysis 35, 40, 43, 175, 176, 194 Stability of Propositional Temporal System 183 Stabilization of emotion by fuzzy temporal reasoning 196 Stabilization of Emotional Dynamics by a Learning Controller 195 Stabilization of emotional dynamics 194, 195 Stabilization Scheme for the Mixed Emotional Dynamics 226 Stable interpretations 177, 198 Stable Points in Propositional Temporal Logic 180 Stable points 38, 298 State transition diagram of emotional dynamics 211 Strength Pareto Evolutionary Algorithm (SPEA) 275 Striate Cortex 95 Stroop interference task 100 Structured approach 297 Sub-cortical control of emotion 12 Superior Colliculus 95 Supervised neural algorithms 296

326

Index

Support Vector Machine (SVM) 253, 259 Support vector machines (SVM) 24 SVM networks 256 Synthesizing Emotions in Voice 271 System Identification Approach to EEG Dynamics 283 System identification 209 Takagi-Sugeno Type Fuzzy Systems 36, 46 Template Matching 89, 90 Template-matching 134, 137 Temporal logic 35, 36, 39, 40 Thresholding 67, 68, 69 Time domain pitch detectors 273 Topographic mapping 236 Train of pulses 121, 122 Training instances 87, 88 Transient response 226 Under-damped dynamics 210 Underdamped 210 Unimodal classifier 278

Unimodal systems 278 Unit Delay 30 Unsupervised Clustering 87, 88 User Assistance Systems 272 Validation 133, 135, 152 van der Pol dynamics 230 Vector quantization 274 Video based recognition 278 Video Photography 261, 262, 269 Video photography/movie-making 261 Virtual characters 263 Visually inspired external stimuli 284 Visual tracking 267 Voluntary Regulation of Sexual Arousals 102 Voluntary Self-Regulation of Emotion 101 Wavelet coefficients 24, 236, 247 Weight Adaptation in Emotion Dynamics by Hebbian Learning 189 Wells & Matthew’s dysfunction model 16

Advances in Agent-Based Complex Automated Negotiations (Studies in Computational Intelligence)

Advances in agent based complex automated negotiations

Computational Intelligence in Engineering (Studies in Computational Intelligence)

Computational Intelligence in Engineering (Studies in Computational Intelligence, 313)

New Trends in Agent-Based Complex Automated Negotiations

Recent Advances in Harmony Search Algorithm (Studies in Computational Intelligence)

Advances in Data Management (Studies in Computational Intelligence,Volume 223)

Computational Intelligence: A Compendium (Studies in Computational Intelligence, Volume 115)

Challenges for Computational Intelligence (Studies in Computational Intelligence, Volume 262)

Challenges for Computational Intelligence (Studies in Computational Intelligence)

Advances in Computational Intelligence: Theory And Applications

Advances in Computational Intelligence: Theory And Applications

Advances in computational intelligence: theory & applications

Advances of Computational Intelligence in Industrial Systems

Innovations in Swarm Intelligence (Studies in Computational Intelligence, Volume 248)

Distributed Computing (Studies in Computational Intelligence)

Intelligent Information Access (Studies in Computational Intelligence)

Computational Intelligence in Control

Applied Pattern Recognition (Studies in Computational Intelligence)

Stigmergic Optimization (Studies in Computational Intelligence)

Applied Pattern Recognition (Studies in Computational Intelligence)

Stigmergic Optimization (Studies in Computational Intelligence)

Computational Intelligence in Archaeology

Computational Intelligence in Control

Metaheuristic Clustering (Studies in Computational Intelligence)

Computational Optimization, Methods and Algorithms (Studies in Computational Intelligence, 356)

Advances in Artificial Intelligence

Advances in computational mathematics

Advanced Computational Intelligence Paradigms in Healthcare 5: Intelligent Decision Support Systems (Studies in Computational Intelligence)

Oppositional Concepts in Computational Intelligence (Studies in Computational Intelligence, Volume 155)

Computational Intelligence in Automotive Applications (Studies in Computational Intelligence, Volume 132)

Advances in Agent-Based Complex Automated Negotiations (Studies in Computational Intelligence)

Advances in agent based complex automated negotiations

Computational Intelligence in Engineering (Studies in Computational Intelligence)

Computational Intelligence in Engineering (Studies in Computational Intelligence, 313)

New Trends in Agent-Based Complex Automated Negotiations

Recent Advances in Harmony Search Algorithm (Studies in Computational Intelligence)

Advances in Data Management (Studies in Computational Intelligence,Volume 223)

Computational Intelligence: A Compendium (Studies in Computational Intelligence, Volume 115)

Challenges for Computational Intelligence (Studies in Computational Intelligence, Volume 262)

Challenges for Computational Intelligence (Studies in Computational Intelligence)

Advances in Computational Intelligence: Theory And Applications

Advances in Computational Intelligence: Theory And Applications

Advances in computational intelligence: theory & applications

Advances of Computational Intelligence in Industrial Systems

Innovations in Swarm Intelligence (Studies in Computational Intelligence, Volume 248)

Distributed Computing (Studies in Computational Intelligence)

Intelligent Information Access (Studies in Computational Intelligence)

Computational Intelligence in Control

Applied Pattern Recognition (Studies in Computational Intelligence)

Stigmergic Optimization (Studies in Computational Intelligence)

Applied Pattern Recognition (Studies in Computational Intelligence)

Stigmergic Optimization (Studies in Computational Intelligence)

Computational Intelligence in Archaeology

Computational Intelligence in Control

Metaheuristic Clustering (Studies in Computational Intelligence)

Computational Optimization, Methods and Algorithms (Studies in Computational Intelligence, 356)

Advances in Artificial Intelligence

Advances in computational mathematics

Advanced Computational Intelligence Paradigms in Healthcare 5: Intelligent Decision Support Systems (Studies in Computational Intelligence)

Oppositional Concepts in Computational Intelligence (Studies in Computational Intelligence, Volume 155)

Computational Intelligence in Automotive Applications (Studies in Computational Intelligence, Volume 132)

Recommend Documents