Modular Neural Networks and Type-2 Fuzzy Systems for Pattern Recognition

Patricia Melin Modular Neural Networks and Type-2 Fuzzy Systems for Pattern Recognition Studies in Computational Inte...

Author: Patricia Melin

24 downloads 470 Views 7MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!

Report copyright / DMCA form

DOWNLOAD PDF

Patricia Melin Modular Neural Networks and Type-2 Fuzzy Systems for Pattern Recognition

Studies in Computational Intelligence, Volume 389 Editor-in-Chief Prof. Janusz Kacprzyk Systems Research Institute Polish Academy of Sciences ul. Newelska 6 01-447 Warsaw Poland E-mail: [email protected] Further volumes of this series can be found on our homepage: springer.com Vol. 366. Mario Köppen, Gerald Schaefer, and Ajith Abraham (Eds.) Intelligent Computational Optimization in Engineering, 2011 ISBN 978-3-642-21704-3 Vol. 367. Gabriel Luque and Enrique Alba Parallel Genetic Algorithms, 2011 ISBN 978-3-642-22083-8 Vol. 368. Roger Lee (Ed.) Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing 2011, 2011 ISBN 978-3-642-22287-0 Vol. 369. Dominik Ry_zko, Piotr Gawrysiak, Henryk Rybinski, and Marzena Kryszkiewicz (Eds.) Emerging Intelligent Technologies in Industry, 2011 ISBN 978-3-642-22731-8 Vol. 370. Alexander Mehler, Kai-Uwe Kühnberger, Henning Lobin, Harald Lüngen, Angelika Storrer, and Andreas Witt (Eds.) Modeling, Learning, and Processing of Text Technological Data Structures, 2011 ISBN 978-3-642-22612-0 Vol. 371. Leonid Perlovsky, Ross Deming, and Roman Ilin (Eds.) Emotional Cognitive Neural Algorithms with Engineering Applications, 2011 ISBN 978-3-642-22829-2 Vol. 372. Ant´onio E. Ruano and Annam´aria R. V´arkonyi-K´oczy (Eds.) New Advances in Intelligent Signal Processing, 2011 ISBN 978-3-642-11738-1 Vol. 373. Oleg Okun, Giorgio Valentini, and Matteo Re (Eds.) Ensembles in Machine Learning Applications, 2011 ISBN 978-3-642-22909-1 Vol. 374. Dimitri Plemenos and Georgios Miaoulis (Eds.) Intelligent Computer Graphics 2011, 2011 ISBN 978-3-642-22906-0

Vol. 378. János Fodor, Ryszard Klempous, and Carmen Paz Suárez Araujo (Eds.) Recent Advances in Intelligent Engineering Systems, 2011 ISBN 978-3-642-23228-2 Vol. 379. Ferrante Neri, Carlos Cotta, and Pablo Moscato (Eds.) Handbook of Memetic Algorithms, 2011 ISBN 978-3-642-23246-6 Vol. 380. Anthony Brabazon, Michael O’Neill, and Dietmar Maringer (Eds.) Natural Computing in Computational Finance, 2011 ISBN 978-3-642-23335-7 Vol. 381. Radoslaw Katarzyniak, Tzu-Fu Chiu, Chao-Fu Hong, and Ngoc Thanh Nguyen (Eds.) Semantic Methods for Knowledge Management and Communication, 2011 ISBN 978-3-642-23417-0 Vol. 382. F.M.T. Brazier, Kees Nieuwenhuis, Gregor Pavlin, Martijn Warnier, and Costin Badica (Eds.) Intelligent Distributed Computing V, 2011 ISBN 978-3-642-24012-6 Vol. 383. Takayuki Ito, Minjie Zhang, Valentin Robu, Shaheen Fatima, and Tokuro Matsuo (Eds.) New Trends in Agent-Based Complex Automated Negotiations, 2012 ISBN 978-3-642-24695-1 Vol. 384. Daphna Weinshall, J¨orn Anem¨uller, and Luc van Gool (Eds.) Detection and Identification of Rare Audiovisual Cues, 2012 ISBN 978-3-642-24033-1 Vol. 385. Alex Graves Supervised Sequence Labelling with Recurrent Neural Networks, 2012 ISBN 978-3-642-24796-5 Vol. 386. Marek R. Ogiela and Lakhmi C. Jain (Eds.) Computational Intelligence Paradigms in Advanced Pattern Classification, 2012 ISBN 978-3-642-24048-5

Vol. 375. Marenglen Biba and Fatos Xhafa (Eds.) Learning Structure and Schemas from Documents, 2011 ISBN 978-3-642-22912-1

Vol. 387. David Alejandro Pelta, Natalio Krasnogor, Dan Dumitrescu, Camelia Chira, and Rodica Lung (Eds.) Nature Inspired Cooperative Strategies for Optimization (NICSO 2011), 2011 ISBN 978-3-642-24093-5

Vol. 376. Toyohide Watanabe and Lakhmi C. Jain (Eds.) Innovations in Intelligent Machines – 2, 2011 ISBN 978-3-642-23189-6

Vol. 388. Tiansi Dong Recognizing Variable Environments, 2012 ISBN 978-3-642-24057-7

Vol. 377. Roger Lee (Ed.) Software Engineering Research, Management and Applications 2011, 2011 ISBN 978-3-642-23201-5

Vol. 389. Patricia Melin Modular Neural Networks and Type-2 Fuzzy Systems for Pattern Recognition, 2012 ISBN 978-3-642-24138-3

Patricia Melin

Modular Neural Networks and Type-2 Fuzzy Systems for Pattern Recognition

123

Author

Prof. Patricia Melin Tijuana Institute of Technology Division of Graduate Studies, Tijuana, Mexico Mailing Address P.O. Box 4207 Chula Vista CA 91909, USA E-mail: [email protected]

ISBN 978-3-642-24138-3

e-ISBN 978-3-642-24139-0

DOI 10.1007/978-3-642-24139-0 Studies in Computational Intelligence

ISSN 1860-949X

Library of Congress Control Number: 2011939329 c 2012 Springer-Verlag Berlin Heidelberg This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typeset & Cover Design: Scientific Publishing Services Pvt. Ltd., Chennai, India. Printed on acid-free paper 987654321 springer.com

Preface

We describe in this book, hybrid intelligent systems using type-2 fuzzy logic and modular neural networks for pattern recognition applications. Hybrid intelligent systems combine several intelligent computing paradigms, including fuzzy logic, neural networks, and bio-inspired optimization algorithms, which can be used to produce powerful pattern recognition systems. The book is organized in three main parts, which contain a group of chapters around a similar subject. The first part consists of chapters with the main theme of theory and design algorithms, which are basically chapters that propose new models and concepts, which can be the basis for achieving intelligent pattern recognition. The second part contains chapters with the main theme of using type-2 fuzzy models and modular neural networks with the aim of designing intelligent systems for complex pattern recognition problems. The third part contains chapters with the theme of evolutionary optimization of type-2 fuzzy systems and modular neural networks in intelligent pattern recognition, which includes the application of genetic algorithms for obtaining optimal type-2 fuzzy integration systems and ideal neural network architectures. In the part of theory and algorithms there are 4 chapters that describe different contributions that propose new models and concepts, which can be the considered as the basis for achieving intelligent pattern recognition. The first chapter offers an introduction to the areas of type-2 fuzzy logic and modular neural networks for pattern recognition applications. The second chapter describes the basic concepts of type-2 fuzzy logic applied to the problem of edge detection in digital images. The third chapter describes a general methodology for applying type-2 fuzzy logic on improving the recognition ability of modular neural networks. The fourth chapter describes the use of type-2 fuzzy systems for improving the performance of response integration in modular neural networks. In the part of type-2 fuzzy systems and modular neural networks for pattern recognition applications there are 4 chapters that describe different contributions on achieving human recognition using hybrid systems based on type-2 fuzzy logic. The first chapter describes the use of a modular neural network with fuzzy response integration for human recognition based on the iris biometric measure. The second chapter deals with the design of modular neural network architecture with fuzzy integration using the ear biometric measure as information for recognition. The third chapter describes the design of a type-2 fuzzy modular neural system for

VI

Preface

signature recognition. The fourth chapter describes the application of type-2 fuzzy logic and modular neural networks for achieving efficient face recognition. In the part of evolutionary optimization of type-2 fuzzy systems and modular neural networks there are 5 chapters that describe different contributions of new algorithms for optimization and their application to designing optimal type-2 fuzzy logic response integrators and ideal modular neural network architectures. The first chapter describes the optimization of type-2 fuzzy response integrators and modular neural networks using genetic algorithms for achieving human recognition based on the face, fingerprint and voice biometric measures. The second chapter deals with an approach for the optimization of number of rules and membership functions for type-2 fuzzy response integrators of modular networks with hierarchical genetic algorithms for human recognition based on face, fingerprint and voice. The third chapter describes the application a general method for designing type-2 fuzzy systems based on genetic algorithms that can be used as response integrators in modular neural networks. The fourth chapter describes the optimal design of the modular architecture and fuzzy response integrator based on genetic algorithms for achieving human recognition based on the iris biometric measure. The fifth chapter describes the application of a hierarchical genetic algorithm for designing optimal fuzzy response integrators and the modular neural network architecture for human recognition based on the ear biometric information. In conclusion, the book comprises chapters on diverse aspects of type-2 fuzzy logic and modular neural networks with evolutionary models for achieving intelligent pattern recognition for different applications, including human recognition. The combination of evolutionary optimization methods with type-2 fuzzy logic and modular neural networks can be considered as a hybrid approach for obtaining efficient and accurate solutions to complex pattern recognition problems.

July 21, 2011

Patricia Melin Tijuana Institute of Technology Mexico

Contents

Part I: Basic Concepts and Theory 1 Introduction to Type-2 Fuzzy Logic in Neural Pattern Recognition Systems .............................................................................................................. 3 2 Type-1 and Type-2 Fuzzy Inference Systems for Images Edge Detection .......................................................................................................... 7 2.1 Introduction ............................................................................................... 7 2.2 Sobel Operators......................................................................................... 8 2.3 Edge Detection by Gradient Magnitude .................................................... 9 2.4 Edge Detection Using Type-1 Fuzzy Logic ............................................ 10 2.5 Edge Detection Using Type-2 Fuzzy Logic ............................................ 14 2.6 Comparison of Results ............................................................................ 16 2.7 Summary ................................................................................................. 19 3 Type-2 Fuzzy Logic for Improving Training Data and Response Integration in Modular Neural Networks .................................................... 21 3.1 Method for Image Recognition ................................................................ 21 3.2 Type-2 Fuzzy Inference System as Edge Detector................................... 22 3.3 The Modular Structure ............................................................................. 25 3.4 Simulation Results.................................................................................... 25 3.5 Summary .................................................................................................. 28 4 Method for Response Integration in Modular Neural Networks Using Type-2 Fuzzy Logic ........................................................................................ 29 4.1 Introduction .............................................................................................. 29 4.2 Proposed Approach for Recognition ........................................................ 31 4.3 Modular Neural Networks........................................................................ 31 4.4 Integration of Results for Person Recognition Using Fuzzy Logic .......... 32 4.5 Modular Neural Networks with Type-2 Fuzzy Logic as a Method for Response Integration ................................................................................ 33

VIII

Contents

4.6 Simulation Results ................................................................................... 35 4.7 Summary .................................................................................................. 39

Part II: Modular Neural Networks in Pattern Recognition 5 Modular Neural Networks for Person Recognition Using the Contour Segmentation of the Human Iris ................................................................... 43 5.1 Introduction.............................................................................................. 43 5.2 Background and Basic Concepts.............................................................. 44 5.3 Proposed Method and Problem Description............................................. 46 5.4 Modular Neural Network Architecture .................................................... 49 5.5 Simulation Results ................................................................................... 50 5.6 Summary .................................................................................................. 58 6 Modular Neural Networks for Human Recognition from Ear Images Compressed Using Wavelets ......................................................................... 61 6.1 Introduction.............................................................................................. 61 6.2 Background .............................................................................................. 64 6.3 Ear Recognition Process........................................................................... 66 6.4 Summary .................................................................................................. 75 7 Signature Recognition with a Hybrid Approach Combining Modular Neural Networks and Fuzzy Logic for Response Integration .................... 77 7.1 Introduction.............................................................................................. 77 7.2 Problem Statement and Outline of Our Proposal ..................................... 78 7.3 Background Theory.................................................................................. 79 7.4 Experiments ............................................................................................. 85 7.5 Summary .................................................................................................. 92 8 Interval Type-2 Fuzzy Logic for Module Relevance Estimation in Sugeno Response Integration of Modular Neural Networks...................... 93 8.1 Introduction.............................................................................................. 93 8.2 Modular Neural Networks........................................................................ 94 8.3 Sugeno Integral for Modules Fusion ........................................................ 96 8.4 The Sugeno Integral ................................................................................. 97 8.5 Fuzzy Logic for Density Estimation ........................................................ 97 8.6 FIS-1 to Estimate Fuzzy Densities ........................................................... 99 8.7 FIS-2 for Estimate Fuzzy Densities........................................................ 100 8.8 Sugeno Integral for Information Fusion ................................................. 101 8.9 Simulation Results ................................................................................. 103 8.10 Summary .............................................................................................. 105

IX

Contents

Part III: Optimization of Modular Neural Networks for Pattern Recognition 9 Optimization of Fuzzy Response Integrators in Modular Neural Networks with Hierarchical Genetic Algorithms ..................................... 109 9.1 Introduction ............................................................................................ 109 9.2 Neural Networks .................................................................................... 110 9.3 Fuzzy Logic............................................................................................ 111 9.4 Genetic Algorithms ................................................................................ 112 9.5 Modular Neural Network with Fuzzy Integration for Face, Fingerprint and Voice Recognition ........................................................................... 113 9.6 Modular Neural Network Results........................................................... 115 9.7 Fuzzy Integration Results....................................................................... 118 9.8 Hierarchical Genetic Algorithm ............................................................. 119 9.9 Comparison with Other Works............................................................... 124 9.10 Summary .............................................................................................. 126 10 Modular Neural Network with Fuzzy Response Integration and Its Optimization Using Genetic Algorithms for Human Recognition Based on Iris, Ear and Voice Biometrics ................................................. 127 10.1 Introduction ........................................................................................ 127 10.2 Background ........................................................................................ 128 10.3 Basic Concepts ................................................................................... 129 10.4 Proposed Method and Results ............................................................ 130 10.5 Summary ............................................................................................ 144 11 A Comparative Study of Type-2 Fuzzy System Optimization Based on Parameter Uncertainty of Membership Functions ............................ 145 11.1 Introduction ........................................................................................ 145 11.2 Preliminaries ...................................................................................... 146 11.3 Optimization Method Description...................................................... 147 11.4 Fuzzy Systems Optimization Based on the Level of Uncertainty ...... 148 11.5 Simulation Results.............................................................................. 150 11.6 Summary ............................................................................................ 161 12 Neural Network Optimization for the Recognition of Persons Using the Iris Biometric Measure........................................................................ 163 12.1 Introduction ........................................................................................ 163 12.2 Methods of Integration ....................................................................... 164 12.3 Iris Image Pre-processing ................................................................... 165 12.4 Statement of the Problem and Proposed Method ............................... 170 12.5 Summary ............................................................................................ 184

X

Contents

13 Optimization of Neural Networks for the Accurate Identification of Persons by Images of the Human Ear as Biometric Measure................ 185 13.1 Introduction ........................................................................................ 185 13.2 Modular Artificial Neural Networks .................................................. 186 13.3 Integration Methods ........................................................................... 187 13.4 Pre-processing the Biometric Image of the Ear.................................. 188 13.5 Problem Statement and Proposed Method ......................................... 190 13.6 Summary ............................................................................................ 204 References .......................................................................................................... 205 Index................................................................................................................... 213

Part I

Chapter 1

Introduction to Type-2 Fuzzy Logic in Neural Pattern Recognition Systems

We describe in this book, new methods for building intelligent systems for pattern recognition using type-2 fuzzy logic and soft computing techniques. Soft Computing (SC) consists of several computing paradigms, including type-1 fuzzy logic, neural networks, and genetic algorithms, which can be used to create powerful hybrid intelligent systems [37, 57]. In this book, we are extending the use of fuzzy logic to a higher order, which is called type-2 fuzzy logic [12]. Combining type-2 fuzzy logic with traditional SC techniques, we can build powerful hybrid intelligent systems that can use the advantages that each technique offers in solving pattern recognition problems [59]. Fuzzy logic is an area of soft computing that enables a computer system to reason with uncertainty [103]. A fuzzy inference system consists of a set of if-then rules defined over fuzzy sets. Fuzzy sets generalize the concept of a traditional set by allowing the membership degree to be any value between 0 and 1. This corresponds, in the real world, to many situations where it is difficult to decide in an unambiguous manner if something belongs or not to a specific class [12]. Fuzzy expert systems, for example, have been applied with some success to problems of decision, control, diagnosis and classification, just because they can manage the complex expert reasoning involved in these areas of application [20, 44, 58, 60]. The main disadvantage of fuzzy systems is that they can't adapt to changing situations [24, 59] For this reason, it is a good idea to combine fuzzy logic with neural networks or genetic algorithms, because either one of these last two methodologies could give adaptability to the fuzzy system [19, 59, 60]. On the other hand, the knowledge that is used to build these fuzzy rules is uncertain. Such uncertainty leads to rules whose antecedents or consequents are uncertain, which translates into uncertain antecedent or consequent membership functions [41]. Type-1 fuzzy systems, like the ones mentioned above, whose membership functions are type-1 fuzzy sets, are unable to directly handle such uncertainties [46, 47]. We also describe in this book, type-2 fuzzy systems, in which the antecedent or consequent membership functions are type-2 fuzzy sets [12, 62]. Such sets are fuzzy sets whose membership grades themselves are type-1 fuzzy sets; they are very useful in circumstances where it is difficult to determine an exact membership function for a fuzzy set [12]. Type-2 fuzzy systems have been applied with relative success

P. Melin: Modular Neural Networks and Type-2 Fuzzy Systems, SCI 389, pp. 3–6. springerlink.com © Springer-Verlag Berlin Heidelberg 2012

4

Chapter 1 Introduction to Type-2 Fuzzy Logic in Neural Pattern Recognition Systems

in many real-world applications [27], like in control [1, 11, 14, 15, 20, 29, 95, 99, 100], time series prediction [6, 42, 76], classification and decision [2, 3, 4, 21, 22, 46, 47, 73, 81, 82, 85, 86, 89, 98, 101, 104, 106], diagnosis [23, 67, 72, 75], and pattern recognition [30, 31, 32, 33, 36, 50, 54, 61, 64, 66, 80, 91, 105]. There have also been recent theoretical advances in type-2 fuzzy logic [1, 13, 26, 68, 69, 71, 77, 78, 88, 92, 94, 96, 102, 108, 109] that have allowed to improve performance in the implementations, and the use of optimization techniques to automate the design of type-2 fuzzy systems [5, 10, 22, 45, 70, 79, 90, 97]. Uncertainty is an inherent part of intelligent systems used in real-world applications. The use of new methods for handling incomplete information is of fundamental importance. Type-1 fuzzy sets used in conventional fuzzy systems cannot fully handle the uncertainties present in intelligent systems. Type-2 fuzzy sets that are used in type-2 fuzzy systems can handle such uncertainties in a better way because they provide us with more parameters [83]. This book deals with the design of intelligent systems using interval type-2 fuzzy logic for minimizing the effects of uncertainty produced by the instrumentation elements, environmental noise, etc. [84]. Neural networks are computational models with learning (or adaptive) characteristics that model the human brain [57]. Generally speaking, biological natural neural networks consist of neurons and connections between them, and this is modeled by a graph with nodes and arcs to form the computational neural network. This graph along with a computational algorithm to specify the learning capabilities of the system is what makes the neural network a powerful methodology to simulate intelligent or expert behavior. Neural networks can be classified in supervised and unsupervised. The main difference is that in the case of the supervised neural networks the learning algorithm uses input-output training data to model the dynamic system, on the other hand, in the case of unsupervised neural networks only the input data is given. In the case of an unsupervised network, the input data is used to make representative clusters of all the data. It has been shown, that neural networks are universal approximators, in the sense that they can model any general function to a specified accuracy and for this reason neural networks have been applied to problems of system identification, control, diagnosis, time series prediction, and pattern recognition [32, 56, 58, 60, 65]. We also describe the basic concepts, theory and algorithms of modular and ensemble neural networks [52, 59]. We will also give particular attention to the problem of response integration, which is very important because response integration is responsible for combining all the outputs of the modules [80, 93]. Basically, a modular or ensemble neural network uses several monolithic neural networks to solve a specific problem [48, 49, 64]. The basic idea is that combining the results of several simple neural networks we will achieve a better overall result in terms of accuracy and also learning can be done faster [50, 51, 80]. Genetic algorithms and evolutionary methods are optimization methodologies based on principles of nature [24]. Both methodologies can also be viewed as searching algorithms because they explore a space using heuristics inspired by

Chapter 1 Introduction to Type-2 Fuzzy Logic in Neural Pattern Recognition Systems

5

nature [25]. Genetic algorithms are based on the ideas of evolution and the biological process that occur at the DNA level. Basically, a genetic algorithm uses a population of individuals, which are modified by using genetic operators in such a way as to eventually obtain the fittest individual. Any optimization problem has to be represented by using chromosomes, which are a codified representation of the real values of the variables in the problem. Both, genetic algorithms and evolutionary methods can be used to optimize a general objective function. As genetic algorithms are based on the ideas of natural evolution, we can use this methodology to evolve a neural network or a fuzzy system for a particular application [10, 11, 25]. The problem of finding the best architecture of a neural network is very important because there are no theoretical results on this, and in many cases we are forced to trial and error unless we use a genetic algorithm to automate this process. A similar thing occurs in finding out the optimal number of rules and membership functions of a fuzzy system for a particular application, here a genetic algorithm can also help us avoid time consuming trial and error. In this book, we use genetic algorithms to optimize the architecture of fuzzy and neural systems [80]. We describe in this book a new approach for face recognition using modular neural networks with a fuzzy logic method for response integration [63, 64]. We describe a new architecture for modular neural networks for achieving pattern recognition in the particular case of human faces. Also, the method for achieving response integration is based on the fuzzy Sugeno integral and type-2 fuzzy logic. Response integration is required to combine the outputs of all the modules in the modular network. We have applied the new approach for face recognition with a real database of faces from students and professors of our institution. Recognition rates with the modular approach were compared against the monolithic single neural network approach, to measure the improvement. The results of the modular neural network approach gives excellent performance overall and also in comparison with the monolithic approach. We describe a new architecture for modular neural networks for achieving pattern recognition in the particular case of human fingerprints [93]. Also, the method for achieving response integration is based on the fuzzy Sugeno integral. Response integration is required to combine the outputs of all the modules in the modular network. We also describe in this book the use of neural networks, fuzzy logic and genetic algorithms for voice recognition [32, 93]. In particular, we consider the case of speaker recognition by analyzing the sound signals with the help of intelligent techniques, such as the neural networks and fuzzy systems. We use the neural networks for analyzing the sound signal of an unknown speaker, and after this first step, a set of type-2 fuzzy rules is used for decision making. We need to use fuzzy logic due to the uncertainty of the decision process. We also use genetic algorithms to optimize the architecture of the neural networks. We illustrate our approach with a sample of sound signals from real speakers in our institution. This book also presents three modular neural network architectures as systems for recognizing persons based on the iris biometric measurement of humans [80]. In these systems, the human iris database is enhanced with image processing

6

Chapter 1 Introduction to Type-2 Fuzzy Logic in Neural Pattern Recognition Systems

methods, and the coordinates of the center and radius of the iris are obtained to make a cut of the area of interest by removing the noise around the iris. The inputs to the modular neural networks are the processed iris images and the output is the number of the person identified. The integration of the modules was done with a gating network method. This book also describes human recognition from ear images as biometric using modular neural networks with preprocessing ear images as network inputs [80]. We propose a new modular neural network architecture composed of twelve modules, in order to simplify the problem making it smaller. Comparing with other biometrics, ear recognition has one of the best performances, even when it has not received much attention. To compare with other existing methods, we used the 2D Wavelet analysis with global thresholding method for compression, and Sugeno Measures and Winner-Takes-All as modular neural network integrator. Recognition results achieved with this approach were excellent. This book also describes a modular neural network with fuzzy response integration for the problem of signature recognition [59]. Currently, biometric identification has gained a great deal of research interest within the pattern recognition community. For instance, many attempts have been made in order to automate the process of identifying a person’s handwritten signature; however this problem has proven to be a very difficult task. In this work, we propose a modular network that has three separate modules, each using different image features as input, these are: edges, wavelet coefficients, and the Hough transform matrix. Then, the outputs from each of these modules are combined using a Sugeno fuzzy integral and a fuzzy inference system. The experimental results obtained using a benchmark database individual’s shows that the modular architecture can achieve a very high recognition accuracy. Therefore, we conclude that the proposed architecture provides a suitable platform to build a signature recognition system. Furthermore we consider the verification of signatures as false acceptance, false rejection and error recognition of the modular neural network. We describe in this book our new approach for human recognition using as information the combination of three biometric measures, iris, ear, and voice of a person [80]. We have described above the use of intelligent techniques for achieving iris recognition, ear, and voice identification. Now we can consider the integration of these three biometric measures to improve the accuracy of human recognition. The new approach will integrate the information from three main modules, one for each of the three biometric measures. The new approach consists in a modular architecture that contains three basic modules: iris, ear, and voice. The final decision is based on the results of the three modules and uses type-2 fuzzy logic to take into account the uncertainty of the outputs of the modules. Recognition results are improved significantly with the use of type 2 fuzzy logic, especially for real world situations, under uncertainty or noise, when decisions are more difficult to make. Of course, the problem is in this case is how to find the optimal type 2 fuzzy system to integrate the information of the modules in an appropriate fashion. Genetic algorithms are used to automatically design the optimal type 2 fuzzy systems for achieving this goal.

Chapter 2

Type-1 and Type-2 Fuzzy Inference Systems for Images Edge Detection

Edges detection in digital images is a problem that has been solved by means of the application of different techniques from digital signal processing, also the combination of some of these techniques with Fuzzy Inference System (FIS) has been experienced [64]. In this chapter a new FIS Type-2 method is implemented for the detection of edges and the results of three different techniques for the same intention are compared [65].

2.1 Introduction In the area of digital signal processing, methods have been proven that solve the problem of image recognition. Some of them include techniques like binarization, bidimensional filter, detection of edges and compression using banks of filters and trees, among others. Specifically in methods for the detection of edges we can find comparative studies of methods like: Canny, Narwa, Iverson, Bergholm y Rothwell. Others methods can group in two categories: Gradient and Laplacian [65]. The gradient methods like Roberts, Prewitt and Sobel detect edges, looking for maximum and minimum in first derived from the image. The Laplacian methods like Marrs-Hildreth do it finding the zeros of second derived from the image. This work is the beginning of an effort for the design of new pre-processing images techniques, using Fuzzy Inference Systems (FIS), which allows feature extraction and construction of input vectors for neural networks with aims of image recognition. Artificial neural networks are one of the most used objective techniques in the automatic recognition of patterns, here some reasons [59]: • • •

Theoretically any function can be determined. Except the input patterns, it is not necessary to provide additional information. They are possible to be applied to any type of patterns and to any data type.

The idea to apply artificial neural networks for images recognition, tries to obtain results without providing another data that the original images, of this form the

P. Melin: Modular Neural Networks and Type-2 Fuzzy Systems, SCI 389, pp. 7–20. springerlink.com © Springer-Verlag Berlin Heidelberg 2012

8

Chapter 2 Type-1 and Type-2 Fuzzy Inference Systems for Images Edge Detection

process is more similar to the form in which the biological brain learns to recognize patterns, only knowing experiences of past [8, 9, 32, 40]. Models with modular neural networks have been designed, that allow recognizing images divided in four or six parts. This is necessary due to the great amount of input data, since an image without processing is of 100x100 pixels, needs a vector 10000 elements, where each one corresponds to pixel with variations of gray tones between 0 and 255. This chapter shows an efficient Fuzzy Inference System for edges detection, in order to use the output image like input data for modular neural networks. In the proposed technique, it is necessary to apply Sobel operators to the original images, and then use a Fuzzy System to generate the vector of edges that would serve as input data to a neural network.

2.2 Sobel Operators The Sobel operator applied on a digital image, in gray scale, calculates the gradient of the intensity of brightness of each pixel, giving the direction of the greater possible increase of black to white, in addition calculates the amount of change of that direction [65]. The Sobel operator performs a 2-D spatial gradient measurement on an image. Typically it is used to find the approximate absolute gradient magnitude at each point in an input grayscale image. The Sobel edges detector uses a pair of 3x3 convolution masks, one estimating the gradient in the x-direction (columns) and the other estimating the gradient in the y-direction (rows). A convolution mask is usually much smaller than the actual image. As a result, the mask is slid over the image, manipulating a square of pixels at a time. The Sobel masks are shown in equation (2.1):

 − 1 0 1 Sobelx = − 2 0 2  − 1 0 1 

2 1 1  Sobel y =  0 0 0  − 1 − 2 − 1

(2.1)

Where Sobely y Sobelx are the Sobel Operators throughout x-axis and y-axis. If we define I as the source image, gx and gy are two images which at each point contain the horizontal and vertical derivative approximations, the latter are computed as in equations (2.2) and (2.3). i =3 j = 3

g x =  Sobel x ,i , j * I r +i − 2,c + j − 2

(2.2)

i =1 j =1

i =3 j = 3

g y =  Sobel y ,i , j * I r +i −2,c + j −2 i =1 j =1

(2.3)

2.3 Edge Detection by Gradient Magnitude

9

Where gx and gy are the gradients along axis-x and axis-y, and * represents the convolution operator. The gradient magnitude g is calculated with equation (2.4).

g = g x2 + g y2

(2.4)

2.3 Edge Detection by Gradient Magnitude Although the idea presented in this chapter, is to verify the efficiency of a FIS for edges detection in digital images, from the approaches given by Sobel operator, is necessary to display first the obtained results using only the gradient magnitude. It will be used as an example the first image of the subject number one of the ORL database (Figure 2.1). The gray tone of each pixel of this image is a value of between 0 and 255. In Figure 2.2 appears the image generated by gx, and figure 8.3 presents the image generated by gy.

20 40 60 80 100 20

40

60

80

Fig. 2.1 Original Image 1.pgm

20

20

40

40

60

60

80

80

100

100 20

40

60

80

Fig. 2.2 Image given by gx

20

40

60

80

Fig. 2.3 Image given by gy

10

Chapter 2 Type-1 and Type-2 Fuzzy Inference Systems for Images Edge Detection

An example of maximum and minimum values of the matrix given by gx, gy and g from the image 1.pgm is shown in Table 2.1. Table 2.1 Maximum and minimum values from 1.pgm, gx, gy y g

Tone

1.pgm

gx

gy

g

Minimum Maximum

11 234

-725 738

-778 494

0 792

After applying equation (2.4), g is obtained as it is in Figure 2.4.

Fig. 2.4 Edges image given by g

2.4 Edge Detection Using Type-1 Fuzzy Logic A Mamdani FIS was implemented using Type-1 Fuzzy Logic [103], with four inputs, one output and 7 rules, using the Matlab Fuzzy Logic Toolbox, which is shown in Figure 2.5.

M

EDGES DETECTOR DH (mamdani)

DV EDGES

Fig. 2.5 FIS in Matlab Fuzzy Logic Tool Box

2.4 Edge Detection Using Type-1 Fuzzy Logic

11

For the Type-1 Fuzzy Inference System, 4 inputs are required, 2 of them are the gradients with respect to x-axis and y-axis, calculated with equation (2.2) and equation (2.3), to which we will call DH and DV respectively. The other two inputs are filters: A high-pass filter, given by the mask of the equation (2.5), and a low-pass filter given by the mask of equation (2.6). The highpass filter hHP detects the contrast of the image to guarantee the border detection in relative low contrast regions. The low-pass filter hMF allow to detects image pixels belonging to regions of the input were the mean gray level is lower. These regions are proportionally more affected by noise, supposed it is uniformly distributed over the whole image. The goal here is to design a system which makes it easier to include edges in low contrast regions, but which does not favor false edges by effect of noise.

 1 − 16  1 hHP =  −  8 − 1  16

1 1 1  hMF = * 1 25  1 1

1 1 −  8 16 3 1 −  4 8 1 1 − −  8 16  −

1 1 1 1 1 1 1 1 1 1 1 1  1 1 1 1 1 1 1 1

(2.5)

(2.6)

Then the inputs for the type 1FIS are: DH=gx DV=gy HP= hHP*I M= hMF*I where * is the convolution operator. For all the fuzzy variables, the membership functions are of Gaussian type. According to the executed tests, the values in DH and DV, go from -800 to 800, then the ranks in x-axis adjusted as it is in Figures 2.6, 2.7 and 2.8, in where the membership functions are: LOW: gaussmf(43,0), MEDIUM: gaussmf(43,127), HIGH: gaussmf(43,255).

12

Chapter 2 Type-1 and Type-2 Fuzzy Inference Systems for Images Edge Detection

LOW MEDIUM

Degree of membership

1

HIGH

0.8 0.6 0.4 0.2 0 -800

-600

-400

-200

0 DH

200

400

600

800

600

800

600

800

Fig. 2.6 Input variable DH

LOW

Degree of membership

1

MEDIUM

HIGH

0.8 0.6 0.4 0.2 0 -800

-600

-400

-200

0 DV

200

400

Fig. 2.7 Input variable DV

LOW

Degree of membership

1

MEDIUM

HIGH

0.8 0.6 0.4 0.2 0 -800

-600

-400

-200

0 HP

200

400

Fig. 2.8 Input variable HP

In the case of variable M, the tests threw values in the rank from 0 to 255, and thus the rank in x-axis adjusted, as it is appreciated in Figure 2.9.

2.4 Edge Detection Using Type-1 Fuzzy Logic

Degree of membership

MEDIUM

LOW

1

13

HIGH

0.8 0.6 0.4 0.2 0 0

50

100

150

200

250

M

Fig. 2.9 Input variable M

In Figure 2.10 is the output variable EDGES that also adjusted the ranks between 0 and 255, since it is the range of values required to display the edges of an image.

LOW

Degree of membership

1

HIGH

MEDIUM

0.8 0.6 0.4 0.2 0 0

50

100

150

200

250

EDGES

Fig. 2.10 Output variable EDGES

The seven fuzzy rules that allow to evaluate the input variables, so that the exit image displays the edges of the image in color near white (HIGH tone), whereas the background was in tones near black (tone LOW). 1. If (DH is LOW) and (DV is LOW) then (EDGES is LOW) 2. If (DH is MEDIUM) and (DV is MEDIUM) then (EDGES is HIGH) 3. If (DH is HIGH) and (DV is HIGH) then (EDGES is HIGH)

14

Chapter 2 Type-1 and Type-2 Fuzzy Inference Systems for Images Edge Detection

4. If (DH is MEDIUM) and (HP is LOW) then (EDGES is HIGH) 5. If (DV is MEDIUM) and (HP is LOW) then (EDGES is HIGH) 6. If (M is LOW) and (DV is MEDIUM) then (EDGES is LOW) 7. If (M is LOW) and (DH is MEDIUM) then (EDGES is LOW)

The result obtained for image of Figure 2.1 is remarkably better than the one than it was obtained with the method of gradient magnitude, as it is in Figure 2.11.

Fig. 2.11 EDGES Image by FIS Type 1

Reviewing the values of each pixel, we see that all fall in the rank from 0 to 255, which is not obtained with the method of gradient magnitude.

2.5 Edge Detection Using Type-2 Fuzzy Logic For the Type-2 FIS, the same method was followed as in Type-1 FIS, indeed to be able to make a comparison of both results. The tests with the type-2 FIS, were executed using the computer program imagen_bordes_fis2.m, which creates a Type-2 Inference System (Mamdani) by intervals. The mentioned program creates the type 2 fuzzy variables as it is seen in Figure 2.12. The wide of the FOU chosen for each membership function was the one that had better results after several experiments. The program imagen_bordes_fuzzy2.m was implemented to load the original image, and to apply the filters before mentioned. Because the great amount of data that the fuzzy rules must evaluate, the image was divided in four parts, and the FIS was applied to each one separately. The result of each evaluation gives a vector with tones of gray by each part of the image, in the end is the complete image with the edges (Figure 2.13).

2.5 Edge Detection Using Type-2 Fuzzy Logic

15

LOW MEDIUM HIGH

0.5

-800

-600

-400

-200

200

400

600

800

M LOW MEDIUM HIGH

0.5

-800

-600

-400

-200

200 400 DH LOW MEDIUM HIGH

600

800

-600

-400

-200

200 400 DV LOW MEDIUM HIGH

600

800

-600

-400

-200

200 400 HP LOW MEDIUM HIGH

600

800

-600

-400

-200

600

800

0.5

-800

0.5

-800

0.5

-800

200

400

EDGES

Fig. 2.12 Type-2 fuzzy variables

Fig. 2.13 EDGES Image by Type 2 FIS

16

Chapter 2 Type-1 and Type-2 Fuzzy Inference Systems for Images Edge Detection

2.6 Comparison of Results The first results of several tests conducted in different images can be appreciated in Table 2.2. Table 2.2 Results of Edge Detection by FIS1 y FIS2 (dark background)

Original Image

EDGES (FIS 1)

EDGES (FIS 2)

At first, the results with Type-1 FIS and Type-2 FIS can be seen to be very similar. However thinking about that to show the images with a dark background it could confuse the contrast of tones, tests were done inverting the consequent of

2.6 Comparison of Results

17

the rules, so that the edges take the dark tone and the bottom the clear tone, the rules changed to the following form: 1. If (DH is LOW) and (DV is LOW) then (EDGES is HIGH) 2. If (DH is MEDIUM) and (DV is MEDIUM) then (EDGES is LOW) 3. If (DH is HIGH) and (DV is HIGH) then (EDGES is LOW) 4. If (DH is MEDIUM) and (HP is LOW) then (EDGES is LOW) 5. If (DV is MEDIUM) and (HP is LOW) then (EDGES is LOW) 6. If (M is LOW) and (DV is MEDIUM) then (EDGES is HIGH) 7. If (M is LOW) and (DH is MEDIUM) then (EDGES is HIGH)

Fuzzy Systems were tested both (Type-1 and Type-2), with the new fuzzy rules and same images, obtaining the results that are in Table 2.3. Table 2.3 Results of Edge Detection by FIS1 y FIS2 (clear background)

EDGES (FIS 1)

EDGES (FIS 2)

18

Chapter 2 Type-1 and Type-2 Fuzzy Inference Systems for Images Edge Detection

In this second test can be appreciated a great difference between the results obtained with the FIS 1 and FIS 2, noticing at first a greater contrast in the images obtained with the FIS 1 and giving to the impression of a smaller range of tones of gray in the type-2 FIS. In order to obtain an objective comparison of the images, histograms were elaborated respectively corresponding to the resulting matrices of edges of the FIS 1 and FIS 2, which are in Table2.4. The histograms show in the y-axis the range of tones of gray corresponding to each image and in x-axis the frequency in which he appears pixel with each tone. Table 2.4 Histograms of the Resulting Images of the Edges by Gradient Magnitud, Fis 1 and Fis 2 Methods IMAGE: 1.PGM METHOD: GRADIENT MAGNITUDE

GM-1.pgm-DB

300

200

100

0 0

50

100

150

200

250

FIS 1-1.pgm-CB

METHOD: FIS 1 (CLEAR BACKGROUND)

1000

500

0 0

50

100

150

200

250

METHOD: FIS 2 (CLEAR BACKGROUND)

FIS 2-1.pgm-CB

4000 3000 2000 1000 0 0

50

100

150

200

250

As we can observe, unlike detector FIS1, with FIS2 the edges of an image could be obtained from very complete form, only taking the tones around 150 and 255.

2.7 Summary

19

As a last experiment, in this occasion to the resulting images of the FIS Type-2 the every pixel out of the range between 50 and 255 was eliminated. Table 2.5 shows the amount of elements that was possible to eliminate in some of the images, we see that the Type-2 Edges Detector FIS allows to using less than half of the original pixels without losing the detail of the images. This feature could be a great advantage if these images are used like input data in neural networks for detection of images instead the original images. We have to mention that type-2 fuzzy systems were implemented using the type-2 fuzzy toolbox developed previously by our group [16, 17, 18]. Table 2.5 Type-2FIS Edges Images Including Only Pixels with Tones between 150 and 255 BORDERS IMAGE

DIMENSION (pixels)

PIXELS INCLUDED

108x88

4661

(9504)

49 %

144x110

7077

(15840)

44.6 %

10 20 30 40 50 60 70 80 90 100 10

20

30

40

50

60

70

80

20

40

60

80

100

120

140 20

40

60

80

100

2.7 Summary The application of Sobel filters was very useful to define the input vectors for the Type-1 FIS and the Type-2 FIS, although in future works we will try to design Neuro-Fuzzy techniques able to extract image patterns without another data that the original image and to compare the results with traditional techniques of digital signal processing.

20

Chapter 2 Type-1 and Type-2 Fuzzy Inference Systems for Images Edge Detection

Thanks to the histograms of the images it was possible to verify the improvement of results of the FIS Type-1 with respect to the FIS Type-2, since with only the appreciation of the human eye was very difficult to see an objective difference. The best result was obtained by the Type-2 Fuzzy Inference System, because it was possible to clear more than half of the pixels without depreciating the image, which will reduce in drastic form the cost of training in a neural network.

Chapter 3

Type-2 Fuzzy Logic for Improving Training Data and Response Integration in Modular Neural Networks

The combination of Soft Computing techniques allows the improvement of intelligent systems with different hybrid approaches [12, 57, 59]. In this work we consider two parts of a Modular Neural Network for image recognition, where a Type-2 Fuzzy Inference System (FIS 2) makes a great difference. The first FIS 2 is used for feature extraction in training data, and the second one to find the ideal parameters for the integration method of the modular neural network. Once again fuzzy logic is shown to be a tool that can help improve the results of a neural system, when facilitating the representation of the human perception.

3.1 Method for Image Recognition At the moment, many methods for image recognition are available. But most of them include a phase of feature extraction or another type of preprocessing closely related to the type of image to recognize. The method proposed in this paper can be applied to any type of images, because the preprocessing phase does not need specific data about the type of image. Even if the method was not designed only for face recognition [64], we have made the tests with the ORL face database (AT&T Laboratories Cambridge) composed of 400 images of size 112x92. There are 40 persons, with 10 images of each person. The images are taken at different times, lighting and facial expressions. The faces are in up-right position of frontal view, with slight left-right rotation. Figure 3.1 shows the 10 samples of one person in ORL database. To explain the proposed steps of the method, we need to separate it them in two phases: the training phase in Figure 3.2 and the recognition phase in Figure 3.3.

P. Melin: Modular Neural Networks and Type-2 Fuzzy Systems, SCI 389, pp. 21–28. springerlink.com © Springer-Verlag Berlin Heidelberg 2012

22

Chapter 3 Type-2 Fuzzy Logic for Improving Training Data and Response Integration

Fig. 3.1 Set of 10 samples of a person in ORL

Fig. 3.2 Steps in Training Phase

Fig. 3.3 Steps in Recognition Phase

3.2 Type-2 Fuzzy Inference System as Edge Detector In previous work we presented an efficient fuzzy inference system for edges detection, in order to use the output image like input data for modular neural networks. In the proposed technique, it is necessary to apply Sobel operators to the original images, then use a Type-2 Fuzzy Inference System [12] to generate the vector of edges that would serve like input data in a neural network. Type-2 Fuzzy Logic enables us to handle uncertainties in decision making [62] and recognition in a more convenient way and for this reason was proposed.

3.2 Type-2 Fuzzy Inference System as Edge Detector

23

For the Type-2 Fuzzy Inference System, 3 inputs are required, 2 of them are the gradients with respect to x-axis and y-axis, calculated with (3.1), to which we will call DH and DV respectively. The Sobel edges detector uses a pair of 3x3 convolution masks, one estimating the gradient in the x-direction (columns) and the other estimating the gradient in the y-direction (rows). 2 1 1  − 1 0 1    0 0  Sobelx = − 2 0 2 Sobely =  0 − 1 − 2 − 1  − 1 0 1 

(3.1)

Where Sobely y Sobelx are the Sobel Operators throughout x-axis and y-axis. If we define I as the source image, gx and gy are two images which at each point contain the horizontal and vertical derivative approximations, the latter are computed as (3.2) and (3.3). i =3 j =3

g x =  Sobel x ,i , j * I r +i − 2,c + j − 2

(3.2)

i =1 j =1

i =3 j =3

g y =  Sobely ,i , j * I r +i −2,c+ j −2 i =1 j =1

(3.3)

Where gx and gy are the gradients along axis-x and axis-y, and * represents the convolution operator. The other input is a filter that calculates when applying a mask by convolution to the original image. The low-pass filter hMF (3.4) allow us to detect image pixels belonging to regions of the input were the mean gray level is lower. These regions are proportionally more affected by noise, supposed it is uniformly distributed over the whole image. The goal here is to design a system which makes it easier to include edges in low contrast regions, but which does not favor false edges by effect of noise. 1 1 1  hMF = * 1 25  1 1

1 1 1 1 1

1 1 1 1 1

1 1 1 1 1

1 1 1  1 1

(3.4)

Then the inputs for FIS type 2 are: DH=gx, DV=gy, M= hMF*I, where * is the convolution operator, and de output is a column vector contains the values of the image edges, and we can represent that in graphics shown in Figure 3.4. The Edges Image is smaller than the original because the result of convolution operation is a central matrix where the convolution has a value. Then in our example, each image with dimension 112x92 is reduced to 108x88.

24

Chapter 3 Type-2 Fuzzy Logic for Improving Training Data and Response Integration

The inference rules and membership function parameters allow us to calculate a gray value between -4.5 and 1.5 for each pixel, where the most negative values corresponds to the dark tone in the edges of the image. Then if we see the rules, only when the increment value of the inputs DH and DV are low the output is HIGH or clear (the background), in the rest of rules the output is LOW or dark (the edges). The complete set of fuzzy rules is given as follows: 1. 2. 3. 4. 5.

If (DH is LOW) and (DV is LOW) then (EDGES is HIGH) (1) If (DH is MEDIUM) and (DV is MEDIUM) then (EDGES is LOW) (1) If (DH is HIGH) and (DV is HIGH) then (EDGES is LOW) (1) If (M is LOW) and (DV is MEDIUM) then (EDGES is LOW) (1) If (M is LOW) and (DH is MEDIUM) then (EDGES is LOW) (1)

1

LOW

MEDIUMHIGH

1

0.5

0 -200

1

MEDIUMHIGH

0.5

0

200 DH

400

600

LOW MEDIUM HIGH

0 -200

1

0.5

0 -200

LOW

0

200 DV

400

LOW

600 HIGH

0.5

0

200 M

400

0 -6

-4

-2 EDGES

0

2

Fig. 3.4 Membership Function for the Type-2 FIS Edge Detector

The edge detector allows us to ignore the background color. We can see in this database of faces, different tones present for the same or another person. Then we eliminate a possible influence of a bad classification by the neural network, without losing detail in the image. Another advantage of edge detector is that the values can be normalized to a homogenous value range, independently the light, contrast or background tone in each image. At the examples in Figure 3.5, all the edges in the images have a minimum value of -3.8 and a maximum value of 0.84. In particular for neural network training, we find these values to make the training faster: the mean of the values is near 0 and the standard deviation is near 1 for all the images.

3.3 The Modular Structure

25

Fig. 3.5 Examples of edge detection with the Type-2 FIS method

3.3 The Modular Structure The design of the Modular Neural Network consists of 3 monolithic feedforward neural networks, each one trained with a supervised method with the first 7 samples of the 40 images [64]. Then the edges vector column is accumulated until the number of samples to form the input matrix for the neural networks as it is in the scheme of Figure 3.7. Once the complete matrix of images is divided in 3 parts, each module is training with a correspondent part, with some rows of overlap. The target to the supervised training method consist of one identity matrix for each sample, building one matrix with dimensions 40x(40*number_of_samples), as shown in Figure 3.8. Each Monolithic Neural Network has the same structure and is trained under the same conditions, like we can see in the next code segment: layer1=200; layer2=200; layer3=number_of_subjects; net=newff(minmax(p),[layer1,layer2,layer3], {'tansig','tansig','logsig'},'traingdx'); net.trainParam.goal=1e-5; net.trainParam.epochs=1000; The average number of epochs to meet the goal in each module is of 240, and the required time of 160 seconds.

3.4 Simulation Results A program was developed in Matlab that simulates each module with the 400 images of the ORL database, building a matrix with the results of the simulation

26

Chapter 3 Type-2 Fuzzy Logic for Improving Training Data and Response Integration

of each module. These matrices are stored in the file “mod.mat” to be analyzed later for the combination of results. We can observe that in the columns corresponding to the training data, the position with a value near one is the image selected correctly. However in the columns that correspond to the test data this doesn’t always happens, reason why it is very important to have a good combination method to recognize more images. According to exhaustive tests made in the simulation matrices, we know that recognition of the images that were used for the training of the neural networks is of the 100%. Therefore the interest is focused on the recognition of the samples that do not belong to the training set, is to say samples 8, 9 and 10. The parameters for the Sugeno Fuzzy Integral that will be inferred will be the Fuzzy Densities, a value between 0 and 1 for each module, which determines the rate for each module. The parameter lambda, according to the theory of fuzzy measures depends on the values of the fuzzy densities, and is calculated by searching for the roots of a polynomial. After the simulation of an image in the Neural Network, the simulation value is the only known parameter to make a decision, then to determine the fuzzy density for each module is the unique available information. For this reason we analyze the values in many simulations matrix and decide that each input to the FIS Type-2 corresponds to the maximum value of each column corresponding to the simulation of each module of each one of the 400 images. The process to recognize each one of the images is shown in Figure 3.6.

Fig. 3.6 Process of recognition using the type-2 fuzzy modular approach

Then each output corresponds to one fuzzy density, to be applied for each module to perform the fusion of results later with the Fuzzy Sugeno Integral. The inference rules found fuzzy densities near 1 when de maximum value in the simulation is between 0.5 and 1, and near 0 when the maximum value in the simulation is near 0. The fuzzy rules are shown below and membership functions in Figure 3.7.

3.4 Simulation Results

27

1. If (max1 is LOW) then (d1 is LOW) (1) 2. If (max2 is LOW) then (d2 is LOW) (1) 3. If (max3 is LOW) then (d3 is LOW) (1) 4. If (max1 is MEDIUM) then (d1 is HIGH) (1) 5. If (max2 is MEDIUM) then (d2 is HIGH) (1) 6. If (max3 is MEDIUM) then (d3 is HIGH) (1) 7. If (max1 is HIGH) then (d1 is HIGH) (1) 8. If (max2 is HIGH) then (d2 is HIGH) (1) 9. If (max3 is HIGH) then (d3 is HIGH) (1) Although the rules are very simple, allows to model the fuzziness to rate de modules when the simulation result don’t reach the maximum value 1. However some of the images don’t reach the sufficient value in the simulation of the three modules, in these cases, do not exists enough information to select an image at the modules combination, and the image is wrongly selected. 1

LOW MEDIUM HIGH

1

0.5 0 -0.5 1

0.5

0

0.5 1 max1 LOW MEDIUM HIGH

1.5

1

0

0.5 1 d1 LOW MEDIUM HIGH

1.5

0

1.5

0.5

0

0.5 1 max2 LOW MEDIUM HIGH

1.5

0 -0.5 1

0.5 0 -0.5

0 -0.5 1

0.5 0 -0.5

LOW MEDIUM HIGH

0.5 1 d2 LOW MEDIUM HIGH

0.5

0

0.5 max3

1

1.5

0 -0.5

0

0.5 d3

1

1.5

Fig. 3.7 Membership functions for the FIS to find fuzzy densities

In order to measure of objective form the final results, we developed a method of random permutation, which rearranges the samples of each person before the training. Once a permutation is made, the modular neural networks are trained and combined four times to obtain the sufficient information to validate the results. The average recognition rate is of 96.5%.

28

Chapter 3 Type-2 Fuzzy Logic for Improving Training Data and Response Integration

We show in Table 3.1 the summary of simulation results for each of the modules and the average and maximum results of the modular network (after fusion or combination of the results). Table 3.1 Summary of the simulation results with the hybrid approach Permutation

1 2 3 4 5

Train 1

92.75 96.5 91.5 94.5 93.75

Train 2

95 95.25 92 94.5 93.5

Image Recognition (%) Train 3 Train 4 Average

92.2 94.25 93.75 93.25 94

93.25 95.5 95.25 94 96

93.3 95.375 93.125 94.0625 94.3125 94.035

Maximum

95 96.5 95.25 94.5 96 96.5

We have to mention that type-2 fuzzy systems were implemented using the type-2 fuzzy toolbox developed previously by our group [16, 17, 18].

3.5 Summary We have shown in this chapter that the combination of Soft Computing techniques allows the improvement of intelligent systems with different hybrid approaches. In this chapter we considered two parts of a Modular Neural Network for image recognition, where a Type-2 Fuzzy Inference System (FIS 2) help us improves the performance results in image recognition. The first FIS 2 was used for feature extraction in training data, and the second one to find the ideal parameters for the integration method of the modular neural network.

Chapter 4

Method for Response Integration in Modular Neural Networks Using Type-2 Fuzzy Logic

We describe in this chapter a new method for response integration in modular neural networks using type-2 fuzzy logic [59]. The modular neural networks were used in human person recognition [64, 80]. Biometric authentication is used to achieve person recognition. Three biometric characteristics of the person are used: face, fingerprint, and voice [32]. A modular neural network of three modules is used. Each module is a local expert on person recognition based on each of the biometric measures. The response integration method of the modular neural network has the goal of combining the responses of the modules to improve the recognition rate of the individual modules. We show in this chapter the results of a type-2 fuzzy approach for response integration that improves performance over type-1 fuzzy logic approaches.

4.1 Introduction Today, a variety of methods and techniques are available to determine unique identity, the most common being fingerprint, voice, face, and iris recognition [7, 8, 9, 36, 40, 52, 54, 55, 74, 107]. Of these, fingerprint and iris offer a very high level of certainty as to a person's identity, while the others are less exact. A large number of other techniques are currently being examined for suitability as identity determinants. These include (but are not limited to) retina, gait (walking style), typing style, body odor, signature, hand geometry, and DNA. Some wildly esoteric methods are also under development, such as ear structure, thermal imaging of the face and other parts of the body, subcutaneous vein patterns, blood chemistry, anti-body signatures, and heart rhythm, to name a few. The four primary methods of biometric authentication in widespread use today are face, voice, fingerprint, and iris recognition. All of these are supported in our approach, some more abundantly than others. Generally, face and voice are considered to be a lower level of security than fingerprint and iris, but on the other hand, they have a lower cost of entry. We describe briefly in this section some of these biometric methods [59].

P. Melin: Modular Neural Networks and Type-2 Fuzzy Systems, SCI 389, pp. 29–39. springerlink.com © Springer-Verlag Berlin Heidelberg 2012

30

Chapter 4 Method for Response Integration in Modular Neural Networks

Face Recognition. Facial recognition has advanced considerably in the last 10 to 15 years. Early systems, based entirely on simple geometry of key facial reference points, have given way to more advanced mathematically-based analyses such as Local Feature Analysis and Eigenface evaluation. These have been extended though the addition of "learning" systems, particularly neural networks. Face recognition systems are particularly susceptible to changes in lighting systems. For example, strong illumination from the side will present a vastly different image to a camera than neutral, evenly-positioned fluorescent lighting. Beyond this, however, these systems are relatively immune to changes such as weight gain, spectacles, beards and moustaches, and so on. Most manufacturers of face recognition systems claim false accept and false reject rates of 1% or better. Voice Recognition. Software systems are rapidly becoming adept at recognising and converting free-flowing speech to its written form. The underlying difficulty in doing this is to flatten out any differences between speakers and understand everyone universally. Alternatively, when the goal is to specifically identify one person in a large group by their voice alone, these very same differences need to be identified and enhanced. As a means of authentication, voice recognition usually takes the form of speaking a previously-enrolled phrase into a computer microphone and allowing the computer to analyse and compare the two sound samples. Methods of performing this analysis vary widely between vendors. None is willing to offer more than cursory descriptions of their algorithms--principally because, apart from LAN authentication, the largest market for speaker authentication is in verification of persons over the telephone. Fingerprint Recognition. The process of authenticating people based on their fingerprints can be divided into three distinct tasks. First, you must collect an image of a fingerprint; second, you must determine the key elements of the fingerprint for confirmation of identity; and third, the set of identified features must be compared with a previously-enrolled set for authentication. The system should never expect to see a complete 1:1 match between these two sets of data. In general, you could expect to couple any collection device with any algorithm, although in practice most vendors offer proprietary, linked solutions. A number of fingerprint image collection techniques have been developed. The earliest method developed was optical: using a camera-like device to collect a highresolution image of a fingerprint. Later developments turned to silicon-based sensors to collect an impression by a number of methods, including surface capacitance, thermal imaging, pseudo-optical on silicon, and electronic field imaging. As discussed, a variety of fingerprint detection and analysis methods exist, each with their own strengths and weaknesses. Consequently, researchers vary widely on their claimed (and achieved) false accept and false reject rates. The poorest systems offer a false accept rate of around 1:1,000, while the best are approaching 1:1,000,000. False reject rates for the same vendors are around 1:100 to 1:1000.

4.3 Modular Neural Networks

31

4.3 Modular Neural Ne

twor ks

4.2 Proposed Approach for Recognition Our proposed approach for human recognition consists in integrating the information of the three main biometric parts of the person: the voice, the face, and the fingerprint [59]. Basically, we have an independent system for recognizing a person from each of its biometric information (voice, face, and fingerprint), and at the end we have an integration unit to make a final decision based on the results from each of the modules. In Figure 4.1 we show the general architecture of our approach in which it is clearly seen that we have one module for voice, one module for face recognition, and one module for fingerprint recognition [32]. At the top, we have the decision unit integrating the results from the three modules. In this paper the decision unit is implemented with a type-2 fuzzy system.

Fig. 4.1 Architecture of the proposed modular approach

4.3 Modular Neural Networks This section describes a particular class of "modular neural networks", which have a hierarchical organization comprising multiple neural networks; the architecture basically consists of two principal components: local experts and an integration unit, as illustrated in Figure 4.2. In general, the basic concept resides in the idea that combined (or averaged) estimators may be able to exceed the limitation of a single estimator [59]. The idea also shares conceptual links with the "divide and conquer" methodology [87]. Divide and conquer algorithms attack a complex problem by dividing it into simpler problems whose solutions can be combined to yield a solution to the complex problem. When using a modular network, a given task is split up among several local experts NNs. The average load on each NN is reduced in comparison with a single NN that must learn the entire original task, and thus the combined model may be able to surpass the limitation of a single NN.

32

Chapter 4 Method for Response Integration in Modular Neural Networks

The outputs of a certain number of local experts (Oi) are mediated by an integration unit. The integrating unit puts the outputs together using estimated combination weights (gi). The overall output Y is given by equation (4.1).

Module 1

Module 2

y1 g1

y2 g2

x

 Module N



y=f(x)

yN g

N  Gating Network

Fig. 4.2 Architecture of a modular neural network

Yi = Σgi OI

(4.1)

Nowlan, Jacobs, Hinton, and Jordan described modular networks from a competitive mixture perspective. That is, in the gating network, they used the "softmax" function [59]. More precisely, the gating network uses a softmax activation gi of ith output unit given by Gi = exp (kui)/ Σj exp (kuj)

(4.2)

Where ui is the weighted sum of the inputs flowing to the ith output neuron of the gating network. Use of the softmax activation function in modular networks provides a sort of "competitive" mixing perspective because the ith local expert's output Oi with a minor activation ui does not have a great impact on the overall output Yi.

4.4 Integration of Results for Person Recognition Using Fuzzy Logic On the past decade, fuzzy systems have displaced conventional technology in different scientific and system engineering applications, especially in pattern recognition

4.5 Modular Neural Networks with Type-2 Fuzzy Logic

33

4.5 Mod ular Neural Networks with Ty pe-2 Fuzzy Logic

and control systems. The same fuzzy technology, in approximation reasoning form, is resurging also in the information technology, where it is now giving support to decision making and expert systems with powerful reasoning capacity and a limited quantity of rules. For the case of modular neural networks, a fuzzy system can be used as an integrator or results. The fuzzy sets were presented by L. A. Zadeh in 1965 to process / manipulate data and information affected by unprobabilistic uncertainty / imprecision [103]. These were designed to mathematically represent the vagueness and uncertainty of linguistic problems; thereby obtaining formal tools to work with intrinsic imprecision in different type of problems; it is considered a generalization of the classic set theory. Type-2 fuzzy sets are used for modeling uncertainty and imprecision in a better way [13, 26, 27, 28, 34, 35, 38, 39, 43, 53]. These type-2 fuzzy sets were originally presented by Zadeh in 1975 and are essentially “fuzzy fuzzy” sets where the fuzzy degree of membership is a type-1 fuzzy set. The new concepts were introduced by Mendel [62] allowing the characterization of a type-2 fuzzy set with a superior membership function and an inferior membership function; these two functions can be represented each one by a type-1 fuzzy set membership function. The interval between these two functions represents the footprint of uncertainty (FOU), which is used to characterize a type-2 fuzzy set [41]. The uncertainty is the imperfection of knowledge about the natural process or natural state [12]. The statistical uncertainty is the randomness or error that comes from different sources as we use it in a statistical methodology.

4.5 Modular Neural Networks with Type-2 Fuzzy Logic as a Method for Response Integration As was mentioned previously, type-2 fuzzy logic was used to integrate the responses of the three modules of the modular network. Each module was trained with the corresponding data, i.e. face, fingerprint and voice. Also, a set of modular neural networks was built to test the type-2 fuzzy logic approach of response integration. The architecture of the modular neural network is shown in Figure 4.3. From this figure we can appreciate that each module is also divided in three parts with the idea of also dividing each of the recognition problems in three parts. Experiments were performed with sets of 20 and 30 persons. The trainings were done with different architectures, i.e. different number of modules, layers and nodes. As can be appreciated from Figure 4.3, the first module was used for training with voice data. In this case, three different words were used for each person. The words used were: access, presentation, and hello.

34

Chapter 4 Method for Response Integration in Modular Neural Networks

MODULE 1 SubMod. 1 SubMod. 2 SubMod. 3 MODULE 2 SubMod. 1 SubMod. 2 SubMod. 3 MODULE 3 SubMod. 1 SubMod. 2 SubMod. 3

Fig. 4.3 Architecture of the Modular Network used for the recognition problem

The second module was used for training with person face data. In this case, two different photos were taken from each person, one in a normal position and the other with noise. The idea is that training with noise will make the recognition more robust to changes in the real world. We show in Figure 4.4 the photos of two persons in a normal situation and in a noisy situation. Normal Images

Images with Noise

Fig. 4.4 Sample Photos of Faces in a Normal and Noisy Situation

4.6 Simulation Results

35

The third module was used with fingerprint data of the group of persons. The fingerprint information was taken with a scanner. Noise was added for training the neural networks. In all cases, each module is subdivided in three sub-modules, in this way making easier the respective recognition problem.

4.6 Simulation Results A set of different trainings for the modular neural networks was performed to test the proposed type-2 fuzzy logic approach for response integration in modular neural networks. We show in Table 4.1 some of these trainings with different numbers of modules, layers and nodes. The training times are also shown in this table to illustrate the performance with different training algorithms and conditions. Table 4.1 Sample Trainings of the Modular Neural Network

36

Chapter 4 Method for Response Integration in Modular Neural Networks

Once the necessary trainings were done, a set of tests were performed with different type-2 fuzzy systems. The fuzzy systems were used as response integrators for the three modules of the modular network. In the type-2 fuzzy systems, different types of membership functions were considered with goal of comparing the results and deice on the best choice for the recognition problem. The best type-2 fuzzy system, in the sense that it produced the best recognition results, was the one with triangular membership functions. This fuzzy system has 3 input variables and one output variable, with three membership functions per variable. We show in Figures 4.5 and 4.6 the membership functions of the type-2 fuzzy system.

Fig. 4.5 Input variables of the type-2 fuzzy system

Fig. 4.6 Output variables of the type-2 fuzzy system

The recognition results of this type-2 fuzzy system for each of the trainings of the modular neural network are shown in Table 4.2. In Table 4.2 we show the results for 15 trainings of the modular neural network. In each row of this table we can appreciate the recognition rate with the type-2 fuzzy system. We can appreciate that in 8 out of 15 cases, a 100% recognition rate was achieved.

4.6 Simulation Results

37

Table 4.2 Results of the Type-2 Fuzzy System with Triangular Membership Functions

The fuzzy systems with the worst results for the modular neural network were the ones with Gaussian and Trapezoidal membership functions. We use 3 input variables and one output variable, as in the previous fuzzy system. We show in Figures 4.7 and 4.8 the Gaussian membership functions of this system.

Fig. 4.7 Input variables for type-2 fuzzy system with Gaussian membership functions

38

Chapter 4 Method for Response Integration in Modular Neural Networks

Fig. 4.8. Output variable for type-2 fuzzy system with Gaussian membership functions

We show in Figures 4.9 and 4.10 the Trapezoidal membership functions of another type-2 fuzzy system.

Fig. 4.9. Input variables for the Type-2 Fuzzy System with Trapezoidal Functions

Fig. 4.10. Output variable for type-2 fuzzy system with Trapezoidal functions.

The results that were obtained with Gaussian and Trapezoidal membership functions are similar. We show in Table 4.3 the recognition results obtained with the type-2 fuzzy system with Trapezoidal membership functions. We can appreciate from Table 4.3 that only in 6 out of the 15 cases a 100% recognition rate is obtained. Also, there are 4 cases with low recognition rates.

4.7 Summary

39

Table 4.3 Recognition rates with the Type-2 System and Trapezoidal Functions

We have to mention that results with a type-1 fuzzy integration of responses were performed in previous work, in which the recognition rates were consistently lower by an average of 5%. We can state in conclusion that the type-2 fuzzy system for response integration is improving the recognition rate in the case of persons based on face, fingerprint and voice. We have to mention that type-2 fuzzy systems were implemented using the type-2 fuzzy toolbox developed previously by our group [16, 17, 18].

4.7 Summary We described in this chapter a new method for response integration in modular neural networks that uses type-2 fuzzy logic to model uncertainty in the decision process. We showed different trainings of the modular neural networks, and tested different type-2 fuzzy systems for response integration. Based on the obtained recognition rates, the best results were achieved with a type-2 fuzzy system with triangular membership functions. The results obtained with this type-2 fuzzy system are better than the previously obtained by a similar type-1 approach.

Part II

Chapter 5

Modular Neural Networks for Person Recognition Using the Contour Segmentation of the Human Iris

This chapter presents three modular neural network architectures as systems for recognizing persons based on the iris biometric measurement of humans [80]. In these systems, the human iris database is enhanced with image processing methods, and the coordinates of the center and radius of the iris are obtained to make a cut of the area of interest by removing the noise around the iris. The inputs to the modular neural networks are the processed iris images and the output is the number of the person identified. The integration of the modules was done with a gating network method [59].

5.1 Introduction This chapter is focused on the area of modular neural networks for pattern recognition based on biometric measures [32], specifically in the recognition by the human iris biometric measurement [80]. At present, biometric measurements are being widely used for person recognition systems [59]. A lot has been said about the use of such measures, particularly for the signature, fingerprint, face and voice [64]. As more research was done in this area further biometric measures were discovered, among which the human iris by its peculiarity of not losing over the years its universality and authenticity. In order to get a good identification, we proposed a modular neural network architecture divided into three modules, each module input is a part of the database of human iris, and some methods or techniques are used for pre-processing the images such as normalization, resizing, cut, edge detection, among several others. The end result in terms of modular neural network architecture and image preprocessing depends on the tests made and also the time to achieve identification [64, 65]. The chapter is organized as follows: Section 2 contains a brief explanation from previous research with human iris for recognition of people and basic concepts relevant to the area, section 3 defines the method proposed for this research and the description of problem addressed in this paper, section 4 presents the results achieved in research and in Section 5 draws conclusions and future work. P. Melin: Modular Neural Networks and Type-2 Fuzzy Systems, SCI 389, pp. 43–59. springerlink.com © Springer-Verlag Berlin Heidelberg 2012

44

Chapter 5 Modular Neural Networks for Person Recognition

5.2 Background and Basic Concepts 5.2.1 Modular Neural Network An artificial neural network (ANN) is a distributed computing scheme based on the structure of the nervous system of humans [59]. The architecture of a neural network is formed by connecting multiple elementary processors, this being an adaptive system that has an algorithm to adjust their weights (free parameters) to achieve the performance requirements of the problem based on representative samples. Therefore we note that an ANN is a distributed computing system characterized by:  A set of elementary units, each of which has low processing capabilities.  A dense interconnected structure using weighted links.  Free parameters to be adjusted to meet performance requirements.  A high degree of parallelism. The most important property of artificial neural networks is their ability to learn from a training set of patterns, i.e. is able to find a model that fit the data. The artificial neuron consists of several parts. On one side are the inputs, weights, the summation, and finally the adapter function. The input values are multiplied by a weights and added:

x w i

i

ij

. This function is completed with

the addition of a threshold amount θ i This threshold has the same effect as an entry with value -1. It serves so that the sum can be shifted left or right of the origin. After addition, we have the function f applied to the resulting set the final value of the output, also called yi. The result of the sum before applying the function f, also often called activation value ai. The modular neural networks are composed of simple networks that behave as functional blocks and these are the neural modules. A modular neural network works similarly to a classical neural network, as it is composed of sigmoidal, linear or discrete activation neurons and are trained with common learning algorithms (gradient descent with adaptive learning algorithm, backpropagation, gradient descent scaling, etc.). What distinguishes it from other neural models, is that it is developed based on functional modules and each module runs a neural network with the same characteristics or different (input layer, hidden layers, output layer, depending activation, learning algorithm, number of neurons per layer, etc.). In this model the modules work independently and in the end a form commonly called integrator performs the function of deciding between the different modules to determine which of them has the best solution (including network of gateways, fuzzy integrator, etc.). Figure 5.1 shows a modular neural network scheme:

5.2 Background and Basic Concepts

45

Fig. 5.1 Schematic of a modular artificial neural network

5.2.2 Historical Development The first use of the iris was presented in Paris [80], where criminals were classified according to the color of their eyes following a proposal by the French ophthalmologist Bertillon in1880. Research in visual identification technology began in 1935. During that year an article appeared in the ’New York State Journal of Medicine’, which suggested that “the pattern of arteries and veins of the retina could be used for unique identification of an individual”. After researching and documenting the potential use of the iris as a tool to identify people, ophthalmologists Flom and Safir patented their idea in 1987; and later, in 1989, they patented algorithms developed with the mathematician Daugman. Thereafter, other authors developed similar approaches. Later in 2001, Daugman also presented a new algorithm for the recognition of people using the biometric measurement of Iris. The literature has well documented the uniqueness of visual identification. The iris is so unique that there are no two irises alike, even twins, in all humanity. The probability of two irises producing the same code is 1 in 1078, becoming known that the earth’s population is estimated at approximately 1010 million, it is almost imposible to ocurr. Biometric identification techniques are very diverse, since any significant element of a person is potentially useful as an element of biometric identification. Even with the diversity of existing techniques, when developing a biometric identification system, this remains a totally independent of the technique. Human beings have many features in common, but also have characteristics that distinguish them and make them unique from each other. Over the years there have been many studies and research about it and developing techniques, methods and systems that enable the use of these patterns for personal identification.

46

Chapter 5 Modular Neural Networks for Person Recognition

5.2.3 Iris Properties The iris is an internal organ of the eye, located behind the cornea and the aqueous humor, which consists of a screening connective tissue, fibers, rings and colors that are a distinctive mark of the people to observe at close range (see Fig. 5.2). The iris texture is not a genetic expression and the morphogenesis is completely random [80].

Fig. 5.2 Human Iris

The properties of the iris that enhance its use for identification of individuals include: a) uniqueness in two individuals, b) inability to modify it without risk of vision loss, c) is a pattern with high randomness, and d) ease of record at close range. But it also presents some disadvantages such as: a) its small size makes it difficult to acquire it at certain distances, b) is a moving target, c) is located on a curved surface, moist and reflective, d), its image is often affected by eyelashes, eyelids and light reflections, and e) the deformations are not elastic when the pupil changes size.

5.3 Proposed Method and Problem Description The proposed methodology for iris recognition can be stated as follows [80]: 1) Search for a database of human iris. 2) Determine the database division in terms of individuals per module for the modular neural network. 3) Search for methods and / or pre-processing techniques for application to the image database and obtain an optimal identification. 4) Implementing in a programming language the different modular neural network architectures (see Fig. 5.3). The RNMs consist of 3 modules. 5) Find a modular integration method to provide the desired results.

5.3 Proposed Method and Problem Description

47

Fig. 5.3 Schematic representation of the division of problem and system operation

5.3.1 Problem Description This work focuses primarily on the identification of individuals. This problem is well known by the scientific community, as innumerable investigations have been developed in this area [8, 40], considering various measures to achieve it with biometrics (fingerprint, voice, palm of hand, signature) and various methods of identification (with particular emphasis on neural networks). The specific problem considered in this work is: “Obtain a good percentage of person identification based on the biometric measurement of the human iris, using modular neural networks”. We used a database of human Iris from the Institute of Automation Chinese Academy of Sciences (CASIA) (see Fig. 5.4). It consists of 14 images (7 right eye - 7 left eye) per person, for a total of 99 individuals, giving a total of 1386 images. The image dimensions are 320 x 280, JPEG format.

Fig. 5.4 Examples of the human iris images from CASIA database

48

Chapter 5 Modular Neural Networks for Person Recognition

5.3.2 Image Pre-processing The pre-processing that has been applied to the images before they are introduced to the neural network is as follows:     

Obtain the coordinates and radius of the iris and pupil using the method developed by Masek and Kovesi. Making the cut in the Iris. Resize the cut of the Iris to 21-21 Convert images from vector to matrix Normalize the images.

1) Obtain coordinates of the center and radius of Iris: To get the coordinates of the center and radius of the iris and pupil of images in the CASIA database, the method developed by Masek and Kovesi was used. This method involves applying a series of filters and mathematical calculations to achieve the desired gain. First we apply edge detection with Canny’s method (see Fig. 5.5 (a)), then the process continues using a gamma adjustment of the image (see Fig 5.5 (b)), to the resulting image obtained above a no maxima suppression is applied (see Fig. 5.5 (c)), and subsequently we applied to the image a threshold method (see Fig. 5.5 (d)).

Fig. 5.5 (a) Edge detection with Canny’s method (b) Image Adjust Gamma (c) No Maxima Suppression (d) Threshold

Finally, we apply the Hough transform to find the maximum in the Hough space and, therefore, the circle parameters (row and column at the center of the iris and the radius).

5.4 Modular Neural Network Architecture

49

2) Cut out the Iris: After obtaining the coordinates of the Iris, the upper right and lower left points are calculated to make the cut (see Fig. 5.6). RowUpLeft = RowIris - RadiusIris; RowLowRight = (RowIris+RadiusIris)-RowUpLeft; ColUpLeft = ColumnIris - RadiusIris; ColLowRight = (ColumnIris + RadiusIris) - ColUpLeft;

Fig. 5.6 Cut of iris, using the “imcrop” Matlab function

5.4 Modular Neural Network Architecture The work was focused on the recognition of persons using a modular neural network. We worked with 3 modules, each module input considers 33 individuals (264 images for training - 198 images for testing). We used the method of integration called Gating Network. The architecture of the modular neural network was defined with an empirical expression. Based on the following expression we can calculate the number of nodes as follows [80]:   

1st hidden layer (2 _ (k + 2)) = 70. 2nd hidden layer (k + m) = 41. Output layer (k) = 33.

In determining the characteristics of the modular neural network, such as the number of modules to be used, the inputs, hidden layers, the number of neurons for each hidden layer and outputs, the initial architecture is specified in Fig. 5.7.

50

Chapter 5 Modular Neural Networks for Person Recognition

Fig. 5.7 Initial Modular Neural Network Architecture

5.5 Simulation Results Experiments were performed with the modular neural network architecture described in the previous section, and we perfomed experiments with 3 types of learning algorithms: gradient descent with adaptive learning (GDA), gradient descent with adaptive learning and momentum (GDX) and scaled conjugate gradient (SCG). After obtaining the results of this architecture, two modified architectures were also tested to improve the accuracy.

5.5.1 Results with the Initial Modular Neural Network Architecture The following results were achieved by each of the 3 modules in terms of percentage of identification. Module 1 In this module the results were better with the scaled conjugate gradient algorithm (SCG), with an identification rate of 96.46% (191/198), goal error of 0.0000001, execution time of 29 seconds and 207 iterations (see Fig. 5.8 and Table 5.1).

5.5 Simulation Results

51

Fig. 5.8 Graph of error of the best training in Module 1 Table 5.1 Results for Module 1

Error

Time

Iterations

Rec.

Ident.

% Ident.

Traingda

0.000001

1.32 m.

1399

264/264

189/198

95.45

Traingdx

0.000001

38 s.

551

264/264

188/198

94.94

Trainscg

0.0000001

29 s.

207

264/264

191/198

96.46

Module 2 In this module the results were better with the gradient descent algorithm with adaptive learning (GDA), with an identification rate of 96.46% (191/198), goal error of 0.000001, execution time of 56 seconds and 833 iterations (see Fig. 5.9 and Table 5.2).

52

Chapter 5 Modular Neural Networks for Person Recognition

Fig. 5.9 Graph of error of the best training in Module 2 Table 5.2 Results for module 2

Error

Time

Iterations

Rec.

Ident.

%Ident.

Traingda

0.000001

56 s.

833

264/264

191/198

96.46

Traingdx

0.000001

40 s.

586

264/264

189/198

95.45

Trainscg

0.0000001

28 s.

207

264/264

189/198

95.45

Module 3 In this module the results were better with the scaled conjugate gradient algorithm (SCG), with a 94.94% of identification rate (188/198), goal error of 0.0000001, execution time of 28 seconds and 144 iterations (see Fig. 5.10 and Table 5.3).

5.5 Simulation Results

53

Fig. 5.10 Graph of error of the best training in Module 3 Table 5.3 Results for Module 3

Error

Time

Iterations

Rec.

Ident.

%Ident.

Traingda

0.000001

66 s.

992

264/264

187/198

94.44

Traingdx

0.000001

29 s.

416

264/264

186/198

93.93

Trainscg

0.0000001

20 s.

144

264/264

188/198

94.94

Integration Analyzing the results obtained in the three modules, we see that in the first and third module, the learning algorithm that showed better results was the scaled conjugate gradient; and in the second module the best results were obtained with the gradient descent with learning adaptive. We decided to use the gating network integration method with the outputs of the three modules. Getting as final result a 95.95% of identification rate (570/594) (see Table 5.4).

54

Chapter 5 Modular Neural Networks for Person Recognition Table 5.4 Results of the integration for the 3 modules

Integrator

MD1

MD2

MD3

Rec.

Ident.

Gating Network Trainscg Traingda Trainscg 792/792 570/594

% Ident. 95.95

5.5.2 First Modification of the Modular Neural Network Architecture In obtaining the results with the proposed modular neural network, we decided to modify manually the number of neurons in a hidden layer of the network, considering the second hidden layer for the modification. There were two changes in the second hidden layer that produced good results: 70 neurons in first hidden layer and 67 neurons in the second hidden layer, and 70 neurons in first hidden layer and 100 neurons in the second hidden layer. Module 1 The results of this module found that the best result was achieved with gradient descent algorithm with adaptive learning (GDA) with 70 hidden neurons in first layer and 67 neurons in second hidden layer with a 96.96% of identification rate (192/198) (see Table 5.5). Table 5.5 Results for Module 1

70-41

70-67

70-100

Identif.

% Identif.

Identif.

% Identif.

Identif.

% Identif.

Traingda

189/198

95.45

192/198

96.96

190/198

95.95

Traingdx

188/198

94.94

189/198

95.45

191/198

96.46

Trainscg

191/198

96.46

189/198

95.45

190/198

95.95

Module 2 The results of this module found that the best result was achieved with gradient descent algorithm with adaptive learning (GDA) with 70 hidden neurons in first layer and 67 neurons in second hidden layer with a 97.97% of identification rate (194/198) (see Table 5.6).

5.5 Simulation Results

55

Table 5.6 Results for module 2

70-41 Identif.

70-67

70-100

% Identif.

Identif.

% Identif.

Identif.

% Identif.

Traingda 191/198

96.46

194/198

97.97

193/198

97.47

Traingdx 189/198

95.45

191/198

96.46

192/198

96.96

Trainscg

95.45

192/198

96.96

193/198

97.47

189/198

Module 3 The results of this module found that the best result was achieved with the scaled conjugate gradient algorithm (SCG) with 70 neurons in first hidden layer and 100 neurons in second hidden layer with a 95.45% of identification rate (189/198) (see Table 5.7). Table 5.7 Results for module 3 70-41

70-67

70-100

Identif.

% Identif.

Identif.

% Identif.

Identif.

% Identif.

Traingda

187/198

94.44

187/198

94.44

188/198

94.94

Traingdx

186/198

93.93

187/198

94.44

188/198

94.94

Trainscg

188/198

94.94

188/198

94.94

189/198

95.45

Knowing the best results for the 3 modules, it is now determined that the modular neural network architecture is modified as shown in Fig. 5.11:

56

Chapter 5 Modular Neural Networks for Person Recognition

Fig. 5.11 Modified Modular Neural Network Architecture

Integration Analyzing the results obtained in the three modules, we see that in the first and second module, the learning algorithm that showed better results was the gradient descent with learning adaptive rate with 70 hidden neurons in first layer and 67 neurons in second hidden layer; and in the third module which showed best results was the scaled conjugate gradient with 7 hidden neurons in first layer and 10 neurons in second hidden layer. The gating network integration is done with the algorithms and numbers of neurons mentioned above for each module. The final result is 96.80% of identification rate (575/594) (see Table 5.8). Table 5.8 Results of the integration for 3 modules with modified architecture

Integrator

MD1

MD2

MD3

Rec.

Ident.

% Ident.

Gating Network Trainscg Traingda Trainscg 792/792 570/594

95.95

Gating Network Traingda Traingda Trainscg 792/792 575/594 Modified Architecture

96.80

Cross Validation 10 Times After obtaining the final results of the integration we chose to carry out crossvalidation of 10 different combinations of training and test images, and averaging the results of identifications (see Table 5.9):

5.5 Simulation Results

57

Table 5.9 Results for cross validation for 10 times

Validator

MD1

MD2

MD3

% Ident.

Cross Validation 91.57 10 Times

92.93

90.25

91.58

5.5.3 Second Modification of the Modular Neural Network Architecture (Extended) In analyzing the results obtained with the integration in the modular neural network we found that the learning algorithm (SCG) identifies the image of human iris of a particular person that other different learning algorithm (GDA) do not identified, as well as the number of neurons in the hidden layers of the neural network. Based on the above ideas, we propose an extended architecture of the modular neural network. In this architecture each module consists of two modules with different learning algorithms or different number of neurons in the second hidden layer (the two modules were more successful results with the modified architecture), see in Fig 5.12:

Fig. 5.12 Extended Modular Neural Network Architecture

58

Chapter 5 Modular Neural Networks for Person Recognition

With the outputs of each module we perform the results integration. Obtaining as final result a 97.13% identification rate (577 images of 594 test images) with the winner takes all integrator and 96.96% identification rate (576 images of 594 test images) with the Mamdani type fuzzy integrator (see Table 5.10). Table 5.10 Results of the integration for 3 modules with extended architecture

Integrator

MD1

MD2

MD3

Rec.

Ident.

Gating Network Traingda Traingda Trainscg 792/792 575/594 192 194 189 Modified Architecture 194 193 190 792/792 577/594 Integrator: Winner Takes All Integrator: Fuzzy 193 194 189 792/792 576/594 Type Mamdani

% Ident. 96.80

97.13 96.96

Cross Validation 10 Times After obtaining the final results of the integration we chose to carry out crossvalidation of 10 different combinations of training and test images, and averaging the results of identifications (see Table 5.11). Table 5.11 Results for cross validation for 10 times with extended architecture

Cross Validation 10 Times Modified Architecture Extended Architecture with Integrator Winner Takes All Extended Architecture with Integrator Fuzzy Type Mamdani

% Ident. 91.58 91.83 91.41

5.6 Summary In this chapter we presented modular neural network architectures, which have as input the database of human iris images, with the database images divided in three parts for the three modules, and each module has two hidden layers. In this work, several methods were used to make the elimination of noise that the original pictures had until the coordinates of the center and radius were obtained, and then make a cut around the iris.

5.6 Summary

59

With the extended architecture we achieved higher results than the initial and modified by achieving a 97.13% identification rate (577 images of 594 test images) with the integrator winner takes all and 96.96% identification rate( 576 images of 594 test images) with the Mamdani type fuzzy integrator. These results demonstrate that the use of the human iris biometric measurement worked with modular artificial neural networks and favorable results of person identification were obtained. Future work consists in considering other different pre-processing of images or that complements the current one and using a method for optimization of the modular artificial neural network, with which we could get beyond the rate of detection obtained in the results shown in this chapter.

Chapter 6

Modular Neural Networks for Human Recognition from Ear Images Compressed Using Wavelets

This chapter is focused in the human recognition from ear images as biometric using modular neural networks with preprocessing ear images as network inputs [80]. We proposed modular neural network architecture composed of twelve modules, in order to simplify the problem making it smaller. Comparing with other biometrics, ear recognition has one of the best performances, even when it has not received much attention [9, 74]. To compare with other existing methods, we used the 2D Wavelet analysis with global thresholding method for compression, and Sugeno Measures and Winner-Takes-All as modular neural network integrator. Recognition results achieved was up to 97%.

6.1 Introduction Is not a surprise to see that actually the human identifying methods by possessions such a cards, badges, keys or by knowledge such a passwords, userid, Personal Identification Number (PIN), are being replaced by biometrics [59]. We can say that every person is unique. There are not two persons with the same face, identical fingerprints or voice [32]. This and other singularities are part of the biometrics. Biometrics is the science of identifying or verifying the identity of a person based on physiological or behavioral characteristics. In physiological also called passive characteristics, the biometric measure is taken from a subject with or without his/her consent, depends of the biometric capture system. In the other way the behavioral also called active characteristics, the subject must show his/her biometric measure doing an action in front of a captured system [93]. Biometrics offer much higher accuracy than the more traditional ones. Possession can be lost, forgot or replicated easily. Knowledge can be forgotten. Both possessions and knowledge can be stolen or shared with other people. In biometrics these drawbacks do exist only in small scale [59].

P. Melin: Modular Neural Networks and Type-2 Fuzzy Systems, SCI 389, pp. 61–75. springerlink.com © Springer-Verlag Berlin Heidelberg 2012

62

Chapter 6 Modular Neural Networks for Human Recognition

Some biometrics are shown at (Figure 6.1). The behavioral characteristics are voice, handwritten signature, keyboard strokes, odor, and more. The physiological ones are fingerprint, iris, face, hand geometry, finger geometry, retina, vein structure, ear, and more. The physiological characteristics systems are generally more reliable than the based ones on behavioral characteristics, despite the last ones can be sometimes simpler to integrate in some specific uses [59].

Fig. 6.1 Sample Biometrics

The most commonly used biometrics according to the International Biometric Group in 2006 were: fingerprint, face, voice, iris, handwritten signature and hand geometry [8, 9, 32, 40, 74]. The Ear biometric measure is not commonly used, yet (Figure 6.2). There are many applications where biometrics can be used [59]. Basically a biometric system may operate in verification mode also known as authentication, or identification mode also known as recognition. Identification mode answer the question Who are you?, the system recognizes a subject by searching the templates of all the users in the database for a match. In verification mode the question is, You are who you claim to be? and the system validates a person’s identity by comparing the captured biometric data with her own biometric. But, what biological measurements qualify to be a biometric measure? Any human physiological and/or behavioral characteristic can be used as a biometric characteristic as long as it satisfies the following requirements: universal, unique,

6.1 Introduction

63

Fig. 6.2 Commonly used Biometrics

permanent, measurable, acceptable, capable and reliable. These concepts are described below: • • • • • • •

Universality (U): each person must own the characteristic. Distinctiveness (D): each person must be differentiable between each subject. Permanence (P): the characteristic does not have to change with time. Collectability (Co): there must exist the capacity to characterize the characteristic measured quantitatively. Acceptability (A): the characteristic must have great acceptance between the societies. Performance (Pf): accuracy, speed, and robustness of technology used. Circumvention (C): reflects how easily the system can be fooled using fraudulent methods.

A brief comparison of different biometrics based on seven factors for an ideal biometric previously mentioned, is provided in (Table 6.1). As you can see, ear, fingerprint and hand geometry have better averages than other biometrics [59]. However, no single technique can outperform all the others in all operational environments. In this sense, each biometric technique is admissible and there is no optimal biometric characteristic.

64

Chapter 6 Modular Neural Networks for Human Recognition

Table 6.1 Comparison for different biometrics. High, Medium, and Low are denoted by H, M, and L, respectively.

Biometric Techniques Ear DNA Face Fingerprint Keystroke Hand G. Iris Odor Retina Signature Voice

U

D

P

Co

A

Pf

C

M H H M L M H H H L M

M H L H L M H H H L L

H H M H L M H H M L L

M L H M M H M L L H M

M H L H L M H L H L L

H L H M M M L M L H H

M L H M M M L L L H H

6.2 Background 6.2.1 Related Work Human ear identification has been interesting in recent years [9, 80]. At 1906 Imhofer found that in a set of 500 ears, only 4 characteristics were needed to state the ears unique. The most famous work among ear identification was made by Alfred Iannarelli at 1989, where he compared over 10.000 ears drawn from a randomly selected sample in California; he found that all ears were different [59]. The following account summarizes previous work on ear [59]. Another study was among identical and non-identical twins using Iannarelli’s measurements. The result was that ears are not identical, even identical twins had similar but not identical ears. After Iannarelli’s classification there have become different and more scientific methods for ear identification: Carreira-Perpiñan in 1995 used outer ear images are proposed for human recognition and compression neural networks to reduce the dimensionality of the images and produce a feature vector of manageable size. Moreno et al. in 1999 presented a multiple identification method, which combines the results from several neural classifiers using feature outer ear points, information obtained from ear shape and wrinkles, and macro features extracted by compression network. Burge and Burger (1998 - 2000) obtained automatic ear biometrics with Voronoi diagram of its curve segments. Hurley, Nixon and Carter (2000) used force field transformations for ear recognition. The image is treated as an array of Gaussian attractors that act as the source of the force field.

6.2 Background

65

Victor, Chang, Bowyer and Sarkar (2002 - 2003) used a Principal Component Analysis (PCA) approach, and made a comparison between ears and faces. Ear Recognition Laboratory at USTB in 2003, they established an image database of 60 subjects (3 images for one subject). Using the method of kernel principal component analysis (KPCA), the ear identification rate was 94%. In 2004, they enlarged the image database to 77 subjects, 4 images for one subject, with pose variation and lighting variation. For ear feature extraction, they proposed a novel recognition method based on local features extraction, which means extracting the shape feature of outer ear and the structural feature of inner ear then using BP neural network for classification. In this way, the recognition rate rises to 85%.

6.2.2 Ear The ear structure is quite complex, it has a great variety of classified zones [80]. The most important ear parts are: Helix, Shell (Concha) and Lobule (Figure 6.3).

Fig. 6.3 Ear parts

Researchers have suggested that the shape and appearance of the human ear is unique to each person, and relatively unchanging during the lifetime of an adult making it better suited for long-term identification when compared to other biometrics, such as face recognition. Ear recognition is not affected by environmental factors such as mood, health, and clothing. However, ear recognition has not received much attention like others biometrics e.g. face, fingerprint, iris, etc. For that reasons, the ear is taken as biometric, since it presents a structural singularity that is practically impossible that two subjects own the same physiological characteristic, even for “identical” twins.

66

Chapter 6 Modular Neural Networks for Human Recognition

We can mention some advantages and disadvantages for ear biometric: • • • • • •

Ears are smaller than other biometrics. Reduced spatial resolution. Ear biometric can be take with or without the subject consent. Biometrics like iris, retina and DNA are more permanent than ear form. At the same level are fingerprint and hand geometry. Less permanent than ear form are signature, face and voice. Ear biometrics are not usable if the ear is covered e.g. with a hat or hair. We have almost none adjectives to describe ears. Ear recognition systems can be fooled with methods like plastic surgery.

6.3 Ear Recognition Process Ear recognition process used for this work is composed by the following steps [80]: • • • •

•

Data acquisition Image pre-processing o Regions of interest (ROI) o Wavelets Neural network structure Neural network training o Scaled Conjugate Gradient (SCG) o Gradient Descent with Momentum and Adaptive Learning Rate (GDX) Modular integration o Sugeno Measures o Winner-Takes-All (WTA)

In the next sections we describe these steps, and we show the results obtained.

6.3.1 Data Acquisition The images were acquired at University of Science and Technology Beijing (USTB), which contains 308 images from 77 subjects, 4 images for one subject with pose variation and lighting variation. The subjects are students and teachers from the department of Information Engineering. The database was formed between November 2003 and January 2005 with 77 subjects, four images each. Two images with angle variation and one with illumination variation. Each image is 24-bit true color image and 300*400 pixels. The first image and the fourth one are both profile image but under different lighting. The second and the third one

6.3 Ear Recognition Process

67

have the same illumination condition with the first while they have separately rotated +30 degree and -30 degree with the first one (Figure 6.4).

Fig. 6.4 USTB Database

6.3.2 Image Pre-processing The original image is true color or RGB. To get the coefficients, the image must to have a color map, it means, the image must be indexed. First of all, we convert the image to grayscale and then from RGB to indexed. The figures (Figure 6.5 and Figure 6.6) below show how the system interprets the original RGB image and how interprets the indexed image. As we can see, the RGB image lost too much information.

Fig. 6.5 RGB Image

68

Chapter 6 Modular Neural Networks for Human Recognition

Fig. 6.6 Indexed Image

The next step for image pre-processing was the image resize from 300*400 to 50*75 pixels taken the region of interest (ROI) to eliminate as much as possible noise (Figure 6.7).

(a)

(b)

Fig. 6.7 (a) 0riginal Image. (b) Resize Image Black square = ROI

6.3 Ear Recognition Process

69

For image compression, we used two-dimensional wavelet analysis with Global Thresholding method, with two decompose levels and near symmetric wavelet (sym8) and 20% for hard-thresholding. The approximation coefficients were stored in a row vector for training. Basically, this method consists of taking the wavelet expansion of the signal and keeping the largest absolute value coefficients. In this case, you can set a global threshold, a compression performance, or a relative square norm recovery performance. Thus, only a single parameter needs to be selected. The following figures (Figure 6.8 and Figure 6.9) show the decomposition and compression process.

Fig. 6.8 Image Decomposition

70

Chapter 6 Modular Neural Networks for Human Recognition

Fig. 6.9 Image Compression

6.3.3 Neural Network Structure The use of monolithic neural networks, such as a multilayer perceptrón, has some drawbacks: e.g. slow learning, weight coupling, and the black box effect [59]. These can be alleviated by the use of a modular neural network (MNN). The creation of a MNN has three steps: task decomposition, module creation and decision integration. For our investigation, we decomposed our network data by this way: 308 training images, divided in 104 images for module 1 and 2, and 100 images for module 3; 77 identification images, divided in 26 images for module 1 and 2, and 25 for module 3 (Figure 6.10). Then, each module was divided as follows (Figure 6.11): •

Module 1 (Subjects 1-26) o Module 4: Helix o Module 5: Concha o Module 6: Lobule • Module 2 (Subjects 27-52) o Module 7: Helix o Module 8: Concha o Module 9: Lobule • Module 3 (Subjects 53-77) o Module 10: Helix o Module 11: Concha o Module 12: Lobule

6.3 Ear Recognition Process

71

Using the expression (2*(k+2), k+m, k): • • •

First hidden layer: 56 neurons module 1 and 2, 54 module 3. Second hidden layer: 30 neurons module 1 and 2, 29 module 3. Output layer: 26 neurons module 1 and 2, 25 module 3.

We used Sugeno measures and Winner-Takes-All for integrate the modules 4, 5 and 6, 7, 8 and 9, and 10, 11 and 12. For modules 1, 2 and 3 we used a Gating Network to take de final decision.

Data Base

Module1 Subject 1-26

Image Preprocessing

Module2 Subject 27- 52

Gating

Module3 Subject 53-77

Fig. 6.10 Schematic representation of the neural network architecture

72

Chapter 6 Modular Neural Networks for Human Recognition 1

1

2

2

56

30

1

1

2

2

56

30

1

1

2

2

56

30

Helix

Concha

Integration

Lobule

Fig. 6.11 Neural network architecture for each module

6.3.4 Neural Network Training For the 77 subjects, we trained the 4 available images and we used every image with cross validation for testing. It means, for each module we used the first image for test at the first iteration, then the second image for test at the second iteration, and so on until the four image. The learning algorithms used for this work were gradient descent momentum and an adaptive learning rate or traingdx, and scaled conjugate gradient or trainscg. The tables below (Table 6.2, Table 6.3 and Table 6.4) show the performance for those learning algorithms, being trainscg the best with less number of iterations but not in time. Table 6.2 Performance for trainscg algorithm. H=Helix, C=Concha, L=Lobule.

Module 1 H C L 377 509 608

Iterations Module 2 H C L 584 669 948

H 444

Module 3 C L 411 400

Table 6.3 Performance for traingdx algorithm. H=Helix, C=Concha, L=Lobule.

Module 1 H C L 1000

Iterations Module 2 H C L 1000

Module 3 H C L

6.3 Ear Recognition Process

73

Table 6.4 Time comparison between SCG and GDX. SCG= trainscg, GDX= traingdx.

Train SCG GDX

Goal 0.00001 0.00001

Time 30.37 24.20

6.3.5 Modular Integration The integrators used in this work were Sugeno Measures and WTA mechanisms. The following tables (Table 6.5, Table 6.6 and Table 6.7) show the recognition results obtained for each module using cross validation and Sugeno Measures as integrator. Table 6.5 Recognition results for Module 1 with Sugeno integration. CV = Cross Validation

Train

Module 1 CV1

CV2

CV3

CV4

%

trainscg

25/26

26/26

24/26

26/26

97.11

traingdx

23/26

23/26

24/26

22/26

88.46

Table 6.6 Recognition results for Module 2 with Sugeno integration. CV = Cross Validation

Train trainscg traingdx

CV1 24/26 22/26

CV2 25/26 23/26

Module 2 CV3 25/26 24/26

CV4 26/26 23/26

% 96.15 88.46

Table 6.7 Recognition results for Module 3 with Sugeno integration. CV = Cross Validation

Train

Module 3 CV1

CV2

CV3

CV4

%

trainscg

24/25

25/25

25/25

25/25

99.0

traingdx

23/25

22/25

21/25

22/25

88.0

74

Chapter 6 Modular Neural Networks for Human Recognition

The tables (Table 6.8, Table 6.9, and Table 6.10) show the recognition results obtained for each module using cross validation and Winner-Take-All as integrator.

Table 6.8 Recognition results for Module 1 with WTA integration. CV = Cross Validation

Train

Module 1 CV1

CV2

CV3

CV4

%

trainscg

23/26

25/26

26/26

25/26

95.19

traingdx

24/26

23/26

25/26

24/26

92.30

Table 6.9 Recognition results for Module 2 with WTA integration. CV = Cross Validation

Train

Module 2 CV1

CV2

CV3

CV4

%

trainscg

24/26

23/26

24/26

26/26

93.26

traingdx

26/26

23/26

25/26

24/26

94.23

Table 6.10 Recognition results for Module 3 with WTA integration. CV = Cross Validation

Train

Module 3 CV1

CV2

CV3

CV4

%

trainscg

24/25

23/25

25/25

23/25

95.00

traingdx

24/25

23/25

24/25

23/25

94.00

The table below (Table 6.11) shows the final average obtained from each module. Table 6.11 Final recognition result for Sugeno and WTA integration

Train trainscg

Sugeno Rec.% 97.42

WTA Rec.% 94.48

traingdx

88.30

93.51

6.4 Summary

75

We can compare our results with the work done by other researchers in the same field using various techniques [9]. Our recognition rate is good because the modular neural network architecture that we used helped us to avoid slow learning and make easier the recognition process.

6.4 Summary In this chapter we proposed a method of human recognition based on human ear images using 2D wavelet analysis for preprocessing. Ear images are resized to a fixed size followed by select regions of interest. After that near symmetric wavelet of level two is used to image compression. Ear database is trained following the proposed modular neural network architecture, which help us to improve the train process. In fact, if we want to add more subjects to the database is not necessary start over, we just need to add more modules to our architecture and that is a great advantage comparing with other similar works. Sugeno Measures integrator obtains the best recognition average being this 97.11% with SCG algorithm versus 94.48% with WTA integrator from SCG algorithm. Both integrators are good; however, in this case it was not enough to choose a winner through the neural weights as do WTA integrator.

Chapter 7

Signature Recognition with a Hybrid Approach Combining Modular Neural Networks and Fuzzy Logic for Response Integration

This chapter describes a modular neural network (MNN) with fuzzy integration for the problem of signature recognition. Currently, biometric identification has gained a great deal of research interest within the pattern recognition community [59]. For instance, many attempts have been made in order to automate the process of identifying a person’s handwritten signature; however this problem has proven to be a very difficult task. In this work, we propose a MNN that has three separate modules, each using different image features as input, these are: edges, wavelet coefficients, and the Hough transform matrix. Then, the outputs from each of these modules are combined using a Sugeno fuzzy integral and a fuzzy inference system [65]. The experimental results obtained using a database of 30 individual’s shows that the modular architecture can achieve a very high 99.33% recognition accuracy with a test set of 150 images. Therefore, we conclude that the proposed architecture provides a suitable platform to build a signature recognition system. Furthermore we consider the verification of signatures as false acceptance, false rejection and error recognition of the MNN.

7.1 Introduction Recently, there has been an increased interest in developing biometric recognition systems for security and identity verification purposes [8, 9, 32, 59, 64, 65, 93, 107]. Such systems usually are intended to recognize different types of human traits, which include a person’s face, their voice, fingerprints, and specific handwriting traits [40, 52, 74, 80]. Particularly, the handwritten signature that each person posses is widely used for personal identification and has a rich social tradition [59]. In fact, currently it is almost always necessary in all types of transactions that involve legal or financial documents. However, it is not a trivial task for a computational system to automatically recognize a person’s signature for the following reasons. First, there can be a great deal of variability when a person signs a document. This can be caused by different factors, such as a person’s mood, free time to write the signature, and the level of concentration during the actual act of signing a document. Second, because signatures can be so diverse it is not evident which type of features should be used P. Melin: Modular Neural Networks and Type-2 Fuzzy Systems, SCI 389, pp. 77–92. springerlink.com © Springer-Verlag Berlin Heidelberg 2012

78

Chapter 7 Signature Recognition with a Hybrid Approach Combining MNNs

in order to describe and effectively differentiate among them. For instance, some signatures are mostly written using straight line segments, and still others have a much smoother form with curved and circular lines. Finally, many signatures share common traits that make them appear quite similar depending on the types of features that are analyzed. In this work, we present a handwritten signature recognition system using Modular Neural Networks (MNNs) with the Sugeno fuzzy integral. We have chosen a MNN because they have proven to be a powerful, robust, and flexible tool, useful in many pattern recognition problems. In fact, we only extract simple and easily computed image features during our preprocessing stage, these features are: image edges, wavelet transform coefficients, and the Hough transform matrix. The MNN we propose uses these features to perform a very accurate discrimination of the input data used in our experimental tests. Therefore, we have confirmed that a MNN system can solve a difficult biometric recognition problem using a simple set of image features.

7.2 Problem Statement and Outline of Our Proposal The problem we address in this chapter is concerned with the automatic recognition of a person’s signature that is captured on a Tablet PC. We suppose that we have a set of N different people, and each has a unique personal signature. The system is trained using several samples from each person, and during testing it must determine the correct label for a previously unknown sample. The system we are proposing consists on a MNN with three separate modules [59]. Each module is given as input the features extracted with different feature extraction methods: edge detection, wavelet transform, and Hough transform. The responses from each of the modules are combined using a Sugeno fuzzy integral [65], which determines the person to whom the input signature corresponds. A general schematic of this architecture is shown in Figure 7.1, where all of the modules and stages are clearly shown.

Fig. 7.1 General architecture of the proposed Modular Neural Network for signature recognition

7.3 Background Theory

79

In the following section we present a brief review of some of the main concepts needed to understand our work.

7.3 Background Theory In this section we provide a general review of artificial neural networks and modular architectures, we discuss how the output from the modular system can be integrated using Sugeno fuzzy integrals, and we describe the feature extraction methods that provide the input for each of the modules in our MNN.

7.3.1

Modular Neural Networks

Artificial Neural Networks (ANNs) are information processing systems that employ a conceptual model that is based on the basic functional properties of biological neural networks [57, 59]. In the past twenty or thirty years, ANN research has grow very rapidly, in the development of new theories of how these systems work, in the design of more complex and intricate models, and in their application to a diverse set problem domains. Regarding the latter, application domains for ANN include pattern recognition, data mining, time series prediction, robot control, and in the development of hybrid methods with fuzzy logic and genetic algorithms, to mention but a few examples [48, 49, 58, 60, 80, 93]. In canonical implementations, most systems employ a monolithic network in order to solve the given task. However, when a system needs to process large amounts of data or when the problem is highly complex, then it is not trivial, and sometimes unfeasible, to establish a good architecture and topology for a single network that can solve the problem. For instance, in such problems a researcher might attempt to use a very large and complex ANN. Nevertheless, large networks are often difficult to train, and for this reason they rarely achieve the desired performance. In order to overcome some of the aforementioned shortcomings of monolithic ANNs, many researchers have proposed modular approaches [32, 59, 80, 93]. MNNs are based on the general principle of divide-and-conquer, where one attempts to divide a large problem into smaller sub-problems that are easier to solve independently. Then, these partial solutions are combined in order to obtain the complete solution for the original problem. MNNs employ a parallel combination of several ANNs, and normally contain two main components: (1) local experts; and (2) an integrating unit. The basic architecture is shown in Figure 7.2. Each module consists of a single ANN, and each is considered to be an expert in a specific task. After the input is given to each module it is necessary to combine all of the outputs in some way, this task is carried out by a special module called an integrator. The simplest form of integration is given by a gating network, which basically switches between the outputs of the different modules based on simple criteria, such as the maximum level of activation. However, a better combination of the responses from each module can be obtained using more elaborate methods of integration, such as the Sugeno fuzzy integral.

80

Chapter 7 Signature Recognition with a Hybrid Approach Combining MNNs

Fig. 7.2 Architecture of a Modular Network

7.3.2 Sugeno Fuzzy Integral The Sugeno fuzzy integral is a nonlinear aggregation operator that can combine different sources of information [12, 59, 65]. The intuitive idea behind this operator is based on how humans integrate information during a decision making process. In such scenarios it is necessary to evaluate different attributes, and to assign priorities based on partially subjective criteria. In order to replicate this process on an automatic system, a good model can be obtained by using a fuzzy representation. Finally, several works have shown that the use of a Sugeno fuzzy integral as a MNN integrator can produce a very high level of performance, and for these reasons we have chosen it for the system we describe here.

7.3.3 Fuzzy Systems Fuzzy theory was initiated by Lotfi A. Zadeh in 1965 with his seminal paper “Fuzzy sets”. Before working on fuzzy theory, Zadeh was a well-respected scholar in control theory [103]. A big event in the 70’s was the birth of fuzzy controllers for real systems. In 1975, Mamdani and Assilian established the basic framework of fuzzy controller and applied the fuzzy controller to control a steam engine. Their results were published in another seminal paper in fuzzy theory “An experiment in linguistic synthesis with a fuzzy logic controller”. They found that the fuzzy controller was very easy to construct and worked remarkably well [37]. The fuzzy inference system is a popular computing framework based on the concepts of fuzzy set theory, fuzzy if- then rules, and fuzzy reasoning. It has found successful applications in a wide variety of field, such as automatic control, data classification, decision analysis, experts systems, times series prediction, robotics, and patter recognition [1, 6, 12, 20, 29, 83, 84].

7.3 Background Theory

81

The basic structure of a fuzzy inference system consists of three conceptual components: a rule base, which contains a selection of fuzzy rules; a database, which defines the membership functions used in the fuzzy rules; and a reasoning mechanism, which performs the inference procedure upon the rules and given facts to derive a reasonable output or conclusion.

7.3.4 Feature Extraction In this work, we employ three individual modules, and each receives different image features extracted from the original image of a person’s signature. Each of these feature extraction methods are briefly described next. 7.3.4.1

Edge detection

For images of handwritten signatures, edges can capture much of the overall structure present within, because people normally write using a single color on a white background. Hence, we have chosen to apply the Canny edge detector [64] to each image that generates a binary image of edge pixels, see Fig 7.3.

(a)

(b)

Fig. 7.3 (a) Original image of a signature. (b) Image edges.

7.3.4.2 Wavelet Transform The wavelet transform decomposes a signal using a family of orthogonal functions, it accounts for both the frequency and the spatial location at each point. The most common application is the Discrete Wavelet Transform (DWT) using a Haar wavelet. The DWT produces a matrix of wavelet coefficients that allows us to compress, and if needed reconstruct, the original image [59]. In Figure 7.4 we can observe the two compression levels used in our work.

82

Chapter 7 Signature Recognition with a Hybrid Approach Combining MNNs

(a)

(b)

(c)

Fig. 7.4 (a) Original image of a signature. (b) First level of decomposition. (c) Second level of decomposition.

7.3.4.3 Hough Transform In the third and final module we employ the Hough transform matrix as our image features. The Hough transform can extract line segments from the image. In Figure 7.5 we show a sample image of a signature and its corresponding Hough transform matrix. Finally, in order to reduce the size of the matrix, and the size of the corresponding ANN, we compress the information of the Hough matrix by 25%.

ρ

-100 0 100

(a)

-50

(b)

0

50

θ

Fig. 7.5 (a) Sample of a signature image with some of the lines found by the Hough transform (b) The Hough transform matrix

7.3 Background Theory

83

7.3.4.4 Verification of Signatures Currently, security practice always involves PIN number, password, and access card. However, these signs are not very reliable, since it can be forgotten or lost. Automatic signature verification is one of the most practical ways to verify human´s identify. Signature verification can be used in many applications such as security, access control, or financial and contractual matters The process of signature verification often consists of a learning stage and a testing stage, as shown in figure 7.6. In the learning stage, the verification system uses the feature extracted from one or several training samples to build a reference signature database. In the testing stage the user inputs the signature into input device. Then the system uses this information to extract the reference in the database, and compares the features extracted from the input signature with the reference. Finally the verification process out whether the test signature is genuine or not.

Input Signature

Data Capture Pre-Procesamiento

Feature Extract MNN Fuzzy integrator

Reference Database Verification Result Fig. 7.6 Signature verification process

84

Chapter 7 Signature Recognition with a Hybrid Approach Combining MNNs

In the research area of signature verification, a type I error rate and type II error rate are usually called false reject rate (FRR) and false acceptance rate (FAR) respectively [59]. To minimize the type II errors, which represent the acceptance of the counterfeited signatures will normally increase the type I errors, which are the rejections of genuine signature. In most case, type II error rate is considered to be more important, but it is not a must. This will depend on the purpose, design, characteristics and application of the verification systems. If the system requests a high security, false accept rate should reduced to its lowest; if the security is not so strict, the system can be adjust to its lowest average false rate. The fuzzy system will answer the greater activation of the 3 modules signing, taking the form of higher activation winner; this means the 27 rules in the system are considered fuzzy. Once the winner module did a signature verification process to know whether the signature that shows fuzzy integrator corresponds to the person. For this we also conducted 30 trainings with 150 different samples of genuine signatures, to make activation and get an average of this activation. The average of the activations is used as to whether a signature is forged or genuine. Typically when the signature is authentic we obtain a high activation and when the signature is false the activation is low, although not necessarily, as may happen if there is a high activation but the signature is false activation or a low but firm is true, so we take into account four different cases: 1. False acceptance (FRA). 2. False rejection (FRR). 3. Error. 4. Signature Authentic. After taking as reference the average of activations, 85 samples were collected from forged signatures of 17 persons and 65 authentic samples of 13 persons, giving a total of 150 samples between false and authentic signatures of 30 persons. In total 210 images of signatures of each module for the training signatures are authentic. Table 7.1 shows the case that can be given upon verification of signatures, taken as a basis the average activations. Table 7.1 Signature verification procedure Recognizes Yes Yes No No No No

Overcome threshold Yes No Yes No No Yes

Original signature No Yes Yes Yes No No

Result False Acceptance False Rejection Error Error Correct Correct

7.4 Experiments

85

7.4 Experiments In this section we present our database of signature images, describe our experimental set-up, and detail the experimental results we have obtained using monolithic and modular networks.

7.4.1 Image Database For this work we build a database of images with the signatures of 30 different people, students and professors from the computer science department at the Tijuana Institute of Technology, BC, México. We collected 12 samples of the signature from each person; this gives a total of 360 images in total. Sample images from the database are shown in Figure 7.7.

Fig. 7.7 Images from our database of signatures. Each row shows different samples from the signature of the same person.

7.4.2 Experimental Setup In this work, we are interested in verifying the performance of our proposed MNN for the problem of signature recognition. Therefore, in order to obtain comparative measures we divide our experiments into four separate tests. 1.

2. 3. 4.

First, we use each module as a monolithic ANN for signature recognition. Therefore, we obtain three sets of results, one for each module, where in each case a different feature extraction method is used. Second, we train our MNN using all three modules concurrently and the Sugeno fuzzy integral as our integration method. Third, we train our MNN using all three modules concurrently and the Fuzzy System as our integration method. Fourth, Signature verification: false acceptance, false rejection.

86

Chapter 7 Signature Recognition with a Hybrid Approach Combining MNNs

In all tests 210 images were chosen randomly and used for training, and the remaining 150 were used as a testing set. Additionally, after some preliminary runs it was determined that the best performance was achieved when the ANNs were trained with the Scaled Conjugate Gradient (Trainscg) algorithm, with a goal error of 0.001. Moreover, all networks had the same basic ANN architecture, with two hidden layers. In what follows, we present a detailed account of each of these experimental tests. 7.4.2.1 Monolithic ANNs The results for the first monolithic ANN are summarized in Table 7.2. The table shows a corresponding ID number for each training case, the total epochs required to achieve the goal error, the neurons in each hidden layer, and the total time required for training. Recognition performance is shown with the number of correct recognitions obtained with the 150 testing images, and the corresponding accuracy score. In this case, the best performance was achieved in the third training run where the algorithm required 78 epochs, and the ANN correctly classified 131 of the testing images. Table 7.2 Performance for a monolithic ANN using edge features; bold indicates best performance No

Epochs

Neurons

Time

Correct

Accuracy (%)

01

80

100-100

00:01:11

123/150

82

02

59

100-100

00:01:18

120/150

80

03

78

100-100

00:01:07

131/150

87

04

90

100-100

00:01:26

117/150

78

05

80

100-100

00:01:08

119/150

79

06

78

100-100

00:01:34

123/150

82

07

53

100-100

00:00:46

123/150

82

08

79

80-90

00:01:08

123/150

82

09

55

80-90

00:00:56

128/150

85

10

58

80-90

00:00:58

122/150

81

The second monolithic ANN uses the wavelet features as input, and the obtained results are summarized in Table 7.3. In this case the best performance was obtained in the third training run, with a total of 5 epochs, and 144 correctly classified images. It is obvious that wavelet features provide a very good discriminative description of the signature images we are testing.

7.4 Experiments

87

Table 7.3. Performance for a monolithic ANN using wavelet features Train

Epochs

Neurons

Time

Correct

Accuracy (%)

01

12

100-100

00:00:18

135/150

90

02

30

100-100

00:00:25

138/150

92

03

05

100-100

00:00:08

144/150

96

04

09

100-100

00:00:11

140/150

93

05

06

80-90

00:00:08

142/150

95

06

05

80-90

00:00:05

140/150

93

07

10

80-90

00:00:14

141/150

94

08

07

80-90

00:00:09

138/150

92

09

10

80-90

00:00:15

140/150

93

10

05

80-90

00:00:06

137/150

91

Finally, the third monolithic ANN uses the Hough transform matrix, and the corresponding results are shown in Table 7.4. The best performance is achieved in the fourth training run, with a total of 6 epochs and 141 correctly classified images. Table 7.4 Performance for a monolithic ANN using the Hough transform Train

Epochs

Neurons

Time

Correct

Accuracy (%)

01

63

100-100

00:00:19

135/150

90

02

65

100-100

00:00:51

140/150

93

03

68

100-100

00:00:19

141/150

94

04

06

80-90

00:00:08

141/150

94

05

04

80-90

00:00:05

140/150

93

06

45

80-90

00:00:11

138/150

92

07

08

80-90

00:00:09

138/150

92

08

05

80-90

00:00:08

137/150

91

09

05

80-90

00:00:06

137/150

91

10

33

50-50

00:00:20

138/150

92

88

Chapter 7 Signature Recognition with a Hybrid Approach Combining MNNs

It is important to note that in all three cases, the monolithic methods did achieve good results. The best performance was obtained using wavelet features, and the Hough transform matrix also produced very similar results. On the other hand, the simple edge features produced a less accurate recognition than the other two methods. 7.4.2.2 Modular Neural Network with Sugeno Fuzzy Integral The final experimental results correspond to the complete MNN described in Figure 7.1, and Table 7.5 summarizes the results of ten independent training runs. For the modular architecture, performance was consistently very high across all runs, and the best recognition accuracy of 98% was achieved in half of the runs. In fact, even the worst performance of 95% is better or equal than all but one of the monolithic ANNs (see Table 7.3). Table 7.5 Results for the Modular Neural Network with fuzzy Sugeno Integral Trian

Epochs

Time

Correct

Accuracy (%)

01 02

55 150

00:00:49 00:01:34

147/150 146/150

98 97

03 04

180 300

00:01:53 00:02:20

144/150 147/150

96 98

05 06

150 155

00:01:30 00:01:45

147/150 146/150

98 97

07 08

320 310

00:02:49 00:02:38

147/150 148/150

98 98

09 10

285 03

00:01:58 00:00:02

145/150 143/150

96 95

7.4.2.3 Modular Neural Network with a Fuzzy System We use a fuzzy systems integrator for the three modules of the network. The fuzzy systems are of Mamdani type, contain three inputs (module 1, module 2, module 3) output (winner module), and 27 rules. Several tests were performed with the fuzzy systems that have the same input, and output rules, but with different functions of membership: Triangular, trapezoidal and Gaussian. In Figures 7.8, 7.9, 7.10 we show the fuzzy systems with trapezoidal Membership functions, Triangular and Gaussian.

7.4 Experiments

89

1. If (MBinaria is Bajo) and (MWavelet is Bajo) and (MHough is Bajo) then (ModGanador is Modulo2) (1) 2. If (MBinaria is Bajo) and (MWavelet is Bajo) and (MHough is Medio) then (ModGanador is Modulo3) 3. If (MBinaria is Bajo) and (MWavelet is Bajo) and (MHough is Alto) then (ModGanador is Modulo3) (1) 4. If (MBinaria is Bajo) and (MWavelet is Medio) and (MHough is Bajo) then (ModGanador is Modulo2) (1) 5. If (MBinaria is Bajo) and (MWavelet is Medio) and

Fig. 7.8 Representation of fuzzy systems with trapezoidal membership functions

1. If (MBinaria is Bajo) and (MWavelet is Bajo) and (MHough is Bajo) then (ModGanador is Modulo2) (1 2. If (MBinaria is Bajo) and (MWavelet is Bajo) and (MHough is Medio) then (ModGanador is Modulo3) ( 3. If (MBinaria is Bajo) and (MWavelet is Bajo) and (MHough is Alto) then (ModGanador is Modulo3) (1) 4. If (MBinaria is Bajo) and (MWavelet is Medio) and (MHough is Bajo) then (ModGanador is Modulo2) (1) 5 If (MBinaria is Bajo) and (MWavelet is Medio) and

Fig. 7.9 Representation of fuzzy systems with Triangular membership functions

90

Chapter 7 Signature Recognition with a Hybrid Approach Combining MNNs

1. If (MBinaria is Bajo) and (MWavelet is Bajo) and (MHough is Bajo) then (ModGanador is Modulo2) (1 2. If (MBinaria is Bajo) and (MWavelet is Bajo) and (MHough is Medio) then (ModGanador is Modulo3) 3. If (MBinaria is Bajo) and (MWavelet is Bajo) and (MHough is Alto) then (ModGanador is Modulo3) (1) 4. If (MBinaria is Bajo) and (MWavelet is Medio)

Fig. 7.10 Representation of fuzzy systems with Gaussian membership functions

Table 7.6 Results for the Modular Neural Network with fuzzy system Train

Membership Funtion

Error goal

Epochs

01

Triangular

0.001

232

00:03:42

148/150

98.66

02

Triangular

0.001

560

00:09:15

146/150

97.33

03

Triangular

0.001

710

00:11:05

148/150

98.66

04

Trapezoidal

0.001

96

00:01:37

147/150

98.00

05

Trapezoidal

0.001

302

00:08:01

147/150

98.00

06

Trapezoidal

0.001

304

00:08:03

146/150

97.33

07

Gaussian

0.001

150

00:02:50

146/150

97.33

08

Gaussian

0.001

257

00:06:02

149/150

99.33

09

Gaussian

0.001

223

00:03:17

149/150

99.33

Time

Correct

Accuracy (%)

7.4 Experiments

91

The results obtained with the fuzzy system as an integrator of MNN with different membership functions (see table 7.6), were good, in this case the best result was obtained in the training with 9 Gaussian membership function, with a total of 223 epochs, and 149 images are classified correctly. The method of training is scaled conjugate gradient (Transcg). Overcoming the best result with fuzzy Sugeno integral (see table 7.5). 7.4.2.4 Modular Neural Network with a Fuzzy System Adding Uniform Random Noise After multiple tests done with the fuzzy system, and taking into account that the best result was obtained with Gaussian membership functions, we applied noise to the images of signatures, using “uniform random noise”. The noise level of 0.5 was applied. Table 7.7 shows the top 10 results. The best training is the one in the second row, with a total of 146 correctly classified images. Table 7.7 Result with uniform random noise

Train

Method

Time

Correct

Accuracy (%)

01

Trainscg

00:02:56

141/150

92.66

02

Trainscg

00:02:57

146/150

97.33

03

Trainscg

00:04:50

144/150

96.00

04

Trainscg

00:03:25

143/150

95.33

05

Trainscg

00:04:20

144/150

96.00

06

Trainscg

00:03:09

141/150

94.00

07

Trainscg

00:06:50

139/150

92.66

08

Trainscg

00:02:54

141/150

94.00

09

Trainscg

00:03:33

146/150

97.33

10

Trainscg

00:05:56

144/150

96.00

7.4.2.5 Results of Verification of Signatures Table 7.8 shows the results as a percentage for each case: false acceptance, false rejection, error recognition and the percentage of correct signatures.

92

Chapter 7 Signature Recognition with a Hybrid Approach Combining MNNs Table 7.8 Results from the verification of signatures

Train

01 02 03 04 05 06 07 08 09 10

Time

00:06:32 00:06:03 00:07:32 00:08:03 00:08:02 00:09:00 00:06:06 00:06:42 00:06:52 00:06:13

False Acceptance (%) 9.33 7.33 10.00 14.66 7.33 16.00 6.66 11.33 15.33 12.00

False Rejection (%) 8.00 18.00 5.33 2.00 18.00 4.00 14.00 10.66 4.66 6.00

Error Recognition (%) 1.33 0.66 2.00 1.33 0.66 1.33 1.33 1.33 2.00 2.66

Correct Sgnatures (%) 81.33 74.00 82.66 82.00 74.00 78.66 78.00 76.66 78.00 79.33

7.5 Summary In this chapter we have addressed the problem of signature recognition, a common behavioral biometric measure. We proposed a modular system using ANNs and three types of image features: edges, wavelet coefficients, and the Hough transform matrix. In our system, the responses from each module were combined using a Sugeno fuzzy integral and a fuzzy inference system. In order to test our system, we built a database of image signatures from 30 different individuals. In our experiments, the proposed architecture achieves a very high recognition rate, results that confirm the usefulness of the proposal. In our tests, we have confirmed that the modular approach always outperforms, with varying degrees, the monolithic ANNs tested here. However, in some cases the difference in performance was not very high, only 3 or 2 percent. Nevertheless, we believe that if the recognition problem is made more difficult then the modular approach will more clearly show a better overall performance.

Chapter 8

Interval Type-2 Fuzzy Logic for Module Relevance Estimation in Sugeno Response Integration of Modular Neural Networks

8.1 Introduction Aggregation has the purpose of making simultaneous use of different pieces of information provided by several sources in order to come to a conclusion or a decision. The aggregation operators are mathematical objects that have the function of reducing a set of numbers into a unique representative number, and any aggregation or fusion process done with a computer underlies numerical aggregation [64]. Most aggregation operators use some kind of parameterization to express additional information about the objects that take part in the aggregation process; then the parameters are used to represent the background knowledge. Among all the existing types of parameters, the fuzzy measures are a rich and important family. They are of interest because they are used for aggregation purposes in conjunction with fuzzy integrals like Choquet and Sugeno Integrals [65]. In this work we describe an image recognition method using Modular Neural Networks combined with Sugeno Integral [65]. The information to be combined is the simulation outputs of the 3 modules trained to recognize a different part of the image. The modular architecture consists in dividing each image in 3 parts after the edges detection process, and uses each part as training data for 3 monolithic neural networks. Then the problem becomes how to combine the simulation of the three modules in order to recognize the maximum number of images possible. We make the final decision using the Sugeno Integral, which is used to combine the simulation vectors into only one vector, then at the end of the method the system decides the best choice of recognition in the same manner than made with only one monolithic neural network, but with the problem of complexity resolved [59]. Then the problem is to find the ideal input parameters for the Sugeno Integral, which means, the values of the fuzzy density for each module, to rank its relevance in the decision process. This problem was solved by building a FIS to estimate the fuzzy densities using only the simulation vectors as input variables. This step was implemented with two fuzzy logic systems, namely using Type-1 and Interval Type-2 Fuzzy Logic respectively [12, 13, 15, 17, 18, 19, 83, 84].

P. Melin: Modular Neural Networks and Type-2 Fuzzy Systems, SCI 389, pp. 93–105. springerlink.com © Springer-Verlag Berlin Heidelberg 2012

94

Chapter 8 Interval Type-2 Fuzzy Logic for Module Relevance Estimation

8.2 Modular Neural Networks 8.2.1 Modular Structure The modular structure was designed for a database of images, like the Olivetty Research Laboratory database of faces (ORL), but it is not limited to this data. To measure the recognition rate in an objective form, we trained the modular neural networks with the set of images in a random ordered fashion, this process is called a random permutation [32, 63, 80, 93]. The design of the Modular Neural Network consists of 3 monolithic feedforward neural networks, each one trained with a supervised method with the first 7 samples of the 40 images of ORL. The edges vector for each image is accumulated into a matrix, as shown in the scheme of Figure 8.1. Then the complete matrix of images is divided into 3 parts, each module is trained with a corresponding part, with the some rows for overlapping.

Fig. 8.1 Input: Seven images for each person

The target to the supervised training method consists of one identity matrix with dimensions 40x40 for each sample, building one matrix with total dimensions (40x40x7), as shown in Figure 8.2.

Fig. 8.2. Target: One identity matrix with dimensions 40x40 for each sample

Each monolithic neural network has the same structure of Figure 8.3 and was trained under the same conditions: •

Three hidden layers with 200 neurons and tansig transfer functions.

8.2 Modular Neural Networks

• •

95

The output layer with 40 neurons and purelin transfer functions. The training function is gradient descent with momentum and adaptive learning rate back-propagation (traingdx).

Fig. 8.3 Structure of each monolithic neural network

8.2.2 Training of the Modules The next code segment was created to train each module, where newff is a Matlab Neural Network Toolbox function to create a feed-forward back-propagation network with the parameters specified as follows. layer1=200; layer2=200; layer3=40; net=newff(minmax(p),[layer1,layer2,layer3],{'tansig','tansig','log sig'},'traingdx'); net.trainParam.goal=1e-5; net.trainParam.epochs=1000;

8.2.3 Modules Simulation A program was developed in Matlab that simulates each module with the 400 images of the ORL database [63], building a matrix with the results of the simulation of each module, as it is shown in Figure 8.4. These matrices are stored in the file “mod.mat” to be analyzed later for the combination of results.

Fig. 8.4 Scheme of simulation matrices for the three modules

96

Chapter 8 Interval Type-2 Fuzzy Logic for Module Relevance Estimation

In the simulation matrix, columns corresponding to the images in the training data, with a value near one, are always selected correctly. However, some outputs of the training data have very low values in all positions, reason why it is very important to have a good combination method to recognize more images.

8.3 Sugeno Integral for Modules Fusion For the recognition of one image we have to divide it in three parts, and then simulate each one using the corresponding module. Then for each image to recognize we have three simulation vectors. To make the final decision is necessary the fusion of the three vectors, using an aggregation operator like Sugeno Integral. In this section we provide some antecedents about Sugeno Measures and Sugeno Integral [32, 63, 64].

8.3.1 Sugeno λ-Meausures The Sugeno measures (λ-measures) are monotone measures, μ that are characterized by the following requirement: For all A, B ∈ P(X), if A ∩ B = Ø, then:

μ ( A ∪ B ) = μ ( A) + μ ( B ) + λμ ( A) μ ( B )

(8.1)

Where λ>-1 is a parameter by which different λ-measures are distinguished. Equation (1) is usually called λ-rule. When X is a finite set and values µ({x}) (called fuzzy densities) are given for all x∈X, then the value µ(A) for any A ∈ P(X), can be determined from this values on singletons by a repeated application of the λ-rule. This value can be expressed as:





 x∈S



μ ( A) = ∏ (1 + λμ ({x})) / λ

(8.2)

Observe that, given values µ({x}) for all x∈X, the values of λ can be determined by the requirement that µ({X})=1. Applying this requirement to the equation (8.2) results in the equation (8.3): n

λ + 1 = ∏ (1 + λμ ({xi }))

(8.3)

i =1

for λ. This equation determines the parameter uniquely under the conditions stated in the following theorem: Theorem 8.1. Let µ({x}) <1 for all x∈X and let µ({x}) >0 for at least two elements on X. Then equation (3) determines the parameter λ uniquely as follows: If  μ ({x}) < 1 , then λ is equal to the unique root of the equation in the interx∈X

val (0, ∞) , which means that µ qualifies as a lower probability, λ>0.

8.5 Fuzzy Logic for Density Estimation 8.5 Fuzzy Logic for Dens ity Est imatio n

If

97

 μ ({x}) = 1 , then λ=0, which is the only root of the equation, which means

x∈ X

that µ is a classical probability measure, λ=0. If  μ ({x}) > 1 , then λ is equal to the unique root of the equation in the interx∈X

val (−1,0) , which means that µ qualifies as an upper probability, λ<0. Once the value of λ is calculated using some numerical method for zero finding, the Sugeno measures can be calculated using equations (8.4) and (8.5), after descendent ordered of the sets X and µ({x}), respect to the elements of set X.

μ ( A1 ) = μ ( x1 )

(8.4)

μ ( Ai ) = μ ( xi ) + μ ( Ai −1 ) + λμ ( xi ) μ ( Ai −1 )

(8.5)

8.4 The Sugeno Integral The Sugeno Integral can be interpreted in a way similar to the weighted maximum, but use the measure instead of possibility distribution [63]. The difference is that now each value is weighted according to the weight (the measure) of all the sources that support the value (8.6).

h (σ 1 ,..., σ n) = max in=1 (min( σ i , μ ( Ai )))

(8.6)

Where σi= σ(xi) y 0 ≤ σ1 ≤…≤ σn ≤ 1 The Sugeno Integral generalizes both weighted minimum and weighted maximum [64]. A weighted maximum with a possibilistic weighting vector u is equivalent to a Sugeno integral with fuzzy measure

μuw max ( A) = max ui

(8.7)

ai ∈ A

A weighted minimum with a possibilistic weighting vector u is equivalent to a Sugeno integral with fuzzy measure

μuw min ( A) = 1 − max ui ai ∉ A

(8.8)

8.5 Fuzzy Logic for Density Estimation After the simulation of an image in the Neural Network, the simulation value is the only known parameter to make a decision, then to estimate the fuzzy density

98

Chapter 8 Interval Type-2 Fuzzy Logic for Module Relevance Estimation

of each module this is the only available information. For this reason we analyze the simulation matrix in many tests and decide that each of the inputs to the FIS corresponds to the maximum value of each column corresponding to the simulation of each module of the 400 images. Then m1, m2 and m3 correspond to the simulation values to combine, max1, max2 and max3 correspond to the maximum values of the simulation vector, and d1, d2 and d3 the fuzzy densities for each module, as shown in Figure 8.5.

Fig. 8.5 Variables for the FIS to find the fuzzy densities

Each output corresponds to one fuzzy density, to be applied to each module to perform the fusion of results later with the Sugeno Integral. The inference rules calculate fuzzy densities near 1 when de maximum value in the simulation is between 0.5 and 1, and near 0 when the maximum value in the simulation is near 0. The fuzzy rules are shown next. 1.If (m1 is LOW) and (max1 is LOW) then (d1 is LOW) 2.If (m1 is MEDIUM) and (max1 is MEDIUM) then (d1 is MEDIUM) 3.If (m1 is HIGH) and (max1 is HIGH) then (d1 is HIGH) 4.If (m2 is LOW) and (max2 is LOW) then (d2 is LOW) 5.If (m2 is MEDIUM) and (max2 is MEDIUM) then (d2 is MEDIUM) 6.If (m2 is HIGH) and (max2 is HIGH) then (d2 is HIGH) 7.If (m3 is LOW) and (max3 is LOW) then (d3 is LOW) 8.If (m3 is MEDIUM) and (max3 is MEDIUM) then (d3 is MEDIUM) 9.If (m3 is HIGH) and (max3 is HIGH) then (d3 is HIGH) 10.If (m1 is LOW) and (max1 is MEDIUM) then (d1 is LOW) 11.If (m1 is MEDIUM) and (max1 is HIGH) then (d1 is MEDIUM) 12.If (m2 is LOW) and (max2 is MEDIUM) then (d2 is LOW) 13.If (m2 is MEDIUM) and (max2 is HIGH) then (d2 is MEDIUM) 14.If (m3 is LOW) and (max3 is MEDIUM) then (d3 is LOW)

8.6 FIS-1 to Estimate Fuzzy Densities

99

15.If (m3 is MEDIUM) and (max3 is HIGH) then (d3 is MEDIUM) 16.If (m1 is LOW) and (max1 is HIGH) then (d1 is LOW) 17.If (m2 is LOW) and (max2 is HIGH) then (d2 is LOW) 18.If (m3 is LOW) and (max3 is HIGH) then (d3 is LOW)

According to exhaustive tests made in the simulation matrices, we know that recognition of the images that were used for the training the neural networks is 100%. Therefore the interest is focused on the recognition of the samples that do not belong to the training set, in other words samples 8, 9 and 10. The parameters for the Sugeno Fuzzy Integral that will be inferred will be the Fuzzy Densities, with values between 0 and 1 for each module, which determines the relevance rate for each module.

8.6 FIS-1 to Estimate Fuzzy Densities

HIGH

0.5

0 0.4

0.6

0.8

0.5

0

1

0

0.2

0.4

0.5

0 0

Degree of membership

MEDIUM HIGH

Degree of membership

LOW

1

0.2

0.4 0.6 max1

LOW

1

0.8

MEDIUM HIGH

0 0.2

0.4

0.6 d1

0.8

1

LOW

1

0.8

0 0

0.2

0.4

0.2

MEDIUM HIGH

0.4 0.6 max2

LOW

1

0.8

MEDIUM HIGH

0 0.2

0.4

0.6 d2

0.8

1

LOW

1

0.8

1

MEDIUM HIGH

0.5

0

1

0.5

0

0.6 m3

0 0

HIGH

0.5

1

0.5

1

0.5

0

0.6

LOW MEDIUM

1

m2

Degree of membership

Degree of membership

m1

Degree of membership

0.2

HIGH

0

Degree of membership

0

LOW MEDIUM

1

Degree of membership

LOW MEDIUM

1

Degree of membership

Degree of membership

The membership functions in Figure 8.6 and the solution surface in Figure 8.7 show the system’s behavior.

0.2

0.4 0.6 max3

LOW

1

0.8

1

MEDIUM HIGH

0.5

0 0

0.2

0.4

0.6

0.8

1

d3

Fig. 8.6 Variables for FIS 1 for find fuzzy densities, before optimization

100

Chapter 8 Interval Type-2 Fuzzy Logic for Module Relevance Estimation

Fig. 8.7 Solution surface for max1, m1 and d1 in FIS 1 to find fuzzy densities

8.7 FIS-2 for Estimate Fuzzy Densities To compare the FIS-1 and FIS-2 to estimate the fuzzy densities, we added a FOU=0.2, to the same fuzzy variables in FIS-1, as we can see in Figure 8.8.

1 LOW MEDIUM

HIGH

1 LOW MEDIUM

0.5 0

0

0.5 m1

1 MEDIUM HIGH

0

0

0.5 m2

1 MEDIUM HIGH

LOW

1

0.5 1 max1 MEDIUM HIGH

0.5

0

0.5 d1

1

0 LOW

1

0

0

0.5 m3

1 MEDIUM HIGH

0.5

0.5 1 max2 MEDIUM HIGH

0.5

0

0

1 LOW

0.5

0

HIGH

0.5

1 LOW

0.5

0

1 LOW MEDIUM

0.5

1 LOW

0

HIGH

0

0 LOW

1

0.5 1 max3 MEDIUM HIGH

0.5

0

0.5 d2

1

0

0

0.5 d3

1

Fig. 8.8 Variables for the FIS-2 to estimate fuzzy densities, using the same centers of the FIS-1 and FOU=0.2

8.8 Sugeno Integral for Information Fusion

101

Fig. 8.9 Solution surface for max1, m1 and d1 in FIS-2 to find fuzzy densities

8.8

Sugeno Integral for Information Fusion

Then, after the simulation of one image divided in three modules we have three simulation vectors of length 40 to combine. The value for each element of the resulting vector is the Sugeno Integral for the values corresponding to the same position in the 3 vectors to combine. Consider the following values corresponding to simulation of sample 8 of person number 13, this sample is not on the training data.

1 2 3 : 13 14 : 40

m1

m2

m3

0.1575

0.0094

0.0247

max1 0.1575

max2 0.0286

max3 0.0574

The fuzzy densities were inferred by a FIS as follows: First the FIS calculates the fuzzy densities for each module. d1= 0.3069 d2= 0.1788 d3= 0.1890 Then we need to solve the function f(λ)=(1+0.3069λ)(1+0.1788λ)(1+0.1890λ) – (1+λ) as we can see in Figure 8.10, according with equation (8.11).

102

Chapter 8 Interval Type-2 Fuzzy Logic for Module Relevance Estimation 0.8 0.6 0.4 0.2 0 -0.2 -1

0

1

2

3

Fig. 8.10 Plot for f(λ)=(1+0.3069λ)(1+0.1788λ)(1+0.1890λ) –(1+λ)

The value of λ can be calculated by equation (8.11), using a numerical method such as Newton-Raphson or bisection, to find the root of f(λ) shown in figure 8.10. To solve this equation we used the Matlab function fzero that finds a root of continuous function of one variable. The solution for λ= 1.9492 in this example, and the Sugeno Measures can be constructed using the recursive formulas (12) and (13), but before this calculation the fuzzy densities must be sorted descendent by σ (xi), that is shown in Table 8.1. Table 8.1 The Values Sorted Descendent By X Before Normalization

xi 0.1575 0.0247 0.0094

σ (xi)=trapmf([0 max(i) max(i) 2],xi) 0.9989 0.0492 0.0249

µ(A1)=µ(x1) µ(Ai)=µ(xi)+µ(Ai-1)+λµ(xi)µ(Ai-1) µ(A1)= 0.3069 µ(A2)= 0.1890+0.3069+(1.9492)(0.1890)(0.3069)= 0.6090 µ(A3)= 0.1788+0.6090+(1.9492)(0.1788)(0.6090)= 1 The Fuzzy Sugeno Integral, can now be calculated using (8.6): h(0.9989,0.0492,0.0249)= max( min(0.9989,0.3069), min(0.0492,0.6090), min(0.0249,1)) h(0.9989,0.0492,0.0249)=max(0.3089,0.0492,0.0249)=0.3089

µ(xi) 0.3069 0.1890 0.1788

8.8 Simulation Results

103

As can be seen in Figure 8.11, for person number 13 it was obtained the highest value of the Sugeno Integral and is correctly selected.

0.35 Sugeno Integral for person 13

Simulation and Sugeno Integral

0.3

0.25

0.2

0.15

0.1

0.05

0

0

5

10

15

20 Person

25

30

35

40

Fig. 8.11 Simulation of the Modules and the Sugeno Integral

In order to measure in an objective form the final results, we developed a method of random permutation, which rearranges the samples of each person before the training. Once a permutation is made, the modular neural networks are trained 5 times and the net weights and permutation order are saved for posterior tests.

8.8 Simulation Results Some of the images don’t reach a sufficient value in the simulation of the three modules, in these cases, there isn’t enough information to select an image at the modules combination, and the image can be wrongly selected. But, in most of the cases the images are correctly selected. In Tables 8.2 and 8.3 we show the recognition rates for FIS-1 and FIS-2. Table 8.2 Recognition rates with FIS-1 Fuzzy Densities Estimation

P 1 2 3 4 5

Train 1 94.00 94.25 94.25 94.00 94.75

Image Recognition (%) Train 2 Train 3 Avg 95.75 94.50 94.75 94.75 94.25 94.41 94.25 95.25 94.58 93.25 93.50 93.58 94.75 94.00 94.36 94.36

Max 95.75 94.75 95.25 94.00 94.75 95.75

104

Chapter 8 Interval Type-2 Fuzzy Logic for Module Relevance Estimation Table 8.3 Recognition rates with FIS-2 Fuzzy Densities Estimation

P 1 2 3 4 5

Train 1 97.25 94.75 95.50 95.25 96.50

Image Recognition (%) Train 2 Train 3 Avg 96.25 95.00 96.17 95.25 95.75 95.25 97.50 96.00 96.33 95.00 95.50 95.25 97.00 96.00 96.50 95.90

Max 97.25 95.75 97.50 95.50 97.00 97.50

To appreciate the performance of each fuzzy system more clearly, in Figure 8.12 we show the recognition rates for the simulation of the Modular Neural Network, for 3 different trainings of type-1 and type-2 fuzzy logic [12, 13, 14, 15, 16].

Fig. 8.12 Recognition Rates for FIS-1 and FIS-2 fuzzy densities estimation

In Figure 8.13 we show the best recognition rates for both types of fuzzy systems. As we can see in this comparison, the best recognition rates using the Type-2 Fuzzy System are better than with Type-1 Fuzzy Systems.

8.9 Summary

105

Fig. 8.13 The best Recognition Rates for FIS-1 and FIS-2 fuzzy densities estimation

8.9 Summary In image recognition with Modular Neural Networks, when an image is divided into several parts, each module is an expert to recognize only one part. Then, in the simulation phase if one of the modules doesn’t reach the target values, is a module with low relevance for the recognition. Fuzzy density is the rate of the relevance of each module, and the estimation of its values improves the recognition rates on Modular Neural Networks combined with Sugeno Integral. Fuzzy densities estimated with the Interval Type-2 Fuzzy System produce better recognition rates than Type-1 Fuzzy Systems.

Part III

Chapter 9

Optimization of Fuzzy Response Integrators in Modular Neural Networks with Hierarchical Genetic Algorithms

In this chapter we describe the development of Fuzzy Response Integrators of Modular Neural Networks (MNN) for Face, Fingerprint and Voice Recognition, and their Optimization with a Hierarchical Genetic Algorithm (HGA). The optimization of the integrators consists of optimizing their membership functions, fuzzy rules, type of model (Mamdani or Sugeno), type of fuzzy logic (type-1 or type-2). The MNN architecture consists of three modules; face, fingerprint and voice [80]. Each of the modules is divided again into three sub modules. The same information is used as input to train the sub modules. Once we have trained and tested the MNN modules, we proceed to integrate these modules with an optimized fuzzy integrator. In this paper we show that using a HGA as an optimization technique for the fuzzy integrators is a good option to solve MNN integration problems [59, 80, 93].

9.1 Introduction Person identification has become a very important activity in different areas of application and with different purposes [59]; in business applications based on punctuality and attendance biometric systems are implemented using the fingerprint or the full hand of the employees; in judicial systems biometric pattern recognition has become a routine tool in the police force during the criminal investigation, allowing the arrest of criminals worldwide, but also other specific applications are known, such as controlling access to any type of transaction or access to data. Neural networks provide a methodology to work in the field of pattern recognition [57]. Modular neural networks are often used to simplify the problem and obtain good results [59]. Because of this, there arises the need to develop methods to integrate the responses provided by the modules of a modular neural network and thus provide a good result. This chapter develops a fuzzy integrator to combine the responses provided by the modular neural network and achieve a good recognition of persons. P. Melin: Modular Neural Networks and Type-2 Fuzzy Systems, SCI 389,, pp. 109–126. springerlink.com © Springer-Verlag Berlin Heidelberg 2012

110

Chapter 9 Optimization of Fuzzy Response Integrators

The work described in this paper is divided into two main areas, the first one is about the Modular Neural Network (MNN) and the second one is about the Fuzzy Integrators. The main goal of the first part is to create and test different MNNs architectures for face, fingerprint and voice recognition [32]. The main goal of the second part is to develop fuzzy integrators to integrate the outputs given by the modules of the modular neural network [63, 64, 80]. This chapter is organized as follows: in sections 2, 3 and 4 we present the basic concepts of neural networks, fuzzy logic theory and genetic algorithms respectively; in sections 5, 6 and 7 we describe the problem and mention how to solve it making use of a MNN with fuzzy integration and the results obtained. In section 8 we describe the HGA to optimize the fuzzy integrator structure and the optimization results; in section 9 we make a comparison with other work. Finally, section 10 presents the conclusions.

9.2 Neural Networks Artificial neural networks (NN) are an abstract simulation of a real nervous system that consists of a set of neural units connected to each other via axon connections [57]. These connections are very similar to the dendrites and axons in biological nervous systems. Figure 9.1 shows the abstract simulation of a real nervous system. Models of artificial neural networks can be classified as: • Biological models: Networks that try to simulate the biological neural systems, as well as the functions of hearing or some basic functions of vision. • Models for applications: Those models are less dependent on models of biological systems. These are models in which their architectures are strongly linked to the application requirements.

Fig. 9.1 Simulation of an abstract real nervous system

9.3 Fuzzy Logic

111

One of the missions of a neural network is to simulate the properties observed in biological neural systems through mathematical models recreated through artificial mechanisms (such as an integrated circuit, a computer or a set of valves). The aim is that the machines give similar responses to those the brain is able to give, which are characterized by their robustness and generalization. Artificial neural networks are models that attempt to reproduce the behavior of the brain. As such model, a simplification is made, identifying the relevant elements of the system, either because the amount of information available is excessive or because it is redundant. An appropriate choice of features and an appropriate structure is the conventional procedure used to construct networks capable of performing certain tasks [56, 58, 60, 63, 93]. Modularity is defined as the ability of a system being studied, seen or understood as the union of several parts interacting with each other and working towards a common goal, each one performing a task necessary to achieve that objective. Each of these parts in which the system is divided is called a module. Ideally, a module must be able to work as a black box, i.e. be independent of other modules and communicate with them (all or only a part) through well-defined inputs and outputs. It is said that a neural network is modular if the computations performed by the network can be decomposed into two or more modules (subsystems) that operate on different inputs with no communication between them [59]. The outputs of the modules are measured by an integration unit, which is not allowed to feed information to the modules. In particular, the unit decides: (1) how the modules are combined to form the final output of the system, and (2) modules that must learn that patterns of training. As mentioned previously, modular neural networks require an integration unit, which allows some form of joining or combining the responses provided by each module. Listed below are some methods that can be used to perform this task [59]: • • • • •

Average operators Gating Network Fuzzy Integrators Voting mechanism using the Softmax function Etc.

9.3 Fuzzy Logic Fuzzy logic has gained a great reputation as a good methodology for a variety of applications, ranging from control of complex industrial processes to the design of artificial devices for automatic deduction through the construction of electronic devices for home use and entertainment, as well as diagnostic systems. It has been considered generally that the concept of fuzzy logic was proposed in 1965 at the University of California at Berkeley, introduced by Lotfi A. Zadeh [103]. The basic structure of a fuzzy inference system consists of three conceptual components: a rule base, which contains a selection of fuzzy rules, a database (or

112

Chapter 9 Optimization of Fuzzy Response Integrators

dictionary) which defines the membership functions used in the rules, and a reasoning mechanism that performs the inference procedure [12, 13, 14, 15, 16, 19, 20, 37, 41, 46]. The basic fuzzy inference system can take either fuzzy or traditional inputs, but the outputs it produces are always fuzzy sets. Sometimes you need a traditional output, especially when a fuzzy inference system is used as a controller. So we need a method of “Defuzzification” to extract the numerical value of the output (as shown in Figure 9.2).

Fig. 9.2 Structure of a Fuzzy Inference System

9.4 Genetic Algorithms Genetic algorithms or evolutionary algorithms are stochastic search techniques guided or inspired by the mechanisms of natural selection, genetics and evolution [24, 25]. Early research on the subject was developed by John Holland at the University of Michigan in the 70's with two objectives: to abstract and rigorously explain the adaptive processes of natural systems and the design of software systems that determine the retention of these important mechanisms. This technique is based on the selection mechanisms used by nature, according to which the fittest individuals of a population are those who survive, which are those who adapt more easily to changes in their environment. We now know that these changes are made in the genes (the basic encoding unit of each of the attributes of a living being) of an individual, and that the most desirable attributes (for example, that allow an individual to better adapt to its environment) of the individuals are transmitted to their descendants, if reproduced sexually [59]. Imitating the mechanics of biological evolution in nature, genetic algorithms operate on a population of possible solutions to the problem. Each element of the population is called "chromosome". A chromosome is the representative within the genetic algorithm, of a possible solution to the problem. The process begins by

9.5 Modular Neural Network with Fuzzy Integration

113

selecting a number of chromosomes to form the initial population. Then we evaluate the role of adaptation for these individuals. The fitness function measures the chromosome's fitness to survive in its environment. This function must be defined so that the chromosomes that represent better solutions have higher values of fitness. The fittest individuals are selected in pairs to breed. Reproduction generates new chromosomes that combine characteristics of both parents. These new chromosomes replace individuals with lower values of adaptation. After this, some chromosomes are randomly selected to be mutated. The mutation consists of applying a random change in its structure. Then, the new chromosomes must be inserted into the population; these chromosomes should replace existing chromosomes. There are different criteria that can be used to select chromosomes to be replaced. The cycle of selection, reproduction and mutation is repeated until it meets the criterion of termination of the algorithm, at which the chromosome that is better suited as a solution is returned. 9.5 Modular Ne ural Networ k wit h F uzzy I ntegration

9.5 Modular Neural Network with Fuzzy Integration for Face, Fingerprint and Voice Recognition The proposed MNN architecture defined in this chapter consists of three modules: face, fingerprint and voice [32]. Each of the mentioned modules will be divided into three sub modules. The same information is used as input to train the sub modules. After the MNN trainings are done, we proceed to integrate the three modules (face, fingerprint ad voice) with a fuzzy integrator [80]. The data used for training the MNN correspond to 30 persons taken from the ORL database and from students of a master degree in computer science from Tijuana Institute of Technology, Mexico [93]. The details are as follows: • Face: 90 images, size of these images is of 268x338 pixels with bmp extension and were preprocessed with the Wavelet transform. Two images are used for training (with different gestures) and one with Gaussian noise to test the recognition. • 60 fingerprint images, the size of these images is of 268x338 pixels with bmp extension and were preprocessed with the Wavelet function. One fingerprint per person for training and one with Gaussian noise to test the recognition. • Voices: There are 3 word files per person; the words are “hola”, “accesar” and “presentacion” in Spanish. The network was trained with the 90 words preprocessed with Mel-cepstral Coefficients and the network is tested with any of them but with noise. Examples of face and fingerprint are shown in Figure 9.3 and a general Scheme of the architecture is shown in Figure 9.4.

114

Chapter 9 Optimization of Fuzzy Response Integrators

Fig. 9.3 Fingerprint and face samples

Fig. 9.4 Architecture of the MNN

9.6 Modular Neural Network Results

115

9.6 Modular Neural Network Results Several trainings were made separately to each of the modules until we accomplish good results. Tables 9.1, 9.2, 9.3 and 9.4 show the results of the MNN trainings for face, fingerprint and voice, respectively. Figure 9.5 shows the results from Table 9.4. Table 9.1 Face training results

Training

Method

ER1

trainscg

ER2

trainscg

ER3

trainscg

ER4

trainscg

ER5

trainscg

ER6

trainscg

ER7

trainscg

Mod 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3

Architecture 350,300 350,300 350,300 300,300 300,300 300,300 300,240 300,250 300,260 350,300 350,300 350,300 300,150 300,150 350,150 290,250 305,210 320,200 300,300 300,300 300,300

Error

Duration Rec.

%

0.01

16:10

19/30

63.33

0.01

20:34

21/30

70

0.001

49:39

28/30

93.33

0.001

40:58

28/30

93.33

01:12:30 28/30

93.33

0.001

0.001

50:50

29/30

96.66

0.001

46:07

30/30

100

We can observe in Table 9.1 that the best result obtained for face module recognition is for training ER7 with 100 percent of recognition. We can observe in Table 9.2 that in several trainings (EH1, EH2, EH3, EH4 and EH5) we obtained good results, having 100 percent of recognition for the fingerprint module.

116

Chapter 9 Optimization of Fuzzy Response Integrators Table 9.2 Fingerprint training results

Training

Method

EH1

trainscg

EH2

trainscg

EH3

trainscg

EH4

trainscg

EH5

trainscg

EH6

trainscg

EH7

trainscg

Mod 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3

Architecture 340,175 280,105 295,138 200,100 200,100 200,100 150,80 150,80 150,80 150,100 150,90 150,110 100,50 100,60 100,55 100,50 100,60 100,55 150,70 117,90 157,87

Error

Duration

Rec.

%

0.01

12:51

30/30

100

0.01

15:56

30/30

100

0.01

59:34

30/30

100

0.01

18:09

30/30

100

0.01

01:39:13

30/30

100

0.02

16:37

26/30

86.66

0.02

08:07

26/30

86.66

Table 9.3 Voice training configuration Training

Method

EV1

trainscg

EV2

trainscg

EV3

trainscg

EV4

trainscg

EV5

trainscg

EV6

trainscg

EV7

trainscg

Architecture 450,240 420,190 410,225 150,80 170,90 170,75 150,100 150,100 150,100 350,140 320,90 310,125 180,80 180,80 180,80 180,120 180,120 180,120 300,100 300,100 300,100

Error

Duration

0.001

01:44

0.001

12:27

0.001

02:03

0.001

02:17

0.001

29:01

0.001

01:41

0.001

02:24

9.6 Modular Neural Network Results

117

Table 9.4 Voice training results

Training

0.1

Recognized persons with noise 0.2 0.3 0.4 0.5 0.6 0.7

0.8

0.9

1.0

EV1

90

90

90

90

90

90

90

86

81

74

EV2

90

90

90

90

89

86

85

74

62

62

EV3

90

90

90

90

90

88

88

75

50

49

EV4

90

90

90

90

90

90

90

83

78

68

EV5

90

90

90

90

90

89

89

79

64

65

EV6

90

90

90

90

90

89

89

72

64

56

EV7

90

90

90

90

90

89

89

88

81

68

We can observe in tables 9.3 and 9.4 that the best result obtained for the voice module recognition is for training EV1, because it has better performance when different noise levels are applied to the input when testing the module. It is important to know that the noise level consists of adding to each voice signal, a data array of normally distributed random numbers with a standard deviation of 0.1 to 1 (where noise level 0.1 represents a standard deviation of 0.1, 0.2 a standard deviation of 0.2…1.0 a standard deviation of 1). When the recognized persons equals 90 we have a 100 percent of recognition.

Fig. 9.5 Results of voice trainings with different noise level

118

Chapter 9 Optimization of Fuzzy Response Integrators

9.7 Fuzzy Integration Results A fuzzy integrator was created using the MATLAB fuzzy logic toolbox, so that we can use it to integrate the outputs of the MNN modules and to generate a final result. This fuzzy integrator consists of 27 rules; three inputs that are provided by the MNN: face activation, fingerprint activation and voice activation; one output: winner activation that will indicate which person is recognized. It is important to notice that each input and output has three Gaussian membership functions. Figure 9.6 shows the architecture of the fuzzy integrator.

Fig. 9.6 Architecture of the fuzzy integrator

Table 9.5 shows the results from the fuzzy integration of the MNN. These results were obtained by making some combinations of the MNN modules. We can observe that when all three modules have 100% of recognition, fuzzy integration is perfect (i.e. No. 8). Even in some cases where one or two of the modules do not give good recognition, fuzzy integration is perfect too (i.e. no.4, 6 and 7). But, when all three modules have poor recognition, the fuzzy integration is not good (i.e. no. 1).

9.8 Hierarchical Genetic Algorithm

119

Table 9.5 Fuzzy integration results No. 1 2 3 4 5 6 7 8

Face ER1 ER1 ER7 ER7 ER1 ER1 ER7 ER7

Rec. Fingerprint 63.33% EH6 63.33% EH1 100% EH6 100% EH1 63.33% EH6 63.33% EH1 100% EH6 100% EH1

Rec. 86.66% 100% 86.66% 100% 86.66% 100% 86.66% 100%

Voice-noise EV3 1.0 EV3 1.0 EV3 1.0 EV3 1.0 EV1 0.5 EV1 0.5 EV1 0.5 EV1 0.5

Rec. 49% 49% 49% 49% 100% 100% 100% 100%

Integration 71/90(78.89%) 90/90 (100%) 80/90(88.89%) 90/90 (100%) 73/90(81.10%) 90/90(100%) 81/90(90%) 90/90(100%)

9.8 Hierarchical Genetic Algorithm Analyzing the results shown in Table 5, we can observe that the integration results for the cases 1, 3, 5 and 7 are not the most desirable, so we decided to develop a genetic algorithm to optimize the structure of the fuzzy integrator and thus increase the percentage of recognition. Figure 9.7 shows the chromosome structure we designed to optimize the fuzzy integrator.

Fig. 9.7 HGA Chromosome structure

120

Chapter 9 Optimization of Fuzzy Response Integrators

The chromosome shown in figure 9.7 consists of 325 gens that will help us optimize the fuzzy integrator structure. Gens 1-4 (Binary) are activation gens that define the structure of the fuzzy integrator such as the logic type (FL type-1 or type-2), the system’s type (Mamdani or Sugeno) and membership function type (Gaussian, Gbell, Triangular or Trapezoidal). Gens 5-298 (Real Numbers) allow us to manage the MFs parameters for the input and output (depending of FL type, system type and MFs type we use part of the gens); gens 299-325 (Binary) allows us to reduce the set of possible fuzzy rules activating or deactivating them. The optimization of the fuzzy integrator is proposed to be made based directly on the performance of the neural network, because we want to: (1) reduce the number of recognition errors and (2) reduce the number of fuzzy rules. The objective Function (equation 9.1) is the one we want to minimize, so that we can achieve our two objectives.

Obj = e + (n / m ) r r r

(9.1)

Where: er is the error of recognition given when simulating the MNN. nr is the number of fuzzy rules of the current individual. mr is the maximum number of fuzzy rules. We executed the HGA 20 times with different parameters (generation, crossover, mutation) for each of the cases we wanted to optimize (in table 5: cases 1, 3, 5 and 7). We show the results in Table 9.6. Table 9.6 Fuzzy integration results comparisson No. Face 1 2 3 4 5 6 7 8

ER1 ER1 ER7 ER7 ER1 ER1 ER7 ER7

Rec. 63.33% 63.33% 100% 100% 63.33% 63.33% 100% 100%

Fingerprint

Rec.

EH6 EH1 EH6 EH1 EH6 EH1 EH6 EH1

86.66% 100% 86.66% 100% 86.66% 100% 86.66% 100%

Voice-noise EV3 EV3 EV3 EV3 EV1 EV1 EV1 EV1

1.0 1.0 1.0 1.0 0.5 0.5 0.5 0.5

Rec. Non Optimized Optimized Integration Integration 49% 71/90(78.89%) 87/90 (96.67%) 49% 90/90 (100%) NO NEED 49% 80/90(88.89%) 90/90 (100%) 49% 90/90 (100%) NO NEED 100% 73/90(81.10%) 85/90 (94.44%) 100% 90/90(100%) NO NEED 100% 81/90(90%) 90/90 (100%) 100% 90/90(100%) NO NEED

We can observe in figure 9.7 that the proposed HGA help us increase the percentage of recognition in all cases. Despite these good results we decided to manually change some rules consequent in the best fuzzy integrator obtained with the HGA for case 5 because we believed that we could improve that result. When we tested the manually changed fuzzy integrator we obtained a 100% of recognition for case 5, so we decided to make some changes to our HGA chromosome structure so that it would be able not only to activate or deactivate fuzzy rules but also to change the rules’ consequents. Figure 9.9 shows the new

9.8 Hierarchical Genetic Algorithm

Fig. 9.8 Integration Comparison

Fig. 9.9 New HGA Chromosome structure

121

122

Chapter 9 Optimization of Fuzzy Response Integrators

HGA chromosome structure; the change we made to it, in comparison with the previous is that we added 27 gens more that will allow the HGA to manage the consequents of each of the 27 rules. We also decided to change the objective function for this new HGA as we can appreciate in equation 9.2. (9.2) Where: er is the error of recognition given when simulating the MNN. nr is the number of fuzzy rules of the current individual. α is the weight assigned to the error recognition objective. β is the weight assigned to the fuzzy rules objective. We executed the HGA 20 times with different parameters (generation, crossover, mutation) and for each case we needed to optimize. We show the results in Table 9.8. Table 9.7 Trainings information for fuzy integration No.

Face

Rec.

Fingerprint

Rec.

1 2 3 4 5 6 7 8

ER1 ER1 ER7 ER7 ER1 ER1 ER7 ER7

63.33% 63.33% 100% 100% 63.33% 63.33% 100% 100%

EH6 EH1 EH6 EH1 EH6 EH1 EH6 EH1

86.66% 100% 86.66% 100% 86.66% 100% 86.66% 100%

Voicenoise EV3 1.0 EV3 1.0 EV3 1.0 EV3 1.0 EV1 0.5 EV1 0.5 EV1 0.5 EV1 0.5

Rec. 49% 49% 49% 49% 100% 100% 100% 100%

Table 9.8 Fuzzy integration results comparison No.

1 2 3 4 5 6 7 8

Non-Optimized Fuzzy Integration

71/90 (78.89%) 90/90 (100%) 80/90 (88.89%) 90/90 (100%) 73/90(81.10%) 90/90(100%) 81/90(90%) 90/90(100%)

Optimized Integration HGA 1

Optimized Integration HGA 2 (α=0.5 β=0.5)

Optimized Integration HGA 2

Optimized Integration HGA 2 (α=0.9 β=0.1)

(α=0.7 β=0.3) 87/90 (96.67%) 84/90 (93.33%) 86/90 (95.56%) 88/90 (97.78%) NO NEED NO NEED NO NEED NO NEED 90/90 (100%) NO NEED NO NEED NO NEED NO NEED NO NEED NO NEED NO NEED 85/90 (94.44%) 90/90 (100%) NO NEED NO NEED NO NEED NO NEED NO NEED NO NEED 90/90 (100%) NO NEED NO NEED NO NEED NO NEED NO NEED NO NEED NO NEED

9.8 Hierarchical Genetic Algorithm

123

Table 9.8 shows a comparison of all the fuzzy integrations that were made. We can observe in the last column that we achieve the best fuzzy integration result for No. 1 with α=0.9 β=0.1. Because the fuzzy integration of case 1 was the most difficult to optimize, Figure 9.10 shows the behavior of the HGA evolutions with different weights (for case 1).

Fig. 9.10 HGA evolutions for case 1

We can observe in Figure 9.10 that the best weights configuration of the HGA for case 1 are α=0.9 and β=0.1, this means that assigning a very high weight to the error recognition objective and a smaller weight to the fuzzy rules objective allowed us to increase the recognition percentage in case 1.

Fig. 9.11 New Fuzzy integration comparison

124

Chapter 9 Optimization of Fuzzy Response Integrators

Figure 9.11 shows the final comparison of non-optimized integration vs. optimized integration.

9.9 Comparison with Other Works Many research works have been made in this area. This chapter is based directly on the work by Hidalgo et al. [32]. In this work a comparison of type-1 and type-2 fuzzy integration method was made for a MNN for face, fingerprint and voice recognition and optimized these systems with a GA (only MFs parameters). The work described in this chapter is different from Hidalgo’s work in the following form: the way of training the MNN (we trained each module separately), Optimization (we used a HGA to optimize the structure of a FLS not only MFs parameters). We used t-student statistic to make the comparison of results. Figures 9.12, 9.13, 9.14 and 9.15 show the results of the hypothesis testing (mean of two independent samples) using the Statdisk 9.5.2 software.

Fig. 9.12 Descriptive statistics for the method proposed in this chapter

We can observe from Figures 9.14 and 9.15 that there is statistical evidence to say that there is significant difference from the results of our paper with respect to the ones in Hidalgo’s work (with almost 90% of confidence). In other words, the HGA provided significant improvement in the recognition rate by improving the responses of the MNN.

9.9 Comparison with Other Works

Fig. 9.13 Descriptive statistics for Hidalgo’s work

Fig. 9.14 Hypothesis testing

125

126

Chapter 9 Optimization of Fuzzy Response Integrators

Fig. 9.15 Hypothesis testing plot

9.10 Summary The analysis of the results presented in this chapter shows that Modular Neural Networks are reliable models for pattern recognition. In this case we considered face, fingerprint and voice as biometric measures. We also demonstrated that a fuzzy integrator is a good choice when we need to integrate the outputs of the modules of a MNN. It is important to note that the results with a non optimized fuzzy integrator were improved using a HGA (Hierarchical Genetic Algorithm). The hierarchical genetic algorithm presented in this paper was designed to cover a considerable number of features attributable to the structure of the fuzzy integrator (type of system, type of logic, type of membership functions, MFs parameters and fuzzy rules), allowing the HGA to explore a very large space of solutions and thus obtain the most optimal possible fuzzy integrator. Based on the results obtained we can say that Hierarchical Genetic Algorithms used to optimize the structure of a fuzzy integrator for MNN responses in face, fingerprint and voice recognition are an efficient optimization technique, through which we can achieve a higher recognition rate compared with non-optimized fuzzy integrators as shown in this chapter.

Chapter 10

Modular Neural Network with Fuzzy Response Integration and Its Optimization Using Genetic Algorithms for Human Recognition Based on Iris, Ear and Voice Biometrics

In this chapter we describe the application of a Modular Neural Network (MNN) for iris, ear and voice recognition for a database of 77 persons. The proposed MNN architecture with which we are working consists of three modules; iris, ear and voice [80]. Each module is divided in other three sub modules. Each sub module contains different information, which, the first 26 individuals are considered in module 1, the following 26 individuals in module 2 and the last 25 in module 3. We considered the integration of each biometric measure separately. Later, we proceed to integrate these modules with a fuzzy integrator [59]. Also, we performed optimization of the modular neural networks and the fuzzy integrators using genetic algorithms, and comparisons were made between optimized results and the results without optimization.

10.1 Introduction Biometrics plays an important role in public security and information security domains. Using various physiological characteristics of the human, such as face, facial thermo grams, fingerprint, iris, retina, hand geometry etc., biometrics accurately identifies each individual and distinguishes one from another [32, 59, 80]. The recognition of people is of great importance, since it allows us to have a greater control about when a person has access to certain information, area, or simply to identify if the person is the one who claims to be [59, 63, 64, 93]. The achieved results indicate that biometric techniques are much more precise and accurate than the traditional techniques. Other than precision, there have always been certain problems which remain associated with the existing traditional techniques. As an example consider possession and knowledge. Both can be shared, stolen, forgotten, duplicated, misplaced or taken away. However the danger is minimized in case of biometric means [59].

P. Melin: Modular Neural Networks and Type-2 Fuzzy Systems, SCI 389,, pp. 127–144. springerlink.com © Springer-Verlag Berlin Heidelberg 2012

128

Chapter 10 MNN with Fuzzy Response Integration and Its Optimization

10.2 Background In this section we present some background of the biometric measurements that were used for this work [59].

10.2.1 Iris The idea of using iris patterns for identification of persons was first proposed in 1936 by the ophthalmologist Frank Burch. However, it was not until 1987, when Leonard Flom and Aran Safir, American ophthalmologists, patented the concept of Burch [80]. Their interest in developing this system, brought them to connect with John G. Daugman, then professor of Harvard University so that he developed the algorithms necessary to perform the biometric recognition through the iris pattern. These algorithms, patented by Daugman in 1994, are the basis of all iris recognition systems that exist today [8, 40, 80]. There are various works undertaken for iris recognition, as the work performed by Ahmad M. Sarhan, which uses neural networks and Discrete Cosine Transform for the identification based on iris.

10.2.2 Ear The possibility of using the appearance of the ear as a means of personal identification was recognized and defended in about 1890 by the French criminologist Alphonse Bertillon. The human recognition based on ear is of particular interest because is not affected by environmental factors such as mood, health, and clothing, as well as not affected by aging, making it more suitable for long-term identification compared with other measures, such as the face [9, 59, 80]. One of the best known works is that of Carreira Perpiñán in 1995, where it uses artificial neural networks (ANN) for feature extraction. Other work is of Ali, Javed and Basit, where they proposed a new ear recognition method using Wavelets.

10.2.3 Voice Automatic speech recognition (ASR) has been one of the most successful technologies allowing the man-machine communications to request some information from it or to request to carry out some given task using the natural oral communication. The artificial intelligence field has contributed in a remarkable way to the development of ASR algorithms [32, 59, 93].

10.3 Basic Concepts

129

The automatic speech recognition is a very complex task due to the large amount of variations involved in it, such as intonation, voice level, health condition and fatigue, and so forth. Therefore, in the automatic speech recognition system, for specific or general tasks, there is an immense amount of aspects to be taken into account. This fact has contributed to increase the interest in this field, and as a consequence, several ASR algorithms have been proposed during the last 60 years. Over the past two decades, voice recognition technology has developed to the point of becoming real-time, continuous speech systems that augment command, security, and content creation tasks with exceptionally high accuracy. In a work more recently presented neural networks and type-2 fuzzy logic were used.

10.3 Basic Concepts In this section we present some basic concepts needed to understand better what has been done in this research work.

10.3.1 Modular Neural Network Neural networks, with their remarkable ability to derive meaning from complicated or imprecise data, can be used to extract patterns and detect trends that are too complex to be noticed by either humans or other computer techniques. A trained neural network can be thought of as an "expert" in the category of information it has been given to analyze. This expert can then be used to provide projections given new situations of interest and answer "what if" questions [57]. A neural network is said to be modular if the computation performed by the network can be decomposed into two or more modules (subsystems) that operate on distinct inputs without communicating with each other [59]. The modular neural networks are comprised of modules which can be categorized on the basis of both distinct structure and functionality which are integrated together via an integrating unit. With functional categorization, each module is a neural network which carries out a distinct identifiable subtask. Also, using this approach different types of learning algorithms can be combined in a seamless fashion.

10.3.2 Fuzzy Logic Fuzzy logic is an area of soft computing that enables a computer system to reason with uncertainty. A fuzzy system consists of a set of if-then rules defined over fuzzy sets [103]. Fuzzy sets generalize the concept of a traditional set by allowing the membership degree to be any value between 0 and 1. This corresponds, in the real world, to many situations where it is difficult to decide in an unambiguous manner if something belongs or not to a specific class. Fuzzy logic is a useful tool

130

Chapter 10 MNN with Fuzzy Response Integration and Its Optimization

for modeling complex systems and deriving useful fuzzy relations or rules. However, it is often difficult for human experts to define the fuzzy sets and fuzzy rules used by these systems. The basic structure of a fuzzy inference system consists of three conceptual components: a rule base, which contains a selection of fuzzy rules, a database (or dictionary) which defines the membership functions used in the rules, and a reasoning mechanism that performs the inference procedure [12, 13, 15, 20, 37, 59].

10.3.3 Genetic Algorithms Genetic Algorithms (GAs) are nondeterministic methods that employ cross-over and mutation operators for deriving offsprings [24, 25]. GAs work by maintaining a constant-sized population of candidate solutions, known as individuals (chromosomes). The power of a genetic algorithm lies in its ability to exploit, in a highly efficient manner, information about a large number of individuals. The search underlying GAs is such that breadth and depth (exploration and exploitation ) are balanced according to the observed performance ( fitness) of the individuals evolved so far. By allocating more reproductive occurrences to above average individual solutions, the overall effect is to increase the population’s average fitness. GAs have proven to be a useful method for optimizing the membership functions of the fuzzy sets used by these fuzzy systems [24].

10.4 Proposed Method and Results This section describes the methodology used and results obtained in this research work.

10.4.1 Methodology The methodology used in this work, consists of the following steps [80]: 1. 2. 3. 4.

Obtain the databases. Programming in Matlab the modular neural network for each of the biometric measurements. Develop the fuzzy integrators. Develop the genetic algorithms.

10.4.2 Databases and Pre-processing The databases that we used and their pre-processing are described below.

10.4 Proposed Method and Results

131

10.4.2.1 Database of Iris We used a database of human Iris from the Institute of Automation of the Chinese Academy of Sciences [80]. It consists of 14 images (7 for each eye) per person, and it consists of 99 persons. The image dimensions are 320 x 280, JPEG format. Only the first 77 persons were used. 8 images were used for training and 6 for testing (see Fig. 10.1).

Fig. 10.1 Examples of the human iris images from CASIA database

In the case of the Iris the following preprocessing steps were performed: • • • •

The coordinates and radius of the iris and pupil using the method developed by Masek and Kovesi are obtained [80]. A cut in the Iris is made. Resize of the new image to 21-21 Convert images from vector to matrix.

10.4.2.2 Database of Ear We used a database of Ear Recognition Laboratory from the University of Science & Technology Beijing (USTB). It consists of 4 images (of one ear) per person, and it is consists of 77 persons[80]. The image dimensions are 300 x 400, BMP format. 3 images were used for training and 1 for testing, and cross-validation was used (see Fig. 10.2).

Fig. 10.2 Examples of Ear Recognition Laboratory from the University of Science & Technology Beijing (USTB)

132

Chapter 10 MNN with Fuzzy Response Integration and Its Optimization

In the case of the Ear for processing the following steps were performed: • • •

A cut of the ear is performed manually Resize of the new image to 132-91 Convert images from vector to matrix.

10.4.2.3 Database of Voice In the case of voice, the database was made in Tijuana Institute of Technology, and it is consist of 7 voice samples (of 77 persons), WAV format. 7 voice samples were used for training and 3 for testing. The word that they said in Spanish was "ACCESAR". To preprocess the voice the Mel Frequency Cepstral Coefficients were used [32, 93].

10.4.3

Architecture and Results of the Modular Neural Network

The general architecture of the modular neural network is shown in Figure 10.3, we can see that consists of 3 modules, one for each biometric measure, each module is divided in other three sub modules. Each sub module contains different information, which is, the first 26 individuals are in module 1, the following 26 individuals in module 2 and the last 25 in module 3. We considered the integration of each biometric measure separately [80].

Fig. 10.3 The general architecture of the modular neural network

10.4 Proposed Method and Results

133

To make simulations, we did perform experiments with 3 types of learning algorithms: gradient descent with scaled conjugate gradient (SCG), gradient descent with adaptive learning and momentum (GDX) and adaptive learning (GDA), also we varied the number of neurons in the first and second hidden layer. 10.4.3.1 Results of the Iris As noted, 8 images were used for training and 6 for testing (per person), in total 462 images were used for testing. To integrate is used Gating Network. We can see in Table 10.1, the best trainings that were obtained. Where training EI2 and training EI4 were having the best recognition rate (97.19%, 448 images from 462), talking of training EI2, in the module 1, the result was with 150 neurons in the first hidden layer and 110 in the second hidden layer, the learning algorithm was adaptive learning (GDA), with an identification rate of 96.15% (150/156), in the module 2, the result was with 150 neurons in the first hidden layer and 118 in the second hidden layer, the learning algorithm was gradient descent with scaled conjugate gradient (SCG), with an identification rate of 96.79% (151/156), and in the module 3, the result was with 150 neurons in the first hidden layer and 110 in the second hidden layer, the learning algorithm was adaptive learning (GDA), with an identification rate of 98.67% (148/150). Table 10.1 The best results of iris

10.4.3.2 Results of the Ear As noted, 3 images were used for training and 1 for testing (per person), in total 77 images were used for testing. Cross-validation was used. To integrate is used the winner take it all. We can see in Table 10.2, the best trainings that we obtained. The best average recognition was 89.29%, in the validation 1 the best result was in the training EO3 with 100% (77/77) of recognition, in the validation 2 the best result was in the training EO1 and the training EO3 with 74.02% (57/77) of recognition, in the

134

Chapter 10 MNN with Fuzzy Response Integration and Its Optimization

validation 3 the best result was in the training EO1 with 85.71% (66/77) of recognition, and in the validation 4 the result in all the trainings was 100% (77/77) of recognition. Table 10.2 The best results of ear

10.4.3.3 Results of the Voice As noted, 4 voice samples were used for training and 3 for testing (per person), in total 321 voice samples were used for testing. To integrate results a gating was used. We can see in Table 10.3, the best trainings that we obtained. Where training EV4 had the best recognition rate (94.81%, 219 voice samples from 231), in the module 1, the result was with 190 neurons in the first hidden layer and 110 in the second hidden layer, the learning algorithm was adaptive learning (GDA), with an identification rate of 96.15% (75/78), in the module 2, the result was with 190 neurons in the first hidden layer and 120 in the second hidden layer, the learning algorithm was adaptive learning (GDA), with an identification rate of 94.87% Table 10.3 The best results of voice

10.4 Proposed Method and Results

135

(74/78), and in the module 3, the result was with 190 neurons in the first hidden layer and 110 in the second hidden layer, the learning algorithm was gradient descent with scaled conjugate gradient (SCG), with an identification rate of 93.33% (70/75).

10.4.4 Genetic Algorithm for MNN Optimization and Results With the purpose of increasing the percentage of recognition, it was decided to use a genetic algorithm that allowed us to optimize some parameters of the modular neural networks. In this case we decided to optimize the number of neurons in two hidden layers, type of learning algorithm and the error goal. Figure 10.4 shows the binary chromosome of 34 genes, which was established for optimization of neural networks. 7 genes were established for the first hidden layer and 6 genes for the second hidden layer, to set the final value of neurons, were added 60 and 50 neurons respectively, this is done to prevent that the neural network was training with a number of neurons too small, with respect to the learning algorithm, 2 genes were established, for this value was added an 1 , this for was to have 3 choices of learning algorithms (trainscg, traingdx and traingda), and the error goal was established with 19 genes, with these genes was realized an operation to set the error goal. The fitness function was (see equation 10.1).

Fitness =

Total errors Total test data.

Fig. 10.4 The binary chromosome for the MNN

(10.1)

136

Chapter 10 MNN with Fuzzy Response Integration and Its Optimization

10.4.4.1 Optimized Iris Results We can see in Table 10.4, the best evolution that we had for each module. In module 1, the best evolution was with 165 neurons in the first hidden layer and 78 in the second hidden layer, the learning algorithm was adaptive learning (GDA), goal error of 0.000002, and with an identification rate of 97.44% (152/156). In module 2, the best evolution was with 68 neurons in the first hidden layer and 69 in the second hidden layer, the learning algorithm was gradient descent with scaled conjugate gradient (SCG), goal error of 0.000002, and with an identification rate of 98.08% (153/156). In module 3, the best evolution was with 131 neurons in the first hidden layer and 77 in the second hidden layer, the learning algorithm was adaptive learning (GDA), goal error of 0.000002, and with an identification rate of 100% (150/150). We had a percent of recognition final of 98.48%, this is greater than the percent of recognition non-optimized (97.19%). Table 10.4 The best evolutions for each module for iris

10.4.4.2 Optimized Ear Results We can see in Table 10.5, the best evolution that we had for each module for the validation 2. In module 1, the best evolution was with 126 neurons in the first hidden layer and 86 in the second hidden layer, the learning algorithm was adaptive learning (GDA), goal error of 0.000004, and with an identification rate of 80.76% (21/26). In module 2, the best evolution was with 58 neurons in the first hidden layer and 49 in the second hidden layer, the learning algorithm was gradient descent with gradient descent with adaptive learning and momentum (GDX), goal error of 0.000002, and with an identification rate of 84.61% (22/26). In module 3, the best evolution was with 117 neurons in the first hidden layer and 69 in the second hidden layer, the learning algorithm was gradient descent with scaled conjugate gradient (SCG), goal error of 0.000004, and with an identification rate of 84% (22/25). We had a percent of recognition final of 83.11%, this is greater than the percent of recognition non-optimized (74.02%).

10.4 Proposed Method and Results

137

Table 10.5 The best evolutions for each module for ear (validation 2)

We can see in Table 10.6, the best evolution that we had for each module for the validation 3. In module 1, the best evolution was with 140 neurons in the first hidden layer and 79 in the second hidden layer, the learning algorithm was adaptive learning (GDA), goal error of 0.000002, and with an identification rate of 92.30% (24/26). In module 2, the best evolution was with 92 neurons in the first hidden layer and 49 in the second hidden layer, the learning algorithm was gradient descent with gradient descent with gradient descent with scaled conjugate gradient (SCG), goal error of 0.000004, and with an identification rate of 92.30% (24/26). In module 3, the best evolution was with 63 neurons in the first hidden layer and 53 in the second hidden layer, the learning algorithm was gradient descent with scaled conjugate gradient (SCG), goal error of 0.000002, and with an identification rate of 92% (23/25). We had a percent of recognition final of 92.20%, this is greater than the percent of recognition non-optimized (85.71%). Table 10.6 The best evolutions for each module for ear (validation 3)

10.4.4.3 Optimized Voice Results We can see in Table 10.7, the best evolution that we had for each module. In the module 1, the best evolution was with 177 neurons in the first hidden layer and 86 in the second hidden layer, the learning algorithm was gradient descent with scaled conjugate gradient (SCG), goal error of 0.000010, and with an identification rate of 98.72% (77/78). In module 2, the best evolution was with 91 neurons in the first hidden layer and 113 in the second hidden layer, the learning algorithm was gradient descent with gradient descent with gradient descent with adaptive learning and momentum (GDX), goal error of 0.000003, and with an identification rate of 96.15% (75/78). In module 3, the best evolution was with 150 neurons in

138

Chapter 10 MNN with Fuzzy Response Integration and Its Optimization

the first hidden layer and 87 in the second hidden layer, the learning algorithm was adaptive learning (GDA), goal error of 0.000012, and with an identification rate of 94.67% (71/75). Table 10.7 The best evolutions for each module for voice

10.4.5 Fuzzy Integration 7 cases were established for combining different trainings of iris, ear and voice, optimized and non-optimized. To combine the responses of the different biometric measurements, we established 2 fuzzy integrators; the first (see Fig. 10.5) ; of Mamdani type with 27 rules and trapezoidal membership functions, the second (see Fig. 10.6); of Mamdani type with 23 rules and Gaussian membership functions. The fuzzy systems have 3 input variables (one for each biometric measure) and 1 output variable, there are 3 membership functions for each variable (ABaja, Amedia and AAlta).

Fig. 10.5 Fuzzy Integrator # 1

10.4 Proposed Method and Results

139

Fig. 10.6 Fuzzy Integrator # 2

10.4.5.1 Fuzzy Integration Results Using the Fuzzy Integrator # 1 We can see in Table 10.8 fuzzy integration results using the fuzzy integrator # 1. Table 10.8 Fuzzy integration results using the fuzzy integrator # 1

10.4.5.2 Fuzzy Integration Results Using the Fuzzy Integrator # 2 We can see in table 10.9 fuzzy integration results using the fuzzy integrator # 2.

140

Chapter 10 MNN with Fuzzy Response Integration and Its Optimization Table 10.9 Fuzzy integration results using the fuzzy integrator # 2

10.4.5.3 Comparison between Both Fuzzy Integrators We can see in Figure 10.7 the comparison between both fuzzy integrators and observed a variation in the percentage of recognition, in most cases the recognition rate was increased using the fuzzy integrator # 2.

Fig. 10.7 Comparison between both fuzzy integrators

10.4 Proposed Method and Results

141

10.4.6 Genetic Algorithm for Fuzzy Integrator and Results With the purpose to increase even more the percentage of recognition, was decided perform a genetic algorithm that allowed us to optimize the fuzzy integrator in the following aspects; • Its type of system (mamdani or sugeno) • Type of membership functions (trapezoidal and GBell) • Parameters of the membership functions. Were worked with type-1 fuzzy logic and the fitness function was: Fitness =

Total errors Total test data.

(10.2)

The chromosome contains 100 genes, the first 13 are binary, genes 14 to 100 are real, its value is between 0 and 1. Is described below: • • • • • • • • • • • • • • • • •

Gene 1: type of systems (Mamdani or Sugeno) Gene 2: Type of membership function for Iris (ABaja) Gene 3: Type of membership function for Iris (AMedia) Gene 4: Type of membership function for Iris (AAlta) Gene 5: Type of membership function for Ear (ABaja) Gene 6: Type of membership function for Ear (AMedia) Gene 7: Type of membership function for Ear (AAlta) Gene 8: Type of membership function for Voice (ABaja) Gene 9: Type of membership function for Voice (AMedia) Gene 10: Type of membership function for Voice (AAlta) Gene 11: Type of membership function for Output (ABaja) (if type of system is Mamdani) Gene 12: Type of membership function for Output (AMedia) (if type of system is Mamdani) Gene 13: Type of membership function for Output (AAlta) (if type of system is Mamdani) Gene 14: Output Value (ABaja) (if type system is Sugeno) Gene 15: Output Value (AMedia) (if type system is Sugeno) Gene 16: Output Value (AAlta) (if type system is Sugeno) Gene 17 to 100: parameters of membership functions.

10.4.6.1 Optimized Fuzzy Integrator Results We can see in table 10.10 the optimized Fuzzy Integrator Results.

142

Chapter 10 MNN with Fuzzy Response Integration and Its Optimization

Table 10.10 Optimized fuzzy integrator results

The genetic algorithm created different fuzzy integrators, for example in Figure 10.8 we can see the best fuzzy integrator for case 3, in this case the type of system was Sugeno. For case 4 we can see the best fuzzy integrator created in Figure 10.9.

Fig. 10.8 The best fuzzy integrator for the case 3

10.4 Proposed Method and Results

143

Fig. 10.9 The best fuzzy integrator for the case 4

10.4.6.2 Comparison between Fuzzy Integrators We can see in Table 10.11 the comparison between the different fuzzy integrators and observe that the best percentages of recognition are for the optimized fuzzy integrators in all cases. Table 10.11 The comparison between fuzzy integrators

144

Chapter 10 MNN with Fuzzy Response Integration and Its Optimization

10.5 Summary We performed the human recognition based on iris, ear and voice biometrics, based on modular neural networks, and for combining their responses we used type-1 fuzzy logic. Also, we used genetic algorithms for optimizing the modular neural network and the fuzzy integrator (in some of their parameters), and thus the percentage of recognition was increased. With the results obtained, we can see that the genetic algorithms are of great help to find the optimal architectures. We will continue working on these cases of recognition, the rules will be optimized and we will work with type-2 fuzzy logic [12, 13, 14, 15, 19, 20], which will surely help us to increase the percentage of recognition.

Chapter 11

A Comparative Study of Type-2 Fuzzy System Optimization Based on Parameter Uncertainty of Membership Functions

A comparative study of type-2 fuzzy inference systems optimization as an integration method of Modular Neural Networks (MNNs) is presented [32]. The optimization method for type-2 fuzzy systems is based on the footprint of uncertainty (FOU) of the membership functions. We use different benchmark problems to test the optimization method for the fuzzy systems [34, 35]. First, we tested the methodology by manually incrementing the percentage in the FOU, later we apply a Genetic Algorithm to find the optimal type-2 fuzzy system. We show the comparative results obtained for the benchmark problems.

11.1 Introduction Biometrics is used for measuring and analyzing a person's unique characteristics [59]. There are two types of biometrics: behavioral and physical. Behavioral biometrics are generally used for verification while physical biometrics can be used for either identification or verification. Examples of biometric measures are: fingerprint, people faces, iris patterns, voice recognition, and shape of the hand signature [8, 9, 32, 64, 93, 107]. Multimodal biometric technology uses more than one biometric identifier to compare the identity of the person [59, 74]. Multimodal biometry is based on using different biometrics characteristics to improve the recognition rate and reliability of the final result of recognition [59, 64]. For this reason, in this chapter, three biometric characteristics of a person are used to achieve a good recognition rate of humans; face, fingerprint and voice. In this chapter we describe an optimization method for the membership functions of type-2 fuzzy systems based on the level of uncertainty [12, 32, 59], in which, the first step is to obtain the optimal type-1 Fuzzy Inference System, which allows us to find the uncertainty of their membership functions using genetic algorithms. This chapter is organized as follows: Section 2 shows an introduction to soft computing techniques, section 3 describes the development of the evolutionary method, section 4 shows fuzzy systems optimization based on the level of uncertainty; section 5 shows simulation results, and section 6 the conclusions.

P. Melin: Modular Neural Networks and Type-2 Fuzzy Systems, SCI 389,, pp. 145–161. springerlink.com © Springer-Verlag Berlin Heidelberg 2012

146

Chapter 11 A Comparative Study of Type-2 Fuzzy System Optimization

11.2 Preliminaries The concept of soft computing, introduced by Lotfi Zadeh in 1991, serves to highlight the emergence of computing methodologies in which the accent is on exploiting the tolerance for imprecision and uncertainty to achieve tractability, robustness and low solution cost [103]. At this juncture, the principal constituents of soft computing are fuzzy logic, neurocomputing, evolutionary computing and probabilistic computing, with the later subsuming belief networks, chaotic systems and parts of learning theory. What is particularly important about soft computing is that it facilitates the use of fuzzy logic, neurocomputing, evolutionary computing and probabilistic computing in combination, leading to the concept of hybrid intelligent systems. Such systems are rapidly growing in importance and visibility. We used three different techniques of soft computing for the optimization method; modular neural networks, type-2 Fuzzy Logic and genetic algorithms [59].

11.2.1 Modular Neural Networks A computational system can be considered to have a modular architecture if it can be split into two or more subsystems in which each individual subsystem evaluates either distinct inputs or the same inputs without communicating with other subsystems [59]. The overall output of the modular system depends on an integration unit, which accepts outputs of the individual subsystems as its inputs and combines them in a predefined fashion to produce the overall output of the system. The modular system design approach has some obvious advantages, like simplicity and economy of design, computational efficiency, fault tolerance and better extendibility. A neural network is said to be modular if the computation performed by the network can be decomposed into two or more modules (subsystems) that operate on distinct inputs without communicating with each other. The outputs of the modules are mediated by an integrating unit that is not permitted to feed information back to the modules. In particular, the integrating unit decides: 1.- How the modules are combined to form the final output of the system, and 2.- Which modules should learn which training patterns.

11.2.2 Type-2 Fuzzy Logic Type-2 Fuzzy Logic can handle uncertainty because it can model and reduce it to the minimum their effects [12, 41]. Also, if all the uncertainties disappear, type-2 fuzzy logic reduces to type-1 fuzzy logic, in the same way that, if the randomness disappears, the probability is reduced to the determinism. Fuzzy sets and fuzzy

11.3 Optimization Method Description

147

logic are the foundation of fuzzy systems, and have been developed looking to model the form as the brain manipulates inexact information. Type-2 fuzzy sets are used to model uncertainty and imprecision [1, 6, 11, 13, 14, 15, 16, 17]; originally they were proposed by Zadeh in 1975 and they are essentially “fuzzy-fuzzy” sets in which the membership degrees are type-1 fuzzy sets [103].

11.2.3 Genetic Algorithms For being able to use a genetic algorithm (GA), one should represent a solution to the problem as a genome (or chromosome) [24]. The genetic algorithm then creates a population of solutions and applies genetic operators such as mutation and crossover to evolve the solutions in order to find the best one. It uses various selection criteria so that it picks the best individuals for mating (and subsequent crossover). The objective function determines the best individual where each individual must represent a complete solution to the problem that you are trying to optimize [25].

11.3 Optimization Method Description Based on the theory described above, a new method for optimization of the membership functions of type-2 fuzzy systems based on the level of uncertainty, applied to the integration of response in Modular Neural Networks (MNN), is proposed [32, 33, 34, 35]. The goal of the research is the development of a method to optimize the modular neural networks including the module of response integration. In particular, the method includes optimizing the membership functions of the type-2 fuzzy system that performs the integration of responses in the modular network, as well as the optimization of the complete architecture of the modular neural network, which consists of the number of modules, layers and neurons. The purpose of obtaining the optimal architecture is to obtain better recognition rates and improve the efficiency of the hybrid system of pattern recognition. In Figure 11.1 we see the specific points to the general scheme of the method. First, the optimization of the complete architecture of the modular neural network, as number of modules, layers and neurons is performed. Later, the genetic algorithm for optimizing the method of integration, where this method of integration uses type-2 fuzzy inference systems [12, 18]. After obtaining the optimization of the architecture of the MNN with Application in Multimodal Biometry and the algorithm of the integration method, then a method of optimization is implemented and then the validation of results and conclusions for comparing statistically the method.

148

Chapter 11 A Comparative Study of Type-2 Fuzzy System Optimization

Optimization Method Optimization the Architecture of MNN Optimization the Method of Integration based of uncertainty Module

Inputs

Module

Evaluate Response Method of Integration Type-2 FIS

Output

Module

Fig. 11.1 General scheme of the optimization method

11.4 Fuzzy Systems Optimization Based on the Level of Uncertainty For the Fuzzy Inference Systems optimization (which are the methods of response integration in the case of MNNs) based in level of uncertainty; the first step is to obtain the optimal type-1 Fuzzy Inference System, which allows us to find the uncertainty of their membership functions. In this case we used a parameter to represent (epsilon) the uncertainty. For the next step we used three cases to manage the using genetic algorithms for all situations [33, 34, 35]. We show in Figures 11.2, 11.3 and 11.4 the graphical representation of differents cases of uncertainty in the membership functions. Case 1. Equal values of uncertainty for all membership function.

Fig. 11.2 Graphical representation of equal values of uncertainty for all membership functions

11.4 Fuzzy Systems Optimization Based on the Level of Uncertainty

149

Case 2. Different values of uncertainty in each input.

Fig. 11.3 Graphical representation of different values of uncertainty in each input

Case 3. Different values of uncertainty for each membership function.

Fig. 11.4 Graphical representation of different value of uncertainty for each membership function

We proposed an index to evaluate the fitness of the fuzzy systems.

Index = N 1+nɛ

150

Chapter 11 A Comparative Study of Type-2 Fuzzy System Optimization

Where N = Data inside the interval output between the total output data; n = Number of fuzzy systems and = Value of increased uncertainty of the membership functions. We consider the optimization of type-2 fuzzy systems based on the above mentioned three cases. As a consequence our proposed method is illustrated in Fig. 11.5.

Type-2 Fuzzy Inference Systems as Integration Method

Fig. 11.5 Representation of Optimization to Type-2 Fuzzy Inference Systems based on level of uncertainty

Figure 11.5 represents the proposed optimization method. First, based on a type-1 fuzzy system we use genetic algorithms to optimize the type-2 membership functions of these fuzzy systems taking into account the three cases described above. Then we can compare them to decide which is best and gives results so get an optimal type-2 fuzzy system.

11.5 Simulation Results The simulation results for two benchmark problems are presented in this section.

11.5 Simulation Results

151

11.5.1 Adaptive Noise Cancellation The first problem is of known as the Adaptive Noise Cancellation (ANC) of a transmitted signal. The goal of ANC is to filter out an interference component by identifying a linear model between a measurable noise source and the corresponding unmeasurable interference [37]. ANC using linear filters has been used successfully in real-world applications such as interference canceling in electro cardiograms, echo elimination on long-distance telephone transmission lines, and antenna sidelobe interference canceling. The original data base contains 601 data points. In Figure 11.6 we can see how the inputs in Fuzzy Inference Systems affect the output, to find the largest number of Data inside the interval output.

Fig. 11.6 Data inside the interval output, where red line indicates the upper value in the interval output, green line indicates the lower value in the interval output and blue line indicates the original output value of the fuzzy inference system.

Jang et al. [37] found the best type-1 fuzzy inference system with two membership function for each input. Based on those results, type-2 fuzzy inference systems are obtained by manually disturbing its value with an epsilon in the membership functions of the two most significant variables and reducing the input data for training. For the first case, 20 fuzzy systems were obtained by increasing the

152

Chapter 11 A Comparative Study of Type-2 Fuzzy System Optimization

values of equal epsilon by 10%. For case two, where the increase of epsilon is different per input, 40% increase for the first input and 20% in second input; and for the third case, where the increase of epsilon is different per membership function, 15% 30% 20% 5% increases, respectively. Table 11.1 shows the simulations results with manually increase for Equal Value of Uncertainty for all Membership functions, for the benchmark problem. Table 11.1 Equal value of uncertainty with manually increase (10% Increase)

No. Base 1 2 3 4

N 0.3794 0.5441 0.7388 0.8170 0.8353

n 20 20 20 20 20

Epsilon ( ɛ ) 0 0.1 0.2 0.3 0.4

Index 0.13 0.18 0.25 0.27 0.28

Data inside the interval output 228/601 327/601 444/601 491/601 502/601

5

0.8386

20

0.5

0.28

504/601

6

0.8469

20

0.6

0.28

509/601

7

0.8552

20

0.7

0.29

514/601

8

0.8686

20

0.8

0.29

522/601

9

0.8752

20

0.9

0.29

526/601

10

0.9085

20

1.0

0.30

546/601

11

0.9168

20

1.1

0.31

551/601

12

0.9218

20

1.2

0.31

554/601

13

0.9251

20

1.3

0.31

556/601

14

0.9251

20

1.4

0.31

556/601

15

0.9268

20

1.5

0.31

557/601

16

0.9285

20

1.6

0.31

558/601

17

0.9285

20

1.7

0.31

558/601

18

0.9285

20

1.8

0.31

558/601

19

0.9285

20

1.9

0.31

558/601

20

0.9285

20

2.0

0.31

558/601

11.5 Simulation Results

153

Table 11.2, shows the simulations results with manually increase for different values of uncertainty in each input for the benchmark problem. Table 11.2 Different value of uncertainty in each input with manually increase (40% Increase first input and 20% in second input)

No.

N

n

Epsilon ( ɛ )

Index

Data inside the interval output

Base

0.3794

20

0

0.05

228/601

1

0.8236

20

0.3

0.12

495/601

2

0.8453

20

0.6

0.12

508/601

3

0.9185

20

0.9

0.13

552/601

4

0.9251

20

1.2

0.13

556/601

5

0.9285

20

1.5

0.13

558/601

6

0.9285

20

1.8

0.13

558/601

7

0.9285

20

2.1

0.13

558/601

8

0.9285

20

2.4

0.13

558/601

9

0.9285

20

2.7

0.13

558/601

10

0.9285

20

3

0.13

558/601

11

0.9285

20

3.3

0.13

558/601

12

0.9285

20

3.6

0.13

558/601

13

0.9285

20

3.9

0.13

558/601

14

0.9285

20

4.2

0.13

558/601

15

0.9285

20

4.5

0.13

558/601

16

0.9285

20

4.8

0.13

558/601

17

0.9285

20

5.1

0.13

558/601

18

0.9285

20

5.4

0.13

558/601

19

0.9285

20

5.7

0.13

558/601

20

0.9285

20

6.0

0.13

558/601

154

Chapter 11 A Comparative Study of Type-2 Fuzzy System Optimization

Table 11.3, shows the simulations results with manually increase for different value of uncertainty in each Membership function (15% 30% 20% 5% Increase respectively) for the benchmark problem. Table 11.3 Different values of uncertainty in each Membership function with manually increase (15% 30% 20% 5% Increase respectively)

No.

N

n

Epsilon ( ɛ )

Index

Data inside the interval output

Base

0.3794

20

0

0.05

228/601

1

0.7238

20

0.35

0.09

435/601

2

0.8186

20

0.7

0.10

492/601

3

0.8303

20

1.05

0.10

499/601

4

0.8436

20

1.4

0.11

507/601

5

0.8619

20

1.75

0.11

518/601

6

0.8852

20

2.1

0.11

532/601

7

0.9085

20

2.45

0.11

546/601

8

0.9218

20

2.8

0.12

554/601

9

0.9268

20

3.15

0.12

557/601

10

0.9285

20

3.5

0.12

558/601

11

0.9285

20

3.85

0.12

558/601

12

0.9285

20

4.2

0.12

558/601

13

0.9285

20

4.55

0.12

558/601

14

0.9285

20

4.9

0.12

558/601

15

0.9285

20

5.25

0.12

558/601

16

0.9285

20

5.6

0.12

558/601

17

0.9285

20

5.95

0.12

558/601

18

0.9285

20

6.3

0.12

558/601

19

0.9285

20

6.65

0.12

558/601

20

0.9285

20

7

0.12

558/601

11.5 Simulation Results

155

Figure 11.7 shows the results graphically for benchmark problem ANC, in which we can see the results for the three cases. We can observe that in case two, where the increase of epsilon is different for each input, we get the best index with less increase of uncertainty in their membership functions.

Fig. 11.7 Simulation results for the three cases for benchmark problem ANC

Later, we used Genetic Algorithms to optimize the epsilon value in the membership functions of a type-2 fuzzy inference system. Next we show the simulation results. Table 11.4, shows the simulations results with a GA for Equal Value of Uncertainty for all Membership Functions for the benchmark problem.

156

Chapter 11 A Comparative Study of Type-2 Fuzzy System Optimization Table 11.4 Equal Value of Uncertainty

No.

Individuals Generation

Mutation

Crossover

Time Execution

Epsilon

1 2

80 150

100 200

0.1 0.3

0.4 0.7

1:50:28 6:36:42

1.00183 1

Data inside the interval output 524/601 525/601

3

100

100

0.001

0.5

2:13:45

1.02672

526/601

4

40

50

0.8

0.9

0:55:31

1

529/601

5

30

30

0.5

0.6

0:12:26

1.00441

530/601

6

50

50

1

1

0:33:36

1.00355

530/601

7

60

30

0.4

0.85

0:46:34

1

538/601

8

35

30

0.95

1

0:26:21

1

552/601

9

25

30

0.8

1

0:19:02

1

556/601

10

15

100

1

1

0:20:36

1

564/601

11

10

100

0.9

1

0:13:20

1

565/601

12

10

30

0.6

0.8

0:08:18

1.502

579/601

13

5

30

0.7

0.9

0:04:04

2

597/601

14

20

30

0.5

0.7

0:16:09

2

600/601

15

15

30

0.35

0.85

0:12:11

3.4934

601/601

Table 11.5, shows the simulations results with a GA for different values of uncertainty in each input for the benchmark problem. Table 11.5 Different value of uncertainty in each input No.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Individuals Generation

80 150 100 40 30 50 60 35 25 15 10 10 5 20 15

100 200 100 50 30 50 30 30 30 100 100 30 30 30 30

Mutation

0.1 0.3 0.001 0.8 0.5 1 0.4 0.95 0.8 1 0.9 0.6 0.7 0.5 0.35

Crossover Time Exe- Epsilon cution First Input

0.4 0.7 0.5 0.9 0.6 1 0.85 1 1 1 1 0.8 0.9 0.7 0.85

0:50:58 3:09:31 1:08:40 0:25:55 0:11:48 0:33:17 0:15:25 0:13:28 0:09:44 0:10:09 0:06:37 0:04:07 0:01:57 0:07:56 0:05:45

1.500 2.000 1.505 1.300 2.000 2.064 2.000 2.000 1.500 2.000 1.500 1.000 1.001 2.000 1.000

Epsilon Data inSecond side the inInput terval output

1.500 1.663 1.501 2.693 1.000 1.061 3.459 2.000 2.500 1.167 1.520 1.000 1.000 1.534 1.500

584/601 597/601 580/601 572/601 598/601 599/601 597/601 597/601 597/601 599/601 588/601 593/601 582/601 601/601 567/601

11.5 Simulation Results

157

Table 11.6, shows the simulations results with a GA for different values of uncertainty in each Membership function for the benchmark problem. Table 11.6 Different value of uncertainty in each Membership function No.

Indi- Genviduals eration

Muta- Cross- Time Exetion over cution

Epsilon Epsilon Epsilon Epsilon Data in1th MF 2th MF 1th MF 2th MF side the 1th Input 1th Input 2th Input 2th Input interval output

1

80

100

0.1

0.4

2:01:39

2.000

1.001

1.000

1.000

592/601

2

150

150

0.3

0.7

5:34:46

1.500

2.000

3.981

3.713

597/601

3

100

100

0.001

0.5

2:16:09

1.510

2.048

3.906

3.571

593/601

4

40

50

0.8

0.9

0:24:59

2.000

1.588

3.237

2.332

598/601

5

30

30

0.5

0.6

0:11:23

1.500

2.000

3.747

3.500

601/601

6

50

50

1

1

0:33:06

2.000

1.000

1.009

1.000

595/601

7

60

30

0.4

0.85

0:23:07

2.000

1.000

1.012

1.047

592/601

8

35

30

0.95

1

0:14:47

4.000

3.860

3.519

4.000

601/601

9

25

30

0.8

1

0:09:40

2.003

3.625

4.000

2.600

601/601

10

15

100

1

1

0:20:56

1.000

1.000

1.025

1.000

560/601

11

10

100

0.9

1

0:13:15

2.031

2.000

3.465

1.830

576/601

12

10

30

0.6

0.8

0:04:00

1.006

2.000

3.252

2.725

601/601

13

5

30

0.7

0.9

0:01:56

1.047

1.698

1.640

1.523

600/601

14 15

20 15

30 30

0.5 0.35

0.7 0.85

23:07:16 0:05:58

3.963 1.000

4.000 2.943

1.979 3.997

1.981 3.594

601/601 601/601

11.5.2 MPG Benchmark Problem The second benchmark problem is of known as automobile MPG (Miles per Gallon) prediction. This problem is a typical nonlinear regression problem, in which several attributes (inputs variables) as number of cylinders, weight, model years, etc., are used to predict another continuous attribute (output variable) in this case MPG [37]. The original data base contains 384 data points, the first 192 were used for training data and the other 192 for testing. Jang et al. [37], found the best model takes "weight" and "model year" as the inputs variables for the type-1 fuzzy inference system with two membership function for input. Based on those results to get a type-2 fuzzy inference system as a base to disturb its value with an epsilon in the membership functions of the two most significant variables and reducing the input data for training, the following results are obtained.

158

Chapter 11 A Comparative Study of Type-2 Fuzzy System Optimization

For the first case, 20 fuzzy systems were obtained by increasing the values of equal epsilon by 10%. For case two where the increase of epsilon is different per input, 30% increase for the first input and 10% in second input; and for the third case where the increase of epsilon is different per membership function 15% 5% 10% 20% increase respectively. Table 11.7 shows the simulations results with manually increase for Equal Value of Uncertainty for all Membership functions, for the second benchmark problem. Table 11.7 Equal value of uncertainty with manually increase (10% Increase)

No. Base 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

N 0.3418 0.602 0.7296 0.7857 0.7908 0.8061 0.8265 0.8469 0.8571 0.8571 0.8622 0.8622 0.8622 0.8622 0.8622 0.8622 0.8622 0.8622 0.8622 0.8673 0.8673

n 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20

Epsilon ( ɛ ) 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2.0

Index 0.11 0.2 0.24 0.26 0.26 0.27 0.28 0.28 0.29 0.29 0.29 0.29 0.29 0.29 0.29 0.29 0.29 0.29 0.29 0.29 0.29

Data inside the interval output 67/192 118/192 143/192 154/192 155/192 158/192 162/192 166/192 168/192 168/192 169/192 169/192 169/192 169/192 169/192 169/192 169/192 169/192 169/192 170/192 170/192

Table 11.8, shows the simulations results with manually increase for different values of uncertainty in each input for the benchmark problem.

11.5 Simulation Results

159

Table 11.8 Different values of uncertainty in each input with manually increase (30% Increase first input and 10% in second input)

No.

N

n

Epsilon ( ɛ )

Index

Data inside the interval output

Base

0.3418

20

0

0.07

67/192

1

0.6531

20

0.2

0.13

128/192

2

0.7602

20

0.4

0.15

149/192

3

0.8214

20

0.6

0.16

161/192

4

0.852

20

0.8

0.17

167/192

5

0.8571

20

1.0

0.17

168/192

6

0.8622

20

1.2

0.17

169/192

7

0.8622

20

1.4

0.17

169/192

8

0.8622

20

1.6

0.17

169/192

9

0.8622

20

1.8

0.17

169/192

10

0.8673

20

2.0

0.17

170/192

11

0.8827

20

2.2

0.18

173/192

12

0.8929

20

2.4

0.18

175/192

13

0.8827

20

2.6

0.18

173/192

14

0.8622

20

2.8

0.17

169/192

15

0.8469

20

3.0

0.17

166/192

16

0.8469

20

3.2

0.17

166/192

17

0.8469

20

3.4

0.17

166/192

18

0.8469

20

3.6

0.17

166/192

19

0.8469

20

3.8

0.17

166/192

20

0.8469

20

4.0

0.17

166/192

Table 11.9, shows the simulations results with manually increase for different values of uncertainty in each Membership function (15% 5% 10% 20% Increase respectively) for the benchmark problem.

160

Chapter 11 A Comparative Study of Type-2 Fuzzy System Optimization

Table 11.9 Different values of uncertainty in each Membership function with manually increase (15% 5% 10% 20% Increase respectively)

No. Base

N 0.3418

n 20

Epsilon ( ɛ )

Index

Data inside the interval output

0

0.1

67/192

1

0.6582

20

0.125

0.19

129/192

2

0.7602

20

0.25

0.22

149/192

3

0.7959

20

0.375

0.23

156/192

4

0.8112

20

0.5

0.23

159/192

5

0.8418

20

0.625

0.24

165/192

6

0.852

20

0.75

0.24

167/192

7

0.8571

20

0.875

0.24

168/192

8

0.8571

20

1

0.24

168/192

9

0.8571

20

1.125

0.24

168/192

10

0.8622

20

1.25

0.25

169/192

11

0.8622

20

1.375

0.25

169/192

12

0.8622

20

1.5

0.25

169/192

13

0.8622

20

1.625

0.25

169/192

14

0.8724

20

1.75

0.25

171/192

15

0.8724

20

1.875

0.25

171/192

16

0.8878

20

2

0.25

174/192

17

0.8673

20

2.125

0.25

170/192

18

0.8571

20

2.25

0.24

168/192

19

0.8469

20

2.375

0.24

166/192

20

0.8469

20

0.125

0.24

166/192

Figure 11.8 shows the results graphically for benchmark problem MPG, in which we can see the results for the three cases. We can observe that in case two, where the increase of epsilon is different for each input, we get the best index with less increase of uncertainty in their membership functions.

11.6 Summary

161

Fig. 11.8 Simulation results for the three cases for benchmark problem MPG

11.6 Summary We presented in this chapter a description of an optimization method for the membership functions of type-2 fuzzy systems based on the level of uncertainty. Simulation results for Adaptive Noise Cancellation benchmark problems are presented. For this problem we can see that varying the uncertainty of the membership functions with the GA changes the fitness of the fuzzy systems.

Chapter 12

Neural Network Optimization for the Recognition of Persons Using the Iris Biometric Measure

This chapter describes the application of modular neural network architecture for the recognition of persons using the human iris images [80]; the iris database was obtained from the Institute of Automation of the Academy of Sciences China (CASIA). We show the results of testing the modular neural network, its optimization using genetic algorithms and the integration with the methods of gating network, type-1 fuzzy integration, and fuzzy integration optimized by genetic algorithms. Simulation results show a good identification using the fuzzy integrators and the best structure found by the genetic algorithm.

12.1 Introduction The recognition of persons using biometrics is a problem which has been around for a long time [59]. This problem has been studied more thoroughly in recent years thanks to advances in computational power that have allowed to implement more complex algorithms using different techniques [8, 9, 32, 40, 59, 64, 93, 107]. Traditional systems used in control systems based on magnetic cards, card systems with bar code systems capture key or combination. These systems involve the use of a card must always carry and which is not exempt from being lost, damaged, be stolen or forged, thus security is more vulnerable to failure. For this reason, if you have a more robust and higher reliability can avoid the problems mentioned above. The pattern recognition system using neural networks have been taking great interest [59]. There are different techniques and methods that can be used for feature extraction, and today it is easier to recognize a person by existing biometric methods, and is recognized for its iris, fingerprints, face, recognizable by the voice, signature, hand geometry, ear, vein structure, retina, facial thermography and others that exist [8, 9, 59, 74, 107]. Within the large field of biometrics where you can highlight, fingerprint recognition, retinal and voice, among others, you can highlight the iris recognition as a biometric tool for the recognition of people uniquely and highly accurate [8, 40, 80].

P. Melin: Modular Neural Networks and Type-2 Fuzzy Systems, SCI 389, pp. 163–184. springerlink.com © Springer-Verlag Berlin Heidelberg 2012

164

Chapter 12 Neural Network Optimization for the Recognition of Persons

This chapter presents research on using modular neural networks, with the database (CASIA), to obtain better identification type-1 fuzzy logic integrators. The optimization method for neural network is based on using genetic algorithms, which are essentially a method that creates a population of individuals find the most suitable simulating evolution. This process is based on natural selection by using features such as crossover and mutation to create new individuals [24]. We also present modular neural network architectures and the chromosomes of genetic algorithms with the best parameters found for the network were tested for their performance and operation, and the results of the different integrators are used, such as Gating Network and Type-1 Fuzzy Logic Integrators.

12.2 Methods of Integration There exist a variety of methods of integration or aggregation of information, and we mention some of these methods below [59]. Integration by Gating Network: A decomposition of a learning task into sub tasks learned through the modules of cooperation, the benefits of working with a Gating Network are: best overall performance, reuse of existing patterns, heterogeneity expert classifiers, which need not be the of same type; different features can be used for different classifiers. There are several implementations of the modular neural network, but the most important is by using the Gating Network. In some cases, the gate corresponds to a single neuron to evaluate the performance of the other modules of experts. Other implementations of the Gating Network are based on a neural network trained with a different data set for training the networks of experts. Below in Figure 12.1 we show the scheme of the gating network integrator.

Fig 12.1 Scheme of the gating network integrator

12.3 Iris Image Pre-processing

165

12.3 Iris Image Pre-processing In the pre-processing phase we applied different methods for feature extraction and noise removal [80], in order to extract the region of interest (iris) of the captured image, we also apply some filters to it, this in order to help modular neural network, to obtain a high recognition rate of the images (Figure 12.2).

Fig. 12.2 General diagram of the pre-processing for the database (CASIA)

The iris database obtained from the Institute of Human automation of China Academy of Sciences (CASIA) is shown below [80] (Figure 12.3).

Fig. 12.3 Example of the image of the Iris

166

Chapter 12 Neural Network Optimization for the Recognition of Persons

The first step is to apply an adjustment to the image, Figure 12.4, which is a basic method of intensity changes in images of gray levels. Then apply a median filter, Figure 12.5, which standardizes the rest of the image of the eye. This is so that the edges of the lashes, pupil and iris are more highlighted, so that they can be better identified in the following stages.

Fig. 12.4 Method of adjusting gray level image

Fig. 12.5 Implementation of the median filter to the image of the iris

Note that at this stage the idea to highlight every detail of the image for coding purposes, which only highlights the edges of the area of interest for future extraction. The pupil of the eye appears in the histogram as a well marked a peak in the low values of gray intensity (because the pupil is black). Edge detection is an essential step of pre-processing in many computer vision algorithms. Within this work we implemented one of these methods, which is the Canny detector [65]. The Canny edge detector is a popular method for detecting edges begin to soften an image by convolution with a Gaussian sigma value given. Based on the smoothed image, resulting in both the x direction and are calculated, which in turn are used to calculate the gradient magnitude image. Once the gradient magnitude image is calculated, a process called "maximum deletion" is performed, in which pixels are deleted if not a local maximum. The final step in the Canny edge detector is the operator of hysteresis, in which the pixels are labeled either as borders, edges, not in the middle, this is done on the basis of threshold values. The next step is to consider each of the pixels that are in the middle, they are connected to the edge pixels, and these are marked as edge pixels as well.

12.3 Iris Image Pre-processing

167

The result of this edge detector is a binary image in which white pixels are really approaching the edges of the original image (Figure 12.6).

Fig. 12.6 Image with Canny edges

The labeling of all objects in the image, choose the largest values, which represents the pupil. At the end of this step we have the image of the pupil (black circle) on a white background, as shown below, Figure 12.7.

Fig. 12.7 Image with Canny edges and applying the fill function

The pupil looks like a circle, but due to light effects, reflections, etc. sometimes is clipped or distorted. These errors are corrected in the next stage, which detects the radio and the center of the pupil within the image with great accuracy (Figure 12.8).

Fig. 12.8 Image with the perimeters

168

Chapter 12 Neural Network Optimization for the Recognition of Persons

12.3.1 Removing the Zone of Interest At this stage the goal is to find the vectors of the circle pointing outward and that are perpendicular to the image edge (circle) (see Figure 12.9), by symmetry of the center of the circle for each vector there will be two vectors in opposite directions, the phase between these two vectors should be approximately 180 degrees.

Fig. 12.9 Image with the circle from the pupil

Once white areas in the image are labeled, we obtain the centroids with their respective areas, then get the best of all of them, with their position, and draw a circle for the perimeter after taking as reference the largest iris to depart from there to find the radius of each circle. The second step is applied to find all pairs of vectors that meet the above conditions. The third step is to consider a candidate circle for each pair of vectors; each has its own center. Finally, the correct circle is drawn from all walks of candidates by storing the coordinates of the center circle and its radius, then there is the histogram of the stored values, which are detected most commonly occurring, which correspond to correct circle (Figure 12.10).

Fig. 12.10 Final image cropped according to the highs and lows of the image

12.3 Iris Image Pre-processing

169

Figure 12.11 shows the resulting image, after making a cut to the image, according to the maximum and minimum, as shown in the image, and can be done in 2 ways one is filling the outer parts of the circle, leaving in black and giving us a better appreciation of the center of the iris, and the other way is to let the image with their property and leaving more features in the image for the network (Figure 12.12).

Fig. 12.11 Result of different techniques of vision for the center of the iris

Fig.12.12 Shows the result of preprocessing the person 1 in 2 cases of pre-processing

Once you have all of the database with the pre-processing and before putting each image into the modular neural network, we compressed the images to 320 x 280 (25x25) using a 2D wavelet transform, it was used “symmlet” order 8, with 2 levels of decomposition (Figure 12.13). We then proceeded to vectorize each image within a matrix containing the first 33 people, another array of vectors the following 33 (34-66), and the last 33 people in the same way. (67-99). In the first 33 people have 14 samples, which means that there are 7 pictures or samples of the right eye and 7 samples of the left eye, and we took 8 sample images for training and 6 images to validate that the images will recognize

170

Chapter 12 Neural Network Optimization for the Recognition of Persons

Fig. 12.13 Application of wavelet transform in 2D

according to 8 above. Then you have two arrays of vectors, the first containing the training images and the other the validation matrix, with (8 * 33), 264 vectors in the matrix of training and (6 * 33), 198 images shown in the validation matrix. The same process was done for the following persons (34-66) and (67-99), for each of the modules of the modular artificial neural network.

12.4 Statement of the Problem and Proposed Method We studied several methods of fuzzy integration that can be applied to modular neural networks for recognition of persons using biometrics iris images as well as develop alternative methods for the integration of the network, such as the gating network and the winner is takes all. Figure 12.14 shows the general architecture used in this work.

Fig. 12.14 General Architecture of Modular Neural Network

12.4 Statement of the Problem and Proposed Method

171

The modular neural network consists of 3 modules, each module consists of a multilayer perceptron, which is an artificial neural network (ANN) consisting of multiple layers, this allows you to solve problems that are not linearly separable. The perceptron uses a matrix to represent neural networks and it is a tertiary discriminator that traces its input x (a binary vector) to a single output value f (x) (a single binary value) through the matrix. The matrix equation of the perceptron is defined as follows: (12.1) Where w is a vector of the actual weights is the dot product (which computes a weighted sum), u is the 'threshold', which represents the degree of inhibition of the neuron, is a constant term is not a value to take the input. The value of f (x) (0 or 1) is used to classify x as a positive event or negative event. The threshold can be thought of as offsetting the activation function, or giving a low level of neuron activity performance. The weighted sum of inputs must yield a value greater than u to change the neuron from 0 to 1. Each module is specialized on the corresponding 33 people who were vectorized, considering a particular modular artificial neural network, training is conducted according to the sequence of modules 1st, 2nd and 3rd. The parameters of the MNN are presented on Table 12.1. Table 12.1 Parameters of the Modular Neural Network (MNN)

To find a suitable architecture for each of the 3 modules, a study in an empirical way to know how many neurons are needed and the appropriate learning for the MNN is acceptable according to the type of training used, will depend on the number of neurons used for such training. The integration links the results of each module and the result is divided by the number of elements, as shown in the following equation. Equation for the recognition rate of each module

(12.2)

172

Chapter 12 Neural Network Optimization for the Recognition of Persons

The main advantage of this method is that each module produces different results, which can be helpful because in other cases all the training is leaved to one expert module, which can affect the outcome of integration, taking into account that each module will have different results. The network consists of three modules, the weight distribution is random for the first step of the network, and once this is completed the network will adjust the weights in each of the connections of the layers of the neural network. The results of each module of our modular network with gating network integrator and arbitrary parameters, without pre-processing, (the result of the integration of 3 modules), are shown in Table 12.2. Table 12.2 Results of the Modular Neural Network without preprocessing in the images

Table 12.2 shows some results obtained from various trainings conducted in each of the modules, the best methods that had a high percentage were traingda, traingdx, trainscg they got a percentage above 90%, with 90.18, 90.91 and 90.40 % respectively with a time of 30 seconds 3m, 2m 3m 15 seconds and 44 seconds, with 8000, 5000, and 8000 times respectively, and a target error of 0.00001 for the MNN, these trainings were conducted without pre-processing the image and uncompressed image, this training as well as the results leave us a little unhappy, because there are parameters that have a very large range, such as neurons, where put the neurons that we will never know that many neurons are optimal and / or appropriate to have a low error margin for this reason we choose to use a genetic algorithm to find an appropriate recognition rate, and thus get the kind of training, the number of neurons and hidden layers once we get these parameters take the mean and standard deviation of how many times the genetic algorithm was executed to find a high percentage of recognition. The results of each module of the modular neural network with the gating network integration method and arbitrary parameters in the pre-processing (the result of the integration of the 3 modules) are shown in Table 12.3.

12.4 Statement of the Problem and Proposed Method

173

Table 12.3 Results of the Modular Neural Network with image preprocessing

Table 12.3 shows the results of the performed trainings with image processing and compression, which can be seen that the recognition rates increased by almost 4% with the same parameters of the training performed previously. The methods with high percentage were traingda, traingdx, trainscg with 95.89, 94.97 and 93.33% respectively, with 8000, 5000 and 5000 epochs respectively, and a goal error of the RNA of 0.00001.

12.4.1 Optimization of the Architecture To optimize the network we used a genetic algorithm that allows us to find architecture, see Figure 12.15.

Fig. 12.15 General Architecture of the Optimized Modular Neural Network

174

Chapter 12 Neural Network Optimization for the Recognition of Persons

Once we have seen that the GA has found a good result, we decided to run the GA 10 times to find the standard deviation and average of neurons, methods, and the number of layers for how to behave in training with the GA as shown in Table 12.4. The chromosome is shown in the table below. Table 12.4 Parameters of the chromosome for the GA M1 M2 M3

Layers

L1

…

L3

Method

Neurons

0:….250

…

0:….250

1:4

Layers

L1

…

L3

Method

Neurons

0:…..250

…

0:….250

1:4

Layers

L1

…

L3

Method

Neurons

0:…..250

…

0:….250

1:4

The real chromosome is composed of 3 layers {1, 2, 3}, and each layer is composed of 250 neurons, which ranged over a range from 0 to 250 values and 4 training methods. The training methods are shown in Table 12.5. Table 12.5 Training methods for the GA

TRAINSCG TRAINGDM TRAINGDX TRAINGDA

SELECTED TRAINING METHODS Scaled conjugate Gradient descent with momentum Gradient descent with momentum and adaptive learning factor Gradient descent with adaptive learning factor

12.4.2 Objective Function of the Genetic Algorithm (12.3) The objective function is the same for each of the modules of the neural network, whether we get a different architecture.

12.4.3 Behavior of Neural Network Training These are the results of running 10 times the genetic algorithm (GA) for (Module 1, Table 12.6), (module 2, Table 12.7) and Module 3 Table 12.8.

12.4 Statement of the Problem and Proposed Method

175

Table 12.6 Results of genetic algorithm for module 1

Table 12.7 Results of genetic algorithm for module 2

Table 12.8 Results of genetic algorithm for module 3

Tables 12.6, 12.7 and 12.8 show the results of the trainings performed by the genetic algorithm, with 20 generations, 10 individuals and 10 runs, showing the average in each generation run and the standard deviation of each generation and better error (B / E / GA) found by the genetic algorithm (GA), better training

176

Chapter 12 Neural Network Optimization for the Recognition of Persons

method and duration of the 10 executions. In the following Figures 12.16 and 12.17 we show the convergence of the error, which shows the minimum and / or the best individual in each generation.

Fig. 12.16 Error convergence graph of the best individuals of the GA Module1 and module 2

Fig. 12.17 Error convergence graph of the best individuals of the GA Module 3

Once the 10 runs of the GA are obtained, we get the best architectures of different training (150 per run). The following Tables 12.9, 12.10 and 12.11 show the different architectures that a good percentage obtained in each module. Table 12.9 Best architecture for module 1

12.4 Statement of the Problem and Proposed Method

177

Table 12.10 Best architecture for the module 2

Table 12.11 Best architecture for the module 3

12.4.4 Validation of the Structure of Modular Neural Network These architectures were used to make cross-validation for 7 times, then we created new vectors containing a different image, in order to obtain the best average recognition in each validation. This process is shown in the following Table 12.12. Table 12.12 Results of applying cross-validation for each of the architectures

178

Chapter 12 Neural Network Optimization for the Recognition of Persons

We choose the best architecture that can fit the percentages obtained from cross-validations for each module if we get a complete architecture, as shown in the following table 12.13. Table 12.13 Best architecture obtained by the GA

These are the results of applying 14-fold cross validation the best architecture, using the integrated network of gates. This is shown in the following Table 12.14.

Table 12.14 Results of 14 cross-validations for the best architecture

As shown in Table 12.14, in some cases we obtained 100% of recognition, but only in some modules of the neural network, the highest percentage of recognition was achieved in validation 14 and 1, which on average has a 98.48% of recognition in 14 of the validations. Once you have found the right structure of the RNA, according to the percentage above with the gate network integrator, we chose to implement a fuzzy integrator to a greater degree of uncertainty in the time it takes a activations in the 2 decision to enter the system integrator. Each module contains 2 sub modules that keep the best architecture obtained by the GA, only the 2nd sub module will have a different training method, which will be the 2nd best trainscg, so each sub module of each module will be trained differently to have a diversity of

12.4 Statement of the Problem and Proposed Method

179

results, for the first module would be as follows, module 1 (traingda and trainscg), 2nd module (trainscg and trainscg) and 3rd module (traingda and trainscg), maintaining the same number of neurons. The architecture of the RNA extended with fuzzy integrator is shown below, Figure 12.18.

Fig. 12.18 General Architecture of the extended RNA with the optimized fuzzy integrator

The integrator is a system of type-1 fuzzy inference rules, in this case we worked with the Mamdani type of reasoning with 2 input linguistic variables and one output linguistic variable, with three membership functions. The three membership functions (MF) are of Triangular type used for the input variables (low, medium, high) for MOD1 and MOD2 in the output variable (response) there are 2 membership functions (FM), which are called (MOD1, MOD2) also of triangular type. There were 2 tests performed with different membership functions, in the first case triangular MFs were used and in the second case Gaussian MFs were used, and their value range from 0 to 1. For the fuzzy systems, 9 fuzzy rules were considered, where each FIS (Fuzzy System inference), specializes in each module of the neural network, which will have the option of deciding which is the highest activation according to the 2 Sub modules found in each module, in order to get a better decision in accordance the activation winner. We show in Figures 12.19 and 12.20 the corresponding membership functions of triangular and Gaussian type, respectively.

180

Chapter 12 Neural Network Optimization for the Recognition of Persons

Fig. 12.19 Fuzzy Inference System (FIS) with FMs of triangular type for the input variables left, right output variable

Fig. 12.20 Fuzzy Inference System (FIS) with FMs of Gaussian type for the left input variable, output variable right

The results of validation with the fuzzy Mamdani integrator for the membership functions with triangular and Gaussian types are shown below Tables 12.15 and 12.16. Table 12.15 Results of the fuzzy Mamdani integrator with FMs of triangular type

12.4 Statement of the Problem and Proposed Method

181

Table 12.16 Results of the fuzzy Mamdani integrator with FMs of Gaussian type

The results shown above, in case 1 where the FMs are triangular in validations 1, 8, 10, 11, 12 achieved a rate of 99.66% and on average 99.37% was obtained for the triangular case. In case 2, where we worked with Gaussian membership functions, validation 1 achieved 99.83% recognition, and on average 99.51% was obtained.

12.4.5 Optimizing the Fuzzy Integrator Figure 12.21 shows the extended architecture with fuzzy integrator optimization.

Fig. 12.21 General Architecture of the RNA extended by optimizing the fuzzy integrator with a genetic algorithm

182

Chapter 12 Neural Network Optimization for the Recognition of Persons

In the optimization of the fuzzy integrator we worked with a genetic algorithm (GA) to move the membership functions for the 2 cases, triangular and Gaussian, and thus achieve a higher percentage of recognition. The genetic algorithm parameters for both cases are 50 generations and 10 individuals. The chromosome is shown below for membership functions of triangular form in Figure 12.22, and in Figure 12.23 for the Gaussian type.

Fig. 12.22 GA chromosome for the fuzzy integrator with membership functions (MF) of Triangular type For the fuzzy integrator with triangular MFs, each gene represents parameters of a triangular membership function that has 3 points so we have 3 genes in a range from 0 to 1 at each point and each variable we have 9 genes, this in the input variables and output variable only 6 points with 2 membership functions, in total the chromosome is composed of 72 genes that the genetic algorithm move to find a best fuzzy inference system (FIS).

Fig. 12.23 GA chromosomes for the fuzzy integrator with Gaussian type FMs

12.4 Statement of the Problem and Proposed Method

183

For the fuzzy integrator with Gaussian FMs, each point of each function is a gene, so a membership function has 2 parameters, which are the mean and standard deviation in each of the input variables. We see that we have 16 genes and the output variable has 4 genes, a total of 48 genes with 1, 2 and 3 modules. Figure 12.24 shows the convergence of the error to 0, according to the used objective function, where we can see that, at least 43 generations are needed to find the results for modules 1, 2, 3, giving us the FIS for each module. Figure 12.25 shows the final MFs obtained with the GA for the third module.

Fig. 12.24 Genetic Algorithm Convergence for triangular MFs

Fig. 12.25 Resulting MFs for the 3rd module

Figure 12.26 shows the final MFs obtained with the genetic algorithm for the case of Gaussian MFs for the third module.

Fig. 12.26 Results of the optimization (FIS) for the 3rd module, Gaussian membership functions

184

Chapter 12 Neural Network Optimization for the Recognition of Persons

Once we have optimized the fuzzy inference systems for each of the cases (triangular and Gaussian), their architectures were validated, making 14 crossvalidations, the results are shown below for the Triangular and Gaussian cases (Table 12.17). Table 12.17 Results validate the Mamdani type fuzzy integrator with triangular membership functions

Table 12.17 shows the results of cross-validations with the fuzzy integrator with membership functions of triangular type. The validation that had a high percentage of recognition was validation number three, this was the validation that went into the genetic algorithm, this validation being one of the lowest in percentage in the previous validations, which on average yielded a 99.64% recognition.

12.5 Summary The best result for the recognition of persons using the iris biometric measurement was obtained using a modular neural network architecture with 3 layers in each module, with 116 and 117 neurons in the first hidden layer, 116 and 113 in the 2nd hidden layer, and 112 to 114 neurons in the 2nd hidden layer (as shown in Table 12.13). Since the results were not satisfactory in the integration of fuzzy systems, decided to apply an evolutionary approach to optimize the membership functions of this system. The method chosen to optimize this system integration is a genetic algorithm. After applying the genetic algorithm we can obtain a better recognition rate, so we have better results with the optimized system.

Chapter 13

Optimization of Neural Networks for the Accurate Identification of Persons by Images of the Human Ear as Biometric Measure

Abstract. This chapter describes the application of modular neural network architecture to improve the recognition of persons using Ear images as a biometric measure [80]. The database used was obtained from the University of Science and Technology Beijing (USTB). We show the results obtained with the modular neural network, its optimization using genetic algorithms and their integration using different methods: Winner Takes All (WTA), type-1 fuzzy integration and fuzzy integration optimized by genetic algorithms. The behavior of the simulations shows a good identification, using the appropriate pre-processing, fuzzy integrators and the best structure found by the genetic algorithm.

13.1 Introduction The methods of human identification with possessions, such as credentials, keys, cards, or knowledge as a password, are becoming obsolete because they are slowly being replaced by biometrics whose advanced computer technology to identify individuals has been enhanced in recent years [8, 32, 40, 59, 74, 93, 107]. The ears have long been considered as a possible means of identification [9, 59, 80]. However, only in recent years the experts began to tackle the idea of using the ears as a biometric measure. French criminologist Alphonse Bertillon was the first to recognize the potential biometric human ears. Empirical evidence supporting the uniqueness of the ear was later found in several experimental studies. The ears have attractive properties for personal identification, they have a rich structure which is very complex, and has a variety of classified areas. Its unique structure makes it impossible for two people having the same physiological characteristics, even among twins, which seems consistent with the age of a few months after birth. Clearly, the ears are not affected by the change in facial expressions. The images of the ears can be acquired without the subject's participation and the ears are large enough to be captured from a distance [9, 80].

P. Melin: Modular Neural Networks and Type-2 Fuzzy Systems, SCI 389, pp. 185–204. springerlink.com © Springer-Verlag Berlin Heidelberg 2012

186

Chapter 13 Optimization of Neural Networks for the Accurate Identification

Within the world of biometrics, there are multiple methods of identification based on certain aspects of the human body such as: voice systems, fingerprints, hand geometry, handwriting verification and shape of the ear, among others [8, 9, 59, 64, 65, 74, 107]. This chapter implements different computer vision techniques for preprocessing the image of the ear using the most important features of the ear, which are the Helix, Shell (Concha) and lobule (included in Figure 13.1). We can use two-dimensional images of the ear, which is important because it allows using less specialized equipment, plus it is computationally less expensive because less information processing is needed. We describe in the following sections the process on the different tests to find the appropriate structure for the modular neural network and type-1 fuzzy logic response integrators.

Fig. 13.1 Main features of the ear. The helix is the outer contour of the ear, the shell (concha) is the cavity where is located the entrance to the ear, and the final roll is called lobe.

13.2 Modular Artificial Neural Networks A neural network is said to be modular if the computation performed by the network can be decomposed into two or more modules (subsystems) that operate on distinct inputs without communicating with each other [57, 59]. Modular neural networks are composed of modules that can be grouped according to both different structure and functionality that are integrated together through an integrated unit. With functional classification, each module is a neural network that performs a different task of identifying sub. Furthermore, using these approach different types of learning algorithms can be combined perfectly [32, 59, 80, 93]. In recent decades, Neural Networks (NNs) have received particular interest as a technology for data mining, since it provides the means to model effective and

13.3 Integration Methods

187

efficient large and complex problems [1, 2, 19, 56, 58, 60, 63]. The NN is a method to solve problems, individually or in combination with other methods for tasks of classification, identification, diagnosis, optimization or prediction on which the balance data / knowledge leans to data and where, additionally, may have learning needs at runtime and fault tolerance. In these cases the NNs dynamically adapt constantly readjusting "weights" of their interconnections.

13.3 Integration Methods We can distinguish different methods of integration or combination of modules that have been considered [59]: Integration by Gating Network: A decomposition of a learning task into sub tasks learned by the cooperation modules (Figure 13.2). The mechanism of "winner takes all: you can only put in systems that perform similar tasks experts and offer consistent results, which is not the case of tasks such as parking a truck in which the outcome (the angle of rotation of the wheel) is a function of the position of the cab and trailer. Models in series: the output of a model is used as input for the next. Fuzzy logic: We define fuzzy membership functions, which provide a smooth transition between the models to give more or less weight to each model according to a set of fuzzy variables [12, 20, 103]. Sugeno Fuzzy Integral: For integration of the modules uses the Fuzzy Sugeno Integral. The reason this method is used is because in past research very good results were obtained in such problems as pattern recognition with modular neural networks [63, 64].

Fig. 13.2 Schematic of a Modular Neural Network

188

Chapter 13 Optimization of Neural Networks for the Accurate Identification

13.4 Pre-processing the Biometric Image of the Ear Performing a pre-processing of the data has several advantages; the most important among them is that it can reduce the dimensionality of data, which improves system performance substantially, especially when using a methodology such as neural networks. In this stage we can pre-process the input patterns so that all of them have the same size (scale) getting from this that the system does not change the scaling. Besides this, also seeks to ensure that the system does not change the translation. When a system does not change the translation and scaling of the patterns, we say that the system has prior knowledge. The used Ear images were obtained from the University of Science and Technology Beijing (USTB). The database was developed from November 2003 and January 2004 with 77 individuals, four images of the left ear for each, giving us a total of 308 images [80]. The images are with different contrast and angle, as shown in Figure 13.3.

Fig. 13.3 Sample images of the ear

Since images of faces are regularly taken at different times, this results in the different images of a person with variations in lighting, orientation, and size of the face. Therefore, it is necessary that the image is preprocessed before it can be used. Among the pre-processing tasks we commonly can find: extracting the facial image of a larger image that contains information irrelevant to the recognition, normalization in size, i.e., all images of the faces are larger similar, and the application of a filtering method to improve the quality of the image. For the development of the pre-processing, we began with the development of an algorithm for cropping the image, we used different techniques of vision to find the area of interest, one of the most important was the binarization of the image where you cut parts leaving only black and white parts, using for and imcut matlab function, once you have the image with the area of interest [80].

13.4 Pre-processing the Biometric Image of the Ear

189

The next step, which is the normalization of the image, where each image is resized to another array, depending on the image containing the largest ear, leaving the image of 300x400 pixels to become 184 * 256 pixels, as shown in Fig. 13.4.

Fig. 13.4 Cut and Standardization of the image of the ear

To improve image quality and achieve better recognition it was decided to apply various vision techniques that are indicated as follows.

13.4.1 Filters Low Pass Filters: Its objective is to soften the image, are useful when it is assumed that the image has a lot of noise and you want to delete it [65]. They can also be used to highlight the information on a certain scale (size of the filter matrix), for example in case you want to remove the variability associated with cover types present in the image thus standardizing response [50]. There are several possibilities: Median Filter, assigned to the central pixel, it is the average of all pixels included in the window. The filtering matrix would consist of some and the divisor would be the total number of elements in the array (Figure 13.5).

Fig. 13.5 Median Filter

190

Chapter 13 Optimization of Neural Networks for the Accurate Identification

13.4.2 Contrast It is defined as the relative difference in intensity between an image point and its surroundings [48, 49]. This was applied to the image with an enhanced level of 0.5, according to the membership of each element of the array, the image that has an enhanced level of greater than 0.5 the image is not applied (Figure 13.6).

Fig. 13.6 Shows the image contrast

13.4.3 2D Wavelets Finally, the images are compressed using wavelet analysis in 2D with a fixed threshold method, which is a technique used to reduce the size of the images without losing the necessary information. This method basically relies on the regularity of the signal in the Wavelet domain and it is for this reason that is a fairly accurate approximation, and can be represented with a minimum number of coefficients [59, 80]. For our wavelet function, we used the "Symmlet" order written as sym8 8, with 2 levels of decomposition. For details of the wavelet coefficients, they were used to eliminate noise with the criteria "hard thresholding" at 20% intensity.

13.5 Problem Statement and Proposed Method This work studied various methods of integration that can be applied to modular neural networks for recognition of persons using images of the ear as biometrics, in addition to developing alternative methods for the integration of the network, such as the gating network, where the image without partitions is the chosen one, and the winner takes it all, for its acronym in English, "Winner takes All" (WTA), where the image is partitioned into three equal parts. Figure 13.7 shows the general architecture used in this work by partitioning the image into three parts [80].

13.5 Problem Statement and Proposed Method

191

Fig. 13.7 General Architecture of Neural Network-integrated Modular winner takes all (WTA)

We first worked with a modular neural network, which we chose to partition the image into 3 parts, to get a better decision when it has to identify the person, in this case we will have 3 parts of the ear shell helix lobe, which are the inputs to enter each module of the network, with 3 sub-modules in each module, 3 modules with a total of 9 sub-modules, as shown in Figure 13.7, each sub module is trained with different training method and different number of neurons, and as we exit the integrator (WTA), which will decide which of the three parts of the ear or that of the three sub-module is the one with the highest activation, so that if we have a winner module ear or even part of the winning we will be able to identify the person. Below in Table 13.1 we show the parameters of the modular neural network, which was tested empirically. Table 13.1 Parameters of the Modular Neural Network

192

Chapter 13 Optimization of Neural Networks for the Accurate Identification

Each sub module is fed with different information, which implies that we need to find a suitable architecture for each of the 9 sub modules, each with different methods and different number of neurons, so that each part has been learned by the network differently. We did study empirically, to see how many neurons for the neural networks provides an acceptable learning, according to the type of training used, will depend on the number of neurons used for such training, and which may learn faster or slower and therefore have a bad or good learning, it must adapt to the neurons with the method of learning along with the times. In modular neural networks, each module receives vectors with different data, leading to architectures that are not uniform. The integration joins the results of each module and the result is divided by the number of elements, as shown in the following equation. Equation for the recognition rate of each module:

(13.1)

Where: Idc: images identified Timg: Total images. The results of each module of our modular network with the integrator Winner takes it all (with arbitrary parameters), without pre-processing (the result of the integration of the 3 modules) is shown in Table 13.2. Table 13.2 Modular Neural Network Results without preprocessing the image

13.5 Problem Statement and Proposed Method

193

Table 13.2 shows some results obtained from various trainings performed in each of the modules, the best trainings were the numbers 4, 2 8, with 8000 epochs respectively, and a goal error of 0.00001 for the neural network, these trainings were performed without pre-processing the image, using an uncompressed image. Based on these results and trainings, which are at best suboptimal, leave us we the need to improve results, because there are parameters that have a very large range, such as the neurons, where the number of neurons that are used, we never know how many neurons are optimal and / or appropriate to have a low margin of error. For this reason we decided to apply a genetic algorithm to find an appropriate percentage of identification, and thus obtain the optimal method of training and the appropriate number of neurons, and once we get these parameters take the mean and standard deviation of many times the genetic algorithm to find a high percentage of identification. The results of each module of the modular neural network with the integrated gating network of and arbitrary parameters with preprocessing are shown in Table 13.3. Table 13.3 shows the results of the performed trainings, with pre-image processing and compression, where one can observe that the identification rates increased considerably with the same parameters used previously. The methods with high percentage were 1, 2 and 4, with 8000 epochs respectively, and a goal error of the neural network of 0.00001.

Table 13.3 Results of the Modular Neural Network with image preprocessing

13.5.1 Architecture Optimization Once we have seen that the GA has found a good result, we decided to run the GA 10 times to find the standard deviation and average of neurons, methods, and the

194

Chapter 13 Optimization of Neural Networks for the Accurate Identification

number of layers for knowing the behavior in training with the GA as shown in Table 13.4. The general architecture is shown in Figure 13.8. The chromosome is described in Table 13.4.

Fig. 13.8 General Architecture of the Optimized Modular Neural Network

Table 13.4 Parameters of the chromosome for the GA

The real chromosome is composed of 2 layers {1, 2}, and each layer is composed of 100 neurons in the first hidden layer and 80 in the 2nd hidden layer, and 3 training methods. The training methods are in Table 13.5.

13.5 Problem Statement and Proposed Method

195

Table 13.5 Training methods for the GA

Method Training Trainscg: Scaled conjugate Traingda: Gradient descent with momentum and adaptive learning factor Traingdx: Gradient descent with adaptive learning factor

13.5.2 Role of the Genetic Algorithm The objective function of the genetic algorithm is expressed as (13.2) The objective function is the same for each of the modules of the neural network, whether we get a different architecture.

13.5.3 Behavior of Neural Network Training The results of executing 10 times the genetic algorithm for (Module 1, Table 13.6), (module 2, Table 13.7) and Module 3 are in Table 13.8.

Table 13.6 Results of the genetic algorithm for module 1

Table 13.7 Results of genetic algorithm for module 2

196

Chapter 13 Optimization of Neural Networks for the Accurate Identification Table 13.8 Results of genetic algorithm for module 3

In Tables 13.6, 13.7 and 13.8 we show the results of the trainings performed by the genetic algorithm, with 15 generations, 10 individuals and 10 runs, showing the average in each evolution, the standard deviation of each evolution and the best error found by the genetic algorithm, the best training method and the duration of the 10 runs. Figure 13.9 shows the convergence of the error, which shows the minimum and / or the best individual in each generation.

Fig. 13.9 Error convergence graph of the best individuals of the GA (left) Module1 (right) module2 (center) module 3

13.5 Problem Statement and Proposed Method

197

Once the 10 runs of the GA were performed, we obtain the best architectures of different executions. Tables 13.9, 13.10 and 13.11 show the different architectures that provided a good percentage in each module. Table 13.9 Best architecture for module 1

Table 13.10 Best architecture for module 2

Table 13.11 Best architecture for module 3

198

Chapter 13 Optimization of Neural Networks for the Accurate Identification

13.5.4 Validation of the Modular Neural Network structure These architectures were used 4 times cross-validation, new vectors were created containing a different image, in order to obtain the best average recognition in each validation. This is shown in the Table 13.12. Table 13.12 Results of applying cross-validation for each architecture

We choose the best architecture can fit the percentages obtained from crossvalidations for each module, whether we get a complete architecture or not, as shown in Table 13.13. Table 13.13 Best architecture obtained by the GA

These are the results of cross validation applied 4 times to the best architecture, using the winner-take-all (WTA) integrator. This is shown in the Table 13.14. Table 13.14 Results of 4 cross-validations for the best architecture

13.5 Problem Statement and Proposed Method

199

As shown in Table 13.14, in some cases 100% of recognition was obtained, but only in some modules of the neural network, with the highest percentage of recognition in validation 4, which on average has 100% recognition for the 4 data validations. Once we have found the right structure of the neural network, according to the percentage above with the WTA integrator, we choose to implement a fuzzy integrator to manage a greater degree of uncertainty in the time of decision. Each module will be trained differently for variety of results, for the first module would be as follows, module 1 (trainscg), 2nd module (trainscg) and 3rd module (trainscg), maintaining the same number of neurons. The architecture of the neural network with the fuzzy integrator is shown below in Figure 13.7.

Fig. 13.7 General Architecture of the extended network with optimized fuzzy integrator

The integrator is a fuzzy inference system of type-1 fuzzy rules, in this case we worked with Mamdani reasoning with 3 input linguistic variables and one output linguistic variable, with three membership functions. The three membership functions (MF) are of Triangular type used for input variables are (low, medium, high) for Helix, Shell (low, medium, high) and lobe (low, medium, high) in the output variable (winner) there are three membership functions (FM), which are named (Helix, Shell, lobe) of triangular type, 2 tests were performed with different membership functions in the first case triangular and Gaussian in the second case, and its range value ranges from 0 to 1. The fuzzy Inference System has 24 fuzzy rules, where each FIS is specialized in each module of the neural network, which will have the option of deciding, which is the highest activation according to the 3 Sub modules found in each module, in order to get a better decision based on the activation winner. Figure 13.8 shows these membership functions.

200

Chapter 13 Optimization of Neural Networks for the Accurate Identification

Fig. 13.8 Fuzzy Inference System (FIS) with FM-type Gaussian (left), triangular (right)

The results of validation with the fuzzy Mamdani integrator with membership functions of triangular and Gaussian types are shown below in Tables 13.15 and 13.16. Table 13.15 Results of the Mamdani fuzzy integrator with triangular type MFs

Table 13.16 Results of the Mamdani fuzzy integrator with Gaussian type MFs.

13.5 Problem Statement and Proposed Method

201

The results shown above, in case 1 where the MFs are triangular in validating one obtains a percentage of 96.72%. In case 2, where we worked with Gaussian membership functions, validation 3 was obtained 98.71% of identification, an average of 97.41%.

13.5.5 Optimizing the Fuzzy Integrator Figure 13.9 shows the general architecture of the neural network with integration of the fuzzy integrator.

Fig. 13.9 General Architecture of the modular neural network with fuzzy integrator optimization using a genetic algorithm

In the optimization of fuzzy integrator we worked with a genetic algorithm (GA) to move the membership functions for the 2 cases, triangular and Gaussian, and thus have a higher percentage of recognition. The genetic algorithm parameters for both cases are 50 generations and 10 individuals. The chromosome is shown below for triangular membership functions in Figure 13.10 and Figure 13.11 for Gaussian. For the fuzzy integrator with triangular MF, every point of each gene function has 3 points so we have 3 genes in a range from 0 to 1, at each point and each variable we have 9 genes, this in the input variables and output variable in the chromosome is composed of total 36 genes, which moves the genetic algorithm to find a better fuzzy inference system (FIS).

202

Chapter 13 Optimization of Neural Networks for the Accurate Identification

For the fuzzy integrator with Gaussian MF, each point of each function is a gene, so a membership function has 2 parameters, which are the mean and standard deviation in each of the input variables we can observe that 18 genes in each one of the modules.

Fig 13.10 GA chromosome for the fuzzy integrator for Triangular MF

Fig. 13.11 GA chromosome for the fuzzy integrator for Gaussian MF Figure 13.12 shows the convergence of the error to 0, according to the proposed objective function, where you can see, that at least 35 generations are needed to find module 1, giving us the FIS for that module. Figures 13.13 and 13.14 show in a similar way the results of the GA for the second and third module, respectively.

13.5 Problem Statement and Proposed Method

Fig. 13.12 Genetic Algorithm Convergence (Right) Result (FIS) for the 1st triangular membership functions

203

module,

Fig. 13.13 Genetic Algorithm Convergence (Right) Result (FIS) for the 2nd module, triangular membership functions

Fig. 13.14 Genetic Algorithm Convergence (Right) Result (FIS) for the 3sd module, triangular membership functions

204

Chapter 13 Optimization of Neural Networks for the Accurate Identification

Figure 13.14 shows the convergence of the error to 0, according to the proposed objective function, where you can see, that at least 25 generations are needed to find the optimal result for module 1, giving us the FIS for that module. Once we have optimized the fuzzy inference systems for each of the cases (triangular and Gaussian), their architecture were validated, by 4 cross-validations, the results are shown below for Triangular and Gaussian. This is shown on Table 13.17. Table 13.17 Validation of Results for the Mamdani type fuzzy integrator with triangular membership functions

Table 13.17 shows the results of cross-validation with the fuzzy Mamdani integrator with triangular membership functions. The validations that had a high percentage of recognition were validations 2 and 4, but even considering the lowest validation, an average of 99.35% was obtained for recognition.

13.6 Summary The best result for the recognition of persons using the biometric measurement of the ear was obtained by modular neural network architecture with 2 layers in each module, with 21 in the first hidden layer and 16 neurons in the second hidden layer (as shown in Table 13.13). Since the results were not satisfactory in the fuzzy system integration, we decided to apply an evolutionary approach to optimize the membership functions of this system. The method chosen to optimize this fuzzy integrator was a genetic algorithm. After applying the genetic algorithm we can obtain a better recognition rate, so we have better results with the optimized system.

References

[1] Abiyev, R.H., Kaynak, O.: Type-2 fuzzy neural structure for identifica-tion and

[2]

[3]

[4]

[5] [6]

[7]

[8]

[9]

[10] [11]

[12] [13]

control of time varying plants. IEEE Transactions on Industrial Electronics 57, 4147–4159 (2010) Abiyev, R.H., Kaynak, O., Alshanableh, T., Mamedov, F.: A type-2 neuro-fuzzy system based on clustering and gradient techniques applied to system identification and channel equalization. Applied Soft Computing Journal 11, 1396–1406 (2011) Albarracin, L.F., Melgarejo, M.A.: An approach for channel equaliza-tion using quasi type-2 fuzzy systems. In: Proceedings of the NAFIPS 2010 Conference, art. no. 5548203 (2010) Aliev, R.A., Pedrycz, W., Guirimov, B.G., Aliev, R.R., Ilhan, U., Babagil, M., Mammadli, S.: Type-2 fuzzy neural networks with fuzzy clustering and differential evolution optimization. Information Sciences 181, 1591–1608 (2011) Astudillo, L., Melin, P., Castillo, O.: A new optimization method base on a paradigm inspired by nature. SCI, vol. 312, pp. 277–283 (2010) Bajestani, N.S., Zare, A.: Application of optimized type-2 fuzzy time series to forecast Taiwan stock index. In: 2nd International Conference on computer, Control and Communication, pp. 275–280 (2009) Benedikt, L., Cosker, D., Rosin, P.L., Marshall, D.: Assessing the uniqueness and permanence of facial actions for use in biometric applications. IEEE Transactions on Systems, Man, Cybernetics, A, Systems, Humans 40, 449–460 (2010) Boddeti, V.N., Kumar, B.V.: Extended depth of field iris recognition using unrestored wavefront coded imagery. IEEE Transactions on Systems, Man, Cybernetics, A, Systems, Humans 40, 495–508 (2010) Bustard, J.D., Nixon, M.S.: Toward unconstrained ear recognition from twodimensional images. IEEE Transactions on Systems, Man, Cybernetics, A, Systems, Humans 40, 486–494 (2010) Castillo, O., Martinez, A.I., Martinez, A.C.: Evolutionary computing for topology optimization of type-2 fuzzy systems. Advances in Soft Computing 41, 63–75 (2007) Castillo, O., Huesca, G., Valdez, F.: Evolutionary computing for topol-ogy optimization of type-2 fuzzy controllers. Studies in Fuzziness and Soft Computing 208, 163–178 (2008) Castillo, O., Melin, P.: Type-2 Fuzzy Logic: Theory and Applications. Springer, Heidelberg (2008) Castillo, O., Aguilar, L.T., Cazarez-Castro, N.R., Cardenas, S.: Systematic design of a stable type-2 fuzzy logic controller. Applied Soft Computing Journal 8, 1274–1279 (2008)

206

References

[14] Castillo, O., Martinez-Marroquin, R., Melin, P., Valdez, F., Soria, J.: Comparative

[15]

[16] [17] [18]

[19]

[20] [21]

[22]

[23]

[24]

[25]

[26] [27]

[28]

[29]

study of bio-inspired algorithms applied to the optimi-zation of type-1 and type-2 fuzzy controllers for an autonomous mobile robot. Information Sciences (2011) (article in press) Castillo, O., Melin, P., Alanis, A., Montiel, O., Sepulveda, R.: Optimization of interval type-2 fuzzy logic controllers using evolutionary algorithms. Journal of Soft Computing 15, 1145–1160 (2011) Castro, J.R., Castillo, O., Melin, P.: An Interval Type-2 Fuzzy Logic Toolbox for Control Applications. In: Proceedings of FUZZ-IEEE, London, pp. 1–6 (2007) Castro, J.R., Castillo, O., Martinez, L.G.: Interval type-2 fuzzy logic toolbox. Engineering Letters 15(1), 14 (2007) Castro, J.R., Castillo, O., Melin, P., Martinez, L.G., Escobar, S., Camacho, I.: Building fuzzy inference systems with the interval type-2 fuzzy logic toolbox. Advances in Soft Computing 41, 53–62 (2007) Castro, J.R., Castillo, O., Melin, P., Rodriguez-Diaz, A.: A hybrid learning algorithm for a class of interval type-2 fuzzy neural networks. Information Sciences 179, 2175–2193 (2009) Cervantes, L., Castillo, O.: Design of a fuzzy system for the longitudinal control of an F-14 airplane. SCI, vol. 318, pp. 213–224 (2010) Chen, X., Li, Y., Harrison, R., Zhang, Y.-Q.: Type-2 fuzzy logic based classifier fusion for support vector machines. Applied Soft Computing Journal 8, 1222–1231 (2008) Chua, T.W., Tan, W.W.: Genetically Evolved Fuzzy Rule-Based Classifiers and Application to Automotive Classification. In: Li, X., Kirley, M., Zhang, M., Green, D., Ciesielski, V., Abbass, H.A., Michalewicz, Z., Hendtlass, T., Deb, K., Tan, K.C., Branke, J., Shi, Y. (eds.) SEAL 2008. LNCS, vol. 5361, pp. 101–110. Springer, Heidelberg (2008) Chumklin, S., Auephanwiriyakul, S., Theera-Umpon, N.: Microcalcification detection in mammograms using interval type-2 fuzzy logic system with automatic membership function generation. In: 2010 IEEE World Congress on Computational Intelligence, WCCI 2010, art. no. 5584896 (2010) Cordon, O., Gomide, F., Herrera, F., Hoffmann, F., Magdalena, L.: Ten years of genetic fuzzy systems: current framework and new trends. Fuzzy Sets and Systems 141, 5–31 (2004) Cordon, O., Herrera, F., Villar, P.: Analysis and guidelines to obtain a good uniform fuzzy partition granularity for fuzzy rule-based systems using simulated annealing. International Journal of Approximate Reasoning 25, 187–215 (2000) Coupland, S., John, R.: New geometric inference techniques for type-2 fuzzy sets. International Journal of Approximate Reasoning 49, 198–211 (2008) Dereli, T., Baykasoglu, A., Altun, K., Durmusoglu, A., Turksen, I.B.: Industrial applications of type-2 fuzzy sets and systems: A concise review. Computers in Industry 62, 125–137 (2011) Garcia, J.C.F.: An evolutive interval type-2 TSK fuzzy logic system for volatile time series identification. In: Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, pp. 666–671 (2009) Hagras, H.: Hierarchical type-2 fuzzy logic control architecture for autonomous mobile robots. IEEE Transactions on Fuzzy Systems 12, 524–539 (2004)

References

207

[30] Herman, P., Prasad, G., McGinnity, T.M.: Support vector-enhanced design of a

[31]

[32]

[33]

[34]

[35]

[36]

[37] [38]

[39]

[40]

[41] [42]

[43]

[44]

T2FL approach to motor imagery-related EEG pattern recognition. In: Proceedings of the IEEE International Conference on Fuzzy Systems, art. no. 4295661 (2007) Herman, P., Prasad, G., McGinnity, T.M.: Design and on-line evaluation of type-2 fuzzy logic system based framework for handling uncertainties in BCI classification. In: Proceedings of the 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS 2008, art. no. 4650146, pp. 4242–4245 (2008) Hidalgo, D., Castillo, O., Melin, P.: Type-1 and type-2 fuzzy inference systems as integration methods in modular neural networks for multimodal biometry and its optimization with genetic algorithms. Information Sciences 179, 2123–2145 (2009) Hidalgo, D., Melin, P., Licea, G., Castillo, O.: Optimization of Type-2 Fuzzy Integration in Modular Neural Networks Using an Evolutionary Method with Applications in Multimodal Biometry. In: Aguirre, A.H., Borja, R.M., Garciá, C.A.R. (eds.) MICAI 2009. LNCS, vol. 5845, pp. 454–465. Springer, Heidelberg (2009) Hidalgo, D., Melin, P., Castillo, O.: Optimal design of type-2 fuzzy membership functions using genetic algorithms in a partitioned search space. In: Proceedings of the IEEE International Conference on Granular Computing, GrC 2010, San Jose, pp. 212–216 (August 2010) Hidalgo, D., Melin, P., Mendoza, O.: Evolutionary optimization of type-2 fuzzy systems based on the level of uncertainty. In: Proceedings of the IEEE World Congress on Computational Intelligence, WCCI 2010, Barcelona (July 2010) Hosseini, R., Dehmeshki, J., Barman, S., Mazinani, M., Qanadli, S.: A genetic type-2 fuzzy logic system for pattern recognition in computer aided detection systems. In: Proceedings of 2010 IEEE World Congress on Computational Intelligence, WCCI 2010, art. no. 5584773 (2010) Jang, J.R., Sun, C.T., Mizutani, E.: Neuro-Fuzzy and Soft Computing. Prentice Hall, Upper Saddle River (1997) Jeng, W.-H.R., Yeh, C.-Y., Lee, S.-J.: General type-2 fuzzy neural network with hybrid learning for function approximation. In: Proceedings of the IEEE Conference on Fuzzy Systems, Jeju, Korea, pp. 1534–1539 (2009) Juang, C.-F., Huang, R.-B., Lin, Y.-Y.: A recurrent self-evolving interval type-2 fuzzy neural network for dynamic system processing. IEEE Transactions on Fuzzy Systems 17, 1092–1105 (2009) Kalka, N.D., Zuo, J., Schmid, N.A., Cukic, B.: Estimating and fusing quality factors for iris biometric images. IEEE Transactions on Systems, Man, Cybernetics, A, Systems, Humans 40, 509–524 (2010) Karnik, N. N., Mendel, J. M.: An Introduction to Type-2 Fuzzy Logic Systems, Technical Report, University of Southern California (1998) Khanesar, M.A., Teshnehlab, M., Kayacan, E., Kaynak, O.: A novel type-2 fuzzy membership function: Application to the prediction of noisy data. In: Proceedings of the IEEE International Conference on Computational Intelligence for Measurement Systems and Applications, CIMSA 2010, pp. 128–133 (2010) Kim, G.-S., Ahn, I.-S., Oh, S.-K.: The design of optimized type-2 fuzzy neural networks and its application. Transactions of the Korean Institute of Electrical Engineers 58, 1615–1623 (2009) Leal-Ramirez, C., Castillo, O., Melin, P., Rodriguez-Diaz, A.: Simulation of the bird age-structured population growth based on an interval type-2 fuzzy cellular structure. Information Sciences 181, 519–535 (2011)

208

References

[45] Li, H., Zhang, X.: A hybrid learning algorithm based on additional momentum

[46]

[47]

[48]

[49]

[50]

[51]

[52]

[53]

[54]

[55]

[56]

[57] [58]

[59] [60]

and self-adaptive learning rate. Journal of Computational Information Systems 6, 1421–1429 (2010) Liang, Q., Mendel, J.M.: Overcoming time-varying co-channel interference using type-2 fuzzy adaptive filters. IEEE Transactions on Circuits and Systems II: Analog and Digital Signal Processing 47, 1419–1428 (2000) Liang, Q., Mendel, J.M.: Equalization of nonlinear time-varying channels using type-2 fuzzy adaptive filters. IEEE Transactions on Fuzzy Systems 8, 551–563 (2000) Lopez, M., Melin, P.: Response integration in ensemble neural networks using interval type-2 fuzzy logic. In: Proceedings of the International Joint Conference on Neural Networks, art. no. 4633995, pp. 1503–1508 (2008) Lopez, M., Melin, P., Castillo, O.: Optimization of response integration with fuzzy logic in ensemble neural networks using genetic algorithms. SCI, vol. 154, pp. 129–150 (2008) Lopez, M., Melin, P.: Topology optimization of fuzzy systems for response integration in ensemble neural networks: The case of finger-print recognition. In: Proceedings of the Annual Conference of the North American Fuzzy Information Processing Society - NAFIPS, art. no. 4531334 (2008) Lopez, M., Melin, P., Castillo, O.: Optimization of response integration with fuzzy logic in ensemble neural networks using genetic algorithms. SCI, vol. 154, pp. 129–150 (2008) Lopez, M., Melin, P., Castillo, O.: Comparative study of feature extraction methods of fuzzy logic type 1 and type-2 for pattern recognition system based on the mean pixels. SCI, vol. 312, pp. 171–188 (2010) Lucas, L.A., Centeno, T.M., Delgado, M.R.: General type-2 fuzzy classifiers to land cover classification. In: Proceedings of the ACM Symposium on Applied Computing, pp. 1743–1747 (2008) Madasu, V.K., Hanmandlu, M., Vasikarla, S.: A novel approach for fuzzy edge detection using type II fuzzy sets. In: Proceedings of SPIE - The International Society for Optical Engineering, art. no. 70750I, vol. 7075 (2008) Marsico, M., Nappi, M., Riccio, D.: FARO: Face recognition against occlusions and expression variations. IEEE Transactions on Systems, Man, Cybernetics, A, Systems, Humans 40, 121–132 (2010) Melin, P., Castillo, O.: A new method for adaptive model-based control of non-linear dynamic plants using a neuro-fuzzy-fractal approach. Journal of Soft Computing 5, 171–177 (2001) Melin, P., Castillo, O.: Modelling, Simulation and Control of Non-Linear Dynamical Systems. Taylor and Francis, London (2002) Melin, P., Castillo, O.: Adaptive intelligent control of aircraft systems with a hybrid approach combining neural networks, fuzzy logic and fractal theory. Journal of Applied Soft Computing 3(4), 353–362 (2003) Melin, P., Castillo, O.: Hybrid Intelligent Systems for Pattern Recognition. Springer, Heidelberg (2005) Melin, P., Castillo, O.: An intelligent hybrid approach for industrial quality control combining neural networks, fuzzy logic and fractal theory. Information Sciences 177, 1543–1557 (2007)

References

209

[61] Melin, P.: Interval type-2 fuzzy logic applications in image processing and pattern

[62] [63]

[64]

[65] [66] [67] [68] [69] [70]

[71]

[72]

[73]

[74]

[75]

[76]

[77]

recognition. In: Proceedings of the 2010 IEEE International Conference on Granular Computing, GrC 2010, pp. 728–731 (2010) Mendel, J.M.: Uncertainty, fuzzy logic, and signal processing. Signal Processing Journal 80, 913–933 (2000) Mendoza, O., Melin, P., Castillo, O., Licea, G.: Type-2 Fuzzy Logic for Improving Training Data and Response Integration in Modular Neural Networks for Image Recognition. In: Melin, P., Castillo, O., Aguilar, L.T., Kacprzyk, J., Pedrycz, W. (eds.) IFSA 2007. LNCS (LNAI), vol. 4529, pp. 604–612. Springer, Heidelberg (2007) Mendoza, O., Melin, P., Castillo, O.: Interval type-2 fuzzy logic and modular neural networks for face recognition applications. Applied Soft Computing Journal 9, 1377–1387 (2009) Mendoza, O., Melin, P., Licea, G.: Interval type-2 fuzzy logic for edges detection in digital images. Int. Journal of Intelligent Systems 24, 1115–1133 (2009) Mitchell, H.B.: Pattern recognition using type-II fuzzy sets. Information Sciences 170, 409–418 (2005) Own, C.-M.: Switching between type-2 fuzzy sets and intuitionistic fuzzy sets: An application in medical diagnosis. Applied Intelligence 31, 283–291 (2009) Ozkan, I., Türkşen, I.B.: Entropy assessment for type-2 fuzziness. In: Proceedings of the IEEE International Conference on Fuzzy Systems, vol. 2, pp. 1111–1115 (2004) Ozkan, I., Turksen, B.: MiniMax ε-stable cluster validity index for type-2 fuzziness. In: Proceedings of the NAFIPS 2010 Conference, art. no. 5548183 (2010) Park, K.-J., Oh, S.-K., Pedrycz, W.: Design of interval type-2 fuzzy neural networks and their optimization using real-coded genetic algorithms. In: Proceedings of the IEEE Conference on Fuzzy Systems, Jeju, Korea, pp. 2013–2018 (2009) Pedrycz, W.: Human centricity in computing with fuzzy sets: An interpretability quest for higher order granular constructs. Journal of Ambient Intelligence and Humanized Computing 1, 65–74 (2010) Phong, P.A., Thien, K.Q.: Classification of cardiac arrhythmias using interval type-2 TSK fuzzy system. In: Proceedings of the 1st International Conference on Knowledge and Systems Engineering, art. no. 5361742, pp. 1–6 Pimenta, A.H.M., Camargo, H.A.: Interval type-2 fuzzy classifier design using genetic algorithms. In: 2010 IEEE World Congress on Computational Intelligence, WCCI 2010, art. no. 5584520 (2010) Poh, N., Kittler, J., Bourlai, T.: Quality-based score normalization with device qualitative information for multimodal biometric fusion. IEEE Transactions on Systems, Man, Cybernetics, A, Systems, Humans 40, 539–554 (2010) Qin, K., Kong, L., Liu, Y., Xiao, Q.: Sea surface temperature clustering based on type-2 fuzzy theory. In: Proceedings of the 18th International Conference on Geoinformatics, Geoinformatics 2010, art. no. 5567484 (2010) Qiu, Y., Zhang, Y.-Q., Zhao, Y.: Statistical genetic interval-valued fuzzy systems with prediction in clinical trials. In: Proceedings of the IEEE International Conference on Granular Computing, San Jose, pp. 129–132 (2007) Qun, R., Baron, L., Balazinski, M.: Type-2 takagi-sugeno-kang fuzzy logic modeling using subtractive clustering. In: Proceedings of the Annual Conference of the North American Fuzzy Information Processing Society - NAFIPS, art. no. 4216787, pp. 120–125

210

References

[78] Ren, Q., Baron, L., Balazinski, M.: High order type-2 TSK fuzzy logic system. In:

Proceedings of the NAFIPS 2010 Conference, art. no. 4531215 (2008) [79] Rhee, F.C.-H., Choi, B.-I.: Interval type-2 fuzzy membership function design and its

[80]

[81]

[82]

[83]

[84]

[85]

[86]

[87]

[88] [89]

[90] [91] [92] [93]

application to radial basis function neural networks. In: Proceedings of the IEEE International Conference on Fuzzy Systems, art. no. 4295680 (2007) Sanchez, D., Melin, P.: Modular neural network with fuzzy integration and its optimization using genetic algorithms for human recognition based on iris, ear and voice biometrics. SCI, vol. 312, pp. 85–102 (2010) Santiago-Sánchez, K., Reyes-García, C.A., Gómez-Gil, P.: Type-2 Fuzzy Sets Applied to Pattern Matching for the Classification of Cries of Infants Under Neurological Risk. In: Huang, D.-S., Jo, K.-H., Lee, H.-H., Kang, H.-J., Bevilacqua, V. (eds.) ICIC 2009. LNCS, vol. 5754, pp. 201–210. Springer, Heidelberg (2009) Sanz, J., Fernandez, A., Bustince, H., Herrera, F.: A genetic algorithm for tuning fuzzy rule based classification systems with interval valued fuzzy sets. In: 2010 IEEE World Congress on Computational Intel-ligence, WCCI 2010, art. no. 5584097 (2010) Sepulveda, R., Castillo, O., Melin, P., Montiel, O.: An efficient computational method to implement type-2 fuzzy logic in control applications. Advances in Soft Computing 41, 45–52 (2007) Sepulveda, R., Castillo, O., Melin, P., Rodriguez-Diaz, A., Montiel, O.: Experimental study of intelligent controllers under uncertainty using type-1 and type-2 fuzzy logic. Information Sciences 177, 2023–2048 (2007) Sharma, P., Bajaj, P.: Performance analysis of vehicle classification system using type-1 fuzzy, adaptive neuro-fuzzy and type-2 fuzzy inference system. In: Proceedings of the 2nd International Conference on Emerging Trends in Engineering and Technology, ICETET 2009, art. no. 5395411, pp. 581–584 (2009) Sharma, P., Bajaj, P.: Accuracy comparison of vehicle classification system using interval type-2 fuzzy inference system. In: Proceedings of the 3rd International Conference on Emerging Trends in Engineering and Technology, ICETET 2010, pp. 85–90 (2010) Starczewski, J.T., Scherer, R., Korytkowski, M., Nowicki, R.: Modular Type-2 Neuro-Fuzzy Systems. In: Wyrzykowski, R., Dongarra, J., Karczewski, K., Wasniewski, J. (eds.) PPAM 2007. LNCS, vol. 4967, pp. 570–578. Springer, Heidelberg (2008) Starczewski, J.T.: Efficient triangular type-2 fuzzy logic systems. International Journal of Approximate Reasoning 50, 799–811 (2009) Tan, W.W., Foo, C.L., Chua, T.W.: Type-2 fuzzy system for ECG ar-rhythmic classification. In: IEEE International Conference on Fuzzy Systems, art. no. 4295478 (2007) Tan, W.-W., Wu, D.: Design of type-reduction strategies for type-2 fuzzy logic systems using genetic algorithms. SCI, pp. 169–187 (2007) Tizhoosh, H.R.: Image thresholding using type II fuzzy sets. Pattern Recognition 38, 2363–2372 (2005) Türkşen, I.B.: Review of fuzzy system models with an emphasis on fuzzy functions. Transactions of the Institute of Measurement and Control 31, 7–31 (2009) Urias, J., Hidalgo, D., Melin, P., Castillo, O.: A method for response integration in modular neural networks with type-2 fuzzy logic for biometric systems. Advances in Soft Computing 41, 5–15 (2007)

References

211

[94] Wagenknecht, M., Hartmann, K.: Application of fuzzy sets of type 2 to the solution

of fuzzy equations systems. Fuzzy Sets and Systems 25, 183–190 (1988) [95] Wagner, C., Hagras, H.: Evolving type-2 fuzzy logic controllers for autonomous

mobile robots. Advances in Soft Computing 41, 16–25 (2007) [96] Walker, C., Walker, E.: Sets with type-2 operations. International Journal of

Approximate Reasoning 50, 63–71 (2009) [97] Wang, C.-H., Cheng, C.-S., Lee, T.-T.: Dynamical optimal training for interval type-

[98] [99]

[100]

[101] [102]

[103] [104] [105]

[106]

[107]

[108]

[109]

2 fuzzy neural network (T2FNN). IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 34, 1462–1477 (2004) Wu, H., Mendel, J.M.: Classification of battlefield ground vehicles based on the acoustic emissions. SCI, vol. 304, pp. 55–77 (2010) Wu, D., Tan, W.-W.: A type-2 fuzzy logic controller for the liquid level process. In: Proceedings of the IEEE Conference on Fuzzy Systems, Budapest, pp. 953–958 (2004) Wu, D., Tan, W.-W.: Genetic learning and performance evaluation of interval type-2 fuzzy logic controllers. Engineering Applications of Artificial Intelligence 19, 829–841 (2006) Yager, R.R.: Fuzzy subsets of type II in decisions. J. Cybernetics 10, 137–159 (1980) Yu, L., Xiao, J., Zheng, G.: Robust interval type-2 possibilistic c-means clustering and its application for fuzzy modeling. In: Proceedings of the 6th International Conference on Fuzzy Systems and Knowledge Discovery, FSKD 2009, art. no. 5359181, vol. 4, pp. 360–365 (2009) Zadeh, L.A.: The concept of a linguistic variable and its application to approximate reasoning. Information Sciences 8, 43–80 (1975) Zarandi, M.H.F., Turksen, I.B., Kasbi, O.T.: Type-2 fuzzy modelling for desulphurization of steel process. Expert Systems with Applications 32, 157–171 (2007) Zeng, J., Liu, Z.-Q.: Type-2 fuzzy hidden Markov models to phoneme recognition. In: Proceedings of the International Conference on Pattern Recognition, vol. 1, pp. 192–195 (2004) Zhang, W.-B., Hu, H.-Z., Liu, W.-J.: Rules extraction of interval type-2 fuzzy logic system based on fuzzy c-means clustering. In: Proceedings - Fourth International Conference on Fuzzy Systems and Knowledge Discovery, FSKD 2007, art. no. 4406083, vol. 2, pp. 256–260 (2007) Zhang, Y., McCullough, C., Sullins, J.R., Ross, C.R.: Hand drawn face sketch recognition by humans and a PCA based algorithm for forensic applications. IEEE Transactions on Systems, Man, Cybernetics, A, Systems, Humans 40, 475–485 (2010) Zheng, G., Wang, J., Zhou, W., Zhang, Y.: A similarity measure between interval type-2 fuzzy sets. In: Proceedings of the 2010 IEEE International Conference on Mechatronics and Automation, ICMA 2010, art. no. 5589072, pp. 191–195 (2010) Zheng, G., Xiao, J., Wang, J., Wei, Z.: A similarity measure between general type-2 fuzzy sets and its application in clustering. In: Proceedings of the World Congress on Intelligent Control and Automation, art. no. 5554327, pp. 6383–6387 (2010)

Index

antecedent 3 architecture for ear recognition 128, 185 for face recognition 5, 31 for fingerprint recognition 5, 31 for iris recognition 7, 61, 128 for human recognition 7, 31 for voice recognition 6, 29, 128 architecture optimization 5 biometric information 7, 128, 186 modalities 29 measures 29, 61 techniques 7, 62, 186 cepstral coefficients 132 chromosomes 5, 113, 135 chromosome representation 113 coding 135 consequents 3 crossover 5 defuzzification 112 divide and conquer principle 31 evolutionary algorithms 5 evolving fuzzy systems 6 network architecture 5 neural networks 4, 79 face detection 5 face recognition 5, 24 face recognition algorithms 5, 24 facial recognition 24 fingerprint image 29 fingerprint recognition 5 fitness evaluation 5 footprint of uncertainty 14, 33

fuzzy if-then rule 24 fuzzy inference system 8 Mamdani 10 Sugeno 143 fuzzy integrals 80 fuzzy modeling 163 fuzzy reasoning 80 fuzzy set 80, 186 fuzzy Sugeno integral 5, 80, 97 fuzzy system for person recognition 99, 100 gate of Experts 31, 49 Gaussian type-2 fuzzy set 37 genetic algorithms 5, 79, 109, 164 hidden layers 49 hierarchical genetic algorithm 109 hierarchical modular architecture 109 human recognition 31 human recognition system 31 identification 43 integration module 5 interval type-2 fuzzy set 36 iris recognition 7, 43, 163 linguistic variable 23 lower membership functions 33 Mamdani model 10 membership function Gaussian 37, 89 generalized bell, 149 trapezoidal 39, 89 triangular 37, 89 mixture of experts 32 modular 4, 79, 163 modular approach 25, 93 modular neural networks 5, 8, 25

214

Index

multilayer perceptron 44 mutation 113

Sugeno fuzzy measure 97 Sugeno model 97, 143

network topology 4

type-2 fuzzy logic 5, 93, 146 type-2 fuzzy logic integration 36 type-2 fuzzy rules 4 type-2 fuzzy set 4 type-2 systems 4, 14

objective function 121 optimization 5, 120, 164 optimizing the networks 5 pattern recognition 4, 62 population 112

upper membership function, 33

recognition system 4, 62 response integration 6

voice recognition 30, 62, 128 voting analysis 111 voting methods 111

selection mechanisms 112 selection operation 113 soft computing 3

weight vector 44 weighted-sum 44 winner takes all 61