Ajith Abraham, Crina Grosan and Witold Pedrycz (Eds.) Engineering Evolutionary Intelligent Systems
Studies in Computational Intelligence, Volume 82 Editor-in-chief Prof. Janusz Kacprzyk Systems Research Institute Polish Academy of Sciences ul. Newelska 6 01-447 Warsaw Poland E-mail:
[email protected] Further volumes of this series can be found on our homepage: springer.com
Vol. 70. Javaan Singh Chahl, Lakhmi C. Jain, Akiko Mizutani and Mika Sato-Ilic (Eds.) Innovations in Intelligent Machines-1, 2007 ISBN 978-3-540-72695-1
Vol. 60. Vladimir G. Ivancevic and Tijana T. Ivacevic Computational Mind: A Complex Dynamics Perspective, 2007 ISBN 978-3-540-71465-1
Vol. 71. Norio Baba, Lakhmi C. Jain and Hisashi Handa (Eds.) Advanced Intelligent Paradigms in Computer Games, 2007 ISBN 978-3-540-72704-0
Vol. 61. Jacques Teller, John R. Lee and Catherine Roussey (Eds.) Ontologies for Urban Development, 2007 ISBN 978-3-540-71975-5 Vol. 62. Lakhmi C. Jain, Raymond A. Tedman and Debra K. Tedman (Eds.) Evolution of Teaching and Learning Paradigms in Intelligent Environment, 2007 ISBN 978-3-540-71973-1 Vol. 63. Wlodzislaw Duch and Jacek Ma´ndziuk (Eds.) Challenges for Computational Intelligence, 2007 ISBN 978-3-540-71983-0 Vol. 64. Lorenzo Magnani and Ping Li (Eds.) Model-Based Reasoning in Science, Technology, and Medicine, 2007 ISBN 978-3-540-71985-4 Vol. 65. S. Vaidya, L.C. Jain and H. Yoshida (Eds.) Advanced Computational Intelligence Paradigms in Healthcare-2, 2007 ISBN 978-3-540-72374-5 Vol. 66. Lakhmi C. Jain, Vasile Palade and Dipti Srinivasan (Eds.) Advances in Evolutionary Computing for System Design, 2007 ISBN 978-3-540-72376-9 Vol. 67. Vassilis G. Kaburlasos and Gerhard X. Ritter (Eds.) Computational Intelligence Based on Lattice Theory, 2007 ISBN 978-3-540-72686-9 Vol. 68. Cipriano Galindo, Juan-Antonio Fernández-Madrigal and Javier Gonzalez A Multi-Hierarchical Symbolic Model of the Environment for Improving Mobile Robot Operation, 2007 ISBN 978-3-540-72688-3 Vol. 69. Falko Dressler and Iacopo Carreras (Eds.) Advances in Biologically Inspired Information Systems: Models, Methods, and Tools, 2007 ISBN 978-3-540-72692-0
Vol. 72. Raymond S.T. Lee and Vincenzo Loia (Eds.) Computation Intelligence for Agent-based Systems, 2007 ISBN 978-3-540-73175-7 Vol. 73. Petra Perner (Ed.) Case-Based Reasoning on Images and Signals, 2008 ISBN 978-3-540-73178-8 Vol. 74. Robert Schaefer Foundation of Global Genetic Optimization, 2007 ISBN 978-3-540-73191-7 Vol. 75. Crina Grosan, Ajith Abraham and Hisao Ishibuchi (Eds.) Hybrid Evolutionary Algorithms, 2007 ISBN 978-3-540-73296-9 Vol. 76. Subhas Chandra Mukhopadhyay and Gourab Sen Gupta (Eds.) Autonomous Robots and Agents, 2007 ISBN 978-3-540-73423-9 Vol. 77. Barbara Hammer and Pascal Hitzler (Eds.) Perspectives of Neural-Symbolic Integration, 2007 ISBN 978-3-540-73953-1 Vol. 78. Costin Badica and Marcin Paprzycki (Eds.) Intelligent and Distributed Computing, 2008 ISBN 978-3-540-74929-5 Vol. 79. Xing Cai and T.-C. Jim Yeh (Eds.) Quantitative Information Fusion for Hydrological Sciences, 2008 ISBN 978-3-540-75383-4 Vol. 80. Joachim Diederich Rule Extraction from Support Vector Machines, 2008 ISBN 978-3-540-75389-6 Vol. 81. K. Sridharan Robotic Exploration and Landmark Determination, 2008 ISBN 978-3-540-75393-3 Vol. 82. Ajith Abraham, Crina Grosan and Witold Pedrycz (Eds.) Engineering Evolutionary Intelligent Systems, 2008 ISBN 978-3-540-75395-7
Ajith Abraham Crina Grosan Witold Pedrycz (Eds.)
Engineering Evolutionary Intelligent Systems With 191 Figures and 109 Tables
123
Ajith Abraham Centre for Quantifiable Quality of Service in Communication Systems (Q2S) Centre of Excellence Norwegian University of Science and Technology O.S. Bragstads plass 2E N-7491 Trondheim Norway
[email protected]
Crina Grosan Department of Computer Science Faculty of Mathematics and Computer Science Babes-Bolyai University Cluj-Napoca, Kogalniceanu 1 400084 Cluj - Napoca Romania
Witold Pedrycz Department of Electrical & Computer Engineering University of Alberta ECERF Bldg., 2nd floor Edmonton AB T6G 2V4 Canada
ISBN 978-3-540-75395-7
e-ISBN 978-3-540-75396-4
Studies in Computational Intelligence ISSN 1860-949X Library of Congress Control Number: 2007938406 c 2008 Springer-Verlag Berlin Heidelberg
This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Cover Design: Deblik, Berlin, Germany Printed on acid-free paper 9 8 7 6 5 4 3 2 1 springer.com
Contents
Engineering Evolutionary Intelligent Systems: Methodologies, Architectures and Reviews Ajith Abraham and Crina Grosan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Architectures of Evolutionary Intelligent Systems . . . . . . . . . . . . . . . . . 3 Evolutionary Artificial Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Evolutionary Search of Connection Weights . . . . . . . . . . . . . . . . . 3.2 Evolutionary Search of Architectures . . . . . . . . . . . . . . . . . . . . . . . 3.3 Evolutionary Search of Learning Rules . . . . . . . . . . . . . . . . . . . . . 3.4 Recent Applications of Evolutionary Neural Networks in Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Evolutionary Fuzzy Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Evolutionary Search of Fuzzy Membership Functions . . . . . . . . . 4.2 Evolutionary Search of Fuzzy Rule Base . . . . . . . . . . . . . . . . . . . . 4.3 Recent Applications of Evolutionary Fuzzy Systems in Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Evolutionary Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Recent Applications of Evolutionary Design of Complex Paradigms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Multiobjective Evolutionary Design of Intelligent Paradigms . . . . . . . 8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1 1 2 5 6 7 8 9 10 12 12 13 14 15 16 18 18
Genetically Optimized Hybrid Fuzzy Neural Networks: Analysis and Design of Rule-based Multi-layer Perceptron Architectures Sung-Kwun Oh and Witold Pedrycz . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 1 Introductory remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2 The architecture of conventional Hybrid Fuzzy Neural Networks (HFNN) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
VI
Contents
3
The architecture and development of genetically optimized HFNN (gHFNN) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Fuzzy neural networks based on genetic optimization . . . . . . . . . 3.2 Genetically optimized PNN (gPNN) . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Optimization of gHFNN topologies . . . . . . . . . . . . . . . . . . . . . . . . . 4 The algorithms and design procedure of genetically optimized HFNN (gHFNN) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 The premise of gHFNN: in case of FS FNN . . . . . . . . . . . . . . . . . 4.2 The consequence of gHFNN: in case of gPNN combined with FS FNN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Experimental studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Nonlinear function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Gas furnace process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 NOx emission process of gas turbine power plant . . . . . . . . . . . . 6 Concluding remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
27 27 31 33 34 35 36 40 40 45 49 53 55 55
Genetically Optimized Self-organizing Neural Networks Based on Polynomial and Fuzzy Polynomial Neurons: Analysis and Design Sung-Kwun Oh and Witold Pedrycz . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 2 The architecture and development of the self-organizing neural networks (SONN) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 2.1 Polynomial Neuron (PN) based SONN and its topology . . . . . . . 62 2.2 Fuzzy Polynomial Neuron (FPN) based SONN and its topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 3 Genetic optimization of SONN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 4 The algorithm and design procedure of genetically optimized SONN (gSONN) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 5 Experimental studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 5.1 Gas furnace process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 5.2 Chaotic time series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 6 Concluding remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 7 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 Evolution of Inductive Self-organizing Networks Dongwon Kim and Gwi-Tae Park . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 2 Design of EA-based SOPNN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 2.1 Representation of chromosome for appropriate information of each PD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 2.2 Fitness function for modelling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
Contents
VII
3
Simulation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 3.1 Gas furnace process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 3.2 Three-input nonlinear function . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 5 Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 Recursive Pattern based Hybrid Supervised Training Kiruthika Ramanathan and Sheng Uei Guan . . . . . . . . . . . . . . . . . . . . . . . . . 129 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 1.2 Organization of the chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 2 Some preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 2.1 Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 2.2 Simplified architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 2.3 Problem formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 2.4 Variable length genetic algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 134 2.5 Pseudo global optima . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 3 The RPHS training algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 3.1 Hybrid recursive training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 3.2 Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 4 Summary of the RPHS algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 5 The two spiral problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 6 Heuristics for making the RPHS algorithm better . . . . . . . . . . . . . . . . . 141 6.1 Minimal coded genetic algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . 142 6.2 Seperability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 6.3 Computation intensity and population size . . . . . . . . . . . . . . . . . . 143 6.4 Validation data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 6.5 Early stopping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 7 Experiments and results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 7.1 Problems considered . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 7.2 Experimental parameters and control algorithms implemented . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 7.3 Experimental results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 Enhancing Recursive Supervised Learning Using Clustering and Combinatorial Optimization (RSL-CC) Kiruthika Ramanathan and Sheng Uei Guan . . . . . . . . . . . . . . . . . . . . . . . . . 157 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 2 Some preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 2.1 Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 2.2 Problem formulation for recursive learning . . . . . . . . . . . . . . . . . . 160 2.3 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
VIII
Contents
3
The RSL-CC algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 3.1 Pre-training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 3.2 Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 3.3 Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 4 Details of the RSL-CC algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 4.1 Illustration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 4.2 Termination criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 5 Heuristics for improving the performance of the RSL-CC algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 5.1 Minimal coded genetic algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . 169 5.2 Population size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 5.3 Number of generations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 5.4 Duplication of chromosomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 5.5 Problems Considered . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 5.6 Experimental parameters and control algorithms implemented . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 5.7 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 6 Conclusions and future directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 Evolutionary Approaches to Rule Extraction from Neural Networks Urszula Markowska-Kaczmar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 2 The basics of neural networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178 3 Rule extraction from neural networks . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 3.1 Problem formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 3.2 The existing methods of rule extraction from neural networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182 4 Basic concepts of evolutionary algorithms . . . . . . . . . . . . . . . . . . . . . . . 184 5 Evolutionary methods in rule extraction from neural networks . . . . . 185 5.1 Local approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 5.2 Evolutionary algorithms in a global approach to rule extraction from neural networks . . . . . . . . . . . . . . . . . . . . . . . . . . . 189 6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 Cluster-wise Design of Takagi and Sugeno Approach of Fuzzy Logic Controller Tushar and Dilip Kumar Pratihar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213 2 Takagi and Sugeno Approach of FLC . . . . . . . . . . . . . . . . . . . . . . . . . . . 215 3 Genetic Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216 4 Clustering and Linear Regression Analysis Using the Clustered Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
Contents
IX
4.1 Entropy-based Fuzzy Clustering (EFC) . . . . . . . . . . . . . . . . . . . . . 217 4.2 Approach 1: Cluster-wise Linear Regression . . . . . . . . . . . . . . . . . 219 5 GA-based Tuning of Takagi and Sugeno Approach of FLC . . . . . . . . . 220 5.1 Genetic-Fuzzy System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221 6 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 6.1 Modeling of Abrasive Flow Machining (AFM) Process . . . . . . . . 223 6.2 Modeling of Tungsten Inert Gas (TIG) Process . . . . . . . . . . . . . . 231 7 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248 8 Scope for Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248 Evolutionary Fuzzy Modelling for Drug Resistant HIV-1 Treatment Optimization Mattia Prosperi and Giovanni Ulivi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251 1.1 Artificial Intelligence in Medicine . . . . . . . . . . . . . . . . . . . . . . . . . . 251 1.2 Road Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252 2 Background on HIV Treatment and Drug Resistance Onset . . . . . . . . 252 2.1 HIV Replication and Treatment Design . . . . . . . . . . . . . . . . . . . . . 253 2.2 Experimental Settings and Data Collection . . . . . . . . . . . . . . . . . . 254 3 Machine Learning for Drug Resistant HIV . . . . . . . . . . . . . . . . . . . . . . . 255 4 Fuzzy Modelling for HIV Drug Resistance Interpretation . . . . . . . . . . 256 4.1 Fuzzy Medical Diagnosis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258 4.2 Fuzzy Relational System for In-Vitro Cultures . . . . . . . . . . . . . . . 259 4.3 Models for In-Vivo Clinical Data . . . . . . . . . . . . . . . . . . . . . . . . . . 260 5 Optimization Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266 5.1 Fuzzy Genetic Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267 5.2 Random Searches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270 5.3 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270 5.4 Feature Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 6 Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273 6.1 Phenotype Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273 6.2 In-Vivo Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276 6.3 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 7 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285 A New Genetic Approach for Neural Network Design Antonia Azzini and Andrea G.B. Tettamanzi . . . . . . . . . . . . . . . . . . . . . . . . 289 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289 2 Evolving ANNs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291 3 Neuro-Genetic Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297 3.1 Evolutionary Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298 3.2 Individual Encoding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299 3.3 Fitness Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
X
Contents
3.4 Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301 3.5 Mutation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302 3.6 Recombination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304 4 Real-World Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307 4.1 Fault Diagnosis Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307 4.2 Brain Wave Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312 4.3 Financial Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315 5 Conclusion and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320 A Grammatical Genetic Programming Representation for Radial Basis Function Networks Ian Dempsey, Anthony Brabazon, and Michael O’Neill . . . . . . . . . . . . . . . . 325 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325 2 Grammatical Evolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326 3 Radial Basis Function Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328 4 GE-RBFN Hybrid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330 4.1 Grammar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330 4.2 Example Individuals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331 5 Experimental Setup & Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331 6 Conclusions & Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334 A Neural-Genetic Technique for Coastal Engineering: Determining Wave-induced Seabed Liquefaction Depth Daeho Cha, Michael Blumenstein, Hong Zhang, and Dong-Sheng Jeng . . 337 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337 1.1 Artificial Neural Networks in Engineering . . . . . . . . . . . . . . . . . . . 337 1.2 Genetic Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338 1.3 ANN models trained by GAs (Evolutionary Algorithms) . . . . . . 338 1.4 Wave-induced seabed liquefaction . . . . . . . . . . . . . . . . . . . . . . . . . . 339 2 A neural-genetic technique for wave-induced liquefaction . . . . . . . . . . 341 2.1 Data preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342 3 Results and discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344 3.1 Neural-genetic model configuration for wave-induced liquefaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344 3.2 ANN model training using GAs for wave-induced liquefaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346 3.3 Results for determining wave-induced liquefaction . . . . . . . . . . . . 347 4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350
Contents
XI
On the Design of Large-scale Cellular Mobile Networks Using Multi-population Memetic Algorithms Alejandro Quintero and Samuel Pierre . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353 2 Background and related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356 3 Memetic Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359 3.1 Basic Principles of Canonical Genetic Algorithms . . . . . . . . . . . . 359 3.2 Basic Principles of Memetic Algorithms . . . . . . . . . . . . . . . . . . . . . 360 3.3 Multi-population Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361 4 Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362 4.1 Memetic Algorithm Implementation . . . . . . . . . . . . . . . . . . . . . . . . 362 4.2 Local Search Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364 4.3 Experimental Setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366 5 Performance Evaluation and Numerical Results . . . . . . . . . . . . . . . . . . 369 5.1 Comparison with Other Heuristics . . . . . . . . . . . . . . . . . . . . . . . . . 369 5.2 Quality of the Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373 6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374 A Hybrid Cellular Genetic Algorithm for the Capacitated Vehicle Routing Problem Enrique Alba and Bernab´e Dorronsoro . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379 2 The Vehicle Routing Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382 3 The Proposed Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383 3.1 Problem Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385 3.2 Recombination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386 3.3 Mutation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386 3.4 Local Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387 4 Looking for a New Algorithm: the Way to JCell2o1i . . . . . . . . . . . . . . 389 4.1 Cellular vs. Generational Genetic Algorithms . . . . . . . . . . . . . . . . 390 4.2 On the Importance of the Mutation Operator . . . . . . . . . . . . . . . 391 4.3 Tuning the Local Search Step . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393 5 Solving CVRP with JCell2o1i . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395 5.1 Benchmark by Augerat et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397 5.2 Benchmark by Van Breedam . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398 5.3 Benchmark by Christofides and Eilon . . . . . . . . . . . . . . . . . . . . . . 400 5.4 Benchmark by Christofides, Mingozzi and Toth . . . . . . . . . . . . . . 401 5.5 Benchmark by Fisher . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403 5.6 Benchmark by Golden et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403 5.7 Benchmark by Taillard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404 5.8 Benchmark of Translated Instances from TSP . . . . . . . . . . . . . . . 406 6 Conclusions and Further Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 406 7 Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407
XII
A B
Contents
Best Found Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411
Particle Swarm Optimization with Mutation for High Dimensional Problems Jeff Achtnig . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423 1.1 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423 1.2 Particle Swarm Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423 1.3 Curse of Dimensionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424 2 PSO Modifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425 2.1 Mutation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425 2.2 Random Constriction Coefficient . . . . . . . . . . . . . . . . . . . . . . . . . . . 426 3 Experimental Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 426 3.1 Standard Test Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427 3.2 Neural Network Test Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 429 4 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 431 4.1 Comparison with Differential Evolution . . . . . . . . . . . . . . . . . . . . . 437 5 Conclusions and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441
Preface
Evolutionary Computation (EC) has become an important and timely methodology of problem solving among many researchers working in the area of computational intelligence. The population based collective learning process, self adaptation and robustness are some of the key features of evolutionary algorithms when compared to other global optimization techniques. Evolutionary Computation has been widely accepted for solving several important practical applications in engineering, business, commerce, etc. As the optimization problems to be tackled in the future will be growing in terms of complexity and volume of data, we can envision a rapidly growing role of the EC over the passage of time. Evolutionary design of intelligent systems is gaining much popularity due to its capabilities in handling several real world problems involving optimization, complexity, noisy and non-stationary environment, imprecision, uncertainty and vagueness. This edited volume is aimed to present the latest state-of-the-art methodologies in ’Engineering Evolutionary Intelligent Systems’. This book deals with the theoretical and methodological aspects, as well as various EC applications to many real world problems originating from science, technology, business or commerce. This volume comprises of 15 chapters including an introductory chapter which covers the fundamental definitions and outlines some important research challenges. These fifteen chapters are organized as follows. In the first Chapter, Abraham and Grosan elaborate on various schemes of evolutionary design of intelligent systems. Generic hybrid evolutionary intelligent system architectures are presented with a detailed review of some of the interesting hybrid frameworks already reported in the literature. Oh and Pedrycz introduce an advanced architecture of genetically optimized Hybrid Fuzzy Neural Networks (gHFNN) resulting from a synergistic usage of the genetic optimization-driven hybrid system generated by combining Fuzzy Neural Networks (FNN) with Polynomial Neural Networks (PNN). FNNs support the formation of the premise part of the rule-based structure of the gHFNN. The consequence part of the gHFNN is designed using
XIV
Preface
PNNs. The optimization of the FNN is realized with the aid of a standard back-propagation learning algorithm and genetic optimization. In the third Chapter, Oh and Pedrycz introduce Self-Organizing Neural Networks (SONN) that is based on a genetically optimized multilayer perceptron with Polynomial Neurons (PNs) or Fuzzy Polynomial Neurons (FPNs). In the conventional SONN, an evolutionary algorithm is used to extend the main characteristics of the extended Group Method of Data Handling (GMDH) method, that utilizes the polynomial order as well as the number of node inputs fixed at the corresponding nodes (PNs or FPNs) located in each layer during the development of the network. The genetically optimized SONN (gSONN) results in a structurally optimized structure and comes with a higher level of flexibility in comparison to the one encountered in the conventional SONN. Kim and Park discuss a new design methodology of the self-organizing technique which builds upon the use of evolutionary algorithms. The selforganizing network dwells on the idea of group method of data handling. The order of the polynomial, the number of input variables, and the optimal number of input variables and their selection are encoded as a chromosome. The appropriate information of each node is evolved accordingly and tuned gradually using evolutionary algorithms. The evolved network is a sophisticated and versatile architecture, which can construct models from a limited data set as well as from poorly defined complex problems. In the fifth Chapter, Ramanathan and Guan propose the Recursive Pattern-based Hybrid Supervised (RPHS) learning algorithm that makes use of the concept of pseudo global optimal solutions to evolve a set of neural networks, each of which can solve correctly a subset of patterns. The patternbased algorithm uses the topology of training and validation data patterns to find a set of pseudo-optima, each learning a subset of patterns. Ramanathan and Guan improve the RPHP algorithm (as discussed in Chapter 5) by using a combination of genetic algorithm, weak learner and pattern distributor. The global search component is achieved by a clusterbased combinatorial optimization, whereby patterns are clustered according to the output space of the problem. A combinatorial optimization problem is therefore formed, which is solved using evolutionary algorithms. An algorithm is also proposed to use the pattern distributor to determine the optimal number of recursions and hence the optimal number of weak learners suitable for the problem at hand. In the seventh Chapter, Markowska-Kaczmar proposes two methods of rule extraction referred to as REX and GEX. REX uses propositional fuzzy rules and is composed of two methods REX Michigan and REX Pitt. GEX takes an advantage of classical Boolean rules. The efficiency of REX and GEX were tested using different benchmark data sets coming from the UCI repository. Tushar and Pratihar deals with Takagi and Sugeno Fuzzy Logic Controllers (FLC) by focusing on their design process. This development by
Preface
XV
clustering the data based on their similarity among themselves and then cluster-wise regression analysis is carried out, to determine the response equations for the consequent part of the rules. The performance of the developed cluster-wise linear regression approach; cluster-wise Takagi and Sugeno model of FLC with linear membership functions and cluster-wise Takagi and Sugeno model of FLC with nonlinear membership functions are illustrated using two practical problems. In the ninth Chapter, Prosperi and Ulivi propose fuzzy relational models for genotypic drug resistance analysis in Human Immunodeficiency Virus type 1 (HIV-1). Fuzzy logic is introduced to model high-level medical language, viral and pharmacological dynamics. Fuzzy evolutionary algorithms and fuzzy evaluation functions are proposed to mine resistance rules, to improve computational performance and to select relevant features. Azzini and Tettamanzi present an approach to the joint optimization of neural network structure and weights, using backpropagation algorithm as a specialized decoder, and defining a simultaneous evolution of architecture and weights of neural networks. In the eleventh Chapter, Dempsey et al. present grammatical genetic programming to generate radial basis function networks. Authors tested the hybrid algorithm considering several benchmark classification problems reporting on encouraging performance obtained there. In the sequel Cha et al. propose neural-genetic model to wave-induced liquefaction, which provides a better prediction of liquefaction potential. The wave-induced seabed liquefaction problem is one of the most critical issues for analyzing and designing marine structures such as caissons, oil platforms and harbors. In the past, various investigations into wave-induced seabed liquefaction have been carried out including numerical models, analytical solutions and some laboratory experiments. However, most previous numerical studies are based on solving complicated partial differential equations. The neuralgenetic simulation results illustrate the applicability of the hybrid technique for the accurate prediction of wave-induced liquefaction depth, which can also provide coastal engineers with alternative tools to analyze the stability of marine sediments. In the thirteenth Chapter, Quintero and Pierre propose a multi-population Memetic Algorithm (MA) with migration and elitism to solve the problem of assigning cells to switches as a design step of large-scale mobile networks. Being well-known in the literature to be an NP-hard combinatorial optimization problem, this task requires the recourse to heuristic methods, which can practically lead to good feasible solutions, not necessarily optimal, the objective being rather to reduce the convergence time toward these solutions. Computational results reported on an extensive suite of extensive tests confirm the efficiency and the effectiveness of MA to provide good solutions in comparison with other heuristics well-known in the literature, especially those for large-scale cellular mobile networks.
XVI
Preface
Alba and Dorronsoro solve the Capacitated Vehicle Routing Problem (CVRP) of 160 instances using a Cellular genetic algorithm (cGA) hybridized with a problem customized recombination operation, an advanced mutation operator integrating three mutation methods, and an inclusion of two well-known local search algorithms formulated for routing problems. In the last Chapter, Achtnig investigates the use of Particle Swarm Optimization (PSO) in dealing with optimization problems of very high dimension. It has been found that PSO with some of the concepts originating from evolutionary algorithms, such as a mutation operator, can in many cases significantly improve the performance of the PSO. Further improvements have been reported with the addition of a random constriction coefficient. We are very much grateful to all the authors of this volume for sharing their expertise and presenting their recent research findings. Our thanks go to the referees for their outstanding service and a wealth of critical yet highly constructive comments. The Editors would like to thank Dr. Thomas Ditzinger (Springer Engineering In house Editor, Studies in Computational Intelligence Series), Professor Janusz Kacprzyk (Editor-in-Chief, Springer Studies in Computational Intelligence Series) and Ms. Heather King (Editorial Assistant, Springer Verlag, Heidelberg) for the editorial assistance and excellent collaboration during the development of this volume. We hope that the reader will share our excitement and find the volume ‘Engineering Evolutionary Intelligent Systems’ both useful and inspiring. Trondheim, Norway Cluj-Napoca, Romania Alberta, Canada
Ajith Abraham Crina Grosan Witold Pedrycz
Contributors
Ajith Abraham Center of Excellence for Quantifiable Quality of Service Norwegian University of Science and Technology, Trondheim Norway
[email protected] Jeff Achtnig Nalisys (Research Division)
[email protected] Enrique Alba Department of Computer Science University of M´ alaga
[email protected] Antonia Azzini Universit` a degli Studi di Milano Dipartimento di Tecnologie dell’Informazione via Bramante 65 26013, Crema - Italy
[email protected] Michael Blumenstein School of Information and Communication Technology Griffith University Gold Coast Campus, QLD 4215 Australia
[email protected]
Anthony Brabazon Natural Computing Research and Applications Group University College Dublin Ireland
[email protected]
Daeho Cha Griffith School of Engineering Griffith University, Gold Coast Campus QLD 4215, Australia
[email protected]
Ian Dempsey Natural Computing Research and Applications Group University College Dublin Ireland ian.dempsey@PipelineFinancial. com
Bernab´ e Dorronsoro Department of Computer Science University of M´ alaga
[email protected]
XVIII Contributors
Crina Grosan Department of Computer Science Faculty of Mathematics and Computer Science Babes-Bolyai University Cluj-Napoca, Romania
[email protected] Sheng Uei Guan School of Engineering and Design Brunel University, Uxbridge Middlesex, UB8 3PH, UK
[email protected] Dong-Sheng Jeng School of Civil Engineering The University of Sydney NSW 2006 Australia
[email protected] Dongwon Kim Department of Electrical Engineering Korea University, 1 5-ka, Anam-dong Seongbuk-ku, Seoul 136-701 Korea
[email protected] Urszula Markowska-Kaczmar Wroclaw University of Technology Institute of Applied Informatics Wyb. Wyspianskiego 27, 50-370 Wroclaw, Poland urszula.markowska-kaczmar@pwr. wroc.pl Sung-Kwun Oh Department of Electrical Engineering The University of Suwon San 2-2, Wau-ri Bongdam-eup, Hwaseong-si Gyeonggi-do, 445-743 South Korea
[email protected]
Michael O’Neill Natural Computing Research and Applications Group University College Dublin Ireland
[email protected] Gwi-Tae Park Department of Electrical Engineering Korea University, 1 5-ka, Anam-dong Seongbuk-ku, Seoul 136-701 Korea
[email protected] Witold Pedrycz Department of Electrical and Computer Engineering University of Alberta, Edmonton AB, Canada T6G 2G6 and Systems Research Institute Polish Academy of Sciences Warsaw, Poland
[email protected] Samuel Pierre Department of Computer Engineering ´ Ecole Polytechnique de Montr´eal C.P. 6079, succ. Centre-Ville Montr´eal, Qu´e. Canada, H3C 3A7
[email protected] Dilip Kumar Pratihar Department of Mechanical Engineering Indian Institute of Technology Kharagpur-721 302 West Bengal, India
[email protected]
Contributors
Mattia Prosperi University of Roma TRE Faculty of Computer Science Engineering Department of Computer Science and Automation
[email protected]
Alejandro Quintero Department of Computer Engineering ´ Ecole Polytechnique de Montr´eal C.P. 6079, succ. Centre-Ville Montr´eal, Qu´e. Canada, H3C 3A7
[email protected]
Kiruthika Ramanathan Department of Electrical and Computer Engineering National University of Singapore 10 Kent Ridge Crescent, Singapore 119260 kiruthika
[email protected]
XIX
Andrea G.B. Tettamanzi Universit` a degli Studi di Milano Dipartimento di Tecnologie dell’Informazione via Bramante 65 26013, Crema - Italy
[email protected] Tushar Department of Mechanical Engineering Indian Institute of Technology Kharagpur-721 302 West Bengal, India
[email protected] Giovanni Ulivi University of Roma TRE Faculty of Computer Science Engineering Department of Computer Science and Automation
[email protected] Hong Zhang Griffith School of Engineering Griffith University, Gold Coast Campus QLD 4215, Australia
[email protected]
Engineering Evolutionary Intelligent Systems: Methodologies, Architectures and Reviews Ajith Abraham and Crina Grosan
Summary. Designing intelligent paradigms using evolutionary algorithms is getting popular due to their capabilities in handling several real world problems involving complexity, noisy environment, imprecision, uncertainty and vagueness. In this Chapter, we illustrate the various possibilities for designing intelligent systems using evolutionary algorithms and also present some of the generic evolutionary design architectures that has evolved during the last couple of decades. We also provide a review of some of the recent interesting evolutionary intelligent system frameworks reported in the literature.
1 Introduction Evolutionary Algorithms (EA) have recently received increased interest, particularly with regard to the manner in which they may be applied for practical problem solving. Usually grouped under the term evolutionary computation or evolutionary algorithms, we find the domains of Genetic Algorithms [34], Evolution Strategies [68], [69], Evolutionary Programming [20], Learning Classifier Systems [36], Genetic Programming [45], Differential Evolution [67] and Estimation of Distribution Algorithms [56]. They all share a common conceptual base of simulating the evolution of individual structures and they differ in the way the problem is represented, processes of selection and the usage/implementation of reproduction operators. The processes depend on the perceived performance of the individual structures as defined by the problem. Compared to other global optimization techniques, evolutionary algorithms are easy to implement and very often they provide adequate solutions. A population of candidate solutions (for the optimization task to be solved) is initialized. New solutions are created by applying reproduction operators (mutation and/or crossover). The fitness (how good the solutions are) of the resulting solutions are evaluated and suitable selection strategy is then applied to determine, which solutions are to be maintained into the next generation. The procedure is then iterated. A. Abraham and C. Grosan: Engineering Evolutionary Intelligent Systems: Methodologies, Architectures and Reviews, Studies in Computational Intelligence (SCI) 82, 1–22 (2008) c Springer-Verlag Berlin Heidelberg 2008 www.springerlink.com
2
A. Abraham and C. Grosan
The rest of the chapter is organized as follows. In Section 2, the various architectures for engineering evolutionary intelligent systems are presented. In Section 3, we present evolutionary artificial neural networks and its recent applications followed by evolutionary fuzzy systems and applications in Section 4. Evolutionary clustering is presented in Section 5 followed by recent applications of evolutionary design of complex paradigms in Section 6. Multiobjective evolutionary design intelligent systems are presented in Section 7 and some conclusions are provided towards the end.
2 Architectures of Evolutionary Intelligent Systems Hybridization of evolutionary algorithms with other intelligent paradigms is a promising research field and various architectures for Evolutionary Intelligent Systems (EIS) could be formulated as depicted in Figures 1–6. By problem, we refer to any data mining/optimization/function approximation type problem and Intelligent Paradigm (IP) refers to any computational intelligence techniques like neural network, machine learning schemes, fuzzy inference systems, clustering algorithms etc.
Evolutionary algorithm
Intelligent paradigm
Problem / Data Solution (Output)
Fig. 1. EIS architecture 1
Evolutionary algorithm
Intelligent paradigm
Problem / Data
Fig. 2. EIS architecture 2
Solution (Output)
Engineering Evolutionary Intelligent Systems
Solution (Output)
Intelligent paradigm
Problem / Data
Evolutionary algorithm
Fig. 3. EIS architecture 3
Intelligent paradigm
Solution (Output)
Error feedback
Problem / Data
Evolutionary algorithm
Fig. 4. EIS architecture 4
Intelligent paradigm
Problem / Data Evolutionary algorithm
Fig. 5. EIS architecture 5
3
Solution (Output)
4
A. Abraham and C. Grosan
Intelligent paradigm
Problem / Data
Solution (Output)
Error feedback
Evolutionary algorithm
Fig. 6. EIS architecture 6
Figure 1 illustrates a transformational architecture where an evolutionary algorithm is used to optimize an intelligent paradigm and at the same time the intelligent paradigm is used to fine tune the parameters and performance of the evolutionary algorithm. An example is an evolutionary - fuzzy system where an evolutionary algorithm is used to fine tune the parameters of a fuzzy inference system (for a function approximation problem) and the fuzzy system is used to control the parameters of the evolutionary algorithm [32]. A concurrent hybrid architecture is depicted in Figure 2, where an EA is used as a pre-processor and the intelligent paradigm is used to fine tune the solutions formulated by the EA. The final solutions to the problem is provided by IP. Both EA and IP are continuously required for the satisfactory performance of the system. EA may be used as a post processor as illustrated in [1], [4] and [28]. Architecture 3 (Figure 3) depicts a cooperative hybrid system where the evolutionary algorithm is used to fine tune the parameters of IP only during the initialization of the system. Once the system is initialized the EA is not required for the satisfactory functioning of the system. Architecture 4 uses an error feed back from the output (performance measure) and based on the error measure (critic information) the EA is used to fine tune the performance of the IP. Final solutions are provided by the the IP as illustrated in Figure 4. An ensemble model is depicted in Figure 5 where EA received inputs directly from the IP and independent solutions are provided by the EA and IP. A slightly different architecture is depicted in Figure 6, where an error feed back is generated by the EA and depending on this input IP performance is fine tuned and final solutions are provided by the IP. In the following Section, some of the well established hybrid frameworks for optimizing the performance of evolutionary algorithm using intelligent paradigms are presented.
Engineering Evolutionary Intelligent Systems
5
3 Evolutionary Artificial Neural Networks Artificial neural networks are capable of performing a wide variety of tasks, yet in practice sometimes they deliver only marginal performance. Inappropriate topology selection and learning algorithm are frequently blamed. There is little reason to expect that one can find a uniformly best algorithm for selecting the weights in a feedforward artificial neural network. This is in accordance with the no free lunch theorem, which explains that for any algorithm, any elevated performance over one class of problems is exactly paid for in performance over another class [75]. At present, neural network design relies heavily on human experts who have sufficient knowledge about the different aspects of the network and the problem domain. As the complexity of the problem domain increases, manual design becomes more difficult and unmanageable. Evolutionary design of artificial neural networks has been widely explored. Evolutionary algorithms are used to adapt the connection weights, network architecture and learning rules according to the problem environment. A distinct feature of evolutionary neural networks is their adaptability to a dynamic environment. In other words, such neural networks can adapt to an environment as well as changes in the environment. The two forms of adaptation: evolution and learning in evolutionary artificial neural networks make their adaptation to a dynamic environment much more effective and efficient than the conventional learning approach. In Evolutionary Artificial Neural Network (EANN), evolution can be introduced at various levels. At the lowest level, evolution can be introduced into weight training, where ANN weights are evolved. At the next higher level, evolution can be introduced into neural network architecture adaptation, where the architecture (number of hidden layers, no of hidden neurons and node transfer functions) is evolved. At the highest level, evolution can be introduced into the learning mechanism [77]. A general framework of EANNs which includes the above three levels of evolution is given in Figure 7 [2]. From the point of view of engineering, the decision on the level of evolution depends on what kind of prior knowledge is available. If there is more prior knowledge about EANN’s architectures that about their learning rules or
Evolutionary Search of learning rules
Slow
Evolutionary search of architectures and node transfer functions
Evolutionary search of connection weights
Fast
Fig. 7. A General framework for evolutionary artificial neural network
6
A. Abraham and C. Grosan
a particular class of architectures is pursued, it is better to implement the evolution of architectures at the highest level because such knowledge can be used to reduce the search space and the lower level evolution of learning rules can be more biased towards this kind of architectures. On the other hand, the evolution of learning rules should be at the highest level if there is more prior knowledge about them available or there is a special interest in certain type of learning rules. 3.1 Evolutionary Search of Connection Weights The neural network training process is formulated as a global search of connection weights towards an optimal set defined by the evolutionary algorithm. Optimal connection weights can be formulated as a global search problem wherein the architecture of the neural network is pre-defined and fixed during the evolution. Connection weights may be represented as binary strings or real numbers. The whole network is encoded by concatenation of all the connection weights of the network in the chromosome. A heuristic concerning the order of the concatenation is to put connection weights to the same node together. Proper genetic operators are to be chosen depending upon the representation used. While gradient based techniques are very much dependant on the initial setting of weights, evolutionary search method can be considered generally much less sensitive to initial conditions. When compared to any gradient descent or second order optimization technique that can only find local optimum in a neighborhood of the initial solution, evolutionary algorithms always try to search for a global optimal solution. Evolutionary search for connection weights is depicted in Algorithm 3.1. Algorithm 3.1 Evolutionary search of connection weights 1. Generate an initial population of N weight chromosomes. Evaluate the fitness of each EANN depending on the problem. 2. Depending on the fitness and using suitable selection methods reproduce a number of children for each individual in the current generation. 3. Apply genetic operators to each child individual generated above and obtain the next generation. 4. Check whether the network has achieved the required error rate or the specified number of generations has been reached. Go to Step 2. 5. End
Engineering Evolutionary Intelligent Systems
7
3.2 Evolutionary Search of Architectures Evolutionary architecture adaptation can be achieved by constructive and destructive algorithms. Constructive algorithms, which add complexity to the network starting from a very simple architecture until the entire network is able to learn the task [23], [52]. Destructive algorithms start with large architectures and remove nodes and interconnections until the ANN is no longer able to perform its task [58], [66]. Then the last removal is undone. For an optimal network, the required node transfer function (Gaussian, sigmoidal, etc.) can be formulated as a global search problem, which is evolved simultaneously with the search for architectures [49]. To minimize the size of the genotype string and improve scalability, when priori knowledge of the architecture is known it will be efficient to use some indirect coding (high level) schemes. For example, if two neighboring layers are fully connected then the architecture can be coded by simply using the number of layers and nodes. The blueprint representation is a popular indirect coding scheme where it assumes architecture consists of various segments or areas. Each segment or area will define a set of neurons, their spatial arrangement and their efferent connectivity. Several high level coding schemes like graph generation system [44], Symbiotic Adaptive Neuro-Evolution (SANE) [54], marker based genetic coding [24], L-systems [10], cellular encoding [26], fractal representation [53], cellular automata [29] etc. are some of the rugged techniques. Global search of transfer function and the connectivity of the ANN using evolutionary algorithms is formulated in Algorithm 3.2. The evolution of architectures has to be implemented such that the evolution of weight chromosomes are evolved at a faster rate i.e. for every architecture chromosome, there will be several weight chromosomes evolving at a faster time scale. Algorithm 3.2 Evolutionary search of architectures 1. Generate an initial population of N architecture chromosomes. Evaluate the fitness of each EANN depending on the problem. 2. Depending on the fitness and using suitable selection methods reproduce a number of children for each individual in the current generation. 3. Apply genetic operators to each child individual generated above and obtain the next generation. 4. Check whether the network has achieved the required error rate or the specified number of generations has been reached. Go to Step 2. 5. End
8
A. Abraham and C. Grosan
3.3 Evolutionary Search of Learning Rules For the neural network to be fully optimal the learning rules are to be adapted dynamically according to its architecture and the given problem. Deciding the learning rate and momentum can be considered as the first attempt of learning rules [48]. The basic learning rule can be generalized by the function ⎛ ⎞ n n k ⎝θi1 ,i2 ,...,ik xij (t − 1)⎠ (1) ∆w(t) = k=1 i1 ,i2 ,...,ik =1
j=1
where t is the time, ∆w is the weight change, x1 , x2 ,. . . .. x n are local variables and the θ’ s are the real values coefficients which is to be determined by the global search algorithm. In the above equation, different values of θ’ s determine different learning rules. The above equation is arrived based on the assumption that the same rule is applicable at every node of the network and the weight updating is only dependent on the input/output activations and the connection weights on a particular node. The evolution of learning rules has to be implemented such that the evolution of architecture chromosomes are evolved at a faster rate i.e. for every learning rule chromosome, there will be several architecture chromosomes evolving at a faster time scale. Genotypes (θ’ s) can be encoded as real-valued coefficients and the global search for learning rules using the evolutionary algorithm is formulated in Algorithm 3.3. In the literature, several research works could be traced about how to formulate different optimal learning rules [8], [21]. The adaptive adjustment of back-propagation algorithm parameters, such as the learning rate and momentum, through evolution could be considered as the first attempt of the evolution of learning rules [30]. Sexton et al. [65] used simulated annealing algorithm for optimization of learning. For optimization of the neural network Algorithm 3.3 Evolutionary search of learning algorithms or rules 1. Generate an initial population of N learning rules. Evaluate the fitness of each EANN depending on the problem. 2. Depending on the fitness and using suitable selection methods reproduce a number of children for each individual in the current generation. 3. Apply genetic operators to each child individual generated above and obtain the next generation. 4. Check whether the network has achieved the required error rate or the specified number of generations has been reached. Go to Step 2. 5. End
Engineering Evolutionary Intelligent Systems
9
learning, in many cases a pre-defined architecture was used and in a few cases architectures were evolved together. Abraham [2] proposed the meta-learning evolutionary evolutionary neural network with a tight interaction of the different evolutionary search mechanisms using the generic framework illustrated in Figure 7. 3.4 Recent Applications of Evolutionary Neural Networks in Practice Cai et al. [11] used a hybrid of Particle Swarm Optimization (PSO) [41], [18] and EA to train Recurrent Neural Networks (RNNs) for the prediction of missing values in time series data. Experimental results illustrate that RNNs, trained by the hybrid algorithm, are able to predict the missing values in the time series with minimum error, in comparison with those trained with standard EA and PSO algorithms. Castillo et al. [12] explored several methods that combine evolutionary algorithms and local search to optimize multilayer perceptrons. Authors explored a method that optimizes the architecture and initial weights of multilayer perceptrons, a search algorithm for training algorithm parameters, and finally, a co-evolutionary algorithm, that handles the architecture, the network’s initial weights and the training algorithm parameters. Experimental results show that the co-evolutionary method obtains similar or better results than the other approaches, requiring far less training epochs and thus, reducing running time. Hui [37] proposed a new method for predicting the reliability for repairable systems using evolutionary neural networks. Genetic algorithms are used to globally optimize the number of neurons in the hidden layer and learning parameters of the neural network architecture. Marwala [51] proposed a Bayesian neural network trained using Markov Chain Monte Carlo (MCMC) and genetic programming in binary space within Metropolis framework. The proposed algorithm could learn using samples obtained from previous steps merged using concepts of natural evolution which include mutation, crossover and reproduction. The reproduction function is the Metropolis framework and binary mutation as well as simple crossover, are also used. Kim and Cho [42] proposed an incremental evolution method for neural networks based on cellular automata and a method of combining several evolved modules by a rule-based approach. The incremental evolution method evolves the neural network by starting with simple environment and gradually making it more complex. The multi-modules integration method can make complex behaviors by combining several modules evolved or programmed to do simple behaviors. Kim [43] explored a genetic algorithm approach to instance selection in artificial neural networks when the amount of data is very large. GA optimizes
10
A. Abraham and C. Grosan
simultaneously the connection weights and the optimal selection of relevant instances. Capi and Doya [13] implemented an extended multi-population genetic algorithm (EMPGA), where subpopulations apply different evolutionary strategies for designing neural controllers in the real hardware of Cyber Rodent robot. The EMPGA subpopulations compete and cooperate among each other. Bhattacharya et al. [7] used a meta-learning evolutionary artificial neural network in selecting the best Flexible Manufacturing Systems (FMS) from a group of candidate FMSs. EA is used to evolve the architecture and weights of the proposed neural network method. Further, a Back-Propagation (BP) algorithm is used as the local search algorithm. All the randomly generated architectures of the initial population are trained by BP algorithm for a fixed number of epochs. The learning rate and momentum of the BP algorithm have been adapted suiting the generated data of the MCDM problem.
4 Evolutionary Fuzzy Systems A conventional fuzzy controller makes use of a model of the expert who is in a position to specify the most important properties of the process. Fuzzy controller consists of a fuzzification interface, which receives the current values of the input variables and eventually transforms to linguistic terms or fuzzy sets. The knowledge base contains information about the domains of the variables, and the fuzzy sets associated with the linguistic terms. Also a rule base in the form of linguistic control rules is stored in the knowledge base. The decision logic determines the information about the control variables with the help of the measured input values and knowledge base. The task of defuzzification interface is to create a crisp control value out of the information about the control variable of the decision logic by using a suitable transformation. The usual approach in fuzzy control is to define a number of concurrent if-then fuzzy rules. Most fuzzy systems employ the inference method proposed by Mamdani [50] in which the rule consequence is defined by fuzzy sets and has the following structure: if x is A 1 and y is B 1 then f = C
(2)
Takagi, Sugeno and Kang (TSK) [72] proposed an inference scheme in which the conclusion of a fuzzy rule is constituted by a weighted linear combination of the crisp inputs rather than a fuzzy set and has the following structure: (3) if x is A 1 and y is B 1 , then f = p 1 x + q 1 y + r In the literature, several research works related to evolutionary design of fuzzy system could be located [59], [62]. Majority of the works are concerned with the automatic design or optimization of fuzzy logic controllers either by
Engineering Evolutionary Intelligent Systems
11
adapting the fuzzy membership functions or by learning the fuzzy if-then rules [55], [33]. Figure 8 shows the architecture of the adaptive fuzzy control system wherein the fuzzy membership functions and the rule bases are optimized using a hybrid global search procedure. An optimal design of an adaptive fuzzy control system could be achieved by the adaptive evolution of membership functions and the learning rules that progress on different time scales. Figure 9 illustrates the general interaction mechanism with the global search of fuzzy rules evolving at the highest level on the slowest time scale. For each fuzzy rule base, global search of membership functions proceeds at a faster time scale in an environment decided by the problem.
Evolutionary search (Adaptation of fuzzy sets and rule base)
Performance measure
Fuzzy sets + -
if-then rules
Process
Knowledge base Fuzzy controller
Fig. 8. Adaptive fuzzy control system architecture
Slow Evolutionary search of fuzzy rules Evolutionary search of membership functions Fast
Fig. 9. Interaction of various search mechanisms in the design of optimal adaptive fuzzy control system
12
A. Abraham and C. Grosan
4.1 Evolutionary Search of Fuzzy Membership Functions The tuning of the scaling parameters and fuzzy membership functions (piecewise linear and/or differentiable functions) is an important task in the design of fuzzy systems and is popularly known as genetic tuning. Evolutionary algorithms could be used to search the optimal shape, number of membership functions per linguistic variable and the parameters [31]. The genome encodes parameters of trapezoidal, triangle, logistic, Laplace, hyperbolic-tangent or Gaussian membership functions etc. Most of the existing methods assume the existence of a predefined collection of fuzzy membership functions giving meaning to the linguistic labels contained in the rules (database). Evolutionary algorithms are applied to obtain a suitable rule base, using chromosomes that code single rules or complete rule bases. If prior knowledge of the membership functions is available, a simplified chromosome representation could be formulated accordingly. The first decision a designer has to make is how to represent a solution in a chromosome structure. First approach is to have the chromosome encode the complete rule base. Each chromosome differs only in the fuzzy rule membership functions as defined in the database. In the second approach, each chromosome encodes a different database definition based on the fuzzy domain partitions. The global search for membership functions using evolutionary algorithm is formulated in Algorithm 4.1. 4.2 Evolutionary Search of Fuzzy Rule Base The number of rules grows rapidly with an increasing number of variables and fuzzy sets. Literature scan reveals that several coding methods were used Algorithm 4.1 Evolution of learning of fuzzy membership functions and its parameters 1. Generate an initial population of N chromosomes using one of the approaches mentioned in Section 4.1. Evaluate the fitness of each fuzzy rule base depending on the problem. 2. Depending on the fitness and using suitable selection methods reproduce a number of children for each individual in the current generation. 3. Apply genetic operators to each child individual generated above and obtain the next generation. 4. Check whether the fuzzy system has achieved the required error rate or the specified number of generations has been reached. Go to Step 2. 5. End
Engineering Evolutionary Intelligent Systems
13
according to the nature of the problem. The rule base of the fuzzy system may be represented using relational matrix, decision table and set of rules. In the Pittsburg approach, [71] each chromosome encodes a whole rule set. Crossover serves to provide a new combination of rules and mutation provides new rules. The disadvantage is the increased complexity of search space and additional computational burden especially for online learning. The size of the genotype depends on the number of input/output variables and fuzzy sets. In the Michigan approach, [35] each genotype represents a single fuzzy rule and the entire population represents a solution. The fuzzy knowledge base is adapted as a result of antagonistic roles of competition and cooperation of fuzzy rules. A classifier rule triggers whenever its condition part matches the current input, in which case the proposed action is sent to the process to be controlled. The fuzzy behavior is created by an activation sequence of mutually collaborating fuzzy rules. In the Michigan approach, techniques for judging the performance of single rules are necessary. The Iterative Rule Learning (IRL) approach [27] is similar to the Michigan approach where the chromosomes encode individual rules. In IRL, only the best individual is considered as the solution, discarding the remaining chromosomes in the population. The evolutionary algorithm generates new classifier rules based on the rule strengths acquired during the entire process. Defuzzification operators and its parameters may be also formulated as an evolutionary search [46], [40], [5]. 4.3 Recent Applications of Evolutionary Fuzzy Systems in Practice Tsang et al. [73] proposed a fuzzy rule-based system for intrusion detection, which is evolved from an agent-based evolutionary framework and multiobjective optimization. The proposed system can also act as a genetic feature selection wrapper to search for an optimal feature subset for dimensionality reduction. Edwards et al. [19] modeled the complex export pattern behavior of multinational corporation subsidiaries in Malaysia using a Takagi-Sugeno fuzzy inference system. The proposed fuzzy inference system is optimized by using neural network learning and evolutionary computation. Empirical results clearly show that the proposed approach could model the export behavior reasonably well compared to a direct neural network approach. Chen et al. [15] proposed an automatic way of evolving hierarchical Takagi - Sugeno Fuzzy Systems (TS-FS). The hierarchical structure is evolved using Probabilistic Incremental Program Evolution (PIPE) with specific instructions. The fine tuning of the if - then rules parameters encoded in the structure is accomplished using Evolutionary Programming (EP). The proposed method interleaves both PIPE and EP optimizations. Starting with random structures and rules parameters, it first tries to improve the hierarchical structure and then as soon as an improved structure is found, it further fine tunes the rules parameters. It then goes back to improve the structure
14
A. Abraham and C. Grosan
and the rules’ parameters. This loop continues until a satisfactory hierarchical TS-FS model is found or a time limit is reached. Pawara and Ganguli [61] developed a Genetic Fuzzy System (GFS) for online structural health monitoring of composite helicopter rotor blades. Authors formulated a global and local GFSs. The global GFS is for matrix cracking and debonding/delamination detection along the whole blade and the local GFS is for matrix cracking and debonding/delamination detection in various parts of the blade. Chua et al. [16] proposed a GA-based fuzzy controller design for tunnel ventilation systems. Fuzzy Logic Control (FLC) method has been utilized due to the complex and nonlinear behavior of the system and the FLC was optimized using the GA. Franke et al. [22] presented a genetic - fuzzy system for automatically generating online scheduling strategies for a complex objective defined by a machine provider. The scheduling algorithm is based on a rule system, which classifies all possible scheduling states and assigns a corresponding scheduling strategy. Authors compared two different approaches. In the first approach, an iterative method is applied, that assigns a standard scheduling strategy to all situation classes. In the second approach, a symbiotic evolution varies the parameter of Gaussian membership functions to establish the different situation classes and also assigns the appropriate scheduling strategies.
5 Evolutionary Clustering Clustering means the act of partitioning an unlabeled dataset into groups of similar objects. Each group, called a ‘cluster’, consists of objects that are similar between themselves and dissimilar to objects of other groups. A comprehensive review of the state-of-the-art clustering methods can be found in [76], [64]. Data clustering is broadly based on two approaches: hierarchical and partitional. In hierarchical clustering, the output is a tree showing a sequence of clustering with each cluster being a partition of the data set. Hierarchical algorithms can be agglomerative (bottom-up) or divisive (top-down). Agglomerative algorithms begin with each element as a separate cluster and merge them in successively larger clusters. Partitional clustering algorithms, on the other hand, attempt to decompose the data set directly into a set of disjoint clusters by optimizing certain criteria. The criterion function may emphasize the local structure of the data, as by assigning clusters to peaks in the probability density function, or the global structure. Typically, the global criteria involve minimizing some measure of dissimilarity in the samples within each cluster, while maximizing the dissimilarity of different clusters. The advantages of the hierarchical algorithms are the disadvantages of the partitional algorithms and vice versa.
Engineering Evolutionary Intelligent Systems
15
Clustering can also be performed in two different modes: crisp and fuzzy. In crisp clustering, the clusters are disjoint and non-overlapping in nature. Any pattern may belong to one and only one class in this case. In case of fuzzy clustering, a pattern may belong to all the classes with a certain fuzzy membership grade. One of the widely used clustering methods is the fuzzy c-means (FCM) algorithm developed by Bezdek [9]. FCM partitions a collection of n vectors xi , i = 1, 2 . . . , n into c fuzzy groups and finds a cluster center in each group such that a cost function of dissimilarity measure is minimized. To accommodate the introduction of fuzzy partitioning, the membership matrix U is allowed to have elements with values between 0 and 1. The FCM objective function takes the form: c c n 2 J(U, c1 , . . . cc ) = Ji = um (4) ij dij i=1
i=1 j=1
where uij , is a numerical value between [0,1]; ci is the cluster center of fuzzy group i; dij = ci − xj is the Euclidian distance between ith cluster center and j th data point; and m is called the exponential weight which influences the degree of fuzziness of the membership (partition) matrix. Usually a number of cluster centers are randomly initialized and the FCM algorithm provides an iterative approach to approximate the minimum of the objective function starting from a given position and leads to any of its local minima [3]. No guarantee ensures that FCM converges to an optimum solution (can be trapped by local extrema in the process of optimizing the clustering criterion). The performance is very sensitive to initialization of the cluster centers. Research efforts have made it possible to view data clustering as an optimization problem. This view offers us a chance to apply EA for evolving the optimal number of clusters and their cluster centers. The algorithm is initialized by constraining the initial values to be within the space defined by the vectors to be clustered. An important advantage of the EA is its ability to cope with local optima by maintaining, recombining and comparing several candidate solutions simultaneously. Abraham [3] proposed the concurrent architecture of a fuzzy clustering algorithm (to discover data clusters) and a fuzzy inference system for Web usage mining. A hybrid evolutionary FCM approach is proposed in this paper to optimally segregate similar user interests. The clustered data is then used to analyze the trends using a Takagi-Sugeno fuzzy inference system learned using a combination of evolutionary algorithm and neural network learning.
6 Recent Applications of Evolutionary Design of Complex Paradigms Park et al. [60] used EA to optimize Hybrid Self-Organizing Fuzzy Polynomial Neural Networks (HSOFPNN)m, which are based on genetically optimized multi-layer perceptrons. The architecture of the resulting HSOFPNN
16
A. Abraham and C. Grosan
combines fuzzy polynomial neurons (FPNs) [57] that are located at the first layer of the network with polynomial neurons (PNs) forming the remaining layers of the network. The GA-based design procedure being applied at each layer of HSOFPNN leads to the selection of preferred nodes of the network (FPNs or PNs) whose local characteristics (such as the number of input variables, the order of the polynomial, a collection of the specific subset of input variables, the number of membership functions for each input variable, and the type of membership function) can be easily adjusted. Juang and Chung [39] proposed a recurrent TakagiSugenoKang (TSK) fuzzy network design using the hybridization of a multi-group genetic algorithm and particle swarm optimization (R-MGAPSO). Both the number of fuzzy rules and the parameters in a TRFN are designed simultaneously by R-MGAPSO. In R-MGAPSO, the techniques of variable-length individuals and the local version of particle swarm optimization are incorporated into a genetic algorithm, where individuals with the same length constitute the same group, and there are multigroups in a population. Aouiti et al. [6] proposed an evolutionary method for the design of beta basis function neural networks (BBFNN) and beta fuzzy systems (BFS). Authors used a hierarchical genetic learning model of the BBFNN and the BFS. Chen at al. [14] introduced a new time-series forecasting model based on the flexible neural tree (FNT). The FNT model is generated initially as a flexible multi-layer feed-forward neural network and evolved using an evolutionary procedure. FNT model could also select the appropriate input variables or time-lags for constructing a time-series model.
7 Multiobjective Evolutionary Design of Intelligent Paradigms Even though some real world problems can be reduced to a matter of single objective very often it is hard to define all the aspects in terms of a single objective. In single objective optimization, the search space is often well defined. As soon as there are several possibly contradicting objectives to be optimized simultaneously, there is no longer a single optimal solution but rather a whole set of possible solutions of equivalent quality. When we try to optimize several objectives at the same time the search space also becomes partially ordered. To obtain the optimal solution, there will be a set of optimal trade-offs between the conflicting objectives. A multiobjective optimization problem is defined by a function f which maps a set of constraint variables to a set of objective values. Delgado and Pegalajar [17], developed a multi-objective evolutionary algorithm, which is able to determine the optimal size of recurrent neural networks for any particular application. Authors analyzed in the case of grammatical inference: in particular, how to establish the optimal size of a recurrent neural network in order to learn positive and negative examples in a certain language,
Engineering Evolutionary Intelligent Systems
17
and how to determine the corresponding automaton using a self-organizing map once the training has been completed. Serra and Bottura [70] proposed a gain scheduling adaptive control scheme based on fuzzy systems, neural networks and multiobjective genetic algorithms for nonlinear plants. A FLC is developed, which is a discrete time version of a conventional one. Its data base as well as the controller gains are optimally designed by using a genetic algorithm for simultaneously satisfying the overshoot and settling time minimizations and output response smoothing. Kelesoglu [47] developed a method for solving fuzzy multiobjective optimization of space truss using GA. The displacement, tensile stress, fuzzy sets, membership functions and minimum size constraints are considered in formulation of the design problem. Lin [48] proposed a multiobjective and multistage fuzzy competence set model using a hybrid genetic algorithm. Author illustrated that the proposed method can provide a sound fuzzy competence set model by considering the multiobjective and the multistage situations simultaneously. Ishibuchi and Nojimaa [38] examined the interpretability-accuracy tradeoff in fuzzy rule-based classifiers using a multiobjective fuzzy genetics-based machine learning (GBML) algorithm which is a hybrid version of Michigan and Pittsburgh approaches. Each fuzzy rule is represented by its antecedent fuzzy sets as an integer string of fixed length. Each fuzzy rule-based classifier, which is a set of fuzzy rules, is represented as a concatenated integer string of variable length. The GBML algorithm simultaneously maximizes the accuracy of rule sets and minimizes their complexity. The accuracy is measured by the number of correctly classified training patterns while the complexity is measured by the number of fuzzy rules and/or the total number of antecedent conditions of fuzzy rules. Garcia-Pedrajas et al. [25] developed a cooperative coevolutive model for the evolution of neural network topology and weights, called MOBNET. MOBNET evolves subcomponents that must be combined in order to form a network, instead of whole networks. The subcomponents in a cooperative coevolutive model must fulfill different criteria to be useful, these criteria usually conflict with each other. The problem of evaluating the fitness on an individual based on many criteria that must be optimized together is approached as a multi-criteria optimization problems. Wang et al. [74] proposed a multiobjective hierarchical genetic algorithm (MOHGA) to extract interpretable rule-based knowledge from data. In order to remove the redundancy of the rule base proactively, authors applied an interpretability-driven simplification method. Fuzzy clustering is used to generate an initial rule-based model and then MOHGA and the recursive least square method are used to obtain the optimized fuzzy models. Pettersson et al. [63] used an evolutionary multiobjective technique in the training process of a feed forward neural network, using noisy data from an industrial iron blast furnace. The number of nodes in the hidden layer, the architecture of the lower part of the network, as well as the weights used in
18
A. Abraham and C. Grosan
them were kept as variables, and a Pareto front was effectively constructed by minimizing the training error along with the network size.
8 Conclusions This Chapter presented the various architectures for designing intelligent paradigms using evolutionary algorithms. The main focus was on designing evolutionary neural networks and evolutionary fuzzy systems. We also illustrated some of the recent generic evolutionary design architectures reported in the literature including fuzzy neural networks and multiobjective design strategies.
References 1. Abraham A, Grosan C, Han SY, Gelbukh A (2005) Evolutionary multiobjective optimization approach for evolving ensemble of intelligent paradigms for stock market modeling. In: Alexander Gelbukh et al. (eds.) 4th Mexican international conference on artificial intelligence, Mexico, Lecture notes in computer science, Springer, Berlin Heidelberg New York, pp 673–681 2. Abraham, A (2004) Meta-learning evolutionary artificial neural networks. Neurocomput J 56c:1–38 3. Abraham A (2003) i-Miner: A Web Usage Mining Framework Using Hierarchical Intelligent Systems, The IEEE International Conference on Fuzzy Systems, FUZZ-IEEE’03, IEEE Press, ISBN 0780378113, pp 1129–1134 4. Abraham A, Ramos V (2003), Web Usage Mining Using Artificial Ant Colony Clustering and Genetic Programming, 2003 IEEE Congress on Evolutionary Computation (CEC2003), Australia, IEEE Press, ISBN 0780378040, pp 1384– 1391, 2003 5. Abraham A (2003), EvoNF: A Framework for Optimization of Fuzzy Inference Systems Using Neural Network Learning and Evolutionary Computation, The 17th IEEE International Symposium on Intelligent Control, ISIC’02, IEEE Press, ISBN 0780376218, pp 327–332 6. Aouiti C, Alimi AM, Karray F, Maalej A (2005) The design of beta basis function neural network and beta fuzzy systems by a hierarchical genetic algorithm. Fuzzy Sets Syst 154(2):251–274 7. Bhattacharya A, Abraham A, Vasant P, Grosan C (2007) Meta-learning evolutionary artificial neural network for selecting flexible manufacturing systems under disparate level-of-satisfaction of decision maker. Int J Innovative Comput Inf Control 3(1):131–140 8. Baxter J (1992) The evolution of learning algorithms for artificial neural networks, Complex systems, IOS, Amsterdam, pp 313–326 9. Bezdek, JC (1981) Pattern recognition with fuzzy objective function algorithms. Plenum, New York 10. Boers EJW, Borst MV, Sprinkhuizen-Kuyper IG (1995) Artificial neural nets and genetic algorithms. In: Pearson DW et al. (eds.) Proceedings of the international conference in Ales, France, Springer, Berlin Heidelberg New York, pp 333–336
Engineering Evolutionary Intelligent Systems
19
11. Cai X, Zhang N, Venayagamoorthy GK, Wunsch II DC (2007) Time series prediction with recurrent neural networks trained by a hybrid PSOEA algorithm. Neurocomputing 70(13–15):2342–2353 12. Castillo PA, Merelo JJ, Arenas MG, Romero G (2007) Comparing evolutionary hybrid systems for design and optimization of multilayer perceptron structure along training parameters. Inf Sci 177(14):2884–2905 13. Capi G, Doya K (2005), Evolution of recurrent neural controllers using an extended parallel genetic algorithm. Rob Auton Syst 52(2–3):148–159 14. Chen Y, Yang B, Dong J, Abraham A (2005) Time-series forecasting using flexible neural tree model. Inf Sci 174(3–4):219–235 15. Chen Y, Yang B, Abraham A, Peng L (2007) Automatic design of hierarchical takagi-sugeno fuzzy systems using evolutionary algorithms. IEEE Trans Fuzzy Syst 15(3):385–397 16. Chu B, Kim D, Hong D, Park J, Chung JT, Chung JH, Kim TH (2008) GA-based fuzzy controller design for tunnel ventilation systems, Journal of Automation in Construction, 17(2):130–136 17. Delgado M, Pegalajar MC (2005) A multiobjective genetic algorithm for obtaining the optimal size of a recurrent neural network for grammatical inference. Pattern Recognit 38(9):1444–1456 18. Eberhart RC, Kennedy J (1995) A new optimizer using particle swarm theory. In: Proceedings of 6th Internationl Symposium on Micro Machine and Human Science, Nagoya, Japan, IEEE Service Center, Piscataaway, NJ, pp 39–43 19. Edwards R, Abraham A, Petrovic-Lazarevic S (2005) Computational intelligence to model the export behaviour of multinational corporation subsidiaries in Malaysia. Int J Am Soc Inf Sci Technol (JASIST) 56(11):1177–1186 20. Fogel LJ, Owens AJ, Walsh MJ (1966) Artificial intelligence through simulated evolution. Wiley, USA 21. Fontanari JF, Meir R (1991) Evolving a learning algorithm for the binary perceptron, Network, vol. 2, pp 353–359 22. Franke C, Hoffmann F, Lepping J, Schwiegelshohn U (2008) Development of scheduling strategies with Genetic Fuzzy systems, Applied Soft Computing Journal, 8(1):706–721 23. Frean M (1990), The upstart algorithm: a method for constructing and training feed forward neural networks. Neural Comput 2:198–209 24. Fullmer B, Miikkulainen R (1992) Using marker-based genetic encoding of neural networks to evolve finite-state behaviour. In: Varela FJ, Bourgine P (eds.) Proceedings of the first European conference on artificial life, France, pp 255–262 25. Garca-Pedrajas N, Hervs-Martnez C, Muoz-Prez J (2002) Multi-objective cooperative coevolution of artificial neural networks (multi-objective cooperative networks). Neural Netw 15(10):1259–1278 26. Grau F (1992)Genetic synthesis of boolean neural networks with a cell rewriting developmental process. In: Whitely D, Schaffer JD (eds.) Proceedings of the international workshop on combinations of genetic algorithms and neural Networks, IEEE Computer Society Press, CA, pp 55–74 27. Gonzalez A, Herrera F (1997) Multi-stage genetic fuzzy systems based on the iterative rule learning approach. Mathware Soft Comput 4(3) 28. Grosan C, Abraham A, Nicoara M (2005) Search optimization using hybrid particle sub-swarms and evolutionary algorithms. Int J Simul Syst, Sci Technol UK 6(10–11):60–79
20
A. Abraham and C. Grosan
29. Gutierrez G, Isasi P, Molina JM, Sanchis A, Galvan IM (2001) Evolutionary cellular configurations for designing feedforward neural network architectures, connectionist models of neurons. In: Jose Mira et al. (eds.) Learning processes, and artificial intelligence, Springer, Berlin Heidelberg New York, LNCS 2084, pp 514–521 30. Harp SA, Samad T, Guha A (1989) Towards the genetic synthesis of neural networks. In: Schaffer JD (ed.) Proceedings of the third international conference on genetic algorithms and their applications, Morgan Kaufmann, CA, pp 360– 369 31. Herrera F, Lozano M, Verdegay JL (1995) Tuning fuzzy logic controllers by genetic algorithms. Int J Approximate Reasoning 12:299–315 32. Herrera F, Lozano M, Verdegay JL (1995) Tackling fuzzy genetic algorithms. In: Winter G, Periaux J, Galan M, Cuesta P (eds.) Genetic algorithms in engineering and computer science, Wiley, USA, pp 167–189 33. Hoffmann F (1999) The Role of Fuzzy Logic in Evolutionary Robotics. In: Saffiotti A, Driankov D (ed.) Fuzzy logic techniques for autonomous vehicle navigation, Springer, Berlin Heidelberg New York 34. Holland JH (1975) Adaptation in Natural and Artificial Systems, The University of Michigan Press, Ann Arbor, MI 35. Holland JH, Reitman JS (1978), Cognitive systems based on adaptive algorithms. In: Waterman DA, Hayes-Roth F (eds.) Pattern-directed inference systems. Academic, San Diego, CA 36. Holland, JH (1980) Adaptive algorithms for discovering and using general patterns in growing knowledge bases. Int J Policy Anal Inf Sys 4(3):245–268 37. Hui LY (2007) Evolutionary neural network modeling for forecasting the field failure data of repairable systems. Expert Syst Appl 33(4):1090–1096 38. Ishibuchi H, Nojima Y (2007) Analysis of interpretability-accuracy tradeoff of fuzzy systems by multiobjective fuzzy genetics-based machine learning. Int J Approximate Reason 44(1):4–31 39. Juang CF, Chung IF (2007) Recurrent fuzzy network design using hybrid evolutionary learning algorithms, Neurocomputing 70(16–18):3001–3010 40. Jin Y, von Seelen W (1999) Evaluating flexible fuzzy controllers via evolution strategies. Fuzzy Sets Syst 108(3):243–252 41. Kennedy J, Eberhart RC (1995). Particle swarm optimization. In: Proceedings of IEEE International Conference on Neural Networks, Perth, Australia, pp 1942– 1948 42. Kim KJ, Cho SB (2006) Evolved neural networks based on cellular automata for sensory-motor controller. Neurocomputing 69(16–18):2193–2207 43. Kim KJ (2006) Artificial neural networks with evolutionary instance selection for financial forecasting. Expert Syst Appl 30(3):519–526 44. Kitano H (1990) Designing neural networks using genetic algorithms with graph generation system. Complex Syst 4(4):461–476 45. Koza JR (1992) Genetic programming: on the programming of computers by means of natural selection, MIT, Cambridge, MA 46. Kosinski W (2007) Evolutionary algorithm determining defuzzyfication operators. Eng Appl Artif Intell 20(5):619–627 47. Kelesoglu O (2007) Fuzzy multiobjective optimization of truss-structures using genetic algorithm. Adv Eng Softw 38(10):717–721
Engineering Evolutionary Intelligent Systems
21
48. Lin CM (2006) Multiobjective fuzzy competence set expansion problem by multistage decision-based hybrid genetic algorithms. Appl Math Comput 181(2):1402–1416 49. Liu Y, Yao X (1996) Evolutionary design of artificial neural networks with different node transfer functions. In: Proceedings of the Third IEEE International Conference on Evolutionary Computation, Nagoya, Japan, pp 670–675 50. Mamdani EH, Assilian S (1975) An experiment in linguistic synthesis with a fuzzy logic controller. Int J Man Mach Stud 7(1):1–13 51. Marwala T (2007) Bayesian training of neural networks using genetic programming. Pattern Recognit Lett 28(12):1452–1458 52. Mascioli F, Martinelli G (1995) A constructive algorithm for binary neural networks: the oil spot algorithm. IEEE Trans Neural Netw 6(3):794–797 53. Merril JWL, Port RF (1991) Fractally configured neural networks. Neural Netw 4(1):53–60 54. Moriarty DE, Miikkulainen R (1997) Forming neural networks through efficient and adaptive coevolution. Evol Comput 5:373–399 55. Mohammadian M, Stonier RJ (1994) Generating fuzzy rules by genetic algorithms. In: Proceedings of 3rd IEEE International Workshop on Robot and Human Communication, Nagoya, pp 362–367 56. Muhlenbein H, Paab G (1996) From recombination of genes to the estimation of distributions I. Binary parameters. In: Lecture notes in computer science 1411: parallel problem solving from nature-PPSN IV, pp 178–187 57. Oh SK, Pedrycz W, Roh SB (2006), Genetically optimized fuzzy polynomial neural networks with fuzzy set-based polynomial neurons. Inf Sci 176(23):3490– 3519 58. Omlin CW, Giles CL (1993) Pruning recurrent neural networks for improved generalization performance. Techincal report No 93-6, CS Department, Rensselaer Institute, Troy, NY 59. Cordon O, Herrera F, Hoffmann F, Magdalena L (2001) Genetic fuzzy systems: evolutionary tuning and learning of fuzzy knowledge bases, World Scientific, Singapore, ISBN 981-02-4016-3, p 462 60. Park HS, Pedrycz W, Oh SK (2007) Evolutionary design of hybrid selforganizing fuzzy polynomial neural networks with the aid of information granulation. Expert Syst Appl 33(4):830–846 61. Pawar PM, Ganguli R (2007) Genetic fuzzy system for online structural health monitoring of composite helicopter rotor blades. Mech Syst Signal Process 21(5):2212–2236 62. Pedrycz W (ed.) (1997), Fuzzy evolutionary computation, Kluwer Academic Publishers, Boston, ISBN 0-7923-9942-0, p 336 63. Pettersson F, Chakraborti N, Saxen H (2007) A genetic algorithms based multiobjective neural net applied to noisy blast furnace data. Appl Soft Comput 7(1):387–397 64. Rokach L, Maimon O (2005) Clustering methods, data mining and knowledge discovery handbook, Springer, Berlin Heidelberg New York, pp 321–352 65. Sexton R, Dorsey R, Johnson J (1999) Optimization of neural networks: a comparative analysis of the genetic algorithm and simulated annealing. Eur J Oper Res 114:589–601 66. Stepniewski SW, Keane AJ (1997) Pruning back-propagation neural networks using modern stochastic optimization techniques. Neural Comput Appl 5:76–98
22
A. Abraham and C. Grosan
67. Storn R, Price K (1997) Differential evolution – a simple and efficient adaptive scheme for global optimization over continuous spaces. J Global Optim 11(4):341–359 68. Rechenberg I, (1973) Evolutions strategie: optimierung technischer Systeme nach Prinzipien der biologischen Evolution, Fromman-Holzboog, Stuttgart 69. Schwefel HP (1977) Numerische Optimierung von Computermodellen mittels der Evolutionsstrategie, Birkhaeuser, Basel 70. Serra GLO, Bottura CP (2006) Multiobjective evolution based fuzzy PI controller design for nonlinear systems. Eng Appl Artif Intell 19(2):157–167 71. Smith SF (1980) A learning system based on genetic adaptive algorithms. PhD thesis, University of Pittsburgh 72. Takagi T, Sugeno M (1983) Derivation of fuzzy logic control rules from human operators control actions. In: Proceedings of the IFAC symposium on fuzzy information representation and decision analysis, pp 55–60 73. Tsang CH, Kwong S, Wang A (2007) Genetic-fuzzy rule mining approach and evaluation of feature selection techniques for anomaly intrusion detection. Pattern Recognit 40(9):2373–2391 74. Wang H, Kwong S, Jin Y, Wei W, Man K (2005) A multi-objective hierarchical genetic algorithm for interpretable rule-based knowledge extraction. Fuzzy Sets Syst 149(1):149–186 75. Wolpert DH, Macready WG (1997) No free lunch theorems for optimization. IEEE Trans Evol Comput 1(1):67–82 76. Xu R, Wunsch D, (2005) Survey of clustering algorithms. IEEE Trans Neural Netw 16(3):645–678 77. Yao X (1999) Evolving artificial neural networks. Proc IEEE 87(9):423–1447
Genetically Optimized Hybrid Fuzzy Neural Networks: Analysis and Design of Rule-based Multi-layer Perceptron Architectures Sung-Kwun Oh and Witold Pedrycz
Summary. In this study, we introduce an advanced architecture of genetically optimized Hybrid Fuzzy Neural Networks (gHFNN) and develop a comprehensive design methodology supporting their construction. A series of of numeric experiments is included to illustrate the performance of the networks. The construction of gHFNN exploits fundamental technologies of Computational Intelligence (CI), namely fuzzy sets, neural networks, and genetic algorithms (GAs). The architecture of the gHFNNs results from a synergistic usage of the genetic optimization-driven hybrid system generated by combining Fuzzy Neural Networks (FNN) with Polynomial Neural Networks (PNN). In this tandem, a FNN supports the formation of the premise part of the rule-based structure of the gHFNN. The consequence part of the gHFNN is designed using PNNs. The optimization of the FNN is realized with the aid of a standard back-propagation learning algorithm and genetic optimization. We distinguish between two types of the fuzzy rule-based FNN structures showing how this taxonomy depends upon the type of a fuzzy partition of input variables. As to the consequence part of the gHFNN, the development of the PNN dwells on two general optimization mechanisms: the structural optimization is realized via GAs whereas in case of the parametric optimization we proceed with a standard least square method-based learning. Through the consecutive process of such structural and parametric optimization, an optimized PNN is generated in a dynamic fashion. To evaluate the performance of the gHFNN, the models are experimented with several representative numerical examples. A comparative analysis demonstrates that the proposed gHFNN come with higher accuracy as well as superb predictive capabilities when comparing with other neurofuzzy models.
1 Introductory remarks Recently, a lot of attention has been devoted towards advanced techniques of modeling complex systems inherently associated with nonlinearity, highorder dynamics, time-varying behavior, and imprecise measurements. It is anticipated that efficient modeling techniques should allow for a selection of pertinent variables and in this way help cope with dimensionality of the problem at hand. The models should be able to take advantage of the existing S.-K. Oh and W. Pedrycz: Genetically Optimized Hybrid Fuzzy Neural Networks: Analysis and Design of Rule-based Multi-layer Perceptron Architectures, Studies in Computational Intelligence (SCI) 82, 23–57 (2008) c Springer-Verlag Berlin Heidelberg 2008 www.springerlink.com
24
S.-K. Oh and W. Pedrycz
domain knowledge (such as a prior experience of human observers or process operators) and augment it by available numeric data to form a coherent dataknowledge modeling entity. The omnipresent modeling tendency is the one that exploits techniques of Computational Intelligence (CI) by embracing fuzzy modeling [1–6], neurocomputing [7], and genetic optimization [8–10]. Especially the two of the most successful approaches have been the hybridization attempts made in the framework of CI [11,12]. Neuro-fuzzy systems are one of them [13–20]. A different approach to hybridization leads to genetic fuzzy systems. Lately to obtain a highly beneficial synergy effect, the neural fuzzy systems and the genetic fuzzy systems hybridize the approximate inference method of fuzzy systems with the learning capabilities of neural networks and evolutionary algorithms [21]. In this study, we develop a hybrid modeling architecture, called genetically optimized Hybrid Fuzzy Neural Networks (gHFNN). In a nutshell, a gHFNN is composed of two main substructures driven by genetic optimization, namely a rule-based Fuzzy Neural Network (FNN) and a Polynomial Neural Network (PNN). From a standpoint of rule-based architectures (with their rules assuming the general form “if antecedent then consequent”), one can regard the FNN as an implementation of the antecedent (or premise) part of the rules while the consequent part is realized with the aid of PNN. The resulting gHFNN is an optimized architecture designed by combining the conventional Hybrid Fuzzy Neural Networks (HFNN [19,20,34,35]) with genetic algorithms (GAs). The conventional HFNNs exhibits FNN architecture treated as the premise part while the PNN structures are used in common as the conclusion part of HFNNs. In this study, the FNNs come with two kinds of network architectures, namely fuzzy-set based FNN and fuzzy-relation based FNN. The topology of the network proposed here relies on fuzzy partitions realized in terms of fuzzy sets or fuzzy relations that its input variables are considered separately or simultaneously. Each of them is placed into the two main categories according to the type of fuzzy inference, namely the simplified and linear fuzzy inference. Moreover the PNN structure is optimized by GAs, that is, a genetically optimized PNN (gPNN) is designed and the gPNN is applied to the consequence part of gHFNN. The gPNN that exhibits a flexible and versatile structure is constructed on a basis of PNN [14,15] and GAs [8–10]. gPNN leads to the effective reduction of the depth of the networks as well as the width of the layer, and the avoidance of a substantial amount of time-consuming iterations for finding the most preferred networks in conventional PNN. In this network, the number of layers and number of nodes in each layer are not predetermined (unlike in case of most neural-networks) but can be generated in a dynamic fashion. The design procedure applied in the construction of each layer of the PNN deals with its structural optimization involving the selection of optimal nodes (or PNs) with specific local characteristics (such as the number of input variables, the order of the polynomial, and a collection of the specific subset of input variables) and addresses specific aspects of parametric optimization.
Genetically Optimized Hybrid Fuzzy Neural Networks
25
The study is organized in the following manner. First, Section 2 delivers a brief introduction to the architecture of the conventional HFNN. In Section 3, we discuss a structure of the genetically optimized HFNN (gHFNN) and elaborate on the development of the networks. The detailed genetic design of the gHFNN model comes with an overall description of a detailed design methodology of the gHFNN presented in Section 4. In Section 5, we report on a comprehensive set of experiments. Finally concluding remarks are covered in Section 6.
2 The architecture of conventional Hybrid Fuzzy Neural Networks (HFNN) The conventional HFNN architecture combined with the FNN and PNN is visualized in Figs. 1–3 [19,20,34,35]. Let us recall that the fuzzy inference (both simplified and linear) -based FNN is constructed with the aid of the space partitioning realized by not only fuzzy set defined for each input variable but also fuzzy relations that effectively capture an ensemble of input variables. These networks arise as a synergy between two other general constructs such as FNN and PNN. Based on the different PNN topologies (see Table 1), the HFNN embraces two kinds of architectures, namely a basic and modified one. Moreover for each architecture of the HFNN, we identified two cases; refer to Fig. 1 for the overall taxonomy. According to the alternative position of two connection points (interface) in case of th usage of FS FNN shown in Fig. 2, we realize a different combination of FNN and PNN while forming the HFNN architecture. Especially when dealing with the interface of FNN realized by means of PNNs, we note that if input variables to PNN used in the consequence part of HFNN are less than three (or four), the generic type of HFNN does not generate a highly versatile structure. As visualized in Figs. 1–3, we identify also two types of the topology, namely a generic and advanced type. Observe that in Figs. 2–3, zi ’(Case 2) in the 2nd layer or higher indicates that the polynomial order of Premise part(FNN)
Linear fuzzy inference
Interface
Simplified fuzzy inference
Consequence part(PNN)
Generic type
Advanced type
Basic
case 1 case 2
Generic type Basic HFNN
Modified
case 1 case 2
Generic type Modified HFNN
Basic
case 1 case 2
Advanced type Basic HFNN
Modified
case 1 case 2
Advanced type Modified HFNN
Fig. 1. Overall diagram for generating the conventional HFNN architecture
26
S.-K. Oh and W. Pedrycz
Fig. 2. FS HFNN architecture combined with FS FNN and PNN
Fig. 3. FR HFNN architecture combined with FR FNN and PNN
Genetically Optimized Hybrid Fuzzy Neural Networks
27
Table 1. Taxonomy of various PNN architectures Layer of PNN
No. of input variables of polynomial
Order of Polynomial
PNN architecture
lst layer
p
Type P
(1) p = q : Basic PNN a) Type P = Type Q: Case1 b) Type P=Type Q: Case2
2nd or higher layer
q
Type Q
(2) p=q : Modified PNN a) Type P = Type Q: Case1 b) Type P = Type Q: Case2
(p = 2,3,4,5, q = 2,3,4,5 ; P = 1,2,3, Q = 1,2,3)
the PD of each node has a different type in comparison with zi of the lst layer. The “NOP” node states that the Ath node of the current layer is the same as the node positioned in the previous layer (NOP stands for “no operation”). An arrow to the NOP node is used to show that the same node moves from the previous layer to the current one.
3 The architecture and development of genetically optimized HFNN (gHFNN) In this section, we elaborate on the architecture and a development process of the gHFNN. This network emerges from the genetically optimized multi-layer perceptron architecture based on fuzzy set or fuzzy relation-based FNN, PNN and GAs. In the sequel, gHFNN is designed by combining the conventional Hybrid Fuzzy Neural Networks (HFNN) with GAs. These networks result as a synergy between two other general constructs such as FNN [24,32] and PNN [14,15]. First, we briefly discuss these two classes of models by underlining their profound features in sections 3.1 and 3.2, respectively. 3.1 Fuzzy neural networks based on genetic optimization We consider two kinds of FNNs (viz. FS FNN and FR FNN) based on two types of fuzzy inferences, namely, simplified and linear fuzzy inferences. The structure of the FNN is the same as the used in the premise of the conventional HFNN. The FNN is designed by using space partitioning realized in terms of the individual input variables or an ensemble of all variables. Its each topology is concerned with a granulation carried out in terms of fuzzy sets defined in each input variable or fuzzy relations that capture an ensemble of input variables respectively. The fuzzy partitions formed for each case lead us to the topologies visualized in Figs. 4–5.
28
S.-K. Oh and W. Pedrycz
Fig. 4. Topology of FS FNN by using space partitioning in terms of individual input variables
Fig. 5. Topology of FR FNN by using space partitioning in terms of an ensemble of input variables
Genetically Optimized Hybrid Fuzzy Neural Networks
29
The notation in these figures requires some clarification. The “circles” denote units of the FNN while “N” identifies a normalization procedure applied tothe membership grades of the input variable xi . The output fi (xi ) of the “ ” neuron is described by some nonlinear function fi . Not necessarily fi is a sigmoid function encountered in conventional neural networks but we allow for more flexibility in this regard. Finally, in case of FS FNN, the output of the FNN y is governed by the following expression; y = f1 (x1 ) + f2 (x2 ) + · · · + fm (xm ) =
m
fi (xi )
(1)
i=1
with m being the number of the input variables (viz. the number of the outputs fi ’s of the “ ” neurons in the network). As previously mentioned, FS FNN is affected by the introduced fuzzy partition of each input variable. In this sense, we can regard each fi given by fuzzy rules as shown in Table 2(a). Table 2(a) represents the comparison of fuzzy rules, inference result and learning for two types of FNNs. In Table 2(a), Rj is the j-th fuzzy rule while Aij denotes a fuzzy variable of the premise of the corresponding fuzzy rule and represents membership function µij . In the simplified fuzzy inference, ωij is a constant consequence of the rules and, in the linear fuzzy inference, ωsij is a constant consequence and ωij is an input variable consequence of the rules. They express a connection (weight) existing between the neurons as we have already visualized in Fig. 4. Mapping from xi to fi (xi ) is determined by the fuzzy inferences and a standard defuzzification. The inference result for individual fuzzy rules follows a standard center of gravity aggregation. An input signal xi activates only two membership functions, so inference results can be written as outlined in Table 2(a) [23,24]. The learning of FNN is realized by adjusting connections of the neurons and as such it follows a standard Back-Propagation (BP) algorithm [23,24]. The complete update formulas are covered in Table 2(a). Where η is a positive learning rate and α is a positive momentum coefficient. The case of FR FNN, see Table 2(b), is carried out in a same manner as outlined in Table 2(a) (the case of FS FNN). The task of optimizing any model involves two main phases. First, a class of some optimization algorithms has to be chosen so that it meets the requirements implied by the problem at hand. Secondly, various parameters of the optimization algorithm need to be tuned in order to achieve its best performance. Along this line, genetic algorithms (GAs) viewed as optimization techniques based on the principles of natural evolution are worth considering. GAs have been experimentally demonstrated to provide robust search capabilities in problems involving complex spaces thus offering a valid solution to problems requiring efficient searching. It is instructive to highlight the main features that tell GA apart from some other optimization methods: (1) GA operates on the codes of the variables, but not the variables themselves. (2) GA searches optimal points starting from a group (population) of points in the search space (potential solutions), rather than a single point. (3) GA’s
30
S.-K. Oh and W. Pedrycz Table 2. Comparison of simplified with linear fuzzy inference-based FNNs
(a) In case of using Simplified fuzzy inference (Scheme I) R1 : If xi is Ai1 then Cyi1 = ωi1 .. . Fuzzy rules Rj : If xi is Aij then Cyij = ωij .. . Rz : If xi is Aiz then Cyiz = ωiz Structure
z
fi (xi ) = Inference result
µij (xi ) · ωij
z
fi (xi ) =
j=1
z
FS FNN Linear fuzzy inference (Scheme II) R1 : · · · then Cyi1 = ωsi1 + xi ωi1 .. . Rj : · · · then Cyij = ωsij + xi ωij .. . Rz : · · · then Cyiz = ωsiz + xi ωiz
j=1
+ µik+1 (xi )
+ µik (xi ) · ωik+1 fi (xi )
Premise part Fuzzy rules Consequence part
n
fi (xi )
i=1
∆ωij = 2 · η · (yp − yp ) · µij (xi )
Structure
· (ωsik+1 + xi wik+1 ) y =
i=1
Learning
µij (xi )
= µik (xi ) · (ωsik + xi ωik )
= µik · (xi )ωik
y =
(µij (xi ) · (ωsij + xi ωij )) z
µij (xi )
j=1
n
j=1
+ α(ωij (t) − ωij (t − 1))
⎧ ⎪ ⎪ ∆ωsij = 2 · η · (y − y) · µij ⎪ ⎪ ⎨ + α(ωsij (t) − ωsij (t − 1)) ⎪ ∆ω = 2 · η · (y − y) · µij · xi ⎪ ⎪ ⎪ ⎩ + α(ωij (t) − ωij (t − 1))
(b) In case of using FR FNN fuzzy inference Linear R1 : If x1 is A11 , · · · , and xk is Ak1 .. . Ri : If x1 is A1i , · · · , and xk is Aki .. . Rn : If x1 is A1n , · · · , and xk is Akn then Cy1 = ω1 .. . then Cyi = ωi .. . then Cyn = ωn
then Cy1 = ω01 + ω11 · x1 + · · · + ωk1 · xk .. . then Cyi = ω0i + ω1i · x1 + · · · + ωki · xk .. . then Cyn = ω0n + ω1n · x1 + · · · + ωkn · xk (continued)
Genetically Optimized Hybrid Fuzzy Neural Networks
31
Table 2. (Continued)
y =
Inference result
= =
n i=1 n i=1 n i=1
fi µ¯i · ωi µi · ωi n µi
y = = =
n i=1 n i=1 n i=1
i=1
Learning
fi µ¯i · (ω0i + ω1i · x1 + ωki · xk ) µi · (ω0i + ω1i · x1 + ωki · xk ) n µi i=1
⎧ ∆ω0i = 2 · η · (y − y) · µ¯i + α(ω0i (t) ⎪ ∆ωi = 2 · η · (y − y) · µ¯i ⎪ ⎪ ⎨ − ω0i (t − 1)) + α(ωi (t) ⎪ = 2 · η · (y − y) · µ¯i · xk ∆ω ki ⎪ ⎪ ⎩ − ωi (t − 1)) + α(ωki (t) − ωki (t − 1))
search is directed only by some fitness function whose form could be quite complex; we do not require its differentiability [8–10]. In order to enhance the learning of the FNN, we use GAs to adjust learning rate, momentum coefficient and the parameters of the membership functions of the antecedents of the rules [19,20,34,35]. 3.2 Genetically optimized PNN (gPNN) As underlined, the PNN algorithm is based upon the GMDH [22] method and utilizes a class of polynomials such as linear, quadratic, modified quadratic, etc. to describe basic processing realized there. By choosing the most significant input variables and an order of the polynomial among various types of forms available, we can obtain the best one - it comes under a name of a partial description (PD). It is realized by selecting nodes at each layer and eventually generating additional layers until the best performance has been reached. Such a methodology leads to an optimal PNN structure [14,15]. In addressing the problems with the conventional PNN (see Fig. 6), we introduce a new genetic design approach; in turn we will be referring to these networks as genetically optimized PNN (to be called “gPNN”). When we construct PNs of each layer in the conventional PNN, such parameters as the number of input variables (nodes), the order of polynomial, and input variables available within a PN are fixed (selected) in advance by the designer. This could have frequently contributed to the difficulties in the design of the optimal network. To overcome this apparent drawback, we resort ourselves to the genetic optimization, see Figs. 8–9 of the next section for more detailed flow of the development activities. The overall genetically-driven structural optimization process of PNN is shown in Fig. 7. The determination of the optimal values of the parameters available within an individual PN (viz. the number of input variables,
32
S.-K. Oh and W. Pedrycz
Polynomial Neural Networks
PN
z1 •
PN
PN
PN PN
z2 • z3 •
PN
z4 •
PN
PN
PN
••
• •
PN
^ f
PN PN
PN PN
Polynomial Neuron(PN) zp zq
Input variables
zp, zq
Polynomial order
2
Partial Description(PD) : Type 2
z
c0+ c1zp+ c2zq+ c3z2p + c4z2q + c5zpzq
Fig. 6. A general topology of the PN-based PNN: note a biquadratic polynomial occurring in the partial description 2nd stage
1st stage 1st layer
Genetic design Selection of the no. of input variables
E
Selection of input variables
S
Layer Generation
2nd layer
Genetic design z1
Selection of the no. of input variables
Selection of input variables
Selection of the polynomial order
Selection of the polynomial order
PNs Selection
PNs Selection
S
Layer Generation
z2
E : Entire inputs, S : Selected PNs, zi : Preferred outputs in the ith stage (zi = z1i, z2i, ..., zWi)
Fig. 7. Overall genetically-driven structural optimization process of PNN
the order of the polynomial, and input variables) leads to a structurally and parametrically optimized network. As a result, this network is more flexible as well as it exhibits simpler topology in comparison to the conventional PNN discussed in the previous research [14,15]. For the optimization of the PNN model, GAs uses the serial method of binary type, roulette-wheel used in the selection process, one-point crossover in the crossover operation, and a binary inversion (complementation) operation in the mutation operator. To retain the best individual and carry it over to the next generation, we use elitist strategy [8,9].
Premise part(FNN)
Consequence part(gPNN) 2nd layer
1st layer
PN
Interface
Simplified & Linear fuzzy inference
GAs S
PN PN PN
Layer generation
GAs
Membership parameters
S PN x2= z1
PN PN PN
Layer generation
GAs
x3= z2
33
Genetically optimized Hybrid Fuzzy Neural Networks (gHFNN)
Genetically Optimized Hybrid Fuzzy Neural Networks
S : Selected PNs, zi : Outputs of the ith layer, xj : Input variables of the jth layer ( j = i + 1)
Fig. 8. Overall diagram for generation of gHFNN architecture
3.3 Optimization of gHFNN topologies The topology of gHFNN is constructed by combining fuzzy set or fuzzy relation-based FNN for the premise part of the gHFNN with PNN being used as the consequence part of gHFNN. These networks emerge through a synergy between two other general constructs such as FNNs and gPNNs. In what follows, the gHFNN is composed of two main substructures driven by genetic optimization; see Figs. 8–9. The role of FNNs arising at the premise part is to support learning and interact with input as well as granulate the corresponding input space (viz. converting the numeric data into their granular representatives emerging at the level of fuzzy sets). Especially, two types of fuzzy inferences-based FNN (viz. FS-FNN or FR-FNN) realized with the fuzzy partitioning of individual input variables or an ensemble of input variables are considered to enhance the adaptability of the hybrid network architecture. One should stress that the structure of the consequent gPNN is not fixed in advance but becomes dynamically organized during a growth process. In essence, the gPNN exhibits an ability of self-organization. The gPNN algorithm can produce an optimal nonlinear system by selecting significant input variables among dozens of those available at the input and forming various types of polynomials. Therefore, for the very reason we selected FNN and gPNN in order to design the gHFNN architecture. One may consider some other hybrid network architectures such as a combination of FNNs and MLPs as well as ANFIS-like models combined with MLPs. While attractive on a surface, such hybridization may lead to several potential problems: 1) The repeated learning and optimization of each of the contributing structure (such as ANFIS and MLP) may result in excessive learning time as well as generate quite complex networks for relatively simple systems and 2) owing to its fixed structure, it could be difficult to generate the flexible topologies of the networks that are required to deal with highly nonlinear dependencies.
34
S.-K. Oh and W. Pedrycz START Premise part of gHFNN : FNN
Agjustment of parameters of MF using GAs & connection weight using BP
Computing activation degrees of linguistic labels Normalization of an activation degree of the rule Multiplying the normalized activation degrees of rules by connection weight Connection Point ? 2
1
Fuzzy inference for output of the rules Output of FS_FNN/FR_FNN
Configuration of input variables for consequence GAs & initial information and gPNN nsequence part
Consequence part of gHFNN : gPNN
Initialization of population
GAs
Generation of a PN by a chromosome in population Reproduction Roulette-wheel selection One-point crossover Invert mutation
Evaluation of PNs(Fitness) x1 = z1, x2 = z2, ..., xW = zW
Elitist strategy & Selection of PNs(W) No
The outputs of the preserved PNs serve as new inputs to the next layer
Stop condition Yes
Generate a layer of gPNN A layer consists of optimal PNs selected by GAs Stop condition
No
Yes
gHFNN gHFNN is organized by FS_FNN and layers with optimal PNs END
Fig. 9. Overall design flowchart of the gHFNN architecture
4 The algorithms and design procedure of genetically optimized HFNN (gHFNN) In this section, we elaborate on the algorithmic details of the design method by considering the functionality of the individual layers in the network architectures. The design procedure for each layer in the premise and consequence of gHFNN comprises of the following steps:
Genetically Optimized Hybrid Fuzzy Neural Networks
35
4.1 The premise of gHFNN: in case of FS FNN [Layer 1] Input layer: The role of this layer is to distribute the signals to the nodes in the next layer. [Layer 2] Computing activation degrees of linguistic labels: Each node in this layer corresponds to one linguistic label (small, large, etc.) of the input variables in layer 1. The layer determines a degree of satisfaction (activation) of this label by the input. [Layer 3] Normalization of a degree activation (firing) of the rule: As described, a degree of activation of each rule was calculated in layer 2. In this layer, we normalize the activation level by using the following expression. µij µik = = µik µ ¯ij = n µik + µik+1 µij
(2)
j=1
where n is the number of membership function for each input variable. An input signal xi activates only two membership functions simultaneously and the sum of grades of these two neighboring membership functions labeled by k and k + 1 is always equal to 1, that is µik (xi ) + µik+1 (xi ) = 1, so that this leads to a simpler format as shown in (2) [23,24]. [Layer 4] Multiplying a normalized activation degree of the rule by connection (weight): The calculated activation degree at the third layer is now calibrated through the connections, that is ¯ij × Cyij = µij × Cyij aij = µ
Simplif ied : Cyij = ωij Linear : Cyij = ωsij + ωij · xi
(3) (4)
If we choose CP (connection point) 1 for combining FS FNN with PNN as shown in Fig. 10, aij is given as the input variable of the PNN. If we choose CP 2, fi (xi ) corresponds to the input signal to the output layer of FNN viewed as the input variable of the PNN. [Layer 5] Fuzzy inference for output of the rules: Considering Fig. 4, the output of each node in the 5th layer of the premise part of gHFNN is inferred µ1j x1
N µij xi
aij
N
CP 2 ∑
w1j
∑
y^
N N
∑
wij
fi(xi)
CP 1
Fig. 10. Connection points used for combining FS FNN (Simplified) with gPNN
36
S.-K. Oh and W. Pedrycz
by the center of gravity method [23,24]. If we choose CP 2, fi is the input variable of gPNN that is the consequence part of gHFNN z j=1
Simplif ied : fi (xi ) = z
z
aij =
µij (xi )
j=1
µij (xi ) · ωij
j=1 z
µij (xi )
(5)
j=1
= µik (xi ) · ωik + µik+1 (xi ) · ωik+1 z j=1
Linear : fi (xi ) = z
j=1
z
aij
µij (xi )
=
µij (xi ) · (ωsij + xi ωij )
j=1 z
µij (xi )
(6)
j=1
= µik (xi ) · ωik + µik+1 (xi ) · ωik+1 [Output layer of FNN] Computing output of basic FNN: The output becomes a sum of the individual contributions from the previous layer, see (1) The design procedure for each layer in FR FNN is carried out in a same manner as the one presented for FS FNN. 4.2 The consequence of gHFNN: in case of gPNN combined with FS FNN [Step 1] Configuration of input variables: Define input variables: xi ’s (i = 1, 2, · · · , n) to gPNN of the consequent structure of gHFNN. If we choose the first option to combine the structures of FNN and gPNN (CP 1), aij , which is the output of layer 4 in the premise structure of the gHFNN, is treated as the input of the consequence structure of gHFNN, that is, x1 = a11 , x2 = a12 , · · · xn = aij (n = i × j). For the second option of combining the structures (viz. CP 2), we have x1 = f1 , x2 = f2 , · · · , xn = fm (n = m). [Step 2] Decision of initial information for constructing the gPNN structure: We decide upon the design parameters of the PNN structure and they include that a) Stopping criterion, b) Maximum number of input variables coming to each node in the corresponding layer, c) Total number W of nodes to be retained (selected) at the next generation of the gPNN, d) Depth of the gPNN to be selected to reduce a conflict between overfitting and generalization abilities of the developed gPNN, and e) Depth and width of the gPNN to be selected as a result of a tradeoff between accuracy and complexity of the overall model. It is worth stressing that the decisions made with respect to (b)–(e) help us avoid building excessively large networks (which could be quite limited in terms of their predictive abilities). [Step 3] Initialization of population: We create a population of chromosomes for a PN, where each chromosome is a binary vector of bits. All bits for each chromosome are initialized randomly.
Genetically Optimized Hybrid Fuzzy Neural Networks
37
[Step 4] Decision of PN structure using genetic design: This concerns the selection of the number of input variables, the polynomial order, and the input variables to be assigned in each node of the corresponding layer. These important decisions are carried out through an extensive genetic optimization. When it comes to the organization of the chromosome representing a PN, we divide the chromosome into three sub-chromosomes as shown in Fig. 12. The 1st sub-chromosome contains the number of input variables, the 2nd sub-chromosome involves the order of the polynomial of the node, and the 3rd sub-chromosome (remaining bits) contains input variables coming to the corresponding node (PN). In nodes (PNs) of each layer of gPNN, we adhere to the notation of Fig. 11. ‘PNn ’ denotes the nth PN (node) of the corresponding layer, ‘N’ denotes the number of nodes (inputs or PNs) coming to the corresponding node, and ‘T’ denotes the polynomial order in the corresponding node. Each sub-step of the genetic design of the three types of the parameters available within the PN is structured as follows. [Step 4-1] Selection of the number of input variables (1st sub-chromosome) Sub-step 1) The first 3 bits of the given chromosome are assigned to the binary bits for the selection of the number of input variables. Sub-step 2) The selected 3 bits are decoded into a decimal. Sub-step 3) The above decimal value is converted into [1 N] and rounded off. N denotes the maximal number of input variables entering the corresponding node (PN). Sub-step 4) The normalized integer value is then treated as the number of input variables (or input nodes) coming to the corresponding node. [Step 4-2] Selection of the order of polynomial (2nd sub-chromosome) Sub-step 1) The 3 bits of the 2nd sub-chromosome are assigned to the binary bits for the selection of the order of polynomial. Sub-step 2) The 3 bits are decoded into a decimal format. Sub-step 3) The decimal value obtained is normalized into [1 3] and rounded off. Sub-step 4) The normalized integer value is given as the polynomial order. a) The normalized integer value is given as 1 ⇒ the order of polynomial is Type 1 nth Polynomial Neuron(PN) xi
PNn N T
xj
z
Polynomial order(Type T) No. of inputs
Fig. 11. Overall diagram for generation of gHFNN architecture
38
S.-K. Oh and W. Pedrycz Selection of node (PN) structrue by chromosome
Related bit items
Bit structure of subchromosome divided for each item
i) Bits for the selection of the no. of input variables
1
0
1
Decoding (Decimal)
Genetic Design
1
1
Decoding (Decimal)
Normalization (less than Max)
Selection of no. of input variables(r)
0
ii) Bits for the selection of the polynomial order
Normalization (1 ~ 3)
Selection of the order of polynomial
1
1
1
iii) Bits for the selection of input variables
0
1
1
0
1
1
1
1
1
1
r
Decoding (Decimal)
Decoding (Decimal)
Normalization (1 ~ n(or W))
Normalization (1 ~ n(or W))
Decision of input variables
Decision of input variables
Selection of input variables
(Type 1~Type 3)
PN
Selected PN
Fig. 12. Overall design flowchart of the gHFNN architecture Table 3. Different forms of regression polynomial forming a PN
PP Number PP PP of inputs P PP Order of PP the polynomial P
2
3
4
1 (Type 1)
Bilinear
Trilinear
Tetralinear
2 (Type 2)
Biquadratic-1
Triquadratic-1
Tetraquadratic-1
2 (Type 3)
Biquadratic-2
Triquadratic-2
Tetraquadratic-2
The following types of the polynomials are used; • Bilinear = c0 + c1 x1 + c2 x2 • Biquadratic-1 (Basic) = Bilinear +c3 x21 + c4 x22 + c5 x1 x2 , • Biquadratic-2 (Modified) = Bilinear +c3 x1 x2
b) The normalized integer value is given as 2 ⇒ the order of polynomial is Type 2 c) The normalized integer value is given as 3 ⇒ the order of polynomial is Type 3
Genetically Optimized Hybrid Fuzzy Neural Networks
39
[Step 4-3] Selection of input variables (3rd sub-chromosome) Sub-step 1) The remaining bits are assigned to the binary bits for the selection of input variables. binary bits for the selection of the number of input variables. Sub-step 2) The remaining bits are divided by the value obtained in step 4-1. Sub-step 3) Each bit structure is decoded into a decimal. Sub-step 4) The decimal value obtained is normalized into [1 n (or W)] and rounded off. n is the overall system’s inputs in the 1st layer, and W is the number of the selected nodes in the 2nd layer or higher. Sub-step 5) The normalized integer values are then taken as the selected input variables while constructing each node of the corresponding layer. Here, if the selected input variables are multiple-duplicated, the multiple-duplicated input variables are treated as a single input variable. [Step 5] Estimation of the coefficients of the polynomial assignedto the selected node and evaluation of a PN: The vector of coefficients is derived by minimizing the mean squared error between yi and y [14,15]. To evaluate the approximation and generalization capability of a PN produced by each chromosome, we use the following fitness function (the objective function is given in Section 5). fitness function =
1 1 + Objective function
(7)
[Step 6] Elitist strategy and Selection of nodes (PNs) with the best predictive capability: The nodes (PNs) obtained on the basis of the calculated fitness values (F1 ,F2 ,· · · ,Fz ) are rearranged in a descending order. We unify the nodes with duplicated fitness values (viz. in case that one node is the same fitness value as other nodes) among the rearranged nodes on the basis of the fitness values. We choose several PNs (W) characterized by the best fitness values. For the elitist strategy, we select the node that has the highest fitness value among the generated nodes. [Step 7] Reproduction: To generate new populations of the next generation, we carry out selection, crossover, and mutation operation using genetic information and the fitness values. Until the last generation, this step carries out by repeating steps 4–7. [Step 8] Construction of a corresponding layer of consequence part of gHFNN: Individuals evolved by GAs produce optimal PNs, W. The generated PNs construct their corresponding layer for the design of consequence part of gHFNN. [Step 9] Check the termination criterion: The termination condition builds a sound compromise between the high accuracy of the resulting model and its complexity as well as generalization abilities. [Step 10] Determine new input variables for the next layer: If the termination criterion has not been met, the
40
S.-K. Oh and W. Pedrycz
model is expanded. The outputs of the preserved nodes (z1 ,z2 , · · · , zW ) serves as new inputs to the next layer (x1 ,x2 , · · · , xW ). Repeating steps 3–10 carries out the gPNN.
5 Experimental studies In this section, the performance of the gHFNN is illustrated with the aid of some well-known and widely used datasets. In the first experiment, the network is used to model a three-input nonlinear function [1,13,19,25,28,29]. In the second simulation, an gHFNN is used to model a time series of gas furnace (Box-Jenkins data) [2–6,26,30–35]. Finally we use gHFNN for NOx emission process of gas turbine power plant [27,29,32]. The performance indexes (object function) used here are: (9) for the three-input nonlinear function and (8) for both gas furnace process and NOx emission process. i) Mean Squared Error (MSE) 1 (yp − yp )2 n p=1 n
E(P I or EP I) =
(8)
ii) Mean Magnitude of Relative Error (MMRE) 1 |yp − yp | × 100(%) n p=1 yp n
E(P I or EP I) =
(9)
Genetic algorithms use binary type, roulette-wheel as the selection operator, one-point crossover, and an invert operation in the mutation operator. The crossover rate of GAs is set to 0.75 and probability of mutation is equal to 0.065. The values of these parameters come from experiments and are very much in line with typical values encountered in genetic optimization. 5.1 Nonlinear function In this experiment, we use the same numerical data as in [1,13,19,25,28,29]. The nonlinear function to be determined is expressed as −1 −1.5 2 y = (1 + x0.5 ) 1 + x2 + x3
(10)
We consider 40 pairs of the original input-output data. The performance index (PI) is defined by (9). 20 out of 40 pairs of input-output data are used as learning set; the remaining part serves as a testing set. Table 4 summarizes the list of parameters related to the genetic optimization of the network. Design information for the optimization of gHFNN distinguishes between information of two networks such as the premise FNN and the consequent gPNN. First, a chromosome used in genetic optimization
Genetically Optimized Hybrid Fuzzy Neural Networks
41
Table 4. Parameters of the optimization environment and computational effort (a) In case of using FS FNN Generation 100 Population size 60 Gas Elite population size (W) 30 Premise structure (FNN) 10 (per one variable) String length Consequence structure (PNN) 3 + 3 + 24 No. of entire system inputs 3 Learning iteration 1000 Learning rate Simplified 0.039 Premise tuned Linear 0.335 (FS FNN) Momentum Simplified 0.004 gHFNN Coefficient tuned Linear 0.058 No. of rules 6 No. of entire CP 1 6 Consequence inputs CP 2 3 (gPNN) Maximal layer 5 No. of inputs to be selected (N) 1 ≤ N ≤ 4 (Max) Type(T) 1≤T≤3 N, T : integer (b) In case of using FR FNN Generation 150 Population size 100 Gas Elite population size (W) 50 Premise structure (FNN) 10 (per one variable) String length Consequence structure (PNN) 3 + 3 + 28 No. of entire system inputs 3 Learning iteration 1000 Learning rate Simplified 0.309 Premise tuned Linear 0.879 gHFNN (FS FNN) Momentum Simplified 0.056 Coefficient tuned Linear 0.022 No. of rules 8 No. of entire inputs 8 Consequence Maximal layer 5 (gPNN) No. of inputs to be selected (N) 1 ≤ N ≤ 4 (Max) Type (T) 1≤T≤3 N, T : integer
of the premise FNN contains the vertices of 2 membership functions of each system input (here, 3 system input variables have been used), learning rate, and momentum coefficient. The numbers of bits allocated to a chromosome are equal to 60, 10, and 10, respectively, that is 10 bits is assigned to each one variable. The parameters such as learning rate, momentum coefficient, and membership parameters are tuned with the help of genetic optimization
42
S.-K. Oh and W. Pedrycz
of the FNN as shown in Table 4. Next, in case of the consequent gPNN, a chromosome used in the genetic optimization consists of a string including 3 sub-chromosomes. The numbers of bits allocated to each sub-chromosome are equal to 3, 3, and 24/28, respectively. The population size being selected from the total population size, 60/100 is equal to 30/50. The process is realized as follows. 60/100 nodes (PNs) are generated in each layer of the network. The parameters of all nodes generated in each layer are estimated and the network is evaluated using both the training and testing data sets. Then we compare these values and choose 30/50 PNs that produce the best (lowest) value of the performance index. The number of inputs to be selected is confined to a maximum of four entries. The order of the polynomial is chosen from three types, that is Type 1, Type 2, and Type 3. Tables 5(a) and (b) summarize the results of the genetically optimized HFNN architectures when exploiting two kinds of FNN (viz. FS FNN and FR FNN) based on each fuzzy inference method. In light of the values of Table 5(a) reported there, we distinguish with two network architectures such as the premise FNN and the overall gHFNN. First, in case of the premise FNN, the network comes in the form of two fuzzy inference methods. Here, the FNN uses two membership functions for each input variable and has six fuzzy rules. In this case, as mentioned previously, the parameters of the FNN are optimized with the aid of GAs and BP learning. When considering the simplified fuzzy inference-based FNN, the minimal value of the performance index, that is PI = 5.217 and EPI = 5.142 are obtained. In case of the linear fuzzy inference-based FNN, the best results are reported in the form of the performance index such that PI = 2.929 and EPI = 3.45. Next the values of the performance index of output of the gHFNN depend on each connection point based on the individual fuzzy inference methods. The values of the performance index vis-` a-vis choice of number of layers of gHFNN related to the optimized architectures in each layer of the network are shown in Table 5. That is, according to the maximal number of inputs to be selected (Max = 4), the selected node numbers, the selected polynomial type (Type T), and its corresponding performance index (PI and EPI) were shown when the genetic optimization for each layer was carried out. For example, in case when considering simplified fuzzy inference and connection point 2 of FS FNN in Table 5(a), let us investigate the 2nd layer of the network (shadowed in Table 5(a)). The fitness value in layer 2 attains its maximum for Max = 4 when nodes 8, 7, 4, 11 (such as z8 , z7 , z4 , z11 ) occur among preferred nodes (W) chosen in the previous layer (the 1st layer) are selected as the node inputs in the present layer. Furthermore 4 inputs of Type 2 (linear function) were selected as the results of the genetic optimization, refer to Fig. 13(b). In the “Input No.” item of Table 5, a blank node marked by period (·) indicates that it has not been selected by the genetic operation. The performance of the conventional HFNN (called “SOFPNN” in the literature [19], which is composed of FNN and PNN with 4 inputs-Type 2 topology) was quantified
Genetically Optimized Hybrid Fuzzy Neural Networks
43
Table 5. Parameters of the optimization environment and computational effort (a) In case of using FS FNN Consequence part CP No. of EPI Layer Input No. inputs 1 4 6 2 5 3 2 4 4 9 7 15 01 3 3 28 7 18 · 4 4 5 1 7 19 5 2 5 3 · · 5.21 5.14 1 3 3 1 2 · 2 4 8 7 4 11 02 3 3 15 5 14 · 4 3 5 3 24 · 5 3 14 25 1 · 1 4 3 2 6 5 2 4 1 7 28 15 01 3 4 28 9 1 14 4 4 16 23 3 22 5 4 11 12 20 7 2.92 3.45 1 3 2 1 3 · 2 4 16 1 2 7 02 3 4 15 2 28 24 4 4 5 15 20 24 5 3 12 3 7 ·
Premise part Fuzzy No. of rules PI Inference (MFs)
Simplified
6 (2 + 2 + 2)
Linear
6 (2 + 2 + 2)
(b) In case of using FR FNN Consequence part No. of EPI Layer Input No. inputs 1 4 5 6 4 7 2 4 2 45 49 25 8 Simplified 3.997 3.269 3 4 42 9 40 49 (2 × 2 × 2) 4 3 36 16 33 · 5 3 43 45 28 · 1 4 6 1 5 7 2 4 49 4 44 34 8 Linear 2.069 2.518 3 2 43 2 · · (2 × 2 × 2) 4 3 33 44 5 · 5 3 41 26 42 · Premise part Fuzzy No. of rules PI Inference (MFs)
T 2 3 3 1 2 2 2 3 3 2 2 2 2 2 1 2 2 3 3 3
T 1 2 3 1 2 2 3 2 3 4
PI
EPI
2.070 0.390 0.363 0.350 0.337 2.706 0.299 0.299 0.299 0.299 0.667 0.087 0.0029 0.0014 0.0014 0.908 0.113 0.029 0.010 0.0092
2.536 0.896 0.642 0.539 0.452 3.946 0.517 0.467 0.412 0.398 0.947 0.315 0.258 0.136 0.112 1.423 0.299 0.151 0.068 0.056
PI
EPI
8.138 10.68 0.382 2.316 0.313 1.403 0.311 0.734 0.309 0.610 0.423 4.601 0.184 2.175 0.105 1.361 0.063 0.761 0.039 0.587
by the values of PI equal to 0.299 and EPI given as 0.555, whereas under the condition given as similar performance, the best results for the proposed network related to the output node mentioned previously were reported as PI = 0.299 and EPI = 0.517. In the sequel, the depth (the number of layers) as well as the width (the number of nodes) of the proposed genetically optimized HFNN (gHFNN) can be lower in comparison to the “conventional HFNN” (which immensely contributes to the compactness of the resulting network), refer to Fig. 13. In what follows, the genetic design procedure at stage (layer) of HFNN leads to the selection of the preferred nodes (or PNs)
44
S.-K. Oh and W. Pedrycz
Fig. 13. Comparison of the proposed model architecture (gHFNN) and the conventional model architecture (SOFPNN [19])
Fig. 14. Optimal topology of genetically optimized HFNN for the nonlinear function (In case of using FS FNN)
with optimal local characteristics (such as the number of input variables, the order of the polynomial, and input variables). In addition, when considering linear fuzzy inference and CP2, the best results are reported in the form of the performance index such as PI = 0.113 and EPI = 0.299 for layer 2, and PI = 0.0092 and EPI = 0.056 for layer 5. Their optimal topologies are shown in Figs. 14 (a) and (b). Figs. 15(a) and (b) depict the optimization process by showing the values of the performance index in successive cycles of both BP learning and genetic optimization when using each linear fuzzy inference-based FNN. Noticeably, the variation ratio (slope) of the performance of the network changes radically around the 1st and 2nd layer. Therefore, to effectively reduce
Genetically Optimized Hybrid Fuzzy Neural Networks
45
16 14
8
Consequence part; gPNN
6 Performance Index
Premise part; FNN
12
Premise part; FNN
5
Performance Index
7
1st layer
E_PI = 3.45
4
2nd layer 3rd layer
Consequence part; gPNN
10 1st layer
8
2nd layer
6
3rd layer
3 4th layer
PI = 2.929
50 200
400
E_PI = 2.069
5th layer
: PI : E_PI
1
4th layer
4
2
E_PI = 0.299 PI = 0.113 E_PI = 0.056 PI = 0.0092
600
Iteration
800 1000 100
200
300
400 500
Generation
(a) In case of FS FNN (linear)
: PI : E_PI
2
50 200
400
5th layer
PI = 2.518
600 800 Iteration
E_PI = 0.587 PI = 0.039
1000 150 300 450 600 750 Generation
(b) In case of FR FNN (linear)
Fig. 15. Optimization procedure of HFNN by BP learning and GAs
a large number of nodes and avoid a substantial amount of time-consuming iterations concerning HFNN layers, the stopping criterion can be taken into consideration. Referring to Figs. 14 and 15 it becomes obvious that we can optimize the network up to maximally the 2nd layer. Table 6 covers a comparative analysis including several previous models. Sugeno’s model I and II were fuzzy models based on linear fuzzy inference method while Shin-ichi’s models formed fuzzy rules by using learning method of neural networks. The study of literature [29] is based on fuzzy-neural networks using HCM clustering and evolutionary fuzzy granulation. SOFPNN [19] is a network being called the “conventional HFNN” in this study. The proposed genetically optimized HFNN (gHFNN) comes with higher accuracy and improved prediction capabilities. 5.2 Gas furnace process We illustrate the performance of the network and elaborate on its development by experimenting with data coming from the gas furnace process. The time series data (296 input-output pairs) resulting from the gas furnace process has been intensively studied in the previous literature [2-6,30-35]. The delayed terms of methane gas flow rate, u(t) and carbon dioxide density, y(t) are used as system input variables such as u(t-3), u(t-2), u(t-1), y(t-3), y(t-2), and y(t1). We use two types of system input variables of FNN structure, Type I and Type II to design an optimal model from gas furnace data. Type I utilize two system input variables such as u(t-3) and y(t-1) and Type II utilizes 3 system input variables such as u(t-2), y(t-2), and y(t-1). The output variable is y(t). Table 7 summarizes the computational aspects related to the genetic optimization of gHFNN.
46
S.-K. Oh and W. Pedrycz Table 6. Performance analysis of selected models Model Linear model [25] GMDH [25,28] Fuzzy model I Sugeno’s [1,25] Fuzzy model II FNN Type 1 Shin-ichi’s [13] FNN Type 2 FNN Type 3 Simplified FNN [29] Linear Simplified Multi-FNN [29] Linear BFPNN SOFPNN [19] MFPNN Simplified FS FNN Linear Simplified FR FNN Linear Proposed model Simplified gHFNN (FS FNN) Linear gFPNN Simplified (FR FNN) Linear
PI 12.7 4.7 1.5 1.1 0.84 0.73 0.63 2.865 2.670 0.865 0.174 0.299 0.116 5.21 2.92 3.997 2.069 0.299 0.299 0.113 0.0092 0.309 0.039
EPI 11.1 5.7 2.1 3.6 1.22 1.28 1.25 3.206 3.063 0.956 0.689 0.555 0.360 5.14 3.45 3.269 2.518 0.517 0.398 0.299 0.056 0.610 0.587
No. of rules
3 4 8(23 ) 4(22 ) 8(23 ) 9(3 + 3 + 3) 9(3 + 3 + 3) 9(3 + 3 + 3) 9(3 + 3 + 3) 6 rules/5th layer 8 rules/5th layer 6(2 + 2 + 2) 6(2 + 2 + 2) 8(2 × 2 × 2) 8(2 × 2 × 2) 6 rules/2nd layer 6 rules/5th layer 6 rules/2nd layer 6 rules/5th layer 8 rules/5th layer 8 rules/5th layer
The GAs-based design procedure is carried out in the same manner as in the previous experiments. Table 8 includes the results of the overall network reported according to various alternatives concerning various forms of FNN architecture, types of fuzzy inference and location of the connection point. When considering the simplified fuzzy inference-based FS FNN with Type I (4 fuzzy rules), the minimal value of the performance index, that is PI = 0.035 and EPI = 0.281 are obtained. In case of the linear fuzzy inference-based FS FNN with Type I (4 fuzzy rules), the best results are reported with the performance index such that PI = 0.041 and EPI = 0.267. When using Type II (6 fuzzy rules), the best results (PI = 0.0248 and EPI = 0.126) were obtained for simplified fuzzy inference and linear fuzzy inference, respectively (in the second case we have PI = 0.0256 and EPI = 0.143). In case of using FR FNN and Type II, the best results are given as the performance index such that PI = 0.026, EPI = 0.115 and PI = 0.033, EPI = 0.119 for simplified and linear fuzzy inference respectively. When using FS FNN and Type I, Fig. 16 illustrates the detailed optimal topology of the gHFNN with 3 layers of PNN; the network comes with the following values: PI = 0.017 and EPI = 0.267. The proposed network enables the architecture to be a structurally optimized and gets simpler than the
Genetically Optimized Hybrid Fuzzy Neural Networks
47
Table 7. Computational aspects of the optimization of gHFNN (a) In case of using FS FNN Generation 150 Population size 60 GAs Elite population size (W) 30 Premise structure (FNN) 10 (per one variable) String length Consequence structure (PNN) 3 + 3 + 24 No. of entire system inputs 2 Learning iteration 300 Premise Simplified 0.0014 Learning rate tuned (FNN) Linear 0.0052 Momentum Simplified 0.0002 Coefficient tuned Linear 0.0004 gHFNN No. of rules 4/6 CP 1 4/6 No. of system inputs Consequence CP 2 2/3 (Gpnn) Maximal layer 5 No. of inputs to be selected (N) 1 ≤ N ≤ 4 (Max) Type (T) 1≤T≤3 N, T : integer (b) In case of using FR FNN and Type II Generation 150 Population size 60 GAs Elite population size (W) 30 Premise structure (FNN) 10 (per one variable) String length Consequence structure (PNN) 3+3+24 No. of entire system inputs 3 Learning iteration 500 Premise Simplified 0.0524 Learning rate tuned (FNN) Linear 0.0144 Momentum Simplified 0.00086 Coefficient tuned Linear 0.00064 gHFNN No. of rules 4/8 No. of entire inputs 4/8 Consequence Maximal layer 5 (Gpnn) No. of inputs to be selected (N) 1 ≤ N ≤ 4 (Max) Type (T) 1≤T≤3 N, T : integer
conventional HFNN. Fig. 17(a) illustrates the optimization process by visualizing the performance index in successive cycles (iteration and generation). It also shows the optimized network architecture when taking into consideration HFNN based on linear fuzzy inference and CP1, refer to Table 8(a) and Fig. 16. As shown in Figs. 17(a) and (b), the variation ratio (slope) of the performance of the network is almost the same around the 2nd through 5th layer.
48
S.-K. Oh and W. Pedrycz Table 8. Performance index of HFNN for the gas furnace
(a) In case of using FS FNN and Type I Premise part Consequence part Fuzzy No. of rules CP No. of PI EPI Layer Input No. Inference (MFs) inputs 1 2 3 1 · · 2 4 24 11 30 18 01 3 3 9 10 23 · Simplified 4 0.035 0.281 4 4 7 13 27 20 (2 + 2) 5 3 1 10 7 · 1 2 1 2 · · 2 4 1 4 5 7 02 3 3 29 27 26 · 4 3 15 13 21 · 5 4 8 4 20 13 1 4 4 2 1 3 2 4 7 12 2 10 01 3 4 20 21 5 3 Linear 4 0.041 0.267 4 3 22 13 29 · (2 + 2) 5 4 25 18 27 9 1 2 1 2 · · 2 3 4 6 5 · 02 3 4 6 14 7 1 4 3 15 3 2 · 5 3 16 6 14 · (b) In case of using FR FNN and Type II Premise part Consequence part Fuzzy No. of rules No. of PI EPI Layer Input No. Inference (MFs) inputs 1 4 2 5 6 7 2 3 26 15 16 · Simplified 8 0.026 0.115 3 3 21 3 27 · (2 × 2 × 2) 4 3 24 3 18 · 5 4 13 12 29 20 1 4 6 5 2 8 2 4 21 18 6 9 Linear 8 0.033 0.119 3 4 4 24 5 6 (2 × 2 × 2) 4 3 28 4 5 · 5 3 21 18 25 ·
T 3 2 2 2 2 3 3 2 3 2 3 2 2 2 3 3 2 2 2 2
T 2 2 2 2 1 1 2 2 2 1
PI
EPI
0.025 0.024 0.020 0.019 0.018 0.024 0.021 0.020 0.019 0.018 0.019 0.018 0.017 0.016 0.015 0.027 0.021 0.018 0.018 0.016
0.328 0.269 0.265 0.262 0.254 0.328 0.282 0.270 0.268 0.265 0.292 0.271 0.267 0.263 0.258 0.310 0.279 0.270 0.263 0.259
PI
EPI
0.021 0.019 0.018 0.018 0.018 0.083 0.028 0.022 0.021 0.021
0.124 0.115 0.114 0.111 0.109 0.146 0.116 0.110 0.106 0.104
Table 9 contrasts the performance of the genetically developed network with other fuzzy and fuzzy-neural networks reported in the literature. It becomes obvious that the proposed genetically optimized HFNN architectures outperform other models both in terms of their accuracy and generalization capabilities.
Genetically Optimized Hybrid Fuzzy Neural Networks
u(t-3)
N
Π
N
Π
PN 1 3 3 PN 2 4 2 PN 4 4 3 PN 5 3 3 PN 13 3 1 PN 16 3 1 PN 18 1 2 PN 21 3 2 PN 23 3 2 PN 24 2 3 PN 29 2 2
∑ ∑
1
y(t-1)
N
Π
N
Π
∑ ∑
1
PN 3 4 2 PN 5 4 2 PN 20 4 2 PN 21 3 2
49
y^
PN 20 4 2
Fig. 16. Genetically optimized HFNN (gHFNN) with FS FNN (linear) 0.9
1.6
: PI : E_PI
0.8
Premise part; FNN
Consequence part; gPNN
0.6
2nd layer 3rd layer
0.5
4th layer
0.4
5th layer
0.3 0.2
E_PI = 0.267 E_PI = 0.258
E_PI = 0.267
Consequence part; gPNN
1.2
1st layer
Performance Index
Performance Index
0.7
: PI : E_PI
1.4
1st layer
1 Premise part; FR_FNN
0.8
2nd layer 3rd layer
0.6 4th layer
0.4 5th layer
0.1
0.2
PI = 0.041
E_PI = 0.119
PI = 0.017 PI = 0.015
30 100 200 300 Iteration
150
300 450 Generation
600
750
E_PI = 0.104
PI = 0.033
30
200 400 500 Iteration
PI = 0.0211
150
300 450 600 Generation
750
(a) In case of using FS FNN (linear) with (b) In case of using FR FNN (linear) Type I and Type II Fig. 17. Optimization procedure of HFNN by BP learning and GAs
5.3 NOx emission process of gas turbine power plant NOx emission process is modeled using the data of gas turbine power plant coming from a GE gas turbine power plant located in Virginia, US. The input variables include AT (Ambient Temperature at site), CS (Compressor Speed), LPTS (Low Pressure Turbine Speed), CDP (Compressor Discharge Pressure), and TET (Turbine Exhaust Temperature). The output variable is NOx [27,29,32]. The performance index is defined by (8). We consider 260 pairs of the original input-output data. 130 out of 260 pairs of input-output data are used as learning set; the remaining part serves as a
50
S.-K. Oh and W. Pedrycz Table 9. Performance analysis of selected models Model Kim, et al.’s model [30] Lin and Cunningham’s mode [31] Simplified Min-Max [5] Linear Simplified Linear Simplified Complex [2] Linear Hybrid [4] Simplified (GAs+Complex) Linear Simplified HCM [3] Linear GAs [5]
Fuzzy
Simplified HCM+GAs [3] Linear Neural Networks [3] Oh’s Adaptive FNN [6] FNN [32] Multi-FNN [33] SOFPNN
Simplified Linear Simplified Linear Generic [34] Advanced [35] Simplified FS FNN Linear Simplified FR FNN Linear
Proposed model gHFNN (FS FNN)
gHFNN (FR FNN)
Simplified Linear Simplified Linear
PI EPI No. of rules 0.034 0.244 2 0.071 0.261 4 0.022 0.335 4(2 × 2) 0.022 0.336 6(3 × 2) 0.024 0.358 4(2 × 2) 0.020 0.362 6(3 × 2) 0.023 0.344 4(2 × 2) 0.018 0.264 4(2 × 2) 0.024 0.328 4(2 × 2) 0.023 0.306 4(2 × 2) 0.024 0.329 4(2 × 2) 0.017 0.289 4(2 × 2) 0.755 1.439 6(3 × 2) 0.018 0.286 6(3 × 2) 0.035 0.289 4(2 × 2) 0.022 0.333 6(3 × 2) 0.026 0.272 4(2 × 2) 0.020 0.2642 6(3 × 2) 0.034 4.997 0.021 0.332 9(3 × 3) 0.022 0.353 4(2 × 2) 0.043 0.264 6(3 + 3) 0.037 0.273 6(3 + 3) 0.025 0.274 6(3 + 3) 0.024 0.283 6(3 + 3) 0.017 0.250 4 rules/5th layer 0.019 0.264 6 rules/5th layer 0.035 0.281 4(2 + 2) 0.024 0.126 6(2 + 2 + 2) 0.041 0.267 4(2 + 2) 0.025 0.143 6(2 + 2 + 2) 0.024 0.329 4(2 × 2) 0.026 0.115 8(2 × 2 × 2) 0.025 0.265 4(2 × 2) 0.033 0.119 8(2 × 2 × 2) 0.018 0.254 4 rules/5th layer 0.018 0.112 6 rules/5th layer 0.015 0.258 4 rules/5th layer 0.018 0.110 6 rules/5th layer 0.017 0.250 4 rules/5th layer 0.018 0.109 8 rules/5th layer 0.016 0.249 4 rules/5th layer 0.021 0.104 8 rules/5th layer
Genetically Optimized Hybrid Fuzzy Neural Networks
51
testing set. Using NOx emission process data, the regression model is y = −163.77341 − 0.06709 x1 + 0.00322 x2 + 0.00235 x3 + 0.26365 x4 + 0.20893 x5
(11)
And it comes with PI = 17.68 and EPI = 19.23. We will be using these results as a reference point when discussing gHFNN models. Table 10 summarizes Table 10. Summary of the parameters of the optimization environment (a) In case of using FS FNN Generation 150 Population size 100 GAs Elite population size 50 Premise structure 10 (per one variable) String length Consequence CP 1 3 + 3 + 70 structure CP 2 3 + 3 + 35 No. of entire system inputs 5 Learning iteration 1000 Premise Simplified 0.052 Learning rate tuned (FNN) Linear 0.034 Momentum Simplified 0.010 Coefficient tuned Linear 0.001 gHFNN No. of rules 10(2 + 2 + 2 + 2 + 2) CP 1 10 No. of system inputs CP 2 5 Consequence Maximal layer 5 (gPNN) No. of inputs to be CP 1 1 ≤ N ≤ 10 (Max) selected (N) CP 1 1 ≤ N ≤ 5 (Max) Type (T) 1≤T≤3 N, T : integer (b) In case of using FR FNN and Type II Generation 150 Population size 100 GAs Elite population size (W) 50 Premise structure (FNN) 10 (per one variable) String length Consequence structure (PNN) 3 + 3 + 105 No. of entire system inputs 5 Learning iteration 1000 Premise Simplified 0.568 Learning rate tuned Linear 0.651 Momentum Simplified 0.044 Coefficient tuned Linear 0.064 gHFNN No. of rules 32(2 × 2 × 2 × 2 × 2) No. of entire inputs 32 Consequence Maximal layer 5 (Gpnn) No. of inputs to be selected (N) 1 ≤ N ≤ 15 (Max) Type (T) 1≤T≤3 N, T : integer
52
S.-K. Oh and W. Pedrycz
the parameters of the optimization environment. The parameters used for optimization of this process modeling are almost the same as used in the previous experiments. Table 11 summarizes the detailed results. When using FS FNN, the best results for the network are obtained when using linear fuzzy inference and CP Table 11. Parameters of the optimization environment and computational effort (a) In case of using FS FNN Premise part Consequence part Fuzzy No. of rules CP No. of PI EPI Layer Type Inference (MFs) inputs 1 7 2 2 6 2 01 3 9 1 4 10 1 Simplified 10 22.331 19.783 5 4 2 (2 + 2 + 2 + 2 + 2) 1 5 2 2 5 2 02 3 5 3 4 3 2 5 4 2 1 9 2 2 5 2 01 3 9 1 4 2 2 Linear 10 8.054 12.147 5 9 1 (2 + 2 + 2 + 2 + 2) 1 5 2 2 5 2 02 3 5 2 4 5 3 5 3 2 (b) In case of using FR FNN Premise part Consequence part Fuzzy No. of rules No. of PI EPI Layer Type Inference (MFs) inputs 1 13 2 Simplified 32 2 12 1 (2 × 2 × 2 × 2 × 2) 0.711 1.699 3 11 1 4 8 1 5 3 3 1 10 2 Linear 32 2 13 1 (2 × 2 × 2 × 2 × 2) 0.079 0.204 3 3 3 4 12 1 5 3 3
PI
EPI
0.916 0.623 0.477 0.386 0.337 1.072 0.176 0.105 0.060 0.049 0.023 0.0095 0.0057 0.0057 0.0045 2.117 0.875 0.550 0.390 0.340
2.014 1.430 1.212 1.077 1.016 2.220 0.291 0.168 0.113 0.081 0.137 0.044 0.029 0.027 0.026 4.426 1.647 1.144 0.793 0.680
PI
EPI
0.149 0.065 0.046 0.044 0.041 0.205 0.049 0.028 0.023 0.019
0.921 0.189 0.134 0.125 0.111 1.522 0.646 0.437 0.330 0.286
Genetically Optimized Hybrid Fuzzy Neural Networks 25
0.3
: PI : E_PI Premise part; FNN
20
Consequence part; gPNN
1st layer
15
3rd layer E_PI = 12.147
4th layer
10 5th layer
Performance Index
Performance Index
2nd layer
0.2 2nd layer
0.15
3rd layer 4th layer
0.1
5th layer
PI = 8.054
5
E_PI = 0.044
0.05
E_PI = 0.026
E_PI = 0.026 PI = 0.0045
400
600
Iteration
800
: PI : E_PI
Consequence part; gPNN
0.25
1st layer
200
53
1000 150 300 450 600 750 Generation
(a) Premise and consequence part
PI = 0.0095
150
300
PI = 0.0045
450
600
750
Generation
(b) Consequence part (extended)
Fig. 18. Optimal procedure of gHFNN with FS FNN (linear) by BP and GAs
1 with Type 1 (linear function) and 9 nodes at input; this network comes with the value of PI equal to 0.0045 and EPI set as 0.026. In case of using FR FNN and simplified fuzzy inference, the most preferred network architecture have been reported as PI = 0.041 and EPI = 0.111. As shown in Table 11 and Fig. 18, the variation ratio (slope) of the performance of the network changes radically at the 2nd layer. Therefore, to effectively reduce a large number of nodes and avoid a large amount of time-consuming iteration of gHFNN, the stopping criterion can be taken into consideration up to maximally the 2nd layer. Table 12 covers a comparative analysis including several previous fuzzyneural network models. The experimental results clearly reveal that the proposed approach and the resulting model outperform the existing networks both in terms of better approximation and generalization capabilities.
6 Concluding remarks In this study, we have introduced a class of gHFNN driven genetic optimization regarded as a modeling vehicle for nonlinear and complex systems. The genetically optimized HFNNs are constructed by combining FNNs with gPNNs. In contrast to the conventional HFNN structures and their learning, the proposed model comes with two kinds of rule-based FNNs (viz. FS FNN and FR FNN based on two types of fuzzy inferences) as well as a diversity of local characteristics of PNs that are extremely useful when coping with various nonlinear characteristics of the system under consideration.
54
S.-K. Oh and W. Pedrycz Table 12. Performance analysis of selected models
Model PI EPI Regression model [32] 17.68 19.23 FNN 5.835 Ahn et al. [27] AIM 8.420 Simplified 6.269 8.778 FNN [32] Linear 3.725 5.291 Simplified 2.806 5.164 Multi-FNN [33] Linear 0.720 2.025 Simplified 22.331 19.783 FS FNN Linear 8.054 12.147 Simplified 0.711 1.699 FR FNN Linear 0.079 0.204 Proposed model 0.176 0.291 Simplified gHFNN 0.049 0.081 (FS FNN) 0.0095 0.044 Linear 0.0045 0.026 0.065 0.189 Simplified gHFNN 0.041 0.111 (FR FNN) 0.049 0.646 Linear 0.019 0.286
No. of rules
30(6 + 6 + 6 + 6 + 6) 30(6 + 6 + 6 + 6 + 6) 30(6 + 6 + 6 + 6 + 6) 30(6 + 6 + 6 + 6 + 6) 10(2 + 2 + 2 + 2 + 2) 10(2 + 2 + 2 + 2 + 2) 32(2 × 2 × 2 × 2 × 2) 32(2×2×2×2×2) 10 rules/2nd layer 10 rules/5th layer 10 rules/2nd layer 10 rules/5th layer 32 rules/2nd layer 32 rules/5th layer 32 rules/2nd layer 32 rules/5th layer
The comprehensive design methodology comes with the parametrically as well as structurally optimized network architecture. A few general notes are worth stressing: 1) as the premise structure of the gHFNN, the optimization of the rule-based FNN hinges on genetic algorithms and back-propagation (BP) learning algorithm: The GAs leads to the auto-tuning of vertexes of membership function, while the BP algorithm helps produce optimal parameters of the consequent polynomial of fuzzy rules through learning; 2) the gPNN that is the consequent structure of the gHFNN is based on the technologies of the extended GMDH algorithm and GAs: The extended GMDH method is comprised of both a structural phase such as a self-organizing and evolutionary algorithm and a parametric phase driven by the least square error (LSE)-based learning. Furthermore the PNN architecture is optimized by the genetic optimization that leads to the selection of the optimal nodes (or PNs) with local characteristics such as the number of input variables, the order of the polynomial, and a collection of the specific subset of input variables. In this sense, we have constructed a coherent development platform in which all components of CI are fully utilized. In the sequel, a variety of architectures of the proposed gHFNN driven to genetic optimization have been discussed. The model is inherently dynamic - the use of the genetically optimized PNN (gPNN) of consequent structure of the overall network is essential to the generation process of the “optimally self-organizing” network by selecting its width and depth. The series of experiments helped compare the network with other models through which we found the network to be of superior quality.
Genetically Optimized Hybrid Fuzzy Neural Networks
55
7 Acknowledgement This work has been supported by KESRI(I-2004-0-074-0-00), which is funded by MOCIE (Ministry of commerce, industry and energy).
References 1. Kang G, Sugeno M (1987) Fuzzy modeling. Trans Soc Instrum Control Eng 23(6cr):106–108 2. Oh SK, Pedrycz W (2000) Fuzzy identification by means of auto-tuning algorithm and its application to nonlinear systems. Fuzzy Sets Syst 115(2):205–230 3. Park BJ, Pedrycz W, Oh SK (2001) Identification of fuzzy models with the aid of evolutionary data granulation. IEE Proc-Control Theory Appl 148(5):406–418 4. Oh SK, Pedrycz W, Park BJ (2002) Hybrid identification of fuzzy rule-based models. Int J Intell Syst 17(1):77–103 5. Park BJ, Oh SK, Ahn TC, Kim HK (1999) Optimization of fuzzy systems by means of GA and weighting factor. Trans Korean Inst Electr Eng 48A(6):789– 799 (In Korean) 6. Oh SK, Park CS, Park BJ (1999) On-line modeling of nonlinear process systems using the adaptive fuzzy-neural networks. Trans Korean Inst Electr Eng 48A(10):1293–1302 (In Korean) 7. Narendra KS, Parthasarathy K (1991) Gradient methods for the optimization of dynamical systems containing neural networks. IEEE Trans Neural Netw 2:252–262 8. Goldberg DE (1989) Genetic algorithms in search, optimization & machine learning. Addison-wesley, Reading 9. Michalewicz Z (1996) Genetic Algorithms + Data Structures = Evolution Programs. Springer, Berlin Heidelberg Newyork 10. Holland JH (1975) Adaptation in natural and artificial systems. The University of Michigan Press, Ann Arbour 11. Pedrycz W, Peters JF (1998) Computational intelligence and software engineering. World Scientific, Singapore 12. Computational intelligence by programming focused on fuzzy neural networks and genetic algorithms. Naeha, Seoul (In Korean) 13. Horikawa S, Furuhashi T, Uchigawa Y (1992) On fuzzy modeling using fuzzy neural networks with the back propagation algorithm. IEEE Trans Neural Netw 3(5):801–806 14. Oh SK, Pedrycz W (2002) The design of self-organizing polynomial neural networks. Inf Sci 141(3–4):237–258 15. Oh SK, Pedrycz W, Park BJ (2003) Polynomial neural networks architecture: Analysis and Design. Comput Electr Eng 29(6):653–725 16. Ohtani T, Ichihashi H, Miyoshi T, Nagasaka K (1998) Orthogonal and successive projection methods for the learning of neurofuzzy GMDH. Inf Sci 110:5–24 17. Ohtani T, Ichihashi H, Miyoshi T, Nagasaka K (1998) Structural learning with M-Apoptosis in neurofuzzy GMHD. In: Proceedings of the 7th IEEE International Conference on Fuzzy Systems:1265–1270
56
S.-K. Oh and W. Pedrycz
18. Ichihashi H, Nagasaka K (1994) Differential minimum bias criterion for neurofuzzy GMDH. In: Proceedings of 3rd International Conference on Fuzzy Logic Neural Nets and Soft Computing IIZUKA’94:171–172 19. Park BJ, Pedrycz W, Oh SK (2002) Fuzzy polynomial neural networks: hybrid architectures of fuzzy modeling. IEEE Trans Fuzzy Syst 10(5):607–621 20. Oh SK, Pedrycz W, Park BJ (2003) Self-organizing neurofuzzy networks based on evolutionary fuzzy granulation. IEEE Trans Syst Man and Cybern A 33(2):271–277 21. Cordon O et al. (2004) Ten years of genetic fuzzy systems: current framework and new trends. Fuzzy Sets Syst 141(1):5–31 22. Ivahnenko AG (1968) The group method of data handling: a rival of method of stochastic approximation. Sov Autom Control 13(3):43–55 23. Yamakawa T (1993) A new effective learning algorithm for a neo fuzzy neuron model. 5th IFSA World Conference:1017–1020 24. Oh SK, Yoon KC, Kim HK (2000) The Design of optimal fuzzy- eural networks structure by means of GA and an aggregate weighted performance index. J Control, Autom Syst Eng 6(3):273–283 (In Korean) 25. Park MY, Choi HS (1990) Fuzzy control system. Daeyoungsa, Seoul (In Korean) 26. Box G.EP, Jenkins GM (1976) Time series analysis, forecasting, and control, 2nd edn. Holden-Day, SanFransisco 27. Ahn TC, Oh SK (1997) Intelligent models concerning the pattern of an air pollutant emission in a thermal power plant, Final Report, EESRI 28. Kondo T (1986) Revised GMDH algorithm estimating degree of the complete polynomial. Trans Soc Instrum Control Eng 22(9):928–934 29. Park HS, Oh SK (2003) Multi-FNN identification based on HCM clustering and evolutionary fuzzy granulation. Int J Control, Autom Syst 1(2):194–202 30. Kim E, Lee H, Park M, Park M (1998) A simply identified sugeno-type fuzzy model via double clustering. Inf Sci 110:25–39 31. Lin Y, Cunningham III GA (1997) A new approach to fuzzy-neural modeling, IEEE Trans Fuzzy Syst 3(2):190–197 32. Oh SK, Pedrycz W, Park HS (2003) Hybrid identification in fuzzy-neural networks. Fuzzy Sets Syst 138(2):399–426 33. Park HS, Oh SK (2000) Multi-FNN identification by means of HCM clustering and its optimization using genetic algorithms. J Fuzzy Logic Intell Syst 10(5):487–496 (In Korean) 34. Park BJ, Oh SK, Jang SW (2002) The design of adaptive fuzzy polynomial neural networks architectures based on fuzzy neural networks and self-organizing networks. J Control Autom Syst Eng 8(2):126–135 (In Korean) 35. Park BJ, Oh SK (2002) The analysis and design of advanced neurofuzzy polynomial networks. J Inst Electron Eng Korea 39-CI(3):18–31 (In Korean) 36. Park BJ, Oh SK, Pedrycz W, Kim HK (2005) Design of evolutionally optimized rule-based fuzzy neural networks on fuzzy relation and evolutionary optimization. International Conference on Computational Science. Lecture Notes in Computer Science 3516:1100–1103 37. Oh SK, Park BJ, Pedrycz W, Kim HK (2005) Evolutionally optimized fuzzy neural networks based on evolutionary fuzzy granulation. Lecture Notes in Computer Science 3483:887–895 38. Oh SK, Park BJ, Pedrycz W, Kim HK (2005) Genetically optimized hybrid fuzzy neural networks in modeling software data. Lecture Notes in Artificial Intelligence 3558:338–345
Genetically Optimized Hybrid Fuzzy Neural Networks
57
39. Zadeh NN, Darvizeh A, Jamali A, Moeini A (2005) Evolutionary design of generalized polynomial neural networks for modeling and prediction of explosive forming process. J Mater Process Technol 164(15):1561–1571 40. Delivopoulos E, Theocharis JB (2004) A modified PNN algorithm with optimal PD modeling using the orthogonal least squares method. Inf Sci 168(3):133–170
Genetically Optimized Self-organizing Neural Networks Based on Polynomial and Fuzzy Polynomial Neurons: Analysis and Design Sung-Kwun Oh and Witold Pedrycz
Summary. In this study, we introduce and investigate a class of neural architectures of self-organizing neural networks (SONN) that is based on a genetically optimized multilayer perceptron with polynomial neurons (PNs) or fuzzy polynomial neurons (FPNs), develop a comprehensive design methodology involving mechanisms of genetic optimization and carry out a series of numeric experiments. The conventional SONN is based on a self-organizing and an evolutionary algorithm rooted in a natural law of survival of the fittest as the main characteristics of the extended Group Method of Data Handling (GMDH) method, and utilized the polynomial order (viz. linear, quadratic, and modified quadratic) as well as the number of node inputs fixed (selected in advance by designer) at the corresponding nodes (PNs or FPNs) located in each layer through a growth process of the network. Moreover it does not guarantee that the SONN generated through learning results in the optimal network architecture. We distinguish between two kinds of SONN architectures, that is, (a) Polynomial Neuron (PN) based and (b) Fuzzy Polynomial Neuron (FPN) based self-organizing neural networks. This taxonomy is based on the character of each neuron structure in the network. The augmented genetically optimized SONN (gSONN) results in a structurally optimized structure and comes with a higher level of flexibility in comparison to the one encountered in the conventional SONN. The GA-based design procedure being applied at each layer of SONN leads to the selection of preferred nodes (PNs or FPNs) with specific local characteristics (such as the number of input variables, the order of the polynomial, and a collection of the specific subset of input variables) available within the network. In the sequel, two general optimization mechanisms of the gSONN are explored: the structural optimization is realized via GAs whereas for the ensuing detailed parametric optimization we proceed with a standard least square method-based learning. Each node of the PN based gSONN exhibits a high level of flexibility and realizes a collection of preferred nodes as well as a preferred polynomial type of mapping (linear, quadratic, and modified quadratic) between input and output variables. FPN based gSONN dwells on the ideas of fuzzy rule-based computing and neural networks. The performance of the gSONN is quantified through experimentation that exploits standard data already used in fuzzy or neurofuzzy modeling. These results reveal superiority of the proposed networks over the existing fuzzy and neural models.
S.-K. Oh and W. Pedrycz: Genetically Optimized Self-organizing Neural Networks Based on Polynomial and Fuzzy Polynomial Neurons: Analysis and Design, Studies in Computational Intelligence (SCI) 82, 59–108 (2008) c Springer-Verlag Berlin Heidelberg 2008 www.springerlink.com
60
S.-K. Oh and W. Pedrycz
1 Introduction Recently, lots of attention has been directed towards advanced techniques of complex system modeling. The challenging quest for constructing models of the systems that come with significant approximation and generalization abilities as well as are easy to comprehend has been within the community for decades. While neural networks, fuzzy sets and evolutionary computing as the technologies of Computational Intelligence (CI) have expanded and enriched a field of modeling quite immensely, they have also gave rise to a number of new methodological issues and increased our awareness about tradeoffs one has to make in system modeling [1–4]. The most successful approaches to hybridize fuzzy systems with learning and adaptation have been made in the realm of CI. Especially neural fuzzy systems and genetic fuzzy systems hybridize the approximate inference method of fuzzy systems with the learning capabilities of neural networks and evolutionary algorithms [5]. When the dimensionality of the model goes up (say, the number of variables increases), so do the difficulties. Fuzzy sets emphasize the aspect of transparency of the models and a role of a model designer whose prior knowledge about the system may be very helpful in facilitating all identification pursuits. On the other hand, to build models of substantial approximation capabilities, there is a need for advanced tools. The art of modeling is to reconcile these two tendencies and find a workable and efficient synergistic environment. Moreover it is also worth stressing that in many cases the nonlinear form of the model acts as a two-edge sword: while we gain flexibility to cope with experimental data, we are provided with an abundance of nonlinear dependencies that need to be exploited in a systematic manner. In particular, when dealing with high-order nonlinear and multivariable equations of the model, we require a vast amount of data for estimating all its parameters [1–2]. To help alleviate the problems, one of the first approaches along the line of a systematic design of nonlinear relationships between system’s inputs and outputs comes under the name of a Group Method of Data Handling (GMDH). GMDH was developed in the late 1960s by Ivakhnenko [6–9] as a vehicle for identifying nonlinear relations between input and output variables. While providing with a systematic design procedure, GMDH comes with some drawbacks. First, it tends to generate quite complex polynomial even for relatively simple systems (experimental data). Second, owing to its limited generic structure (that is quadratic two-variable polynomials), GMDH also tends to produce an overly complex network (model) when it comes to highly nonlinear systems. Third, if there are less than three input variables, GMDH algorithm does not generate a highly versatile structure. To alleviate the problems associated with the GMDH, Self-Organizing Neural Networks (SONN) (viz. polynomial neuron (PN)-based SONN and fuzzy polynomial neuron (FPN)based SONN, or called SOPNN/FPNN) were introduced by Oh and Pedrycz [10–13] as a new category of neural networks or neuro-fuzzy networks. In a nutshell, these networks come with a high level of flexibility associated with
Genetically Optimized Self-organizing Neural Networks
61
each node (processing element forming a Partial Description (PD) (viz. Polynomial Neuron (PN) or Fuzzy Polynomial Neuron (FPN)) can have a different number of input variables as well as exploit a different order of the polynomial (say, linear, quadratic, cubic, etc.). In comparison to well-known neural networks or neuro-fuzzy networks whose topologies are commonly selected and kept prior to all detailed (parametric) learning, the SONN architecture is not fixed in advance but becomes generated and optimized in a dynamic way. As a consequence, the SONNs show a superb performance in comparison to the previously presented intelligent models. Although the SONN has a flexible architecture whose potential can be fully utilized through a systematic design, it is difficult to obtain the structurally and parametrically optimized network because of the limited design of the nodes (viz. PNs or FPNs) located in each layer of the SONN. In other words, when we construct nodes of each layer in the conventional SONN, such parameters as the number of input variables (nodes), the order of the polynomial, and the input variables available within a node (viz. PN or FPN) are fixed (selected) in advance by the designer. Accordingly, the SONN algorithm exhibits some tendency to produce overly complex networks as well as a repetitive computation load by the trial and error method and/or the repetitive parameter adjustment by designer like in case of the original GMDH algorithm. In order to generate a structurally and parametrically optimized network, such parameters need to be optimal. In this study, in addressing the above problems with the conventional SONN as well as the GMDH algorithm, we introduce a new genetic design approach; as a consequence we will be referring to these networks as genetically optimized SONN (gSONN). The determination of the optimal values of the parameters available within an individual PN or FPN (viz. the number of input variables, the order of the polynomial, and a collection of preferred nodes) leads to a structurally and parametrically optimized network. As a result, this network is more flexible as well as exhibits simpler topology in comparison to the conventional SONN discussed in the previous research. Let us reiterate that the objective of this study is to develop a general design methodology of gSONN modeling, come up with a logic-based structure of such model and propose a comprehensive evolutionary development environment in which the optimization of the models can be efficiently carried out both at the structural as well as parametric level [14]. This chapter is organized in the following manner. First, Section 2 gives a brief introduction to the architecture and development of the SONNs. Section 3 introduces the genetic optimization used in SONN. The genetic design of the SONN comes with an overall description of a detailed design methodology of SONN based on genetically optimized multi-layer perceptron architecture in Section 4. In Section 5, we report on a comprehensive set of experiments. Finally concluding remarks are covered in Section 6. To evaluate the performance of the proposed model, we exploit two well-known time series data [10, 11, 13, 15, 16, 20–41]. Furthermore, the network is directly contrasted with several existing neurofuzzy models reported in the literatures.
62
S.-K. Oh and W. Pedrycz
2 The architecture and development of the self-organizing neural networks (SONN) Proceeding with the overall SONN architecture, essential design decisions have to be made with regard to the number of input variables, the order of the polynomial, and a collection of the specific subset of input variables. We distinguish between two kinds of the SONN architectures (PN-based SONN and FPN-based SONN). 2.1 Polynomial Neuron (PN) based SONN and its topology As underlined, the SONN algorithm is based on the GMDH method and utilizes a class of polynomials such as linear, quadratic, modified quadratic, etc. to describe basic processing realized there. By choosing the most significant input variables and an order of the polynomial among various types of forms available, we can obtain the best one - it comes under a name of a partial description (PD). It is realized by selecting nodes at each layer and eventually generating additional layers until the best performance has been reached. Such methodology leads to an optimal SONN structure. Let us recall that the input-output data are given in the form (Xi , yi ) = (x1i , x2i , · · · , xN i , yi ),
i = 1, 2, 3, · · · , n.
(1)
where N is the number of input variables, i is the datanumber of each input and output variable, and n denotes the number of data in the dataset. The input-output relationship for the above data realized by the SONN algorithm can be described in the following manner y = f (x1 , x2 , · · · , xN ).
(2)
Where, x1 , x2 , · · · , xN denote the outputs of the 1st layer layer of PN nodes (the inputs of the 2nd layer (PN nodes)). The estimated output y reads as y = c0 +
N i=1
ci xi +
N N i=1 j=1
cij xi xj
N N N
cijk xi xj xk · · ·
(3)
i=1 j=1 k=1
Where, C(c0 , ci , cij , cijk , · · · ) (i, j, k, · · · : 1, 2, · · · , N ) and X(xi , xj , xk , · · · ), (i, j, k, · · · : 1, 2, · · · , N ) are vectors of the coefficients and input variables of the resulting multi-input single-output (MISO) system, respectively. The design of the SONN structure proceeds further and involves a generation of some additional layers. These layers consist of PNs (PDs) for which the number of input variables, the polynomial order, and a collection of the specific subset of input variables are genetically optimized across the layers. The detailed PN involving a certain regression polynomial is shown in Table 1.
Genetically Optimized Self-organizing Neural Networks
63
Table 1. Different forms of regression polynomial building a PN
PP PP Number of PP inputs PP Order PP
1
2
1 (Type 1) Linear Bilinear 2 (Type 2) Biquadratic-1 Quadratic 2 (Type 3) Biquadratic-2 The following types of the polynomials are used; • Bilinear = c0 + c1 x1 + c2 x2 • Biquadratic-1 (Basic) = Bilinear + c3 x21 + c4 x22 + c5 x1 x2 • Biquadratic-2 (Modified) = Bilinear + c3 x1 x2 1 st la y e r
3 Trilinear Triquadratic-1 Triquadratic-2
2 n d la y e r o r h ig h e r
PN PN x1
PN
PN
x2
PN
x3
PN
PN PN PN PN
PN x4
yˆ
PN
PN
PN
PN
Input variables zp
zp
zq
zq
Partial Description
2
C0 +C1 zp +C2 zq+C3 z 2p+C4 z 2q +C5 zp zq
z
Polynomial order
Fig. 1. A general topology of the PN based-SONN : note a biquadratic polynomial in the partial description (z: intermediate variable)
The architecture of the PN based SONN is visualized in Fig. 1. The structure of the SONN is genetically optimized on the basis of the design alternatives available within a PN occurring in each layer. In the sequel, the SONN embraces diverse topologies of PN being selected on the basis of the number of input variables, the order of the polynomial, and a collection of the specific subset of input variables (as shown in Table 1). The choice of the number of input variables, the polynomial order, and input variables available within each node itself helps select the best model with
64
S.-K. Oh and W. Pedrycz
respect to the characteristics of the data, model design strategy, nonlinearity and predictive capabilities. 2.2 Fuzzy Polynomial Neuron (FPN) based SONN and its topology In this section, we introduce a fuzzy polynomial neuron (FPN). This neuron, regarded as a generic type of the processing unit, dwells on the concepts of fuzzy sets and neural networks. We show that the FPN encapsulates a family of nonlinear “if-then” rules. When arranged together, FPNs build a selforganizing neural network (SONN). In the sequel, we investigate architectures arising therein. 2.2.1 Fuzzy polynomial neuron (FPN) As visualized in Fig. 2, the FPN consists of two basic functional modules. The first one, labeled by F, is a collection of fuzzy sets (here Al and Bk ) that form an interface between the input numeric variables and the processing part realized by the neuron. Here q and xp denote input variables. The second module (denoted here by P) is about the function-based nonlinear (polynomial) processing. This nonlinear processing involves some input variables (xi and xj ). Quite commonly, we will be using a polynomial form of the nonlinearity, hence the name of the fuzzy polynomial processing unit. The use of polynomials is motivated by their generality. In particular, they include constant and linear mappings as their special cases (that are used quite often in rule-based systems). In other words, FPN realizes a family of multiple-input single-output rules. Each rule, refer again to Fig. 2, reads in the form if xp is Al and xq is Bk then z is Plk (xi , xj , alk )
FPN x i ,x j
F xp
xq
(4)
P
µ1
µˆ 1
P1
µ2
µˆ 2
P2
µ3
µˆ 3
P3
µK
µˆ K
PK
{A l }
∑
z
{Bk }
Fig. 2. A general topology of the generic FPN module (F: fuzzy set-based processing part, P: the polynomial form of mapping)
Genetically Optimized Self-organizing Neural Networks
65
Table 2. Different forms of the regression polynomials standing in the consequence part of the fuzzy rules
PP Number of PP PP inputs P PP Order of PP the polynomial P P 0 1 2 2
(Type (Type (Type (Type
1) 2) 3) 4)
1
2
3
Constant Linear
Constant Bilinear Biquadratic-1 Biquadratic-2
Constant Trilinear Triquadratic-1 Triquadratic-2
Quadratic
1: Basic type, 2: Modified type
where alk is a vector of the parameters of the conclusion part of the rule while Plk (xi , xj , alk ) denotes the regression polynomial forming the consequence part of the fuzzy rule which uses several types of high-order polynomials (linear, quadratic, and modified quadratic) besides the constant function forming the simplest version of the consequence; refer to Table 2. The types of the polynomial read as follows • • • • • •
Bilinear = c0 + c1 x1 + c2 x2 Biquadratic-1 = Bilinear + c3 x21 + c4 x22 + c5 x1 x2 Biquadratic-2 = Bilinear + c3 x1 x2 Trilinear = c0 + c1 x1 + c2 x2 + c3 x3 Triquadratic-1 = Trilinear + c4 x21 + c5 x22 + c6 x23 + c7 x1 x2 + c8 x1 x3 + c9 x2 x3 Triquadratic-2 = Trilinear + c4 x1 x2 + c5 x1 x3 + c6 x2 x3
Alluding to the input variables of the FPN, especially a way in which they interact with the two functional blocks shown there, we use the notation FPN (xp , xq ; xi , xj ) to explicitly point at the variables. The processing of the FPN is governed by the following expressions that are in line of the rule-based computing existing in the literatures [15, 16]. The activation of the rule “K ” is computed as an and-combination of the activations of the fuzzy sets occurring in the rule. This combination of the subconditions is realized through any t-norm. In particular, we consider the minimum and product operations as two widely used models of the logic connectives. Subsequently, denote the resulting activation level of the rule by µK . The activation levels of the rules contribute to the output of the FPN being computed as a weighted average of the individual condition parts (functional transformations) PK (note that the index of the rule, namely “K” is a shorthand notation for the two indexes of fuzzy sets used in the rule (4), that is K = (l, k)). all rules all rules all rules z= µK PK (xi , xj , aK ) µK = µ K PK (xi , xj , ak ) (5) K=1
K=1
K=1
66
S.-K. Oh and W. Pedrycz
In the above expression, we use an abbreviated notation to describe an activation level of the “K”th rule to be in the form µK µ K = all rules (6) µL L=l
2.2.2 The topology of the fuzzy polynomial neuron (FPN) based SONN The topology of the FPN based SONN implies the ensuing learning mechanisms; in the description below we indicate some of these learning issues that permeate the overall architecture. First, the network is homogeneous in the sense it is constructed with the use of the FPNs. It is also heterogeneous in the sense that FPNs can be very different (as far as the detailed architecture is concerned) and this contributes to the generality of the architecture. The network may contain a number of hidden layers each of them of a different size (number of nodes). The nodes may have a different number of inputs and this triggers a certain pattern of connectivity of the network. The FPN itself promotes a number of interesting design options, see Fig. 3. These alternatives distinguish between two categories such as designer-based and GA-based. The former concerns a choice of the membership function (MF) type, the consequent input structure of the fuzzy rules, and the number of MFs per each input variable. The latter is related to a choice of the number of inputs, and a collection of the specific subset of input variables and its associated order of the polynomial realizing a consequence part of the rules based on fuzzy inference method. Proceeding with the FPN-based SONN architecture, see Fig. 4, essential design decisions have to be made with regard to the number of input variables
Fig. 3. The design alternatives available within a single FPN
Genetically Optimized Self-organizing Neural Networks
67
Fig. 4. Configuration of the topology of the FPN-based SONN Table 3. Polynomial type according to the number of input variables in the conclusion part of fuzzy rules
H Input HH vector Selected input HH Type of variables in the HH the consequence premise part HH polynomial H H Type T Type T*
A A
Selected input variables in the consequence part
Entire system input variables
A B
B B
and the order of the polynomial forming the conclusion part of the rules as well as a collection of the specific subset of input variables. The consequence part can be expressed by linear, quadratic, or modified quadratic polynomial equation as mentioned previously. Especially for the consequence part, we consider two kinds of input vector formats in the conclusion part of the fuzzy rules of the 1st layer, namely i) selected inputs and ii) entire system inputs, see Table 3. i) The input variables of the consequence part of the fuzzy rules are same as the input variables of premise part. ii) The input variables of the consequence part of the fuzzy rules in a node of the 1st layer are same as the entire system input variables and the input variables of the consequence part of the fuzzy rules in a node of the 2nd layer or higher are same as the input variables of premise part. Where notation A: Vector of the selected input variables (x1 , x2 , · · · , xi ), B: Vector of the entire system input variables (x1 , x2 , · · · , xi , xj · · · ), Type T:f (A) = f (x1 , x2 , · · · , xi ) - type of a polynomial function standing in the consequence part of the fuzzy rules, Type T*: f (B) = f (x1 , x2 , · · · , xi , xj · · · ) - type of a polynomial function occurring in the consequence part of the fuzzy rules.
68
S.-K. Oh and W. Pedrycz
As shown in Table 3, A and B describe vectors of the selected input variables and the entire collection (set) of the input variables, respectively. Proceeding with each layer of the SONN, the design alternatives available within a single FPN can be carried out with regard to the entire collection of the input variables or its selected subset as they occur in the consequence part of fuzzy rules encountered at the 1st layer. Following these criteria, we distinguish between two fundamental types (Type T, Type T*), namely Type T- the input variables in the conditional part of fuzzy rules are kept as those in the conclusion part of the fuzzy rules (Zi of (A)F P N (xp , xq ; xi , xj )) Type T*- the entire collection of the input variables is kept as input variables in the conclusion part of the fuzzy rules (Zi of (B)F P N (xp , xq ; x1 , x2 , x3 , x4 )). In the Fuzzy Polynomial Neuron (FPN) shown in Fig. 4, the variables in the FPN(•) are enumerated in the form of two lists that are separated by a semicolon. The former and latter part denote the premise input variables (xp , xq ) and the input variables (xp , xq or x1 , x2 , x3 , x4 ) of the consequence regression polynomial of the fuzzy rules respectively. In other words, xp and xq of both the former and latter part stand for the selected input variables to be used in both the premise and consequence part of the fuzzy rules, and x1 , x2 , x3 , and x4 of the latter part stand for system input variables to be used in the consequence polynomial of the fuzzy rules.
3 Genetic optimization of SONN The task of optimizing any complex model involves two main phases. First, a class of some optimization algorithms has to be chosen so that it is applicable to the requirements implied by the problem at hand. Secondly, various parameters of the optimization algorithm need to be tuned in order to achieve its best performance. Genetic algorithms (GAs) are optimization techniques based on the principles of natural evolution. In essence, they are search algorithms that use operations found in natural genetics to guide a comprehensive search over the parameter space. GAs have been theoretically and empirically demonstrated to provide robust search capabilities in complex spaces thus offering a valid solution strategy to problems requiring efficient and effective searching. For the optimization applied to real world problems, many methods are available including gradient-based search and direct search, which are based on techniques of mathematical programming [17]. In contrast to these, genetic algorithm are aimed at stochastic global search and involving a structured information exchange [18]. It is eventually instructive to highlight the main features that tell GA apart from some other optimization methods: (1) GA operates on the codes of the variables, but not the variables themselves. (2) GA searches optimal points starting from a group (population) of points in the search space (potential solutions), rather than a single point. (3) GA’s search
Genetically Optimized Self-organizing Neural Networks
69
is directed only by some fitness function whose form could be quite complex; we do not require it need to be differentiable. In this study, for the optimization of the SONN model, GA uses the serial method of binary type, roulette-wheel used in the selection process, one-point crossover in the crossover operation, and a binary inversion (complementation) operation in the mutation operator. To retain the best individual and carry it over to the next generation, we use elitist strategy [19]. The overall geneticallydriven structural optimization process of SONN is shown in Figs. 5–6. As mentioned, when we construct PNs or FPNs of each layer in the conventional SONN, such parameters as the number of input variables (nodes),
Fig. 5. Overall genetically-driven structural optimization process of the PN-based SONN
Fig. 6. Overall genetically-driven structural optimization process of FPN-based SONN
70
S.-K. Oh and W. Pedrycz
Fig. 7. A general flow of genetic design of SONNs
the order of polynomial, and input variables available within a PN or a FPN are fixed (selected) in advance by the designer. This could have frequently contributed to the difficulties in the design of the optimal network. To overcome this apparent drawback, we resort ourselves to the genetic optimization, see Fig. 7 for more detailed flow of the development activities.
4 The algorithm and design procedure of genetically optimized SONN (gSONN) The genetically-driven SONN comes with a highly versatile architecture both in the flexibility of the individual nodes as well as the interconnectivity between the nodes and organization of the layers. Evidently, these features contribute to the significant flexibility of the networks yet require a prudent design methodology and a well-thought learning mechanisms based on genetic optimization. Let us stress that there are several important differences that make this architecture distinct from the panoply of the well-known neurofuzzy architectures existing in the literature. The most important is that the
Genetically Optimized Self-organizing Neural Networks
71
learning of SONNs dwells on the extended GMDH algorithm that is crucial to the structural development of the network. In the sequel, The GMDH method is comprised of both a structural phase such as a self-organizing and an evolutionary algorithm (rooted in natural law of survival of the fittest), and a parametric phase of Least Square Estimation (LSE)-based learning. Therefore the structural and parametric optimization help utilize hybrid method (combining GAs with a structural phase of GMDH) and LSE-based technique in the most efficient way. Overall, the framework of the design procedure of the SONN based on genetically optimized multi-layer perceptron architecture comprises the following steps. [Step 1] Determine system’s input variables. [Step 2] Using available experimental data, form a training and testing data set. [Step 3] Decide initial information for constructing the SONN structure. [Step 4] Decide a structure of the PN or FPN based SONN using genetic design. • Selection of the number of input variables • Selection of the polynomial order (PN) or the polynomial order of the consequent part of fuzzy rules (FPN) • Selection of a collection of the specific subset of input variables [Step 5] Estimate the coefficient parameters of the polynomial in the selected node. • In case of a PN – Estimate the coefficients of the polynomial assigned to the selected node (PN) • In case of a FPN – Carry out fuzzy inference and coefficients estimation for fuzzy identification in the selected node (FPN) [Step 6] Select nodes (PNs or FPNs) with the best predictive capability and construct their corresponding layer. [Step 7] Check the termination criterion. [Step 8] Determine new input variables for the next layer. In what follows, we describe each of these steps in more detail. [Step 1] Determine system’s input variables. Define system’s input variables xi (i = 1, 2, · · · , n) related to the output variable y. If required, the normalization of input data is carried out as well. [Step 2] Form a training and testing data. The input-output data set (xi , yi ) = (x1i , x2i , · · · , xni , yi ), i = 1, 2, . . . , N (with N being the total number of data points) is divided into two parts, that is, a training and testing dataset. Denote their sizes by Nt and Nc respectively. Obviously we have N = Nt + Nc . The training data set is used to construct
72
S.-K. Oh and W. Pedrycz
the SONN. Next, the testing data set is used to evaluate the quality of the network. [Step 3] Decide initial information for constructing the SONN structure. We decide upon the design parameters of the SONN structure and they include: a) According to the stopping criterion, two termination methods are exploited: – Criterion level for comparison of a minimal identification error of the current layer with that occurring at the previous layer of the network. – The maximum number of layers (predetermined by the designer) with an intent to achieve a sound balance between model accuracy and its complexity. b) The maximum number of input variables coming to each node in the corresponding layer. c) The total number W of nodes to be retained (selected) at the next generation of the SONN algorithm. d) The depth of the SONN to be selected to reduce a conflict between overfitting and generalization abilities of the developed SONN. e) The depth and width of the SONN to be selected as a result of a tradeoff between accuracy and complexity of the overall model. In addition, in case of FPN-based SONN, parameters related to the following item are considered besides what are mentioned above. f) The decision of initial information for fuzzy inference method and fuzzy identification: – Fuzzy inference method – MF type: Triangular or Gaussian-like MF – No. of MFs per each input of a node (or FPN) – Structure of the consequence part of fuzzy rules [Step 4] Decide a structure of the PN or FPN based SONN using genetic design. This concerns the selection of the number of input variables, the polynomial order, and the input variables to be assigned in each node of the corresponding layer. These important decisions are carried out through an extensive genetic optimization. When it comes to the organization of the chromosome representing (mapping) the structure of the SONN, we divide the chromosome to be used for genetic optimization into three sub-chromosomes as shown in Figs. 8–9. The 1st sub-chromosome contains the number of input variables, the 2nd subchromosome involves the order of the polynomial of the node, and the 3rd sub-chromosome (remaining bits) contains input variables coming to the corresponding node (PN or FPN). All these elements are optimized when running the GA. In nodes (PN or FPNs) of each layer of SONN, we adhere to the notation of Fig. 10. ‘PNn’ or ‘FPNn’ denotes the nth node (PN or FPN) of the
Genetically Optimized Self-organizing Neural Networks
73
Fig. 8. The PN design used in the SONN architecture - structural considerations and mapping the structure on a chromosome
Fig. 9. The FPN design used in the SONN architecture - structural considerations and mapping the structure on a chromosome
74
S.-K. Oh and W. Pedrycz n th Polynomial or Fuzzy Polynomial Neuron(PN or FPN) xi
PN n or FPNn N T
xj No. of inputs
z
Polynomial order(Type T)
Fig. 10. Formation of each PN or FPN in SONN architecture
corresponding layer, ‘N’ denotes the number of nodes (inputs or PNs/FPNs) coming to the corresponding node, and ‘T’ denotes the order of polynomial used in the corresponding node. Each sub-step of the genetic design of the three types of the parameters available within the PN or the FPN is structured as follows: [Step 4-1] Selection of the number of input variables (1st sub-chromosome) Sub-step 1) The first 3 bits of the given chromosome are assigned to the binary bits for the selection of the number of input variables. The size of this bit structure depends on the number of input variables. For example, in case of 3 bits, the maximum number of input variables is limited to 7. Sub-step 2) The 3 bits randomly selected as β = (22 × bit(3)) + (21 × bit(2)) + (20 × bit(1))
(7)
are then decoded into decimal. Here, bit(1), bit(2) and bit(3) show each location of these three bits and are denoted as “0”, or “1” respectively. Sub-step 3) The above decimal value is rounded off γ = (β/α) × (M ax − 1) + 1
(8)
where Max denotes the maximal number of input variables entering the corresponding node (PN or FPN) while α is the decoded decimal value corresponding to the situation when all bits of the 1st sub-chromosome are set up as 1’s. Sub-step 4) The normalized integer value is then treated as the number of input variables (or input nodes) coming to the corresponding node. Evidently, the maximal number (Max) of input variables is equal to or less than the number of all system’s input variables (x1 , x2 , · · · , xn ) coming to the 1st layer, that is, Max ≤ n. [Step 4-2] Selection of the order of polynomial (2nd sub-chromosome) Sub-step 1) The 3 bits of the 2nd sub-chromosome are assigned to the binary bits for the selection of the order of polynomial. Sub-step 2) The 3 bits randomly selected using (7) are decoded into a decimal format. Sub-step 3) The decimal value obtained by means of (8) is normalized and rounded off. The value of Max is replaced with 3 (in case of PN) or 4 (in case of FPN), refer to Tables 1and 2.
Genetically Optimized Self-organizing Neural Networks
75
Sub-step 4) The normalized integer value is given as the selected polynomial order, when constructing each node of the corresponding layer. [Step 4-3] Selection of input variables (3rd sub-chromosome) Sub-step 1) The remaining bits are assigned to the binary bits for the selection of input variables. Sub-step 2) The remaining bits are divided by the value obtained in step 4-1. If these bits are not divisible, we apply the following rule. For example, if the remaining are 22 bits and the number of input variables obtained in step 4-1 has been set up as 4, the 1st , 2nd , and 3rd bit structures (spaces) for the selection of input variables are assigned to 6 bits, respectively. The last (4th ) bit structure (spaces) used for the selection of the input variables is assigned to 4 bits. Sub-step 3) Each bit structure is decoded into decimal (through relationship (7)) Sub-step 4) Each decimal value obtained in sub-step 3 is then normalized following (8); moreover we round off the values obtained from this expression. We replace Max with the total number of inputs (viz. input variables or input nodes), n (or W ) in the corresponding layer. Note that the total number of input variables denotes the number of the overall system’s inputs, n, in the 1st layer, and the number of the selected nodes, W , as the output nodes of the preceding layer in the 2nd layer or higher. Sub-step 5) The normalized integer values are then taken as the selected input variables while constructing each node of the corresponding layer. Here, if the selected input variables are multiple-duplicated, the multiple-duplicated input variables (viz. same input numbers) are treated as a single input variable while the remaining ones are discarded. [Step 5] Estimate the coefficient parameters of the polynomial in the selected node (PN or FPN). [Step 5-1] In case of a PN The vector of coefficients Ci is derived by minimizing the mean squared error between yi and zmi Ntr 1 E= (yi − zm i)2 (9) Ntr i=0 Using the training data subset, this gives rise to the set of linear equations Y = Xi Ci
(10)
Evidently, the coefficients of the PN of nodes in each layer are expressed in the form (11) Ci = (XiT Xi )−1 XiT Y where Y = [y1 y2 · · · yntr ]T , Xi = [X1i X2i · · · Xki · · · Xntr i ]T , T m m m T = [1 xki1 xki2 · · · xkin · · · xm Xki ki1 xki1 xki2 · · · xkin ], Ci = [c0i c1i c2i · · · cn t ]
76
S.-K. Oh and W. Pedrycz µ (x )
Small( µ S (x ))
Big(µB ( x ) = 1 − µ S ( x ))
1 µ S (x )
0.5 0
x min
x
x max
0
if x ≥ x max
x max − x x max − x min
if x min < x < x max
1
if x ≤ x min
(a) Triangular MF µ (x )
Small( µ S (x))
Big(µB ( x) = 1 − µ S ( x))
1
if c ≥ x max
0 2
µ S (x)
0.5 0
x min
x max
x
−0.6931
µ ( x) = e
1
( x −c ) σ
if x min < c < x max if c ≤ x min
(b) Gaussian-like MF Fig. 11. Triangular and Gaussian-like membership functions and their parameters
with the following notation i: node number, k: data number, ntr : number of the training data subset, n: number of the selected input variables, m: maximum order, n : number of estimated coefficients. [Step 5-2] In case of a PN At this step, the regression polynomial inference is considered. The inference method deals with regression polynomial functions viewed as the consequents of the rules. Regression polynomials (polynomial and in the very specific case, a constant value) standing in the conclusion part of fuzzy rules are given as different types of Type 1, 2, 3, or 4, see Table 2. In the fuzzy inference, we consider two types of membership functions, namely triangular and Gaussianlike membership functions. The regression fuzzy inference (reasoning scheme) is envisioned: The consequence part can be expressed by linear, quadratic, or modified quadratic polynomial equation as shown in Table 2. The use of the regression polynomial inference method gives rise to the expression Ri : If x1 is Ai1 , · · · , xk is Aik then yi = fi (x1 , x2 , · · · , xk )
(12)
Genetically Optimized Self-organizing Neural Networks
77
where, Ri is the i-th fuzzy rule, xl (l = 1, 2, k) is an input variable, Aik is a membership function of fuzzy sets, k denotes the number of the input variables, fi (•) is a regression polynomial function of the input variables. The calculation of the numeric output of the model are carried out in the well-known form n
y =
µi fi (x1 , x2 , · · ·, xk )
i=1
n
= µi
n
µ i fi (x1 , x2 , · · ·, xk )
(13)
i=1
i=1
where, n is the number of the fuzzy rules, y is the inferred value, µi is the premise fitness of Ri and µ i is the normalized premise fitness of µi . Here we consider a regression polynomial function of the input variables. The consequence parameters are produced by the standard least squares method that is = (XjT Xj )−1 XjT Y a (14) where aTj = [a10j , · · ·, anoj , a11j , · · ·, an1j , · · ·, a1kj , · · ·, ankj ], Xj = [x1j , x2j , · · ·, xij , · · ·, xmj ]T , µlj1 , · · ·, µ ljn , xlj1 µ lj1 , · · ·, xlj1 µ ljn , · · ·, xljk µ lj1 , · · ·, xljk µ ljn ], xTlj = [ Y = [y1 , · · ·, ym ]T j is the node number, m denotes the total number of data points, and k stands the number of the input variables. Subsequently l is the data index while n is the number of the fuzzy rules. The procedure described above is implemented iteratively for all nodes of the layer and also for all layers of SONN; we start from the input layer and move towards the output layer. [Step 6] Select nodes (PNs or FPNs) with the best predictive capability and construct their corresponding layer. Fig. 12 depicts the genetic optimization procedure for the generation of the optimal nodes in the corresponding layer. As shown in Fig. 12, all nodes of the corresponding layer of SONN architecture are constructed through the genetic optimization. The generation process can be organized as the following sequence of steps Sub-step 1) We set up initial genetic information necessary for generation of the SONN architecture. This concerns the number of generations and populations, mutation rate, crossover rate, and the length of the chromosome. Sub-step 2) The nodes (PNs or FPNs) are generated through the genetic design. Here, a single population assumes the same role as the node (PN or FPN) in the SONN architecture and its underlying processing is visualized
78
Generation 1
S.-K. Oh and W. Pedrycz Population (1~T)
Array of nodes (PNs or FPNs:1~T)
Fitness operation of nodes (PNs or FPNs)
Reproduction Roulette-wheel selection One-point crossover Invert mutaion
Generation 2
Population (1~T)
Fitness operation of nodes (PNs or FPNs)
Elitist strategy
Selection of the optimal node
Selected Population Array of nodes
Highest fitness value
(preferred nodes:
(PNs or FPNs:1~2W)
PNs orFPNs) (1~W)
Array of nodes (PN or FPNs:1~T)
Reproduction Roulette-wheel selection One-point crossover Invert mutaion
Selected Population (preferred nodes: PNs orFPNs) (1~W)
Selected Population (preferred nodes: PNs orFPNs) (1~W)
Selection of the optimal node Highest fitness value
Selected Population Array of nodes
(preferred nodes:
(PNs or FPNs:1~2W)
PNs orFPNs) (1~W)
T : Total population size Generation 3
Population (1~T)
Fitness operation of nodes (PNs or FPNs)
Elitist strategy
Array of nodes (PNs or FPNs:1~T)
Selected Population (preferred nodes: PNs orFPNs) (1~W)
W : Selected population size : GA operation flow
: Optimization operation of SOFPNN within GA flow
Fig. 12. The genetic optimization used in the generation of the optimal nodes in the given layer of the network
in Figs. 8–9. The optimal parameters of the corresponding polynomial are computed by the standard least squares method. Sub-step 3) To evaluate the performance of nodes (PNs or FPNs) constructed using the training dataset, the testing dataset is used. Based on this performance index, we calculate the fitness function. The fitness function reads as F(fitness function) =
1 1 + EPI
(15)
where EPI denotes the performance index for the testing data (or validation data). In this case, the model is obtained by the training data and EPI is obtained from the testing data (or validation data) of the SONN model constructed by the training data. Sub-step 4) To move on to the next generation, we carry out selection, crossover, and mutation operation using genetic initial information and the fitness values obtained via sub-step 3. Sub-step 5) The nodes (PNs or FPNs) obtained on the basis of the calculated fitness values (F1 , F2 ,· · · , Fz ) are rearranged in a descending order. We unify the nodes with duplicated fitness values (viz. in case that one node is the same fitness value as other nodes) among the rearranged nodes on the basis of the fitness values. We choose several nodes (PNs or FPNs) characterized by the best fitness values. Here, we use the pre-defined number W of nodes (PNs or FPNs) with better predictive capability that need to be preserved to assure an optimal operation at the next iteration of the SONN algorithm. The outputs
Genetically Optimized Self-organizing Neural Networks
79
of the retained nodes serve as inputs to the next layer of the network. There are two cases as to the number of the retained nodes, that is (i) If W * < W , then the number of the nodes (PNs or FPNs) retained for the next layer is equal to z. Here, W * denotes the number of the retained nodes in each layer that nodes with the duplicated fitness values were moved. (ii) If W * ≥ W , then for the next layer, the number of the retained nodes (PNs or FPNs) is equal to W . The above design pattern is carried out for the successive layers of the network. For the construction of the nodes in the corresponding layer of the original SONN structure, the nodes obtained from (i) or (ii) are rearranged in ascending order on a basis of initial population number. This step is needed to construct the final network architecture as we trace the location of the original nodes of each layer generated by the genetic optimization. Sub-step 6) For the elitist strategy, we select the node that has the highest fitness value among the selected nodes (W ). Sub-step 7) We generate new populations of the next generation using operators of GAs obtained from Sub-step 4. We use the elitist strategy. This sub-step carries out by repeating sub-step 2–6. Especially in sub-step 5, we replace the node that has the lowest fitness value in the current generation with the node that has reached the highest fitness value in the previous generation obtained from sub-step 6. Sub-step 8) We combine the nodes (W populations) obtained in the previous generation with the nodes (W populations) obtained in the current generation. In the sequel, W nodes that have higher fitness values among them (2W ) are selected. That is, this sub-step carries out by repeating sub-step 5. Sub-step 9) Until the last generation, this sub-step carries out by repeating sub-step 7–8. The iterative process generates the optimal nodes of the given layer of the SONN. Step 7) Check the termination criterion. The termination condition that controls the growth of the model consists of two components, that is the performance index and a size of the network (expressed in terms of the maximal number of the layers). As far as the performance index is concerned (that reflects a numeric accuracy of the layers), a termination is straightforward and comes in the form, F1 ≤ F∗
(16)
Where, F1 denotes a maximal fitness value occurring at the current layer whereas F∗ stands for a maximal fitness value that occurred at the previous layer. As far as the depth of the network is concerned, the generation process is stopped at a depth of less than five layers. This size of the network has been experimentally found to build a sound compromise between the high accuracy of the resulting model and its complexity as well as generalization abilities.
80
S.-K. Oh and W. Pedrycz
In this study, we use two measures (performance indexes) that is the Mean Squared Error (MSE) and the Root Mean Squared Error (RMSE). i) Mean Squared Error E(P Is or EP Is ) =
N 1 (yp − yp )2 N p=1
(17)
ii) Root Mean Squared Error N 1 E(P Is or EP Is ) = (yp − yp )2 N p=1
(18)
where, yp is the p − th target output data and yp stands for the p-th actual output of the model for this specific data point. N is training (P Is ) or testing (EP Is ) input-output data pairs and E is an overall (global) performance index defined as a sum of the errors for the N. [Step 8] Determine new input variables for the next layer. If (16) has not been met, the model is expanded. The outputs of the preserved nodes (zli , z2i , · · ·, zW i ) serves as new inputs to the next layer (x1j , x2j , · · ·, xW j )(j = i + 1). This is captured by the expression x1j = z1i , x2j = z2i , · · · , xwj = zwi
(19)
The SONN algorithm is carried out by repeating steps 4–8 of the algorithm.
5 Experimental studies In this section, we illustrate the development of the SONN and show its performance for a number of well-known and widely used datasets. The first one is a time series of gas furnace (Box-Jenkins data) which was studied previously in [10, 11, 13, 15, 16, 20–34]. The other one deals with a chaotic time series data (Mackey-Glass time series data) [35–41]. 5.1 Gas furnace process We illustrate the performance of the network and elaborate on its development by experimenting with data coming from the gas furnace process. The time series data (296 input-output pairs) resulting from the gas furnace process has been intensively studied in the previous literature [10, 11, 13, 15, 16, 20–34]. The delayed terms of methane gas flow rate, u(t) and carbon dioxide density, y(t) are used as system input variables such as u(t − 3), u(t − 2), u(t − 1), y(t − 3), y(t − 2), and y(t − 1). The output variable is y(t). The first part of the dataset (consisting of 148 pairs) was used for training. The remaining part
Genetically Optimized Self-organizing Neural Networks
81
Table 4. System’s input vector formats for the design of SONN Number of System Inputs (SI) 2 (Type I) 3 (Type II) 4 (Type III) 6 (Type IV)
Inputs and output u(t−3), y(t−1):y(t) u(t−2), y(t−2), y(t−1):y(t) u(t−2), u(t−1), y(t−2), y(t−1):y(t) u(t−3), u(t−2), u(t−1), y(t−3), y(t−2), y(t−1):y(t)
of the series serves as a testing set. We consider the MSE given by (17) to be a pertinent performance index. We choose the input variables of nodes in the 1st layer of SONN architecture from these input variables. We use four types of system input variables of SONN architecture with vector formats such as Type I, Type II, Type III, and Type IV as shown in Table 4. The forms Type I, II, III, and IV utilize two, three, four, and six system input variables respectively. Table 5 summarizes the list of parameters used in the genetic optimization of the PN-based and the FPN-based SONN. In the optimization of each layer, we use 100 generations, 60 populations, a string of 36 bits, crossover rate equal to 0.65, and the probability of mutation set up to 0.1. A chromosome used in the genetic optimization consists of a string including 3 sub-chromosomes. The 1st chromosome contains the number of input variables, the 2nd chromosome contains the order of the polynomial, and finally the 3rd chromosome contains input variables. The numbers of bits allocated to each sub- chromosome are equal to 3, 3, and 30, respectively. The population size being selected from the total population size (60) is equal to 30. The process is realized as follows. 60 nodes (PNs or FPNs) are generated in each layer of the network. The parameters of all nodes generated in each layer are estimated and the network is evaluated using both the training and testing data sets. Then we compare these values and choose 30 nodes (PNs or FPNs) that produce the best (lowest) value of the performance index. The maximal number (Max) of inputs to be selected is confined to two to five (2–5). In case of PN-based SONN, the order of the polynomial is chosen from three types that is Type 1, Type 2, and Type 3 (refer to the Table 1), while in case of FPN-based SONN, the polynomial order of the consequent part of fuzzy rules is chosen from four types, that is Type 1, Type 2, Type 3, and Type 4 as shown in Table 2. As usual in fuzzy systems, we may exploit a variety of membership functions in the condition part of the rules and this is another factor contributing to the flexibility of the network. Overall, triangular or Gaussian fuzzy sets are of general interest. The first class of triangular membership functions provides with a very simple implementation. The second class becomes useful because of an infinite support of its fuzzy sets.
82
S.-K. Oh and W. Pedrycz
Table 5. Computational aspects of the genetic optimization of PN-based and FPNbased SONN
GA
Parameters
1st layer
2nd layer
3rd layer
4th layer
5th layer
Maximum generation
100
100
100
100
100
Total population size
60
60
60
60
60
Selected population size (W )
30
30
30
30
30
Crossover rate
0.65
0.65
0.65
0.65
0.65
Mutation rate
0.1
0.1
0.1
0.1
0.1
String length
3 + 3 + 30
3 + 3 + 30
3 + 3 + 30
3 + 3 + 30
3 + 3 + 30
Maximal no. (Max) of inputs 1 ≤ l ≤ Max 1 ≤ l ≤ Max 1 ≤ l ≤ Max 1 ≤ l ≤ Max 1 ≤ l ≤ Max (2–5) (2–5) (2–5) (2–5) (2–5) to be selected
PNbased SONN Polynomial type (Type T) (# )
1≤T≤3
1≤T≤3
1≤T≤3
1≤T≤3
1≤T≤3
Maximal no. (Max) of inputs 1 ≤ l ≤ Max 1 ≤ l ≤ Max 1 ≤ l ≤ Max 1 ≤ l ≤ Max 1 ≤ l ≤ Max (2–5) (2–5) (2–5) (2–5) (2–5) to be selected Polynomial type (Type T) of the consequent part of fuzzy rules (## ) FPNConsequent based input type to be SONN used for Type T (### )
Membership Function (MF) type
1≤T≤4
1≤T≤4
1≤T≤4
1≤T≤4
1≤T≤4
Type T
Type T
Type T
Type T
Type T
Type T∗
Type T
Type T
Type T
Type T
Triangular
Triangular
Triangular
Triangular
Triangular
Gaussian
Gaussian
Gaussian
Gaussian
Gaussian
2
2
2
2
2
No. of MFs per input l, T, Max: integers,
#
,
##
and
###
: refer to Tables 1–3 respectively.
5.1.1 PN-based SONN Fig. 13 shows an example of the PN design that is driven by some specific chromosome, refer to the case that the performance values are PI = 0.022, EPI = 0.136 in the 1st layer when using Max = 3 in Type IV as shown in Table 6(b). Here, the number of the input variables considered in the 1st layer is given as 6 that are the number of the entire system input variables. Especially, in the 2nd layer or higher, the number of entire input variables is given as W that is the number of the nodes selected in the current layer, as the output nodes of the preceding layer. Refer to sub-step 4 of step 4-3 of the introduced
Genetically Optimized Self-organizing Neural Networks
83
Selection of node(PN) structure by a chromosome
i) Bits for the selection of the no. of input variables
Related bit items
Bit structure of subchromosome divided for each item
1
1
1
1
1
0
Decoding
Decoding
7
6
Normalization
Normalization
3
3
1
1
1
1
0
0
0
0
1
No. of selected input variables(3)
Selected polynomial order (Type 3)
1
0
0
0
1
Normalization : 5
Selected input variables : 5
1
0
1
1
1
1
1
0
1
0
2
3
Decoding : 7
Decoding : 60
Normalization : 2
Normalization : 6
Selected input variables : 2
Selected input variables : 6
1 Decoding : 50
Genetic Design
iii) Bits for the selection of input variables
ii) Bits for the selection of the polynomial order
0
0
Selected input variables : 2, 5, 6
3 input variables : x 2 , x 5 , x 6
Selected PN
Modified quadratic polynomial : a0 +
a1x2 + a2x5 + a3x6 + a4x2x5 + a5x2x6 + a6x5x6
PN
Fig. 13. The example of the PN design guided by some chromosome (Type IV and Max = 3 in layer 1)
design process. The maximal number of input variables for the selection is confined to 3 over nodes of each layer of the network. Table 6 summarizes the results when using Type II and Type IV: According to the maximal number of inputs to be selected (Max = 2 to 5), the selected node numbers, the selected polynomial type (Type T), and its corresponding performance index (PI and EPI) were shown when the genetic optimization for each layer was carried out. “Node” denotes the nodes for which the fitness value is maximal in each layer. For example, in case of Table 6(b), the fitness value in layer 1 is maximal for Max = 5 when nodes 3, 4, 5, 6 occurring in the previous layer are selected as the node inputs in the present layer. Only 4 inputs of Type 1 (linear function) were selected as the result of the genetic optimization. Here, node “0” indicates that it has not been selected by the genetic operation. Therefore the width (the number of nodes) of the layer can be lower in comparison to the conventional PN-based SONN (which immensely contributes to the compactness of the resulting network). In that case, the minimal value of the performance index at the node, that is PI = 0.035, EPI = 0.125 are obtained. Fig. 14 depicts the values of the performance index of the PN-based gSONN with respect to the maximal number of inputs to be selected when
6
1 0.035 0.125 6
21
28
EPI 8
3 0.042 0.112 1
2 0.043 0.115 12
EPI
1 0.021 0.123 4
2 0.026 0.134
PI 2 22
PI
EPI 23
23 3 0.041 0.112 14
2 0.024 0.114
T
18
0
21
30
Node
8 25 29 3 0.039 0.108 5 15 16
16
16
Node
PI
EPI
1 0.038 0.106
2 0.036 0.109
2 0.024 0.112
T
5th layer
EPI 8
3 0.017 0.113 4
2 0.020 0.112 12
1 0.023 0.126
PI
0
19
PI
EPI
2 0.014 0.110 4
2 0.022 0.124
T
4th layer
7 6
Node
7 10 28 2 0.016 0.106 27 28 29
13
Node
0
18
9
PI
EPI
3 0.014 0.100
2 0.014 0.102
2 0.022 0.120
T
5th layer
9 18 27 0 3 0.015 0.107 3 23 30 0 0 2 0.013 0.096 6 24 25 0 0 3 0.012 0.091
19
28
T
3rd layer
25
Node
(b) In case of Type IV
9 15 27 3 0.018 0.122 6 12 13
0
27
T
3 4 5 6 0 1 0.035 0.125 4 15 22 0 0 2 0.015 0.112 7
25
Node
2nd layer
5
24
2 0.105 0.199 24
2 0.105 0.199
EPI
18
2 0.045 0.121 8 13 14
PI
2 0.024 0.119
T
4th layer
7 18 0 0 2 0.045 0.121 1 15 30 0 0 2 0.022 0.115 23 29 30 0 0 2 0.021 0.110 13 14 23 30 0 1 0.019 0.108
0
14 2 0.045 0.121 16
12
Node
3rd layer
(a) In case of Type II
10
3 4 5
6
6
PI
EPI
4
5
T
1st layer
PI
2 0.048 0.124
T
2
5
Node
7
10
Node
2nd layer
3
2
Max
1 2 3 0 0 3 0.022 0.135 4
6
3 0.022 0.135 5 15 16
3 0.022 0.135 5
5
0
1 2 3
3
EPI
1
2
PI
2 0.104 0.198
T
4
3
Node
2
3
2
Max
1st layer
Table 6. Performance index of the PN-based SONN viewed with regard to the increasing number of the layers
84 S.-K. Oh and W. Pedrycz
Genetically Optimized Self-organizing Neural Networks Maximal number of inputs to be selected(Max) 2(A) ; , 3(B) ; , 4(C) ; ,5 (D) ;
Maximal number of inputs to be selected(Max) 2(A) ; , 3(B) ; , 4(C) ; ,5 (D) ;
0.1 0.09
Training error
0.2
A : (2 3) B : (1 2 3) C : (1 2 3 0) D : (1 2 3 0 0) A : (6 10) B : (5 7 14) C : (5 15 16 0) D : (4 7 18 0 0)
0.08 0.07
0.19 0.18 A : ( 8 16) B : (12 16 23) C : ( 1 8 25 29) D : (23 29 30 0 0)
A : (10 12) B : (16 18 28) C : ( 8 13 14 21) D : ( 1 15 30 0 0)
0.06 0.05
Testing error
0.11
A : (23 30) B : (14 18 21) C : (10 16 28 0) D : (13 14 23 30 0)
0.04
0.17 0.16 0.15 0.14 0.13 0.12
0.03
0.11
0.02 0.01
1
2
3
4
0.1
5
1
2
Layer
Maximal number of inputs to be selected(Max) 2(A) ; , 3(B) ; , 4(C) ; ,5 (D) ;
0.11
0.07
0.05 0.04 0.03
(a-2) Testing error
0.18
A : (2 25) B : (4 22 28) C : (6 12 13 19) D : (7 9 18 27 0) A : ( 8 19) B : (12 13 0) A : ( 7 9) C : ( 4 7 10 28) B : ( 4 6 18) D : ( 3 23 30 0 0) C : (27 28 29 0) D : ( 6 24 25 0 0)
0.06
5
Maximal number of inputs to be selected(Max) 2(A) ; , 3(B) ; , 4(C) ; ,5 (D) ;
Testing error
0.08
4
0.2
A : (5 6) B : (2 5 6) C : (3 4 5 6) D : (3 4 5 6 0) A : (24 27) B : (24 25 0) C : ( 6 9 15 27) D : ( 4 15 22 0 0)
0.1 0.09
3
Layer
(a-1) Training error (a) In case of Type II
Training error
85
0.16 0.14 0.12 0.1
0.02 0.01
1
2
3
4
5
0.08
1
Layer
(b-1) Training error (b) In case of Type IV
2
3
4
5
Layer
(b-2) Testing error
Fig. 14. Performance index treated as a function of the maximal number of inputs to be selected in Type II or Type IV
the number of system inputs is equal to 6 (Type IV). Next, Fig. 15 shows the performance index of the gSONN with respect to the number of entire system inputs when using Max = 5. Considering the training and testing data sets in Type IV with Max = 5, the best results for network in the 2nd layer happen with the Type 2 polynomial (quadratic function) and 3 nodes at the inputs (nodes numbered as 4, 15, 22); the performance of this network is quantified by the values of PI equal to 0.015 and EPI given as 0.112. The best results for the network in the 5th layer coming with PI = 0.012 and EPI = 0.091 have been reported when using Max = 5 with the polynomial of Type 3 and 3 nodes at the inputs (the node numbers are 6, 24, 25). In Fig. 14, A(•)–E(•) denote the optimal node numbers at each layer of the network, namely those with the best predictive performance. Here, the node numbers of the 1st layer represent system input numbers, and the node numbers of each layer in the 2nd layer or higher represent the output node numbers of the preceding layer, as the optimal node which has the best output performance in the current layer. Fig. 16 illustrates the detailed optimal topologies of the network with 2 or 3 layers, compared to the conventional optimized network of Fig. 17. That is, in Fig. 17, the
86
S.-K. Oh and W. Pedrycz ,3;
,4;
No. of system inputs : 2 ;
,6; 0.4
0.045
0.35
0.04
Testing error
Training error
No. of system inputs : 2 ; 0.05
0.035 0.03 0.025 0.02
0.2 0.15 0.1 0.05
3
4
5
,6;
0.3
0.01
2
,4;
0.25
0.015 1
,3;
1
2
3
4
5
Layer
Layer
(a) Training error
(b) Testing error
Fig. 15. Performance index regarded as a function of the number of system inputs (SI) (in case of Max = 5) u(t-3)
PN 4
PN4
2 1
y(t-2)
PN 6
PN 1
2 2
2 3
y(t-2)
1 2
3 3
y ^
3 2
PN30
PN 14
y(t-1)
PN 7
PN15
PN 7
4 3
2 3
4
u(t-2) u(t-1)
2 1
PN5
PN15 4
y(t-3) y(t-2)
PN 17
3
3
2
^
y
2
PN22 5
y(t-1)
1
(a) In case of Type II with 3 layers and (b) In case of Type IV with 2 layers and Max = 5 Max = 5 Fig. 16. PN-based genetically optimized SONN (gSONN) architecture PD 11
PD 1
PD 3
PD 2
u(t-2) y(t-2)
PD 4 PD 1 PD 2 PD 1
PD 3 PD 4
y(t-1)
PD 9 PD 11 PD 12
PD 3 PD 12
PD 5
u(t-3) • PD
PD 20
PD 15
^
y u(t-2)
•
u(t-1) •
PD 18 PD 20
y(t-3) •
PD 7 PD 8 PD 9 PD 10 PD 11 PD 12
NOP5
y(t-2) •
NOP6 NOP7 PD 2
PD 9
PD 3
PD 10
NOP25
PD 44
PD 4
(a) In case of Type II with 5 layers
y(t-1) •
PD 13 PD 14 PD 15
PD 8 PD 9 PD 10 PD 12 PD 13 PD 16 PD 20 PD 21 PD 23
PD 2
PD 12
PD 17
PD 7
PD 20 PD 24
PD 9
PD
PD 26 PD 28
PD 30
PD 29
PD 24 PD 25 PD 26
(b) In case of Type IV with 5 layers
Fig. 17. Conventional optimized PN-based SONN architecture
^
f
Genetically Optimized Self-organizing Neural Networks
87
0.17
0.04
0.16
0.035
0.025
1st layer
2nd layer
3rd layer
4th layer
Testing error
Training error
0.15 0.03 5th layer
0.02
0.14 0.13
1st layer
2nd layer
3rd layer
4th layer
5th layer
0.12 0.11
0.015 0.01
0.1 0
100
200
300
Generation
(a) Training error (PI)
400
500
0.09
0
100
200
300
400
500
Generation
(b) Testing error (EPI)
Fig. 18. The optimization process reported in terms of PI and EPI
performance of the conventional optimized SONN in Type II or IV with 5 layers was quantified by the values of PI = 0.020, EPI = 0.119 or PI = 0.017, EPI = 0.101 respectively whereas under the condition given as similar performance values, two types of gSONN architectures were depicted as shown in Fig. 16 (a) Type II with 3 layers and Max = 5 (PI = 0.022, EPI = 0.115) (refer to Table 6(a), PI = 0.019, EPI = 0.108 in 5 layers with Max = 5) and (b) Type IV with 2 layers and Max = 5 (PI = 0.015, EPI = 0.112) (refer to Table 6(b), PI = 0.012, EPI = 0.091 in 5 layers with Max = 5). As shown in Fig. 16, the genetic design procedure at each stage (layer) of SONN leads to the selection of the preferred nodes (or PNs) with optimal local characteristics (such as the number of input variables, the order of the polynomial, and input variables). Fig. 18 illustrates the optimization process by visualizing the performance index in successive generations of the genetic optimization in case of Type IV with Max = 5. It also shows the optimized network architecture over 5 layers. Noticeably, the variation ratio (slope) of the performance of the network changes radically around the 2nd layer from the viewpoint of PI and EPI. Therefore, to effectively reduce a large number of nodes and avoid a substantial amount of time-consuming iterations concerning SONN layers, the stopping criterion can be taken into consideration. Referring to Fig. 16(b), it becomes obvious that we optimized the network up to the maximally 2nd layer. Table 7 summarizes the detailed results of the optimal architecture according to each Type of the network. 5.1.2 FPN-based SONN Fig. 19 shows an example of the FPN design that is driven by some specific chromosome, refer to the case that the performance values are PI = 0.012, EPI = 0.145 in the 1st layer when using Type IV, Gaussian-like MF, and Max = 4 in Table 8(b-2). In FPN of each layer, two Gaussian-like membership
88
S.-K. Oh and W. Pedrycz
Table 7. Performance index of PN based SONN architectures for Types I, II, III, and IV PN-based SONN
Max = 4 Type I (SI: 2) Max = 5
Max = 4 Type II (SI: 3) Max = 5
Max = 4 Type III (SI: 4) Max = 5
Max = 4 Type IV (SI: 6) Max = 5
Layer 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
Optimal Polynomial Neuron No. of inputs (Node no.) Polynomial type 2(1,2) Type 1 4(1,4,5,6) Type 2 4(1,2,14,25) Type 2 3(17,22,25) Type 2 4(4,14,25,26) Type 2 2(1,2) Type 1 4(3,4,5,7) Type 2 5(7,9,15,24,28) Type 3 3(21,27,30) Type 2 3(2,15,13) Type 2 3(1,2,3) Type 3 3(5,15,16) Type 2 4(8,13,14,21) Type 3 4(1,8,25,29) Type 3 3(10,16,28) Type 1 3(1,2,3) Type 3 3(4,17,18) Type 2 3(1,15,30) Type 2 3(23,29,30) Type 2 4(13,14,23,30) Type 1 3(1,3,4) Type 3 4(24,26,29,30) Type 3 4(7,11,18,26) Type 2 3(2,28,29) Type 2 3(3,12,22) Type 1 3(1,3,4) Type 3 5(12,13,15,18,29) Type 2 4(7,9,10,14) Type 2 3(1,3,5) Type 2 4(8,9,23,24) Type 1 4(3,4,5,6) Type 1 4(6,9,15,27) Type 3 4(6,12,13,19) Type 3 4(4,7,10,28) Type 2 3(27,28,29) Type 3 4(3,4,5,6) Type 1 3(4,15,22) Type 2 4(7,9,18,27) Type 3 3(3,23,30) Type 2 3(6,24,25) Type 3
PIs EPIs 0.022 0.019 0.020 0.018 0.018 0.022 0.020 0.020 0.017 0.017 0.022 0.045 0.042 0.039 0.038 0.022 0.045 0.022 0.021 0.019 0.022 0.016 0.014 0.014 0.013 0.022 0.015 0.014 0.014 0.013 0.035 0.018 0.017 0.016 0.014 0.035 0.015 0.015 0.013 0.012
0.335 0.282 0.273 0.268 0.265 0.335 0.282 0.271 0.263 0.259 0.135 0.121 0.112 0.108 0.106 0.135 0.121 0.115 0.110 0.108 0.135 0.124 0.113 0.102 0.099 0.135 0.119 0.104 0.100 0.098 0.125 0.122 0.113 0.106 0.100 0.125 0.112 0.107 0.096 0.091
Genetically Optimized Self-organizing Neural Networks
89
Selection of node(FPN) structure by a chromosome
Related bit items
Bit structure of subchromosome divided for each item
i) Bits for the selection of the no. of input variables
0
1
1
0
1
iii) Bits for the selection of input variables
ii) Bits for the selection of the polynomial order
1
Decoding
Decoding
3
3
0
0
1
1
0
1
0
Normalization
2
2
No. of selected input variables(2)
1
1
0
0
0
Decoding : 11992
Normalization : 2
Normalization : 3
Selected input variable : 2
Selected input variable : 3 Selected input variables : 2, 3
Selected polynomial order (Type 2)
Input variables :x 2,x 3
Regression polynomial fuzzy inference
Fuzzy inference & fuzzy identification
0
2
1
Normalization
1
0 0 1 1 0 1 1 0 1 1 1 0 0 0 0 0 1 0 1 1 1 0 1 1 0 1 1 0 0 0
Decoding : 7024
Genetic Design
1
Gaussian
2 MFs
Entire system input variables
R 1 : If x 2 is Small and x3 is Small, then y 1 = f1(x 1 , x 2 , x 3 , x 4 ) with Type 2
R 4 : If x 2 is Big and x 3 is Big, then y4 = f 4 (x 1 , x 2 , x 3 , x 4) with Type 2
Selected FPN
FPN
Fig. 19. The example of the FPN design guided by some chromosome (Type IV, Gaussian-like MF, Max = 4, 1st layer)
functions for each input variable are used. Here, the number of entire input variables (here, entire system input variables) considered in the 1st layer is given as 6 (Type IV). The polynomial order selected is given as Type 2. Furthermore the entire system input variables for the regression polynomial function of the consequent part of fuzzy rules in the first layer were considered. That is, “Type 2∗ ” as the consequent input type in the 1st layer is used and in the 2nd layer or higher, the input variables in the conclusion part of fuzzy rules are kept as those in the conditional part of the fuzzy rules. Especially, in the 2nd layer or higher, the number of entire input variables is given as W that is the number of the nodes selected in the current layer, as the output nodes of the preceding layer. Refer to sub-step 4 of step 4–3 of the introduced design process. As mentioned previously, the maximal number (Max) of input variables for the selection is confined to 4, and two variables (such as x2 and x3 ) were selected among them. The parameters of the conclusion part (polynomial) of fuzzy rules can be determined by the standard least-squares method. Table 8 summarizes the results when using Types II and IV: According to the maximal number of inputs to be selected (Max = 2 to 5), the selected node numbers, the selected polynomial type (Type T), and its corresponding
2 3 4 5
Max
2 3 4 5
Max
2 3 4 5
Max
2 3 4 5
Max
2nd layer EPI Node T PI 0.133 8 19 1 0.049 0.133 8 20 0 1 0.049 0.133 18 21 0 0 1 0.049 0.133 4 27 0 0 0 1 0.049
2nd layer EPI Node T PI 0.142 8 14 2 0.017 0.142 25 16 17 1 0.025 0.142 3 9 14 25 1 0.021 0.142 5 6 9 24 0 1 0.020
2nd layer EPI Node T PI 0.149 25 0 3 0.012 0.136 3 14 0 1 0.021 0.136 12 27 0 0 1 0.021 0.136 16 27 0 0 0 1 0.021
2nd layer EPI Node T PI 0.145 25 0 3 0.012 0.145 10 29 0 4 0.012 0.145 19 29 0 0 4 0.011 0.145 5 27 28 0 0 1 0.015
1st layer Node T PI 1 0 4 0.020 1 0 0 4 0.020 1 0 0 0 4 0.020 1 0 0 0 0 4 0.020
1st layer Node T PI 1 2 4 0.017 1 2 0 4 0.017 1 2 0 0 4 0.017 1 2 0 0 0 4 0.017
1st layer Node T PI 2 3 2 0.013 2 5 6 1 0.021 2 5 6 0 1 0.021 2 5 6 0 0 1 0.021
1st layer Node T PI 2 3 2 0.012 2 3 0 2 0.012 2 3 0 0 2 0.012 2 3 0 0 0 2 0.012 EPI 0.146 0.139 0.125 0.138
EPI 0.148 0.124 0.124 0.124
EPI 0.134 0.131 0.125 0.127
EPI 0.129 0.129 0.129 0.129
(b) In case of Type IV (b-1) Triangular MF 3rd layer Node T PI EPI 2 23 2 0.013 0.143 2 4 6 2 0.015 0.117 19 25 0 0 3 0.018 0.122 5 6 10 0 0 4 0.015 0.106 (b-2) Gaussian-like MF 3rd layer Node T PI EPI 2 0 3 0.013 0.147 11 13 23 1 0.023 0.129 4 13 29 0 1 0.020 0.104 11 27 29 0 0 1 0.021 0.114
(a) In case of Type II (a-1) Triangular MF 3rd layer Node T PI EPI 16 0 2 0.049 0.127 5 0 0 2 0.049 0.127 1 4 13 19 1 0.019 0.126 11 22 0 0 0 2 0.019 0.124 (a-2) Gaussian-like MF 3rd layer Node T PI EPI 17 26 2 0.017 0.132 10 20 26 1 0.024 0.127 2 3 9 10 1 0.023 0.111 4 7 17 20 0 1 0.022 0.117
4th layer T PI 19 0 3 0.015 2 3 7 1 0.016 29 0 0 0 3 0.011 3 5 0 0 0 2 0.009 Node
4th layer Node T PI 3 26 3 0.013 10 12 30 1 0.017 8 14 15 19 1 0.017 5 27 0 0 0 2 0.015
4th layer Node T PI 15 25 4 0.017 2 4 0 2 0.016 10 20 21 28 1 0.030 17 29 0 0 0 2 0.014
4th layer T PI 7 9 3 0.018 11 20 24 1 0.017 10 20 27 30 1 0.018 5 23 0 0 0 2 0.018 Node
5th layer EPI Node T PI 0.149 8 14 4 0.010 0.116 7 14 0 2 0.008 0.106 2 20 24 27 1 0.008 0.106 5 15 20 0 0 1 0.016
5th layer EPI Node T PI 0.141 21 30 2 0.012 0.114 4 12 21 4 0.011 0.114 6 10 14 28 1 0.011 0.105 7 19 0 0 0 3 0.015
5th layer EPI Node T PI 0.130 1 5 4 0.016 0.119 16 25 0 2 0.015 0.108 3 13 0 0 2 0.018 0.114 19 29 0 0 0 3 0.014
5th layer EPI Node T PI 0.122 22 30 2 0.018 0.123 9 17 0 4 0.017 0.123 14 23 0 0 1 0.018 0.119 17 21 26 27 0 1 0.016
EPI 0.152 0.119 0.093 0.100
EPI 0.125 0.103 0.106 0.103
EPI 0.128 0.117 0.101 0.108
EPI 0.119 0.119 0.119 0.114
Table 8. Performance index of the network of each layer versus the increase of maximal number of inputs to be selected for Types II and IV with Type T∗
90 S.-K. Oh and W. Pedrycz
Genetically Optimized Self-organizing Neural Networks
91
performance index (PI and EPI) were shown when the genetic optimization for each layer was carried out. “Node” denotes the nodes for which the fitness value is maximal in each layer. For example, in case of Table 8(b-2), the fitness value in layer 5 is maximal for Max = 5 when nodes 5, 15, 20 occurring in the previous layer are selected as the node inputs in the present layer. Only 3 inputs of Type 1 (constant) were selected as the result of the genetic optimization. Here, node “0” indicates that it has not been selected by the genetic operation. Therefore the width (the number of nodes) of the layer can be lower in comparison to the conventional FPNN (which immensely contributes to the compactness of the resulting network). In that case, the minimal value of the performance index at the node, that is PI = 0.016, EPI = 0.100 are obtained. Fig. 20 shows the values of performance index vis-` a-vis number of layers of the gSONN with respect to the maximal number of inputs to be selected as optimal architectures of each layer of the network included in Table 8(b) while Fig. 21 summarizes the values of the performance index represented vis-` a-vis the increasing number of the layers with regard to the number of Maximal number of inputs to be selected(Max) , 3(B) ;
2(A) ;
, 4(C) ;
Maximal number of inputs to be selected(Max)
, 5 (D) ;
0.022 A : (2 3) B : (2 5 6) C : (2 5 6 0) D : (2 5 6 0 0) A : (25 0) B : ( 3 14 0) C : (12 27 0 0) D : (16 27 0 0 0)
0.018 0.016
A : (21 30) B : ( 4 12 21) C : ( 6 10 14 28) D : ( 7 19 0 0 0)
0.014 A : ( 2 23) B : ( 2 4 6) C : (19 25 0 0) D : ( 5 6 10 0 0)
0.012
1
2
A : ( 3 26) B : (10 12 30) C : ( 8 14 15 19) D : ( 5 27 0 0 0)
3
0.12 0.11
4
0.1
5
1
2
,5 (D) ;
0.02 0.018 0.016
0) 3 7) 0 0 0) 5 0 0 0) A : (8 14) B : (7 14 0) C : (2 20 24 27) D : (5 15 20 0 0)
0.014 A : (2 0) B : (11 13 22) C : ( 4 13 29 0) D : (11 27 29 0 0)
0.012 0.01 0.008
1
2
3
Layer
5
, 3(B) ;
, 4(C) ;
,5 (D) ;
0.15
Testing error
0.022
Training error
A : (19 B:( 2 C : (29 D:( 3
4
(a-2) Testing error
2(A) ;
0.16
0.024 A : (25 0) B : (10 29 0) C : (19 29 0 0) D : ( 5 27 28 0 0) A : (2 3) B : (2 3 0) C : (2 3 0 0) D : (2 3 0 0 0)
3
Layer
Maximal number of inputs to be selected(Max)
Maximal number of inputs to be selected(Max) , 4(C) ;
,5 (D) ;
0.13
(a-1) Training error error (a) Triangular MF
, 3(B) ;
, 4(C) ;
0.14
Layer
2(A) ;
, 3(B) ;
0.15
Testing error
Training error
0.02
0.01
2(A) ;
0.16
0.14 0.13 0.12 0.11 0.1
4
5
0.09
1
(b-1) Training error error (b) Gaussian-like MF
2
3
4
5
Layer
(b-2) Testing error
Fig. 20. Performance index of gSONN for Type IV according to the increase of number of layers
92
S.-K. Oh and W. Pedrycz No. of system inputs : 2 ;
,3;
,4;
No. of system inputs : 2 ;
, 6;
0.05 0.045
,4;
,6;
: Selected input variables : Entire system input variables
0.3
0.04
Testing error
Training error
,3;
0.35 : Selected input variables : Entire system input variables
0.035 0.03 0.025 0.02
0.25
0.2
0.15
0.015 0.01
1
2
3
4
0.1
5
Layer
1
2
(a-1) Training error error (a) Triangular MF
No. of system inputs : 2 ;
0.045
,4;
,6;
4
5
(a-2) Testing error
No. of system inputs : 2 ;
,3;
,4;
,6;
0.3 : Selected input variables
: Selected input variables : Entire system input variables
0.04
: Entire system input variables
0.25
0.035
Testing error
Testing error
,3;
3
Layer
0.03 0.025 0.02 0.015
0.2
0.15
0.1
0.01 0.005
1
2
3
4
5
0.05
1
Layer
(b-1) Training error error (b) Gaussian-like MF
2
3
4
5
Layer
(b-2) Testing error
Fig. 21. Performance index of gSONN with respect to the increase of number of system inputs (in case of Max = 5)
entire system inputs being used in gSONN when using Max = 5. In Fig. 20, A(•)–D(•) denote the optimal node numbers at each layer of the network, namely those with the best predictive performance. Here, the node numbers of the 1st layer represent system input numbers, and the node numbers of each layer in the 1nd layer or higher represent the output node numbers of the preceding layer, as the optimal node that has the best output performance in the current layer. Fig. 22 illustrates the detailed optimal topology of the network with 2 layers, compared to the conventional optimized network of Fig. 23. That is, in Fig. 23, the performance of two types of the conventional optimized SONNs was quantified by the values of PI = 0.020, EPI = 0.130 for Type II, and PI = 0.016, EPI = 0.128 for Type III whereas under the condition given as similar performance values, two types of gSONN architectures were depicted as shown Fig. 22(a) and (b). As shown in Fig. 22, the genetic design procedure at each stage (layer) of SONN leads to the selection of the preferred nodes (or FPNs) with optimal local characteristics (such as the number of input variables, the order
Genetically Optimized Self-organizing Neural Networks u(t-3)
FPN5
2
u(t-2)
4
u(t-2)
FPN6
2
y(t-2)
3
FPN23
4
y^
1
1
2
u(t-1)
FPN29
2
y(t-3)
y^
4
FPN 29
y(t-2)
FPN24
1
FPN19
2
2
FPN9
y(t-1)
93
3
4
1
y(t-1)
(a) FPN-based gSONN with 2 layers for Type II, Max = 5, and Gaussianlike MF
(b) FPN-based gSONN with 2 layers for Type IV, Max = 4, and Gaussian-like MF
Fig. 22. Genetically optimized FPNN (gFPNN) architecture FPN 3 FPN 5
u(t-2)
FPN 1
FPN 6
FPN 28 FPN 29
FPN22
y(t-2)
FPN11
y^
NOP 7
y(t-1) NOP 2
FPN 10
NOP31
NOP 31
NOP 3 NOP 4
(a) Type II u(t-2)
FPN 1
FPN1
FPN 2
FPN5
FPN 3
FPN6
FPN4
FPN8
FPN5
FPN14
FPN6
FPN15
FPN 1
c
u(t-1)
c
y(t-2)
c
y(t-1)
c
FPN 17
FPN 3
FPN 20
FPN 24
FPN 7
y^
FPN 30
(b) Type III Fig. 23. Genetically optimized FPNN (gFPNN) architecture
of polynomial of the consequent part of fuzzy rules, and a collection of the specific subset of input variables). In the sequel, the best results for Type II network in the 2nd layer are obtained when using Max = 5 and Gaussian-like MF, and this happens with the Type 1 (constant) and 4 nodes at the inputs (nodes numbered as 5, 6, 9, 24); the performance of this network is quantified by the values of PI = 0.020 and EPI = 0.127. In addition, the most preferred results for the Type IV network in the 2nd layer coming with PI = 0.011 and EPI = 0.125 have been reported when using Max = 4 and Gaussian-like MF with the polynomial of Type 4 and 2 nodes at the inputs (the node numbers are 19, 29). Therefore the width (the number of nodes) of the layer as well as the depth (the number of layers) of the network can be lower in comparison
94
S.-K. Oh and W. Pedrycz
Table 9. Performance index of FPN based SONN architectures for Types I, II, III, and IV PN-based SONN Max = 5 (Triangular MF) Type I (SI: 2) Max = 5 (Gaussian-like MF)
Max = 5 (Triangular MF) Type II (SI: 3) Max = 4 (Gaussian-like MF)
Max = 5 (Triangular MF) Type III (SI: 4) Max = 5 (Gaussian-like MF)
Max = 5 (Triangular MF) Type IV (SI: 6) Max = 4 (Gaussian-like MF)
Layer 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
Optimal Polynomial Neuron No. of inputs (Node no.) Polynomial type 1(1) Type 2 4(7, 9, 10, 11) Type 1 4(6, 12, 25, 28) Type 1 4(6, 11, 12, 27) Type 1 3(10, 14, 19) Type 1 2(1,2) Type 3 3(6, 9, 11) Type 2 4(5, 13, 15, 25) Type 1 2(3, 6) Type 3 2(28, 30) Type 2 1(1) Type 4 2(4, 27) Type 1 2(11, 22) Type 2 2(5, 23) Type 2 4(17, 21, 26, 27) Type 1 2(1, 2) Type 4 4(3, 9, 14, 25) Type 1 4(2, 3, 9, 10) Type 1 4(10, 20, 21, 28) Type 1 2(3, 13) Type 2 3(1, 3, 4) Type 1 2(1, 26) Type 1 4(2, 6, 13, 15) Type 1 4(10, 14, 17, 28) Type 1 4(9, 14, 17, 24) Type 1 3(1, 3, 4) Type 2 4(1, 11, 24, 27) Type 1 4(1, 8, 19, 24) Type 1 2(2, 22) Type 3 4(8,9,23,24) Type 1 3(2, 5, 6) Type 1 2(16, 27) Type 1 3(5, 6, 10) Type 4 2(5, 27) Type 2 2(7, 19) Type 3 2(2, 3) Type 2 2(19, 29) Type 4 3(4, 13, 29) Type 1 1(29) Type 3 4(2, 20, 24, 27) Type 1
PIs
EPIs
0.022 0.018 0.017 0.017 0.016 0.022 0.020 0.020 0.017 0.017 0.020 0.049 0.019 0.018 0.016 0.017 0.021 0.023 0.030 0.018 0.021 0.021 0.017 0.010 0.010 0.016 0.043 0.024 0.012 0.014 0.021 0.021 0.015 0.015 0.015 0.012 0.011 0.020 0.011 0.008
0.335 0.271 0.270 0.268 0.264 0.281 0.267 0.260 0.252 0.249 0.133 0.129 0.124 0.119 0.114 0.142 0.125 0.111 0.108 0.101 0.135 0.125 0.123 0.120 0.115 0.146 0.128 0.117 0.104 0.099 0.136 0.124 0.106 0.105 0.103 0.145 0.125 0.104 0.106 0.093
to the conventional SONN (which immensely contributes to the compactness of the resulting network). Table 9 summarizes the detailed results of the optimal architecture according to each Type of the network. Table 10 contrasts the performance of the genetically developed network with other fuzzy and neuro-fuzzy models studied in the literatures. The experimental results clearly reveal that the proposed approach and the resulting model outperforms the existing networks both in terms of better approximation capabilities (lower values of the performance index on the training data, PIs ) as well as superb generalization abilities (expressed by the performance index on the testing data, EPIs ). In addition, the structurally optimized gSONN leads to the effective reduction of the depth of network as well as the width of the layer and the avoidance of a substantial amount
Genetically Optimized Self-organizing Neural Networks
95
Table 10. Comparative analysis of the performance of the network; considered are models reported in the literature Performance index Model PI PIs EPIs Box and Jenkin’s model [20] 0.710 Tong’s model [21] 0.469 Sugeno and Yasukawa’s model [22] 0.355 Sugeno and Yasukawa’s model [23] 0.190 Xu and Zailu’s model [24] 0.328 Pedrycz’s model [15] 0.320 Chen’s model [25] 0.268 Gomez-Skarmeta’s model [26] 0.157 Oh and Pedrycz’s model [16] 0.123 0.020 0.271 Kim et al.’s model [27] 0.055 Kim et al.’s model [28] 0.034 0.244 Leski and Czogala’s model [29] 0.047 Lin and Cunningham’s model [30] 0.071 0.261 NNFS model [31] 0.128 CASE I (SI = 4, 5th layer) 0.016 0.116 FPNN [32] CASE II (SI = 4, 5th layer) 0.016 0.128 Basic (SI = 4, 5th layer) 0.021 0.110 PNN [33] Modified (SI = 4, 5th layer) 0.015 0.103 Triangular (SI = 4, 5th layer) 0.019 0.134 HFPNN [34] Gaussian (SI = 4, 5th layer) 0.021 0.119 Basic SOPNN (SI = 4, 5th layer) 0.027 0.021 0.085 Generic SOPNN [10] Modified SOPNN (SI = 4, 5th layer) 0.035 0.017 0.095 Basic SOPNN (SI = 4, 5th layer) 0.020 0.119 Advanced SOPNN [11] Modified SOPNN (SI = 4, 5th layer) 0.018 0.118 Basic Case 1(5th layer) 0.016 0.266 Type I SONN Case 2(5th layer) 0.016 0.265 (SI = 2) Modified Case 1(5th layer) 0.013 0.267 SONN∗ [13] SONN Case 2(5th layer) 0.013 0.272 Basic Case 1(5th layer) 0.016 0.116 Type II SONN Case 2(5th layer) 0.016 0.128 (SI = 4) Modified Case 1(5th layer) 0.016 0.133 SONN Case 2(5th layer) 0.018 0.131 Type I 2nd layer (Max = 4) 0.019 0.282 (SI = 2) 5th layer (Max = 5) 0.017 0.259 Type II 1st layer (Max = 5) 0.022 0.135 PN(SI = 3) 3rd layer (Max = 5) 0.022 0.115 based Type III 2nd layer (Max = 5) 0.015 0.119 (SI = 4) 5th layer (Max = 5) 0.018 0.098 Type IV 2nd layer (Max = 5) 0.015 0.112 Proposed (SI = 6) 3rd layer (Max = 5) 0.012 0.091 gSONN Type I Gaussian1st layer (Max = 5) 0.017 0.281 (SI = 2) like 5th layer (Max = 5) 0.014 0.249 Type II Gaussian1st layer (Max = 5) 0.017 0.142 FPN(SI = 3) like 5th layer (Max = 5) 0.014 0.108 based Type III Gaussian1st layer (Max = 5) 0.016 0.146 (SI = 4) like 5th layer (Max = 5) 0.014 0.099 Type IV Gaussian2nd layer (Max = 5) 0.012 0.145 (SI = 6) like 5th layer (Max = 4) 0.008 0.093 PI - performance index over the entire data set, PIs - performance index on the training data, EPIs - performance index on the testing data. *: denotes “conventional optimized FPN-based SONN”.
96
S.-K. Oh and W. Pedrycz
of time-consuming iterations for finding the most preferred network in the conventional SONN. PIs (EPIs ) is defined as the mean square errors(MSE) between the experimental data and the respective outputs of the model (network). 5.2 Chaotic time series In this section, we demonstrate how the GA-based SONN can be utilized to predict future values of a chaotic Mackey-Glass time series. The performance of the network is also contrasted with some other models existing in the literature [35–41]. The time series is generated by the chaotic Mackey-Glass differential delay equation [42] comes in the form x(t) ˙ =
0.2x(t − τ ) − 0.1x(t) 1 + x10 (t − τ )
(20)
The prediction of future values of this series arises as a benchmark problem that has been used and reported by a number of researchers. To obtain the time series value at each integer point, we applied the fourth-order RungeKutta method to find the numerical solution to (20). From the Mackey-Glass time series x(t), we extracted 1000 input-output data pairs of three types of vector formats such as Type I, Type II, and Type III as shown in Table 11. We choose the input variables of nodes in the 1st layer of SONN architecture from these input variables. The first 500 pairs were used as the training data set while the remaining 500 pairs formed the testing data set. To come up with a quantitative evaluation of the network, we use the standard RMSE performance index as given by (18). The parameters used for optimization of this process modeling are the same as used in the previous experiments. Especially in the optimization of each layer of the PN-based SONN, we use 100 generations and 150 populations. The population size being selected from the total population size (150) is equal to 100. The GA-based design procedure is carried out in the same manner as in the previous experiments as well. The consequent input type to be used in this process is the same as that in case of gas furnace process. Table 11. System’s input vector formats for the design of SONN Number of System Inputs (SI) 4(Type I) 5(Type II) 6(Type III) where t = 118 to 1117.
Inputs and output x(t−18), x(t−12), x(t−6), x(t);x(t+6) x(t−24), x(t−18), x(t−12), x(t−6), x(t);x(t+6) x(t−30), x(t−24), x(t−18), x(t−12), x(t−6), x(t);x(t+6)
Genetically Optimized Self-organizing Neural Networks
97
5.2.1 PN-based SONN Table 12 summarizes the results for Type III when using the maximal number, 2, 3, 4, 5, 10 of inputs to be selected. The best results for the network in the 5th layer are reported as PI = 0.0009, EPI = 0.0009 when using Max = 10. Fig. 24 depicts the values of the performance index of the PN-based gSONN with respect to the maximal number of inputs to be selected when the number of system inputs is equal to 6 (Type III). Next, Fig. 25 shows the performance index of the gSONN with respect to the number of entire system inputs when using Max = 10. 5.2.2 FPN-based SONN Table 14 summarizes the performance of the 1st to 5th layer of the network when changing the maximal number of inputs to be selected; here Max was set up to 2 through 5. Fig. 26 depicts the performance index of each layer of FPNbased gSONN according to the increase of maximal number of inputs to be selected. Fig. 27 summarizes the values of the performance index represented vis-` a-vis the increasing number of the layers with regard to the number of selected inputs (or entire system inputs) being used in gSONN when using Max = 5. Fig. 28(a)–(b) illustrate the detailed optimal topologies of gSONN for 1 layer when using triangular MF: the results of the network have been reported as PI = 4.0e-4, EPI = 4.0e-4 for Max = 3, and PI = 7.7e-5, EPI = 1.6e-4 for Max = 5. Table 13 summarizes the detailed results of the optimal architecture according to each Type of the network. And also Fig. 28(c)–(d) illustrate the detailed optimal topologies of gSONN for 1 layer in case of Gaussian-like MF: those are quantified as PI = 3.0e-4, EPI = 3.0e-4 for Max = 2, and PI = 3.6e-5, EPI = 4.5e-5 for Max = 5. As shown in Fig. 28, the proposed network enables the architecture to be a structurally more optimized and simplified network than the conventional SONN. Fig. 29 illustrates the optimization process by visualizing the values of the performance index obtained in successive generations of GA. It also shows the optimized network architecture when using Gaussian-like MF (the maximal number (Max) of inputs to be selected is set to 5 with the structure composed of 5 layers). As shown in Fig. 29, the variation ratio (slope) of the performance of the network is almost the same up to the 2nd through 5th layer, therefore in this case, the stopping criterion can be taken into consideration up to maximally 1 or 2 layers for the purpose to effectively reduce the number of nodes as well as layers of the network (from the viewpoint of the width and depth of gSONN for the compact network). Table 15 summarizes the results of the optimal architecture according to each Type (such as Types I, II, and III) of the network when using Type T∗ as the consequent input type.
EPI 17
10
2 0.0211 0.0203
73
50
73
2 0.0036 0.0034
2 0.0106 0.0105 47 51 61 69
2 0.0161 0.0159 1
64
Node
PI
EPI 31
2 0.0021 0.0019
2 0.0063 0.0062 6 31 46
66
93
86
Node
2 0.0107 0.0103 15 80
2 0.0215 0.0211
T
PI
EPI 70
14
43
73
Node
2 0.0013 0.0012
2 0.0052 0.0052 29 40 75 95
2 0.0080 0.0078 8
2 0.0173 0.0170
T
PI
EPI
2 0.0009 0.0009
2 0.0045 0.0044
2 0.0067 0.0065
2 0.0156 0.0154
T
1 3 4 5 6 2 0.0231 0.0223 2 51 70 74 83 2 0.0126 0.0124 74 76 79 86 95 2 0.0066 0.0063 16 36 43 53 58 2 0.0049 0.0046 10 21 34 46 67 2 0.0041 0.0038
39
PI
2 0.0309 0.0305
T
5
6 2 0.0347 0.0339 31
23
Node
1 3 4 6 2 0.0293 0.0283 4 25 59 63
12
5th layer
3 4
EPI
4th layer
4
PI
2 0.0502 0.0497
3rd layer
3
6
2nd layer
4
Node T
1st layer
2
Max
Table 12. Performance index of the PN-based SONN of each layer versus the increase of maximal number of inputs to be selected for Type III
98 S.-K. Oh and W. Pedrycz
Genetically Optimized Self-organizing Neural Networks Maximal number of inputs to be selected(Max) 2(A) ;
0.06
, 4(C) ;
,5 (D) ;
Maximal number of inputs to be selected (Max) , 3(B) ; , 4(C) ; ,5 (D) ; , 10(E) ;
, 10(E) ;
2(A) ;
0.05
A : (4 6) A : (17 64) B : (3 4 6) B : ( 1 50 73) C : (1 3 4 6) C : (47 51 61 69) D : (1 3 4 5 6) D : (74 76 79 86 95) E : (1 2 3 4 5 6 0 0 0 0) E : ( 2 3 4 31 52 61 72 78 81 85) A : (12 23) B : (31 39 73) A : (70 73) C : ( 4 25 59 63) B : ( 8 14 43) D : ( 2 51 70 74 83) C : (29 40 75 95) E : ( 1 35 38 42 45 56 57 70 91 97) D : (10 21 34 46 67) E : ( 3 8 23 38 50 53 56 63 85 99)
0.04 0.03
0.045 0.04 0.035
Testing error
0.05
Training error
, 3(B) ;
A : (31 86) B : (15 80 93) C : ( 6 31 46 66) D : (16 36 43 53 58) E : ( 3 4 20 27 38 57 80 82 91 96)
0.02
99
0.03 0.025 0.02 0.015 0.01
0.01
0.005 0
1
2
3
4
0
5
1
2
3
Layer
4
5
Layer
(a) Training error
(b) Testing error
Fig. 24. Performance index according to the increase of number of layers in case of Type III No. of system inputs : 4 ;
,5;
No. of system inputs : 4 ;
,6;
0.035
0.03
0.03
0.025
Testing error
Training error
0.025 0.02 0.015 0.01
,6;
0.02 0.015 0.01 0.005
0.005 0
,5;
1
2
3
Layer
(a) Training error
4
5
0
1
2
3
4
5
Layer
(b) Testing error
Fig. 25. Performance index of gPNN with respect to the increase of number of system inputs (Max = 10)
Table 16 gives a comparative summary of the network with other models. The experimental results clearly reveal that it outperforms the existing models both in terms of better approximation capabilities (lower values of the performance index on the training data, PIs ) as well as superb generalization abilities (expressed by the performance index on the testing data EPIs ). PIs (EPIs ) is defined as the root mean square errors (RMSE) computed for the experimental data and the respective outputs of the network.
100
S.-K. Oh and W. Pedrycz
Table 13. Performance index of PN based SONN architectures for Types I, II, and III PN-based SONN
Max = 5 Type I (SI: 4) Max = 10
Max = 5 Type II (SI: 5) Max = 10
Max = 5 Type III (SI: 6) Max = 10
Layer 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
Optimal Polynomial Neuron No. of inputs (Node no.) Polynomial type 4(1, 2, 3, 4) Type 2 5(19, 26, 28, 34, 41) Type 2 5(2, 17, 23, 48, 86) Type 2 5(3, 25, 71, 75, 78) Type 2 5(1, 3, 54, 94, 98) Type 2 4(1, 2, 3, 4)) Type 2 10(2,4,8,10,11,12,13,15,21,23) Type 2 10(12,34,35,43,52,60,62,72,89,95) Type 2 10(12,13,28,29,32,41,57,73,91,94) Type 2 9(44,51,53,67,69,75,86,92,93) Type 2 5(1, 2, 3, 4, 5) Type 2 5(39, 59, 71, 74, 84)) Type 2 5(46, 51, 66, 67, 88) Type 2 4(29, 38, 58, 72, 83) Type 2 5(10, 21, 23, 63, 85) Type 2 5(1, 2, 3, 4, 5) Type 2 10(7,38,39,40,47,48,56,59,60,68) Type 2 10(3,11,18,35,42,50,52,53,67,73) Type 2 10(2,8,26,30,33,35,48,79,81,85) Type 2 10(3,7,10,31,54,59,79,89,95,97) Type 2 5(1, 3, 4, 5, 6) Type 2 5(2, 51, 70, 74, 83) Type 2 5(74, 76, 79, 86, 95) Type 2 5(16, 36, 43, 53, 58) Type 2 5(10, 21, 34, 46, 67) Type 2 6(1, 2, 3, 4, 5, 6) Type 2 10(1,35,38,42,45,56,57,70,91,97) Type 2 10(2,3,4,31,52,61,72,78,81,85) Type 2 10(3,4,20,27,38,57,80,82,91,96) Type 2 10(3,8,23,38,50,53,56,63,85,99) Type 2
PIs
EPIs
0.0302 0.0082 0.0063 0.0050 0.0043 0.0302 0.0047 0.0039 0.0032 0.0026 0.0282 0.0088 0.0054 0.0042 0.0037 0.0282 0.0026 0.0019 0.0015 0.0012 0.0231 0.0126 0.0066 0.0049 0.0041 0.0211 0.0036 0.0021 0.0013 0.0009
0.0294 0.0081 0.0061 0.0049 0.0043 0.0294 0.0045 0.0038 0.0031 0.0026 0.0274 0.0087 0.0052 0.0041 0.0035 0.0274 0.0024 0.0018 0.0014 0.0012 0.0223 0.0124 0.0063 0.0046 0.0038 0.0203 0.0034 0.0019 0.0012 0.0009
6 Concluding remarks In this study, we introduced a class of genetically optimized self-organizing neural networks, discussed their topologies, came up with a detailed genetic design procedure, and used these networks to nonlinear system modeling. The comprehensive experimental studies involving well-known datasets quantify a superb performance of the network in comparison to the existing fuzzy and neuro-fuzzy models. The key features of this approach can be enumerated as follows: • The gSONN is sophisticated and optimized architecture capable of constructing models out of a few number of entire system inputs as well as a limited data set. • The proposed design methodology helps reach a compromise between approximation and generalization capabilities of the constructed gSONN model. • The depth (layer size) and width (node size of each layer) of the gSONN can be selected as a result of a tradeoff between accuracy and complexity of the overall model.
14
EPI
19 2 3.2e-5 4.1e-5 7
17 28
21
18
12 18
3 0.0002 0.0002
4 29 2 2.7e-5 3.6e-5 15
14
EPI
23
EPI
0
0
2 2.4e-5 3.3e-5
3 2.7e-5 3.5e-5
4 0.0002 0.0002
5th layer T PI 21
Node
4 2.8e-5 3.6e-5 2 3 18 0 2 2.4e-5 3.4e-5 7 19 0
2 3.1e-5 3.9e-5 6
3 0.0002 0.0002
4th layer T PI
1 4 5 0 0 3 3.6e-5 4.5e-5 12 14 18 0 0 2 3.2e-5 4.1e-5 7 8 13 14 20 4 2.7e-5 3.6e-5 6 14 15 0 0 3 2.4e-5 3.4e-5 18 26 28 0 0 4 2.3e-5 3.3e-5
18
13
Node
5
5 3 3.6e-5 0.199 3
4 0.0002 0.0002
EPI
3 5.6e-5 9.9e-5
1 4 5 0 3 3.6e-5 4.5e-5 7 16 19 24 2 2.8e-5 4.1e-5 11 17 23
20
3rd layer T PI
0
25 3 0.0002 0.0002
3 0.0015 0.0013
1 4
18
20
16
4
3 0.0003 0.0003
Node
22 4 0.0002 0.0003 14
3 0.0015 0.0014
3
6
EPI
14
22
4 6.0e-5 1.2e-4 20 25 29 0 4 5.8e-5 1.0e-4 10 13 15
4 0.0003 0.0003 2
(b) Gaussian-like MF
0
24
10
5th layer T PI
4
EPI
2nd layer Node T PI
10
3 0.0016 0.0014
Node
2
Max
1st layer Node T PI
13 1 0.0003 0.0003 4
7
EPI
3 4 5 6 0 3 7.7e-5 1.6e-4 6 16 23 0 0 4 6.7e-5 1.2e-4 12 18 0 0 0 4 6.1e-5 1.0e-4 3 15 28 0 0 3 5.6e-5 9.7e-5 13 15 19 21 0 1 5.7e-5 9.4e-5
3
4
4th layer T PI
5
5 3 0.0004 0.0004 2
4 0.0016 0.0015
Node
3 4 5 6 3 7.7e-5 1.6e-4 4 14 23 0 2 6.9e-5 1.3e-4 6 10 14
25
EPI
3 4
14
3rd layer T PI
4
3 0.0019 0.0017
Node
3
5
EPI
(a) Triangular MF
4
EPI
2nd layer Node T PI
2
Max
1st layer Node T PI
Table 14. Performance index of the network of each layer versus the increase of maximal number of inputs to be selected for Type III (in case of Type T∗ )
Genetically Optimized Self-organizing Neural Networks 101
102
S.-K. Oh and W. Pedrycz
2
x 10 −3
Maximal number of inputs to be selected (Max) 2(A) ; , 3(B) ; , 4(C) ; ,5 (D) ;
1.8
1.8
1.4
1.4
A : (14 25) B : ( 2 3 13) C : ( 4 14 23 0) D : ( 6 16 23 0 0)
1.2 1 0.8
A : (10 22) B : ( 2 14 22) C : (20 25 29 0) D : ( 3 15 28 0 0)
A : (14 16) B : (14 20 25) C : (10 13 15 0) D : (13 15 19 21 0)
Testing error
Training error
Maximal number of inputs to be selected (Max) 2(A) ; , 3(B) ; , 4(C) ; ,5 (D) ;
1.6 A : (4 5) B : (3 4 5) C : (3 4 5 6) D : (3 4 5 6 0)
1.6
A : ( 4 7) B : ( 4 10 24) C : ( 6 10 14 0) D : (12 18 0 0)
0.6
1.2 1 0.8 0.6
0.4
0.4
0.2
0.2
0
x 10 −3
1
2
3
Layer
4
0
5
1
2
3
Layer
4
5
(a-1) Training error error (a-2) Testing error (a) Triangular MF
4
x 10 −4
Maximal number of inputs to be selected (Max) 2(A) ; , 3(B) ; , 4(C) ; ,5 (D) ;
4
A : (4 6) B : (1 4 5) C : (1 4 5 0) D : (1 4 5 0 0)
2.5
3
Testing error
3
Training error
Maximal number of inputs to be selected (Max) 2(A) ; , 3(B) ; , 4(C) ; ,5 (D) ;
3.5
3.5
A : (18 20) B : ( 3 18 19) A : ( 4 21) C : ( 7 16 16 24) B : (15 23 0) D : (12 14 18 0 0) C : ( 7 19 0 0) A : (13 18) D : (18 26 28 0 0) B : ( 7 17 21) A : (12 14) C : (11 17 23 28) B : ( 6 18 29) D : ( 7 8 13 14 20) C : ( 2 2 18 0) D : ( 6 14 15 0 0)
2 1.5 1
2.5
2 1.5
1 0.5
0.5 0
x 10 −4
1
2
3
Layer
4
5
0
1
2
3
4
5
Layer
(b-1) Training error error (b-2) Testing error (b) Gaussian-like MF Fig. 26. Performance index according to the increase of number of layers
• The structure of the network is not predetermined (as in most of the existing neural networks) but becomes dynamically adjusted and optimized during the development process. • With a properly selected type of membership functions and the organization of the layers, FPN based gSONN performs better than other fuzzy and neurofuzzy models. • The gSONN comes with a diversity of local neuron characteristics such as PNs or FPNs that are useful in copying with various nonlinear characteristics of the nonlinear systems. GA-based design procedure at each stage (layer) of gSONN leads to the optimal selection of these preferred nodes (PNs or FPNs) with local characteristics (such as the number of input variables, the order of the polynomial, and a collection of specific subset of input variables) available within a single node, and then based on these selections, we build the flexible and optimized architecture of gSONN.
Genetically Optimized Self-organizing Neural Networks −3 1.8 x 10
No. of system inputs : 4 ;
,5;
,6;
−3 1.8 x 10
No. of system inputs : 4 ;
1.6
: Entire system input variables
: Entire system input variables
1.4
Testing error
Training error
1.4
,6;
: Selected input variables
: Selected input variables
1.6
,5;
103
1.2 1 0.8 0.6
1.2 1
0.8 0.6
0.4
0.4
0.2
0.2
0
1
2
3
4
0
5
1
2
Layer
3
4
5
Layer
(a-1) Training error error (a-2) Testing error (a) Triangular MF −3 1.4 x 10
No. of system inputs : 4 ;
,5;
,6;
−3 1.4 x 10
No. of system inputs : 4 ;
: Selected input variables
1.2
: Entire system input variables
1
Testing error
Training error
1.2
0.8 0.6 0.4 0.2 0
,5;
,6;
: Selected input variables : Entire system input variables
1 0.8 0.6 0.4 0.2
1
2
3
4
5
0
1
Layer
2
3
4
5
Layer
(b-1) Training error error (b-2) Testing error (b) Gaussian-like MF Fig. 27. Performance index of gFPNN with respect to the increase of number of system inputs (Max = 5) x(t-30)
x(t-30)
x(t-24)
x(t-24)
x(t-18)
x(t-18) x(t-12)
FPN13 3
y^
3
x(t-12)
x(t-6)
x(t-6)
x(t)
x(t)
(a) Triangular MF (1 layer and Type III, Max = 3)
FPN16 4
3
(b) Triangular MF (1 layer and Type III, Max = 5)
x(t-30) x(t-24) x(t-18) x(t-12)
FPN18 2
3
y^
x(t-6) x(t)
(c) Gaussian-like MF (1 layer and Type III, Max = 2)
(d) Gaussian-like MF (1 layer and Type III, Max = 5)
Fig. 28. FPN-based gSONN architecture
y^
104
6.5
S.-K. Oh and W. Pedrycz x 10 −5
8 7.5
6
7 6.5
5
Testing error
Training error
5.5
4.5 1st layer
2nd layer
3rd layer
4th layer
5th layer
4 3.5
6 5.5
1st layer
2nd layer
3rd layer
4th layer
5th layer
5 4.5
3
4
2.5
3.5
2
x 10 −5
0
100
200 300 Generation
400
(a) Training error
500
3
0
100
200 300 Generation
400
500
(b) Testing error
Fig. 29. The optimization process of each performance index by the genetic algorithms (Type III, Max = 5, Gaussian) Table 15. Performance index of FPN based SONN architectures for Types I, II, and III PN-based SONN Max = 5 (Triangular MF) Type I (SI: 4) Max = 5 (Gaussian-like MF)
Max = 5 (Triangular MF) Type II (SI: 5) Max = 4 (Gaussian-like MF)
Max = 5 (Triangular MF) Type III (SI: 6) Max = 5 (Gaussian-like MF)
Layer 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
Optimal Polynomial Neuron No. of inputs (Node no.) Polynomial type 4(1, 2, 3, 4) Type 3 5(8, 24, 25, 27, 28) Type 2 4(2, 5, 9, 15) Type 2 4(2, 6, 17, 26) Type 3 4(1, 3, 17, 24) Type 1 4(1, 2, 3, 4) Type 3 5(7, 12, 18, 22, 29) Type 4 4(10, 18, 20, 22) Type 2 4(3, 7, 21, 26) Type 2 2(2, 15) Type 4 5(1, 2, 3, 4, 5) Type 4 4(1, 8, 9, 14) Type 4 4(2, 17, 23, 24) Type 3 5(8, 11, 18, 21, 26) Type 1 4(7, 9, 14, 29) Type 1 3(2, 3, 5) Type 3 4(9, 11, 21, 30) Type 2 4(2, 8, 19, 21) Type 4 2(22, 24) Type 4 3(8, 18, 27) Type 3 4(3, 4, 5, 6) Type 3 3(6, 16, 23) Type 4 2(12, 18) Type 4 3(3, 15, 28) Type 3 4(13, 15, 19, 21) Type 1 3(1, 4, 5) Type 3 3(12, 14, 18) Type 2 5(7, 8, 13, 14, 20) Type 4 3(6, 14, 15) Type 3 3(18, 26, 28) Type 4
PIs
EPIs
0.0017 0.0014 0.0011 0.0009 0.0009 0.0005 0.0004 0.0003 0.0003 0.0003 0.0003 0.0003 0.0002 0.0002 0.0002 0.0004 0.0003 0.0002 0.0002 0.0002 7.7e-5 6.7e-5 6.1e-5 5.6e-5 5.7e-5 3.6e-5 3.2e-5 2.7e-5 2.4e-5 2.3e-5
0.0016 0.0013 0.0011 0.0010 0.0010 0.0006 0.0006 0.0005 0.0005 0.0005 0.0004 0.0003 0.0003 0.0003 0.0003 0.0003 0.0003 0.0002 0.0002 0.0002 1.6e-4 1.2e-4 1.0e-4 9.7e-5 9.4e-5 4.5e-5 4.1e-5 3.6e-5 3.4e-5 3.3e-5
Genetically Optimized Self-organizing Neural Networks
105
Table 16. Comparative analysis of the performance of the network; considered are models reported in the literature Model Wang’s model [35]
PI 0.044 0.013 0.010
Performance index PIs EPIs NDEI∗
Cascaded-correlation NN [36] 0.06 Backpropagation MLP [36] 0.02 6th-order polynomial [36] 0.04 ANFIS [37] 0.0016 0.0015 0.007 FNN model [38] 0.014 0.009 Recurrent neural network [39] 0.0138 Basic Case 1 0.0011 0.0011 0.005 Type I (5th layer) Case 2 0.0027 0.0028 0.011 Modified Case 1 0.0012 0.0011 0.005 SONN∗∗ [40] (5th layer) Case 2 0.0038 0.0038 0.016 Basic Case 1 0.0003 0.0005 0.0016 Type II (5th layer) Case 2 0.0002 0.0004 0.0011 Modified Case 1 0.000001 0.00009 0.000006 Type III (5th layer) Case 2 0.00004 0.00007 0.00015 PNType I (5th layer) Max = 10 0.0026 0.0026 based Type II (5th layer) Max = 10 0.0012 0.0012 Type III (5th layer) Max = 10 0.0009 0.0009 Proposed Type I Triangular Max = 5 0.0017 0.0016 gSONN FPN- (1st layer) Gaussian Max = 5 0.0005 0.0006 based Type II Triangular Max = 5 0.0003 0.0004 st FPN- (1 layer) Gaussian Max = 5 0.0004 0.0003 based Type III Triangular Max = 5 7.7e-5 1.6e-4 (1st layer) Gaussian Max = 5 3.6e-5 4.5e-5 *Non-dimensional error index (NDEI) as used in [41] is defined as the root mean square errors divided by the standard deviation of the target series. ** is called “conventional optimized FPN-based SONN”.
• The design methodology comes with hybrid structural optimization and parametric learning viewed as two phases of modeling building. The GMDH method is comprised of both a structural phase such as a selforganizing and an evolutionary algorithm (rooted in natural law of survival of the fittest), and a parametric phase of Least Square Estimation (LSE)based learning. Therefore the structural and parametric optimization help utilize hybrid method (combining GAs with a structural phase of GMDH) and LSE-based technique in the most efficient way.
7 Acknowledgements This work was supported by the Korea Research Foundation Grant funded by the Korean Government (MOEHRD)(KRF-2006-311-D00194 and KRF-2006D00019).
106
S.-K. Oh and W. Pedrycz
References 1. Cherkassky V, Gehring D, Mulier F (1996) Comparison of adaptive methods for function estimation from samples. IEEE Trans Neural Netw 7:969–984 2. Dickerson JA, Kosko B (1996) Fuzzy function approximation with ellipsoidal rules. IEEE Trans Syst Man Cybern Part B 26:542–560 3. Sommer V, Tobias P, Kohl D, Sundgren H, Lundstrom L (1995) Neural networks and abductive networks for chemical sensor signals: a case comparison. Sens Actuators B 28:217–222 4. Kleinsteuber S, Sepehri N (1996) A polynomial network modeling approach to a class of large-scale hydraulic systems. Comput Elect Eng 22:151–168 5. Cordon O, et al. (2004) Ten years of genetic fuzzy systems: current framework and new trends. Fuzzy Set Syst 141(1):5–31 6. Ivakhnenko AG (1971) Polynomial theory of complex systems. IEEE Trans Syst Man Cybern SMC-1:364–378 7. Ivakhnenko AG, Madala HR (1994) Inductive learning algorithms for complex systems modeling. CRC, Boca Raton, FL 8. Ivakhnenko AG, Ivakhnenko GA (1995) The review of problems solvable by algorithms of the group method of data handling (GMDH). Pattern Recogn Image Anal 5(4):527–535 9. Ivakhnenko AG, Ivakhnenko GA, Muller JA (1994) Self-organization of neural networks with active neurons. Pattern Recogn Image Anal 4(2):185–196 10. Oh SK, Pedrycz W (2002) The design of self-organizing polynomial neural networks. Inf Sci 141:237–258 11. Oh SK, Pedrycz W, Park BJ (2003) Polynomial neural networks architecture: analysis and design. Comput Electr Eng 29(6):703–725 12. Park HS, Park BJ, Oh SK (2002) Optimal design of self-organizing polynomial neural networks by means of genetic algorithms. J Res Inst Eng Technol Dev 22:111–121 (in Korean) 13. Oh SK, Pedrycz W (2003) Fuzzy polynomial neuron-based self-organizing neural networks. Int J Gen Syst 32(3):237–250 14. Pedrycz W, Reformat M (1996) Evolutionary optimization of fuzzy models in fuzzy logic: a framework for the new millennium. In: Dimitrov V, Korotkich V (eds.) Studies in fuzziness and soft computing, pp 51–67 15. Pedrycz W (1984) An identification algorithm in fuzzy relational system. Fuzzy Set Syst 13:153–167 16. Oh SK, Pedrycz W (2000) Identification of fuzzy systems by means of an auto-tuning algorithm and its application to nonlinear systems. Fuzzy Set Syst 115(2):205–230 17. Michalewicz Z (1996) Genetic algorithms + Data structures = Evolution programs. Springer, Berlin Heidelberg Newyork 18. Holland JH (1975) Adaptation in natural and artificial systems. The University of Michigan Press, Ann Arbour 19. Jong DKA (1996) Are genetic algorithms function Optimizers?. In Manner R, Manderick B (eds.) Parallel problem solving from nature 2, North-Holland, Amsterdam 20. Box DE, Jenkins GM (1976) Time series analysis forcasting and control. Holden Day, California 21. Tong RM (1980) The evaluation of fuzzy models derived from experimental data. Fuzzy Set Syst 13:1–12
Genetically Optimized Self-organizing Neural Networks
107
22. Sugeno M, Yasukawa T (1991) Linguistic modeling based on numerical data. IFSA 91, Brussels, Comput Manage & Syst Sci:264–267 23. Sugeno M, Yasukawa T (1993) A fuzzy-logic-based approach to qualitative modeling. IEEE Trans Fuzzy Syst 1(1):7–31 24. Xu CW, Zailu Y (1987) Fuzzy model identification self-learning for dynamic system. IEEE Trans Syst Man Cybern SMC 17(4):683–689 25. Chen JQ, Xi YG, Zhang ZJ (1998) A clustering algorithm for fuzzy model identification. Fuzzy Set Syst 98:319–329 26. Gomez-Skarmeta AF, Delgado M, Vila MA (1999) About the use of fuzzy clustering techniques for fuzzy model identification. Fuzzy Set Syst 106:179–188 27. Kim ET, et al. (1997) A new approach to fuzzy modeling. IEEE Trans Fuzzy Syst 5(3):328–337 28. Kim ET, et al. (1998) A simple identified Sugeno-type fuzzy model via double clustering. Inf Sci 110:25–39 29. Leski J, Czogala E (1999) A new artificial neural networks based fuzzy inference system with moving consequents in if-then rules and selected applications. Fuzzy Set Syst 108:289–297 30. Lin Y, Cunningham III GA (1995) A new approach to fuzzy-neural modeling. IEEE Trans Fuzzy Syst 3(2):190–197 31. Wang Y, Rong G (1999) A self-organizing neural-network-based fuzzy system. Fuzzy Set Syst 103:1–11 32. Park HS, Oh SK, Yoon YW (2001) A new modeling approach to fuzzy-neural networks architecture. J Control Autom Syst Eng 7(8):664–674 (in Korean) 33. Oh SK, Kim DW, Park BJ (2000) A study on the optimal design of polynomial neural networks structure. Trans Korean Inst Electr Eng 49D(3):145–156 (in Korean) 34. Oh SK, Pedrycz W, Kim DW (2002) Hybrid fuzzy polynomial neural networks. Int J Uncertain, Fuzziness Knowl-Based Syst 10(3):257–280 35. Wang LX, Mendel JM (1992) Generating fuzzy rules from numerical data with applications. IEEE Trans Syst Man Cybern 22(6):1414–1427 36. Crowder III RS (1990) Predicting the mackey-glass time series with cascadecorrelation learning. In: Touretzky D, Hinton G, Sejnowski T (eds.) Proceedings of the 1990 connectionist models summer school, Carnegic Mellon University 37. Jang JSR (1993) ANFIS: adaptive-network-based fuzzy inference system. IEEE Trans Syst Man Cybern 23(3):665–685 38. Maguire LP, Roche B, McGinnity TM, McDaid LJ (1998) Predicting a chaotic time series using a fuzzy neural network. Inf Sci 112:125–136 39. James LC, Huang TY (1999) Automatic structure and parameter training methods for modeling of mechanical systems by recurrent neural networks. Appl Math Model 23:933–944 40. Oh SK, Pedrycz W, Ahn TC (2002) Self-organizing neural networks with fuzzy polynomial neurons. Appl Soft Comput 2(IF):1–10 41. Lapedes AS, Farber R (1987) Non-linear signal processing using neural networks: prediction and system modeling. Technical Report LA-UR-87-2662, Los Alamos National Laboratory, Los Alamos, New Mexico 42. Mackey MC, Glass L (1977) Oscillation and chaos in physiological control systems. Science 197:287–289 43. Cho MW, Kim GH, Seo TI, H YC, Cheng HH (2006) Integrated machining error compensation method using OMM data and modified PNN algorithm, International Journal of Machine Tool and Manufacture 46(2006):1417–1427
108
S.-K. Oh and W. Pedrycz
44. Menezes LM, Nikolaev NY (2006) Forecasting with genetically programmed polynomial neural networks. International Journal of Forecasting 22(2):249–265 45. Pei JS, Wright JP, Smyth AW (2005) Mapping polynomial fitting into feedforward neural networks for modeling nonlinear dynamic systems and beyond. Compt Methods Appl Mech Eng 194(42–44): 4481–4505 46. Nariman-Zadeh N, Darvizeh A, Jamali A, Moeini A (2005) Evolutionary design of generalized polynomial neural networks for modelling and prediction of explosive forming process. J Mater Process Technol 164–165:1561–1571 47. Delivopoulos E, Theocharis JB (2004) A modified PNN algorithm with optimal PD modeling using the orthogonal least squares method. Inf Sci 168(1–4):133–170 48. Huang LL, Shimizu A, Hagihara Y, Kobatake H (2003) Face detection from cluttered images using a polynomial neural network. Neurocomputing 51:197–211
Evolution of Inductive Self-organizing Networks Dongwon Kim and Gwi-Tae Park
Summary. We discuss a new design methodology of the self-organizing approximate technique using evolutionary algorithm (EA). In this technique, the selforganizing network dwells on the idea of group method of data handling. The performances of the network depend strongly on the number of input variables available to the model, the number of input variables, and type (order) of the polynomials to each node. They must be fixed by the designer in advance before the architecture is constructed. Therefore, the trial and error method has a heavy computation burden and low efficiency. Moreover, it does not guarantee that the obtained model is the best one. In this chapter, we propose evolved inductive self-organizing networks to alleviate these problems. The order of the polynomial, the number of input variables, and the optimum input variables are encoded as a chromosome, and the fitness of each chromosome is computed. The appropriate information of each node is evolved accordingly and tuned gradually throughout the EA iterations. The evolved network is a sophisticated and versatile architecture, which can construct models from a limited data set as well as from poorly defined complex problems. Comprehensive comparisons showed that the performance of the proposed model, which has a much simpler structure than the conventional model as well as previous identification methods, was significantly improved with respect to approximation and prediction abilities.
1 Introduction System modelling and identification is important in system analysis, control, and automation as well as scientific research, so much effort has been directed to developing advanced techniques of system modelling. Neural networks (NNs) and fuzzy systems have been widely used for modelling nonlinear systems. The approximation capability of neural networks, such as multilayer perceptrons, radial basis function (RBF) networks, or dynamic recurrent neural networks, has been investigated [1–3]. On the other hand, fuzzy systems can approximate nonlinear functions with arbitrary accuracy [4–5]. But the resultant neural network representation is very complex and difficult to understand, and fuzzy systems require too many fuzzy rules for accurate function D. Kim and G.-T. Park: Evolution of Inductive Self-organizing Networks, Studies in Computational Intelligence (SCI) 82, 109–128 (2008) c Springer-Verlag Berlin Heidelberg 2008 www.springerlink.com
110
D. Kim and G.-T. Park
approximation, particularly in the case of multidimensional inputs. Alternatively, there is the GMDH-type algorithm. Group method of data handling (GMDH) was introduced by Ivakhnenko in the early 1970’s [6–10]. GMDHtype algorithms have been extensively used since the mid-1970’s for prediction and modelling complex nonlinear processes. GMDH is mainly characterized as being self-organizing algorithm, which can provide automated selection of essential input variables without the use of prior information about the relationship among input-output variables [11]. Self-organizing Polynomial Neural Networks (SOPNN) [12–13], a GMDHtype algorithm, is a useful approximate technique. SOPNN has an architecture similar to that of the feedforward neural networks whose neurons are replaced by polynomial nodes. The output of the each node in SOPNN structure is obtained by using several types of high-order polynomials such as linear, quadratic, and modified quadratic of input variables. These polynomials are called partial descriptions (PDs). SOPNNs have fewer nodes than NNs, but its nodes are more flexible. Although the SOPNN is structured by a systematic design procedure, it has some drawbacks that must be solved. If there are sufficiently large number of input variables and data points, SOPNN algorithm has a tendency to produce overly complex networks. On the other hand, for a small number of available input variables, SOPNN does not maintain good performance. Moreover, the performances of SOPNN depend strongly on the number of input variables available to the model as well as the number of input variables and polynomial types or order in each PD. They must be chosen in advance before the architecture of SOPNN is constructed. In most cases, they are determined by a trial and error method, which must bear a heavy computational burden at low efficiency. Moreover, the SOPNN algorithm is a heuristic method so it does not guarantee that the obtained SOPNN is the best one for nonlinear system modelling. Therefore, these above-mentioned drawbacks must be overcome. In this chapter, we present a new design methodology of SOPNN using evolutionary algorithm (EA) to alleviate the above-mentioned drawbacks of the SOPNN. This new network is called the EA-based SOPNN. Evolutionary Algorithm (EA) has been widely used as a parallel global search method for optimization problems [14–16]. The EA is used to determine how many input variables are chosen to each node, which input variables are optimally chosen among many input variables, and what is the appropriate type of polynomials in each PD. This chapter is organized as follows. The conventional SOPNN and its drawbacks are briefly explained to illustrate the proposed modelling algorithm in Section 2. The new algorithm, the design methodology of EA-based SOPNN, is described and the coding of the key factors of the SOPNN, the representation of chromosome, and fitness function are also discussed in Section 2. The proposed EA-based SOPNN is applied to nonlinear systems modelling to assess its performances, and its simulation results are compared with those of other methods including the conventional SOPNN in Section 3. Conclusions
Evolution of Inductive Self-organizing Networks
111
are given in Section 4. Finally Appendix contains a summary of the design procedure of the conventional SOPNN algorithm.
2 Design of EA-based SOPNN The conventional SOPNN algorithm is based on the GMDH method and utilizes various types of polynomials such as linear, quadratic, and modified quadratic. By choosing the most significant input variables and polynomial types, the PDs in each layer can be obtained. The framework of the design procedure of the conventional SOPNN is summarized in Appendix, and further discussion on the conventional SOPNN can be obtained in [13]. As provided in the Appendix, when the final layer is constructed, the node with the best predictive capability is selected as the output node. All remaining nodes except the output node in the final layer are discarded. Furthermore, all the nodes in the previous layers that do not influence the output node are also removed by tracing the data flow path of each layer. The SOPNN is a flexible neural architecture whose structure is developed through a modeling process. Each PD can have a different number of input variables and can exploit a different order of the polynomial. As a result, SOPNN provides a systematic design procedure. But the number of input variables and the polynomial order type must be fixed by the designer in advance before the architecture is constructed. As a result, the trial and error method must bear a heavy computation burden at low efficiency. That is, it does not guarantee the best model, only that a good model for a certain system. Therefore, its performances depend strongly on a few factors stated in the section 1. In this section, we propose a new design procedure using EA for the systemic design of SOPNN with the optimum performance. In the SOPNN algorithm, the key problems are the determination of the optimal number of input variables, selection of input variables, and selection of the order of the polynomial forming a PD in each node. In [13], these factors are determined in advance by the trial and error method. But in this chapter, these problems are solved by using EA automatically. The EA is implemented using crossover and mutation probability rates for better exploitation of the optimal inputs and order of polynomial in each node of SOPNN. All of the initial EA populations are randomized to use minimum heuristic knowledge. The appropriate inputs and order are evolved accordingly and are tuned gradually throughout the EA iterations. In the evolutionary design procedure, the key issues are the encoding process of the order of the polynomial, the selection of the number of input variables, and the selection of the optimum input variables as a chromosome and the defining of a criterion to compute the fitness of each chromosome. A detailed representation of the coding strategy and choice of fitness function are given in the following sections.
112
D. Kim and G.-T. Park
2.1 Representation of chromosome for appropriate information of each PD When the SOPNN is designed by using EA, the most important consideration is the representation strategy, that is, the process by which the key factors of the SOPNN are encoded into the chromosome. A binary coding is employed for the available design specifications. The order and the inputs of each node in the SOPNN are coded as a finite-length string. Our chromosomes are made of three sub-chromosomes. The first one consists of 2 bits for the order of polynomial (PD), the second one consists of 3 bits for the number of inputs of PD, and the last one consists of N bits, which are equal to the total number of input candidates in the current layer. These input candidates are the node outputs of the previous layer. The representation of binary chromosomes is illustrated in Fig. 1. The 1st sub-chromosome is made of 2 bits, which represent the several types of the order of PD. The relationship between bits in the 1st sub-chromosome and the order of PD is shown in Table 1. Thus, each node can exploit a different order of the polynomial. The 3rd sub-chromosome has N bits, which are concatenated bits of 0’s and 1’s coding. The input candidate is represented by a ‘1’ bit if it is chosen as the input variable to the PD and by a ‘0’ bit it is not chosen. This type of representation solves the problem of which input variables are to be chosen. If many input candidates are chosen for a model design, the modelling becomes computationally complex, and normally requires a lot of time to achieve good results. Also, complex modelling can produce inaccurate results and poor generalizations. Good approximation performance does not necessarily guarantee good The 3rd sub-chromosome: N bits equal to input candidates in the current layer
The 1st sub-chromosome: 2 bits for the order of PD
The 2nd sub-chromosome: 3 bits for the number of inputs of PD
0
1
1
0 0
0
0
1
• • •
1
1
Fig. 1. Structure of binary chromosome for a PD Table 1. Relationship between bits in the 1st sub-chromosome and order of PD Bits in the 1st sub-chromosome
Order of polynomial (PD)
00
Type 1 - Linear
01 10
Type 2 - Quadratic Type 2 - Quadratic
11
Type 3 - Modified quadratic
Evolution of Inductive Self-organizing Networks
113
Table 2. Relationship between bits in the 2nd sub-chromosome and number of inputs to PD Bits in the 2nd sub-chromosome
Number of inputs to PD
000
1
001 010
2 2
011 100
3 3
101 110
4 4
111
5
generalization capability [18]. To overcome this drawback, we introduced the 2nd sub-chromosome into the chromosome. The 2nd sub-chromosome consists of 3 bits and represents the number of input variables to be selected. The number based on the 2nd sub-chromosome is shown in the Table 2. The number of input variables selected for each node among all input candidates is as many as the number represented in the 2nd sub-chromosome. Designer must determine the maximum number, considering the characteristic of system, design specification, and prior knowledge of the model. With this method, we can solve problems such as the conflict between overfitting and generalization and excessive computation time. The relationship between chromosome and PD information is shown in Fig. 2. The PD corresponding to the chromosome in Fig. 2 is described briefly in Fig. 3. Figure 2 shows an example of PD. The various pieces of required information are obtained from its chromosome. The 1st sub-chromosome shows that the polynomial order is Type 2 (quadratic form). The 2nd sub-chromosome identifies two input variables to this node. The 3rd sub-chromosome indicates that x1 and x6 are selected as the input variables. The node with PD corresponding to Fig. 2 is shown in Fig. 3. Thus, the output of this PD, can be expressed as (1). yˆ = f (x1 , x6 ) = c0 + c1 x1 + c2 x6 + c3 x21 + c4 x26 + c5 x1 x6
(1)
where coefficients c0 , c1 , . . . , c5 are evaluated by using the training data set and the standard least square estimation (LSE). Therefore, the polynomial function of PD is formed automatically according to the information provided by the sub-chromosomes. The design procedure of EA-based SOPNN is shown in Fig. 4. At the beginning of the process, the initial populations are comprised of a set of chromosomes that are scattered all over the search space. The populations
114
D. Kim and G.-T. Park Input cadidates
Chromosome
Information on PD
Forming a PD
1 0
1st subchromosome
Order of polynomial
2nd subchromosome
No. of inputs
0 1 0 x1
1
x2
f
selected
^ y
ignored
0
x3
0
x4
0
x5
0
ignored
x6
1
selected
ignored
3rd subchromosome
ignored
Fig. 2. Example of PD whose various pieces of required information is obtained from its chromosome :quadratic (Type 2) x1 x6
2 2
^ y
PD : 2 inputs
Fig. 3. Node with PD corresponding to chromosome in Fig. 2
are all randomly initialized; thus, the use of heuristic knowledge is minimized. The fitness assignment in EA serves as a guide in the search toward the optimal solution. The fitness function for specific cases of modeling will be explained later. After each chromosome is evaluated and associated with a fitness, the current population undergoes the reproduction process to create the next generation of population. The roulette-wheel selection scheme is used to determine the members of the new generation of population. After the new group of population is built, a mating pool is formed for the crossover. The crossover proceeds in three steps. First, two newly reproduced strings are selected from the mating pool, which is produced by reproduction. Second, a position (one point) along the two strings is selected uniformly at random. In the third step, all characters are exchanged by following the crossing site. We use one-point crossover operator with a crossover probability of pc (0.85). This crossover is then followed by the mutation operation. The mutation is the occasional alteration of a value at a particular bit position (we flip the states of a bit from 0 to 1 or vice versa). The mutation serves as an insurance policy for recovering the loss of a particular piece of information (any simple bit). The mutation rate used is fixed at 0.05 (pm ). Generally, after these three operations, the overall fitness of the population improves. Each of the
Evolution of Inductive Self-organizing Networks
115
Start Generation of initial population : the parameters are encoded into a chromosome
The fitness values of the new chromosomes are improved trough generations with genetic operators Reproduction: roulette wheel
Evaluation: each chromosome is evaluated and has its fitness value
A: 0 0 0 0 0 0 0 1 1 1 1
B: 1 1 0 0 0 1 1 0 0 1 1
One-point crossover
NO Termination condition
YES Results: chromosomes which have good fitness value are selected for the new input variables of the next layer
---: crossover site
A: 0 0 0 0 0 0 0 1 1 1 1 B: 1 1 0 0 0 1 1 0 0 1 1 before crossover
A`: 0 0 0 0 0 0 0 0 0 1 1 B`: 1 1 0 0 0 1 1 1 1 1 1 after crossover
Invert mutation ---: mutation site A`: 0 0 0 0 0 0 0 0 0 1 1 A`: 0 0 0 1 0 0 0 0 0 1 1 before mutation
after mutation
End: one chromosome (PD) characterized by the best performance is selected as the output when the 3rd layer is reached
Fig. 4. Block diagram of the design procedure of EA-based SOPNN
population generated then goes through a series of evaluation, reproduction, crossover, and mutation, and the procedure is repeated until a termination condition is reached. After the evolution process, the final generation of population consists of highly fit bits that provide optimal solutions. After the termination condition is satisfied, one chromosome (PD) with the best performance is selected as the output PD in the final generation of the population. All other remaining chromosomes are discarded, and all the nodes that do not influence this output PD in the previous layers are also removed. Finally, the EA-based SOPNN model is obtained. 2.2 Fitness function for modelling In EA, the fitness function must be determined. The genotype representation encodes the problem into a string, and the fitness function measures the performance of the model. It is quite important for evolving systems to find a good fitness measurement. To construct models with superior approximation and generalization ability, we introduce the error function such as E = Θ × P I + (1 − Θ) × EP I
(2)
where Θ ∈ [0, 1] is a weighting factor for PI and EPI, which the performance index values of the training data and testing data, respectively,
116
D. Kim and G.-T. Park
as expressed in (4). Then the fitness value [12] is determined as follows: 1 (3) F = 1+E Maximizing F is identical to minimizing E. The choice of Θ establishes a certain tradeoff between the approximation and generalization ability of the EA-based SOPNN.
3 Simulation Results In this section, we show the performance of our new EA-based SOPNN for two well known nonlinear system modeling. One is a time series of a gas furnace (Box-Jenkins data) [19], which was studied previously in [19–21], and the other is a nonlinear system already exploited in fuzzy modeling [22–26]. 3.1 Gas furnace process The delayed terms of methane gas flow rate u(t) and carbon dioxide density y(t) such as u(t−3), u(t−2), u(t−1), y(t−3), y(t−2), and y(t−1)are used as input variables to the EA-based SOPNN. The actual system output y(t) is used as the target output variable for this model. We choose the input variables of nodes in the 1st layer from these input variables. The total data set consisting of 296 input-output pairs is split into two parts. The first part (consisting of 148 pairs) is used as the training data set, and the remaining part of the data set serves as a testing data set. Using the training data set, the EA-based SOPNN can estimate the coefficients of the polynomial by using the standard LSE. The performance index is defined as the mean squared error 1 (yi − yˆi )2 m i=1 m
P I(EP I) =
(4)
where yi is the actual system output, yˆi is the estimated output of each node, and m is the number of data. The design parameters of EA-based SOPNN for modeling are shown in Table 3. In the 1st layer, 20 chromosomes are generated and evolved during 40 generations, where each chromosome in the population is defined as a corresponding node. So 20 nodes (PDs) are produced in the 1st layer based on the EA operators. All PDs are estimated and evaluated using the training and testing data sets, respectively. They are also evaluated by the fitness function of (3) and ranked according to their fitness value. We choose nodes as many as a predetermined number w from the highest ranking node, and use their outputs as new input variables to the nodes in the next layer. In other words, the chosen PDs (w nodes) must be preserved for the design of the next layer, and the outputs of the preserved PDs serve as inputs to the
Evolution of Inductive Self-organizing Networks
117
Table 3. Design parameters of EA-based SOPNN for modeling Parameters
1st layer
2nd layer
3rd layer
Maximum generations
40
60
80
Population size:(w)
20:(15)
60:(50)
80
String length
11
20
55
Crossover rate (pc )
0.85
Mutation rate (pm )
0.05
Weighting factor (Θ)
0.1˜ 0.9
Type (order)
1˜ 3
Table 4. Values of performance indices of the proposed EA-based SOPNN Weighting factor
1st layer PI – EPI
2nd layer PI – EPI
3rd layer PI – EPI
0.1
0.0214 – 0.1260
0.0200 – 0.1231
0.0199 – 0.1228
0.25
0.0214 – 0.1260
0.0149 – 0.1228
0.0145 – 0.1191
0.5
0.0214 – 0.1260
0.0139 – 0.1212
0.0129 – 0.1086
0.75
0.0214 – 0.1260
0.0139 – 0.1293
0.0138 – 0.1235
0.9
0.0173 – 0.1411
0.0137 – 0.1315
0.0129 – 0.1278
next layer. The value of w is different for each layer, which is also shown in Table 3. This procedure is repeated for the 2nd layer and the 3rd layer. Table 4 summarizes the values of the performance indices, PI and EPI, of the proposed EA-based SOPNN according to the weighting factor. These values are the lowest in each layer. The overall lowest values of the performance indices are obtained at the third layer when the weighting factor is 0.5. If this model is designed to have a fourth or higher layer, the performance values become much lower, and the computation time increases considerably for the model with such a complex network. Fig. 5 depicts the trend of the performance index values produced in successive generations of the EA for the weighting factor Θ of 0.5. Fig. 6 illustrates the values of the error function and fitness function in successive EA generations when Θ = 0.5. Fig. 7 shows the proposed EA-based SOPNN model with 3 layers and its identification performance for Θ = 0.5. The model output follows the actual output very well. The values of the performance indices of the proposed method are equal to PI = 0.012, EPI = 0.108, respectively.
D. Kim and G.-T. Park 0.145
0.022
Performance index(EPI)
Performance index(PI)
118
PI 0.020 0.018 0.016 0.014 1st layer 0.012
0
20
40
2nd layer
60
3rd layer
0.140 0.130 0.125 0.120 0.115 0.110 0.105
80 100 120 140 160 180 Generations
EPI
0.135
2nd layer
1st layer 0
20
40
60
(a)
3rd layer
80 100 120 140 160 180 Generations
(b)
0.945
0.080
Value of fitness function(F)
Value of error function(E)
Fig. 5. (a) performance index for the training data set (b) performance index for the testing data set: Trend of performance index values with respect to generations through layers
0.075 0.070 0.065 1st layer
0.060 0
20
40
2nd layer 60
3rd layer
0.940 0.935 0.930 0.925
80 100 120 140 160 180 Generations
1st layer 0
20
2nd layer
40
60
(a)
3rd layer
80 100 120 140 160 180 Generations
(b)
Fig. 6. (a) error function (b) fitness function : Values of the error function and fitness function for successive generations 1 4 1 5 1 3 3 5 2 5
u(t-3) u(t-2) u(t-1) y(t-3)
PD PD PD 3 5 PD 3 5 PD 2 4 PD
PD PD
2 4 PD
y(t-2)
3 PD 3
^ y
2 5 PD 2 4 PD 1 4 PD
y(t-1)
(a) Actural output Model output
60
2.25
58
1.50
56
0.75
54
Error
CO2 concentration
62
52 50 48
−1.50
46 44
0.00 −0.75
training data 0
50
100
testing data
150 200 Data number
(b)
250
−2.25 300
0
50
100
150 200 Data number
250
300
(c)
Fig. 7. (a) Proposed EA-based SOPNN model with 3 layers, (b) actual output versus model output, (c) error: Proposed EA-based SOPNN model with 3 layers and its identification performance
Evolution of Inductive Self-organizing Networks
119
Fig. 8. (a) Basic SOPNN & Case 1, (b) Modified SOPNN & Case 2: Conventional SOPNN models with 5 layers
For the comparison of the network size of the proposed EA-based SOPNN with that of conventional SOPNN, conventional SOPNN models are visualized in Fig. 8. The structure of the basic SOPNN & Case 1 in Fig. 8(a) is obtained by use of 4 input variables and Type 3 polynomial for every node in all layers to the fifth layer. Its performance is as follows. PI = 0.012 and EPI = 0.084. On the other hand, the structure of the modified SOPNN and Case 2 in Fig. 8(b) is obtained by use of 2 input variables and Type 1 polynomial for every node in the 1st layer and 3 input variables and Type 2 polynomial for every node from the 2nd layer to the 5th layer. In this model, PI is 0.016, and EPI is 0.101. Figures 7 and 8 show that the structure of the EA-based SOPNN is much simpler than that of the conventional SOPNN in terms of the number of nodes and layers, despite their similar performance. Table 5 provides a comparison of the proposed model with other techniques. The comparison is realized on the basis of the same performance index for the training and testing data sets. Additionally, PI represents the performance index of the model for the training data set and EPI for the testing data. The proposed architecture, EA-based SOPNN model, outperforms other models both in terms of accuracy and generalization capability.
120
D. Kim and G.-T. Park Table 5. Values of performance index of some identification models Model
PI
EPI
Kim’s model [20]
0.034
0.244
Lin’s model [21]
0.071
0.261
Kim’s model [12]
0.013
0.126
0.012 0.016
0.084 0.101
0.012
0.108
SOPNN (5 layers) [13]
Basic & case 1 Modified & case 2
EA-based SOPNN (3 layers)
3.2 Three-input nonlinear function This example demonstrates the application of the proposed EA-based SOPNN model to identify highly nonlinear functions. The performance of this model is compared with those of earlier models. The function to be identified is a three-input nonlinear function given by (5) −1 −1.5 2 y = (1 + x0.5 ) 1 + x2 + x3
(5)
which has been widely used by Takagi and Hayashi [22], Sugeno and Kang [23], and Kondo [24] to test their modelling approaches. Table 6 shows 40 pairs input-output data obtained from (5) [26]. The input x4 is a dummy variable, which is not related to (5). The data in Table 6 is divided into the training data set (Nos. 1–20) and the testing data set (Nos. 21–40). To compare the performance, the same performance index, the average percentage error (APE), adopted in [22–26] is used. 1 |yi − yˆi | × 100 m i=1 yi m
AP E =
(6)
where m is the number of data pairs and yi and yˆ1 are the i-th actual output and model output, respectively. Again, a series of comprehensive experiments was conducted, and the results are summarized. The design parameters of EA-based SOPNN for each layer are shown in Table 7. The simulation results of the EA-based SOPNN are summarized in Table 8. The lowest values of the performance indices, PI = 0.188 EPI = 1.087, are obtained at the third layer when the weighting factor (Θ) is 0.25. Figure 9 illustrates the trend of the performance index values produced in successive generations of the EA for the weighting factor Θ of 0.25. Figure 10 shows the values of the error function and fitness function in successive EA generations when Θ is 0.25.
Evolution of Inductive Self-organizing Networks
121
Table 6. Input-output data of three-input nonlinear function No.
x1
x2
x3
x4
y
No.
x1
x2
x3
x4
y
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
1 1 1 1 1 5 5 5 5 5 1 1 1 1 1 5 5 5 5 5
3 5 1 3 5 1 3 5 1 3 5 1 3 5 1 3 5 1 3 5
1 2 3 4 5 4 3 2 1 2 3 4 5 4 3 2 1 2 3 4
1 1 5 5 1 1 5 5 1 1 5 5 1 1 5 5 1 1 5 5
11.11 6.521 10.19 6.043 5.242 19.02 14.15 14.36 27.42 15.39 5.724 9.766 5.87 5.406 10.19 15.39 19.68 21.06 14.15 12.68
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
1 1 1 1 1 5 5 5 5 5 1 1 1 1 1 5 5 5 5 5
1 3 5 1 3 5 1 3 5 1 3 5 1 3 5 1 3 5 1 3
5 4 3 2 1 2 3 4 5 4 3 2 1 2 3 4 5 4 3 2
1 1 5 5 1 1 5 5 1 1 5 5 1 1 5 5 1 1 5 5
9.545 6.043 5.724 11.25 11.11 14.36 19.61 13.65 12.43 19.02 6.38 6.521 16 7.219 5.724 19.02 13.39 12.68 19.61 15.39
Table 7. Design parameters of EA-based SOPNN for modeling Parameters
1st layer
2nd layer 3rd layer
Maximum generations
40
60
80
Population size:(w)
20:(15)
60:(50)
80
String length
8
20
55
Crossover rate (pc )
0.85
Mutation rate (pm )
0.05
Weighting factor (Θ)
0.1˜0.9
Type (order)
1˜3
Figure 11 depicts the proposed EA-based SOPNN model with 3 layers for Θ = 0.25. The structure of EA-based SOPNN is very simple and gives good performance. However, the conventional SOPNN has difficulty in structuring the model for this nonlinear function. Therefore, only a few
122
D. Kim and G.-T. Park Table 8. Values of performance indices of the proposed EA-based SOPNN Weighting factor
1st layer PI – EPI
2nd layer PI – EPI
3rd layer PI – EPI
0.1
5.7845 – 6.8199
2.3895 – 3.3400
2.2837 – 3.1418
0.25
5.7845 – 6.8199
0.8535 – 3.1356
0.1881 – 1.0879
0.5
5.7845 – 6.8199
1.6324 – 5.5291
1.2268 – 3.5526
0.75
5.7845 – 6.8199
1.9092 – 4.0896
0.5634 – 2.2097
0.9
5.7845 – 6.8199
2.5083 – 5.1444
0.0002 – 4.8804
7 PI
5
Performance index(EPI)
Performance index(PI)
6
4 3 2 1 1st layer
0 0
20
2nd layer 40
60
3rd layer
80 100 120 Generations
140
EPI
6 5 4 3 2 1st layer
1 160
180
0
20
2nd layer 40
60
(a)
3rd layer
80 100 120 Generations
140
160
180
(b)
Fig. 9. (a) performance index for the training data set (b) performance index for the testing data set: Trend of performance index values with respect to generations through layers
Value of fitness function(F)
Value of error function(E)
7 6 5 4 3 2 1 0
2nd layer
1st layer 0
20
40
60
3rd layer
80 100 120 Generations
(a)
140
160
180
0.6 0.5 0.4 0.3 0.2 1st layer
0.1 0
20
40
2nd layer 60
80 100 120 Generations
3rd layer 140
160
180
(b)
Fig. 10. (a) error function (b) fitness function : Values of the error function and fitness function for successive generations
number of input candidates are considered. Fig. 12 shows the identification performance and error of the proposed EA-based SOPNN when Θ is 0.25. The output of the EA-based SOPNN follows the actual output very well.
Evolution of Inductive Self-organizing Networks
X1
X2
X3
2 3 2 3 2 3 2 3 2 3 1 3 2 2
123
PD 2 2 3 2 3 3 3 2 3 1
PD PD PD PD PD
PD PD 3 5 PD
PD
y^
PD PD
PD
Fig. 11. Structure of the EA-based SOPNN model with 3 layers (Θ = 0.25)
20
30
Actual output Model output
25
10 5 Errors
20 y_tr
15
15
−5 −10
10
−15
5
−20 5
10 15 Data number
20
5
10 Data number
15
20
5
10 Data number
15
20
20
30
Actual output Model output
15
25
10 5 Errors
20 y_te
0
15
0 −5 −10
10
−15
5
−20 5
10 15 Data number
20
Fig. 12. (a) actual output versus model output of training data, (b) errors of (a)(c) actual output versus model output of testing data, (b) errors of (c): Identification performance and errors of EA-based SOPNN model with 3 layers
Table 9 shows the performances of the proposed EA-based SOPNN model and other models studied in the literature. The experimental results clearly showed that the proposed model outperformed the existing models both in terms of approximation capability (PI) as well as generalization ability (EPI). But the conventional SOPNN cannot be applied properly to the identification of this example.
124
D. Kim and G.-T. Park Table 9. Performance comparison of various identification models Model
PI
GMDH model[24]
EPI
4.7
5.7
Fuzzy model [23]
model 1 model 2
1.5 0.59
2.1 3.4
FNN [26]
type 1 type 2 type 3
0.84 0.73 0.63
1.22 1.28 1.25
GD-FNN [25]
2.11
1.54
SOPNN (5 layers) [13]
Basic & case 1 2.59 8.52 Modified Impossible Impossible
EA-based SOPNN (3 layers)
0.188
1.087
4 Conclusions In this chapter, we proposed a new design methodology of SOPNN using an evolutionary algorithm, which is called the EA-based SOPNN and studied the properties of the EA-based SOPNN. The EA-based SOPNN is a sophisticated and versatile architecture that can construct models from a limited data set and poorly-defined complex problems. Moreover, the architecture of the model is not predetermined, but can be self-organized automatically during the design process. The conflict between overfitting and generalization can be avoided by using a fitness function with a weighting factor. The experimental results showed that the proposed EA-based SOPNN is superior to the conventional SOPNN models as well as other previous models in terms of modeling performance.
5 Acknowledgement The authors would like to thank the financial support of the Korea Science & Engineering Foundation. This work was supported by grant No. R01-2005-00011044-0 from the Basic Research Program of the Korea Science & Engineering Foundation. The authors are also very grateful to the anonymous reviewers for their valuable comments.
Evolution of Inductive Self-organizing Networks
125
APPENDIX SELF-ORGANIZING POLYNOMIAL NEURAL NETWORK [13] This appendix summarizes the design procedure of the conventional SOPNN algorithm. Step 1: Determine system’s input variables We define input variables such as x1i , x2i , . . . , xN i , related to output variables yi , where N and i are the number of all input variables and input-output data set, respectively. The normalization of the input data is also performed if required. Step 2: Form training and testing data The input - output data set is separated into the training (ntr ) data set and the testing (nte ) data set. Then, we have n = ntr + nte . The training data set is used to construct a SOPNN model. And the testing data set is used to evaluate the constructed SOPNN model. Step 3: Choose a structure of the SOPNN The structure of SOPNN is strongly dependent on the number of input variables and the order of PD in each layer. Two kinds of SOPNN structures, namely, the basic SOPNN structure and the modified SOPNN structure are available. Each of them is specified with two cases. (a) Basic SOPNN structure - The number of input variables of PDs is the same in every layer. Case 1. The polynomial order of the PDs is the same in each layer of the network. Case 2. The polynomial order of the PDs in the 2nd or higher layer is different from the one of PDs in the 1st layer. (b) Modified SOPNN structure - The number of input variables of PDs varies from layer to layer. Case 1. The polynomial order of the PDs is same in every layer. Case 2. The polynomial order of the PDs in the 2nd layer or higher is different from the one of PDs in the 1st layer. Step 4: Determine the number of input variables and the order of the polynomial forming a PD We determine arbitrarily the number of input variables and the type of the polynomial in PDs. The polynomials are different according to the number of input variables and the polynomial order. The total number of PDs located at the current layer is determined by the number of the selected input variables (r) from the nodes of the preceding layer, because the outputs of the nodes of the preceding layer become the input variables to the current layer. The total number of PDs in the current layer is equal to the combination, that is N! r!(N −r)! , where N is the number of nodes in the preceding layer. Step 5: Estimate the coefficients of the PD The vector of coefficients of the PDs is determined using standard mean squared errors (MSE) by minimizing the following index
126
D. Kim and G.-T. Park Choice of estimated models / stop conditions
Possible inputs
x1i
j th layer
1
PD
x2i
Z1 1
x3i
PD
x4i
PD
xNi
z
j-1
1
PD
Z2
∧
1 Z N!/{(N-r)!r!}
xN-1i
Optimal model
yi
z j-1 p z j-1q
zi PD
selected inputs: (j-1)th layer
z j-1p z j-1q
z j-1p , z j-1q
order Type 2
zi
PD: j th layer c0+c1z j-1p+c2z j-1q+c3(zj-1p)2+c4(zj-1q)2+c5zj-1pzj-1q
Fig. 13. Overall architecture of the SOPNN
Ek =
ntr 1 (yi − zki )2 , ntr i=1
k = 1, 2, . . . ,
N! r!(N − r)!
(7)
where, zki denotes the output of the k-th node with respect to the i-th data. This step is completed repeatedly for all the nodes in the current layer and, subsequently, all layers of the SOPNN starting from the input to the output layer. Step 6: Select PDs with the good predictive capability The predictive capability of each PD is evaluated by the performance index ! using the testing data set. Then, we choose w number of PDs among r!(NN−r)! PDs in the order of the best predictive capability (the lowest value of the performance index). Here, w is the pre-defined number of PDs that must be preserved to next layer. The outputs of the chosen PDs serve as inputs to the next layer. Step 7: Check the stopping criterion The SOPNN algorithm terminates when the number of layers predetermined by the designer is reached. Step 8: Determine new input variables for the next layer If the stopping criterion is not satisfied, the next layer is constructed by repeating step 4 through step 8. The overall architecture of the SOPNN is shown in Fig. 13.
References 1. Hornik K, Stinchcombe M, White H (1989) Multilayer feedforward networks are universal approximators. IEEE Trans Neural Netw 2:359–366
Evolution of Inductive Self-organizing Networks
127
2. Chen T, Chen H (1995) Approximation capability to functions of several variables, nonlinear functions, and operators by radial basis function neural networks. IEEE Trans Neural Netw 6:904–910 3. Li K (1992) Approximation theory and recurrent networks. Proc IJCNN 2: 266–271 4. Wang LX, Mendel JM (1992) Generating fuzzy rules by learning from examples IEEE Trans Syst Man Cybern 22:1414–1427 5. Wang LX, Mendel JM (1992) Fuzzy basis function, universal approximation, and orthogonal least-squares learning. IEEE Trans Neural Netw 3:807–814 6. Ivakhnenko AG (1971) Polynomial theory of complex systems. IEEE Trans Syst Man Cybern SMC-1:364–378 7. Ivakhnenko AG, Ivakhnenko NA (1974) Long-term prediction by GMDH algorithms using the unbiased criterion and the balance-of-variables criterion (1974) Sov Automat Control 7:40–45 8. Ivakhnenko AG, Ivakhnenko NA (1975) Long-term prediction by GMDH algorithms using the unbiased criterion and the balance-of-variables criterion. Part 2 Sov. Automat Control 8:24–38 9. Ivakhnenko AG, Vysotskiy VN, Ivakhnenko NA (1978) Principal version of the minimum bias criterion for a model and an investigation of their noise immunity. Sov Automat Control 11:27–45 10. Ivakhnenko AG, Krotov GI, Ivakhnenko NA (1970) Identification of the mathematical model of a complex system by the self-organization method. In: Halfon E (ed.) Advances and case studies Theoretical Systems Ecology, Academic, New York 11. Farlow SJ (1984) Self-organizing methods in modeling: GMDH type-algorithms. Marcel Dekker, New York 12. Kim DW (2002) Evolutionary design of self-organizing polynomial neural networks. MA Thesis, Wonkwang University, Korea 13. Oh SK, Pedrycz W (2002) The design of self-organizing polynomial neural networks. Inf Sci 141:237–258 14. Shi Y, Eberhart R, Chen Y (1999) Implementation of evolutionary fuzzy systems. IEEE Trans Fuzzy syst 7:109–119 15. Kristinnson K, Dumont GA (1992) System identification and control using genetic algorithms. IEEE Trans Syst Man Cybern 22:1033–1046 16. Uckun S, Bagchi S, Kawamura K (1993) Managing genetic search in job shop scheduling. IEEE Expert 8:15–24 17. Goldberg DE (1989) Genetic algorithms in search, optimization and machine learning. Addison-Wesley, Reading, MA 18. Kung SY, Taur JS (1995) Decision-based neural networks with signal/image classification applications. IEEE Trans Neural Netw 6:170–181 19. Box GEP, Jenkins FM (1976) Time series analysis: forecasting and control. Holden-day, Sanfrancisco, CA 20. Kim E, Lee H, Park M, Park M (1998) A simple identified Sugeno-type fuzzy model via double clustering. Inf Sci 110:25–39 21. Lin Y, Cunningham GA (1995) A new approach to fuzzy-neural modeling. IEEE Trans Fuzzy Syst 3:190–197 22. Takagi H, Hayashi I (1991) NN-driven fuzzy reasoning. Int J Approx Reasoning 5:191–212 23. Sugeno M, Kang GT (1988) Structure identification of fuzzy model. Fuzzy Sets Syst 28:15–33
128
D. Kim and G.-T. Park
24. Kondo T (1986) Revised GMDH algorithm estimating degree of the complete polynomial. Trans Soc Instrum Control Eng 22:928–934 25. Wu S, Er MJ, Gao Y (2001) A fast approach for automatic generation of fuzzy rules by generalized dynamic fuzzy neural networks. IEEE Trans Fuzzy Syst 9:578–594 26. Horikawa SI, Furuhashi T, Uchikawa Y (1992) On fuzzy modeling using fuzzy neural networks with the back-propagation algorithm. IEEE Trans Neural Netw 3:801–806
Recursive Pattern based Hybrid Supervised Training Kiruthika Ramanathan and Sheng Uei Guan
Summary. We propose, theorize and implement the Recursive Pattern-based Hybrid Supervised (RPHS) learning algorithm. The