The Sixth International Symposium on Neural Networks (ISNN 2009) Advances in Intelligent and Soft Computing

Advances in Intelligent and Soft Computing Editor-in-Chief: J. Kacprzyk

56

Advances in Intelligent and Soft Computing Editor-in-Chief Prof. Janusz Kacprzyk Systems Research Institute Polish Academy of Sciences ul. Newelska 6 01-447 Warsaw Poland E-mail: [email protected] Further volumes of this series can be found on our homepage: springer.com Vol. 42. O. Castillo, P. Melin, O. Montiel Ross, R. Sepúlveda Cruz, W. Pedrycz, J. Kacprzyk (Eds.) Theoretical Advances and Applications of Fuzzy Logic and Soft Computing, 2007 ISBN 978-3-540-72433-9

Vol. 50. J.M. Corchado, S. Rodriguez, J. Llinas, J.M. Molina (Eds.) International Symposium on Distributed Computing and Artificial Intelligence 2008 (DCAI 2008), 2009 ISBN 978-3-540-85862-1

Vol. 43. K.M. W˛egrzyn-Wolska, P.S. Szczepaniak (Eds.) Advances in Intelligent Web Mastering, 2007 ISBN 978-3-540-72574-9

Vol. 51. J.M. Corchado, D.I. Tapia, J. Bravo (Eds.) 3rd Symposium of Ubiquitous Computing and Ambient Intelligence 2008, 2009 ISBN 978-3-540-85866-9

Vol. 44. E. Corchado, J.M. Corchado, A. Abraham (Eds.) Innovations in Hybrid Intelligent Systems, 2007 ISBN 978-3-540-74971-4 Vol. 45. M. Kurzynski, E. Puchala, M. Wozniak, A. Zolnierek (Eds.) Computer Recognition Systems 2, 2007 ISBN 978-3-540-75174-8 Vol. 46. V.-N. Huynh, Y. Nakamori, H. Ono, J. Lawry, V. Kreinovich, H.T. Nguyen (Eds.) Interval / Probabilistic Uncertainty and Non-classical Logics, 2008 ISBN 978-3-540-77663-5 Vol. 47. E. Pietka, J. Kawa (Eds.) Information Technologies in Biomedicine, 2008 ISBN 978-3-540-68167-0 Vol. 48. D. Dubois, M. Asunción Lubiano, H. Prade, M. Ángeles Gil, P. Grzegorzewski, O. Hryniewicz (Eds.) Soft Methods for Handling Variability and Imprecision, 2008 ISBN 978-3-540-85026-7 Vol. 49. J.M. Corchado, F. de Paz, M.P. Rocha, F. Fernández Riverola (Eds.) 2nd International Workshop on Practical Applications of Computational Biology and Bioinformatics (IWPACBB 2008), 2009 ISBN 978-3-540-85860-7

Vol. 52. E. Avineri, M. Köppen, K. Dahal, Y. Sunitiyoso, R. Roy (Eds.) Applications of Soft Computing, 2009 ISBN 978-3-540-88078-3 Vol. 53. E. Corchado, R. Zunino, P. Gastaldo, Á. Herrero (Eds.) Proceedings of the International Workshop on Computational Intelligence in Security for Information Systems CISIS 2008, 2009 ISBN 978-3-540-88180-3 Vol. 54. B.-y. Cao, C.-y. Zhang, T.-f. Li (Eds.) Fuzzy Information and Engineering, 2009 ISBN 978-3-540-88913-7 Vol. 55. Y. Demazeau, J. Pavón, J.M. Corchado, J. Bajo (Eds.) 7th International Conference on Practical Applications of Agents and Multi-Agent Systems (PAAMS 2009), 2009 ISBN 978-3-642-00486-5 Vol. 56. H. Wang, Y. Shen, T. Huang, Z. Zeng (Eds.) The Sixth International Symposium on Neural Networks (ISNN 2009), 2009 ISBN 978-3-642-01215-0

Hongwei Wang, Yi Shen, Tingwen Huang, Zhigang Zeng (Eds.)

The Sixth International Symposium on Neural Networks (ISNN 2009)

ABC

Editors Hongwei Wang Department of Control Science and Engineering Huazhong University of Science and Technology No. 1037, Luoyu Road Wuhan, Hubei, 430074 China Yi Shen Department of Control Science and Engineering Huazhong University of Science and Technology No. 1037, Luoyu Road Wuhan, Hubei, 430074 China

ISBN 978-3-642-01215-0

Tingwen Huang Texas A&M University at Qatar PO Box 23874 Doha Qatar E-mail: tingwen.huang@ qatar.tamu.edu Zhigang Zeng Department of Control Science and Engineering Huazhong University of Science and Technology No. 1037, Luoyu Road Wuhan, Hubei, 430074 China E-mail: [email protected]

e-ISBN 978-3-642-01216-7

DOI 10.1007/978-3-642-01216-7 Advances in Intelligent and Soft Computing

ISSN 1867-5662

Library of Congress Control Number: Applied for c 2009 

Springer-Verlag Berlin Heidelberg

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable for prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.

Typeset & Cover Design: Scientific Publishing Services Pvt. Ltd., Chennai, India. Printed in acid-free paper 543210 springer.com

Preface

This volume of Advances in Soft Computing and Lecture Notes in Computer Science vols. 5551, 5552 and 5553, constitute the Proceedings of the 6th International Symposium of Neural Networks (ISNN 2009) held in Wuhan, China during May 26–29, 2009. ISNN is a prestigious annual symposium on neural networks with past events held in Dalian (2004), Chongqing (2005), Chengdu (2006), Nanjing (2007) and Beijing (2008). Over the past few years, ISNN has matured into a well-established series of international conference on neural networks and their applications to other fields. Following this tradition, ISNN 2009 provided an academic forum for the participants to disseminate their new research findings and discuss emerging areas of research. Also, it created a stimulating environment for the participants to interact and exchange information on future research challenges and opportunities of neural networks and their applications. ISNN 2009 received 1,235 submissions from about 2,459 authors in 29 countries and regions (Australia, Brazil, Canada, China, Democratic People's Republic of Korea, Finland, Germany, Hong Kong, Hungary, India, Islamic Republic of Iran, Japan, Jordan, Macao, Malaysia, Mexico, Norway, Qatar, Republic of Korea, Singapore, Spain, Taiwan, Thailand, Tunisia, United Kingdom, United States, Venezuela, Vietnam, and Yemen) across six continents (Asia, Europe, North America, South America, Africa, and Oceania). Based on rigorous reviews by the Program Committee members and reviewers, 95 high-quality papers were selected to be published in this volume. These papers cover all major topics of the theoretical research, empirical study and applications of neural network research. In addition to the contributed papers, the ISNN 2009 technical program included five plenary speeches by Anthony Kuh (University of Hawaii at Manoa, USA), Jose C. Principe (University of Florida, USA), Leszek Rutkowski (Technical University of Czestochowa, Poland), Fei-Yue Wang (Institute of Automation, Chinese Academy of Sciences, China) and Cheng Wu (Tsinghua University, China). Furthermore, the ISNN 2009 also featured five special sessions focusing on emerging topics of neural network research. ISNN 2009 would not have achieved its success without the support and contributions of many volunteers and organizations. We would like to express our sincere thanks to the Huazhong University of Science and Technology, The Chinese University of Hong Kong, and the National Natural Science Foundation of China for their sponsorship, to the IEEE Wuhan Section, the IEEE Computational Intelligence Society, the International Neural Network Society, the Asia Pacific Neural Network Assembly, and the European Neural Network Society for their

VI

Preface

technical co-sponsorship, and to the Systems Engineering Society of Hubei Province and the IEEE Hong Kong Joint Chapter on Robotics & Automation and Control Systems for their logistic co-operations. We would also like to sincerely thank the General Chair and General Co-Chairs for their overall organization of the symposium. Also we would like to thank the members of the Advisory Committee and Steering Committee for their invaluable assistance and guidance in enhancing the scientific level of the event, the members of the Program Committee and additional reviewers for reviewing the papers, and members of the Publications Committee for checking the accepted papers in a short period of time. Particularly, we deeply appreciate Prof. Janusz Kacprzyk (Editor-in-Chief), Dr. Thomas Ditzinger (Senior Editor, Engineering/Applied Sciences) and other Springer-Verlag staff for their help and collaboration in this demanding scientific publication project – it is always a great pleasure to work with them. There are still many more colleagues, associates, friends, and supporters who helped us in many ways, we would like to say “Thank you so much” to all of them. We also would like to express our heartfelt gratitude to the plenary and panel speakers for their vision and discussions on the state-of-the-art research development in the field as well as promising future research directions, opportunities, and challenges. Last but not least, we would like to express our most cordial thanks to all of the authors of the papers constituting this volume in Advances in Soft Computing: it is the excellence of their research work that gives value to the book.

May 2009

Hongwei Wang Yi Shen Tingwen Huang Zhigang Zeng

Organization

General Chair Shuzi Yang, China

General Co-chairs Youlun Xiong, China Yongchuan Zhang, China

Advisory Committee Chairs Shoujue Wang, China Paul J. Werbos, USA

Advisory Committee Members Shun-ichi Amari, Japan Zheng Bao, China Tianyou Chai, China Guanrong Chen, China Shijie Cheng, China Ruwei Dai, China Jay Farrell, USA Chunbo Feng, China Russell Eberhart, USA David Fogel, USA Walter J. Freeman, USA Kunihiko Fukushima, Japan Marco Gilli, Italy Aike Guo, China Xingui He, China Zhenya He, China Petros Loannou, USA Janusz Kacprzyk, Poland

VIII

Nikola Kasabov, New Zealand Okyay Kaynak, Turkey Frank L. Lewis, USA Deyi Li, China Yanda Li, China Chin-Teng Lin, Taiwan Robert J. Marks II, USA Erkki Oja, Finland Nikhil R. Pal, India Marios M. Polycarpou, USA Leszek Rutkowsk, Poland Jennie Si, USA Youxian Sun, China Joos Vandewalle, Belgium DeLiang Wang, USA Fei-Yue Wang, USA Donald C. Wunsch II, USA Lei Xu, China Xin Yao, UK Gary G. Yen, USA Bo Zhang, China Nanning Zheng, China Jacek M. Zurada, USA

Steering Committee Chairs Jun Wang, Hong Kong Derong Liu, China

Steering Committee Members Jinde Cao, China Shumin Fei, China Chengan Guo, China Min Han, China Zeng-Guang Hou, China Xiaofeng Liao, China Bao-Liang Lu, China Fuchun Sun, China Zhang Yi, China Fuliang Yin, China Hujun Yin, UK Huaguang Zhang, China Jianwei Zhang, Germany

Organization

Organization

Organizing Committee Chairs Hongwei Wang, China Jianzhong Zhou, China Yi Shen, China

Program Committee Chairs Wen Yu, Mexico Haibo He, USA Nian Zhang, USA

Special Sessions Chairs Sanqing Hu, USA Youshen Xia, China Yunong Zhang, China

Publications Chairs Xiaolin Hu, China Minghui Jiang, China Qingshan Liu, China

Publicity Chairs Tingwen Huang, Qatar Paul S. Pang, New Zealand Changyin Sun, China

Finance Chair Xiaoping Wang, China

Registration Chairs Charlie C.L. Wang, China Zhenyuan Liu, China Weifeng Zhu, China

Local Arrangements Chairs Zhigang Zeng, China Chao Qi, China Liu Hong, China

IX

X

Organization

Program Committee Members José Alfredo, Brazil Sabri Arik, Turkey Xindi Cai, USA Yu Cao, USA Matthew Casey, UK Emre Celebi, USA Jonathan Chan, Thailand Sheng Chen, UK Yangquan Chen, USA Ji-Xiang Du, China Hai-Bin Duan, China Andries Engelbrecht, South Africa Péter érdi, USA Jufeng Feng, China Chaojin Fu, China Wai Keung Fung, Canada Erol Gelenbe, UK Xinping Guan, China Chengan Guo, China Ping Guo, China Qing-Long Han, Australia Hanlin He, China Daniel Ho, Hong Kong Zhongsheng Hou, China Huosheng Hu, UK Jinglu Hu, Japan Junhao Hu, China Marc van Hulle, Belgium Danchi Jiang, Australia Haijun Jiang, China Shunshoku Kanae, Japan Rhee Man Kil, Republic of Korea Sungshin Kim, Korea Arto Klami, Finland Rakhesh Singh Kshetrimayum, India Hon Keung Kwan, Canada Chuandong Li, China Kang Li, UK Li Li, China Michael Li, Australia Ping Li, Hong Kong Shutao Li, China

Xiaoli Li, UK Xiaoou Li, Mexico Yangmin Li, Macao Hualou Liang, USA Jinling Liang, China Wudai Liao, China Alan Liew, Australia Ju Liu, China Li Liu, USA Meiqin Liu, China Wenxin Liu, USA Yan Liu, USA Jianquan Lu, Hong Kong Jinhu Lu, China Wenlian Lu, China Jinwen Ma, China Ikuko Nishkawa, Japan Seiichi Ozawa, Japan Jaakko Peltonen, Finland Juan Reyes, Mexico Jose de Jesus Rubio, Mexico Eng. Sattar B. Sadkhan, Iraq Gerald Schaefer, UK Michael Small, Hong Kong Qiankun Song, China Humberto Sossa, Mexico Bingyu Sun, China Norikazu Takahashi, Japan Manchun Tan, China Ying Tan, China Christos Tjortjis, UK Michel Verleysen, Belgium Bing Wang, UK Dan Wang, China Dianhui Wang, Australia Meiqing Wang, China Rubin Wang, China Xin Wang, China Zhongsheng Wang, China Jinyu Wen, China Wei Wu, China Degui Xiao, China

Organization

Rui Xu, USA Yingjie Yang, UK Kun Yuan, China

XI

Xiaoqin Zeng, China Jie Zhang, UK Liqing Zhang, China

Publications Committee Members Guici Chen Huangqiong Chen Shengle Fang Lizhu Feng Junhao Hu Feng Jiang Bin Li Yanling Li Mingzhao Li Lei Liu Xiaoyang Liu Cheng Wang Xiaohong Wang

Zhikun Wang Shiping Wen Ailong Wu Yongbo Xia Li Xiao Weina Yang Zhanying Yang Tianfeng Ye Hongyan Yin Lingfa Zeng Yongchang Zhang Yongqing Zhao Song Zhu

Technical Committee Members Helena Aidos Antti Ajanki Tholkappia AraSu Hyeon Bae Tao Ban Li Bin Binghuang Cai Lingru Cai Xindi Cai Qiao Cai Chao Cao Hua Cao Jinde Cao Kai Cao Wenbiao Cao Yuan Cao George Cavalcanti Lei Chang Mingchun Chang Zhai Chao Cheng Chen Gang Chen Guici Chen

Ke Chen Jiao Chen Lei Chen Ming Chen Rongzhang Chen Shan Chen Sheng Chen Siyue Chen TianYu Chen Wei Chen Xi Chen Xiaochi Chen Xiaofeng Chen XinYu Chen Xiong Chen Xuedong Chen Yongjie Chen Zongzheng Chen Hao Cheng Jian Cheng Long Cheng Zunshui Cheng Rong Chu

XII

Bianca di Angeli C.S. Costa Jose Alfredo Ferreira Costa Dadian Dai Jianming Dai Jayanta Kumar Debnath Spiros Denaxas Chengnuo Deng Gang Deng Jianfeng Deng Kangfa Deng Zhipo Deng Xiaohua Ding Xiuzhen Ding Zhiqiang Dong Jinran Du Hongwu Duan Lijuan Duan Xiaopeng Duan Yasunori Endo Andries Engelbrecht Tolga Ensari Zhengping Fan Fang Fang Haitao Fang Yuanda Fang June Feng Lizhu Feng Yunqing Feng Avgoustinos Filippoupolitis Liang Fu Ruhai Fu Fang Gao Lei Gao Ruiling Gao Daoyuan Gong Xiangguo Gong Fanji Gu Haibo Gu Xingsheng Gu Lihe Guan Jun Guo Songtao Guo Xu Guo Fengqing Han Pei Han Qi Han

Organization

Weiwei Han Yishan Han Yunpeng Han Hanlin He Jinghui He Rui He Shan He Tonejun He Tongjun He Wangli He Huosheng Hu Li Hong Liu Hong Ruibing Hou Cheng Hu Jin Hu Junhao Hu Hao Hu Hui Hu Ruibin Hu Sanqing Hu Xiaolin Hu Xiaoyan Hu Chi Huang Darong Huang Diqiu Huang Dongliang Huang Gan Huang Huayong Huang Jian Huang Li Huang Qifeng Huang Tingwen Huang Zhangcan Huang Zhenkun Huang Zhilin Huang Rey-Chue Hwang Sae Hwang Hui Ji Tianyao Ji Han Jia Danchi Jiang Shaobo Jiang Wei Jiang Wang Jiao Xianfa Jiao

Organization

Yiannis Kanellopoulos Wenjing Kang Anthony Karageorgos Masanori KaWakita Haibin Ke Seong-Joo Kim Peng Kong Zhanghui Kuang Lingcong Le Jong Min Lee Liu Lei Siyu Leng Bing Li Changping Li Chuandong Li Hui Li Jian Li Jianmin Li Jianxiang Li Kelin Li Kezan Li Lei Li Li Li Liping Li Lulu Li Ming Li Na Li Ping Li Qi Li Song Li Weiqun Li Wenlong Li Wentian Li Shaokang Li Shiying Li Tian Li Wei Li Wu Li Xiang Li Xiaoli Li Xiaoou Li Xin Li Xinghai Li Xiumin Li Yanlin Li Yanling Li

XIII

Yong Li Yongfei Li Yongmin Li Yuechao Li Zhan Li Zhe Li Jinling Liang Wudai Liao Wei Lin Zhihao Lin Yunqing Ling Alex Liu Bo Liu Da Liu Dehua Li Dayuan Liu Dongbing Liu Desheng Liu F.C. Liu Huaping Liu Jia Liu Kangqi Liu Li Liu Ming Liu Qian Liu Qingshan Liu Shangjin Liu Shenquan Liu Shi Liu Weiqi Liu Xiaoyang Liu Xiuquan Liu Xiwei Liu XinRong Liu Yan Liu Yang Liu Yawei Liu Yingju Liu Yuxi Liu Zhenyuan Liu Zijian Liu Yimin Long Georgios Loukas Jinhu Lu Jianquan Lu Wen Lu

XIV

Wenlian Lu Wenqian Lu Tongting Lu Qiuming Luo Xucheng Luo Chaohua Ma Jie Ma Liefeng Ma Long Ma Yang Ma Zhiwei Ma Xiaoou Mao Xuehui Mei Xiangpei Meng Xiangyu Meng Zhaohui Meng Guo Min Rui Min Yuanneng Mou Junichi Murata Puyan Nie Xiushan Nie Gulay Oke Ming Ouyang Yao Ouyang Seiichi Ozawa Neyir Ozcan Joni Pajarinen Hongwei Pan Linqiang Pan Yunpeng Pan Tianqi Pang Kyungseo Park Xiaohan Peng Zaiyun Peng Gao Pingan Liquan Qiu Jianlong Qiu Tapani Raiko Congjun Rao Fengli Ren Jose L. Rosseilo Gongqin Ruan Quan Rui Sattar B. Sadkhan Renato Jose Sassi Sassi

Organization

Sibel Senan Sijia Shao Bo Shen Enhua Shen Huayu Shen Meili Shen Zifei Shen Dianyang Shi Jinrui Shi Lisha Shi Noritaka Shigei Atsushi Shimada Jiaqi Song Wen Song Yexin Song Zhen Song Zhu Song Gustavo Fontoura de Souza Kuo-Ho Su Ruiqi Su Cheng Sun Dian Sun Junfeng Sun Lisha Sun Weipeng Sun Yonghui Sun Zhaowan Sun Zhendong Sun Manchun Tan Xuehong Tan Yanxing Tan Zhiguo Tan Bing Tang Hao Tang Yili Tang Gang Tian Jing Tian Yuguang Tian Stelios Timotheou Shozo Tokinaga Jun Tong Joaquin Torres Sospedra Hiroshi Wakuya Jin Wan B.H. Wang Cheng Wang

Organization

Fan Wang Fen Wang Gang Wang Gaoxia Wang Guanjun Wang Han Wang Heding Wang Hongcui Wang Huayong Wang Hui Wang Huiwei Wang Jiahai Wang Jian Wang Jin Wang Juzhi Wang Kai Wang Lan Wang Lili Wang Lu Wang Qilin Wang Qingyun Wang Suqin Wang Tian Wang Tianxiong Wang Tonghua Wang Wei Wang Wenjie Wang Xiao Wang Xiaoping Wang Xiong Wang Xudong Wang Yang Wang Yanwei Wang Yao Wang Yiping Wang Yiyu Wang Yue Wang Zhanshan Wang Zhengxia Wang Zhibo Wang Zhongsheng Wang Zhihui Wang Zidong Wang Zhuo Wang Guoliang Wei Li Wei

XV

Na Wei Shuang Wei Wenbiao Wei Yongchang Wei Xiaohua Wen Xuexin Wen Junmei Weng Yixiang Wu You Wu Huaiqin Wu Zhihai Wu Bin Xia Weiguo Xia Yonghui Xia Youshen Xia Zhigu Xia Zhiguo Xia Xun Xiang Chengcheng Xiao Donghua Xiao Jiangwen Xiao Yongkang Xiao Yonkang Xiao Yong Xie Xiaofei Xie Peng Xin Chen Xiong Jinghui Xiong Wenjun Xiong Anbang Xu Chen Xu Hesong Xu Jianbing Xu Jin Xu Lou Xu Man Xu Xiufen Yu Yan Xu Yang Xu Yuanlan Xu Zhaodong Xu Shujing Yan Dong Yang Fan Yang Gaobo Yang Lei Yang

XVI

Sihai Yang Tianqi Yang Xiaolin Yang Xing Yang Xue Yang Yang Yang Yongqing Yang Yiwen Yang Hongshan Yao John Yao Xianfeng Ye Chenfu Yi Aihua Yin Lewen Yin Qian Yin Yu Ying Xu Yong Yuan You Shuai You Chenglong Yu Liang Yu Lin Yu Liqiang Yu Qing Yu Yingzhong Yu Zheyi Yu Jinhui Yuan Peijiang Yuan Eylem Yucel Si Yue Jianfang Zeng Lingjun Zeng Ming Zeng Yi Zeng Zeyu Zhang Zhigang Zeng Cheng Zhang Da Zhang Hanling Zhang Haopeng Zhang Kaifeng Zhang Jiacai Zhang Jiajia Zhang Jiangjun Zhang Jifan Zhang Jinjian Zhang

Organization

Liming Zhang Long Zhang Qi Zhang Rui Zhang Wei Zhang Xiaochun Zhang Xiong Zhang Xudong Zhang Xuguang Zhang Yang Zhang Yangzhou Zhang Yinxue Zhang Yunong Zhang Zhaoxiong Zhang YuanYuan Bin Zhao Jin Zhao Le Zhao Leina Zhao Qibin Zhao Xiaquan Zhao Zhenjiang Zhao Yue Zhen Changwei Zheng Huan Zheng Lina Zheng Meijun Zheng Quanchao Zheng Shitao Zheng Ying Zheng Xun Zheng Lingfei Zhi Ming Zhong Benhai Zhou Jianxiong Zhou Jiao Zhou Jin Zhou Jinnong Zhou Junming Zhou Lin Zhou Rong Zhou Song Zhou Xiang Zhou Xiuling Zhou Yiduo Zhou Yinlei Zhou

Organization

Yuan Zhou Zhenqiao Zhou Ze Zhou Zhouliu Zhou Haibo Zhu Ji Zhu

XVII

Jiajun Zhu Tanyuan Zhu Zhenqian Zhu Song Zhu Xunlin Zhu Zhiqiang Zuo

Contents

Session 1: Theoretical Analysis The Initial Alignment of SINS Based on Neural Network . . . . Tingjun Li

1

Analysis on Basic Conceptions and Principles of Human Cognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xiaorui Zhang, Minyong Li, Zhong Liu, Meng Zhang

7

Global Exponential Stability for Discrete-Time BAM Neural Network with Variable Delay . . . . . . . . . . . . . . . . . . . . . . . . Xiaochun Lu

19

The Study of Project Cost Estimation Based on Cost-Significant Theory and Neural Network Theory . . . . . . . . Xinzheng Wang, Liying Xing, Feng Lin

31

Global Exponential Stability of High-Order Hopfield Neural Networks with Time Delays . . . . . . . . . . . . . . . . . . . . . . . . . Jianlong Qiu, Quanxin Cheng

39

Improved Particle Swarm Optimization for RCP Scheduling Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Qiang Wang, Jianxun Qi

49

Exponential Stability of Reaction-Diffusion Cohen-Grossberg Neural Networks with S-Type Distributed Delays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yonggui Kao, Shuping Bao

59

XX

Contents

Global Exponential Robust Stability of Static Reaction-Diffusion Neural Networks with S-Type Distributed Delays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shuping Bao A LEC-and-AHP Based Hazard Assessment Method in Hydroelectric Project Construction . . . . . . . . . . . . . . . . . . . . . . . . . . Jian-lan Zhou, Da-wei Tang, Xian-rong Liu, Sheng-yu Gong A Stochastic Lotka-Volterra Model with Variable Delay . . . . . Yong Xu, Song Zhu, Shigeng Hu

69

81 91

Extreme Reformulated Radial Basis Function Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 Gexin Bi, Fang Dong Research of Nonlinear Combination Forecasting Model for Insulators ESDD Based on Wavelet Neural Network . . . . . . . . . 111 Haiyan Shuai, Jun Wu, Qingwu Gong Parameter Tuning of MLP Neural Network Using Genetic Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 Meng Joo Er, Fan Liu Intelligent Grid of Computations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 Samia Jones Method of Solving Matrix Equation and Its Applications in Economic Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 Qingfang Zhang, June Liu Efficient Feature Selection Algorithm Based on Difference and Similitude Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 Weibing Wu, Zhangyan Xu, June Liu Exponential Stability of Neural Networks with Time-Varying Delays and Impulses . . . . . . . . . . . . . . . . . . . . . . . . . . 153 Haydar Ak¸ca, Val´ery Covachev, Kumud Singh Altmayer

Session 2: Machine Learning Adaptive Higher Order Neural Networks for Effective Data Mining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 Shuxiang Xu, Ling Chen

Contents

XXI

Exploring Cost-Sensitive Learning in Domain Based Protein-Protein Interaction Prediction . . . . . . . . . . . . . . . . . . . . . . . 175 Weizhao Guo, Yong Hu, Mei Liu, Jian Yin, Kang Xie, Xiaobo Yang An Efficient and Fast Algorithm for Estimating the Frequencies of 2-D Superimposed Exponential Signals in Presence of Multiplicative and Additive Noise . . . . . . . . . . . . . . . 185 Jiawen Bian, Hongwei Li, Huiming Peng, Jing Xing An Improved Greedy Based Global Optimized Placement Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197 Luo Zhong, Kejing Wang, Jingling Yuan, Jingjing He An Alternative Fast Learning Algorithm of Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205 Pin-Hsuan Weng, Chih-Chien Huang, Yu-Ju Chen, Huang-Chu Huang, Rey-Chue Hwang Computer Aided Diagnosis of Alzheimer’s Disease Using Principal Component Analysis and Bayesian Classifiers . . . . . . 213 ´ Miriam L´ opez, Javier Ram´ırez, Juan M. G´ orriz, Ignacio Alvarez, Diego Salas-Gonzalez, Fermin Segovia, Carlos Garc´ıa Puntonet Margin-Based Transfer Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 Bai Su, Wei Xu, Yidong Shen

Session 3: Support Vector Machines and Kernel Methods Nonlinear Dead Zone System Identification Based on Support Vector Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235 Jingyi Du, Mei Wang A SVM Model Selection Method Based on Hybrid Genetic Algorithm and Empirical Error Minimization Criterion . . . . . . 245 Xin Zhou, Jianhua Xu An SVM-Based Mandarin Pronunciation Quality Assessment System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255 Fengpei Ge, Fuping Pan, Changliang Liu, Bin Dong, Shui-duen Chan, Xinhua Zhu, Yonghong Yan An Quality Prediction Method of Injection Molding Batch Processes Based on Sub-Stage LS-SVM . . . . . . . . . . . . . . . . . . . . . . 267 XiaoPing Guo, Chao Zhang, Li Wang, Yuan Li

XXII

Contents

Soft Sensing for Propylene Purity Using Partial Least Squares and Support Vector Machine . . . . . . . . . . . . . . . . . . . . . . . . 275 Zhiru Xu, Desheng Liu, Jingguo Zhou, Qingjun Shi Application of Support Vector Machines Method in Credit Scoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 Leilei Zhang, Xiaofeng Hui

Session 4: Pattern Recognition Improving Short Text Clustering Performance with Keyword Expansion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291 Jun Wang, Yiming Zhou, Lin Li, Biyun Hu, Xia Hu Nonnative Speech Recognition Based on Bilingual Model Modification at State Level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299 Qingqing Zhang, Jielin Pan, Shui-duen Chan, Yonghong Yan Edge Detection Based on a PCNN-Anisotropic Diffusion Synergetic Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311 Mario I. Chacon-Murguia, Mitzy Nevarez-Aguilar, Angel Licon-Trillo, Oscar Mendoza-Vida˜ na, J. Alejandro Martinez-Ibarra, Francisco J. Solis-Martinez, Lucina Cordoba Fierro Automatic Face Recognition Systems Design and Realization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323 Zhiming Qian, Chaoqun Huang, Dan Xu Multi-view Face Detection Using Six Segmented Rectangular Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333 Jean Paul Niyoyita, Zhao Hui Tang, Jin Ping Liu Level Detection of Raisins Based on Image Analysis and Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343 Xiaoling Li, Jimin Yuan, Tianxiang Gu, Xiaoying Liu English Letters Recognition Based on Bayesian Regularization Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351 Xiaoli Huang, Huanglin Zeng Iris Disease Classifying Using Neuro-Fuzzy Medical Diagnosis Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359 Sara Moein, Mohamad Hossein Saraee, Mahsa Moein

Contents

XXIII

An Approach to Dynamic Gesture Recognition for Real-Time Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369 Jinli Zhao, Tianding Chen Dynamic Multiple Pronunciation Incorporation in a Refined Search Space for Reading Miscue Detection . . . . . . . . . 379 Changliang Liu, Fuping Pan, Fengpei Ge, Bin Dong, Shuiduen Chen, Yonghong Yan Depicting Diversity in Rules Extracted from Ensembles . . . . . 391 Fabian H.P. Chan, A. Chekima, Augustina Sitiol, S.M.A. Kalaiarasi A New Statistical Model for Radar HRRP Target Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401 Qingyu Hou, Feng Chen, Hongwei Liu, Zheng Bao Independent Component Analysis of SPECT Images to Assist the Alzheimer’s Disease Diagnosis . . . . . . . . . . . . . . . . . . . . 411 ´ Ignacio Alvarez, Juan M. G´ orriz, Javier Ram´ırez, Diego Salas-Gonzalez, Miriam L´ opez, Carlos Garc´ıa Puntonet, Fermin Segovia The Multi-Class Imbalance Problem: Cost Functions with Modular and Non-Modular Neural Networks . . . . . . . . . . . . . . . 421 Roberto Alejo, Jose M. Sotoca, R.M. Valdovinos, Gustavo A. Casa˜ n Geometry Algebra Neuron Based on Biomimetic Pattern Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433 Wenming Cao, Feng Hao A Novel Matrix-Pattern-Oriented Ho-Kashyap Classifier with Locally Spatial Smoothness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441 Zhe Wang, Songcan Chen, Daqi Gao An Integration Model Based on Non-classical Receptive Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 451 Xiaomei Wang, Hui Wei Classification of Imagery Movement Tasks for Brain-Computer Interfaces Using Regression Tree . . . . . . . . . . . 461 Chiman Wong, Feng Wan MIDBSCAN: An Efficient Density-Based Clustering Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 469 Cheng-Fa Tsai, Chun-Yi Sung

XXIV

Contents

Detection and Following of a Face in Movement Using a Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 481 Jaime Pacheco Mart´ınez, Jos´e de Jes´ us Rubio Avila, Javier Guillen Campos

Session 5: Intelligent Modelling and Control Nonparametric Inter-Quartile Range for Error Evaluation and Correction of Demand Forecasting Model under Short Product Lifecycle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 491 Wen-Rong Li, Bo Li Simulated Annealing and Crowd Dynamics Approaches for Intelligent Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 501 Qingpeng Zhang Accomplishing Station Keeping Mode for Attitude Orbit Control Subsystem Designed for T-SAT . . . . . . . . . . . . . . . . . . . . . 507 Montenegro Salom´ on, Am´ezquita Kendrick Nonlinear System Identification Based on Recurrent Wavelet Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 517 Fengyao Zhao, Liangming Hu, Zongkun Li Approximation to Nonlinear Discrete-Time Systems by Recurrent Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527 Fengjun Li Model-Free Control of Nonlinear Noise Processes Based on C-FLAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535 Yali Zhou, Qizhi Zhang, Xiaodong Li, Woonseng Gan An Empirical Study of the Artificial Neural Network for Currency Exchange Rate Time Series Prediction . . . . . . . . . . . . 543 Pin-Chang Chen, Chih-Yao Lo, Hung-Teng Chang Grey Prediction with Markov-Chain for Crude Oil Production and Consumption in China . . . . . . . . . . . . . . . . . . . . . . 551 Hongwei Ma, Zhaotong Zhang Fabric Weave Identification Based on Cellular Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 563 Suyi Liu, Qian Wan, Heng Zhang Cutting Force Prediction of High-Speed Milling Hardened Steel Based on BP Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . 571 Yuanling Chen, Weiren Long, Fanglan Ma, Baolei Zhang

Contents

XXV

BP Neural Networks Based Soft Measurement of Rheological Properties of CWS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 579 Dong Xie, Changxi Li A Parameters Self-adjusting ANN-PI Controller Based on Homotopy BP Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 587 Shuguang Liu, Mingyuan Liu

Session 6: Optimization and Genetic Algorithms Study on Optimization of the Laser Texturing Surface Morphology Parameters Based on ANN . . . . . . . . . . . . . . . . . . . . . 597 Zhigao Luo, Binbin Fan, Xiaodong Guo, Xiang Wang, Ju Li A Combined Newton Method for Second-Order Cone Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 605 Xiaoni Chi, Jin Peng MES Scheduling Optimization and Simulation Based on CAPP/PPC Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 613 Yan Cao, Ning Liu, Lina Yang, Yanli Yang An Improved Diversity Guided Particle Swarm Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 623 Dongsheng Xu, Xiaoyan Ai Research on Intelligent Diagnosis of Mechanical Fault Based on Ant Colony Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 631 Zhousuo Zhang, Wei Cheng, Xiaoning Zhou A New Supermemory Gradient Method without Line Search for Unconstrained Optimization . . . . . . . . . . . . . . . . . . . . . . 641 June Liu, Huanbin Liu, Yue Zheng A Neural Network Approach for Solving Linear Bilevel Programming Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 649 Tiesong Hu, Bing Huang, Xiang Zhang Fuzzy Solution for Multiple Targets Optimization Based on Fuzzy Max-Min Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 659 Pengfei Peng, Jun Xing, Xuezhi Fu Fixed-Structure Mixed Sensitivity/Model Reference Control Using Evolutionary Algorithms . . . . . . . . . . . . . . . . . . . . . . 669 Pitsanu Srithongchai, Piyapong Olranthichachat, Somyot Kaitwanidvilai

XXVI

Contents

Session 7: Telecommunication and Transportation Systems ANN-Based Multi-scales Prediction of Self-similar Network Traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 677 Yunhua Rao, Lujuan Ma, Cuncheng Zhao, Yang Cao Application of DM and Combined Grey Neural Network in E-Commerce Data Transmission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 685 Zhiming Qu Application of Prediction Model in Monitoring LAN Data Flow Based on Grey BP Neural Network . . . . . . . . . . . . . . . . . . . . 693 Zhiming Qu Monitoring ARP Attack Using Responding Time and State ARP Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 701 Zhenqi Wang, Yu Zhou A Study of Multi-agent Based Metropolitan Demand Responsive Transport Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 711 Jin Xu, Weiming Yin, Zhe Huang

Sesson 8: Applications The Diagnosis Research of Electric Submersible Pump Based on Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 721 Ding Feng, Cheng Yang, Bianyou Tan, Guanjun Xu, Yongxin Yuan, Peng Wang The Application of BP Feedforward Neural Networks to the Irradiation Effects of High Power Microwave . . . . . . . . . . . . 729 Tingjun Li A Novel Model for Customer Retention . . . . . . . . . . . . . . . . . . . . . 739 Yadan Li, Xu Xu, Panida Songram Neural Network Ensemble Approach in Analog Circuit Fault Diagnosis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 749 Hong Liu, Guangju Chen, Guoming Song, Tailin Han Research on Case Retrieval of Case-Based Reasoning of Motorcycle Intelligent Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 759 Fanglan Ma, Yulin He, Shangping Li, Yuanling Chen, Shi Liang

Contents

XXVII

Improving Voice Search Using Forward-Backward LVCSR System Combination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 769 Ta Li, Changchun Bao, Weiqun Xu, Jielin Pan, Yonghong Yan Agent Oriented Programming for Setting Up the Platform for Processing EEG / ECG / EMG Waveforms . . . . . . . . . . . . . . 779 Tholkappia Arasu Govindarajan, Mazin Al-Hadidi, Palanisamy V. A Forecasting Model of City Freight Volume Based on BPNN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 791 Peina Wen, Zhiyong Zhang The Estimations of Mechanical Property of Rolled Steel Bar by Using Quantum Neural Network . . . . . . . . . . . . . . . . . . . . . 799 Jen-Pin Yang, Yu-Ju Chen, Huang-Chu Huang, Sung-Ning Tsai, Rey-Chue Hwang Diagnosis of Epilepsy Disorders Using Artificial Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 807 Anupam Shukla, Ritu Tiwari, Prabhdeep Kaur Neural Forecasting Network for the Market of Pleione Formosana Hayata Orchid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 817 Chih-Yao Lo, Cheng-I Hou, Tian-Syung Lan Harmonic Current Detection Based on Neural Network Adaptive Noise Cancellation Technology . . . . . . . . . . . . . . . . . . . . . 829 Ziqiang Xi, Ruili Tang, Wencong Huang, Dandan Huang, Lizhi Zheng, Pan Shen Study on Dynamic Relation between Share Price Index and Housing Price: Co-integration Analysis and Application in Share Price Index Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 837 Jin Peng Application of RBF and Elman Neural Networks on Condition Prediction in CBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 847 Chao Liu, Dongxiang Jiang, Minghao Zhao Judging the States of Blast Furnace by ART2 Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 857 Zhiling Lin, Youjun Yue, Hui Zhao, Hongru Li Research on Dynamic Response of Riverbed Deformation Based on Theory of BP Neural Network . . . . . . . . . . . . . . . . . . . . . 865 Qiang Zhang, Xiaofeng Zhang, Juanjuan Wu

XXVIII

Contents

Adaboosting Neural Networks for Credit Scoring . . . . . . . . . . . . 875 Ligang Zhou, Kin Keung Lai An Enterprise Evaluation of Reverse Supply Chain Based on Ant Colony BP Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . 885 Ping Li, Xuhui Xia, Zhengguo Dai Ultrasonic Crack Size Estimation Based on Wavelet Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 893 Yonghong Zhang, Lihua Wang, Honglian Zhu Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 901

The Initial Alignment of SINS Based on Neural Network Tingjun Li 1

Abstract. Strap-down Inertial Navigation System (SINS) is the development direction of the inertial navigation technology. Its working precision based not only on the inertial measurement units, but also on the initial alignment. This paper uses the N-Tupple neural network in their research, gives principal algorithm and the training algorithm of N-Tupple, and the computer simulation is done. The simulation shows that, the alignment precision of the neural network method is higher than the normal methods, and the greatly reduced alignment time shows the advantage of the new method. Keywords: Strap-down Inertial Navigation System (SINS), N-Tupple neural network, Training algorithm of N-Tupple.

1 Introduction Modern intelligence technique has been successfully used in both pattern recognition and function approximation task, and has been used in the inertial alignment of INS. It adopts BP neural network as well RBF neural network to carry out the inertial alignment of INS. This technique simplifies the algebra structure of system operation, and the real-time of system is better than that of distributed Kalman filter, and the precision is almost the same as that of Kalman filter. Theory N-Tupple neural networks have been successfully applied to both patters recognition and function approximation tasks. Their main advantages include a single layer structure, capability of realizing highly non-linear mapping and simplicity of operation. This network is capable of approximating complex probability density functions (PDFS), and deterministic arbitrary function mappings. The main advantages of N-Tupple network is the fact that the training set points are stored by network implicitly, rather than explicitly, and thus the operation speed remains constant and independent of the training set size. Therefore, the network performance can be guaranteed in practical implementations. This paper uses N-Tupple network to realize the initial alignment of SINS, the computer simulation results show that this method meet the need of engineer. Tingjun Li Naval Aeronautical and Astronautical University, Yantai 264001, China [email protected] H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 1–6. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

2

T. Li

2 Algorithm of N-Tupple Network We consider a general system taking a D-dimensional real-valued vector, X, as its input and producing a scalar real-valued output, Y (a scalar rather than vector output form is considered for simplicity). The input and output are realizations of random variables X and Y, respectively. It is assumed that X and Y are distributed according to a continuous joint probability density function (pdf). We seek to find an input/output relationship of the system in terms of the regression or a conditional mean of the dependent variable Y for any particular value of the input.

m( x ) = E{Y / x} = E{Y / X = x} : ℜ D → ℜ

(1)

Where it is assumed that the conditional mean exists, i.e., ∀x E m( x) < ∞ . For a known underlying PDF the regression function is given by ∞

m( x) = E{Y / x} = ∫ y ⋅ f ( y / x)dy −∞

(2)

And for any particular (x, y) pair generated by the system: y = m( x ) + ε , where the random error component, ε , disappears in the average (i.e., m ( x ) = E{Y / x} ). However, when no explicit knowledge about the system is available, the regression function can only be estimated from a finite set of point according to its distribution. Regression analysis plays a major role in statistics. In this work we are concerned only with one kind of non-parameter regression, based on the kernel method for probability density estimation. From the definition of the conditional mean it is apparent that if an estimate of the system joint PDF was available, it could be used directly for estimating the regression function. The kernel method provides a means of estimating f ( x, y ) with no assumptions being made about its form, allowing approximation of the regression function in the general case. The unvaried, real and even kernel function, ϕ ( x ) , satisfies the following conditions. Additionally, if hT denotes a smoothing parameter (also called bandwidth or window width of the kernel function) dependent on the number of training samples, T, and satisfying the condition. lim hr = 0 lim T ⋅ hT = ∞

T →∞

T →∞

(3)

Then the estimator, fˆ ( x ) , given by ∞

x − xi fˆ ( x) = ∑ φ ( ) [T ⋅ ∫ φ ( x)dx] hT i =1 −∞ T

(4)

Approaches asymptotically the unvaried distribution density f ( x) . This provides a consistent and asymptotically unbiased estimate of fˆ ( x ) .

The Initial Alignment of SINS Based on Neural Network

3

∫ ϕ ( x) dx = 1

φ ( x) ≥ 0

(5)

ℜ

Approaches asymptotically the univariate distribution density f ( x) , this provides a consistent and asymptotically unbiased estimate of fˆ ( x ) .

ϕ ( x, y ) = ϕ x ( x )φ y ( y )

(6)

Where ϕ y ( y ) is a univariate kernel function satisfying the conditions (3) and (6).

：

Thus, PDF is

1 fˆ ( X , y ) = T

T

∑Φ i =1

X

( X − X i ) ⋅ϕ y ( y − y i )

(7)

And the regression function can be estimated as: T

Eˆ (Y / X ) =

∑y

i

i =1

⋅ΦX (X − X i )

(8)

T

∑ΦX (X − X i ) i =1

Where Φ X ( X ) satisfies the following conditions

sup Φ( x) < ∞

X ∈ℜ D

sup Φ( x) < ∞

X ∈ℜ D

sup Φ( x) < ∞

(9)

X ∈ℜ D

According to the above analysis, we know the fact that if the function Φ X ( X ) can be realized, then this network can response the input/output relationship. The literature (3) gives the N-Tupple network. The network consists of an R-bit binary array, and a set of K memory nodes, each having a N-bit long address word (i.e., having 2 N addressable locations), the structure is showed in Fig.1. The ℜ D → ℜ mapping performed by the network consists essentially of three stages:

%LQDU\PRGHORILQSXW

Address of N -bit

LQSXW RXWSXW

Fig. 1 Structure of N-Tupple Network for Initial Alignment

4

T. Li

z z z

Conversion of the real vector input into a binary format and projecting it onto the network retina. Sampling of the R-bit binary array to form its address with N randomly selected array bits. Combining the contents of the addressed memory location to produce the network response.

3 N-Tupple Network of Initial Alignment 3.1 Initial Alignment Principle of SINS Assume that the state and measurement equation of initial system is:

⎧⎪ X ( k +1) = Φ ( k +1, k ) X ( k ) + T( k +1, k )U ( k ) + Γ ( k +1, k )W( k ) ⎨ Z ( k +1) = H ( k +1) X ( k +1) + V( k +1) ⎪⎩

(10)

Where X ∈ R n is state vector, Z ∈ R m is measurement vector, U ( k ) is controlling vector, W( k ) ,

V( k +1)

are the system’s dynamic noise vector and measurement noise

vector, are white noise, its’ variance matrix is Q( k ) and

R( k +1) ,

the rest are all coefficient

matrix. Taking SINS of 8-order for example, the process and precision of initial alignment system using N-Tupple network will be discussed concretely. Assume that the state vector x = ⎡⎣Φ x , Φ y , Φ z , ε x , ε y , ε z , ∇ x , ∇ y ⎤⎦T , measurement vector z = ⎡⎣ zx , z y ⎤⎦T , controlling vector u = ⎡⎣u x , u y , u z ⎤⎦T , measurement noise vector V = ⎡⎣Vx ,Vy ⎤⎦T , and then the mathematic model is: ⎧ dx (t ) = Ax(t ) + Bu (t ) ⎪ ⎨ dt ⎪⎩ z (t ) = Hx (t ) + V (t )

(11)

Where Φ x , Φ y , Φ z are the error angles of SINS, ε x , ε y , ε z are the random constant excursion of three gyro (east and north are 0.01°/h, sky direction is 0.03°/h), ∇ x , ∇ y are the random constant excursion of accelerometer (east and north are 10-4g). A, B, H is the coefficient matrix, the item whose the value is respectively is A(1,2) = - A(2,1) = ωiesinL, H(1,2) = -H(2,1)= -g, A(1,3) = - A(3,1) = -ωiecosL, A(1,4) = - A(2,5) = A(3,6) = 1, H(1,7) = H(2,8) = 1, B(1,1) = B(2,2) = B(3,3) = 1.

3.2 Study Algorithm of N-Tupple As each memory location contains a real number, any particular choice of tupple addresses results in a selection of K numerical weights {w1 ( x), w2 ( x), , wk ( x)} . When the input is x, let ak ( x) designate the counter value corresponding to the location addressed in the kth tupple memory by the input X, thus any input to the

The Initial Alignment of SINS Based on Neural Network

5

network results in a unique selection of K-Tupple addresses together with their associated weight and counter values.

⎧ {t1 ( x), t2 ( x), , tk ( x )} ⎪ x → ⎨{w1 ( x), w2 ( x), , wk ( x)} ⎪ {a ( x), a ( x), , a ( x)} ⎩ 1 2 k

(12)

Initially, all network tupple memory locations (both the weight and counter values) are set to zero. During the training phase the network is presented with T training pairs (Xi, Yi) drawn according to the PDF of the system being modeled, where Xi is the D-dimensional input vector, and Yi denotes the corresponding output. For each tupple location addressed by Xi the value of Yi is added to the corresponding weight, and the location counter is incremented: ⎧⎪ wk ( X i ) ← wk ( X i ) + y i ⎨ i i ⎪⎩ ak ( X ) ← ak ( X ) + 1

(13)

During the recall phase the network output, yˆ( X ) , is obtained by normalizing the sum of addressed weights with the sum of their corresponding counter values. K

yˆ( X ) = ∑ wk ( X ) k =1

K

K

∑ a ( X ) , ∑ a ( X ) = 0 → yˆ ( X ) = 0 k =1

k

k =1

(14)

k

3.3 Simulation and Analysis of N-Tupple SINS Initial Alignment The initial data for simulation is L=45°, E{Φ 2x } = E{Φ 2y } = (10′)2 , E{Φ 2z } = (60′) 2 ,

E{ε x2 } = E{ε y2 } = (0.01° h)2 , E{ε z2 } = (0.03° h) 2 , E{∇2x } = E{∇ 2y } = (10−4 g )2 . When introducing N-tupple network, the simulated curves for error angles Φ x, N , Φ y , N and Φ z , N of SINS initial alignment is showed in Fig.2. 0.1 X,N

0.05

Ф-0.05 0

Y,N

0.05 0

Ф-0.05 0.5

Z,N

Fig. 2 Simulated Curves Using N-tupple

0

Ф-0.5 0

50

100

150 200 250 T ime (s)

300

350

400

6

T. Li

4 Conclusions As discussion, we finished the initial alignment of SINS using N-Tupple network. From the simulated results, we can reach the following conclusions: (1) The initial alignment using N-Tupple network is feasible. (2) The initial alignment using N-Tupple network can decrease greatly the alignment time, especially can be applied in the field of rapid alignment; At the same time from the point of view of hardware realization, N-Tupple network has great predominance.

References 1. Yang, L., Wang, S.H.: Initial Alignment System of SINS Using BP Neural Network. Transactions of Nanjing University of Aeronautics & Astronautics 28, 487–499 (1996) 2. Wang, D.L.: Initial Alignment System of SINS Using RBF Neural Network. Astronautics Control 2, 48–59 (1999) 3. Aleksander, K.: N-Tupple Regression Network. Neural Network 9, 855–869 (1996) 4. Yuan, X., Yu, J., Chen, Z.: Navigation System. China Aviation Industry Press, Beijing (1994) 5. Li, T.J.: Data Acquiring System Based on Vxi bus. In: Proceedings of the Second International Conference on Active Media Technology, vol. 5, pp. 688–692 (2004) 6. Li, T.J.: Design of Computer Management System. In: Proceedings of the Third International Conference on Wavelet Analysis and Applications, vol. 5, pp. 744–749 (2004) 7. Li, T.J.: Design of Boot Loader in Embedded System. In: Proceedings of the 6th International Progress Wavelet Analysis and Active Media Technology, vol. 6, pp. 458–463 (2005) 8. Li, T.J., Lin, X.Y.: Research on Integrated Navigation System by Rubidium Clock. Journal on Communication 8, 144–147 (2006)

Analysis on Basic Conceptions and Principles of Human Cognition Xiaorui Zhang, Minyong Li, Zhong Liu, and Meng Zhang 1

Abstract. The knowledge on consciousness, thinking and cognition is the base of the researches about human intelligence. The basic elements of cognition and their relations between each other are discussed, and breakthrough on cognition has been made. It is concluded that thinking is one-dimensional kinetic process that is based on memory function and is composed of two or more static consciousnesses with certain relationships between them in one’s conscious focus. The concepts about consciousness in several scientific subjects and their relationships are summarized, and conscious forms are classified newly. According to the thought about the contemporary system of science and technology, an opinion about human cognitive levels has been proposed. Keywords: Systems science, Cognitive science, Consciousness, Thinking, Language.

1 Introduction Cognitive researches in China belong to the field of noetic science [1] that is in fact no other than cognitive science, because they both have the same research objects, but there exist some dissimilarities between their viewpoints. Noetic science is dedicated to disclose the essence and laws of thinking and cognition, and aims at understanding human cognition. It is very significant to study cognition. First, it can promote cognitive abilities and broaden our fields of vision and help us to cognize the world better. Second, it can facilitate thoughts and cognitions owned by different persons to be better communicated and imparted, and make individual cognition become social cognition. Third, it can accelerate one’s consciousness and body to coevolve better and answer some basic problems, e.g. what life is, why life forms are so various, why I exist, how human come, whether there is life in other planets, if there is limit in the scales of matter forms, etc. Fourth, it can improve individual’s life, enhance his inner psychic realm, and increase the morality at the aspect of his behavior and make individuals pursue to be saints. Uncovering human Xiaorui Zhang . Minyong Li . Zhong Liu . Meng Zhang Dept. of Command Automatization, Naval Univ. of Engineering, Wuhan 430033, China [email protected] H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 7–17. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

8

X. Zhang et al.

cognition entirely can help individuals to heighten their cognitive abilities and make it easier to exceed themselves in cognition. Studying cognition means that it is studying life especially human brains that are very complex huge systems. Any object correlated to cognition can be included in the research range, so the research about cognition is a field that has strong characteristics of systems and synthesis.

2 Consciousness and Thinking 2.1 Reflection on Consciousness and Thinking Marxism philosophy considers that consciousness is the reflection of matter in brains. From the viewpoint of psychological motion, consciousness may be classified as psychological imitation and knowledge [2]. Psychological imitation is subjective consciousness, and is a kind of psychological experiences and mental states, and includes emotion, feeling, interest, attention, will and so on. Knowledge is formed from thought, is objective consciousness abstracted and refined by rational thinking. In a broad sense, consciousness includes explicit consciousness and potential consciousness. Consciousness usually means explicit consciousness and is a category in a narrow sense. Explicit consciousness can be grasped and controlled by our spirit, but potential consciousness can’t. Potential consciousness is the base of explicit consciousness which comes from potential consciousness. Instinct is an innate capability or patterns of behaviors that are often responses to specific environmental stimuli, e.g. babies needn’t study and are able to cry and suckle after birth. Instinct also includes the abilities that have been posteriorly possessed by bodies and seem not to need any feeling to enter brains, e.g. at the time of emergencies, human usually can erupt self-protecting behaviors under the condition of self-unknowing and sometimes can appear supernormal behavioral capability. In fact, instinct is a kind of potential consciousness. Potential consciousness shows itself more at the aspect of functionality, in which one part is owned with birth and seems solidified in neural organizations, and the other part is formed after birth and possessed by studying in the course of living, e.g. writing, riding, driving, typing on a keyboard, etc. Consciousness is the higher form of potential consciousness and is a mental form which can be felt, grasped and controlled. Based on consciousness, thinking comes into being. Thinking is a form that belongs to consciousness. In a point of time, consciousness is quiescent, and is like a picture depicting the form of a material object, so thinking is a dynamic course that is like a film comprised of some pictures between which there are some relations. Human’s thinking is one-thread or one-dimensional. The consciousness which is being paid attention to is named as conscious focus, then in a time point there is only one conscious focus for any person. A thinking course in which there exists change is formed by different consciousnesses between which there usually still exist other relations besides temporal and spatial logical relations. Thinking is interrelated with memories that supply the relations between consciousnesses in a thinking course. Memory is also a kind of consciousnesses, and the difference is that memories are the consciousnesses which are saved in brains and can be taken out and then reappear in conscious focus and feelings.

Analysis on Basic Conceptions and Principles of Human Cognition

9

2.2 Cognitive Forms on Thinking in Scientific Subjects There is meta-cognition in any cognitive field. All cognition comes from methodologies that include relations, information, scale views [3], points of view, methods, etc. Different cognitive bases lead to different cognitive contents. The conceptions about thinking in different subjects are different. In human history, mental phenomena were originally researched by philosophy. With the development of human cognition, the researches on thinking formed logic which gradually separated from philosophy, afterward cognitive science comes into being, and in China this research field is called noetic science. In Marxism philosophy, the meaning of thinking has the difference between broad sense and narrow sense [2]. Thinking in a broad sense equals consciousness, spirit and cognition. It expresses the meaning as the object opposite to existence, and is a general category. In a narrow sense, thinking just denotes the rational cognition in a cognitive course. Obviously, these are two different conceptions, but they are not distinguished and used by a mixed mode in Marxism philosophy. Brain science is interested in the inner laws of neural physiological functions which are the base of consciousness and thinking, and researches how consciousness and thinking emerge from the operations of neural organizations. Psychology science pays attention to the operational mechanism of thinking and mental behaviors, and commits itself to find out the inner laws contained in the transformation from psychological activities to physical behaviors. Noetic science is the scientific category which studies the laws relevant with thinking. It covers the research results of brain science, psychology, thinking, cognitive science, philosophy, systems science and so on, and synthesizes all cognitive contents about thinking that comes from different scale views or viewpoints in order to understand how human cognize the objective world and how the information gotten from feeling the world is saved and processed to become the knowledge about the world. Systems science thinks much of the consistency, difference and systematization about thinking and cognition between all relevant scientific subjects, emphasizes the relations, cross and amalgamation of subjects. Systems science is also interested in the function which thinking and cognition perform. In fact, what systems science here studies is the same as noetic science’s, the difference is just their viewpoints, so noetic science can also be claimed as noetic systems science from the viewpoint of system idea. Cognition has hierarchical structure from concreteness to abstraction. Philosophy grasps the most ultimate and general properties in the abstract. Concrete subjects such as brain science, psychology, neurophysiology and somatology research some subsidiary or concrete properties, laws and processes in a less scale under the macro direction of philosophy. The conception about thinking which is formed from each subject is different from other subject’s due to different methodologies. This is like a focus lens which is facing an object and shows different pictures according to different focus positions and ranges, but the object doesn’t change with the change of focus and it is objective. Though there are some differences between correlated concrete subjects, they all faces the same object as thinking, they should be theoretically consistent with each other. The consistent problem is just the place where systems science usually performs its functions. Systems science will harmonize all concrete subjects which are correlated with noetic science and do its best to form synthetical and uniform conceptions about thinking and other relevant objects from concreteness to abstraction and

10

X. Zhang et al.

from feelings to logos. The way doesn’t accord with cognitive characteristics that the essence of thinking is understood through the definition of several general sentences. Thinking should be grasped by synthesizing the cognitive contents that come from different methodologies.

2.3 Classification on Thinking The classification on thinking can be said that the benevolent see benevolence and the wise see wisdom. It is bewildering that the same cognitive object has so many different opinions, which to some extent shows that it isn’t easy to completely cognize thinking. Combined with Hsue-shen Tsien’s thought, according to the basic units of thinking, thinking is classified as behavioral thinking, visual thinking, abstract thinking; with the result and function of thinking, thinking is divided into reappearing thinking and creating thinking; from the viewpoints of main bodies, thinking is composed of individual thinking and social thinking [2]. Based on the restudy about consciousness and thinking, thinking forms are here newly classified. According to the relational modes in thinking, thinking contains two kinds of thinking forms that are logical thinking and alogical thinking. In logical thinking there are some laws that have been grasped by human. Logical thinking mainly is symbolic thinking that is so particular that many people don’t think they can think if they don’t use symbols. Symbolic thinking includes language thinking, all kinds of computer language thinking, mathematical symbolic thinking, graphical thinking and other kinds of symbolic logical thinking. Language thinking is a tool that plays a key role in the development of human cognition. Symbolic thinking especially language thinking spans all levels of cognition from concreteness to abstraction, and carves up consciousness and cognition precisely and punctiliously. In the course of cognitive development, this kind of carving is continuously carried on and affirms and records each result gotten in cognitive progress, so cognition gets accumulated and can be handed down or be inherited through a series of symbols. Alogical thinking denotes the thinking forms in which human haven’t yet found out laws, and has many forms, i.e. replaying thinking, dreaming thinking, associational thinking, random relating thinking and inspirational thinking, etc. The boundary between logical thinking and alogical thinking isn’t changeless. Human cognitive development shows that the logical range has been ceaselessly expanding and the boundary changes as if the logical range had been nibbling the range of alogical thinking at all times and found out laws in the range of alogical thinking and transferred them to the logical range of thinking. Replaying thinking denotes that based on memories what has been felt reappears in conscious focus. Dreaming thinking figures the thinking forms in one’s dream. Associational thinking is the thinking activities in which some consciousnesses have been correlated according to some relations and have been reflected orderly in conscious focus. Random relating thinking is the least restricted form and can be completely divorced from the usual relations among other kinds of thinking. Alogical thinking usually contains both logical thinking contents and alogical thinking contents. Compared with logical thinking, the laws contained in alogical thinking appear faint. Inspirational thinking has close relations with other thinking forms, and it seems to emerge from a kind of

Analysis on Basic Conceptions and Principles of Human Cognition

11

thinking chaos and has strong creativity. According to the concrete forms of consciousness contained in thinking, thinking forms can be divided into perceptional reflection thinking, conscious presentation thinking, abstract idea thinking and symbolic thinking. Perceptional reflection thinking is the direct and sensory reflection on objective objects and forms the perception about objects. Conscious presentation thinking is composed of pure conscious perceptions without the help of symbolic languages and begins to grasp the content and essence of objective objects to some extent. Abstract idea thinking breaks away from sensory reflection, synthetic perception and conscious presentation, and purely moves about in the range of content and essence about objective world. Symbolic thinking is the thinking activities that depend on symbols which are correlated with concrete consciousnesses. The forms of things are consistent with their contents. According to the concrete contents of consciousnesses contained in thinking, thinking forms can be divided into sensory thinking, perceptional thinking, presentative thinking, idea thinking and symbolic thinking. Sensory thinking is about the original feelings on objects, and is an intuitionistic thinking. Perceptional thinking synthesizes some relevant sensory thinking and forms a total sensory grasp. Presentative thinking is the process that is based on relevant perceptional thinking and can partly grasp the content and essence of objects. Idea thinking is a purely abstract thinking and is the highest cognitive forms of human thinking. Thinking can also be classified into concrete thinking and abstract thinking, and concrete thinking is also named as intuitionistic thinking or imagery thinking. From the participant extent of perception and sensibility, thinking can be divided into sensorial thinking and rational thinking. The most attractive thinking form is creating thinking that can find out new ideas. Creation is weighed by the results of thinking, so creating thinking is always associated with concrete problems that make creating thinking pragmatic. Creating thinking has two types, one is pattern logic evolvement, and the other is pattern extension or reformation. Most of past researches belonged to the former that can’t direct the development of era, but the latter can. Like the work of Aristotle, Copernicus, Galileo, Heng Zhang, Newton, Hegel, Marx, Einstein, Planck, Hawking, etc, what they do are the greatest creations.

3 Thinking and Language 3.1 The Antiquity and Evolvement of Language At the beginning of the form of a language, every syllable was correlated with a concrete object in life, e.g. in Chinese such words are like sun, moon, white, black, eat, look, etc. With the development of cognition, the ability to abstract was also developed, the denotation of language symbols had been changed or extended. In the meanings of words, most of archaic words originally only had one kind of meanings, but most of modern words have been developed to have several kinds of meanings and some words still keep the original meanings. A language becomes more complex and richer with its development. The pronounciation of words has changed from single syllable to multisyllable because of human’s characteristics of physiology and thinking, and multisyllable can make up the shortcoming that the acuteness of human audition isn’t as perfect as possible and in fact is usually dull. The

12

X. Zhang et al.

evolvement of languages obviously shows that human cognition is evolving from simpleness and sensibility and is becoming more and more complex and rational.

3.2 Analysis on the Broad and Narrow Meanings of Words The broad and narrow meanings of words mean the denotation of a word is a range that is like a circle which has core and periphery. The content of core denotes the narrow meanings, and the periphery that is formed from the relations with the core denotes the broad meanings. Although the form of a word has not any change, its denotative content isn’t single or identical in different lingual contexts. When a word shows its broad meanings, the word is like a point correlated with the body that is looked as its meanings, so the same meanings can be denoted by different points, i.e. different words can denote the same meanings, e.g. in Marxism philosophy consciousness have the same meanings as spirit or cognition, they are identical. The broad and narrow meanings of words come from two factors, one is human’s cognitive ability to abstract, and the other is the reduction, resolution and exactness of cognition about cognitive objects. Abstraction means holism, resolution means reductionism, so the cognitive abilities pertinent to holism and reductionism jointly forms the emergence of the broad and narrow meanings of words. It is the difference of scales of view in cognition that is the essential reason which leads to the emergence of the broad and narrow meanings of words. The broad meaning of a word is formed from the magnification of the range of its denotation so as to contain the correlated contents in a bigger range. Although the denotation has changed with the context, the symbol is still the word which hasn’t changed.

3.3 The Relation between Thinking and Language Our life bodies are living fossils that have been witnessing human’s history and also the earth’s history when there had existed any evolving form from which human ancestors’ lives came. The language we use is also a historical witness and is easier to study. There are more than six thousand languages in the world, but they all belong to eight kinds of phyla [4]. Many languages have evolved from a same language, e.g. Zang language, Burmese and Chinese have obvious common base, which has proved that long ago they belonged to one people. Language is a tool used to measure cognition. The least metrical unit is morpheme. Language is composed of a series of symbols that are used to segment and match nearly all things that belong to one’s conscious world. Thinking continues segmenting and makes itself more precise with language, and each progress gotten by thinking about cognition is recorded by new language symbols, so things that are invisible, blurry or dormant are transformed into things that are visible, specific and tangible. The development of language toward complication shows that human thinking ability is evolving to be more intelligent. Not all thinking is embodied or exhibited by language because thinking can operate without language, but thinking does its best to be expressed by language, because without language any thinking or cognition can’t be inherited. Thinking needs memories, and each tiny

Analysis on Basic Conceptions and Principles of Human Cognition

13

progress needs the help of memories, which are recorded in brains, but most are finally transferred and recorded out of brains in some mode such as language or picture. Thinking and language help and accelerate each other’s evolvement.

4 Thinking and Cognition 4.1 Cognition Definition Each individual is a conscious unit or body. Cognition emphasizes the result of the knowledge about objects, that is, a conscious body does his best to approach the objective existence of things through conscious reflection, and under the context of cognitive historical level accurately hold the truth about things to the greatest extent. Comprehending as a kind of thinking modes emphasizes cognitive process, but cognition is the result of comprehending and emphasizes cognitive result.

4.2 The Hierarchical Structure of Cognition The contemporary system of science and technology [5] proposed by Hsue-shen Tsien divided human knowledge into six levels and eleven big branches, as is shown in figure 1. The knowledge has been longitudinally carved up into six levels, which from top to bottom are philosophy, bridges, basic theories, technical science, applied technologies and pre-science. Scientific knowledge has been latitudinally divided into natural science, social science, mathematical science, systems science, noetic science, anthropic science, geographical science, military science, behavioral science, architectural science, and literature and art. The content of cognition is hierarchical, and accordingly knowledge also has multilevel. According to the thought of the contemporary system of science and technology, from the viewpoint of cognition, individual’s cognitive levels has been abstracted and formed. As is shown in figure 2, human’s cognition has six levels, which from abstraction to visualness are respectively view, methodology, science, method, art, technology and work. Work is here tightly correlated with

Mathematical science

Dialectics of Nature

Systemic methodology

Mathematical Philosophy

Systems science

Objective historical view

Epistemology Noetic science

Social science

Human-world view Anthropic science

Natural science

Military philosophy

Geographical philosophy Geographical science

Human ology

Military science

Behavioral science

Unwritten practical perception

Applied technologies

Technical science

Basic theories

Practical experiential knowledge repository and philosophical thinking

Pre-science

Aesthetics

Architectural philosophy

Literary theory

Literary activitis

Architectural science

Quantitative intelligence

Bridges

Philosophy

——Science of cognizing the subjective and objective world

Marxism philosophy Qualitative intelligence

Literary creation

Fig. 1 Contemporary system of science and technology is proposed by Tsien Hsue-shen

14

X. Zhang et al.

Fig. 2 The abstract structure of one’s cognition has six levels

practice, it may be said that what work denote here is just practice. Through work, human transforms his thought and ideas into behavioral activities, and communicates with practical objects and the world, and is able to transform nature, and feeds back information through practice to validate and to improve cognition. Work is here concrete behavioral activities, and has its specific behavioral object. Technology here denotes abilities at the aspects of thinking and skills, and is combined with individual body and can’t separate from it. Without the help of technology, work can’t be carried on. Art is here a kind of rules or strategies that is relevant to concrete behavioral activities or operational processes and can’t separate from the operations and has its concrete operational object, i.e. an art is the art of some activity or object and isn’t the art of other activities or objects. It is possible that the arts of some activities or objects are the same, and this situation belongs to method levels. Method here denotes a kind of arts that are the same or a set of arts that are pertinent, and a method is faced with a kind of collectivities that may be a set or a range. Some methods about problems in a field can be logically correlated to form a logical system. Different methodologies and world views lead to different logical systems about the same things, and each kind of logical systems can be called a school. A science is here a general uniform theory that has syncretized, synthesized, combined and integrated nearly all the schools in a field and has formed a uniform acknowledged logical system. In the process of forming a science, each school needs abstract theoretical grasp or direction about some problems to coordinate and to harmonize their opinions, this kind of abstract theoretical grasps is called their methodology. Based on all kinds of methodologies, one can have a whole general abstract cognition about the world or problems, which is here called view. When one faces a problem, he always integrates and grasps his cognition from the most concrete work up to the most abstract view, finally through thinking gives birth to one or more ideas. Once the idea has been performed, then one will show the idea by his face’s expressions, physical actions or behaviors, which can be organized to form the state of work.

4.3 The Relation between Thinking and Cognition The realization and improvement of cognition is a process which goes through from sense to perception, then to conscious presentation, abstract view, and finally to idea, from visualness to abstraction, from concreteness to generalness, from

Analysis on Basic Conceptions and Principles of Human Cognition

15

microcosmicness to macroscopicalness, from phenomena to essence, from practice to rationalness, from chance to necessity and from finity to infinity. The realization of this process owes to the function of consciousness, which innately has this kind of talents and its necessary organizational material base. Each individual is unique and different from other conscious bodies. The uniqueness is confirmed by life’s material constitution that can’t be substituted by other lives. Furthermore, life growth and cognitive development are also unique in the world, so each conscious body is inevitably different from others. On the other hand, every individual can coexist and communicate with others, which explains that there are similarities, commonness, congenericness or homogenousness among conscious bodies. The same kind of life bodies has the same physiological material bases and similar habitations, lives and evolves in the mode of humanity which makes the existence of commonness between conscious bodies, and commonness is the base for the communication between conscious bodies. Consciousness can’t communicate directly with others by the forms in which it exists in brains, so consciousness needs some explicit forms to have contact with other individual’s consciousness. Face expressions and body actions are the original forms for the embodiment of consciousness, and doodling, depicting, drawing or painting such as ancient cave frescoes belong to another kind of the most intuitionistic forms, all of these forms can be called denotations that are used to pass conscious information. The emergence of concrete denotations that separate from human bodies is a qualitative progress. The significance of denotations lies in the representation of nearly any complex conscious content in a concise mode. Language is a kind of advanced denotation systems that can denotes nearly most conscious contents and the relations between them. Each word is relevant to a kind of special consciousnesses in a certain range. If there aren’t the relevant conscious memories of a word in one’s brain, he can’t understand the meanings of the word. It has been known that there always exists the difference between any two conscious bodies at the aspect of concreteness, people’s consciousnesses or conscious backgrounds correlated with a word are different, how they manage to communicate? Here one’s conscious background is composed of concrete and abstract consciousnesses relevant to something and is a mass of all kinds of relevant consciousnesses, e.g. the word mama makes everyone think of his mother and then relate to others’ mothers and other life forms’ mothers, finally in his conscious world has grasped the most abstract content that mama is a kind of relations that identifies the object that has given birth to a new life or has fostered a new life. It is by this way that the cognition about mama comes into being. When one has heard a child’s pronunciation on mama, he firstly searches out the relation between the denotation in the sound form with the most abstract rational consciousness that has already been recorded in his brain in the form of memories, then downwards relates to others’ mothers, through associating these consciousnesses with the child, he is conscious of the child’s mother. Therefore, although each individual has his unique consciousness background that is different from others’, all of them are able to form identical rational consciousness through the ability to abstract some consciousnesses. Although these identical rational consciousnesses respectively exist in each of conscious bodies, closed cognitive channels that achieve all human’s cognition are

16

X. Zhang et al.

formed based on these pivotal rational consciousnesses which are the upward peaks in the closed cognitive channels. One’s thinking is cultivated and improved in the process of his growth. For a student, the contents studied in school are made up of two aspects, one is the exercises and improvement of the ability to think and to cognize, the other is extending fields of vision i.e. obtaining knowledge. Although human brains have congenital cognitive material base that is far more dominant than other species’, but the ability to think and to cognize intelligently is reached and acquired by a lot of methodical exercises. Without necessary exercises, even if one has good genetic aptitude, his potential intelligent ability can’t get aroused, and after he has grown up, usually he is maybe clever due to his life experiences, but he couldn’t be intelligent or knowledgeable. Language can boost up the ability to think, but in most cases thinking can operate without language, which can be regarded as a kind of thinking ability. Most of people are usually accustomed to think with language, they think that it is incredible to think without language, which is because people used to so excessively rely on the help of language that the ability to think without language hasn’t been initially evoked or has degenerated. A lot of abilities are correlated with cognition. Since life is a kind of material motion [6], human abilities aren’t changeless. Through study and exercises, one can learn an ability and reach the extent of proficiency, but if he hasn’t use the ability in a long time, he will feel rusty at this aspect when he needs to use the ability to tackle some problems, i.e. the ability gets degenerated. Thinking is a kind of abilities, so it has this characteristic about abilities. Thinking has many forms, and it can be said that any forms is a kind of abilities to think. Any kind of thinking needs to be evoked, cultivated and improved, after it is gotten, it also needs to be kept through frequently using or exercising it. Cognition is reached through thinking. Different cognitive abilities can form different opinions. Even cognition on the same level can lead to different cognitive results due to different bases and methods. Thinking abilities decide the level and profundity of cognition. One’s cognitive abilities get more liberated, i.e. the better his thinking abilities are, the deeper and the more self-contained his cognition about the essence of cognitive objects is. It may be said that the abilities of thinking and cognition aren’t inherent or congenital but cognitive parochialism is. Consequentially conscious bodies lying in certain cognitive levels coexist with the bondages of cognitive parochialism. Since one doesn’t improve himself up to a higher cognitive level, which means that he doesn’t break away from his current level, then he can’t feel the parochialism of his cognition formed on the current cognitive level and he will think that his opinions is very correct, on the contrary, he maybe doubt or deny some significant opinions, which shows their conservativeness on thought and restriction with cognition. When he has gotten away from his current cognitive level and has risen on a new higher level, he will naturally realize the parochialism of his opinions formed on the former level. Therefore, it isn’t proper to deny others’ cognition or optionally make comment on others’ opinions, because they aren’t fault based on their cognitive level. Anybody can’t get away from the parochialism of his historical existence as if today can’t exceed tomorrow forever. The development of one’s cognition is the course in which he is ceaselessly exceeding himself and wiping

Analysis on Basic Conceptions and Principles of Human Cognition

17

off current cognitive parochialism through self-conscious cognitive evolvement and improvement, and it is also a process of frequently denying himself in cognition, through which conscious bodies rise toward a higher level of cognitive states.

References 1. Tsien, H.S.: On Noetic Science. Nature Magazine 8, 563–567 (1983) 2. Lu, M.S.: Research on the Mystery of Thinking. Beijing Agricultural University Publishing House, China (1994) 3. Zhang, X.R., Li, M.Y.: Scale View and Thinking Principle in System Mode. Science and Technology Philosophy Doctoral Forum of China, Taiyuan (2008) 4. Pinker, S.: The Language Instinct—How the Mind Creates Language. Translated by Lan Hong. Shantou University Press, China (2004) 5. Tsien, H.S.: The Structure of Contempary Science—Rediscussion on systematology of contemporary science and technology. Philosophical Researches 3, 19–22 (1982) (in Chinese) 6. Zhang, X.R., Li, M.Y., Liu, Z., Zhang, M.: System Cognition Idea. In: 4th National Conference of Logic System, Intelligent Science and Information Science, Guiyang, China (2008)

Global Exponential Stability for Discrete-Time BAM Neural Network with Variable Delay Xiaochun Lu

Abstract. In this paper, the existence and the global exponential stability of the equilibrium point for a class of discrete-time BAM neural network with variable delay are investigated via Lyapunov stability theory and some analysis techniques such as using an important inequality and using norm inequalities in matrix theory. Several delay-independent sufficient conditions for the existence and the global exponential stability of the equilibrium point are derived by constructing different Lyapunov functions for different cases. Finally, two illustrative examples are given to demonstrate the effectiveness of the obtained results. Keywords: BAM neural network, Discrete-time system, Global exponential stability, Lyapunov stability theory.

1 Introduction The bi-directional associative memory(BAM) neural network model, known as an extension of the unidirectional auto associator of Hopfield neural network, was first introduced by Kosko in 1987. It has been widely used in many fields such as pattern recognition, optimization and automatic control and so on. Up to now, a lot of works concerning the global exponential stability of BAM networks have been done. Many scholars have derived various sufficient conditions for the global exponential stability for continuous-time BAM neural networks(see [1-8]) by constructing suitable Lyapunov functions and using some different inequality techniques. Nowadays, more and more researchers studied more general BAM networks model without delays described as the following form: Xiaochun Lu School of Water Resources and Hydropower, Wuhan University, Wuhan 430072, China H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 19–29. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

20

X. Lu

⎧ p  ⎪ ⎪ cji fj (yj (t)) + Ii ], i = 1, . . . , m, ⎨ x˙ i (t) = −αi (xi (t))[ai (xi (t)) − j=1

m  ⎪ ⎪ dij gi (xi (t)) + Jj ], j = 1, . . . , p, ⎩ y˙ j (t) = −βj (yj (t))[bj (yj (t)) −

(1)

i=1

or the model with delays described as the following form: ⎧ p  ⎪ ⎪ sji fj (yj (t − τji (t))) + Ii ], i = 1, . . . , m, ⎨ x˙ i (t) = −αi (xi (t))[ai (xi (t)) − j=1

m  ⎪ ⎪ tij gi (xi (t − σij (t))) + Jj ], j = 1, . . . , p, ⎩ y˙ j (t) = −βj (yj (t))[bj (yj (t)) − i=1

(2) where m ≥ 2, p ≥ 2 are numbers of neurons in the network, xi , yj denote the state variable associated with the neuron, and ai , bj are appropriately behaved functions. The connection matrices C = (cji )p×m , D = (dij )m×p tell us how the neurons are connected in the network and S = (sji )p×m and T = (tij )m×p indicate the strength of the neuron interconnections within the network with time delay parameters τji (t) and σij (t). And the activation functions fj and gi show how neurons respond to each other. However, because of implementing the continuous-time neural networks for simulation or computational purposes, it is essential to formulate a discretetime system which is an analogue of the continuous-time networks. There is no unique schemes ways of obtaining a discrete-time analogue from continuoustime networks. There exist many numerical schemes such as Euler scheme and Runge-Kutta scheme which can be used to obtain the discrete-time version of the continuous-time system. Certainly, the discrete-time system is desired to preserve the dynamical characteristics of the continuous-time system. For the stability problem of discrete-time(see [9-13]) bi-directional associative memory neural networks, results are much less than those of continuoustime neural networks. For example, in [13] authors have studied the global exponential stability of a class of discrete-time Cohen-Grossberg neural network(CGNN) with delay or without delays. Mohamad and Gopalsamy[11,12] have investigated the exponential stability of continuous-time and discretetime cellular neural networks with delays via Halanay-type inequalities and Lyapunov methods. Based on the linear matrix inequality (LMI), in [14], authors have derived some sufficient delay-independent and delay-dependent conditions for the existence, uniqueness and global exponential stability of the equilibrium point of discrete-time BAM neural networks with variable delays. With the development of the Cohen-Grossberg neural network(see [15-17]), in this paper, we consider the corresponding discrete-time version of the bi-directional associative memory (BAM) neural network model (2) described as the following equation

Global Exponential Stability for Discrete-Time BAM Neural Network

21

⎧ p  ⎪ ⎪ sji fj (yj (n − kji (n))) + Ii ], ⎨ xi (n + 1) = xi (n) − αi (xi (n))[ai (xi (n)) − j=1 (3) m  ⎪ ⎪ tij gi (xi (n − lij (n))) + Jj ]. ⎩ yj (n + 1) = yj (n) − βj (yj (n))[bj (yj (n)) − i=1

The initial conditions associated with (4) are the form:  xi (s) = φi (s), i = 1, . . . , m, s ∈ [−k, 0], yj (s) = ψj (s), j = 1, . . . , p, s ∈ [−l, 0], where k =

max

1≤i≤m,1≤j≤p

{kji (n)}, l =

max

(4)

{lij (n)}. And the delays

1≤i≤m,1≤j≤p

satisfy 1 < kji (n + 1) < 1 + kji (n) and 1 < lij (n + 1) < 1 + lij (n). The rest of this paper is organized as follows: in Section 2 we give some Preliminaries; next, in Section 3 we obtain some sufficient conditions to ensure that the equilibrium of system (3) is globally exponentially stable via constructing two different Lyapunov functions; one example and a figure are given to illustrate the effectiveness of our results in Section 4; finally, in Section 5 the conclusions are given.

2 Preliminaries Throughout this paper, we need the following assumptions: (H1 ) Functions αi (·), βj (·) are both bounded, positive and Lipschitz con+ − m tinuous, furthermore, 0 ≤ α− i ≤ αi (x) ≤ αi < ∞, x ∈ R ; 0 ≤ βj ≤ + βj (y) ≤ βj < ∞, y ∈ Rp . (H2 ) Behaved functions ai (·), bj (·) are bounded and Lipschitz continuous with Lipschitz constants λi , hj such that |ai (μ) − ai (ν)| ≤ λi |μ − ν|, |bj (μ) − bj (ν)| ≤ hj |μ − ν|; furthermore they are reversible and satisfy a˙ i (x) ≥ ξi > 0, b˙ j (y) ≥ rj > 0. (H3 ) Activation functions fj (·), gi (·) are bounded or globally Lipschitz continuous and there exist Lipschitz constants Mi , Nj such that |fj (μ) − fj (ν)| ≤ Nj |μ − ν|, |gi (μ) − gi (ν)| ≤ Mi |μ − ν|. Let R denote the set of real numbers and Rm = R × . . . × R and Rp =

  m

R × . . . × R. Let Z = {· · · , −1, 0, 1 · · ·}, Z0+ = {0, 1, 2, . . . , m + p}; kji ∈ 

 p

Z0+ , lij ∈ Z0+ . For arbitrary x ∈ Rm and y ∈ Rp , xT and y T denote the 1 transpose of x and y, respectively. Define x2 = (xT x) 2 be the vector’s 2norm. Let Rm×m denote the set of all m × m real matrices. For A ∈ Rm×m , the spectral norm of A is defined as A2 = (max{|λ| : λ is an eigen value 1 of AT A}) 2 . Suppose (x∗ , y ∗ ) = (x∗1 , . . . , x∗m , y1∗ , . . . , yp∗ )T be an equilibrium point of system (3). Let ui (n) = xi (n) − x∗i , vj (n) = yj (n) − yj∗ , then we can rewrite (3) into

22

X. Lu

⎧ ui (n + 1) = ui (n) − αi (ui (n) + x∗i )[ai (ui (n) + x∗i ) − ai (x∗i ) ⎪ ⎪ ⎪ p ⎪  ⎪ ⎪ sji [fj (vj (n − kji (n)) + yj∗ ) − fj (yj∗ )]], − ⎨ j=1

⎪ vj (n + 1) = vj (n) − βj (vj (n) + yj∗ )[bj (vj (n) + yj∗ ) − bj (yj∗ ) ⎪ ⎪ m ⎪  ⎪ ⎪ − tij [gi (ui (n − lij (n)) + x∗i ) − gi (x∗i )]]. ⎩

(5)

i=1

For convenience, we let αi (ui (n)) = αi (ui (n)+x∗i ), βj (vj (n)) = βj (vj (n)+yj∗ ), ai (ui (n)) = ai (ui (n) + x∗i ) − ai (x∗i ), bj (vj (n)) = bj (vj (n) + yj∗ ) − bj (yj∗ ), fj (vj (n)) = fj (vj (n) + yj∗ ) − fj (vj∗ ), gi (ui (n)) = gi (ui (n) + x∗i ) − gi (x∗i ), then (5) can be further reduced into ⎧ p  ⎪ ⎪ sji fj (vj (n − kji (n)))], ⎨ ui (n + 1) = ui (n) − αi (ui (n))[ai (ui (n)) − j=1 (6) m  ⎪ ⎪ tij gi (ui (n − lij (n)))]. ⎩ vj (n + 1) = vj (n) − βj (vj (n))[bj (vj (n)) − i=1

Obviously, functions in (6) satisfy H1 and:  (H2 ) Behaved functions ai (·), bj (·) are bounded with constants λi , hj such that |ai (u)| ≤ λi |u|, |bj (v)| ≤ hj |v|; furthermore its are reversible and satisfy a˙ i (u) ≥ ξi > 0, b˙ j (v) ≥ rj > 0.  (H3 ) Activation functions fj (·), gi (·) are bounded with constants Mi , Nj such that |fj (v)| ≤ Nj |v|, |gi (u)| ≤ Mi |u|, for ∀u ∈ Rm , ∀v ∈ Rp .

3 Existence of the Equilibrium Point and Globally Exponential Stability for System (6) It is obvious that the stability of the equilibrium of system (6) implies the stability of the equilibrium of system (3). So we only need to study system (6). 



Theorem 1. For system (6), under assumptions H1 , H2 , H3 , if it also satp m   isfies 1 ≥ ξi Nj |sji |, 1 ≥ rj Mi |tij |, then there exists at least one zero i=1

j=1

equilibrium. The proof of this Theorem is similar to that of existed results [12]. Here we omit it. Our main purpose is to proof the equilibrium of system (6) is globally exponentially stable via constructing proper Lyapunov-Krasovskii functions. 



Theorem 2. Under assumptions H1 , H2 , H3 and conditions of Theorem 1, if there exist γi , ηj (i = 1, . . . , m, j = 1, . . . , p) such that uai (u) ≥ γi u2 , vbj (v) ≥ p m   + − − α+ βj+ Mi |tij |. ηj v 2 and α+ i λi ≤ 1, βj hj ≤ 1, βj ηj > i Nj |sji |, αi γi > i=1

j=1

Global Exponential Stability for Discrete-Time BAM Neural Network

23

Then the zero equilibrium of (6) is unique and globally exponentially stable, i.e. for every solution (u, v) = (u1 , . . . , um , v1 , . . . , vp )T of (6) such that m

p m 1 n |ui (n)| + |vj (n)| ≤ ν( ) [ sup |φi (s)| + sup |ψj (s)|], (7) ξ i=1 s∈[−l,0] i=1 j=1 j=1 s∈[−k,0]

where l =

p

max

1≤i≤m,1≤j≤p

ξ > 1, ν > 1, for n ∈

{lij }, k =

max

1≤i≤m,1≤j≤p

{kji }. ξ, ν are constants and

Z0+ .

Proof. By Theorem 1, there exists an equilibrium and the uniqueness of the equilibrium will be guaranteed by (7). So we only need to certify (7). We can consider functions G1 (·), G2 (·) given by: ˜ = 1−ξ+ ˜ ξα ˜ − γi−ξ˜ G1 (ξ) i

p

˜ =1−ξ+ ˜ ξβ ˜ − ηj −ξ˜ βj+ Mi |tij |ξ˜l , G2 (ξ) j

j=1

m

˜k α+ i Nj |sji |ξ ,

i=1

for ξ˜ ∈ [1, +∞). We note that G1 (1) = α− i γi −

p

βj+ Mi |tij | > 0, G2 (1) = βj− ηj −

j=1

m

α+ i Nj |sji | > 0.

i=1

Using the continuity of G1 (·), G2 (·) on [1, +∞), they will follow that there exists a real number ξ, such that G1 (ξ) = 1 − ξ +

ξα− i γi

−ξ

G2 (ξ) = 1 − ξ + ξβj− ηj − ξ

p j=1 m

βj+ Mi |tij |ξ l ≥ 0,

(8)

k α+ i Nj |sji |ξ ≥ 0.

(9)

i=1

Now let us consider functions Wi (n) = ξ n |ui (n)|, Zj (n) = ξ n |vj (n)|. We obtain Wi (n + 1) = ξ n+1 |ui (n + 1)| = ξ n+1 |ui (n) − αi (ui (n))(ai (ui (n)) −

p

sji fj (vj (n − kji (n))))|

j=1

≤ ξ n+1 |ui (n) − αi (ui (n))ai (ui (n))| + ξ n+1

p

|sji ||αi (ui (n))|

j=1

×|fj (vj (n − kji (n))| ≤ξ

n+1

(1 −

α− i γi )|ui (n)|

+

ξ n+1 α+ i

p j=1

Nj |sji ||vj (n − kji (n))|

24

X. Lu

= ξWi (n) −

α− i γi ξWi (n)

+

α+ i

p

Nj |sji |ξ kji (n)+1 Zj (n − kji (n))

j=1 + ≤ ξWi (n) − α− i γi ξWi (n) + αi

p

Nj |sji |ξ k+1 Zj (n − kji (n)).(10)

j=1

Similarly, we can calculate Zj (n + 1) = ξ n+1 |vj (n + 1)| ≤ ξZj (n) − βj− ηj ξZj (n) + βj+

m

Mi |tij |ξ l+1 Wi (n − lij (n)), (11)

i=1

for i = 1, . . . , m, j = 1, . . . , p, n ∈ Z0+ . According to (10) and (11), we can construct a Lyapunov function V (·) as V (n) = V1 (n) + V2 (n), V1 (n) =

m

[Wi (n) +

i=1

V2 (n) =

α+ i

p

Nj |sji |ξ

n−1

k+1

j=1

p

[Zj (n) + βj+

j=1

m

Zj (q)],

q=n−kji (n) n−1

Mi |tij |ξ l+1

i=1

Wi (q)].

q=n−lij (n)

Calculating the difference V1 (n) = V1 (n + 1) − V1 (n), we have V1 (n) = V1 (n + 1) − V1 (n) p m + [Wi (n + 1) + αi Nj |sji |ξ k+1 = i=1

−

j=1

Zj (q)]

q=n−kji (n+1)+1

p m [Wi (n) + α+ Nj |sji |ξ k+1 i i=1

≤−

n

j=1

n−1

Zj (q)]

q=n−kji (n)

p m m + (1 − ξ + α− γ ξ)W (n) + α Nj |sji |ξ k+1 Zj (n). (12) i i i i i=1

i=1

j=1

Similarly, we can have V2 (n) ≤ −

p

(1 − ξ + βj− ηj ξ)Zj (n) +

j=1

p j=1

βj+

m

Mi |tij |ξ l+1 Wi (n). (13)

i=1

Thus V (n) ≤ −

m i=1

[1 − ξ + α− i γi ξ −

p j=1

βj+ Mi |tij |ξ l+1 ]Wi (n)

Global Exponential Stability for Discrete-Time BAM Neural Network

−

p

[1 − ξ +

βj− ηj ξ

j=1

−

m

k+1 α+ ]Zj (n). i Nj |sji |ξ

25

(14)

i=1

By using (8) and (9), we assert that V (n) ≤ 0, for n ∈ Z0+ , which implies V (n) ≤ V (0), for n ∈ Z0+ . Since V (0) = V1 (0) + V2 (0) =

m

[Wi (0) +

α+ i

i=1

+

p

+

m i=1 p

Nj |sji |ξ

[Zj (0) + βj+

m

p

−1

Mi |tij |ξ l+1

j=1

m

βj+ Mi |tij |ξ l+1 (lij (0) − 1)) sup |Wi (s)| s∈[−l,0]

k+1 α+ (kji (0) − 1)) sup |Zj (s)| i Nj |sji |ξ s∈[−k,0]

i=1 m

≤ ν{ sup

Wi (q)]

q=−lij (0)

j=1

(1 +

Zj (q)]

q=−kji (0)

i=1

(1 +

−1

k+1

j=1

j=1

≤

p

|Wi (s)| +

s∈[−l,0] i=1

sup

p

|Zj (s)|},

(15)

s∈[−k,0] j=1

where ν = max{ max {1 + 1≤i≤m

max {1 +

1≤j≤p

We have

m 

Wi (n) +

i=1 m

ui (n) +

i=1

p 

m

p

βj+ Mi |tij |ξ l+1 (l − 1)},

j=1 k+1 α+ (k − 1)}} > 1. i Nj |sji |ξ

i=1

Zj (n) ≤ V (n). Thus

j=1 p

p m 1 vj (n) ≤ ν( )n { sup |φi (s)| + sup |ψj (s)|}, ξ s∈[−l,0] i=1 s∈[−k,0] j=1 j=1

holds. This completes the proof. 



Corollary 1. Under assumptions H1 , H2 , H3 and the conditions of Theorem 4, if there exist γi (i = 1, . . . , m), ηj (j = 1, . . . , p), such that uai (u) ≥ m  + − γi u2 , vbj (v) ≥ ηj v 2 and α+ α+ i λi ≤ 1, βj hj ≤ 1, βj ηj > i Nj |sji |, α− i γi

>

p  j=1

i=1

βj+ Mi |tij |.

Then the zero equilibrium of (6)is unique and globally

26

X. Lu

exponentially stable, i.e. for every solution (u, v) = (u1 , . . . , um , v1 , . . . , vp )T p p m m     of (6)such that |ui (n)| + |vj (n)| ≤ ν( 1ξ )n [ |ui (0)| + |vj (0)|], where i=1

j=1

i=1

j=1

ξ, ν are constants and ξ > 1, ν > 1, n ∈ Z0+ . Proof. Similarly to the proof of Theorem 2, we consider another Lyapunov p m   function V (n) = Wi (n) + Zj (n), for n ∈ Z0+ .The remaining part of the i=1

j=1

proof is similar to that of Theorem 5 and it’s omitted here. This completes the proof. Considering the following system ⎧ p  ⎪ ⎪ ui (n + 1) = ui (n) − αi (ui (n))[ai (ui (n)) − cji fj (vj (n)) ⎪ ⎪ ⎪ j=1 ⎪ ⎪ p ⎪  ⎪ ⎪ − sji fj (vj (n − kji (n)))], ⎨ j=1 (16) m  ⎪ ⎪ ⎪ v (n + 1) = v (n) − β (v (n))[b (v (n)) − d g (u (n)) j j j j j j ij i i ⎪ ⎪ ⎪ i=1 ⎪ m ⎪  ⎪ ⎪ tij gi (ui (n − lij (n)))], − ⎩ i=1

Obviously, according to the proof of Theorem 1 and Corollary 1, we have the following corollary. 



Corollary 2. Under assumptions H1 , H2 , H3 , if there exist γi (i = 1, . . . , m), ηj (j = 1, . . . , p), such that uai (u) ≥ γi u2 , vbj (v) ≥ ηj v 2 and α+ i λi ≤ 1, m p  + − + − + βj hj ≤ 1, βj ηj > αi Nj (|cji | + |sji |), αi γi > j=1 βj Mi (|dij | + |tij |). i=1

Then the zero equilibrium of (16)is unique and globally exponentially stable, i.e. for every solution (u, v) = (u1 , . . . , um , v1 , . . . , vp )T of (16)such that m i=1

|ui (n)| +

p

p m 1 |vj (n)| ≤ ν( )n [ |ui (0)| + |vj (0)|], ξ i=1 j=1 j=1

where ξ, ν are constants and ξ > 1, ν > 1, n ∈ Z0+ . Remark 1. In this paper, we have derived some analogous conditions via constructing different Lyapunov functions and applying different distinct analysis technique in the proof. Comparing Theorem 1 with Theorem 2, we can conclude that our results are independent on delays. Remark 2. In [14], the authors considered a special case of the model (3) as αi (xi (n)) = 1, βj (yj (n)) = 1, ai (xi (n)) = 1, bj (yj (n)) = 1. In other words, this shows that our results are general and new.

Global Exponential Stability for Discrete-Time BAM Neural Network

27

4 Example In this section, some examples are presented to illustrate the feasibility and effectiveness of our results. Consider the following discrete-time CohenGrossberg type BAM neural network model with delay. Considering the neural network (6) with parameters: f1 (y) = f2 (y) = sin(y), g1 (x) = g2 (x) = 1 2 sin(x), 



2+sin(x1 (n)) 4−sin(y1 (n)) 0 0 4 6 , β(y) = , α(x) = 2+cos(x2 (n)) 4−cos(y2 (n)) 0 0 4 6      y1 (n) x1 (n) 10 10 , b(y) = , 01 01 x2 (n) y2 (n)

√

√ √  √  2 2 2 2 −√ − 16 16 √4 √4 √ S= , T = . 2 2 − 162 − 162 4 4 

a(x) =

+ Thus we can obtain N1 = N2 = 1, M1 = M2 = 0.5. α+ 1 = α2 = 0.75, + + − − − − 5 β1 = β2 = 6 , α1 = α2 = 0.25, β1 = β2 = 0.5, λ1 = λ2 = 1, h1 = h2 = 1, η1 = η2 = 1, γ1 = γ2 = 1. So, the conditions of Theorem 2, α+ i λi = 0.75 < 1, βj+ hj = 56 < 1,

√ 2 1 < 0, + + = −0.25 + × 2 4 √ 2 1 + × < 0, −βj− ηj + Nj (α+ 1 |sj1 | + α2 |cj2 |) = −0.5 + 2 16

−α− i γi

Mi (β1+ |ti1 |

β2+ |di2 |)

hold, where i, j = 1, 2. Consequently the origin of the system (17) is globally exponentially stable. We can also see from Fig.1. In Fig.1, we also take the Fig. 1 The numeric simulation of state variables x(n) and y(n) of system (17)

1 x1(n) x2(n) y1(n) y2(n)

0.8

0.6

x1(n), x2(n), y1(n), y2(n)

0.4

0.2

0

−0.2

−0.4

−0.6

−0.8

−1

0

2

4

6

8 n

10

12

14

28

X. Lu

initial values as (x1 (0), x2 (0), y1 (0), y2 (0), )T = (1, 0.5, −0.5, −1)T and the delay kji = 1, lij = 1. However, using the criteria in [14], it is difficult to make certain of the exponential stability of examples (17). This means that our results are new and general.

5 Conclusions In this paper, the main purpose is to study the existence and the global exponential stability of the equilibrium point for discrete-time Cohen-Grossberg type BAM networks with variable delay. Our model is more general and we generalize the prevent authors’ techniques and results. Therefore we have derived some sufficient independent conditions ensuring global exponential stability of the equilibrium of discrete-time Cohen-Grossberg type BAM networks via Lyapunov functional approach, inequality analysis technique and matrix theory.

References 1. Arika, S., Tavsanoglu, V.: Global asymptotic stability analysis of bidirectional associative memory neural networks with constant time delays. Neurocomputing 68, 161–176 (2005) 2. Gopalsamy, K., He, X.Z.: Delay-independent stability in bidirectional associative memory networks. IEEE Trans. on Neural Networks 5, 998–1002 (1994) 3. Li, C.D., Liao, X.F., Zhang, R.: Dealy-dependent exponential stability analysis of bi-directional associative memory neural networks with time delay: an LMI approach. Chaos, Solution and Fractals 24, 1119–1134 (2005) 4. Liao, X.F., Wong, K.W., Yang, S.Z.: Convergence dynamics of hybrid bidirectional associative memory neural networks with distributed delays. Physics Letters A 316, 55–64 (2003) 5. Liu, Y.R., Wang, Z.D., Liu, X.H.: Global asymptotic stability of generalized bidirectional associative memory networks with discrete and distributed delays. Chaos, Solitons and Fractals 28, 793–803 (2006) 6. Lou, X.Y., Cui, B.T.: Absolute exponential stability analysis of delayed bidirectional associative memory neural networks. Chaos, Solitons and Fractals 31, 695–701 (2007) 7. Wang, H.X., He, C., Yu, J.B.: Analysis of global exponential stability for a class of bi-directional associative memeory networks. Circuits and Systems 5, 673–676 (2003) 8. Zhao, H.Y.: Exponential stability and periodic oscillatory of bi-directional associative memory neural network involving delays. Neurocomputing 69, 424–448 (2006) 9. Feng, Z.S., Michel, N.: Robustness analysis of a class of discrete-time recurrent neural networks under perturbations. IEEE Trans. on Circuits and Systems I 46, 1482–1486 (1999)

Global Exponential Stability for Discrete-Time BAM Neural Network

29

10. Ma, K.L., Peng, J.G., Xu, Z.B., Yiu, K.F.C.: A new stability criterion for discrete-time neural networks: Nonlinear spectral radius. Chaos, Solitons and Fractals 31, 424–436 (2007) 11. Mohamad, S., Gopalsamy, K.: Dynamics of a class of discrete-time neural networks and their continuous-time counterparts. Mathematics and Computers in Simulation 53, 1–39 (2000) 12. Mohamad, S., Gopalsamy, K.: Exponential stability of continuous-time and discrete-time cellular neural networks with delays. Applied Mathematics and Computation 135, 17–38 (2003) 13. Xiong, W.J., Cao, J.D.: Global exponential stability of discrete-time CohenGrossberg neural networks. Neurocomputing 64, 433–446 (2005) 14. Liang, J.L., Cao, J., Ho, D.W.C.: Discrete-time bidirectional associative memory neural networks with variable delays. Physics Letters A 335, 226–234 (2005) 15. Cao, J., Liang, J.L.: Boundedness and stability for Cohen-Grossberg neural network with time-varying delays. Journal of Mathematical Analysis and Applications 296, 665–685 (2004) 16. Cao, J., Li, X.L.: Stability in delayed Cohen-Grossberg neural networks: LMI optimization approach. Physica D: Nonlinear Phenomena 212, 54–65 (2005) 17. Liao, X.F., Li, C.G., Wong, K.W.: Criteria for exponential stability of CohenGrossberg neural networks. Neural Networks 17, 1401–1414 (2004) 18. Song, Q.K., Cao, J.D.: Stability analysis of Cohen-Grossberg neural network with both time-varying and continuously distributed delays. Journal of Computational and Applied Mathematics 197, 188–203 (2006) 19. Liu, X.G., Tang, M.L., Martin, R., Liu, X.B.: Discrete-time BAM neural networks with variable delays. Physics Letters A 367, 322–330 (2007)

The Study of Project Cost Estimation Based on Cost-Significant Theory and Neural Network Theory Xinzheng Wang, Liying Xing, and Feng Lin 1

Abstract. Based on the reference to domestic and foreign correlative theories and methods, cost-significant theory and neural network theory are used to estimate project cost in the paper. The cost-significant theory is put forward to solve the tedious operation issues by finding out significant items to simplify the operational difficulty of engineering cost estimation. Then the BP neural network is applied to “distill” the data of CSIs and csf from the completed projects. It has realized the accurate prediction of project investment by using the two nonlinear theories The basic theories of CS and BP neural network are illustrated by an example From the example it shows that the relative errors are so small that they can meet the accurate demands of cost estimation. Meanwhile, the test results show that the model based on cost- significant theory and neural network theory is successful and effective for practical engineering

． ．

，

．

Keywords: Cost estimation, Cost-significant theory, BP neural network, Investment control.

1 Introduction The construction investment estimation is an important content of the project’s feasibility study. The accuracy of the cost estimation directly affects the project’s decision, construction’s scale, design scheme and economic effects, and affects the project’s proceeding. It is significant for the management and control of the project estimation to process the estimate handily, quickly and exactly. The main feature of the engineering’s estimation is that there are too many factors which can affect the project cost, and there is a highly nonlinear mapping relationship between the project cost and these uncertain factors. According to the traditional estimation method, quota index has certain and static character (such as Investment Estimate Fixed Estimation), and the traditional estimation method is to build model XinzhengWang . Liying Xing . Feng Lin School of Civil Engineering, Nanyang Normal University, Nanyang, Henan 473061,China [email protected] H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 31–38. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

32

X. Wang, L. Xing, and F. Lin

according to a linear relationship between the project’s cost and its factors (such as Units of Production Capacity Estimation, Production Capacity Index Estimation, Proportion Estimation, Lange Coefficient Method, Capital Turnover Method etc), which is not accurate of the practical condition. To solve these problems, this paper tries to research our country project cost estimation method with the combination of the cost-significant theory and the BP neural network theory. Firstly, the estimation budget and the budget estimate programming according to the cost-significant theory is simplified. Secondly, based on this foundation, this paper aims to “discover and seek” the similar cost-significant items (CSIs) and factors (csf ) data from a large quantity of project’s historical information that based on the BP neural network. Then we will discover and obtain the nonlinear relationship between the uncertain factors (which influence the project cost) and the construction cost, therefore, realize the investment estimation of the proposed project and reach to the purpose of simplifying estimate procedure and improving estimation accuracy.

2 Basic Thoughts of Cost-Significant Theory CS theory is the cost-significant theory, which theoretical thoughts originated from Italy economists Vilfred Pareto founds, that is social wealth is not well-distributed, about twenty percent population own eighty percent of social wealth. British Dundee university Malcolm Horner applied the "cost-significant theory" to the construction cost estimation research, founding that about twenty percent total quantity items bear about eighty percent of construction total cost, we say that the twenty percent items is the Cost-significant Items (CSIs), others are the non-Cost-significant Items (non-CSIs) [1]. Although the non-Cost-significant Items have less effect on the total cost, its computing workload is far more than the CSIs’ computing workload. Therefore we just only pay attention to the cost-significant items, catch and study on the key sectional engineering from various sectional engineering according to the key ideas, thereby speculate the whole project’s and the similar project’s cost, which can achieve to the purpose of both simplifying calculation workload and guarantee the accuracy of the investment estimation[2-4].

2.1 The Search Steps of Csis (1) Calculate the total construction cost C and the total amount N. Here need to explain the count of quantities items, firstly select out the level of N construction, which is counted in the total by construction scale, secondly account the number N, which only occur expenses item among the same level standard items. (2) Calculate the average cost T of the construction. The average cost of projects is the ratio of the total project cost and the total project number, namely T = C/N. (3) Find the CSIs by comparing with the single construction cost and the average cost. Generally, if the single project cost is more than the average cost, we called it as cost-significant items (CSIs); if less than the average cost, called it as non-cost-significant items (non - CSIs). In case the CSIs cannot guarantee within thirty percent, we can carry on the second average [5].

The Study of Project Cost Estimation

33

2.2 The Consistency of CSIs Foreign studies show that, the similar construction has the same CSIs, and the"cost-significant factor" is very stable, which is the ratio of CSIs cost and total project cost. Although the CSIs may not exactly the same in different project quantities detailed list, CSIs exists a lot of similarities in the similar project. Many studies also show that[6], similar project quantities detailed lists have almost an identical CSIs. In foreign constructions, there have many CSIs models for the construction cost estimation. The results indicate that, the csf data of different projects is between 10% and 27%, the investment estimation precision is between 4% and 10%, which satisfies the precision demand of investment estimation.

2.3 The Application Step of CSIs Model If the same type completed projects have the same CSIs, and its " cost-significant factor" is stable, proposed projects may carry on the investment estimation by means of the following ways: choose the CSIs from the similar completed project’s quantities detailed list by Mean theory, calculate the similar project’s thus estimate the proposed project CSIs’ cost and its cost-significant factor; cost-significant factor; the proposed project cost equals the ratio of CSIs cost and mean of cost-significant factor(csf). Thus, the CSIs can greatly simplify the investment estimation’s computation procedure of the feasibility study phase, and will not affect the accuracy of measurement. However, to reach this purpose, the distinguish between the same type projects and the similar projects plays an important role. If artificial distinguish, it must cause a big error and a bad stability. This paper discusses to adopt the neural network method, under having many completed construction’s CSIs and csf data condition, judge the same type projects or similar engineering process by simulating the people’s brain "experience", thus predict the proposed project’s CSIs cost and the cost-significant factor.

② ③

①

3 The Applications of BP Network Artificial neural network is a method of information processing, which is developed by the biological neural systems inspired. Based on the learning sample process, the artificial neural network analysis data, a model is built and then finds new knowledge. Neural network can automatically adjust the neurons input and output in accordance with the rules through learning, to change the internal state[7-9]. In all kinds of neural network models, the BP neural network is a widen-applied neural network model. The standard BP network is made up of three kinds of neural unit layers. The lowest layer is called as the input layer. The middle one is called as the implication layer (can be multi-layer). And the top one is called as the output layer. Every layer of neural unit forms fully-connection, and the neural units in each layer have no connection. The study course of BP algorithm is made up of

34

X. Wang, L. Xing, and F. Lin

Error reverse pass

Excepted output

Output layer

Input layer Hidden layer

Fig. 1 BP neural network basic structure [10-11]

propagation and back-propagation . In the propagation course, the input information is transferred and processed through input layer and implication layer. The state of every neural unit layer only affects the state of next layer. If the expected information cannot be got in the output layer, the course will turn into the back-propagation and return the error signal along the former connection path. Altering the connection weight between each layer, the error signal is transmitted orderly to the input layer, and then passes the propagation course. The repeated application of these two courses makes the error become smaller and smaller, until it meets the demand. The specific structure can be seen in Fig.1. In this picture, the relationship between the input and output of neural unit (except the input layer) is nonlinear mapping, and S(Sigmoid) function is always been adopted. f(x)=1/(1+e-x) is the node-outputting function, and the differential coefficient is f’(x)=f(x)(1-f(x)). Its advantage is that, whenever inputting any data can be transformed into the numbers that are in 0,+1 .

（ ）

3.1 BP Neural Network Application in Csis Estimates For a large number of typical completed projects budget (or bill of quantities) cost information, we can firstly analysis every project item’s CSIs sectional item, total project cost, and the cost-significant factor data by mean theory. And secondly sort out the information by some certain format according to project data analysis and engineering properties, which is taken as the training sample. Thirdly input in the neural network to training, therefore complete the mapping from input layer (engineering characteristics) to output layer (CSIs data). That is a highly nonlinear neural network model, that can automatically extract this knowledge, and store the network weights in neural network inside. So project cost personnel can get the proposed project’s CSIs and csf, by inputting the relative information into the neural network about the characteristics of proposed project.

The Study of Project Cost Estimation

35

3.2 Build BP Neural Network Model of Investment Estimation The investment estimation model of neural network is composed of input preprocessing module, neural network module and output processing module. Input preprocessing module is mainly pre-processing the input data, changing the qualitative things into the quantitative data, what is easy for neural network operating. The output processing module will transform the neural network output into the investment estimate data what we need, whose core is the neural network module.

3.3 Case Analysis

，

This article takes the highway investment estimation for example analyzes the different constructions characteristic’s CSIs and cfs data, by collecting and sorting out the material cost, labor cost, levy land cost and project construction cost of completed highway construction projects. The selection of the project’s feature should consult statistics and analysis of the historical projects’ materials. And it can be sure by the expert experience. We can analyze the effect that the cost of the typical highway project and the change of construction’s parameter make for the estimation of the project’s investment. Then, we confirm eight main factors that are landform, cross-section’s type ( cutting excavation, embankment, half-digging and half-filling), height, and width, foundation processing type , road surface’s material and thickness, guard project’s type and so on as the project’s features. And then list the different types of project’s feature, and lead to the change of every kilometer highway engineering’s cost according to quota standard and engineering characteristic to being mutually related nature of cost influence, and sequence them from childhood to the big, and the Tab.1 is seen to the subjective preset corresponding quantification data at last. To illustrate this problem, we can simply calculation, choose twenty typical construction CSIs remit total(taking the seventeen and twentieth groups data as testing samples, the first and sixteen groups data as learning samples) to set up a BP neural network investment estimation model. From Table 1, it shows that any highway project’s model can be given a quantitative description. Taking Ti = (ti1 , ti 2 , , ti 8 ) as an example, Ti is the serial

number of project i (i = 1,2, ) ; tij ( j = 1, 2, ,8) is the quantitative numeric value of the first j feature of the first i project. For instance, some highway project (the serial number is assumed as i ) in the plain, and the type of cross section is embankment and highway grade for at a high speed with height 1.8m of roadbed cross section, width 35m and manson grizzly screen, pitch concrete, common guard and thickness 0.45m of road surface’s structure. Therefore, its quantitative description is Ti = (3, 2, 4, 6,3,1,1, 4) . If some feature is made up of several kinds, count its weighed average according to the proportion and make it as its quantification result. This model adopts three layers of BP network model, and chooses f(x)=1/(1+e-x) as the node-outputting function. The units of model’s input layer are eight, and they stand for project’s characteristic vectors, like landform, cross-section’s type, cross-section’s height, cross-section’s width, foundation processing type, material of road surface’s structure, road surface’s thickness and protection type, and they

36

X. Wang, L. Xing, and F. Lin

are expressed by I1~I8; the two output units are the per- kilometer CSIs millon yuan cost and the csf, and respectively expressed by O1 O2 [12]. The value of implication layer’s unit is seventeen according to Ke Ermoge luofu’s theorem. Commonly, choose the random numbers whose original weight is between(-1 1). Collect sixteen training samples and four testing samples according to the complexity degree of the inputting-outputting mapping. List the twenty typical samples’ characteristic fixed quantity data and the data of O1 O2, in Tab.2.

、

，

、

3.4 The Analysis of Test’s Result Use the network after constringency to test the data of group seventeen and twentieth,, and take its average as the predictive value of O1 O2, and the results are showed in Tab.3, O1/O2 is the per-kilometer project investment total estimation of the predictive sample. It can be seen from the test’s result that the relative error between the real value and predictive value is less than 8%, the overall error ratio is small, and the need for the estimation of engineering feasibility study can be basically satisfied. This shows that the model’s generalization ability is better and the estimation model is successful.

、

Table 1 The types of achievement highway project’s features fixed quantity table The fixed quantitative value of feature

1

2

3

Landform

maintain area

hill

plain

The type of foundation’s cross-section

cutting excavation

4

5

6

Embankment

half-diggin g and half-filling

cutting excavationǃ embankment

cutting excavationǃ half-digging and half-filling

embankmentǃ half-digging and half-filling

0~0.5

0.5~1

1~1.5

1.5~2

2~2.5

more than 2.5

0~20

20~30

30~40

40~50

50~60

more than 60

The type of foundation

general replacement

plastic board drain concretion

mason grizzly screen

sand pile drain concretion

punning, stir pile

geo-textile

The material of road surface’s structure

bitumen& concrete

cement& concrete

Guard project

common guard

anchor plate revetment

gravitation retaining wall

spray-net strut

Plate girder strut

grass slope preserve

The thickness of road surface’s structure/m

0~0.2

0.2~0.3

0.3~0.4

0.4~0.5

0.5~0.6

more than 0.6

The height of foundation’s cross-section/ m The width of foundation’s cross-section/ m

The Study of Project Cost Estimation

37

Table 2 The table of typical sample’s characteristic fixed quantity data and budget material

Serial number

Output(ten thousand yuan/Km)

Entry

I1

I2

I3

I4

I5

I6

I7

I8

O1

O2

1 2 3 4 5

3 2 1 2 3

2 1 4 4 1

1 1 2 2 1

2 2 2 3 3

1 3 2 6 1

1 2 1 2 1

2 1 5 3 1

6 4 3 3 4

1608 1823 2400 2528 1723

0.782 0.807

6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

3 1 1 3 2 2 3 1 3 1 3 3 1 1 3

6 1 4 2 1 2 2 4 2 4 1 4 4 3 2

4 1 3 5 1 6 6 5 6 5 6 1 2 4 2

5 5 3 3 4 5 6 4 6 4 6 3 2 3 3

6 1 3 6 4 6 2 4 3 3 1 1 6 1 1

1 2 4 1 1 2 1 1 1 1 1 2 1 1 1

6 5 5 1 4 6 4 3 1 1 6 6 3 6 1

4 3 2 3 4 5 5 5 5 4 5 4 3 4 6

2346 3113 2361 1742 2240 2468 2521 2904 2232 2704 2400 1906 2511 2389 1826

0.824

0.811 0.794 0.778 0.813 0.806 0.814 0.794 0.782 0.815 0.808 0.825 0.802 0.805 0.798 0.801 0.806 0.812

Table 3 Results analysis Serial number predictive value of O1 actual value of O1 relative error of O1 (%) predictive value of O2 actual value of O2 relative error of O2 (%) predict Km/millon yuan actual Km/millon yuan relative error%

17 1906 1990 -4.22 0.817 0.798 2.4 2332.9 2493.7 -6.4

18 2511 2492 0.8 0.790 0.801 -1.3 3178.5 3111.1 2.1

19 2389 2490 -4.4 0.824 0.806 2.2 2899.3 3089.3 -6.2

20 1826 1855 -1.6 0.790 0.812 -2.7 2311.4 2284.5 1.2

4 Conclusion In the paper, the cost-significant theory and the neural network theory are used to estimate the engineering cost. According to the thought of CS theory, some key items, namely the significant items, are excavated from numerous subentry projects so as to reduce the operational difficulty of the project cost estimation and the calculation work in a large degree. Meanwhile, neural network theory is applied to extract the similar cost-significant items (CSIs) and the cost-significant factors (csf) from a large number of historical project cost data automatically. In addition, for its highly fault-tolerance, the neural network has the automatic function to correct the

38

X. Wang, L. Xing, and F. Lin

deviation of the completed projects, which is caused by someone or other factors. The neural network also has an automatic matching and correcting function for the similarities between the cost-significant project and factors. Furthermore, because the neural network has a function of parallel processing data, the data can be processed in a high speed, and thus the neural network can meet the requirements of rapid estimation. It shows that it is efficient, feasible and accurate to use the cost-significant theory and the neural network to estimate the investment of highway project. At the same time, it has some reference value and academic significance for adopting new scientific methods to promote the study on the estimation of project investment.

References 1. Malcolm, R., Horner, W.: New Property of Numers-The Mean Value and its Application to Data Simplification. The Papers to the Royal Society, London (2004) 2. Zakieh, R.: Quantity-significance and Its Application to Construction Project Modelling. University of Dundee, Dundee (1991) 3. AI-Hubail, J.: Modelling Operation and Maintenance Expenditures in the Offshore Oil and Gas Industry. Department of Civil Engineering of The University of Dundee, Dundee (2000) 4. Asif, M.: Simple Generic Models for Cost-Significant Estimating of Construction Project Cost. Department of Civil Engineering of The University of Dundee, Dundee (1988) 5. Duan, X.C., Zhang, J.W.: The Government Investment Project Overall Investment Control Theory and Study Method. Science press, Beijing (2007) 6. Wang, N.: New Approaches to Optimising the Whole Life Performance of Building and Civil Engineering Projects, Department of Civil Engineering of the University of Dundee, Dundee (2005) 7. Shao, F.J., Yu, Z.Q.: Principle and method of Data mining. China HydropowerPress, Beijing (2003) 8. Wang, W.: Artificial Neural Network Theory-Application and Introductory. Beijing Aeronautics and Astronautics University Press, Beijing (1996) 9. Jiao, L.C.: Neural Network Theory. Xi’an Electronic Technology University Press, Xi’an (1994) 10. Cheng, M.: Neural Network Model. Dalian University of Science and Technology Press, Dalian (1995) 11. Yan, P.F., Zhang, C.S.: Artificial Neural Network and Simulation Calculation. Tsinghua University Press, Beijing (2000) 12. Zhou, L.P., Hu, Z.F.: The Application of Neural Network in the Cost Estimation of Construction. Journal of Xi’an University of Architecture& Technology 37(2), 261–264 (2005)

Global Exponential Stability of High-Order Hopfield Neural Networks with Time Delays Jianlong Qiu and Quanxin Cheng

Abstract. In this paper, the global exponential stability is studied for a class of high-order Hopfield neural networks (HHNNs) with time delays by employing Lyapunov method and linear matrix inequality (LMI) technique. Simple sufficient conditions are given ensuring global exponential stability of HHNNs. The proposed results improve some previous works and do not require the symmetry of weight matrix. In addition, the proposed conditions are easily checked by using the Matlab LMI Control Toolbox.

1 Introduction It is well known that neural networks play an important role in many fields, such as pattern recognition, signal processing, associate memory, and optimization. These applications of neural network heavily depend on its strong approximation property. In this point, high-order neural networks does better than ordinary neural networks, that is to say, high-order neural networks have stronger approximation property, faster convergence rate, great stronger capacity, and higher fault tolerance. Due to this, recently HHNNs have attracted considerable attention, see [1-8]. In [2], the absolute stability of highorder neural networks is studied. In [7], some criteria are derived to ascertain global asymptotic stability for high-order Hopfield type neural networks, and in [8] some sufficient conditions are presented for the exponential stability of Jianlong Qiu School of Automation, Southeast University, Nanjing 210096, China and Department of Mathematics, Linyi Normal University, Linyi 276005, China Quanxin Cheng Department of Mathematics, Southeast University, Nanjing 210096, China [email protected] H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 39–47. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

40

J. Qiu and Q. Cheng

high-order BAM neural networks. These previous literature only consider the stability of high order neural networks. In reality, studies on neural dynamical systems not only involve discussion of stability property, but also involve many dynamics behavior such as periodic oscillatory, bifurcation and chaos. In many applications, the property of periodic solutions is of great interest. Motivated by the above discussion, this paper studys the exponential stability of HHNNs. The proposed results are given in the form of linear matrix inequality which do not require the symmetry of the weight matrix. Moreover, an illustrative example is given to demonstrate the effectiveness of the obtained results. The remaining paper is organized as follows. In Section 2 the model formulation and some preliminaries are given. The main results are stated in Section 3. Finally, concluding remarks are made in Section 4. Notation. Throughout this paper the following notations will be used. Let A = (aij ) be a n × n dimensional real matrix. AT , A−1 , λmax (A), λmin (A) denotes, respectively, the transpose of, the inverse of, the minimum eigenvalue of and the maximum eigenvalue of a square matrix A. The notation A > 0 means that A is symmetric and positive definite. For x ∈ Rn , its norm is n √  defined by ||x|| = xT x, M ∗ = Mj2 , K = diag(K1 , K2 , . . . , Kn ), L = j=1

diag(L1 , L2 , . . . , Ln ).

2

Model Formulation and Preliminaries

Consider the following second-order neural networks with time-delays modelling by the set of differential equations x˙ i (t) = −ai xi (t) +

n j=1

+

n n

bij fj (xj (t)) +

n

cij gj (xj (t − τ ))

j=1

dijl gj (xj (t − τ )) gl (xl (t − τ )) + Ii (t),

(1)

j=1 l=1

where i = 1, 2, . . . , n; t > 0; xi (t) denote the potential (or voltage) of the ith neuron at time t; ai are positive constants, and denote the rate with which the ith unit will reset its potential to the resting state in isolation when disconnected from the network and external inputs at time t; time delay τ is non-negative constant, which correspond to finite speed of axonal signal transmission; bij , cij , dijl are the first-order and second-order connection weights of the neural network, respectively; Ii denote the ith component of an external input source introduced from outside the network to the ith neuron. Throughout this paper, the activation functions are assumed to satisfy the following assumptions

Global Exponential Stability of High-Order Hopfield Neural Networks

41

j > 0 such that | j , for all x ∈ R, j = (H1 ) There exist numbers M gj (x)| ≤ M 1, 2, . . . , n;  i > 0, K  j > 0 such that (H2 ) There exist numbers L 0≤

fi (x) − fi (y) i, ≤L x−y

0≤

gj (y) gj (x) −   j , ≤K x−y

for all x, y ∈ R (i, j = 1, 2, . . . , n). The initial conditions associated with system (1) are of the form xi (t) = ϕi (t),

−τ ≤ t ≤ 0,

(2)

in which ϕi (t) (i = 1, 2, . . . , n) are continuous functions. Under the assumptions (H1 ) and (H2 ) system (1) has an equilibrium point X ∗ , where X ∗ = [x∗1 , x∗2 , . . . , x∗n ]T . Denote yi (t) = xi (t) − x∗i , fj (yj (t)) = fj (yj (t) + x∗j ) − fj (x∗j ), gj (yj (t)) = gj (yj (t) + x∗j ) − gj (x∗j ), so it is obvious that functions fj (·), gj (·) have the following properties (A1 ) There exist numbers Mj > 0 such that |gj (x)| ≤ Mj , for all x ∈ R, j = 1, 2, . . . , n; (A2 ) There exist numbers Li > 0, Kj > 0 such that 0≤

fi (x) − fi (y) ≤ Li , x−y

0≤

gj (x) − gj (y) ≤ Kj , x−y

for all x, y ∈ R (i, j = 1, 2, . . . , n). System (1) is transformed into y˙ i (t) = −ai yi (t) +

n j=1

+

n n

bij fj (yj (t)) +

n

cij gj (yj (t − τ ))

j=1

  dijl gj (xj (t − τ )) −  gj (x∗j )  gl (xl (t − τ ))

j=1 l=1

   + gl (xl (t − τ )) − gl (x∗l ) gj (x∗j ) n n = −ai yi (t) + bij fj (yj (t)) + cij gj (yj (t − τ )) j=1

+

n n

j=1

(dijl + dilj ) ξl gj (yj (t − τ )),

(3)

j=1 l=1

where ξl = dijl /(dijl + dilj ) gl (xl (t − τ )) + dilj /(dijl + dilj ) gl (x∗l ) when ∗ dijl + dilj = 0, it lies between gl (xl (t − τ )) and  gl (xl ); otherwise ξl = 0.

42

J. Qiu and Q. Cheng

Denote Y (t) = (y1 (t), y2 (t), . . . , yn (t))T , F (Y (t)) = (f1 (y1 (t)), f2 (y2 (t)), . . .,fn (yn (t)))T ; G(Y (t − τ )) = (g1 (y1 (t − τ )), g2 (y2 (t − τ )), . . . , gn (yn (t − τ )))T ,A = diag(a1 , a2 , . . . , an ), B = (bij )n×n , C = (cij )n×n (where we do not assume that matrixes B, C to be symmetric); D = (D1 + D1T , D2 + D2T , . . . , Dn + DnT )T , where Di = (dijl )n×n , S = diag(ξ, ξ, . . . , ξ)n×n , where ξ = [ξ1 , ξ2 , . . . , ξn ]T . Then system (3) becomes in the following vector-matrix form Y˙ (t) = −AY (t) + BF (Y (t)) + CG(Y (t − τ )) + S T DG(Y (t − τ )).

(4)

It is obvious that the global exponential stability of the origin of system (4) is equivalent to the global exponential stability of the equilibrium point X ∗ of system (1). Definition 1. The equilibrium point X ∗ of system (1) is said to be globally exponentially stable, if there exist constants k > 0 and γ ≥ 1 such that, for t ≥ 0, ||X(t) − X ∗ || = ||Y (t)|| ≤ γe−k(t−τ ) sup ||Y (s)||. s∈[−τ, 0]

To give our main results, we also need the following lemmas. Lemma 1 [12]. Suppose W, U are any matrices,  is a positive number and matrix H = H T > 0, then the following inequality holds W T U + U T W ≤ W T HW + −1 U T H −1 U. Lemma 2 [13]. (Schur complement) The following linear matrix inequality (LMI)   Q(x) S(x) > 0, S T (x) R(x) where Q(x) = QT (x), R(x) = RT (x), and S(x) depend affinely on x, is equivalent to (1) R(x) > 0, Q(x) − S(x)R−1 (x)S T (x) > 0, (2) Q(x) > 0, R(x) − S(x)Q−1 (x)S T (x) > 0.

3 Global Exponential Stability of HHNNs In this section, some criteria are given for checking the global exponential stability of HHNNs by constructing suitable Lyapunov functional and employing LMI method. Theorem 1. Under the assumptions (H1 )–(H2 ), the equilibrium point X ∗ of system (1) is unique and globally exponentially stable if there exist positive

Global Exponential Stability of High-Order Hopfield Neural Networks

43

definite matrices P , Q, Σ, positive diagonal matrix R = diag(r1 , r2 , · · · , rn ) and constants β, i > 0, i = 1, 2, 3, 4 such that ⎡ ⎤ √ H P B P C βLB T P ⎢ B T P In×n 0 ⎥ 0 0 ⎢ T ⎥ ⎢ C P ⎥ > 0, 0  Σ 0 0 (5) 1 ⎢√ ⎥ ⎣ βBL 0 ⎦ 0 In×n 0 2 P 0 0 0 M ∗ In×n and 1 Σ + (2 + β4 )DT D + β3 C T C − 2Q ≤ 0,

(6)

−1 ∗ 2 T where H = AT P + P A − 2KQK − L2 − β(2 + −1 3 + 4 M )KR K − βA A.

Proof. From Lemma 2, we know that condition (5) is equivalent to −1 ∗ 2 T AT P + P A − 2KQK − L2 − β(2 + −1 3 + 4 M )KR K − βA A −1 −1 ∗ 2 T −1 T T −2 M P − P BB P − 1 P CΣ C P − βLBB L > 0

then there exists a scalar k > 0 such that −1 ∗ 2 AT P +P A−2kP −2e2kτ KQK − L2 − βAT A − β(2 + −1 3 + 4 M )KR K

−1 ∗ 2 T −1 T C P − βLBB T L ≥ 0. (7) − 2kβRK − −1 2 M P −P BB P −1 P CΣ

Consider the Lyapunov functional n  y (t)  V (Yt ) = e2kt Y T (t)P Y (t) + 2βe2kt ri 0 i gi (s)ds i=1 t +2 t−τ e2k(s+τ ) GT (Y (s))QG(Y (s))ds.

Calculate the derivative of V (Yt ) along the trajectory of (4), we have V˙ (Yt )|(4) = e2kt 2kY T (t)P Y (t) + 2Y T (t)P Y˙ (t) ! yi (t) n n +4βk ri gi (s)ds + 2β ri gi (yi (t))y˙i (t) +2e

i=1 2kτ T

0

i=1

" G (Y (t)QG(Y (t)) − 2GT (Y (t − τ ))QY (Y (t − τ ))

= e2kt Y T (t)[2kP − P A − AT P ]Y (t) + 2e2kτ GT (Y (t)QG(Y (t)) +2Y T (t)P BF (Y (t)) + 2Y T (t)P CG(Y (t − τ )) −2βGT (Y (t))RAY (t) + 2βGT (Y (t))RBF (Y (t)) ! yi (t) n ri gi (s)ds +2βGT (Y (t))RCG(Y (t − τ )) + 4βk i=1

0

" +2Y T (t)P S T DG(Y (t−τ ))−2GT (Y (t − τ ))QG(Y (t−τ )) . (8)

44

J. Qiu and Q. Cheng

By Lemma 1, we have 2Y T (t)P BF (Y (t)) ≤ Y T (t)[P BB T P + L2 ]Y (t), 2Y (t)P CG(Y (t − τ )) ≤ T

2Y (t)P S DG(Y (t − τ )) ≤ T

0≤

T

n i=1

! ri 0

yi (t)

gi (s)ds ≤

(9)

T −1 T −1 C P Y (t) 1 Y (t)P CΣ T +1 G (Y (t − τ ))ΣG(Y (t − τ )), T T −1 2 Y (t)P S SP Y (t) T +2 G (Y (t − τ ))DT DG(Y (t − τ )), ! yi (t) n 1 ri ki sds = Y (t)T RKY (t), 2 0 i=1

− 2βGT (Y (t))RAY (t) ≤ βY T (t)(KR2 K + AT A)Y (t),

(10) (11) (12)

(13)

2βGT (Y (t))RBF (Y (t)) ≤ βY T (t)(KR2 K + LB T BL)Y (t), T 2 2βGT (Y (t))RCG(Y (t − τ )) ≤ −1 3 βY (t)KR KY (t)

(14)

+3 βG(Y (t − τ ))T C T CG(Y (t − τ )), (15) ∗ T 2 2βGT (Y (t))RS T DG(Y (t − τ )) ≤ −1 4 βM Y (t)KR KY (t) +4 βG(Y (t − τ ))T DT DG(Y (t − τ )). (16) Substituting (9)–(16) into (8), and from conditions (6)–(7), we have V˙ (Yt )|(4) ≤ e2kt Y T (t)[2kP − P A − AT P + 2βkRK + 2e2kτ KQK −1 T ∗ 2 T 2 +P BB T P + −1 C P + −1 1 P CΣ 2 M P + βLB BL + L −1 ∗ 2 T +β(2 + −1 3 + 4 M )KR K + βA A]Y (t)

+GT (Y (t − τ ))[1 Σ + (2 + β4 )DT D " +β3 C T C − 2Q]G(Y (t − τ )) ≤ 0, which means V (Yt ) ≤ V (Y0 ),

∀t ≥ 0.

Since V (Yt ) ≥ e2kt λmin (P )||Y (t)||2 ,

V (Y0 ) ≤ (λmax (P ) + β||R||||K||)||Y (0)||2 +

∀t ≥ 0,

 1 ||Q||||K||2(e2kτ − 1) sup 2k s∈[−τ,

by the above two inequalities, we easily obtain ||Y (t)|| ≤ γe−k(t−τ )

sup s∈[−τ, 0]

||Y (s)||,

0]

||Y (s)||

2

.

Global Exponential Stability of High-Order Hopfield Neural Networks

45

for all t ≥ 0, where γ ≥ 1 is a constant. By Definition 1, this implies that the equilibrium X ∗ = (x∗1 , x∗2 , · · · x∗n )T is globally exponentially stable.  Corollary 1. Under the assumptions (H1 )–(H2 ), the equilibrium point X ∗ of system (1) is unique and globally exponentially stable if there exist positive definite matrices P , Q, positive diagonal matrix R = diag(r1 , r2 , · · · , rn ) and constants β, i > 0, i = 1, 2, 3, 4 such that any one of the following conditions holds: (i) ⎡

 H PB PC ⎢ B T P In×n 0 ⎢ ⎢ CT P 0 In×n ⎢√ ⎣ βBL 0 0 P 0 0

√ βLB T 0 0 In×n 0

⎤ P ⎥ 0 ⎥ ⎥ > 0, 0 ⎥ ⎦ 0 1 M ∗ In×n

(17)

and In×n + (1 + β)DT D + βC T C − 2Q ≤ 0,

(18)

 = AT P + P A − 2KQK − L2 − β(3 + M ∗ )KR2 K − βAT A. where H (ii) ⎡

H PB ⎢ B T P In×n ⎢√ ⎣ βBL 0 P 0

⎤ √ βLB T P ⎥ 0 0 ⎥ > 0, ⎦ In×n 0 −1 −1 +  ) I 0 (−1 n×n 1 2

(19)

and (1 + β3 )C T C + (2 M ∗ + β4 )DT D − 2Q ≤ 0,

(20)

−1 ∗ 2 T where H = AT P + P A − 2KQK − L2 − β(2 + −1 3 + 4 M )KR K − βA A.

Theorem 2. Under the assumptions (H1 )–(H2 ), the equilibrium point X ∗ of system (1) is unique and globally exponentially stable if there exist positive definite matrices P , Q, Σ, positive diagonal matrix R = diag(r1 , r2 , · · · , rn ) and constants β, i > 0, i = 1, 2, 3, 4 such that ⎡ ⎤ √ H P B P C βLB T P ⎢ B T P In×n 0 ⎥ 0 0 ⎢ ⎥ ⎢ CT P ⎥ > 0, (21) 0  Σ 0 0 1 ⎢√ ⎥ ⎣ βBL 0 ⎦ 0 In×n 0 2 P 0 0 0 M ∗ In×n and

46

J. Qiu and Q. Cheng

1 KΣK + (2 + β4 )KDT DK + β3 KC T CK − 2Q ≤ 0,

(22)

−1 ∗ 2 T where H = AT P + P A − 2Q − L2 − β(2 + −1 3 + 4 M )KR K − βA A.

Proof. Consider the Lyapunov functional n  y (t)  ri 0 i gi (s)ds V (Yt ) = e2kt Y T (t)P Y (t) + 2βe2kt i=1 t +2 t−τ e2k(s+τ ) Y T (s)QY (s)ds.

Calculate the derivative of V (Yt ) along the trajectory of ( 4) and apply Lemma 1, 2, we have V (Yt ) ≤ V (Y0 ), ∀t ≥ 0. The remaining part of the proof is similar to that of Theorem 1, and is omitted here.  If dijl = 0, i, j, l = 1, 2, · · · , n, the system (1) degenerate into the following first-order neural networks: x˙ i (t) = −ai xi (t) +

n

bij fj (xj (t)) +

j=1

n

cij  gj (xj (t − τ )) + Ii ,

(23)

j=1

where i = 1, 2, . . . , n, t > 0. We have the following corollaries. Corollary 2. Under the assumptions (H1 )–(H2 ), the equilibrium point X ∗ of system (23) is unique and globally exponentially stable if there exist positive definite matrices P , Q, Σ, positive diagonal matrix R = diag(r1 , r2 , · · · , rn ) and constants β, i > 0, i = 1, 2 such that the following condition holds: ⎡ ⎢ ⎢ ⎢ ⎢ ⎣

AT P + P A − L2 − KQK 2 −βA2 − (2 + −1 2 )βKR K T B P T √C P βBL



⎤ √ T P B P C βLB ⎥ ⎥ ⎥ > 0, In×n 0 0 ⎥ ⎦ 0 1 Σ 0 0 0 In×n

(24)

and 1 Σ + 2 C T C − Q ≤ 0,

(25)

4 Conclusion In this paper, we have studied the exponential stability of high-order neural networks with time delay. Several new sufficient conditions have been derived to show the HHNNs are exponentially stable and have periodic solutions by using Lyapunov method and LMI technique. The conditions are easy to check and the results are useful in applications of manufacturing high quality neural networks.

Global Exponential Stability of High-Order Hopfield Neural Networks

47

Acknowledgements. This work was jointly supported by the National Natural Science Foundation of China under Grant No. 60574043 and the Natural Science Foundation of Shandong Province of China under Grant No. Y2008A32.

References 1. Karayiaaanis, N., Nvenetsaopoulos, A.: On the Dynamics of Neural Networks Realizing Associative Memories of First and High-order. Network: Comp. Neural Syst. I, 345–364 (1990) 2. Dembo, A., Farotimiand, O., Kaillath, T.: High-order Absolutely Stable Neural Networks. IEEE Trans. Circuits Syst. 38, 57–65 (1991) 3. Karayiaaanis, N., Nvenetsaopoulos, A.: On the Training and Performance of High-order Neural Networks. Math. Biosci. 129, 143–168 (1995) 4. Kosmatopoulos, E., Christodoulou, M.: Structural Properties of Gradient Recurrent High-order Neural Networks. IEEE Trans. Circuits Syst. II 42, 592–603 (1995) 5. Kosmatopoulos, E., Polycarpou, M., Christodoulou, M., et al.: High-order Neural Networks Structures for Identification of Dynamical Systems. IEEE Trans. Neural networks 6, 442–431 (1995) 6. Brucoli, M., Carnimeo, L., Grassi, G.: Associative Memory Design Using Discrete-time Second-order Neural Networks with Local Interconnections. IEEE Trans. Circuits Syst. I 44, 153–158 (1997) 7. Xu, B., Liu, X., Liao, X.: Global Asymptotic Stability of High-order Hopfield Type Neural Networks with Time Delays. Comput. Math. Appl. 45, 1729–1737 (2003) 8. Cao, J., Liang, J., Lam, J.: Exponential Stability of High-order Bidirectional Associative Memory Neural Networks with Time Delays. Phys. D 199(3-4), 425–436 (2004) 9. Cao, J., Wang, L.: Exponential Stability and Periodic Oscillatory Solution in BAM Networks with Delays. IEEE Trans. Neural Networks 13, 457–463 (2002) 10. Cao, J., Wang, J.: Absolute Exponential Stability of Recurrent Neural Networks with Lipschitz-continuous Activation Functions and Time Delays. Neural Networks 17, 379–390 (2004) 11. Cao, J.: A Set of Stability Criteria for Delayed Cellular Neural Networks. IEEE Trans. Circuits Syst. I 48, 494–498 (2001) 12. Cao, J., Ho, D.W.C.: A General Framework for Global Asymptotic Stability Analysis of Delayed Neural Networks Based on LMI Approach. Chaos, Solitons and Fractals 24(5), 1317–1329 (2005) 13. Boyd, S., Ghaoui, L., Feron, E., et al.: Linear Matrix Inequalities in System and Control Theory. SIAM, Philadelphia (1994)

Improved Particle Swarm Optimization for RCP Scheduling Problem Qiang Wang and Jianxun Qi

*

Abstract. In this paper, an improved particle swarm optimization (IPSO) algorithm is presented to solve RCP Scheduling Problem. Firstly, a mapping is created between the feasible schedule and the position of the particle, then the IPSO begin to search the global best and the local best until the stop criteria is satisfied. A case study is presented and a comparison is made between IPSO and some traditional heuristic methods. Results show that the IPSO algorithm is more satisfying than those of the heuristic methods in terms of feasibility and efficiency. Keywords: RCP scheduling problem, Improved particle swarm optimization, Project management, Heuristic methods, Project scheduling, Genetic algorithm.

1 Introduction The resource constrained project scheduling problem –RCPSP is difined by Davis (1973) as “the method of scheduling activities within fixed amounts of resources available during each time period of project duration so as to minimize the increase in project duration” [1] Blazewicz et al.(1983)proved that the RCPSP is a general form developed from Job shop problem, so that it is a kind of NP-hard problem [2]. This type of problem is characterized by factorial growth in the amount of computation required to consider all possible solutions as problem size increase. There are two classes of method to solve this kind of problem, exact method and Heuristic methods [3]. The exact method includes branch and bound method, mathematical programming such as 0-1 programming [4] and dynamic programming [5]; Heuristic procedures employ some rule of thumb or experience to determine priorities among activities competing for available resources [6-8]. Article [9-10] made a summary of all these heuristic methond. These studies, however, do not contain any computational performance analysis and discuss only small problem examples. In most instances, these exact algorithms may be computationally infeasible or face “combinatorial explosion” problem if the project under study is larger or more complicated [11]. Due to these drawbacks, some evolutionary computation techniques such as genetic algorithm [12-14] have been developed and widely Qiang Wang . Jianxun Qi School of Business and Management, North China Electric Power University Beijing 102206, China [email protected] *

H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 49–57. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

50

Q. Wang and J. Qi

employed to solve the RCP Scheduling Problem. Although the GA approach can overcome the drawbacks of the analytical and heuristic algorithms, it meets difficulties such as premature convergence or slow convergence process. The Particle Swarm Optimization (PSO) algorithm developed by Kennedy and Eberhart is a stochastic, population-based optimization method which consists of a swarm of particles [15-18]. It has shown to be less susceptible to premature convergence. Previous work also illustrated that it is an efficient and effective algorithm to map high-dimensional dada sets to a lower-dimension with minimal mapping error [19]. With all these advantages, PSO has been applied to other industrial areas like electrical engineering for power optimization [20], resource-constrained project scheduling [21] and so on. To our knowledge, however, there have not been any applications of PSO to RCP Scheduling Problem. In this paper, an improved PSO (IPSO) algorithm is proposed for the RCP Scheduling Problem in project management.

2 The RCP Scheduling Problem 2.1 Activities-on-Node Network A project consists of a number of events (milestones) and activities or tasks that have to be performed in accordance with a set of precedence constrains. Each activity has duration and normally requires resources. Resources may be of different types, including financial resources, manpower, machinery, equipment, materials energy, space, etc. The problem here is to arrange a suitable schedule for the activities, so that all these resources will be used within the limits. The precedence relationship in this paper is the finish-start relationship with a zero time lag: an activity can only start as soon as all its predecessor activities have finished. Usually, there are two possible modes of representation of a project network: the activity-on-arcs mode and the activity-on- node mode.The projects in this paper are represented by an activity-on-node (AoN) network G = (V , E ) , where the set of nodes V represents the activities and the set of arcs E represents finish-start precedence constrains with a time-lag equal to zero. The activities are numbered from the dummy start activity1 to the dummy end activity n . So for each arc ( i , j ) ∈ E in the network, the activity i is the immediate predecessor of activity.

2.2 The Performance Measure When

Tn denotes the finished time of the last activity, the objective function of

RCP Scheduling Problem can be formulated conceptually as follows:

min Tn

(1)

Tn = S n + d n

(2)

Improved Particle Swarm Optimization for RCP Scheduling Problem

51

The above two equations are subjected to the following conditions: For all ( i , j ) ∈ E

si + d i ≤ s j s1 = 0 For k

i , j = 1, 2, … , N .

(3)

sn ≤ δ n .

(4)

and

= (1, … , m ) and t = ( s1 , … , sn )

∑r i∈st

ik

= ukt ≤ Rkt

(5)

where N is the total number of the activities of the project under study; si denotes the start time of activity

i ( i = 1, 2,… , N ) ; d j is the duration of the activity

i ; Rkt represents the available amount of resource k during time period t ; rik is

st is taken as the set of ongoing activities at time t ; ukt denotes the resource usage for resource type k during time period t . the amount of resource k required by activity i ;

3 The Principle of Improved Particle Swarm Optimization 3.1 The Standard PSO Algorithm The standard PSO simulates a social behavior such as bird flocking to a promising position for food or other objective in an area or apace. By sharing the swarm experience and generalizing its own experience, the birds can easily adjust its own behavior police toward the best destination. Just like the birds, the particles fly through the problem space by updating their velocities based on the particle’s personal best position and the best previous position attained by any member of its th

neighbor. In a N dimensional space, the position of the i particle in the t

generation can be represented as X i ( t ) = ( xi1 ( t ) , velocity is denoted as Vi ( t ) = ( vij ( t ) ,

…,

xi 2 ( t ) ,

vij ( t ) ,

…,

…,

th

xin ( t ) ) ; its

vin ( t ) ) . During

iterations, the local bests and the global bests are determined through evaluating the performance, i.e., fitness values or objective, of the current population of parth

ticles. The i particle’s new position is calculated as: X i ( t + 1) = X i ( t ) + Vi ( t )

(6)

where Vi ( t ) denotes the particle’s velocity and X i ( t ) represents the current position in the i

th

generation. The particle velocity is updated according to:

52

Q. Wang and J. Qi

Vi ( t + 1) = ϕ 0Vi ( t ) + ϕ1 ⋅ r1 ( t ) ⋅ [ pi Best − X i ( t )] + ϕ 2 ⋅ r2 ( t ) ⋅ [ g Best − X i ( t )]

(7)

where 0 ≤ ϕ 0 ≤ 1 is an inertia weight taken to control the impact particle’s previous velocity on the current velocity; ϕ1 , ϕ 2 are acceleration constants; r1 ( t ) and r2 ( t ) are random variables sampled from ( 0,

1) ; piBest is the personal best posi-

th

tion found by i particle, and g Best is the global best among all the population of particles achieved so far.

3.2 The Improved PSO Algorithm As the search mechanism of PSO is based on its personal best and the global best, no information of other particles is considered. Therefore, the search in the solution space is single direction. No doubt, it will suffer from premature convergence, where the particles cannot escape local minima. The improved particle swarm optimization (IPSO) is designed to overcome this disadvantage [22]. In the IPSO, all the particles are ranked according to their fitness value which is decided by their personal best piBest . The information from the top n particles is selected to adjust the behavior police for each particle in its next iteration (for the instance of Fig 1.2 in this paper n =2 is selected). Thereby the search of IPSO turned out to be even and multi-directional, the precision and global convergence ability of the algorithm are also improved. The basic formulation for IPSO algorithm is as the follows: Vi ( t + 1) = ϕ 0Vi ( t ) + ϕ1 ⋅ r1 ( t ) ⋅ [ pi Best − X i ( t )] +

1 n

n

∑ϕ

2j

[

⋅ r2 j ( t ) ⋅ g jBest − X i ( t )

]

(8)

j =1

X i ( t + 1) = X i ( t ) + Vi ( t )

(9)

where ϕ 2 j is the acceleration constant of the j particle; r2 j ( t ) is its random varith

able sampled from ( 0,

1) ; g jBest is the personal best position of the j particle th

associated with the best fitness encountered after ( t − 1) iterations. The j particle th

is among top n particles which are ranked according to their fitness value decided by their personal best piBest .

4 IPSO Framework for the RCP Scheduling Problem 4.1 Particle-Represented Schedule For an AoN network with N activities, if its schedule fulfills (3), (4) and (5), it is called a feasible schedule. Each of its arbitrary feasible schedules S ( t ) may be represented by the start times s j ( t ) of all its activities, using:

Improved Particle Swarm Optimization for RCP Scheduling Problem

S ( t ) = ( s1 ( t ) ,

…,

s j (t ) ,

…,

sn ( t ) )

53

(10)

In order to apply the IPSO, A mapping is created between the feasible schedules and the IPSO particles. In this mapping, the particle position X ( t ) = ( x ( t ) , … , x ( t ) , … , x ( t ) ) can be represented by the Activities’ start time (the candidate solutions to the resource leveling problem) as follows: i

i1

X i (t ) =

ij

Si ( t )

δn

in

⎛ si1 ( t ) , … , sij ( t ) , … , sin ( t ) ⎞ δn δn δ n ⎟⎠ ⎝

=⎜

(11)

Where Si ( t ) represents a feasible schedule found by the i particle after ( t − 1) itth

erations; sij ( t ) represents the start time of j activity of i particle after ( t − 1) itth

erations, and δ n

th

n

= ∑ di is the longest duration of the project. It can be learned i =1

from (11) that the value of xij ( t ) varies between [ 0,1] . Also a feasible project schedule can be transformed from the particle position using:

Si ( t ) = X i ( t ) ⋅ δ n

(12)

4.2 Parameters Configuration for IPSO Usually the IPSO size (total amount of the particles) between 40~60 will lead to a good result. It can be adjusted according to the concrete problem. For the instance Fig. 1 in this paper, m = 10 is selected as the size of the IPSO. For the AoN network mentioned above, all parameters of the particle position, either initialized or updated during search should represent a feasible schedule. In order to avoid infeasible particle position, we adjust (9) as follows: If [ X i ( t ) + Vi ( t )] ⋅ δ n represent a feasible schedule

X i ( t + 1) = X i ( t ) + Vi ( t )

(13)

If [ X i ( t ) + Vi ( t )] ⋅ δ n represent an infeasible schedule

X i ( t + 1) = X i ( t )

(14)

In order to balance the global search ability and the local search ability, a suitable the maximum velocity Vmax should be confirmed and the velocity of the particles should vary within interval [ −Vmax , Vmax ] . Usually Vmax is selected between the 10% and 20% of the total variable scope. In this paper, 10% of the deadline is used as

54

Q. Wang and J. Qi

the maximal velocity ( Vmax = 10% ). During the IPSO search, if the i particle’s th

velocity in j

th

direction ( vij ( t ) ) is beyond interval [ −Vmax , Vmax ] , its velocity

should be adjusted as follows: If vij ( t ) ≥ Vmax , then

vij ( t ) = Vmax = 10%

(15)

vij ( t ) = −Vmax = −10%

(16)

If vij ( t ) ≤ −Vmax , then

The convergence ability of the IPSO has been discussed by Eberhart R C and Shi Ti in 2000; some parameters are suggested to guarantee the convergence and effective calculation [23]. In this paper some parameters are selected as the follows:

ϕ 0 = 0.729844 , ϕ1 = 1.49618 and

1

n

∑ϕ n

2i

= 1.49618 .

i =1

4.3 Procedure of the IPSO Algorithms Some feasible schedules can be attained through heuristic methods based on priority rules. In the computational study of section 7.4.1 in Klein (2000) [24], a total of 73 priority rules were evaluated. In the heuristic algorithms, schedulable activities (whose predecessor are all completed and which require no more resources than available amounts at the time) with a higher priority should be assigned the resource and scheduled prior to the ones with lower priorities. Although the schedules thus acquired are not optimal, they are feasible and can be transformed into initial particles for the IPSO. In this paper, 40 initial particles are randomly created with these heuristic methods such as SAD rule, MILFT rule and so on. Beginning with these initial particles, the IPSO search process is as the follows: Step1: Set iteration counters as 0; calculate the positions of the 40 initialized particles with (11); go to step 5 Step2: update the particles’ velocity using (8) and the adjustment condition represented by (15) and (16). Step3: update the particles’ position using (9) and the adjustment condition represented by (13) and (14) Step4: A feasible projected schedule can be transformed from the particle position on the basis of (12). Step5: Calculate the fitness value (the total weighted sum of squired resource usage) of the particles at current time, using (1) and (2). For each particle, if its current fitness value is smaller than that of its previous local best position piBest ,

Improved Particle Swarm Optimization for RCP Scheduling Problem

55

the piBest will be updated as the current position; according to the IPSO Principle, the top n piBest among the particles are selected as the g jBest for (8) ( n =2 for the instance of Fig.1 in this paper). Step6: update the count number using t = t + 1 and test if the approach fulfills the stop criteria. If the stop criteria are fulfilled, then go to step 7; otherwise, go to step 2. [Stop criteria. The IPSO will be terminated if the current iteration meets any one of the following termination signals: (1) maximum number of iterations since last updating of the global best, and (2) maximum total number of iterations.] Step7: Recorded the top

n piBest among the particles, the best one, p1 pest are se-

lected as the g Best

5 Computational Analyses A simple instance as Fig.1 is created to try the IPSO algorithm. It is an AoN network of 10 nodes with fumy start node 1 and terminate node 10. The duration for each activity is indicated above the corresponding node, while the requirement for the single renewable resource, which is assumed for simplicity in this instance, is give below the node. The resource availability limit R kt of this instance is 9 and δ n is 36. The IPSO size is taken as 10 as it is only a simple network.Other parameters can be referred to section 4.2 in this paper. Some other heuristic methods such as GA, PSO have been applied to make a comparison, the result of which is shown as Tab.1

Fig. 1 An example instance for the resource leveling problem

56

Q. Wang and J. Qi

Table 1 Results Comparison

ALGORITHMS GA PSO IPSO

SIZE 20 10 10

THE BEST FITNESS VALUE

SCHEDULE SEARCHED

TIMES

27 25 17

20 19 18

(0,0,0,2,5,5,10,8,18,20) (0,0,0,2,6,5,10,8,17,19) (0,0,5,2,0,6,12,12,16,18)

ITERATION

It can be easily found from Tab.1 that the IPSO presented in this paper is the best one among all these methods compared. Its solution with a fitness value of 18 is better than that of the GA (genetic algorithm) GA, and its iteration times (17) is much smaller than that of GA (27) and traditional PSO (25). Though this is just a small network, it can be deduced that its advantages in optimization and convergence ability will be showed of f for large and complex project networks.

6 Conclusions In this paper, an improved particle swarm optimization (IPSO) method is designed to solve the RCP Scheduling Problem. A mapping between the feasible schedule and the particles’ position is created, and the particle’s best position searched in the solution space by IPSO will be transformed into the best schedule in terms of project duration in return. The computation analysis show that the approach based on IPSO has the ability to search for the global optima, and is more efficient than the traditional PSO method and GA approach. Acknowledgments. This paper is supported by National Natural Science Foundation 80579101 and Doctor Funds of the China Education Ministry 20050079008.

References 1. Davis, E.W.: Project Scheduling under Resource Constraints—a Historical Review and Categorization of Procedures. AIIE Transactions 5, 297–313 (1973) 2. Erik, L., Demeulemeester, Willy, S.H.: Project Scheduling: A Research Handbook, pp. 10–15. Kluwer Academic Publishers, Dordrecht (2002) 3. Herroelen, W., Reyck, B.D., Demeulemeester, E.: Resource-constrained Project Scheduling: A Survey of Recent Developments. Computers & Ops. Res. 25, 279–302 (1998) 4. Patterson, J.H., Huber: A Horizon-varying Zero-one Approach to Project Scheduling. Management Science 20, 990–998 (1974) 5. Bell, C.E., Park, K.: Solving Resource Constrained Project Scheduling Problems by A*-search. Naval Research Logistics 37, 61–84 (1990) 6. Kolisch, R., Kolisch, A.: Adaptive Search for Solving Hard Project Scheduling Problem of Operational Research. Naval Research Logistics 43, 23–40 (1996) 7. Bouleimen, K., Lecocq, H.: A New Efficient Simulated Annealing Algorithm for Resource Constrained Scheduling Problem. Technical Report, Service de Robotique et Automatisation, University de Liege, pp. 1–10 (1998)

Improved Particle Swarm Optimization for RCP Scheduling Problem

57

8. Hartmann, S.: A Competitive Genetic Algorithm for Resource Constrained Project Scheduling. Naval Research Logistics 45, 733–750 (1998) 9. Lawrence, S.R.: Resource-constrained Project Scheduling-A Computational Comparison of Heuristic Scheduling Techniques. Technical Report, Graduate School of Industrial Administration, Carnegie-Mellon University, Pittsburgh, pp. 10–18 (1985) 10. Alvarez-Valdes, R., Tamarit, J.M.: Advances in Project Scheduling, pp. 113–134. Elsevier, Amsterdam (1989) 11. Lee, J.K., Kim, Y.D.: Search Heuristics for Resource Constrained Project Scheduling. Journal of Operat. Res. Soc. 47, 678–689 (1996) 12. Zhang, L.Y., Zhang, J.P., Wang, L.: Genetic Algorithms Based on MATLAB of Construction Project Resource Leveling. Journal of Industrial Engineering/Engineering Management 18, 52–55 (2004) (Chinese) 13. Leu, S.S., Yang, C.H., Huang, J.C.: Resource Leveling in Construction by Genetic Algorithm-based Optimization and in Decision Support System Applicaiton. Automation in Construction 10, 27–41 (2000) 14. Tarek, H.: Optimization of Resource Allocation and Leveling Using Genetic Algorithms. Journal of Construction Engineering and Management 6, 167–175 (1999) 15. Kennedy, R., Eberhart, C.: Particle Swarm Optimization. In: Proceedings of International Conference on Neural Networks, pp. 1942–1948 (1995) 16. Wang, J.W., Wang, D.W.: Experiments and Analysis on Inertia Weight In Particle Swarm Optimization. Journal of Systems Engineering 20, 194–198 (2005) (Chinese) 17. Clerc, M., Kennedy, J.: The Particle Swarm. Explosion, Stability, and Convergence in a Multi- dimensional Complex Space. IEEE Trans. on Evolutionary Computation 6, 58–73 (2002) 18. Trelea, I.: The Particle Swarm Optimization Algorithm. Convergence Analysis and Parameter Selection. Information Processing Letters 85, 317–325 (2003) 19. Edwards, A., Engelbrecht, A.P.: Comparing Particle Swarm Optimisation and Genetic Algorithms for Nonlinear Mapping. In: 2006 IEEE Congress on Evolutionary Computation, pp. 694–701 (2006) 20. Naka, S., Genji, T., Yura, T., et al.: A Hybrid Particle Swarm Optimization for Distribution State Estimation. IEE Trans. on Power System 18, 60–68 (2003) 21. Zhang, H., Li, H., Tam, C.M.: Particle Swarm Optimization for Resource-constrained Project Scheduling. International Journal of Project Management 14, 83–92 (2006) 22. Zhao, B., Cao, Y.J.: An Improved Particle Swarm Optimization Algorithm For Power System Unit Commitment. Power System Technology 28, 6–10 (2004) 23. Eberhart, R.C., Shi, Y.: Comparing Inertia Weights and Constriction Factors in Particle Swarm Optimization. In: Proceedings of the Congress on Evolutionary Computing, pp. 84–88. IEE Service Center, California (2000) 24. Klein, R.: Scheduling of Resource-Constrained Projects. Kluwer Academic Publisher, Boston (2000)

Exponential Stability of Reaction-Diffusion Cohen-Grossberg Neural Networks with S-Type Distributed Delays Yonggui Kao and Shuping Bao

Abstract. This paper is devoted to investigation of the existence of the equilibrium point and its globally exponential stability for reaction-diffusion Cohen-Grossberg neural networks with variable coefficients and distributed delays by means of the homotopic mapping theory and Lyapunov-functional method. The sufficient conditions obtained which are easily verifiable, have a wider adaptive range. Finally, a numerical example verifies the theoretical analysis. Keywords: Homotopic mapping theory, Cohen-Grossberg neural networks, Reaction-diffusion terms, Exponential stability.

1 Introduction Cohen-Grossberg neural networks (CGNNs) has been widely applied in parallel computation, associative memory and optimization problems, which relys on the stability, especially, global asymptotic or exponential stability of neural networks. Many researchers have presented various criteria for the uniqueness and global asymptotic or exponential stability for the equilibrium point of CGNNs with or without time varing delays. In practice, the delays in artificial neural networks are usually continuously distributed [1-10], because neural networks usually has a spatial extent due to the presence of an amount of parallel pathways with a variety of axon sizes and lengths. Moreover, diffusion effect cannot be avoided in the neural networks model when electrons Yonggui Kao Department of Mathematics, Harbin Institute of Technology, Weihai 264209, China Shuping Bao College of information Science and Technology, Qingdao University of Science and Technology, Qingdao 266061, China H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 59–68. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

60

Y. Kao and S. Bao

are moving in asymmetric electromagnetic field[11-13]. Besides, it is common to consider the diffusion in biological systems (such as immigration) [6,8,10]. However, few author have studied the existence of the equilibrium point and its exponential stability for reaction-diffusion CGNNs with distributed delays. We consider the following reaction-diffusion CGNNs model: ⎧ # $ m  ∂ui (t,x) ∂ui (t,x) ∂ ⎪ D − ai (ui (t, x)) = ⎪ il ∂t ∂x ∂x ⎪ l l ⎪ l=1 ⎪ ⎪ n 0 ⎪  ⎪ ⎪ aij sj ( −∞ dηj (θ)uj (t + θ, x)) ×[bi (ui (t, x)) − ⎪ ⎨ j=1 n K   (k) ⎪ ⎪ tij sj (uj (t − τk (t), x)) + Ii ], t ≥ 0, x ∈ Ω − ⎪ ⎪ ⎪ j=1 k=0 ⎪ ⎪ ⎪ ⎪ ui (t0 + s, x) = ϕi (s, x), −∞ < s ≤ 0, x ∈ Ω ⎪ ⎩ ∂ui (t,x) = 0, x ∈ ∂Ω, ∂n

(1)

where i = 1, . . . , n; n is the number of neurons in the network; ui (t, x) denotes the state variable associated with the i th neurons at time t and in space x; ai (t, x) represents an amplification function; bi (t, x) represents an appropriately behaved function; aij represents the strength of the neuron interconnections within the network; sj (t, x) shows how the jth neuron reacts (k) to the input; tij represents the interconnection with delay τk (t); τk (t) are delay functions satisfying 0 ≤ τk (t) ≤ τ and 0 ≤ τ˙k (t) ≤ τ¯ ≤ 1; Ii is the constant input from outside the system; Dik ≥ 0 corresponds to the transmission diffusion operator along the ith neuron. u(t, x) = (u1 (t, x), . . . , un (t, x))T , x = (x1 , . . . , xm )T . Throughout the paper, we assume that system (1) has a continuous solution denoted by u(t, 0, ϕ; x) or simply u(t, x) . we also assume that (H1) There exist mi and Mi , such that 0 < mi ≤ ai (ui ) ≤ Mi , i = 1, . . . , n. Δ (H2) bi (·) is differentiable, and αi = inf ui ∈R {b˙ i (ui )} > 0, where b˙ i (·) is the derivative of bi (·), bi (0) = 0, i = 1, . . . , n.

(H3) There exist constants βj > 0(j = 1, . . . , n), such that |sj (x) − sj (y)| ≤ βj |x − y| , ∀x, y ∈ R. ∗ , such that (H4) There exist constants pij , qij , p∗ij , qij n  2q 2q 2−2qji 2−2qji Mi |aij |2pij βj ij δj ij + Mj |aji |2−2pji βi δi j=1 j=1 % %2p∗ij % %2−2p∗ji n n K  K  ∗   2q∗ 2−2qji % (k) % % (k) % + Mi %tij % βj ij + (1 + τ¯) Mj %tji % βi < 0.

−2mi αi +

k=0 j=1

0

n 

k=0 j=1

(H5) −∞ dηj (θ)uj (t + θ, x), j = 1, 2, ..., n are Lebesgue-Stieltjes integrable and the delay kernels ηj : (−∞, 0] → R are nondecreasing bounded variation functions satisfying

Exponential Stability of Reaction-Diffusion CGNNs

0 −∞

61

dηj (θ) = δj > 0.

Δ

Let C = C((−∞, 0] × Rm , Rn ) be the Banach space of continuous functions which map (−∞, 0] × Rm into Rn with the topology of uniform convergence. Ω be an open bounded domain in Rm with smooth boundary ∂Ω, and mesΩ > 0 denotes the measure of Ω. L2 (Ω) is the space of real functions on Ω which are L2 for the Lebesgue measure. It is a Banach n  space for the norm u(t) = ui (t)2 where u(t) = (u1 (t), · · · , un (t))T , ui (t)2 =

  Ω

define ϕ =  

i=1

2

1/2

|ui (t, x)| dx n 

i=1 1/2

|ϕi (x)|2τ dx

Ω

. For any ϕ(t, x) ∈ C[(−∞, 0] × Ω, Rn ], we

ϕi 2 , where ϕ(t, x) = (ϕ1 (t, x), · · · , ϕn (t, x))T , ϕi 2 = , |ϕi (x)|∞ =

sup

−∞≤s≤0

|ϕi (s, x)| .

Definition 1. u(t, x) ≡ u∗ ∈ Rn is said to be an equilibrium point of system (1), if the constant vector u∗ = (u∗1 , . . . , u∗n )T satisfies ∗

bi (u ) −

n

aij sj (δj u∗j )

j=1

−

K n

tij sj (u∗j ) + Ii = 0, i = 1, . . . , n. (k)

(2)

k=0 j=1

Lemma 1. (Cronin[8] Kroneckers theorem). Assume that f : Ω → Rn is a continuous function, deg(f, Ω, p) = 0, then there exists x0 ∈ Ω, such that f (x0 ) = p. Lemma 2. (Cronin[8] Homotopy invariance theorem). Assume that H(x, λ) : Ω × [0, 1] → Rn is a continuous function, denote hλ (x) = H(x, λ). When λ ∈ [0, 1] and p ∈ / hλ (∂Ω), deg(hλ , Ω, p) is independent of λ.

2 Existence and GES of the Equilibrium Point In order to study the existence and uniqueness of the equilibrium point, we consider the following algebraic equations associated with system (1) bi (ui (t, x)) −

n j=1

aij sj (δj uj (t, x)) −

K n

(k)

tij sj (uj (t, x)) + Ii = 0,

(3)

k=0 j=1

where i = 1, . . . , n. System (3) can be rewritten in the following vector form: B(u) − AS(δu) − T S(u) + I = 0,

(4)

62

Y. Kao and S. Bao

where u = (u1 , . . . , un )T , B(u) = (b1 (u1 ), . . . , bn (un ))T , A = (aij )n×n , δ = K (k) diag(δ1 , . . . , δn ),T = ( k=0 tij )n×n , S(δu) = (s1 (δ1 u1 ), . . . , sn (δn un ))T , I = (I1 , . . . , In )T . Theorem 1. If (H1)-(H5) holds, then system (1) has an unique equilibrium point u∗ . Proof. Let h(u) = B(u)−AS(δu)−T S(u)+I. Define the homotopic mapping Δ ¯ × [0, 1] → Rn as follows: H(u, λ) = (H1 (u, λ), . . . , Hn (u, λ))T = H(u, λ) : Ω λh(u) + (1 − λ)B(u), λ ∈ [0, 1], in which α = diag(αi ). It follows from (H2) and (H4) that |bi (ui )| ≥ αi |ui | ,

i = 1, . . . , n,

(5)

|sj (uj )| ≤ βj |uj |+|sj (0)| , |sj (δj uj )| ≤ βj δj |uj |+|sj (0)| , j = 1, . . . , n. (6) Thus we have |Hi (u,

λ)| = |λ bi (ui ) −

n 

aij sj (δj uj ) −

j=1 n 

≥ αi |ui | −

|aij | βj δj |uj | −

j=1

% n % K   % (k) % − %tij % |sj (0)| − |Ii | .

n K  

k=0 j=1 n 

 (k) tij sj (uj )

|aij | |sj (0)| −

j=1

+ Ii

+ (1 − λ)bi (ui )|

% n % K   % (k) % %tij % βj |uj |

k=0 j=1

k=0 j=1

By (7), we obtain n 

Mi |ui | |Hi (u, λ)|

% n n n K  n %     % (k) % Mi |ui | αi |ui | − |aij | βj δj |uj | − |aij | |sj (0)| − ≥ %tij % βj |uj | i=1 j=1 j=1 k=0 j=1  % K  n %  % (k) % − %tij % |sj (0)| − |Ii |

i=1

k=0 j=1

% ∗ ∗ n % K   % (k) %2pij 2qij |aij | |sj (0)| |ui | − 12 βj |ui |2 %tij % j=1 k=0 j=1 % ∗ % ∗ K  K  n % n %   % (k) %2−2pij 2−2qij % (k) % 1 −2 βj |uj |2 − %tij % %tij % |sj (0)| |ui | − |Ii | |ui | k=0 j=1 k=0 j=1

n n n    2q 2q 2−2qji 2−2qij Mi αi − 12 Mi |aij |2pij βj ij δj ij − 12 Mj |aji |2−2pji βi δj ≥ i=1 j=1 j=1  % % ∗ % % ∗ ∗ ∗ n n K  K    % (k) %2pij 2qij % (k) %2−2pji 2−2qji − 12 Mi %tij % βj − 12 Mj %tji % βi |uj |2 k=0 j=1 k=0 j=1 

% n n K  n %    % (k) % |s − Mi |aij | |sj (0)| + (0)| + |I | |ui | . %tij % j i

−

n 

i=1

j=1

k=0 j=1

(7)

Exponential Stability of Reaction-Diffusion CGNNs

63

So,  we obtain  n   M |u | |H (u, λ)| dx i i i Ω i=1 & n n    2q 2q Mi αi − 12 ≥ Ω Mi |aij |2pij βj ij δj ij − i=1

j=1

n  j=1

Mj |aji |2−2pji  

% %2−2p∗ji ∗ 2−2qji % (k) % |uj |2 Mj %tji % βi k=0 j=1 k=0 j=1

 ' % n % n n K     % (k) % − Mi |aij | |sj (0)| + %tij % |sj (0)| + |Ii | |ui | dx − 21

n K  

i=1

% %2p∗ij 2q∗ % (k) % Mi %tij % βj ij −

1 2

j=1

n K  

k=0 j=1

2 δ0 u(t)2 (

≥

1 2

− L u(t)2 ,

δ0 = min Mi αi −

1 2

i

n 

where,

2pij

Mi |aij |

−

j=1

1 2

 n 

=

2−2pji

Mj |aji |

j=1

2qij 2qij δj ,

βj

and

2−2qji 2−2qji δi

βi )

% %2p∗ij % %2−2p∗ji n K  ∗  2q∗ 2−2qji % (k) % % (k) % > 0, − 21 Mi %tij % βj ij − 12 Mj %tji % βi k=0 j=1 k=0 j=1 % % * # $ + K n % (k) % n L = maxi Ω Mi |s |a | |s (0)| + (0)| + |I | dx . %t % ij j j i j=1 k=0 j=1 ij n Take U (R 0 }. For any u ∈ ∂U (R0 ),  0n) = {U ∈ R | u(t)  2 < R0 = (L + 1)/δ # $   Mi |ui | |Hi (u, λ)| dx ≥ δ0 u(t)2 u(t)2 − δL0 > 0, ∀λ ∈ and Ω n K  

i=1

[0, 1], i.e., H(u, λ) = 0, obtain:

∀λ ∈ [0, 1]. From Lemma 2, we

∀u ∈ ∂U (R0 ),

deg(h(u), U (R0 ), 0) = deg(H(u, 1), U (R0 ), 0) = deg(H(u, 0), U (R0 ), 0) = 1. It follows from Lemma 1 that there exist at least an u∗ ∈ U (R0 ), such that h(u∗ ) = 0. i.e., system (1) has at least an equilibrium point u∗ . Next, we prove the uniqueness of the equilibrium point. Suppose that u ¯∗ is also an equilibrium point of system (1), then u∗i ) − bi (¯

n

aij sj (δj u ¯∗j ) −

j=1

K n

bi (u∗i ) +

(k)

−

bi (¯ u∗i )

n K  

k=0 j=1

=

n  j=1

(k) tij sj (u∗j )

aij sj (δj u∗j )

−

n K   k=0 j=1

From

(H2),(H4) and (H5), we obtain n 

i=1

+ 12

−Mi αi + K  n  k=0 j=1

1 2

(8)

k=0 j=1

Thus

tij sj (¯ u∗j ) + Ii = 0, i = 1, . . . , n

n 

j=1

2qij 2qij δj

Mi |aij |2pij βj

% % ∗ ∗ % (k) %2pij 2qij Mi %tij % βj +

1 2

K  n  k=0 j=1

+

1 2

−

n  j=1

 aij sj (δj u ¯∗j )

(k) tij sj (¯ u∗j )

n  j=1



(9) .

2−2qji 2−2qji δi

Mj |aji |2−2pji βi 

% % ∗ ∗ % (k) %2−2pji 2−2qji Mj %tji % βi

|u∗i − u ¯∗i |2 ≤ 0,

64

Y. Kao and S. Bao

which implies u∗i = u ¯∗i (i = 1, . . . , n). Hence system (1) has an unique equilib∗ rium point u . This completes the proof. Denoting yi (t, x) = ui (t, x) − u∗i , i = 1, . . . , n, system (1) can be rewritten as follows: ⎧ # $ m  ∂yi (t,x) ∂yi (t,x) ∂ ⎪ D − ai (yi (t, x)) × [bi (ui (t, x)) − bi (u∗i ) = ⎪ il ∂xl ∂t ∂xl ⎪ ⎪ l=1 ⎪ ⎪ n n 0 0 ⎪   ⎪ ∗ ⎪ ⎪ ⎨ − j=1 aij sj ( −∞ dηj (θ)uj (t, x)) + j=1 aij sj ( −∞ dηj (θ)uj ) (10) K n K n ⎪ −   t(k) s (u (t − τ (t), x)) +   t(k) s (u∗ )], t ≥ 0, x ∈ Ω ⎪ j j k j ⎪ j ij ij ⎪ ⎪ k=0 j=1 k=0 j=1 ⎪ ⎪ ⎪ yi (t, x) = ψi (t, x), −τ ≤ t ≤ 0, x ∈ Ω ⎪ ⎪ ⎩ ∂yi (t,x) = 0, x ∈ ∂Ω, ∂n where i = 1, . . . , n, ψi (t, x) = ϕi (t, x)−u∗i . y(t, x) = (y1 (t, x), . . . , yn (t, x))T , ψ(t, x) = (ψ1 (t, x), . . . , ψn (t, x))T . Obviously, u∗ of system (1) is GES if and only if the equilibrium point O of system (10) is GES. Thus in the following, we only consider GES of the equilibrium point O for system (10). Theorem 2. If (H1)-(H5) hold, then the equilibrium point O of system (10) is GES. Proof. From (H4), there exists a sufficiently small constant 0 < λ < mini {mi αi } , such that n n   2q 2q 2p 2−2pji 2−2qji 2−2qji 2λ − 2mi αi + Mi |aij | ij βj ij δj ij + Mj |aji | βi δi j=1 j=1 % %2p∗ij % %2−2p∗ji n n K  K  ∗   2q∗ 2−2qji % (k) % % (k) % + Mi %tij % βj ij + Mj %tji % βi e2λτ k=0 j=1 k=0 j=1 % %2−2p∗ji n K  ∗  2−2qji % (k) % +¯ τ Mj %tji % βi ≤ 0. k=0 j=1

Taking Lyapunov functional and calculating the rate of change of V (t): % %2−2p∗ij n n K  ∗    2−2qij % (k) % 2 Mi %tij % βi V (t) = Ω ( |yi (t, x)| e2λt + i=1 k=0 j=1 t × t−τk (t) |yj (s, x)|2 e2λ(s+τ ) ds)dx n  

2

(11)

(2λ |yi (t, x)| e2λt + 2 |yi (t, x)| e2λt sign(yi (t, x))y˙i (t, x) i=1 % %2−2p∗ij n K   2−2q∗ % (k) % 2 + Mi %tij % βj ij |yj (t, x)| e2λ(t+τ ) k=0 j=1 % %2−2p∗ij n K   2−2q∗ % (k) % 2 −(1 − τ˙k (t)) Mi %tij % βj ij |yj (t − τk (t), x)| e2λ(t−τk (t)+τ ) )dx D+ V (t) =

Ω

k=0 j=1

Exponential Stability of Reaction-Diffusion CGNNs 

65

n 

[2λ |yi (t, x)|2 e2λt + 2 |yi (t, x)| e2λt sign(yi )  # $ i (t,x) Dil ∂y∂x − ai (yi (t, x)) k 0 0 ×[bi (ui (t, x)) − bi (u∗i ) − aij sj ( −∞ dηj (θ)uj (t + θ, x)) + aij sj ( −∞ dηj (θ)u∗j ) K  K  n n   (k) (k) − tji sj (uj (t − τk (t), x)) + tji sj (u∗j )] k=0 j=1 k=0 j=1 ∗ % % ∗ K  n  % (k) %2−2pij 2−2qij + Mi %tij % βj |yj (t, x)|2 e2λ(t+τ ) k=0 j=1 % % ∗ K  ∗ n  % (k) %2−2pij 2−2qij −(1 − τ˙k (t)) Mi %tij % βj |yj (t − τk (t), x)|2 e2λ(t−τk (t)+τ ) ]dx =

Ω

 mi=1  ∂ × ∂xl l=1

k=0 j=1

From the boundary condition, we have # $ # $ m m     ∂yi (t,x) ∂ ∂ i (t,x) |yi (t, x)| dx = Dil ∂y∂x dx |y (t, x)| ∂x ∂xl Dil ∂xl Ω Ω i l l l=1 l=1 # $ m ,   ∂yi (t,x) ∂yi (t,x) ∂ i (t,x) = dx− Ω ∂y∂x D dx il Ω ∂xl |yi (t, x)| Dil ∂xl ∂x l l l=1 # $2 m   ∂yi (t,x) =− |yi (t, x)| dx. Ω Dil ∂xl l=1

Combining (H1)-(H5) with 2ab ≤ a2 + b2 , we obtain D+ V (t) ≤

n  

2

2

[2λ |yi (t, x)| e2λt + 2e2λt (−mi αi |yi (t, x)| i=1 % % n n K    % (k) % + |aij | βj δj |yi (t, x)| |yj (t, x)| + Mi %tij % βj |yi (t, x)| ) j=1 k=0 j=1 % %2−2p∗ij n K  ∗  2−2qij % (k) % Mi %tij % βj |yj (t, x)|2 e2λ(t+τ ) + j=1 k=0 % %2−2p∗ij n K   2−2q∗ % (k) % 2 −(1 − τ¯) Mi %tij % βj ij |yj (t − τk (t), x)| e2λ(t−τ (t)+τ ) ]dx n  

Ω

k=0 j=1

2 2 [2λ |yi (t, x)| e2λt + 2e2λt (−mi αi |yi (t, x)| Ω i=1 n  q q 1−q 1−q p 1−p + Mi |aij | ij βj ij δj ij |yi (t, x)| |aij | ij βj ij δj ij |yj (t, x)| j=1 % %p∗ij ∗ % %1−p∗ij n K   q 1−q∗ % (k) % % (k) % Mi %tij % βj ij |yi (t, x)| %tij % βj ij |yj (t − τk , x)|) + k=0 j=1 % %2−2p∗ij n K   2−2q∗ % (k) % Mi %tij % βj ij |yj (t, x)|2 e2λ(t+τk ) + k=0 j=1 % %2−2p∗ij n K   2−2q∗ % (k) % 2 − Mi %tij % βj ij |yj (t − τk , x)| e2λt ]dx k=0 j=1 n   2 2 ≤ Ω [2λ |yi (t, x)| e2λt + e2λt (−2mi αi |yi (t, x)| i=1 n n (12) 2q 2p 2 2−2pji 2−2qji 2 Mi |aij | ij βj ij |yi (t, x)| + Mi |aij | βj |yj (t, x)| + j=1 j=1

≤

66

Y. Kao and S. Bao

% %2p∗ij % %2−2p∗ij n K   2q∗ 2−2q∗ % (k) % % (k) % 2 Mi %tij % βj ij |yi (t, x)| + Mi %tij % βj ij  k=0 j=1 k=0 j=1 % %2−2p∗ij n K  ∗  2−2qij % (k) % 2 Mi %tij % βj |yj (t, x)| e2λ(t+τk ) + k=0 j=1 % %2−2p∗ij n K   2−2q∗ % (k) % 2 − Mi %tij % βj ij |yj (t − τk , x)| e2λt ]dx +

n K  

k=0 j=1 n 

n n   2q 2p 2−2pji 2−2qji [2λ − 2mi αi + Mi |aij | ij βj ij + Mi |aij | βj i=1 j=1 j=1 ∗ ∗ % % % % n n K  K  ∗ ∗   % (k) %2pij 2qij % (k) %2−2pji 2−2qji + Mi %tij % βj + Mj %tji % βi e2λτ ] k=0 j=1 k=0 j=1 % %2−2p∗ij n K   2−2q∗ % (k) % 2 Mi %tij % βj ij |yj (t − τk , x)| yi (t)2 e2λt +

≤

k=0 j=1

≤ 0,

Where  = |yj (t − τk , x)|. It implies V (t) ≤ V (0). By (11) we have % % ∗ K  ∗  n n    % (k) %2−2pij 2−2qij t V (0) = Ω (|ψi (t, x)|2 + Mi %tij % βi |yj (s, x)|2 ℘ds)dx −τk i=1 k=0 j=1 

% % ∗ n K  ∗  n   % (k) %2−2pij 2−2qij 0 2 2λ(s+τk ) Mi %tij % βi y (s) e ds ≤ ψi (t)22 + j 2 −τk i=1 k=0 j=1 ( ) ∗ % ∗ K  n %  % (k) %2−2pij 2−2qij ≤ max 1 + βi (eτ − 1) ψ22 , %tij % i

k=0 j=1

2

where ℘ = e2λ(s+τk )(. And V (t) ≥ y(t)2 e2λt . Hence ) ∗ % n % K  ∗  % (k) %2−2pij 2−2qij 2 2λt y(t)2 e ≤ max 1 + βi (eτ − 1) ψ22 , %tij % i

k=0 j=1

which leads to y(t)2 ≤ γ ψ2 e−λt , where ( )1/

∗ 2 % n % K  ∗  % (k) %2−2pij 2−2qij . βi (eτ − 1) γ = max 1 + %tij % i

k=0 j=1

This completes the proof. Corollary 1. System (1) has an unique equilibrium point which is GES point, if the conditions (H1)-(H3) hold. Furthermore, assume one of the following conditions hold: % %2 n n K    % (k) % 2 (H5) −2mi αi + Mi |aij | βj2 + Mi %tij % βj2 < 0. (H6)

−2mi αi +

j=1 n  j=1

+

n K   k=0 j=1

k=0 j=1

Mi |aij | βj +

% % % (k) % Mj %tji % βi < 0.

n  j=1

Mj |aji | βi +

n K   k=0 j=1

% % % (k) % Mi %tij % βj

Exponential Stability of Reaction-Diffusion CGNNs

67

∗ Proof. In (H4), let pij = qij = p∗ij = qij = 1, then (H4) turns to (H5). ∗ Furthermore, we suppose that pij = qij = p∗ij = qij = 12 , then (H4) turns to (H6).By Theorems 1 and 2, system (1) has a unique equilibrium point which is GES.

Remark 1. When Dik ≡ 0, then system (1) becomes the system analyzed in [2,9,14,15,16]. It is worth noting that, in the paper, we did not need sj is bounded. Thus, we improve the results in Refs. [2,9,14,15,16]. Remark 2. From Theorems 1 and 2, we see if reaction- diffusion terms satisfy a weaker condition Dik ≥ 0, then the effects for the existence and GES of the equilibrium point just come from the networks parameters, the stability is completely expressed by the relations of these parameters. Remark 3. Although the assertions of exponential stability in Theorems 1 and 2 are independent of the delays, the convergence rate λ do depend on the delays τk .

3 An Example Example 1. Consider Cohen-Grossberg neural networks with delays and reaction-diffusion terms ⎧ # $ ∂u1 (t,x) (t,x) ∂ ⎪ D11 ∂u1∂x − a1 (u1 (t, x)) = ∂x ⎪ ⎪ ∂t ⎪ ' & ⎪ ⎪ 1 2 2  ⎪ 0   ⎪ (k) ⎪ a1j sj ( −∞ dηj (θ)uj ) − t1j sj (uj (t − τk , x)) ⎪ ⎨ × b1 (u1 (t, x)) − j=1 j=1 k=0 # $ ∂u2 (t,x) (t,x) ∂ ⎪ D21 ∂u2∂x − a2 (u2 (t, x)) = ∂x ⎪ ⎪ ∂t ⎪ & ' ⎪ ⎪ 1 2 2  ⎪ 0   ⎪ (k) ⎪ a2j sj ( −∞ dηj (θ)uj ) − t2j sj (uj (t − τk , x)) . ⎪ ⎩ × b2 (u2 (t, x)) − j=1

j=1 k=0

(13) Let D11 > 0, D21 > 0, ai = 4 + sin ui , bi (ui ) = 4ui , s1 (u1 ) = arctan u1 , s2 = u2 . Clearly, ai satisfies (H1) with mi = 3, Mi = 5, sj satisfies (H2) with βj = 1, bi satisfies (H3) with αi = 4, i, j = 1, 2. Moreover, we choose (0) (0) (1) (1) 1 1 1 1 1 a11 = 12 , a12 = 12 , a21 = 16 , a22 = 24 , t11 = 12 , t12 = 16 , t11 = 12 , t12 = (1) (1) 1 (0) 1 (0) 1 1 1 4 , t21 = 6 , t22 = 24 , t21 = 12 , t22 = 24 . By simple calculation, we show that (H6) holds. It follows from Theorems 1 and 2 that system (13) has a unique equilibrium point (0, 0)T which is GES.

4 Conclusions In this paper, the dynamics of Cohen-Grossberg neural networks model with delays and reaction-diffusion is studied. By employing homotopic mapping theory and constructing Lyapunov functional method, some sufficient condi-

68

Y. Kao and S. Bao

tions have been obtained which guarantee the model to be GES. The given algebra conditions are useful in design and applications of reaction-diffusion Cohen-Grossberg neural networks. Moreover, our methods in the paper may be extended for more complex networks. Acknowledgements. This paper was supported by the National Natural Science Foundations of China under Grant 60673101 and 60674020. And supported by the Foundations under Grant HITWHXB200807.

References 1. Xu, Z., Qiao, H., Peng, J., Zhang, B.: A Comparative Study of Two Modeling Approaches in Neural Networks. J. Neural Networks 17, 73–85 (2004) 2. Cao, J.: An Estimation of the Domain of Attraction and Convergence Rate for Hopfield Continuous Feedback Neural Networks. Phys. Lett. A 325, 370–374 (2004) 3. Cao, J., Liang, L.: Boundedness and Stability for Cohen-Grossberg Neural Network with Time-varying Delays. J. Math. Anal. Appl. 296, 665–685 (2004) 4. Cao, J., Wang, J.: Globally Exponentially Stability and Periodicity of Recurrent Neural Networks with Time Delays. IEEE Trans. Circuits Syst. I 52, 920–931 (2005) 5. Zhang, J., Suda, Y., Komine, H.: Global Exponential Stability of CohenGrossberg Neural Networks with Variable Delays. Phys. Lett. A 338, 44–50 (2005) 6. Hasting, A.: Global Stability in Lotka-Volterra Systems with Diffusion. J. Math. Biol. 6, 163–168 (1978) 7. Liang, J., Cao, J.: Global Exponential Stability of Reaction-Diffusion Recurrent Neural Networks with Time-varying Delays. Phys. Lett. A 314, 434–442 (2003) 8. Liao, X., Li, C.: Stability in Gilpin-Ayala Competition Models with Diffusion. Nonlinear Anal. TMA. 18, 1751–1758 (1997) 9. Liao, X., Li, C., Wong, K.: Criteria for Exponential Stability of CohenGrossberg Neural Networks. Neural Networks 17, 1401–1414 (2004) 10. Rothe, F.: Convergence to the Equilibrium State in the Volterra-Lotka Diffusion Equations. J. Math. Biol. 3, 319–324 (1976) 11. Song, Q., Cao, J.: Global Exponential Stability and Existence of Periodic Solutions in BAM Networks with Delays and Reaction-diffusion Terms. Chaos Solitons & Fractals 23, 421–430 (2005) 12. Song, Q., Zhao, Z., Li, Y.: Global Exponential Stability of BAM Neural Networks with Distributed Delays and Reaction-diffusion Terms. Phys. Lett. A 335, 213–225 (2005) 13. Wang, L., Xu, D.: Global Exponential Stability of Reaction-diffusion Hopfield Neural Networks with Variable Delays. Sci. China Ser. E 33, 488–495 (2003) 14. Xiong, W., Cao, J.: Global Exponential Stability of Discrete-time CohenGrossberg Neural Networks. Neurocomputing 64, 433–446 (2005) 15. Yuan, K., Cao, J.: Global Exponential Stability of Cohen-Grossberg Neural Networks with Multiple Time-varying Delays. LNCS, vol. 73, pp. 78–83 (2004) 16. Zhang, J., Suda, Y., Komine, H.: GlobalExponential Stability of CohenGrossberg Neural Networks with Variable Delays. Phys. Lett. A 338, 44–50 (2005)

Global Exponential Robust Stability of Static Reaction-Diffusion Neural Networks with S-Type Distributed Delays Shuping Bao

Abstract. In this letter, by using homotopic invariance, toplogical degree theory and Lyapunov functional method, we investigate the global exponential robust stability of static neural network models with reaction-diffusion terms and S-type distributed delays. We present a theorem and a corollary which generalize the results of related literature. Moreover, the exponential convergence rate is estimated. Keywords: Static neural networks, Distributed delays, Exponential robust stability, Liapunov functional, Reaction-diffusion.

1 Introduction On the basis of the difference of basic variables (local field states or neuron states), a dynamical neural network can frequently be cast either as a local field neural network model or as a static neural network model [1,2]. Generally speaking, Hopfield neural networks [3], bidirectional associative memory networks [4] and cellular neural networks [5] all are local field neural network models. The recurrent back-propagation networks [6], brain-statein-a-box/domain type networks [7,8] and optimization-type neural networks in [9-11] are all in the static neural network model forms. The basic form of the local field model is n dxi (t) = −ai xi (t) + wij fj (xj (t)) + Ii , i = 1, 2, · · · , n, dt j=1

(1)

where n denotes the number of neurons; wij is the value of the synaptic connectivity from neuron j to i; fi (·) is the nonlinear activation function of Shuping Bao College of information Science and Technology, Qingdao University of Science and Technology, Qingdao 266061, China H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 69–79. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

70

S. Bao

neuron i; Ii is the external input imposed on neuron i [1, 2]. With the same notation, the static model can be written as ⎛ ⎞ n dyi (t) = −ai yi (t) + fi ⎝ wij yj (t) + Ii ⎠ , i = 1, 2, · · · , n. (2) dt j=1 Local field neural network models have attracted a great deal of attention. Many deep theoretical results have been obtained for local field neural network, we can refer to [3,4,5,12-18] and so on. Recently, more and more attention has been paid to the static models for its great potential of applications. In [2], the authors presented a reference model approach for investigating the stability of system (2), but did not consider the influence of time delays. In [19], the authors investigated the global robust asymptotic stability of static neural network models with S-type distributed delays on a finite interval, In [20], the authors discussed local robust stability of static neural network (SNN) with S-type distributed delays, but they both didn’t consider the effect of reaction-diffusion phenomina. In this paper, we will study the global exponential robust stability of the following static reaction-diffusion neural networks with S-type distributed delays on a finite interval: ⎧ m ∂ui ∂ ∂ui ⎪ ⎪ ⎪ = (Dik ) − ai (t, λ)ui (t, x) ⎪ ⎪ ∂t ∂x ∂x k k ⎪ ⎪ k=1  ⎪ ! 0 ⎪ ⎪ n  ⎪ ⎨ +fi uj (t + θ, x)dωij (θ, λ) + Ii (t) (3) j=1 −r(λ) ⎪ ⎪ ⎪ ⎪ ∂ui ∂ui ∂ui ⎪ ⎪ := col( ,..., ) = 0, t ≥ σ ≥ 0, x ∈ ∂Ω, ⎪ ⎪ ⎪ ∂n ∂x ∂x 1 m ⎪ ⎪ ⎩ yi (σ + θ) = φi (θ), t ≥ σ, θ ∈ [−r(λ), 0], i = 1, 2, · · · , n, where n denotes the number of neurons; ai (t) > 0 and Dik (t, x, u) ≥ 0 represent the neuron charging time functions and smooth diffusion operator, respectively; xi (i = 1, ..., n) corresponds to the neuron ith coordinate in the space X; Ω is a bounded compact set with smmoth boundary ∂Ω and measure μ(Ω) > 0 in Rm ; is the nonlinear activation function of neuron i; Ii is the external input imposed on neuron i; λ ∈ Λ ⊂ R is the parameter, σ ∈ R, φi (θ, x) ∈ C([−r(λ), 0] × Rm ) is the initial conditions; ωij (θ, λ), i, j = 1, 2, · · · , n, are nondecreasing bounded variation functions ! 0 on [−r(λ), 0], uj (t + θ, x)dωij (θ, λ), i, j = 1, 2, · · · , n, are Lebesgue−r(λ)

Stieltjes integrable. ∗ There exist positive constants r, ai , ai , ωij , i, j = 1, 2, · · · , n, such that for 0 ∗ < any λ ∈ Λ, 0 < ai ≤ ai (λ, t) ≤ ai , 0 ≤ r(λ) ≤ r, | −r(λ) dωij (θ, λ)| ≤ ωij

∗ ∞, i, j = 1, 2, · · · , n. Let W = (ωij )n×n , A = diag(ai ), L = diag(li ),

Global Exponential Robust Stability

71

i = 1, 2, · · · , n, B = {φ(θ, x)|φ(θ, x) = (φ1 (θ, x), φ1 (θ, x), · · · , φn (θ, x)), φi (θ, x) ∈ C([−r(λ), 0] × Rm ), i = 1, 2, · · · , n}. Define φmax = max ! 1≤i≤n {max−r(λ)≤θ≤0 φi (θ)} as the maximum norm, 1

|φi (θ)|2 dx) 2 . Then B is a Banach space. For any

where φi (θ) = ( Ω

φ(θ) ∈ B and σ ∈ R, a solution of system (3) is a vector function u(t, x) = u(σ, φ, t, x) = (u1 (t, x), · · · , un (t, x)) satisfying (3) for t ≥ σ.

2 Preliminaries Definition 1. Suppose that u(t, x) = u(σ, φ, t, x) is a solution of system (3). The equilibrium point u∗ of system (3) is said to be globally exponentially stable if there exist constants M > 0, α > 0, such thatu−u∗ ≤ M e−α(t−t0 ) , where  ·  denotes the Euclidean norm. Definition 2. System (3) is said to be robust stable or globally exponentially robust stable if its equilibrium u∗ = (u∗1 , u∗2 , · · · , u∗n ) is stable or globally exponentially stable for any r(λ) ∈ [0, r] and ai (λ) ∈ [ai , ai ], i = 1, · · · , n. Lemma 1. If f (x, y, θ) is continuous on [a, b; c, d; α, β], ω(θ) is a non! β decreasing bounded variation function on [α, β], dω(θ) = ω ∗ < ∞, then α ! β f (x, y, θ)dω(θ) is continuous on [a, b; c, d]. g(x, y) = α

Proof. f (x, y, θ) is continuous on [a, b; c, d; α, β], so it is uniformly continuous, i.e., ∀ε ≥ 0, ∃δ ≥ 0 such that when |x1 −x2 | ≤ δ,|y1 −y2 | ≤ δ and |θ1 −θ2 | ≤ δ, we have |f (x1 , y1 , θ1 ) − f (x2 , y2 , θ2 )| ≤ ε. Since −|f (x1 , y1 , θ)−f (x2 , y2 , θ)| ≤ f (x1 , y1 , θ)−f (x2 , y2 , θ) ≤ |f (x1 , y1 , θ)−f (x2 , y2 , θ)|,

!

!

β

−

β

|f (x1 , y1 , θ) − f (x2 , y2 , θ)|dω(θ) ≤ α

|f (x1 , y1 , θ) − f (x2 , y2 , θ)|dω(θ), α

we have %! % % β % % % (f (x1 , y1 , θ) − f (x2 , y2 , θ))dω(θ)% ≤ εω ∗ . % % α % %! % % β % % % (f (x1 , y1 , θ) − f (x2 , y2 , θ))dω(θ)% ≤ εω ∗ . |g(x1 , y1 ) − g(x2 , y2 )| = % % α % Thus g(x, y) is continuous on [a, b; c, d].



72

S. Bao

From Lemma 1, we know that when the conditions of Lemma 1 are satisfied, ! lim ∗

x→x , y→y ∗

!

β

f (x, y, θ)dω(θ) =

β

lim

∗ ∗ α x→x ,y→y

α

f (x, y, θ)dω(θ).

∂ f (x, y, θ)and fy (x, y, θ) = Lemma 2. If f (x, y, θ), fx (x, y, θ) = ∂x ∂ f (x, y, θ)are continuous on [a, b; c, d; α, β], ω(θ) is a nondecreasing ∂y ! β bounded variation function on [α, β], dω(θ) = ω ∗ < ∞, then α ! β  ! β ∂ f (x, y, θ)dω(θ) = fx (x, y, θ)dω(θ), ∂x α α ! β  ! β ∂ f (x, y, θ)dω(θ) = fy (x, y, θ)dω(θ) ∂y α α ! β Proof. Let g(x, y) = f (x, y, θ)dω(θ). From the mean value theorem, α

g(x + Δx, y) − g(x, y) = Δx

!

β

α

!

f (x + Δx, y, θ) − f (x, y, θ) dω(θ), Δx

β

fx (x + ξΔx, y, θ)dω(θ)

= α

ξ ∈ [0, 1]. From Lemma 1, g(x + Δx, y) − g(x, y) ∂ g(x, y) = lim = lim Δx→0 Δx→0 ∂x Δx ! ! β lim fx (x + ξΔx, y, θ)dω(θ) = = α Δx→0

Similarly,

∂ ∂y

!

β

 ! f (x, y, θ)dω(θ) =

α

!

β

fx (x + ξΔx, y, θ)dω(θ) α β

fx (x, y, θ)dω(θ). α

β

fy (x, y, θ)dω(θ). α

3 Global Exponential Robust Stability Theorem 3. Assume that (T1 ) |fi (y1 ) − fi (y2 )| ≤ li |y1 − y2 |, i = 1, · · · , n (T2 ) A − LW ∗ is an M -matrix. Then system (3) is globally exponentially robust stable.



Global Exponential Robust Stability

73

Proof. Part I. Existence of the equilibrium. We can prove that for any constant inputs I ∈ Rn ,λ ∈ Λ, r(λ) ∈ [0, r], ai (t, λ) ∈ [ai , ai ] , i = 1, · · · n, system (3) has at least an equilibrium, by homotopic invariance, topological degree theory. From (T1 ), it follows that |fi (s)| ≤ li |s| + |fi (0)|, i = 1, 2, · · · , n, ∀s ∈ R. Let h(u, t) = A(t)u − f (u) = 0,

(4)

Obviously, the solutions of (4) are the equilibria of system (3). Define the ¯ × [0, 1] → Rn as follows homotopic mapping H(u, t) : Ω H(u, t, η) = (H1 (u, t, η), ..., Hn (u, t, η)) = ηh(u, t) + (1 − η)u, η ∈ J = [0, 1].

|Hi (u, t, η)| = |η[ai (λ, t)ui − fi

n 

!

j=1



0 −r(λ)

+ (1 − η)ui |

≥ [1 + η(ai − 1)]|ui | − η|fi & ≥ [1 + η(ai − 1)]|ui | − η li | ≥ [1 + η(ai − 1)]|ui | − ηli

uj (t + θ, x)dωij (θ, λ) + Ii ] !

n  j=1

n  j=1

n  j=1

0

uj (t + θ, x)dωij (θ, λ) + Ii −r(λ) '

 |

∗ wij uj | + li |Ii | + |fi (0)|

∗ |wij ||uj | − η(li |Ii | + |fi (0)|)

i = 1, 2, . . . , n.

(5) That is, H + ≥ [E + η(A − E)][u]+ − ηLW ∗ [u]+ − η[LI + + f + (0)] = (1 − λ)[u]+ + η(A − σW ∗ )[u]+ − η[LI + + f + (0)]

(6)

where H + = [|H1 |, |H2 |, . . . , |Hn |]T ,[u]+ = [|u1 |, . . . , |un |]T , I + = [|I1 |, |I2 |, . . . , |In |]T , f + (0) = [|f1 (0)|, |f2 (0)|, . . . , |fn (0)|]T , E is an identity matrix. Since C = A−LW ∗ is an M-matrix, we have (A−LW ∗ )−1 ≥ 0(nonnegative matrix) and there exists Q = (Q1 , Q2 , · · · , Qn )T > 0. namely Qi > 0, i = 1, 2, . . . , n. such that (A − LW ∗ )Q > 0. Let " U (R0 ) = u : [u]+ ≤ R0 = Q + (A − LW ∗ )−1 [LI + + f + (0)] (7) Then, U (R0 )is not empty and it follows from (7) that for any u ∈ ∂U (R0 )(boundary of U (R0 )), " H + ≥ (1 − η)[u]+ + η(A − LW ∗ ) [u]+ − (A − LW ∗ )−1 [LI + + f + (0)] (8) = (1 − η)[u]+ + η(A − LW ∗ )Q > 0, η ∈ J = [0, 1].

74

S. Bao

That is,H(u, t, η) = 0, ∀u ∈ ∂U (R0 ), η ∈ [0, 1]. So, from homotopy invariance theory[21], we have d(h, U (R0 ), 0) = d(H(x, 1), U (R0 ), 0) = d(H(x, 0), U (R0 ), 0) = 1, where d(h, U (R0 ), 0)denotes topological degree. By topological degree theory, we can conclude that h(u,t)=0 has at least one solution in U (R0 ). That is, system (3) has at least one equilibrium. Part II. Uniqueness of equilibrium and its global exponential robust stability. Let u∗ = (u∗1 , u∗2 , · · · , u∗n )T be an equilibrium of system (3) and u(t, x) = (u1 (t, x), u2 (t, x), · · · , un (t, x))T = (u∗1 , u∗2 , · · · , u∗n )T is any solution of system (3). Rewrite system (3) as m ∂ui − u∗ ∂ ∂ui − u∗ = (Dik ) − ai (t, λ)(ui − u∗ ) ∂t ∂xk ∂xk

 ! 0k=1 n  +fi uj (t + θ, x)dωij (θ, λ) + Ii (t) (9) j=1 −r(λ)

 ! 0 n  ∗ uj dωij (θ, λ) + Ii (t) , i = 1, 2, · · · , n, −fi j=1

−r(λ)

Multiply both sides of (9) by ui − u∗ and integrate it, we get

! ! m ! 1 d ∂ui − u∗ ∂ (ui − u∗ )2 dx = (ui − u∗ ) (Dik )dx − ai (t, λ) (ui − 2 dt Ω ∂xk ∂xk Ω k=1 Ω  u∗ )2 dx + Ω (ui − u∗ )Υ dx

(10)

Where Υ =

fi

n !

−r(λ)

j=1



0

uj (t + θ, x)dωij (θ, λ) + Ii (t)

− fi

n ! j=1

0 −r(λ)

 u∗j dωij (θ, λ)

+ Ii (t)

by the boundary condition of equation (3), we get n ! Ω

k=1

!

= ∂Ω

=−

(ui − u∗i )

∂(ui − u∗i ) ∂ (Dik dx = ∂xk ∂xk

((ui − u∗i )Dik

m  k=1

Dik ( Ω

∂(ui − ∂xk

u∗i ) m )k=1 dx

∂(ui −u∗ i) 2 ) dx ∂xk

−

! Ω

(ui − u∗i )∇(Dik

m ! k=1

Ω

Dik (

∂(ui − u∗i ) m )k=1 dx ∂xk

∂(ui − u∗i ) 2 ) dx ∂xk

(11) ∂ ∂ T in which ∇ = ( ∂x , . . . , ) is the gradient operator. From (10) and (11), ∂x 1 m assumption (T1 ) and H¨ older inequality, we have n ! 0 d ∗ 2 ∗ 2 ||ui − ui ||2 ≤ −2ai (t, λ)||ui − ui ||2 + 2li uj (t + θ, x) − u∗j × dt −r(λ) j=1 ! 0 n  ∗ ∗ 2 ||ui − ui ||2 dωij (θ, λ) ≤ −2ai ||ui − ui ||2 + 2li uj (t + θ, x) − Θ j=1

−r(λ)

(12)

Global Exponential Robust Stability

75

Where Θ=u∗j .||ui − u∗i ||2 dωij (θ, λ). I.e., n ! 0 d ∗ ∗ ||ui − ui ||2 ≤ −ai ||ui − ui ||2 + li uj (t + θ, x) − u∗j dωij (θ, λ) dt −r(λ) j=1

(13) We can choose a sufficient small positive constant ε such that βi ai − li

n

∗ βj ωji > ε, i = 1, · · · n.

(14)

j=1

Let us consider functions Fi (ξi ) = βi (ai − ξi ) − li

n

! βj

j=1

0 −r(λ)

e−ξi θ dωij (θ, λ), ξi ∈ [0, +∞), i = 1, 2, ..., n.

From (14), we get Fi (0) > ε > 0 and Fi (ξ) is continuous for ξi ∈ [0, +∞), moreover,Fi(ξ) → −∞ as ξ → +∞, thus there exist constant αi ∈ (0, +∞) such that ! 0 n Fi (αi ) = βi (ai − αi ) − li βj e−αi θ dωij (θ, λ) = 0, (15) j=1

−r(λ)

for i ∈ 1, 2, ..., n. By choose α = max1≤i≤n αi , we have ! 0 n Fi (α) = βi (ai − α) − li βj e−αθ dωij (θ, λ) ≥ 0, j=1

Define the Lyapunov functional ⎛

! n n ∗ αt V (t) = βi ⎝ui − ui 2 e + li i=1

(16)

−r(λ)

j=1

0

−r(λ)

⎞ Ξdωij (θ, λ) ⎠ .

$ # t Where Ξ= t+θ uj (s, x) − u∗j 2 eα(s−θ) ds . Since 0 ≤ r(λ) ≤ r, Let ωij (θ, λ) = ωij (−r(λ), λ) when θ ∈ [−r, −r(λ)), we have that θe−αθ and ωij (θ, λ) can be% continued as %the bounded variation function on [−r, 0] and ! % 0 %  −αθ ∗ % |θe |≤M , % dωij (θ, λ)%% ≤ ωij , i, j = 1, · · · , n. −r

From Lemma 2, the upper right Dini-derivative of V (t) along the solution of system (3) can be calculated as DV + =

n  i=1

+

n  j=1

! li

 βi 0

−r(λ)

αeαt ui − u∗i 2 + eαt

dui − u∗i 2 dt )

[uj (t, x) −

u∗j 2 eα(t−θ)

− uj (t + θ, x) −

u∗j 2 eαt ]dωij (θ, λ)

76

S. Bao

≤

(

n 

βi e

α − ai +

αt

i=1

= −e

i=1

li

βi (ai − α) −

)

0

e

−αθ

−r(λ)

j=1

&

n 

αt

!

n 

n 

! l i βj

'

0

e

−αθ

−r(λ)

j=1

||ui − u∗i ||2

dωij (θ, λ)

||ui − u∗i ||2

dωij (θ, λ)

From (16), we have DV + ≤ 0, t > 0 . So V (t) ≤ V (0), t > 0. Since V (t) =

n

βi eαt ui − u∗i 2 ≥ min {βi }eαt 1≤i≤n

i=1 n 

n

n 

u∗i 2 eαt

!

ui − u∗i 2

i=1

!

0





0

ui (0, x) − + li ϑ(s, x)ds dωij (θ, λ) j=1 −r(λ) θ

n ui (0, x) − u∗i 2 eαt + ≤ max {βi } 1≤i≤n i=1 

! 0 n  n  ∗ α(ρ−θ) li |θ|uj (ρ, x) − uj 2 e dωij (θ, λ) (θ ≤ ρ ≤ 0) i=1 j=1 −r(λ)

 n n  ∗ ≤ max {βi } 1 + max {li M ωij } × sup ui (s, x) − u∗i 2

V (0) =

i=1

βi

1≤i≤n

1≤i≤n

−r≤s≤0 i=1

j=1

Where ϑ(s, x) = uj (s, x) − u∗j 2 eα(s−θ) . We let

max1≤i≤n {βi } 1 + max1≤i≤n {li

n  j=1

M=

min1≤i≤n {βi }

 

∗ M ωij }

,

And then M ≥ 1 and n

ui −

u∗i 2

≤ Me

j=1

−αt

sup

n

−r≤s≤0 i=1

ui (s, x) − u∗i 2 ,

So the equilibrium u∗ of system (3) is exponentially stable, and the exponential converging velocity index α = min1≤i≤n {αi } from (16). Because of the arbitrariness of u, u∗ is globally exponentially stable. Thus, system (3) is globally exponentially robust stable.From[14,15], we have: Corollary 4. Assume that system(3) satisfies (T1)and one of the following: (T3) A − LW ∗ is a matrix with strictly diagonal dominance of column (or row). n  ∗ βj li wji ∗ (T4) ai − li wij >

j=1,j =i

βi > 0

, βi > 0, i = 1, 2, ..., n

Global Exponential Robust Stability n 

(T5) max

1≤i≤n

(T6) max

1≤i≤n

j=1 n  j=1

77

∗ lj wji

ai

<1

∗ ∗ li wij + lj wji

2ai

<1

Then system (3) is globally asymptotically robust stable.

4 Remark and Conclusion System (3) includes many models as special cases. For example, when the smooth operators Dik = 0, (i = 1, 2, ..., n; k = 1, 2, ..., m), ( ωij , θ = 0 1 1 ωij (θ, λ) = , r(λ) = r, ai (λ) = , fi (·) = gi (·), τ τ 0, −r ≤ θ < 0 System (3) is model (2). When ⎧ m  k ⎪ ⎪ ωij (λ), θ = r0 = 0 ⎪ ⎪ ⎪ k=0 ⎪ ⎪ ⎪ ⎪ m  ⎪ ⎪ k ⎪ ωij (λ), r1 ≤ θ < 0 ⎪ ⎪ ⎪ k=1 ⎪ ⎪ ⎨ m ωij (θ, λ) = ω k (λ), r2 ≤ θ < r1 ⎪ k=2 ij ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ...... ⎪ ⎪ ⎪ ⎪ (m) ⎪ ⎪ ωij (λ), rm ≤ θ < rm−1 ⎪ ⎪ ⎪ ⎪ ⎩ 0, −r ≤ θ < rm i, j = 1, . . . , n System (3) becomes a static neural network model with discrete time delays: ∂ ∂ui ∂ui (Dik ) − ai (λ)ui (t, x) + fi = ∂t ∂xk ∂xk m

k=1

n m

 k ωij (λ)uj (t

− rk , x) + Ii

.

k=0 j=1

(17)

When ωij (θ, λ) ∈ C 1 [−r, 0], i, j = 1, . . . , n, the value of the synaptic connectivity from neuron j to i is a continuous function on [−r, 0], which means that time delays influence the network continuously, system (3) belongs to the static model with continuous time delays:

78

S. Bao

∂ ∂ui ∂ui (Dik ) − ai (λ)ui (t, x) + fi = ∂t ∂xk ∂xk m

k=1

n ! j=1

0 −r

 

uj (t`, x)ωij (θ, λ)dθ + Ii

,

.

(18) Where t` = t + θ. As many neural networks are described by static models (see[2]), system(3) is widely representative. The global exponential robust stability of neural networks can be widely applied in solving control and optimization problems, and the M -matrix is easy to verify, so the results in this paper are significant and practical in both theory and application. Acknowledgements. This paper was supported by the National Natural Science Foundations of China under Grant 60673101 and 60674020.

References 1. Xu, Z., Qiao, H., Peng, J., Zhang, B.: A Comparative Study of two Modeling Approaches in Neural Networks. J. Neural Networks 17, 73–85 (2004) 2. Qiao, H., Peng, J., Xu, Z., Zhang, B.: A Reference Model Approach to Stability Analysis of Neural Networks. IEEE Trans. on Systems, Man and Cybernetics 33, 925–936 (2003) 3. Hopfield, J.: Neurons with Graded Response have Collective Computational Properties Like Those of Two-state Neurons. In: National Academy of Sciences of the United States of America, pp. 3088–3092. NASA Press (1984) 4. Kosko, B.: Bidirectional Associative Memories. IEEE Trans. Syst., Man Cybernetics 18, 49–60 (1988) 5. Chua, L., Yang, L.: Cellular Neural Networks: Theory and applications. IEEE Trans. Circuits Syst. 35, 1257–1290 (1988) 6. Pineda, F.J.: Generalization of Back-propagation to Recurrent Neural Networks. Phys. Rev. Lett. 59, 2229–2232 (1987) 7. Li, J., Michel, A.N., Porod, W.: Analysis and Synthesis of A Class of Neural Networks Linear Systems Operating on a Closed Hypercube. IEEE Trans. Circuits Syst. 36, 1406–1422 (1989) 8. Varga, I., Elek, G., Zak, H.: On the Brain-state-in-a-convex-domain Neural Models. Neural Networks 9, 1173–1184 (1996) 9. Bouzerdoum, A., Pattison, T.R.: Neural network for quadratic optimization with bound constraints. IEEE Trans. Neural Networks 4, 293–303 (1993) 10. Forti, M., Tesi, A.: New Conditions for Global Stability of Neural Networks with Applications to Linear and Quadratic Programming Problems. IEEE Trans. Circuits Syst. I 42, 354–366 (1995) 11. Friesz, T.L., Bernstein, D.H., Mehta, N.J., et al.: Day-to-day Dynamic Network Disequilibrium and Idealized Traveler Information Systems. Operat. Res. 42, 1120–1136 (1994) 12. Song, Q., Cao, J., Zhao, Z.: Periodic and Its Exponential Stability of Reactiondiffusion Recurrent Neural Networks with Continuously Distributed Delays. Nonlinear Analysis Series B 7, 65–80 (2006)

Global Exponential Robust Stability

79

13. Linshan, W., Zhe, Z., Yangfan, W.: Stochastic Exponential Stability of the Delayed Reaction-diffusion Recurrent Neural Networks with Markovian Jumping Parameters. Physics Letters A 372, 3201–3209 (2008) 14. Linshan, W., Yuying, G.: Global Exponential Robust Stability of Reactiondiffusion Interval Neural Networks with Time-varying Delays. Physics Letters A 350, 342–348 (2006) 15. Yang, X.F., Liao, X.F., Tang, Y.Y.: Guaranteed Attractivity of Equilibrium Points in a Class of Delayed Neural Networks. Int. J. Bifurcat. Chaos 16, 2737– 2743 (2006) 16. Mohamad, S.: Global Exponential Stability in DCNNs with Distributed Delays and Unbounded Activations. J. Comput. Appl. Math. 205, 161–173 (2007) 17. Mohamad, S., Gopalsamy, K., Haydar, A.: Exponential Stability of Artifical Neural Networks with Distributed Delays and Large Impulses. Nonlinear Anal. Real World Appl. 9, 872–888 (2008) 18. Huang, Z.K., Xia, Y.H., Wang, X.H.: The Existence and Exponential Attractivity of j-almost Periodic Sequence Solution of Discrete Time Neural Networks. Nonlinear Dyn. 50, 13–26 (2007) 19. Wang, M., Wang, L.: Global Asymptotic Robust Stability of Static Neural Network Models with S-type Distributed Delays. Mathematical and Computer modelling 44, 218–222 (2006) 20. Guo, D.J., Sun, J.X., Liu, Z.L.: Functional Methods of Nonlinear Ordinary Differential Equations. Shandong Science Press, Jinan (1995)

A LEC-and-AHP Based Hazard Assessment Method in Hydroelectric Project Construction Jian-lan Zhou, Da-wei Tang, Xian-rong Liu, and Sheng-yu Gong

*

Abstract. Aiming at the defects of simply using LEC method to evaluate the risk factors in working system, this paper thoroughly analyses the main factors which can influence the hazard assessment; adapts LEC and AHP assessment methods with the combination of the former accident cases and experts’ experiences; associates multiple factors that can lead to accidents, assigns reasonable weights, thus determines the importance order of the factors. The statistical data verifies that the above method implements the quantitative assessment objectively, which can guarantee the consistency of the evaluation result and statistical data. Keywords: Hydroelectric project construction, Hazard assessment, LEC, AHP, Statistical data on accidents.

1 Introduction As we know, the Three-Gorges Project’ commencement ceremony was held in 1993, which is 15 years ago. Recently, the other two hydroelectric projects on the Yangtse River, namely Xiluodu hydroelectric project and Xiangjiaba hydroelectric project, are also on their way in construction. All these projects share some common features: the scale of the project is huge; it will take a rather long time to fully finish the project; the construction process of the project is extremely complicated and the project involves very high risks. Thus, it is very urgent to carry out the identification, evaluation as well as the monitoring of the risk sources, to strengthen the control of the potential accidents, to prevent and respond to the severe accidents during the process of the construction of hydroelectric project. However, there is little systematic research on the above issues, and there are no Jian-lan Zhou . Xian-rong Liu Department of Work Safety, China Three Gorges Project Corporation, Yichang 443002, China

*

Jian-lan Zhou . Da-wei Tang Institute of Systems and Engineering, Huazhong University of Science & Technology, Wuhan 430074, China Sheng-yu Gong Sinohydro Engineering Bureau No. 3, Ankang 725011, China {ZhouJian-lan,TangDa-wei,LiuXian-rong,GongSheng-yu, ZHOUJL1999}@163.com H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 81–90. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

82

J.-l. Zhou et al.

existing regulations or laws, which can cover the relevant issues of risk sources in construction of hydroelectric project, either. Therefore, it is very necessary and emergent to carry out some research on them [1].

2 Identification and Evaluation of the Risk Sources The risk sources in the hydroelectric project construction refer to those working positions, working process and working tools which may lead to human injury or death, economic loss, working environment breach or the combination of the above consequences. To be more specifically, the risk sources include earth excavation, rock excavation, shaft excavation, slope excavation, mechanical equipments, electrical equipments, dynamotor installation and so on. Accordingly, the identification of the risks sources include the analysis of working environment, working layout, working process, working tools, transportation routes, equipments with high risks, workers’ psychology and so on. Through the identification, we can get various factors which may cause accidents in the construction process, including physical factors (like equipment defects, protection defects, electricity hazards etc.), chemical factors (like flammable materials, explosive materials, poison materials etc.), biological factors (like pathogenic microorganism etc.), psychological factors (like overloaded, abnormality in health etc.), behavioral factors (like miss-leading, miss-operating etc.) and so on. The evaluation of the risk sources is a dynamic process which involves many interactive aspects and the criteria of the evaluation should be scientific, rational and comprehensive. Therefore, based on the features of the risk sources in hydroelectric project construction, this paper will carry out the evaluation in the following steps: first, LEC will be used to identify and roughly evaluate the factors which may cause the accidents; then, incorporated with the accidents cases and expert experience, AHP will be applied to perform the correlation analysis on those factors, and through the specification of the relevant weights, the importance of each factor concerning the potential accidents will be determined. The paper is based on the example of scaffold operation. The corresponding risk factors which may lead to accidents as well as the evaluation framework are given as follows: A

B1

C1

C2

C3

C4

B2

C5

C6

C7 C8

B3

C9

C1 0

B4

C1 1

C1 2

C1 3

B5

C1 4

C1 5

C1 6

Fig. 1 The risk evaluation framework of working on the scaffold

B6

C1 7

C1 C1 8 9

C2 0

C2 1

C2 2

A LEC-and-AHP Based Hazard Assessment Method

83

In the above figure: A stands for ‘risk evaluation of working on scaffold’; B1 stands for ‘falling from high places’; B2 stands for ‘collapse’; B3 stands for ‘objects hitting’; B4 stands for ‘fire’; B5 stands for ‘electric shock’; B6 stands for ‘others’ C1 stands for ‘haven’t made proper construction plan or developed safety related measures according to the features of the work before setting up and taking down the scaffold’; C2 stands for ‘the men who set up and take down the scaffold have not been properly trained’; C3 stands for ‘haven’t informed or haven’t completely informed the workers about the potential risks before they are working’; C4 stands for ‘working in abominable weather without any protection measures’; C5 stands for ‘setting up and taking down the scaffold without obeying design regulations’; C6 stands for ‘setting up the scaffold with unqualified materials’; C7 stands for ‘improper guidance from the commander or workers disobey the commander’; C8 stands for ‘working without obeying the rated load, overload, or unbalance load’; C9 stands for ‘improper or inadequate protection measures on working place; C10 stands for ‘workers don’t carry tools for protecting themselves’; C11 stands for ‘inadequate or improper cooperation when delivering objects among workers’; C12 stands for ‘intercrossing work by different groups simultaneously without proper protection measures’; C13 stands for ‘construction residues, tools and materials are not cleared in time’; C14 stands for ‘unauthorized people entering into working places because of no protection fence around the working places’; C15 stands for ‘rock falling along the slope’; C16 stands for ‘the bases of scaffold are not solid enough’; C17 stands for ‘fire caused by improper disposition of welding residues’; C18 stands for ‘flammable materials ignited by naked fire’; C19 stands for ‘improper layout of the temporary electric wires’; C20 stands for ‘electric leakage due to damp and naked wires’; C21 stands for ‘slipping, being hit or falling down due to carelessness’; C22 stands for ‘insufficient light in working place, slippery on the surface of working place’.

3 Working Conditions Hazard Analysis (LEC Method) LEC method uses the product of 3 factors to determine the extent of hazard of a certain object. These 3 factors are: the probability of risks (L); the frequency of exposure to the risks (E); and the severity of consequences (C). The value of each

84

J.-l. Zhou et al.

factor and its corresponding meanings are shown in table 1- table 3 [2]. Practically, the values of those factors are determined through the discussion of the experts or through the Delphi Method. Based on those values, the final value (D), which is used to specify the hazardous level, can be got from the product of L, E and C, i.e., D = LEC. The value of D and its corresponding meanings are described in table 4. Table 1. Probability of risks (L) Value

Description

10 6 3 1 0.5 0.2 0.1

Certain Quite likely Likely Unlikely Quite unlikely Nearly impossible Impossible

Table 2. Frequency of exposure to the risks (E) Value 10 6 3 2 1 0.5

Description

Constant exposure Daily exposure Weekly exposure Monthly exposure Occasional exposure Rare exposure

Table 3. Severity of consequences (C) Value 100 40 15 7 3 1

Description Extremely severe, more than 10 deaths or more than 50 seriously injured Very severe, 3-9 deaths or 10-49 seriously injured Quite severe, 1-2 deaths or 6-9 seriously injured Severe, 2-5 seriously injured Severe, 1 seriously injured Not so severe, slightly injured

Table 4. The extent of hazard and the corresponding level Value

Description

Level

>=320

Extremely hazardous, must stop working Very hazardous, countermeasures are needed immediately Quite hazardous, countermeasures are needed hazardous, attentions should be paid to Not so hazardous, acceptable

Level 1

[160, 320) [70, 160) [20, 70) <20

Level 2 Level 3 Level 4 Level 5

A LEC-and-AHP Based Hazard Assessment Method

85

Table 5. The risk evaluation results of a certain operation process with scaffold Accident type Factor

B1

B2

B3

B4

B5

B6

L

E

L*E

C

D

Hazardous level

C1

10

6

60

7

420

1

C2

10

6

60

7

420

1

C3

10

6

60

7

420

1

C4

6

1

6

7

42

4

C5

10

6

60

40

2400

1

C6

6

6

36

40

1440

1

C7

3

2

6

15

90

3

C8

10

6

60

7

420

1

C9

6

6

36

7

252

2

C10

10

2

20

7

140

3

C1

10

6

60

7

420

1

C2

6

6

36

7

252

2

C3

10

6

60

7

420

1

C5

10

6

60

7

420

1

C6

10

6

60

40

2400

1

C7

6

6

36

7

252

2

C8

6

6

36

7

252

2

C16

10

6

60

40

2400

1

C10

6

6

36

7

252

2

C11

3

6

18

15

270

2

C12

10

6

60

7

420

1

C13

10

6

60

3

180

2

C14

10

6

60

7

420

1

C15

10

6

60

7

420

1

C17

10

6

60

7

420

1

C18

6

6

36

7

252

2

C4

3

1

3

15

45

4

C19

6

6

36

15

540

1

C20

6

6

36

7

252

2

C21

6

6

36

1

36

4

C22

10

6

60

3

180

2

In our example of a certain operation process with scaffold, after consulting the experts, the value of L, E and C can be summarized in table 5. Note that, those values can only reflect the intrinsic features of the corresponding risks, and influences of countermeasures against them are not incorporated.

86

J.-l. Zhou et al.

From the above table, we can see that the same factor can lead to different types of accidents, different values of L, E, C and D. This is reasonable. For example, factor C5 may lead to accident B1 and B2, although the hazardous level are both 1, their values of D have a significant different.

4 AHP The basic idea of AHP is to decompose the target objectives into different levels and perform the qualitative and quantitative analysis based on the hierarchical structure. Based on the analysis of the essence and factors of a complex decision problem, as well as the relationship among those factors, AHP can mathematically model the decision process without much quantitative information. In this way, it can conveniently analyze the complex and ill-structured decision problems with multi-objective, and multi-criteria [3]. The process of AHP can be roughly divided into 4 steps: determine the weights, specify the hierarchy structures, construct pair-wise comparison matrix and rank. In order to perform the ranking, the maximum Eigen-value of the comparison matrix and its corresponding Eigen-vector should be specified. Concerning the consistency test, AHP introduces the factor of CR, which equals to CI/RI. When CR is less than 0.1, the consistency is acceptable. Note that CI = (λmax − n) /(n − 1) ( λmax is the maximum Eigen-value of the comparison matrix, n is the dimension of the matrix), and RI is a series of constant depend on n. The value of RI is shown in table 6. Table 6. The value of RI n

1

2

3

4

5

6

7

8

9

10

RI

0

0

0.58

0.90

1.12

1.24

1.32

1.41

1.46

1.49

5 Hazard Assessment Based on AHP and LEC The product of L and E in table 5 can represent the possibility of the accident during a certain period, in addition, the product of L and E can also be seen as the weight of the consequence C. In table 5, the possible values of L × E include 60, 36, 20, 18, 6 and 3. Further, these values can be categorized into 5 levels: Level A for 60, Level B for 36, Level C for 2o and 18, Level D for 6 and Level E for 3. Based on these levels, the meaning of the scale in the comparison matrix can be determined as follows: According to the scales and their corresponding meanings in table 7, and incorporating the values of L × E in table 6, the comparison matrixes can be got. Table 8 gives the factor comparison matrix of B1: b1 =10.2311 , CRb1 =0.0172 . Since CR is much From the above matrix, we can get λmax less than 0.1, the consistency is quite satisfactory. In addition, the corresponding Eigen-vector which can be seen as the weight vector is:.

A LEC-and-AHP Based Hazard Assessment Method

87

Table 7. The scales in comparison matrix and their meanings scale

Corresponding relation between the factors

1

A / A, B / B, C / C , D / D, E / E

Corresponding meaning of the scale The

two

factors

are

equally

important 3

One factor is slightly more important

A / B , B / C , C / D, D / E

than the other 5

One factor is more important than

A / C , B / D, C / E

the other 7

One factor is much more important

A / D, B / E

than the other 9

One factor is extremely more im-

A/ E

portant than the other The comparison between factor i and factor j is expressed by bij, The

Reciprocal

comparison between factor j and factor i is expressed by bji= 1/bij

Table 8. Factor comparison matrix of B1

b1 c1 c2 c3 c4 c5 c6 c7 c8 c9 c10

c1

c2

c3

c4

c5

c6

c7

c8

c9

c10

1

1

1

7

1

3

7

1

3

5

1

1

1

7

1

3

7

1

3

5

1

1

1

7

1

3

7

1

3

5

1/7

1/7

1/7

1

1/7

1/5

1

1/7

1/5

1/3

1

1

1

7

1

3

7

1

3

5

1/3

1/3

1/3

5

1/3

1

5

1/3

1

3

1/7

1/7

1/7

1

1/7

1/5

1

1/7

1/5

1/3

1

1

1

7

1

3

7

1

3

5

1/3

1/3

1/3

5

1/3

1

5

1/3

1

3

1/5

1/5

1/5

3

1/5

1/3

3

1/5

1/3

1

ωb1 =(0.158, 0.158, 0.158, 0.019, 0.158, 0.068, 0.019, 0.158, 0.068, 0.035) Similarly, λmax RC ω of the other comparison matrix concerning B2, B3, B4,

、 、

B5 and B6 can also be specified as follows: b2 λmax = 8, CR b2 = 0, ω b 2 =(0.167, 0.056, 0.167, 0.167, 0.167, 0.056, 0.056, 0.167) b3 λmax = 6.0393, CR b3 = 0.0063, ω b 3 =(0.082, 0.041, 0.219, 0.219, 0.219, 0.219)

88

J.-l. Zhou et al.

Table 9. Evaluation value of each factor and the corresponding ranking factor

D’

ranking

D

Original ranking

B1C5

1.896

1

2400

1

B2C6

1.196

2

2400

1

B2C16

1.196

3

2400

1

B5C19

0.918

4

540

5

B1C6

0.816

5

1440

4

B3C12

0.518

6

420

6

B3C14

0.518

6

420

6

B3C15

0.518

6

420

6

B5C20

0.428

9

252

18

B1C1

0.332

10

420

6

B1C2

0.332

10

420

6

B1C3

0.332

10

420

6

B1C8

0.332

10

420

6

B3C13

0.222

14

180

25

B2C1

0.209

15

420

6

B2C3

0.209

15

420

6

B2C5

0.209

15

420

6

B3C11

0.208

18

270

17

B3C10

0.194

19

252

18

B1C9

0.143

20

252

18

B4C17

0.137

21

420

6

B5C4

0.132

22

45

29

B1C7

0.086

23

90

28

B1C10

0.074

24

140

27

B2C2

0.070

25

252

18

B2C7

0.070

25

252

18

B2C8

0.070

25

252

18

B6C22

0.059

28

180

25

B4C18

0.046

29

252

18

B1C4

0.040

30

42

30

B6C21

0.007

31

36

31

A LEC-and-AHP Based Hazard Assessment Method

89

b4 λmax = 2, CR b4 = 0, ω b 4 =(0.75,0.25) b5 λmax = 3, CR b5 = 0, ω b 5 =(0.067, 0.467, 0.467) b6 λmax = 2, CR b6 = 0, ω b 6 =(0.25, 0.75)

Since all the value of CR above is much less than 0.1, the consistencies are all very satisfactory. According to the actual severe accidents data from the 3 hydroelectric projects mentioned in the introduction, B1 happens 17 times, B2 happens 8 times, B3 happens 21 times, B4 happens 0 times, B5 happens 4 times, B6 happens 0 times. Based on the experts’ opinion, the values of λmax RC ω of comparison matrix concerning A in figure 1 are:

、 、

a λmax = 6.4031, CR a = 0.065, ω a =(0.3, 0.179, 0.338, 0.026, 0.131, 0.026) .

Using Dbj ' = ω aj ω bj cbj ( i = 1, , k j ; j = 1, 2, , 6 ) , we can get the evaluation value of D of each factor and the corresponding ranking. The results are shown in the table 9: i

i

i

6 Result Analysis From table 9, we can see that the importance of the first 3 factors remain unchanged. These factors are C5 which can cause B1, C6 which can cause B2 and C16 which can cause B2. The reason for why B1C5 ranks first is that it is the most primary factor which can cause B1, another primary factor is B1C6, which ranks 5th. The rankings of other factors which can cause B1 have a slight decrease, for example, the ranking of B1C1 has decreased to 10 from 6. As for B2, C6 and C16 are two primary causes, which rank 2nd and 3rd respectively. B5 happens a lot in actual cases, and the main reason for B5 is C19, ranking 4th, another cause of B5 ---- C20 ranks 9th. There are negligible changes in the rankings of the factors which can cause B3. Among the factors which can cause B4 and B6, B4C17 ranks highest, but it only ranks 21st, the reason for this phenomenon is that B4 and B6 never happen in reality. Besides the intrinsic features of the risk sources, the occurrences of accidents also depend on the establishment and fulfillment of the security management regulations. Based on the analysis above, and considering the accident investigation results, the following facts are contributes to the frequent occurrences of the accidents in operation with scaffold: The security management regulation concerning the scaffold is insufficient; Security criteria are not strictly obeyed; Supervising companies misunderstand the relationship between the production and security.

7 Conclusion Based on the example of operation with scaffold during the construction of hydroelectric projects, we proposed a LEC-and-AHP based hazard assessment method by constructing a hierarchy model of risk factors and the comparison matrixes. With the help of actual cases and experts’ experience, the method

90

J.-l. Zhou et al.

determines the importance of risk factors which can lead to different kinds of accidents through the analysis on those factors and the assignment of different weights accordingly. The future research may include the establishment of a set of security assessment criteria and a unified security checking list of the construction of hydroelectric projects.

References 1. Lv, A., Wang, P., Feng, L.: Identification, Appraisal and Control of Hazard Installations in Construction Engineering. Journal of Huazhong University of Science and Technology 1, 43–45 (2006) 2. Liu, Zhang, X., Liu, G.: The Application Guide of Safety Assessment method. Chemical Industry Press, Safety Science and Engineering Publishing Center (2005) 3. Wang, L., Xu, S.: AHP Introduction. China Renmin University Press (1990)

A Stochastic Lotka-Volterra Model with Variable Delay Yong Xu, Song Zhu, and Shigeng Hu

Abstract. Since population models are often subject to environmental noise, in this paper we stochastically perturb the Lotka-Volterra model with   variable delay x(t) ˙  = diag(x(t)) b + Ax(t) + Bx(t − δ(t)) into the Itˆ o form   dx(t) = diag(x(t)) b + Ax(t) + Bx(t − δ(t)) dt + σdw(t) . We show that under certain conditions, the deterministic delay equation and the perturbed delay equation have similar behaviour in the sense that both have positive solutions which will not explode to infinity in a finite time and, in fact, will be ultimately bounded. Keywords: Stochastic Lotka-Volterra model, Variable delay, Ultimate boundedness, Itˆ o s formula. AMR Subject Classifications: 34K40, 60H10, 60J65.

1 Introduction The variable delay Lotka-Volterra model for n interacting species is described by the n-dimensional variable delay differential equation n n , x˙ i (t) = xi (t) bi + aij xj (t) + bij xj (t − δj (t)) , j=1

i = 1, · · · , n ,

(1)

j=1

where xi (t) represents the population size of ith species, bi is the inherent net birth rate of ith species, aij and bij (i, j = 1, · · · , n) represent their Yong Xu · Shigeng Hu School of Mathematics and Statistics, Huazhong University of Science and Technology, Wuhan 430074, China Song Zhu Department of Control Science and Engineering, Huazhong University of Science and Technology, Wuhan 430074, China [email protected] H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 91–100. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

92

Y. Xu, S. Zhu, and S. Hu

interactions rates and all these parameters are constants, δi (t) (i = 1, · · · , n) is the delay function. There is an extensive literature concerned with the dynamics of this variable delay model and here we only mention Gopalsamy and He [1], Huo and Li [2], Freedman and Wu [3], Liu and Chen [4], Li and Kuang [5], Teng el.ct. [6–8]. In [1], Gopalsamy and He discuss the following variable delay Lotka-Volterra model for two interacting species (   N˙ 1 (t) = N1 (t) b1 (t) − a11 (t)N1 (t − τ (t)) − a12 (t)N2 (t − τ (t)) (2)   N˙ 2 (t) = N2 (t) b2 (t) − a21 (t)N1 (t − τ (t)) − a22 (t)N2 (t − τ (t)) , where all bi (t) and aij (t) are positive periodic functions. They show that, in order to guarantee the existence and global asymptotic stability of the positive periodic solution of the system (2), the coefficient functions bi (t) and ai (t) need to satisfy a set of algebraic conditions. On the other hand, population systems are often subject to environmental noise. It is therefore useful to reveal how the noise affects on the population systems. Recently, stochastic Lotka-Volterra system receives an increasing attention. For example, Mao and his coauthors [9–11] show the presence of environmental noise may suppress explosion in population dynamics. Bahar et al. [12] show that under the sufficient large noise, the population will become extinct with probability one. Rudnicki et al. [13, 14] mainly examine the stationary distribution and extinction. However, little is known about the variable delay Lotka-Volterra system (1) with perturbation. This paper will fill the blank. To consider the environment noise, we may replace the rate bi by bi + σi w, ˙ where w˙ is a white noise (i.e., w(t) is a Brownian motion). Hence, Eq. (1) becomes the following Itˆo stochastic differential equation with variable delay n n ,# $ dxi (t) =xi (t) bi + aij xj (t) + bij xj (t − δj (t)) dt + σi dw(t) j=1

(3)

j=1

on t ≥ 0, where i = 1, · · · , n. In this paper we will show that under certain conditions, the deterministic systems (1) and the stochastic systems (3) behave similarly in the sense that both have positive solutions which will not explode to infinity in a finite time and, in fact, will be ultimately bounded. In other words, we show that under certain condition the noise will not spoil these nice properties. Moreover, under certain conditions, we find that for any p > 0, the pth moment of the solutions for the stochastic systems (3) is ultimately bounded. This paper is organized as follow: In the next Section, some notations and a useful lemma are given. In Section 3, we show Eq. (3) has a global positive solution. From the biological point of view, the asymptotic ultimate bound property is obtain in Section 4.

A Stochastic Lotka-Volterra Model with Variable Delay

93

2 Preliminaries Throughout this paper unless otherwise specified, we use the following notations. Let | · | denote the Euclidean norm in Rn . If A is a vector or matrix, its transpose is denoted by AT . If A is matrix, its trace norm is denoted by 2 |A| = trace(AT A). Let R+ = (0, ∞). For any x ∈ R, denote x+ = x∨0. For any x = (x1 , · · · , xn )T ∈ Rn , denote x∞ = max1≤i≤n xi . For any matrix A = (aij )n×n ∈ Rn×n , we use the following notations, Ai =

n

a+ ij ,

A¯i =

n

A¯ i =

a+ ij ,

j=1

j =i

n

a+ ji ,

i = 1, · · · , n.

j=1

Let (Ω, F , P) be a complete probability space with a filtration {Ft }t≥0 satisfying the usual conditions (i.e. it is right continuous and F0 contains all P -null sets). Assume that w(t) is a scalar Brownian motion defined on the complete probability space. If x(t) is an Rn -valued stochastic process on t ∈ [−τ, ∞), we let xt = {x(t + θ) : θ ∈ [−τ, 0]} for t ≥ 0. In this paper we consider the following Itˆo stochastic differential equation with variable delay n n ,# $ dxi (t) =xi (t) bi + aij xj (t) + bij xj (t − δj (t)) dt + σi dw(t) j=1

(4)

j=1

on t ≥ 0, where i = 1, · · · , n. This takes the matrix form ,  dx(t) = diag(x(t)) b + Ax(t) + By(t) dt + σdw(t) ,

(5)

where x(t) = (x1 (t), · · · , xn (t))T , y(t) = (y1 (t), · · · , yn (t))T = (x1 (t − δ1 (t)), · · · , xn (t − δn (t)))T , diag(x(t)) represents the n × n matrix with all elements zero except those on the diagonal which are x1 (t), · · · , xn (t), b = (b1 , · · · , bn )T , σ = (σ1 , · · · , σn )T , A = [aij ]n×n , B = [bij ]n×n , w(t) is a scalar Brownian motion, the delay function δi (t) ∈ C 1 ([0, ∞); [0, τ ]) (i = 1, · · · , n) and satisfies the following condition η := max inf (1 − δi (t)) > 0. 1≤i≤n t≥0

The initial data {x(t) : −τ ≤ t ≤ 0} = {ξ(t) : −τ ≤ t ≤ 0} ∈ C([−τ, 0]; Rn+ ), which is the family of continuous Rn+ -valued function ϕ with norm ϕ = sup−τ ≤t≤0 |ϕ(t)| < ∞. Before we state our main result, let us introduce a useful lemma. Lemma 1. Let p > 0. Assume that there a matrix A = (aij )n×n ∈ Rn×n such that aii < −Ai ,

i = 1, · · · , n.

(6)

94

Y. Xu, S. Zhu, and S. Hu

Then, for any x ∈ Rn+ , Ip (x) :=

n n

aij xpi xj ≤ −Kp xp+1 ∞ ,

(7)

i=1 j=1

where −Kp = max

1≤k≤n

, akk + Ak +

pp (p + 1)p+1

n

. |aii |−p Ap+1 i

i=1,i =k

Proof. Fixing x ∈ Rn+ , we may compute that Ip (x) ≤

n #

aii xp+1 + i

i=1

n

n $ p := a+ x x ϕi (xi ), ∞ ij i

(8)

i=1

j =i

where ϕi (t) = aii tp+1 + Ai tp x∞ , i = 1, · · · , n. By condition (6), we have ϕi (x∞ ) = (aii + Ai )xp+1 ∞ < 0,

i = 1, · · · , n.

Therefore, it is easy to get that max 0≤t≤ x ∞

ϕi (t) =

pp |aii |−p Ap+1 xp+1 ∞ , i (p + 1)p+1

i = 1, · · · , n.

On the other hand, by the definite of x∞ , there must exist some k such that xk = x∞ . Hence, by (8), we have that Ip (x) ≤ ϕk (x∞ ) +

n

ϕi (xi )

i=1,i =k

≤ max

1≤k≤n

= max

1≤k≤n

n , ϕk (x∞ ) + i=1,i =k

, akk + Ak +

max

0≤t≤ x ∞

pp (p + 1)p+1

ϕi (t)

n

xp+1 |aii |−p Ap+1 ∞ i

i=1,i =k

as required.

3 Positive Global Solutions It is well known that, in order for a stochastic differential equation to have a unique global solution for any given initial data, the coefficients of the equation are generally required to satisfy the linear growth condition and the local Lipschitz condition (cf. Mao [15]). However, the coefficients of (5)

A Stochastic Lotka-Volterra Model with Variable Delay

95

do not satisfy the linear growth condition, though they are locally Lipschitz continuous, so the solution of (5) may explode at a finite time (cf. Mao [16], Wu [17]). In this section, we shall show that the solutions of Eq. (3) is not only positive but will also not explode to infinite at any finite time. Theorem 2. Assume that parameter matrix A, B of Eq. (5) satisfy that ¯i ). max (aii + Ai ) ≤ − max (nB

1≤i≤n

(9)

1≤i≤n

Then, for any given initial data x0 = ξ ∈ C([−τ, 0]; Rn+ ), there is a unique solution x(t) to Eq. (5) on t ≥ −τ . Moreover, this solution remains in Rn+ with probability 1, namely x(t) ∈ Rn+ for all t ≥ −τ almost surely. Proof. Since the coefficients of Eq. (5) are locally Lipschitz continuous, for any given positive initial data ξ ∈ C([−τ, 0]; Rn+ ), there is a unique maximal local solution x(t) on t ∈ [−τ, τe ), where τe is the explosion time (cf. [16] Theorem 3.2.2, p. 95). To show this solution is global, we only need to prove that τe = ∞ a.s. Let k0 > 0 be sufficiently large in the sense k0−1 <

min |ξ(θ)| ≤ max |ξ(θ)| < k0 .

−τ ≤θ≤0

−τ ≤θ≤0

For each integer k ≥ k0 , define the stopping time τk = inf{t ∈ [−τ, τe ) : xi (t) ∈ / (k −1 , k) for some i = 1, 2, · · · , n} with usual setting inf ∅ = ∞,where ∅ denotes the empty set. Clearly, τk is increasing as k → ∞. Set τ∞ = limk→∞ τk , whence τ∞ ≤ τe a.s. If we can prove τ∞ = ∞ a.s., then τe = ∞ a.s., which implies the desired result. To prove this statement, for any p > 0, let us define a C 2 -function V : Rn+ → R+ by V (x) =

n

u(xi ),

(10)

i=1

where u(xi ) = xpi − 1 − p log(xi ). Clearly, u(·) ≥ 0 and u(0+ ) = u(∞) = ∞. Let T > 0 be arbitrary. For 0 ≤ t ≤ τk ∧ T , applying the Itˆo formula to V (x(t)) and taking expectation yield !

t

EV (x(t)) = EV (x(t)) + E

LV (x(t), y(t))dt, 0

where LV : Rn+ × Rn+ → R is defined by LV (x, y) =p

n i=1

n n n , n n , xpi bi + aij xj + bij yj − p aij xj + bij yj bi + j=1

j=1

n n p(p − 1) 2 p p 2 σi x i + σ . + 2 2 i=1 i i=1

i=1

j=1

j=1

(11)

96

Y. Xu, S. Zhu, and S. Hu

By the condition (9) and the Lemma 1, we have that n n

aij xpi xj ≤ −Kp xp+1 ∞ ,

i=1 j=1

where −Kp = max

1≤k≤n

, akk + Ak +

pp (p + 1)p+1

n

. |aii |−p Ap+1 i

i=1,i =k

Noting that Ak < |akk | for k = 1, · · · , n, we have K∞ := lim Kp = − max (akk + Ak ) > 0. p→∞

1≤k≤n

So we may choose sufficiently large p such that Kp > 0. Clearly, nxp+1 ∞ . We therefore have n n

aij xpi xj

≤ −n

−1

Kp

i=1 j=1

n

n i=1

xp+1 ≤ i

xp+1 . i

(12)

i=1

By the Young inequality, we may compute that n n

bij xpi yj ≤

i=1 j=1

n n

b+ ij

i=1 j=1

, p 1 p+1 xp+1 y + p+1 i p+1 j

1 , ¯ ¯ xp+1 pBi + η −1 B = i i p + 1 i=1 n

1 + , p+1 . bij yj − η −1 xp+1 j p + 1 i=1 j=1 n

n

+

Substituting (12) and (13) into (11) gives LV (x, y) ≤

n i=1

+p

p + , p+1 bij yj − η −1 xp+1 j p + 1 i=1 j=1 n

Fi (x) + n

n

, |bji | yi − η −1 xi ,

j=1

where , ¯ - p+1 ¯i + η −1 B   pB i xi + p bi + (p − 1)σi2 /2 xpi Fi (x) := − p n−1 Kp − p+1 + p(η −1 |bji | − aji )xi + p(σi2 /2 − bi ).

(13)

A Stochastic Lotka-Volterra Model with Variable Delay

97

By condition (9), we have , ¯i ¯i + η −1 B pB ¯i > 0, lim n−1 Kp − = n−1 max (akk + Ak ) − B p→∞ 1≤k≤n p+1

i = 1, · · · , n.

Therefore we may choose sufficiently large p such that ¯

¯i + η −1 B pB i > 0, p+1

n−1 Kp −

i = 1, · · · , n,

which implies Fi (x) is bounded above by a positive constant H for any x ∈ Rn+ . On the other hand, for any θ > 0, we may compute that !

!

t

0

≤η

−1

≤ η −1

!

t

xθi (s − δi (s))ds =

yiθ (s)ds = 0

!

t

−τ ! t

xθi (r)dr

=η

−1

t−δi (t)

δi (0)

!

t

xθi (r)dr

+η

−1

!

xθi (r) dr (1 − δi (s)) 0

−τ

0

ξiθ (r)dr

xθi (r)dr + η −1 τ ξθ .

(14)

0

Therefore, we have p + bij E p+1 n

EV (x(t)) ≤EV (ξ(0)) + nHt + +p

n n i=1 j=1

! |bij |E

n

i=1 j=1

0 −τ

≤EV (ξ(0)) + nHt + η

!

0

−τ

η −1 xp+1 (s)ds j

η −1 xj (s)ds

−1

n , n b+ ij ξp+1 + p|bij |ξ τ p+1 i=1 j=1

= : Ht . Let t = τk ∧ T . We obtain that EV (x(τk ∧ T )) ≤ HT . By the definition of τk , xi (τk ) = k or 1/k for some i = 1, 2, . . . , n, P(τk ≤ T )[u(k −1 ) ∧ u(k)] ≤ P(τk ≤ T )V (x(τk ∧ T )) ≤ EV (x(τk ∧ T )) ≤ HT , which implies that lim sup P(τk ≤ T ) ≤ lim k→∞

k→∞

HT = 0. u(k −1 ) ∧ u(k)

Since T > 0 is arbitrary, we must have P(τ∞ < ∞) = 0 as required.

98

Y. Xu, S. Zhu, and S. Hu

4 Asymptotic Bound Properties In Section 3, we show that the solution of Eq. (5) is positive and will not explode in any finite time. This nice positive property allows us to further discuss various asymptotic properties for the solution of Eq. (5). Comparison with nonexplosion of the solution, stochastically ultimate boundedness is more interesting from the biological point of view. To discuss stochastically ultimate boundedness, we first examine the pth moment boundedness, which is also interesting. Theorem 3. Let condition (9) hold and p > 0. Then there is a positive constant Kp , which is independent of the initial data x0 = ξ ∈ C([−τ, 0]; Rn+ ), such that the solution x(t) of Eq. (5) has the property that lim sup E|x(t)|p ≤ Hp .

(15)

t→∞

n Proof. By Theorem 2 , the solution x(t) will remain n in pR+ for all t ≥ −τ . 2 n Define a C -function U : R+ → R+ by U (x) = i=1 xi . Applying the Itˆ o formula to et V (x(t)) and taking expectation yields

!

t

et EU (x(t)) = EU (ξ(0)) + E

, es LU (x(s), y(s)) + U (x(s)) ds,

(16)

0

where LV : Rn+ × Rn+ → R is defined by LU (x, y) = p

n i=1

xpi

n n n , - p(p − 1) bi + aij xj + bij yj + σi2 xpi . 2 j=1 j=1 i=1

Using the same computation (12) and (13) in the proof of Theorem 2.1, we have that p + , p+1 LU (x, y) ≤ , Gi (x) + bij yj − eτ η −1 xp+1 j p + 1 i=1 j=1 i=1 n

n

n

(17)

where # # ¯ $ p+1 ¯i + η −1 eτ B pB p − 1 2$ p i xi + p bi + σi xi Gi (x) := −p n−1 Kp − p+1 2 By condition (9), we have , ¯i + η −1 eτ B ¯i pB ¯k > 0, lim n−1 Kp − = n−1 max (akk + Ak ) − B p→∞ 1≤k≤n p+1

i = 1, · · · , n.

A Stochastic Lotka-Volterra Model with Variable Delay

99

Therefore, these exists p∗ > 0, for any p ≥ p∗ , such that n−1 Kp −

¯i + η −1 eτ B ¯

pB i > 0, p+1

i = 1, · · · , n,

which implies Gi (x) is bounded above by a positive constant H1 for any x ∈ Rn+ . On the other hand, we may compute that !

! t es yip+1 (s)ds ≤ eτ e(s−δi (s)) xp+1 (s − δi (s))ds i 0 0 ! t ! t τ −1 es xp+1 (s)ds ≤ e η es xp+1 (s)ds + eτ η −1 τ ξp+1 . ≤ eτ η −1 i i t

−τ

0

Therefore, we get !

t

et EU (x(t)) ≤ EU (ξ(0)) + nH1

es ds + 0

n n peτ (η)−1 τ ξp+1 + bij . p+1 i=1 j=1

This immediately implies that lim sup EU (x(t)) ≤ nH1 . t→∞

Recall the elementary inequality: for any xi > 0, α > 0, then α/2 |x|α ≤ nα/2 max xα i ≤n 1≤i≤n

n

xα i ,

(18)

i=1

where x = (x1 , · · · , xn )T . Therefore, for any p ≥ p∗ , we have lim sup E|x(t)|p ≤ Hp , t→∞

by setting Hp = n1+p/2 H1 . Recall the Liapunov inequality: 

E|x(t)|r

1/r

1/s  ≤ E(|x(t)|s ,

0 < r < s < ∞.

Then the assertion (15) follows for any p > 0. Then we introduce a simple corollary, by which the pth moment boundedness implies the stochastically ultimate boundedness. Corollary 4. For any p > 0, if the stochastic process x(t) is bounded in pth moment, i.e., lim supt→∞ E|x(t)|p ≤ Kp , where Kp is a constant dependent on p, then x(t) is ultimately bounded, namely, for any  ∈ (0, 1), there exists a constant M = M () such that lim sup P{|x(t)| ≤ M } ≥ 1 − . t→∞

(19)

100

Y. Xu, S. Zhu, and S. Hu 1/p

Proof. For any  ∈ (0, 1), let M = Kp /1/p . Then by the Chebyshev inequality, P{|x(t)| > M } ≤ Hence,

E|x(t)|p . Mp

lim sup P{|x(t)| ≤ M } ≥ 1 − ,

(20)

t→∞

as required.

References 1. Gopalsamy, K., He, X.: Oscillations and Convergence in an Almost Periodic Competition System. Acta. Appl. Math. 46, 247–266 (1997) 2. Huo, H., Li, W.: Periodic Solutions of a Periodic Lotka-Volterra System with Delays. Appl. Math. Comput. 156, 787–803 (2004) 3. Freedman, H.I., Wu, J.: Periodic Solutions of Single-species Models with Periodic Delay. SIAM J. Math. Anal. 23, 689–701 (1992) 4. Liu, S., Chen, L.: Profitless Delays for Extinction in Nonautonoumous LotkaVolterra System. Commun. Nonlinear Sci. Numer. Simul. 4, 210–216 (2001) 5. Li, Y., Kuang, Y.: Periodic Solutions of Periodic Delay Lotka-Volterra Equations and Systems. J. Math. Anal. Appl. 255, 260–280 (2001) 6. Teng, Z.: On the Positive Almost Periodic Solutions of a Class of Lotka-Volterra Type Systems with Delays. J. Math. Anal. Appl. 249, 433–444 (2000) 7. Teng, Z.: Nonautonomous Lotka CVolterra Systems with Delays. J. Differential Equations 179, 538–561 (2002) 8. Teng, Z., Chen, L.: Global Asymptotic Stability of Periodic Lotka-Volterra Systems with Delays. Nonlinear Anal. 45, 1081–1095 (2001) 9. Bahar, A., Mao, X.: Stochastic Delay Lotka-Volterra Model. J. Math. Anal. Appl. 292, 364–380 (2004) 10. Mao, X., Marion, G., Renshaw, E.: Environmental Noise Supresses Explosion in Population Dynamics. Stochastic Process. Appl. 97, 96–110 (2002) 11. Mao, X., Sabanis, S., Renshaw, E.: Asymptotic Behaviour of the Stochastic Lotka-Volterra Model. J. Math. Anal. Appl. 287, 141–156 (2003) 12. Bahar, A., Mao, X.: Stochastic Delay Population Dynamics. Int. J. Pure Appl. Math. 4, 377–400 (2004) 13. Rudnicki, R.: Long-time Behaviour of a Stochatic Prey-predator Model. Stochastic Process. Appl. 108, 93–107 (2003) 14. Rudnicki, R., Pichor, K.: Influence of Stochastic Perturbation on Prey-predator Systems. Math. Biosci. 206, 108–119 (2007) 15. Mao, X.: Stochastic Differntial Equations and Applications. Horwood, Chichester (1997) 16. Mao, X.: Exponential Stability of Stochastic Differential Equations. Dekker, New York (1994) 17. Wu, F., Hu, S.: Stochastic Functional Kolmogorov-type Population Dynamics. J. Math. Anal. Appl. 347, 534–549 (2008)

Extreme Reformulated Radial Basis Function Neural Networks Gexin Bi and Fang Dong

Abstract. Gradient descent based learning algorithms are generally very slow due to improper learning steps or may easily converge to local minima. And many iterative learning steps may be required by such learning algorithms in order to obtain better learning performance. This paper proposes a new learning algorithm for R-RBFNs which randomly chooses hidden nodes and analytically determines the output weights of R-RBFNs. The experimental results based on a few benchmark problems has shown that the proposed algorithm tends to provide better generalization performance at extremely fast learning speed. Keywords: Reformulated radial basis function neural network, Gradient descent based learning algorithm, Admissible radial basis function, Generator function.

1 Introduction Radial basis function neural networks (RBFNNs) are function approximation models that can be trained to implement a desired input-output mapping [1], [2]. The performance of RBFNNs depend on the number and centers of the radial basis functions, their shapes, and the methods used for learning the input-output mapping. Broomhead and Lowe [3] suggested that the centers of the radial basis functions can either be distributed uniformly within the region of the input space for which there is data, or chosen to be a subset of the training vectors by analogy with strict interpolation. Moody and Darken [4] proposed a hybrid learning process for training radial basis function neural networks with Gaussian radial basis functions, which employs a supervised scheme for updating the output weights,i.e., the weights that Gexin Bi · Fang Dong College of Navigation, Dalian Maritime University, 1 Linghai Road, 116026 Dalian, China [email protected],[email protected]

This work is supported by Application Fundamental Research Foundation of China Ministry of Communications under grant 200432922505.

H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 101–110. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

102

G. Bi and F. Dong

connect the radial basis functions with the output units, and an unsupervised clustering algorithm for determining the centers of the radial basis functions. Karayiannis et al., proposed a systematic approach for constructing reformulated radial basis function neural networks (R-RBFNs), which was developed to facilitate their training by supervised learning algorithms based on gradient descent learning algorithms. This approach reduces the development of reformulated radial basis function models to the selection of admissible generator functions. Experiments involving a variety of reformulated radial basis function models generated by linear and exponential generator functions [5] perform considerably better than conventional RBF models trained by existing algorithms [1], [6]. However, It is clear that the learning speed of gradient descent based learning algorithms is in general far slower than required and it has been a major bottleneck in their applications for past decades. Further the learning parameters (i.e., learning rate, number of learning epochs, stopping criteria, and other predefined parameters) must be properly chosen to ensure convergence [7]. Huang, et al [7], [8], [9] proved that single hidden layer feedforward neural networks (SLFNs) with randomly generated RBF nodes and a widespread of piecewise continuous activation functions can universally approximate any continuous target function on any compact subspace of the Euclidean space Rn [10]. After the hidden layer parameters are chosen randomly, SLFNs can be simply considered as a linear system and the output weights of SLFNs can be analytically determined through simple generalized inverse operation of the hidden layer output matrices [11], [12], [13]. In this paper, we present a extreme reformulated radial basis function neural networks (ER-RBFNNs) as a systematic and axiomatic approach for extremely designing R-RBFNNs. ER-RBFNNs provides an axiomatic approach for R-RBFNNs by using basis functions formed in terms of admissible generator functions. ERRBFNNs need not be tuned during training and may simply be assigned with random values. After the hidden layer parameters are chosen randomly, ER-RBFNNs can be simply considered as a linear system and the output weights of ER-RBFNNs can be analytically determined through simple generalized inverse operation of the hidden layer output matrices.

2 Construction OF R-RBFNs 2.1 Mathematical Description of RBF Models Consider the Rn → Rp mapping implemented by the model ˜ N f( ωi φ(x − vi )) = y.

(1)

i=1

where f (·) is a nondecreasing, continuous and differentiable function, x = [x1 , x2 , . . . , xn ]T ∈ Rn , y = [y1 , y2 , . . . , yp ]T ∈ Rp , vi = [v1 , v2 , . . . , vn ]T ∈ Rn

Extreme Reformulated Radial Basis Function Neural Networks

103

is the prototype of the radial basis function, ωi = [ωi1 , ωi2 , . . . , ωim ]T ∈ Rm is the weight vector connecting the ith hidden node and the input nodes, and φ(x) are radial basis functions.

2.2 Admissible Radial Basis Functions The interpretation of an RBF neural networks as a composition of receptive fields requires that the responses of all RBF’s to all inputs are always positive. If the prototypes are interpreted as the centers of receptive fields, it is required that the response of any RBF becomes stronger as the input approaches its corresponding prototype. And, it is required that the response of any RBF becomes more sensitive to an input vector as this input vector approaches its corresponding prototype. Finally, it is reasonable to require that the responses of all radial basis functions to all inputs be bounded [1], [6]. In order for the model (1) to satisfy the desired properties mentioned above, any admissible radial basis function φ(x) = g(x2 ) must satisfy the following three basic axiomatic requirements: A1) g(x − v2 ) > 0 for all x, v ∈ Rn ; A2) g(x−v2 ) > g(y−v2 ) for all x, y, v ∈ Rn such that x−v2 < y−v2 ; A3) If ∇x g = ∇x g(x − v2 ) enotes the gradient with respect to x of g(x− v2 ) at x, then ∇x g2 /x − v2 > ∇y g2 /y − v2 for all x, y, v ∈ Rn such that x − v2 < y − v2 ; A4) g(x − v2 ) > ∞ for all x, v ∈ Rn . The four basic axiomatic requirements impose some mathematical restrictions on the search for admissible radial basis functions that can lead to R-RBFNs. The selection of admissible radial basis functions in accordance with the four basic axiomatic requirements can be facilitated by the following theorem. Theorem 1: The model described by (1) represents an radial basis function neural network in accordance with all four axiomatic requirements if g(x) is infinitely differentiable on (0, ∞) such that: 1) g(x) > 0, ∀x ∈ (0, ∞); 2) g (x) < 0, ∀x ∈ (0, ∞); 3) g

(x) > 0, ∀x ∈ (0, ∞); 4) limx→0+ g(x) = L, where L are finite numbers. Theorem 1 requires that g(x) is infinitely differentiable on (0, ∞). So theorem 1 is more restrictive than Karayiannis et al.’s theorem . A radial basis function is said to be admissible in the wide sense if it satisfies the three basic axiomatic requirements, that is, the first three conditions of Theorem 1. If a radial basis function satisfies all four axiomatic requirements, that is, all four conditions of Theorem 1, then it is said to be admissible in the strict sense.

104

G. Bi and F. Dong

2.3 Admissibility Generator Functions The search for admissible radial basis functions can be simplified by considering basis functions of the form φ(x) = g(x2 ) ,with each g(x) defined in terms of a generator function g0 (x) as g(x) = (g0 (x))1/1−m , m = 1, can be facilitated by the following theorem [1], [6]: Theorem 2: Consider the model (1) and let each g(x) be defined as g(x) = (g0 (x))1/1−m , m = 1, where g0 (x) is a generator function that is continuous on (0, ∞) and has continuous first- and second-order derivatives. If m > 1, then this model represents an radial basis function neural network in accordance with all four axiomatic requirements if: 1) g0 (x) > 0, ∀x ∈ (0, ∞); 2) g0 (x) > 0, ∀x ∈ (0, ∞); 3) r0 (x) = [m/m − 1](g0 (x))2 − g0 (x)g0

(x) > 0, ∀x ∈ (0, ∞); 4) limx→0+ g0 (x) = L1 , where L1 ∈ (0, ∞). If m < 1, then this model represents an radial basis function neural network in accordance with all four axiomatic requirements if: 1) g0 (x) > 0, ∀x ∈ (0, ∞); 2) g0 (x) < 0, ∀x ∈ (0, ∞); 3) r0 (x) = [m/m − 1](g0 (x))2 − g0 (x)g0

(x) < 0, ∀x ∈ (0, ∞); 4) limx→0+ g0 (x) = L2 , where L2 ∈ (0, ∞). Any generator function that satisfies the first three conditions of Theorem 2 leads to admissible radial basis functions in the wide sense [1], [6]. Admissible radial basis functions in the strict sense can be obtained from generator functions that satisfy all four conditions of Theorem 2.

2.4 Constructing Admissible Generator Functions Theorem 2 essentially reduces the construction of admissible radial basis function models to the search for admissible generator functions. A broad variety of admissible generator functions can be determined by a constructive approach based on the admissibility conditions of Theorem 2 [1], [6]. The construction of wide-sense admissible generator functions can be attempted by assuming that g0 (x) = p(g(x)), where p(·) satisfies certain conditions in accordance with Theorem 2. Theorem 2 requires that g0 (x) > 0, ∀x ∈ (0, ∞), for m > 1 and g0 (x) < 0, ∀x ∈ (0, ∞), for m < 1. Since it is also required by Theorem 2 that g0 (x) > 0, ∀x ∈ (0, ∞), the function p(x) must be selected so that p(x) > 0, ∀x ∈ (0, ∞), if m > 1 and p(x) < 0, ∀x ∈ (0, ∞), if m < 1. For such functions, the admissibility conditions of Theorem 2 are satisfied by all solutions g0 (x) > 0, ∀x ∈ (0, ∞), of the differential equation g0 (x) = p(g0 (x)) that satisfy the conditions r0 (x) > 0, ∀x ∈ (0, ∞), for m > 1 and r0 (x) < 0, ∀x ∈ (0, ∞), for m < 1.

Extreme Reformulated Radial Basis Function Neural Networks

105

Generator functions admissible in the strict sense can be obtained by determining the subset of the resulting wide-sense admissible generator functions that satisfy the fourth condition of Theorem 2. The constructive approach outlined above is employed here to produce increasing generator functions that can be used for m > 1. The same constructive approach can be extended to produce decreasing generator functions that can be used for m < 1. Assume that m > 1 and let the function p(x) be of the form p(x) = kxn , k > 0. The function g0 (x) can be obtained in this case by solving the differential equation g0 (x) = k(g0 (x))n , k > 0. According to (2),

g0

(x)

=

kn(g0 (x))n−1 g0 (x)

r0 (x) = (g0 (x))2 (

2

(2) 2n−1

= k n(g0 (x))

m − n). m−1

. In this case (3)

If m > 1, it is required that r0 (x) > 0, ∀x ∈ (0, ∞), which holds for all m/(m − 1) > n. For m > 1, m/(m − 1) > 1 and the inequality m/(m − 1) > n holds for all n < 1. For n = 1, m/(m − 1) − n = 1/(m − 1) > 0. Thus, the condition r0 (x) > 0, ∀x ∈ (0, ∞), is satisfied for all n ≤ 1. For n = 1, p(x) = kx and the solutions of (2) are g0 (x) = θexp(βx), where θ > 0 and β = k/θ > 0. These generator functions also satisfy the fourth axiomatic requirement since limx→0+ g0 (x) = θ > 0. The exponential generator functions g0 (x) = exp(βx), β > 0, obtained for θ = 1 correspond to g(x) = exp(βx/(1 − m)), which lead to Gaussian radial basis functions φ(x) = g(x2 ) = exp(−x2 /σ 2 ), with σ 2 = (m − 1)/β. For n < 1, the solution of (2) are of the form g(x) = (ax + b)1/(1−n) , where a = k(1 − n) > 0 and b ≥ 0. For n = 0, p(x) = and (2) leads to linear generator functions g0 (x) = ax + b, a > 0, b ≥ 0. For g0 (x) = ax + b, the fourth axiomatic requirement is satisfied only if limx→0+ g0 (x) = b > 0. Thus, the fourth axiomatic requirement excludes generator functions of the form g0 (x) = ax. Linear generator function produce radial basis functions of the form φ(x) = g(x2 ) = (ax2 + b)1/(1−m) , with m > 1. If a = 1 and b = r2 , then the linear generator function g0 (x) = ax + b becomes g0 (x) = x + γ 2 and g(x) = (x + γ 2 )1/(1−m) . If m = 3, g(x) = (x + γ 2 )−1/2 corresponds to the inverse multiquadratic radial basis function φ(x) = g(x2 ) = 1/(x2 + γ 2 )1/2 . Another useful generator function for practical applications can be obtained from g0 (x) = ax + b by selecting b = 1 and a = δ > 0. For g0 (x) = 1 + δx, limx→0+ g(x) = limx→0+ g0 (x) = 1. For this choice of parameters, the corresponding radial basis function φ(x) = g(x2 ) is bounded by 1, which is also the bound of the Gaussian radial basis function.

3 ER-RBFNNs For N arbitrary distinct samples (xk , tk ), where xk = [xk1 , xk2 , . . . , xkn ]T ∈ Rn ˜ hidden nodes and admissiand tk = [tk1 , tk2 , . . . , tkp ]T ∈ Rp , R-RBFNNs with N ble RBF φ(x) are mathematically modeled as

106

G. Bi and F. Dong ˜ N

1

ωi (g0 (xk − vi 2 )) 1−m = yk , k = 1, . . . , N.

(4)

i=1

˜ hidden nodes with activation function φ(x) can approximate R-RBFNNs with N N˜ these N samples with zero error means that i=1 yk − tk  = 0, i.e., there exist ωi , vi such that ˜ N

1

ωi g0 ((xk − vi 2 )) 1−m = tk , k = 1, . . . , N.

(5)

i=1

The above N equations can be written compactly as Hω = T,

(6)

where HN ×N˜ = ⎞ ⎛ 1 1 g0 (x1 − v1 2 ) 1−m · · · g0 (x1 − vN˜ 2 ) 1−m ⎟ ⎜ .. .. ⎠ ⎝ . ··· . 1 1 2 1−m 2 1−m g0 (xN − v1  ) · · · g0 (xN − vN˜  ) ⎛ T⎞ ⎛ T⎞ ω1 t1 ⎜ .. ⎟ ⎜ .. ⎟ and T = ⎝ . ⎠ . ω=⎝ . ⎠ T T ωN˜ N˜ ×m tN N ×m

(7)

(8)

H is called the hidden layer output matrix of the R-RBFNs; the ith hidden node output with respect to inputs x1 , x2 , . . . , xN . Huang et al. proved that for SLFNs with additive or RBF hidden nodes, one may randomly choose and fix the hidden node parameters and then analytically determine the output weights when approximating any continuous target function. Note that the proof of universal approximation is shown to be valid for R-RBFNs with general type of hidden nodes generated by linear and exponential generator functions. In this regard, for N arbitrary distinct samples (xk , tk ), in order to obtain arbi˜ (≤ N ) hidden trarily small non-zero training error, one may randomly generate N nodes generated by linear and exponential generator functions with random parameters. Eq. (4) then becomes a linear system and the output weights ω are estimated as ω ˆ = H † T, (9) where H † is the Moore-Penrose generalized inverse of the hidden layer output matrix H. Calculation of the output weights is done in a single step here. Thus this avoids any lengthy training procedure where the network parameters are adjusted iteratively with appropriately chosen control parameters (learning rate and learning epochs, etc.).

Extreme Reformulated Radial Basis Function Neural Networks

107

4 Performance Evaluation In this section, the performance of the proposed ER-RBFNNs is evaluated on seven different benchmark regression data sets, which are summarized in table 1. All the simulations have been conducted in MATLAB 7.1 environment running on ordinary PC with 2.0 GHZ CPU and 512M RAM. In our simulations, the input and output attributes of regression applications are normalized into the range [-1,1]. The prototypes representing the locations of the RBF’s in the input space are randomly chosen from the range [-1,1]. There are 20 hidden nodes assigned for our ER-RBFNNs algorithm . The experimental results reported here are based on the average of 20 independent trials. Table 1 Specification of real-world regression cases Data sets Training data Testing data Attributes Ailerons 7154 6596 40 bank domains 4500 3692 32 Delta ailerons 3000 4129 5 Delta elevators 4000 5517 6 Elevators 8752 7846 18 Kinematics 4000 4192 8 Puma32H 4500 3692 32

4.1 Comparison with Different Value of Linear Generator Functions We first compare the generalization performance of ER-RBFNNs generated by linear generator functions g0 (x) = x + γ 2 , m = 3 and γ = 0, γ = 0.1, γ = 1. The 1 radial basis functions of the form φ(x) = g(x2 ),with g(x) = (g0 (x)) 1−m . It is clear from Table 2 that the best performance of ER-RBFNNs on the testing root mean square error (RMSE) was achieved for γ = 0. However, the best performance of standard deviaton (Dev) on the testing set was achieved for γ = 1. As observed from table 2, the learning time of ER-RBFNNs is very stable on a wide range of value of linear generator functions. According to table 2, the learning phase of ERRBFNNs can be completed in seconds or less than seconds.

4.2 Comparison with Different Value of Exponential Generator Functions Next, we compare the generalization performance of ER-RBFNNs generated by exponential generator functions g0 (x) = exp(βx), m = 3 and β = 0.5, β = 1, β = 5. It is clear from Table 3 that the best performance of ER-RBFNNs on the testing root mean square error (RMSE) and exponential were achieved for β = 0.5, ER-RBFNNs algorithm run faster than R-RBFNNs algorithm. As observed from

108

G. Bi and F. Dong

Table 2 Comparison with different value of linear generator functions Data sets Ailerons

γ

0.0 0.1 1.0 bank 0.0 domains 0.1 1.0 Delta 0.0 ailerons 0.1 1.0 Delta 0.0 elevators 0.1 1.0 Elevators 0.0 0.1 1.0 Kinematics 0.0 0.1 1.0 Puma32H 0.0 0.1 1.0

Testing Mean Dev. 1.7286e-004 6.9348e-006 1.7397e-004 5.3345e-006 1.7305e-004 4.7740e-006 0.1132 0.0038 0.1139 0.0033 0.1137 0.0027 4.5021e-004 4.2689e-005 4.5627e-004 4.0186e-005 4.6176e-004 2.7356e-005 0.0015 1.4005e-005 0.0015 2.1459e-005 0.0015 8.2240e-006 0.0031 1.5747e-004 0.0032 2.2819e-004 0.0031 1.4728e-004 0.1967 0.0072 0.2039 0.0074 0.2079 0.0054 0.0282 3.7659e-004 0.0284 3.5732e-004 0.0285 2.8951e-004

Training time(s) 0.4609 0.4261 0.4500 0.4461 0.4727 0.4877 0.0711 0.0727 0.0711 0.1000 0.1008 0.1047 0.3586 0.3570 0.3438 0.1078 0.1187 0.1078 0.2383 0.2305 0.2445

table 3, the performance of ER-RBFNNs improved as the value of β decreased from 5 to 0.5. The networks trained with β = 5, did not achieve satisfactory performance. This experiments indicate that there exists a certain range of values of β that guarantee satisfactory performance. This determines the responses of the RBF’s and plays a critical role in the ability of the network to implement the desired input-output mapping.

4.3 Comparison between Linear and Exponential Generator Functions with Random Value In this section, we compare the generalization performance of ER-RBFNNs generated by linear generator function g0 (x) = x + γ 2 and exponential generator functions g0 (x) = exp(βx) with m = 3. Each of these networks was tested with random value chosen from the range [-1,1]. In order to comparison to be fair, the same set of randomly selected prototypes were used in each trials. Table 4 shows the testing root mean square error (RMSE) ,standard deviaton (Dev) and training time in 20 trials. The experiments verified that training speed is one of the major advantages of ER-RBFNNs. According to table 4, the root mean square error produced on the average in 20 trials by ER-RBFNNs were not significantly affeced by different type of generator function with random value. Compared with ER-RBFNNs generated

Extreme Reformulated Radial Basis Function Neural Networks

109

Table 3 Comparison with different value of exponential generator functions Data sets

β

Ailerons

0.5 1.0 5.0 bank 0.5 domains 1.0 5.0 Delta 0.5 ailerons 1.0 5.0 Delta 0.5 elevators 1.0 5.0 Elevators 0.5 1.0 5.0 Kinematics 0.5 1.0 5.0 Puma32H 0.5 1.0 5.0

Testing Mean Dev. 3.0253e-004 3.1286e-005 5.6511e-004 5.4372e-005 0.0031 0.0078 0.1195 0.0016 0.1187 0.0015 4.3878 15.3158 4.4930e-004 1.0320e-005 4.6034e-004 2.1009e-005 4.6053e-004 8.4129e-005 0.0014 1.0338e-005 0.0015 3.3189e-006 0.0017 8.3193e-005 0.0038 3.1361e-004 0.0057 0.0010 52.9946 199.1462 0.2181 0.0053 0.2774 0.0091 0.6623 0.0151 0.0291 1.3406e-004 0.0300 2.4612e-004 0.2458 0.3968

Training time(s) 0.5016 0.5070 0.4781 0.5338 0.2703 0.2609 0.0813 0.0914 0.0836 0.1164 0.1187 0.1211 0.3844 0.3984 0.3742 0.1344 0.1281 0.1227 0.2555 0.2531 0.2539

by liner generator functions, ER-RBFNNs generated by exponential generator functions produced a slightly lower standard deviaton on the testing set. Table 4 Comparison between linear and exponential generator functions with random value Data sets Mode Ailerons linear expo bank linear domains expo Delta linear ailerons expo Delta linear elevators expo Elevators linear expo Kinema linear -tics expo Puma linear -32H expo

Testing Mean Dev. 1.7168e-004 4.1466e-006 2.1784e-004 2.0198e-005 0.1133 0.0030 0.1175 0.0016 4.5690e-004 3.7404e-005 4.3316e-004 2.2301e-005 0.0015 4.0118e-006 0.0014 1.6391e-005 0.0031 1.9121e-004 0.0035 3.1569e-004 0.2025 0.0070 0.1922 0.0057 0.0284 2.7302e-004 0.0293 1.9312e-004

Training time(s) 0.8542 0.5094 0.4852 0.2664 0.0656 0.0766 0.1055 0.1148 0.3555 0.3906 0.1102 0.1242 0.2500 0.2648

110

G. Bi and F. Dong

5 Conclusion In this paper, a fast and axiomatic approach algorithm (ER-RBFNNs) has been developed for R-RBFNNs generated by linear and exponential generator functions. The parameters of linear and exponential generator function can be randomly choosed and analytically determines the output weights of ER-RBFNNs. The learning speed of ER-RBFNNs is extremely fast.

References 1. Karayiannis, N.B., Randolph-Gips, M.M.: On the Construction and Training of Reformulated Radial Basis Function Neural Networks. IEEE Trans. Neur. Netw. 14, 835–846 (2003) 2. Micchelli, C.A.: Interpolation of Scattered Data: Distance Matrices and Conditionally Positive Definite Functions. Constr. Approx. 2, 11–22 (1986) 3. Broomhead, D.S., Lowe, D.: Multivariable Functional Interpolation and Adaptive Networks. Comp. Sys. 2, 321–355 (1988) 4. Moody, J.E., Darken, C.J.: Fast Learning in Networks of Locally-Tuned Processing Units. Neur. Comput. 1, 281–294 (1989) 5. Chen, T., Chen, H.: Approximation Capability to Functions of Several Variables, Nonlinear Functionals, and Operators by Radial Basis Function Neural Networks. IEEE Trans. Neur. Netw. 6, 904–910 (1995) 6. Karayiannis, N.B.: Reformulated Radial Basis Neural Networks Trained by Gradient Descent. IEEE Trans. Neur. Netw. 10, 657–671 (1999) 7. Liang, N.Y., Huang, G.B., Saratchandran, P., Sundararajan, N.: A Fast and Accurate Online Sequential Learning Algorithm for Feedforward Networks. IEEE Trans. Neur. Netw. 17, 1411–1423 (2006) 8. Huang, G.B., Zhu, Q.Y., Siew, C.K.: Extreme Learning Machine: Theory and Applications. Neur. Comput. 70, 489–501 (2006) 9. Huang, G.B., Liang, N.Y., Rong, H.J., Saratchandran, P., Sundararajan, N.: On-Line Sequential Extreme Learning Machine. In: Proc. IASTED International Conference on Computational Intelligence, Calgary, Canada (2005) 10. Huang, G.B., Chen, L., Siew, C.K.: Universal Approximation Using Incremental Constructive Feedforward Networks with Random Hidden Nodes. IEEE Trans. Neur. Netw. 17, 879–892 (2006) 11. Huang, G.B., Zhu, Q.Y., Siew, C.K.: Extreme Learning Machine: Theory and Applications. Neur. Computing 70, 489–501 (2006) 12. Huang, G.B., Zhu, Q.Y., Siew, C.K.: Extreme Learning Machine: A New Learning Scheme of Feedforward Neural Networks. In: Proceedings of International Joint Conference on Neural Networks, pp. 985–990. IEEE Press, New York (2004) 13. Huang, G.B., Siew, C.K.: Extreme Learning Machine with Randomly Assigned RBF Kernels. Inter. J. Inform. Tech. 11, 16–24 (2005) 14. Chen, S., Cowan, C., Grant, P.: Orthogonal Least Squares Learning Algorithm for Radial Basis Function Networks. IEEE Trans. Neur. Netw. 2, 302–309 (1991)

Research of Nonlinear Combination Forecasting Model for Insulators ESDD Based on Wavelet Neural Network* *

Haiyan Shuai, Jun Wu, and Qingwu Gong

Abstract. Wavelet neural network (WNN) effectively overcomes the intrinsic shortcomings of artificial neural network, namely, slow learning speed, difficult determination of structure and existence of local minima, combination forecasting model fully integrates the information of each model, and nonlinear one effectually conquers the difficulties and drawbacks in combined modeling non-stationary time serial by using linear model , hence, nonlinear combination forecasting model based on WNN possesses more flexible structure, higher data fitting and forecasting accuracy. Simulation experiments show, compared with other forecasting models, the predicted results are closer to the actual ones which show the nonlinear combination forecasting model for insulators ESDD based on WNN can efficaciously improve the speed and accuracy of the forecasting. Therefore, the method presented provides a doable thought for the computerization of pollution area map of power network. Keywords: Wavelet neural network(WNN), Equal salt deposit density(ESDD), Nonlinear combination forecasting, Insulators.

1 Introduction Exposed to dirty environment, surfaces of insulators will be polluted. After being wetted, the polluted layers will deduce insulators’ insulation capability and often invite pollution flashover. According to the operation experiences of power Haiyan Shuai . Qingwu Gong School of Electrical Engineering of Wuhan University, Wuhan 430074, China *

Haiyan Shuai Wuhan Technical College of Communications, Wuhan 430074, China Jun Wu Three Gorges Vocational College of Electric Power,Yichang 443000, China [email protected] *

Project Supported by Ministry of Science and Technology of China (NCSIE-2006-JKZX-174).

H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 111–119. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

112

H. Shuai, J. Wu, and Q. Gong

sectors, pollution flashover is one of the main factors causing power accidents. In recent years, there have happened several large area pollution flashovers in some locals or the whole country, which resulted in large area power blackout. For example, in the early of 2001, the large area flashovers happened in Northeast China, North China and Central China respectively l. Among all the causes resulting in pollution flashover, the lag and the inaccuracy of pollution area map of power network is a primary one. Equal Salt Deposit Density (ESDD) is the equivalent amount of NaCl that would yield the same conductivity at complete dilution [1]. ESDD is a main factor to classify contamination severity and map pollution areas. Therefore, forecasting ESDD plays an important role in the safety, economy and reliability of power system. At present, the methods of ESDD forecasting are mainly traditional multivariate regression analysis [2-3], artificial neural network [4-6], support vector machines [7], et al.. By far, there is no report about using combination forecasting model to predict ESDD at home and abroad. So-called combination forecasting method is the way to use two or more than two different forecasting methods to make predictions for an object respectively, carry out a proper combination of each single prediction result , and regard the combination as the final forecasting result [8]. Combination forecasting integrates the useful information of all single forecasting models and generally considers each forecasting result; hence it can more systemically and comprehensively reflect the changes of an object than a single model. Bates J.M and Granger C.J.W. [9] proved the combination of two or more than two agonic single forecasting models can produce the results better than those of each single model, which showed combination forecasting method can increase prediction accuracy. According to the different ways of combination of each single forecasting model, combination forecasting models can generally be divided into two classes: linear one and nonlinear one. Most of former combination forecasting methods belonged to linear ones. Although linear combination forecasting is relatively simple, there exist some limitations in it. Reference [10] points out that the value of a linear combination forecasting is merely a convex combination of the values of different single forecasting models. In coordinate system, when the actual values of an object being predicted are over, under or intersecting the values of the two different forecasting models, linear method tends to be disable. In order to conquer these limitations, reference[11] puts forward a nonlinear combination forecasting method, namely, supposing a set of the actual values of an object being predicted are y ( d ) ( d = 1, 2, , D ) ,then using L forecasting methods to get L values of the object: z (i ) (i = 1, 2, , L) ,and finally utilizing the L values to construct a nonlinear combination function: ∧

y = ϕ ( Z ) = ϕ ( z (1), z (2), Where,

ϕ

, z ( L))

(1)

is the nonlinear function. According to the nonlinear combination ∧

forecasting theory, under a certain measure, the metric of

y will be better than those

Research of Nonlinear Combination Forecasting Model

113

of z (i ) (i = 1, 2, , L) . References [11-12] uses BP neural network to construct nonlinear combination functions respectively and all get relatively good results. However, the researches in recent years show that, although BP network has very strong ability to self-study and nonlinearly map, but in the view of function expression, the network is an inferior one because there does not exist a unified and integrated theory to instruct the selection of BP structure which is selected depending on experiences, leading to over-fitting and low generalization, especially under small samples. Besides, BP algorithm belonging to a local optimization method also suffers from many local minima, which will influence the reliability and veracity of nonlinear combination forecasting model. Wavelet neural network (WNN) [13] is a mathematical modeling method which is developed based on the combination of wavelet transform [14] with artificial neural network. By virtue of wavelet outstanding performances on analyzing of non-fixed signals and constructing of nonlinear function models, WNN which has been combined with wavelet basis function has more advantages than traditional neural network, and been effectively used in signal processing ,data compression and fault diagnosis ,et al.. The paper applies the nonlinear combination forecasting model based on WNN to carry out a study on insulators’ ESDD prediction, and compares the ESDD values of the model with those of BP nonlinear combination model, WNN single model and BP single model. The experiments show that the model being discussed, compared to other models, has intelligent combined-modeling process, more flexible structure, broader applicability, higher accuracy of data fitting and forecasting, et al..

2 Nonlinear Combination Forecasting Model for ESDD Based on WNN In fact, the nature of wavelet transform [15] is an integral transformation between different parameters.

wab ( a, b ) =

+∞

∫

f ( x) g (a, b, x)dt

(2)

−∞

Where, g ( a, b, x) = mother wavelet.

a

1 x−b g( ) is called wavelet basis and g ( x) called a |a|

、 b are termed as scaling factor and translation factor of

g (a, b, x) respectively. To f ( x ) , the resolution of its local structure can be realized through adjusting wavelet basis window.

a

、 b ,namely, gearing the scale and position of

114

H. Shuai, J. Wu, and Q. Gong

WNN is a model which, based upon wavelet analysis, possesses neural network thread. In other words, WNN adopts nonlinear wavelet basis to replace common-used Sigmoid function in traditional neural network. Through linear superposition of nonlinear wavelet basis selected, the combination of the ESDD data of all single forecasting models is realized. The nonlinear combination function in Equation. (1) can be fitted as follow, by adopting wavelet basis g ( a, b, x ) : L

∧

m

ϕ ( Z ) = ∑ ωk g ( k =1

∑v j =1

jk

z ( j ) − bk

(3)

)

ak

∧

Where ϕ ( Z ) is the ESDD values of nonlinear combination forecasting corresponding to the actual ones y ( d ) ; z ( j ) expresses the prediction value of the v jk bk ak are the weighted No.j model, Z = ( z (1), z (2),...z ( L )) ; ωk coefficient between output terminal and the No.k hidden layer node, the weighted one between the No.j input and the No.k hidden layer node, the translation factor and scaling factor of the No.k wavelet basis respectively; m is the number of wavelet basis (the number of hidden layer nodes ). Considering Morlet wavelet possessing relatively good localization and smoothness, it will be selected in Equation (3).

、 、 、

g ( x) = cos(1.75 x) exp(− Fig.1

shows

the

structure

of

WNN. ∧

x2 ) 2

There

(4) are

L input nodes-

z (1), z (2),...z ( L) and one output node- ϕ ( Z ) .

Fig. 1 The structure of WNN

、 、 、 、

The objective of using WNN to carry out regression analysis is to determine the ∧ network parameters- ωk v jk bk and ak by which ϕ ( Z ) can be fitted optimally with y ( d ) . ωk v jk bk and ak can be optimized via Minimum Mean Square Error(MMSE) energy function as follow.

Research of Nonlinear Combination Forecasting Model L

D

m

1 E = ∑ [ ∑ ωk g ( 2 d =1 k =1

∑v j =1

jk

115

z ( j ) − bk ) − y (d )]2

ak

(5)

Where, D is the number of training samples and y ( d ) the No.d actual value. To gain the optimal ωk v jk bk and ak is to minimize Equation (5). In this article, gradient descent algorithm is used as WNN learning principle. The details are as followings:

、 、

1) Set the objective error function value; 2) Define input z ( j ) and corresponding output 3) Initialize network parameters;

y (d ) , and normalize them;

Table 1 Initial parameters of the WNN



4) Calculate the gradient of each parameter; L

Let

z * = ∑ v jk z j ( j ) j =1

， S = z a− b ，then the gradients of Equation（5）are *

k

k

respectively: D ∧ z * − bk ∂E = − ∑ [ϕ ( Z ) − y ( d )] g [ ] ∂ ωk ak t =1 D ∧ ∂E S2 S = − ∑ [ϕ ( Z ) − y ( d )]ω k [ − cos(1.75 S ) exp( − ) − ∂ v jk 2 ak t =1

1.75sin(1.75 S ) exp( −

S2 1 ) ]z ( j ) 2 ak

D ∧ S2 S ∂E = − ∑ [ϕ ( Z ) − y ( d )]ω k [cos(1.75 S ) exp( − ) + 2 ak ∂ bk t =1

1.75sin(1.75 S ) exp( −

S2 1 ) ] 2 ak

116

H. Shuai, J. Wu, and Q. Gong D ∧ ∂E S2 S2 + = − ∑ [ϕ ( Z ) − y ( d )]ω k [cos(1.75S ) exp( − ) 2 ak ∂a t =1 k

1.75sin(1.75 S ) exp( − 5) Introduce momentum factor

α

S2 S ) ] 2 ak

to amend each parameter;

ωk (t + 1) = ωk (t ) − η

∂E + α ω k (t ) ∂ωk

v jk (t + 1) = v jk (t ) − η

∂E + α v jk (t ) ∂v jk

bk (t + 1) = bk (t ) − η

∂E + α bk (t ) ∂bk

ak (t + 1) = ak (t ) − η

∂E + α ak (t ) ∂ak

Where, η is the learning ratio and α the momentum factor. 6) Compute current output of WNN: put current parameters into Equation (3) to get current output of the network; 7) Numerate error function value. When the error is less than the set one, the learning process is terminated; otherwise, turns to step 4). The basic principles of nonlinear combination forecasting model for insulators ESDD based on WNN are to regard the ESDD of each single model as the input of WNN and the corresponding actual ESDD as output, and then utilize sufficient samples to train the model in the interest of establishing a nonlinear mapping relationship between the ESDD of each single model and the actual one. When its precision is reached after many training and testing, the model can be used as an effective way to predict insulator’s ESDD.

3 Appraisal of the Prediction In order to evaluate the prediction effects of the combination model comprehensively, according to the practice and principle, the paper uses relative error δ and average relative error δ to evaluate the accuracy of the prediction.

yi − yˆi

δ=

δ =

1

yi

×100%

l

yi − yˆ i

i =1

yi

∑ l

× 100%

(6)

(7)

Research of Nonlinear Combination Forecasting Model

117

Where yi are actual values and yˆ i predicted ones. These values reflect comprehensively the prediction results. The smaller δ is, the better the generalization of the model and the corresponding parameters are.

4 Experimental Results On the basis of the model principles and modeling steps in the text, Matlab 7.0 is adopted to write the ESDD prediction programs based on WNN. In the experiment, the historical meteorological data and ESDD data in the ESDD monitoring spot of Qingshan District, Wuhan from April to June in 2005 are regarded as training set, and those from April to June in 2006 as forecasting set. Tab.2 shows the comparisons between WNN combination forecasting model and BP combination forecasting model, WNN single forecasting model, and BP single forecasting model.

Table 2 Comparison of forecasting results of ten ESDD Single foreca forecasting ng

Combination m ination forecasting mb ng Times



WNN ecasting g Actuall Forecasting 2  ˄PJFP ˅˄mg/cm ˅ 0.0268

0.0281

  0.0325

0.0341

  0.0346

0.0328



0.0376

0.0387



0.0230

0.0243

 0.0255 

0.0242

 0.0494 

0.0517



0.0422

0.0448

  0.0467

0.0450

  0.0474

0.0491

G (%)

WNN

BP Forecasting g G (%) %)˄mg/cm2˅ 4.85 4.92 5.20 2.93 5.65 5.10 4.66 6.16 3.64 3.59

4.670

G (%)

Forecasting i ˄mg/cm2˅

BP

G (%) %)

Forecasting ˄mg/cm2˅

G (%)

0.0252

5.97

0.0286

6.72

0.0249

7.09

0.0346

6.46

0.0354

6.77

0.0360

10.77

0.0367

6.07

0.0322

6.94

0.0372

8.00

0.0395

5.05

0.0397

5.99

0.0408

8.51

0.0213

7.39

0.0248

7.83

0.0211

8.26

0.0245

3.92

0.0239

6.27

0.0238

6.67

0.0526

6.48

0.0542

9.72

0.0534

8.10

0.0403

4.50

0.0452

7.11

0.0398

5.69

0.0451

3.43

0.0441

5.57

0.0443

5.14

0.0500

5.49

0.0509

7.38

0.0516

8.86

5.476

7.03

7.79

From Table 2, it can be seen that the maximum relative errors of WNN single model and BP one are 9.72% and 10.77% respectively, and the average relative errors of them 7.03% and 7.79% respectively. The maximum relative error and the average relative error of BP nonlinear combination model are 7.39% and 5.467% respectively; while those of WNN nonlinear combination model are 6.16% and 4.670% respectively. This indicates that the prediction algorithm based on WNN is stable and practical, and can improve forecasting precision. Using the ESDD predicted by WNN combination forecasting model to divide pollution area can greatly increase the accuracy of power area pollution map.

118

H. Shuai, J. Wu, and Q. Gong

5 Conclusions ESDD is a main factor to classify contamination severity and draw pollution areas map of power network, hence its accuracy directly influences the precision of pollution areas map, and further affects the insulation capability of power system. There are some conclusions according to the study. (1) By virtue of WNN forecasting model for ESDD conquering some intrinsic shortcomings of traditional BP network, such as low convergence, difficulty to select structure, over-fitting &low generalization, existence of local minima, et al., its forecasting accuracy is higher than that of BP network. (2) The nonlinear combination forecasting model for ESDD fully takes use of the raw data and the information of all single forecasting models which compensates the flaws of only using single model, therefore its prediction precision is higher than that of single model. (3) The model discussed above can effectively improve ESDD forecasting accuracy. Compared with other models, its values are closer to the actual ones. Using its predicting ESDD to divide pollution area will largely boost the accuracy of pollution area map of power network. The way provides a new thread for the computerization of drawing pollution area map of power network.

References 1. Su, Z.Y., Wu, G.Y.: Q/GDW152-2006 (4) (2006) 2. Almad, A. S., Ahmad, H., Salam, M. A.: Regression Technique for Prediction of Salt Contamination Severity on High Voltage Insulators. In: Annual Report Conference on Electrical Insulation and Dielectric Phenomena, Victoria BC, vol. 1, pp. 218–221 (2000) 3. Almad, A.S., Ahmad, H., Salam, M.A.: Prediction of Salt Contamination on High Voltage Insulators in Rainy Season Using Regression Technique. In: Proceedings of TENCON, vol. 3, pp. 24–27 (2000) 4. Almad, A.S., Ghosh, P.S., Aljunid, S.A.K.: Modeling of Various Meteorological Effects on Contamination Level for Suspension Type of High Voltage Insulators Using ANN. In: IEEE/PES Transmission and Distribution Conference and Exhibition 2002: Asia Pacific, Yokohama, Japan, pp. 1030–1035 (2002) 5. Almad, A.S., Ghosh, P.S., Aljunid, S.A.K.: Assessing of ESDD on High-voltage Insulators Using Artificial Neural Network. Electric Power System Research, 131–136 (2004) 6. Zhang, H., Wen, X.S., Ding, H.: Extrapolation of Insulator’s ESDD Based on Climate Factor with Artificial Neural Network. High Voltage Apparatus 39, 31–32 (2003) 7. Jiao, S.B., Liu, D., Zheng, G.: Forecasting the ESDD of Insulator Based on Least Squares Support Vector Machine. In: Proceedings of the CSEE, vol. 26, pp. 149–153 (2006) 8. Tang, X.W.: Research of Combination-forecasting Calculation Methods. Forecasting 10, 35–40 (1991) 9. Bate, J.M., Granger, C.W.J.: Combination of Forecasting. Operation Research Quarterly 20, 451–468 (1969)

Research of Nonlinear Combination Forecasting Model

119

10. Zhang, G.P.: Anatomy of B-G Combination-forecasting. Forecasting 12, 24–27 (1988) 11. Wen, G.X., Niu, M.J.: A New Nonlinear Combined Forecasting Method on the Basis of Neural Networks. Systems Engineering-Theory &Practice 12, 66–72 (1994) 12. Dong, J.R., Yang, X.T.: Neural Network Approach to Combination Forecast in Ownerships of Chinese motor vehicle. In: Proceedings of 1997 international conference on management science & engineering, pp. 98–105. Harbin Institute of Technology Press (1997) 13. Cao, L., Hong, Y.: Predicting Chaotic Time Series with Wavelet Networks. Physics D (1995) 14. Li, S.X.: Wavelet Transform and its Applications. Higher Education Press, Beijing (1997) 15. Yang, J.G.: Wavelet Analysis and its Engineering Applications. China Machine Press, Beijing (2005)

Parameter Tuning of MLP Neural Network Using Genetic Algorithms Meng Joo Er and Fan Liu

*

Abstract. In this paper, a hybrid learning algorithm for a Multilayer Percept-rons (MLP) Neural Network using Genetic Algorithms (GA) is proposed. This hybrid learning algorithm has two steps: First, all the parameters (weights and biases) of the initial neural network are encoded to form a long chromosome and tuned by the GA. Second, as a result of the GA process, a quasi-Newton method called BFGS method is applied to train the neural network. Simulation studies on function approximation and nonlinear dynamic system identification are presented to illustrate the performance of the proposed learning algorithm. Keywords: Genetic algorithms, Backpropagation, Function approximation, Nonlinear dynamic system identification.

1 Introduction The most noticeable applications of Neural Networks (NNs) are in the areas of communication channel equalization, signal processing, pattern recognition, system identification, prediction and financial analysis, and process control [1~5]. Such a widespread use of NN is mainly due to its behavioral emulation of the human brain and its learning ability. A NN can be termed as a parallel and distributed process that consists of multiple processing elements. NNs have emerged as an important tool in terms of learning speed, accuracy, and noise immunity and generalization ability. In this paper, the most popular models of NN, such as feedforward NNs are considered. Supervised Learning (SL) is one of the most effective weight training algorithms, whereby efforts are made to find an optimal set of connective weights for a NN according to some optimality criteria. One of the most popular SL training algorithms for feedforward NN is backpropagation (BP). The BP is a gradient descent search algorithm. It is based on minimization of the total mean square Meng Joo Er . Fan Liu School of Electrical and Electronic Engineering, Nanyang Technological Universigy Singapore 639798, Singapore {Emjer,liuf0009}@ntu.edu.sg

*

*

H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 121–130. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

122

M.J. Er and F. Liu

error between the actual output and the desired output [6]. This error is used to guide the search of the BP algorithm in the weight space. The BP is widely used in many applications in that it is not necessary to determine the exact structure and parameters of NN in advance. However, the problem of the BP algorithm is that it is very often trapped in local minima and the learning and the adaptation speed is very slow in searching for global minimum of the search space. The speed and robustness of the BP algorithm are sensitive to several parameters of the algorithm and the best parameters vary from problems to problems. Recently, some developed evolutionary algorithms, notably Genetic Algorithms (GA), have attracted a great attention in the NN areas [6]-[12]. The GA is proposed to train a feedforward NN and its performance is investigated in [6]. In [7], the GA is proposed to optimize the NN topology and connective weights. The GA is used to identify unimportant neurons and delete those neurons to yield a compact structure in [9]. By working with a population of solutions, the GA can seek many local minima, and thus increase the likelihood of finding global minimum. This advantage of the GA can be applied to NNs to optimize the topology and weight parameters. In this paper, a hybrid algorithm based on GA technique is proposed to optimize parameters of the MLP Neural Network (MLPNN). A simple chromosome representation is used, which contains information about connections, weights and biases of the MLPNN. The parameter learning process, based on GA technique and BP algorithm, is a two-step learning process. In the first step, the initial parameters, such as weights and biases of the NN are tuned by the GA. In the second step, the BP algorithm and the quasi-Newton method is introduced to train the initial NN to yield optimal values of weights and biases of the NN. This paper is organized as follows: In Section 2, the general structure of the MLPNN and the process of the GA optimal algorithm are briefly reviewed. Details of the proposed hybrid algorithm are given in Section 3. To demonstrate the effectiveness of the proposed algorithm, simulation studies on function approximation and nonlinear dynamic system identification are carried out in Section 4. Conclusions are given in Section 5.

2 MLPNN MLPNN has become very popular in past decades. The development of the BP training algorithm represents a landmark in NNs in that it provides a computationally efficient method for training the MLPNN. A general three-layer MLPNN is depicted in Fig. 1. In Fig. 1, the NN is a three-layer feedforward NN. Here, multi-input and singleoutput (MISO) systems are considered. However, all results could be extended to multi-input and multi-output (MIMO) systems. The mathematical description of the network is as follows. Input layer: Each node represents an input variable pi , i = 1, 2, , n .

Parameter Tuning of MLP Neural Network Using Genetic Algorithms

θ1

b1 W2

W1

b2

p1

123

θ2 y

p2

pn

bm

Input Layer

Layer 1

θ

r

Layer 2

Output layer

Fig. 1 Topology structure of a three-layer feedforward NN

Layer 1: Each node represents a neuron of layer 1. The term w ji denotes the weight of the link between the jth neuron of layer 1 and the ith input variable. Here, W1 is the corresponding weight matrix of layer 1. Layer 2: Each node represents a neuron of layer 2. The term wkj denotes the weight of the link between the kth neuron of layer 2 and the jth neuron of layer 1. W2 is the corresponding weight matrix of layer 2. Output layer: Each node represents an output as the summation of the incoming signals from layer 2. In such a network, a neuron sums a number of weighted inputs and a bias and then it passes the result through a nonlinear activation function. Fig. 2 shows a typical structure of a MLP neuron with n -input connections and a single output.

Fig. 2 A typical MLP neuron

124

M.J. Er and F. Liu

The output of the neuron is given by y= f(

n

∑w

1i

p i + b) ,

(1)

i =1

where p1 , p 2 , w11 , w12 ,

, p n : Input variables; , w1n : Connective weights;

b : Bias; f : Activation function.

Before a NN can be used for any purpose, the weights of neurons and bias values should be adjusted by a learning algorithm so that outputs of the NN will match the desired patterns for specific sets of inputs. One of the most widespread learning algorithms is the BP algorithm. However, difficulties faced by the BP algorithm are poor convergence rate [13] and an entrapment of local minimum. Here, we choose a quasi-Newton method called the BFGS method for NN training [14].

3 Hybrid BP Learning Algorithm Based on GA The GA is a derivative-free stochastic optimization method based on the features of natural selection and biological evolution. It has several advantages over other optimization algorithms. It can be applied to both continuous and discrete optimization problems. Compared with the BP algorithm, the GA is less likely to get trapped in local minima [6]. It is a computational model inspired by population genetics. It has been used mainly as function optimizers and it has been demonstrated to be an effective global optimization tool, especially for multi-model and non-continuous functions. The GA evolves a multi-set of elements, called a population of individuals. Each individual X i (i = 1,2, , p ) ( p , the size of the population) of population X represents a solution of the problem. Individuals are usually represented by strings and each element of which is called a gene. The value of a gene is called its allelic value, and its range is usually restricted to [0, 1], but it can also be continuous and even structured. We use real-valued strings in our approach. The GA is capable of maximizing a given fitness function F computed on each individual of the population. The block diagram of the proposed hybrid algorithm is depicted by Fig. 3. This model describes a hybrid learning algorithm of MLPNN by using the GA to optimize the parameters of the network. All the parameters of the network are encoded to form a long chromosome and tuned by the GA. Then, as a result of the GA process, the BP algorithm is used to train the network. The procedure of the hybrid BP learning algorithm is presented as follows.

Parameter Tuning of MLP Neural Network Using Genetic Algorithms

125

Start

Creating initial MLPNN

GA Start Initialization GA parameters, pc ,pm Evaluation Fitness

Satisfy the stopping criteria

Output Optimal Weights and Biases to MLPNN

YES

NO Selection New Population

Reproduce

Crossover

Mutation

Training New MLPNN New generation End

Fig. 3 Flowchart of the proposed learning algorithm method

3.1 Chromosome Representation A MLPNN can be represented by a directed graph, encode on a chromosome with each parameter (weights and biases). All these parameters are memorized by a row vector C = (ci ), i = 1,2, , N , where N is the number of all NN parameters. We can write the chromosome as

C = [W1 ,W2 , b1 , b2 ,

, bm ,θ 1 ,θ 2 ,

,θ r ] ,

(2)

where W1 denotes the connective weight of link between the input layer and the first hidden layer, W2 is the connective weight of link between the first hidden layer and the second hidden layer, b1 , b2 ,

, bm are the biases of neurons of the

126

M.J. Er and F. Liu

first hidden layer, θ1 ,θ 2 , ,θ r are the biases of neurons of the second hidden layer. We use the real-value encoding in this paper, W1 , W2 , b1 , b2 , , bm ,

θ1 , θ 2 ,

,θ r are the real values of the connective weights, biases respectively.

3.2 Fitness Function The fitness function is dependent on problem and is used to evaluate the performance of each individual. The error signal of the output neuron j at iteration n (i.e., presentation of the nth training example) is defined by

e j ( n) = d j ( n) − y j (n) .

(3)

We defined the instantaneous value of the error energy for neuron j as 1 2 1 e j (n) . Correspondingly, the value of ξ (n) is obtained by summing e 2j (n) 2 2 over all neurons in the output layer

ξ ( n) =

1 2

∑ e ( n) , 2 j

j∈C

(4)

where the set C includes all the neurons in the output layer of the network. For MLPNN it is the sum squared error. The fitness is defined as by summing ξ (n) over all n with respect to the set size N , as shown by N

F=

∑ ξ ( n) .

(5)

n =1

Obviously the objective is to minimize F (⋅) subject to weights w ji , wkj and biases bi (i = 1,2,

, m) and θ j ( j = 1,2,

, r) .

3.3 Selection Selection operator is to select individuals from the population for reproduction based on the relative fitness value of each individual. The extraction can be carried out in several ways. We use “NormGeomSelcet” ranking selection method in this paper, it is a ranking selection function based on the normalized geometric distribution.

3.4 Crossover To apply the standard crossover operator the individuals of the population are randomly paired. The two mating chromosomes are cut once at corresponding points and the sections after the cuts exchanges. The crossover point can be chosen randomly.

Parameter Tuning of MLP Neural Network Using Genetic Algorithms

127

3.5 Mutation After crossover, the new individuals are subjected to mutation. Mutation prevents the algorithm to be trapped in a local minimum. A variable is selected with a certain probability and its value is modified by a random value. Here, we choose non-uniform mutation method. Non-uniform mutation changes one of the genes of the parent based on a non-uniform probability distribution.

4 Illustrative Examples One of the most powerful uses of an NN is function approximation. Two examples are simulated in this section to demonstrate the effectiveness of the proposed hybrid learning algorithm. They are the Hermite polynomial function and the nonlinear dynamic system identification.

4.1 Function Approximation Here, the underlying function to be approximation is the Hermite polynomial, which is defined as: f ( x) = 1.1(1 − x + 2 x 2 ) exp(−

x2 ). 2

(6)

A set of 100 one-input normalized data and the corresponding target data is used as the training data. Another 100 input-target data are also used as the testing data. The parameter of GA is set as, crossover probability: pc = 0.09 , number of generation: gen = 100 . We construct a 1-15-1 MLPNN and the activation functions are set as “tansig”, “tansig” and “purelin” respectively. “Tansig” denotes hyperbolic tangent sigmoid transfer function. “Purelin” denotes linear transfer function. A set of 17 neurons is generated. The total number of parameters is 46. Simulation results are demonstrated in Fig. 4 ~ Fig. 6. The comparisons of the proposed method and BP method are depicted in Fig. 4 and Fig. 5. The mean squared error arrived at goal value at the 121 epoch in Fig. 4 and at 123 epoch in Fig. 5. The fitness value is shown in Fig. 6.

4.2 Nonlinear Dynamic System Identification The system can be described as

y (t + 1) =

y (t ) y (t − 1)[ y (t ) + 2.5] + u (t ) 1 + y 2 (t ) + y 2 (t − 1)

y (1) = 0, u (t ) = sin( 2πt / 25)

.

(7)

128

M.J. Er and F. Liu

Fig. 4 Training performance with GA

Fig. 5 Training performance without GA

Fig. 6 Fitness evolution (Example 1)

Fig. 7 Fitness evolution (Example 2)

Fig. 8 Training performance with GA

Fig. 9 Training performance without GA

The model is identified in series-parallel mode and is given by yˆ (t + 1) = f ( f (t ), y (t − 1), u (t )) .

(8)

Parameter Tuning of MLP Neural Network Using Genetic Algorithms

129

It is a three-input-single-output model. There are 200 input-data sets chosen as training data. The parameter of GA is set as, crossover probability: pc = 0.095 , number of generation: gen = 100 . We construct a 3-3-1 MLPNN and the activation functions are set as “tansig”, “tansig” and “purelin” respectively. A set of 7 neurons is generated. The total number of parameters is 16. Simulation results are demonstrated in Fig. 7 ~ Fig. 9. The comparisons of the proposed method and BP method are depicted in Fig. 8 and Fig 9. The mean squared error arrived at goal value at the 67 epoch in Fig. 7 and at about 71 epoch in Fig. 9. The fitness value is shown in Fig. 7.

5 Conclusions In this paper, a hybrid learning algorithm using GA to optimize parameters of MLPNN is presented. The initial structure of the MLPNN is created, and all the parameters of the network are tuned by GA. Simulations show that the hybrid learning algorithm has good performance in function approximation and nonlinear dynamic system identification. The GA, as a global search tool, can avoid the local minima problem of the BP. In our recent works, we propose that GA can be used to optimize the parameters of MLPNN. Further research will focus on how to use the GA to obtain a more compact structure of the network with higher accuracy. Not only the parameters of the network, but also the network topology can be optimize by the GA is a future research work.

References 1. Kadirkamanathan, V., Niranjan, M.: A Function Estimation Approach to Sequential Learning with Neural Networks. Neural Computation 5, 954–975 (1993) 2. Juang, C.F., Chin, C.T.: An On-Line Self-Constructing Neural Fuzzy Inference Network and its Applications. IEEE Trans. Fuzzy Systems 6, 12–32 (1998) 3. Widrow, B., Rumelhart, D.E., Lehr, M.A.: Neural Networks Applications in Industry, Business and Science. Communication of the ACM 37, 93–105 (1994) 4. Narendra, K.S., Kannan, P.: Identification and Control of Dynamical Systems Using Neural Networks. IEEE Trans. Neural Networks 1, 4–27 (1990) 5. Vellido, A., Lisboa, P.J.G., Vaughan, J.: Neural Networks in Business: A survey of applications. Expert Systems with Applications 17, 51–70 (1999) 6. Siddique, M.N.H., Tokhi, M.O.: Training Neural Networks: Backpropagation vs Genetic Algorithms. In: Proceeding of the International Joint Conference on Neural Networks, vol. 4, pp. 2673–2678 (1999) 7. Tang, K.S., Chan, C.Y., Man, K.F., Kwong, S.: Genetic Structure for NN Topology and Weights Optimization. In: IEEE Conference Publication, vol. 414, pp. 250–255 (1995) 8. Leng, G., McGinnity, T.M., Prasad, G.: Design for Self-Organizing Fuzzy Neural Networks Based on Genetic Algorithms. IEEE Trans. on Fuzzy Systems 14, 755–765 (2006)

130

M.J. Er and F. Liu

9. Chen, S., Wu, Y., Luk, B.L.: Combined Genetic Algorithm Optimization and Regularized Orthogonal Least Squares Learning for Radial Basis Function Networks. IEEE Trans. on Neural Networks 10, 1239–1243 (1999) 10. Maniezzo, V.: Genetic Evolution of Topology and Weight Distribution of Neural Networks. IEEE Trans. on Neural Networks 5, 39–53 (1994) 11. Billings, S.A., Zheng, G.L.: Radial Basis Function Network Configuration Using Genetic Algorithms. Neural Networks 8, 877–890 (1995) 12. Frank, H., Leung, F., Lam, H.K., Ling, S.H., Peter, K., Tam, S.: Tuning of the Structure and Parameters f a Neural Network Using an Improved Genetic Algorithm. IEEE Trans. on Neural Networks 14, 79–88 (2003) 13. Lippmann, R.P.: An Introduction to Computing with Neural Nets. IEEE ASSP Magazine 1, 4–22 (1987) 14. Watrous, R.L.: Learning Algorithms for Connectionist Networks: Applied Gradient Meth-ods for Nonlinear Optimization. In: Proceeding IEEE First Int. Conf. Neural Net., vol. 2, pp. 619–627 (1987)

Intelligent Grid of Computations Samia Jones*

Abstract. The Web can be considered a massive information system with interconnected databases and remote applications providing various services. While these services are becoming more and more user oriented, the concept of smart applications on the Web is increasing. Most sites still measure success by hits and page views. Instead, building an intelligent infrastructure to track visitors and their activities could be useful. Web intelligence accurately measures site success and guide future direction. Once built, visitor profile, event, and scenario models will clarify relevant hit measurements. To track users, a three-tiered infrastructure that aggregates, stores, and distributes intelligence across the organization could be build. A middleware platform is required to deal with multiple very-large data sources for multi-aspect analysis intelligence by creating a grid-based of web data mining agents known as Data Mining Grid. As users click banners, view products, and make purchases, and commerce software from the vendors will be filed in a central repository (data Warehouse), typically built on Oracle or Microsoft SQL Server. This Web warehouse will become the definitive repository for clean and consistent organization information. To test hypotheses, non-technical decisionmakers will use interactive analysis tools (OLAP and query tools ) from vendors like SAS. OLAP and query tools only answer the questions put to them -- they don’t reveal what users should have asked. But Data mining tools uncover hidden trend to find less-obvious knowledge. Data mining tools are available from vendors like DataSage, kapowtech and DBMiner. A multi-level control data mining architecture model included. Keywords: Intelligent Grid Data mining.

1 Introduction Central to human intelligence is the process of learning or adapting. Likewise, machine learning may be the most important aspect of Artificial Intelligence (AI), including behavior, cognition, symbolic manipulation, and achieving goals. This suggests that AI software should be concerned with being changeable or adaptable. The challenge for AI is to learn capabilities for helping people derive specifically targeted knowledge from diverse information sources, such as, the Web. Subsequently, one of the challenges facing Web Services includes developing a Samia Jones Texas A&M University at Qatar PO Box 23874 - Doha, Qatar Texas A&M Engineering Building-Education City *

H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 131–135. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

132

S. Jones

Table 1 The total verified number of search results Total Total Totalfrom from from March2002 Aug. 2001 (in Dec. 2002 Search Engine (in millions) (in millions) millions) Google

9,732

8,371

6,567

AlltheWeb

6,757

4,388

4,969

AltaVista

5,419

3,432

3,112

WiseNut

4,664

5,009

4,587

HotBot MSN Search Teoma NLResearch Gigablast

3,680 3,267 3,259 2,352 2,352

2,869 2,523 1,839 3,610 NA

3,277 3,005 2,219 3,321 NA

global consensus for an architecture that lets applications (using object-oriented specialized software components) plug into an "application bus" and call a Web AI service. Smart Web applications are security controlled to allow users access only to information that they are authorized to see. Multi-layered security (via password control and the Oracle database) allows some users to view only; while others can also edit directory information, send pages or update on-call schedules. Information Portals are one of the most sophisticated applications on the Web today. During 1998, the first wave of Internet Portals became very popular. They provided consumers with personalized points of entry to a wide variety of information on the Internet. Examples included; MyYahoo (Yahoo), NetCenter (Netscape), MSN (Microsoft) and AOL. The following table gives the total verified number of search results (including Web pages, PDFs, other file types) from all searches. Since the exact same queries were used in Dec. 31, 2002., March 2002 and August 2001. Based on All the Web reported size and percentages, by Greg R. Notess. Moreover, It should be obvious that the current trend is using a grid technology of the Web services structure. The aim of this paper is to develop an intelligent grid and test it.

2 Research It is believed that the next paradigm shift in the Web is towards smart choices and knowledgeable queries. Now, the question is: How to evolve through this paradigm and generate association rules with services to support the choices that may aggregate automatically knowledgeable queries?

Intelligent Grid of Computations

133

2.1 Web Intelligence System Basic System of WIS (Web Intelligence System) may look like this: Portal → search engine→ database ↔ updates → cookies →retrieve info ↔ back to user. (Figure 1 shows that graphically)

Internet And Data Base, Search engine & Networks

Queries

Web Services

Usage Mining

HTML

VML

HTTP

WAP

TCP

UDP IP

Web logs, Cookies Users Profiles

Content Mining

Data Grid

Fig. 1 A basic Web Mining Intelligence model

Nevertheless, based on Russell & Norvig’s scheme, WIS can be classified into four categories as follows: Table 2 Designing philosophy of WIS and Ability, Functionality Of WIS

System that thinks like humans

System that thinks rationally

System that acts like humans

System that acts rationally

Moreover, to capture data transaction for future analysis, there is the need of Web mining techniques to track the clicks of the mouse that define where the visitor has been on the website. In order for the smart choices to cooperate and compete among themselves to optimize themselves as well as others’ resources and utilities, a new platform as the middleware is required to deal with the multiple very-large data sources for multi-aspect analysis in portals for the Web Intelligence System (Figure 2). Creating a grid-based, of Web mining agents, called Data Mining Grid maybe used to: 1. 2. 3. 4.

develop various data mining agents for different targeted tasks; organize the agents into a grid with multi-layer under the Web as a middleware; use the gird for multi-analysis in distributed, multiple data sources; manage the grid by a multi-level control techniques

134

S. Jones

Fig. 2 A multi-level control intelligent infrastructure model

Data Base & Warehouse

Internet And Web Services

AI

Intelligent infrastructure

Adaptive SQL

HTML

VML

HTTP

WAP

Dynamic Languages Logic Layer TCP

Results

UDP

IP

Data Grid

XML schema

Compute Grid

So, the purpose of the paper is to study the pattern of the visited sites and the relation between them as well as to analyze the visitors and possibly their motives of visiting the site. Designing a system that acts like humans based on their preferences and the wide variety of information that are available on the web. Therefore, data will be collected about the visited sites, and study of the association between the pages visited to determine if it is random or related.

3 Hypothesis The visitor is known to the visited sites The visitor is new to the visited sites, but seems to be Savvy user (computer knowledgeable) The visitor is new to the visited sites, but seems to be Ignorant user (a computer Ill literal) The visitor is new to the visited sites but seems to be a random user (No pattern on the visits) Four groups will be formed: • • • •

Recurrent user; is known to the visited sites, and finds the information quick. Savvy user: is new to the website, and knowledgeable to find the information of hyperlinks. Ignorant user; is new to the website, but computer Ill literal, too many mistakes or clicks to get the right hyperlinks. Random user: has no strong intention to get any inks no pattern among the choices, just wanders among pages.

4 Data Collection Multiple data sources that are obtained from multiple customer touch points that would include the Web, wireless webs, call centers, and brick-and-mortar store data. All will be integrated into a distributed data warehouse that provides a multifaced view of the customers.

Intelligent Grid of Computations

135

5 Conclusion It should be obvious from the description of the proposed model, and the current trend of grid technology's move toward a Web services structure, that the two are converging. In simple terms, the grid is a distributed system for sharing resources, while this model is a distributed architecture most concerned with easy integration, and simple, extensible, secure access. Both systems have common problems which is the partial failures. This research could provide better scalable performance and/or an improved data mining model quality by using Social Network theory which is now significantly influencing search engine and portal development for the Web. This could also be compared to an existing implementations of database-integrated mining kernels. Finally, the present approach can lead to greater flexibility in the deployment of a data mining application, leading to better overall workload management on an enterprise data servers that support a complex mix of transactional, decision support, data management and data mining applications.

References 1. Yao, Y., Zhong, N., Liu, J., Ohsuga, S.: Web Intelligence (WI). In: Zhong, N., Yao, Y., Ohsuga, S., Liu, J. (eds.) WI 2001. LNCS(LNAI), vol. 2198, pp. 1–17. Springer, Heidelberg (2001) 2. Berners-Lee, T., Hendler, J., Lassila, O.: The Semantic Web. Scientific American 284, 34–43 (2001) 3. Alesso, H.P., Smith, C.F.: The Intelligent Wireless Web. Addison-Wesley, Reading (2000) 4. Kumar, R., Raghavan, P., Rajagopalan, S., Tomkins, A.: The Web and Social Networks. IEEE Computer Special Issue on Web Intelligence 35, 32–36 (2002) 5. Congiusta, A., Pugliese, A., Talia, D., Trunfio, P.: Designing Grid Services for Distributed Knowledge Discovery. Web Intelligence and Agent Systems 1(2) (2003) 6. Zhong, N., Yao, Y., Ohshima, M.: Peculiarity Oriented Multi-Database Mining. IEEE TKDE 15(4) (2003) 7. Friedman, N., Getoor, L.: Efficient Learning Using Constrained Sufficient Statistics. In: Proceedings of the 7th International Workshop on Artificial Intelligence and Statistic (1999) 8. Apte, C., Natarajan, R., Pednault, E., Tipu, F.: A Probabilistic Framework for Predictive Modeling Analytics. IBM Systems Journal 41(3) (2002) 9. Apte, C., Grossman, E., Pednault, E., Rosen, B., Tipu, F., White, B.: Probabilistic Estimation Based DataMining for Discovering Insurance Risks. IEEE Intelligent Systems 14 (1999) 10. Natarajan, R., Pednault, E.P.: Segmented Regression Estimators for Massive Data Sets. In: Proc. Second SIAM Conference on Data Mining, Crystal City, VA (2002) 11. Pednault, E.: Transform Regression and the Kolmogorov Superposition Theorem. IBM Research Report RC 23227, IBM Research Division, Yorktown Heights, NY 10598 (2004) 12. Fraley, C.: Algorithms for Model-Based Gaussian Hierarchical Clustering. SIAM J. Sci. Comput. 20, 270–281 (1998)

Method of Solving Matrix Equation and Its Applications in Economic Management Qingfang Zhang and June Liu*

Abstract. By changing the conditions of linear system of equations, some important conclusions about matrix equation are obtained. Furthermore, the applications of matrix equation in economic management are discussed. Keywords: Matrix equation, Solution of matrix equation, Economic management.

1 Introduction In our production, management and commodities circulation, there are some questions can be expressed directly or indirectly by more than one system of linear equations. The matrix equation is a powerful tool to solve this kind of problem. Because of concise algorithm rules, the matrix equation makes the linear relationship with many variables easier. Furthermore, the matrix equation not only supply the theorem foundation, but also optimize programs and improve the strategic ability in practise. Thence, we should attach importance to the study of matrix equation. In many linear algebra books, there are satisfied solutions to linear equations. Unfortunately, the researches into matrix equation application are limitedly. In this paper, the author has further studied solution of the matrix equation, with a preliminary probe into matrix equation application on related issues of economic management.

2 Preliminaries We denote: A : m × s matrix, B : m × n matrix, X matrix, R( A) = r : rank of matrix A .

、 b : n ×1 matrix, E : identity

Lemma 1. Linear system of equations AX = b has solutions if and only if

R( A) = R( A, b ) = r ; Qingfang Zhang . June Liu College of Mathematical and Information Sciences, Huanggang Normal University, Huanggang 438000, China [email protected]

*

H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 137–141. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

138

Q. Zhang and J. Liu

(1) AX = b has unique solution when r = s ; (2) AX = b has infinite solutions when r < s . Lemma 2. Let A = (ail ) ≠ O be matrix with rank m , then A is an inverse matrix m

∑| a

if

i≠ j

ij

| <| a jj | ( j = 1, 2,

m) .

Column matrix b in Lemma 1 is applied by matrix B , we have Theorem 1. The necessary and sufficient condition that matrix equation AX = B has solutions is R( A) = R ( A, B ) = r .

(1) AX = B has unique solution when r = s ; (2) AX = B has infinite solutions when r < s . Proof. Let X = ( X 1 , X 2 ,

, X n ), B = ( B1 , B2 ,

, Bn ) , then matrix equation AX = B

is equal to vector equation AX i = Bi (i = 1, 2, Since ( A, B) = ( A, B1 , B2 ,

, n) .

, Bn ) ⎯⎯ →( A, B1 , B2 ,

where A is the row simple matrix of A . →( A, Bi ) (i = 1, 2, Then we have ( A, Bi ) ⎯⎯

, Bn )

, n) by primary translation.

Suppose that R( A) = r ,then A has r nonzero rows and others are zero. Using Lemma 1, we have AX = B ⇔ AX i = Bi (i = 1, 2,

, n) ⇔ R( A) = R( A, Bi )(i = 1, 2,

, n) ,

which implies that the last m − r elements of Bi are zero (i = 1, 2, , n) . In other words, R( A, B) = r = R ( A) . The matrix equation AX = B is equal to AX i = Bi (i = 1, 2, , n) according to the proof above. Thence, AX i = Bi (i = 1, 2, , n) has unique solution when r = s . So does AX = B . And, AX i = Bi (i = 1, 2, , n) has infinite solutions when r < s , in this case AX = B has infinite solutions. Theorem 2. Let A = (aij )m× m be a matrix with

aij ≥ 0 ( i, j = 1, 2,

m

m),

∑a i =1

ij

< 1( j = 1, 2,

, m) .

Then ( E − A) X = b has unique solution and X = ( E − A) −1 b . m

Proof.

∑a i =1

ij

= a1 j + a2 j +

+ amj < 1( j = 1, 2,

a1 j + a2 j + a j −1, j + a jj + a j +1, j +

, m) shows + amj < 1 − a jj ,

Method of Solving Matrix Equation and Its Applications in Economic Management m

∑a

i.e.,

i≠ j

ij

< 1 − a jj .

aij =| aij |=| −aij | is obvious by aij ≥ 0(i, j = 1, 2, m

Then

∑ | −a i≠ j

ij

139

, m) .

| <|1 − a jj | (or E − A ) is obtained. The Lemma 2 implies that E − A

is an inverse matrix and R ( E − A) = m . By Lemma 1, matrix equation ( E − A) X = b has unique solution and X = ( E − A) −1 b .

3 The Methods of Solving Matrix Equation Not only the matrix equation ( E − A) X = b , but also AX = B , we can solute them as following: ⎛ E B1 ⎞ Case 1: r = s . First, ( A, B) ⎯⎯ →⎜ r ⎟ by primary row translation, then ⎝O O⎠ ⎛E ⎞ ⎛B ⎞ AX = B and ⎜ r ⎟ X = ⎜ 1 ⎟ have same solution X = B1 . O ⎝ ⎠ ⎝O⎠ Case 2: r < s .

⎛ A B⎞ →⎜ First, ( A, B) ⎯⎯ ⎟ by primary row translation where A are r × s row ⎝O O⎠ simple matrix and B are r × n matrix. Let X = ( X 1 , X 2 ,

, X n ), B = ( B1 , B2 ,

, Bn ) ,

then the same solution equation AX i = Bi of AX = B has infinite solutions for every i and has s − n free variables. Suppose that X 0i were is a special solution of AX i = Bi (i = 1, 2, , n) , then X is the general solution. So the all solutions of AX i = Bi are X i = { X 0i + X } (i = 1, 2, , n). The all solutions of AX = B are X = ( X1 , X 2 ,

, X n ) = ( X 01 + X , X 02 + X ,

, X 0n + X ) .

4 Application Examples in Economic Management Matrix equation is a power tool to express the relationship between many variables. And it can be applied to describe the world as a mathematics module. Some problems in the economics can be solved by using matrix equation.

140

Q. Zhang and J. Liu

Table 1 The amounts will be sold A

B

I

60

64

II

100

48

Example 1 Two companies I and II all sell two products A and B. The Table 1 represents the amounts will be sold in a year. How much are the price of two this products and the profit in order to achieve the objective in Figure 2. Table 2 Income and profit in a year

I II

Income (a year)

Profit (a year)

4360 4920

2780 3460

Now, suppose that the price and profit are showed in Tabel 3. Table 3 Income and profit Price

Profit

A

X 11

X 12

B

X 21

X 22

⎛ 60 64 ⎞ A=⎜ ⎟ describe the amounts of products sold in a ⎝100 48 ⎠ ⎛ X 11 X 12 ⎞ ⎛ 4360 2780 ⎞ year, B = ⎜ ⎟ the price ⎟ the income and profit a year, X = ⎜ ⎝ 4920 3460 ⎠ ⎝ X 21 X 22 ⎠ and profit one product,respectively. Then the problem can be solved by matrix equation AX = B as following: Let

⎛ 60 64 ( A B) = ⎜ ⎝100 48

4360 2780 ⎞ primary row translation ⎛ 1 0 ⎟ ⎯⎯⎯⎯⎯⎯⎯→ ⎜ 4920 3460 ⎠ ⎝0 1

30 25 ⎞ ⎟ 40 20 ⎠

⎛ 30 25 ⎞ and X = ⎜ ⎟. ⎝ 40 20 ⎠ Example 2 Let I II III be three workshops. The expense coefficients of these three workshops in a product period are represented in Tabel 4. The products of this three workshops are 235,125,210,respectively. Now we will compute the total value of output.

、、

Method of Solving Matrix Equation and Its Applications in Economic Management

141

Table 4 Expense coefficients of these three workshops in a product period I

II

III

I

0.25

0.10

0.10

II

0.20

0.20

0.10

III

0.10

0.10

0.20

⎛ −0.25 −0.1 −0.1 ⎞ ⎛ 235 ⎞ ⎜ ⎟ ⎜ ⎟ Let A = ⎜ −0.2 −0.2 −0.1 ⎟ describe the expense coefficients, b = ⎜ 125 ⎟ ⎜ −0.1 −0.1 −0.2 ⎟ ⎜ 210 ⎟ ⎝ ⎠ ⎝ ⎠ ⎛ x1 ⎞ ⎜ ⎟ the total products of three workshops and X = ⎜ x2 ⎟ the total value of output. ⎜x ⎟ ⎝ 3⎠ ⎧(1 − 0.25) x1 − 0.1x2 − 0.1x3 = 235 ⎪ Then we have the system of equation ⎨ −0.2 x1 + (1 − 0.2) x2 − 0.1x3 = 125 . ⎪ −0.1x − 0.1x + (1 − 0.2) x = 210 1 2 3 ⎩ And the matrix equation ( E − A) X = b can be obtained. At last, the solution of ⎛ 400 ⎞ ⎛ 0.75 −0.1 −0.1⎞ ⎜ ⎟ ⎜ ⎟ matrix equation is X = ⎜ 300 ⎟ by E − A = ⎜ −0.2 0.8 −0.1⎟ = A1 and ⎜ 350 ⎟ ⎜ −0.1 −0.1 0.8 ⎟ ⎝ ⎠ ⎝ ⎠ ⎛ 0.75 −0.1 −0.1 0.8 −0.1 ⎜ −0.1 −0.1 0.8 ⎝

(A1 b ) = ⎜ −0.2 ⎜

235 ⎞ ⎛1 0 0 ⎟ primary row translation ⎜ 125 ⎟ ⎯⎯⎯⎯⎯⎯⎯ →⎜ 0 1 0 ⎜0 0 0 210 ⎟⎠ ⎝

400 ⎞ ⎟ 300 ⎟ . 350 ⎟⎠

References 1. Guo, J., Wang, W.: The solutions of Matrix Equation AX=B. Journal of Science of Teachers College and University 7, 86–88 (2008) 2. Ma, H., Liu, X.: Structured Solutions to Matrix Equation XB=C. Journal of Applied Mathematics 1, 49–51 (2008) 3. Sheng, X., Chen, G.: Research to the Solution of Matrix Equation AXB=D. Journal of Lanzhou Universtiy 6, 101–104 (2006) 4. Wu, C.: Linear Algebra. Higher Education Press, Beijing (2003) 5. Yang, G.: Some Issues on Linear Equation. College Mathematics 8, 161–167 (2008) 6. Zheng, G.: A Block Matrix Solution to Matrix equation. College Mathematics 4, 124– 127 (2005) 7. Zhang, G.: Solving Matrix Equations by Primary Row Transformation of Matrix. College Mathematics 12, 117–120 (2003) 8. Zheng, H.: On Solving General Solutions of the Linear Matrix Equation. Journal of Mathematics for Technology 6, 83–86 (2002)

Efficient Feature Selection Algorithm Based on Difference and Similitude Matrix Weibing Wu, Zhangyan Xu, and June Liu*

Abstract. Feature selection algorithm based on method-difference-similitude matrix (DSM) is a better method of data mining. In this method, for storing Dmatrix and S-matrix, the efficiency of the algorithm is seriously affected when the massive data sets are considered. So we use the idea of the old algorithm to design a new feature selection algorithm which need not store D-matrix and S-matrix. The complexity of the new algorithm are better than that of the old. At last, an example is used to illustrate the efficiency of the new algorithm. Keywords: Rough set, Feature selection, Difference matrix, Similitude matrix.

1 Introduction Reduction of pattern dimensionality via feature extraction and feature selection belongs to the most fundamental steps in data preprocessing[1-5]. Feature selection techniques aim at reducing the number of unnecessary, irrelevant, or unimportant features. It is common practice to use a measure to decide the importance and necessity of features. In 1992, Professor Skowron[6] proposed discernibility matrix methods to reduce the unnecessary attributes and select necessary attributes of information system. Since the frequencies of attributes appearing in the discernibility matrix implicate their abilities to distinguish different classes, the attribute with higher frequency is preferred to being selected. However in rules reducing, the numbers of reduced rules with the discernibility matrix method may not be the least and the reduction result may not be the simplest rules, because the similar characteristics are not revealed in discernibility matrix. So Professor Yan etc. [7,8,9,10] proposed a new knowledge reduction method based on methoddifference-similitude matrix (short by DSM). The new method can synthesize the Weibing Wu . June Liu College of Mathematics and Information Science, Huanggang Normal University, Huanggang, 438000, China [email protected]

*

Zhangyan Xu Department of Computer, Guangxi Normal University, Guilin, 541004, China [email protected] H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 143–152. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

144

W. Wu, Z. Xu, and J. Liu

similar rules into as fewer ones as possible. So the DSM is a better method for feature selection in rough set. Professor Wu proposed an efficient algorithm for feature selection with DSM. The time complexity of this algorithm is O(| C |2 | U |2 ) . This algorithm must store D-matrix and S-matrix. So the space complexity of this algorithm is O (| C || U |2 ) , and we need about | C | × | U |2 units for storing them. For example,

if the number of the objects and attributes are 10 5 and 10 2 respectively, the D-matrix and S-matrix need about 1012 units in the worst case. This is unacceptable. Therefore, the size of units which the D-matrix and S-matrix require is a restriction when massive object sets are considered. For using a restricted space, we used the idea of radix sorting in [11] to design a new algorithm for feature selection based on DSM, but in the new algorithm, we did not construct D-matrix and S-matrix. The time complexity and space complexity of the new algorithm are cut to O(| C || U |2 ) + O(| C |2|U |) and O(| U |) + O(| C |) respectively. So the new algorithm is better than the old algorithm.

2 The Old Algorithm Based on DSM In this section, we briefly introduce the old algorithm based on DSM [10], and briefly review the concepts in [10]. A decision table is defined as S=(U,C,D,V,f), where U is the set of objects, C is the set of condition attributes, D is the set of decision attributes, and C ∩ D =∅; V = ∪ Va , where Va is the value range of attribute a. f : ( C ∪ D ) × U → V is a∈C ∪ D

information function, which one information value for each attribute of each object, that is ∀ a ∈ C ∪ D , x ∈ U , f ( a, x ) ∈Va holds. For ∀P ⊆ (C ∪ D) there is a binary indiscernibility relation IND(P):

IND(P) ={(x, y)∈U ×U | ∀a∈P, f (a, x) = f (a, y)} . The partition of U, generated by IND(P) is denoted as U / IND(P) (for short U/P). If

，

(x,y)∈IND(P) then x and y are indiscernible by attributes from P. Any element [ x]P = { y | ∀a ∈ P, f ( x, a) = f ( y, a)} in U/P is called equivalence class. Suppose there are n objects in S, and they are categorized into m categories by D. Let us denotes the ith object as xi, the union of objects in the jth category as U j, the value of attribute a of xi as f ( a , xi ) . Difference sets: The difference set of xi, denoted as S D ( xi ) .

S D ( xi ) = {md md = ∪ {a | f ( a, xi ) ≠ f ( a , xi ) ∧ f ( D, xi ) ≠ f ( D, xi )}, a ∈ C , j = 1, 2,..., n} The difference set of the jth category, denoted as SD (U j )( j = 1, 2,..., m) is defined as:

Efficient Feature Selection Algorithm Based on Difference and Similitude Matrix

145

S D (U j ) = {S D ( xi ) | xi ∈ U j } . The difference set of S, denoted as S D ( S ) , is defined as:

S D ( S ) = {S D ( xi ) | xi ∈ U } . The core of S, denoted as Core(C ) , is defined as:

Core(C ) = {a | a ∈ C ∧ ∃md = {a} ∈ SD ( xi ) | xi ∈U }. Similitude sets: The similitude of xi, denoted as S S ( xi ) .

S S ( xi ) = {ms ms = ∪ {a | f ( a, xi ) = f ( a , xi ) ∧ f ( D, xi ) = f ( D , xi )}, a ∈ C , j = 1, 2,..., n} The similitude set of the jth category, denoted as SS (U j ) , is defined as:

S S (U j ) = {S S ( xi ) | xi ∈ U j } . The similitude set of S, denoted as S S ( S ) , is defined as:

S S ( S ) = {S S ( xi ) | xi ∈ U } . The important factor of attribute a, denoted as F(a) , is defined as below:

F (a ) = ∑ i =1 q( a , i ) / | S S (U i ) |2 m

where q(a,i) denotes the frequency of attribute a appearing in S S (U i ) . The rank function of attribute a, denoted as R(a), is defined as below:

R(a ) = N (a ) × F (a ) where N(a) denotes the frequency of attribute a appearing in S D ( S ) . Only the attribute with the greatest rank value will be selected, and the rank values will be re-calculated after each selection. The followings are the old algorithm [10] . Input: Object number: n; category number: m; condition attribute set: C; decision attribute set: D; attribute value set: V; information function: f. Output: Reduct. Initial state: Reduct=Null, Core(C ) =Null. Steps: (1) Calculate difference-similitude set of the whole decision table. Generate difference set of the decision table S D ( S ) ; for i=1 to m do Generate difference set S D (U i ) ; (2) Find out the core attributes. Computing Core(C ) ;

146

W. Wu, Z. Xu, and J. Liu

Reduct← Core(C ) ;

S D ( S ) = S D ( S ) − {md | md ∩ Core(C ) ≠ ∅} ; X = C − Core(C ) ; (3) Calculate F(a), which denotes the important factor of each attribute a in X. (4) Generate reduction result. (4.1) For each attribute a in X, computing R(a); (4.2) Choose attribute a with the greatest R(a) in X (4.3) S D ( S ) = S D ( S ) − {md | md ∩ {a}) ≠ ∅} ; (4.4) Reduct←Reduct+{a}; (4.5) X=X − {a}; (4.6) If | S D ( S ) | ≠ 0 then go to step (4.1); In this old algorithm, it must store the D-matrix and S-matrix. The larger of data set is, the worst of efficiency of the old algorithm is. Because the size of units for storing the D-matrix and S-matrix is unacceptable when the massive data sets are considered. In the next section, we use the idea of the old algorithm, but we need not store the D-matrix and S-matrix.

3 The New Algorithm Based on DSM Because the core of the decision table is the part of any reductions of the decision table, the core is first to be computed. For cutting the space complexity of the new algorithm, we need not store the D-matrix and S-matrix. Now we first propose the algorithm for computing the core of the decision table. Algorithm 1: computing the core(C) Input: Object sets: U = {x1, x2, , xn} ; condition attribute set: C = {c1, c2 , , cr } ; decision attribute set: D; attribute value set: V; information function: f. Output: Core(C ) Initial state: Core(C ) =∅. for (i=1;i
for (j=i+1;j
B ← ∅;

for (k=1;k
if (|B|>1) break; } if (|B|==1) Core(C ) = Core(C ) ∪ B; }

Efficient Feature Selection Algorithm Based on Difference and Similitude Matrix

147

It is obviously that the time and space complexity of the algorithm for computing the Core(C) are O (| U |2 | C |) and O (| C |) in the worst case respectively. In the old algorithm, S-matrix is used to compute the m F (a) = ∑ q(a, i) / | SS (Ui ) |2 . In fact, there is another method to compute F(a) that i =1

do not use the S-matrix. Now we provide a new method to calculate F(a) without S-matrix. Algorithm 2: calculate F(a) Input: Object sets:

; category number: U = { x1 , x 2 , , x n } U / D = {U1,U2 , ,Um} ; condition attribute set: C; decision attribute set: D; attribute value set: V; information function: f. Output: F(a), ∀a ∈ C . (1) To a, calculate the maximum and minimum of f (a, x j ) ( j =1,2, ,n) and de-

note M and m respectively; (2) F(a)=0; (3) for ( k=1;k<m+1;k++) (3.1) Use static list to store the objects U k = {xk 1 , xk 2 , , xkt } in turn; let the head pointer of the list point to xk1 ; (3.2) Construct Mi-mi+1 empty queues, let frontk and end k (k=0,1,…, Mimi) be the head pointer and tail pointer of the kth queue respectively. Distribute the object x of the list Uk to the f ( x, ci ) − mi th queue according to the elements order of list Uk. (3.3) Calculate the number of objects in each non-empty queue, donate t1,t2, ,ts (t1 + t2 + + ts = t) ; calculate F(a) as follow: F (a ) = F (a ) + ∑ s ti2 /t 2 . i =1

It is obviously that the time and space complexity of the algorithm for calculating F(a) are O (| U |) and O (| U |) respectively. Now we only calculate N(a). It also has a new method to calculate N(a) without D-matrix. To calculate N(a), we first propose an efficient algorithm for computing U/R(R⊆C). Algorithm 3: calculate U/R(R⊆C) Input: Object sets: U = {x1, x2, , xn} ; condition attribute set: R = {r1, r2, , rl }; decision attribute set: D; attribute value set: V; information function: f. Output: U/R. (1) To each ri ( i = 1, 2, , l ) , calculate the maximum and minimum of f ( x j , ri ) ( j = 1,2, , n) and denote M i and mi respectively;

(2) Use static list to store the objects of the list point to x1 ; (3) for (i=1;i
x1 , x2 , , xn in turn ; let the head pointer

148

W. Wu, Z. Xu, and J. Liu

(3.1) The ith “distribution”: construct Mi-mi+1 empty queues, let frontk and

end k (k=0,1,…, Mi-mi ) be the head pointer and tail pointer of the kth queue respectively. Distribute the object x of the list U to the f ( x , ri ) − mi th queue according to the elements order of list U. (3.2) The ith “collection”: the head pointer of the list points to the head pointer of the first non-empty queue, modify the tail pointer of each non-empty and let it point to the head object of the next nonempty queue. In this way, recombine M i − mi + 1 queues to a new list; (4) Let the objects sequence of list from Step 3 be x1′, x2′ , t=1; Bt = {x1′} ;

, x′n ;

for (j=2;j
else { t=t+1; Bt = {x′j } ;} The time and space complexity of the algorithm 3 are O (| R || U |) and O ( | U |) respectively [11,12]. Now we give the following algorithm to calculate N(a). Algorithm 4: calculate N(a) Input: current reduction reduce(C) , U/ reduce(C ) = {R1 , R2 , , Rz } ; condition attribute set C; decision attribute set: D; attribute value set: V; information function: f. Output: N(a) , U/( reduce(C ) ∪a)= {R1′, R2′ , , Rz′′} , a ∈ C − reduce(C ) . (1) To a, calculate the maximum and minimum of f (a, x j ) ( j =1,2, ,n) and denote M and m respectively; (2) N(a)=0; (3) for ( k=1;k
i =1

ki

(3.2) If t≠1 we use the algorithm 3 to compute Rk /{a} = {Rk 1 , Rk 2 ,

, Rkt } ;

h = h1 − h2 ; if | Rki / D |== 1 , then delete Rki ( Rki ∈ Rk /{a } ); To the rest of Rkj ∈ R k /{ a } , calculate h2 = ∑t | Rki |(| Rki | −1)/2 . i=1

(3.3) N ( a ) = N ( a ) + h ; Notation: Any two objects of each Rk (k = 1,2,..., Z ) are indiscernible by attributes set {a,d}. But any two objects in different Ri and R j are discernible by attributes

set {a,d}. Since N(a) denotes the frequency of attribute a appearing in SD(S), according to the old algorithm, only two objects in each Rk (k = 1,2,..., Z ) may be obtained attribute a in SD(S). However, for any two objects in each Rki ∈ Rk / D can

Efficient Feature Selection Algorithm Based on Difference and Similitude Matrix

149

not be obtained attribute a because the value of decision attribute D of their are the same. According to the old algorithm, the frequencies of attribute a in SD(S)may be h1. On the other hand, to any two objects in each Rki ∈ Rk /{a} can not be obtained attribute a because the value of conditional attribute {a} of their are the same. So to Rk (k = 1,2,..., Z ) , the frequencies of attribute a in SD(S) is h1 –h2. The time and space complexity of the algorithm 4 are O (| U |) and O (| U |) in the worst case respectively according to the algorithm 3. Now, we can design a new as follow. Algorithm 5: efficient algorithm based on DSM Input: Object number: n; category number: m; condition attribute set: C; decision attribute set: D; attribute value set: V; information function: f. Output: Reduct. Initial state: Reduct=∅, Core(C ) =∅. (1) Calculate Core(C ) according to the algorithm 1; (2) Reduct= Reduct ∪Core(C ) ; X=C- Core(C ) ; (3) Calculate U / Core(C) according to the algorithm 3; (4) To ∀ a ∈ X , calculate according to the algorithm 2 and calculate N(a) according to the algorithm 4, R(a)= F(a)× N(a); (5) Choose attribute a with the greatest R(a) in X; (6) If | U / Reduct | ≠ 0 then go to step (4); Analysis of the complexity for the algorithm 5. The time complexity of the step 1 is O (| C || U |2 ) in the worst case. The time complexity of the step 3 is O (| C || U |) in the worst case. The time complexity of the step 4 is O (| X || U |) . So all the time complexity of the algorithm from the step 4 to the step 7 is O(| C || U |) + O((| C | −1) |U |) + + O(2| U |) + O(|U |) = O(| C |2|U |) in the worst case. So the time complexity of the new algorithm is O(| C || U |2 ) + O(| C |2|U |) . So it is better than the time complexity of the old algorithm with O(| C |2|U |2 ) . It is obviously that the space complexity of the new algorithm is O(| C |) + O(|U |) . It is also better than the space complexity of the old algorithm with O(| C || U |2 ) .

4 An Example The reduction on the following decision table will illustrate the reducing in detail. For an decision table is given in table 1[9]. According to the step 1 of the algorithm 5, we use the algorithm 1 to calculate the core of decision table 1. So Core(C)={a,d}. According to the step 3 of the algorithm 5, we can calculate U/{a,d} as follow. According to the step 3.1of Algorithm 3, there is :

front[0] → X1 → X 2 → X 4 → X10 → X12 ← end[0] ;

150

W. Wu, Z. Xu, and J. Liu

Table 1 Decision table 1

X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13 X14

a 1 1 3 1 3 2 3 3 2 1 3 1 2 2

b 1 1 3 2 2 1 2 3 3 1 2 2 2 1

c 1 1 2 1 1 1 1 2 2 2 2 1 1 2

d 1 2 2 1 2 1 1 1 2 1 1 2 2 1

e 1 2 2 2 2 1 1 2 2 2 1 1 2 1

f 1 2 1 1 1 1 1 1 1 1 2 2 1 2

D 1 1 1 1 1 2 2 2 2 2 2 2 2 2

front[1] → X 6 → X 9 → X 13 → X 14 ← end [1] ; front[2] → X 3 → X 5 → X 7 → X 8 → X11 ← end[2] ; According to the step 3.2 of Algorithm 3, there is:

X1→X2 →X4 →X10 →X12 →X6→X9→X13→X14 → X 3 → X 5 → X 7 → X 8 → X11; According to the step 3.1 of Algorithm 3, there is :

front[0] → X1 → X 4 → X10 → X 6 → X14 → X 7 → X 8 → X 11 ← end [0] ; front[1] → X 2 → X12 → X9 → X13 → X3 → X5 ←end[1]; According to the step 3.2 of Algorithm 3, there is

X1→X4 →X10 →X6 →X14 →X7 →X8→X11→X2 → X12 → X 9 → X13 → X 3 → X 5 According to the step 4 of Algorithm 3, there is U/{a,d}={{X1,X4,X10}, {X6, X14}, {X7,X8,X11},{X2,X12},{ X9,X13},{X3,X5}}. According to the step 4 of the algorithm 5, we can calculate R(b)= F(a)× N(a). We first use the algorithm 2 to calculate F(b). Now the calculation of F(b). The maximum and minimum of f (b, Xj) ( j =1,2, ,14) are M=3 and m=1 respectively according to the step 1 of the algorithm 2. To X 1 → X 2 → X 3 → X 4 → X 5 , according to the step 3.2 of Algorithm 2, there is: front[0] → X 1 → X 2 ← end [0] ; front[1] → X 4 → X 5 ← end [1] ; front[2] → X 3 ← end [2] ;

Efficient Feature Selection Algorithm Based on Difference and Similitude Matrix

151

According to the step 3.3 of Algorithm 2, there is t1 = t2 = 2, t3 = 1. So

F(b) = 0 + (22 + 22 +12 )/52 =9/25; To X 6 → X 7 → X 8 → there is:

→ X 14 , according to the step 3. 2 of Algorithm 2,

front[0] → X 6 → X 10 → X 14 ← end [0] ; front[1] → X 7 → X 11 → X 12 → X 13 ← end [1] ; front[2] → X 8 → X 9 ← end [2] ; According to the step 3.3 of Algorithm 2, there is t1 = 3, t 2 = 4 , t 3 = 2 . So

F(b) = 9/ 25 + (32 + 42 + 22 )/92 = 9/25 + 29/81 = 0.718 ;

Now we can use the algorithm 4 to calculate N(b). For the {X1,X4,X10}, we can get {X1,X4,X10}/D ={{X1,X4},{X10}}. So h1=2×1=2. And {X1,X4, X10}/{b}={{X1,X10},{X4}}. And {X4} is deleted. h2=2×1/2=1. Therefore N(b)= h1- h2=2-1=1. For the {X2,X12},there is h1=1 and h2=0. So N(b)= N(b)+1=2. For the {X6,X14},{X7,X8,X11}, {X9,X13} and {X3, X5}, according to the step 3.1 of the algorithm 4, they are deleted because their decision attribute values are the same. So

N(b)=2. And R(b)=0.72×2=1.44, U/{a,d,b}= {{X1,X10}}.

For the same reason, there are R(c)=1. 19×2=2.38, U/{a,d,c}={{X2,X12}}; R(e)=1.19×2=2.3,U/{a,d,e}={{X4,X10}};R(f)=1. 24×0=0, U/{a,d,f}={{X1,X4, X10}, {X2,X12}}. According to the step 5 of the algorithm 5, R(c) and R(e) are the maximum. So we can select one of c and e. In here, we select attribute c. So Reduct={a,d,c}. Because U/{a,d,c}={{X2,X12}}, the algorithm go to the step 4. At this time, R(b)=0.72, U/{a,d,c,b}=∅. R(e)=1. 19, U/{a,d,c,e}=∅. R(f)=0, U/{a,d,c,f}={{X2, X12 }}. In this time, we select attribute e. And Reduct={a,d,c,e}. Now U/{a,d,c,e}=∅, so algorithm is end. We can get a reduction of decision table 1 which is Reduct={a,d,c,e}.

5 Conclusions In this paper, for improving the complexity of algorithm for feature selection based on DSM, we proposed a new algorithm with the same idea of the algorithm in [10]. However, the new algorithm is not to store D-matrix and S-matrix, and the space complexity of the new algorithm is O(|U|). So it can drastically cut down the space complexity of the old algorithm with O(|C||U|2) in [10]. On the other hand, the time complexity of the new algorithm with O(|C2||U|)+ O(|C||U|2) is better than that of the old algorithm with O(|C|2|U|2). In the new algorithm, if the core is not calculated, the time complexity of the new algorithm is O(|C2||U|).

152

W. Wu, Z. Xu, and J. Liu

When the objects |U|>>|C|, if the core is not calculated, the algorithm is very good. So how to improve the efficiency of computing the core is our next work. Acknowledgments. This work is supported by the Education Department Foundation of GuangXi, China (200826) and the Scientific Research Foundation of Huanggang Normal University, China (06CB50) and the Doctor Grant of Guangxi Normal University.

References 1. Kudo, M., Sklansky, J.: Comparison of Algorithms that Select Features for Pattern Classifiers. Pattern Recognit. 33, 25–41 (2000) 2. Langley, P.: Selection of Relevant Feature in Machine Learning. In: Proceedings of the AAAI Fall Symposium on Relevance, pp. 140–144. IEEE Press, New York (1994) 3. Liu, H., Setiono, R.: A Probabilistic Approach to Feature Selection-A Filter Solution. In: Proceedings of the 13th International Conference on Machine Learning, pp. 319– 327. IEEE Press, New York (1996) 4. Zhong, N., Dong, J.Z., Ohsuga, S.: Using Rough Sets with Heuristics for Feature Selection. Journal of Intelligent Information Systems 16, 199–214 (2001) 5. Swiniarski, R.W., Skowron, A.: Rough Set Methods in Feature Selection and Recognition. Pattern Recognition Letters 24, 833–849 (2003) 6. Skowron, A., Rauszer, C.: The Discernibility Matrixes and Functions in Information Systems, pp. 331–362. Kluwer Academic Publishers, Dordrecht (1992) 7. Xia, D.L., Yan, P.L.: A New Method of Knowledge Reduction for Information System-DSM Approach. Wuhan University, Wuhan (2001) 8. Jiang, H., Yan, P.L., Xia, D.L.: A New Reduction Algorithm Difference-Similitude Matrix. In: Proceedings of the 2nd International Conference on Machine Learning and Cybernetics, pp. 1533–1537. IEEE Press, Xi’an (2003) 9. Wu, M., Xia, D. L., Yan, P. L.: Difference-Similitude Set Theory. Intelligent Computing: Theory and Applications III, SPIE, Orlando, pp. 1–11. IEEE Press, New York (2005) 10. Wu, M., Yan, P.L.: Feature Selection Based on Difference and Similitude in Data Mining. Wuhan University Journal of Natural Sciences 12, 467–470 (2007) 11. Yang, B.R., Xu, Z.Y., Song, W.: An Efficient Algorithm for Computing Core Based on Positive Region. In: Proceedings of the 2007 International Conference on Artificial Intelligence (ICAI 2007), Las Vegas, Nevada, USA, vol. I, pp. 124–132 (2007)

Exponential Stability of Neural Networks with Time-Varying Delays and Impulses Haydar Ak¸ca, Val´ery Covachev, and Kumud Singh Altmayer

Abstract. We present sufficient conditions for the uniqueness and exponential stability of equilibrium points of impulsive neural networks which are a generalization of Cohen-Grossberg neural networks. Keywords: Neural networks, delays, impulses, stability.

1 Introduction A neural network is a network that performs computational tasks such as associative memory, pattern recognition, optimization, model identification, signal processing, etc. on a given pattern via interaction between a number of interconnected units characterized by simple functions. From the mathematical point of view, an artificial neural network corresponds to a nonlinear transformation of some inputs into certain outputs. Haydar Ak¸ca Mathematical Sciences Department, Faculty of Sciences, United Arab Emirates University, P.O. Box 17551, Al Ain, UAE [email protected] Val´ery Covachev Department of Mathematics & Statistics, College of Science, P.O. Box 36, Sultan Qaboos University, Muscat 123, Sultanate of Oman and Institute of Mathematics, Bulgarian Academy of Sciences, Sofia 1113, Bulgaria [email protected], [email protected] Kumud Singh Altmayer Department of Mathematical and Physical Sciences, Philander Smith College, Little Rock, AR 72202 [email protected] H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 153–163. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

154

H. Ak¸ca, V. Covachev, and K.S. Altmayer

In the present paper we consider a neural networks model described by the following system of differential equations with time-varying delays and impulses: n n d ui (t) = −ai (ui (t)) + wij fj (uj (t)) + vij gj (uj (t − τij (t))) + Ii , dt j=1 j=1

t > t0 ,

t = tk ,

i = 1, n,

ui (tk +) = Jik (ui (tk )),

(1)

i = 1, n,

t0 < t1 < t2 < · · · < tk → ∞

k = 1, 2, 3, . . . ,

as k → ∞,

where n ≥ 2 is the number of neurons in the network; ui (t) is the state potential of the i-th neuron at time t; i = 1, n means i ∈ {1, 2, . . . , n} or i = 1, 2, . . . , n; fj (·) and gj (·) are the normal and the delayed activation functions; wij and vij are the normal and the delayed synaptic connection weights from the j-th neuron on the i-th neuron; Ii are constant external inputs from outside the network to the i-th neuron; the functions ai (·) show how the neuron self-regulates or resets its potential when isolated from other neurons and inputs; τij (t) ≥ 0 are the time-varying transmission delays; tk (k = 1, 2, 3, . . .) are the instants of impulse effect. By a solution of (1) we mean u(t) = (u1 (t), u2 (t), . . . , un (t))T ∈ IRn , in which u(·) is piecewise continuous on [t0 , α) for some α > t0 such that u(tk +) and u(tk −) exist and u(·) is differentiable on the intervals of the form (tk−1 , tk ) ⊂ (t0 , α) and satisfies the differential equations in (1); we assume that u(t) is left continuous with u(tk ) = u(tk −); the left and right limits of u(·) are related by the impulse operators Jk : IRn → IRn , Jk (u) = (J1k (u1 ), J2k (u2 ), . . . , Jnk (un ))T . The model (1) is a generalization of Cohen-Grossberg neural networks first proposed by Cohen and Grossberg [3]. Special cases of this model are Hopfield-type neural networks with time-varying delays [5], cellular neural networks with time-varying delays [7, 8, 9, 10] and bi-directional associative memory neural networks with discrete delays [2]. Throughout the present paper, we will use the following assumptions. H1. The functions ai : IR → IR are continuous and there exist constants λi > 0 such that (x1 − x2 )[ai (x1 ) − ai (x2 )] ≥ λi (x1 − x2 )2 for all x1 , x2 ∈ IR and i = 1, n. H2. The transfer functions fi and gi are continuous and monotonically increasing. H3. The delays τij (t) are bounded, that is, there exists a constant b such that 0 ≤ τij ≤ b for all t = tk and i, j = 1, n. H4. The number i(t0 , t) = max{k ∈ IN : tk < t} of instants of impulse effect between t0 and t satisfies

Exponential Stability of Neural Networks

lim sup t→+∞

155

i(t0 , t) = p < +∞ t

and the impulse functions Jik are monotonically increasing. In our previous paper [1] ai (ui ) = ai ui , where ai (i = 1, n) are positive constants, fi ≡ 0, gi are Lipschitz continuous and monotonically increasing. Here the Lipschitz continuity of fi and gi will be replaced by a weaker property. However, in [1] we were able to obtain conditions for the existence of an equilibrium point of (1). Here we have to assume the existence and uniqueness of the solution of the initial value problem for (1), and the existence of an equilibrium point for (1). This paper is organized as follows. In Sect. 2, we introduce a more general time-delay impulsive system (2), (3) and present the necessary notations and concepts of the stability analysis of this system. Then we present sufficient conditions for the uniqueness and stability of equilibrium points of the impulsive system (2), (3). Finally, in Sect. 3 we apply the results obtained to the neural network with time-varying delays and impulses (1).

2 Notations and Preliminaries Here we follow [5, 6] adapting the approach expounded therein to impulsive systems. In order to study the stability analysis of the general time delay system we rewrite system (1) as du(t) = F (u(t)) + G(uτ (t)), t > t0 , t = tk , dt u(tk +) = Jk (u(tk )), k = 1, 2, 3, . . . ,

(2) (3)

where Jk (u) = (J1k (u1 ), . . . , Jnk (un ))T and the operators Jik satisfy the condition H4, u(t) = (u1 (t), u2 (t), . . . , un (t))T is the state vector of the neural network, both F and G are continuous mappings from an open subset Ω of IRn into IRn , and G(uτ (t)) is defined as G(u) = (G1 (u), G2 (u), . . . , Gn (u))T and Gi (uτ (t)) = Gi (u1 (t − τi1 (t)), u2 (t − τi2 (t)), . . . , un (t − τin (t))), τij (i, j = 1, n) are the delays which satisfy the condition H3. Also, we suppose that the initial value problem for (2), (3) has a unique solution. Let IRn be the n-dimensional real vector space with vector norm  ·  n  defined as follows: If x = (x1 , x2 , . . . , xn )T ∈ IRn , then x = |xi |. i=1

Definition 1. Let Ω ⊂ IRn , f : Ω → IRn and x0 ∈ Ω. The minimal Lipschitz constant of f with respect to x0 is defined by

156

H. Ak¸ca, V. Covachev, and K.S. Altmayer

LΩ (f, x0 ) = inf {α > 0 : f (x) − f (x0 ) ≤ αx − x0 , x ∈ Ω}. Definition 2. Let Ω ⊂ IRn , f : Ω → IRn and x0 ∈ Ω. Then the constant mΩ (f, x0 ) =

f (x) − f (x0 ), sgn (x − x0 ) x − x0  x∈Ω\{x0 } sup

is called the relative nonlinear measure of f with respect to x0 on Ω. Here ·, · is the inner product in IRn and sgn (x) = (sgn (x1 ), sgn (x2 ), . . . , sgn (xn ))T . Remark 1. If f (x) = (f1 (x1 ), f2 (x2 ), . . . , fn (xn ))T and fi (i = 1, n) are monotonically increasing, then mΩ (f, x0 ) =

f (x) − f (x0 ) . x − x0  x∈Ω\{x0 } sup

We will also need the following simple assertion from calculus. Lemma 1. [5] If a > c ≥ 0, then, for each nonnegative real number b, the equation λ − a + ceλb = 0 has a unique positive solution. In fact, the left-hand side of this equation is a strictly monotonic function for λ ≥ 0, which is nonnegative at λ = a and negative at λ = 0. If u∗ is an equilibrium point of system (2), then it satisfies the equation F (u∗ ) + G(u∗ ) = 0.

(4)

If u∗ is an equilibrium point of the impulsive system (2), (3), then it satisfies (4) and u∗ = Jk (u∗ ), k = 1, 2, 3, . . . , that is, u∗ is a fixed point of the impulse operators Jk . Definition 3. The time-delay impulsive system (2), (3) is said to be exponentially stable on a neighbourhood Ω of an equilibrium point u∗ if there are two positive constants α and M such that u(t) − u∗  ≤ M e−α(t−t0 )

sup

t0 −b≤s≤t0

u0 (s) − u∗ ,

t ≥ t0 ,

where u(t) is the unique trajectory of the system initiated from u0 (s) ∈ Ω with s ∈ [t0 − b, t0 ]. Proposition 1. Let u∗ ∈ Ω be an equilibrium point of system (2). If mΩ (F + G, u∗ ) < 0, then u∗ is the unique equilibrium point in Ω.

Exponential Stability of Neural Networks

157

Proof. Suppose that x∗ is another equilibrium point of system (2) in Ω. Then both u∗ and x∗ satisfy equation (4) and 0 > mΩ (F + G, u∗ ) = ≥

F (u) + G(u) − F (u∗ ) − G(u∗ ), sgn(u − u∗ ) u − u∗  u∈Ω\{u∗ } sup

F (x∗ ) + G(x∗ ), sgn(x∗ − u∗ ) = 0, x∗ − u∗ 

which is a contradiction. Let us introduce some notation: d = sup mΩ (Jk , u∗ ), k∈IN

D = max { sup mΩ (Jk−1 , u∗ ), 1}, k∈IN

ν = sup (i(t0 , t) − i(t0 , t − b)) . t>t0

From condition H4 it follows that ν < ∞. Proposition 2. Let Ω ⊂ IRn be a neighbourhood of an equilibrium point u∗ of the system (2), (3). If D < ∞ and for some matrix A = diag(a1 , a2 , . . . , an ) with ai > 0 we have mA−1 (Ω) (F A, A−1 u∗ ) + Dν LA−1 (Ω) (GA, A−1 u∗ ) < 0,

(5)

then by Lemma 1 the equation λ min ai + mA−1 (Ω) (F A, A−1 u∗ ) + Dν LA−1 (Ω) (GA, A−1 u∗ )ebλ = 0

(6)

i=1,n

has a unique positive solution λ. If d < eλ/p ,

(7)

that is, p ln d < λ, then system (2), (3) is exponentially stable on Ω. More ˜ ∈ (0, λ − p ln d) there exists a constant M such that precisely, for any λ ˜

u(t) − u∗  ≤ M e−λ(t−t0 )

sup

t0 −b≤s≤t0

u0 (s) − u∗ 

for all t ≥ t0 .

(8)

Proof. For any vector w ∈ IRn we have w = w, sgn(w) and w ≥ w, sgn(z) for all z ∈ IRn . Therefore for any s ∈ IR, s > 0 we have 1 u(t) − u∗  − u(t − s) − u∗  ≤ u(t) − u(t − s), sgn(u(t) − u∗ ). s s

158

H. Ak¸ca, V. Covachev, and K.S. Altmayer

So, from system (2) we have du(t) du(t) − u∗  ≤ , sgn(u(t) − u∗ ) dt dt = F (u(t)) + G(uτ (t)), sgn(u(t) − u∗ ) = F (u(t)) − F (u∗ ), sgn(u(t) − u∗ ) + G(uτ (t)) − G(u∗ ), sgn(u(t) − u∗ ) = F A(A−1 u(t)) − F A(A−1 u∗ ), sgn(A−1 u(t) − A−1 u∗ ) + GA(A−1 uτ (t)) − GA(A−1 u∗ ), sgn(A−1 u(t) − A−1 u∗ ) ≤ mA−1 (Ω) (F A, A−1 u∗ )A−1 (u(t) − u∗ ) + LA−1 (Ω) (GA, A−1 u∗ )

A−1 (u(s) − u∗ )

sup t−b≤s≤t

* ≤ mA−1 (Ω) (F A, A−1 u∗ )u(t) − u∗  + LA−1 (Ω) (GA, A−1 u∗ )

u(s) − u∗ 

sup t−b≤s≤t

+# min ai

$−1

.

i=1,n

By virtue of (5), by Halanay’s inequality [4] and taking into account the presence of impulses, we have u(t) − u∗  ≤ e−λ(t−t0 ) di(t0 ,t)

sup

t0 −b≤s≤t0

u(s) − u∗ ,

(9)

where λ is the unique positive solution of equation (6). In fact, for t ∈ [t0 , t1 ] by Halanay’s inequality we derive u(t) − u∗  ≤ e−λ(t−t0 )

sup

t0 −b≤s≤t0

u(s) − u∗ .

Now let t ∈ (tk , tk+1 ] for some k ∈ IN. In order to apply Halanay’s inequality, we extend u(t) as a continuous function from (tk , tk+1 ] back to t0 − b as follows: ⎧ ⎪ u(t), t ∈ (tk , tk+1 ], ⎨ v(t) = Jk Jk−1 · · · J u(t), t ∈ (t−1 , t ],  = 2, k, ⎪ ⎩ Jk Jk−1 · · · J1 u(t), t ∈ [t0 − b, t1 ]. If tk−μ−1 < t − b ≤ tk−μ < · · · < tk < t, then sup

u(s) − u∗  ≤ Dμ

t−b≤s≤t

sup

v(s) − u∗ 

t−b≤s≤t

and μ ≤ ν. Thus we have dv(t) − u∗  * ≤ mA−1 (Ω) (F A, A−1 u∗ )v(t) − u∗  dt $−1 +# + Dν LA−1 (Ω) (GA, A−1 u∗ ) sup v(s) − u∗  min ai . t−b≤s≤t

i=1,n

Exponential Stability of Neural Networks

159

Now from Halanay’s inequality we have v(t) − u∗  ≤ e−λ(t−t0 )

sup

t0 −b≤s≤t0

v(s) − u∗ .

To derive (9) it suffices to notice that v(s) − u∗  ≤ dk u(s) − u∗  for s ∈ [t0 − b, t0 ]. Let ε > 0 be such that λ − (p + ε) ln d > 0. Then i(t0 , t) ≤ (p + ε)(t − t0 ) for all t large enough and there exists a constant M ≥ 1 such that i(t, t0 ) ≤ (p + ε)(t − t0 ) + ln M/ ln d for all t ≥ t0 . Then di(t0 ,t) ≤ M exp{[(p + ε) ln d](t − t0 )} ˜ = λ − (p + ε) ln d. and the desired estimate (8) follows with λ

3 Main Results Now, extending some results of [1, 5, 6], we present some sufficient conditions for uniqueness and exponential stability of the equilibrium of the impulsive network (1). In order to apply Propositions 1 and 2, we define F, G : IRn → n n   IRn by Fi (u) = −ai (ui ) + wij fj (uj ) and Gi (u) = vij gj (uj ) + Ii . For j=1

j=1

Ω ⊂ IRn we denote by Ωi the projection of Ω on the i-th axis of IRn . Theorem 1. Let Ω be a neighbourhood of an equilibrium point u∗ = (u∗1 , u∗2 , . . . , u∗n )T of system (1), mi = mΩi (fi , u∗i ) and Mi = mΩi (gi , u∗i ). If ri (i = 1, n) are positive real numbers such that ⎫ ⎧ n n ⎬ 1 ⎨ rj r j |wji | + Dν Mi |vji | < 1, (10) max mi ⎭ r r i=1,n λi ⎩ j=1 i j=1 i then the equilibrium point u∗ of system (1) is unique in Ω. Proof. For each i = 1, n the transfer functions fi and gi are increasing, or equivalently   fi (t) − fi (s) sgn(t − s) = |fi (t) − fi (s)|,   gi (t) − gi (s) sgn(t − s) = |gi (t) − gi (s)| for all t, s ∈ R. Moreover, from condition H1 we have % %   ai (t) − ai (s) sgn(t − s) = %ai (t) − ai (s)% ≥ λi |t − s|.

160

H. Ak¸ca, V. Covachev, and K.S. Altmayer

An equilibrium point u∗ of system (1) corresponds to a solution of the equation (4). Let us suppose that u and u∗ are two distinct solutions of (4) in Ω. Then for R = diag(r1 , r2 , . . . , rn ) we have   0 = R F (u) + G(u) − F (u∗ ) − G(u∗ ) , sgn(u − u∗ ) ⎧ n n ⎨     = ri − ai (ui ) − ai (u∗i ) + wij fj (uj ) − fj (u∗j ) ⎩ i=1 j=1 ⎫ n ⎬   vij gj (uj ) − gj (u∗j ) sgn(ui − u∗i ) + ⎭ j=1 ⎧ n n ⎨ % % % %  |wij |%fj (uj ) − fj (u∗j )% ri −%ai (ui ) − ai (u∗i )% + ≤ ⎩ i=1 j=1 ⎫ ⎬ % %  + |vij |%gj (uj ) − gj (u∗j )% ⎭ ≤−

n

ri λi |ui − u∗i | +

i=1

=−

n

n n   ri |wij |mj |uj − u∗j | + |vij |Mj |uj − u∗j | j=1 i=1

ri λi |ui −

u∗i |

+

i=1

n j=1

&

mj

n

ri |wij | + Mj

i=1

n

' ri |vij | |uj − u∗j |

i=1

⎫ ⎬ rj |wji | − Mj rj |vji | |ui − u∗i | =− ri λi − mi ⎭ ⎩ i=1 j=1 j=1 ⎧ ⎤⎫ ⎡ n n n ⎬ ⎨ 1 ⎣ rj rj ri λi 1 − |wji | + Mj |vji |⎦ |ui − u∗i | < 0. =− mi ⎭ ⎩ λi r r i=1 j=1 i j=1 i ⎧ n ⎨

n

n

As in Proposition 1, the contradiction obtained proves the uniqueness of the equilibrium point u∗ of system (1) in Ω. Theorem 2. Let all assumptions of Theorem 1 hold. Suppose further that the unique positive solution λ of the equation 1 − 1 + Dν qeλb = 0 i=1,n pi

λ min with

pi = λi − mi

n rj j=1

ri

|wji |

Exponential Stability of Neural Networks

161

⎫ ⎧ n ⎬ ⎨M rj i q = max |vji | ⎭ r i=1,n ⎩ pi j=1 i

and

satisfies (7). If u(t) is the trajectory of system (1) initiated from u0 (s) ∈ Ω with s ∈ [t0 − b, t0 ], then max ri ∗

u(t) − u  ≤ M e

i=1,n ˜ −λ(t−t 0)

min ri i=1,n

sup

t0 −b≤s≤t0

u0 (s) − u∗ ,

(11)

˜ ∈ (0, λ − p ln d). where λ Proof. We can first note that by virtue of Theorem 1 the equilibrium of system (1) is unique. By the change x = Ru system (1) takes the form ( d −1 x(t)) + RG(R−1 xτ (t)), t = tk , dt x(t) = RF (R (12) x(tk +) = RJk (R−1 x(tk )), k = 1, 2, 3, . . . It is easy to see that x∗ = Ru∗ is an equilibrium point of system (12) and mR(Ω) (RJk R−1 , Ru∗ ) = mΩ (Jk , u∗ ) ≤ d. Denote P = diag(p1 , p2 , . . . , pn ). From inequality (10) it follows that pi > 0, i = 1, n. As in Theorem 1, for all x ∈ P R(Ω) we have RF (R−1 P −1 x) − RF (u∗ ), sgn(x − P Ru∗ ) ⎫ ⎧ n n ⎬ ⎨ % % % % ∗ % ∗ % ≤ ri −%ai (ri−1 p−1 |wij |%fj (rj−1 p−1 i xi ) − ai (ui ) + j xj ) − fj (uj ) ⎭ ⎩ i=1 j=1 ⎫ ⎧ n n ⎬ ⎨ λ % % % % m i % j %xj − pj rj u∗j % xi − pi ri u∗i % + ≤ ri − |wij | ⎭ ⎩ ri pi rj pj i=1 j=1 ⎞ ⎛ n n % % r j ⎝λi − mi =− p−1 |wji |⎠ %xi − pi ri u∗i % i r i=1 j=1 i =−

n

% % ∗% ∗ % p−1 i pi xi − pi ri ui = −x − P Ru 

i=1

and thus

mP R(Ω) (RF R−1 P −1 , P Ru∗ ) ≤ −1.

(13)

162

H. Ak¸ca, V. Covachev, and K.S. Altmayer

Further on, for all x ∈ P R(Ω) % % % n % n % %   −1 −1 ∗ % %ri g v (r p x ) − g (u ) RG(R−1 P −1 x) − RG(u∗ ) = ij j j j j j % j % % i=1 % j=1 n n % % % % Mi rj %xj − pj rj u∗ % = |vij |Mj rj−1 p−1 |vji |%xi − pi ri u∗i % j j pi j=1 ri i=1 j=1 i=1 ⎫ ⎧ n n ⎬ ⎨M % % rj i %xi − pi ri u∗ % = qx − P Ru∗ , ≤ max |vji | i ⎭ ri i=1,n ⎩ pi

≤

n

ri

n

j=1

i=1

which implies that LP R(Ω) (RGR−1 P −1 , P Ru∗ ) ≤ q.

(14)

From inequalities (13) and (14) we deduce mP R(Ω) (RF R−1 P −1 , P Ru∗ ) + Dν LP R(Ω) (RGR−1 P −1 , P Ru∗ ) ≤ −1 + Dν q ) ') ( ( & n n Mi rj rj ν −1 ν = −1 + D max |vji | = max pi |vji | −pi + D Mi pi j=1 ri ri i=1,n i=1,n j=1 ( & ') n n rj rj −1 ν = max pi mi <0 |wji | + D Mi |vji | − λi ri ri i=1,n j=1 j=1

in view of inequality (10). Thus we can apply Proposition 2 to system (12) with A = P −1 and deduce the estimate ˜

x(t) − Ru∗  ≤ M e−λ(t−t0 )

sup

t0 −b≤s≤t

x0 (s) − Ru∗ 

for all t ≥ t0 . Since x = Ru, this yields estimate (11). Acknowledgements. The authors express their gratitude to the reviewer whose suggestions allowed them to considerably improve the quality of the paper, in particular, the proof of Proposition 2.

References 1. Ak¸ca, H., Alassar, R., Covachev, V.: Stability of Neural Networks with Time Varying Delays in the Presence of Impulses. Adv. Dyn. Syst. Appl. 1, 1–17 (2006) 2. Chen, A.P., Cao, J.D., Huang, L.H.: Exponential Stability of BAM Neural Networks with Transmission Delays. Neurocomputing 57, 435–454 (2004)

Exponential Stability of Neural Networks

163

3. Cohen, M., Grossberg, G.: Absolute Stability and Global Pattern Formation and Parallel Memory Storage by Competitive Neural Networks. IEEE Trans. Systems Man Cybernet. 13, 815–821 (1983) 4. Driver, R.D.: Ordinary and Delay Differential Equations. Springer, New York (1977) 5. Peng, J., Qiao, H., Xu, Z.: A New Approach to Stability of Neural Networks with Time-Varying Delays. Neural Networks 15, 95–103 (2002) 6. Song, X., Peng, J.: Stability of a Class of Time-Delayed Neural Networks with Non-Lipschitz Continuous Activation Functions. Neurocomputing (submitted) 7. Zhang, Q., Wei, X.P., Xu, J.: Global Exponential Convergence Analysis of Delayed Neural Networks with Time-Varying Delays. Phys. Lett. A 318, 537– 544 (2003) 8. Zhang, Q., Wei, X.P., Xu, J.: Global Asymptotic Stability Analysis of Neural Networks with Time-Varying Delays. Neural Process. Lett. 21, 61–71 (2005) 9. Zhang, Q., Wei, X.P., Xu, J.: Delay-Dependent Exponential Stability of Cellular Neural Networks with Time-Varying Delays. Chaos Solitons Fractals 23, 1363– 1369 (2005) 10. Zhou, D.M., Cao, J.D.: Globally Exponential Stability Conditions for Cellular Neural Networks with Time-Varying Delays. Appl. Math. Comput. 131, 487– 496 (2002)

Adaptive Higher Order Neural Networks for Effective Data Mining Shuxiang Xu and Ling Chen*

Abstract. A new adaptive Higher Order Neural Network (HONN) is introduced and applied in data mining tasks such as determining automobile yearly losses and edible mushrooms. Experiments demonstrate that the new adaptive HONN model offers advantages over conventional Artificial Neural Network (ANN) models such as higher generalization capability and the ability in handling missing values in a dataset. A new approach for determining the best number of hidden neurons is also proposed. Keywords: Neural network, Higher order neural network, Adaptive activation function, Data mining.

1 Introduction Data mining tools can answer many business questions that traditionally were too time-consuming to resolve. They search databases for hidden patterns, finding predictive information that business experts may overlook because it lies outside their expectations. An example of pattern discovery is the analysis of retail sales data to identify seemingly unrelated products that are often purchased together. Another pattern discovery example is detecting fraudulent credit card transactions from collected data [1, 2]. Data mining techniques can be implemented rapidly on existing software and hardware platforms to enhance the value of existing information resources. When implemented on high performance client/server or parallel processing systems, data mining tools can analyze massive databases to deliver answers to questions such as, "Which customers are most likely to buy this new product, and why?" [3, 4]. Data mining is usually supported by the following technologies: massive data collection, powerful multiprocessor computers, and Shuxiang Xu School of Computing and IS, University of Tasmania, Launceston, Tasmania 7250, Australia [email protected] *

Ling Chen Information Services, Department of Health and Human Services, Hobart, Tasmania 7000, Australia H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 165–173. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

166

S. Xu and L. Chen

data mining algorithms. Whilst data collection is a relatively easy task and powerful computers as well as distributed systems are readily available, commonly used data mining algorithms include Artificial Neural Networks (ANNs), Decision Trees, Rule Induction, Nearest Neighbor Classification, and Cluster Analysis [2, 5]. Data mining is often applied to two separate processes: knowledge discovery and prediction. Knowledge discovery provides explicit information that has a readable form and can be understood by users. Forecasting, or predictive modeling provides predictions of future events and may be transparent and readable in some approaches (e.g. Rule Induction), but opaque in others such as ANNs. However, several recent research findings have shown that it’s possible to retrieve transparent rules from certain ANNs [35]. This paper addresses using ANNs for data mining, for the following reasons. First, although usually considered a black-box approach, ANNs are a natural technology for data mining. ANNs are non-linear models that resemble biological neural networks in structure and learn through training. ANNs present a model based on the massive parallelism and the pattern recognition and prediction abilities of the human brain. ANNs learn from examples in a way similar to how the human brain learns. Then ANNs take complex and noisy data as input and make educated guesses based on what they have learned from the past, like what the human brain does. Next, ANNs (especially higher order ANNs) are able to handle incomplete or noisy data [6, 7]. Finally, ANNs may hold superior predictive capability, compared with other data mining approaches [8-11]. While conventional ANN models have been able to bring huge profits to many businesses, they suffer from several drawbacks. First, conventional ANN models do not perform well on handling incomplete or noisy data [6, 7, 12]. Next, conventional ANNs can not deal with discontinuities (which contrasts with smoothness: small changes in inputs produce small changes in outputs) in the input training data set [8, 10, 13]. Finally, conventional ANNs lack capabilities in handling complicated business data with high order nonlinearity [10, 13]. To overcome these limitations some researchers have proposed the use of Higher Order Neural Networks (HONNs) [14, 15]. HONNs are networks in which the net input to a computational neuron is a weighted sum of products of its inputs (instead of just a weighted sum of its inputs, as with conventional ANNs). In [8] HONN models have been used in several business applications. The results demonstrate significant advantages of HONNs over conventional ANNs such as much reduced network size, faster training, as well as reduced forecasting errors. In [16] HONNs are used for data clustering which offer significant improvement when compared to the results obtained from using self-organising maps. In [17] global exponential stability and exponential convergence issues of HONNs are studied. In [10], HONNs have been used for dealing with non-linear and discontinuous financial time-series data, and are able to offer roughly twice the performance of conventional ANNs on financial time-series prediction. [13] employs HONNs for financial data automodeling. Their algorithms are further shown to be capable of automatically finding an optimum model, given a specific application. In [18] a HONN model is applied to the classification into age-groups of abalone shellfish, a difficult benchmark to which previous researchers have tried to handle using different ANN architectures.

Adaptive Higher Order Neural Networks for Effective Data Mining

167

Adaptive HONNs are HONNs with adaptive activation functions. Such activation functions are adaptive because there are free parameters in the activation functions which can be adjusted (in the same way as connection weights) to adapt to different problems. In [24], an adaptive activation function is built as a piecewise approximation with suitable cubic splines that can have arbitrary shape and allows them to reduce the overall size of the neural networks, trading connection complexity with activation function complexity. In [25], real variables a (gain) and b (slope) in the generalized sigmoid activation function are adjusted during learning process. A comparison with classical ANNs to model static and dynamical systems is reported, showing that an adaptive sigmoid (ie, a sigmoid with free parameters) leads to an improved data modeling. In this paper, Section 2 proposes a new adaptive HONN model with an adaptive activation function. Section 3 addresses the issue of optimizing the number of hidden layer neurons, one of the key issues yet to be resolved. Section 4 gives experiments to justify our new adaptive HONN model. Section 5 offers a summary of this paper and possible directions for further work in the future.

2 Adaptive HONNs HONNs were first introduced by [14]. The network structure of a three input second order HONN is shown below:

Fig. 1 Left, MLP (multi-layer perceptron) with three inputs and two hidden nodes; Right, second order HONN with three inputs

Adaptive HONNs are HONNs with adaptive activation functions. The network structure of an adaptive HONN is the same as that of a multi-layer ANN. That is,

168

S. Xu and L. Chen

it consists of an input layer with some input units, an output layer with some output units, and at least one hidden layer consisting of intermediate processing units (see next section on the number of hidden units). We will only use one hidden layer as it has been mathematically proved that ANNs with one hidden layer is a universal approximator [26]. Usually there is no activation function for neurons in the input layer and the output neurons are summing units (linear activation), the activation function in the hidden units is an adaptive one. Our adaptive activation function has been defined as the following:

Ψ( x ) = A1⋅ sin (B1⋅ x ) + A2 ⋅ e − B 2⋅x

2

(1)

where A1, B1, A2, B2, are real variables which will be adjusted (as well as weights) during training. Justification of the use of free parameters in the neuron activation function (2.1) can be found in [8, 25, 34]. In our experiments (Section 4) we use an HONN learning algorithm that is based on an improved steepest descent rule [8] to adjust the free parameters in the above adaptive activation function (as well as connection weights between neurons). We will see that such approach provides more flexibility and better data mining ability compared with more traditional approaches.

3 Optimizing the Number of Hidden Units for HONNs Optimizing the number of hidden layer neurons for an ANN to solve a practical problem remains one of the unsolved tasks in this research area. Setting too few hidden units causes high training errors and high generalization errors due to under-fitting, while too many hidden units results in low training errors but still high generalization errors due to over-fitting. It is argued that the best number of hidden units depends in a complex way on: the numbers of input and output units, the number of training cases, the amount of noise in the targets, the type of hidden unit activation function, the training algorithm, etc [27]. A dynamic node creation algorithm for ANNs is proposed in [28]. [29] proposes an approach which is similar to [28] but removes nodes when small error values are reached. In [30] an algorithm is developed to optimize the number of hidden nodes by minimizing the mean-squared errors over noisy training data. In this paper we propose an approach for determining the best number of hidden nodes based on [31], which reports that, using ANNs for function approximation, the rooted mean squared (RMS) error between the well-trained neural network and a target function f is shown to be bounded by ⎛ C 2f O⎜ ⎜ n ⎝

⎞ ⎟ + O⎛⎜ nd log N ⎞⎟ ⎟ ⎠ ⎝N ⎠

(2)

where n is the number of hidden nodes, d is the input dimension of the target function f, N is the number of training pairs, and Cf is the first absolute moment of the Fourier magnitude distribution of the target function f. The two important points of (2) are the approximation error and the estimation error between the

Adaptive Higher Order Neural Networks for Effective Data Mining

169

well-trained neural network and the target function. For this research we are interested in the approximation error which refers to the distance between the target function and the closest neural network function of a given architecture (which represents the simulated function). To this point, [31] mathematically proves that, with n ~ Cf (N/(d log N))1/2 nodes, the order of the bound on the RMS error is optimized to be O(Cf ((d/N) log N)1/2). Based on the above result, we can conclude that if the target function f is known then the best number of hidden layer nodes (which leads to a minimum RMS error) is n = Cf (N/(d log N))1/2

(3)

Note that the above equation is based on a known target function f. However, in most practical cases the target function f is not known, instead, we are usually given a series of training input-output pairs. In these cases, [31] suggests that the number of hidden nodes may be optimized from the observed data (training pairs) by the use of a complexity regularization or minimum description length criterion [32]. This is a criterion which reflects the trade-off between residual error and model complexity and determines the most probable model (in this research, the HONN with the best number of hidden nodes). Based on this, when f is unknown we use a complexity regularization approach to determine the constant C in the following n = C (N/(d log N))1/2

(4)

The approach is to try an increasing sequence of C to obtain different number of hidden nodes, train an ANN for each number of hidden nodes, and then observe the n which generates the smallest RMS error (and note the value of the C). The maximum of n has been proved to be N/d [32]. Please note the difference between the equation (3) and the equation (4): in (3), Cf depends on a known target function f, which is usually unknown (so (3) is only a theoretical approach), whereas in our approach as shown in (4), C is a constant which does not depend on any function. Based on our experiments conducted so far we have found that for a small or medium-sized dataset (with less than 10000 training pairs), when N/d is less than or close to 30, the optimal n most frequently occurs on its maximum, however, when N/d is greater than 30, the optimal n is close to the value of (N/(d log N))1/2.

4 Adaptive HONN Experiments Our first experiment is to use our adaptive HONN to process the Automobile Dataset from the UCI Machine Learning Repository [33]. This dataset is made of 205 instances, with 25 attributes (inputs) and 1 class attribute (output). This data set consists of three types of entities: (a) the specification of an auto in terms of various characteristics, (b) its assigned insurance risk rating, and (c) its normalized losses in use as compared to other cars. The second rating corresponds to the degree to which the auto is more risky than its price indicates. Cars are initially assigned a risk factor symbol associated with its price. Then, if it is more risky (or less), this symbol is

170

S. Xu and L. Chen

adjusted by moving it up (or down) the scale. Actuarians call this process "symboling". A value of +3 indicates that the auto is risky, -3 that it is probably pretty safe. The third factor is the relative average loss payment per insured vehicle year. This value is normalized for all autos within a particular size classification (two-door small, station wagons, sports, etc...), and represents the average loss per car per year. This third factor is considered the output attribute. The 26 attributes are: Attribute: Attribute Range. 1. symboling: -3, -2, -1, 0, 1, 2, 3. 2. normalized-losses: continuous from 65 to 256. 3. make: alfa-romero, audi, bmw, chevrolet, dodge, honda, isuzu, jaguar, mazda, mercedes-benz, mercury, mitsubishi, nissan, peugot, plymouth, porsche, renault, saab, subaru, toyota, volkswagen, Volvo. 4. fuel-type: diesel, gas. 5. aspiration: std, turbo. 6. num-of-doors: four, two. 7. body-style: hardtop, wagon, sedan, hatchback, convertible. 8. drive-wheels: 4wd, fwd, rwd. 9. engine-location: front, rear. 10. wheel-base: continuous from 86.6 120.9. 11. length: continuous from 141.1 to 208.1. 12. width: continuous from 60.3 to 72.3. 13. height: continuous from 47.8 to 59.8. 14. curb-weight: continuous from 1488 to 4066. 15. engine-type: dohc, dohcv, l, ohc, ohcf, ohcv, rotor. 16. num-of-cylinders: eight, five, four, six, three, twelve, two. 17. engine-size: continuous from 61 to 326. 18. fuel-system: 1bbl, 2bbl, 4bbl, idi, mfi, mpfi, spdi, spfi. 19. bore: continuous from 2.54 to 3.94. 20. stroke: continuous from 2.07 to 4.17. 21. compression-ratio: continuous from 7 to 23. 22. horsepower: continuous from 48 to 288. 23. peak-rpm: continuous from 4150 to 6600. 24. city-mpg: continuous from 13 to 49. 25. highway-mpg: continuous from 16 to 54. 26. price: continuous from 5118 to 45400. There are missing values in this dataset. Based on our new approach, the optimal number of hidden layer neurons for this experiment is n=8. For this experiment, the data set is divided into a training set made of 85% of the original set and a test set made of 15% of the original set. A validation set is not needed because the optimal number of hidden layer neurons has been determined. After the adaptive HONN (with 8 hidden layer units) has been well trained over the training data pairs, it is used to forecast over the test set. The correctness rate reaches 96.1%. To verify that for this example the optimal number of hidden layer neuron is 8, we try to apply the same procedure by setting the number of hidden layer neurons to 7, 9, and 11, which results in correctness rates of 81.2%, 83.2%, and 79.3% on the test set, respectively.

Adaptive Higher Order Neural Networks for Effective Data Mining

171

To verify the advantages of our adaptive HONN model we establish a conventional ANN with the sigmoid activation function (and one hidden layer) for the same experiments. With 8 hidden neurons and the same training set, the conventional ANN reaches a correctness rate of only 81.4% on the test set. After we change the number of hidden neurons to 7, 9, and 11, the correctness rates obtained are 70.5%, 72.1%, and 69.1%, respectively. These results seem to suggest that the adaptive HONN model holds better capability in generalization and in handling datasets with missing values. For our second experiment, a Mushroom Data Set is considered [33]. This data set includes descriptions of hypothetical samples corresponding to 23 species of gilled mushrooms in the Agaricus and Lepiota Family. Each species is identified as definitely edible, definitely poisonous, or of unknown edibility and not recommended. This latter class was combined with the poisonous one. There are 8124 instances with 22 attributes. There are missing values in this data set. Based on our approach, the optimal number of hidden layer neurons for this experiment is n=10. For this experiment, the data set is divided into a training set made of 80% of the original set and a test set made of 20% of the original set. Again a validation set is not needed because the optimal number of hidden layer neurons has been determined. After the adaptive HONN (with 10 hidden layer units) has been well trained over the training data pairs, it is used to forecast over the test set. The correctness rate reaches 95.3%. To verify that for this example the optimal number of hidden layer neuron is 10, we try to apply the same procedure by setting the number of hidden layer neurons to 9, 11, and 14, which results in correctness rates of 82.1%, 84.5%, and 78.8.9% on the test set, respectively. To verify the advantages of the proposed adaptive HONN model we use a conventional ANN with the sigmoid activation function (and one hidden layer) for the same experiments. With 10 hidden neurons and the same training set, the conventional ANN reaches a correctness rate of only 77.1% on the test set. After we change the number of hidden neurons to 9, 11, and 14, the correctness rates obtained are 70.5%, 72.1%, and 62.3%, respectively. These results also confirm that the adaptive HONN model holds better capability in generalization and in handling datasets with missing values, compared with conventional ANNs.

5 Summary and Discussions In this paper a new adaptive HONN model is introduced and applied in data mining tasks such as predicting average yearly losses for automobiles and determining edibility of mushrooms. Such model offers significant advantages over conventional ANNs such as more accurate predictions, and the ability of handling missing values in a dataset. A new approach for determining the best number of hidden nodes has been proposed. For future work, it would be a good idea to extend the research to involve large applications which contain training datasets of over 10000 input-out pairs. Further comparison studies between our adaptive HONN and other ANN approaches should also be conducted to demonstrate the advantages and disadvantages of each method.

172

S. Xu and L. Chen

References 1. Cios, K.J., Pedrycz, W., Swiniarski, R.W., Kurgan, L.A.: Data Mining: A Knowledge Discovery Approach. Springer, Heidelberg (2007) 2. Han, J., Kamber, M.: Data Mining: Concepts and Techniques. Elsevier, Amsterdam (2006) 3. Masseglia, F., Poncelet, P., Teisseire, M.: Successes and New Directions in Data Mining. Information Science Reference (2007) 4. Witten, I.H., Frank, E.: Data Mining: Practical Machine Learning Tools and Techniques. Elsevier/Morgan Kaufman (2005) 5. Bramer, M.: Principles of Data Mining. Springer, Heidelberg (2007) 6. Peng, H., Zhu, S.: Handling of incomplete data sets using ICA and SOM in Data Mining. Neural Computing & Applications 16, 167–172 (2007) 7. Wang, S.H.: Application Of Self-Organising Maps for Data Mining with Incomplete Data Sets. Neural Computing & Applications 12, 42–48 (2003) 8. Xu, S.: Adaptive Higher Order Neural Network Models and Their Applications in Business. In: Zhang, M. (ed.) Artificial Higher Order Neural Networks for Economics and Business, ch. XIV. IGI Global (2008) 9. Zhang, M., Xu, S.X., Fulcher, J.: ANSER: an Adaptive-Neuron Artificial Neural Network System for Estimating Rainfall Using Satellite Data. International Journal of Computers and Applications 29, 215–222 (2007) 10. Fulcher, J., Zhang, M., Xu, S.: Application of Higher-Order Neural Networks to Financial Time-Series Prediction. In: Artificial Neural Networks in Finance and Manufacturing, ch. V (2006) 11. Kohonen, T., Kaski, S., Lagus, K., Salojarvi, J., Honkela, J., Paatero, V., Saarela, A.: Self Organization Of A Massive Document Collection. IEEE Trans. Neural Networks 11, 574–585 (2000) 12. Dong, G., Pei, J.: Sequence Data Mining (Advances in Database Systems). Springer, Heidelberg (2007) 13. Zhang, M., Xu, S.X., Fulcher, J.: Neuron-Adaptive Higher Order Neural-Network Models for Automated Financial Data Modeling. IEEE Transactions On Neural Networks 13, 188–204 (2002) 14. Giles, L., Maxwell, T.: Learning Invariance and Generalization in High-Order Neural Networks. Applied Optics 26, 4972–4978 (1987) 15. Redding, N., Kowalczyk, A., Downs, T.: Constructive High-Order Network Algorithm That is Polynomial Time. Neural Networks 6, 997–1010 (1993) 16. Ramanathan, K., Guan, S.U.: Multiorder Neurons for Evolutionary Higher-Order Clustering and Growth. Neural Computation 19, 3369–3391 (2007) 17. Ho, D.W.C., Liang, J.L., Lam, J.: Global Exponential Stability of Impulsive HighOrder BAM Neural Networks with Time-Varying Delays. Neural Networks 19, 1581– 1590 (2006) 18. Abdelbar, A.M.: Achieving Superior Generalisation with a High Order Neural Network. Neural Computing & Applications 7, 141–146 (1998) 19. Chen, Y.H., Jiang, Y.L., Xu, J.X.: Dynamic Properties and a New Learning Mechanism in Higher Order Neural Networks. Neurocomputing 50, 17–30 (2003) 20. Cho, J.S., Kim, Y.W., Park, D.J.: Identification Of Nonlinear Dynamic Systems Using Higher Order Diagonal Recurrent Neural Network. Electronics Letters 33, 2133–2135 (1997)

Adaptive Higher Order Neural Networks for Effective Data Mining

173

21. Burshtein, D.: Long-Term Attraction in Higher Order Neural Networks. IEEE Transactions On Neural Networks 9, 42–50 (1998) 22. Psaltis, D., Park, C.H., Hong, J.: Higher Order Associative Memories and Their Optical Implementations. Neural Networks 1, 149–163 (1988) 23. Reid, M.B., Spirkovska, L., Ochoa, E.: Simultaneous Position, Scale, Rotation Invariant Pattern Classification Using Third-Order Neural Networks. Int. J. Neural Networks 1, 154–159 (1989) 24. Campolucci, P., Capparelli, F., Guarnieri, S., Piazza, F., Uncini, A.: Neural Networks with Adaptive Spline Activation Function. In: Proceedings of IEEE Melecon, vol. 96, pp. 1442–1445 (1996) 25. Chen, C.T., Chang, W.D.: A Feedforward Neural Network with Function Shape Autotuning. Neural Networks 9(4), 627–641 (1996) 26. Leshno, M., Lin, V.Y., Pinkus, A., Schocken, S.: Multilayer Feedforward Networks with a Nonpolynomial Activation Function Can Approximate Any Function. Neural Networks 6, 861–867 (1993) 27. Haykin, S.: Neural Networks: A Comprehensive Foundation. Prentice Hall, NJ (1999) 28. Ash, T.: Dynamic Node Creation in Backpropagation Networks. Connection Science 1, 365–375 (1989) 29. Hirose, Y., Yamashita, I.C., Hijiya, S.: Back-Propagation Algorithm Which Varies the Number Of Hidden Units. Neural Networks 4, 365–375 (1991) 30. Rivals, I., Personnaz, L.: A Statistical Procedure For Determining the Optimal Number of Hidden Neurons of A Neural Model. In: Second International Symposium on Neural Computation (2000) 31. Barron, A.R.: Approximation and Estimation Bounds for Artificial Neural Networks. Machine Learning 14, 115–133 (1994) 32. Barron, A.R., Cover, T.M.: Minimum Complexity Density Estimation. IEEE Transactions on Information Theory 37, 1034–1054 (1991) 33. UCI Machine Learning Repository (2008), http://archive.ics.uci.edu/ml/index.html 34. Xu, S., Zhang, M.: Justification of a Neuron-Adaptive Activation Function. In: IJCNN 2000 (CD-ROM) Proceeding (2000) 35. Malone, J., McGarry, K., Wermter, S., Bowerman, C.: Data Mining Using Rule Extraction From Kohonen Self-Organising Maps. Neural Computing & Applications 15, 9–17 (2006)

Exploring Cost-Sensitive Learning in Domain Based Protein-Protein Interaction Prediction Weizhao Guo, Yong Hu, Mei Liu, Jian Yin, Kang Xie, and Xiaobo Yang*

Abstract. Protein interactions are of great biological interest because they orchestrate nearly all cellular processes and can further our understandings in biological processes and diseases. Protein interaction data like many real world datasets are imbalanced in nature. Most protein pairs belong to the non-interaction class and few belong to the interaction class. Most existing protein interaction prediction methods assume equal distribution of the positive and negative interaction data. In this study, we first analyze effects of various portions of negative samples on the performance of domain-based protein interaction prediction methods using Artificial Neural Network (ANN), Bayesian Network (BN), and SVM. Then we introduce cost-sensitive learning to address the class imbalance problem. Experimental results demonstrated that the addition of cost-sensitive learning to each classifier: ANN, BN, and SVM, indeed yields an increase in accuracy. Keywords: Cost-sensitive learning, Imbalance data, Protein-protein interactions. Weizhao Guo . Jian Yin School of Information Science and Technology, Sun Yat-sen University, Guangzhou 510275, China [email protected], [email protected] *

Yong Hu Business Intelligence and Knowledge Discovery, Guangdong University of Foreign Studies, Sun Yat-sen University, Guangzhou 510275, China [email protected] Mei Liu Bioinformatics and Computational Life-Sciences Laboratory, ITTC, Department of Electrical Engineering and Computer Science, The University of Kansas, 1520 West 15th Street, Lawrence, KS 66045, U.S.A. [email protected] Kang Xie School of Business, Sun Yat-sen University, Guangzhou 510275 [email protected] Xiaobo Yang The 2nd Affiliated Hospital, Guangzhou University of TCM, Guangzhou 510120, China H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 175–184. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

176

W. Guo et al.

1 Introduction Proteins are essence of life because they orchestrate virtually every process and function in a living cell. The multiplicity of functions that proteins execute in most cellular processes and biochemical events is attributed to their interactions with other proteins. Protein-protein interaction (PPI) information can contribute to ANNotations of uncharacterized proteins according to the classification of known proteins within interaction network. Moreover, information about these protein interactions can improve our understanding in diseases and directly contribute to new therapeutic approaches. In a simple yeast cell, there are about 6,000 proteins which consequently results in an enormous pool of all possible protein interaction pairs to analyze. It is thus impractical to rely on experimental techniques for obtaining a complete protein interaction map. Over the years, research in seeking complementary in silico methods that are capable of accurately predicting interactions has been actively sought. These researches have focused on machine learning techniques such as artificial neural network (ANN) [1], Bayesian network [2,3], and SVM [4-7]. Despite the preliminary successes achieved, these machine learning techniques may perform poorly in learning from imbalanced data, in which one class is represented by a large number of examples while the other is represented by only a few. The reason is that traditional classifiers seek to maximize classification accuracy over a full range of instances. High predictive accuracy may be produced for the majority class but poor predictive accuracy is produced for the minority class, which is usually the important class. Protein interaction prediction problem is a typical imbalanced data classification problem. There exist much more protein pairs that do not interact than the ones that do interact. It is apparent that failing to identify one of the few true interacting protein pairs is much worse than inaccurately classifying one of the many non-interacting pairs as interacting. For instance, consider two classifiers A and B trained to recognize a class of histones consisting 15 genes. Assume that on a test set of 1000 genes, classifier A identified 30 genes, in which 14 are correct. Classifier B, on the other hand, identified everything in the test set as negative. Undoubtedly, the classifier A has learned something regarding histones, whereas the classifier B has learned nothing. Most existing computational methods bypassed the class imbalance problem by assuming equal number of interaction and non-interaction data samples. In this research, we address the class imbalance problem in protein interaction prediction by employing cost-sensitive learning. Protein interaction prediction is formulated as a classification problem based on their constituent domain information. Protein domains are structural and/or functional units of proteins that are conserved through evolution to represent protein functions or structures. Proteins interact with each other through their domains is a well accepted concept. In this study, three classification algorithms, Artificial Neural Network (ANN), Bayesian network (BN), and SVM are analyzed using different proportions of positive and negative samples. Performances of the classifiers before and after utilizing costsensitive learning are compared. Our experimental results showed that the prediction accuracy can be improved accordingly by the addition of cost-sensitive learning.

Exploring Cost-Sensitive Learning in Domain

177

The paper is organized into four sections. Section 2 introduces feature representation used in protein interaction prediction and cost-sensitive learning that can be applied to different classification algorithms to improve the prediction accuracy. Experimental results and discussion are presented in Section 3 and conclusions are drawn in Section 4.

2 Methods 2.1 Domain Feature Representation In our study, protein-protein interaction (PPI) prediction problem is formulated as a two-class classification problem. Each protein pair is a sample classified as either interacting or non-interacting, where the interacting class is labeled as 1 and the non-interacting class is labeled as 0. A protein pair can be characterized by the domains existing in each protein. Thus, a protein pair is represented as a vector of n features where each feature corresponds to a unique Pfam domain in our data set [8]. Let X = [x1, x2 … xn, y] represents a sample with n feature attributes belonging to the class y which equals to either 0 for non-interacting or 1 for interacting. Each feature xi has a discrete value of 0, 0.5 or 1. The feature value xi = 0 means that neither proteins in the protein pair contain the corresponding domain. Similarly, xi = 0.5 means that one protein in the protein pair contains the domain, and xi = 1 means that both proteins in the pair contain the domain. This ternary-valued feature representation is different from other domain-based binary-valued protein methods. It allows us to distinguish between protein pairs with interaction evidence existing in one protein and those existing in both proteins [9].

2.2 Cost-Sensitive Learning Recent research in machine learning algorithms strives to minimize various errors by considering unequal costs of those errors. This is the case in many real-world applications. For example, diagnosing a healthy person mistakenly to be sick may be less serious than diagnosing a patient erroneously to be healthy because the latter may result in loss of a life. Thus, the inclusion of costs of errors into learning algorithms has been suggested as a potential solution for the class imbalance problem [11]. In Section 2.1, X = [x1, x2, …, xn, y] represents a sample with n feature attributes xi belonging to the class y, where y has a discrete value of 0 and 1. Therefore for a two-class case, the cost matrix has the following structure in Table 1[11]. Table 1. Two-class Cost Matrix

Actual Class

Negative Positive

Predicted Class Negative C(0, 0) = C00 C(1, 0) = C10

Positive C(0, 1) = C01 C(1, 1) = C11

178

W. Guo et al.

In Table 1, the cost matrix columns correspond to the predicted class while rows represent the actual class. C01 stands for the cost of false positives and C10 stands for the cost of false negatives. Similarly, C11 stands for the cost of true positives and C00 for the cost of true negatives. Normally, the values of C00 and C11 are both 0. C01+C10 represent the cost of error for misclassification while C11+C00 represent the cost for correct classification. In our application, the purpose of cost-sensitive learning is to train a more adaptive classifier by adjusting diverse costs of misclassification errors, so that the classifier concentrates on the misclassified protein pairs during every training iteration. Considering a two-case classification problem, the cost of a protein pair X predicted as the class I is W(X, I) where I = 0 or 1. Accordingly, the key of obtaining the optimal prediction of the protein pair X is to minimize the cost W(X, I). Assume that the probability of the protein pair X being in class J is P(J|X) where J = 0 or 1, the cost W(X, I) can be computed by the following formula:

W ( X , I ) = min( P (0 | X )C ( I ,0) + P (1 | X )C ( I ,1))

(1)

Protein interaction prediction is a typical class imbalance problem where the number of ‘non-interacting’ proteins is much more than that of ‘interacting’ proteins. Traditional classifiers do not work well with imbalance data. Therefore, we expect the prediction accuracy of the classifiers to increase with the addition of costsensitive learning. In this paper, three classification algorithms, Artificial Neural Network (ANN) [1], Bayesian network (BN), and SVM [10], are employed. ANN is a biologically inspired learning method that imitates the brain made up of closely interconnected set of neurons. It takes a number of real-valued inputs but produces only one single-valued output and adjusts the connected weights based on the output values obtained for a specific input instance. BN can combine highly irrelevant types of protein instances and converts the different types of data to a common probabilistic framework. SVM is based on structural risk minimization that provides principle means to minimize the predicted error by adjusting kernel function. It is readily adaptable to new types of data and trains mass data fast [4]. Over the years, there has been a growing interest in the application of SVM to biological problems such as prediction of protein-protein interaction [4-7]. In the next section, we analyze the effects of negative samples in domain-based protein interaction prediction by changing the proportion of negative and positive examples and compare performances of different classifiers before and after utilizing the cost-sensitive learning.

3 Experiments 3.1 Data Sources Protein-protein interaction data for the yeast organism was collected from the database of interacting proteins (DIP) [12], Deng et al. [13], and Schwikowski et al. [14]. The dataset used by Deng et al. [13] is a combined interaction data experimentally obtained through two-hybrid assays on Saccharomyces cerevisiae by

Exploring Cost-Sensitive Learning in Domain

179

Uetz et al. [15], Schwikowski et al. [14] gathered their data from yeast two-hybrid, biochemical, and genetic data. We obtained 15,409 interacting protein pairs for the yeast organism from DIP, 5,719 pairs from Deng et al. [13], and 2,238 pairs from Schwikowski et al. [14]. The data sets were then combined by removing the overlapping interaction pairs. Because domains are the basic units of protein interactions, proteins without domain information cANNot provide any useful information for our prediction. Therefore, we excluded the pairs where at least one of the proteins has no domain information. To further reduce noise in our data, pairs with both proteins containing only one domain, which only occurred once among all proteins, were also excluded. Finally, we have 9,834 protein interaction pairs among 3,713 proteins. Negative samples are generated by randomly picking a pair of proteins. A protein pair is considered to be a negative sample if the pair does not exist in the interaction set. The protein domain information was gathered from Pfam [8], a protein domain family database, which contains multiple sequence alignments of common domain families. Hidden Markov model profiles were used to find domains in new proteins. The Pfam database consists of two parts: Pfam-A and Pfam-B. Pfam-A is manually curated, and Pfam-B is automatically generated. Both Pfam-A and Pfam-B families are used here. In total, there are 4,293 Pfam domains defined by the set of proteins.

3.2 Evaluation Measures Three classification methods [1,10] for the protein-protein interaction prediction problem, namely ANN, BN, and SVM, are evaluated by computing sensitivity (SN), specificity (SP) and accuracy (AC). SN is the proportion of matched interactions over total number of observed interactions. SP is defined as the percentage of matched non-interactions over total number of observed non-interactions. AC is the percentage of all protein pairs that are correctly classified by the classifier. Detailed formulas for SN, SP and AC are listed below. Here, TP, TN, FP and FN stand for true-positive, true-negative, false-positive and false-negative, respectively. SN =

TP TP + FN

(2)

SP =

TN TN + FP

(3)

TP + TN TP + TN + FP + FN

(4)

AC =

3.3 Experiment Result 3.3.1 Preprocessing There exists vast amount of noise in protein-protein interaction dataset. Especially, we have tremendous amount of features (i.e. 4293) to consider. Intuitively, not all features are equally important. In contrast to most protein interaction prediction

180

W. Guo et al.

algorithms, data preprocessing is utilized here such that only a set of more important features are used for learning. We implemented entropy and information gain based feature selection to eliminate the noise. In our application, we define entropy as H(x)=-∑Pi Pi, where Pi is the probability of i-th message. For a set of protein pairs, we compute the information expectation using the formula:

㏒

E = p log p + (1 − p) log(1 − p)

(5)

Where p stands for the probability obtained by computing how often one class occurs in the PPI data set and (1-p) for the other class. In Section 2.1, Let X = [x1, x2, …, xn, y] represent a sample with n feature attributes xi which has a discrete value of 0, 0.5 or 1. Therefore, each input attribute xi divides the set of protein pairs into three subsets {Li1, Li2, L i3}. We can get the probability pij for each subset Lij by computing how often each subset Lij occurs, respectively (Note: j=1, 2, 3). Then, the entropy H(Lij) for each subset Lij is computed. At last, the information gain for each input attribute xi is obtained by the formula below: 3

InformationGain( xi ) = E − ∑ pij H ( Lij ) j =1

(6)

Attributes with low information gain are eliminated because they are identified as redundant noise. Based on feature selection described above, we selected 1426 attributes with high information gain from the original 4293 attributes described in section 3.1. Now the protein pairs in our dataset is processed to only consider the selected features and is used to train our model. 3.3.2 Training In order to predict protein-protein interactions, we need to train our model first. After data preprocessing by feature selection, 9834 samples (4917 positives and 4917 negatives) with 1426 feature attributes are utilized to train different models. To construct the SVM model, we use the linear kernel to find a separating hyper plane which is approximated by minimizing the prediction error. In our application, the kernel tries to find more common domains between different pairs of PPIs. The more common domains the pairs of PPIs have, the larger the similarity value is. For instance, let X and Y represent two different PPI samples with n feature attributes and the kernel value K(X, Y) is used to measure the similarity between X and Y in the feature space, the larger the similarity values are, the more likely X and Y are classified into the same class. For our artificial neural network model, the radius basis function (RBF) is employed. Like other ANN methods, it has three layers: input, hidden and output. Different layers have different functions. For example, each node in the input layer represents a feature of an input instance and the output for a given sample depends on the similarity between its domain features and domain features in other protein pairs. The hidden layer is what makes the network nonlinear through hidden neurons and is a bridge that links the input layer and the output layer. ANN

Exploring Cost-Sensitive Learning in Domain

181

predicts a protein pair to be interacting if the value assigned by the output layer is larger than or equal to a certain threshold. At last, the BN model is constructed by computing the Bayes equation (7). Assume that Dj stands for each class in the training set and H refers to the sample to be classified. We can estimate the prior probability P(Dj) by computing how often each class Dj occurs in the training data. Similarly, P(H|Dj) can be obtained by computing how often H occurs in each class for the training data. P(H) is estimated by computing ∑P(H|Dj)P(Dj). Finally, we obtain the posterior probability P(Dj|H) for each class by formula (7). By comparing the value P(Dj|H) for each class, we can classify the sample H as the class with the highest posterior probability P(Dj|H). P( D j | H ) =

P ( H | D j ) P( D j )

(7)

P( H )

3.3.3 Testing Protein-protein interaction predictions are made using the models constructed above. First, we compared different classification methods, SVM, ANN and BN, using 5-fold cross-validation with equal number of positive and negative data samples. Here, the output of three methods is a real number between 0 and 1. Thus, we can set a certain threshold and if the output for a protein pair is greater than or equal to the threshold, the protein pair is predicted as interacting. And the threshold can be changed to produce ROC curves (Figure 1). As clearly shown in Table 2 Predicted results comparing SVM, ANN and BN

Specificity (SP) Sensitivity (SN) Accuracy (AC)

SVM 68.4% 76.8% 72.6%

ANN 67.6% 77.6% 72.6%

Fig. 1 The ROC comparison for SVM, ANN and BN

BN 67.6% 79.4% 73.5%

182

W. Guo et al.

Table 2, for our domain-based protein interaction predictions, Bayesian network outperforms both SVM and ANN. When we fix the specificity at approximately the same level 68% across different classification models, the sensitivity are 76.8%, 77.6% and 79.4% for SVM, ANN and BN respectively. Meanwhile, the accuracies are 72.6%, 72.6% and 73.5%.

3.4 Effects of Adding Cost-Sensitivity In this section, we address the class imbalance problem in protein interaction prediction by changing the proportion of positive and negative samples and employing cost-sensitive learning. Here, we compare performances of the classifiers before and after utilizing cost-sensitive learning. All three classification algorithms, BN, ANN, and SVM are analyzed using different proportions of positive and negative samples (Figure 2). In this section, we validate our result based on the test dataset. In section 2.5, cost matrix for a two-class case was introduced. In this section, cost matrix is represented as [C00 C01; C10 C11]. For example, in the cost matrix [0.0 0.95; 1.05 0.0], 0.95 stands for the cost of false positive and 1.05 for the cost of false negative. Similarly, 0.0 stands for the cost of true positive and the cost of true negative, respectively. Now, we compare the effects of adding cost-sensitivity for BN, SVM and ANN using different proportions of positive and negative samples below. It can be observed that the accuracies with the addition of costsensitive learning are adaptively improved comparing to those without cost-sensitive learning. Table 3 compares the performances for the classification method BN before and after utilizing cost-sensitive learning as the proportion of positive and negative training samples is varied. When the proportion of positive and negative training examples is 1000: 2000, 1000: 3000 and 1000: 4000, the addition of cost matrix, [0.0 0.90; 1.25 0.0], yields an increase in accuracy.



1XPEHU

Fig. 2 Setting different number of positive and negative training examples

   



3RVLWLYH





1HJDWLYH

Table 3 Comparing the results for BN before and after utilizing cost-sensitive learning

Size (positive: negative) 1000: 2000 1000: 3000 1000: 4000

Without/ With the addition of cost-sensitive SP SN AC 92.3%/89.1% 30.4%/39.8% 61.3%/64.2% 93.9%/92.2% 27.1%/32.8% 60.4 %/62.5% 94.5%/93.0% 25.4%/30.1% 59.9%/61.6%

Exploring Cost-Sensitive Learning in Domain

183

Table 4 Comparing the results for SVM before and after utilizing cost-sensitive learning

Size (positive: negative) 1000: 2000 1000: 3000 1000: 4000

Without/ With the addition of cost-sensitive SP SN AC 86.8%/84.4% 42.9%/46.5% 64.8%/65.5% 91.5%/87.7% 33.2%/40.9% 62.3%/64.2% 93.9%/89.3% 26.2%/36.8% 60.0%/63.0%

Table 4 compares the performances for SVM before and after utilizing costsensitive. When the proportion is 1000: 2000, adding cost matrix [0.0 0.95; 1.05 0.0] improves the accuracy. When 1000: 3000, the corresponding cost matrix is [0.0 0.95; 1.25 0.0]. When 1000: 4000, the corresponding cost matrix is [0.0 0.90; 1.25 0.0]. Table 5 compares the classification result for ANN before and after using costsensitive learning. When the proportions of positive and negative examples are 1000: 2000, the accuracy increases with the addition of [0.0 0.95; 1.25 0.0]. When 1000: 3000 and 1000: 4000, the corresponding cost matrix is [0.0 0.90; 1.25 0.0]. Table 5 Comparing the results for ANN before and after utilizing cost-sensitive learning

Size (positive: negative) 1000: 2000 1000: 3000 1000: 4000

Without/ With the addition of cost-sensitive SP SN AC 88.1%/82.7% 40.1%/49.7% 64.1%/66.2% 89.7%/89.0% 30.6%/39.0% 60.2%/64.0% 94.4%/89.3% 25.4%/32.5% 59.9%/60.9%

4 Conclusion In this paper, we first used feature selection based on entropy and information gain to eliminate existing noise in our data set. The useful features are used to train our models and can obtain the classification result more quickly and more accurately. Then we addressed class imbalance problem in protein interaction prediction by applying cost-sensitive learning. Three classification algorithms, ANN, BN and SVM are analyzed in domain-based protein-protein interactions prediction and we compared the classification results for the three algorithms before and after employing cost-sensitive learning as the proportion of positive and negative samples are varied. Experimental results suggested that adding cost-sensitive learning to each classifier yields an increase in accuracy. Acknowledgements. This research was partly supported by National Natural Science Foundation of China (NSFC, Project No.: (70801020) (60773198) (60573097) (70572053)). Natural Science Foundation of Guangdong Province (7300272), Research Foundation of Science, Technology Plan Project in Guangdong Province (2007B031403003), Sun Yat-sen University “211 Project” Construction Projects of Phase Key Discipline, NECT-06-0737, GDUFS(399-X3207018) and GDUFS(GWQ0718).

Ⅲ

184

W. Guo et al.

References 1. Chen, X.W., Liu, M.: Domain Based Predictive Models for Protein-Protein Interaction Prediction. EURASIP Journal on Applied Signal Processing, 1–8 (2006) 2. Jansen, R., Yu, H., Greenbaum, D., Kluger, Y., Krogan, N.J.: A Bayesian Networks Approach for Predicting Protein-Protein Interactions from Genomic Data. Science 302, 449–453 (2003) 3. Rhodes, D.R., Tomlins, S.A., Varambally, S., Mahavisno, V., Barrette, T.: Probabilistic Model of the Human Protein-protein Interaction Network. Nat. Biotechnol., 951– 959 (2005) 4. Bock, J.R., Gough, D.A.: Predicting Protein-protein Interactions from Primary Structure. Bioinformatics 17, 455–460 (2001) 5. Ben-Hur, A., Noble, W.S.: Kernel Methods for Predicting Protein-protein Interactions. Bioinformatics, i38–i46 (2005) 6. Martin, S., Roe, D., Faulon, J.L.: Predicting Protein-protein Interactions Using Signature Products. Bioinformatics 21, 218–226 (2005) 7. Chen, X.W., Han, B., Fang, J.: Large-scale Protein-protein Interaction Prediction Using Novel Kernel Methods. Int. J. Data Mining and Bioinformatics 2, 145–156 (2008) 8. Finn, R.D., Tate, J., Mistry, J., Coggill, P.C., Sammut, J.S.: The Pfam Protein Families Database. Nucleic Acids Res. 36, D138–D141 (2008) 9. Chen, X.W., Liu, M.: Prediction of Protein-protein Interactions Using Random Decision Forest Framework. Bioinformatics 21, 4394–4400 (2005) 10. Burges, C.J.C.: A Tutorial on Support Vector Machines for Pattern Recognition. Data Mining and Knowledge Discovery 2, 121–167 (1998) 11. Elkan, C.: The Foundation of Cost-Sensitive Learning. In: 17th International Joint Conference on Artificial Intelligence, pp. 973–978 (2001) 12. Salwinski, L., Miller, C.S., Smith, A.J., Pettit, F.K., Bowie, J.U.: The Database of Interacting Proteins: 2004 update. Nucleic Acids Res. 32, D449–D451 (2004) 13. Deng, M., Mehta, S., Sun, F., Chen, T.: Inferring Domain-domain Interactions from Protein-protein Interactions. Genome Res. 12, 1540–1548 (2002) 14. Schwikowski, B., Uetz, P., Fields, S.: A Network of Protein-protein Interactions in Yeast. Nature Biotechnol. 18, 1257–1261 (2000) 15. Uetz, P., Giot, L., Cagney, G., Mansfield, T.A., Judson, R.S.: A Comprehensive Analysis of Protein-protein Interactions in Saccharomyces cerevisiae. Nature 403, 623–627 (2000)

An Efficient and Fast Algorithm for Estimating the Frequencies of 2-D Superimposed Exponential Signals in Presence of Multiplicative and Additive Noise Jiawen Bian, Hongwei Li, Huiming Peng, and Jing Xing

Abstract. In this paper, we consider the two-dimensional (2-D) superimposed exponential signals in independently and identically distributed (i.i.d.) multiplicative and additive noise. We use a three step iterative(TSI) algorithm to estimate the frequencies of the considered model. It is observed that the estimator is consistent and works quite well in terms of biases and mean squared errors. Moreover, the convergence rate of the estimators attain Op (M −3/2 N −1/2 ) and Op (M −1/2 N −3/2 ) for each pair of frequencies. It attains the convergence rate of the least squares estimators (LSEs) in presence of only additive noise. Keywords: 2-D superimposed exponential signals, Three step iterative algorithm, Least squares estimators, Consistent estimators.

1 Introduction We consider the following 2-D model of superimposed exponential signals in multiplicative and additive noise: y(m, n) =

p

ξk (m, n)ei(uk m+vk n+φk ) + ε(m, n), m = 1, 2, . . . , M, n = 1, 2, . . . , N,

k=1

(1) Jiawen Bian · Hongwei Li · Huiming Peng School of Mathematics and Physics, China University of Geosciences, Wuhan 430074, China Jing Xing Department of Statistics and Applied Mathematics, Hubei University of economics, Wuhan 430205, China [email protected] H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 185–195. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

186

J. Bian et al.

√ where y(m, n) are observed values, i = −1, uk ∈ (0, 2π) and vk ∈ (0, 2π) are the unknown frequencies. Multiplicative noise {ξk (m, n)} is an array of independent identically distributed (i.i.d) real random variables with mean Ak = 0 and finite variance σk2 , {φk } ∈ [0, 2π) is the unknown phase. Additive noise {ε(m, n)} is an array of complex random variable with mean zero and finite variance σ02 /2 for both the real and imaginary parts which are assumed to be independent. The number of components p is assumed to be known in advance. In this paper we mainly consider the estimation of frequencies (uk , vk ), given a sample of size M and N , namely y(1, 1), . . . , y(M, N ). We make the following assumptions: (1) The multiplicative noise {ξk (m, n)} and additive noise {ε(m, n)} are independent mutually; (2) The frequencies u j s are different from each other, and it is the same for vj s; (3) 0 < limmin{M,N }→+∞ M N < +∞. The problem of estimating 2-D exponential signals is an important problem in multidimensional statistical signal processing. This modeling and estimation problem has fundamental importance, as well as various applications in texture estimation of image(see [1] and the reference therein) and in wave propagation problems (see [2] and the references therein). This problem has been intensively studied in presence of additive noise. Many techniques have been developed to estimate the parameters of 2-D superimposed exponential signals in white or color noise. Such as maximum-likelihood estimator (MLE)(see [3]), least squares estimators (LSEs)([4]), Pisarenko’s method([5]), singular value decomposition ([6]), cumulant based method([7]). However, multiplicative noise may also occur in a variety of applications (see [8], [9], [10]). But there are not many methods for the 2-D harmonic retrieval in the presence of multiplicative noise and additive noise. [11] and [12] considered this problem using cyclic statistic method. It was also considered in [13] using 2-D chirp z-transform. It is known [4] that the LSEs have the best convergence rate among all the estimators and attains the CRB in presence of white i.i.d. noise. It needs multidimensional searching, so it is time consuming and hard to implement for large sample size. It will be very meaningful to find a iterative algorithm for the LSEs. But nowhere, at least not known to the authors, the iterative procedure for the LSEs of the frequencies for a 2-D superimposed exponential signal model with multiplicative and additive noise has been considered. Recently, a computationally efficient three step iterative (TSI) algorithm was proposed by [14] to estimate the frequencies of 1-D sinusoidal signals in presence of additive stationary noise, which shares the same technique with the seven step iterative algorithm proposed by [15], it is observed that the TSI algorithm is statistically efficient and attains the same convergence rate as the LSEs. Moreover, it need only a little time to work and can be serve as online implementation.

An Efficient and Fast Algorithm

187

Stimulated by the paper [14], in this paper, we generalized the TSI algorithm for the 1-D signal to estimate the frequencies of a 2-D superimposed exponential signal in presence of i.i.d multiplicative and additive noise. This method uses a correction term based on A1M,N (j), A2M,N (j) and BM,N (j) to be defined in Section 3, which are functions of the data vector and the j-th available frequency estimators. It is observed that if the initial estimators are accurate up to the order Op (M −1 ) and Op (N −1 ) for the frequencies uj and vj respectively (here Op (M δ ) means M −δ Op (M δ ) is bounded in probability), then the three step iterative algorithm produces fully efficient frequency estimators which have convergence rate of Op (M −3/2 N −1/2 ) and Op (M −1/2 N −3/2 ) for uj and vj respectively, which is the convergence rate of the LSEs ([4]). We use the 2-D periodogram maximizers over Fourier frequencies as the initial estimators. It is known that the 2-D periodogram maximizers over Fourier frequencies do not generally provide estimators up to the order Op (M −1 ) and Op (N −1 ) for the frequencies uj and vj respectively. To overcome this problem, we do not use the fixed sample size available for estimation at each step. At first step we use a fraction of it and at the last step we use the whole data set by gradually increasing the effective sample sizes. The rest of the paper is organized as follows. In Section 2, we give the initial estimator based on the 2-D periodogram, which is implemented by 2D FFT. The TSI algorithm is presented in Section 3. In Section 4, we present the numerical experiments and finally we conclude the paper in Section 5. All the proofs are provided in the Appendix.

2

Initial Estimator

We consider the initial estimator as the 2-D periodogram maxmizer at the Fourier frequencies, which is defined as follows: %2 % M N % 1 %% −i(mu+nv) % I(u, v) = y(m, n)e (2) % . % % MN % m=1 n=1

The estimators obtained by finding p local maxima of I(u, v) achieve the best possible rate and are asymptotically equivalent to the LSEs. It is known as approximate least squares estimators (ALSE) in the literature, but it needs multidimensional searching in the frequency domain and is very time consuming for large sample size. In this paper, we employ 2-D FFT to find the p local maxima of I(u, v). The initial estimates obtained by 2-D FFT are 2πj Fourier frequencies which have the form u = 2πi M and v = N for some integer 1  i  M, 1  j  N , so the convergence rate of the initial estimators are Op (M −1 ) and Op (N −1 ) for uj and vj respectively.

188

J. Bian et al.

3 TSI Algorithm In this section, we will discuss the TSI algorithm. Given a consistent estimator u ˜j and v˜j of model (1), we compute u ˆj and u ˆj for j = 1, . . . , p as follows:     12 A1M,N (j) 12 A2M,N (j) , vˆj = v˜j + 2 Im , (3) u ˆj = u˜j + 2 Im M BM,N (j) N BM,N (j) where A1M,N (j) =

N M

M −i(˜uj m+˜vj n) )e , 2

(4)

N −i(˜uj m+˜vj n) )e , 2

(5)

y(m, n)e−i(˜uj m+˜vj n) ,

(6)

y(m, n)(m −

m=1 n=1

A2M,N (j) =

N M

y(m, n)(n −

m=1 n=1

BM,N (j) =

M N m=1 n=1

and Im[.] denotes the imaginary part of a complex number. We can start with any consistent estimators u ˜j and v˜j and improve them step by step using Eq. (3). The motivation of the algorithm is based on the following theorem. In the following we note u = (u1 , ..., up ), v = (v1 , ..., vp ), u ˆ = (ˆ u1 , ..., u ˆp ), v ˆ = (ˆ v1 , ..., vˆp ) for convenience. Theorem 1. If for j = 1, 2, . . . , p, u ˜j − uj = Op (M −1−δ ) and v˜j − vj = −1−δ Op (N ), where δ ∈ (0, 1], then (a) u ˆj − uj = Op (M −1−2δ ), vˆj − vj = Op (N −1−2δ ), if δ < 12 , 3

1

1

where Σjj =

σ2 −σj2 , f or A2j

3

L

u − u), M 2 N 2 (ˆ v − v)) −→ N2p (0, 6Σ), if δ  12 , (b) (M 2 N 2 (ˆ j = 1, ..., p; Σjj =  1, ..., 2p; Σij = 0, f or i = j. σ 2 = pk=1 σk2 + σ02

2 σ2 −σj−p , f or A2j−p

j = p +

Proof. See Appendix. We start with the initial estimates obtained by 2-D FFT and improve it step (m) by step by the recursive algorithm below. The m-th step estimators u ˆj and (m)

(m−1)

ˆj vˆj are computed from the (m − 1)-th step estimators u respectively by the formula (m) u ˆj

=

(m−1) u ˆj +

(m−1)

and vˆj

    12 A1Mm ,Nm (j) 12 A2Mm ,Nm (j) (m) (m−1) , vˆj = vˆj , Im + 2 Im 2 Mm BMm ,Nm (j) Nm BMm ,Nm (j) (7)

where A1Mm ,Nm (j), A2Mm ,Nm (j) and BMm ,Nm (j) can be obtained from Eq. (m−1) (m−1) (4-6) by replacing M , N , u ˜j and v˜j with Mm , Nm , u ˆj and vˆj respectively. We repeatedly choose suitably Mm and Nm at each step as follows:

An Efficient and Fast Algorithm

Step 1. (0) vˆj

189 (0)

With m = 1, choose M1 = M 0.8 , N1 = N 0.8 , u ˆj

= u ˜j and

= v˜j , which are the initial estimates obtained by 2-D FFT. −1− 14

Note that u ˜j − uj = Op (M −1 ) = Op (M1 −1− 1 Op (N1 4 ).

) and v˜j − vj = Op (N −1 ) = (0)

(0)

Taking M1 = M 0.8 , N1 = N 0.8 , u ˆj = u ˜j and vˆj (7), and applying part (a) of Theorem 1 , we obtain −1− 12

(1)

u ˆj −uj = Op (M1 Step 2. (2) vˆj (1) vˆj

6

6

) = Op (N − 5 ). (2)

(1) u ˆj

and

6

−1− 23

(2)

u ˆj −uj = Op (M2

3

−1− 23

) = Op (M − 2 ), vˆj −vj = Op (N2 (2)

3

) = Op (N − 2 ) (3)

With m = 3, choose M3 = M and N3 = N , computing uˆj

from

(2) u ˆj

and 3

and

6 −1− 1 (1) Since u ˆj − uj = Op (M − 5 ) = Op (M2 3 ) and −1− 1 = Op (N2 3 ), using part (a) of Theorem 1 again,

(1) vˆj .

− vj = Op (N − 5 ) we have

(3) vˆj

(1)

With m = 2, choose M2 = M 0.9 and N2 = N 0.9 , compute uˆj

from

Step 3.

−1− 12

) = Op (M − 5 ), vˆj −vj = Op (N1

= v˜j in

(2) vˆj 1

and

and applying part (b) of Theorem 1, we have 1

3

L

(M 2 N 2 (ˆ u − u), M 2 N 2 (ˆ v − v)) −→ N2p (0, 6Σ). Remark 1. Note that the exponents we use above are not unique. There are several other ways they can be chosen so that the iterative process will converge in three steps. For example another set of choices can be M1 = M 0.6 , N1 = N 0.6 ; M2 = M 0.8 , N2 = N 0.8 ; M3 = M , N3 = N , it is not possible to choose a set of exponents to make the iterative process converge in less than three steps, but it is sure for several sets of exponents to take more than three steps to converge.

4 Numerical Experiment In this section we present some numerical results to observe how the proposed method works for finite sample size. We consider the following model: y(m, n) = ξ1 (m, n)ei(0.5m+1.5n+π/4) + ξ2 (m, n)ei(0.6m+1.7n+π/3) + ε(m, n), where {ξ1 (m, n)} and {ξ2 (m, n)} are real Gaussian random variables with the same mean 3 and variance 0.5 and 0.6 respectively, {ε(m, n)} is an array of i.i.d. exponential complex random variables with mean zero and the same variance σ02 /2 for the real and the imaginary parts. To asset the sensitivity of the model to different noise levels we plot three different signal to min{A2 } noise ratio (SNR), defined as: SN R = 10 log10  2 σk2 , namely SNR = -5, 0 k=0

k

190

J. Bian et al.

and 5. To present the consistency we take the sample sizes as M=N=64, 128 and 256. Now, for each sample size, we estimate the frequencies based on the TSI algorithm. In all cases we consider the initial estimator as the periodogram maxmizer at the Fourier frequencies. We report the average estimates of the two pairs of frequencies (Fr11, Fr12), (Fr21, Fr22) and the corresponding square root of the mean squared errors (SRMSEs) over 100 replications. For comparison purpose, we also report the initial estimates as well as the true values of parameters and the corresponding square root of the asymptotic variances (SRAVs). All the results are reported in Tables 1-3 corresponding to M=N=64, M=N=128, M=N=256 respectively. In each table and for each SNR, the first line represents the true parameter values and the initial estimates are reported at the second line, the third line represents the TSI estimates and the SRMSEs of the TSI estimates are reported at the fourth line. Finally, we reported the SRAVs at the last line. The following observations are very clear from the numerical experiments. It is observed that the TSI estimates are very close to the true parameter values and are better than the initial estimates in nearly all the cases considered. It is immediate that the biases decrease as the SNR increase. Therefore, the TSI estimates provide asymptotically unbiased estimators of the frequencies. It can be seen that the SRMSEs of all the parameters decrease gradually and approach the SRAVs as the sample size increases. It verifies the consistency of the the TSI estimates. It is also observed that the TSI estimates are also fairly good even when the sample size is small and the SNR is low when the initial estimates are bad, so it verifies the effectiveness of the TSI algorithm. Table 1 The average estimates of the initial and TSI estimators based on 100 replications, along with the SRMSEs and SRAVs of the two pairs of frequencies when M=N=64 SNR Estimate Parameter Initial -5 TSI SRMSE SRAV Parameter Initial 0 TSI SRMSE SRAV Parameter Initial 5 TSI SRMSE SRAV

Fr11 0.500000 0.490874 0.500148 1.19256e-3 1.05406e-3 0.500000 0.490874 0.499922 5.89485e-4 5.81172e-4 0.500000 0.490874 0.499906 3.08272e-4 3.05329e-4

Fr12 1.500000 1.472622 1.498587 1.06496e-3 1.05406e-3 1.500000 1.472622 1.498533 6.59702e-4 5.81172e-4 1.500000 1.472622 1.498566 3.14107e-4 3.05329e-4

Fr21 0.600000 0.589049 0.599263 1.23070e-3 1.05217e-3 0.600000 0.589049 0.599473 6.01880e-4 5.77743e-4 0.600000 0.589049 0.599791 3.05855e-4 2.98750e-4

Fr22 1.700000 1.668971 1.701828 1.28560e-3 1.05217e-3 1.700000 1.668971 1.701556 5.73435e-4 5.77743e-4 1.700000 1.668971 1.701638 3.03656e-4 2.98750e-4

An Efficient and Fast Algorithm

191

Table 2 The average estimates of the initial and TSI estimators based on 100 replications, along with the SRMSEs and SRAVs of the two pairs of frequencies when M=N=128 SNR Estimate Parameter Initial -5 TSI SRMSE SRAV Parameter Initial 0 TSI SRMSE SRAV Parameter Initial 5 TSI SRMSE SRAV

Fr11 0.500000 0.490874 0.499873 2.68336e-4 2.63515e-4 0.500000 0.490874 0.499851 1.54061e-4 1.45293e-4 0.500000 0.490874 0.499912 7.64692e-5 7.63322e-5

Fr12 1.500000 1.521709 1.499992 2.64891e-4 2.63515e-4 1.500000 1.521709 1.499996 1.45544e-4 1.45293e-4 1.500000 1.521709 1.499980 7.67465e-5 7.63322e-5

Fr21 0.600000 0.589049 0.600115 2.86920e-4 2.63043e-4 0.600000 0.589049 0.600042 1.45542e-4 1.44436e-4 0.600000 0.589049 0.600173 7.48406e-5 7.46875e-5

Fr22 1.700000 1.718058 1.699986 2.97158e-4 2.63043e-4 1.700000 1.718058 1.699976 1.45130e-4 1.44436e-4 1.700000 1.718058 1.700016 7.49614e-5 7.46875e-5

Table 3 The average estimates of the initial and TSI estimators based on 100 replications, along with the SRMSEs and SRAVs of the two pairs of frequencies when M=N=256 SNR Estimate Parameter Initial -5 TSI SRMSE SRAV Parameter Initial 0 TSI SRMSE SRAV Parameter Initial 5 TSI SRMSE SRAV

Fr11 0.500000 0.490874 0.499982 6.88426e-5 6.58787e-5 0.500000 0.490874 0.500003 3.68782e-5 3.63233e-5 0.500000 0.490874 0.499985 1.96997e-5 1.90830e-5

Fr12 1.500000 1.497165 1.499999 6.88849e-5 6.58787e-5 1.500000 1.497165 1.499998 3.65143e-5 3.63233e-5 1.500000 1.497165 1.500000 1.91082e-5 1.90830e-5

Fr21 0.600000 0.589049 0.600015 6.68669e-5 6.57607e-5 0.600000 0.589049 0.600033 3.66264e-5 3.61089e-5 0.600000 0.589049 0.600022 1.917771e-5 1.86719e-5

Fr22 1.700000 1.693515 1.700016 6.88854e-5 6.57607e-5 1.700000 1.693515 1.700008 3.61466e-5 3.61089e-5 1.700000 1.693515 1.700011 1.89554e-5 1.86719e-5

5 Conclusions In this paper, we considered the estimation of the frequencies of signals of a 2-D superimposed exponential model in presence of i.i.d. multiplicative and additive noise. We generalized the three step iterative algorithm [14] to our

192

J. Bian et al.

considered model (1). We proved the efficiency of the TSI estimators and obtained the asymptotic distribution of the TSI estimators. It is observed that the TSI algorithm works quite well and the TSI estimators have the same convergence rate as the LSEs in presence of additive noise [4]. It can be easily seen that the TSI algorithm needs only three steps to converge from the given starting value, it naturally saves computational time and can be used for online implementation. Acknowledgements. This research is supported by the National Nature Science Foundation of China (No. 60672049) and the Research Foundation for Outstanding Young Teachers, China University of Geosciences (Wuhan) (No. CUGQNL0822 and No. CUGQNL0732).

References 1. Francos, J.M., Narashimhan, A., Woods, J.W.: Maximum-likelihood Estimation of Textures Using a Wold Decomposition Model. IEEE Trans. Image Process. 4(12), 1655–1666 (1995) 2. Ward, J.: Space-Time Adaptive Processing for Airborne Radar. Lincoln Lab., MIT, Lexington, MA, Tech. Rep. 1015 (1994) 3. Rao, C.R., Zhao, L., Zhou, B.: Maximum Likelihood Estimation of 2-D Superimposed Exponential Signals. IEEE Trans. Signal Process. 42(7), 1795–1802 (1994) 4. Kunda, D., Mitra, A.: Asymptotic Properties of the Least Squares Estimators of 2-D Exponential Signals. Multidimensional Systems and Signal Processing 7, 135–150 (1996) 5. Lang, S.W., McClellan, J.H.: The Extension of Pisarenko’s Method to Multiple Dimensions. In: Proc. Int. Conf. Acoustics, Speech, Signal Processing, Paris, France, pp. 125–128 (1982) 6. Kumaresan, R., Tufts, D.W.: A Two-dimensional Technique for FrequencyWavenumber Estimation. Proc. IEEE 69(11), 1515–1517 (1981) 7. Ibrahim, H.M., Gharieb, R.R.: Estimating Two-dimensional Frequencies by a Cumulant Based FBLP Method. IEEE Trans. Signal Process. 47(1), 262–266 (1999) 8. Giannakis, G.B., Zhou, G.: Harmonics in Multiplicative and Additive Noise: Parameter Estimation Using Cyclic Statistics. IEEE Trans. Signal Process. 43(9), 2217–2221 (1995) 9. Besson, B., Castanie, F.: On Estimating the Frequency of a Sinusoid in Autoregressive Multiplicative Noise. Signal Processing 30, 65–83 (1993) 10. Swami, A.: Multiplicative Noise Models: Parameter Estimation Using Cumulants. Signal Processing 36, 355–373 (1994) 11. Wang, F., Wang, S., Dou, H.: Two-dimensional Parameter Estimation Using Two-dimensional Cyclic Statistics. Acta Electronica Sinica 31(10), 1522–1525 (2003) 12. Yang, S., Li, H.: Two-dimensional Harmonics Parameters Estimation Using Third-order Cyclicmoments. Acta Electronica Sinica 33(10), 1808–1811 (2005)

An Efficient and Fast Algorithm

193

13. Yang, S., Li, H.: 2-D Harmonic Retrieval in Multiplicative and Additive Noise Based on Chirp Z-tranform. In: Proc. Int. Conf. Signal Processing, Beijing, China, pp. 16–20 (2006) 14. Nandi, S., Kundu, D.: An Efficient and Fast Algorithm for Estimating the Parameters of Sinusoidal Signals. Sankhya 68(2), 283–306 (2006) 15. Bai, Z.D., Rao, C.R., Chow, M., Kundu, D.: An Efficient Algorithm for Estimating the Parameters of Superimposed Exponential Signals. J. Statist. Plan. Infer. 110, 23–34 (2003) 16. Mangulis, V.: Handbook of Series for Scientists and Engineers. Academic Press, New York (1965)

Appendix To prove Theorem 1, we need the following lemmas. These results are elementary, so the proofs are omitted, readers can refer to [16]. Lemma 1. for k, l=1,2,3, α = 0 or β = 0, N M 1 1 mk−1 nl−1 cos2 (mα + nβ) = min{M,N }→∞ M k N l 2kl m=1 n=1

lim

for u = α or v = β, N M 1 mk−1 nl−1 cos(mα + nβ) cos(mu + nv) = 0 min{M,N }→∞ M k N l m=1 n=1

lim

Lemma 2. if {ε(m, n)} is a sequence of i.i.d. random variable with mean zero and finite variance σ 2 , then for any positive integer k and l, sup

N M

1

1

ε(m, n)mk nl ei(mα+nβ) = Op (M k+ 2 N l+ 2 ).

α,β m=1 n=1

Proof. (Proof of Theorem 1) We will give a simple proof of Theorem 1 in the following. Now we compute BM,N (j), A1M,N (j) and A2M,N (j) respectively. Firstly, we will compute BM,N (j). Since ξk (m, n) is an array of i.i.d. random variable with mean Ak and variance σk2 , so if we note ηk (m, n) = ξk (m, n) − Ak , then ηk (m, n) is an array of i.i.d. variable with mean zero and variance σk2 . BM,N (j) =

p k=1

+

Ak

N M

˜ j )m+(vk −˜ vj )n+φk ] ei[(uk −u

m=1 n=1

p N M

˜ j )m+(vk −˜ vj )n+φk ] ηk (m, n)ei[(uk −u +

k=1 m=1 n=1

=

p k=1

Ak eiφk J1k (M, N ) + R1 (M, N ), (say)

N M

˜ j m+˜ vj n) ε(m, n)e−i(u

m=1 n=1

(8)

194

J. Bian et al.

where for k = j, J1k (M, N ) = Op (1); for k = j, using Taylor series approximation of ei(uj −˜uj )m and ei(vj −˜vj )n all up to first order, we have J1j (M, N ) = M N i[(uj −˜ uj )m+(vj −˜ vj )n] = M N (1 + Op (M −δ ) + Op (N −δ )). m=1 n=1 e 1 1 Using Lemma 2, we have R1 (M, N ) = Op (M 2 N 2 ). It is immediate that BM,N (j) = Aj eiφj M N (1 + Op (M −δ ) + Op (N −δ )). For A1M,N (j) A1M,N (j) =

&

N M m=1 n=1

=

p

p

k=1

iφk

Ak e

' (Ak + ηk (m, n))ei(uk m+vk n+φk ) + ε(m, n) (m −

M −i(u ˜ j m+˜ vj n) )e 2

J2k (M, N ) + R2 (M, N ) (say),

(9)

k=1

where for k = j, J2k (M, N ) = Op (M ); for k = j, using Taylor series approximation of ei(uj −˜uj )m and ei(vj −˜vj )n all up to first order, we have M N 3 i(uj −˜ uj )m i(vj −˜ vj )n = i(uj − u ˜ j ) M12N (1 + J2j (M, N ) = m=1 (m − M n=1 e 2 )e Op (M −δ ) + Op (N −δ )). vj )n] j )m+(vj −˜ Using Taylor series approximation of ei[(uj −˜u up to first N M order and Lemma 2, we have R2 (M, N ) = ε(m, n)(m − p M m=1 N n=1 M −i(uj m+vj n) i[(uj −˜ uj )m+(vj −˜ vj )n] )e e + η (m, n)(m − k=1 m=1 n=1 k 2 M i[(uk −uj )m+(vk −vj )n+φk ] i[(uj −˜ uj )m+(vj −˜ vj )n] e = R21 (M, N ) + R22 (M, N ) 2 )e (say). M N Where R21 (M, N ) = − m=1 n=1 ε(m, n)(m 3 1 3 1 M −i(uj m+vj n) 2 −δ N 2 ) 2 N 2 −δ ), )e + O (M + O (M R (M, N ) = p p 22 2 M N  p M i[(uk −uj )m+(vk −vj )n+φk ] η (m, n)(m − )e + k k=1 m=1 n=1 2 3 1 3 1 Op (M 2 −δ N 2 ) + Op (M 2 N 2 −δ ). So, A1M,N (j) =

N M

ε(m, n)(m −

m=1 n=1

+iAj eiφj (uj − u ˜j ) +

p N M

M −i(uj m+vj n) )e 2

M 3N (1 + Op (M −δ ) + Op (N −δ )) 12

ηk (m, n)(m −

k=1 m=1 n=1

M i[(uk −uj )m+(vk −vj )n+φk ] )e . (10) 2

Therefore

  12 A1M,N (j) ˜j + 2 Im u ˆj = u M BM,N (j) ˜j )(Op (M −δ ) + Op (N −δ )) = uj + (uj − u & M N 12 m=1 n=1 ε(m, n)(m − + 3 Im M N Aj

M −i(uj m+vj n+φj ) 2 )e

An Efficient and Fast Algorithm

p

M

k=1

+

195

N

M i[(uk −uj )m+(vk −vj )n+(φk −φj )] n=1 ηk (m, n)(m − 2 )e Aj

m=1

= uj + (uj − u˜j )(Op (M −δ ) + Op (N −δ )) +

12 Xj (say), M 3N

'

(11)

where M

m=1

Xj =

N

n=1 [Im{ε(m, n)}(m

−

M 2

) cos(uj m + vj n + φj )

Aj M 2

−Re{ε(m, n)}(m − p

k=1

+

M

m=1

N

) sin(uj m + vj n + φj )]

Aj

n=1

ηk (m, n)(m −

M 2

) sin[(uk − uj )m + (vk − vj )n + (φk − φj )] Aj

(12) Similarly, we have   12 A2M,N (j) vˆj = v˜j + 2 Im N BM,N (j) = vj + (vj − v˜j )(Op (M −δ ) + Op (N −δ )) +

12 Yj (say), MN3

(13)

where Yj is similar with Xj and can be obtained by only replacN ing (m − M 2 ) in Xj by (n − 2 ). Using Lemma 1-2, we have that when min{M, N } → ∞ var( 6(σ2 −σj2 ) , Cov( 312 1 A2j M2N2 12

Cov(

have

0, Cov(

12

3

3

1

M2N2 1

M2N2

Xj ,

Xj ,

Xj , 12

1

12

1

3

1

M2N2 3

M2N2

3

M2N2 12

12

M

3 2

N

Yj )

Xk )

Yk ) → 0.

1 2

Xj ) → →

→

0,

6(σ2 −σj2 ) , var( 112 3 A2j M2N2

and

0, Cov(

for 12

1

3

M2N2

Yj ,

=

j

12

1

3

M2N2

Yj ) → k,

we →

Yk ) 3

1

From assumption (3), we have Op (M −δ ) = Op (N −δ ), Op (M − 2 N − 2 ) = 1 3 Op (M − 2 N − 2 ) = Op (M −2 ) = Op (N −2 ). Therefore, if u ˜j − uj = Op (M −1−δ ), 1 −1−δ v˜j − vj = Op (N ) and 0 < δ  2 , then from Eq. (11) and Eq. (13), u ˆj − uj = Op (M −1−2δ ), vˆj − vj = Op (N −1−2δ ). If 12 < δ  1, from Eqs. 3 1 (11-13) and using the Central Limit Theorem, it follows that (M 2 N 2 (ˆ u− 1

3

L

v − v)) −→ N2p (0, 6Σ). u), M 2 N 2 (ˆ

An Improved Greedy Based Global Optimized Placement Algorithm Luo Zhong, Kejing Wang, Jingling Yuan, and Jingjing He*

Abstract. An improved Greedy based global optimized Placement algorithm was proposed in this paper. Probability controlling was applied to avoid local minimization of traditional Greedy algorithm. Some new technique such as Marker bit method, dynamical selection and weight matrix were introduced to improve efficiency of this algorithm. From the experiments and comparison with Simulated Annealing algorithm, this novel algorithm shows better stability and accuracy. Keywords: Greedy, Global selection, Marker bit, weight matrix.

1 Introduction

，

， ，

，

The problem of optimized Placement is appeared in many areas. There are a lot of algorithm to solve the problem such as SLA SA Greedy GA and so on. The article propose a improved Greedy algorithm to avoid the problem of getting local optimization that is appear in Greedy usually .The article will introduction some common problems of optimizing Placement algorithm and two method to improve Greedy algorithm.

2 Problems of Placement Algorithm 2.1 Overlap and Connection Overlap and connection are common problems in placement, so the effective solution is to separate different modules correctly. For example, there are three tasks in 4*4 as follows. Task1 0010 1111 0110 0000

Task 2 1000 1111 1000 1000

Task 3 0000 0001 0001 0001

Luo Zhong . Kejing Wang . Jingling Yuan . Jingjing He Computer Science and Technology School, Wuhan University of Technology, 430070, Wuhan, China [email protected] *

H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 197–204. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

198

L. Zhong et al.

Marker bit method is applied in this paper to merge and separate different modules for the purpose of solving overlapping and connection problems. For example, if the first task is represented by 0000 0001 and the second task is represented by 0000 0010, the overlapping area is the sum of both tasks is represented by 0000 0011, So if the solution is 0011 0100 that means the third one, the fifth one and the sixth one are overlapped in this point. The empty matrix is as follows. Fig. 1 16*16 Matrix Space

There are three tasks are overlapped in the yellow module as follows

Task1 0010 1111 0110 0000

Task 2 1000 1111 1000 1000

Task 3 0000 0001 0001 0001

There are two tasks are overlapped in the red module as follows

Task 4 100 100 100

Task 5 010 010 010

There is only one task in the pink module as follows

Task 6 1111 1110 1111 That means there are 6tasks at the beginning

An Improved Greedy Based Global Optimized Placement Algorithm

199

Task 1, Task 2 and Task 3 are overlapped in the yellow module. Task 4 and Task 5, are connected in the red module. Task 6 is independent in the pink module. Do the same operations for each module as follows (1)Assign the first task as 1,the second task as 2,the third task as 4,the nth task is (2) Combine different tasks in the same module, the value is the sum of the values in the same position. (3) Write the results into the matrix. So, the matrix is represented as Fig. 2 Representation of overlapping and connection

So the advantage of marker bit method is to merge and separate different modules effectively. But the number of modules is limited. For example, one integer value only represent at most 16 overlapping and connective matrix at once because of the limitation of integer representative range.

2.2 Two Standards There are two standards that decide the value of solution in Placement. ¾ ¾

Overlapping evaluation value maximal free matrix

2.2.1 Overlapping Evaluation Value The overlapping evaluation value shows overlapping situation. The formula is as follows. Cvalue = v2 *1 + v3* 2 + v4 * 4 + v5*8 + v6 *16.... + vi * 2i−2 + …… (1)

vi represented the number of points in which there are overlapping i modules . Just like the above example, there are three points resulted from two modules overlapped and one point produced by three modules overlapped, so the overlapping evaluation value is 3*1+1*2=5. In a word, the overlapping evaluation value is bigger, the overlapping is more.

200

L. Zhong et al.

2.2.2 Maximal Free Matrix The maximal free matrix is defined as the free matrix having the biggest free such as the following figure Fig.3. The yellow is the maximal free matrix whose parameter is 9*4=36 Fig. 3 A solution of Placement

Obviously, the less of overlapping evaluation value and bigger of maximal free matrix, the solution is much better. In practice, the overlapping evaluation value has higher priority

3 Improved Greedy In order to avoid getting local optimization, the solution is obtained from global space rather than local space by probability controlling.

3.1 Global Optimized Placement 3.1.1 Movement And Rotation New solution is obtained by movement and rotation of some modules. Movement is to change module position to produce a new solution. For example, Fig.a is an initial solution includes 5 modules, so we get a new solution Fig.5 through moving module 3 from the coordinate (12,1) to the coordinate (12,8). Through rotating module 5, we also get new solution Fig.6 .

Fig. 4 Initial solution

Fig. 5 Movement

Fig. 6 Rotation

An Improved Greedy Based Global Optimized Placement Algorithm

201

3.1.2 Global Selection It is an important problem to get new solution in global arrange which rests the required number of movement and rotation at every time. All solution can be produced if all modules are moved and rotated at each time. The less is the number of movement, the smaller is the solution space, which will result in local minimization. But the number of movement is too much, randomcity and time will increase. So we proposed a dynamical selection to solve this problem. The basic idea of dynamical selection is that if there are i modules in one solution space, k modules are selected to do movement and rotation. K is a random value is chosen by the following rules k is in the range of (1…i) Given, f(v)is the probability of k=v. Then,

(2)

For example, a solution including 5 modules selected k modules to operate.

， ， ， ，

The probability of K=1 is 50% The probability of k=2 is 25% The probability of k=3 is 12.5% The probability of k=4 is 6.25% The probability of k=5 is 6.25%

This probability controlling will not only guarantee selection to be in a small range to get the best solution quickly but also have a certain probability to get new solution in larger range even in the global space. Pseudo code Begin GetRandomNum(A) V=GetRan(1);//get a Random number in (0…1) K=1; While(v<1) Begin V=V*2; If(k>=A) return A; K++; End while Return

；

k

End GetRandomNum

202

L. Zhong et al.

3.2 Optimization Generally, Moving modules to the corner can get better solutions. So the paper propose the weight matrix to decide the movement. The basic idea of placement optimization is the probability of moving to the corner is bigger than moving to the center. the formula is Map[i][j]=|i-m/2|+|j-n/2|+1; i=0…m j=0…n

(3)

m,n is the range that the module can move. For example,There is a module. 0101 0111 1100 The top-left point is the basic point A, the position of A in the range of 5*9 is 000000000 000000000 000000000 000000000 000000000 So point A only move in the yellow range, that is map[3][5]. Based the formula (3), The weight matrix is

The nearer to the corner, the bigger is the weight from the above matrix, which will optimize the solution. On the basis of weight matrix, add the former date to the later data to produce selection matrix is as follows.

From selection matrix, we can get movement coordinate. The detail steps are as follows.

∈

1. Get a random value in [0…43] such as 17 . 2.decide which area k belong to. Such as 17 [16,19] this is the area between map[0][4] and map[1][0]. 3.Get the point that module move. Such as we get point (1,0). 4.move the module to the point. Such as We get a new solution :

An Improved Greedy Based Global Optimized Placement Algorithm

203

000000000 010100000 011100000 110000000 000000000

3.3 Test Data Based the test data,The inproved Greedy can get better solution and cost less time than other algoritms. Improved Greedy avoid local solution successfully and become more efficient. Table 1 Test data between algorithms Parameters Algorithm

Complex O

Maximal Free Matrix

Executions Time

Test Case

（Average） 98.3 （Average） 16×3（48）

103.4

2.103s

Test1

3.067s

Test1

4.766s

Test1

（Average） 89.7 （Average） 8×5（40）

100.6

1.943s

Test2

3.264s

Test2

4.747s

Test2

86.6

1.487s

Test3

3.177s

Test3

4.664s

Test3

1.016

Test4

1.432

Test4

4.101s

Test4

Greedy

T*n* mn

SA

T*L*n* mn

GA

n×mn×P

Greedy

T*n* mn

SA

T*L*n* mn

GA

n×mn×P

Greedy

T*n* mn

SA

T*L*n* mn

GA

n×mn×P

（Average） 84.3 （Average） 6×8（48）

Greedy

T*n* mn

132

SA

T*L*n* mn

GA

n×mn×P

） （

（ Average

136.7 Average 9×12 108

） （ ）

n is the number of pattern, mn is the number of units of the pattern.

204

L. Zhong et al.

4 Conclusion The paper propose two method—Dynamical selection and The weight matrix. Through the dynamical selection, Greedy is provided with an ability of global selection, which avoid to get local solution. What is more, The weight matrix make Greedy more efficient. On the other hand, Marker bit method that is used in placement to solve the problem of overlap and connection make Greedy own better stability and accuracy. Acknowledgments. The work is a part of NOC. My Guide teacher, Teacher Yuan, suggest a lot of good advice for my article. On the other hand, My classmate help me complete the article. I would like to thank my teacher and my classmate for so many days to help me.

References 1. Bazargan, K., Kastner, R., Sarrafzadeh, M.: Fast Template Placement for Reconfigurable Computing Systems. IEEE Design and Test of Computers 17, 68–83 (2000) 2. Walder, H., Plazner, M., Thiele, L.: Online Scheduling and Placement of real-time asks to partially Reconfigurable Devices. In: Real-Time Systems Symposium (RTSS), pp. 224–225 (2003) 3. Walder, H., Steige, C., Plazner, M.: Fast Online Task Placement on FPGAs: Free Space Partitioning and 2D Hashing. In: International Parralel and Distribute Processing Symposium (IPDPS), p. 178 (2003) 4. Ahmadinia, A., Bodda, C., Teich, J.: A Dynamic Scheduling and Placement Algorithm for Reconfigurable Hardware. In: Müller-Schloer, C., Ungerer, T., Bauer, B. (eds.) ARCS 2004. LNCS, vol. 2981, pp. 125–139. Springer, Heidelberg (2004) 5. Handa, M., Vemuri, R.: An Efficient Algorithm for Finding Empty Space for Online FPGA placement. In: Design Automation Conference (DAC), June 2004, pp. 960–965 (2004) 6. Handa, M., Vemuri, R.: An Integrated online Scheduling and Placement Methodology. In: Becker, J., Platzner, M., Vernalde, S. (eds.) FPL 2004. LNCS, vol. 3203, pp. 444–453. Springer, Heidelberg (2004) 7. Cui, J., Deng, Q., He, X., Gu, Z.: An Efficient Algorithm for Online Management of 2D Area of Partially Reconfigurable FPGAs. In: Design Automation and Test in Europe (DATE), pp. 129–134 (2007) 8. Cui, J., Gu, Z., Liu, W., Deng, Q.: An Efficient Algorithm for Online Soft Real-time Task Placement on Reconfigurable Hardware Device. In: IEEE International Symposium on Object/Component/Service-Oriented Real-time Distributed Computing (ISORC), pp. 321–328 (2007)

An Alternative Fast Learning Algorithm of Neural Network Pin-Hsuan Weng, Chih-Chien Huang, Yu-Ju Chen, Huang-Chu Huang, and Rey-Chue Hwang*

Abstract. In this paper, an alternative fast learning algorithm for supervised neural network was proposed. Both linear multi-regression and back-propagation learning methods were used alternately in the network’s training process. This new learning way is expected to improve the learning efficiency and accuracy of neural network in the real applications. To demonstrate the superiority of the learning method we developed, several examples were simulated. The conventional back-propagation learning method was also performed as the comparison with the new method proposed. From the simulation results shown, the new method we proposed not only has the faster learning speed, but also has the better learning efficiency. Keywords: Alternative, Fast learning, Neural network, Linear multi-regression, Back-propagation.

1 Introduction In the past two decades, neural network (NN) has been widely and successfully applied into many areas due to its powerful capabilities of learning and adaptation [1-3]. Usually, error back-propagation (BP) is the most known and popular learning algorithm used in the NN applications. It is an iterative gradient method to define the parameters of a multi-layer network. However, the slow convergence and easy plunging into the local minimum are two main drawbacks of BP learning algorithm. Pin-Hsuan Weng, Chih-Chien Huang, Rey-Chue Hwang Electrical Engineering Department, I-Shou University, Kaohsiung County 840, Taiwan, China

*

Yu-Ju Chen Information Management Department, Cheng Shiu University, Kaohsiung 833, Taiwan, China Huang-Chu Huang Electric Communication Department, Kaohsiung Marine University, Kaohsiung 811, Taiwan, China [email protected] H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 205–212. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

206

P.-H. Weng et al.

Basically, the learning rate and momentum are two parameters associated with BP learning. They are selected as constants in the most of NN applications. An improper setting for these two values may either slow down the learning process or may even prevent the convergence of the network. Therefore, how to speed up the training and improve the learning accuracy of NN have been proposed by many researches [4-13]. In these past studies, linear multi-regression (LMR) method was also investigated and used for the optimization of NN [9-12]. Such an idea is to partition the overall weights of NN into two parts, i.e. nonlinear part and linear part. It assumed that the relationship between output layer and hidden layer are linear so that the weights between these two layers can be solved by LMR method directly. The weights between hidden layer and input layer are still nonlinear so that the optimal values of weights are searched by using gradient error BP method. According to our research experiences, undoubtedly, LMR algorithm certainly can approach the best solution for a two-layer neural model, i.e. NN includes input and output layers only. But, for the multi-layer NN model, how to determine the appropriate number of training epoch for the weights between input and hidden layers is still a critical study problem. In other words, after the process of LMR in the linear part of NN model, the follow-up nonlinear training between hidden layer and input layer becomes a very important step for an optimal NN model. The inappropriate training number setting easily makes NN model be divergent or become a worse trained model. In this study, an alternative supervised learning algorithm for NN model was proposed. Both LMR and BP learning methods were used alternately in the network’s training. The mechanism for the determination of training epoch in nonlinear BP learning part was developed and reported. In Section 2, Both BP learning method and LMR algorithm are briefly described. Section 3 reports the learning mechanism we proposed. The simulations and comparisons by using two methods are presented in Section 4 and then a conclusion is made in Section 5.

2 BP Learning and LMR Algorithms 2.1 BP Learning Method Fig. 1 shows the architecture of a three-layer neural model. The sigmoid function, f ( x) = 1 /(1 + exp(− x)) , is used as the signal transfer function for each neuron in the model. At here, we assume that the size of NN is n-m-1, i.e. n nodes in the input layer, m nodes in the hidden layer and one node in the output layer. Denote

ωij1

is

j th unit of hidden layer and i th unit of input layer th and ω 2j is the weight connection between j unit of hidden layer and output

the weight connection between

layer. Let zik ,

sk , ok and r jk be the i th input, desired output, actual output and

An Alternative Fast Learning Algorithm of Neural Network

207

the output of j th hidden node for input pattern k. δˆk and δ jk are defined as the error terms of output node and j th hidden node, respectively. Based on the steepest gradient descent algorithm, BP learning rule can be summarized as follows [3]. Step 1: Initialize all weights to small random values. Step 2: Present an input pattern k and specify the desired output sk . Step 3: Calculate the network output using the present weights. Step 4: Find the error terms. For the output node:

δˆk = ( s k - o k )o k (1 - o k )

(1)

For the hidden node j:

δ jk = r jk (1 - r jk )δˆk ω 2j

(2)

ω 2j (k + 1) = ω 2j (k ) + ηδˆk rjk + α (ω 2j (k ) − ω 2j (k − 1))

(3)

ω ij1 ( k + 1) = ω ij1 ( k ) + ηδ jk z ik + α (ω ij1 ( k ) − ω ij1 ( k − 1))

(4)

Step 5: Adjust weights by

Where (k+1), (k) and (k-1) are next, present and previous iterations, respectively η is the learning rate and α is the momentum. Step 6: Present next input pattern k+1 and then go back to step 2. All training inputs are presented cyclically until the weights are stabilized. Fig. 1 A three-layer neural model

2.2 LMR Method Regression analysis is the method for exploring relationships between response variable and predictor variables. LMR can obtain the optimal solution, if the data

208

P.-H. Weng et al.

base is linear and good for estimation. Here, the LMR method is briefly described as follows. Usually, a prediction linear model can be expressed as

yˆ = bˆ0 + bˆ1 x1 + bˆ2 x 2 + In this expression,

+ bˆn x n

(5)

yˆ represents a predicted value for the model based on speci-

fied values of the variables x1 , x 2 , ..., x n . The estimated regression coefficients are

bˆ0 , bˆ1, bˆ2 , ..., bˆn . Suppose the data is denoted as ( x1i , x2i , , xni , y ni ) , i = 1, , m . Then, the multiple-linear-regression model can be represented as yi = b0 + b1 x1i + b2 x2 i + + bn x ni , for i = 1, , m . It can be rewritten as the matrix form Y = Xβ

(6)

Where, X , Y , and β

⎡1 ⎢1 ⎢ X = ⎢1 ⎢ ⎢ ⎢⎣1

x11

x21

x12 x13

x22 x23

x1m

x2 m

⎡ y1 ⎤ xn1 ⎤ ⎢y ⎥ ⎥ … xn 2 ⎥ ⎢ 2⎥ , ⎢ y3 ⎥ and β = [b0 , b1 , b2 , Y = ⎥ xn 3 ⎢ ⎥ ⎥ ⎢ ⎥ ⎥ ⎢⎣ y m ⎥⎦ xnm ⎥⎦

, bn ]T

Based on the least-squares estimation, the following equation can be obtained.

( X T X )β = X T Y

(7)

Thus, the least-squares estimator can be expressed as

βˆ = ( X T X ) −1 ( X T Y )

(8)

3 Alternative Learning Algorithm In articles [9-12], the training process by combining LMR and BP techniques was presented for improving the learning speed of NN. The training procedure can be briefly described as follows. Step 1: Initialize all weights to small random values. Step 2: Calculate the output of each hidden node for each input pattern. (Or, the NN model can be trained with some iteration firstly, then calculate the output of each hidden node.) Step 3: Develop a linear system of equations and then use LMR method to calculate the weights between output layer and hidden layer.

An Alternative Fast Learning Algorithm of Neural Network

209

Step 4: Evaluate the training error and use BP method to propagate this error to update the weights between input layer and hidden layer. Step 5: Go back to Step 2, until the convergence is achieved. However, according to our research experiences, for making NN model could be efficiently convergent, the follow-up nonlinear training between input layer and hidden layer becomes a very important step in the whole training process. In above procedure, the inappropriate training number setting easily makes NN model be divergent or become a worse trained model. To improve the learning efficiency of NN, an alternative learning method with LMR and BP technique was proposed. In this method, LMR and BP algorithms are alternately used in the training process of NN model. The training procedure is presented as follows. Step 1: Initialize all weights to small random values. Step 2: Train NN by BP learning with a certain number of iterations. In our study, 100 training epochs were set for each simulation. Step 3: Calculate the average of weight changes for all hidden nodes on the 100th 2 . epoch and denote it as Δω BP Step 4: Fix the weights between input and hidden layers, then use LMR method to find the optimal increments of weights ( Δω j 2 , j=1,…, m) between hidden and output layers. Furthermore, update the weights ω 2j for j=1,…, m. 2 . Step 5: Calculate the average value of Δω 2j , for j=1,…,m and denote it as Δω LR

2 2 . Step 6: Let N1 = Δω LR Δω BP Step 7: Fix the weights between hidden layer and output layer and then retrain the weights between input layer and hidden layer by BP method N1 epochs. In our study, we set a limit for N1 value, i.e. if N1>1000, then N1=1000 and if N1<20, then N1=20. Step 8: Go back to Step 4, until the convergence of NN is achieved.

4

Simulations

To demonstrate the superiority of learning method we proposed, the identifications of two nonlinear systems were studied and simulated. For a comparison, both systems were also performed by NN model with BP learning rule only. In order to make a clear comparison, all initial parameters of NN models with different learning techniques are the same. In addition, all neural models are executed by nine different learning rates, from 0.1 to 0.9. Simulation 1: The first plant needs to be identified is

y (k ) = x1 (k ) 2 + x2 (k ) ⋅ x3 ( k ) + (1 x2 (k ) ⋅ x4 (k )) 2

(9)

210

P.-H. Weng et al.

Fig. 2 The error curves performed by BP and proposed methods with different learning rates for simulation 1 (solid line – BP method, dotted line – proposed method)

In our study, 1000 data points are generated from above system. 700 points are used for training and 300 points are used for testing. For all simulations, the size of NN model is 4-4-1. The input vector of neural model is [ x1 (k ), x2 (k ), x3 (k ), x4 (k )] . Fig. 2 shows the error curves performed by BP and proposed methods with 9 different learning rates. From the simulation results shown, the performance of NN with proposed learning method obviously is much better than the NN model with BP learning method only.

Simulation 2: The second plant needs to be identified is

y ( k ) = 10 sin(π ⋅ x1 (k ) ⋅ x 2 ( k )) + 20( x3 (k ) − 0.5) 2 + 10 x 4 (k ) + 5 x 5 (k ) + ε

(10)

where ε is the noise term which is uniformly distributed between [0, 1]. For this plant, 1000 data points are generated. 700 points are used for training and 300 points are used for testing. The size of NN model is 5-4-1. The input vector of neural model is [ x1 (k ), x2 (k ), x3 (k ), x4 (k ), x5 (k )] . Fig. 3 shows the error curves performed by two methods with 9 different learning rates. From the simulation

An Alternative Fast Learning Algorithm of Neural Network

211

Fig. 3 The error curves performed by BP and proposed methods with different learning rates for simulation 2 (solid line – BP method, dotted line – proposed method)

results, same as the simulation 1, the performance of NN with proposed learning method obviously is much better than the NN model with BP learning method only.

5 Conclusion In this paper, an alternative fast learning algorithm for supervised NN was proposed. Both LMR and BP learning methods were used alternately in the network’s training process. From the simulation results, it is clearly to find that the performance of NN model with alternative learning method is much better than the NN model with BP learning method only. This method not only makes NN have the faster learning efficiency, but also can improve the drawbacks existing in the methods proposed in past related researches. However, only the stationary signal processing problem was studied in this paper. In our advanced study, the nonstationary signal processing problem will be continuously studied in our future works.

212

P.-H. Weng et al.

Acknowledgments. This work is supported by the National Science Council of Republic of China under contract No. NSC-97-2221-E-214-069.

References [1] Carpenter, G.A., Grossberg, S.: A Massively Parallel Architecture for a Selforganizing Neural Pattern Recognition Machine. Computer Vision, Graphics, and Image Processing 37, 55–115 (1987) [2] Chen, S., Billings, S.A., Grant, P.M.: Non-linear System Identification Using Neural Network. International Journal of Control 51, 1191–1214 (1990) [3] Kaotanzad, A., Hwang, R.C., Abaye, A., Maratukulam, D.: An Adaptive Modular Artificial Neural Network Hourly Load Forecaster and Its Implementation at Electric Utilities. IEEE Trans. On Power System 10, 1716–1722 (1995) [4] Leonard, J., Kramer, M.A.: Improvement to The Back-propagation Algorithm for Training Neural Network. Computers and Chemical Engineer 14, 337–341 (1986) [5] Azmi, M.R., Liou, R.: Fast Learning Process of MLPNN Using Recursive Least Squares Methods. IEEE Trans. on Signal Processing 4, 446–450 (1993) [6] Biegler, F., Omatu, S.: A Learning Algorithm for Multilayer Neural Networks Based on Linear Least Squares Problems. Neural Networks 6, 127–131 (1993) [7] Verma, B.K., Mulawka, J.J.: Training of The Multiplayer Perceptron Using Direct Solution Methods. In: Proc. 12th IASTED Int. Conf. Appl. Inform., pp. 18–24 (1994) [8] Mulawka, J.J., Verma, B.K.: Improving The Training Time of the Backpropagation Algorithm. Int. Journal of Microcomputer Application 13, 85–89 (1994) [9] Verma, B.: Fast Learning of Multilayer Perceptrons. IEEE Trans. on Neural Networks 8, 1314–1320 (1997) [10] Stager, F., Agarwal, M.: Three Methods to Speed up The Training of Feedforward and Feedback Perceptrons. Neural Networks 10, 1435–1443 (1997) [11] Xu, C.H., Xu, X.D.: Research of New Learning Method of Feedforward Neural Network. Journal of TSINGHUA University (Science and Technology), 301–307 (1999) [12] Stubberud, P., Bruce, J.W.: LMS Algorithm for Training Single Layer Globally Recursive Neural Networks. In: IEEE International Conference on Neural Networks Conference Proceedings, vol. 3, pp. 2214–2217 (1998) [13] Abid, S., Fnaiech, F., Najim, M.: A Fast Feedforward Training Algorithm Using a Modified Form of The Standard Backpropagation Algorithm. IEEE Trans. on Neural Networks 12, 424–430 (2001)

Computer Aided Diagnosis of Alzheimer’s Disease Using Principal Component Analysis and Bayesian Classifiers ´ Miriam L´ opez, Javier Ram´ırez, Juan M. G´orriz, Ignacio Alvarez, Diego Salas-Gonzalez, Fermin Segovia, and Carlos Garc´ıa Puntonet

Abstract. Functional brain imaging with PET (Positron Emission Tomography) and SPECT (Single Photon Emission Computed Tomography) has a definitive and well established role in the investigation of a variety of conditions such as Alzheimer’s Disease (AD). Nowadays the inspection of PET and SPECT images is performed by expert clinicians, but usually entails time consuming and subjective steps. This work aims at providing an automatic tool to assist the interpretation of SPECT and PET images for the diagnosis of AD. The main problem to be handled is the so-called small size sample, which consists in having a small number of available images compared to the large number of features. This problem is faced up by reducing intensively the dimension of the feature space by means of Principal Component Analysis (PCA). Our approach is based on bayesian classifiers, which uses the a posteriori information to determine to which class the subject belongs, yielding 88.6% and 98.3% accuracy for SPECT and PET images respectively. Keywords: SPECT, PET, Alzheimer type dementia, Principal component analysis, Bayesian classification.

1 Introduction Alzheimer’s Disease (AD) is a neuro-degenerative disease that causes memory loss, behavioral changes and cognitive impairment. Mainly elderly are ´ Miriam L´ opez · Javier Ram´ırez · Juan M. G´ orriz · Ignacio Alvarez · Diego Salas-Gonzalez · Fermin Segovia Dept. of Signal Theory, Networking and Communications University of Granada, Spain Carlos Garc´ıa Puntonet Dept. of Architecture and Computer Technology, University of Granada, Spain H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 213–221. springerlink.com Springer-Verlag Berlin Heidelberg 2009

214

M. L´ opez et al.

affected by this disease. Because of the aging populations, the number of Alzheimer’s patients is expected to increase dramatically in coming years, straining the health care system. Single Photon Emission Computed Tomography (SPECT) and Positron Emission Tomography (PET) are being largely used for the study of the regional Cerebral Blood Flow (rCBF) and cell metabolism respectively, and provide unique information for the identification of functional abnormalities relevant to AD. With these methods it is possible to detect early rCBF changes seen in this dementia (even before clinical symptoms) and differentiate AD from other dementias by means of the rCBF pattern change. In the interpretation of these cerebral functional images, well-trained classifiers provide effectively diagnostic information. Both SPECT and PET are non-invasive, three-dimensional functional imaging modalities that provide clinical information regarding biochemical and physiologic processes in patients. The evaluation of these images is usually done through visual ratings performed by experts. However, maybe due to the large amounts of data represented in comparison with the number of available imaged subjects (typically <100), statistical classification methods have not been widely used in this area. Several approaches have been recently proposed in the literature aiming at providing an automatic tool that guides the clinicians in the AD diagnosis process. The Voxels-As-Features (VAF) approach [1], which will be used as a reference, uses the brain voxels intensities to train a Support Vector Machine (SVM) classifier. The main problem to face up is the small size sample, that is, the number of available samples to train a classifier is much lower than the features used in the training step. Other approaches [2, 3, 4] try to resolve this problem by searching the most discriminant features or the brain regions of interest (ROIs), which are used to train the classifier. In this work, a fully computer aided diagnosis (CAD) system for the early detection of AD is shown. The information used in the classification step is obtained from a previous feature extraction phase. The selected features to be used in the training step undergo a Principal Component Analysis (PCA) transformation which allows us to reduce drastically the dimension of the feature space so the small size sample problem is solved. The resultant data are used to make up a bayesian classifier which makes use of the a posteriori information to classify the coming images of a new patient.

2 Image Preprocessing As in previous approaches [1] the classification task is based on the assumption that the same position in the volume coordinate system in different volumes corresponds to the same anatomical position. However this assumption is not met by the images without pre-processing: The subject who is being imaged is not always positioned at the same position in the reference frame of the

CAD of Alzheimer’s Disease Using Principal Component Analysis

215

Fig. 1 Average image of the dataset, seen as consecutive transaxial slice images

imaging device and the anatomy does not always have the same shape and size among different subjects. This means that registering the volumes spatially is needed. Spatial registration is done by an implementation of the algorithms proposed in [5]. SPECT imaging generates volumes that only give a relative measure of the blood flow, whereas PET images represent information about the cells metabolism. In order to make possible a direct comparison of the voxel intensities between SPECT or PET images, a previous normalization of the intensities is needed. For all the experiments, we normalize the images by applying an affine transformation to the intensities as suggested also in [5]. All the images of the database are transformed using affine and non-linear spatial normalization, thus the basic assumptions are met. After the normalization steps, about 105 voxels remain for each patient. As proposed in [2, 3], we first construct a binary mask which selects the voxels of interest and discards the rest. This is done by taking the voxels whose mean intensity value averaged over all images exceeds the half of the maximum mean intensity value, and this mask is applied to the original images. In the resulting averaged images the irrelevant information has been removed or reduced. Fig. 1 represents the average image along the transaxial axis. The mask application will reject those voxels whose intensity values are lower than 0.5.

3 Principal Component Analysis and Eigenbrains PCA generates a set of orthonormal basis vectors, known as principal components, which maximize the scatter of all the projected samples. After the preprocessing steps, the remaining voxels are rearranged in a vector form. Let X = [X1 , X2 , ..., Xn ] be the sample set of these vectors, where n is the number of patients, each of dimensionality N . After normalizing the vectors to unity norm and subtracting the grand mean, a new vectors set Y = [Y1 , Y2 , ..., Yn ] is derived. Therefore, each Yi represents a normalized voxels vector with dimensionality N , Yi = (yi1 , yi2 , ..., yiN )t , i = 1, 2, ..., n. The covariance matrix of the normalized vectors set is defined as 1 1 ΣY = Yi Yit = YYt n i=1 n n

(1)

and the eigenvector and eigenvalue matrices Φ, Λ are computed as ΣY Φ = ΦΛ.

(2)

216

M. L´ opez et al.

Note that YYt is an N × N matrix while Yt Y is an n × n matrix. If the sample size n is much smaller than the dimensionality N , then diagonalizing Yt Y instead of YYt reduces the computational complexity [6] (Yt Y)Ψ = ΨΛ1

(3)

T = YΨ

(4)

where Λ1 = diag{λ1 , λ2 , ..., λn } and T = [Φ1 , Φ2 , ..., Φn ]. Derived from the eigenface concept [6] and due to its still brain-like appearance, the eigenbrains correspond to the dominant eigenvectors of the covariance matrix. In this approach, only m leading eigenvectors are used, which define the matrix P P = [Φ1 , Φ2 , ..., Φm ].

(5)

The criterion to choose the most discriminant eigenbrains is set by their separation ability, which is measured by the Fisher Discriminant Ratio (FDR), defined as F DR =

(μ1 − μ2 )2 σ12 + σ22

(6)

where μi and σi denote the i-th class within class mean value and variance, respectively. Fig. 2 represents the aspect of the three main eigenbrains of three slices along the transaxial axis. For the classification task we project each patient vector into the previously defined eigenbrain space, producing a vector of weights. For the whole database, a matrix of weights can be constructed, given by: Z = Pt Y. Fig. 2 Three representative transaxial slices for the first three eigenbrains

(7)

CAD of Alzheimer’s Disease Using Principal Component Analysis

217

4 Bayes Classifier For pattern recognition, the Bayes classifier is the best classifier in terms of minimum Bayes error, therefore the a posteriori probability functions will be evaluated [7]. Let ω1 and ω2 denote the object classes (AD and NORMAL), and Z a patient voxels vector in the reduced PCA subspace. The a posteriori probability function of ωi given Z is defined as P (ωi |Z) =

p(Z|ωi )P (ωi ) , p(Z)

i = 1, 2.

(8)

where P (ωi ) is a priori probability, p(Z|ωi ) the conditional probability density function of Z given ωi , and p(Z) is the mixture density. The maximum a posteriori (MAP) decision rule for the Bayes classifier is defined as p(Z|ωi )P (ωi ) = max{p(Z|ωj )P (ωj )}, j

Z ∈ ωi .

(9)

The brain projected data Z is classified to ωi of whom the A Posteriori probability given Z is the largest between the classes. Usually there are not enough samples to estimate the conditional probability density function for each class (within class density). A compromise, therefore, is to make an assumption of a particular density form, and convert the general density estimation question into a parametric one. The within class densities are usually modeled as normal distributions  8 1 1 t −1 p(Z|ωi ) = × exp − (Z − Mi ) Σi (Z − Mi ) 2 (2π)m/2 |Σi |1/2

(10)

where Mi (see (11)) and Σi are the mean and covariance matrix of class ωi , respectively.

4.1 Probabilistic Reasoning Model (PRM) Under the unified Bayesian framework, a new probabilistic reasoning model, PRM-1 is derived in [8], which utilizes the within class scatters to derive averaged estimations of within class covariance matrices. For the PRM-1 model, in the reduced PCA subspace, we assume all the within class covariance matrices are identical, diagonal, and each diagonal element is estimated by the sample variance in the one dimensional PCA subspace. As a result, the conditional probability density function specifies the MAP rule as a quadratic classifier characterized by Mahalanobis distance. In particular, let ω1 , ω2 and N1 , N2 denote the classes and number of patients within each class, respectively. Let M1 , M2 be the means of the classes in the reduced PCA subspace span [Φ1 , Φ2 , ..., Φm ]. We then have

218

M. L´ opez et al.

Mi =

Ni 1 (i) Z , Ni j=1 j

i = 1, 2.

(11)

(i)

where Zj , j = 1, 2, ..., Ni represents the sample voxels vector for the ωi class. The PRM-1 model assumes the within class covariance matrices are identical and diagonal " 2 Σi = diag σ12 , σ22 , ..., σm . (12) Each component σi2 can be estimated by sample variance in the one dimensional PCA subspace ⎧ ⎫ Nk # L ⎨ $2 ⎬ 1 1 (k) σi2 = zij − mki (13) ⎩ Nk − 1 ⎭ L j=1 k=1

(k)

(k)

where zij is the i-th element of the sample Zj , mki the i-th element of Mk , and L the number of classes (two in our approach). From (10) and (12) it follows ⎧ ⎫ m ⎨ 2⎬ (zj − mij ) 1 1 9m p(Z|ωi ) = × exp − (14) ⎩ 2 ⎭ σj2 (2π)m/2 j=1 σj j=1

Thus the MAP rule (9) specifies a quadratic classifier characterized by Mahalanobis distance (note that the priors are set to be equal). ⎧ ⎫ m m ⎨ 2 2⎬ (zj − mij ) (zj − mkj ) = min (15) ⇒ Z ∈ ωi ⎭ k ⎩ σj2 σj2 j=1

j=1

5 Evaluation Results SPECT and PET images used in this work were taken with a PRISM 3000 machine and a SIEMENS ECAT 47 respectively. Initially they were labeled by experienced clinicians of the “Virgen de las Nieves” Hospital (Granada, Spain) and “Cl´ınica PET Cartuja” (Seville, Spain) respectively. The database consists of 79 SPECT patients (41 labeled as NORMAL and 38 labeled as AD) and 60 PET patients (18 NORMAL and 42 AD). Initially, the original brain image 79 × 95 × 69 voxel sized is reduced by averaging over subsets of 4 × 4 × 4 voxels. After applying the mask, the remaining voxels are rearranged into a vector form so that PCA can be applied to the training set and eigenbrains are obtained. The new patient to be classified is projected into the eigenbrain space, the a posteriori probabilities P (Z|ω1 ) and P (Z|ω2 ) are

CAD of Alzheimer’s Disease Using Principal Component Analysis

219

100

50

AD NORMAL

0

−50

−100

−150 100 50 0 −50 −100

−30

−20

−10

30

20

10

0

Fig. 3 For PET images, distributions of the three first principal coefficients for AD and NORMAL patients

computed (where ω1 = NORMAL and ω2 = AD) and the MAP rule (14) is applied. Fig. 3 represents the three main PCA coefficients as 3D points for NORMAL and AD patients. Fig. 4 shows the accuracy values obtained for SPECT and PET images when the number of considered principal component m increases. The accuracy results were obtained by testing the classifier with the LeaveOne-Out method, that is, the classifier is trained using all but one patient, which is used in the test phase. This procedure is repeated for each patient and an average accuracy value for all the experiments is obtained. The VAF CLASSIFICATION RESULTS 100

90

Accuracy

80

70

60 PET SPECT

50

40 0

5

10

15

20 25 30 # PCA Coefficients

35

40

45

50

Fig. 4 Accuracy for SPECT and PET images using bayesian classifiers when the number m of considered principal components increases

220

M. L´ opez et al.

Table 1 Accuracy results obtained from SPECT and PET images SPECT PET VAF approach 77.2% 96.7% Eigenbrains approach 88.6% 98.3%

approach was also implemented and tested by the same cross-validation strategy. Results can be compared in Table 1.

6 Conclusions A computer aided diagnosis system for the early detection of the Alzheimer disease was shown in this paper. The system was developed by performing the principal components analysis to a subset of remaining voxels after the preprocessing steps, which allows us to face up the small size problem and reduce drastically the feature space dimension. The most important components in terms of ability to separate are chosen to make up a bayesian classifier. The a posteriori information is used when a new patient is needed to be classified. With this approach, 88.6% and 98.3% accuracy values are obtained for SPECT and PET images respectively. These results outperform the accuracy values attained by VAF approach, which has been taken as reference work. Acknowledgements. This work was partly supported by the Spanish Government under the PETRI DENCLASES (PET2006-0253), TEC2008-02113, NAPOLEON (TEC2007-68030-C02-01) projects and the Consejer´ıa de Innovaci´ on, Ciencia y Empresa (Junta de Andaluc´ıa, Spain) under the Excellence Project (TIC-02566).

References 1. Fung, G., Stoeckel, J.: SVM Feature Selection for Classification of SPECT Images of Alzheimer’s Disease Using Spatial Information. Knowledge and Information Systems 2(11), 243–258 (2007) 2. G´ orriz, J.M., Ram´ırez, J., Lassl, A., Salas-Gonzalez, D., Lang, E.W., Puntonet, ´ C.G., Alvarez, I., L´ opez, M., G´ omez-R´ıo, M.: Automatic Computer Aided Diagnosis Tool Using Component-Based SVM. In: Medical Imaging Conference, Dresden (2008) ´ 3. Alvarez, I., L´ opez, M., G´ orriz, J.M., Ram´ırez, J., Salas-Gonzalez, D., Puntonet, C.G., Segovia, F.: Automatic Classification System for the Diagnosis of Alzheimer Disease Using Component-Based SVM Aggregations. In: Proc. International Conference on Neural Information Processing (2008) 4. Ram´ırez, J., G´ orriz, J.M., Salas-Gonzalez, D., Lassl, A., L´ opez, M., Puntonet, C.G., G´ omez, M., Rodr´ıguez, A.: Computer Aided Diagnosis of Alzheimer Type Dementia Combining Support Vector Machines and Discriminant Set of Features. Accepted in Information Sciences (2008)

CAD of Alzheimer’s Disease Using Principal Component Analysis

221

5. Friston, K.J., Ashburner, J., Kiebel, S.J., Nichols, T.E., Penny, W.D.: Statistical Parametric Mapping: The Analysis of Functional Brain Images. Academic Press, London (2007) 6. Turk, M., Pentland, A.: Eigenfaces for Recognition. Journal of Cognitive Neuroscience 13(1), 71–86 (1991) 7. Fukunaga, K.: Introduction to Statistical Pattern Recognition, 2nd edn. Academic Press, London (1991) 8. Cheng, j.L., Wechsler, H.: A Unified Bayesian Framework for Face Recognition. In: International Conference on Image Processing, ICIP 1998, pp. 151–155 (1998)

Margin-Based Transfer Learning Bai Su, Wei Xu, and Yidong Shen

Abstract. To achieve good generalization in supervised learning, the training and testing examples are usually required to be drawn from the same source distribution. However, in many cases, this identical distribution assumption might be violated when a task from one new domain(target domain) comes, while there are only labeled data from a similar old domain(auxiliary domain). Labeling the new data can be costly and it would also be a waste to throw away all the old data. In this paper, we present a discriminative approach that utilizes the intrinsic geometry of input patterns revealed by unlabeled data points and derive a maximum-margin formulation of unsupervised transfer learning. Two alternative solutions are proposed to solve the problem. Experimental results on many real data sets demonstrate the effectiveness and the potential of the proposed methods.

1 Introduction The problem of transfer learning, where information gained in one learning task is used to improve performance in another related task, is an important new area of research. This paper concerns the subproblem of domain adaptation, in which the information learned from an auxiliary domain is generalized to help learning in a related target domain, where these two domains data are distributed similarly, but not identically. Previous works [1, 13, 7, 6] have studied the supervised version of this problem in which Bai Su · Yidong Shen State Key Laboratory of Computer Science, Institute of Software, Chinese Academy of Sciences {subai,ydshen}@ios.ac.cn Bai Su · Wei Xu NEC Laboratories America, Inc. [email protected] H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 223–233. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

224

B. Su, W. Xu, and Y. Shen

labeled data from both auxiliary and target domains are available for training. In this work, however, we study the more challenging problem of unsupervised transfer learning, where no labeled data from the target domain are available at training time, but instead, unlabeled target test data are available during training. We are specially interested in applying maximum margin method to transfer learning. Maximum margin methods, such as Support Vector Machine and its extensions, have significantly advanced the state of the art for classification. In particular, SVMs have excellent learning, classification ability and generalization ability which use structural risk minimization instead of empirical risk minimization. However, these methods usually perform poorly as traditional classification methods do in the scenario of transfer learning since they all assume the source and target domains are distributed identically. Recently, there have been some research [1, 13] to apply maximum margin methods to transfer learning, but they are all restricted to semi-supervised transfer learning, i.e. a small portion of target data have been labeled for training. In this paper, we introduce the maximum margin method into unsupervised transfer learning for the first time. The major obstacle to apply maximum margin method to unsupervised transfer learning is the missing labels of target data while traditional maximum margin methods require supervised information to find the maximum margin separating different classes. To overcome this difficulty, we treat the supervised information provided by the classifier trained in auxiliary domain as the prior information and proposed an iterative approach based on alternating optimization to make the classification results consistent with the maximum margin in the target data. The main advantage of this tranductive learning is that we can use any classifier as the source of prior information, which provide a certain flexibility in practice. However it does not provide an appropriate model for the target domain and a new round of computation is required when new data arrives. To avoid this problem, we further propose to train a SVM classifier for the target domain by modifying the regularizer in the objective function which is solved under the same alternating optimization framework as before.

2 Problem Formulation We consider the problem of a binary classification task with respect to a target data set Dt , where all examples are unlabeled. Besides, there is a fully-labeled auxiliary data set Da which comes from the auxiliary domain, and an auxiliary classifier f (x) has been trained from it. A classifier is a function mapping a data point xi (i = 1..n) with m attributes xji (j = 1..m) to yi where yi = {±1}. The target data is drawn from a distribution that

Margin-Based Transfer Learning

225

is related to, but different from the auxiliary data in a way unknown to the learner. The objective is to label the data in Dt as accurate as possible.

3 Maximum Margin Method The goal of an SVM is to find the linear discriminant f (x) = wT φ(x) + b (φ(x) is the kernel function) that maximizes the minimum misclassification margin by solving the following optimization problem: 1 w2 + Cξ T e 2

min

w,b,ξi

s.t. yi (wT φ(xi ) + b) ≥ 1 − ξi , ξi ≥ 0  where i ξi measures the total classification error, w2 is a regularization term that is reversely related to margin between data points of two classes, C > 0 is a regularization parameter, and e is the vector of ones. We aim at extending maximum margin methods to transfer learning. A trivial solution could be to assign all data points to the same class, as the class label y is unknown, and the resultant margin could be infinite. To avoid such a meaningless solution, a class balance constraint is introduced as − ≤ eT y ≤ 

(1)

where  ≥ 0 is a constant controlling the class imbalance. Then, the margin can be maximized by optimizing both the unknown y and the unknown parameter (w, b), as 1 w2 + C ξ T e 2 s.t. yi (wT φ(xi ) + b) ≥ 1 − ξi , ξi ≥ 0.

min

y,w,b,ξi

(2)

yi = {±1}, − ≤ et y ≤ . Recall that the dual of the SVM is 1 max αT e − αT (K ◦ yy T )α α 2 s.t. αT y = 0, 0 ≤ α ≤ Ce

(3)

where α = [α1 , . . . , αn ]T , K is the kernel matrix, and ◦ denotes the elementwise product between matrices. Hence, (2) can also be written as 1 min max αT e − αT (K ◦ yy T )α y α 2 s.t. αT y = 0, 0 ≤ α ≤ Ce yi = {±1}, − ≤ et y ≤ .

(4)

226

B. Su, W. Xu, and Y. Shen

3.1 Iterative Approach As (4) is non-convex integer programming problem and thus difficult to solve, we use a iterative approach based on alternating optimization [2]. First, fix y and maximize (3) w.r.t. α, which is standard SVM training. Since the auxiliary data is similar to the target data, the prediction result made by f (x) can be treated as a weak evidence of y. Thus we use it as the initial value of y. Then, fix α and minimize (2) w.r.t. y. Note that with a fixed α, both w and b can be determined from the KKT conditions, so the second step reduces to min ξ T e y,ξi

s.t. yi (wT φ(xi ) + b) ≥ 1 − ξi , ξi ≥ 0. Obviously, this yields yi = sign(wT φ(xi )+b). The two steps are then repeated until convergence. The second step is supposed to correct y to maintain consistency with the maximum margin through the target data set. However, the performance of the iterative approach is not satisfactory in practice. To understand this, we consider the hinge loss used by the SVM:  0 if yi f (xi ) > 1 li = 1 − yi f (xi ) otherwise. As shown in Figure 1(a), SVM tries to push yi f (xi ) to the right of the turning point where yi f (xi ) = 1. Changing the label of any point (e.g. point P) on the right of the turning point will incur relatively larger loss (Figure 1(b)) compared with the original one. Therefore, the procedure over-commits to the initial label, and thus easily gets stuck in local optima.

3.2 Iterative Approach Using Laplacian Loss Function To solve this problem, the loss function has to be changed to reduce the loss incurred by changing class label in the iterative approach. This can be done by using the -sensitive loss function (Figure 2(a)) used in support (a)

1

(b)

3 2.5

hinge losee

Fig. 1 Hinge loss and the loss incurred by changing label

2 0.5

1.5 1 0.5

0 0

0.5

1 yif(xi)

1.5

2

0 −2

−1

0 yif(xi)

1

P

2

Margin-Based Transfer Learning

227

Fig. 2  sensitive loss and Laplacian loss

(a)

(b) 1.5 Laplacian loss

ε sensitive loss

1

0.5

0 −1

−0.5 −ε 0 ε y −f(x ) i

0.5

1

i

1

0.5

0

0

0.5

1 y f(x ) i

1.5

2

i

vector regression (SVR). When  = 0, it reduced to Laplacian loss function l = |f (xi ) − yi | where yi = {±1} different from the one in standard SVR. Laplacian loss function is shown under the same reference frame as that of hinge loss in Figure 2(b). Note that because Laplacian loss is symmetric around the turning point, the yi f (xi )’s will tend to lie around the turning point in order to reduce the loss. As changing the label will not incur big loss as before, SVR can more easily get out of a poor solution. By replacing the hinge loss function with Laplacian loss function, the objective function becomes min

w,y,b,ξi ,ξi∗

n 1 w2 + C (ξi + ξi∗ ) 2 i=1

(5)

s.t. yi − (wT φ(xi ) + b) ≤ ξi −yi + (wT φ(xi ) + b) ≤ ξi∗ ξi ≥ 0, ξi∗ ≥ 0, yi = ±1 − ≤ et y ≤ . First we can fix y and then obtain w by solving the dual form of (5) and using the KKT conditions. As discussed at the beginning of Section 3, the class balance constraint (1) has to be maintained to avoid trivial solutions. Hence, each of the y’s obtained throughout the iterative process is required to satisfy (1). To guarantee this, the bias b cannot be computed according to the KKT conditions as usual. Instead, we obtain b by minimizing (5) with w being fixed. Then (5) can be reduced to min y,b

n

|wT φ(xi ) + b − yi |

(6)

i=1

s.t. yi = ±1, − ≤ et y ≤ . In this optimization problem, b practically defines a boundary on the sorted list of wT φ(xi ) + b between positive and negative. Apparently, in order to minimize the objective value, we should label those patterns whose f (xi ) value is positive as +1, while label the rest as −1. Therefore, we can shift b in the range such that the class balance constraint is satisfied, compute the objective value in (6) at each position and set b to be the position with the

228

B. Su, W. Xu, and Y. Shen

minimum objective. Sorting takes O(nlog(n)) time, and search takes O(n) time. So the procedure takes O(nlog(n)) time. y can also be obtained by solving (6) which gives yi = sign(wT φ(xi ) + b). The complete procedure is presented at Algorithm 1. Algorithm 1. Maximum Margin Transfer Learning 1: 2: 3: 4: 5: 6:

Initialize y with the classification results of f  (x). Perform SVR training with Laplacian loss. Compute w from the KKT conditions. Compute the bias b as described in Section 3.2 Assign the labels as yi = sign(wT φ(xi ) + b) Repeat steps 2-5 until convergence.

So far, we have proposed an unsupervised transfer learning approach in which the labels of target data are refined iteratively during the learning process until they are consistent with the maximum margin through target data. The main goal of this approach is to gain correct label of target data, not a model over target data. Although we do get a linear discriminant in the end, the resultant solution will not be sparse due to using Laplacian loss, which degrades its generalization ability.

3.3 Model Learning In this section, we propose another approach to unify the two goals: labeling target data and learning a new model under the same iterative optimization framework. Based on the observation that the distribution of target data is similar to that of auxiliary data, it is reasonable to assume that the discrepancy between the model of target data and the model of auxiliary data is also small. Thus we revise (5) to incorporate this assumption as: min

w,y,b,ξi ,ξi∗

n 1 w − w 2 + C (ξi + ξi∗ ) 2 i=1

(7)

s.t. yi − (wT φ(xi ) + b) ≤ ξi −yi + (wT φ(xi ) + b) ≤ ξi∗ ξi ≥ 0, ξi∗ ≥ 0, yi = ±1 − ≤ et y ≤  where w is the counterpart of w in f (x). The objective function combines two goals, i.e. minimizing the the discrepancy of two models and classification error in target data, together. Note that w +w −w  ≥ w. Because w is inversely related to the margin and w  is constant, minimizing w − w 

Margin-Based Transfer Learning

229

is also equivalent to maximizing the lower bound of the margin. As a result, the solution would have better generalization performance than the original one. The dual of (7) is: : :2 n n :   1: : : min max − : (αi − βi )φ(xi ): + (αi − βi ) yi − w T φ(xi ) (8) y : αi ,βi 2 : i=1 i=1 s.t.

n

(αi − βi ) = 0, 0 ≤ αi , βi ≤ C

i=1

yi = {±1}, − ≤ et y ≤ . With y being fixed, this turns out to be a convex problem that can be solved by Platt’s sequential minimal optimization algorithm [3]. Thus an iterative algorithm can be derived in the same manner as algorithm 1. The difference is that we will get an appropriate classification model for target domain in the end. We denote the first approach as MM1 and the second as MM2. Several differences between the two approaches need to be clarified. Comparing with MM1, MM2 utilize not only the prior label information but also the model information of auxiliary domain. For MM1, we can use any classifier desired as the auxiliary classifier, while only SVM can be applied for MM2. The goals of MM1 and MM2 are also different. The former is only to label target data, while the latter has an additional goal, i.e. to learn an appropriate SVM model.

3.4 Computational Complexity MM1 and MM2 share the same computational complexity. The main computational load of the two algorithms is a sequence of quadratic programming. Modern implementations for quadratic programming typically have an empirical time complexity between O(n) and O(n2.3 ). The number of iterations of both algorithms is usually small (under 10) in practice.

4 Related Work Transfer learning aims to apply knowledge learned from auxiliary tasks, where labeled data are usually plenty, to develop an effective model for a related target task with limited labeled data or even no labeled data. There is no formal definition of “related tasks”, and in practice it refers to either related learning problems on the same data set or the same learning problem on different data sets. Different from semi-supervised learning in which both auxiliary data and target data are distributed identically, transfer learning deals with data sets with different distributions. One line of transfer learning research focused on learning with sample selection bias or covariance shift. In this problem setting, the training samples

230

B. Su, W. Xu, and Y. Shen

is governed by an unknown distribution p(x|λ) while the unlabeled test data is governed by a different unknown distribution p(x|θ). The training and test distribution may differ arbitrarily, but there is only one true unknown conditional distribution p(x|y). Zadronzy [4] and Bickel and Scheffer [5] directly estimate the ratio p(s = 1|x, λ, θ) ∝ p(x|θ)/p(x|λ), where s is a selector variable that decides whether an example x drawn under the test distribution p(x|θ) is moved into the training set or not. In practice, p(s = 1|x, λ, θ) can be estimated by a discriminative approach [5]. Despite the advance on sample bias correction methods, several difficulties limit their applicability on our problem. One of them is the difficulty of estimating the distribution of p(x), not to mention its change across domains. Moreover, the class-conditional p(y|x) often changes, which violates the assumption of sample selection bias. Another line of transfer learning research is about semi-supervised transfer learning. In this scenario, a typical assumption being made is that the number of labeled target data is limited. Thus the problem is how to utilize auxiliary data, which is typically sufficient, to learn a high-quality classification model over target data. Several works based on SVM [1, 13], logistic regression [6], and AdaBoost [7] have revealed that auxiliary data play the role of additional training data and boost the performance of target classification model. In contrast to these works, we focus on a different scenario: unsupervised transfer learning in which no labeled target data is available. There have been a few works to deal with transfer learning in this scenario. [8] proposed a co-clustering based algorithm (CoCC) to overcome the domain difference. CDSC [9] is a spectral classification based method in which the main idea is to regularize two objectives, namely, minimizing the cut size on all the data with the least inconsistency of the auxiliary data, and at the same time maximizing the separation of the target data. We compare our algorithm with CoCC and CDSC in the next section and demonstrate the better performance of our algorithm.

5 Experiments In this section we evaluate our algorithm empirically. We focus on binary text classification problems in the experiments, although the algorithm can be extended for multi-class problems by performing binary classification recursively.

5.1 Data Sets The cross-domain data sets are generated in specific strategies using 20 Newsgroups1 , Reuters-215782 and SRAA3 . Each of these data sets has at least two 1 2 3

http://people.csail.mit.edu/jrennie/20Newsgroups/ http://www.daviddlewis.com/resources/testcollections/ http://www.cs.umass.edu/ mccallum/data/sraa.tar.gz

Margin-Based Transfer Learning

231

level hierarchical structure. Suppose A and B are two root categories in one data collection, and A1, A2 and B1, B2 are sub-level categories of A and B respectively. Let A1, B1 be the positive and negative examples in the training set respectively. Let A2, B2 be the positive and negative examples in the test set respectively. Because training and test data are in different subcategories, they distributions are different but related. The 20 Newsgroups is a text collection of approximately 20,000 newsgroup documents. Six different data sets are generated from 20 Newsgroups for evaluating cross-domain classification algorithms. Each data set contains two top categories: one as positive and the other as negative classes. Then, we split the data based on subcategories. For instance, comp vs. sci indicates that the top category comp is treated as positive class and sci is as negative. The other data sets are named in the same way. SRAA is a Simulated/Real/Aviation/Auto UseNet data set for document classification. 73,218 UseNet articles are collected from four discussion groups about simulated autos (sim-auto), simulated aviation (sim-aviation), real autos (real-auto) and real aviation (real-aviation). We use the documents in real-auto and sim-auto as training data, while real-aviation and sim-aviation as test data. The auto vs aviation data set is generated in the similar way. Reuters-21578 is one of the most used test collections for evaluating automatic text-categorization techniques. The data sets are generated for crossdomain classification in the similar ways as what we have done on the 20 Newsgroups and SRAA corpora. Three data sets, orgs vs. people, orgs vs. places and peoples vs. places, are generated for cross-domain classification. Since there are too many sub-categories, we do not list the detailed description here.

5.2 Compared Algorithms Different types of algorithms are employed for comparison with our algorithms. The first is a supervised learning algorithm SVM. SVM has been successfully applied in many applications like text classification. To implement SVM algorithm, We use libSVM with radial basis kernel function (RBF). For SVM algorithm, the error rates are recorded both in the case of transfer learning and in the case of single domain classification. We also compared our algorithm with two semi-supervised algorithms, Transductive SVM (TSVM) [11] and Spectral Classifier (SC) [12]. SVMlight is employed to implement TSVM with a linear kernel function as in [11]. Two transfer learning algorithms, CoCC and CDSC, are selected as the state-of-art transfer learning algorithms to compare with our algorithm. All parameters are set default by the software. We use test error rate as the evaluation measure.

232

B. Su, W. Xu, and Y. Shen

Table 1 Performance Comparison for Different Data Sets Data Sets

K-L

auto vs aviation real vs simulated orgs vs people orgs vs places people vs places comp vs sci rec vs talk rec vs sci sci vs talk comp vs rec comp vs talk

1.126 1.161 0.303 0.329 0.307 0.874 1.102 1.021 0.854 0.866 0.967

SVM Da − Dt Dt − CV 0.289 0.034 0.235 0.031 0.300 0.106 0.418 0.094 0.241 0.083 0.305 0.015 0.204 0.004 0.202 0.009 0.207 0.009 0.182 0.010 0.114 0.006

Semi-Supervised Transfer Learning TSVM SC CoCC CDSC MM1 MM2 0.188 0.160 0.142 0.120 0.080 0.080 0.316 0.278 0.250 0.188 0.044 0.044 0.294 0.276 0.232 0.232 0.240 0.230 0.424 0.386 0.400 0.318 0.309 0.309 0.256 0.230 0.226 0.202 0.154 0.154 0.334 0.270 0.192 0.098 0.091 0.091 0.118 0.428 0.092 0.092 0.069 0.069 0.162 0.192 0.160 0.124 0.019 0.019 0.148 0.362 0.100 0.044 0.034 0.034 0.104 0.086 0.090 0.042 0.020 0.020 0.024 0.042 0.042 0.024 0.048 0.041

5.3 Experimental Results In Table 1, Kullback-Leibler divergence values [10] between the training and test sets are presented in the second column, which indicate the distances between different distributions. It can been seen that the KL-divergence values for all the data sets are much larger than the case when we simply split the same data set into test and training data, which has a KL value of nearly zero. The next column Da − Dt shows the performance when we use the labeled auxiliary data Da to train the SVM classifier for classification and test the target data Dt , while the column Dt − CV shows the best case of performing 10-fold cross-validation on Dt . Dt − CV is the case of single domain classification, which gives the best possibly result for transfer learning classifiers. In most cases, our algorithms achieve the best performance against other methods.

6 Conclusion In this paper, we address the issue of unsupervised transfer learning and introduce the maximum margin method to it for the first time. We formulated the problem as an iterative optimization problem. Two alternative approaches have be derived, namely MM1 and MM2. While MM1 utilize the prior label information provided by the auxiliary classifier, MM2 combine both the prior label information and the model information from the auxiliary classifier. With different characters, the two approaches can be applied alternatively according to different requirements in practice. The Experimental results on many data sets show that our algorithms consistently outperform the previously proposed unsupervised transfer learning algorithms, which demonstrate the effectiveness and potential of our proposed algorithms.

Margin-Based Transfer Learning

233

References 1. Wu, P., Dietterich, T.G.: Improving SVM Accuracy by Training on Auxiliary Data Sources. In: ICML (2004) 2. Zhang, K., Tsang, I.W., Kwok, J.T.: Maximum Margin Clustering Made Practical. In: ICML (2007) 3. Platt, J.: Fast Training of Support Vector Machines Using Sequential Minimal Optimization. In: Sch¨ olkopf, B., Burges, C., Smola, A. (eds.) Advances in Kernel Methods-Support Vector Learning, pp. 185–208. MIT Press, Cambridge 4. Zadrozny, B.: Learning and Evaluating Classifiers under Sample Selection Bias. In: ICML (2004) 5. Bickel, S., Scheffer, T.: Dirichlet-Enhanced Spam Filtering Based on Biased Samples. In: NIPS (2007) 6. Liao, X., Xue, Y., Carin, L.: Logistic Regression with an Auxiliary Data Source. In: ICML (2005) 7. Dai, W., Yang, Q., Xue, G.R., Yu, Y.: Boosting for Transfer Learning. In: ICML (2007) 8. Dai, W., Xue, G.R., Yang, Q., Yu, Y.: Co-Clustering Based Classification for Out-of-Domain Documents. In: SIGKDD (2007) 9. Ling, X., Dai, W., Xue, G.R., Yang, Q., Yu, Y.: Spectral Domain-Transfer Learning. In: SIGKDD (2008) 10. Kullback, S., Leibler, R.A.: On Information and Sufficiency. Annals of Mathematical Statistics 22(1), 79–86 (1951) 11. Joachims, T.: Transductive Inference for Text Classification Using Support Vector Machines. In: ICML (1999) 12. Kamvar, S.D., Klein, D., Manning, C.D.: Spectral Learning. In: IJCAI (2003) 13. Yang, J., Yan, R., Hauptmann, A.: Adapting Svm Classifiers to Data with Shifted Distributions. In: Workshop on Knowledge Discovery and Data Mining from Multimedia Data and Multimedia Applications, Omaha, NE (2007) 14. Porter, M.: An Algorithm for Suffix Stripping Program. Program 14(3), 130– 137 (1980)

Nonlinear Dead Zone System Identification Based on Support Vector Machine Jingyi Du and Mei Wang *

Abstract. Aiming at a kind of nonlinear dead zone system, a novel identification method based on support vector machines was presented. In the method, the nonlinear dynamic characteristic of systems was expressed by using the Hammerstein model and the dead zone nonlinear model was created by using the least squares support vector machine. Furthermore, the strategy of the modified cost function was adopted to improve the accuracy of the dead zone model based on the inputs. In addition, the convergence condition of the algorithm is analyzed theoretically. Finally, the simulation on a hydraulic positioning servo system was carried out, and it is verified that the accuracy of the model identification is improved efficiently by using the proposed method. Keywords: Support vector machines, Dead zone nonlinearities, Hammerstein model, Modified cost function, Hydraulic position servo system.

1 Introduction In the industrial processes, the different degrees of dead zone (insensitive zone) exist in the mechanical and electrical systems and the hydraulic systems and control systems. This is a typical non-linear aspect. The difficulty to model this kind of systems which contains the dead zone is increased by the dead zone[1]. If the system input and output are measurable (known), the transmission characteristics of the system can be identified. By using the relationships between the inputs and the outputs in which both the high-frequency harmonic component and the fundamental frequency component exist, Li Shu-shen and Zhu Ji proposed a method to identify the systems which contain the dead zone link[2]. The method can estimate the system model parameters but fail to manipulate the asymmetric dead zone link. Dai Yi-ping et al. applied the genetic algorithms to simulate the steam turbine system which involves in the non-linear adjusted part containing the dead zone section[3]. The result can recognize the system parameters accurately, but Jingyi Du . Mei Wang School of Electricaland Control Engineering, Xi’an University of Science and Technology, Xi’an 710054, China [email protected] *

H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 235–243. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

236

J. Du and M. Wang

the dead zone is the symmetric dead zone whose slope is assumed to be 1. Xue-qi and QiuhaiCao [4] suggested the two step recognizing method based on the Legendre polynomial optimal square approximation to handle the dead zone in the system containing the asymmetric link. In practice, the systems comprising the dead-zone links probably are not only non-symmetrical, and the non-linear relationships between the inputs and the outputs are often shown [5,6]. Neural networks which are of the inherent adaptability and the learning ability are essentially the non-linear transform information systems. They carry out the huge amounts of parallel processing by using the neurons and identify effectively the dynamic characteristics of the nonlinear systems. Then the models of any non-line systems are obtained [7]. Tsai and Chuang [8], Selmic and Lewis [9] presented the methods to identify the models of the nonlinear dead zone links and achieved the good application results. However, the applications of neural networks are limited to a certain degree because of the over-Fitting learning, the local minimum, and the dependence of structures and designs on the expertise. Support Vector Machine (SVM) is a novelty machine learning algorithm proposed by Vapnik [10]. It has the better characteristics of the less samples learning, the global optimum, and the generalization ability, and so on. Then it overcomes the above shortcomings of the neural networks. Suykens and the other researchers [11, 12] presented a new type of SVM (the least squares support vector machines, LS-SVM). Being the extension of the SVM, the LS-SVM converts the quadratic programming problem of the SVM to the solving linear equation group problem. For the required precision, the LS-SVM shows the characteristics of the faster learning speed, the simpler architecture and the concise arithmetic. The Hammerstein model is suitable for describing the non-linear process, such as the dead zone and the switch on or off and so on. After the above analyses, the correctness of the identification result is ensured in this paper by the LS-SVM algorithm and the changed cost function which compensates for model error together yield good identification performance. Moreover, the convergence condition of the algorithm is deduced from the Lyapunov stability theorem.

2 System Description The Hammerstein model of the non-linear system containing the dead zone is shown in figure 1. There are 2 parts in the model. The first part is f (v(k )) which describes the non-linear dead zone. The second part is the linear discrete transmitting function G(z) to describe the linear system. v(k ) and y (k ) are the input and the output of controlled objective, respectively. u (k ) is the non-measurable middle

v(k)

u(k)

f (v ( k ))

y(k )

G (z )

Fig. 1 The Hammerstein model of the non-linear system containing the dead zone

Nonlinear Dead Zone System Identification Based on Support Vector Machine

237

variable. In our specific application (a hydraulic pressure servo-system), u (k ) would be the volume of the ratio threshold. (In practice, the direct measure of the volume of the ratio threshold of the hydraulic pressure servo system is very difficult. So it is very tough to model the nonlinear dead zone of the ratio threshold in the servo system.) Most of the references[15,16,17] tend to describe the dead zone as a symmetrically standard non-sensitive part. In practice, the essence of the dead zone system is not only asymmetric, but also is over reflects the non-linear relationship between inputs of the over positive or the negative dead zone edge and the outputs (Such as the dead zone characteristic of the volume in the servo system of the hydraulic pressure). f (v(k )) is shown as the non-linear equation below (figure 2).

⎧ f1 (v) ⎪ u(k ) = f (v(k )) = ⎨0 ⎪ f (v) ⎩ 2

if

v(k ) > br −bl ≤ v (k ) ≤ br

if if

(1)

v(k ) < −bl

Fig. 2 Nonlinear dead zone model

u f1 (v)

bl br

v

f 2 (v )

The linearly dynamic systems can be expressed by the transfer function below. G( z) =

y(k ) B( z −1 ) b0 + b1 z −1 + + bm z −m = = u(k ) A( z −1 ) 1+ a1 z −1 + + an z −n

(2)

where m and n are the power of the shift operator polynomial, and m ≤ n. Its difference equation is n

m

i =1

j =0

y (k ) = −∑ ai y (k − i ) + ∑ b j u (k − j )

(3)

3 System Model Based on LS-SVM Given the training sample set ( xk , yk ) kN=1 , the function in the form of

y ( x) = wT ϕ ( x) + b is used to estimate the unknown function by the LS-SVM

238

J. Du and M. Wang

method [11,12]. The final non-linear model is obtained by solving the equality constraints optimization problem. N

∑α K ( x, x ) + b

y ( x) =

k

(4)

k

k =1

where K ( x, xk ) is a kernel function which satisfies the Mercer condition. The different SVMs can be constructed by selecting the different kernel functions. The last term b is not suitable for the sequential calculation and be removed. Set T ϕ T ( x) = (ϕ ( x1 ), ϕ ( x2 ), , ϕ ( xN ), λ ) and w = ( w1 , w2 , , wN , b / λ ) , and λ is a constant. Namely, the dimension of the SMV is increased one more and the dimension of the corresponding weight vector is also increased one more[18].Then (4) becomes: N

∑α ′ K ′( x, x )

y ( x) =

k

(5)

k

k =1

where K′(x, xk ) = K(x, xk ) + λ2 . For simplicity, K ′( x, xk ) and α ′ are written as α and K ( x, xk ) . The output y(k) of the non-linear Hammerstein model which contains the dead area includes two parts. The first part uses the LS-SVM to estimate the nonlinear dead zone model between v( k ) and u (k ) . The other part y(k) is a linear model which depends on the past inputs and the output. That is N

∑α K (v(k ), v (k ) )

u (k ) =

j

(6)

j

j =1

where {v (k )}Nj=1 is the past N input data. For convenience, (6) is written as the following. N

∑α K

u (k ) =

j

j (v ( k ))

(7)

j =1

Combining (3) and (7), the output of the dead area non-linear Hammerstein model is n

y (k ) = −

∑

N

ai y (k − i ) + b0 (

i =1

∑α K (v(k ))

m

+

j

j =1

j

(8)

∑ b u (k − l ) l

l =1

It can also be expressed in the matrix form. y (k ) = Wn X n (k )

(9)

Nonlinear Dead Zone System Identification Based on Support Vector Machine

239

where Wn = [−a1, −a2 , , −an , b0α1, b0α2 , , b0αN , b1, b2 , b3 , , bm ] is the weight coefficient vector. The input vector is

X n = [ y (k − 1), y (k − 2),

, K N (v(k )), u (k − 1),

K1 (v(k )), K 2 (v(k )), u (k − 2),

, y (k − n), (10)

, u (k − l )]

T

If the parameter λ and the kernel function (such as the radial basis kernel function K ( x1 , x2 ) = exp(− x1 − y2

2

2s 2 ) ) are determined in the previous equation,

the part of {K j (v(k ))}Nj =1 will be obtained. Then equation (9) is a linear equation group and the solution of the weight coefficient vector can be searched out by the no constraint-free gradient descent algorithm. The definition of the cost function is needed when the gradient descend method is used. Usually the error function can be adopted. So the cost function is 2 J (k ) = e (k )

(11)

2

where e(k ) = y (k ) − d (k ) , d (k ) is the target output. If the input signal v(k ) is located in the dead area, −bl ≤ v( k ) ≤ br , the output u (k ) of the non-linear dead area approximately equals to zero. The definition is shown as follows. N

h( k ) = u ( k ) =

∑α K j

j (v ( k ))

≈0

(12)

j =1

or h(k ) = WP X P = Wn X n

(13)

Where Wp = [0,0, ,0, b0α1, b0α2 , , b0αN ,0,0,0, ,0] ,

X p = [0,

, 0 K1 (v(k )),

, K N (v(k )), 0,

, 0]T .

Then, the cost function becomes 2 J ( k ) = e (k )

2

+

Δυ h ( k )

2

2

(14)

Where the υ is penalty factor, the value of Δ is

⎧1 Δ=⎨ ⎩0

if

−ε l ≤ v(k ) ≤ ε r

if

otherwise

Where ε l and ε r are the estimated range of the dead zone.

(15)

240

J. Du and M. Wang

According to the gradient descend method, the learning factor is selected to be η , and the value of the weight vector is refreshed. Wn (k + 1) = Wn (k ) − η

∂J (k ) ∂Wn (k )

(16)

or Wn (k +1) = Wn (k) −η(e(k)Xn (k) +Δυh(k)Xp (k))

(17)

If the defined ideal weight value is W′ , the error function is ′ n (k ) = Wn (k )Xn (k ) e(k) = Wn (k)Xn (k) − WX

(18) 2

The orthogonal discrete Lyapunov function is defined as V ( k ) = W n ( k ) . It varies in the training process. 2

ΔV(k +1) = V(k +1) −V(k) = Wn (k +1) − Wn (k)

2

=η2 e(k) Xn (k) −υ2η2 h(k) Xp (k) 2

2

2

2

+2η2tr{e(k)Xn (k)T υh(k)Xp (k)}

(19)

−2ηtr{e(k)Xn (k) Xn (k) } T

T

−2ηυtr{h(k)Xp (k)T Xn (k)T } or

ΔV (k +1) = η2 ( e(k) Xn (k) +υ2 h(k) Xp (k) )2 −2η e(k)2 − 2υηh(k)2

(20)

Based on the Lyapunov stability theorem, there must be ΔV ( k + 1) < 0 to ensure the algorithm convergence or the system stable. Equivalently, it can ensure the algorithm convergence that the learning factor is properly selected in the range below. 2 2 0<η <2(e(k) +υh(k) )

( e(k) Xn(k) +υ2 h(k) Xp(k) )2

(21)

4 Simulation After the above formula illation, the simulation of the modeling problem of the Servo hydraulic system which contains the dead zone is carried out. The mathematical model of the system can be depicted as a series of a dead zone nonlinear part and a linear part [8]. The nonlinear dead zone part is depicted in figure 2, br=2.5, bl=2.1. The discrete model of the linear part is

Nonlinear Dead Zone System Identification Based on Support Vector Machine v( k )

Dead zone

u (k )

241 y (k )

Linear system t

yˆ (k ) Linear model

LS-SVM

d l Fig. 3 The identification structure block diagram of the hydraulic servo system

G(z) =

y (k ) 0.1499 z 2 + 0.0435 z − 0.0001 = 3 u ( k ) z − 0.8898 z 2 − 0.1472 z + 0.0414

(22)

The support vector machines can imitate any non-linear characteristics. A LSSVM is designed as the identification machine to replace the non-linear dead zone. It can be seen in figure 3 that the required sample data for the identification are the given stimulant signals v(k) and the actual output signals of the Piston location. The location output signals are relatively easy to measure. For the Servo hydraulic system identification, the position signals are given, v = 5sin(5t ) . For the LS-SVM, the most commonly used radial basis function is selected as the kernel function. After several simulation tests, the kernel width is finally determined s = 0.25 , and λ = 5.6 . Set bl = br = 2.5 for the non-linear dead zone identification. This means that the dead zone is symmetric in the system. The variable cost function is used in figure 4, the cost function is the equation (14), and the penalty factor v=10. The common cost function is used in figure 5, and the cost function is the equation (11). It can be seen by comparing figure4 with figure5 that the nonlinear dead zone model using the variable cost function strategy is better because of the consideration of the specific case of the small inputs. The actual output signals of the system and the model output signals are shown in Figure 6. The solid curve represents the actual output signals and the dashed curve represents the model output signals. The mean square error MSE = 0.0832 . Fig. 4 The estimated dead zone using the varying cost function method

6

4

u

2

0

-2

-4

-6 -6

-4

-2

0 v

2

4

6

242

J. Du and M. Wang 6

Fig. 5 The estimated dead zone using the common cost function method

4

u

2

0

-2

-4

-6 -6

-4

-2

0 v

2

4

6

1.5

Fig. 6 The actual output signals of the system and the model output signals

1

0.5

mm

0

-0.5

-1

-1.5

-2 0

1

2

3

4

5 s

6

7

8

9

10

5 Conclusion The dead zone phenomena have an important impact on the accuracy and stability of system control because of the seriously non-linear character. In order to reduce or eliminate the influence on the system output, the most important thing is to deeply analyze and understand the dead zone non-linear systems. A Hammerstein model method is proposed to identify the nonlinear dead zone based on the LSSVM. Aiming at the input signals, the variable cost function strategy is adopted to improve the accuracy of the model identification. Moreover, the convergent range is given according to the Lyapunov stability theorem. The simulation results show the effectiveness of this method. A new optional method is provided for the nonlinear dead zone system modeling by this paper.

References 1. Merzouki, R., Davila, J.A., Fridman, L., et al.: Backlash Phenomenon Observation and Identification in Electromechanical System. Control Engineering Practice 15(4), 447– 457 (2007) 2. Li, S.X., Zhu, J.: A Modeling Method for the Non-Linear Servo System Containing a Dead Zone. Automation Technology and Application 18, 4–8 (1999) 3. Dai, Y.P., Deng, R., Liu, J.: Research on Parameter Identification for Nonlinear Steam Turbine Governing System Based on Genetic Algorithm. Power Engineering 23(1), 2215–2218 (2003)

Nonlinear Dead Zone System Identification Based on Support Vector Machine

243

4. Cao, X.Q., Lu, Q.H.: Two-Step Method for Identifying Physic Parameters of Systems with Dead Zone Nonlinearity. Journal of Tsinghua University (Natural Science) 47, 2045–2047 (2007) 5. Wang, X.S., Sub, C.Y., Henry, H.: Robust Adaptive Control of a Class of Nonlinear Systems with Unknown Dead-Zone. Automatica 40, 407–413 (2004) 6. Lee, S.W., Kim, J.H.: Control of System with Deadzones Using Neural Network Based on Learning Controller. In: Proc. IEEE Int. Conf. Neural Networks, pp. 2535– 2538 (1994) 7. Zhu, Q.B.: Neural Network Learning Correction Method of the Nonlinear Sensing Characteristics and Its Application. Instruments Journal 25(6), 505–805 (2004) 8. Tsai, C.H., Chuang, H.T.: Deadzone Compensation Based on Constrained RBF Neural Network. Journal of the Franklin Institute 341, 361–374 (2004) 9. Selmic, R.R., Lewis, F.L.: Deadzone Compensation in Motion Control Systems Using Neural Networks. IEEE Trans. Autom. Control 45, 602–613 (2000) 10. Vapnik, V.: The Nature of Statistical Learning Theory. Springer, New York (1995) 11. Suykens, J., Vandewalle, J.: Least Squares Support Vector Machines Classifiers. Neural Processing Letters 9, 293–300 (1999) 12. Suykens, J., Van, G., De Brabanter, J., et al.: Least Squares Support Vector Machines. World Scientific Publishing Co Pte Lte, Singapore (2002) 13. Tsai, C.H., Chuang, H.T.: Deadzone Compensation Based on Constrained RBF Neural Network. Journal of the Franklin Institute 341, 361–374 (2004) 14. Chen, Z.H.: A New Identification Method of Nonlinear System Based on the Hammerstein Model. Control Theory and Applications 24(1), 143–147 (2007) 15. Shen, Q.K., Zhang, T.P.: Robustly Adaptive Fuzzy Control for the Unknown Nonlinear Dead-zone. Control and Decision 21(4), 368–375 (2006) 16. Cho, H., Bai, E.W.: Convergence Results for an Adaptive Deadzone Inverse. Int. J. Adaptive. Control Signal Processing 12, 451–466 (1998) 17. Tao, G., Kokotovic, P.V.: Adaptive Sliding Control of Plants with Unknown Deadzone. IEEE Trans. Autom. Control 39, 59–68 (1994) 18. Vijayakumar, S., Wu, S.: Sequential Support Vector Classifiers and Regression. In: Proc. Int. Conf. on Soft Computing, vol. 5, pp. 610–619 (1999)

A SVM Model Selection Method Based on Hybrid Genetic Algorithm and Empirical Error Minimization Criterion Xin Zhou and Jianhua Xu*

Abstract. The generalization capacity of support vector machine (SVM) depends largely on the selection of kernel function and its parameters, and penalty factor, which is regarded as model selection of SVM. When various forms of differentiable and loose generalization bounds are considered as the objective functions, the traditional optimization algorithms easily fall into the local optimal solutions, whereas the modern techniques difficultly find out really optimal ones. Recently the empirical error criterion on a validation set is used as a new objective function which is optimized by the classical optimization methods. In this paper, we propose a new SVM model selection based on hybrid genetic algorithm and empirical error minimization criterion. The hybrid genetic method integrates the gradient descent method into the genetic algorithm to search for a better parameter of RBF kernel. The experiments on 13 benchmark datasets demonstrate that our method can work well on some real applications. Keywords: Support vector machine (SVM), Hybrid genetic algorithm, Empirical error minimization criterion.

1 Introduction The support vector machine (SVM) is a prominent classifier introduced by Vapnik and his co-workers [1, 2], and has been applied to many different classification problems successfully. But its success depends on the tuning of several parameters which affect the generalization capacity, including the penalty parameter C and the kernel function parameters such as σ for radial basis function (RBF) kernel. In order to achieve optimal generalization performance, SVM model selection technique should optimally determine these parameters. The adjustment of the parameters is usually conducted through minimizing an error estimator. Several methods [3-9] were developed for choosing the best parameter values, such as the k-fold cross-validation or the leave-one-out (LOO) Xin Zhou . Jianhua Xu Department of Computer Science, Nanjing Normal University, Nanjing 210097, China [email protected] *

H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 245–253. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

246

X. Zhou and J. Xu

methods. Ratsch et al. [3] used 5-fold cross validation to optimize the penalty parameter C and kernel parameter on the first five realizations, and average their estimations as the final model parameters. Lee and Lin [4] made full use of training samples to adopt the exhaustive search for the optimal parameters, which minimizes the expectation test of the LOO error rate. However, two algorithms are time consuming. Since it is difficult to calculate the real LOO error, researchers used some estimated upper bounds of the LOO error as the objective functions for SVM model selection, such as radius-margin bound, span bound and VC dimension bound. Chapelle et al. [5] proposed firstly an automatic method for selecting parameters for SVM by using an approximate error of the LOO procedure. Keerthi [6] applied Quasi-Newton algorithms efficiently to minimize radius-margin bound for tuning parameters of RBF kernel SVM. Duan et al. [7] showed that radius margin bound can provide good prediction for L2-SVM, but does not work well for L1-SVM. In this case, Chung et al. [8] proposed a modified radius margin bound for L1-SVM. Recently, Ayat et al. [9] defined the empirical error on a validation set as a new criterion of model selection, and optimized it through the Quasi-Newton method. It is attractive since this criterion does not need to solve another QP problem (for example, to calculate the minimal radius containing all training examples) [10, 11]. Adankon [12] proposed a new formulation for SVM model selection by using the criterion of empirical error technique, which is based on generalization error minimization through a validation set. However, among the above gradient-based (such as Newton and conjugate gradient methods, etc.) model selection algorithms, the final generalization capacity of SVM are largely affected by the initial point of the parameters since these classical methods can only obtain local optimal solutions. On the other hand, some researchers began to use modern optimization techniques (such as genetic, simulated annealing, and PSO algorithms) to study the SVM model selection. Zheng [13] used genetic algorithm to minimize radius margin bound for SVM parameters. Javier [14] proposed a method for tuning L1SVM parameters with the simulated annealing and modified radius margin bound. But their experiments show that since genetic algorithms are lack of local optimization capability and the upper bounds are loose, the final results of model selection are not satisfactory yet, especially for kernel parameters. In this paper, we propose a new method for SVM model selection, which is based on hybrid genetic algorithm and empirical error minimization criterion. The hybrid genetic method integrates the gradient descent algorithm into the genetic algorithm. This can help us find out a better parameter of RBF kernel. Experiments on 13 benchmark datasets show our new model selection method can work well.

2 Support Vector Machine Given a binary class training set {xi , yi } , where xi ∈ R n , yi = {1, −1} and i = 1,…, l . The goal of a support vector machine is to solve the following quadratic problem:

A SVM Model Selection Method Based on Hybrid Genetic Algorithm l

max

∑αi − i =1 l

s.t.

∑α y i

i =1

i

1 l l ∑∑ α iα j yi y j k (xi , x j ) 2 i =1 j =1

,

247

(1)

= 0, 0 ≤ α i ≤ C

where C is the penalty factor and k ( xi , x j ) the kernel function, for example, RBF kernel:

⎛ x −x i j k ( xi , x j ) = exp ⎜ − 2 ⎜ 2σ ⎝

2

⎞ ⎟ = exp ⎛ − θ x − x i j ⎜ ⎟ ⎝ 2 ⎠

2

⎞ ⎟, ⎠

(2)

where σ is referred to as the width parameter of RBF kernel. Here let θ = 1 σ 2 for convenience. Through solving the problem (1), we can obtain a nonlinear classifier: l

f ( xi ) = ∑ y jα j k ( x j , xi ) + b, i = 1,

,l ,

(3)

j =1

where the class is predicted to be +1 if f > 0 and −1 otherwise. To obtain a good performance, some parameters in SVM have to be chosen elaborately. When applying RBF kernel, the parameters include the penalty factor C (determining the tradeoff between the training error and model complexity) and the parameter of the kernel function θ (implicitly defining the nonlinear mapping from input space to some high-dimensional feature space).

3 An Empirical Error Criterion In this section, we briefly introduce the empirical error criterion used in [9, 10]. It is attractive to use a subset of the training data to approximate the expected risk of SVM [5]. Given an independent validation data set, the number of classification errors is: T=

1 N

∑ Ψ(− y f ( x )) , i

i

(4)

i

where Ψ is a step function (i.e., Heaviside function) and N is the size of the validation set. The larger the validation set is, the smaller the variance of the estimated error will be according to the statistical viewpoints. Since such an error function is not differentiable, we have to replace it using some smooth function. For the SVM’s output f ( xi ) (or simply f i ) of example xi , Platt [15] estimated its corresponding posterior probability, pi =

1 , 1 + exp( Af ( xi ) + B)

(5)

248

X. Zhou and J. Xu

which indicates the possibility that the example xi belongs to the positive class, where A and B are constants. Now we substitute a new variable ti = ( yi + 1) 2 ∈ {0,1} for the real class label yi = ±1 of example xi . Through minimizing the test error of validation data set by using a variant of Newton’s algorithm, we can get the optimal values of A and B: N sv

( A* , B* ) = arg max ∑ ( ti log( pi ) + (1 − ti ) log(1 − pi ) ) . A, B

(6)

i =1

Using two optimal values ( A* , B* ) , we define the error probability for a given example xi as,

Ei =| ti − pi |= pi1−ti (1 − pi )ti , i = 1,

,l .

(7)

For a validation set, the average estimation of the errors could be written as: E=

1 N

N

∑E

i

i =1

=

1 N

N

∑p i =1

1− ti i

(1 − pi )ti .

(8)

This is our empirical error criterion for SVM model selection. Generally speaking, such a function is associated with all SVM parameters. In this paper, we particularly apply the gradient descent method to adjust the kernel parameter of RBF (i.e., θ ). In this case, we have to calculate its derivative with respect to θ : ∂E ∂ ⎛1 = ∂θ ∂θ ⎜⎝ N

⎞

N

1

N

∂Ei

∑ E ⎟⎠ = N ∑ ∂θ i =1

i

=

i =1

1 N

N

∂Ei ∂f i . ⋅ ∂θ i

∑ ∂f i =1

(9)

Now we derive the derivative of Ei with respect to f i , ∂Ei ∂Ei ∂pi = ⋅ , ∂f i ∂pi ∂f i where

(10)

∂pi ∂Ei ∂ ti − pi ⎧−1 if ti = 1 = − Api (1 − pi ) . Then we = =⎨ = yi , and 1 if t 0 + = ∂f i pi ∂pi i ⎩

have ∂Ei = Ayi pi (1 − pi ) . ∂f i

(11)

The derivative of f i with respect to θ is:

⎞ ∂fi ∂ ⎛ N sv = ⎜ ∑ α j y j k ( xi , x j ) + b ⎟ , ∂θ ∂θ ⎝ j =1 ⎠ where N sv is the number of support vectors. Finally, we get

(12)

A SVM Model Selection Method Based on Hybrid Genetic Algorithm

∂α ⎤ ∂b ∂fi N sv ⎡ ∂k ( x j , xi ) α j + j k ( xi , x j ) ⎥ + . = ∑ yj ⎢ ∂θ j =1 ⎣ ∂θ ∂θ ⎦ ∂θ

249

(13)

It is noted that this derivative is composed of three parts. Theoretically the last two ∂α j ∂b k ( xi , x j ) and are parts depend on θ , but in our experimnet, we find that ∂θ ∂θ ∂k ( xi , x j ) α j . Finally, we obtain the derivative of the negligible in comparison to ∂θ empirical error function with respect to the kernel parameter θ of RBF kernel:

⎧ N sv ∂k ( xi , x j ) ⎫ ∂Ei αj⎬, = Ayi pi (1 − pi ) ⋅ ⎨∑ y j ∂θ ∂θ ⎩ j =1 ⎭

(14)

where the derivative of k ( xi , x j ) with respect to its parameter θ is: ∂k ( xi , x j ) ∂θ

⎛ θ = exp ⎜ − xi − x j ⎝ 2

2

⎞ ⎛ 1 ⎟ ⋅ ⎜ − xi − x j ⎠ ⎝ 2

2

⎞ ⎟. ⎠

(15)

4 A Novel SVM Model Selection Method The empirical error criterion (8) is a nonlinear function with respect to SVM model parameters. The genetic algorithm is a best candidate to optimize such a criterion. But in practice it is time consuming and does not guarantee to find out a global optimal solution without a good initialization. On the other hand, the gradient descent method is a widely used classical optimization tool because of its simplicity and easy implementation. However, it could get stuck in some local optimum generally. It is found that the generalization capacity of SVM is affected by the kernel parameter of RBF more seriously than by the penalty one. In this paper, the gradient descent method is integrated into the genetic algorithm to construct a hybrid genetic technique. It can help us find out a better kernel parameter. We choose the best individual as an initial solution, and then search for its optimal kernel parameter using gradient descent method as its iterative solution. After evaluating a new generation, we pick up a new best individual from three candidates: the best solution of the old generation, the best one of the new generation, and the iteratively improved version of the old best solution. The worst individual is replaced by the iterative solution too. Such an optimization method can sufficiently utilize advantages of two different methods to find out the better model parameters. Our SVM model selection procedure based on hybrid genetic algorithm is listed as follows: 1) Randomly generate an initial population; 2) Calculate the fitness function (Eq. (8)) for each individual by SVM algorithm, and save the best one as I best ;

250

X. Zhou and J. Xu

3) Regard I best as the initial solution, and then search for a local solution using the gradient descent method as an iterative solution. It is noted that only kernel parameter of RBF is optimized in this step; 4) Generate a new generation of population using three basic operation (i.e., selection, crossover and mutation) in genetic method; 5) For a new generation, calculate fitness function for each individual using SVM; 6) Find out the best individual and the worst one from the current generation; compare the current best individual with I best and the iterative solution, and save the best individual in I best ; replace the worst one with the iterative solution; 7) Return to step 3 until meeting the stop conditions. If converged, the parameters corresponding to the optimal individual are defined as the SVM optimal model parameters. In such a hybrid genetic algorithm for SVM model selection, both the penalty factor C and the RBF kernel parameter θ are optimized at the same time. It is very valuable for SVM applications.

5 Experimental Results In this paper, we only focus on SVM with RBF kernel, and carry out experiments on 13 datasets from [16]: banana, breast cancer, diabetes, flare-solar, german, heart, image, ringnorm, splice, thyroid, titanic, twonorm, and waveform. Each dataset is composed of 100 or 20 different realizations of the binary classification problem. The general information about these data sets is given in Table 1. The search space of model parameters are C ∈ [0.1,30] and θ ∈ [0.01,5.0] . Following the similar experi-mental setting in [3] and [5], we randomly extract one third examples of each training set as the validation dataset, and the left two third ones as the training set. Table 1 General information about 13 data sets Datasets Banana Breast cancer Diabetes Flare-solar German Heart Image Ringnorm Splice Thyroid Titanic Twonorm Waveform

Training size 400 200 468 666 700 170 1300 400 1000 140 150 400 400

Test size 4900 77 300 400 300 100 1010 7000 2175 75 2051 7000 4600

Dimension 2 9 8 9 20 13 18 20 60 5 3 20 21

Realizations 100 100 100 100 100 100 20 100 20 100 100 100 100

A SVM Model Selection Method Based on Hybrid Genetic Algorithm

251

Table 2 Test error rates and their standard deviation found by different algorithms for selecting the SVM model parameters. The values in the squared brackets are the corresponding model parameters, i.e., [C, θ ] Adankon’s GA-SVM Method [12] Breast 25.79±4.21 25.62±4.49 26.04±4.7 26.84±4.71 25.59±4.18 [1.080,0.100] [0.654,0.058] cancer 23.25±1.82 23.32±1.61 Diabetes 23.53±1.73 23.25±1.70 23.19±1.67 [0.500,0.060] [1.458,0.036] 15.98±3.32 15.48±3.33 Heart 15.92±3.18 16.13±3.11 15.95±3.26 [0.666,0.010] [0.428,0.021] 4.64±2.13 4.57±2.17 Thyroid 4.80±2.19 4.62±2.03 4.56±1.97 [10.000,0.183][0.558,0.327] 22.93±1.17 22.57±0.89 Titanic 22.42±1.02 22.88±1.23 22.5±0.88 [1.100,0.120] [2.884,0.227] Datasets

5-fold CV[3]

R-M Bound[5]

Span Bound[5]

Hybrid GA-SVM 25.51±4.25 [0.785,0.101] 23.11±1.74 [1.879,0.018] 15.37±3.26 0.490,0.010] 4.54±2.08 [0.469,0.328] 22.57±0.89 [2.833,0.229]

Firstly we conduct our experiments on five data sets (i.e., breast cancer, diabetes, heart, thyroid, and titanic). Taking account of the time efficiency, on each of the first ten training and testing sets, the penalty parameter C and kernel parameters θ are optimized by using our hybrid genetic algorithm (simply Hybrid GA-SVM). The final model parameters are computed from the average of the ten estimations. For comparison, we also carry out general genetic algorithm as SVM model selection tool on these five sets (simply GA-SVM). Additionally, some results from 5-fold cross validation (5-fold CV) [3], radiusmargin bound (R-M bound) [5], span bound [5] and Adankon’s method [12] are Table 3 The test error rates and their standard deviation obtained with 5-fold CV, GASVM, and hybrid GA-SVM. The values in the square brackets are the corresponding optimal model parameters, i.e., [C, θ ] Datasets

5-fold CV[3]

Banana

11.53±0.66

Flare-solar

32.43±1.82

German

23.61±2.07

Image

2.96±0.60

Ringnorm

1.66±0.12

Splice

10.88±0.66

Twonorm

2.96±0.23

Waveform

9.88±0.43

GA_SVM 11.34±0.49 [3.271, 0.327] 32.45±1.80 [1.450, 0.043] 23.60±2.23 [1.259, 0.034] 3.56±0.67 [3.237, 0.327] 1.47±0.08 [0.223, 0.077] 10.83±0.67 [2.528, 0.016] 2.43±0.13 [0.176, 0.040] 9.86±0.48 [0.623, 0.042]

Hybrid GA_SVM 11.30±0.50 [3.036, 0.339] 32.33±1.80 [1.811, 0.018] 23.58±2.21 [2.825, 0.026] 3.21±0.68 [3.239, 0.528] 1.46±0.08 [0.194, 0.088] 10.76±0.69 [2.624, 0.018] 2.42±0.13 [0.163, 0.044] 9.86±0.45 [0.721, 0.045]

252

X. Zhou and J. Xu

cited. All these experimental data are shown in Table 2, where the best results are indicated in the bold font. According to Table 2, our model selection method performs best on four datasets. On titanic data set, our results are worse than 5-fold cross validation and span bound methods. For the other eight data sets, we execute two model selection methods: hybrid GA-SVM and GA-SVM, as shown in Table 3. The results from 5-fold cross validation are listed too, but we could not find comparable results for the other methods on these data sets in the literature. From this table we can show that our method can work best on six data sets, while 5-fold cross validation and GA-SVM perform best on one data set respectively. According to the above experimental results, we can conclude that for most of the 13 data sets our model selection based on hybrid genetic algorithm for SVM can obtain better results than the 5-fold cross validation, radius margin bound or span bound based ones did.

6 Conclusions In this paper, an effective model selection for SVM is proposed by combining hybrid genetic algorithm and empirical error minimization criterion. The hybrid genetic algorithm integrates the gradient descent algorithm and the genetic one. It helps us find out a better parameter of RBF kernel. The iterative solution from the gradient descent method is also regarded as a candidate of the best and worst individuals. It makes it easy to search for better model parameters. The empirical error rate on a validation set is a better estimate of generalization error, compared with various generalization error bounds. The simulation results on 13 benchmark dataset demonstrate that our proposed approach can work well. In our further work, we will test more real applications including some unbalanced sets to validate our method and consider other kernel functions (for example, polynomial and sigmoid kernels). On the other hand, we will utilize some heuristic strategies to speed up our model selection method. Acknowledgments. This work is supported by National Natural Science Foundation of China (No. 60875001).

References 1. Vapnik, V.N.: The Nature of Statistical Learning Theory. Springer, New York (1995) 2. Cortes, C., Vapnik, V.N.: Support Vector Networks. Machine Learning 20(3), 273– 297 (1995) 3. Ratsch, G., Onoda, T., Muller, K.R.: Soft Margins for AdaBoost. Machine Learning 42, 287–320 (2001) 4. Lee, J.H., Lin, C.J.: Automatic Model Selection for Support Vector Machines. Technical Report, Department of Computer Science and Information Engineering, National Taiwan University (2000) 5. Chapelle, O., Vapnik, V.N., Bousquet, O., Mukherjee, S.: Choosing Multiple Parameters for Support Vector Machines. Machine Learning 46(1-3), 131–159 (2002)

A SVM Model Selection Method Based on Hybrid Genetic Algorithm

253

6. Keerthi, S.S.: Efficient Tuning of SVM Hyperparameters Using Radius Margin Bound and Iterative Algorithms. IEEE Trans. on Neural Networks 13, 1225–1229 (2002) 7. Duan, K., Keerthi, S.S., Poo, A.N.: Evaluation of Simple Performance Measures for Tuning SVM Hyperparameters. Neurocomputing 51, 41–59 (2003) 8. Chung, K.M., Kao, W.C., Sun, C.L., Wang, L.L., Lin, C.J.: Radius Margin Bounds for Support Vector Machines with the RBF Kernel. Neural Computation 15, 2643–2681 (2003) 9. Ayat, N.E., Cheriet, M., Suen, C.Y.: Optimization of the SVM Kernels Using an Empirical Error Minimization Scheme. In: Lee, S.W., Verri, A. (eds.) SVM 2002. LNCS, vol. 2388, pp. 354–369. Springer, Heidelberg (2002) 10. Adankon, M.M., Cheriet, M., Ayat, N.E.: Optimizing Resources in Model Selection for Support Vector Machines. In: 2005 International Joint Conference on Neural Networks, pp. 925–930. IEEE Press, Montreal (2005) 11. Ayat, N.E., Cheriet, M., Suen, C.Y.: Automatic Model Selection for the Optimization of the SVM Kernels. Pattern Recognition 38, 1733–1745 (2005) 12. Adankon, M.M., Cheriet, M.: New Formulation of SVM for Model Selection. In: 2006 International Joint Conference on Neural Networks, pp. 1900–1907. IEEE Press, Vancouver (2006) 13. Zheng, C.H., Li, C.J.: Automatic Parameters Selection for SVM Based on GA. In: 5th World Congress on Intelligent Control and Automation, pp. 1869–1872. IEEE Press, Hangzhou (2004) 14. Javier, A., Saturnino, M., Philip, S.: Tuning L1-SVM Hyperparameters with Modified Radius Margin Bounds and Simulated Annealing. In: Sandoval, F., Prieto, A.G., Cabestany, J., Graña, M. (eds.) IWANN 2007. LNCS, vol. 4507, pp. 284–291. Springer, Heidelberg (2007) 15. Platt, J.: Probabilistic Outputs for Support Vector Machines and Comparisons to Regula-rized Likelihood Methods. In: Bartlett, P.J., Scholkopf, B., Schuurmans, D., Smola, A.J. (eds.) Advances in large margin classifiers, pp. 67–74. MIT Press, Cambridge (1999) 16. Ratsch, G.: Benchmark data sets, http://ida.first.fhg.de/projects/bench/benchmarks.htm

An SVM-Based Mandarin Pronunciation Quality Assessment System Fengpei Ge, Fuping Pan, Changliang Liu, Bin Dong, Shui-duen Chan, Xinhua Zhu, and Yonghong Yan

Abstract. This paper presents our Mandarin pronunciation quality assessment system for the examination of Putonghua Shuiping Kaoshi (PSK) and investigates a novel Support Vector Machine (SVM) based method to improve its assessment accuracy. Firstly, an selective speaker adaptation module is introduced, in which we select well pronounced speech from results of the first-pass automatic pronunciation scoring as the adaptation data, and adopt Maximum Likelihood Linear Regression to update the acoustic model (AM). Then, compared with the traditional triphone based AM, the monophone based AM is studied. Finally, we propose a new method of incorporating all kinds of posterior probabilities using SVM classifier. Experimental results show that the average correlation coefficient between machine and human scores is improved from 83.72% to 85.48%. It suggests that the two methods of selective speaker adaptation and multi-model combination using SVM are very effective to improve the accuracy of pronunciation quality assessment. Keywords: SVM, Pronunciation Quality Assessment, CALL, Speaker Adaptation, Speech Recognition.

1 Introduction This work is to investigate automatic pronunciation quality assessment (APQA), as the main part of a computer-aided Putonghua Shuiping Kaoshi Fengpei Ge · Fuping Pan · Changliang Liu · Bin Dong · Yonghong Yan ThinkIT Speech Lab, Institute of Acoustics, Chinese Academy of Sciences, Beijing 100871, P.R. China {gefengpei,panfuping,liuchangliang,dongbin}@hccl.ioa.ac.cn Shui-duen Chan · Xinhua Zhu Department of Chinese & Bilingual Studies, the Hong Kong Polytechnic University {chsdchan,Ctxhzhu}@polyu.edu.hk, [email protected] H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 255–265. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

256

F. Ge et al.

(PSK) system. Over the last decades, many research groups have studied it based on speech recognition techniques. The evaluation scores are obtained by combining various machine measurements, including scores of Hidden Markov Model (HMM) log-likelihood, timing, phone log-posterior probability and segment duration. Mandarin is a syllabic language, whose syllable has two parts: the initial part is consonant and the final is vowel. So the segmental quality is a major problem especially for nonnative speakers. A classical measurement is the phonetic posterior probability, which was used in several previous works [1]-[5]. However, it does not correlate with human judge directly. Witt and Young studied the pronunciation quality assessment at the phone level, and suggested improving evaluation performance by using a likelihood-based GOP measure and explicit error modeling [6]. This paper describes our Mandarin APQA system, and some improvement measures are investigated. Firstly, a novel speaker adaptation module is present. We select well pronounced speech from results of the first-pass APQA as the adaptation data, and adopt Maximum Likelihood Linear Regression (MLLR) to update acoustic model (AM). Then, monophone based AM is studied, which can describe the acoustic characteristics of isolated phones more elaborately. At last, we propose a new method of incorporating all kinds of posterior probabilities using SVM to class pronunciation qualities. Since pronunciation assessment is highly subjective, three measures has been used, each of which measures a different aspect of how well computerderived scores agree with human assessment. Experiment results suggest that selective speaker adaptation is useful to improve the performance of forced alignment, and that the method of multi-model combination using SVM is helpful for increasing the accuracy of classing pronunciation qualities and can achieve usable performance. The rest of the paper is organized as follows: section 2 describes our baseline system; the improvement measures are shown in section 3; in section 4 experiment results are presented; section 5 concludes this paper; and finally the acknowledgement is given.

2 Baseline System Our system evaluates the pronunciation quality of Mandarin speech, in which syllable is the fundamental assessment unit. All Mandarin syllables can be considered as a combination of initial and final parts. The initial is articulated with a final to form a syllable pronunciation. Mandarin is also a tonal language. The tone is mainly specified by the pattern of pitch contour of vowel portion of the syllable. In our system, the syllable pronunciation is evaluated from three aspects: the quality of consonant, vowel and tone. The first two parts are evaluated using the ASR techniques of HMM and Viterbi search [7], which is the core of the system. The tone quality is evaluated using a Gaussian Mixture Model

An SVM-Based Mandarin Pronunciation Quality Assessment System

257

Input Speech

Feature Extraction

Learning Text

MFCC

Pronunciation Dictionary

One-Pass Viterbi Decoding (Forced Alignment) Phonetic Segments

HMM Model

Vowel Part

Pitch Extraction

Post Probability Calculation Predefined Threshold

Model Matching

Detector Vowel Score

Tone GMM

Tone Score

Consonant Score

Integrate Scores Syllable Score

Fig. 1 Our CALL system structure

(GMM) classifier, which is not the focus of this paper, but can be found in [8][9]. A block diagram of the system is shown in Fig.1. The front-end feature extraction converts the speech waveform to a sequence of Mel-Frequency Cepstral Coefficients (MFCC), which is then fed into decoder to do onepass Viterbi decoding. The HMM net only consists of the models of the learning text, and the Viterbi decoding is a forced alignment between the speech frames and the HMM models. With the frame index of each HMM state and the accumulated observation probability of the phone segment, the phonetic posterior probability score is calculated as the pronunciation quality measurement. There are two kinds of phonetic posterior probabilities in our system. One is the average of logarithm of the frame-based posterior probabilities (AFBPP) [3][4][5]. 1 log P (st |ot ), e−b+1 e

ρAF BP P (P H|O) =

(1)

t=b

where O is the forced-aligned observation sequence of P H; b is its beginning frame and e is its end frame; S is the state sequence corresponding to O; P (st |ot ) is the state posterior probability. The other is the phone log-posterior probability (PLPP) [6]: ρP LP P (P H|O) =

p(O(q) )|q 1 1 log P (q|O(q) ) = log  , τ τ p(O(q) |p) p∈Q

(2)

258

F. Ge et al.

where τ is the number of frames in the acoustic segment O(q) ; Q is the set of Mandarin consonants when q is a consonant, otherwise it is the set of Mandarin vowels when q is a vowel. We combine the two posterior probability scores as follows. P (P H|O) = ω ∗ ρAF BP P + ρP LP P ,

(3)

where P (P H|O) is the combined posterior probability, and ω is the combination coefficient. The confidence of P (P H|O) is an absolute measurement describing how close the utterance is to the standard pronunciation. It can be used for phonetic pronunciation assessment directly [1][4]. We classify phonetic pronunciation quality into three classes: good, medium and bad, corresponding to score 2, 1 and 0 respectively. So, the final stage of evaluation is using predetermined thresholds to map the posterior probability scores to grades. The thresholds used for consonant and vowel are context-dependent. Evaluation of the tone is in parallel with the phone pronunciation assessment. Finally, the phone pronunciation scores and the tone score are integrated by Eq.(4) to form the syllable score. All syllable scores for one speaker are summed up to the speaker score. Ssyllable = min (SConsonant , SV owel , ST one )/2,

(4)

3 Improved System 3.1 Selective Speaker Adaptation As described in section 2, forced alignment is significant for pronunciation quality assessment, from which the frame index of each HMM state and the accumulated observation probability of the phone segment are obtained. During it, AM plays an important role. Ideally, we should adopt standard speech as the evaluation criterion to assess pronunciation quality, and AM should be pre-trained with standard Mandarin speech. However, speaker independent (SI) AM often mismatch the test speech, because most users are with strongly accent. For such speakers, their articulations are different from those of native speakers. They often have difficulty in pronouncing some initials/finals. So they use some typical strategies of language learners to compensate for such difficulties, including phonological transfer, overgeneralization, prefabrication, epenthesis, etc. Therefore, they pronounce in a way close to standard Mandarin, but not accurately. This mismatch between AM and test speech degrades the APQA performance severely. Therefore, Adjusting SI AM to fit the users is necessary for improving the phone segmentation (forced alignment) of strongly accented Mandarin speech, and also non-native speaker’s speech.

An SVM-Based Mandarin Pronunciation Quality Assessment System

259

Fig. 2 Our CALL system structure with speaker adaptation

In this paper, we adopt MLLR [10] to adapt SI AM to the special testee. A block diagram of the improved system is shown in Fig.2. Firstly, preassessment which is the first-pass pronunciation quality assessment is done using the baseline system and the scores for every consonant, vowel and tone for the testee are obtained. Using the first-pass assessment results, we select well pronounced speech as the adaptation data, whose score obtained from the system is 2. Then the SI acoustic model is adapted using MLLR. In order to achieve speaker normalization without adapting to specific phone error patterns, this adaptation is limited to a single global transform of the HMM mixture component means. Finally, we use the speaker-adapted (SA) AM to assess his/her pronunciations again.

3.2 Monophone Based AM Used in Pronunciation Quality Assessment In baseline system, we adopt triphone based AM to do forced alignment and posterior probability calculation, which is popularly used in ASR systems. The triphone based AM can model the variability of context-dependent phones better. In fact, the PSK examination focuses on the single syllables and two-syllable words. Moreover, most of the testees can not pronounce them fluently, and they often speak phone by phone. In this case, the context of a phone is not necessary to be modeled subtly, and the elaborate AM with triphones may cause some phonetic confusions. Using monophone as the modeling unit can elaborate the acoustic characteristic of different phones with more GMMs. So the monophone based AM is also used in APQA to model acoustic characteristics for the isolated phones. In our improved system, the triphone and monophone based AM are combined to assess Mandarin pronunciation qualities.

3.3 Multi-model Combined System Using SVM Triphone based AM can model the variability of context-dependent phones better, while monophone based AM can describe the acoustic characteristics of the isolated phones more elaborately. The posterior probabilities calculated

260

F. Ge et al.

Fig. 3 Multi-model combined system using SVM

with different AMs contain acoustic characteristic from different aspects. So combining them is useful to improve system performance. In baseline system, we combine the two type posterior probabilities of AFBPP and PLPP using the method of the linear weighted sum shown as Eq.(4) and classify the combined confidence with thresholds. This method can not resolve complex learning problems, while SVM is an outstanding statistical learning method for such classify problem[11] . We use the toolkit “(lib) SVM” of Chih-Chung Chang and Chih-Jen Lin [12][13] to implement the system combination of triphone and monophone AMs. A block diagram of the combined system is shown in Fig.3. It mainly includes two parts: training and classifying. The latter is implemented with models pre-trained by the former. The input and output of SVM are confidence vectors and pronunciation qualities (0,1 and 2) respectively. What is the confidence vector is an important problem. In this paper, there are two kinds of phonetic posterior probabilities for every phone described as section 2. From the two systems, based on triphone and monophone, we can obtain four-dimensional vectors per phone as the input of SVM. During training models, confidence vectors of the developed corpora are fed into the SVM training module and the model for every phone is obtained. While classifying, a evaluation corpora is used.

4 Experiments 4.1 Performance Measures For similarity measurements between machine and human assessment, three kinds of evaluation metrics are used. They are the assessment correct rate (ACR), the average correlation coefficient (ACC), and the speaker correctrank rate (SCRR). The three evaluation measures reflect the performance of our system from different aspects. ACR can be defined as the overall fraction of phones/syllables which are correctly assessed. ACC measures overall similarity of the machine and human scores, which is calculated as follows:

An SVM-Based Mandarin Pronunciation Quality Assessment System

 T T (fki · gki ) 1 < fk , gk > 1 2i 2 ACC = , = 2 2 T  fk  ·  gk  T i fki i gki k=0

261

(5)

k=0

where fki and gki are the machine and human scores respectively for the syllable i of testee k. And T is the testees number. The speaker correct-rank rate (SCRR) is usually used in the PSK examination, which denotes one’s Putonghua level [14]. SCRR is the ratio of the number of correctly ranked speaker number, whose machine and human scored rank are the same, and the number of all speakers.

4.2 Experimental Setup The AMs used in our experiments are gender-dependent, continuous mixture density, state-tied, cross-word HMMs, which model monophone or triphone. For the triphone based AMs, the HMMs consist of about 5100 tiedstates, each of which has 16 Gaussian components. For the monophone based ones, there are about 700 tied-states and the number of Gaussian components per state is investigated. The front end used MFCC analysis to get the 39-dimensional feature, including 12-dimensional static cepstra and 1dimensional energy with 1st and 2nd order derivatives. The training data are the same for the above two kinds of AMs, about 400 hours native Mandarin speech with 200 hours female and 200 hours male data. We collected three test sets of PSK, which are named as PSK1, PSK2 and PSK3 including 289, 201 and 400 testees respectively. They are all spoked by Hong Kong native undergraduates with strongly accent, and recorded in the actual PSK examination. Transcriptions for each test set are the same, but that for different sets are distinct. They are annotated by five trained phoneticians, each of whom grades the pronunciation quality of consonant, vowel and tone for every syllable with 2, 1 or 0, standing for good, medium and bad pronunciation respectively. According to voting mechanism, the combined human scores is obtained. We randomly draw out seventy percents of every test set as its development corpus and the rest are used as its evaluation corpus.

4.3 System Performance Comparison Using Monophone Based AM with Different GMMs Using monophone as the modeling unit can elaborate the acoustic characteristic of different phones, and the GMM number used to describe the distribution of monophones affects the system performance greatly. Fig. 4 shows the system performance using monophone based AM with different number of GMMs in PSK3 eval. From this figure, it appears that the AM with 64 GMMs is the best in terms of the average CC.

262

F. Ge et al.

Fig. 4 System performance using monophone based AM with different number of GMMs in PSK3

4.4 Evaluation This section presents our experiment results with the three kinds of performance measures described in 4.1, illustrating the performance improvement from different aspects. In the below tables, “Baseline monophone AM” and “Baseline triphone AM” stand for the baseline systems using monophone based and triphone based AM respectively; the improved systems with speaker adaptation module are labeled as “Monophone AM + adaptation” and “Triphone AM + adaptation”. “SVM combination system” denotes the multi-model combination system using SVM, which also adopts speaker adaptation module. Table 1 shows the increase of ACR from baseline system to the SVM combination system. In this table, C-ACR, V-ACR, T-ACR and S-ACR is the assessment correct rate for consonants, vowels, tones and syllables respectively. Comparing the improved systems of speaker adaptation with the baseline systems, it is believed that the speaker adaptation module is helpful to improve ACR, which is the most outstanding on syllables. The line of “SVM combination system” presents that the multi-model combination using SVM increases the assessment accuracy very much. Table 2 indicates the performance improvement in terms of the average CC between machine and human scores. From baseline system to the SVM combination system, the average CC is increased step by step. In particular, the SVM combination system increases the average CC with 1.39% in PSK1, 1.47% in PSK2 and 1.15% in PSK3, compared with the system “Triphone AM + adaptation”. Especially, With PSK2 corpora, the average CC is improved from 83.72% to 85.48%. In the other two evaluation corpora, the average CC based on the SVM combination system is close to that of the inter-expert. So the SVM-based system can achieve usable performance.

An SVM-Based Mandarin Pronunciation Quality Assessment System

263

Table 1 System performance in terms of ACR PSK1 System Description C-ACR Baseline triphone AM 0.890 Baseline monophone AM 0.888 Triphone AM + adaptation 0.892 Monophone AM + adaptation 0.889 SVM combination system 0.900 PSK2 System Description C-ACR Baseline triphone AM 0.896 Baseline monophone AM 0.896 Triphone AM + adaptation 0.899 Monophone AM + adaptation 0.900 SVM combination system 0.909 PSK3 System Description C-ACR Baseline triphone AM 0.884 Baseline monophone AM 0.892 Triphone AM + adaptation 0.886 Monophone AM + adaptation 0.895 SVM combination system 0.905

V-ACR T-ACR 0.890 0.868 0.892 0.870 0.895 0.869 0.901 0.871 0.903 0.875

S-ACR 0.835 0.835 0.838 0.838 0.848

V-ACR T-ACR 0.897 0.894 0.893 0.899 0.903 0.895 0.901 0.899 0.905 0.902

S-ACR 0.854 0.851 0.856 0.856 0.867

V-ACR T-ACR 0.873 0.895 0.871 0.898 0.879 0.897 0.880 0.899 0.885 0.905

S-ACR 0.809 0.809 0.815 0.816 0.829

Table 2 System performance with the average CC between machine and human scores System Description Baseline triphone AM Baseline monophone AM Triphone AM + adaptation Monophone AM + adaptation SVM combination system

PSK1 0.8379 0.8388 0.8410 0.8393 0.8549

PSK2 0.8372 0.8376 0.8401 0.8420 0.8548

PSK3 0.8471 0.8456 0.8505 0.8483 0.8620

Table 3 System performance in terms of SCRR System Description Baseline triphone AM Baseline monophone AM Triphone AM + adaptation Monophone AM + adaptation SVM combination system

PSK1 82.04% 81.67% 83.33% 82.72% 84.20%

PSK2 90.31% 90.20% 92.06% 92.16% 93.20%

PSK3 76.96% 77.37% 78.74% 78.53% 79.44%

The increase of SCRR is shown in Table 3. From the table, we know that SCRR of the SVM combination system is the highest. So we believe that

264

F. Ge et al.

speaker adaptation and multi-model combination using SVM are useful to improve the performance of our Mandarin pronunciation quality assessment system.

5 Conclusion This paper has presented our Mandarin pronunciation quality assessment system for PSK and investigated some measures in order to improve its assessment accuracy. In conclusion, this work indicates that our SVM-based system is capable of providing similar feedback to a testee with regard to which pronunciation can be accepted as correct or not and which rank his/her Putonghua Shuiping is. Future work will concentrate on increasing the agreement of machine scores with human assessment more. Acknowledgements. This work is partially supported by MOST (973 program, 2004CB318106), the National Natural Science Foundation of China (10574140, 60535030), the National High Technology Research and Development Program of China (863 program, 2006AA01010, 2006AA01Z195).

References 1. Neumeyer, L., Franco, H., Weintraub, M., Price, P.: Automatic TextIndependent Pronunciation Scoring of Foreign Language Student Speech. In: Proc. of ICSLP 1996, Philadelphia, Pennsylvania, pp. 1457–1460 (1996) 2. Tatsuya, K., Masatake, D., Yasushi, T.: Practical Use of English Pronunciation System for Japanese Students in the CALL Classroom. In: INTERSPEECH 2004, pp. 1689–1692 (2004) 3. Franco, H., Neumeyer, L., Kim, Y., Ronen, O.: Automatic Pronunciation Scoring for Language Instruction. In: Proc. Int’l. Conf. on Acoust., Speech and Signal Processing, Munich, pp. 1471–1474 (1997) 4. Neumeyer, L., Franco, H., Digalakis, V., Weintraub, M.: Automatic Scoring of Pronunciation Quality. Speech Communication 30(2-3), 83–93 (2000) 5. Franco, H., Neumeyer, L., Digalakis, V., Ronen, O.: Combination of Machine Scores for Automatic Grading of Pronunciation Quality. Speech Communication 30 (2000) 6. Witt, S.M., Young, S.J.: Phone-level Pronunciation Scoring and Assessment for Interactive Language Learning. Speech communication 30(2)-32(3), 95–108 (2000) 7. Bernstein, J., Cohen, M., Murveit, H., Rtischev, D., Weintraub, M.: Automatic Evaluation and Training in English Pronunciation. In: ICSLP Kobe, Japan (1990) 8. Chen, J.C., Jang, J.S.R., Li, J.Y., Wu, M.J.: Automatic Pronunciation Assessment for Mandarin Chinese. In: IEEE International Conference on Multimedia and Expo., Taipei, Taiwan (June 2004)

An SVM-Based Mandarin Pronunciation Quality Assessment System

265

9. Pan, F.P., Zhao, Q.W., Yan, Y.H.: Improvements in Tone Pronunciation Scoring for Strongly Accented Mandarin Speech. In: Proceedings of ISCSLP 2006, pp. 592–602 (2006) 10. Leggetter, C., Woodland, P.: Speaker adaptation of HMMs using linear regression. Technical Report CUED/F-INFENG/TR. 181. Cambridge University Engineering Department, Cambridge, UK (1994) 11. Vapnik, V.: The Nature of Statistical Learning Theory. Springer, Heidelberg (1995) 12. http://www.csie.ntu.edu.tw/~ cjlin/libsvm/ 13. http://ntu.csie.org/~ piaip/svm/svm_tutorial.html 14. Implementation Outline for Putonghua Shuiping Kaoshi. Commercial Press, Beijing (2006)

An Quality Prediction Method of Injection Molding Batch Processes Based on Sub-Stage LS-SVM XiaoPing Guo, Chao Zhang, Li Wang, and Yuan Li*

Abstract. Injection molding process, a typical batch process, due to multistage, nonlinear and unavailability of direct on-line quality measurements, it is difficult for on-line quality control. A sub-stage LS-SVM quality prediction method is proposed for dedicating to reveal the nonlinearly relationship between process variables and final qualities at different stages. Firstly, using an clustering arithmetic, PCA P-loading matrices of time-slice matrices are clustered and the batch process is divided into several operation stages, the most relevant stage to the quality variable is defined, and then applying correlation analysis to get irrelevant variables of un-fold stage data according to time as input of LS-SVM for end-of-batch product quality prediction. For comparison, a sub-MPLS quality prediction method is applied. The experimental results prove that the effectiveness of the proposed quality prediction method is superior to the one of the sub-MPLS method. Keywords: Quality prediction, Batch process, LS-SVM, Sub-Stage.

1 Introduction Injection molding processes, a typical multistage batch process, is an important polymer processing technique and transforms polymer materials into various shapes and types of products. However, due to the process high dimensionality, complexity, batch-to-batch variation, and also limited product-to-market time, the final product quality are usually available at the end of the batch, which is analysed (mostly offline) after the batch completion. It is difficult for on-line quality control. Several statistical modeling methods such as multi-way partial least square (MPLS) models, have been reported recently for batch processes[1], Nevertheless, MPLS method is inefficient in revealing time-specific relationships for some multi stage processes and is a linear method. Consequently, it performs poorly in predicting response variables of nonlinear batch processes. XiaoPing Guo . Chao Zhang . Li Wang . Yuan Li Information Engineering School, Shenyang Institute of Chemical Technology, Shenyang 110142, China [email protected] *

H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 267–273. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

268

X. Guo et al.

However, an artificial neural network (ANN) had the capability to handle the modeling problems associated with nonlinear static or dynamic behaviors. The NNPLS method [2]differs from the direct ANN approach in that, the input–output datas are not directly used to train the NN, but are preprocessed by the PLS outer transform. Jia bo Zhu et al (1998) [3] proposed a time-delay neural network (TDNN) modeling method for predicting the treatment result. Vapnik (1998) presented the Support Vector Machines (SVM), which can give attention to both the expectation risk and the generalization performance and can be used to approximatenonlinear functions. Suykens and Vandewalle (1999) presented LS-SVM method, in which the objective function includes an additional sum squared error term. LS-SVM is one of the methods by which the statistical learning theory can be introduced to practical application. It has its own advantages in solving the pattern recognition problem with small samples, nonlinearity, and higher dimension. And it can be easily introduced into learning problem such as function estimation[4,5]. For multi-stage batch processes,each stage has its own underlying characteristics, and a batch process can exhibit significantly different behaviors over different stages. It is therefore natural to develop stage-based statistical modeling methods to reflect the inherent stage nature to improve the performances of quality control. A stage-based sub-PCA modeling method[6] has been developed, and it has been shown that the stage PCA modeling can overcome many difficulties of MPCA-based monitoring for batch processes. Thus, in the present paper, for injection molding process, a sub-stage LS-SVM quality prediction method is proposed for dedicating to reveal the nonlinear relationship between process variables and final qualities at different stages, and to build a stage-based quality prediction model. Firstly, using an clustering arithmetic, PCA P-loading matrices of time-slice matrices is clustered and the batch process is divided into several operation stages according to the change of process correlation, the most relevant stage to the quality variable is defined, then applying correlation analysis to get inrelevant variables to un-fold stage data according to time as input of quality prediction model, finally, sub-stage LS-SVM models are developed for every stage for quality prediction. For comparison purposes sub-MPLS quality models are established. The results prove the effectiveness of the proposed quality prediction method is superior to the one of the sub-MPLS quality prediction method.

2 Process Description Injection molding, an important polymer processing technique, transforms polymer materials into various shapes and types of products. As a typical multistage process, injection molding operates in stages, among which, filling, packingholding, and cooling are the most important phases. During filling, the screw moves forward and pushes melt into the mold cavity. Once the mold is completely filled, the process then switches to the packing-holding stage, during which additional polymer is “packed” at a high pressure to compensate for the material shrinkage associated with the material cooling and solidification. The packingholding continues until the gate freezes off, which isolates the material in the mold from that in the injection unit. The process enters the cooling stage; the part in the

An Quality Prediction Method of Injection Molding Batch Processes

269

mold continues to solidify until it is rigid enough to be ejected from the mold without damage. Concurrently with the early cooling phase, plastication takes place in the barrel where polymer is melted and conveyed to the front of barrel by screw rotation, preparing for next cycle[7]. For injection molding, high degree of automation is possible. After the process conditions are properly set, the process repeats itself to produce molded part at a high rate. The process is, however, susceptible to the production of off-spec products due to various process malfunctions, drifting of process conditions, changes in materials, and unknown disturbances. Abrupt, gross faults in the key process variables can be easily and reliably detected by the conventional SPC chart. Slow drift or faults involving multiple process variables, however, can not be detected. These process faults, even if they are small and not common, can lead to production of large quantity of bad parts, if they are not detected earlier. The material used in this work is high-density polyethylene (HDPE). Ten process variables are selected for modeling, they are Nozzle Pressure, Stroke, Injection Velocity, Hydraulic Pressure, Plastication Pressure, Cavity Pressure, Screw Rotation Speed, SV1 valve opening, SV2 valve opening, and Mold Temperature, respectively. The sampling rates of these process variables are 20 ms. The operating conditions are set as follows: injection velocity is 25mm/sec; mold temperature equals 25ºC; seven-band barrel temperatures are set to be (200, 200, 200, 200, 200, 180, 160, 120) ºC; packing-holding time is fixed to be 3 second. Quality variables are part weight. Totally, 60 batch runs are conducted under 19 different operation conditions, which can cover all the normal operation range. Based on these data, an stage-based sub-MPLS model and LS_SVM model are developed for quality prediction.

3 A Stage-Based LS-SVM Modeling 3.1 LS-SVMs(Least Square-Support Vector Machines (SVMs)) LS-SVMs follow the approach of a primal-dual optimization formulation, where this technique makes use of a so-called feature space where the inputs have been transformed by means of a (possibly infinite dimensional) nonlinear mapping.This is converted to the dual space by means of Mercer’s theorem and the use of a positive definite kernel, without computing explicitly the mapping. It has its own advantages in solving the pattern recognition problem with small samples, nonlinearity, and higher dimension. Suykens and Vandewalle (1999) presented the LS-SVM approach, in which the following function is used to approximate the unknown function,

，

y ( xi ) = ω ⋅ φ( xi ) + b

，

(1)

where, xi ∈ Χ ⊂ R n yi ∈ Υ = R φ(⋅) : R n → R nh is a nonlinear function which maps the input space into a higher dimension feature space.Given training data,

270

X. Guo et al.

T = {( x1 , y1 ), ,( xl , yl )} ∈ (Χ × Υ )l , the goal is to estimate a model of the form, LSSVM defines an optimization problem as follows, 1 1 l 2 min J (ω, e) = ω + γ ∑ ei2 , (γ > 0) (2) ω,b , e 2 2 i =1

subject to the equality constraints yi = ωT ⋅ φ( xi ) + b + ei , i = 1, 2,… l To solve this optimization problem, one defines the following Lagrange function, L(ω, b, e, α ) = J (ω, e) − ∑ αi ( ωT ⋅ φ( xi ) + b + ei − yi ) l

(3)

i =1

Where, α = {αi }i =1 is the Lagrange multiplier set. l

Calculating the partial derivatives of L(ω, b, e, α ) with respect to ω, b, e, α , one gets the optimal condition for Eq.(2) as l

∇ b L = ∑ αi = 0, i =1

l

∇ ω L = ω − ∑ αi φ( xi ) = 0,

(4)

i =1

∇ ei L = αi − γei = 0, i = 1, 2,… l , ∇ αi L = ωT ⋅ φ( xi ) + b + ei − yi = 0, i = 1, 2,… l ,

Expressing ei and w with αi and b, one can transform the above equality into ⎡Ω + γ −1 I ⎢ T ⎣ 1

1 ⎤ ⎡α ⎤ ⎡ y ⎤ ⎥⎢ ⎥ = ⎢ ⎥ 0⎦ ⎣b ⎦ ⎣ 0 ⎦

(5)

where y = [ y1 ,… , yl ]T , α = [α1 ,… , αl ]T ,1 = [1,… ,1]T and Ω is a square matrix in which the element located on lth column and mth row is Ωij = φT ( xi )φ( x j ) = K ( xi , x j ), i, j = 1,… , l.

(6)

Choosing γ > 0 , ensures that the matrix ⎡ Ω + γ −1 I Φ=⎢ T ⎣ 1

1⎤ ⎥ is invertible. Then we have the analytical solution of a and b 0⎦ ⎡α ⎤ −1 ⎡ y ⎤ ⎢b ⎥ = Φ ⎢ 0 ⎥ ⎣ ⎦ ⎣ ⎦

(7)

Substituting the obtained b and a into Eq.(4), we get l

y ( x) = ∑ αi K ( x, xi ) + b

(8)

i =1

where K(x, xk) is the Kernel function, which can be any symmetric function satisfying Mercer’s condition(Smola, 1996).

An Quality Prediction Method of Injection Molding Batch Processes Fig. 1 Sub-stage modeling method

271

k

i

K

j

1

k

2

1 ~ X1

PCA

~ P1

2 ~ X2

~ P2

k

K

~ Xk

~ XK

~ PK

~ Pk

Clustering algorithm

stage1 Sub-model 1

stage c

Y

Sub-model c

3.2 Sub Stage LS-SVMS Modeling The data gathered from injection molding process forms a three-dimensional data matrix, X( I × J × K ) , where for batch process applications, I denotes cycle number, J denotes variable number, and K denotes the number of samples within a cycle. The X ( I × J × K ) is unfolded, with each of the K time slabs concatenated to produce a two-way array, Xnew ( I × JK ) . The multi-way unfolding procedure is graphically shown in Fig. 1. Based on the work of sub-PCA stage partition strategy[6] an batch run is divided into several stages. An un-fold data of one stage represented as X ( I × m ) , where I is batch numbers, and m is the number of process variables after un-fold data of the stage. All un-fold data of every stage is proposed to extract the local covariance information of process variable and get irrelevant variables as input variables of model. Using irrelevant input variables data and quality data established sub LS-SVM models. For comparison purposes sub-MPLS models were established.

，

4 Experimental Results Without using any prior process knowledge, using sub-PCA based stage-division algorithm, the trajectories of an injection molding batch run is divided into four stages according to the change of local covariance structure, correspond to the four physical operation stages, i.e., injection, packing-holding, plastication and cooling stages shown in Fig. 2. The final quality variables have weak relation with the plastication and cooling stage. The on-line quality prediction model is a distributed model, the weight variables are estimated by the sub-MPLS model and the sub-LSSVM model in packing stage.

272

X. Guo et al. Stages defined by the proposed method

0 1 72

2

3

219 250

0 72 248 Injection Packing-Holding

4

565 578

557 Plastication Physical Operating Stages

Cooling

1000

1000

Fig. 2 Stage division results for injection molding process

，

For illustration, the results of the LS_SVM and he sub-MPLS output prediction for 30 batches are shown in Fig. 3. In Fig. 3 the solid line with circle symbols indicates the weight measurements, and the solid line with triangle symbols plots the corresponding weight prediction using sub-MPLS model and the solid line with square symbols plots the corresponding weight prediction using sub-LSSVM model. It is clear that the predictions of two methods are much closer to the actual weight measurements, indicating significant improvement by using tail data of the packing stage prediction model. But there exists an obvious offset between the measured and predicted values for sub-MPLS methods, and the value of the offset varies with different operating conditions, for instance, the offset for batch 5 is much smaller than that of batch 20. An analysis suggests that the problem was caused by nonlinear of packing stages because sub-MPLS model is linear. But the product weight predicted by sub-LSSVM model can be more exactly predicted.

28.5

Measurment Sub_LSSVM Sub_MPLS

28.0

Weight

27.5

27.0

26.5

26.0 5

10

15

20

25

30

Batch Number

Fig. 3 Predicted results for weight variables using sample time data of packing stages

An Quality Prediction Method of Injection Molding Batch Processes

273

5 Conclusion For injection molding process, an typical multi-stage batch process, a new quality prediction method has been applied. Firstly, Process stages are determined by analyzing the change of process covariance structure and partitioning time-slice PCA loading matrices using a new clustering algorithm. Then applying correlation analysis to get irrelevant variables of un-fold stage data according to time as input, then modelling sub LS-SVM model, finally, the predicted result of sub LS-SVM model is compared with the one of the sub-MPLS model, the predicted precision of the proposed model is superior to the one of the sub-MPLS model. The results have demonstrated the effectiveness of the proposed method. Acknowledgments. This work was supported by the National Science Foundation of China under Grant 60776070 and project of Department of Education of Liaoning under Grant 2008566.

References 1. Nomikos, P., Macgregor, J.F.: Multiway Partial Least Squares in Monitoring Batch Processes. J. Chemometrics Intell. Lab. Syst. 30, 97–108 (1995) 2. Qin, S.J., McAvoy, T.J.: Nonlinear PLS Modeling using Neural Networks. J. Comput. Chem. Eng. 16, 379–391 (1992) 3. Jiabao, Z., Zurcher, J., Ming, R., Meng, Q.H.: An Online Wastewater Quality Predication System Based on a Time-delay Neural Network. J. Engineering Applications of Artificial Intelligence 11, 747–758 (1998) 4. Suykens, J.A.K., Vandewalle, J.: Least Squares Support Vector Machine classifiers. J. Neural Processing Letters 9, 293–300 (1999) 5. Smola, A.J.: Regression Estimation with Support Vector Learning Machines. Master’s Thesis, Technische Universität München (1996) 6. Lu, N., Gao, F., Yang, Y., Wang, F.: A PCA-based Modeling and On-line Monitoring Strategy for Uneven-length Batch processes. J. Ind. Eng. Chem. Res. 43, 3343–3352 (2004) 7. Yang, Y.: Injection Molding: from Process to Quality Control, Ph.D. Thesis, The Hong Kong University of Science & Technology (2004)

Soft Sensing for Propylene Purity Using Partial Least Squares and Support Vector Machine Zhiru Xu, Desheng Liu, Jingguo Zhou, and Qingjun Shi*

Abstract. Many important process variables in propylene distillation actual system is difficultly detected directly or not easy online survey, especially propylene purity which will vary with other process parameter. This paper presents an online soft sensing method by combining partial least squares and support vector machine. First multivariate data analysis was performed using partial least squares, then soft sensing regression model was constructed. Simulation shows that the proposed measuring scheme guarantees parameter estimation and predicting accuracy. Keywords: Soft sensing, Partial least squares, Support vector machine, Propylene purity.

1 Introduction Batch distillation is one of the most important separation processes used in many chemical industries, especially those related to the manufacture of fine and specialty chemicals and in the recovery of volatile compounds from industrial waste[1]. The propylene rectifying tower is the essential in the ethylene product installment, it can separate the propylene of chemistry level or polymerization level from the propane binary mixture and propylene and can get propane product [2]. But one of the main disadvantages of a propylene rectifying is that the process exhibits nonlinear characteristics such as multiple steady states and high sensitivity to operating variables due to the coupling between separation and chemical reaction [3]. Furthermore, in propylene purity actual observation and control system, many important process variable is be difficultly detected directly or not easy online survey Hence special consideration must be given to the design of its detecting and control system. In order to solve this kind of estimate and control variables, the soft-sensing technology came into being. Research and development of soft sensing has been comprehensively reviewed by Arijit Bhattacharya, Pandian Vasan [4], L. Fortuna,

，

Zhiru Xu . Desheng Liu . Jingguo Zhou . Qingjun Shi School of Electrics and Information Engineering, Jiamusi University Heilongjiang 154007, China [email protected] *

H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 275–281. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

276

Z. Xu et al.

S. Graziani, M. G. Xibilia [5], Weiwu Yan, Huihe Shao, Xiaofan Wang[6]. The basic idea soft-sensing technology is to choose a set of measurable or easily measured parameters (referred to as secondary or ancillary variable variables), the auxiliary variables and the estimated variable-led are be inferred by constructing a mathematical relationship, measurement function is achieved in place of hardware. In this paper, the support vector machine (SVM) is employed to predicting propylene purity. First of all, using partial least-squares(PLS) methods principal component analysis for distillation’s parameters is realized to simplify dimension of the variable parameters, these parameters include refluxing irrigation liquid-level, quantity, overhead temperature of b, reflux temperature column pressure, overhead temperature of a overhead temperature difference and so on, they will affect propylene purity. And then SVM regression model is established through off-line data, and on-line forecast of the propylene purity is accomplished base on the model. The outline of the paper is as follows. Section 2 concentrates on the process of Propylene distillation. The concept of partial least squares and Correlation analysis results are given in Section 3. The experimental results and discussion are presented in Section 4. Some final remarks and overall conclusions are given in Section 5.

，

，

2 Propylene Distillation Process The control process of propylene distillation sees Figure 1[7]. Petroleum gas from the catalytic cracking unit by washing the desulfurizing liquefied goes into the gas separation plant, the light components above C3 are separated in the column 1, and then by the ethane from column 2 removal them into propylene distillation column (column 3, 4), propylene, propylene oxide are isolated. From the analysis of product quality, the column1, 2 have the better separation precision. In good working order, propane do not carry propane C4 component, and propylene not also carry C2 components. As a result, the operation focused on the distillation column on propylene.

，

Fig. 1 Process of propylene distillation Condenser

Mixture feed

Reflux tank

Propylene product Distillation Column Reboil

Reboil

Propane product

Soft Sensing for Propylene Purity Using Partial Least Squares

277

3 Correlation Analysis Based on Partial Least Squares PLS is a new kind of multivariate statistical data analysis, partial least squares is designed to explain the variable space to find some combination of linear, in order to better explain the variations in response to variable information[8]. Assuming that all explanatory variables are associated with the response variable, PLS basic idea is as follows as shown in Figure 2:

Reaction component

Interpretation composition PLS

OLS Interpretation variable

Reaction variable

Fig. 2 Modeling diagram partial least-squares 0.2 0.15

Propylen Purity Value

0.1 0.05 0 -0.05 -0.1 -0.15 -0.2 -0.25 -0.3

Product Output Refluxing Irrigation LiquidQuantity of Reflux Overhead Temperature of B Reflux Temperature Column Pressure Overhead Temperature of A Overhead Temperature Reflux Ratio

Fig. 3 Principal component analysis on process parameters

278

Z. Xu et al.

Ordinary least-squares regression OLS (ordinal least squares) establishes a direct response to variable interpretation the variable linear regression model, reflecting the linear relationship between the two (virtual arrow in Figure 2); but the establishment of PLS is to explain latent variable on the response of the latent variable linear regression model, reflecting the indirect explanation of variables and the relationship between the response variables (the arrow in Figure 2). In this study multivariate data analysis was performed using PLS. variable preprocessing was performed. Principal component analysis (PCA) was performed to get an overview on the offline data sets. The information contained in original variables was summarized by calculation of new latent variables. The compounds, which could not be well explained with the latent variables, were classified as outliers in PCA. Analysis showed impact factor by a variety of parameters on the purity of propylene (see Figure 3), the relevance of the parameters are different (in which, quantity 0.1886441 reflux temperature-0.241395 column pressure-0.276399 overhead temperature of a 0.261821), therefore in this paper the four parameters with a large proportion were choose to be a forecast of input parameters of the propylene purity.

，

，

，

4 Soft Sensing Methods of Propylene Purity Based on SVM 4.1 SVM Regression Analysis Support vector machine (SVM) are learning systems that use a hypothesis space of linear function in a high dimensional feature space, trained with a learning algorithm from optimization theory that implements a learning bias derived from statistical learning theory. Least squares support vector machines (LSSVM) are reformulations to the standard SVM which result in a set of linear equations instead of a quadratic programming problem of SVM[9]. Based on the statistical learning theory and the structural risk minimization principle, SVM obtain the solution by solving the quadratic programming problem while avoiding the local minima, which provides an advantage over other regression techniques[10]. On the other hand, the least squares version of the SVM algorithms finds the solution by solving a set of linear equations. The main advantage of LSSVMs over SVMs is solving a linear system instead of solving a quadratic programming problem. The motivation for choosing LSSVMs as the approximation tool is their higher generalization capability, as well as the achievement of an almost global solution within a reasonably short training time. In feature space, an LSSVM regression model is of the form,

f ( x) = wT φ ( x) + b

(1)

where the weight vector w ∈ R and φ (⋅) is a nonlinear mapping from the input space to a feature space and b is the bias term. Dual form of the model is given by n

N

f ( x) = ∑ σ k K ( xk , x) + b k =1

(2)

Soft Sensing for Propylene Purity Using Partial Least Squares

where

279

σ k s are positive real constants and k (⋅) is a kernel, which is defined to be

a function such that,

k ( x, z ) = (φ ( x), φ ( z ))

(3)

The mapping φ (⋅) is not needed to be defined explicitly since, according to Mercer’s theorem, it can be connected to a kernel function by (3) under some conditions. In this work, a Gaussian function (4) is adopted as a kernel function,

k ( x, z ) = exp(−

x−z

σ2

2

)

(4)

where σ is the width of the function. The attainment of the kernel function is cumbersome and it will depend on each case[11]. However, the kernel function more used is the radial basis function (RBF), a simple Gaussian function, and polynomial functions. For σ of the RBF kernel and d of the polynomial kernel it should be stressed that it is very important to do a careful model selection of the tuning parameters, in combination with the regularization constant, in order to achieve a good generalization model.

4.2 Soft Sensing Model of Propylene Purity Based on SVM Actual production, because of the complex industrial processes, propylene purity is affected by many factors. There is a certain degree of relevance and redundancy between the impact of factors, it will lead to the importation of soft sensing model number is very large, prone to over-fit and seriously affect computing speed and accuracy of forecasts of the soft sensing model. PLS for feature extraction from the original the relevant isolated characteristics, can effectively deal with colinearity between variable, lower variable dimension to feature extraction and selection of the purpose, and effectively address co-linear and data redundancy. In this paper, a soft sensing model combined PLS and SVM is proposed, shown in Figure 4.

x1

Z1(x)

x2

Z2(x)

x3 . . xn .

Mean Normalization

. . .

PLS Feature Extraction

Z3(x) . . Zn.(x)

Fig. 4 Soft sensing model combining PLS and SVM

SVM Regression

Fine Propylene Purity(%)

280

Z. Xu et al.

Measured sample value

99.77

predictive value 99.76 99.75 99.74 99.73 0

20

40 60 sampling point

80

100

Fig. 5 Output of soft sensing model

The input–output data were divided into training sets(100 data ) and test sets(100 data), which consisted of different batches, respectively. Thus, a highly informative process database was generated economically consisting of quantity reflux temperature column pressure overhead temperature of a. Kernel of SVM regression Choices radial basis function of nuclear, the degree of punishment parameters of Samples gam=100, radial basis kernel parameters of the test sig2 = 50; The predictive results are as figure 5:

，

，

，

0.03

Error Analysis

0.02 0.01 0 -0.01 -0.02

0

20

Fig. 6 Predictive error

40 60 sampling point

80

100

Soft Sensing for Propylene Purity Using Partial Least Squares

281

It can be observed from simulation results (fig. 5) that the LSSVM model exhibits an acceptable reconstruction performance for the soft sensing of the Propylene purity. Red dot discrete curve is predictive value of SVM model, Black curve is actual sensor measurements. As the Environmental interference and measurement error of instruments and so on, measurements are often volatile. So soft sensing model can avoided effectively these problems brought by instrument. Fig 6 shows that errors are within the range of the production process.

5 Conclusion In this paper, a novel approach based on combining PLS and LSSVM has been applied on the task of predicting propylene purity. PLS has a strong feature extraction capabilities, can effectively reduce the sample-dimensional, provide samples which is simple and information which is full to input; for the model; SVM has a good ability of non-linear function approximation. In this paper, Soft-sensing modeling method combines the advantages of both. Experiments have been conducted on the actual distillation tower. The simulation the results strongly suggest that LSSVM can aid in the predicting propylene purity. It is hoped that more interesting results will follow on further exploration of data.

References 1. Fileti, A.M.F., Cruz, S.L., et al.: Control strategies analysis for a batch distillation column with experimental testing. Chemical Engineering and Processing 39, 121–128 (2000) 2. Gadalla, M., Olujic, Z., et al.: A design method for internal heat integrated distillation columns (iHIDiCs). Computer Aided Chemical Engineering 24, 1041–1046 (2007) 3. Ali, M.A.-H., Betlem, B., et al.: Non-linear model based control of a propylene polymerization reactor. Chemical Engineering and Processing 46, 554–564 (2007) 4. Bhattacharya, A., Vasant, P.: Soft-sensing of level of satisfaction in TOC product-mix decision heuristic using robust fuzzy-LP. European Journal of Operational Research 177, 55–70 (2007) 5. Fortuna, L., Graziani, S., et al.: Soft sensors for product quality monitoring in debutanizer distillation columns. Control Engineering Practice 13, 499–508 (2005) 6. Yan, W., Shao, H., et al.: Soft sensing modeling based on support vector machine and Bayesian model selection. Computers & Chemical Engineering 28, 1489–1498 (2004) 7. Dayi, D., Xionglin, L.: Aplacation of advanced control in propylene rectifying tower of gas ends plant. Petroleum refinery engineering 32, 48–51 (2002) (in China) 8. Sellin, N.: Partial least square modeling in research on educational achievement. Reflections on Educational Achievement, 256–257 (1995) 9. Li, S., Yao, X., et al.: Prediction of T-cell epitopes based on least squares support vector machines and amino acid properties. Analytica Chimica Acta 584, 37–42 (2007) 10. Iplikci, S.: Dynamic reconstruction of chaotic systems from inter-spike intervals using least squares support vector machines. Nonlinear Phenomena 216, 282–293 (2006) 11. Borin, A., Ferrao, M., et al.: Least-squares support vector machines and near infrared spectroscopy for quantification of common adulterants in powdered milk. Analytica Chimica Acta 579, 25–32 (2006)

Application of Support Vector Machines Method in Credit Scoring Leilei Zhang and Xiaofeng Hui*

Abstract. Credit scoring has attracted lots of research interests in the literature. The credit scoring manager often evaluates the consumer’s credit with intuitive experience. However, with the support of the credit classification model, the manager can accurately evaluate the applicant’s credit score. Support Vector Machine (SVM) classification is currently an active research area and successfully solves classification problems in many domains. This article introduces support vector machines (SVM), to the problem in attempt to provide a model with better explanatory power. We used backpropagation neural network (BNN) as a benchmark and obtained prediction accuracy around 80% for both BNN and SVM methods for the Australian and German credit datasets from UCI. Keywords: Credit scoring, SVM, BNN.

1 Introduction Credit scoring is a method of modelling potential risk of credit applications. In general it employs statistical techniques and historical data to produce a score that financial institutions can use to evaluate credit applicants in terms of risk.[1] With the rapid growth in the credit industry, credit scoring models have been extensively used for the credit admission evaluation. In the last two decades, several quantitative methods have been developed for the credit admission decision. The credit scoring models are developed to categorize applicants as either accepted or rejected with respect to the applicants’ characteristics such as age, income, and marital condition. [2-4]Credit officers are faced with the problem of trying to increase credit volume without excessively increasing their exposure to default. Therefore, to screen credit applications, new techniques should be developed to help predict credits more accurately. The benefits of credit scoring involve reducing the credit analysis cost, enabling faster credit decisions, closer monitoring of existing accounts and prioritizing credit collections. Although rating agencies and many institutional writers emphasize the importance of analysts’ subjective judgment in determining credit scoring, many researchers have obtained promising Leilei Zhang . Xiaofeng Hui School of Managment, Harbin Institute of Technology, Harbin, China [email protected]

*

H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 283–290. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

284

L. Zhang and X. Hui

results on credit scoring prediction applying different statistical and Artificial Intelligence (AI) methods. [5-7]. The overall objective of credit scoring is to build models that can extract knowledge of credit risk evaluation from past observations and to apply it to evaluate credit risk of companies or persons with much broader scope. The most popular methods adopted in the scoring industry are linear discriminant analysis, logistic regression and their variations. They are relatively easy to implement and are able to generate straightforward results that can be readily interpreted, but this usually requires domain expert knowledge and in-depth understanding of the data. In addition, these methods are not effective for problems with high-dimensional inputs and small sample size (e.g. new creditors). In real world application, assumptions regarding the data must be held, such as being linear separable and that the data should follow certain distributions. In this study, we experimented with using a relatively new learning method for the field of credit scoring prediction, support vector machines, together with a frequently used high-performance method, backpropagation neural networks, to predict credit scoring.

2 Inductive Learning of Classifiers A classifier is a function that maps an input attribute vector, x = ( x1,x2,x3,...xn ) , to the confidence that the input belongs to a class – i.e., f ( x ) = confidence(class). In the case of text classification, the attributes are words in the document and the classes are the categories of interest. The key idea behind SVMs and other inductive learning approaches is to use a training set of labeled instances to learn the classification function automatically. SVM classifiers resemble the second example above – a vector of learned feature weights. The resulting classifiers have many advantages: they are easy to construct and update, they depend only on information that is easy for people to provide (i.e., examples of items that are in/out of categories), and they allow users to smoothly tradeoff precision and recall depending on their task.

3 The Proposed Support Vector Machine Algorithm The support vector machine (SVM) is a training algorithm for learning classification and regression rules from data, for example the SVM can be used to learn polynomial, radial basis function (RBF) and multi-layer perceptron (MLP) classifiers SVMs were first suggested by Vapnik in the 1960s for classification [810]and have recently become an area of intense research owing to developments in the techniques and theory coupled with extensions to regression and density estimation. SVMs arose from statistical learning theory; the aim being to solve only the problem of interest without solving a more difficult problem as an intermediate step. SVMs are based on the structural risk minimisation principle, closely related

Application of Support Vector Machines Method in Credit Scoring

285

to regularisation theory. This principle incorporates capacity control to prevent over-fitting and thus is a partial solution to the bias-variance trade-off dilemma. Two key elements in the implementation of SVM are the techniques of mathematical programming and kernel functions. The parameters are found by solving a quadratic programming problem with linear equality and inequality constraints; rather than by solving a non-convex, unconstrained optimisation problem. The flexibility of kernel functions allows the SVM to search a wide variety of hypothesis spaces. Here we focus on SVMs for two-class classification, the classes being P, N

yi = +1,−1 respectively. This can easily be extended to k − class classification by constructing k two-class classifiers The geometrical interpretation of for

support vector classification (SVC) is that the algorithm searches for the optimal separating surface, i.e. the hyperplane that is, in a sense, equidistant from the two classes. This optimal separating hyperplane has many nice statistical properties. SVC is outlined first for the linearly separable case. Kernel functions are then introduced in order to construct non-linear decision surfaces. Finally, for noisy data, when complete separation of the two classes may not be desirable, slack variables are introduced to allow for training errors. Maximal Margin Hyperplanes If the training data are linearly separable then there exists a pair that

(w, b) such

w T xi + b ≥ 1, for all xi ∈ P

(1)

w T xi + b ≤ −1, for all xi ∈ N with the decision rule given by

f w ,b (x) = sgn( wT x + b)

(2)

w is termed the weight vector and b the bias (or − b is termed the threshold). The inequality constraints (1) can be combined to give

yi (w T x i + b) ≥ 1, for all x i ∈ P ∪ N

(3)

Without loss of generality the pair (w , b) can be rescaled such that

min w T xi + b = 1 ,

i =1,…, l

this constraint defines the set of canonical hyperplanes on ℜ . In order to restrict the expressiveness of the hypothesis space, the SVM searches for the simplest solution that classifies the data correctly. The learning N

problem is hence reformulated as: minimize

w = wT w 2

subject to the

286

L. Zhang and X. Hui

constraints of linear separability (3). This is equivalent to maximising the distance, normal to the hyperplane, between the convex hulls of the two classes; this distance is called the margin. The optimisation is now a convex quadratic programming (QP) problem

1 2 w 2 T subject to yi (w xi + b) ≥ 1, i = 1,…, l. Minimize Φ (w ) = w ,b

(4)

This problem has a global optimum; thus the problem of many local optima in the case of training e.g. a neural network is avoided. This has the advantage that parameters in a QP solver will affect only the training time, and not the quality of the solution. This problem is tractable but in order to proceed to the non-separable and non-linear cases it is useful to consider the dual problem as outlined below. The Lagrangian for this problem is

L ( w , b, Λ ) = where

[

]

l 1 2 w − ∑ λi yi (wT x i + b) − 1 2 i =1

(5)

Λ = (λ1 ,…, λl )T are the Lagrange multipliers, one for each data point.

The solution to this quadratic programming problem is given by maximising L with respect to Λ ≥ 0 and minimising with respect to w , b . Differentiating with respect to w and b and setting the derivatives equal to 0 yields l ∂ L(w, b, Λ ) = w − ∑ λi yi xi = 0 ∂w i =1

and l ∂ L(w, b, Λ ) = −∑ λi yi = 0 ∂b i =1

(6)

So that the optimal solution is given by (2) with weight vector l

w * = ∑ λ*i yi xi

(7)

i =1

Substituting (6) and (7) into (5) we can write l

F (Λ ) = ∑ λi − i =1

l 1 1 l l 2 T w = ∑ λi − ∑∑ λi λ j yi y j xi x j 2 2 i =1 j =1 i =1

which, written in matrix notation, leads to the following dual problem

(8)

Application of Support Vector Machines Method in Credit Scoring

Maximize F (Λ ) = ΛT I −

1 T Λ DΛ 2

287

(9)

subject to Λ ≥ 0, Λ y = 0 T

where

y = ( y1 , … , y l ) T and D is a symmetric l × l matrix with elements

Dij = yi y j xi x j . Note that the Lagrange multipliers are only non-zero when T

yi (wT x i + b) = 1 , vectors for which this is the case are called support vectors since they lie closest to the separating hyperplane. The optimal weights are given by (10) and the bias is given by T

b* = yi − w * xi for any support vector

(10)

xi (although in practice it is safer to average over all sup-

port vectors). The decision function is then given by

⎛ l ⎞ f (x) = sgn ⎜ ∑ yi λ*i xT xi + b* ⎟ ⎝ i =1 ⎠ The solution obtained is often sparse since only those

(11)

x i with non-zero Lagrange

multipliers appear in the solution. This is important when the data to be classified are very large, as is often the case in practical data mining situations. However, it is possible that the expansion includes a large proportion of the training data, which leads to a model that is expensive both to store and to evaluate. Alleviating this problem is one area of ongoing research in SVMs.

4 Backpropagation Neural Network Backpropagation neural networks have been extremely popular for their unique learning capability and have been shown to perform well in different applications in our previous research such as medical application and game playing. A typical backpropagation neural network consists of a three layer structure: input-layer nodes, output-layer nodes and hidden-layer nodes. In our study, we used financial variables as the input nodes and rating outcome as the output layer nodes. Backpropagation networks are fully connected, layered, feed-forward models. Activations flow from the input layer through the hidden layer, then to the output layer. A backpropagation network typically starts out with a random set of weights. The network adjusts its weights each time it sees an input–output pair. Each pair is processed at two stages, a forward pass and a backward pass. The forward pass involves presenting a sample input to the network and letting activations flow until they reach the output layer. During the backward pass, the network’s actual output is compared with the target output and error estimates are

288

L. Zhang and X. Hui

computed for the output units. The weights connected to the output units are adjusted to reduce the errors (a gradient descent method). The error estimates of the output units are then used to derive error estimates for the units in the hidden layer. Finally, errors are propagated back to the connections stemming from the input units. The backpropagation network updates its weights incrementally until the network stabilizes.

5 Experimental Studies The two real world data sets, the Australian and German credit data sets, are available from the UCI Repository of Machine Learning Databases (Murphy & Aha, 2001) and are adopted herein to evaluate the predictive accuracy. The Australian credit data consists of 307 instances of creditworthy applicants and 383 instances where credit is not creditworthy. Each instance contains 6 nominal, 8 numeric attributes, and 1 class attribute (accepted or rejected). This dataset is interesting because there is a good mixture of attributes: continuous, nominal with small numbers of values, and nominal with larger numbers of values. There are also a few missing values. To protect the confidentiality of data, the attributes names and values have been changed to meaningless symbolic data. The German credit scoring data are more unbalanced, and it consists of 700 instances of creditworthy applicants and 300 instances where credit should not be extended. For each applicant, 24 input variables describe the credit history, account balances, loan purpose, loan amount, employment status, personal information, age, housing, and job title. This data set only consists of numeric attributes.

6 Prediction Accuracy Analysis To evaluate the prediction performance, we followed the 10-fold cross-validation procedure, which has shown good performance in model selection. Because some credit data had a very small number of data points for both the Australian and German datasets, we also conducted the leave one-out cross-validation procedure to access the prediction performances. When performing the cross validation procedures for the neural networks, 10% of the data was used as a validation set. Table 1 summarizes the prediction accuracies of the two models using both crossvalidation procedures. The outcome show that support vector machines outperformed neural networks model; and the 10-fold and leave-one-out cross-validation procedures obtained comparable prediction accuracies. We can draw several conclusions from the experiment results obtained. First, our results conform to prior research results indicating that analytical models based on publicly available financial information built by machine learning algorithms could provide accurate predictions. In our study, we obtained the highest

Application of Support Vector Machines Method in Credit Scoring

289

Table 1 Prediction accuracies (SVM: support vector machines, NN: neural networks)

Data

10-fold

Leave one-out

Cross validation

Cross validation

（%）

SVM

（%）

NN

（%）

SVM

（%）

NN

Australian credit data sets

81.63

76.54

79.87

75.26

German credit data sets

80.00

79.25

80.00

75.68

Fig. 1 ROC curve (SVM)

prediction accuracies of 81.63% for Australian data set and 79.25% for the German data set. Second, support vector machines slightly improved the credit scoring prediction accuracies.

7 Conclusion In this study, we applied a newly introduced learning method based on statistical learning theory, support vector machines, together with a frequently used high performance method, backpropagation neural networks, to the problem of credit scoring prediction. We used two data sets for Australian and German credit datasets as our experiment testbed. The results showed that support vector machines achieved accuracy comparable to that of backpropagation neural networks.

290

L. Zhang and X. Hui

References 1. David, W.: Neural Network Credit Scoring Models. Computers & Operations Research 27, 1131–1152 (2000) 2. Reshmi, M., Malhotra, D.K.: Differentiating between Good Credits and Bad Credits Using Neuro-Fuzzy Systems. Computing, Artificial Intelligence and Information Technology 136, 190–211 (2002) 3. Malhotra, R., Malhotra, D.K.: Evaluating Consumer Loans Using Neural Networks. Omega 31, 83–96 (2003) 4. Baesens, B.: Benchmarking State-of-the-Art Classification Algorithms for Credit Scoring. Journal of the Operational Research Society 54(6), 627–634 (2003) 5. Haärdle, W.: Predicting Corporate Bankruptcy with Support Vector Machines Humboldt University and the German Institute for Economic Research (2003) 6. Schebesch, K.B., Stecking, R.: Support Vector Machines for Classifying and Describing Credit Applicants: Detecting Typical and Critical Regions. Journal of the Operational Research Society 56(9), 1082–1088 (2005) 7. Shin, K.S.: An Application of Support Vector Machines in Bankruptcy Prediction Model. Expert Systems and Applications 28, 127–135 (2005) 8. Syed, N.A.: Incremental Learning with Support Vector Machines. In: Proceedings of the International Joint Conference on Artificial Intelligence (1999) 9. Cauwenberghs, G., Poggio, T.: Incremental and Decremental Support Vector Machine Learning. In: Advances in Neural Information Systems, Cambridge, MA, pp. 409–415 (2000) 10. Hosmer, D.W., Lemeshow, S.: Applied Logistic Regression. Wiley & Sons, New York (1989)

Improving Short Text Clustering Performance with Keyword Expansion Jun Wang, Yiming Zhou, Lin Li, Biyun Hu, and Xia Hu*

Abstract. Most of traditional text clustering methods are based on bag of words representation, which ignore the important information on semantic relationship between key terms. To overcome this problem, researchers have recently proposed several new methods for improving short text clustering accuracy based on enriching short text representation. However, the computational costs of these methods based on expanding words appeared in short texts are usually time-consuming. In this paper, we improve previous work by enriching short text representation with keyword expansion. Empirical results show that the proposed method can greatly save time without sacrificing clustering accuracy. Keywords: Short text clustering, Keyword expansion, Text representation.

1 Introduction Text clustering plays an important role in text-related task and application area such as text mining, web information retrieval and search engine. However, short text clustering poses new challenges. Short texts commonly contain no more than fifty words. Because of the short length, the texts do not provide enough contexts for pattern matching. Conventional bag of words representation used for clustering methods is often unsatisfactory. In order to overcome this difficulty, researchers have recently proposed several new methods for improving the accuracy of short text clustering focused on enriching short text representation. Banerjee Somnath, Ramanathan Krishnan[1] proposed a method of improving the accuracy of clustering short texts by enriching their representation with additional features from Wikipedia. In their method, the conventional bag of words representation of text items is augmented with the titles of select Wikipedia articles. Hotho A., Staab S., Stumme.G[2] compiled background knowledge into the text document representation. They used concepts from WordNet[3] to expand text items in the conventional bag of words representation. In [1] and [2], both of them employed the term expansion method. However, expanding every term with Jun Wang . Yiming Zhou . Lin Li . Biyun Hu . Xia Hu School of Computer Science and Engineering, Beihang University, Beijing 100191, China [email protected] *

H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 291–298. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

292

J. Wang et al.

additional feature has two weak points. One is that the method will be rather timeconsuming. As the number of terms in bag of words representation is usually very high, the number of terms will become huge after expansion. Secondly, it may add noise to the representation and may incur a loss of information. In this paper, we improve previous work by enriching short text representation with keyword expansion. Our method is based on an initial text representation as a bag of words. First, we extract keywords from the short text corpus based on TF/IDF score. Secondly, after getting the keywords list, we compiled the background knowledge into the keywords list. The background knowledge is obtained from WordNet. WordNet assigns words of English language to sets of synonyms called ‘synsets’. We consider the ‘synsets’ as the background knowledge of a word. Thirdly, we use the keywords list with background knowledge to expand the initial text representation. The clustering is then performed with K-means. We presented an experimental comparison with previous work. The experiment was made on two short text corpus which both came with a set of categorizing labels attached to the documents, the Reuters-21578 Corpus[4], and OHSUMED test collection[5], a clinically-oriented MEDLINE subset. Results indicate that with our proposed method, the efficiency of the short text clustering can be greatly improved without sacrificing clustering accuracy.

2 Enriching Text Representation with Keyword Expansion Our method first extracts keywords as candidate words to be extended, then expands original short texts with keywords background knowledge.

2.1 Keyword Expansion The first step is to extract keywords from short texts. A popular algorithm for indexing is the TF/IDF measure, which extracts keywords that appear frequently in a document, but not appear frequently in the remainder of the corpus[6]. In this paper, we extract keywords from the short text corpus based on. The principal idea is that for each short text, we select the top x percent terms based on their TF/IDF score. A keywords list consisted of the top x percent terms. The parameter x can be set depended on different applications. In this paper, we select the top 45% words from each short text, and then choose noun, verb, adjective and adverb from the top 45% words as candidate words to be expanded. After obtaining the keywords list, we use background knowledge to expand these keywords. The background knowledge is obtained from WordNet. WordNet is semantically-oriented dictionary of English, similar to a traditional thesaurus but with a richer structure. WordNet assigns words of English language to sets of synonyms called ‘synsets’. We consider the ‘synsets’ as the background knowledge of the term in keywords list.

Improving Short Text Clustering Performance with Keyword Expansion

293

2.2 Compiling Background Knowledge into Text Representation When compiling background knowledge into text representation, we first construct a keywords list for each short text. Then, for every term in the list, we use term’s ‘synsets’ from WordNet to replace the original one. For example, consider the following short text segments: 1. Benz is credited with the invention of the motorcar. First, we remove the stop words and do stemming, and represent the short text segments (1) as bag of words, like: 2. BOW={‘Benz’, ,’credit’, ’invention’ ,’motorcar’} Suppose motorcar is a keyword of this text, then we select motorcar into keyword list. Next, we find the ‘synsets’ of motorcar in WordNet. For instance, the ‘synsets’ of motorcar is: 3. Syn(‘motorcar’) = [‘car’, ‘auto’, ‘machine’, ‘automobile’, ‘motorcar’]. Then the original short text is expanded with Syn(‘motorcar’), like: 4.Extended_BOW={‘Benz’, ’credit’, ’invention’, ’motorcar’, ‘car’, ‘auto’, ‘machine’, ‘automobile’} After we have processed all the short texts in the corpus, TF/IDF score of each short text should be calculated to obtain the representation of the corpus. Then, clustering algorithms can be employed based on this representation.

2.3 Algorithm Description This section provides the precise algorithm description. The enriched short text representation with keyword expansion is derived as follows: Stop words removal and stemming Let Dn be the set of N short texts, for each short text d in Dn, apply stop words removal and stemming. Extract Keywords For each short text di in Dn, construct the term vector vi, where each element is a term tj and it’s weighting score wij. The weighting score is defined as follows:

wi , j = tf ij × log(

N ) df j

(1)

Where tfij is the term frequency of tj in di, N is the total number of documents in Dn and dfj is the document frequency of term tj. According to the weighting score, we select top x percent terms from the term vector. Here we set x to be 45%. Then we select noun, verb, adjective and adverb from the top 45% words to construct a keywords list keywords-di. Enrich original short text with keyword expansion For each short text di in Dn, and each term tj in keywords list keywords-di, let Syn_tj be tj’s ‘synsets’ in WordNet, and use Syn_tj to replace tj in di .

294

J. Wang et al.

Weighting the extended short text document Let EX_Dn be the set of N extended short texts, for each ex_di in EX_Dn, construct the term vector ex_vi, where each element is the weighting score wij of term tj, defined as follows:

wi , j = tf ij × log(

N ) df j

(2)

Where tfij is the term frequency of tj in ex_di, N is the total number of documents in EX_Dn for calculating document frequencies, and dfj is the document frequency of term tj.

3 Experiment and Discussion 3.1 Short Text Corpus Information Our experiment data comes from two corpus, Reuters-21578 Corpus and OHSUMED test collection. Some texts from the two corpuses are multiple categories and some contains more than 50 words. To perform our method, we select those texts which belong to only one category and contain no more than 50 words to construct a subset from Reuters-21578 Corpus and OHSUMED test collection respectively. We refer these two subsets as Reuters-subset and OHSUMEDsubset. Table 1. Short text corpora information Corpus

Category 'earn' 'money-fx' ‘money-supply’ Reuters-subset ‘trade’ ‘acq’ ‘interest’ ‘crude’ ‘Disorders of Environmental Origin’ OHSUMED-subset ‘Neoplasm’ ‘Musculoskeletal Diseases’

articles Total articles 1086 63 61 1715 27 332 93 53 15 57 30 12

3.2 Experimental Methodology and Evaluation Metrics Baseline It has been observed previously that additional features from WordNet can improve clustering accuracy. For comparison, we let the method used in [1] as baseline and refer it as all-word-expansion based representation method and call our method as keyword-expansion based representation method.

Improving Short Text Clustering Performance with Keyword Expansion

295

Clustering Solution Evaluation Metrics We focus our experiment on clustering accuracy and efficiency with different text representation between all-word-expansion and keyword-expansion. We have clustered the above two corpuses using K-means and compared the pre-categorization with our clustering results using three different metrics. The first metric is the widely used F-measure which combines recall and precision into global measures of utility. The second measure is the Entropy that looks how the various classes of texts are distributed within each cluster, and the third measure is the Purity that measures the extend to which each cluster contained texts from primarily one class[7,8].The derivation of the formula of F-measure, Purity and Entropy is beyond the scope of this paper, more details can be found in [7,8]. In general, lower Entropy value indicates better clustering solution and higher value of Purity and F-measure means the clustering results is more convinced. Efficiency Evaluation Metrics We employed the program executing time to evaluate the method’s efficiency. Our program included four parts: Extract-Words, Enrich-Text, TF/IDF-Weighting and Kmeans-Clustering. In the Extract-Words process, the program extracts words from text to construct a keywords list from which the word will be expanded with background knowledge from WordNet. In All-words-expansion representation method, this process is just to remove stop words from short texts and all the rest will be selected into the keywords list. The Enrich-Text process is to use the background knowledge to replace the words in short text which also appeared in the keywords with background knowledge list. The TF/IDF-Weighting process is to represent the enriched short texts with TF/IDF score.

3.3 Results and Discussion The performance on each of the representation method was explored by averaging its performance twenty times. The program executing time is measure by CPU time(The CPU time is measured in seconds on a Intel(R) Core(TM)2 CPU 1.73GHz running Python). The quality of the clustering solution was evaluated by F-measure, Purity and Entropy. Experiment Results Fig.1 showed when the OHSUMED-subset was represented by two methods, the executing time of Extract-Words, Enrich-Text, and TF/IDF-Weighting and Kmeans-Clustering process. Fig. 2 showed when the Reuters-subset was represented by the two methods, the executing time used by Extract-Words, Enrich-Text, TF/IDF-Weighting and Kmeans-Clustering process. Table 2 recorded when OHSUMED-subset and Reuters-subset were represented by the two methods respectively, the total executing time used by the entire program. Table 3 showed the quality of clustering solutions measured by F-measure, Purity and Entropy of the two methods respectively.

296

J. Wang et al.

Fig. 1 Executing time with OHSUMED-subset represented by All-words-expansion representation method and keywordexpansion representation method

OHSUMED-subset 1.8

CPU Time(seconds)

1.6 1.4 1.2 1 0.8 0.6 0.4 0.2 0 All-words Extract Words T F/IDF Weighting

Enrich Short T ext Kmeans Clusteirng

Reuters-subset

350 300 CPU Time(seconds)

Fig. 2 Executing time with Reuters-subset represented by All-words-expansion representation method and Key-wordsexpansion representation method

Key-words

250 200 150 100 50 0 All-words

Key-words

Extract Words

Enrich Short Text

TF/IDF Weighting

Kmeans Clusteirng

Discussion For convenience, we refer the All-words-expansion based representation method as the first one, and our Key-words-representation based method as the second. From Fig.1 and Fig.2, we can see that when the corpus was represented by the first method, the executing time of the Extract-Words process is less than the executing time when the corpora was represented by the second method. The reason is that the second method had to extract keywords from the corpus while the first method didn’t. However, when executing the Enrich-Text, TF/IDF-Weighting and K-means-Clustering process, the time used by the first method was much longer than the second method.

Improving Short Text Clustering Performance with Keyword Expansion

297

Table 2 Executing time Corpus Expansion:

OHSUMED -subset Reuters -subset All words Keywords All words Keywords

Total CPU Time 3.314539744 2.6565999 438.2793 263.36194 Table 3 Clustering results evaluation Corpus Representation method F-measure Purity

Reuters-subset All words expansion 0.466704585096 0.761516034985

Keywords expansion 0.473993525225 0.781535471331

OHSUMED-subset All words expansion 0.502852281363 0.56140350877192979

Entropy

0.390147579326

0.389705295309 0.858800812261

Keywords expansion 0.492494239648 0.559122807018 0.86573901042695078

Table 3 showed the quality of clustering solutions measured by F-measure, Purity and Entropy for the two methods respectively. From this figure, we see that the quality of the clustering is almost the same. From the experiment results given by fig.1, fig2, table 2 and table 3, we can see that our method saves time while obtaining almost the same clustering accuracy.

4 Conclusions In this paper, we studied the short text clustering problem. We improve previous work on short text clustering by enriching text representation. Compared to previous work, most of which are based on all words expansion, our proposed method, key words expansion method, can greatly improve the clustering efficiency while without sacrificing clustering accuracy. Although we use the ‘synsets’ from WordNet as the background knowledge of our expanded representation in this paper, an interesting direction of future work would be to use a variety of other sources of external text, such as web search results, query reformulation logs, and Wikipedia, etc. Acknowledgement. The work is supported by the National Basic Research Program of China under Grant No.2007CB310803. We are also grateful to anonymous reviewers for their valuable comments.

References 1. Banerjee, S., Ramanathan, K., Gupta, A.: Clustering Short Texts Using Wikipedia. In: 30th Annual International ACM SIGIR Conference on Research and Development In Information Retrieval, pp. 787–788. ACM Press, New York (2007) 2. Hotho, A., Staab, S., Stumme, G.: Ontologies Improve Text Document Clustering. In: Third IEEE International Conference on Data Mining. IEEE Computer Society Press, Florida (2003)

298

J. Wang et al.

3. Fellbaum, C.: An Electronic Lexical Database (Language, Speech, and Communication). MIT Press, Cambridge (1998) 4. The Reuters-21578 benchmark corpus, http://nltk.googlecode.com/svn/trunk/nltk_data/index.xml 5. Hersh, W., Buckley, C., Leone, T.J.: OHSUMED: an Interactive Retrieval Evaluation and New Large Test Collection for Research. In: 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 192– 201. Springer, New York (1994) 6. Li, J.Z., Fan, Q., Kuo, Z.: Keyword Extraction Based on tf/idf for Chinese News Document. Wuhan University Journal of Natural Sciences (2007) 7. Zhao, Y., Karypis, G.: Empirical and Theoretical Comparisons of Selected Criterion Functions for Document Clustering. Machine Learning 55(3), 311–331 (2004) 8. George, H., Adam, S.: Agreement, the F-Measure, and Reliability in Information Retrieval. J. Am. Med. Inform. Assoc. 12, 296–298 (2005) 9. Diego, I., David, P., Paolo, R.: Evaluation of Internal Validity Measures in Short-Text Corpor. In: Gelbukh, A. (ed.) CICLing 2008. LNCS, vol. 4919, pp. 555–567. Springer, Heidelberg (2008) 10. Resnik, P.: Using Information Content to Evaluate Semantic Similarity in Taxonomy. In: Proceedings of the 14th International Joint Conference on Artificial Intelligence, Canada, pp. 448–453 (1995)

Nonnative Speech Recognition Based on Bilingual Model Modification at State Level Qingqing Zhang, Jielin Pan, Shui-duen Chan, and Yonghong Yan

Abstract. This paper presents a novel bilingual model modification approach to improve nonnative speech recognition accuracy when the variations of accented pronunciations occur. Each state of baseline nonnative acoustic model is modified with several candidate states from the auxiliary acoustic model, which is trained on speakers’ mother language. State mapping criterion and n-best candidates are investigated, and different numbers of Gaussian mixtures of the auxiliary acoustic model are compared based on a grammarconstrained speech recognition system. Using this bilingual model modification approach, compared to the nonnative acoustic model which has already been well trained by adaptation technique MAP, the Phrase Error Rate further achieves a 5.83% relative reduction, while only a small relative increase on Real Time Factor occurs.

1 Introduction With globalization, nonnative speech recognition is becoming a popular issue in automatic speech recognition (ASR). As apposed to native one, current speech recognition is known to perform considerably worse when recognizing nonnative speech, which results in word error rates of two to three times native error rates [1]. Qingqing Zhang · Jielin Pan · Yonghong Yan ThinkIT Speech Laboratory Institute of Acoustics Chinese Academy of Sciences Beijing 100190, China Shui-duen Chan Department of Chinese and Bilingual Studies The Hong Kong Polytechnic University [email protected] H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 299–309. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

300

Q. Zhang et al.

The difficulty in recognizing nonnative speech results mainly from the mismatch between acoustic models automatically determined from training data and characteristics of test data. Usually, acoustic models are trained on native speech, which represent characteristics of native pronunciation. Nonnative speakers’ pronunciations of the test speech, however, different from those native speakers’ observed during training, dramatically decreases the recognition performance [2]. It is shown that when the acoustics of the accents are taken into account, large gains in performance can be achieved [3, 4]. This indicates that if acoustic models are trained with the nonnative speech covering the pronunciation variation, the performance of nonnative speech recognition can be improved. In fact, the variation among nonnative speakers, even with the same accent, is potentially very large [5]. The characteristics of the nonnative speakers’ pronunciation may differ in fluencies, levels of familiarity with the target language, individual tendencies in mapping unfamiliar sounds to their own native language sounds, etc. In order to build robust nonnative acoustic models, much more training data with different kinds of variations are required. However, to obtain such training data is very difficult, especially for those with strong accents, since most of the nonnative training data are recorded by speakers who are relatively familiar with the second language. In recent years, speaker adaptation techniques such as MAP and MLLR have been widely used to handle nonnative speech [6], by which native acoustic models can be adapted to nonnative ones with limited amounts of adaptation speech. In these methods the similarity between adaptation speech and test speech is the key point which determines the performance. Although these techniques gain in nonnative speech recognition accuracy, however, the typical nonnative accuracy following adaptation still falls substantially below that of native speech. [7] studies the limitations of adaptation. Results suggest that the target language phones that do not exist in the speakers’ own language as well as the large acoustic variability reduces the performance heavily. How to make acoustic models robust and tolerant with these pronunciation variations is the motivation of our research presented in this paper. [8] argues that nonnative speakers may produce speech sounds which are either part of their own native language or which are established via merging characteristics of a native sound with a nonnative speech sound. Thus it can be speculated that training data from the speakers’ mother language will be useful for nonnative speech recognition, especially when a heavy accent occurs (where speakers tend to use the phonemes of their own native language to substitute ones of the target language), since the training data represents pronunciation characteristics of the speakers’ own native language. Inspired by this, we examine the problem of Mandarin-accented English speech recognition when only a limited amount of nonnative training data is available. To make the baseline acoustic model better adapted to the characteristics of Mandarin-accent, the baseline nonnative English acoustic model is modified with an auxiliary Mandarin acoustic model at the state level. The

Nonnative Speech Recognition Based on Bilingual Model Modification

301

similarity between states from the nonnative English acoustic model and Mandarin acoustic model is investigated, and different numbers of state candidates are compared based on the designed testing database. Because computation cost is an important factor for real world applications, different numbers of Gaussian mixtures of the auxiliary acoustic model are also compared in our research to investigate their impacts on computation cost and recognition accuracy. The paper is structured as follows: Training and testing databases are presented in section 2. In section 3, we describe the baseline acoustic models of our experiments and in section 4 we document how state-level bilingual model modification can help improve the baseline nonnative recognizer performance efficiently. Section 5 gives a brief conclusion of this paper.

2 Database Description Our study is restricted to nonnative English spoken by native speakers of Mandarin. All the training and testing data described were recorded and digitized at 8 KHz sampling rate with 16-bit resolutions. The speech feature vector used throughout this paper consists of 36 components (12 PLP parameters, and their first and second order time derivatives), which is analyzed at a 10msec frame rate with a 25msec window size. Cepstral Mean Subtraction (CMS) is employed. Our training data are divided into three categories: native Mandarin database, native English database and Mandarin-accented English database. The native Mandarin training database consists of native Chinese speech database of National 863 Hi-Tech Project (DB863) [12]. It is a standard database published by governmental research program 863 for read speech in Mandarin. The English training database is Wall Street Journal (WSJ), etc. The Mandarin-accented English database was collected in our lab. It includes English read speech with text from everyday conversations, spoken by hundreds of Mandarin-accented speakers. Table 1 summarizes the main information about these three databases. In order to study different variations of nonnative pronunciations, a representative testing database is designed in the experiment. This testing database contains two types of phrases that are composed of words with different degrees of familiarity to speakers. Some of these phrases have the Table 1 Summary of three training databases Training Corpus Type TrainM Native Mandarin TrainE Native English Mandarin-accented TrainA English

Time Source (hours) DB863 500 WSJ 232 Lab

20

302

Q. Zhang et al.

most common and simple words in English such as ”baby”, ”hello”, so the speakers can pronounce them more easily; the other include some words that are rarely used in general conversations, such as names of English people: ”Fitzgerald”,” orchestra”, which are hard for Mandarin speakers, and in this situation speakers tend to pronounce them with heavier accents. This database consists of 1568 utterances in total, which were recorded by four Chinese female and six Chinese male. All the experiments use a grammar-constrained speech recognition system. The grammar consists of all of the phrases in the test set, which has about 400 different phrases in total. The English phone set used in the experiments is supplied by the ARPABET and the dictionary is based on the CMU pronunciation dictionary [9]. This dictionary consists of approximately 53,000 words with associated phonetic transcriptions. Performances measured in this paper are at the phrase level.

3 Baseline Models In this section, one native English acoustic model and different nonnative English acoustic models are investigated and compared. The acoustic model that has the best performance will be selected to be the baseline and modified by the state-level bilingual model modification approach in order to achieve further higher nonnative speech recognition accuracy. Additionally, a native Mandarin acoustic model is established in this section too. This model will be used as an auxiliary acoustic model in the model modification approach.

3.1 Native Acoustic Model For comparison, the native English acoustic model is established first. The native English acoustic model is trained on TrainE database only, which is labeled as Model Nat. This acoustic model contains 6000 tied HMM states with 32 Gaussian mixture components per state. Table 2 gives the Phrase Error Rate (PhrER) of the native English acoustic model. Additionally, we use the Mandarin training data TrainM to build native Mandarin acoustic model Model Mand, which will be used in the model modification approach as the auxiliary acoustic model. This acoustic model contains about 6000 tied HMM states with 32 Gaussian mixture components per state. The phone set of the model is supplied by 49 toneless Mandarin phones based on the phonetic inventory of the International Phonetic Association (IPA) [10].

3.2 Nonnative Acoustic Models In order to improve nonnative speech recognition accuracy, adding nonnative speech data into training may be the simplest way to reduce the mismatch

Nonnative Speech Recognition Based on Bilingual Model Modification

303

Table 2 Performances of the native and nonnative acoustic models on the testing database Acoustic Models PhrER (%) Model Nat 46.9 Model ReT 39.3 Model MLLR 37.2 Model MAP 34.3

between training/testing data. We explored different acoustic modeling methods to add the nonnative speech data TrainA into training for fitting the characteristics of Mandarin accent. Table 2 shows performances of these nonnative acoustic models on the testing data and gives comparison with native one. Model ReT (short for ReTrained acoustic Model) is the acoustic model retrained by pooling non-native data TrainA and native data TrainE together. Model MLLR (short for MLLR adaptive acoustic Model) refers to acoustic model to apply adaptation technique MLLR on native acoustic model Model Nat with TrainA. Model MAP (short for MAP adaptive acoustic Model) is the acoustic model using adaptation technique MAP on native acoustic model Model Nat with TrainA. As expected, all of these nonnative models perform better than do the native one on the nonnative testing data. Compared to Model ReT and Model MLLR, Model MAP reaches the lowest PhrER. Therefore, Model MAP is selected to be the baseline nonnative acoustic model, and all of the improvements with state-level bilingual model modification approach below will be based on this adaptive nonnative acoustic model.

4 State-Level Model Modification Adaptation technique is shown to be effective for nonnative speech recognition. This technique depends on the consistency between the types of adaptive data and test data. In practical use, however, the nonnative speech recognizer may encounter speakers who are just beginning to learn the foreign language or who have heavy accents. In these cases, nonnative speakers tend to substitute sounds of their mother language for those foreign sounds they can not produce [11]. Usually, these pronunciation substitutions are difficult to predict and hard to capture in the training data. Due to lack of adaptive data, the gain achieved by adaptation technique will be limited. In our application, nonnative speakers may produce English sounds which are more likely as parts of Mandarin sounds. It can be speculated that modifying nonnative English acoustic models with Mandarin ones may be useful to capture the pronunciation substitutions. Since Mandarin have different phone sets from English, in order to implement the modification, a statelevel mapping criterion to define which English model state combines with

304

Q. Zhang et al.

which Mandarin model states is required. In the following section, the statelevel mapping algorithm is presented first, and then different n-best state candidates are investigated and compared based on the nonnative test set. Lastly, different numbers of Gaussian mixtures for the auxiliary acoustic models used in modifying the nonnative one are also compared to measure computational cost.

4.1 State-Level Mapping Algorithm The state-level mapping algorithm used in this paper is an extension of a phone clustering algorithm based on confusion matrix [2]. In the state-level mapping, each baseline acoustic model state is modified with its corresponding auxiliary model states by combining the Gaussian mixture into a new mixture. In our application, the nonnative English acoustic model Model MAP is chosen to be the baseline acoustic model and the Mandarin acoustic model is used as the auxiliary acoustic model. The detailed algorithm can be described as below: (For convenience, Mandarin and English are referred as source and target language respectively.) 1. Target reference: Force align target language states in small amounts of target language speech data using target language acoustic model in order to get the time-label information. These are considered as the target states reference. 2. Source hypothesis: The source language state recognizer is applied using source language acoustic model on these speech data to decode the source phonetic representation of each utterance. This yields parallel phonetic segmentations of the target language speech data in the source language state inventories. The source phonetic representation is considered as the source states hypothesis. 3. Co-occurrence criterion: Define a criterion for co-occurrence between two phonetic labelings of the reference and hypothesis. In our system, when the number of overlapping frames between the reference and hypothesis states is more than sixty percent of the reference state duration, we can arrange the state of the source language into a target language matrix that contains the counts of co-occurrences between the ith and j th states of the source and target languages. This language matrix of cooccurrences is the confusion matrix. Figure 1 shows an example of the co-occurrence between state ” tj en” and state ” si ch” when English is taken as the target language. 4. Confusion probability calculation: Let M, N be the numbers of states in source and target language. Let AS,T (M, N) be the confusion matrix and Ai,j be the ith row and j th column element of this matrix. Given the target language state tj and the source language state si , the confusion probability can be computed as:

Nonnative Speech Recognition Based on Bilingual Model Modification

305

Fig. 1 Example of the co-occurrence between state ” tj en” and state ” si ch” when English is taken as the target language. (Note: the Mandarin states and English states are labeled by tag ” ch” and ” en” respectively)

Ai,j =

count(si |tj ) M  count(sn |tj )

(1)

n=1

where Ai,j ∈ AS,T (M, N),i = 1...M, j = 1...N. 5. State mapping information: After the confusion probability calculation Ai,j is obtained, the state mapping information can be derived from this matrix. Given the ith row and j th column element of Ai,j has the maximal value in the j th column, the ith state from source language is the best matching state to the j th state of the target language. If k th row has the second maximal value in the j th column, the k th state from source language is the 2nd-best matching state to the j th target language state, and so on. Based on this rule, every target language state can find n-best (n<M) matching states from the source language. 6. State-candidate model modification: When the state mapping information and the number of n-best candidates are determined, the new output probability density pj (ot ) of target language state j th for the observation vector ot is modified by the n-best state candidates from the source language as follow: pj (ot ) = αpj (ot )tar + (1 − α)

n l=1

where n l=1

alj = 1

alj pl (ot )sou

(2)

306

Q. Zhang et al.

Table 3 Performance of the nonnative acoustic model with state-level model modification approach on the testing database Acoustic Models PhrER (%) Model MAP 34.3 Model MM C1 31.8

Here α denotes the weight for state from baseline acoustic model, alj is the confusion probability of corresponding lth -best state candidate and n denotes the number of n-best candidates. pj (ot )tar and pl (ot )sou refer to output probability densities of original target language state j th and source language state lth respectively. Table 3 shows the performance of the nonnative acoustic model with statelevel model modification approach on the testing database. Model MM C1 refers to the acoustic model modified by the state-level bilingual model modification approach when Model MAP is regarded as the baseline nonnative English acoustic model and Model Mand is regarded as the auxiliary acoustic model, respectively. ” C1”, short for ”1-best Candidate”, means that each state of baseline acoustic model is modified by just using the best matching state from Mandarin model states via combining the Gaussian mixture into a new mixture. As shown, Model MM C1 performs remarkably better than the baseline nonnative English acoustic model, achieving a 7.3% relative PhrER reduction.

4.2 N-Best Candidate Selection Given the state mapping criterion, another aspect concerning model optimality that we have investigated is the number of n-best candidates selected to modify the baseline nonnative acoustic model. Appropriate number of candidates can improve the recognition accuracy further. In our study, different candidate numbers are investigated and compared. Experiment shows that 2-best candidate is the most appropriate choice. Table 4 presents the experimental results on the testing database when selecting different numbers of n-best candidates. Tags” C1”,” C2” and” C3” refer to three numbers of n-best candidates. As can be seen, Model MM C2 Table 4 Performances of state-level model modification approach with different n-best candidates on the testing database Acoustic Models PhrER (%) Model MAP 34.3 Model MM C1 31.8 Model MM C2 31.6 Model MM C3 32.0

Nonnative Speech Recognition Based on Bilingual Model Modification

307

outperforms Model MM C1 and Model MM C3, achieving a 7.9% relative PhrER reduction compared to the baseline nonnative acoustic model. Thus, 2-best state candidate is regarded as the preferable number of state candidates to be selected in the state-level model modification approach.

4.3 Number of Gaussian Mixture Components It can be seen that state-level bilingual model modification approach improves nonnative speech recognition accuracy significantly. However, adding the auxiliary acoustic model into recognition processing will cost more time to decode. In speech recognition systems, the acoustic computation cost often occupies most of the decoding time. Thus, controlling the number of Gaussian mixture components in the acoustic models is needed before the system can be put into practical use. To measure computation, Real Time Factor (RTF) is used as the criterion in our experiments. If it takes time P to process an input of duration T, the RTF is defined as: P (3) T Table 5 shows results on RTF and PhrER for the model modification approach with different numbers of Gaussian mixtures of auxiliary acoustic models. Tags ” 4GMM”, ” 8GMM” and ” 32GMM” refer to three different numbers of Gaussian mixtures for the auxiliary Mandarin acoustic model. With the number of Gaussian mixtures increasing, the RTF increases remarkably. Even though Model MM 32GMM C2 achieves the best performance on PhrER, it costs much more time to decoder speech, which has a 152% relative increase on RTF compared to the baseline. That is too slow to be used in practice. As we can see in Table 5, when the number of Gaussian mixtures is reduced to 4, the computation cost decreases significantly, and at the same time over 5% relative reduction on PhrER can still be achieved. In fact, the auxiliary Mandarin acoustic model, which represents the characteristics of Mandarin pronunciations, is used to modify the nonnative acoustic model. It does not need to be too detailed. As shown in Table 5, 4 Gaussian mixture components for the auxiliary acoustic model is enough to capture most of the characteristics of Mandarin pronunciations. RT F =

Table 5 The comparison results of model modification approach with different numbers of Gaussian mixture components in the auxiliary acoustic models Acoustic Models Model MAP Model MM 4GMM C2 Model MM 8GMM C2 Model MM 32GMM C2

RTF PhrER (%) 1.05 34.3 1.32 32.3 1.53 32.7 2.65 31.6

308

Q. Zhang et al.

Therefore, considering the computation cost and recognition accuracy, the nonnative acoustic model whose states are modified with corresponding 2best state candidates from the auxiliary Mandarin model with 4 Gaussian mixture components achieves the best performances.

5 Conclusion This paper investigates how to improve nonnative speech recognition accuracy when large variations of accented pronunciations occur. In order to capture the characteristics of accents from speakers’ own native language, a novel state-level bilingual model modification approach is proposed and presented. Each state of the baseline nonnative acoustic model is modified by 2-best candidate states from the auxiliary acoustic model with 4 Gaussian mixture components, which is trained on speakers’ mother language. In our experiments, significant improvement is achieved on recognition accuracy while only a small relative increase on computation cost occurs. This approach is proven to be effective and efficient for the development of nonnative speech recognition systems in which training data are limited to cover large variations of accented pronunciations. Acknowledgements. This work is partially supported by MOST (973program,2004CB318106), National Natural Science Foundation of China (10574140, 60535030), The National High Technology Research and Development Program of China (863 program, 2006AA010102,2006AA01Z195).

References 1. Tomokiyo, L.M., Waibel, A.: Adaptation Methods for Nonnative Speech. In: Proceedings of Multilinguality in Spoken Language Processing (2001) 2. Zhang, Q., Pan, J., Yan, Y.: Mandarin-English Bilingual Speech Recognition for Real World Music Retrieval. In: ICASSP 2008, paper 1147, Las Vegas, March 30 - April 4 (2008) 3. Humphries, J., Woodland, P., Pearce, D.: Using accent-specific pronunciation modeling for robust speech recognition. In: Proc. ICSLP 1996, Philadelphia, PA, October 1996, pp. 2324–2327 (1996) 4. Teixeira, C., Trancoso, C., Serralheiro, A.: Recognition of Non-native Accents. In: Proc. Eurospeech 1997, Rhodes, Greece, September 1997, pp. 2375–2378 (1997) 5. Livescu, K.: Analysis and Modeling of Non-native Speech for Automatic Speech Recognition. Master’s thesis, MIT (August 1999) 6. Wang, Z., Schultz, T., Waibel, A.: Comparison of Acoustic Model Adaptation Techniques on Non-native Speech. In: Proc. ICASSP (2003) 7. Clarke, Constance, Jurafsky, Daniel: Limitations of MLLR Adaptation with Spanish-accented English: an Error Analysis. In: INTERSPEECH 2006, paper 1611-Tue2BuP.7 (2006)

Nonnative Speech Recognition Based on Bilingual Model Modification

309

8. Bohn, O.-S., Flege, J.E.: The production of New and Similar Vowels by Adult German Learners of English. Stud. Second Lang. Acquis. 14, 131–158 (1992) 9. The CMU Pronouncing Dictionary v0.6, The Carnegie Mellon University, http://www.speech.cs.cmu.edu/cgi-bin/cmudict 10. IPA. The International Phonetic Association (revised to 1993) IPA Chart. Journal of the International Phonetic Association 23 (1993) 11. Flege, J.E.: Production and Perception of a Novel, Second-language Phonetic Contrast. Journal of the Acoustical Society of America 93, 1589–1608 (1993) 12. Li, A., Yin, Z., Wang, T., Fang, Q., Hu, F.: RASC863 - A Chinese Speech Corpus with Four Regional Accents. In: ICSLT-o-COCOSDA, New Delhi, India (2004)

Edge Detection Based on a PCNN-Anisotropic Diffusion Synergetic Approach Mario I. Chacon-Murguia, Mitzy Nevarez-Aguilar, Angel Licon-Trillo, Oscar Mendoza-Vidaña, J. Alejandro Martinez-Ibarra, Francisco J. Solis-Martinez, and Lucina Cordoba Fierro*

Abstract. A new synergetic model based on a pulse neural network and the Perona-Malik anisotropic diffusion algorithm is introduced. The proposed model was developed because of the failure of conventional edge detectors to perform the extraction of geometric structures from low contrast edges and noisy assumed uniform regions. The synergetic model is a variation of the Perona-Malik algorithm that incorporates a gradient update during its iteration process. The gradient update is related to enhance real edges and to reduce the noise content in uniform regions. The proposed approach is compared with the Canny and watershed methods as well as with a variation of the Perona-Malik algorithm. Results indicate that the synergetic approach yields a more complete geometric structure extraction compared with the others methods.

1 Introduction Edge detection has been a challenge area in image processing. The importance of solving the edge detection problem correctly is related to the effect it has in the segmentation stage for image analysis. Different schemes have been proposed to face the edge detection problem, and many more are still being proposed because edge detection in general requires specific edge detection methods for specific situations. In fact, edge detection can be considered an open problem due to the variety of situation found in image processing applications. Although many edge detection methods have been reported in the literature [1-4], it is not common to find synergetic approaches aimed to improve the edge detection methodology performance in complex situations. One kind of complex situation for edge Mario I. Chacon-Murguia . Mitzy Nevarez-Aguilar . Oscar Mendoza-Vidaña Chihuahua Institute of Technology DSP & Vison Lab *

Angel Licon-Trillo . Francisco J. Solis-Martinez . Lucina Cordoba Fierro Facultad de Medicina, Universidad Autónoma de Chihuahua J. Alejandro Martinez-Ibarra Centro Universitario del Sur, Universidad de Guadalajara, Mexico [email protected] H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 311–321. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

312

M.I. Chacon-Murguia et al.

Fig. 1 Structure samples found on the Nose Bug eggs’ surface

detection is when the image under processing has low contrast edges and noisy assumed uniform regions. In these cases most of the edge methods present a high rate of false positives from the edge pixel point of view. An interesting approach to deal with edge detection in low contrast images is the anisotropic diffusion filters [5-8]. A well known anisotropic filter methodology is the Perona-Malik method, P-M, [5]. A modification of this filter has been reported in the literature to have good performance in the detection of edges in glass substrates images, which are low contrast images [6]. A specific situation of low contrast images with a low signal (edges) to noise ratio can be found in surface images of the Meccus Phyllosomus eggs. The Meccus Phyllosomus or Nose Bug is an insect that can infect humans with a parasite that may originate cardiovascular problems termed Chagas disease. Due to this situation the Nose Bug has been a topic of many studies for many years [9,10]. One of the studies related to the Nose Bug concerns the taxonomy of the families and their possible hybrids. A possible solution proposed to determine the correct taxonomy is to relate the geometric structures found in the eggs’ surface with a corresponding family or hybrid. Figure 1 illustrates two examples of such geometric structures. The images are acquired with a scanning electron microscope. Before the geometric structures of the egg can be analyzed for classification purposes, it is necessary to extract them from the egg’s surface using image processing techniques, which is not a trivial task because of the low contrast and noisy composition of the images. The geometric structures extraction problem through image edge detection is the topic cover in this research. The proposed method to achieve this task is a synergetic approach. The method incorporates information extracted with a Pulse Coupled Neural Network, PCNN, which is incorporated into an anisotropic diffusion method. The organization of this paper includes a description of the possible solution using the Perona-Malik method as well as one proposed modification, Section 2. Section 3 presents the PCNN, methodology to obtain synergetic information. The proposed methodology combining the PCNN information with the P-M method is explained in Section 4. Finally the results and conclusions are discussed in Section 5.

Edge Detection Based on a PCNN-Anisotropic Diffusion Synergetic Approach

313

2 Anisotropic Filtering Anisotropic filtering is a method based on the solution of the heat equation or diffusion equation. Its main idea is to smooth assumed uniform regions based on a diffusion behavior, and preserve the edges of a region by stopping the diffusion process. The method developed by Perona–Malik [5], is described in the next subsection.

2.1 Perona Malik Approach The Perona-Malik model, P-M, is derived from the continuous anisotropic diffusion defined by

∂I t ( x, y ) ∂t

= div ⎡⎣ct ( x, y ) ∇I t ( x, y ) ⎤⎦

(1)

From this continuous model a discrete anisotropic diffusion model is derived and is represented by 1 4 I t +1 ( x, y ) = I t ( x, y ) + ∑ ⎡⎣ ctj ( x, y ) ∇I ti ( x, y ) ⎤⎦ (2) 4 i =1 where I t ( x, y ) is the current image under processing at time t. ctj ( x, y ) is the diffusion coefficient at time t with direction j={North, South, East, West}, and ∇I it ( x, y ) represents the gradient at the coordinates (x,y) in the directions j, defined by

∇I tN ( x, y ) = I t ( x − 1, y ) − I t ( x, y )

(3)

∇I tS ( x, y ) = I t ( x + 1, y ) − I t ( x, y )

(4)

∇I tE ( x, y ) = I t ( x, y + 1) − I t ( x, y )

(5)

∇IWt ( x, y ) = It ( x, y ) − I t ( x, y − 1)

(6)

The diffusion coefficient ctj ( x, y ) can be defined as a function of ∇I it ( x, y ) ctj ( x, y ) = g ( ∇I it ( x, y ) )

(7)

where g ( ∇I ( x, y ) ) is a non-negative monotonically decreasing function, such i t

that

⎧1 g ( 0) ⎪ g ( ∇I it ( x, y ) ) = ⎨0 lim g ( ∇I i ( x, y ) ) t ⎪ ∇Iit ( x , y ) →∞ ⎩

(8)

which warranties that edge regions, high gradient values, will be preserved, and uniform regions will be diffused, smoothed.

314

M.I. Chacon-Murguia et al.

A possible g() function in the P-M model is g ( ∇I it ( x, y ) ) =

1 ⎛ ∇I ( x , y ) 1+ ⎜ ⎜ k ⎝ i t

⎞ ⎟ ⎟ ⎠

2

(9)

where k is a parameter that controls the diffusion process, such that, oversmoothing ⎧ Diffusion process = ⎨ diffsuion stops at early stages ⎩

if k is too large if k is too small

2.2 Perona – Malik Variations and Parameter Estimation Chao and Tsai propose a modification to the P-M model, P-MM, to carry out edge sharpening besides the smoothing process [6]. This model is formulated as

I t +1 ( x, y ) = I t ( x, y ) +

1 4 ⎡ g ( ∇I ti ( x, y ) ) − v ( ∇I ti ( x, y ) ) ⎤∇I ti ( x, y ) ∑ ⎦ 4 i =1 ⎣

(10)

The sharpening function is defined as

v ( ∇I it ( x, y ) ) = α ⎡⎣1 − g ( ∇I it ( x, y ) ) ⎤⎦

(11)

where α is sharpening parameter with 0 ≤ α ≤ 1. The v() function is a nonnegative monotonically increasing function, such that ⎧0 v ( 0) ⎪ v ( ∇I it ( x, y ) ) = ⎨1 lim v ( ∇I i ( x, y ) ) t ⎪ ∇Iit ( x , y ) →∞ ⎩

(12)

The parameters for the P-M and P-MM models were adjusted to process the eggs’ images following the recommendations reported in the literature [5,6,8]. The best edge detection results were obtained with the values shown in Table 1. Table 1 P-M and P-MM parameters P-M 20 Iterations K=maximum gradient value,λ=1/4

P-MM 10 Iterations K= 70, λ=1/4,α= 0.03

2.3 Anisotropic Diffusion Results The P-M and the P-MM models were tested in the edge detection problem. The findings indicate that edge detection in these images is even problematic for these two methods. The problem arises from the fact that the image composition incorporates low contrast edges and high content of noise. Therefore, this leads us to a

Edge Detection Based on a PCNN-Anisotropic Diffusion Synergetic Approach

315

new paradigm where the main idea of diffusion is preserved but a solution to the low contrast edges and noisy uniform regions is incorporated. This new paradigm based on the PCNN image preprocessing is described in the next section.

3 PCNN Preprocessing The PCNN has been used as a preprocessor in different situations reported in the literature [11-13]. The idea of incorporating the PCNN model is to try to generate uniform regions of the images. If this is possible then these regions may be used to reinforce the edges and diffuse the other pixels using the P-M approach.

3.1 PCNN Model The PCNN architecture has three main modules: the dendrite tree, the linking and the pulse generator [11]. The linking and feeding elements form the dendrite tree. The neuron element receives information from its neighborhood through the linking. The input signal information is obtained through the feeding. The pulse generator module compares the internal activity, U(t), with a dynamic threshold to decide if the neuron element fires or not. A PCNN model can be defined by (13)-(17) F (t ) = GFeed e

−α F Δt

L ( t ) = GLink e

F ( t − 1) + S + Y ( t − 1) ∗ W

−α LΔt

L ( t − 1) + Y ( t − 1) ∗ M

U ( t ) = F ( t ) ⎡⎣1 + β L ( t ) ⎤⎦ θ (t ) = e

− 1

αθ θ ( t − 1) + VY (t )

⎧1 if U ( t )〉θ ( t ) Y ( t ) = ⎪⎨ ⎪⎩0 otherwise

(13) (14) (15) (16)

(17)

The feeding region is represented by Eq. (13), where GFeed is the feed gain, S is the input image, αFΔt is the time constant of the leakage filter of the feeding region, Y(t) is the neuron output at time t, and W is the feeding kernel. The outputs Y(t) of the PCNN can be observed as output images called pulsed images of the PCNN. Eq. (14) describes the linking activity. Here GLink is the linking gain, αLΔt is the time constant of the leakage filter of the linking region, and M is the linking kernel. Equation (15) corresponds to the internal activity of the neuron element. The internal activity depends on the linking and feeding activity. In (15) β is the linking coefficient. β defines the amount of modulation of the feeding due to

316

M.I. Chacon-Murguia et al.

the linking activity. The dynamic threshold is implemented by (16), where αθ is the time constant of the leakage filter of the threshold and V is the threshold gain. Finally the output of the neuron is defined by (17). In the case of an image processing task, each pixel is related to a neural element.

3.2 Image Segmentation The generation of the information that will be included in the P-M model is presented in this section. The main interest is to generate information that may increase the performance of the P-M method. This information must be related to data to enhance real edges and to reduce the noise content in uniform regions. In this respect, the PCNN model parameters are set to generate uniform regions, the parameter values used are indicated in Table 2. These parameters are derived from previous works [14,15]. Using these parameters the PCNN was tested and it was found that the best information is generated in pulsation 14th. Figure 2 illustrates an original image and the pulsation 14th, I14(x,y). Once the main information is generated by the PCNN, it is morphologically processed to yield the final information that is provided to the P-M algorithm. These morphologic operations are listed next. The first operation on I14(x,y) is a close operation with a structure element B corresponding to an octagon of size 3 to generate uniform areas I C ( x, y ) = I14 ( x, y ) • B

(18)

Table 2 PCNN parameter settings to generate uniform regions

β = 1.0 αLΔt = -0-1

G = 0.1 Feed α

θ

=5

αFΔt = -0.1 V =5

G

Link

= 1.0

The linking and feeding kernels are 3X3 average filters.

Fig. 2 a) Original image, b) Regions generated by the PCNN at pulsation 14th

Edge Detection Based on a PCNN-Anisotropic Diffusion Synergetic Approach

a)

b)

c)

d)

317

e) Fig. 3 a) IC(x,y), b) IE(x,y), c) IH(x,y), d) Small holes dilated and added to IE(x,y), e) ITH(x,y)

Then remove connections between uniform areas through erosion with C being a square of size 3 I E ( x , y ) = I C ( x, y ) ○ C

(19)

Now find small holes in IC(x,y) to fill them in IE(x,y), this gives image IH(x,y). Dilate the small holes found in IC(x,y) with D equal to a disk of size 2, add them to IE(x,y) , and apply thinning to obtain the image ITH(x,y). The image ITH(x,y) contains useful information related to possible edge regions and uniform regions that can be incorporated into the P-M model. The complete process is illustrated in Figure 3. ITH ( x, y ) = ⎡⎣( I H ( x, y ) ⊕ D ) + I E ( x, y ) ⎤⎦

(20)

4 Synergetic Model The PCNN–PM synergetic model consists on improving the edge information quality before it is used by the PM algorithm. This improvement is achieved by

318

M.I. Chacon-Murguia et al.

increasing the gradient values of possible edge pixels and decreasing the gradient values of non possible edge pixels using the information provided by ITH ( x, y ) .

4.1 Edge Estimation, Edge Reinforcement and Noise Reduction The information presented in ITH(x,y) is used to estimate the possible edges in the original image, such that the P-M algorithm is reinforced by better edge pixel estimation. Pixel edges information is improved by modifying the term Iit(x,y) in Eq. (2) and (10) as indicated in Eq. (21).

⎧max ∇ ( I it ( x, y ) ) if p ( x, y ) ∈ ITHE ( x, y ) ⎪ I ( x, y ) = ⎨ i ⎪⎩ min ∇ ( I t ( x, y ) ) if p ( x, y ) ∉ ITHE ( x, y ) i t

(21)

where, max ∇ ( I it ( x, y ) ) , and min ∇ ( I it ( x, y ) ) are the maximum gradient vector and the minimum gradient vector in the image under processing, and ITHE(x,y) are the set of edge pixels in ITH(x,y). This step has the effect of reducing the uncertainty on the information used by the P-M algorithm, improving the edge to noise ratio. Thus the synergetic anisotropic model can be expressed for the P-M and PMM models by I t +1 ( x, y ) = I t ( x, y ) +

1 4 ⎡⎣ctj ( x, y ) ∇I it , n ( x, y ) ⎤⎦ ∑ 4 i =1

(22)

1 4 ∑ ⎡ g ( ∇Iit ,n ( x, y ) ) − v ( ∇Iit ,n ( x, y ))⎤⎦∇Iit , n ( x, y ) (23) 4 i =1 ⎣ where n in the gradient term is the number of times equation (21) is applied in the P-M process. In this work the best result was achieved with n=2. Applying the anisotropic synergetic model yields a new algorithm that is faster in its edge convergence and edge definition performance, due to the uncertainty reduction accomplished by the PCNN information. I t +1 ( x, y ) = I t ( x, y ) +

4.2 Edge Definition Once the PCNN-PM model is applied, the final edges are defined with the Canny edge detector method. Some examples of the performance of the new model over the images in Figure 1 are shown in Figure 4 and 5 in comparison with other edge detector methods; Canny, Watershed, P-M, and P-MM. It can be observed from these figures that the Canny detector is completely mixed up with the noise presented in the internal regions. The Watershed method found acceptable edges but they are not completely defined. The P-M approaches deal better with the low contrast edges and the noise content. Among the P-M paradigm the PCNN-PM and the PCNN-PMM models yield more closed structures than the P-M and the PMM. The PCNN-PM is selected over the PCNN-PMM because the PCNN-PMM

Edge Detection Based on a PCNN-Anisotropic Diffusion Synergetic Approach

a)

b)

c)

d)

e)

f)

319

Fig. 4 a) Canny alone, b) Watershed, c) P-M normal 20 iterations K= maximum gradient, d) P-MM, e) PCNN-PM, 8 iterations, λ=1/4, n=2, f) PCNN-PMM, 8 iterations, λ=1/4, n=2

a)

b)

c)

d)

e)

f)

Fig. 5 a) Canny alone, b) Watershed, c) P-M normal 20 iterations K= maximum gradient, d) P-MM, e) PCNN-PM, 8 iterations, λ=1/4, n=2, f) PCNN-PMM, 8 iterations, λ=1/4, n=2

requires the computation of the sharpening function v(), which in this cases does not provide significant advantages due to the noisy content in the assumed uniform regions.

320

M.I. Chacon-Murguia et al.

5 Results and Conclusions 5.1 Experimental Results This work described a synergetic approach to determine the edges of geometric structures found in scanning electron microscope surface images of the Meccus Phyllosomus eggs. The challenge is to extract those structures from a low contrast edges and noisy information. Results of this research showed that the synergetic approach yield better results than ordinary edge detection operators, as well as better results than the anisotropic approach alone. Incorporation of the PCNN information speeded up the PM algorithm, and it also improved the detection of real edges and decreases the false positive cases.

5.2 Conclusions and Future Work The work showed that common edge detector operators are not able to deal satisfactorily with this problem. Instead, an anisotropic diffusion approach was followed. Albeit the anisotropic approach is a more robust method to face the problem it was necessary to incorporate information generated with a PCNN to increase the performance of edge detection. Future work will be related to remove the small isolated blobs, reduce the false positives edges, and to link discontinuous edges.

References 1. Canny, J.: A Computational Approach to Edge Detection. IEEE Trans. on Pattern Analysis and Machine Intelligence 8, 679–698 (1986) 2. Walczak, A., Puzio, L.: Adaptive Edge Detection Method for Images. OptoElectronics Review 16, 60–67 (2008) 3. Yuan, W.Q., Li, D.S.: Edge Detection Based on Directional Space. Frontiers of Electrical and Electronic Engineering in China, 135–140 (2006) 4. Wang, H., Li, H., Ye, X., Gu, W.: Training a Neural Network for Moment Based Image Edge Detection Journal of Zhejiang University 1, 398–401 (2000) 5. Perona, P., Malik, J.: Scale-Space and Edge Detection Using Anisotropic Diffusion. IEEE Trans. on Pattern Analysis and Machine Intelligence 12, 629–639 (1990) 6. Chao, S.M., Tsai, D.M.: An Anisotropic Diffusion-based Defect Detection for LowContrast Glass Substrates. Image and Vision Computing 26, 187–200 (2008) 7. Alvarez, L., Pierre, L., Morel, J.: Image Selective Smoothing and Edge Detection by Nonlinear Diffusion. SIAM J. Numeric Analysis 29, 845–866 (1992) 8. Jahne, B., HauBecker, H.: Computer Vision and Applications. Academic Press, London (2000) 9. Lent, H., Wygodzinsky, P.: Revision of the Triatominae (Hemiptera, Reduviidae), and Their Significance as Vectors of Chagas’ Disease. Bull. Amer. Museum nat. Hist. 163, 125–520 (1979)

Edge Detection Based on a PCNN-Anisotropic Diffusion Synergetic Approach

321

10. Cruz-Reyes, A., Pickering-López, J.M.: Chagas Disease in Mexico: an Analysis of Geographical Distribution During the Past 76 years - A Review. Mem Inst Oswaldo Cruz, Rio de Janeiro 101, 345–354 (2006) 11. Lindbland, T., Kinser, J.: Image Processing Using Pulse-Coupled Neural Netw. Springer, Heidelberg (1998) 12. Johnson, J.L., Padgett, M.L.: PCNN Models and Applications. IEEE Trans. on Neural Netw. 10, 480–498 (1999) 13. Gu, X., Zhang, L.: Orientation Detection and Attention Selection Based Unit-linking PCNN. In: International Conference on Neural Networks and Brain, vol. 3, pp. 1328– 1333 (2005) 14. Chacón, M., Zimmerman, A., Rivas, P.: Image Processing Applications with a PCNN. In: Liu, D., Fei, S., Hou, Z., Zhang, H., Sun, C. (eds.) ISNN 2007. LNCS, vol. 4493, pp. 884–893. Springer, Heidelberg (2007) 15. Chacón, M., Zimmerman, A.: License Plate Location Based on a Dynamic PCNN Scheme. In: Proceedings of the IEEE IJCNN (2003)

Automatic Face Recognition Systems Design and Realization Zhiming Qian, Chaoqun Huang, and Dan Xu*

Abstract. This paper provides new insights into three issues in face recognition: designs a complete face recognition system and gives the realization process; Presents a iris location method on K-means algorithm; Provides a face recognition method on Sobel and LBP. Experiments show that the system has good recognition results. Keywords: Face recognition, Adaboost algorithm, K-means clustering, Feature extraction, Feature analysis.

1 Introduction In recent years, with the development of artificial intelligence and E-commerce, face recognition becomes the most potential biometric technology and authentication means. The research of computer face recognition began in the late 1960s. The earliest face recognition system was created by Bledsoe in terms of the distance between facial feature point and rate parameter [1]. Face recognition has a progressive development in recent decades, and in the 1990s it became a hot research. Face recognition is the leading edge of a field of pattern recognition domain, but it is now still at the stage of research topics and it can not become active in the field of practical field. Although human can be easy to identify a familiar face, there are still many difficulties in automatically recognizing human faces from frontal views. For example: occlusion and corruption; varying illumination and expression; imaging perspective and distance; and so on. Although the classification criteria on face recognition may not be the same, the current study has two main directions [2]. One is the global analysis, classifying and recognizing based on whole face image. It considered the attributes of the global pattern. The other is based on the local feature information of face image, such as eye, mouth and nose, then it classifies and recognizes in terms of certain criteria. Zhiming Qian . Chaoqun Huang . Dan Xu YunNan University, China [email protected], [email protected], [email protected] *

Zhiming Qian Chuxiong Normal University, China H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 323–331. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

324

Z. Qian, C. Huang, and D. Xu

The subspace methods of face recognition such as PCA, Fisher were successfully tested in many situations [3,4]. As a result, they made large progresses in face recognition field. However, if we analyze them from the practical angle, these methods still have the following deficiencies: First of all, in pace with the new flourished face samples, there is no guarantee that there won’t be any trouble in finding the required subspace. Secondly, the subspace method cannot tackle the nonlinear problems in face recognition. From what have been discussed above, the local feature analysis are preferred in practical face recognition systems. At present, there is large number of part face recognition methods in numerous documents, such as SIFI, differential features [5,6], etc. It is a big pity that these documents only emphasize the effects in a specific face database; moreover, most of the face recognition systems are finished on the form of semi automation, which leading to no complete process of face recognition. Focus on the difficulties, the study shows a thorough automatic face recognition system, besides, it also give the concrete realization process. The system adopts popular techniques such as Adaboost and LBP [7,8], etc. Furthermore, it perfects the system so as to produce better recognition effects.

2 System Overview In the system analysis, we mainly think about the following factors: 1. Velocity: It is demanding that the system must keep higher processing speed. During the detection stage, we adopt the Adaboost algorithm to carry out the face detection. The reasons why we choose it result from two characteristics: the first one is that it enjoys the fastest detection speed. The second is that it has better detection capability; and during feature extraction stage, between the Gabor wavelet [9] and LBP which are popular, we choose the latter one. LBP is a kind of local feature extraction which can calculate fast and has a good recognition capability. 2. Effect: to ensure the system’s running speed simultaneously, we try best to improve the identifying effects. During the detection period, in order to ensure the iris’s accurate location, we apply K—means clustering to locate iris. In the recognition period, we add weight analysis in Soble and LBP scheme so as to make advancement of the identifying effect of LPB. Figure 1 shows the main framework of our face recognition system. Fig. 1 The frame map of the system

Automatic Face Recognition Systems Design and Realization

325

3 System Details According to Figure 1, there are 6 main parts in the face recognition system. Consequently, we will explain the specific progresses in each part.

3.1 Face Finder In our face recognition system, we choose Adaboost algorithm to locate face, Adaboost was first put forward by Freund and Schapire in 1995. The algorithm’s basic idea is boosting many weaker classifiers which could make a simple classification to construct a stronger classifier. Theory proves that while each weaker classifier makes a better classification than random speculation, if we have infinite weaker classifiers, the error rate of stronger classifier will tend to zero. In 2001, P.Viola presented a new face detection algorithm using boosted cascade of Haar features [7].

3.2 Iris Finder In accordance with the distribution of face organs, we could estimate the locations where the irises appear. Let the two iris points be (x1, y1) and (x2, y2), where x1=1.3w/4 y1=1.6h/5 x2=2.8w/4 y2=1.6h/5 (w and h are respectively the width and height of a face image). Next, get the designated eye-window based two iris points. The window size is w/6, as the Figure 2(a) shows. Afterwards, separate the colors of the iris and the skin with K-mean clustering method. The algorithm as follows:

，

，

，

(1) Initialize the three classes, the centers of these classes are class1=min (img) (the minimum of gray value), class2= max (img) (the maximum of gray value), class3= (class1+class2)/2, respectively. (2) Scan the window image. According the neighboring principle, classify each pixel to the near self gray value. (3) Calculate the new class center. Then, find the difference between the values of old and new centers. If there is no difference, go to the fourth step, otherwise, go to the second step. (4) Conclude the classification. After assorting, the class1 which is the dark area may contain iris. We suppose the class1 is white, while the class2 and class3 are skin areas which are black, as the Figure 2(b). Fig. 2 Eyes windows cluster analysis

(a)

(b)

326

Z. Qian, C. Huang, and D. Xu

Fig. 3 Sub-window images Fig. 4 3x3-neighborhood operator

Fig. 5 Face and eyes localization

For the white area, we will find the eye center using the following conditions to justify: (1) The heights of the eye lumps are smaller than their widths. (2) The eye lumps are symmetrical, and the corresponding two areas have no obvious difference. (3) Assuming there are two qualified lumps in one window image. The coordination of the center of the two lumps are (x3, y3) and (x4, y4) respectively. Comparing y3 and y4, select the corresponding lump of the bigger one as the center of the eye. Finally, the iris center will be found by the center of the eye. We get the designated sub-window using two eye points. The window size is 31×13 pixels, as the Figure 3 shows. Use the operator to every pixel of a sub-window by adding the 3x3-neighborhood of each pixel replacing the center pixel value. See Figure 4 for an illustration of the operator. Next, calculating the maximum of the sub-window, select the corresponding coordinate of the maximum as the iris center. The final result is showed in Figure 5.

3.3 Normalization Based on the iris location, make the two centers’ positions at an alignment level through the rotation of the image. Then zooming the images until the distance between the two centers comes to 70 pixels. Next adopt the template in figure 6(a) to make the face sample to be normalized to 150×180 pixels, like Figure 6(b).

3.4 Preprocessing We use Xiao-yang Tang’s preprocessing method [10] that eliminates most of the effects of changing illumination while still preserving the essential appearance details that are needed for recognition. It includes three main steps.

Automatic Face Recognition Systems Design and Realization

327

Fig. 6 Face image normalization

(a)

(b)

(1) Gamma correction: It has the effect of enhancing the local dynamic range of the image in dark or shadowed regions, while compressing it in bright regions and at highlights. This is a non-linear gray-level transformation that replaces graylevel I with I γ , where γ ∈ [0,1] is a user-defined parameter. (2) Difference of Gaussian (DoG) filtering: DoG filtering is a convenient way to obtain the resulting bandpass behavior. Fine spatial detail is critically important for recognition. We use inner Gaussian(σ 0 = 1)and outer Gaussian(σ 1 = 3). (3) Contrast equalization: globally rescales the image intensities to standardize a robust measure of overall contrast or intensity variation. Using the following two stage process: P

P

B

B

B

I ( x, y ) (mean(| I ( x' , y ' ) |α ))1 / α

(1)

I ( x, y ) (mean(min(τ , | I ( x' , y ' ) |)α ))1 / α

(2)

I ( x, y ) = I ( x, y ) =

B

Here, α is a strongly compressive exponent that reduces the influence of large values, τ is a threshold used to truncate large values after the first phase of normalization, and the mean is over the whole image. By default we use α=0.1 and τ=10. The result is showed in Figure 7. Fig. 7 Face image preprocessing

3.5 Sobel Edge Detection Refer to the preprocessing face image, if it belongs to the test image, we use Sobel operator to do edge detection with it. Then use the method of adapting threshold to get the binary image. As shown in Figure 8(a). Divide the edge binary image into 63 sub-blocks, each one is 21×15 pixels. As shown in Figure 8(b). Then calculate the numbers of the pixel point. Assuming there are n w white pixels in the ith B

B

328

Z. Qian, C. Huang, and D. Xu

Fig. 8 Sobel weight analysis

(a)

(b)

sub-block, the total number of the pixel is N in the ith sub-block. Then the weight of this sub-block is P= n w /N+0.5. Calculate the weight of all the sub-blocks one by one. Among them, the left and right down areas (the red part in the Figure 8(b)) are vulnerable to the interference of noise. Also, they are comparatively useless for recognition. Set the weight of each sub-block as 0.25, then a vector of 63 dimensions will be got. It represents the weight of the 63 sub-blocks. B

B

3.6 LBP Analysis The LBP operator is one of the best performing texture descriptors and it has been widely used in various applications. It has proven to be highly discriminative and its key advantages, namely its invariance to monotonic gray level changes and computational efficiency, make it suitable for demanding image analysis tasks. The LBP operator was originally designed for texture description. The operator assigns a label to every pixel of an image by thresholding the 3x3-neighborhood of each pixel with the center pixel value and considering the result as a binary number. See Figure 9 for an illustration of the basic LBP operator. To be able to deal with textures at different scales, the LBP operator was later extended to use neighborhoods of different sizes(n, r) which means n sampling points on a circle of radius of r. Another extension to the original operator is the definition of so called uniform patterns. A local binary pattern is called uniform if the binary pattern contains at most two bitwise transitions from 0 to 1 or vice versa when the bit pattern is considered circular. For example, the patterns 00000000(0 transitions), 01110000(2 transitions) and 11001111(2 transitions) are uniform where as the patterns 11001001(4 transitions) and 01010011(6 transitions) are not. Timo Ahonen have found that 90.6% of the patterns in the (8, 1) neighborhood in case of preprocessed FERET face database [8]. In our experiment, the face images are analyzed using the uniform patterns in the (8, 1) neighborhood. As shown in Figure 10. Fig. 9 Basic LBP operator

Automatic Face Recognition Systems Design and Realization

329

Fig. 10 Face image analyzed using LBP

Fig. 11 Face feature extraction

3.7 Feature Analysis and Matching After the analysis with LBP, we divide the image into 63 average sub-clock each one occupies 21×18 pixels. To each sub-block, weighted processing using the edge template we got from Figure 6, and then got a vector which contains a length of 59 through local histogram statistic. Link the vector of each sub-block together to form a reference feature of 63×59=3717, store it in the database. As shown in Figure 11. For one waiting—recognition image, using Equation (3) get the similarity values of the test image and all registration image, then choose the corresponding face image of the smallest one and treat it as the final recognition result. 63

59

i =1

n =1

S (img 1 , img 2 ) = ∑ (q i ∑ min(hn1 , hn2 ))

(3)

Where q i denotes the weight of the ith sub-block, h n denotes the nth bin in a subblock’s histogram. B

B

B

B

4 Experiment The database adopted by this experiment includes two parts: colorful Feret face images and the self-collected face images. Register face images contain 150 persons with one frontal image per person (some images are showed in Figure 12). Then make tests using other three images which contain different poses and facial expression. Compare the results with the original LBP method. The final experiment result is showed in Table 1. The face database we have collected contains 26 persons, each one 4 pictures, and the fours are different in poses, facial expressions and decorations. (Some images are showed in Figure 13). Choose the frontal image of each person and register it. Then test the other three pictures, the result is showed in Table 2.

330

Z. Qian, C. Huang, and D. Xu

Fig. 12 Some images in colorful Feret face database

Table 1 Comparison of recognition rates in colorful Feret database Method LBP Sobel+LBP

Top 1 recognition rate 91.3% 94.6%

Fig. 13 Some images in our own face database

Table 2 Comparison of recognition rates in our own face database Method LBP Sobel+LBP

Top 1 recognition rate 93.2% 97.1%

5 Conclusion This study designed a completely face recognition system and described the realization process. Providing the methods of clustering iris location and LBP which applies the Sobel weighted processing and they have proved to be effective through the experiment. Our continuing work will focus on these two aspects: 1. Keeping on perfecting the accuracy of the iris location. In our experiment, some faces can’t be recognized. This phenomenon is mainly caused by the inaccurate location of the iris, which directly leads to the distortion of the face picture after the capturing process. 2. Obviously, the scale of the face database isn’t big enough in our experiment. In the future, we need to go on experimenting on a large scale database and make a comparison of recognition rate between this way and other methods.

Automatic Face Recognition Systems Design and Realization

331

References 1. Bledsoe, W.: Man-Machine Facial Recognition. Technical Report PRI 22, Panoramic Research Inc., Palo Alto, CA (1966) 2. Hjelmas, E., Low, B.K.: Face Detection: A Survey. Computer Vision and Image Understanding 83, 236–274 (2001) 3. Turk, M., Pentland, A.: Eigenfaces for Recognition. Journal of Cognitive Neuroscience 13, 71–86 (1991) 4. Belhumeur, P., Hespanha, J., Kriegman, D.: Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection. IEEE Trans. on Pattern Analysis and Machine Intelligence 19, 711–720 (1997) 5. Bicego, M., Lagorio, A., Grosso, E., Tistarelli, M.: On the Use of SFIT Features for Face Authentication. In: Proceddings Of IEEE Int. Workshop on Biometrics, in association with CVPR, New York (2006) 6. Ravela, S., Hanson, A.: On Multi-Scale Differential Features for Face Recognition. In: Proc. Vision Interface, Ottawa, Canada, pp. 15–21 (2001) 7. Viola, P., Jones, M.: Robust Real Time Object Detection. In: 8th IEEE International Conference on Computer Vision, Vancouver (2001) 8. Timo, A., Abdenour, H., Matti, P.: Face Description with Local Binary Patterns: Application to Face Recognition. IEEE Trans. on Pattern Analysis and Machine Intelligence 28, 2037–2041 (2006) 9. Liu, C., Wechsler, H.: Gabor Feature Based Classification Using the Enhanced Fisher Linear Discriminant Model for Face Recognition. IEEE Trans. on Image Processing 11, 467–476 (2002) 10. Tan, X.Y., Triggs, B.: Enhanced Local Texture Feature Sets for Face Recognition under Difficult Lighting Conditions. In: Proceedings of the 2007 IEEE International Workshop on Analysis and Modeling of Faces and Gestures. LNCS, vol. 4778, pp. 168–182. Springer, Heidelberg (2007)

Multi-view Face Detection Using Six Segmented Rectangular Features Jean Paul Niyoyita, Zhao Hui Tang, and Jin Ping Liu*

Abstract. This paper presents a multi-view face detection system which combines skin color detection and adaptive boosting (Adaboost) algorithm. The aim of this combination is to satisfy the accuracy and speed, the two important characteristics of real time face detection. The second contribution of this paper is a new type of rectangular features for face detection, represented in a 2 X 3 matrix form. With these new features the training time becomes significantly very short: five times faster than using the traditional feature set. The experimental results demonstrate the effectiveness of our method in detecting profile and rotated faces over a wide range of variations in color. The method detects 97.5% of positive faces while 5% is declared as false positive. The system also detects the occluded faces as well as lean faces and rotated faces. Keywords: Face detection, rectangular filter, skin color information, Adaboost.

1 Introduction Face detection from images is currently a very active research area due to its potential applications, such as automated face recognition, surveillance and security system, human–computer interaction, etc [1], [2]. Although the task of face detection is so trivial for human brain, yet it still remains a challenging and difficult problem to enable a computer to deal with face detection. Numerous methods have been proposed to detect faces in a single image of intensity or color images [3]. Among the face detection methods, the ones based on learning algorithms have attracted much attention recently and have demonstrated excellent results [2]. With appearance-based methods, face detection is treated as a problem of classifying each scanned sub window as one of two classes (i.e. face and non-face). Appearance-based methods avoid difficulties in modeling 3D structures of faces by considering possible face appearances under various conditions. A face/nonface classifier may be learned from a training set composed of face examples Jean Paul Niyoyita . Zhao Hui Tang . Jin Ping Liu School of Information Science and Engineering Central South University, Changsha 410083, China *

H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 333–342. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

334

J.P. Niyoyita, Z.H. Tang, and J.P. Liu

taken under possible conditions and non-face examples. Building such a classifier is possible because pixels on a face are highly correlated, whereas those in a non-face sub window present much less regularities. However, large variations caused by changes in facial appearance, lighting, and expression impede the face detection. Changes in facial view (head pose) further complicate the situation. The speed is also an important issue for real-time performance. Moreover, the image based techniques are slow. Recently, several face detection systems based on image based technique have been made [4], [5]. Those techniques require a large number of training samples and a long training time. To overcome this drawback we combine the image based technique with skin color technique. We first prevent the problem of lighting condition, which may change the skin color, by compensating the light in image. As we know a digital image is composed by pixels; however we need just those ones for the skin. To save the time we remove the non-skin pixels. After extracting the skin tone, we have a set of face candidates which have to be scanned by a proposed six segmented rectangular filter (see the section 4.2) to gain the rapidity and training time. By using the adaptive boosting algorithm (Adaboost), these face candidates will be classified as face or non face. The paper is structured as follows. In the next section we present the résumé of our system. Section 3 then presents operations needed before carrying out the face detection. Having face candidates obtained from this phase, section 4 proposes a novel feature set to classify faces. The experimental results are presented in section 5, and then we conclude in section 6.

2 System Overview We have summarized the structure of our algorithm in Fig. 1. Our system is similar to the one of Rein-Lien Hsu et al. [1] for lighting compensation and skin color detection. As for processing, we apply a six-segmented rectangular filter to classify faces regions. The system contains two main parts: image preparation and processing. In order to alleviate the variation of lighting conditions, a pre-processing procedure must be done. To identify the existence of human face, we have to scan the image for detecting the skin color regions and remove unnecessary pixels. To reduce the search region, we need to locate the possible face region. For this we classify the pixels into face-color or non-face color based on their hue component only. After segmentation, we have face candidates. In this processing phase, we use classifiers. Here the image is scanned by a six-segmented rectangular filter. To achieve that, the adaptive boosting algorithm (Adaboost) will be used. In case that we have a grayscale image instead of color one; the preprocessing phase is not considered and we go directly to the classification step.

Multi-view Face Detection Using Six Segmented Rectangular Features

335

3 Preprocessing Phase Human skin has its own color distribution that differs from that of most of non face objects [1]. For this reason, it can be used to filter the input image to obtain candidate regions of face. To achieve this filtering process, we need to transform the color space of our image. The following sections deal with these operations.

3.1 Lighting Compensation The appearance of skin tone color depends on lighting conditions. For this reason the first step in preparation phase is to normalize the color appearance in image. For this we have used the “reference white” approach as proposed in [1]. We first sort the image and obtain the top 5% of pixels with high gray level. If the number of those pixels is larger than 100, their gray level is set to 255 and for remain pixels value, the value is normalized to the ratio of those 5% top pixels (ratio is obtained by dividing the value 255 by the mean of the 5% top pixels). The color components are unaltered if a sufficient number of reference white pixels is not detected.

3.2 Color Transformation Most images acquired with a color video camera are in RGB space. The reason for transforming to another color space when aiming at skin color detection is to achieve invariance for changing illumination and/or skin tones. In the RGB space, the triple component (R, G, B) represents not only color but also luminance. Luminance may vary across a person's face due to the ambient lighting and is not a reliable measure in separating skin from non-skin region [6]. Luminance can be removed from the color representation in the chromatic color space. Numerous color spaces have been developed and suggested for various applications and purposes. They are usually obtained via a transform from RGB space or other “basic” color space. Those color spaces can be classified in two categories: those that only use chrominance channels such as normalized RG, HS, CbCr and color vectors that consist of all color channels (RGB, HSV, YCbCr, CIE-Lab). In our paper we benefit from a color vector that use all color channels since, as revealed in [7], segmentation performance degrades when only chrominance channels are used in pixels classification. For this, we prefer the YCbCr color space, since it is widely used in video compression standard.

3.3 Skin Color Segmentation and Region Merging As we have seen in previous section, human skin has its own color distribution that differs from that of most of non-face objects. Although different people have different skin color, several studies have shown that the major difference lies largely between their intensity later than their chrominance [3].

336

J.P. Niyoyita, Z.H. Tang, and J.P. Liu

Skin segmentation is commonly used in algorithms for face detection, hand gesture analysis, a filter for pornographic content on the internet, and other uses in video applications. In these applications, the search space for objects of interest, such as faces or hands, can be reduced through the detection of skin regions. To this end, skin segmentation is very effective because it usually involves a small amount of computation, it can be done regardless of pose, and it is robust under partial occlusion and resolution changes. To segment the skin, we first have to detect it in order to build a decision rule that will classify skin and non-skin pixels. For this fact, we take advantage of skin modelling technique. Having skin region, filters will no longer have to scan the whole image but just that region. After this segmentation, we will have to merge the detected skin regions then eliminate the small skin regions in order to facilitate the image scan.

4 Features and Classification Haar features are based on Haar wavelets, which are functions that consist of a brief positive impulse, followed of a brief negative impulse. Features usually encode knowledge about the domain, which is difficult to learn from the raw and finite set of input data. These haar-like features are interesting for two reasons: (1) powerful face/nonface classifiers can be constructed based on these features; and (2) they can be computed efficiently by using the summed-area table or integral image technique (see the next section). In previous image based face detection systems, many feature sets have been used: four feature types for Paul Viola and Michael Jones [8] and Lienhart et al. [5] extended them. To get the value of a Haar-like feature, we have to make the difference between the sums of the pixel gray level values within the black and white rectangular regions.

4.1 Integral Image Technique As explained in [4], the technique consists in rapidly computing the rectangle features using the intermediate representation for the image. The integral image II(x, y) at location x, y contains the sum of the pixels above and to the left of x, y, defined as follow.

II ( x , y ) =

∑ I ( x, y )

x '≤ x , y '≤ y

(1)

The image can be computed in one pass over the original image using the following pair of recurrences:

S ( x, y ) = S ( x, y − 1) + I ( x, y )

(2)

Multi-view Face Detection Using Six Segmented Rectangular Features Fig. 1 The sum of the pixels within rectangle D can be computed with four array references

A

337

B 1

C 3

II ( x, y ) = II ( x − 1, y ) + S ( x, y )

2 D 4

(3)

Where S(x, y) is the cumulative row sum, S(x, −1) = 0 and II (−1, y) = 0. Using the integral image, any rectangular sum can be computed in four array references, as illustrated in Fig. 3. The value of the integral image at location 1 is the sum of the pixels in rectangle A. The value at location 2 is A+B, at location 3 is A+C, and at location 4 is A+B +C +D. The sum within D can be computed as (4+1)-(2+3) [4]. The use of integral images leads to enormous savings in computation for features at varying locations and scales.

4.2 Six Segmented Rectangular Filter In [4] Viola and Jones, proposed the concept of rectangular filter which is based on bright-dark relation. Oraya Sawettanusorn et al. [9] applied the concept to detect the face region around the point between the eyes. This point is a face representative because it is common to most people and easy to find for wide range of face orientation. As characteristics, the point between the eyes has dark part (eyes and eyebrows) on both sides, and has comparably bright part on upper side (forehead), and lower side (nose and cheekbone). Inspired by this idea, Shinjiro, Kawato, Tetsutani, and Hosaka [10] have developed the face detection and tracking in real time with SSR filter and support vector machine. The system extracts the face candidates with a Six-Segmented Rectangular (SSR) filter and face verification by a support vector machine. They also benefit in motion cue in selecting face candidates and template matching. Although we share the use of six segmented filter with those others, our system is different from theirs in the fact that, after prior preprocessing steps, the face candidates are selected by using Adaboost algorithm. This is different from Kawato, S. & Tetsutani who have used the support vector machine (SVM). At this point, the Adaboost take advantage over SVM because the later depends on parameter settings. It has also revealed that the Adaboost feature selection provides a final hypothesis model that can be easily interpreted but the high dimensional support vectors of SVM approach do not provide any [11]. For Oraya Sawettanusorn et al. they didn’t use the trained data. From those features we calculate a feature response which is a difference of the sum of pixels in neighbor regions. These responses then permit to distinguish

338

J.P. Niyoyita, Z.H. Tang, and J.P. Liu

(a)

(g)

(b)

(c)

(d)

(e)

(f)

(h)

Fig. 2 A feature set of six segmented rectangular filters ((a), (b), (c), (d), (e) and (f)). (g) The nose area is brighter than the right and left eye area. (h) The region containing both eyes and eyebrows is relatively darker than the cheekbone area

positive and negative examples. The overall classification decision is made from the combined weighted classification decisions of the group of classifiers. In training, we learn the group of classifiers one at a time. These weak classifiers are typically composed of single-variable decision trees called “stumps”. In training, the decision stump learns its classification decisions from the data and also learns a weight for its “vote” from its accuracy on the data. Between training each classifier one by one, the data points are re-weighted so that more attention is paid to data points where errors were made. This process continues until the total error over the data set, arising from the combined weighted vote of the decision trees, falls below a set threshold.

4.3 The Motivation The feature types used in [5] and [8] have been used to track and detect the human faces in image. However they have used a large number of features that increase the calculation time, therefore the training time is too long. For this, we opted for new feature type to solve the training time and improve the detection. These new features can also detect the diagonal face due to their property of being used in oblique position. In the next paragraph we will deal with calculation of different feature types and the advantage of the new feature set. Number of features: As explained in [5], the number of features obtained from each feature prototype differs from prototype to prototype. In order to calculate the number of features, suppose that we have a window of size W x H as a basic unit for testing the presence of face and rectangle of size w X h to slide it, as shows the Fig. 5. With: 0<w<W; 0 < h
Multi-view Face Detection Using Six Segmented Rectangular Features

x + 1⎞ ⎛ y +1⎞ ⎛ F = Y.⎜W +1− w ⎟ ⎟.⎜ H +1− h 2 ⎠⎝ 2 ⎠ ⎝

339

(4)

While a rotated feature generate

X + 1⎞ ⎛ Y +1⎞ ⎛ XY .⎜W + 1 − z ⎟.⎜ H + 1 − z ⎟ with z=w+h 2 ⎠⎝ 2 ⎠ ⎝

(5)

Using the above formulas we can calculate the number of features according to their type. The results show that number of features significantly decreases if we use the new prototype feature (1.1/10 compared with former features). For details about feature calculations, we suggest the reader to read the paper wrote by Lienhart [5].

4.4 Classification in Cascade We know how to select a small number of critical features and to combine them into a strong classifier. Now, we want to increase the detection performance while reducing the computation time. That objective will be achieved by constructing a cascade of classifiers. The principle is to reject quickly the majority of negative windows while keeping almost all positive examples and then focus on more sensitive sub windows with more complete classifiers. To do that, the first stages in the cascade will contain only few features, which achieve very high detection rates (about 100 %) but will have a false positive rate of roughly 40 %. It is clear that it is not acceptable for a face detection task but combining successively many of these stages which are more and more discriminating will permit to reach the goal of fast face detection. We can just compare this cascade structure with a degenerated decision tree. If a sub-window is classified as positive at one stage, it proceeds down in the cascade and will be evaluated by the next stage. It will be like this until this subwindow is found negative by one stage or if all the stages classify it as positive. In this last case, it will finally be considered as a positive example. The fig 3 shows the procedure where each boosted classifier group is organized into nodes of a rejection cascade. In the figure, each of the nodes (F1, F2, F3,..., FN) contains an entire boosted cascade of groups of decision stumps trained on the Haar-like features from faces and non-faces. Typically, the nodes are ordered from least to most complex so that computations are minimized (simple nodes are tried first) when rejecting easy regions of the image. At the earliest stage, the cascade attempts to reject as many negative as possible. Although the boosting in each node is tuned to have a very high detection rate, it does have many false positives. But this will be salved because of using many nodes. During the run mode, a search window of different sizes is swept over the original image. This quick and early rejection vastly speeds up face detection.

340 Fig. 3 Rejection cascade: A series of classifiers is applied to every subwindow. The initial classifier eliminates a large number of negative examples with very little processing. Subsequent layers eliminate additional negatives but require additional computation

J.P. Niyoyita, Z.H. Tang, and J.P. Liu

No face

F1 Face

No face

F2 Face F3

No face Face Face

F4 No face

5 Experiments As explained with the definition of the integral image in 4.1, the scanning of an input image is quite simple and efficient with the integral image representation. To detect faces of different sizes and places in an image, we will apply scaled and shifted detectors all over the image. Our basic detector is a 20x20. All the images used to train the model, faces and non faces are of this size, and accordingly, all the selected rectangular features that we have to apply in the windows are defined in this 20x20 basic window. The training and classification have been done by using the gentle Adaboost algorithm. During this operation, 20 cascades (layers) have been created. We have trained our data using a set of 4160 images divided into two subsets: MIT-CBCL face recognition database which contains 2000 face images and background set of 2160 images taken arbitrarily. The training lasted just eight hours on a PC with Intel 3.2 GHz CPU, and 1 gigabytes of random access memory while it takes forty three hours to train the same set on the same PC using traditional features presented in [4] and [5]. At the same PC, our system uses 100 ms to detect an image of 1024 by 768 mega pixels i.e. 2/3 of time if we compare it with the one in [4]. We tested our detector on a new test set of CMU face database that contains 65 images with 187 faces. With that database test, 119 of 121 (98.3%) upright faces have been detected while 38 of 39 (97.7%) and 25 of 27 (92.5%) for profile and rotated faces respectively were detected. It can detect 97.5% of positive faces while 5% is declared as false positive. Unfortunately we couldn’t compare other systems ours due to different testing datasets; the reason why we use the MIT+CUM dataset, to make a comparison with the Paul Viola and Michel Jones [4]’s system as shows the table 1. The system also detects the occlusion faces as well as lean faces and oblique faces. In Fig. 5, we show the detection results where the occluded faces, rotated faces, and lean faces are detected.

Multi-view Face Detection Using Six Segmented Rectangular Features

341

Table 1 Detection results with MIT+CMU test set containing 130 images and 507 faces

Detector Paul viola [4] Our detector

Upright face 93.7% 98.6%

Profile face 97.5%

Rotated face 96%

W

Detection rate 93.7% 97.3%

False alarm 5% 6%

W

Window w h

H

Window h w w H

Upright Rectangle h Rotated rectangle Fig. 4 Upright and rotated rectangular features inside the testing window

Fig. 5 Pictures taken in different position, different orientation and different lighting conditions

6 Conclusion In this paper, we have presented a method of detecting and tracking faces in video sequences in real time, which is based on skin color detection and learning

342

J.P. Niyoyita, Z.H. Tang, and J.P. Liu

method. This method first compensates the light in image then selects the skin tone for getting the face candidates. Our basic strategy for detection is fast training with a Six-Segmented Rectangular (SSR) filters and face verification by Adaboost algorithm. We have evaluated our algorithm on various images and face databases. The images have been taken in different positions and lighting conditions. Not only has improved the detection speed, our algorithm is also effective in detecting non-frontal faces and non-upright faces. Acknowledgments. This research has been supported by the National Natural Science Foundation of China under grant number 60634020.

References 1. Hsu, R.L., Abdel-Mottaleb, M., Jain, A.K.: Face Detection in Color Images. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(5), 696–706 (2002) 2. Li, S.Z., Jain, A.K.: Handbook of Face Recognition, pp. 371–390. Springer Science+Business Media, Inc., Heidelberg (2005) 3. Yang, M.H., Kriegman, D.J., Ahuja, N.: Detecting Faces in Images: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(1), 34–58 (2002) 4. Viola, P., Jones, M.J.: Robust Real-time Face Detection. International Journal of Computer Vision, 137–154 (2004) 5. Lienhart, R., Kuranov, A., Pisarevsky, V.: Empirical Analysis of Detection Cascades of Boosted Classifiers for Rapid Object Detection. MRL Technical Report (2002) 6. Cai, J., Goshtasby, A., Yu, C.: Detecting Human Faces in Color Images, Wright State University, U. of Illinois. In: International Workshop on Multimedia Database Management Systems, pp. 124–131 (1998) 7. Phung, S.L., Bouzerdoum, A., Chai, D.: Skin Segmentation Using Color Pixel Classification: Analysis and Comparison. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(1) (2005) 8. Viola, P., Jones, M.J.: Rapid Object Detection using a Boosted Cascade of Simple Features. In: IEEE CVPR (2001) 9. Sawettanusorn, O., Senda, Y., Kawato, S., Tetsutani, N., Yamauchi, H.: Real-Time Face Detection Using Six-Segmented Rectangular Filter. In: International Symposium on Intelligent Signal Processing and Communication Systems, Japan (2003) 10. Kawato, S., Tetsutani, N., Hosaka, K.: Scale-Adaptive Face Detection and Tracking in Real Time with SSR Filters and Support Vector Machine. The Iinstitute of Electronics, Information and Communication Engineers, pp. 2857–2863 (2005) 11. Silapachote, P., Karuppiah, D.R., Hanson, A.R.: Feature Selection Using Adaboost for Face Expression Recognition. University of Massachusetts Amherst (2005)

Level Detection of Raisins Based on Image Analysis and Neural Network Xiaoling Li , Jimin Yuan, Tianxiang Gu, and Xiaoying Liu*

Abstract. Utilizing image processing technology, the researcher calculated the length of the long-short-axis, marked the location of it and calculated the 7 parameters, chroma, length, width and etc, 4 of which were chosen as the key characteristics of the input to build a neural network and identify the level of raisins through analysis of the external characteristics of raisins. The method was based on traditional characteristics detection, used by boundary tracking algorithm and the length of the new long-short-axis detection algorithm. The result of the experiment indicates that the calculating method and judging of the level of raisins are precise and accurate, with an average recognition rate of 92%. Therefore, the method has a great practical value, which can be applied to other agricultural products classification. Keywords: Image manipulation, Characteristic parameter, Neural network, Raisin, Level detection.

1 Introduction The image detection technology to identify agricultural research has been extensively researched and applied. Rehkugler and He Dong detected defects of apples by utilizing methods of gray-scale value detection of image and color classification [1,2]; Miller used colored and near-infrared image to analyze the injury area of peaches to classify peaches [3]; Shearer invented new ways of colors grading on round chilis from the perspective of machine vision, with the correct rate up to 96% [4,5]; application of computer vision technology to test the quality of mango and surface corruption [6]; and theory of image processing and neural network to determine long fruit, with accuracy rate over 96% with regard to grading of cucumber levels [7]. Xiaoling Li Information college of Panzhihua University, Panzhihua 617000, SiChuan, China

*

Jimin Yuan . Tianxiang Gu Automatic College of UESTC, Chengdu 610054, SiChuan, China Xiaoying Liu Computer College of PanZhiHua University, PanZhiHua 617000, SiChuan, China [email protected] H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 343–350. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

344

X. Li et al.

At present, there are photoelectric sorting and manual separation with regard to distinguish the grading of raisins. Methods of Manual separation were based mainly upon human observation to determine the level. There is lack of objectivity if relying solely on color characteristics and the naked eyes because of the raisins sizes and the complex situations of surface. Method of photoelectric separation is based primarily on color characteristics, utilizing surface testing to determine level of raisins, which is made up of material delivering equipment, light boxes, electronically controlled lines and pneumatic system to distinguish the colors of raisins [8]. It controls circuits complicatedly and needs users of higher level, which restricts its application. In this paper, image processing and analysis technology are combined with artificial neural networks to identify and grade raisins. The key of this way is image manipulation algorithms and characteristic testing, linked to artificial neural networks through extracting valid characteristic parameter. The experiment takes Turpan nuclear-free green raisins as samples [9].

2

Image Pre-manipulation

Use digital cameras COOLPIX4500 to get photos by adjusting brightness and location, the calibration coefficient of X and Y: X_SCALE=0.2135675; SCALE=0.214416667.The calibration coefficient of diagonal line XY_SCALE can be worked out by formula 1.

XY _ SCALE = X _ SCALE 2 + Y _ SCALE 2

(1)

The photos of raisins, 640 × 480 pixel, 24-bit true color BMP, as shown in Figure 1. In the process of image-getting, the quality of image declined because of uneven light and transmission lines etc. Therefore, the images need pre-treating for extracting characteristic data. The image Pre-treatment shown in Figure 2:

Fig. 1 Raisin image (The left: First-grade raisin image; The right: Second-grade raisin image)

Level Detection of Raisins Based on Image Analysis and Neural Network

345

Fig. 2 Process of image pretreatment

When segmenting the background of color images, the researcher first conducted threshold as far as the gray image was concerned, and then compared the original color image with treated image. If pixel is 255(having treated image, the background is white), set the corresponding pixel of the original image to 255 [10]. In this way, non-background part of the original image is retained, making the background white. The colored image of raisins can be indicated as R(red), G(green) and B(blue). Based on the formula 2, the threshold is shown as follows:

R0 (i, j ) ≤ Tr

⎧ R0 (i, j ) R(i, j ) = ⎨ ⎩255 ⎧ G0 (i, j ) G (i, j ) = ⎨ ⎩255

G0 (i, j ) ≤ Tg G0 (i, j ) > Tg

⎧ B (i, j ) B(i, j ) = ⎨ 0 ⎩255

B0 (i, j ) ≤ Tb B0 (i, j ) > Tb

R0 (i, j ) > Tr (2)

R0 ( i , j ),G0 ( i , j ) , B0( i , j ) and R( i , j ),G( i , j ),B( i , j ) are gray value of three-channel pixel before and after the background segmenting.; Tr, Tg, Tb are the threshold of segmenting background for three channels R, G and B. The result of segmenting is shown in Figure 3: Compared the three pictures, the researcher found that while segmentation in B channel, the information of raisins remained integral; However, Segmentation in R & G channel, part of information lost. Therefore, because of the largest instance

Fig. 3 Results of background segmentation in three channels(The left: Segmentation in R channel; The middle: Segmentation in G channel; The right: Segmentation in B channel)

346

X. Li et al.

between the two peaks which is helpful for the correct choice of the threshold, the effect in histogram B with background segmentation is quite good.

3 Selection and Extraction of Characteristics Parameter In order to identify and grade raisins, the characteristics of a single raisin need extracting. The following experiment focuses on extraction method of color, shape, and other characteristics. Transforming from RGB to HSI can be done as formula 3, 4, 5 and 6:

I = ( R + G + B) / 3

S = 1−

(3)

3 [min( R + G + B)] ( R + G + B)

[( R − G ) + ( R − B )] / 2 H = arccos{ } [( R − G ) 2 + ( R − B)(G − B)1/ 2

(4)

（if B ≤ G ）

[( R − G ) + ( R − B)]/ 2 H = 360。− arccos{ } if B > G [( R − G ) 2 + ( R − B)(G − B)1/ 2

（

(5)

）

(6)

Moreover, calculation the mean of HSI can be done as formula 7,8,9:

H =

1 n ∑ Hi n i =1

(7)

S=

1 n ∑ Si n i =i

(8)

I=

1 n ∑ Ii n i =1

(9)

、 S 、 I － the color, saturation and brightness of raisins picture element H －the mean of color; S －the mean of saturation; I － the mean of brightness; n－the number of raisins picture element.

In the above formula,

Hi

i

i

According to the area feature in the Green theorem as formula 10:

A=

1 ( xdy − ydx) 2∫

(10)

Integral goes along with the closed curve. After the discretization, the researchers get the following formula 11 to calculate the area:

Level Detection of Raisins Based on Image Analysis and Neural Network

347

Table 1 Average of raisin features Grade

H

S

I

P

A

L

M

First

44.7560

0.4811

0.3355

133.7500

140.2401

19.9255

9.6232

Second

36.1069

0.4506

0.3576

124.8903

102.6010

17.3508

7.9682

Third

32.1093

0.5245

0.4028

93.0393

67.4188

14.1359

6.3648

A= In the above formula, N b

1 Nb ∑ [ xi yi +1 − xi +1 yi ] 2 i =1

(11)

－the number of boundary point.

Calculate three levels(each contain 40) of raisin samples, and get mean score of 7 characteristics, identify the parameters color, saturation, brightness, long, area, length and width as a means of identification features. As shown in table 1.

4 Raisin Grade-Level under BP Network Checkup The relation between the form feature and grading is comparatively complex which is hard to distinguish one from another, so the researchers make use of BP internet to establish the relationship between form feature and grading which is helpful to distinguish different grades. The input of BP network is 7, the output nodes are 3 and there is an implied layer. The input has 7 corresponding feature parameters, while the output is corresponding with the 3 grades of the raisin. The initial study velocity is 0.01.self study velocity is adopted. The target error is 0.001. The NN grading figure is as figure 4:

First Grade Raisin Characteristic Parameter of Raisin

Neural Network Classifier

Second Grade Raisin Third Grade Raisin

Fig. 4 Grading sketch map of neural network

348

X. Li et al.

The 180 samples are made up of 60 raisins in 3 different grades respectively. Identifiable samples are 120 raisins with 40 raisins of each grading. In order to make them belonging to the same scope, the data is normalized. PT is the input matrix, among which there are 7 feature parameters and 5 samples of different grades.

T is the target matrix, and each set of parameters is corresponding to an output vector. Each element stands for a grade. In the target output, 1 stands for the grade of the target output.

5 Network Training and the Results of Recognition The experiment is carried out with the help of 7 inputs, 3 outputs and a middle layer with 17 nerve cells, indicating that the training speed is fast, the ratio of identification is very high. The result is shown in table 2. Table 2 shows that chroma, area, length, width are relatively important parameters. At the same time, these 4 features are more efficient than those 7 features. Therefore, the cyber structure adopts the system with 7 inputs, 3 outputs and a middle layer with 17 nerve cells.

Level Detection of Raisins Based on Image Analysis and Neural Network

349

Table 2 Recognition rate of different parameters

parameter number 7

H √

S √

6

√

√

5

√ √

5

I √

epoch of learning

Recognition rate of sample %

（ ）

P √

A √

L √

M √

976

91

√

√

√

√

973

91

√

√

√

√

1181

91

√

√

√

√

9302

86

√

√

1961

90

1183

88

√

930

88

√

907

92

4

√

√

4

√

√

√

4

√

√

√

4

√

√

√ √

6 Conclusion Separating raisins is based on image manipulation technology, through the border tracking algorithms. The researchers put forward a new method of calculation, which is to test the length of the long-short-axis, mark the location of it and calculate the 7 parameters, chroma, length, width and etc, 4 of which are chosen as the key characteristics of the BP input of network to build a network and identify the level of raisins. The result of the experiment indicates that the calculating method and judging of the level of raisins are precise and accurate, with an average recognition rate of 92%.

References 1. Rehkugler, G.E., Throop, J.A.: Apple Sorting with Machine Vision. Transaction of the ASAE 29, 1388–1395 (1985) 2. He, D.J., Yang, Q.X., Shao, P.: Computer Vision for Color Sorting of Fresh Fruits. Transaction of the Chinese Society of Agricultural Engineering 14, 202–205 (1998) (in Chinese) 3. Miller, B.K., Delwiche, M.J.: A Color Vision System for Peach Grading. Trans. of the ASAE 32, 1484–1490 (1989) 4. Li, P., Zhu, J.Y., Liu, Y.D.: Application and Developing Trend of Computer Version Technology in Detection and Classification of Agricultural Products. Acta Agriculturae Universitis Jiangxiensis 27, 796–800 (2005) (in Chinese) 5. Shearer, S.A., Payne, F.A.: Color and Defect Sorting of Bell Peppers Using Machine Vision. Trans. of the ASAE 33, 2045–2050 (1990) 6. Wang, J.F., Luo, X.W., Hong, T.S.: Application of Computer Vision Technology in Detecting Mango Weight and Surface Bruise. Trans. of the Chinese Society of Agricultural Engineering 14, 186–189 (1998) (in Chinese)

350

X. Li et al.

7. Zhang, R.Y., Liu, S.S.: Researches and Applications of Computer Vision Technique in Fruit and Vegetables Commercialization After Harvesting. Journal of Yuzhou University 21, 497–501 (2004) (in Chinese) 8. Xie, F.Y.: On Photoelectric Technology of Colour Sorting for Raisin. Journal of Hunan Agricultural University 30, 71–73 (2004) 9. Liu, X.Y.: Study on Raisin Grading Technology Based on Image Analysis. Northwest A & F University, Yangling (2006) (in Chinese) 10. Ying, Y.B.: Study on Background Segment and Edge Detection of Fruit Image Using Machine Vision. Journal of Zhejiang Agricultural University 26, 35–38 (2000) (in Chinese)

English Letters Recognition Based on Bayesian Regularization Neural Network Xiaoli Huang and Huanglin Zeng*

Abstract. In this paper, we study the incorporation of Bayesian Regularization into constructive neural networks. The degree of regularization is automatically controlled in the Bayesian inference framework and hence does not require manual setting. Simulation shows that regularization with input training using a full Bayesian approach produces networks with better recognition performance and lower susceptibility as the noise increases. Regularization with input training under the gradient descent algorithm, however, does not produce significant improvement on the problems tested. Keywords: Bayesian regularization, Neural network, The gradient descent algorithm, English letters recognition.

1 Introduction Multi-layer feed forward networks have been popularly used in many pattern classification and regression problems. BP algorithm is one of the fully developed algorithms. According to the Kolmogolov theorem, this type of network consists of the input, the hidden and the output layers which can precision approach for any non-linear function. This conclusion is strictly set up only in the infinite network conditions. Generally speaking, we do not know the exact capacity of a limited network to solve a given problem. Because different problems need different ability of the network, it is important to select the appropriate size of the neural network when it is designed to solve specific problems. If the network is too small, it could not be a good model on the problems. At the same time, over-fitting problem or poor generalization capability happens when a neural network over learns during the training period. As a result, such a too well-trained model may not perform well on unseen data set due to its lack of generalization capability. The first method is an early learning stopping mechanism in which the training process is concluded as soon as the overtraining signal appears. The signal can be observed when the Xiaoli Huang . Huanglin Zeng School of Automation and Electronic Information, Sichuan University of Science & Engineering, Zigong 643000, China [email protected] *

H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 351–357. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

X. Huang and H. Zeng

352

prediction accuracy of the trained network applied to a test set, at that stage of training period, gets worsened. The second approach is the Bayesian Regularization for regular-insertion and model comparison described in the companion paper `Bayesian interpolation' [1]. This approach minimizes the over-fitting problem by taking into account the goodness-of-fit as well as the network architecture. My work is based on the same probabilistic framework and extends it using concepts and techniques adapted from Gull and Skilling's Bayesian image reconstruction methods [2].The Bayesian Regularization approach is considered in this study and demonstrated on the letters in the alphabet which are added noise.

2 BP Neural Network’s Overview BP ANN is a kind of multilayer forward propagation NN based on the inverse error propagation algorithm. The BP model is consisted of one input layer, some hidden layers and one output layer, with every layer consisting of some neurons, connection of the different layer neurons by connective weight and threshold and nonexistence of connection among same layer neurons. The sketch of standard BP model is shown in Fig.1. The error back-propagation learning consists of 2 passes through the different layers of the network: a forward and a backward passes. In the forward pass, an activity pattern is applied to the input neurons of the network and its effects propagate through the network, layer by layer, until an output is produced the network. Weights between neurons of successive layers are initially assigned in random. In the backward pass, the error observed between the network and the desired responses is computed and used to amend the weights. The standard BP learning algorithm [3] is to modify the network connective weights and thresholds to make the error function descending along negative grad direction. Let I = [ I1 , I 2 , , I i , , I m ] be the function implemented by a network with

i hidden units directly connected to the output unit, O = [O1 , O2 , , Om ] be the expect output , w be the set of values of the connections in the network, the error sum of squares function

ED ( w) =

Fig. 1 Structure of three-layer BP Neural Network

1 m ∑ (Yk − Ok )2 2 k =1

(1)

English Letters Recognition Based on Bayesian Regularization Neural Network

353

w

be the corresponding residual error. The task of `learning' is to find a set of which fits the training set well, i.e. has small error

ED .

3 Bayesian Regularization It is also hoped that the learned connections will ‘generalize’ well to new examples. We will address medications to the standard back prop algorithm which implicitly or explicitly modify the objective function, with decay terms or regularizes. Some of the ‘hints’ in [4] also fall into the category of additive weight-dependent energies. A sample weight energy term is:

Ew = ∑ i

1 2 wi N

(2)

Where the N is number of the samples. Gradient-based optimization is then used to minimize the combined function:

H = α EW + β ED

(3)

Where the α is a decay rate or regularizing constant which controls of other parameters’ (the weight and the threshold) distribution. The α , β = 1 − α are parameters which are to be optimized in Bayesian framework of MacKay [7], [5]. Training algorithms are designed to minimize the error if α << β . At the same time training algorithm is designed to have more smooth response, that is, as far as possible to reduce the effective network parameters in order to make up a larger network error if α >> β . Connections between probabilistic inference and neural networks have been discussed in [6]. Let us now review the probabilistic interpretation of network learning. A network with connections w is viewed as making predictions about the target outputs as a function of input I in accordance with the probability distribution:

P ( I , w, β ) = Where

exp − β E ( I , w) Si ( β )

(4)

Si ( β ) = ∫ (exp − β E )dI , E is the error for a single datum, and β is a

measure of the presumed noise included in I . If E is the quadratic error function then this corresponds to the assumption that I includes additive Gaussian noise with variance δ = 1/ β . A prior probability is assigned to alternative network connection strengths written in the form: 2

w,

X. Huang and H. Zeng

354

P ( w, α ) = Where

exp − α Ew ( w) sw (α )

(5)

S w ( β ) = ∫ (exp − α Ew )d k w , α is a measure of the characteristic ex-

pected connection magnitude. If

Ew is quadratic as specified in (2) then weights

are expected to come from a Gaussian with zero mean and variance The posterior probability of the network connections

P( w, α , β ) =

δ w2 = 1/ α .

w is:

exp− (α Ew + β ED ) S (α , β )

(6)

∫

Where S (α , β ) = [exp − (α Ew + β ED )]d k w . So under this framework, minimization of A = (α Ew + β ED ) is identical to finding the (locally) maximum a posteriori parameters

wMP ; minimization of ED

by back propagation is identical to finding the maximum likelihood parameters wML . Thus an interpretation has been given to back propagation’s energy functions

ED and Ew , and to the parameters α and β . Bayesian Regularization can make effectively weights as little as possible under the circumstances of ensuring the network error as small as possible through employing new performance function; this is equivalent to automatically reduce the size of the network. Bayesian methods can adaptive adjust parameters to achieve the best value. In the MATLAB toolbox, the Bayesian Regularization is achieved through the trainbr function.

4 Results and Discussions The neural network input neurons are 5 × 7 and output neurons are 26 because the matrix of English letters is 5 × 7 and the number of English letters is 26; Experience has shown that 15 neurons is selected as the hidden layer of neural network is the best. In order to have a network of anti-interference capability, first of all, the ideal noise-free letters are used to train the network, and then two sets of letters with random noise are used to train the network, with the last the ideal noise-free letter are used to train the network. The output of the network will be put through a competition layer to ensure that the goal output value is 1 and the other output is 0. Let the network goal error be 0.0001, the largest training number be 1000. First of all, Bayesian Regularization neural network is used to train the network, the parameters after the convergence is shown in Fig. 2; and then the gradient descent algorithm that includes momentum and self-adjusting learning rate is used to train

English Letters Recognition Based on Bayesian Regularization Neural Network

355

Fig. 2 The parameters after the convergence by Bayesian regularization

（a）No noise

(b) With noise

the network on the same scale and same error conditions, the parameters after the convergence is shown in Fig. 3. From Fig.2 and Fig.3, both methods can be compared with the results of training parameters in Table 1: From Table 1 we can see that the Bayesian Regularization method can speed up network convergence while improve the accuracy of the network compare with gradient descent algorithm. Table 1 The results of training parameters of the two methods

Sum squared error

The Bayesian Regularization algorithm The gradient descent algorithm

Convergence number in training

No noise

With noise

No noise

With noise

25.060500

0.0000865078

135

40

0.0961202

0.5898880000

217

87

356

X. Huang and H. Zeng

Fig. 3 The parameters after the convergence by the gradient descent algorithm

(a) No noise

(b) With noise

Finally, performance tests are carried out on the network that trained by both algorithms. Any letter by adding 0.01 to 1.00 random noise to generate 1000 input vector are entered the network to perform the recognition experiments on the both trained network. The relationship between the recognition error rate and the noise indicators are shown in Fig. 4 (a) and (b). From Fig.4, we can see that: (1) The Bayesian Regularization algorithm’s recognition error rate was 0% (recognition rate can reach 100%) when the noise is less than 0.2 while the rate of recognition error rise slowly when the noise is between 0.2 and 1.0 to achieve 6%. But no matter how the noise increased, the largest error rate is 6%. (2)The gradient descent algorithm’s recognition error rate was 0% (recognition rate can reach 100%) when the noise is less than 0.15 while the rate of recognition error rise slowly when the noise is between 0.15 and 0.4. But the recognition error rate increased dramatically after the noise is bigger than 0.4. The largest error rate is 20%. From the above information, we can draw the conclusions that on the same scale and network error conditions, the Bayesian Regularization algorithm generalization ability is superior to the gradient descent algorithm because it has faster convergence, better identification and better fault-tolerant ability.

English Letters Recognition Based on Bayesian Regularization Neural Network

357

Fig. 4 The relationship between the recognition error rate and the noise indicators

(a) By the Bayesian Regularization algorithm

(b) By the gradient descent algorithm

References 1. MacKay, D.J.C.: Bayesian Interpolation. Neural Computation (1991) 2. Skilling, J.: Maximum Entropy and Bayesian Methods, pp. 45–52. Kluwer Academic Publishers, Dordrecht (1989) 3. Wang, Z.W., ZhangLiu, R.P.: One Improved BP Neural Network Learning Algorithm. Mathematical Theory and Applications 25, 31–34 (2005) 4. Abu-Mostafa, Y.S.: Learning from Hints in Neural Networks. J. Complexity 6, 192–198 (1990) 5. MacKay, D.J.C.: Bayesian Interpolation. Neural Computation 4(3), 415–447 (1992) 6. Tishby, N., Levin, E., Solla, S.A.: Consistent Inference of Probabilities in Layered Networks: Predictions and Generalization. In: Proc. IJCNN, Washington (1989) 7. MacKay, D.J.C.: A Practical Bayesian Framework for Back Propagation Networks. Neural Computation 4(3), 448–472 (1992)

Iris Disease Classifying Using Neuro-Fuzzy Medical Diagnosis Machine Sara Moein, Mohamad Hossein Saraee, and Mahsa Moein*

Abstract. Disease diagnosis is an essential task in the medical world. The use of computers in the practice of medicine is becoming more and more crucial. In this paper, we propose an intelligent system to help us diagnose the Iris disease. This system is based on Artificial Neural Network (ANN) approach. In order to evaluate our proposed approach we apply the system on a dataset which includes all related symptoms. Next multilayer perceptron ANN is trained to be able to classify. In order to obtain the best results we use different measure values. Finally data fuzzification is employed to improve the stem performance. Keywords: Artificial Neural Network, Medical Diagnosis, Fuzzification.

1 Introduction Medical diagnosis as an important problem for human. Medical diagnosis deals with human body and life. The amount of information a physician can memorized and remember is limited. In addition there are enormous number of difference diseases each one with distinct feature. Medical diagnosis directly related to the experience and intelligence of a physician. However the most experience medic, will do the same as the new graduated ones, in facing a new disease. Therefore, many medical experts and scientists are concerned about computerizing tools to diagnose disease in medical science [13], [14], [16]. Fuzzy logic, genetic algorithm, artificial neural network and hybrid neurofuzzy systems have tried to solve some of medical diagnosis problems. Sara Moein Dept. of Computer Engineering, Islamic Azad University of Najafabad branch, Isfahan, Iran [email protected] *

Mohamad Hossein Saraee Dept. of Electrical and Computer Engineering, Isfahan University of Technology, Isfahan, Iran [email protected] Mahsa Moein Dept. of Computer Engineering, Islamic Azad University of Najafabad branch, Isfahan, Iran [email protected] H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 359–368. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

360

S. Moein, M.H. Saraee, and M. Moein

1.1 Intelligent Medical Diagnosis Intelligent medical diagnosis machines do the same process as the physician perform for medical diagnosis. First, they arrange an interview with patient and to get appropriate knowledge after accurate examining. It is probable that additional laboratory examination should be taken. After all, the physician can diagnose the disease. Medical diagnosis is not a 100% reliable process. Either machine or a physician does make mistake. But it is safe to think that all physicians don't make the same medical diagnosis decision. Also, not all the intelligent machines have the same performance and they don't have the same reliability. The more intelligent and experienced machine, the more reliable medical decision making [3], [6], [9]. Artificial neural networks are helpful and powerful. Since not only they are capable to recognize patterns with the aid of the expert, they can also detect hidden information and strike features that can't be seen from data or even video pictures. Actually, artificial neural network shows that experience from expert is not enough in medical diagnosis. Nowadays, physicians combined this opportunity that gives by neural network. Scientists used ANN for various diseases like hepatitis, breast cancer, etc and they could achieve to goal results. Present paper is effort to use ANN for Iris medical diagnosis [10], [11]. Meanwhile there are different points to be concerned, when an intelligent machine has been implemented: • Comprehensibility: the symptoms should be comprehensible by the intelligent machine. • Reliability: how much the machine processing result is reliable? Is it trustable? • Performance: does the intelligent machine perform the medical diagnosis task with high quality?

1.2 Existing WISER (Wireless Information System for Emergency Responders) is a system which has been designed to assist First Responders in hazardous material incidents. Developed by the National Library of Medicine, WISER provides a wide range of information on hazardous substances, including substance identification support, physical characteristics, human health information, and containment and suppression guidance [19],[20]. The followings are some of the features of the WISER system: • Mobile support, providing First Responders with critical information in the palm of their hand. • Comprehensive decision support, including assistance in identification of an unknown substance and, once the substance is identified, providing guidance on immediate actions necessary to save lives and protect the environment. • Rapid access to the most important information about a hazardous substance by an intelligent synopsis engine and display called "Key Info".

Iris Disease Classifying Using Neuro-Fuzzy Medical Diagnosis Machine

361

• Radiological support, including radioisotope substance data, tools, and reference materials. • Intuitive, simple, and logical user interface developed by working with experienced first responders.

2 Proposed Method Implementation of artificial medical diagnosis machine is involved few steps. The first is arrange an interview with a physician to collect standard data about Iris disease. Iris diagnosis uses the structure, color, and shape of the Iris and pupil of the eye to determine an individual's illness [1], [3], [5]. Illustration of the proper symptoms and reference values to Iris Diagnosis is given below, while Table 1 shows the symptoms values Maximum and minimum range. Attribute Information: 1. 2. 3. 4. 5.

Sepal length in cm Sepal width in cm Petal length in cm Petal width in cm Class: − Iris Setosa − Iris Versicolour − Iris Virginica

In next step the selected features for many visitors of a particular clinic are measured during 3 months.

2.1 Iris Dataset It is vital to have acceptable and valid data of Iris dataset. We have obtained the data set with cooperation of Iris clinic. This dataset consists of four major symptoms to diagnose Iris disease for 150 records of patients to classify them to 3 classes: Iris Setosa, Iris Versicolour and Iris Virginica, where each class refers to a type of iris plant. Table 2 shows a part of used dataset. As mentioned before, the complete dataset contains the measured features of 150 patients.

2.2 ANN Training Schema What we opted the algorithm of ANN, is the three layer perceptron to keep the structure as so simple. Also, we applied an ordinary back propagation training GDR algorithm on that. The main point is to change the training cycle number and also the number of nodes in hidden layer, to examine which structure can classify more accurately. Also fuzzification effect on symptoms values is investigated [18], [19]. Table 3 is the base to assign labels to each class. Fig. 2 shows the error

362

S. Moein, M.H. Saraee, and M. Moein

Table 1 Summary statistics for attributes(Symptoms values)

Sepal length Sepal width Petal length Petal width

Min

Max

Mean

SD

Class correlation

4.3 2.0 1.0 0.1

7.9 4.4 6.9 2.5

5.84 3.05 3.76 1.20

0.83 0.43 1.76 0.76

0.7826 -0.4194 0.9490 0.9565

Table 2 Part of provided dataset of Iris

Table 3 List of classes and assigned lables Assigned label 1 2 3

Class name Iris Setosa Iris Versicolour Iris Virginica

Iris Disease Classifying Using Neuro-Fuzzy Medical Diagnosis Machine

363

rate in some epochs. The MATLAB package and NETLAB toolbox are the simulator softwares that we used for ANN training. Fig. 1 is a patch of MATLAB code, developed to train ANN.

Fig. 1 Patch of Developed MATLAB code to Train ANN

Cycle Cycle Cycle Cycle Cycle Cycle Cycle Cycle Cycle Cycle Cycle Cycle Cycle Cycle Cycle Cycle

1 Error 2 Error 3 Error 4 Error 5 Error 6 Error 7 Error 8 Error 9 Error 10 Error 11 Error 12 Error 13 Error 14 Error 15 Error 16 Error

35.740504 Scale 1.000000e+000 20.335462 Scale 5.000000e-001 20.335462 Scale 2.500000e-001 20.335462 Scales 1.000000e+000 20.335462 Scales 4.000000e+000 20.335462 Scales 1.600000e+001 20.335462 Scales 6.400000e+001 20.335462 Scale 2.560000e+002 15.305778 Scale 1.024000e+003 12.937717 Scale 5.120000e+002 8.209793 Scale 2.560000e+002 7.977332 Scale 1.280000e+002 5.693968 Scale 6.400000e+001 5.470339 Scale 3.200000e+001 5.186173 Scale 1.600000e+001 5.081094 Scale 8.000000e+000

Fig. 2 Error rate in cycles after runing matlab code

364

S. Moein, M.H. Saraee, and M. Moein

3 Experiment and Discussion K-folding is a successfully experienced technique that is used in this proposed System. Based on the number of records, k-folding scheme with k=7 was applied. To be exact, each time 80% of the samples are as training dataset, and the left 20% are as the testing dataset, while training procedure has to be done 7 times. Fig. 3

Fig. 3 Training cycle = 150, hidden layer node = 30, Above the output error, below the assigned label/ disease, Performance: 80%

Fig 4a The fuzzy toolbox

Iris Disease Classifying Using Neuro-Fuzzy Medical Diagnosis Machine

365

(b)

(c) Fig. 4 (b,c) Fzzy membership function for two of Iris attributes

illustrates the test result based the non-fuzzy continuous data. Deduced performance is 80%. In the other stage, the fuzzufied data is trained. Basically a fuzzy membership function is used on all symptoms. Fig. 4 illustrates using the fuzzy toolbox and the fuzzy membership function for one of the symptoms. It is wonderful that

366

S. Moein, M.H. Saraee, and M. Moein

Fig. 5 Training cycle = 500, hidden layer node = 20, Fuzzified data, Performance = 100%

Fig. 6 Training cycle = 1000, hidden layer node = 10, Fuzzified data, performance = 100%

performance evaluated to 100%, after data fuzzification. Error rate decreases to less than 0.3. Fig. 5 and fig. 6 show it. Also trade off between training cycle and hidden layer node number can be notified. While the above part of figure shows the error rate, and the below part clear the difference between the training result and the real value [19].

4 Conclusion In this paper, we presented the power of Artificial Neural Network (ANN) in artificial medical diagnosis. Our approached showed improved performance. Most of

Iris Disease Classifying Using Neuro-Fuzzy Medical Diagnosis Machine

367

the research works in this area have been focused on diagnosis of disease like hepatitis, breast cancer. Iris is a new disease that has been classified using ANN approach. In addition, we show that the performance does improve enormously by performing fuzzifing, however there are different effective parameters to increase the validity of results. For the future works, advanced ANN can be trained with fuzzy or non fuzzy data to evaluate the performance for unknown problems. Also focusing on the process of data collecting based on interview with patients can be worthy of consideration toward an intelligence medical diagnosis system.

References 1. Laura, D., Camacho, A.C., Badr, A., Armando, D.G.: Images Compression for Medical Diagnosis using Neural Networks. Universidad Nacional de La Plata (1996) 2. Michie, D., Spiegelhalter, F.D., Taylor, S.: Machine Learning, Neural and Statistical Classification. Ellis Horwood (1994) 3. Salim, J.: Medical Diagnosis Using Neural Networks. International Journal of Health and Medical Computation (2004) 4. Pomi, A., Olivera, H.: BMC Medical Informatics and Decision Making: Contextsensetive auto associative memories as expert systems in medical diagnosis. BioMed. Central (2006) 5. Sasikala, K.R., Petrou, J., Kittler, M.: Fuzzy Classification with a GIS as an aid to decision making. Department of Electronic and Electrical Engineering (2006) 6. Steimanm, F., Adlassnig, K.P.: Fuzzy Medical Diagnosis. Institut fur Rechnergestutzte Wissensverarbeitung, Universitat Hannover (2004) 7. Siganos, D.: Neural Networks in Medicine the Public health care case study. PortRoyal Paris (1995) 8. Sordo, M.: Introduction to Neural Networks in Healthcare. Open Clinical: Knowledge Management for Medical Care (2002) 9. Ultsch, A., Korus, D., Kleine, T.O.: Integration of Neural Networks and KnowledgeBased Systems in Medicine. Hans-Meerwein-Straße / Lahnberge, Marburg (1995) 10. Wesley, K., Haisty, J.: Agreement between Artificial Neural Networks and Human Expert for the Electrocardiographic Diagnosis of Healed Myocardial Infarction. Journal of the American College of Cardiology, 28, 1012-1016LU TP 95-9 (1996) 11. Wolff, J.G.: Medical Diagnosis as Pattern Recognition in a Framework of Information Compression by Multiple Alignment. Unification and Search (2005) 12. Weigand, S., Huberman, A., Rumelhart, D.E.: Predicting the Future: A Connectionist Approach. International Journal of Neural Systems (1995) 13. Yao, X., Liu, Y.: Involving Artificial Neural Network in Medical. The University of South Wales, Australia (1995) 14. Yao, X., Liu, Y.: Neural Network for Breast Cancer Diagnosis. Biomedical and Bio Engineering Journal, Birhmingham (1999) 15. Zhou, Z.H., Jiang, Y., Yang, Y.B., Chen, S.F.: Lung Cancer Cell Identification Based on Artificial Neural Network Ensembles. Nanjing University, Nanjing (2001) 16. Zrimec, T., Kononenko, I.: Feasibility Analysis of Machine Learning in Medical Diagnosis from Aura Images. University of Ljubljana, Bangladesh (2004)

368

S. Moein, M.H. Saraee, and M. Moein

17. Moein, S., Monadjemi, S.A., Moallem, P.: A Novel Fuzzy Neural Based Medical Diagnosis system. In: WASET, Egypt, vol. 26 (February 2008) 18. Moein, S.: Hepatitis Diagnosis by Training A MLP Artificial Neural Network, Las Vegas, USA (July 2008) 19. Software for Neural Network Training (March 2006), http://www.Neuralware.com 20. Wireless Information System for Emergency Responders (2006), http://wiser.nlm.nih.gov/ 21. An expert system for Medical decision (March 2006), http://www.Dxplain.com

An Approach to Dynamic Gesture Recognition for Real-Time Interaction Jinli Zhao and Tianding Chen*

Abstract. Recently, the new Human Computer Interactive (HCI) technologies such as gesture recognition have drawn extensive attention, which have been extended to such field as the interaction between human and machine. In this paper, the critical technology about the gesture recognition is discussed, a method to locate human hand in high speed is proposed, based on the combination of skincolour based Gauss model and the revised optical flow. Considering the characters of dynamic hand gesture, we adopt the Hidden Markov Model and use it in ‘the remote robot control system based on p2p’. Keywords: Gesture Recognition, Skin-colour model, Optical Flow Tracking, Hidden Markov Model.

1 Introduction As personal computer becomes more pervasive in people’s life, the technology of Human Computer Interaction (HCI) will get more attention. The center of HCI has been transferred from computer to human [1-3]. Gesture recognition will be applied to the interaction between human and robot beside the interaction between computer and human. Robot will be used extensively in dangerous work and service industry, especially the service industry, which needs a natural human computer interface [4]. Unlike works where users are required to wear data-gloves, which are not natural, gesture recognition based on vision becomes into the main direction of the research. The advantage is non-glove and non-contact, and the fast computer power makes the real-time vision computer possible. Dynamic hand gesture has been regarded as a highly difficult task mainly due to two aspects of signal characteristics: segmentation ambiguity and spatiotemporal variability. The segmentation ambiguity problem concerns how to determine the start point and end point in a continuous hand trajectory. The other Jinli Zhao . Tianding Chen College of Information & Electronic Engineering Zhejiang Gongshang University *

No.18, Xuezheng Str., Xiasha University Town, Hangzhou 310018, P.R. China [email protected], [email protected] H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 369–377. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

370

J. Zhao and T. Chen

difficulty comes from the fact that the same gesture varies dynamically in shape and duration, even for the same gesture. In the thesis, a method is proposed for hand trajectory recognition based on HMM, which can model spatio-temporal information in a natural way. In order to distinguish the undefined gesture, a modified threshold model is proposed, and also a gesture spotting method is given to work perfectly.

2 System Overview The flow chart of the system is shown in Figure 1.

Fig. 1 The system block diagram

3 Dynamic Gesture Recognition Based on Vision 3.1 Hand Segmentation Based on Skin Color Model The RGB color model in common use includes the information both of color and brightness, which is sensitive to the variation of the illumination and environment. To ensure the skin color not be affected by the illumination and the color, we use

YCb Cr color model [6]. Where Y represents the brightness, Cb represents the color green and Cr represents red. The relationship between YCb Cr model the

and RGB model show as follows:

Y = 0.29900* R + 0.58700* G + 0.11400* B

(1)

Cb = −0.16874 * R − 0.33126 * G + 0.50000 * B + 128

(2)

Cr = 0.50000 * R − 0.41869 * G − 0.08131* B + 128

(3)

Then we model the skin color based on Gauss model,

(

)

M = Σ, μCb , μCr .

An Approach to Dynamic Gesture Recognition for Real-Time Interaction

Where

the

mean

N

value

of

Cb and Cr

respectively

371

is

given:

N

1 1 , C μ = ∑ ∑ Cri , and the variance matrix is given: bi C b r N i=0 N i =0 2 2 ⎛ σC 2 σC C ⎞ N N 2 2 ∑=⎜ ⎟ σ = ( C − μ ) σ = ( C − μ ) ∑ ∑ Cr ri Cr Cb bi Cb ⎜σ C C σ C 2 ⎟ ⎝ ⎠ .where i =0 i =0 , ,

μC =

b

b r

r b

r

N

σ C C = σ C C = ∑ (Cri − μC )(Cbi − μC ) b r

r

b

i =0

r

b

We use Gauss model to approach the distribution of the skin color with a certain number of the skin samples. After the Gauss model training of the skin color [7], we can get the probability of the pixels that need to be judged, as follow: P(Cb , Cr ) = exp(−0.5(Cr − μCr , Cb − μCb )T ∑ (Cr − μCr , Cb − μCb )) −1

(4)

So we can get the probability of the every pixel of the image to be tested. We take the adaptive threshold by observing to get the segmentation of the skin: when the threshold gets lower, the region of the skin increase, but the rate of increase reduce. When the threshold reduces to a certain value, the region of the skin increase significantly, since non-skin is region also included. Record the incremental minimum when the threshold reduces, and this is the optimal threshold. In this paper, threshold reduces from 0.75 to 0.05, reduces by 0.1 every time. Observing the incremental minimum, if the threshold reduces from 0.45 to 0.35, we set the threshold 0.4. The example of the segmentation shows as Figure 2.

Fig. 2 An example of the segmentation based on skin-color

3.2 Gesture Tracking Based on Optical Flow Optical flow means the speed of the campaign mode in the image. Optical flow tracking reflects the change in the image because of the motion in the interval dt . Therefore, optical flow only connected with the change from motion not from others [8].

372

J. Zhao and T. Chen

Fig. 3 Schematic of SAD

Optical flow calculates usually based on the assumption as follows: The brightness of any point on the object to be observed remain unchanged as the time changes. The campaign mode of the adjacent points is similar in the image plane; it is an assumption that the brightness in the hand region and the parameters such as the direction and the speed of the motion basically maintain the same, in the system of the dynamic gesture recognition based on vision. Calculating the sum of the absolute value of the gray difference between the adjacent frames in the interested pixels, we can seek for the optical flow, as show in Figure3. The block of No. f frame is the reference block(size is 8*8 pixels), it is at search block S next frame, block S is the expansion of the original reference block, which expand for 8 pixels at the top and left, 7 pixels at the bottom and right. The formula of the optical flow is SAD(Sum of Absolute Difference),

Ⅰ Ⅱ

D(u, v) =

7

∑ R( x, y ) − S ( x + μ + 8, y + ν + 8)

(5)

x , y =0

R ( x, y ) means the gray value of the point ( x, y ) in the block S, S ( x + μ + 8, y + ν + 8) means the gray value of the point ( x + μ + 8, y + ν + 8) , D( μ ,ν ) is the sum of the gray value in the block R and block S. we can know the range of ( μ ,ν ) is 8~7. The point ( μ ,ν ) corresponding to the smallest D ( μ ,ν ) is at the center of the block R in the next frame.

Where

The original image can be divides into many reference blocks, and the center of the block in the next frame can be calculated by the algorithm above.

3.3 Feature Extraction The features of the feature tracking basically are location, speed, and angle. We mainly consider the plane movement, and can calculate the values in the polar

An Approach to Dynamic Gesture Recognition for Real-Time Interaction

373

coordinate ( ρ , ϕ ) . The judgment suspension can be seen in figure 4 and figure 5. According to the algorithm of the optical flow, we can get the original data— the median point ( x, y ) of the hand in the rectangular coordinate. To make the features not sensitive to the change of the location, we should regularize ( x, y ) .

1 T ⎛1 T ⎞ ( x c , y c ) = ⎜ ∑ xt , ∑ y t ⎟ ⎝ T t =1 T t =1 ⎠

rt =

( (x − x ) t

c

ϕt = tan −1 ( Where and

2

+ ( yt − yc ) 2

)

yt − yc ) / 2π xt − xc

(6)

(7)

(8)

( xc , yc ) is the center of gesture track, rt is the distance of ( xc , yc )

( x, y ) . For φt , we have a discretization with 16 direction chain code.

To set the discretization dis_N, calculate the dis_N: Sita=180* φt ; FOR (i=0; i<=15; i++) If ((sita >= i*22.5) && (sita < (i+1) *22.5)) Dis_N=I; Thus, a specific gesture G can be represented by a series of discrete teature extraction [9].

G = {dis _ N i } i=1, 2, …T, where T is the symbol number of the quantified observation. To get the high recognition rate, the number of the observation quantified is very important. After several trials, the optimal quantified number is 24.

3.4 Suspension Point Judgment In the dynamic gesture recognition system, every movement of the hand will cause the system estimation [11] whether it is the suspension point. When the dynamic gesture almost ends, it has the corresponding max likelihood value. The start point can be obtained by the Viterbi algorithm whenever meets the suspension point candidate. And the optimal candidate suspension point can be the one we need. We choose as following principle:

ａ ｂ

choose the last candidate suspension point before the pattern B when it’s not a gesture after the pattern B. when it’s a gesture after the pattern B, we have two options:

374

J. Zhao and T. Chen

Fig. 4 Trackview of the start gesture

Fig. 5 Schematic of the suspension judgement

A will be a part of the gesture when the start point of B is before the first candidate suspension point of A, concluding both A and B and neglecting all the candidate suspension point of A. Choose the last point to be the suspension point when B starts before the first and the last candidate suspension point of A.

3.5 The Application of the Gesture Recognition In this paper, we have the recognition test with eight gestures show in the Table 1 to interact between the robot and human, applying to the remote robot control system based on p2p [12-13]. It can be divided to 3 steps: firstly, to choose a patter of the hand. In the paper, we choose the center of the gravity to be the pattern of the hand. Secondly, design a method of recognition. We adopt the discrete HMM with 8 states [14]. Finally, we train the HMM [15] in the training data. We can see from the Table 2 and Table 3 that the accuracy rate of the recognition is high to above 90%, which can basically satisfy the remote robot control.

An Approach to Dynamic Gesture Recognition for Real-Time Interaction Table 1 Eight dynamic gestures in the application

Table 2 Training sample and testing data of the test

Table 3 The results of the recognition test

375

376

J. Zhao and T. Chen

4 Conclusions In this paper, we considered a vision-based system that can interpret a user’s gestures in real time. Hand segmentation utilized color and motion information and we use the mathematics morphology of expansion and corrosion to get an accurate region of the hand. The current location of the hand can be represented by the coordinate of the center of gravity in 2D plane. HMM with discrete and continuous densities were trained and tested. The disadvantages of the system are that: it is difficult to segment the hand when the color of the background is similar to the hand, and in the condition that non-defined gestures happen, the system recognition is low to 88%.

References 1. Wu, Y., Huang, T.S.: Human Hand Modeling, Analysis and Animation in the Context of HCI. In: Proc. of the Int’l. Conf. on Image Processing, pp. 6–10 (1999) 2. Zhu, Y.X., Ren, H.B., Xu, G.Y., Lin, X.Y.: Toward Real-time Human-computer Interaction with Continuous Dynamic Hand Gestures. In: Proc. of the IEEE Int’l. Conf. on Automatic Face and Gesture Recognition, pp. 544–551 (2000) 3. Nilsen, M., Storring, M., Moeslund, T.B., Granum, E.: A Prodedure for Developing Intuitive and Ergonomic Gesture Interface for HCI. In: Camurri, A., Volpe, G. (eds.) GW 2003. LNCS(LNAI), vol. 2915, pp. 409–420. Springer, Heidelberg (2003) 4. Mo, Z.Y., Lewis, J.P., Nwumann, U.: SmartCanvas: A Gesture-driven Intelligent Drawing Desk System. In: Proc. of the IUI 2005, pp. 239–243 (2005) 5. Malik, S., Laszlo, J.: Visual touchpad: A Two-handed Gestural Input Device. In: Proc. of the ACM ICMI 2004, pp. 289–296 (2004) 6. Kjeldsen, R., Kender, J.: Finding Skin in Color Images. In: Proceedings of International Conference on Autaomatic Face and Gesture Recognition, Killington, pp. 312– 317 (1996) 7. Yang, J., Lu, W., Waibel, A.: Skin-color Modeling and Asaptation. In: Proceedings of ACCV 1998, HonhKong, pp. 687–694 (1998) 8. Cutler, R., Turk, M.: View-based Interpretation of Real-time Optical Flow for Gesture Recognition. In: IEEE Inter. Conf. on Automatic Face and Gesture Recognition, Nara, Japan (April 1998) 9. Zhu, Y.X., Huang, Y., Xu, G.Y., et al.: Motion-Based Segmentation Scheme to Feature Extraction of Hand Gesture. In: Zhou, J., ain, A.K., Tian-xu, Z., et al. (eds.) Proceedings of SPIE, vol. 3545, pp. 228–231. SPIE, Washington (1998) 10. Ren, H.B., Xu, R.Y., Zhu, Y.X., Lin, X.Y., Tao, L.M.: Motion-and-color Based Hand Segmentation and Hand Gesture Reocgnition. In: Proceedings of the First International Conference on Image and Graphics; Journal of Image and Graphics (JIG), 5(suppl.), 384–388 11. Wren, C., Azarbayejani, A., Darrell, T., Pentland, A.: Pfinder: Real-Time Tracking of the Human Body. IEEE Trans. on Pattern Analysis and Machine Intelligence 19(7), 780–785 (1997) 12. Waldherr, S., Romero, R., Thrun, S.: A Gesture Based Interface for Human-robot Interaction. Autonomous Robots 9, 151–173 (2000)

An Approach to Dynamic Gesture Recognition for Real-Time Interaction

377

13. Segen, J., Kumar, S.: Fast and accurate 3D gesture recognition interface. In: On: Proceedings of International Conference on Pattern Recognition, Brisbane, Australia, pp. 86–91 (1998) 14. Andrew, D., Bobick, Aaron, F.: Parametric Hidden Markov Models for Gesture Recognition Wilson. IEEE Trans. on Pattren Anlysis and Machine Intelligence 21(9), 884– 900 (1999) 15. Rabiner, L.R., Juang, H.: An Introduction to Hidden Markov Models. IEEE ASSP Magazine, January 4-16 (1985)

Dynamic Multiple Pronunciation Incorporation in a Refined Search Space for Reading Miscue Detection Changliang Liu, Fuping Pan, Fengpei Ge, Bin Dong, Shuiduen Chen, and Yonghong Yan

Abstract. Error prediction is important for detecting reading miscues by a reading tutor. In order to incorporate the error prediction into the decoder of a conventional speech recognizer, this paper proposes an algorithm of Dynamic Multiple Pronunciation Incorporation (DMPI). It solves the confliction between the coverage of errors and the perplexity increase of search space. A multiple pronunciation model (MPM) is developed to model the misreading errors. The pronunciation variants referred to in current reference are extracted from MPM and added to the search space of the recognizer – a refined state network before recognizing. The original state network is redeveloped to reserve some redundant fan-in and fan-out nodes which make the merging of the original state network and the additional state network very easy. The experiment result proved effectiveness of this algorithm. The EER is decreased by about 9.5%. Keywords: CALL, Reading tutor, Decoder, State network, Multiple pronunciation model.

1 Introduction Computer Assisted Language Learning (CALL) has been proved very effective for language learners. Some CALL systems concentrate on pronunciation assessment or pronunciation training (CAPT) [1, 2] and some concentrate on improving reading proficiency and comprehension of learners (reading tutor) Changliang Liu · Fuping Pan · Fengpei Ge · Bin Dong · Yonghong Yan ThinkIT Speech Lab, Institute of Acoustics, Chinese Academy of Sciences {chliu,fpan,fge,bdong,yyan}@hccl.ioa.ac.cn Shuiduen Chen Department of Chinese & Bilingual Studies, The Hong Kong Polytechnic University [email protected] H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 379–389. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

380

C. Liu et al.

[3, 4, 5, 6]. A reading tutor can listen to the learner’s reading and provides help when the learner needs or it thinks the learner needs. A reading tutor is a tool for helping language learners (i.e. children, foreigners) to learn language more easily. One of its main task is to detect and classify miscues in learners’ reading speech given the reference, so that the tutor can provide reliable error analysis and appropriate feedbacks. Reading miscues consist of insertion, omission, substitution, etc.. Most reading tutors use an automatic speech recognizer (ASR) to detect reading miscues [3, 4, 5, 6]. However, a broad range of pronunciation variations caused by misreading or accent make it difficult to get sufficiently high speech recognition performance because there are no correct definitions in a common pronunciation dictionary. The poor recognition result impairs the detection performance greatly. In this case, efficient error predication becomes important for the system. It can help the recognizer to anticipate and remediate the reading miscues. In [5] and [4], error predictions help to expand the simple reference to a syllable or phone graph. The graph with error predictions is then aligned with the lattice resulted from the recognizer to generate the detection results. However, in the recognizing process, they don’t consider the error predictions and only use a common pronunciation dictionary. As we have said, the misdefinitions of the reading miscues will make decoder make mistakes. We attempt to incorporate error predictions in the decoder of a large vocabulary continuous speech recognition (LVCSR) system. However, we don’t simply add all pronunciation variants into the pronunciation dictionary, which will largely increase the perplexity of search space and won’t obtain a good performance [8]. For a reading tutor, what the reader will read is known beforehand (reference), which is the most different with conventional LVCSR. Therefore, we proposed an algorithm of Dynamic Multiple Pronunciation Incorporation (DMPI). It only adds the pronunciation variants in the dictionary relevant to the reference and will rarely increases the perplexity search space. In practical LVCSR, the pronunciation dictionary is not an individual component. It and other knowledge sources such as hidden Markov model and context dependent phones are pre-compiled to a search space. In our system, the search space is a memory-efficient state network [7]. Dynamically adding pronunciation variants to the dictionary means that the search space has to be re-compiled, which is very inconvenient and inefficient. DMPI directly modifies the search space itself and needn’t re-compile the whole search space. The rest of this paper is structured as follows: section 2 describes an overview of the whole system; section 3 introduces the constructions of the multiple pronunciation model and in section 4 details of the algorithm DMPI is described; the experiments are presented in section 5 and the conclusion is given in section 6.

Dynamic Multiple Pronunciation Incorporation

381

2 System Overview Our system of detecting reading miscues is based on LVCSR framework, which is shown in figure 1. A decoder incorporates all the knowledge sources: language model (LM), acoustic model (AM), pronunciation dictionary (PD), reference and multiple pronunciation model (MPM), and translates the input speech into a hypothesis lattice. Then, the lattice is transferred into a confusion network by a lattice processing module. Confusion network is a new representation of a set of candidate hypothesis which specifies the sequence of syllable-level confusion in a compact lattice format [9]. It supplies many candidates for each word result, and can compensate the recognition errors when aligning it with the reference. Finally, the confusion network is aligned with the reference by an improved dynamic programming algorithm (DP), then the reading miscues – insertions, omissions and substitutions are extracted from the aligned path. Compared with a conventional LVCSR [7], the system has another two knowledge sources: • Reference: what is supposed to be read by the reader which is the special information for a reading tutor. • MPM: model the pronunciation variants and their probabilities.

/DQJXDJH 0RGHO

)HDWXUH ([WUDFW

$FRXVWLF 0RGHO

3URQXQFLDWLRQ 'LFWLRQDU\

/DWWLFH 3URFHVV

'HFRGH

$OLJQPHQW

'HWHFWLRQ 5HVXOW

6SHHFK

5HIHUHQFH

0XOWLSOH 3URQXQFLDWLRQ 0RGHO

5HIHUHQFH

Fig. 1 System overview of detecting reading miscues

3 Multiple Pronunciation Model The multiple pronunciation model contains all possible pronunciations (including the normative pronunciation and pronunciation variances) or words. And it quantifies the occasion of each pronunciation v of word w with a probability PMP (v|w). It can be derived by linguistic analysis [3], data-driven approaches or their combination [6]. In our system, the MPM is generated by a data-driven approach. An example of the multiple pronunciation model is shown in Fig.2. For the training of the multiple pronunciation model, a lot of reading speech by language learners is collected and then, the actual pronunciations of

382

C. Liu et al.

Fig. 2 An example of multiple pronunciation model

Ӊ ch ang2 0.59 Ӊ zh ang3 0.41 ӉӬ ch ang2 ch eng2 0.85 ӉӬ zh ang3 ch eng2 0.15 ......

the corpus is transcribed by human experts. After aligning the transcriptions and the references by DP, we count all pronunciations of each word and compute its pronunciation probability by the following equation. PMP (v|w) =

count of w with pronunciation v count of all word w

Several methods of exploiting MPM are introduced in [8]. A common method is to add all pronunciation variants to the pronunciation dictionary. However, it will not work well here. In our system, the dictionary has more than 40000 vocabularies and the multiple pronunciations have a large range because of readers’ different knowledge backgrounds. If all pronunciation variants are added, the search space will become very large and the increase of perplexity will reduce the accuracy of the recognition and impair the performance of detecting reading miscues. Considering both the pronunciation variants coverage and the perplexity of the search space, the algorithm DMPI is proposed.

4 Dynamic Multiple Pronunciation Incorporation In this algorithm, we did not add all pronunciation variants into the search space, but only those belong to words of current reference. Before decoding, they are extracted from the multiple pronunciation model based on analysis of current reference, and then added to the search space. Other pronunciation definitions in the search space keep unchanged. Thus, the perplexity increases only slightly. Before adding the pronunciation variants to the search space, which is a state network in our system, they will be converted to state paths. Then, the state paths are merged into the original state network.

4.1 The State Network The state network is pre-compiled from a linear lexicon and phonetic decision trees. The linear lexicon defines a list of words with their monophone sequence. Together with phonetic decision trees which give a clustering of the HMM states, the linear lexicon is finally transferred into a state network. The state network is then optimized by forward and backward node-merging

Dynamic Multiple Pronunciation Incorporation

383

Fig. 3 A three-word linear lexicon

࣫Ҁ%HLMLQJ EHLMLQJVS ⃶䖢:HOFRPH KXDQ\LQJVS Դ
HLML

LQJE 

LE

LQJEHL      

 5227

LQE





 

 



 



࣫Ҁ 

MLE

 



 ML





LEHL



 

 







फҀ LQ







LQQ

MLQJ

 LQJQDQ



 



 LQDQ





  

 QDQM

 



 

 

 MLQ 

HLMLQJ

LQJQ



࣫ᵕ

EHLM

 MLQJE 

 

 









VS

ZH





VS

ZH





VS

ZH





VS

ZH



 DQMLQJ

 



MLQJQ  

 

Fig. 4 The state network for the three-word lexicon

process. A three-word linear lexicon and its final state network are shown in Fig.3 and Fig.4. The state network is a left-to-right graph and structured in three parts: individual mid-part, shared initial part (fan-in triphones) and final part (fanout triphones). There are two types of nodes in the network: state nodes and dummy nodes. Each state node is associated with an HMM state index, while dummy node is not related to any acoustic event and used to compress network or mark special situations. In Fig.4, symbol “” denotes state node and symbol “” denotes dummy node. Some important features about the state network are listed below: • Two layers of dummy nodes: FI nodes and FO nodes are introduced to cluster the cross-word (CW) context dependent fan-in and fan-out triphones; • A so-called “WI layers” are introduced to store the word identities and putting the nodes of this layer in the non-shared mid-part of the network; • Optimizing the network at the state level by a sufficient forward and backward node-merging process; • A language model (LM) look-ahead model is built on the state network to factor the LM probabilities over the state nodes.

384

C. Liu et al.

4.2 Algorithm of Dynamic Multiple Pronunciation Incorporation The liner lexicon added with an additional pronunciation is shown in Fig.5. The additional gray rectangle denotes that the word Кࣘ may also be pronounced as “bei3 jin1”. The insertion of pronunciation variants to the state network is done one by one. The state path which is converted from the additional pronunciation in figure 5 is shown in Fig.6. The state path is a mini state network. In order to merge the two state networks, we make some change during the building of the state network. In the original state network, only the essential FI and FO nodes are added. That is to say, if the CW triphone does not exist in the linear lexicon, the corresponding FI or FO node will not appear in the state network. Thus, if the mini state network has some FI or FO nodes which do not exist in the original state network, the merge of them will be very difficult. Therefore, we reserve the the whole set of FI and FO nodes when building the state network, no matter whether the corresponding triphones exist in the lexicon or not. Thus, some FI or FO nodes may be unconnected in the modified state network ,such as nodes 46 and 49 in figure 4. In our system of recognizing Chinese Mandarin, the total tonal mono phone number is 213, including 27 initials, 184 finals, a ‘sp’ and a ‘sil’. The total number of FI or FO nodes is 4968. In figure 4, not all the fan-in and fan-out nodes are listed due to space reason. ࣫ᵕQRUWKSROH EHLMLVS

Fig. 5 The linear lexicon which is inserted with an additional pronunciation of Кࣘ

࣫Ҁ%HLMLQJ EHLMLQJVS ࣫Ҁ%HLMLQJ EHLMLQVS फҀ1DQMLQJ QDQMLQJ 6(17B3$86( VLO

6(17B67$57 VLO 6(17B67$57 VLO

࣫Ҁ%HLMLQJ EHLMLQ

LQE 

LQEHL 

 







 







 

 

 



MLQJ







  

MLQE 











VS

ZH

 MLQQ  











VS

ZH

ĂĂ

ĂĂ

LQQ 

 LEHL  

 

 LE



HLMLQ

EHLM

࣫Ҁ

Fig. 6 A pronunciation variant and its corresponding mini state network

Dynamic Multiple Pronunciation Incorporation

385

When transferring a pronunciation definition to a mini state network, the whole set of FI and FO nodes are also built in. The derived state network contains all CW triphones. Similarly with the large state network, there are also some unconnected nodes in the mini state network, such as FI nodes: 76, 77 and FO node: 71. The corresponding triphone ‘i2-b+ei3’ and ‘j-in1+n’ do not exist in the pronunciation sequence of ’b-ei3-j-ing1’. They take a role of placeholder. With them, the merge of the large state network and the mini state network will become very easy. The merging process is operated in three steps: Step 1: For each FI node in the additional state network, move all its preceding and successive links to the same node in the original state network and remove the redundant links. Step 2: For each FO node in the additional stat network, move all its preceding and successive links to the same node in the original state network and remove the redundant links. Step 3: Link the ROOT node to all the first state nodes of the additional state network. The final merged state network is illustrated in Fig.7. The grey nodes belong to the additional state network and the dotted lines are the new links to the additional nodes. All multiple pronunciations referred to in the reference will be added to the original state network as the step above. Another problem of the additional state network is that they do not have corresponding LM look-ahead model. TO resolve this problem, the additional state network will not be optimized by forward and backward merging and the WI node of each pronunciation path are regularly put at the begin of the state sequence. When decoding,

HLML

LQJE

࣫ᵕ

  LQE

EHLM

LQJEHL 

 







 



LEHL

 

LE





 ࣫Ҁ

MLE



 

  







ML









MLQ 

HLMLQJ





5227 

LQJQ







LQ



फҀ





 











LQQ





LE 





DQMLQJ



QDQM



LQDQ



EHLM 

 

  LEHL 





MLQJQ 

࣫Ҁ 

HLMLQ

  



 

 ZH





VS

ZH





VS

ZH

 









 VS













 





  

LQEHL LQE

MLQJ

 LQJQDQ

 MLQJE

 

 

 ZH

MLQE 



 VS



MLQJ

















 LQQ 

Fig. 7 The final state network after merging

 



 MLQQ





VS

ZH





VS

ZH

  



386

C. Liu et al.

the decoder can obtain the LM probability as soon as it appears in this path and do not need the LM look-ahead probability. All pronunciations variants of the same word share the same language model probability during the decoding process because they stand for the same word identity. Their pronunciation probability PMP (v|w) is introduced and the word probability will be computed according to the equation below: P (w|o)  PAM (o|v) · PMP (v|w) · PLM (w) where PMP (v|w) is the probability of the pronunciation v given word w, and PAM (o|v) is the acoustic model probability of pronunciation v given observation vector o. PMP (v|w) is the probability of pronunciation v given word w, which is referred to in section 3.

5 Experiment 5.1 Data Corpus and Evaluation Metrics We use a data corpus of Hongkong college students’ reading Chinese Mandarin to evaluate the algorithm DMPI. The reading materials are four passages from “Hongkong Putonghua Shuiping Kaoshi” (PSK). Each of 665 students read one of the four passages. All speech were transcribed with syllable sequences (pin yin) by Chinese native teachers. The whole corpus is divided into two parts: 100 paragraphs as test set and the rest as develop set. The develop set is used to adapt the acoustic model and train the multiple pronunciation model. The test set is used to evaluate performance of the algorithm. We use three metrics to measure the performance of the system: syllable recognition error rate (SER) is used to evaluate the accuracy of recognition; miscue detection error rate (DER) and false alarm rate (FAR) are used to evaluate the detecting performance. SER is computed by comparing the 1best recognizing result with the transcription using DP. DER is defined as the number of miscues which have not been detected divided by the number of all miscues; and FAR is defined as the number of words erroneously detected as reading miscues divided by the number of all miscues. The real miscue information is obtained by aligning the reference with the transcription. Different points of DER and FAR can be obtained by changing some parameters in the system. They can be plotted as a detection error tradeoff (DET) curve. For the convenience of comparison, equal error rate (EER) is also used, which is the rate when the DER equals to FAR.

5.2 Experiment Setup The front end of the system extracts 39 dimensions MFCC feature, including 12 dimensions static cepstrum and 1 dimension energy, with their 1st and

Dynamic Multiple Pronunciation Incorporation

387

2nd derivatives. The HMM acoustic model was trained on about three hundred hours of speech from Chinese native speakers, then was adapted by the develop set of the Hongkong corpus referred to in section 5.1 with a MAP algorithm. The final model has about five thousand states and each state has 32 Gauss mixtures. The language model is a general tri-gram model which was trained on about 2G bytes text materials from news and internet. The dictionary has about forty thousand words, including all characters and most words of Chinese Mandarin.

5.3 Experimental Result In this section, performance of the algorithm DMPI is investigated. The baseline system is a conventional LVCSR decoder followed by a confusion network alignment module, without the multiple pronunciation model. The multiple pronunciation model is trained on the develop set referred to in section 5.1. Table 1 shows the recognizing and detecting performance of DMPI compared with baseline. The recognizing SER is reduced by 9.5% relatively and the detecting EER decreases by 9.3% respectively, which proves the effectiveness of DMPI. The DET curves of DMPI is illustrated in Fig.8. Compared with the baseline, the detecting performance is significantly improved. The learner’s poor language level results in many arbitrary reading miscues. There are many pronunciations that do not exist in the original Table 1 Recognizing and detecting performance of DMPI SER EER baseline 31.0% 46.3% DMPI 27.8% 42%

Fig. 8 DET Performance of the algorithm DMPI compared with baseline

388

C. Liu et al.

pronunciation dictionary corresponding to some word. They may be called out-of-pronunciations (OOP). The additional pronunciations from the multiple pronunciation model resolve these OOPs. They give the decoder more choices in the decoding process. The dynamically incorporation algorithm hardly increase the perplexity of the search space but it can cover the most necessary error predictions. The decoding time after incorporating additional pronunciations does not increase compared with the baseline.

6 Conclusion This paper proposed an algorithm of Dynamic Multiple Pronunciation Incorporation (DMPI) to incorporate the error prediction into the decoder of a conventional speech recognizer. The algorithm solves the confliction between the coverage of errors and the perplexity of search space. It dynamically adds the pronunciation definition paths into the search space, which increases the perplexity slightly. We redeveloped the the original state network to reserve some redundant fan-in and fan-out nodes. These placehold nodes make the merge of the original state network and the additional state network very easy. The experiment result proved the effectiveness of this algorithm. The EER of miscue detecting error rate and false alarm rate is decreased by 9.5%. The Dynamic Multiple Pronunciation Incorporation is a novel method to apply the multiple pronunciation model. It is a general framework, not limited by special multiple pronunciation model, or special searching space – state network. In our system, the multiple pronunciation model is a statistical multiple pronunciation dictionary. Some more complex models also can be used, such as decision trees, artificial neural network (ANN), etc. [8]. The decoding space can also has be other types, such as phonetic graphs or weighted finite-state transducers (WFST), etc.

References 1. Witt, S.M., Young, S.J.: Phone-level Pronunciation Scoring and Assessment for Interactive Language Learning. Speech Communciation 30, 95–108 (2000) 2. Leonardo, N., Horacio, F., Vassilios, D., Mitchel, W.: Automatic Scoring of Pronunciation Quality. Speech Communciation 30, 83–93 (2000) 3. Jack, M., Steven, F.R., Alexander, G.H., Matthew, K.: A Prototype Reading Coach that Listens. In: Proceedings of the Twelfth National Conference on Artificial Intelligence (AAAI 1994), Seattle, WA (1994) 4. Daniel, B., Wayne, W., Sarel, V.V., Javier, G.: Syllable Lattices as a Basis for a Children’s Speech Reading Tracker. In: InterSpeech (2007) 5. Jacques, D., Mari, W., Kris, D., Hugo, V.H.: A Flexible Recogniser Architecture in a Reading Tutor for Children. In: InterSpeech (2006) 6. Hongcui, W., Tatsuya, K.: Effective Error Prediction Using Decision Tree for Grammar Network in Call System. In: ICASSP (2008)

Dynamic Multiple Pronunciation Incorporation

389

7. Shao, J., Li, T., Zhang, Q., Zhao, Q., Yan, Y.: A One-Pass Real-Time Decoder Using Memory-Efficient State Network. IEICE Transactions on Information and Systems 91, 529–537 8. Helmer, S., Catia, C.: Modeling Pronunciation Variation for ASR: A Survey of the Literature. Speech Communication 29, 225–246 (1999) 9. Lidia, M., Eric, B., Andreas, S.: Finding Consensus in Speech Recognition: Word Error Minimization and Other Applications of Confusion Networks. Computer Speech and Language 14, 373–400 (2000)

Depicting Diversity in Rules Extracted from Ensembles Fabian H.P. Chan, A. Chekima, Augustina Sitiol, and S.M.A. Kalaiarasi 1

Abstract. Ensembles are committees of neural networks used to achieve better classification and generalization accuracy in contrast to just using a single neural network. Traditionally incomprehensible models such as artificial neural networks presents a problem in its application to safety critical domains where not only accuracy but also model transparency is a requirement. This problem is not only inherited but further multiplied in ensembles. Furthermore, the aspect of diversity by which ensembles achieve improved classification ability cannot be reflected in rule extraction methods designed for single neural networks hence the need for rule extraction methods specifically designed for ensembles. This paper presents a decompositional rule extraction algorithm for ensembles which is able to approximately decompose the neural networks in an ensemble and reflect their collective diversity to identify significantly contributing inputs. Keywords: Safety critical domain, Ensemble neural network, Rule extraction, Comprehensibility, Data mining.

1 Introduction Safety critical domains refer to an area whereby a mistake on the application of programmable systems could lead to injury, death or other serious adverse effects on people. Healthcare or medical systems are an obvious example of a safety critical domain [1]. In order to facilitate the application of advanced computation models such as neural networks it is important therefore that these models can be tested and verified. Verification typically involves examination of the underlying logic of such systems or models by human experts of the related domains. Hence the need of humanly comprehensible rules which can be extracted from incomprehensible Fabian H.P. Chan . A. Chekima . Augustina Sitiol . S.M.A. Kalaiarasi School of Engineering and Information Technology Universiti Malaysia Sabah 88999 Kota Kinabalu Sabah Malaysia [email protected] H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 391–400. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

392

F.H.P. Chan et al.

models such as neural networks via rule extraction algorithms to allow transparency to the model and subsequent verification. Ensemble neural networks which are made up of multiple single neural networks achieve its improved classification and generalization ability through the diversity of its members [2,3]. As long as each singular neural network in the ensemble is able to classify sufficiently on different areas of the data, their combined output tends to suppress individual errors of any particular network. This implies that the output of an ensemble for any instance of data may reflect the decision of all or only a subset of the ensemble members. The aspect of diversity in ensembles is a scope for which single neural network rule extraction algorithms are not designed to handle. The rule extraction method presented in this paper attempts to address this aspect by extracting rules which retains the frequency of input usage across the positively voting components of the ensemble. The ensemble rule extraction approach used in the design of this algorithm aims for a decompositional rule extraction algorithm in which the criterion emphasized upon is the fidelity of the rules extracted. An explanation of common rule extraction algorithm categorization and criteria is given in [4].

2 Ensemble Rule Extraction The ensemble rule extraction approach used in this paper divides the ensemble rule extraction process into two stages. The first stage called the component stage, deals with rule extraction for the individual network components used in the ensemble. The second stage is the combination stage which combines the rules obtained from the component stage in a relevant way to represent the voting combination method used in the ensemble. Because the components that contribute to the output of the ensemble can potentially vary for each instance of data, the rule extraction is performed on a case by case basis using one instance of data at a time. An explanation of the merit of this rule extraction approach for ensembles is given in [5].

2.1 Component Stage The basis for the rule extraction method employed here follows from the fact that artificial neural networks, being modeled after real neurons have activation functions in each neuron with a firing threshold. The output of a neuron y can be written in formula as expressed in equation (1).

⎛ n ⎞ y = S ⎜ ∑ wi pi + b ⎟ . ⎝ i =1 ⎠

(1)

where S is the activation function, wi and pi are respectively the i-th weights and inputs to the current neuron and b is the bias or threshold. The rule extraction algorithm employed here tries to find the minimum subset of links that contributes to a neuron firing for any given input. The algorithm does

Depicting Diversity in Rules Extracted from Ensembles

393

this by incrementally eliminating the smallest absolute valued weight and input product shown in (2) which still allows the neuron to fire.

wi ⋅ pi .

(2)

By repeatedly finding the minimum subset of links from each neuron identified by a minimum subset of links in the previous layer, the significant input neurons that contribute most to the network output can be identified. This rule extraction algorithm is applied to individual networks at the component stage of the ensemble rule extraction. The algorithm is given in Fig 1. 1 2 3 4 5

Start with the output neuron in the output layer of the network. Find the minimal subset of weight links that is overcome the neuron threshold by incrementally removing the least absolute valued input weight product. Use the minimal subset to identify contributing neurons in the hidden layer For each identified contributing neuron, repeat step 2 and represent the results in a binary vector representing which inputs are used. Combine the binary input vectors obtained in step 4 with a logical OR function.

Fig. 1 Component Stage Rule Extraction Algorithm

In the following example, a neural network was trained for the Pima Indian Diabetes dataset [5]. The architecture of the network is shown in Fig. 2. Fig. 2 Network Architecture

The weights and biases for the network are given in the following where wij represents the weights between the input and hidden layer and vjk represents the weights between the hidden and the output layer. bj and ck represent the biases for the neurons in the hidden and output layer respectively. Input to hidden layer weights (wij), w13 = -0.5679 w11 = -0.0363 w12 = -1.5289 w23 = 0.9485 w21 = 2.0053 w22 = 0.2514 w33 = -1.2278 w31 = 0.4351 w32 = 0.8049 w41 = -0.1003 w42 = -1.5084 w43 = 1.3426 w53 = 1.3842 w51 = 2.0678 w52 = -0.2926

394

F.H.P. Chan et al.

w61 = 1.0435 w62 = -1.3746 w71 = 2.0320 w72 = 1.4391 w81 = 0.8230 w82 = -0.2552

w63 = 0.7555 w73 = 1.7399 w83 = -0.4560

Hidden to output layer weights (vjk): v11 = 3.3256 v21 = 2.3636 v31 = -2.5335 Hidden nodes bias (bj): b1 = 2.4522 b2 = -0.5033 b3 = -3.1981 Output node bias (ck): c1 = -3.2974 The neurons in the hidden and output layer use a log-sigmoid activation function as given in equation (3).

f ( x) =

1 . 1 + e−x

(3)

The following network inputs (Pi) is one instance of inputs taken from the Pima Indian Diabetes dataset which have already been bipolar normalized using the formula in (4).

f ( x) =

(1 − (−1))( x − x min ) − 1. x max − x min

(4)

[p1, p2, p3, p4, p5, p6, p7, p8] = [-0.3846, 0.1980, 0.1475, -1, -1, 0.6182, -0.2701, - 0.7727]

Applying equation (1) to each of the neurons in the hidden layer (hj) we get the following,

⎛ n ⎞ h j = S ⎜ ∑ wij ⋅ pi + b j ⎟ ⎝ i =1 ⎠ h1 = S(0.014+0.397+0.0642+0.1003 -2.0678+0.6451-0.5488-0.6359 +2.4522) = S(0.4203) = 0.6036 h2 = 0.7336 h3 = 0.047

Similarly, applying the same equation to the neuron in the output layer (Ok) we get the following output for the network, O1 = S(2.007+1.733-0.1191-3.2974) = 0.5802 The rule extraction begins by looking at the output neuron where the inputs are the outputs of the neurons in the hidden layer as shown in the earlier feedforward calculations of the network. The next step attempts to find a minimal subset of absolute valued weight and input product links necessary to overcome the neuron threshold which is expressed in equation (5). Find minimal subset of links so that, n

∑v j =1

jk

h j + ck ≥ 0 .

(5)

Depicting Diversity in Rules Extracted from Ensembles

395

From the example, v11h1 + v21h2 + v31h3 + c1 = 2.007 + 1.733 + -0.1191-3.2974 = 0.3235 The algorithm attempts to find the minimal subset by incrementally removing the smallest absolute valued weight input product link. Smallest |vjkhj| is: |v31h3| = |-0.1191| = 0.1191 Removing v31h3 will yield: v11·h1 + v21h2 + c1 = 2.007 + 1.733 -3.2974 = 0.4426 Smallest | vjkhj | is: |v21h2| = 1.733 Removing v21h2 will yield: v11h1 + c1 = 2.007 + -3.2974 = -1.2904 When equation (5) is no longer satisfied the algorithm stops and uses the last subset which still satisfied the equation as the minimal subset of links. The minimal subset of links identifies the significantly contributing neurons in the previous layer which in the example are h1 and h2. The next step of the algorithm repeats the minimal subset search for each of the hidden neurons identified. Output at h1 is given by, w11p1 + w21p2 + w31p3 + w41p4 + w51p5 + w61p6 + w71p7 + w81p8 + b1 = 0.014 + 0.397 + 0.0642 + 0.1003 - 2.0678 + 0.6451 - 0.5488 - 0.6359 + 2.4522 = 0.4203 Smallest |wijpi| is: |w11p1| = 0.014 Removing w11p1 will yield: w21p2 + w31p3 + w41p4 + w51p5 + w61p6 + w71p7 + w81p8 + b1 = 0.397 + 0.0642 + 0.1003 - 2.0678 + 0.6451- 0.5488 - 0.6359 + 2.4522 = 0.4063 Smallest |wijpi| is: |w31p3| = 0.0642 Removing w31p3 will yield: w21p2 + w41p4 + w51p5 + w61p6 + w71p7 + w81p8 + b1 = 0.397 + 0.1003 - 2.0678 + 0.6451 - 0.5488 - 0.6359 + 2.4522 = 0.3421 Smallest |wij pi| is: |w41p4 | = 0.1003 Removing w41p4 will yield: w21p2 + w51p5 + w61p6 + w71p7 + w81p8 + b1 = 0.397 - 2.0678 + 0.6451 - 0.5488 - 0.6359 + 2.4522 = 0.2418 Smallest |wijpi| is: |w21p2| = 0.397 Removing w21p2 will yield:, w51p5 + w61p6 + w71p7 + w81p8 + b1 = -2.0678 + 0.6451 - 0.5488 - 0.6359 + 2.4522 = -0.1552

396

F.H.P. Chan et al.

The algorithm stops here because equation (5) is no longer satisfied. In order to combine the results of different neurons in the hidden layer, the algorithm represents the significant inputs in a binary vector with 1 denoting significant and 0 as insignificant. For h1: [p1 p2 p3 p4 p5 p6 p7 p8] = [0 1 0 0 1 1 1 1] By applying the same procedure to H2 we obtain: [p1 p2 p3 p4 p5 p6 p7 p8] = [0 0 0 1 0 0 0 0] Combining both vectors with a logical OR function yields: [0 1 0 0 1 1 1 1] ∨ [0 0 0 1 0 0 0 0] = [0 1 0 1 1 1 1 1] The final output for the component rule extraction stage is this binary vector which represents significant inputs used in that particular neural network component.

2.2 Combination Stage The combination stage of an ensemble combines the outputs from the neural network components to produce the final output of the ensemble. The rule extraction in this stage attempts to take into consideration how the ensemble achieves this combination and reflects it in the final rules that is produced. The algorithm for the combination stage rule extraction is presented in Fig 3. 1 2 3

For each network component that votes positively towards the ensemble output, perform component stage rule extraction. Sum the binary input vectors from the components to obtain an input usage indication. Convert the input usage into rules.

Fig. 3 Combination Stage Rule Extraction Algorithm

In the following example an ensemble with 5 network components are used. The ensemble uses network components trained for the Pima Indian Diabetes dataset with a majority vote as the combination method to produce the final output [5]. Table1 shows the voting results for 5 data instances put into the ensemble and Table 2 shows the binary input matrices for each network component. Table 1 Voting results of each component Network Components 1 2 3 4 5

Votes for 5 data instances 1 1 1 1 1

0 0 0 0 0

1 1 0 1 0

1 1 1 1 1

0 0 0 0 0

Depicting Diversity in Rules Extracted from Ensembles

397

Table 2 Component binary input matrices Network Components Binary input matrices for the 5 data instances

1

2

3

4

5

11000110 11111111 0 0 0 1 0 00 1 11000010 11111111

11111111 11111111 00011010 00000000 11111111

11111111 11111111 11111111 00000000 11111111

11101111 11111111 11111111 00000000 11111111

00000000 11111111 11111111 0 00 0 0 0 0 0 11111111

Since the Pima Indian Diabetes dataset has a binary target, we are only interested in the positive outputs of the ensemble. Therefore the rules are only extracted when the ensemble outputs a positive result. Step 1 of the algorithm looks for components that vote positively towards the ensemble output. Table 1 shows that out of the 5 data instances entered into the ensemble only 3 of them produces positive outputs which are data instances 1, 3 and 4. In the first instance of data entered into the ensemble all the components voted positively. The corresponding binary input vectors from Table 2 for each positively voting component in this instance is then summed to obtain an input usage indication. Input usage indication at data instance 1, = Component1 + Component2 + Component3 + Component 4 + Component5 = [1 1 0 0 0 1 1 0] +[1 1 1 1 1 1 1 1] + [1 1 1 1 1 1 1 1] + [1 1 1 0 1 1 1 1] + [0 0 0 0 0 0 0 0] = [4 4 3 2 3 4 4 3] Input usage indications are similarly obtained at each data instance which produces a positive result. Input usage indication at data instance 3, = Component1 + Component2 + Component 4 = [0 0 0 1 0 0 0 1]+[0 0 0 1 1 0 1 0]+ [1 1 1 1 1 1 1 1] = [1 1 1 3 2 1 2 2] Input usage indication at data instance 4, = Component1 + Component2 + Component3 + Component 4 + Component5 = [1 1 0 0 0 0 1 0]+[0 0 0 0 0 0 0 0]+ [0 0 0 0 0 0 0 0]+[0 0 0 0 0 0 0 0]+ [0 0 0 0 0 0 0 0] = [1 1 0 0 0 0 1 0] IF att1(4)=6 AND att2(4)=148 AND att3(3)=72 AND att4(2)=35 AND att5(3)=0 AND att6(4)=33.6 AND att7(4)=0.627 AND att8(3)=50 THEN 1 IF att1(1)=10 AND att2(1)=115 AND att3(1)=0 AND att4(3)=0 AND att5(2)=0 AND att6(1)=35.3 AND att7(2)=0.134 AND att8(2)=29 THEN 1 IF att1(1)=2 AND att2(1)=197 AND att7(1)=0.158 THEN 1 Fig. 4 Final extracted rules

398

F.H.P. Chan et al.

Each of the input usage matrix are then converted into a rule showing the original data of that particular instance with the frequency of which a particular attributewas referenced in the ensemble denoted in brackets. The rule set produced from the 3 data instances are given in Fig 4 .

3

Experimentation

This section gives the experiment setup and results for the rule extraction algorithm applied on diabetes dataset. The details of the dataset are first given followed by the ensemble creation procedure and finally the rule extraction results applied on the ensembles. The datasets used in the experiment were obtained from the UCI Machine Learning Repository [6]. This dataset have been commonly used in other literature and the availability of this datasets to other researchers allows experiments to be replicated or compared. The details of a few relevant aspects of the datasets used in this experiment are given in Table 3. Table 3 Dataset details Aspect Number of instances Class type Class Distribution Attributes Division for Train, Test and Validation set (%)

Description/Value 768 Binary 0 = 500, 1 = 268 8 Train = 60, Test = 20, Validation = 20

3.1 Ensemble Training The neural network components for the ensemble were trained using feedforward backpropagation neural networks with gradient descent and momentum. The activation functions used in the hidden and output layers were both log-sigmoid. Parameters were selected through successive network trainings using different sets of parameters. The final parameters used are given in Table 4. Table 4 Network training parameters used Parameter Hidden Neurons Learning Rate Momentum Constant

Value 10 0.2 0.6

Network component generation used simple bootstrapping to increase diversity between components. A pool of 100 network components was generated. Genetic

Depicting Diversity in Rules Extracted from Ensembles

399

algorithm was then used to optimize a selection of 5 components to constitute the ensemble. Genetic algorithm parameters are given in Table 5. The ensemble creation procedure was repeated 5 times for diabetes dataset thus giving 5 separately trained ensembles from their own pool of 100 network components. Table 5 Genetic algorithm parameters Parameter Generations Population Size Crossover Fraction Migration Interval Migration Fraction Mutation

Value 100 20 0.8 0.8 0.2 Adaptive

3.2 Ensemble Rule Extraction Results The rule extraction algorithm was applied on each of the ensembles created. The experimentation results are given in Table 6 and Table 7. Each table gives the overall accuracy of the ensemble, the classification accuracy of the rules, and the rule fidelity. Table 6 Results for Pima Indian Diabetes No

Ensemble Accuracy

Rule Accuracy

Rule Fidelity

1 2 3 4 5

80.599 80.0781 80.0781 79.9479 79.8177

73.8281 69.9219 67.0573 64.7135 76.1719

90.625 84.8958 78.125 74.6094 95.5729

The results from Table 6 shows that the rule extraction algorithm can produce rule sets with relatively good accuracy although this ability is not entirely consistent as can be seen in the rule extraction results for the third ensemble. A possible exception to this algorithm causes too many links to be removed thus reducing the fidelity and accuracy in the rules obtained. By implementing a limit on the minimum number of links to retain however the rule extraction results improved greatly as can be seen in Table 7 which shows the rule extractions results with the algorithm modified to retain at least two links. Although the rule extraction results from Table 7 shows near perfect results, the rule comprehensibility has suffered greatly and nearly all the rules have as many antecedents as inputs to the network.

400

F.H.P. Chan et al.

Table 7 Results for Pima Indian Diabetes No

Ensemble Accuracy

Rule Accuracy

Rule Fidelity

1 2 3 4 5

80.599 80.0781 80.0781 79.9479 79.8177

80.599 80.0781 80.0791 79.9479 79.8177

100 100 100 100 100

4 Conclusion This paper has presented and illustrated a novel ensemble rule extraction algorithm capable of addressing an important aspect of ensembles which is diversity. The rules extracted by this algorithm depict diversity by showing the frequency of input usage used by the ensemble neural network components in producing its output. The application results of this algorithm on diabetes dataset is presented and although the rule extraction results is not shown to be entirely consistent, further experimentation with simple constraints show that sacrificing rule comprehensibility will allow higher fidelity. A possible solution to address this issue would be to accept that comprehensibility and fidelity are inversely related and therefore design an approach to rule extraction that allows the user to choose between them.

References 1. Wall, R., Cunningham, P.: Exploring the Potential for Rule Extraction from Ensembles of Neural Networks. In: 11th Irish Conference on Artificial Intelligence & Cognitive Science, pp. 52–68 (2000) 2. Tsymbal, A., Pechenizkiy, M., Cunningham, P.: Diversity in Ensemble Feature Selection. Technical report, Trinity College Dublin (2003) 3. Opitz, D., Shavlik, J.: Actively Searching for an Effective Neural-Network Ensemble. Connection Science 8, 337–353 (1996) 4. Tickle, A., Golea, M., Hayward, R., Diederich, J.: The Truth Is in There: Current Issues in Extracting Rules from Feedforward Neural Networks. In: Proceedings of the International Conference on Neural Networks, vol. 4, pp. 2530–2534 (1997) 5. Wall, R., Cunningham, P., Walsh, W.: Explaining Predictions from a Neural Network Ensemble One at a Time. In: Elomaa, T., Mannila, H., Toivonen, H. (eds.) PKDD 2002. LNCS, vol. 2431, pp. 449–460. Springer, Heidelberg (2002) 6. Asuncion, A., Newman, D.J.: UCI Machine Learning Repository. University of California, Department of Information and Computer Science, Irvine (2007), http://www.ics.uci.edu/~mlearn/MLRepository.html

A New Statistical Model for Radar HRRP Target Recognition Qingyu Hou, Feng Chen, Hongwei Liu, and Zheng Bao 1

Abstract. To resolve the problem of how to determine the proper number of the mixture models for radar high-resolution range profile (HRRP) target recognition. This paper develops a variational Bayesian mixture of factor analyzers (VBMFA) model. This method can automatically determine the optimal number of models by birth-death moves and can accurately describe the statistical characteristics of HRRP. So the VBMFA method should have better recognition performance than factor analysis and mixtures of factor analyzers method, and experimental results for measured data proved this conclusion. Keywords: Radar automatic target recognition (RATR), High-resolution range profile (HRRP), Variational Bayesian mixtures of factor analyzers (VBMFA), Variational Bayesian(VB), Mixtures of factor analyzers (MFA).

1 Introduction A high-resolution range profile (HRRP) contains the target structures, such as target size, scatterer distribution, etc. Several issues have to be considered when HRRP is applied to radar automatic target recognition (RATR). The first one is the well-known target-aspect sensitivity of HRRP [1-4], and one effective approach to overcome this problem is to establish a template database by dividing HRRPs from all target aspects into several target-aspect frames. The second one is the time-shift sensitivity of HRRP. In order to decrease the computation complexity, an HRRP is only a part of received radar echo extracted by a range window, in which target signal is included. Thus the position of the target signal in an HRRP can vary with the measurement. The last one is the amplitude-scale sensitivity of HRRP. This comes from the fact that the intensity of an HRRP is a function of radar transmitting power, target distance, radar antenna gain, radar receiver gain, radar system losses and so on. Thus, HRRPs measured by different radars or under different conditions will have different amplitude-scales. Therefore, it is a prerequisite of HRRP-based statistical recognition to deal with the three sensitivity Qingyu Hou . Feng Chen . Hongwei Liu . Zheng Bao National Laboratory of Radar Signal Processing, XiDian University, 710071 Xi’an, China [email protected] H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 401–409. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

402

Q. Hou et al.

problems of HRRP. Reference [2], [3] and [4] make a detailed analysis of the effect of the sensitivity problems of HRRP on statistical recognition, and propose a corresponding solution. After solving the above three sensitivity problems of HRRP, the main challenging task is how to accurately describe HRRPs statistical characteristics. Based on the hypothesis that the range cells of an HRRP approximate to be statistically independent, [3] developed a statistical model comprising two distribution forms to model echoes of different types of range cells as the corresponding distribution forms. According to [4], this hypothesis is not totally true, [4] discussed a more accurate statistical recognition methods based on HRRP samples’ jointly statistical characteristics. Three different joint-Gaussian models, Principle Component Analysis (PCA) model, Probabilistic Principle Component Analysis (PPCA) model, and Factor Analysis (FA) model were discussed in [4]. The recognition performance of PPCA model is better than PCA model, as the PPCA model considers both signal subspace and noise subspace. The base vectors of signal subspace must be orthogonal in both of the above two models. As the variables of Gaussian with zero mean are independent if they are orthogonal, the latent variables of those two models are independent each other. But actually independent variables are not always orthogonal, PCA model and PPCA model may not the best statistical model for HRRP. FA model just need the latent variables are independent, so it could describe HRRPs statistical characteristics accurately. Mixture of Factor Analyzers (MFA) is applied to this paper, as radar HRRP data is usually the multiple models. But how to determine the proper number of the models is a problem [5], [6]. A simple solution is to fix the number of mixture models, usually could improve the recognition performance, but not strict. Via variational Bayesian theory [7], [6] presented an algorithm of variational inference for Bayesian Mixtures of Factor Analyzers (VBMFA) that can automatically determine the optimal number of models by birth-death moves. This paper we apply the VBMFA to model radar HRRP data, it can describe HRRPs statistical characteristics accurately.

2 VBMFA in Radar HRRP Statistical Recognition Factor analysis (FA) is a method for modeling the joint-Gaussian distributed correlations in multidimensional data, but radar HRRP samples do not satisfy the joint-Gaussian distributed strictly. Therefore, FA mixture model, which is a jointGaussian mixture model, can describe HRRPs statistical characteristics accurately. Assume we have a mixture of m factor analyzers indexed by ws , s = 1,2, , m . In radar automatic target recognition, the generative MFA model of a ddimensional HRRP sample in the

Tk (k = 1,2,

K ) is given by

l th (l = 1,2,

(

Lk ) aspect-frame of the target

)

p( xkli | Θ kl ) = ∑ s =1 p xkli | μ kls , Akls ,Ψ kl p(ws ) m

(1)

A New Statistical Model for Radar HRRP Target Recognition

where parameter tions,

{

}

Θkl = (μkls , Akls , π s )s=1,Ψkl , π s m

403

is the vector of mixing propor-

π s = p(ws ) . Here we define that each model has a same Ψ kl

(

)

p xkli | μ kls , Akls ,Ψ kl = (2π )

−d 2

Ψ kl + Akls Akls

)(

T T ⎛ 1 × exp⎜ − xkli − μ kls Ψ kl + Akls Akls ⎝ 2

(

T −1 2

)( −1

matrix. And

(2)

)

⎞ xkli − μ ⎟ ⎠ s kl

But how to determine the proper number of the models-m is a problem. We can fix the number of models in our MFA model like [8], this makes a simple computation, but is not reasonable. And the method in [8] utilized the maximumlikelihood (ML) method, which fails to take into account model complexity. So this may lead to overfitting and the inability to determine the best model size and structure without utilizing costly cross-validation procedures. To deal with the problem of overfitting, we apply the Bayesian approach, which is often computationally and Bayesian integration is usually intractable in practice. So some different approximation method were developed [9], [10], such as Markov chain Monte Carlo (MCMC) methods and the Laplace approximation. And reference [6] developed a third method, variational Bayesian (VB) inference. We can obtain a lower bound on the log evidence using the Jensen’s inequality:

L ≡ ln p( x) = ln ∫ dΘp( x,Θ ) ≥ ∫ dΘq(Θ ) ln

p( x,Θ ) ≡F q(Θ )

(3)

It converts computation of log evidence into an optimization problem of minimizing the Kullback-Leibler (KL) distance between q (Θ ) and p ( x,Θ ) , which is

ln p( x) . To make the optimization problem tractable, it is assumed that the variational distribution q (Θ ) is suffiequivalent to maximizing a lower bound of

ciently simple - fully factorized with each factorized component in the exponential Fig. 1 A Bayesian formulation for MFA. Circles denote random variables, rectangles denote hyperparameters, and the plate notation is used to denote repetitions over data xkli and over the m analyzers in the generative model.

a* , b*

νs

s = 1, 2,..., m k = 1, 2,..., K l = 1, 2,..., Lk

μ * ,ν *

μkls

Akls

α *u * Ψ *kl

xkli i = 1, 2,..., N k = 1, 2,..., K l = 1, 2,..., Lk

si ykli

π

404

Q. Hou et al.

family. Under this assumption, an analytic solution of the optimal obtained by taking functional derivatives. We use

Θ kl = {α *u * , a* , b* , μ * ,ν * ,Ψ kl } to

q (Θ ) can be

denote the set of hyperparame-

ters of the Bayesian MFA model. The directed acyclic graph for the generative model is shown graphically in fig.1. Deriving from reference [6], we obtain the log marginal likelihood of the HRRPs by utilizing the equation (3): N ⎛ ⎞ L ≡ ln p ( xkl ) = ln⎜⎜ ∫ dΘ kl p (Θ kl )∏ p( xkli | Θ kl ) ⎟⎟ i =1 ⎝ ⎠

⎛ dπp (π | α *u * ) dνp(ν | a* , b* ) d Akl p ( Akl | ν ) dμ kl p( μ kl | μ * ,ν * ) ⋅ ⎞ ∫ ∫ ∫ ⎜∫ ⎟ ⎟ = ln⎜ N ⎡ m ⎜ ⎢ p( s | π ) dy p( y ) p ( x | s , y , A , μ ,Ψ )⎤⎥ ⎟ ∑ i kli kli kli i kli kl kl kl ∫ ⎜∏ ⎟ ⎦ ⎝ i =1 ⎣ si =1 ⎠ ≥ ∫ dπq (π ) ln

p (π | α * , u * ) q (π )

~ ⎡ p (ν s | a* , b* ) ~s ~s p ( Akls | ν s , μ * ,ν * ) ⎤ + ∑ ∫ dν q (ν ) ⎢ln + ∫ dAkl q ( Akl ) ln ~ ⎥ q (ν s ) q ( Akls ) s =1 ⎣ ⎦ m

s

s

N m ⎡ p ( si | π ) p ( ykli ) + ∑∑ q ( si ) ⎢ ∫ dπq (π ) ln + ∫ dykli q ( ykli | si ) ln q ( si ) q ( ykli | si ) i =1 si =1 ⎣ ~ ~ ~ + ∫ d Akl q ( Akl ) ∫ dykli q ( ykli | si ) ln p ( xkli | si , ykli , Akl ,Ψ kl )

(4)

]

≡F

~

= [ Akl , μ kl ] , k = 1,2,

, K and l = 1,2, , Lk . To make the VB trac~ q(π ,ν , Akl ) is fully factorized: table, we assume that ~ q(π ,ν , Akl , μ kl ) ≈ q(π )q(ν )q( Akl ) . The variational posteriors q (⋅) are given Here Akl

in the Appendix. The model hyperparameters which govern the priors also are estimated in the Appendix. VBMFA method can infer the number of analyzers automatic via a heuristic approach [6].

3 Main Procedure of Radar HRRP Target Recognition Based On the Proposed Statistical Model The main procedure of radar HRRP target recognition based on the proposed statistical model-VBMFA model will described in the following. The FA model and MFA model have the same procedure in our experiment.

A New Statistical Model for Radar HRRP Target Recognition

A. Training Phase Step 1) Divide the training samples of target frames {x

}

Lk kl l =1 ,

xkl = {xkli | i = 1,2,

where

and amplitude normalized HRRP frame. Step 2) Compute the mean

mx | kl (k = 1,2,

, K ; l = 1,2,

Step 3) Estimate the parameters ters

{A , μ s kl

s kl

,Ψ kl , π s

}

K ; Lk ; m

k =1; l =1; s =1

405

Tk (k = 1,2,

, K ) into HRRP

, N } denotes the l th range aligned vector

, Lk ) by mx |kl =

of

HRRP

frame

1 N x . ∑ i =1 kli N

Akl andΨ kl in FA model. Estimate the paramein MFA model and in VBMFA model.

Step 4) Store both the mean vector and the parameters for each frame of each target. B. Test Phase Step 1) The amplitude normalized test sample

x is time-shift compensated with mx | kl , the mean vector of frame HRRP xkl (k = 1,2, , K ; l = 1,2, , Lk ) ,

by the slide correlation processing.

x into the l th (l = 1,2, , Lk ) frame model of , K ) . The joint distribution of x , p ( x | l ; k ) can be ob-

Step 2) Substitute test sample target

Tk (k = 1,2,

tained by equation (1), and FA model is a special instance for equation (1) by m = 1 . Step 3) The class-conditional density of target

Tk (k = 1,2,

, K ) is

p ( x | k ) = max p ( x | l ; k ) . According to Bayes’ rule, the posterior probability l

of target

Tk is p(k | x) =

p( x | k ) p(k ) K

∑ p( x | k ) p(k ) k =1

Step 4) If C

= arg max p(k | x) , then x belongs to target TC . k

4 Experiment Result 1) Data Description: The results presented in this paper are based on measured airplane data [2]-[4]. The parameters of the targets and radar are shown in Table 1, and the projections of target trajectories onto ground plane are shown in Fig. 2, from which the aspect angle of an airplane can be estimated according to its relative position of radar. As shown in Fig.2 (a-c), the measured data are segmented. Training data and test data are from different data segments, and the training data

406

Q. Hou et al. 15

60

15

5 1

4

7

10

5

10

40 2 20

5

7

6

7

4

2

5

1

1

km

6

km

km

3

5

2

3 3

4 0 -5

Radar 0

5

km

0 -20

-15

(a)

-10 km

-5

Radar 0

0 -20

Radar -15

(b)

-10 km

-5

0

(c)

Fig. 2 Projection of target trajectories onto ground .plane (a) Yak-42; (b) Cessna Citation S/II; and (c) An-26

cover almost all of the target-aspect angles of the test data. Therefore, the second and the fifth segments of Yak-42, the fifth and the sixth segments of An-26, the sixth and seventh segments of Cessna S/II are taken as the training data, other data segments are taken as the test data in our experiments. In addition, the measured HRRP is a 256-dimensional vector. In our experiments, the aspect-frames in the training data are divided by the equal interval partition approach, and the latent variables in all aspect-frames have the same dimensionalities. There are 35 aspect-frames for Yak-42, 50 for Cessna Citation S/II, and 50 for An-26, which each frame has 1024 HRRP samples. 2) Recognition Experiments: The average recognition rates of FA model, MFA model and VBMFA model are shown in Table 2. The dimensionalities of latent variables in our experiments are the same q=20. In MFA model, the number of components with a constant M=2, and in VBMFA model, the number of components is not fixed, because this method can automatically determine the optimal number of components. From reference [4], we know that FA model has the best performance than PCA model and PPCA model (both the training data and test data are having a pretreatment with power transformation in reference [4], and we do not utilize it in our experiments). MFA model [8] has a better recognition performance than FA model, which indicated that the statistical model of radar HRRP data is usually a non-Gaussian distribution, not a single model like FA model. And the MFA model via multiple joint-Gaussian models to describe the statistical characteristics of non-Gaussian distributed correlation in multidimensional radar HRRP data. But Table 1 Parameters of Palnes and Radar in the Isar Experiment radar parameters

center frequency bandwidth

5520 MHz 400MHz

planes

length (m)

width (m)

height (m)

Yak-42 Cessna Citation S/II An-26

36.38

34.88

9.83

14.40

15.90

4.57

23.80

29.20

9.83

A New Statistical Model for Radar HRRP Target Recognition

407

Table 2 Confusion Matrices and Average Recognition Rates of FA Model, MFA Model, VBMFA Model FA model (q=20) Yak42 Yak-42

100

Cessna Citation S/II 2

MFA model (q=20;M=2)

VBMFA model (q=20)

An-26

Yak42

Cessna Citation S/II

An-26

Yak42

Cessna Citation S/II

An26

17.75

100

2

15

100

0

8

Cessna Citation S/II

0

84.25

1.75

0

84.50

1.50

0

89

11

An-26

0

13.75

80.50

0

13.50

83.50

0

11

81

Average Recognition rate (%)

88.25

89.30

90.00

the number of analyzers with a fixed constant (i.e. M=2 in our experiment) is not strict, and it does not assure that the fixed constant is the optimal number of analyzers. VBMFA model does not fix the number of analyzers to a constant, contrary it can infer the number of analyzers automatic via a heuristic approach, therefore VBMFA model has a best recognition performance, the average recognition rate of which is approximately 2 percentage points larger than that of FA model.

5 Conclusion From the theoretical analysis and some experiments for the FA model, MFA model and VBMFA model, we can have the conclusion that a jointly nonGaussian distributed HRRP frame can be described more accurately via the jointGaussian mixture model(i.e. MFA model and VBMFA model) than single model(i.e. FA model). The VB method is first introduced into radar HRRP statistical recognition, and can obtain a better recognition performance than FA model and MFA model. It is worth pointing out that there are only three targets in our experiments. Therefore, the performance evaluation results will descend if more targets are concerned. And VBMFA method has a small drawback that it is a little sensitive to random initialization of the parameters, so how to find an algorithm with more robust and a better recognition performance for multi-class targets, is our next work. Acknowledgments. This work is partially supported by the Program for Cheung Kong Scholars and Innovative Research Team in University (PCSIRT, grant no. IRT0645) and NSFC with grant of 60772140.

408

Q. Hou et al.

References 1. Jocobs, S.P.: Automatic Target Recognition Using High-resolution Radar Rang Profiles. Ph. D. Dissertation. Washington University (1999) 2. Chen, F., Du, L., Liu, H.W., Bao, Z., et al.: An Amplitude-scale and Time-shift Jointly Optimization for Radar HRRP Recognition. In: Proceeding of 2008 International Conference on Radar, Adelaide, Australia, pp. 515–518 (2008) 3. Du, L., Liu, H.W., Bao, Z., et al.: A Two-Distribution Compounded Statistical Model for Radar HRRP target Recognition. IEEE Transactions on Signal Processing 54(6), 2226–2238 (2006) 4. Du, L., Liu, H.W., Bao, Z.: Radar HRRP Statistical Recognition Parametric Model and Model Selection. IEEE Transactions on Signal Processing 56(5), 1931–1944 (2008) 5. Ueda, N., Nakano, R., Ghahramani, Z., Hinton, G.E.: SMEM Algorithm for Mixture Models. Neural Computation 12(9), 2109–2128 (2000) 6. Ghahramani, Z., Beal, M.: Variational Inference for Bayesian Mixture of Factor Analysers. In: Advances in NIPS, vol. 12, pp. 449–455 (2000) 7. Attias, H.: Inferring Parameters and Structure of Latent Variable Models by Variational Bayes. In: Proceeding of 15th Conference on Uncertainty in Artificial Intelligence (1999) 8. Ghahramani, Z., Hinton, G.E.: The EM Algorithm for Mixtures of Factor Analyzers. Technical Report CRG-TR-96-1, Dept. of Computer Science, University of Toronto (1996) 9. Stephen, P.B.: Markov Chain Monte Carlo Method and Its Application. The Statistician 47(1), 69–100 (1998) 10. MacKay, D.J.C.: Probable Networks and Plausible Predictions—a Review of Practical Bayesian Methods for Supervised Neural Networks. Network: Computation in Neural Systems 6, 469–505 (1995)

Appendix: Optimal q (⋅) Distribution and Hyperparameters Via the VB approximation, we can obtain the full variational posterior [6]

( )

m N m ~ p(π ,ν , Akl , μ kl , s, ykl | xkl ) ≈ q (π )∏ q (ν s )q Akls ⋅ ∏∏ q(si )q( ykli | si ) s =1

i =1 si =1

• q ( ykli | si ) ~ N ( yklis , Σ s ) s ykli = Σ s Akls Ψ kl−1 ( xkli − μkls T

−1

Σ s = I + Akls Ψ kl−1 Akls T

d d ~s ~s ~s q, s • q ( Akl ) = ∏ q( Aklq⋅ ) ~ ∏ N ( Aklq ) ⋅, Σ q =1

~s

where Aklq ⋅

s s = [ Aklq ⋅ , μ klq ]

~ q ( Akls )

q =1

~ q ( Akls )

(5)

(6)

A New Statistical Model for Radar HRRP Target Recognition N

409

s s s −1 q, s Aklq ⋅ = [ Akl ]q = [Ψ kl ∑ q ( si ) xkli ykli Σ Akl Akl ]q T

(7)

i =1

⎡⎡

N

⎤

i =1

⎦

⎤

s μ klq = [ μ kls ]q = ⎢ ⎢Ψ kl−1 ∑ q ( si ) xkli + ν *μ * ⎥ Σ μq , sμ ⎥

⎣⎣

Σ Aq , sA kl

−1

kl

N

T = Ψ kl−1 ∑ q( s i ) ykli ykli i =1

Σ μq , sμ kl

Ψ

−1 kl

⎡ N 1 = diag ⎢ ∑ ⎢ N i =1 ⎣⎢

q ( y kli | s i )

−1 kl

kl

kl

⎦q

+ diag ν s

(8)

(9) q (ν s )

N

= ν q* +Ψ kl−1 ∑ q ( si )

(10)

i =1

⎛ ~ ⎡ y ⎤ ⎞⎛ ~ ⎡y ⎤⎞ ⎜ xkli − Akls ⎢ kli ⎥ ⎟⎜ xkli − Akls ⎢ kli ⎥ ⎟ ⎟ ⎟ ⎜ ⎜ ⎣ 1 ⎦ ⎠⎝ ⎣ 1 ⎦⎠ ⎝

T

⎤ ⎥ (11) ⎥ ~ ⎥ q ( Akls ) q ( s i ) q ( y kli | s i ) ⎦

where here we use the diag as the operator which sets off-diagonal terms to zero. ks

ks

j =1

j =1

• q(ν s ) = ∏ q(ν ⋅sj ) ~ ∏ Ga (a sj , b sj ) where

Ga denotes the Gamma distribution, k s is the number of column of s th

factor loading matrix

Akls .

a sj = a * +

1 d d s s 2 * , b j = b + ∑ Aklqj 2 2 q =1

(12) q ( Akls )

• q(π ) ~ Dir (αu ) where Dir denotes the Dirichlet distribution.

αu s =

α* m

N

+ ∑ q( si )

(13)

i =1

~ 1 • ln q( si ) = [ψ (αus ) − ψ (α )] + ln Σ s + ln p( xkli | ykli , si , Akls ,Ψ kl ) + c 2 ∂ where ψ (x ) is the digamma function ψ ( x ) ≡ ln Γ ( x) , Γ (x) is the Gamma ∂x function.

πs =

1 N ∑ q(si ) N i =1

(14)

Independent Component Analysis of SPECT Images to Assist the Alzheimer’s Disease Diagnosis ´ Ignacio Alvarez, Juan M. G´ orriz, Javier Ram´ırez, Diego Salas-Gonzalez, Miriam L´ opez, Carlos Garc´ıa Puntonet, and Fermin Segovia

Abstract. Finding sensitive and appropriate technologies for non-invasive observation and early detection of the Alzheimer’s Type Dementia (ATD) are of fundamental importance to develop early treatments. Single Photon Emission Computed Tomography (SPECT) images are commonly used by physicians to assist the diagnosis, rating them by visual evaluations. In this work, we present a computer aided diagnosis method in which a selection of relevant features was extracted from each patient image by means of Independent Component Analysis (ICA). An average image was computed within the normal or Alzheimer’s disease brain image class, to be later used to extract a set of independent sources that best symbolized each class characteristics. Each brain image was projected onto the space spanned by this independent sources basis, and the extracted information was used to train a SVM classifier which could classify new subjects in a unsupervised manner.

1 Introduction The Alzheimer’s Dementia (AD) is still incurable and the affected population, mostly elder, is growing through the past decades. The diagnostic of AD still remains a challenge, specially during the early stage of the disease that offers better opportunities to treat its symptoms or allows to test new medical treatments. Anatomical changes in the affected brains caused by the AD dementia take long time to occur, while physiological functions can give information about the damage earlier. A study ´ Ignacio Alvarez · Juan M. G´ orriz · Javier Ram´ırez · Diego Salas-Gonzalez · Miriam L´ opez · Fermin Segovia Dept. of Signal Theory, Networking and Communications, University of Granada, Spain Carlos Garc´ıa Puntonet Dept. of Architecture and Computer Technology, University of Granada, Spain H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 411–419. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

412

´ I. Alvarez et al.

of the regional cerebral blood flow (rCBF) of the brain is frequently used as a diagnostic tool in addition to the clinical findings. SPECT is a noninvasive, three-dimensional functional imaging modality that provide clinical information regarding biochemical and physiologic processes in patients, i.e. the rCBF. Many studies have examined the predictive abilities of nuclear imaging with respect to AD and other dementia illnesses. The evaluation of these images is usually done through visual ratings performed by experts. However, statistical classification methods have not been widely used in this area, quite possibly due to the fact that images represent large amounts of data and most imaging studies have relatively few subjects (generally <100)[1], producing the small sample size problem. Previous approaches consider the use of voxel intensities of the whole brain image as features to apply statistical learning methods for the classification [2]. Due to the large number of voxels, such approaches suffer the so-called small sample size problem, which occurs if the number of training samples is much smaller than the dimensionality of the feature space [3]. However, there exist approaches that successfully deal with the small sample size problem, as the PCA application of [4]. Nevertheless, PCA only takes into account pairwise relationships between voxels of the brain images. It seems reasonable to expect that important information may be contained in the high-order relationships among pixels, for which ICA is a sensitive method, being this reason the motivation to use it. Also, ICA has been successfully applied to other fields that can suffer the small sample size problem, as the face detection problem [5], fMRI brain imaging [6] or glaucoma detection in retina [7]. In the other hand, also an effective and robust method is needed to manage the classification task. Support Vector Machines are an example of classification technique that has shown adequate generalization properties in small datasets [8]. Support Vector Machines (SVMs) have marked the beginning of a new era in the learning from examples paradigm [9]. Recently, SVMs have attracted attention from the pattern recognition community due to a number of theoretical and computational merits derived from [9]. These techniques have been successfully applied to many fields including voice activity detection (VAD) [10], content-based image retrieval [11], and medical imaging diagnosis [2]. Somehow, the application of SVM to high dimensional and small sample size problems is still a challenge and improving the accuracy of SVM based-approaches is still a field in development [12]. This paper shows a computer aided diagnosis (CAD) system for the early detection of ATD using SVM classifiers applied to the projection of each brain image into the ’independent sources brain images space’ (see Sect.2). The proposed method, tested over SPECT images, is developed with the aim of reducing the subjectivity in visual interpretation of these scans by clinicians, thus improving the accuracy of diagnosing Alzheimer disease in its early stage.

Independent Component Analysis of SPECT Images

413

2 Estimation of the ICA Model The task in ICA is to find a solution to the noiseless blind source separation problem [13, 14], which can be stated as follows: let X be an observed random vector and A a full rank matrix such that: X = AS

(1)

where the source signals S are commonly assumed to be stochastically independent: ps (S1 , ..., Sn ) = ps1 (S1)...psn (Sn ). Thus, ICA recovers both the sources Sj and the mixing process using the independence assumption. In the linear case, the latter task consists of finding the mixing matrix A. A popular approach is to find a demixing or separating matrix W so that variables Yj in Y = WX are estimates of Sj up to scaling and permutation. In the deflationary approach the sources are estimated one by one, by finding a column vector wj (this will be stored as a row of W) such that Yj = wTj X is an estimate of Sj . Hence W is an estimate of the (pseudo)inverse of A up to scaling and permutation of the rows of W. The estimation of the independent components and the mixing matrix is done with the help of FastICA [15], which is an iterative fixed-point algorithm with the following update for w: w ←− E{Xg(wT X)} − E{g (wT X)}w

(2)

where w is one of the rows of the demixing matrix W, and g is the derivative of the contrast function, chosen to be a cubic polynomial. After each iteration step (2), w is normalized to have unit norm, ensuring that the rows wj of the demixing matrix are orthogonal.

2.1 ICA Application to Functional Brain Images The approach taken in this work to the ICA method consist of the search of a few element basis, which can describe the classes of our database in the most independent way. In brain images, the dataset is an ensemble of 3D brain images Γi , which size M is typically 79 × 95 × 69 ∼ 5 · 105 voxels, each pertaining to a class C k , k = 1, 2, ..., K, where K is the total number of classes. Let the full 3D brain image database be Γ1 , Γ2 , ..., ΓN , each understood as a vector of dimension M . A within class average brain image X can be defined as: Xk =

1 Nk



Γi ,

k = 1, 2, ..., K

(3)

{Γi ⊂C k }

where Nk denotes the number of images in the class C k . These average images, which number is the same as the number of classes, are the original mixed signals X = [X1 , X2 , ..., XK ]. ICA algorithm is expected to separate

´ I. Alvarez et al.

414

Fig. 1 Separation process of four original signals into four independent sources. Three representative coronal brain slices are displayed

them into a independent set of sources through WX = Y, where W is the estimated separating matrix, and Y is the estimated set of K independent sources Y = [Y1 , Y2 , ..., YK ] (see Fig 1). These latent variables are the essence of the different classes. Furthermore, these independent sources Yk define a orthogonal basis which span a independent components subspace of the ‘brain images space’. Thus, if seen as a linear operator, Y can be used to project each brain image onto the independent component subspace, yielding to a coefficients matrix Ω: Ωki = Yk Γi ,

i = 1, ..., N

k = 1, 2, ..., K

(4)

This coefficients denote the contribution of each original source in representing a specific brain image (see Fig. 2). Thus, the matrix Ω contains the most significant information extracted from the independent component analysis, stacked in a K × N data ensemble. We used this matrix Ω as features for the classification task, that is N K-dimensional patterns: xi = [Ω1i , Ω2i , ..., ΩKi ], i = 1, 2, ..., N

(5)

each of them with its corresponding class label yi ∈ {±1}.

3 Background on SVMs The classification is achieved through a SVM, that separates a given set of binary labeled training data with a hyperplane that is maximally distant from

Independent Component Analysis of SPECT Images

415

the two classes (known as the maximal margin hyper-plane). The objective is to build a function f : IRK −→ {±1} using training data, consisting of K-dimensional patterns xi and class labels yi :   (x1 , y1 ), (x2 , y2 ), ..., (xN , yN ) ∈ IRK × {±1} ,

(6)

so that f will correctly classify new examples (x, y). When no linear separation of the training data is possible, SVM can work effectively in combination with kernel techniques using the kernel trick, so that the hyperplane defining the SVM corresponds to a non-linear decision boundary in the input space [9]. In this way the decision function f can be expressed in terms of the support vectors only [9]: f (x) = sign{

NS

αi yi K(si , x) + w0 },

(7)

i=1

where K(., .) is the kernel function, αi is a weight constant derived form the SVM process and si are the support vectors [9]. Common kernels that are used by SVM practitioners for the nonlinear feature mapping are: • Polynomial

K(x , y) = [γ(x · y) + c]d .

(8)

• Radial basis function (RBF) K(x , y) = exp(−γ||x − y ||2 ).

(9)

as well as the linear kernel, in which K(., .) is simply a scalar product.

4 Database The database consists of a set of 3D SPECT brain images produced with an injected gamma emitting 99m Tc-ECD radiopharmeceutical and acquired by a three-head gamma camera Picker Prism 3000. Images of the brain cross sections are reconstructed from the projection data using the filtered backprojection (FBP) algorithm in combination with a Butterworth noise removal filter. The classification task is based on the assumption that the same position in the volume coordinate system within different images corresponds to the same anatomical position. The SPECT images are spatially normalized using the SPM software [16]. The preprocessing and spatial normalization procedure, described in detail in [8], is achieved using Affine and Non-linear spatial normalization [17], and essentially guarantees meaningful voxel-wise comparisons between images. SPECT images consists of functional information and direct comparison of the voxel intensities, even different acquisitions of the same subject, is thus not possible without normalization of the

416

´ I. Alvarez et al.

intensities. Intensity level of the images is normalized to the maximum intensity, and consequently the basic assumptions are met. The images were initially labeled by experienced physicians of the “Virgen de las Nieves” hospital (Granada, Spain), using 4 different labels: normal (NOR) for patients without any symptoms of ATD and possible ATD (ATD1), probable ATD (ATD-2) and certain ATD (ATD-3) to distinguish between different levels of the presence of typical characteristics for ATD. In total, the database consists of N =79 patients: 41 NOR, 20 ATD-1, 14 ATD-2 and 4 ATD-3. We considered the patient label positive when belonging to any of the ATD classes, and negative otherwise.

5 Experiments Two different experiments were performed to construct the within class average images (3): • Method I: Using 2 classes; NOR and ATD, combining the three ATD labels into one. • Method II: Using 4 classes; NOR, ATD-1, ATD-2 and ATD-3. Once the matrix Ω was obtained using (4), a SVM was trained using 4 different kernels: linear, quadratic, Radial Basis Function (RBF) and polynomial, and was tested using a leave-one-out cross-validation strategy. It is to be noted that, in the first method, four classes were used with the purpose of constructing the matrix Ω. But for the classification task, the feature vector class labels (5) were always reduced to two, combining the three ATD labels into one. Fig. 2 ICA representation of an Alzheimer’s affected patient following method I

Independent Component Analysis of SPECT Images

417

Table 1 Statistical performance measures of the ICA model Parameter (%) Method I

Accuracy Specificity Sensitivity

Linear 84.81 87.80 81.58

Method II

Accuracy Specificity Sensitivity

83.54 85.37 81.58

Kernel function Quadratic RBF 87.34 87.34 92.68 90.24 81.58 84.21 86.07 87.80 84.21

91.14 92.68 89.47

VAF

Polynomic 87.34 92.68 81.58

72.15 78.05 65.79

88.60 90.24 86.84

74.68 82.93 65.79

5.1 Analysis The results summarized in table 1 reveal that the idea of finding a few element image basis for characterizing the AD is meaningful. The two described methods of constructing Ω yielded to significant improvement over other CAD systems based on the Voxels-As-Features (VAF) approximation[18], which performance values are displayed as reference. As expected from theoretic reasons [9], non linear kernel methods generalize better as linear kernels when the number of features is small. Both method I and II represent a high compression of the large amount of brain image data to a small number of features; 2 or 4 weight values respectively. Method I exhibit interesting performance results, but the compression into 2 features may be too drastic, and brain image information may be lost. Method II shows to be more adequate to characterize the AD, and best performance is obtained when combined with a RBF kernel, reaching 91.1% accuracy. Specificity took the value 92.7% and sensitivity 89.5% in that case.

6 Conclusions ICA applied to SPECT brain images is an effective way of extracting the relevant information for classification. Different architectures are possible for applying the ICA algorithm, like finding a equal number of sources as observed signals, or performing the unmixing process to the image voxels generating a new images made up of independent information. Nevertheless, the original aim of the ICA application within this work was to find a set of independent sources that best symbolize each Alzheimer’s disease stage. The final results show the success of this method when combined with a RBF kernel SVM, which estimated error rate lower than 9%, ourperforming previous results like the VAF approximation.

418

´ I. Alvarez et al.

Acknowledgements. This work was partly supported by the MICINN under the PETRI DENCLASES (PET2006-0253), TEC2008-02113, NAPOLEON (TEC200768030-C02-01) and HD2008-0029 projects and the Consejer´ıa de Innovaci´ on, Ciencia y Empresa (Junta de Andaluc´ıa, Spain) under the Excellence Project TIC-02566.

References 1. Ishii, K., Kono, A.K., Sasaki, H., Miyamoto, N., Fukuda, T., Sakamoto, S., Mori, E.: Fully Automatic Diagnostic System for Early- and Late-onset Mild Alzheimer’s Disease Using FDG PET and 3D-SSP. European Journal of Nuclear Medicine and Molecular Imaging 33(5), 575–583 (2006) 2. Fung, G., Stoeckel, J.: SVM Feature Selection for Classification of SPECT Images of Alzheimer’s Disease Using Spatial Information. Knowledge and Information Systems 11(2), 243–258 (2007) 3. Duin, R.P.W.: Classifiers in Almost Empty Spaces. In: Proceedings 15th International Conference on Pattern Recognition, vol. 2, pp. 1–7. IEEE, Los Alamitos (2000) 4. Nobili, F., Salmaso, D., Morbelli, S., Girtler, N., Piccardo, A., Brugnolo, A., Dessi, B., Larsson, S.A., Rodriguez, G., Pagani, M.: Principal Component Analysis of fdg pet in Amnestic MCI. Eur. J. Nucl. Med. Mol. Imaging 35(12), 2191–2202 (2008) 5. Bartlett, M., Movellan, J., Sejnowski, T.: Face Recognition by Independent Component Analysis. IEEE Transactions on Neural Networks 13(6), 1450–1464 (2002) 6. Theis, F.J., Gruber, P., Keck, I.R., Lang, E.W.: Functional MRI analysis by a novel spatiotemporal ICA algorithm. In: Duch, W., Kacprzyk, J., Oja, E., Zadro˙zny, S. (eds.) ICANN 2005. LNCS, vol. 3696, pp. 677–682. Springer, Heidelberg (2005) 7. Fink, F., Worle, K., Gruber, P., Tome, A.M., Gorriz, J.M., Puntonet, C.G., Lang, E.W.: Ica Analysis of Retina Images for Glaucoma Classification. In: 30th Annual International Conference of the Engineering in Medicine and Biology Society, pp. 4664–4667. IEEE, Los Alamitos (2008) 8. Ram´ırez, J., G´ orriz, J.M., G´ omez-R´ıo, M., Romero, A., Chaves, R., Lassl, A., Rodr´ıguez, A., Puntonet, C.G., Theis, F., Lang, E.: Effective emission tomography image reconstruction algorithms for SPECT data. In: Bubak, M., van Albada, G.D., Dongarra, J., Sloot, P.M.A. (eds.) ICCS 2008, Part I. LNCS, vol. 5101, pp. 741–748. Springer, Heidelberg (2008) 9. Vapnik, V.N.: Statistical Learning Theory. John Wiley and Sons, Inc., New York (1998) 10. Ram´ırez, J., Y´elamos, P., G´ orriz, J.M., Segura, J.C.: SVM-based Speech Endpoint Detection Using Contextual Speech Features. Electronics Letters 42(7), 877–879 (2006) 11. Tao, D., Tang, X., Li, X., Wu, X.: Asymmetric Bagging and Random Subspace for Support Vector Machines-based Relevance Feedback in Image Retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence 28(7), 1088– 1099 (2006)

Independent Component Analysis of SPECT Images

419

12. G´ orriz, J.M., Ram´ırez, J., Lassl, A., Salas-Gonzalez, D., Lang, E.W., Puntonet, ´ C.G., Alvarez, I., L´ opez, M., G´ omez-R´ıo, M.: Automatic Computer Aided Diagnosis Tool Using Component-based SVM. In: Medical Imaging Conference, Dresden. IEEE, Los Alamitos (2008) 13. Comon, P.: Independent Component Analysis, a new concept? Signal Process. 36(3), 287–314 (1994) 14. Bingham, E.: Advances in Independent Component Analysis with Applications to Data Mining. PhD thesis, Helsinki University of Technology (2003) 15. Oja, E.: A Fast Fixed-point Algorithm for Independent Component Analysis. Neural Computation 9, 1483–1492 (1997) 16. Friston, K., Ashburner, J., Kiebel, S., Nichols, T., Penny, W. (eds.): Statistical Parametric Mapping: The Analysis of Functional Brain Images. Academic Press, London (2007) 17. Salas-Gonzalez, D., G´ orriz, J.M., Ram´ırez, J., Lassl, A., Puntonet, C.G.: Improved Gauss-newton Optimization Methods in Affine Registration of Spect Brain Images. IET Electronics Letters 44(22), 1291–1292 (2008) 18. Stoeckel, J., Malandain, G., Migneco, O., Koulibaly, P.M., Robert, P., Ayache, N., Darcourt, J.: Classification of SPECT Images of Normal Subjects Versus Images of Alzheimer’s Disease Patients. In: Niessen, W.J., Viergever, M.A. (eds.) MICCAI 2001. LNCS, vol. 2208, pp. 666–674. Springer, Heidelberg (2001)

The Multi-Class Imbalance Problem: Cost Functions with Modular and Non-Modular Neural Networks Roberto Alejo, Jose M. Sotoca, R.M. Valdovinos, and Gustavo A. Casa˜n

Abstract. In this paper, the behavior of Modular and Non-Modular Neural Networks trained with the classical backpropagation algorithm in batch mode and applied to classification problems with Multi-Class imbalance is studied. Three different cost functions are introduced in the training algorithm in order to solve the problem in four different databases. The proposed strategies show an improvement in the classification accuracy with three different types of Neural Networks. Keywords: Multi-Class, imbalance, backpropagation, cost function.

1 Introduction Typically supervised learning methods are designed to work with reasonably balanced Training Sets (TS) [1], but many real world applications have to face imbalanced data sets [2]. A TS is said to be imbalanced when several classes are under-represented (minority classes) in comparison with others (majority classes). Roberto Alejo Lab. Reconocimiento de Patrones, Instituto Tecnol´ogico de Toluca Av. Tecnol´ogico s/n, 52140 Metepec, M´exico and Centro Universitario UAEM Atlacomulco, Universidad Aut´onoma del Estado de M´exico Carretera Toluca-Atlacomulco Km. 60, 50450 Atlacomulco, M´exico R.M. Valdovinos Centro Universitario UAEM Valle de Chalco, Universidad Aut´onoma del Estado de M´exico Hermenegildo Galena No.3, Col. Ma. Isabel, 56615 Valle de Chalco, M´exico Jose M. Sotoca · Gustavo A. Casa˜n Dept. Llenguatges i Sistemes Inform´atics, Universitat Jaume I Av. Sos Baynat s/n, 12071 Castell´o de la Plana, Spain

H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 421–431. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

422

R. Alejo et al.

A feed-forward Neural Network (NN) trained on an imbalanced dataset is not able to learn with sufficient discrimination among classes [3]. Particularly, in the back-propagation algorithm with batch-mode, the majority class dominates the training process, therefore the minority classes converge very slowly [4]. In the machine learning field, most of the work in imbalanced problems are addressed to solve the problem for two classes [5], and only a few studies discuss the multi-class imbalance problem [4, 6]. This paper is focused mainly in the evaluation of different cost functions designed to improve the NN performance. Thus, the backpropagation algorithm is modified to deal with the multi-class imbalance problem. These cost functions will be calculated in relation to the proportion of samples used in to train the NN. The main contributions of this paper are the comparison between different approaches that apply cost functions in the multi-class imbalance problem learning directly and the effect of decoupling multi-class problems and solving two-class imbalance problem using a modular strategy.

2 Modular and Non Modular Neural Networks The Modular Neural Networks (Mod-NNs) present a new trend in NN architectural designs [7]. It has been motivated by the highly-modular nature in biological networks and based on the “divide and conquer” approach [8]. The use of Mod-NNs implies a significant improvement in the learning process in comparison with a Non-Modular NN (Non-Mod-NN) [8]. The Non-Modular classifiers tend to introduce high internal interferences because a strong coupling among their hidden-layer weights can appear [9]. The Mod-NNs show the following computational advantages [4]: a) the numbers of iterations needed to train the individual modules is less than the number of iterations needed to train a Non-Mod-NN for the same task; b) the modules in a Mod-NN are smaller than Non-Mod-NN; and c) the modules can be trained independently and in parallel. Here, we use the Mod-NN architecture to face the multi-class imbalance problem. In this Mod-NN architecture, each module is a single-output NN (see Fig.1) which determines if a pattern belongs to a particular class. Thereby, a K-class problem is reduced to a set of K two-class problems. A module for class ck is trained to distinguish between patterns belonging to ck respect to the patterns of the rest of classes. So, in our Mod-NN, given a instance test xi , the two class network with the highest rating is taken as the class label for that instance. Non-Mod-NN and Mod-NNs modules are studied in this work with Radial Basis Function NN (RBFNN), Random Vector Functional Link Net Networks (RVFLNN) and Multilayer Perceptron (MLP). MLP and RBFNN are two well-known NN in the pattern recognition field [10]. The main difference from the MLP is that the activations of the hidden neurons of RBFNN depend on the distance of an input vector to a prototype vector whereas MLP calculate the inner product of the input

The Multi-Class Imbalance Problem

423

(a) Non-Modular NN

(b) Modular NN

Fig. 1 The NN architectures

vector and the weight vector [11]. Both NN can be trained by supervised methods [10]. All parameters are adapted simultaneously by an optimization procedure. The RVFLNN is a variant of the RBFNN. The RVFLN of Pao [12] is added to the RBFNN in order to obtain the last one, and it gives a extra connectivity of the FLN along with any functions put into the offset hidden neurons. The addition of connections between the hidden neurons adds extra learning power [10].

3 The Backpropagation Algorithm and the Class Imbalance Problem Empirical studies of the backpropagation algorithm [13], show that the class imbalance problem does not generate equal contributions to the mean square error (MSE) in the training phase. Obviously the main contribution to the MSE is produced by the majority class. m Let us consider a TS with two classes such that N = i ni and ni is the number of samples from class i. Suppose that the MSE by class may be expressed as Ei (U ) =

ni L 1 (y n − Fpn )2 , N n=1 p=1 p

(1)

so that the overall MSE can be expressed as E(U ) =

m

Ei = E1 (U ) + E2 (U ) .

(2)

i=1

If n1 << n2 then E1 (U ) << E2 (U ) and ∇E1 (U ) << ∇E2 (U ). Then ∇E(U ) ≈ ∇E2 (U ). So, −∇E(U ) it is not always the best direction to minimize the MSE in both classes.

424

R. Alejo et al.

Considering that the TS imbalanced problem affects negatively the backpropagation algorithm due to the disproportionate contributions to the MSE, it is possible to consider a cost function (γ) that balances the TS class imbalance as follows m E(U ) = i=1 γ(m)Ei = γ(1)E1 (U ) + γ(2)E2 (U ) (3) nm L M n n 2 γ(m) (y − F ) , = N1 p n=1 m=1 p=1 p where γ(1)∇E1 (U ) ≈ γ(2)∇E2 (U ) avoiding the fact that the minority class would be ignored in the learning process [14]. The previous process can be generalized to multi-class problems [6] to obtain the following cost functions: • Option 0: γ(m) = 1, is the backpropagation algorithm without modifications. • Option 1: γ(m) = nmax /nm ; where m = 1, ..., M and nmax is the number of samples of the majority class. • Option 2: γ(m) = N/nm ; where m = 1, ..., M and N is the total number of samples. • Option 3: γ(m) = ∇Emax (U )/∇Em (U ), where ∇Emax (U ) is the majority class. This function is a simplification of [13].

4 Data Sets For the experimental phase, Cayo, Ecoli6, Feltwell and Satimage databases with multiple classes were used. Feltwell is related to an agriculture region near to Felt Ville, Feltwell (UK) and is divided in training data (5124 samples) and test data (5820 samples). Cayo, which represents a particular region in the gulf of Mexico, was partitioned using the holdout method (50% training and 50% test). Both are remote sensing images. Ecoli6 is obtained from Ecoli, a biological database created by the Institute of Molecular and Cellular Biology from Osaka University, Japan. It was originally distributed in eighth classes but in this work classes 7 and 8 have been eliminated since these only have two samples making it difficult to apply the method of cross validation. The Ecoli6 database was split using the five cross validation method in 10 parts (5 training and 5 testing), using the 80-20 proportion. Satimage were obtained from the UCI Machine Learning Database Repository, and used

Table 1 A brief summary of the some basic characteristics of the databases Dataset Cayo Ecoli6 Feltwell Satimage

Size 6019 332 10944 6430

Attr. 4 7 15 36

Class 11 6 5 6

Class distribution 838/293/624/322/133/369/324/722/789/833/772 5/143/77/52/35/20 3531/2441/896/2295/1781 1508/1531/703/1356/625/707

The Multi-Class Imbalance Problem

425

without changes. The data in Satimage is divided into: 4435 training and 200 testing samples. In Table 1, the most important characteristics of each database are summarized.

5 Methodology The NN was trained with the backpropagation algorithm in batch mode. This process was repeated five times and the results correspond with the average. The KHold-Out Paired t and K-Fold Cross-Validation Paired t statistical tests [15] were applied. We assume that the set of differences has an independently drawn sample from an approximately normal distribution. Then, under the null hypothesis (equal classification accuracies), the following statistic has a Test Student distribution with K - 1 degrees of freedom. The learning rate (η) was set to 0.0001 for RBFNN and RVFLNN, and a value of 0.9 was used for the MLP. Only one hidden layer was used in the last case. The number of neurons for the hidden layer (for all NN) was established to 16, 15, 6 and 12 for Cayo, Ecoli6, Feltwell and Satimage respectively. In this study, Accuracy, g-mean and Kappa coefficient are used as measure criteria for the classifiers performance. These measure criteria are often obtained from the confusion matrix, where real classes are in columns whereas predicted ones appears in rows (Table 2). The table built on this way is a general vision assignment, the right ones (diagonal elements) like the wrong ones  (elements out of the diagok

n

ii nal). From Table 2, the measure criteria Accuracy = i=1 is obtained, where n njj n is the total number of samples and n+j is the Accuracy by class. The proportions of the samples pij in the cell (i, j) correspond to the number of samples nij , i.e, k k pij = nij /n. So, define pi+ and p+j as pi+ = j=1 pij , and p+j = i=1 pij . The Kappa coefficient is used as a quality parameter and takes into consideration the marginal distributions for the confusion matrix. Its value provides gives us an idea of the right percentage obtained in the classification process, # $ once o −pc the random part has been eliminated. It is defined as Kappa = p1−p where c k k po = i=1 pii is the well predicted percentage, and pc = i=1 pi+ p+j the random coefficient.

Table 2 Confusion matrix

Predicted Classes 1 2 k total (n+j )

1 n11 n21 nk1 n+1

Real Classes 2 n12 n22 nk2 n+2

k n1k n2k nkk n+k

total (ni+ ) n1+ n2+ nk+ n

426

R. Alejo et al.

Other measure used to quantify the classifier performance in the class imbalance problem is the geometric mean (g-mean) [14]. The geometric mean is defined as 9k 1 g-mean = ( i=1 pii ) k , where pii is the class accuracy i.

6 Experiments and Discussions As can be seen in Table 1, all databases have different TS imbalance levels. From a moderate imbalance up to a severe imbalance in the same data set. As example, in Ecoli6 database, classes 1 and 2 present a severe imbalance between them (majority class has 143 samples and minority class has only 5), while classes 5 and 6 show a reasonable imbalanced problem. A similar case is observed in Cayo, while in Feltwell and Satimage databases only a moderate imbalance between classes can be noticed. The first column in the tables of classification performance shows the NN used and the second the applied evaluation criteria. The following columns contain the observed values due to the applied strategies. The data in parenthesis represents the standard deviation.

6.1 Results with Non-Mod-NN The experience with the Cayo dataset shows the low performance of the classifier when imbalanced samples were used in the training phase without a cost function (Option 0). In table 3, the zero value of g-mean is observed in the three network models without applying cost functions. As expected, the classes not correctly identified are minority classes (4 and 5) whose impact is small in the global performance. In the three network models using cost functions improve the results when the g-mean has a low value in Option 0 (for Cayo database it took value zero). The improvements of classifier accuracy are due to the increment in the minority classes accuracy when cost functions are applied in the training phase and consequently are

Table 3 Classification performance of Cayo dataset with Non Modular Networks

MLP

RBFNN

RVFLNN

Acc g-mean Kappa Acc g-mean Kappa Acc g-mean Kappa

Option 0 76.7(1.8) 0.0(0.0) 0.7(0.0) 75.0(4.5) 0.0(0.0) 0.7(0.1) 70.8(4.3) 0.0(0.0) 0.7(0.1)

Option 1 85.2(0.4) 82.6(0.3) 0.8(0.0) 75.6(3.9) 72.7(4.5) 0.7(0.0) 75.4(2.1) 73.1(2.4) 0.7(0.0)

Option 2 85.9(0.3) 84.1(0.7) 0.8(0.0) 78.4(4.8) 73.7(4.8) 0.8(0.1) 78.8(2.5) 73.5(3.1) 0.8(0.0)

Option 3 85.1(0.2) 80.2(0.4) 0.8(0.0) 81.0(1.5) 74.9(2.4) 0.8(0.0) 82.0(1.4) 74.1(2.4) 0.8(0.0)

The Multi-Class Imbalance Problem

427

Table 4 Classification performance of Ecoli6 dataset with Non Modular Networks

MLP

RBFNN

RVFLNN

Acc g-mean Kappa Acc g-mean Kappa Acc g-mean Kappa

Option 0 87.1(2.7) 84.5(5.7) 0.8(0.0) 83.4(5.5) 66.2(37.4) 0.8(0.1) 84.9(4.3) 62.0(35.4) 0.8(0.1)

Option 1 82.8(6.8) 82.1(6.2) 0.8(0.1) 83.1(5.4) 84.1(3.6) 0.8(0.1) 83.1(7.4) 84.4(6.2) 0.8(0.1)

Option 2 82.2(5.9) 81.7(5.3) 0.8(0.1) 84.6(4.9) 84.4(5.4) 0.8(0.1) 84.9(4.4) 84.1(6.2) 0.8(0.1)

Option 3 85.8(5.2) 84.4(6.7) 0.8(0.1) 84.6(5.5) 84.3(7.3) 0.8(0.1) 83.1(6.7) 84.1(6.6) 0.8(0.1)

identified correctly in the classifier phase. The options with better results are Option 2 for MLP and Option 3 for RBFNN and RVFLNN. In data set Ecoli6 (Table 4) the g-mean values (see Option 0) are rather high specially for MLP, thus, the use of cost functions does not suppose an improvement in the results, and in same cases it obtains worse results. It should probably be related to the small size of the database. Feltwell dataset presents a similar situation as Ecoli6 with high g-mean values, but when we use cost functions the accuracy results are similar or bit better. The difference seems to come from the number of samples, that is significantly bigger in Feltwell. Option 2 shows the best results in MLP and RBFNN models, while Option 1 and 3 do the same for RVFLNN. In Satimage dataset (Table 6) all implemented Options report improvements in the global classifier performance. Important accuracy, g-mean values improvements can be observed (specially in MLP) when options 1, 2 and 3 are applied. The best results are obtained with Option 2 in MLP and RBFNN and Option 3 in RVFLNN. Table 5 Classification performance of Feltwell dataset with Non Modular Networks

MLP

RBFNN

RVFLNN

Acc g-mean Kappa Acc g-mean Kappa Acc g-mean Kappa

Option 0 88.6(1.3) 84.7(3.2) 0.9(0.0) 87.0(0.8) 81.1(2.5) 0.8(0.0) 86.9(1.8) 74.0(15.6) 0.8(0.0)

Option 1 88.8(0.7) 87.0(0.8) 0.9(0.0) 86.5(1.8) 84.9(2.2) 0.8(0.0) 88.0(0.8) 85.8(0.7) 0.8(0.0)

Option 2 89.3(1.2) 87.4(1.3) 0.9(0.0) 87.8(1.8) 86.3(1.7) 0.8(0.0) 87.0(2.9) 84.5(4.1) 0.8(0.0)

Option 3 88.8(0.4) 86.6(0.4) 0.9(0.0) 84.1(3.0) 78.2(7.2) 0.8(0.0) 88.1(1.1) 84.7(2.4) 0.8(0.0)

428

R. Alejo et al.

Table 6 Classification performance of Satimage dataset with Non Modular Networks

MLP

RBFNN

RVFLNN

Acc g-mean Kappa Acc g-mean Kappa Acc g-mean Kappa

Option 0 82.2(0.2) 50.8(2.8) 0.78(0.0) 83.2(0.7) 76.8(1.8) 0.8(0.0) 82.8(1.2) 74.1(1.9) 0.8(0.0)

Option 1 85.6(0.6) 84.2(0.6) 0.8(0.0) 83.3(0.6) 81.6(1.2) 0.8(0.0) 83.2(1.3) 81.1(1.2) 0.8(0.0)

Option 2 87.4(0.5) 86.3(0.4) 0.8(0.0) 85.6(0.5) 84.6(0.5) 0.8(0.0) 84.5(0.3) 82.8(0.5) 0.8(0.0)

Option 3 86.3(0.5) 83.6(1.1) 0.8(0.0) 84.3(1.3) 81.4(2.5) 0.8(0.0) 85.3(0.3) 82.2(0.5) 0.8(0.0)

Table 7 Classification performance of Cayo with Modular Networks

MLP

RBFNN

RVFLNN

Acc g-mean Kappa Acc g-mean Kappa Acc g-mean Kappa

Option 0 71.4(1.4) 0.0(0.0) 0.7(0.0) 76.6(4.2) 0.0(0.0) 0.74(0.0) 83.18(1.3) 74.0(3.5) 0.8(0.0)

Option 1 83.2(0.3) 79.1(1.0) 0.8(0.0) 82.3(0.5) 78.8(1.9) 0.8(0.0) 79.8(5.9) 68.1(13.9) 0.8(0.1)

Option 2 83.4(0.4) 79.8(0.3) 0.8(0.0) 80.6(1.3) 77.8(2.4) 0.78(0.0) 81.5(1.8) 77.3(2.9) 0.8(0.0)

Option 3 83.0(0.7) 77.4(0.5) 0.8(0.0) 81.7(3.0) 76.5(2.6) 0.8(0.0) 83.19(1.2) 77.5(2.0) 0.8(0.0)

6.2 Results with Mod-NNs For Cayo dataset with Modular NN, as happened Non-Modular NN, the use of cost functions improved the accuracy results, except with RVFLNN. It is important to note that the behavior of Modular and Non-Modular NN is different for the same database (compare the accuracy classification for the three different NN with Option 0 in tables 3 and 7). In Cayo dataset with Modular NN the best results where obtained for Option 1 for the MLP and RBFNN, and Option 3 for RVFLNN. Ecoli6 database with Modular NN presents a similar behavior as the NonModular NN, using cost functions does not suppose an improvement in the results (table 8). When Modular NN and cost functions are used with Feltwell dataset, several behaviours can be seen (table 9). We can be note that in MLP the results are worse when the Options are applied, but in RBFNN and RVFLNN we can talk of a light improvement in the global performance. Only Option 3 reports better classifier performance.

The Multi-Class Imbalance Problem

429

Table 8 Classification performance of Ecoli6 with Modular Networks

MLP

RBFNN

RVFLNN

Acc g-mean Kappa Acc g-mean Kappa Acc g-mean Kappa

Option 0 87.1(2.9) 83.7(5.9) 0.8(0.0) 86.4(3.0) 84.6(4.9) 0.8(0.0) 87.9(2.8) 85.0(5.7) 0.8(0.0)

Option 1 84.9(3.5) 83.3(3.5) 0.8(0.0) 85.2(7.4) 84.7(8.2) 0.8(0.1) 85.9(5.8) 84.9(4.6) 0.8(0.1)

Option 2 85.2(3.9) 83.8(3.6) 0.8(0.0) 84.9(5.0) 84.9(5.4) 0.8(0.0) 85.2(3.4) 85.6(3.6) 0.8(0.0)

Option 3 85.5(3.9) 83.3(5.9) 0.8(0.0) 85.2(6.0) 86.0(4.8) 0.8(0.0) 87.0(5.0) 85.6(6.0) 0.8(0.1)

Table 9 Classification performance of Feltwell with Modular Networks

MLP

RBFNN

RVFLNN

Acc g-mean Kappa Acc g-mean Kappa Acc g-mean Kappa

Option 0 89.8(0.1) 87.1(0.2) 0.9(0.0) 89.5(0.6) 87.4(1.0) 0.9(0.0) 88.3(2.0) 82.4(4.2) 0.8(0.0)

Option 1 87.2(0.3) 84.7(0.3) 0.8(0.0) 89.0(1.0) 87.1(1.7) 0.9(0.0) 88.7(1.0) 86.1(1.0) 0.9(0.0)

Option 2 86.8(0.4) 84.3(0.4) 0.8(0.0) 89.0(1.0) 87.0(1.1) 0.8(0.0) 89.9(1.2) 88.1(1.2) 0.8(0.0)

Option 3 88.6(0.4) 85.9(0.5) 0.8(0.0) 90.4(0.5) 88.2(0.2) 0.8(0.0) 90.7(0.6) 88.2(1.0) 0.8(0.0)

Table 10 Classification performance of Satimage with Modular Networks

MLP

RBFNN

RVFLNN

Acc g-mean Kappa Acc g-mean Kappa Acc g-mean Kappa

Option 0 81.8(0.8) 47.5(2.8) 0.8(0.0) 83.9(1.5) 73.9(5.9) 0.8(0.02) 84.9(1.0) 77.3(1.3) 0.8(0.0)

Option 1 85.4(0.6) 82.8(1.0) 0.8(0.0) 85.4(1.4) 84.0(1.6) 0.8(0.0) 86.0(0.4) 83.9(0.2) 0.8(0.0)

Option 2 84.7(0.5) 82.8(0.9) 0.8(0.0) 86.2(0.8) 84.4(0.9) 0.8(0.0) 86.0(0.1) 83.9(0.4) 0.8(0.0)

Option 3 84.8(0.3) 81.4(0.8) 0.8(0.0) 85.0(1.3) 81.5(1.5) 0.8(0.0) 85.9(1.0) 82.5(1.0) 0.8(0.0)

In Satimage, a better classifier performance is observed when the cost functions are used (in table 10). Option 1 reports better accuracy results in models MLP and RVFLNN, and a similar improvement is shown with Option 2 in RBFNN.

430

R. Alejo et al.

7 Conclusions In this work, the class imbalance problem for multiple classes is analyzed by means of Modular and Non Modular NN trained with the backpropagation algorithm in batch mode for four different databases. Three strategies have been studied in order to balance the MSE for each class contribution. Each strategy consists of using different kinds of cost functions in the training algorithm. The proposed strategies improve the Modular and Non Modular NN (implemented for MLP, RBFNN and RVFLNN models) performance over the less representative classes contained in the TS, reducing the class imbalance problem in the training process with the added objective of improving the general performance during the classification phase. But in some cases it does not improve the global results. The most clear cases is the Ecoli6 database, which has a very reduced sample set. We can not decide which cost function (Option 1, 2, 3) is better: in Non Modular NN the Option 1 has significance statistical improvements in 8.3% of the experiments, Option 2 in about 41.6% and Option 3 in 33.3%. While in the Modular NN we have 33.3% for Option 1 and Option 3, and 16.7% for Option 2. Future research must consider the relationship between TS imbalance and data complexity (overlapping, noise or decision frontiers), and include the techniques for to reduce the data complexity. Also severe TS imbalance (for example in remote perception images) must be further considered. Acknowledgements. This work has been partially supported by grants DPI2006-15542C04-03 from the Spanish CICYT, SEP-2003-C02-44225 from the Mexican CONACyT, Generalitat Valenciana under the project GV/2007/105, CB-2008-01-107085 from the Mexican CONACyT and PROMEP/103.5/08/3016 from the Mexican SEP.

References 1. Japkowicz, N., Stephen, S.: The Class Imbalance Problem: a Systematic Study. Intelligent Data Analysis 6, 429–449 (2002) 2. Kotsiantis, S., Pintelas, P.: Mixture of Expert Agents for Handling Imbalanced Data Sets. Annals of Mathematics and Computing & TeleInformatics 1, 46–55 (2003) 3. Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: Smote: Synthetic Minority Over-Sampling Technique. J. Artif. Intell. Res. (JAIR) 16, 321–357 (2002) 4. Anand, R., Mehrotra, K., Mohan, C.K., Ranka, S.: Efficient Classification for Multiclass Problems Using Modular Neural Networks. IEEE Transactions on Neural Networks 6, 117–124 (1995) 5. Zhou, Z.H., Liu, X.Y.: Training Cost-Sensitive Neural Networks with Methods Addressing the Class Imbalance Problem. IEEE Transactions on Knowledge and Data Engineering 18, 63–77 (2006) 6. Bruzzone, L., Serpico, S.B.: Classification of Imbalanced Remote-Sensing Data by Neural Networks. Pattern Recognition Letters 18, 1323–1328 (1997) 7. Auda, G., Kamel, M.: Modular Neural Network Classifiers: A Comparative Study. Journal of Intelligent and Robotic Systems 21, 117–129 (1998)

The Multi-Class Imbalance Problem

431

8. Eric, R., Gawthrop, P.: Modular Neural Networks: A State of the Art. Technical Report CSC-95026, Centre for System and Control. Faculty of mechanical Engineering, University of Glasgow, UK (1995) 9. Jacobs, R.A., Jordan, M.I., Nowlan, S.J., Hinton, S.J.: Adaptive Mixtures of Local Experts. Neural Computation 3, 79–87 (1991) 10. Looney, C.: Pattern Recognition Using Neuronal Networks - Theory and Algorithms for Engineers and Scientists, 1st edn. Oxford University Press, New York (1997) 11. Ding, C., Xiang, S.Q.: From Multilayer Perceptrons to Radial Basis Function Networks: A Comparative Study. In: EEE. Conference on Cybernetics and Intelligent Systems, vol. 1, pp. 69–74 (2004) 12. Pao, Y.H., Park, G.H., Sobajic, D.J.: Learning and Generalization Characteristics of the Random Vector Functional-Link Net. Neurocomputing 6, 163–180 (1994) 13. Anand, R., Mehrotra, K.G., Mohan, C.K., Ranka, S.: An Improved Algorithm for Neural Network Classification of Imbalanced Training Sets. IEEE Transactions on Neural Networks 4, 962–969 (1993) 14. Alejo, R., Garcia, V., Sotoca, J.M., Mollineda, R.A., Sanchez, J.S.: Improving the Performance of the RBF Neural Networks with Imbalanced Samples. In: 9th International Work-Conference on Artificial Neural Networks, pp. 162–169. Springer, Heidelberg (2007) 15. Kuncheva, L.I.: Combining Pattern Classifiers: Methods and Algorithms. WileyInterscience, Hoboken (2004)

Geometry Algebra Neuron Based on Biomimetic Pattern Recognition Wenming Cao and Feng Hao 1

Abstract. Biometric Pattern Recognition aim at finding the best coverage of per kind of sample’s distribution in the feature space. It is based on the analysis of relationship of sample points in the feature space. According to the principle of “same source”, research the same kind of samples’ distribution in the feature space can get eigenvector information with low data amount. This can be realized by ‘coverage recognizing method of complex geometric body in high dimensional space’. Self-adaptive topological structure of high dimensional geometrical neuron model offers theoretical basis for its realization. But it has been investigated in 2D sample space. In this paper ,we extend to Multispectral Image sample space by Clifford Algebra,and propose geometry algebra neuron by biomimetic pattern recognition theory. The experiment result proves the efficiency of our theory. Keywords: Biometric Pattern Recognition, Geometry Algebra,Geometry Algebra Neuron, Multispectral Image sample.

1 Introduction There are finite or infinite sample points between two different samples of a same kind of things. We call this as “same source”. In mathematical language it is described as, there is a set:

B = {x1 = x, x2 ,

, xn = y ρ ( xm , xm+1 ) < ξ , m ∈ [1, n − 1], m, n ∈ N , ξ > 0,} ⊂ A

(1) n

Where A is a point set consists of things belong to A in the feature space R . In this paper, based on biomimetic pattern recognition, we consider the concept of Wenming Cao . Feng Hao Jiaxing, Zhejiang 100083, China Wenming Cao School of Information Engineering, Shenzhen University, Guangdong 518060, China [email protected], [email protected] H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 433–440. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

434

W. Cao and F. Hao

color and multispectral image recognition[1]. A multispectral (multicolor,multicomponent) image contains more than one component. An RGB image is an example of a color image featuring three separate image components R (red), G(green), and B (blue). In the classical approach every multicolor pixel (in particular, color pixel) is associated to a point of a kD multicolor vector space (to a point of a 3D RGB vector space for color images)[2-4]. Our proposed approach is that each image multicolor pixel is considered not as a kD vector, but as a multiplet number, and multicolor (color) space is identified with the so-called multiplet (triplet) algebra. The aim is to show that the use of Clifford algebras[5] fits more naturally to the tasks of multicolor image processing and recognition of multicolor patterns than does the use of color vector spaces. We give algebraic models of multispectral image using different hypercomplex[6] and Clifford algebras and mention new methods of image recognition based on an algebraic-geometric theory of invariants. In this approach, each color or multicolor pixel is considered not as a kD vector, but as a kD hypercomplex number. The remainder of this paper is organized as follows. Section 2 describes Clifford algebras as models of physical spaces, and Biomimetic Pattern Recognition Theory based on Clifford Algebra,and proposes the metric between two Clifford numbers.. Section 3 introduces 3D face recognition algorithm based on Clifford algebras.Experiments results, experimental analysis and conclusion are given in Section 4 and Section 5 respectively.

2 Geometric Algebra We suppose that some hypercomplex-valued invariants of an image are calculated when recognizing it. Hypercomplex algebras generalize the algebras of complex numbers, quaternions and octonions. Of course, the algebraic nature of hypercomplex numbers must correspond to the spaces with respect to geometrically perceivable properties. For recognition of 2D, 3D and nD images we turn the

R 3 and R n into corresponding algebras of hypercomplex numbers. Let n “small” nD space R be spanned by the orthonormal basis of n space

spaces R

2

hyperimaginary units I i , i = 1, 2,..., n . We assume

⎧ +1 ⎪ I = ⎨ −1 ⎪0 ⎩ 2 i

i = 1, 2,..., p, i = p + 1, p + 2,...., p + q, i = p + q + 1, p + q + 2,....., p + q + r = r

and I i I j = − I i I j . Now, we construct the “big”

(2)

n

2n D hypercomplex space R 2 .

Let b = (b1 , b2 ,......., bn ) ∈ B2n be an arbitrary n-bit vector, where bi ∈ B2 = {0,1} n

and B2 is the nD Boolean algebra. Let us introduce I b := I1b1 I 2b2 ......I nbn . Then

2n elements I b form a basis of 2n D space, i.e., for all M ∈ R 2 we n

Geometry Algebra Neuron Based on Biomimetic Pattern Recognition

have M :=

∑c I

b b

b∈B2n

435

. We denote these algebras by A A2Spn ( p ,q ,r ) ( R | I1 ,....., I n ) ,

A2Spn ( p ,q ,r ) or A2Spn , if I1 ,....., I n , p, q, r are fixed. The conjugation operation in

A2Spn ( p ,q , r ) maps every Clifford number

e := c0 I 0 + ∑ b ≠ 0 cb I b to the number e = c0 I 0 − ∑ b ≠ 0 cb I b . The algebras A2Spn ( p ,q , r ) are transformed into 2n D pseudo-metric spaces designated by ( p ,q ,r ) p ,q ,r eLSp or eL2 n , if the pseudo-distance between two Clifford numbers A and 2n

B is defined by ρ ( A,B ) = ( A − B )( A − B ) . Subspaces of pure vector Clifford numbers The

x1 I1 + ...+ xn I n ∈ Vec1 ( A2Spn ) are nD spaces R n := gR nSp ( p ,q ,r ) .

pseudo-metrics

constructed Sp ( p , q , r )

corresponding pseudo-metrics in gR n We can make

( p ,q ,r ) eLSp 2n

in

induce

.

A2Spn , be a ranked and Z / 2 Z -graded algebra. Let r(b) be the

Hamming weight (= rank) of b, i.e., a functional r : B2 → [0, n − 1] defined by n

n

r (b) := ∑ bi , and let ∂ (b) = r (b)(mod 2) be the grad of b. Then A2Spn can be i =1

represented as the ranked and Z / 2 Z -graded sums

A2Spn = ⊕ nr =0 A2[nr ] and

R 2 = ⊕1∂= 0 A2{n∂} , where the dimension of the vector space A2[nk ] equals the n

k

binomial coefficient Cn and

n

∑C k =0

The dimensions of

k n

= 2n .

{1} [k ] n −1 A2{0} and A2 n are equal to 2 . The subspaces A2 n are n

spanned by the k-products of units I i1 I i2 ...I ik (i1 < i2 < ... < ik ) , i.e., by all basis vectors with r (b) = k . Every element e := tations: e = e

[0]

∑

b∈B2n

cb I b of A2Spn has the represen-

+ e [1] + ... + e [ n ] and e = e {0} + e {1} , where e [0] ∈ A2[0] is the n

scalar part of the Clifford numbers, bivector part, . . . ,

e [1] ∈ A2[1]n is its vector part, e [2] ∈ A2[2] is its n

e [ n ] ∈ A2[nn ] is its n-vector part, and, finally, e {0} and e {1} are

even and odd parts of the Clifford number e . If

e ∈ A2{nl } , we put ∂ (e ) = l and

say that l is the degree of e .Multiplication of two Clifford numbers of ranks k and s gives the sum of Clifford numbers from

436

W. Cao and F. Hao

|k −s|

to

p = min(k + s, 2n − k − s )

[|k − s|]

[|k − s|+2]

with

increment

2,

i.e.,

A B =e +e + ... + e . In A2 ( R ) we introduce a conjugation operation which maps every element z = x1 + Ix2 to the element z = x1 − Ix2 . Now, the generalized complex plane [k ]

[s]

[ p]

is turned into the pseudo-metric space A2 ( R ) → ge 2

Sp ( p , q , r )

if one defines the

pseudo-distance as:

⎧ ( x − x ) 2 + ( y − y ) 2 , if z ∈ A ( R | i ) 2 1 2 1 2 ⎪ ⎪ 2 2 ρ ( z1 , z2 ) = ⎨ ( x2 − x1 ) − ( y2 − y1 ) , if z ∈ A2 ( R | e) ⎪| x − x |, if z ∈ A2 ( R | ε ) ⎪⎩ 2 1

(3)

where z1 := ( x1 , x2 ) = x1 + Ix2 , z2 := ( y1 , y2 ) = y1 + Iy2 . So, the plane of the Sp (2,0,0)

classical complex numbers is the 2D Euclidean space ge 2 numbers plane is the 2D Minkowskian space ge

Sp (1,1,0) , 2

, the double

and the dual numbers

Sp (1,0,1) . 2

plane is the 2D Galilean space ge When one speaks about all three algebras (or geometries) simultaneously, then the corresponding algebra (or geometry) is that of generalized complex numbers, denoted by

A2Sp ( p ,q , r ) (or

ge 2Sp ( p ,q , r ) ).

3 Geometric Algebra Neuron 3.1 Geometric Algebra Neuron A mathematical description of GA Neuron P of class A:

P = ∪ Pi

(4)

Pi = { X | ρ ( X , Y ) ≤ k , Y ∈ Βi , X = e + x, x ∈ G n }

(5)

i

Bi = { X | X = α si + (1 − α ) si +1 , α ∈ [ 0,1]}

(6)

n

Where k is a constant, G is n -grade Geometric Algebra, si is a sample of a clas s A in feature space. Let

ρ ( X , L ) be

L = p ∧ v , p is a point in line

the distance from X to line vector L ,

L , v is the direction vector of L . Then

Geometry Algebra Neuron Based on Biomimetic Pattern Recognition

⎧ 2 ρ ( X , p ) ≤ ρ ( X , p + v) and ⎪ ρ ( X , p), ⎪ ⎪ 2 ρ ( X , L ) = ⎨ ρ 2 ( X , p + v), ρ ( X , p + v ) ≤ ρ ( X , p ) and ⎪ ⎪ ρ 2 ( X , X '), otherwise ⎪ ⎩ where

i=

X '=

437

X '− p =− X '− p X '− p v = X '− p v

(7)

x ∧ (exp(−iπ / 2)v) p∧v exp(−iπ / 2)v , v+ (exp(−iπ / 2)v) ∧ v v ∧ (exp(−iπ / 2)v)

x∧ p And the GA Neurons S ( L; r ) is: x∧ p S ( L; r ) = { X | ρ 2 ( X , L ) < r 2 }

If L is a point (it means v is a scalar) in feature space, then lent to d ( x, p ) , and

(8)

d ( X , L ) is equiva-

S ( L; r ) is equivalent to S ( p; r ) (GA Neurons be-

comes a hypersphere). If v is a vector, then a GA Neuron S ( L; r ) is a compact

coverage compared to a hypersphere. A new neuron model, a GA Neuron is defined by the input-output transfer function: f ( X ; L ) = φ ( ρ ( X , L)) Where

φ (⋅)

(9)

is the non-linearity of the neuron, the input vector X = e + x, x ∈ G n ,

and L = p ∧ v , p is a point in line L , v is the direction vector of L . A typical choice of φ ( ⋅) was the Gaussian function used in data fitting, and the threshold function was a variant of the Multiple Weights Neural Network(MWNN) [9,10,11]. The network used in data fitting or system controlling consisted of an input layer of source nodes, a single hidden layer, and an output layer of linear weights. The network implemented the mapping ns

f s ( X ) = λ0 + ∑ λiϕ ( ρ ( X , Li ) )

(10)

i =1

where

λi ,0 ≤ i ≤ ns

were weights or parameters. The output layer made a

decision on the outputs of the hidden layer. One mapped

438

W. Cao and F. Hao

f s ( X ) = max φ ( ρ ( X , Li ) ) ns

(11)

i =1

where φ ( ⋅) was a threshold function. In this paper, ni = 3 .

3.2 GA Neuron Classifier The function of classifier in Patten Recognition is: according to the eigenvectors from feature extractor to mark the tested objects[8]. A classifier could be considered as a net or machine, which compute r distinguish function and select a sort of model corresponding to maximum of the functions. There are many ways to describe pattern classifier. The most usual used is the form: g k ( x), k = 1, , r . for all j ≠ k , If g k ( x) > g j (x)

(12)

。Distinguish function of GA N euron Classifier is equation (12), that is g ( X ) = f ( X ) = max φ ( ρ ( X , L ) ) 。

The classifier judge the eigenvector x as pattern k

ns

k

s

i =1

i

3.3 GA Neuron Net Learning Algorithm Let S denote the filtered set that contains the expression pattern which determine the network and X denote the original set that contains all the expression pattern in the order. Begin 1. Put the first expression pattern into the result set S and let it be the fiducially expression pattern sb , and the distance between the others and it will be com-

S = {sb } . smax = sb and ρ max = 0 2. If no expression pattern in the original set X , stop filtering. Otherwise, check the next expression pattern in X , then compute its distance to sb ,i.e., pared. Set

ρ = s − sb . 3. If ρ > ρ max , go to step 6. Otherwise continue to step 4. 4. If ρ < ε , set smax = s , ρ max = ρ , go to step 2. Otherwise continue to step 5. 5. Put

s into the result set: S = S ∪ {s} , and let sb = s , smax = s , and

ρmax = ρ . Then go to step 2.

Geometry Algebra Neuron Based on Biomimetic Pattern Recognition

6. If | ρ max

439

− ρ |> ε 2 , go to step 2. Otherwise put smax into the result set:

S = S ∪ {s

max

} , and let sb = smax , d max = s − smax go to step2. 3.3 GA Neu-

ron Net learning Algorithm.

4 Experiment and Analysis This experiment uses 300 space target samples with 6 classes for training and testing. Each class has 50 pictures from different angles as illustrated in figure 1.

Fig. 1. Sample pictures the United States, "Cassini" probe, Chang-e IV,Mars Global Surveyor,0809geoeye,Pierwszy polski satellite, polish satellite,Spy sateline 40 samples in each class are selected as training samples for recognition respectively, meanwhile 5 new samples in each class are selected as testing samples that means there are 270(40*6+5*6) samples totally for recognition Table 1 Experiment result Training samples numbe Testing samples numbe Accurate recognized number

20*6

25*6

30*6

35*6

40*6

120

120

120

120

120

92

103

108

110

115

Accurate rate

76.66%

85.81%

90.00%

91.66%

95.8%

Error recognized number

12

11

5

5

4

Error rate

10.00%

0.911%

4.16%

4.16%

3.83%

6

5

5

4

4

5%

4.81%

4.81%

3.33%

3.33%

Refused recognized number Refused rate

440

W. Cao and F. Hao

Table 1 illustrates the recognition result with different numbers of training samples. The result proves the efficiency of our method. Some testing samples are refused to be recognized from table 1 due to the threshold. It also illustrates the efficiency of the method to recognize the samples that do not belong to any trained class. The quality of image and the space target sample in image can result the measurement error which increases the distance between testing sample and sub-manifold centroid. This may be the reason of error rate and refused rate.

5 Conclusions The hypercomplex algebras and geometric algebra neuron are used to analyzed the problems found in the color and multispectral image processing and recognition effectively. We research the property of space color image, and propose biomimetic pattern recognition based on Clifford algebra with high dimensional space theory[8]. The experiment result proves the efficiency of our theory, and lays the foundation for next job. Acknowledgment. This paper is supported by National Natural Science Foundation of China (NNSF) No. 60871093, Pre-Research and Defense Fund of China: No.9140C80002080C80.

References 1. Labunets, R.E.V., Labunets, V.G., Astola, J.: Is the Visual Cortex a Fast Clifford Algebra Quantum Compiler. In: A NATO Advanced Research Workshop Clifford Analysis and Its Applications, pp. 173–183 (1999) 2. Labunets, R.E.V.: Fast Fourier–Clifford Transforms Design and Application in Invariant Recognition. PhD Thesis of Tampere University Technology, Tampere, Finland, pp. 262–300 (2000) 3. Labunets, V.G., Rundblad, E.V., Astola, J.: Is the Brain Clifford Algebra Quantum Computer. In: Proc. of SPIE Materials and Devices for Photonic Circuits, pp. 134–145 (2001) 4. Labunets, V.G., Rundblad, E.V., Astola, J.: Fast Invariant Recognition of Color 3D Images Based on Spinor–Valued Moments and Invariants. In: Proc. Of SPIE Vision Geometry X, pp. 22–33 (2001) 5. Lasenby, A.N., Doran, C.J.L., Gull, S.F.: Lectures in Geometric Algebra.Clifford (Geometric) Algebras with Applications to Physics. In: Mathematics and Engineering, Birkhouser, Boston, pp. 256–288 (1996) 6. Vela, M.: Explicit Solutions of Galois Embedding Problems by Means of Generalized Clifford algebras. In: Symbolic Computation, pp. 811–842 (2000) 7. Wang, S.J.: A New Development on ANN in China – Biomimetic Pattern Recognition and Multi Weight Vector Neurons. LNCS(LNAI), pp. 35–43. Springer, Heidelberg (2003) 8. Wang, S.J., et al.: Multi Camera Human Face Personal Identification System Based on Biomimetic Pattern Recognition. Acta Electronica Sinica, 1–3 (2003)

A Novel Matrix-Pattern-Oriented Ho-Kashyap Classifier with Locally Spatial Smoothness Zhe Wang, Songcan Chen, and Daqi Gao

Abstract. The previous work matrix-pattern-oriented Ho-Kashyap classifier (MatMHKS) can directly deal with images in matrix representation n1 × n2 such that the spatial information within these images is not destroyed. Although MatMHKS works with n1 × n2 per image, it is far less that this spatial correlation with the matrix form n1 × n2 can suggest the real number of freedom. MatMHKS just keeps the relationship of the pixels in the same row or column of images. In this paper we further consider the relationship of the pixels that close to each other may be correlated, and thus develop a new matrix-pattern-oriented Ho-Kashyap classifier named MatHKLSS that is introduced with a locally spatial smoothness. The experimental results here demonstrate that the proposed MatHKLSS has a superior advantage to MatMHKS in terms of classification.

1 Introduction The existing classifier design strategies generally base on vector pattern. When a non-vector pattern such as images is input, the non-vector pattern first has to be vectorized by concatenating its feature elements [1]. The socalled vector-pattern-oriented classifier design indeed brings a convenience while dealing with problems. However, such a vectorization especially for images, not only increases the computational complexity [2], but also breaks down the original spatial or structural information between the elements of Zhe Wang · Daqi Gao Department of Computer Science & Engineering, East China University of Science & Technology, Shanghai 200237, China Songcan Chen Department of Computer Science & Engineering, Nanjing University of Aeronautics & Astronautics, Nanjing 210016, China

H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 441–449. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

442

Z. Wang, S. Chen, and D. Gao

the image [3]. Several researchers have paid attention to the problems about the vectorization, and independently proposed a technique that can directly operate on matrix pattern without the vectorization preprocessing. Firstly, a series of matrixized (two-dimensional) feature extraction methods [4, 5] were independently proposed. Two-Dimensional Principal Component Analysis (2DPCA) [5] can extract features directly from image matrices and is shown to be better than the classical Principal Component Analysis (PCA) [6] in terms of both image classification performance and the reduction of computational complexity. Two-Dimensional Linear Discriminant Analysis (2DLDA) [4] is shown to not only increase the classification performance, but also overcome the singularity problem implicitly in classical Linear Discriminant Analysis (LDA) [7]. We found that during the classification processing of the matrixized or twodimensional feature extraction [4, 5], the classifiers following these matrixized feature extractors still resort to the traditional vector-based technique. In other words, the operating pattern of the subsequent classifier is still of vector representation rather than matrix representation. Intuitively, directly manipulating images in design of a classifier seems simpler and also not to lose too much spatial or local contextual information of the original image. To this end, our previous work was proposing a Matrix-pattern-oriented Modified Ho-Kashyap classifier with Squared approximation of the misclassification errors (MatMHKS) [8]. The advantages of MatMHKS are: 1) preventing the structural information of images from being fully destroyed; 2) decreasing the memory required for the weight vector from n1 × n2 to n1 + n2 ; 3) avoiding overtraining. The objective function of our previous work MatMHKS [8] is given as follows min I = Remp + cRreg , (1) where the empirical risk term Remp considers an image with the matrix representation n1 × n2 , and the regularization term Rreg tries to make the discriminant function smooth. However, even though MatMHKS takes n1 × n2 per image, it is far less that this spatial correlation can reveal the real number of freedom for images. Since the Remp of the equation (1) adopts the mathematical form uT A˜ v where the given image A ∈ Rn1 ×n2 and the weight n1 n2 vectors u ∈ R , v˜ ∈ R , the Remp only considers the relationship of the pixels in the same row or column of images. The Rreg of MatMHKS plays a role as a generalization regularization term but just makes the discriminant function smoothness. But Rreg still does not fully consider the image case, i.e., the spatial information of images. In a word, our previous work MatMHKS should be spatially rough and fails to fully explore the spatial information. The aim of this paper is to remedy the disadvantage of our previous work MatMHKS and to propose a new Matrix-pattern-oriented Ho-Kashyap classifier with Locally Spatial Smoothness named MatHKLSS. The proposed

A Novel Matrix-Pattern-Oriented Ho-Kashyap Classifier

443

MatHKLSS not only considers the relationship of the pixels in the same row or column of images, but also incorporates the prior spatial information of the correlated neighboring pixels. Thus while dealing with images, the proposed MatHKLSS here can utilize the local spatial information that the neighboring pixels of images should be similar. In practice, the MatHKLSS of this paper is designed by introducing a new regularization term Rloc into the objective function of MatMHKS. And the introduced term Rloc exactly suggests the prior information that the neighboring pixels of images should be similar. The experiments here compare the proposed MatHKLSS with our previous work MatMHKS, and demonstrate the effective classification of the newly proposed MatHKLSS. The rest of this paper is organized as follows. Section 2 gives the formulation of the proposed new Matrix-pattern-oriented Ho-Kashyap classifier with Local Spatial Smooth (MatHKLSS). Section 3 reports on all the experimental results. Finally, conclusion and future work are given in Section 4.

2 Matrix-Pattern-Oriented Ho-Kashyap Learning with Locally Spatial Smoothness (MatHKLSS) Suppose that there is an image matrix sample set S = {(A1 , ϕ1 ), ..., (AN , ϕN )}, where N is the sample number, Ap ∈ Rn1 ×n2 , and the corresponding class label ϕp ∈ {+1, −1}. In our previous work MatMHKS [8], its objective function is given as follows min I(u, v ˜, v0 , bp ) =

N

2

(ϕp (uT Ap v ˜ + v0 ) − 1 − bp ) + c(uT u + v ˜T v ˜), (2)

p=1

˜ ∈ Rn2 , the bias v0 ∈ R, the c ≥ 0 is a where the weight vectors u ∈ Rn1 , v regularized parameter, and bi ≥ 0. It has been stated that for a given image Ai , the first term of the right side of the optimization (2) just utilizes the relationship information between the pixels in the same row or column and the second term just keeps the global smoothness of the discriminant function of MatMHKS. Thus while dealing with images where the prior local smoothness need be emphasized than the global one, a local regularized method or a local spatial smoothness should be introduced. In this paper, for a given image A ∈ Rn1 ×n2 , we introduce the local spatial smoothness into the matrix-pattern-oriented classifier through the newly-proposed regularized term Rloc = (ui vj − us vt )2 , (i,j) (s,t)∈Nij

where ui , us are the ith and sth elements of the weight vector u respectively, vj , vt are the jth and tth elements of the weight vector v ˜ respectively, (i, j) denotes the pixel position that lies in the coordinate with the ith row and

444

Z. Wang, S. Chen, and D. Gao

the jth column of the image A, and (s, t) ∈ Nij denotes that all the (s,t) lie in the 4 or 8 neighbors of the (i, j). Thus, minimizing the term Rloc directly suggests that the neighboring pixels of images have similar values. In the other words, the corresponding element values should be similar if the elements are spatially near. Further, we give the objective function of the newly-proposed Matrix-pattern-oriented Ho-Kashyap learning with Locally Spatial Smoothness (MatHKLSS) as follows min J(u, v ˜, v0 , bp ) =

N p=1

+c

2

(ϕp (uT Ap v ˜ + v0 ) − 1 − bp )

(3)



(ui vj − us vt )2 .

(i,j) (s,t)∈Nij

The first term of the right side of the equation (3) plays the same role as that of the equation (2), i.e., measures the number of the misclassified samples and keeps the values of the pixels in the same row or column similar through the mathematic form uT Ap v ˜, which also accords with the statement that the values in the same row or column of images have a common divisor [9]. As the above said, the second term of the right side of the equation (3) suggests the local spatial smoothness, i.e., that the element values in neighbors would be similar if the elements are spatially near. Here, to get a simply mathematical form of the equation (3), we simplify the newly-proposed term Rloc as following   Rloc = (ui vj − us vt )2 ≈

(i,j) (s,t)∈Nij n 1 −1

(ui−1 − 2ui + ui+1 )2 +

i=2

where

n 2 −1

(vj−1 − 2vj + vj+1 )2 ,

j=2

n 1 −1

(ui−1 − i=2 n 1 −1 T

2ui + ui+1 )2

(u (ei−1 − 2ei + ei+1 ))2

=

(4)

i=2 n 1 −1 T

(5)

(ei−1 − 2ei + ei+1 )(ei−1 − 2ei + ei+1 )T ]u

=u [

i=2

= uT Ru u, Ru =

n 1 −1

(ei−1 − 2ei + ei+1 )(ei−1 − 2ei + ei+1 )T , ei = [0...1...0]T ∈ Rn1 and

i=2

the ith element of ei is 1, ui = uT ei . Similarly, we can also get n 2 −1

(vj−1 − 2vj + vj+1 )2 = v ˜T Rv v ˜.

j=2

(6)

A Novel Matrix-Pattern-Oriented Ho-Kashyap Classifier

445

Consequently, by introducing the equation (5) and (6) into the objective function (3), we can get a simplified form as following min J(u, v ˜, v0 , bp ) =

N

2

(ϕp (uT Ap v ˜ + v0 ) − 1 − bp ) + c(uT Ru u + v ˜T Rv v ˜),

p=1

(7) where Ru = Rv =

n 1 −1

(ei−1 − 2ei + ei+1 )(ei−1 − 2ei + ei+1 )T , ei ∈ Rn1

i=2 n 2 −1

(ej−1 − 2ej + ej+1 )(ej−1 − 2ej + ej+1 )T , ej ∈ Rn2 .

j=2

It can be found that both Ru and Rv are constant values while n1 , n2 are given, which can bring a simpleness for optimizing the function (7). Then, by setting Y = [y1 , ..., yN ]T , yp = ϕp [uT Ap , 1]T , i = 1, ..., N, v = [˜ vT , v0 ]T , the function (7) can be rewritten in matrix form as following min J(u, v ˜, v0 , b) = (Y v − 1 − b)T (Y v − 1 − b) + c(uT Ru u + vT R˜v v), (8)   0 R v N . From the equations (7) or (8), we where 1, b ∈ R and R˜v = 0 0 cannot directly find closed-form optimal weights, instead, use the gradient descent technique to iteratively seek them. The gradients of the objective functions (7) and (8) with respect to u, v and b are N ∂J =2 ϕp Ap v ˜[ϕp (uT Ap v ˜ + v0 ) − 1 − bp ] + 2cRu u, ∂u p=1

∂J = 2Y T (Y v − 1 − b) + 2cR˜v v, ∂v ∂J = −2(Y v − 1 − b). ∂b Further through setting

∂J ∂u

= 0 and

∂J ∂v

(10) (11)

= 0, we can get

N N u=( Ap v ˜v ˜T ATp + cRu )−1 [ ϕp (1 + bp − ϕp v0 )Ap v ˜], p=1

(9)

(12)

p=1

v = (Y T Y + cR˜v )−1 Y T (1 + b).

(13)

From the equations (12) and (13), it can be seen that the weight vectors u and v are all determined by the margin vector b whose components determine the distance of the corresponding sample to the separation hyperplane. Moreover, since u and v are also mutually dependent, we seek the u and v by initializing b and the iteration. The design procedure for MatHKLSS is shown in Table 1.

446

Z. Wang, S. Chen, and D. Gao

Table 1 Algorithm MatHKLSS Input: S = {(A1 , ϕ1 ), ..., (AN , ϕN )} OutPut: the weight vectors u, v ˜, and the bias v0 1. fix c ≥ 0, 0 < ρ < 1; initialize b(1) ≥ 0 and u(1); set the iteration index k = 1; 2. Y = [y1 , ..., yN ]T , where yp = ϕp [u(k)T Ap , 1]T ; 3. v(k) = (Y T Y + cR˜v )−1 Y T (1 + b(k)); 4. e(k) = Y v(k) − 1 − b(k); 5. b(k + 1) = b(k) + ρ(e(k) + |e(k)|); 6. if  b(k + 1) − b(k) > ξ, then go to Step 7, else stop; N N   7. u(k + 1) = ( Ap v ˜(k)˜ v(k)T ATp + cRu )−1 ( ϕp (1 + bp (k) − ϕp v0 )Ap v ˜(k)), p=1

p=1

k = k + 1, go to Step 2.

In Table 1, k denotes the iteration step, b(k + 1) = b(k) + ρ(e(k) + |e(k)|) prevents all the components of b from decreasing zero, and ξ is a preset terminated parameter. Finally, the discriminant function of the proposed MatHKLSS for input pattern A ∈ Rn1 ×n2 can be given in terms of  > 0 , A ∈ class + 1 T f(A) = u A˜ (14) v + v0 < 0 , A ∈ class − 1 where the weight vectors u and v ˜ and the bias v0 are obtained in the process of training.

3 Experiments In this section, we compare the newly-proposed matrix-pattern-oriented classifier with local spatial smoothness MatHKLSS with our previous work MatMHKS [8] in terms of classification and computational complexity. The used images are Olivetti Research Laboratory (ORL) face database (available at http://www.cam-orl.co.uk) and Letter text database (available at http://sun16.cecs.missouri.edu/pgader/CECS477/NNdigits.zip), respectively. The ORL data set has images with the size 28 × 23 from 40 persons, each of which provides ten different images. The main challenge on this data set is of pose and expression variations. For ORL data set, we have employed the first six images per person for training and the rest for testing. Letter data set contains ten text classes consisted of the digits from ’0’ to ’9’, each of which provides 50 different images of the size 24 × 18. For Letter data set, we have employed 30 images per digit for training and the rest for testing. All computations are run on Pentium IV 1.40-GHz processor running, Windows XP Professional and MATLAB environment. The involved parameters

A Novel Matrix-Pattern-Oriented Ho-Kashyap Classifier

447

of both MatHKLSS and MatMHKS are given as follows: bp1 = 10−4 , ρp = 0.99, ξ = 10−4 , and the regularization parameter c is selected from the set {10−4 , 10−3 , ..., 103 , 104 } by N -fold cross validation [10] that randomly splits the samples into two parts (the training and testing sets) and repeats the procedure N times. In our experiments, N is set to 10.

3.1 Classification Performance Comparison This subsection compares the newly-proposed MatHKLSS with the previous MatMHKS in terms of classification. The difference between MatHKLSS and MatMHKS is the second regularized term of their objective function of (2) and (7). Since MatHKLSS emphasizes the neighboring relationship of pixels of images, its second regularized term (uT Ru u + v ˜T Rv v ˜) requires that the corresponding pixel values should be similar if the pixels are spatially near. In contrast, the previous work MatMHKS just requires the global smoothness of its discriminant function through (uT u + v ˜T v ˜). Intuitively, the local or neighboring information plays an important role in the image classification. Thus, we implement MatHKLSS and MatMHKS on the image datasets Letter and ORL, respectively. Since both MatHKLSS and MatMHKS can directly deal with image samples, both Letter and ORL are kept the original size, i.e., Letter with 24 × 18 and ORL with 28 × 23. Figure 1 gives the classification accuracies of MatHKLSS and MatMHKS on Letter and ORL, respectively. In this figure, the horizontal axis denotes the logarithm value of the regularized parameter c and the vertical axis denotes the classification accuracies. From Figure 1, it can be found that while the c decreases the accuracies of MatHKLSS is close to that of MatMHKS. This phenomenon attributes to that both (2) and (7) would degenerate the same empirical risk

100

100

MatHKLSS MatMHKS

90

80 percentage of Recognition(%)

percentage of Recognition(%)

80 70

60 50 40

30

70 60

50 40 30

20

20

10

10

0 −4

MatHKLSS MatMHKS

90

−3

−2

−1 0 1 value of log(c) (Letter)

2

3

4

0 −4

−3

−2

−1 0 1 value of log(c) (ORL)

2

3

4

Fig. 1 The classification accuracy (%) comparison of MatHKLSS and MatMHKS with varying values of the regularized parameter c on Letter and ORL, respectively

448

Z. Wang, S. Chen, and D. Gao

N 2 term p=1 (ϕp (uT Ap v ˜ + v0 ) − 1 − bp ) if the value c becomes zero. On the other hand, while the value c becomes large, the objective functions (2) and (7) both focus on their regularized terms, respectively. In our opinion, since the Rloc of the newly-proposed MatHKLSS utilizes the prior neighboring pixels of images, MatHKLSS should outperform MatMHKS, which is validated by the left sub-figure of Figure 1. But the right sub-figure of Figure 1 shows that MatMHKS is better than MatHKLSS while the c equals to 102 and 103 , which should be further discussed. It should be noted that Figure 1 shows that the accuracy of the previous MatMHKS fast decreases while the c becomes large, which demonstrates that the newly-proposed MatHKLSS introduces more information than MatMHKS.

3.2 Computational Complexity Analysis In this subsection, we give a discussion on the running time of the newlyproposed MatHKLSS and MatMHKS. All the experiments are implemented in the same environment. Figure 2 gives the average training time of MatHKLSS and MatMHKS with the variation of the regularized parameter c on Letter and ORL, respectively. Since both Ru and Rv of the objective function (7) can be computed in advance, MatHKLSS has the same computational complexity as MatMHKS in theory. From this figure, it can also be found that MatHKLSS has a comparable computational complexity to MatMHKS in the c with a smaller value. But the training time of MatMHKS fast decreases in the c with a larger value. Simultaneously, MatMHKS with a much less running time also has a much worse classification performance than its corresponding MatHKLSS comparing Figure 1 with Figure 2.

100

400

90

MatHKLSS MatMHKS

350

80 300 running time (in s)

running time (in s)

70

60 50 40

250 200 150

30 100

20 MatHKLSS MatMHKS

10 0 −4

−3

−2

50

−1 0 1 value of log(c) (Letter)

2

3

4

0 −4

−3

−2

−1 0 1 value of log(c) (ORL)

2

3

4

Fig. 2 The running time (in s) comparison of MatHKLSS and MatMHKS with varying values of the regularized parameter c on Letter and ORL, respectively

A Novel Matrix-Pattern-Oriented Ho-Kashyap Classifier

449

4 Conclusion and Future Work In this paper, we propose a new matrix-pattern-oriented linear classifier MatHKLSS. The contributions of the proposed MatHKLSS are: 1) that MatHKLSS utilizes the prior neighboring relationship between pixels of images through the newly-proposed regularized term Rloc and consequently introduces more information than our previous work MatMHKS [8]; 2) that MatHKLSS need not take more computation than MatMHKS since both Ru and Rv can be got before training. In future, we should implement the proposed MatHKLSS on more image datasets and give more discussion. On the other term Rloc here is just given as one form  side, the newly-proposed  2 (i,j) (s,t)∈Nij (ui vj − us vt ) and in future can be generated to another form if the form can suggest some intuitive information. Acknowledgment. The authors thank Natural Science Foundations of China under Grant No. 60575027, the High-Tech Development Program of China (863) under Grant No. 2006AA10Z315, and High University’s Doctor foundation under grant No. 200802870003 for partial supports respectively.

References 1. Beymer, D., Poggio, T.: Image Representations for Visual Learning. Science 272, 1905–1909 (1996) 2. Chen, L., Liao, M., Lin, J., Yu, G.: A New Lda-based Face Recognition System Which Can Solve the Small Sample Size Problem. Pattern Recognition 33, 1713–1726 (2000) 3. Wang, H., Ahuja, N.: Rank-r Approximation of Tensors: Using Image-as-matrix Representation. In: IEEE CVPR (2005) 4. Li, M., Yuan, B.: 2D-lda: A Statistical Linear Discriminant Analysis For Image Matrix. Pattern Recognition Letters 26, 527–532 (2005) 5. Yang, J., Zhang, D., Frangi, A., Yang, J.: Two-dimension pca: A New Approach to Appearance-based Face Representation and Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 26, 131–137 (2004) 6. Jolliffe, I.: Principle Component Analysis. Springer, Heidelberg (1986) 7. Fisher, R.: The Use of Multiple Measurements in Taxonomic Problems. Annals of Eugenics 7, 179–188 (1936) 8. Chen, S., Wang, Z., Tian, Y.: Matrix-pattern-oriented Hockashyap Classifier with Regularization Learning. Pattern Recognition 40, 1533–1543 (2007) 9. Cai, D., He, X., Han, J., Huang, T.: Learning a Spatially Smooth Subspace for Face Recognition. In: CVPR (2007) 10. Xu, Q., Liang, Y.: Monte Carlo cross Validation. Chemometrics and Intelligent Laboratory Systems 56, 1–11 (2001)

An Integration Model Based on Non-classical Receptive Fields Xiaomei Wang and Hui Wei 1

Abstract. In this paper, we improve a new model of retinal ganglion cell receptive fields (RFs), which is different from conventional model of classical receptive fields (CRFs), and take into account the characteristic that the sizes of receptive fields vary with eccentricities. According to the information processing mechanism of retinal ganglion cell RFs including the non-classical receptive fields (nCRFs), we use the model to integrate image information. The results show that the model can not only enhance the edge contrast, but also transfer the low spatial frequencies which are filtered by the center/surround mechanism of the classical receptive field. So the image processing results are benefit for the integration of image information. The results are consistent with the spatial summation characteristic of ganglion cell RFs. Keywords: Ganglion cell, Non-classical receptive field, Size change, Integration.

1 Introduction Machine vision is of great significance to intelligent computing systems. However, the vast majority of machine vision researchers overlooked the principle of visual physiology, and proposed various algorithms from the engineering point of view. Therefore machine vision lacks a unified and effective algorithm to solve the problem of complex scenes segmentation and the current machine vision can only deal with single local information. But most scenes are composed of a wide variety of local information. In contrast, the human vision system has a strong processing ability to adapt to different environments. When a person observes a scene, the brain will well combine various local visual information and produce visual perception, which can give a harmonious feeling and identify objects of various sizes and characteristics from the very complex background. Therefore, by applying theories of human vision, neuropsychology and cognitive psychology, we are entirely possible to find some new breakthroughs to provide a new prototype of computer vision theory and to solve the difficulties encountered in computer vision. Xiaomei Wang . Hui Wei School of Computer Science, Fudan University, Shanghai 200433, China [email protected] H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 451–459. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

452

X. Wang and H. Wei

Receptive field is the basic unit of structure and function of visual information processing system. And an in-depth progress has been made on the receptive field research in recent years. Every visual cell has a classical receptive field (CRF), which is a small area in the retina that is responsible for eliciting the response of the neuron. The spatial summation property of CRF is to extract boundaries of images. Since the 1960's, many researchers [1, 2] have found that there is still a large scope of area beyond the CRF of a visual neuron, which is called as nonclassical receptive field (nCRF). Light spot stimuli in it can’t cause a direct reaction of the cell, but they can facilitate, inhibit or disinhibit the behavior of that cell. Li Chaoyi et al. studied the spatial summation property of disinhibitory nCRF of cat retinal ganglion cells [3] and lateral geniculate neurons [4]. They found that disinhibitory nCRF can not only enhance the bolder contrast but also compensate the loss of low-frequency which was caused by the antagonistic center/surround mechanism of the CRF. So, it plays an important role in transmitting the image information of area luminance contrast and luminance gradient [5]. In this paper, a receptive field is composed of a CRF and an nCRF. In addition, Li Chaoyi et al. also observed two phenomena: (1) the algebraic sum of the inhibition to the cell in the CRF induced by cells in different regions of the disinhibitory nCRF which are respectively stimulated by two independent light spots is bigger than that induced by those cells which are synchronously stimulated [6]; (2) the light spot array within the disinhibitory nCRF strongly influence the sensitivity of the cell response to the light spot within the CRF. The denser the array is, the lower the sensitivity is, vice versa. This fact shows that, there is mutual inhibition among the cell responses within a disinhibitory nCRF [7]. On the basis of the above fact, we improved the three Gaussians model proposed by Ghosh et al. in order to show the characteristic that mutual inhibition exists between any two cell responses within a disinhibitory nCRF. And we built a two-layer neural network model of retinal ganglion cell RFs in which all RFs were same in sizes [8]. In fact, the sizes of RFs are very different. The ganglion cell RFs in the fovea are much smaller than those in the periphery. And they increase with increasing eccentricities from fovea [9]. So, based on the two-layer neural network model, our study is to discuss the role of RFs with varying size in image information integration, which is of great significance to image segmentation, edge detection, object recognition, etc.

2 An Integration Model Based on Non-classical Receptive Fields 2.1 Neurophysiological Mechanism of Retinal Ganglion Cell NCRF The neurophysiological study shows that [10], the synchronous activities of horizontal cells in large-area affect the activities of photo-receptors by means of spatial summation and feedback, and then cause the ganglion cell response to the stimulates within the disinhibitory nCRF through bipolar cells. Moreover, Li Chaoyi et al. proposed that the bipolar cells far from the CRF center may form the large-scale nCRF of a ganglion cell via amacrine cells, and the RFs of these bipolar cells may form the subunits of the nCRF of a retinal ganglion cell respectively

An Integration Model Based on Non-classical Receptive Fields

453

Fig. 1 Structural diagram of retinal ganglion cell RF, where the red area is the CRF center, the green area is the CRF surround, the blue area is large-scale nCRF, and an area within a yellow circle is a subunit of the nCRF CRF center

CRF surroud

nCRF

[11]. Hence, horizontal cells, bipolar cells and amacrine cells all participate in the formation of the nCRF of a retinal ganglion cell. The structure of retinal ganglion cell RF (including nCRF) is shown in Fig.1.

2.2 The Integration Model Base on the above Neurophysiological mechanism, we design a three-layer neural network model to integrate image information, as shown in Fig. 2. The first layer is composed of photo-receptors. The second one is composed of retinal ganglion cells. And the third one is composed of the integration units. In this model, a RF is composed of an excitatory CRF center, an inhibitory CRF surround and a disinhibitory nCRF. The output of the second layer can be expressed as follow:

GC ( X , Y ) =

∑ ∑ W ⋅ RC ( x, y) − ∑σ ∑σ W

y∈σ1 x∈σ1

1

y∈

2

x∈ 2

2

⋅ RC ( x, y )

⎛ ⎞ + log B ⎜1 + ∑ ∑ W3 ⋅ RC ( x, y ) ⎟ , ⎝ y∈σ 3 x∈σ 3 ⎠ − A1 ⋅e W1 = 2πσ 1

( x − x0 )2 + ( y − y0 ) 2 2σ 12

− A2 ⋅e W2 = 2πσ 2 − A3 W3 = ⋅e 2πσ 3

(1)

,

(2)

,

(3)

( x − x0 ) 2 + ( y − y0 ) 2 2σ 22

( x − x0 )2 + ( y − y0 ) 2 2σ 32

,

(4)

454

X. Wang and H. Wei

Fig. 2 The simulate model of retinal ganglion cell RFs, where the green units represent photo-receptors located in the CRF center, the red units represent the ones located in the CRF surround, the blue units represent the ones located in the nCRF, and the units within a yellow circle compose a subunit of the nCRF

where GC(x, y) is the response of a ganglion cell; RC(x, y) is the image stimulus within a RF ; W1, W2 and W3 are the weighting functions of the photo-receptors within the CRF center, the CRF surround and the disinhibitory nCRF respectively; A1, A2 and A3 are the sensitivity of the CRF center, the CRF surround and the nCRF respectively; σ1, σ2 and σ3 are the radii of the CRF center, the CRF surround and the nCRF respectively; x and y are position coordinates of a photo-receptor; x0 and y0 are the center coordinates of a RF; X and Y are the position coordinates of a retinal ganglion cell; B is the logarithmic base and B>1, which represents the mutual inhibition among the cell responses within a disinhibitory nCRF. The third layer is the integration layer, in which similar outputs of the second layer are integrated together according to the topological neighborhood and the output response of the second layer. The output of the third layer can be expressed as follow:

⎧1 (2n ), ⎪3 (2n ), ⎪ Oi ( x, y ) = ⎨ ⎪ ⎪⎩(2n − 1) (2n ),

mean(GCi ) ≤ 1 n 1 n < mean(GCi ) ≤ 2 n (5)

(n − 1) n < mean(GCi ) ≤ 1

An Integration Model Based on Non-classical Receptive Fields

mean(GCi ) =

1 ∑ ∑ GC ( x, y), k y∈Ωi x∈Ωi

(i = 1, 2,

455

, m)

(6)

where Ωi is the No.i sub-region of the integration layer (i=1,2,…,m); Oi(x,y) is the outputs of the units within Ωi ; mean(GCi) is the input average of the units within Ωi; x and y are position coordinates of an integration unit; k is the number of units within Ωi; n is the number of integration sub-regions. For convenience of calculation, in the model, the layer of photo-receptors covers the range of 10 degrees eccentricity from fovea. Objects are imaged on the fovea, which is the central region of a retina and has the strongest sensitivity. The mean diameter of the fovea is about 5.2 degrees [12]. Therefore, the 10-degree range can sufficiently cover center field of vision. The first layer can receive input images with various sizes. So the amount of photoreceptors in the first layer is not fixed and is the same as that of pixels of each image. According to the fact that the amount of photoreceptors is 100 times that of ganglion cells in physiology, the amount of the cells in second layer is set to about one percent of that in the first layer. The amount of units in the integration layer is the same as that in ganglion cell layer. There is little reference on the study of the precise statistical data about the size change of ganglion cell RFs. According to the data in the paper written by Croner L.J. and Kaplan E. [13], we do a statistical analysis of the sizes of ganglion cell RFs located at different positions in retina. The cells are grouped into five eccentricity ranges (0°–2°, 2°–4°, 4°–6°, 6°–8°, 8°–10°), and the size values of the CRF center within them are shown in table 1. The ratio of CRF surround radius to CRF center radius does not vary systematically with eccentricity from fovea. And for the vast majority of ganglion cells, the ratio is more than 2. The sizes of ganglion cell CRFs are 3-6 times that of nCRF [14]. Table 1 Size values of ganglion cell CRF centers in the range of 10 degrees eccentricity from the fovea (in units of degree) Eccentricity range Radius

0-2 0.03

2-4 0.04

4-6 0.05

6-8 0.06

8-10 0.07

3 Results We use the model to integrate image information, and the processing results of the ganglion cell layer are shown in Fig. 3. Fig. 3(a) is an original picture of a horse. Fig. 3(b) is the processed image by only using CRF, which only retains the image edge information (high spatial frequency components) while completely lose the information of slow change in the brightness gradient (low spatial frequency components).But what we see is not a world only composed of edges. So our complex visual systems are not only limited to edge enhancement when they process image information. On the basis of the edge processing, they should transfer image information to brains as completely

456

X. Wang and H. Wei

Fig. 3 The image processing results of the ganglion cell layer. (a): the original image; (b): the processed image by only using CRF, and the model parameters are A1=1, A2=0.18, σ1=33, σ2=6.7×σ1; (c): the processed images by using RFs (CRFs + nCRFs), which have a constant size in different positions, the model parameters are A3=0.005, σ3=4×σ2, B=1.2; (d): the processed images by using RFs (CRFs + nCRFs), which have a varying constant size in different positions, σ1 take value according to table 1, and the other parameters are the same as those in (c)

An Integration Model Based on Non-classical Receptive Fields

457

as possible. Fig. 3(c) and Fig. 3(d) are the processed images by the whole RF (CRF + nCRF). The RFs in different positions in Fig. 3(c) have a constant size, which is the same as that of the smallest RF in Fig. 3(d). RFs, which have the smallest sizes in the center, increase with the increasing distance from the center in Fig. 3(d). Compared with Fig. 3(b), both Fig. 3(c) and 3(d) can efficiently restore the low spatial frequency information of the original picture, and the restoration don’t counteract the edge enhancement produced by the center/surround mechanism of CRFs, which is consistent with the spatial summation property of nCRF. The similar inputs in the photoreceptor layer can produce the similar outputs in the ganglion cell layer. On the principles of topological neighborhood and output similarity, the integration layer group the similar outputs of the ganglion cell layer together so as to achieve the integration of image information. In addition, by comparing Fig. 3(c) with Fig. 3(d) in the head and neck regions, we can see that Fig. 3(d) filters out more change information of the brightness gradient and better highlights the image edges. This is because the sizes of RFs in the neck and head regions in Fig. 3(d) are larger than those in Fig. 3(c). Hence it is evident that big RF is helpful to the integration of regional image information and small RF is beneficial to the display of image detail information. Fig. 3(d) shows the characteristic that RF increases with increasing eccentricity from fovea, which accord with the physiological characteristics. The processing results of the integration layer are shown in Fig. 4.

Fig. 4 The image processing results of the integration layer. (a): the image processed by the ganglion cell layer; (b): the histogram of the figure (a); (c): the output of the integration layer, when the border values are taken into account, the model parameters are n=4; (d): the output of the integration layer, when the border values are not taken into account, the model parameters are n=2

458

X. Wang and H. Wei

Fig. 4(a) is the image processing results of the ganglion cell layer. Fig. 4(b) is the histogram of Fig. 4(a). Because the model enhances the edge contrast and results in Mach band effect, the bright belt and the dark band can be seen on both sides of the edge of the horse. So the highest and the lowest values in the histogram correspond respectively to the bright band and the dark band of the border. When we take into account the border in Fig. 4(a) and Fig. 4(b), the values in histogram can be divided into four integration sub-regions, namely, the bright belt of the edge, the dark band of the edge, the horse and the background. The integration result is shown in Fig. 4(c). When we don’t take into account the border in Fig. 4(a) and Fig. 4(b), the values in histogram can be divided into two integration regions, namely, the horse and the background. The integration result is shown in Fig. 4(d). Contrasting Fig. 4(c) with Fig. 4(d), we can see that the integration result is better when the border values are not taken into account.

4 Discussions (1) Based on a new model of retinal ganglion cell receptive fields, we add the characteristic that the sizes of ganglion cell RFs increase systematically with increasing eccentricities from fovea, and further discuss the role of RFs with changing sizes in image information integration. Using the model to process images, we can see that it can not only enhance the edges of images but also retain the low spatial frequency components of images. Obviously, this result is consistent with the spatial summation property of nCRF and is benefit to the further image segmentation. (2) Research shows that ganglion RF can dynamically change its size [15]. According to different brightness, different lengths of time of stimulations, different background images, or the changes of speed of moving objects, the size of RF will be changed. For example, in the dark environment, visual neurons will expand RF and receive weak lights by spatial summation at the cost of reducing the spatial resolution. In the case of needing to identify the fine structures of images, RFs become smaller in order to improve the spatial resolution capability. Thus, visual systems can meet the needs of different tasks in different situations by adjusting themselves. In this paper, although the sizes of ganglion cell RFs increase with the increasing eccentricities from fovea, the size of the RF in fixed position don’t dynamically change with the different tasks. We will improve our work with the further research. We will make RFs adjust the sizes of their own according to nature of stimulus. That is to say, RFs will expand in the region of image details needing to be finely distinguished and reduce in the continuous region within which the image information changes slightly. By this way, images are accurately characterized, i.e., not only the image edges are extracted, but also region brightness contrasts are retained. Finally, image integration is achieved through spatial summation of neuron's RFs. Acknowledgment. Our research gets financial aid from National Natural Science Foundation of China (NSFC) project 60303007 and Shanghai Science and Technology Development Fund 08511501703.

An Integration Model Based on Non-classical Receptive Fields

459

References 1. Ikeda, H., Wright, M.J.: The Outer Disinhibitory Surround of the Retinal Ganglion Cell Receptive Field. J. Physiol. 226, 511–544 (1972) 2. Krüger, J., Fischer, B.: Strong Periphery Effect in Cat Retinal Ganglion Cells. Excitatory Responses in ON- and OFF-center Neurons to Single Grid Displacements. Exp. Brain Res. 18, 316–318 (1973) 3. Li, C., Zhou, Y., Pei, X., Qiu, F., Tang, C., Xu, X.: Extensive Disinhibitory Region Beyond the Classical Receptive Field of Cat Retinal Ganglion Cells. Vision Res. 32, 219–228 (1992) 4. Li, C., He, Z.: Effects of Patterned Backgrounds on Responses of Lateral Geniculate Neurons in Cat. Exp. Brain Res. 67, l6-26 (1987) 5. Li, C., Pei, X., Zhou, Y.: Role of the Extensive Area Outside the X-cell Receptive Field in Brightness Information Transmission. Vision Res. 31, 1529–1540 (1991) 6. Li, C.: Mutual Interactions and Steady Background Effects within and Beyond the Classical Receptive Field of Lateral Geniculate Neurons. In: Yew, D.T., So, K.F., Tsang, D.S. (eds.) Vision: Structure and Function, Singapore, pp. 259–280 (1988) 7. Li, C., He, Z.: Effects of Patterned Backgrounds on Responses of Lateral Geniculate Neurons in Cat. Exp. Brain Res. 67, 16–26 (1987) 8. Wang, X., Wei, H.: Two Improved Models of Retinal Ganglion Cell Receptive Field. In: 2008 International Conference on Humanized Systems, pp. 384–387. Posts and Telecom Press, Beijing (2008) 9. Kolb, H.: How the Retina Works. American Scientist 91, 28–35 (2003) 10. Yang, X., Gao, F., Samuel, M.: Modulation of Horizontal Cell Function by GABA(A) and GABA(C) Receptors in Dark- and Light-adapted Tiger Salamander Retina. Vis. Neurosci. 16, 967–979 (1999) 11. Qiu, F., Li, C.: Mathematical Simulation of Disinhibitory Properties of Concentric Receptive Field. Acta Biophysica Sinica 11, 214–220 (1995) 12. Druid, A.: Vision Enhancement System – Does Display Position Matter? [Master’s Thesis: Department of Computer and Information Science]. Linköping University, Sweden (2002) 13. Croner, L.J., Kaplan, E.: Receptive Fields of P and M Ganglion Cells Across the Primate Retina. Vision Res. 35, 7–24 (1995) 14. Li, C.: Integration Fields beyond the Classical Receptive Field Organization and Function Properties. News Physiol. Sci. 11, 181–186 (1996) 15. Li, C.: New Advances in Neuronal Mechanisms of Image Information Processing. Bulletin of National Natural Science Foundation of China 3, 201–204 (1997)

Classification of Imagery Movement Tasks for Brain-Computer Interfaces Using Regression Tree Chiman Wong and Feng Wan 1

Abstract. Classification of EEG (electroencephalographic) signals recorded during right and left motor imagery tasks is a technique for designing BCI (Braincomputer interfaces). In this paper, the regression tree is used to separate the right/left patterns that are extracted by ERD time courses. The regression tree is a statistical method to identify complex patterns without rigorous theoretical and distributional assumptions. The simulation result shows that the proposed BCI can provide satisfactory offline classification error rate and mutual information. Keywords: Brain-computer interface (BCI), Decision tree, Event-related desynchronization (ERD), Motor imagery.

1 Introduction Since the human’s EEG was first described by Hans Berger in 1929, people have speculated that it might be used for communication and control. In recent few decades, the BCI based EEG has attracted more and more attention in research since the EEG signal can reflect brain activity [1, 4]. In first BCI international meeting, the BCI was defined that it is a communication system that does not depend on the brain’s normal output pathways of peripheral nerves and muscles [4]. In other words, the patients who are paralyzed or have other severe movement deficits can have an alternative communication method for acting on the world. Thus, BCI translates the continuous EEG signal or brain activity into a discrete command for computer processing [3]. For this purpose, many research teams study to use many different algorithms and EEG signals to design the BCIs. Present-day BCIs based on the different EEG signals can fall into 5 groups. They are using VEPs (Visual Evoked Potential), slow cortical potentials, P300 evoked potentials, mu/beta rhythms and cortical neuronal action potentials, respectively [1]. In addition, a great variety of classifiers have Chiman Wong . Feng Wan Department of Electrical and Electronics Engineering Faculty of Science and Technology, University of Macau H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 461–468. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

462

C. Wong and F. Wan

been tried to apply in BCI in order to find out the best classification algorithm to the BCIs based on different EEG signals. The more details can be found in [2]. This paper proposed a BCI using mu/beta rhythms feature from EEG signals that are recorded during left and right hand imagery movement from a healthy subject. Then the extracted features are classified by the regression tree. Decision tree is one of the most popular classification algorithms in data mining and machine learning because it is easily interpreted and comprehended by humans. Furthermore, decision tree can classify the complex patterns with only using several simply discriminants. The basic idea is to break up a complex decision into a union of several simpler decisions, thus providing a clear solution which is often easier to interpret. Note that the regression tree is the same kind of decision tree that is used to predict the numeric quantities. The performance of the proposed BCI is tested by BCI competition 2003 dataset III. The main purpose of this work is to investigate whether the decision tree is applied in BCI.

2 Methodology 2.1 Experimental Data The BCI competition 2003 dataset III is used to evaluate the proposed decision tree classifier algorithm in this work. The EEG dataset that provided by Graz BCI group was recorded from a 25 years old female during a feedback session [13]. Three pairs of bipolar EEG channels were positioned over C3, Cz and C4. It was sampled at 128 Hz and bandpass filtered between 0.5 and 30Hz. This dataset consists of 280 trials (140 left trials and 140 right trials) of 9 s length. In each trial, the first 2 s was quite. At 3 s, an arrow (left or right) was displayed as cue while the female was asked to imagine left or right hand movement according to the cue. One half of dataset (140 trials: 70 left trials and 70 right trials) are used to train the BCI system, the others are used for testing.

2.2 Feature Extraction An internally or externally paced event results not only in the generation of an event-related potential (ERP) but also in a change in the ongoing EEG in form of an event-related desynchronization (ERD) or event-related synchronization (ERS). Note that these events, sensory stimuli, can induce changes in the activity of neuronal populations that are generally called event-related potentials (ERPs). Furthermore, it is known since Berger (1930) that the events can block or desynchronize the ongoing alpha activity. This phenomenon represents frequency specific changes of the ongoing EEG activity and may consist either of decreases or increases of power in the specific frequency bands. It also may be considered to be a decrease or an increase in synchrony of the underlying neuronal populations, respectively. This former case is called event-related desynchronization (ERD) and the latter is called event-related synchronization (ERS) [5, 6]. Event-related desynchronization (ERD) of mu and beta rhythms is a distinct characteristic for imagery movement classification task. One of the basic

Classification of Imagery Movement Tasks for Brain-Computer Interfaces

463

measurements of ERD/ERS in time domain is that the EEG power within identified frequency bands is displayed relative to the power of the same EEG derivations recorded during the reference or baseline period a few seconds before the event occurs introduced in [6]. This is a very convenient, fast and simply method to extract the ERD time course feature from filtered EEG data. The aforementioned EEG dataset was sampled from 3 channels where are C3, Cz and C4, respectively. Only C3 and C4 channel data are employed here since the ERD time course in Cz is not clear. First, the C3 and C4 channel data is filtered by FIR (Finite Impulse Response) bandpass filter (10~12 Hz). Then the two time courses of ERD ec3(k) and ec4(k) are computed at k-th sampling point by equation (1) [11]. xc3(k) and xc4(k) are the filtered data from C3 and C4 channel, respectively, w −1

w−1

ec 3 (k ) = ∑ xc 3 (k − i ) , ec 4 (k ) = ∑ xc 4 (k − i ) 2 2

i =0

(1)

i =0

where w is the length of window. Each feature vector ec(k) is constructed by two column vectors ec3(k) and ec4(k), and denoted as.

ec (k , m ) = [ec 3 (k , m), ec 4 ( k , m)]

(2)

where k represents the k-th sampling point and m denoted the m-th trial. As the period of each trial is 9s length and the sampling frequency is 128Hz, the total sampling points should be 1152. In each trial, it may generate (1152-w+1) feature vectors. For every sampling point, there are 140 feature vectors for growing the tree. In other words, there are (1152-w+1) trees for feature classification totally.

2.3 Feature Classification Decision tree T is made up of nodes and branches. Node t is designated as an internal node or a terminal node (leaf node). Internal node can split into two children (tL and tR) while terminal node does not have any children. A terminal node has a class label associated with it such that observations that fall into particular terminal node are assigned to that class [7, 8, 9, 12]. Decision trees are designed for predicting categories rather than numeric quantities. Regression trees are designed for predicting numeric quantities, the same kind of tree representation can be used, but the terminal nodes of the tree would contain a numeric value that is the average of all the training set values to which the leaf, that represent the average output for observations that reach the leaf. Because statisticians use the term “regression” for the process of computing an expression that predicts a numeric quantity, decision trees with averaged numeric values at the leaves are called “regression tree” [8]. To construct a regression tree, it is performed in a manner similar to decision tree. Much of the work in designing the decision trees focus on deciding which property should be tested at each node. The fundamental principle underlying tree is that of simplicity. The expected target is to obtain a simple, compact decision

464

C. Wong and F. Wan

tree with few nodes. The method is to seek a property test at each node that makes the data reaching the branched nodes as “pure” as possible. The impurity of the node is measured by squared difference between the predicted values from the tree and the real values (or class label) [12, 14]. During the growing of the tree, the minimum impurity of every split is guaranteed. As a result, the growing algorithm is searching a threshold value at each node that splits two branches, left and right, whose total impurity is as small as possible. For instance, the splitting at node t is determined by one of the element’s threshold value in the feature vector ec(k,m) in this paper. At first, selecting one column of feature vector which is ec3(k) or ec4(k) and sorting the specific column in ascending order, denoted by esc3(k) or esc4(k) respectively. The matrix of threshold value splitj(k,i) at node t is calculated by (3)

split j ( k , i ) = es j (k , i ) +

es j ( k , i + 1) − es j (k , i ) 2

(3)

where i=1,2,…,m-1, k=1,2,…, (1152-w+1) and j represents C3 or C4. It means that the feature vector ec(k,m) at node t is classified to left (or right) branch if the feature ej(k,m) is larger (smaller) than splitj(k,i). Then the impurity R(t) of node t, tL and tR can be calculated by (4). tL and tR are the left and right child nodes of t.

R(t ) =

1 ( ym − y (t )) 2 ∑ n(t ) ec ( k , m )∈t

(4)

1 ∑ ym n(t ) ec ( k , m )∈t

(5)

y (t ) =

where ym is the class label of ec(k,m) and n(t) represents the number of feature vectors that fall into node t. Thus, for a splitting threshold value splitj(k,i), and node t, the difference in the mean squared error between before and after splitting is

ΔR ( split j ( k , i ), t ) = R (t ) − R (tL ) − R (tR )

(6)

where tL and tR are the left and right child nodes of t. Obviously, only R(t) is independent on splitj(k,i). It is easy to know that the summation of impurity of tL and tR is smaller, the difference ΔR(splitj(k,i),t) is bigger. The maximum of ΔR(splitj(k,i),t) implies that the child nodes tL and tR are the purest. Hence, the optimal threshold value splitj(k,i) may be obtained by maximizing the ΔR(splitj(k,i),t). The aim of splitting is minimizing the node’s impurity. If the impurity of tL (tR) is still large, then the similar splitting is continue. Otherwise, the node tL (tR) is terminal node. And the predicted value ŷ of the node t is estimated by the average of ym. Note that the predicted value ŷ is also the output of regression tree,

Classification of Imagery Movement Tasks for Brain-Computer Interfaces

yˆ (t ) =

465

1 ∑ ym . n(t ) ec ( k , m )∈t

(7)

The regression tree T is obtained by iteratively splitting nodes. The main problem is when to stop the splitting during the growing. One method is to set a small threshold value in impurity. Splitting is stopped if the impurity of node is less than the threshold value. Another method is to stop the splitting when a node in terminal represents fewer than some threshold number of data, say 5, for instance, after growing the tree, each terminal node is assigned to less than 5 training samples. However, it is often difficult to find the best threshold value in these two methods. An alternative principal method is pruning. In this method, the tree is growing fully at first. It indicates that the impurity of terminal node is zero or each terminal node does not contain more than 5 training data. This large tree fits the training data very well. However,it does not generalize well to the testing data. Thus, it is necessary to prune the meaningless branches from this large tree. As a result, a sequence of subtrees can be obtained by successively pruning branches [14]. The N subtrees are denoted T1, T2, T3, … , TN (T1=T, TN={t1}, t1 is the root node). The 140 training feature vectors ec(k) are given to train the regression tree Tk. Then the least training error of the tree T* will be chosen for testing. In addition, T* is pruned to N subtrees which are T*,1, T*,2, T*,3,…, T*,N before the testing. All subtrees except T*,N are used to classify the 140 testing feature vectors etc(k), respectively. If the output of tree Y is smaller than 1.5 (the mean of the class label of left and right trial), the corresponding feature vector is classified as a left trial. On the contrary, if Y is larger than 1.5, it is classified a right trial.

Y*,n (k , m) = T*,n (etc (k , m))

(8)

where T*,n is n-th subtree of T* (n=1,2,…,N). Y*,n(k,m) indicates the confidence of the classification result. If Y*,n(k,m) is closer to 1 (2), it gives more confidence to say that the trial is left (right) imagination. Finally, the error rate of T* is the minimum error rate of N subtrees.

3 Result and Discussion In general, classification error rate and mutual information are usually used to qualify the BCI’s performance. [10] proposed a measurement for information transfer based on mutual information, where is shown in equation (9) and (10)

MI n (k ) = 0.5 ⋅ log 2 (1 + SNRn (k )) SNRn (k ) =

2 var {Y*,n (k , m)} m∈{ L , R}

var {Y*,n (k , mL )} + var {Y*,n (k , mR )}

mL ∈{ L}

mR ∈{ R}

(9)

−1

(10)

466

C. Wong and F. Wan

Fig. 1 Box plot of Mutual information and Classification error rate for different window length (w=4~33) by cross validation. When w=19, the error rate is 0.1714, 0.1643, 0.1357, 0.1357, 0.1571, 0.2143, 0.1571, 0.1357, 0.1643 and 0.1643, the mutual information is 0.4136, 0.5189, 0.5340, 0.5519, 0.4616, 0.3547, 0.6173, 0.5025, 0.5681 and 0.5393, respectively

where {L} and {R} are the sets of left and right trials, mL and mR is the corresponding index, and var{∗} is the variance. The mutual information of T* is the maximum mutual information of N subtrees. The performance of regression tree is primarily determined by its training data (such as the order in which the training data are presented to the growing algorithm). Thus, each experiment is repeated 10 times for cross-validation to find out the general performance. For this purpose, the training and testing data should be selected randomly and they are non-overlapping. In addition, the training data still consists of 70 left trials and 70 right trials. Then the least training error of regression tree T* is learning from training feature vectors. Consequently, the error rate and mutual information of T* can be obtained. The same procedure is iterated 10 times and then the 10 error rate and 10 mutual information can be obtained. Furthermore, the parameter w in (1) also effects the classification result. Fig. 1 is the box plot of classification error rates and mutual information based on different window length. The box plot displays the minimum, first quartile (Q1), the median (Q2), the third quartile (Q3) and the maximum (Q4) of the data. The “+” represents the outlier since that point beyond the whiskers whose length is interquartile range (Q3-Q1). When w=19, the median value of mutual information can be achieved the maximum. It can be found that the optimal w in Fig. 1 is 19. So when w=19, the median value of mutual information is about 0.52 and the median value of error rate is 0.16. Fig. 2 shows the classification error rate and mutual information at every sampling point and the averaged output for the left and right trials when select w=19. From Fig. 2(a) ~ (b), the minimum error rate is 0.1357 at

Classification of Imagery Movement Tasks for Brain-Computer Interfaces

467

Fig. 2 The Classification error rate, Mutual information and Averaged tree output for the left or right trials at every sampling point. (a) Classification error rate; (b) Mutual information; (c) Averaged tree output for the left or right trial

4.75 s and the maximum mutual information is 0.5739 at 4.75 s. It is reasonable because the subject began the motor imagery after the cue ended at 3 s. It is also a very good result compared with BCI competition 2003 [13]. The Fig.2 (c) illustrates that the averaged tree output of right or left motor imagery. It can be found that there is a different pattern between imagination of left and right hand movement from 3.5 s to 6 s. The training time of fully growing tree is definitely longer than the training time of linear classifier (such as LDA). In Matlab simulation, the training time of growing all regression trees (1152-w+1) for 140 training trials using non-optimized code adds up to approximately 124s, and the processing time of regression tree is about 9 s. It is quite shorter than the growing time because the classification operation in tree only includes several properties testing. Fig. 3 shows the pruned tree that only has several branches. Fig. 3 This is the pruned tree that is applied to classify the testing data. The corresponding testing result is illustrated in Fig. 2. x1 and x2 are denoted ec3(k,m) and ec4(k,m) respectively

468

C. Wong and F. Wan

4 Conclusion A classical tree classifier is applied to the BCI based on the time course of ERD features. The simulation result shows that the proposed BCI has a satisfactory offline performance basically. The classification performance of decision tree is mainly determined by training data and the generalization of tree is related to the pruning technology. And this dataset is only based on a specified subject. Accordingly, the cross validation result may not reflect the real performance of proposed BCI completely in this paper. In future, the proposed BCI can be tested by more subjects’ datasets to investigate the generalization.

References 1. Wolpaw, J.R., Birbaumer, N., McFarland, D.J., Pfurtscheller, G., Vaughan, T.M.: Brain-Computer Interface for Communication and Control. Journal of Clinical Neurophysiology 113, 767–791 (2002) 2. Lotte, F., Congedo, M., Lecuyer, A., Lamarche, F., Arnaldi, B.: A Review of Classification Algorithms for EEG-based Brain-Computer Interfaces. Journal of Neural Engineering 4, R1–R13 (2007) 3. Bashashati, A., Fatourechi, M., Ward, R.K., Birch, G.E.: A Survey of Signal Processing Algorithms in Brain-Computer Interfaces based on Electrical Brain Signals. Journal of Neural Engineering 4(2), R32–R57 (2007) 4. Wolpaw, J.R.: Brain-Computer Interface Techology: A Review of the First International Meeting. IEEE Transactions on Rehabilitation Engineering 8(2), 164–173 (2000) 5. Pfurtscheller, G., Neuper, C.: Motor Imagery Activates Primary Sensorimotor Area in Humans. Neuroscience Letters 239, 65–68 (1997) 6. Pfurtscheller, G., da Silva, F.H.L.: Evented-Related EEG/MEG Synchronization and Desynchronization: Basic Principles. Clinical Neurophysiology 110, 1842–1857 (1999) 7. Breiman, L., Friedman, J., Stone, C.J., Olshen, R.A.: Classification and Regression Trees. Wadsworth (1984) 8. Witten, I.H., Frank, E.: Data Mining: Practical Machine Learning Tools and Techniques, 2nd edn. Morgan Kaufmann Publishers, San Francisco (2005) 9. Safavian, S.R., Landgrebe, D.: A survey of Decision Tree Classifier Methodology. IEEE Transactions on Systems, Man, and Cybernetics 22(3), 660–674 (1991) 10. Schlogl, A., Keinrath, C., Scherer, R., Pfurtscheller, G.: Information Transfer of an EEG-based Brain-Computer Interface. In: Proceedings of the 1st International IEEE EMBS Conference on Neural Engineering, pp. 641–644 (2003) 11. Jia, W.Y.: Classification of Single Trial EEG during Motor Imagery based on ERD. In: Proceedings of the 26th Annual International Conference of the IEEE EMBS, pp. 5–8 (2004) 12. Duda, R.O., Hart, P.E., David, G.S.: Pattern Classification, 2nd edn. John Wiley & Sons Inc., Chichester (2001) 13. BCI competition II, http:// ida.first.fraunhofer.de/projects/bci/competition_ii/ 14. Martinez, W.L., Martinez, A.R.: Computational Statistics Handbook with MATLAB. Chapman & Hall/CRC, Boca Raton (2002)

MIDBSCAN: An Efficient Density-Based Clustering Algorithm Cheng-Fa Tsai and Chun-Yi Sung

Abstract. This investigation presents a clustering algorithm that incorporates neighbor searching into the density-based IDBSCAN algorithm. The rooted algorithm performs fewer searches than standard IDBSCAN. Experimental results indicate that the proposed MIDBSCAN algorithm has a lower execution time cost than DBSCAN, IDBSCAN or KIDBSCAN. MIDBSCAN has a maximum deviation in clustering correctness rate of 0.1%, and a maximum deviation in noise data filtering rate of 0.3%. Keywords: Data mining, Data clustering, Density-based clustering algorithm.

1 Introduction Data mining is widely employed in business management and engineering. The major objective of data mining is to discover helpful and accurate information among a vast amount of data, providing a reference basis for decision makers. Data clustering is currently a very popular and frequently applied analytical method in data mining. Research in data clustering focuses mainly on increasing the accuracy and reducing the clustering time cost [1]-[17]. Clustering schemes are classified as partitioning, hierarchical, density-based, grid-based and mixed methods. Partitioning methods, such as K-means, are the most popular clustering algorithms. The advantage of partitioning approaches is fast clustering, while the disadvantages are the instability of the clustering result, and inability to filter noise data. Hierarchical methods, such as CURE and CHAMELEON, involve constructing a hierarchical tree structure, and adopting it to perform clustering. These methods have high Cheng-Fa Tsai · Chun-Yi Sung Department of Management Information Systems, National Pingtung University of Science and Technology, 91201, Pingtung, Taiwan {cftsai,n9656018}@mail.npust.edu.tw H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 469–479. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

470

C.-F. Tsai and C.-Y. Sung

clustering accuracy, but suffer from continuously repetitive merging and partitioning: each instance must compare the attribute of all objects, leading to a high calculation complexity. Density-based methods, including DBSCAN, IDBSCAN and KIDBSCAN perform expansion and clustering based on density. These approaches can filter noise, and perform clustering in disordered patterns, but take a long time to perform clustering. Grid-based clustering algorithms, such as STING, segment data space into various grids, where each data point falls into a grid, and perform clustering with the data points inside the grids, thus significantly reducing the clustering time. Section 2 of this work reviews literature about the K-means, DBSCAN, IDBSCAN and KIDBSCAN algorithms. Section 3 describes in detail the concept, algorithm and implementation steps of MIDBSCAN mentioned in this investigation. Section 4 illustrates the experimental and analytical results. A summary is presented in Section 5.

2 Related Works This section introduces K-means, DBSCAN, IDBSCAN and KIDBSCAN density-based clustering methods, which are related to the proposed MIDBSCAN algorithm.

2.1 K-Means Algorithm K-means, presented by McQueen in 1967 [5], was the first clustering algorithm, and only needs to set the K centroid. The clustering method of K-means is as follows: (1) Generate K points randomly, and define these as the centroid. (2) Input all of the data from the database; calculate the closest distance from the point, then set it as the same cluster. (3) Recalculate the centroid; if it is unchanged, then conclude the algorithm; otherwise, repeat step (2). K-means is a partitioning clustering method, and therefore has fast execution. However, K-means produces unstable clustering, and does not perform noise filtering.

2.2 DBSCAN Algorithm DBSCAN, proposed by Ester et al. in 1996 [2], was the first clustering algorithm to employ density as a condition. It utilizes density clustering to place data points into the same cluster when their density within their data points is higher than a set threshold value, and sets this cluster as the seed for outward expansion. This algorithm must set two parameters, the radius () and the minimum number of included points (M inP ts). To search for sets, DBSCAN scans all data points in the database, and draws a circle of radius

MIDBSCAN: An Efficient Density-Based Clustering Algorithm

471

Fig. 1 DensityReachability of Core Point [2]

() around each data point Q. The number of data points within a circle is set as a neighborhood. If a neighborhood is greater than the set M inP ts, then its data points are set as expansion seeds, if P is the core point, and its neighborhood is less than the set M inP ts, then these data points are considered as a Border Point, as depicted in Fig. 1. DBSCAN can conduct clustering on disordered patterns, and has noise filtering capacity, as well as clusters that can be stabilized. However, parameter setting is difficult, which also affects the clustering result.

2.3 IDBSCAN Algoritm IDBSCAN is a density-based data clustering scheme developed by Borah et al. in 2004 [1]. This method applies a Marked Boundary Object to determine the data point of an expansion seed when searching for neighborhood to add in expansion seeds. Assuming that the√core point is P (0, 0), eight √ √ marked √ objects may be√defined 2), C(, 0), D(/ 2, -/ 2), √ as: A(0,), B(/ 2, √ / √ E(0, -), F (-/ 2, -/ 2), G(-, 0), H(-/ 2, / 2), as show in Fig. 2. where every quadrant contains three boundary objects. If P indicates the core point, and it satisfies the set density condition, then the algorithm finds within the neighborhood the closest point to these eight marked boundary objects, and sets these data points as the expansion seeds. Since these seeds may be selected using multiple marked boundary objects, the algorithm requires only one instance of input is needed. The number of seeds added is below (3d − 1), where d represents the dimension of the database. Fig. 2 Eight Marked Boundary Object of IDBSCAN [1]

472

C.-F. Tsai and C.-Y. Sung

2.4 KIDBSCAN Algorithm KIDBSCAN is a density-based clustering method presented by Tsai and Liu in 2006 [8]. They searched for marked boundary objects with IDBSCAN, and found that inputting data sequentially from low-density database causes remnant seed searching, resulting in poor expansion results. To decrease the number of sample instances, KIDBSCAN performs expansion by inputting elite points. It has three parameters, elite point, radius and M inP ts. The execution steps are as follows. (1) Adopt K-means algorithm to find the K numbers of the centroid within the database, then find the K data points closest to these centroid and define them as elite points, because K-means can discover these elite points quickly. (2) Move the K elite points to the very front of the database. (3) Execute the IDBSCAN algorithm. Experimental results prove that KIDBSCAN performs data clustering quickly.

3 The Proposed MIDBSCAN Algorithm The processing procedure of searching for Neighbors (neighborhood data points) is very time consuming in the DBSCAN and IDBSCAN algorithms. Therefore, to shorten the time consumed, this study focuses lowering the number of expansion seeds added into the neighborhood data in this procedure, thus reducing the time cost of searching for Neighbors. This section introduces the method, concept, implementation procedures and algorithm for MIDBSCAN. Searching for neighborhood data in density-based clustering algorithms is time-consuming. The time taken in DBSCAN is defined by SetofPoints.region Query(), such that the time complexity of searching for neighborhood data is O(n) where n denotes the database size. Data points that have been clustered do not need to be input again into the neighborhood search criteria when search-

Time cost of search neighbors : 100% Cluster 1(50%) Cluster 2(30%)

Noise(20%)

Time cost of search neighbors : 50% Cluster 1(50%) Cluster 2(30%)

Noise(20%)

Time cost of search neighbors : 20% Cluster 1(50%) Cluster 2(30%)

Noise(20%)

Fig. 3 Conceptual illustration of neighborhood search, where after each completion of a clustering task, the time cost of neighborhood search will be lowered to the number of those clustered

MIDBSCAN: An Efficient Density-Based Clustering Algorithm

473

ing for neighborhood points (See Fig. 3). Thus, the index array required for the neighborhood search is readjusted after each completion of clustering task. This neighborhood search index array is defined as for use at the beginning of the algorithm, and it stores the unclassified data points, with storage in Boolean type. MIDBSCAN requires n bytes of memory in the entire database. Although MIDBSCAN uses more memory space than conventional IDBSCAN, it reduces the time consumed. The time cost of the next neighborhood search following completion of a clustering task is O(n − c), where n denotes the database size, and c represents the size of the cluster set data size after the clustering task. The parameters are set when implementing MIDBSCAN: (1) Radius (), (2) Minimum number of included points (M inP ts). The MIDBSCAN clustering algorithm can be described as follows: Input: Datasets, Eps, MinPts Output: Clusters MIDBSCAN (Datasets, Eps, MinPts) Intialization; ClusterID := NextID(First); FOR i FROM 1 TO Datasets.Size DO Point := Datasets.Get(i); IF Point.CID = UNCLASSIFIED THEN IF ExpandCluster(Datasets, Point, ClusterID, Eps, MinPts) THEN ClusterID := NextID(ClusterID); UnclassfiedData.Adjust(); END IF END IF END FOR END; ExpandCluster(Datasets, Point, CID, Eps, MinPts) : Boolean; Neighbors := UnclassfiedData.RegionQuery(Point, Eps); IF Neighbors.size < MinPts THEN Datasets.ChangeCIDs(Point, NOISE); RETURN False; ELSE Datasets.ChangeCIDs(Point,CID); Neighbors.AddMBOs(); FOR i FROM 1 TO Neighbors.size DO neighborPoint := Neighbors.Get(i); IF neighborPoint.CID = UNCLASSIFIED || neighborPoint.CID = NOISE THEN Datasets.ChangeCID(neighborPoint,CID); END IF; END FOR; WHILE Seeds <> Empty DO seedPoint := Seeds.First(); Seeds.Delete(seedPoint); Neighbors := UnclassfiedData.RegionQuery(seedPoint, Eps); IF Neighbors.size >= MinPts THEN Neighbors.AddMBOs(); FOR i FROM 1 TO Neighbors.size DO neighborPoint := Neighbors.Get(i); IF neighborPoint.CID = UNCLASSIFIED || neighborPoint.CID = NOISE THEN Datasets.ChangeCID(neighborPoint,CID); END IF; END FOR; END IF; END WHILE; RETURN True; END IF END;

474

C.-F. Tsai and C.-Y. Sung

The implementation steps for the MIDBSCAN algorithm are described as follows: Step 1. Initialize all parameters, and define a new Cluster ID. Step 2. Begin scanning all data points within the entire database. For data points belonging to the Cluster ID of those unclassified data, implement the Expand Cluster processing procedure. The database is the set of data points; the Point is the core point; the Cluster ID is the current cluster ID;  represents the radius, and M inP ts denotes the minimum number of included points. Step 3. If the data point returned by the expansion procedure function is a noise data point, then go directly to Step 2, until the Datasets database has been fully scanned. If an expansion data point is returned, then update the new Cluster ID, and alter the index array of unclassified data, and then go to Step 2. Step 4. End the algorithm when all data points have been processed. The implementation steps for the Expand Cluster processing procedure are as follows. Step 1. Search for Neighbors within the range of radius  in the unclassified cluster index. If the number of Neighbors is less than M inP ts, then leave the procedure, and return the core point as the noise data point. Otherwise, go to Step 2. Step 2. Set the core point as the current cluster set ID. Step 3. If the set of seeds is empty, then end the expansion processing procedure, otherwise go to Step 4. Step 4. Search for the marking boundary point within Neighbors, and add in the expansion Seeds. Step 5. Set all unclassified data points and Neighbors that are noise data points as the current cluster ID. Step 6. Extract the first seed from the expansion seeds; and define it as the core point, and delete it. Step 7. In the unclassified data index, search for Neighbors within the range of radius  of the core point. If the number of Neighbors is greater than M inP ts, then go to Step 3.

4 Experimental Results The clustering algorithm was implemented in the C# language in Microsoft Visual Studio 2005 on a notebook computer with a 1.6 GHz Intel CPU, with 1G of RAM, running Windows XP. The clustering correctness rate and noise filtering rate experiments were performed with six different pattern datasets. An additional pattern dataset was designed for the time cost experiment. The pattern image size was 900x700 pixels. The number of datasets

MIDBSCAN: An Efficient Density-Based Clustering Algorithm

475

Fig. 4 The experimental results obtained by MIDBSCAN

within Dataset1 to Dataset 6 was 115,000 of 2-dimensional data, which includes 15,000 (15%) noise data points, and a fixed radius (= 6). Dataset 7 contained 110,000-1,100,000 instances of data, with datasets of six different sizes (each containing 10% noise data points), and a fixed radius (= 16) for experimentation with datasets of varying sizes, as shown in Fig. 4. The proposed algorithms were applied to the Neighbors search processing procedures in the DBSCAN, IDBSCAN and KIDBSCAN algorithms, to form DBSCAN Plus, MIDBSCAN, KIDBSCAN Plus. Thus, the experiments were performed on six algorithms in total. The radius () was fixed, then according to the pattern data density to adjust its M inP ts to conduct the experiment. The experiments measured (1) Execution time, (2) Clustering Correctness Rate and (3) Noise Filtering Rate. Table 1 lists the datasets source and characteristics. Experimental results involving various pattern datasets comparing the original DBSCAN, IDBSCAN and KIDBSCAN algorithms and those with

Table 1 Benchmark Datasets Benchmark

Cluter

Points

Noise Rate

Source

Dataset 1

10

115,000

15%

Guha et al., 1998

Dataset 2

4

115,000

15%

Ester et al., 1996

Dataset 3

14

115,000

15%

C.F. Tsai and C.C.Yen, 2007

Dataset 4

2

115,000

15%

C.F. Tsai and C.C.Yen, 2007

Dataset 5

4

115,000

15%

C.F. Tsai and C.C.Yen, 2007

Dataset 6

4

115,000

15%

C.F. Tsai and C.C.Yen, 2007

Dataset 7

5

110K-1,100K

10%

2010 Olympic Winter Games

476

C.-F. Tsai and C.-Y. Sung

Table 2 A comparison between DBSCAN, IDBSCAN, KIDBSCAN, DBSCAN Plus, MIDBSCAN and KIDBSCAN Plus algorithms, using 6 kinds of pattern datasets, with data size being 115,000 (containing 15% noise data). Item 1 represents Execution Time (in seconds); Item 2 denotes Clustering Correctness Rate; Item 3 indicates Noise Filtering Rate Algorithm

Item

DataSet 1

DataSet 2

DataSet 3

DataSet 4

DataSet 5

DataSet 6

DBSCAN

1 2 3

1217.10 100% 95.06%

1210.50 99.99% 95.55%

1211.34 99.83% 92.43%

1211.86 99.96% 96.93%

1211.75 99.99% 95.72%

1246.51 99.97% 96.25%

1 2 3

807.18 100% 95.02%

877.39 99.99% 95.40%

770.95 99.83% 92.31%

920.03 99.98% 96.82%

874.25 99.99% 95.62%

857.40 99.96% 96.14%

IDBSCAN

1 2 3

278.84 100% 95.42%

422.95 99.97% 96.23%

425.81 99.99% 92.88%

465.43 99.96% 97.40%

381.06 99.99% 95.94%

435.17 99.95% 96.52%

MIDBSCAN

1 2 3

71.48 100% 95.77%

132.92 99.91% 96.78%

130.84 99.94% 93.20%

166.09 99.90% 97.66%

112.39 99.97% 96.20%

133.70 99.94% 96.81%

KIDBSCAN

1 2 3

275.54 100% 95.43%

415.95 99.98% 96.24%

424.73 99.97% 92.91%

459.00 99.96% 97.40%

379.17 99.98% 95.96%

436.06 99.95% 96.56%

1 2 3

124.64 100% 95.38%

248.53 99.99% 95.06%

233.17 99.99% 92.75%

299.25 99.96% 97.30%

216.82 99.98% 95.90%

255.92 99.96% 96.42%

DBSCAN Plus

KIDBSCAN Plus

Neighbor search processing indicate a variation in the clustering error rate of about 0.1%, and the variation in the noise filtering rate never exceeded 0.3%, revealing that the proposed algorithms do not influence the clustering quality and noise filtering capacity of the original algorithms. Experimental results on the pattern dataset of Dataset 1 demonstrate that the particularly high density of the dataset led to a large number of M inP ts and a high execution time cost. Additionally, the execution time cost of KIDBSCAN Plus algorithm in these datasets was also higher, since the algorithm also contains the K-means algorithm, making it likely to produce instability in the execution time cost. The time cost of DBSCAN and DBSCAN Plus algorithms in the 440K dataset exceeded 10,000 seconds (see Table 3). Because the experiment objective was to determine the efficiency that could be achieved by incorporating the proposed method into each of the algorithms, no further experiment were performed on the algorithms on datasets larger than 660K. Hence, Table 3 does not list all of the simulation results for DBSCAN and DBSCAN Plus (N/A means that the simulations were not performed). The adding the proposed method to IDBSCAN and KIDBSCAN significantly changed their time costs with datasets of size 1,100K. The Neighbors search processing reduced the time wasted in unnecessary searches, thus

MIDBSCAN: An Efficient Density-Based Clustering Algorithm

477

Table 3 Using Dataset 7 pattern to construct experiments with various dataset sizes to test the execution time cost (in seconds) of various clustering algorithms, with each dataset size containing 10% noise data points Algorithm

110K

220K

1116.46

4466.46

17840.89

DBSCAN Plus

769.25

3184.87

12444.76

N/A

N/A

N/A

IDBSCAN

133.23

439.65

1699.25

1098.75

1510.66

9794.62

MIDBSCAN

26.76

82.50

300.53

372.20

4976.26

1451.60

KIDBSCAN

133.57

459.92

1751.96

1077.65

1480.67

9631.90

KIDBSCAN Plus

43.10

188.04

335.29

1101.82

1465.18

1687.39

DBSCAN

440K

660K

880K

1,110K

N/A

N/A

N/A

improving the efficiency in searching large databases, and significantly reducing the execution time cost. The time cost of KIDSCAN Plus algorithm did not vary significantly from that of KIDBSCAN in datasets of size 660K and 880K. The KIDBSCAN algorithm first uses the K-means algorithm to seek its centroid; move the data points away from the centroid to the front end of the database, then runs the IDBSCAN algorithm. Since the K-Means algorithm first employs a random value to extract the K centroid, an unstable centroid extraction may compromise the implementation efficiency of IDBSCAN.

Fig. 5 Execution time cost (in seconds) of processing various dataset sizes by the MIDBSCAN, IDBSCAN, KIDBSCAN and KIDBSCAN Plus algorithms

478

C.-F. Tsai and C.-Y. Sung

5 Conclusion The proposed processing method for Neighbors search was added to the DBSCAN, IDBSCAN and KIDBSCAN algorithms, and tested with various patterns and sizes of datasets. The IDBSCAN algorithm with the new method, called MIDBSCAN, had better execution time cost, clustering correctness rate, and noise filtering rate and clustering stability than IDBSCAN alone. Acknowledgement. The author would like to thank the National Science Council of Republic of China, Taiwan for financially supporting this research under contract no. NSC 96-2221-E-020-027.

References 1. Borah, B., Bhattacharyya, D.K.: An Improved Sampling-Based DBSCAN for Large Spatial Databases. In: Proceedings of International Conference on Intelligent Sensing and Information, pp. 92–96 (2004) 2. Ester, M., Kriegel, H., Sander, J., Xu, X.: A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. In: Proceedings of 2nd International Conference on Knowledge Discovery and Data Mining, pp. 226–231 (1996) 3. Guha, S., Rastogi, R., Shim, K.: CURE: An Efficient Clustering Algorithm for Large Data Bases. In: Proceedings of the ACM SIGMOD International Conference on Management of Data, vol. 27(2), pp. 73–84 (1998) 4. Karypis, G., Han, E.H., Kumar, V.: CHAMELEON: Hierarchical Clustering Using Dynamic Modeling. IEEE Computers 32(8), 68–75 (1999) 5. McQueen, J.B.: Some Methods of Classification and Analysis of Multivariate Observations. In: Proceedings of the 5th Berkeley Symposium on Mathematical Statistics and Probability, pp. 281–297 (1967) 6. Tsai, C.F., Chen, Z.C., Tsai, C.W.: MSGKA: An Efficient Clustering Algorithm for Large Databases. In: IEEE International Conference on Systems, Man, and Cybernetics, vol. 5, pp. 6–13 (2002) 7. Tsai, C.F., Lee, J.C.: DK-Means: A Robust New Clustering Technique in Data Mining for Databases. Electronic Commerce Studies 5(4), 419–438 (2007) 8. Tsai, C.F., Liu, C.W.: KIDBSCAN: A New Efficient Data Clustering Algorithm for Data Mining in Large Databases. In: Rutkowski, L., Tadeusiewicz, ˙ R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2006. LNCS(LNAI), vol. 4029, pp. 702–711. Springer, Heidelberg (2006) 9. Tsai, C.F., Shih, D.C., Liu, C.W.: FICA: A New Data Clustering Technique Based on Partitional Approach for Data Mining. In: IEEE International Conference on Machine Learning and Cybernetics, Hong Kong, vol. 2, pp. 739–744 (2007) 10. Tsai, C.F., Tsai, C.W., Wu, H.C., Yang, T.: ACODF: A Novel Data Clustering Approach for Data Mining in Large Databases. Journal of Systems and Software 73, 133–145 (2004)

MIDBSCAN: An Efficient Density-Based Clustering Algorithm

479

11. Tsai, C.F., Wu, H.C., Tsai, C.W.: A New Data Clustering Approach for Data Mining in Large Databases. In: The 6th IEEE International Symposium on Parallel Architectures, Algorithms, and Networks, Manila, Philippine, pp. 278– 283 (2006) 12. Tsai, C.F., Yang, T.: An Intuitional Data Clustering Algorithm for Data Mining in Large Databases. In: IEEE International Conference on Informatics, Cybernetics and Systems, Taiwan, pp. 1487–1492 (2003) 13. Tsai, C.F., Yen, C.C.: ANGEL: A New Effective and Efficient Hybrid Clustering Technique for Large Databases. In: Zhou, Z.-H., Li, H., Yang, Q. (eds.) PAKDD 2007. LNCS(LNAI), vol. 4426, pp. 817–824. Springer, Heidelberg (2007) 14. Tsai, C.F., Yen, C.C.: G-TREACLE: A New Grid-Based and Tree-Alike Pattern Clustering Technique for Large Databases. In: Washio, T., Suzuki, E., Ting, K.M., Inokuchi, A. (eds.) PAKDD 2008. LNCS, vol. 5012, pp. 739–748. Springer, Heidelberg (2008) 15. Tsai, C.F., Yen, C.C.: Unsupervised Anomaly Detection Using HDG-Clustering Algorithm. In: Ishikawa, M., Doya, K., Miyamoto, H., Yamakawa, T. (eds.) ICONIP 2007, Part II. LNCS, vol. 4985, pp. 356–365. Springer, Heidelberg (2008) 16. Wang, W., Yang, J., Muntz, R.: STING: A Statistical Information Grid Approach to Spatial Data Mining. In: Proceedings of 23rd International Conference on Very Large Data Bases, pp. 186–195 (1997) 17. Xu, R., Wunsch, D.: Survey of Clustering Algorithm. Proceedings of IEEE Transactions on Neural Networks 16(3), 645–678 (2005) 18. Vancouver 2010 XXI Olympic Winter Games, International Olympic Committee, http://www.olympic.org/uk/games/vancouver/index_uk.asp

Detection and Following of a Face in Movement Using a Neural Network Jaime Pacheco Mart´ınez, Jos´e de Jes´ us Rubio Avila, and Javier Guillen Campos

Abstract. In this paper, a system that uses smaller time in the detection of one person in one environment with controlled light is proposed. To reach this result the following 3 steps are used: 1) Detection of the movement; this fact consist in detecting the object (the object is a person); 2) Detecting of the face, the face is found in one image in movement, considering a camera and a fuzzy neural network. 3) Detecting of the person in movement. The camera is moved in the direction of the person in movement using the prediction of the position using the Kalman filter algorithm. This system is implemented in real time.

1 Introduction The systems of computer vision have used some processes as manufacture plants or in military plants. For example, in manufacture plants is used in the inspection or in the quality control of the products. In military plants it is used to find and follow the enemy to destroy it. In one system of computer vision to follow persons the first step is the detection of the moving object, the second step is to use one algorithm to interpret the image, and the last step is the prediction and the movement of the camera. The first researching about the face recognitions started in the sixties [3]. It consists in one system machine human where the computer saves and classifies a face in a base of data that are constructed from some photos of humans [23]. The research said that the eyes and the mouth are the first places to be recognized, the distance between the eyes, the noise and the mouth are the second parameters to be recognized. Jaime Pacheco Mart´ınez · Jos´e de Jes´ us Rubio Avila · Javier Guillen Campos Secci´ on de Estudios de Posgrado e Investigaci´ on, Instituto Polit´ecnico Nacional - ESIME Azcapotzalco, Av. de las Granjas 682, Col. Sta. Catarina. Azcapotzalco, M´exico D.F [email protected]

H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 481–490. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

482

J.P. Mart´ınez, J. de Jes´ us Rubio Avila, and J.G. Campos

Actually, there exist some methods to detect the faces in images. Yang [25] classify this method in 4 categories: method based in the knowledge, in the invariant characteristics, the corresponding of the photos and in the characteristics of the facials and textures. Wang [15] converts the images in one reduced image to eliminate the textures, later he takes the characteristics of the eyes of possible faces. He obtain the 89.3% of right detection of 402 images. Froba and Ernest [8] detect the faces being in front taking the local characteristics with a mask of 3x3 pixels. Hamouz [11] present the method to detect and localize the faces being in front. First, he looks and filters 10 characteristics independent of the face, as are the corner and the center of the eyes, the noise and the mouth. Hichem [14] presents a fast method of learning to detect the color of the skin, using a method based in the form of the face and a neural network. Henrry [20] in the first step looks faces being in front and later he looks and presents in faces being inclined with some angle. Garcia [9] uses a neural network that deduces automatically the best filter, this network detects faces being inclined with an angle from 20 to 60 degrees. Maslabre [17], uses a radial basis neural network to distribute the faces of different dimensions. In Rafael [19] and Cesar [5] they design a device that is a camera that follows a white point based is several images and it predicts the movement with the Kalman filter. In this paper, it is proposed a system that follows one object (a person) using a camera over a mechanic and rotational system. It makes it detects the movements using the difference between consecutive images. It looks for the face using a fuzzy neural network. And finally it predicts the next movement with the Kalman filter algorithm.

2 The General System The general problem is to find and to look a person, to get this objective, it is proposed the system given in Fig. 1 and the flow diagram of the algorithm is given in Fig. 2. To implement the system is necessary to take a system to capture some consecutive images and to save these images in the hard disc to a future analysis. The system includes a target of video of 720x480 pixels and a camera of video of 512x480 pixels. The images are taken in black and white. The flow diagram of the Fig. 2 can be summarized in 3 steps: the first step is the capture of the image; the second step is the processing, the segmentation and the analysis of the image; and the last step is the recognizing and the interpretation of the image.

Detection and Following of a Face in Movement Using a Neural Network

483

Fig. 1 The system proposed

2.1 Capture of the Image The time of the capture of the image depends of the scanning that is used in the standard formats of TV NTSC and PAL which capture only the half of the horizontal lines for each line.

2.2 Looking for the Movement Looking for the movement of one person in one image is one of the first steps for the segmentation as can be seen in Fig. 1. Let us consider the following equations [10]: C = |A(to ) − B(t1 )| (1) If C < 0 → C = 0

(2)

Where A is the image in the time 0 and B is the image in the time 1. C is the absolute difference between the two images. Using the difference of one person in two images called T 0 and T 1 with the equations (1) and (2) can be detected the movement. In addition it can eliminate the noisy.

2.3 Segmenting of the Movement Sometimes, depending of the cloths and of the lights, the result of the difference of two images whit the threshold shows only the segmentation of the image in movements with holes. To overcome the problems related with the

484

J.P. Mart´ınez, J. de Jes´ us Rubio Avila, and J.G. Campos

Fig. 2 Flow diagram of the algorithm

illumination and the color, it is used first the dilatation and later the erosion given in the following equations [10]: X · B = (X ⊕ B) ⊗ B

(3)

Detection and Following of a Face in Movement Using a Neural Network

485

Where: X ⊕ B = d ∈ E 2 : x + b for each x ∈ X and b ∈ B X ⊗ B = d ∈ E 2 : d + b ∈ X for each b ∈ B

"

"

(4) (5)

The result dilatation and the erosion is a mask that will be used later.

2.4 Analysis of the Area In the process to look for some person in movement in one image, it is assumed that there is not other body or object in movement. The true is that it is uncommon. That is way it was obtained the mask to find that biggest area which is the person.

2.5 The Mask The process of the binary mask has two purposes: 1. Making the segmentation of the two images T 0 and T 1. 2. Determining how big is the body of the person that moves. To make the segmentation of images it is used the product operation. As the mask is one image that has the numbers 0 and 1, taking the product of one image (T 0 or T 1) with the mask the movement of the body will be segmented.

2.6 Looking for the Face It is applied the derivative for the mask combined with one of the images in a vertical form with the objectives to detect the form of the person and to take a measure of the width of the face. The width of the face gives a measure of how far is the person. This algorithm has the disadvantage that if the person has long hair it can have a mistake with the distance of the person. To look the face, it was used a fuzzy neural network with 12 inputs. The images of the neural network were 3 with 3 kind of skin, the first was a blond person, the second was a brown person and the last was a black person. If the part of the body of tested person is approximated to one of the 3 kind of persons (blond, brown or black), it is the face of the person. The fuzzy neural network used in this study was the found in the toolbox of Matlab.

2.7 The Square After to find the face, it is saved the coordinate of the face. It will be the upper and left part of the square. To make the square for the face will be used the following equations:

486

J.P. Mart´ınez, J. de Jes´ us Rubio Avila, and J.G. Campos

Scaling =

Coordenate of the mask Scale

Original coordinate = Original Image + Scaling

(6) (7)

2.8 The Calculus and the Prediction After to mark with a square the original face, the position of the square and the image are sent to the Kalman filter algorithm. The Kalman filter algorithm will find the position and the velocity of the image form the origin of the square and other images (T 2 and T 3) to follow the localized person. The square of the face of the person, has two objectives: 1. Localizing the person and determining the average position. 2. Letting and specific analysis of the person. To use the Kalman filter algorithm, it is used a mathematical model of the person who walks in straight line. To determine the state of the system it uses the position of the person (p) and the velocity (v); the input (u) is the acceleration and the output (y) is the position which in this case is the center of the square. The velocity v of the object is given by the following equation: vk+1 = vk + T uk

(8)

The change of the velocity is affected by noisy, i.e., the noisy needs to be included to the equation as follows: vk+1 = vk + T uk + vk

(9)

Where vk is the noisy added to the velocity. A similar equation is obtained for the position p: 1 pk+1 = pk + T uk + T 2 uk + pk 2

(10)

Where pk is the noisy added to the position. Let us define the state of the systems as:   p xk = k (11) vk Finally, the equations of the system are:    2  1T T /2 uk + wk xk + xk+1 = T 0 1   yk = 1 0 xk + zk

(12)

Where zk is the measure produced by the instrumentation errors which may be the minimum.

Detection and Following of a Face in Movement Using a Neural Network

487

Fig. 3 Results 1 and 2

2.9 Sending of the Date After to predict the movement with the Kalman filter algorithm, the algorithm send a data proportional to the angle of the turn for the camera. This process uses the communication of two applications: the first is Matlab which find a program to make the communication, the flow of the data and the end of the communication; the second id a microcontroller (P IC16F 877) with a serial cable and a M AX232 circuit reaching velocities of 921Kbps.

3 Simulations The following figures shows the final results of the faces founded in many persons. The Table 1 shows the time used for each program to finish its run.

Fig. 4 Results 3 and 4

488

J.P. Mart´ınez, J. de Jes´ us Rubio Avila, and J.G. Campos

Fig. 5 Results 5 and 6

Fig. 6 Results 7 and 8

Table 1

Image T0 Image T1 Difference Filter Dilatation and erosion Area and center Mixing Looking for the face Kalman filter Turn of the servomotor Total

Minimum times (sec) Maximum times (sec) 0.333 0.333 0.333 0.333 0.016 0.047 0.125 0.235 2.281 2.267 0.547 1.016 0.0001 0.00016 182.34 236.5 0.192 0.92 0.002 0.28 186.1691

241.93

Detection and Following of a Face in Movement Using a Neural Network

489

4 Conclusions In this study an algorithm for the detection and the following of one person in movement in images was proposed. It used some techniques. The mask lets to make the segmentation of the image and to get the width and the area of the body. The fuzzy neural network determine if there exists a face considering 12 average inputs. The Kalman filter algorithm gives a prediction of the movement of a person who moves in front in an straight line. This system was implemented in real time. Acknowledgement. The authors are thankful with the editor to invite them to be part of the committee.

References 1. Baron, R.J.: Mechanisms of Human Facial Recognition. International Journal in Man 15, 137–178 (1981) 2. Barron, J.L., Fleet, D.J., Beauchemin, S.S.: Performance Optical Flow 3. Bledsoe, W.W.: Man Machine Facial Recognition. Technical Report, Palo Alto California (1966) 4. Castleman, K.R.: Digital Image Processing. Prentice-Hall, Englewood Cliffs (1996) 5. Cesar, D.: Seguimiento de Objetos Por Medio de Visi´ on activa. Technical Report of the INAOE (2002) 6. Davies, G.M., Ellis, H.D., Shepherd, J.W. (eds.): Perceiving and Remembering Faces. Academic Press Series in Cognition and Perception. Academic Press, London (1981) 7. Diccionario, L.E.: Diccionario de la Real Academia Espa˜ nola. Vig´esima Segunda Edici´ on (2001) 8. Froba, B., Ernst, A.: Face Detection with the Modified Census Transform. In: Sixth IEEE International Conference Automatic Face and Gesture Recognition, pp. 91–96 (2004) 9. Garcia, C., Delakis, M.: A Neural Architecture for Fast and Robust Face Detection. In: IEEE IAPR International Conference on Pattern Recognition, vol. 2, pp. 40–43 (2002) 10. Gonzalo, P.: Vision Por Computadora. Editorial Alfaomega (2002) 11. Hamouz, M., Kittler, J., Kamarainen, J.K., Palanen, P., Kalviainen, H.: Affine Invariant Face Detection and Localization Using Gmm-Based Feature Detector and Enhanced Aparence Model. In: Sixth IEEE International Conference on Automatic Face and Gesture Recognition, pp. 67–72 (2004) 12. Haralick, R.M., Shapiro, L.G.: Computer and Robot Vision. Addison-Wesley, Reading (1992) 13. Hartley, R.I., Zisserman, A.: Multiple View Gometry in Computer Vision. Cambridge University Press, Cambridge (2000) 14. Hichen, S., Bougemma, N.: Coarce to Fine Face Detection Based on Kkin Color Adaptation. INRIA France, http://www-rocq.inria.fr/imedia

490

J.P. Mart´ınez, J. de Jes´ us Rubio Avila, and J.G. Campos

15. Kongqiao, W.: Automatical Face Detection in Images with Complex Background. Neural Networks and Signal Processing 2, 1027–1030 (2003) 16. Kuo, B.C.: Sistemas de Control Digital. Editorial CECSA (1996) 17. Malasne, N., Yang, F., Paindavoine, M.: Real Time Implementation of a Face Tracking. IEEE Transactions on Neural Networks 14 (2003) 18. Martines, R.K.: Control de Movimientos de Robots Manipuladores. Editorial Prentice Hall, Englewood Cliffs (2003) 19. Rafael, J.: Dise˜ no del Control de un Robot de dos Grados de Libertad Para Aplicaciones de Seguimiento de Objetos. Technical Report of the INAOE (2003) 20. Rowley, H.A., Baluja, S., Takeo, K.: Neural Network-Based Face Detection. IEEE Transactions on Analysis and Machine Intelligence 20, 23–38 (1998) 21. Shing, Y., Jang, R.: Neuro Fuzzy and Soft Computing. Editorial Prentice Hall, Englewood Cliffs (1997) 22. Siyan, K.: Windows 2000 TCP/IP. Prentice-Hall, Englewood Cliffs (2001) 23. Taylor, W.K.: Machine Learning and Recognition of Faces. Electronics Letters 3, 436–437 (1967) 24. Welch, G., Bishop, G.: An Introduction to the Kalman Filter. Department of Computer Science, University of North California at Chapel Hill 25. Yang, M., Kriegman, D.J., Narendra, A.: Detecting Faces in Images: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 24 (2002)

Nonparametric Inter-Quartile Range for Error Evaluation and Correction of Demand Forecasting Model under Short Product Lifecycle Wen-Rong Li*and Bo Li

Abstract. In manufacturing, many important decision-makings are based on demand forecasting of products. However, so many uncertainties lead to demand forecasting error. Especially, semiconductor manufacturing is characterized by its short product lifecycle, which means only limited historical data of a single product can be used to support error evaluation and correction of demand forecasting model, while traditional forecasting error evaluation and correction methods need large sample size to ensure the quality of results. To solve this problem, a new method titled “Nonparametric Inter-Quartile Range” (NIQR), which combines nonparametric density kernel estimation with cumulative probability distribution function, is proposed for error evaluation and model selection, and second quartile is used to correct model’s forecasting error. Numerical experiments in semiconductor manufacturing are used to show the feasibility and effectiveness of the proposed NIQR method for small sample size from short lifecycle product. Keywords: Uncertainty, Demand forecasting error, Short product lifecycle, Nonparametric inter-quartile range, Evaluation, Correction, Small sample size.

1 Introduction Demand forecasting in manufacturing is often affected by uncertainty, and at the same time, to consider all the factors impact product demand in the demand forecasting model is impossible. Demand forecasting error is the primary root of irrationality in production planning decision-makings. So, how to evaluate and correct error of demand forecasting model has been one of the hottest topics in demand forecasting field. Researches have been carried on popularly in error evaluation and correction of demand forecasting models. In Semiconductor Demand Forecast Accuracy Model Wen-Rong Li . Bo Li Institute of Astronautics & Aeronautics, University of Electronic Science and Technology of China, Chengdu 610054, China [email protected], [email protected] *

Corresponding author.

H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 491–500. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

492

W.-R. Li and B. Li

(SeDFAM) [1], variance and covariance had been used to describe demand forecasting error. Geometric Brownian motion (GBM) process [2][3] had been borrowed to calibrate demand forecasting error, showed demand forecasting error has a linear accretion with increase of forecasting period. Forecasting errors have been collected as a new data series for future forecasting error dynamic prediction using different forecasting model, such as GM (1, 1) model and Neural Networks to correct future model’s forecasting error in order to improve forecasting precision [4][5]. Impacts of other important uncertainty factors beside model independent variables are considered in the forecasting model using weight coefficients to increase forecasting precision [6]. However, above methods are popular in research field, while statistical methods are widely used in practice in the manufacturing enterprises. The existing practical statistical methods are as following. Spread of forecast error includes variance, covariance, CV (Coefficient of Variation) [7], Sigma’ rule, range [8] and Inter-Quartile Range (IQR) [9], are used to help forecasting model evaluation and better model selection. Center of forecast error, contains mean, median of forecasting error or relative forecasting error, which are often opted to correct forecasting error. In this paper, we focus on statistical methods. Median and IQR are robust methods, which are resistant to extreme values are popular with manufacturing manager. They can be gained easily using commercial statistical analysis software such as SPSS or JMP. It’s well known that traditional statistical methods all need large samples to ensure the quality of results. However, in semiconductor manufacturing, historical data of one single product is limited for its short product lifecycle. Also these statistical results may dynamic change over time, so it leads smaller samples for median and IQR calculation in one short forecasting period. So far, there are no existing methods suitable for small samples effectively, so it’s high time for us to find an effective way to solve it. Nonparametric kernel density estimation [10] is borrowed to fit the unknown distribution using the small samples from short lifecycle product by its discrete sampling function of model’s independent variable, and then combined with cumulative probability function to calculate IQR and median of forecasting errors to evaluate and correct error of demand forecasting models. This new method called Nonparametric Inter-Quartile Range is proposed for error evaluation and correction of demand forecasting models more effectively, by using relative forecast errors as distribution statistic. It avoids the requirement of large sample size, and this is a breakthrough over traditional statistical methods. This paper is organized as follows. In section 2, propose the framework of NIQR method. Section 3 provides details of NIQR method, and shows its benefits. In section 4, Numerical experiments are used to validate distinction of the proposed new method, by comparing with widely used traditional methods. Finally, conclusion is reported in Section 5.

2 The Framework of Nonparametric Inter-Quartile Range This section will propose and describe the framework of new method Nonparametric Inter-Quartile Range, which solves the problem how to evaluate and correct error of demand forecasting models with small sample size.

Nonparametric Inter-Quartile Range for Error Evaluation and Correction

493

In order to overcome the small sample size problem from short product lifecycle in semiconductor manufacturing, Nonparametric Inter-Quartile Range is proposed to improve. It borrows the benefits of nonparametric kernel density estimation to estimate more simulative samples of model’s independent variable using limited small samples from short product lifecycle. This proposed NIQR method makes of three steps: nonparametric kernel density estimation, forecasting error evaluation using nonparametric IQR and forecasting error correction using nonparametric median. The steps can be described as Fig.1:

Choose statistic, sample size for error evaluation Nonparametric kernel density estimation Step 1

Select optimization bandwidth h

Density estimation for each discrete sampling point

Discrete probability density distribution Step 2 Calculate NIQR using cumulative probability distribution function to evaluate error of forecasting model for better model selection

Select sample size for error correction through DOE Step 3 Use Q2 (median) to correct error of demand forecasting models to reduce its impact to production planning decision-makings

Fig. 1 Steps of Nonparametric Inter-Quartile Range method

3 Details of Nonparametric Inter-Quartile Range and Its Benefits In this paragraph, how to make out the three steps of NIQR will be presented respectively and detailedly. Then discuss the benefits of new proposed NIQR method through comparing it with traditional methods.

494

W.-R. Li and B. Li

3.1 Nonparametric Kernel Density Estimation 3.1.1 Statistic and Sample Size Choice For different products and different market cycles, the demands are in different quantitative levels, such as 100 units or 10000units. If forecasting error is adopted as statistic, it may lead to large gaps among IQR of different products So, RFE (Relative Forecast Error) as Equation 1 is used as statistic in order to reduce iterative calculation error. For denominator can’t be zero, if demand is zero, it will be set as 1unit to ensure continuity of the calculations. RFE =

forecast − actual . actual

(1)

The sample size of error evaluation of demand forecasting models for better model selection can be chose based on the size of historical data. 3.1.2 Two Key Parameters Choice If nonparametric kernel density estimation is selected to fit distribution of demand forecasting error accurately, firstly M(I)SE ((Integrated) Mean Square Error) [10] must be made sure minimal. M(I)SE is lied on the selection of kernel function K(y) and bandwidth h, so these two parameters determine the precision of distribution fitting. According to long-term research both at home and abroad, Epanechnikov kernel is chose as the kernel function in our research, for it minimizes M(I)SE. The function of Epanechnikov kernel is as Equation 2 [11]: ⎧3 ⎪ (1 − x 2 ) | x |≤ 1 . k(x) = ⎨ 4 ⎪⎩ 0 | x |> 1

(2)

Now, let’s introduce an automatic and convenient method ——cross validation method which selects bandwidth of nonparametric kernel density estimation. The foundational thought is to cancel one point at each calculation, and this cancelled point is used to validate. Combining parametric maximum likelihood with cross validation method, we can get the equation for optimization estimate of bandwidth as Equation 3 [10]: N

N

hˆ = arg max L(h) = arg max Π fˆh* (X j ) = arg max Π [ h

h

j =1

h

j =1

1 ( N − 1)h

N

∑ k( i≠ j

xi − x j h

)] ,

(3)

where, X j -the cancelled point at j cross validation;

L(h) -maximum likelihood when bandwidth is h . The meaning of this equation is to select bandwidth where maximum likelihood with cross validation is maximal.

Nonparametric Inter-Quartile Range for Error Evaluation and Correction

495

3.1.3 Probability Density Estimation After opted kernel function and optimization bandwidth h, let’s go on to estimate corresponding probability density fˆ(x) of any point x in (−∞,+∞) , and fit out any unknown distribution by using probability density function as Equation 4 [10] [11]: 1 fˆ ( x ) = nh

n

∑ K( i =1

x − Xi ), h

(4)

where, K(⋅) -kernel function;

h -optimization bandwidth; X i -No.i point in the samples X;

n-samples quantity. For instance, to sort the discrete sampling points in ascending, from − ∞ to + ∞ , sample a point when cross a spacing of Δ . In practice application, we usually use reasonable large values instead of ± ∞ . Equation 4 is used to work out probability density for each discrete sampling point respectively, and fit out population distribution from samples with any unknown distribution. This method can increase quantity of samples for distribution fitting by discrete sampling of model’s independent variable. From this method, we can get the distribution form not a mathematic function but a series of discrete probability densities in numerical value form.

3.2 Error Evaluation and Correction of Demand Forecasting Models For unknown distributed of samples got by nonparametric kernel density estimation, IQR can be got though cumulating by using cumulative probability distribution statistic. Firstly, nonparametric kernel density estimation is used to fit population distribution with known samples, and then quartiles of IQR are estimated with discrete cumulative probability distribution function as Equation 5: X

Fˆn ( x) =

∑ fˆ (x) × Δ .

(5)

−∞

In Equation 5, fˆ ( x ) is a series of discrete probability density values in population distribution, which are simulated by using nonparametric kernel density estimation, Δ is the spacing between sampling points. Use Equation 5 to calculate the quartiles of NIQR: the first quartile (25th percentile) NIQR-Q1, the second quartile (50th percentile) NIQR-Q2 and the third quartile (75th percentile) NIQR-Q3. Then subtract NIQR-Q1 from NIQR-Q1 to calculate the Nonparametric Inter-Quartile Range: NIQR = NIQR-Q3 –NIQR-Q1, in order to help model evaluation and selection. In error correction step, the sample size of error correction can be chose through DOE by NIQR method, such as 3 to 10 samples before current time point to select

496

W.-R. Li and B. Li

an optimum sample size which minimizes forecasting Mean Square Error (MSE) dynamically over a period of time. Then the second quartile NIQR-Q2 of RFE can be used to correct future error of demand forecasting model as Equation 6. corrected forecasted demand = model forecast result × (1 + NIQR - Q2 of RFE) .

(6)

3.3 Comparison between Methods By comparing traditional model’s forecasting error evaluation and correction methods with the proposed NIQR a conclusion can be drawn that the obvious difference between these methods are that sample size of traditional methods are used for distribution fitting equal to historical points, while Nonparametric InterQuartile Range has larger samples over traditional methods for its discrete sampling function, larger sample size can be got by choosing smaller sampling spacing of independent variable X.

4 Numerical Experiments and Results 4.1 Question Presentation and Data Format In semiconductor manufacturing enterprises, many important decision-makings are based on demand forecasting. There are so many models can be used for product demand forecasting, but which one is a better option? How to evaluate and correct the forecasting error with small samples to support more reasonable decision-makings, as semiconductor manufacturing is characterized by short product lifecycle? In this section, nine forecasting models are built for one special product from an international semiconductor manufacturing company, and then use traditional methods and NIQR to evaluate and correct forecasting errors of these models respectively. Let’s go to introduce the data format used in numerical experiments: T1, T2…, T8, …are the forecasted time with equal interval; A, B…, H, …are the actual values of demand at each forecasted time; A1, A2…, B1, B2…, H1, H2, …are the forecasting values of demand at forecasted time with different forecast models. Then according to statistic discussion in subsection 3.1.1, RFE is chose as statistic in our research to get the data used for error evaluation and correction, as Table 1. Table 1 RFE data RFE Forecasted time Model 1

Model 2

Model 3

…

Model 9

T1

(A1-A)/A

(A2-A)/A

(A3-A)/A

…

(A9-A)/A

T2

(B1-B)/B

(B2-B)/B

(B3-B)/B

…

(B9-B)/B

…

…

…

…

…

…

T8

(H1-H)/H

(H2-H)/H

(H3-H)/H

…

(H9-H)/H

…

…

…

…

…

…

Nonparametric Inter-Quartile Range for Error Evaluation and Correction

497

There are 33 samples of this product, 27 samples are used for error evaluation and model selection, while 6 samples are used for error correction. The optimum sample size for error correction is chose by DOE from 3 to 10 samples.

4.2 Experiments of Models Evaluation and Selection Use the first 27 samples to calculate IQR by traditional and nonparametric methods respectively. Fig.2 and Fig.3 are the results of traditional IQR method and nonparametric IQR method. From both methods, mode 9 is chose as the best demand forecasting model as its minimal IQR. But the difference between the two methods will be discussed in the coming subsection.

4.3 Why NIQR Different? Now let’s move to study why NIQR method is different from traditional methods via their sample size for distribution fitting. In Fig.2, every model has 27 samples for distribution fitting, while as the discussion in subsection 3.1.3, NIQR method has discrete sampling function of independent variable X, so there are much more 0.8 0.7 0.6 0.5 0.4 tri 0.3 au Q 0.2 0.1 0.0 -0.1 -0.2 -0.3

IQR Q1 Q2(Median) Q3

1

2

3

4

5

6

7

8

9

Model No.

Fig. 2 Traditional IQR method for error evaluation and model selection 0.8 0.7 0.6 0.5 it 0.4 aru 0.3 Q 0.2 0.1 0.0 -0.1 -0.2 -0.3

IQR Q1 Q2(Median) Q3

1

2

3

4

5

6

7

8

9

Model No.

Fig. 3 Nonparametric IQR method for error evaluation and model selection

498

W.-R. Li and B. Li

Table 2 Sample size of Nonparametric IQR method for each model

Model No.

1

Sample size

122

0.05

0.045 li 0.04 abb 0.035 o 0.03 pr 0.025 ty sin 0.02 de 0.015 0.01 0.005 0 -0.9 -0.8 -0.6 -0.5 -0.4 -0.3 -0.1 0.01 0.14 0.27 0.4 0.53 0.66 0.79 0.92 1.05 1.18 1.31 1.44 1.57 1.7 RFE

a) model 1 l bi bao pr yt sin de

0.05 0.045 0.04 0.035 0.03 0.025 0.02 0.015 0.01 0.005 0 -0.9 -0.8 -0.6 -0.5 -0.4 -0.3 -0.1 0.01 0.14 0.27 0.4 0.53 0.66 0.79 0.92 1.05 1.18 1.31 1.44 1.57 1.7 RFE

d) model 4 0.05 0.045 li 0.04 abb 0.035 o 0.03 pr 0.025 yt is 0.02 ne 0.015 d 0.01 0.005 0 -0.9 -0.8 -0.6 -0.5 -0.4 -0.3 -0.1 0.01 0.14 0.27 0.4 0.53 0.66 0.79 0.92 1.05 1.18 1.31 1.44 1.57 1.7 RFE

g) model 7

2

3

4

5

6

7

8

9

134

117

241

197

235

180

140

116

0.05

0.045 li 0.04 b 0.035 bao 0.03 pr 0.025 ty sin 0.02 de 0.015 0.01 0.005 0 -0.9 -0.8 -0.6 -0.5 -0.4 -0.3 -0.1 0.01 0.14 0.27 0.4 0.53 0.66 0.79 0.92 1.05 1.18 1.31 1.44 1.57 1.7 RFE

b) model 2 0.05 0.045 li 0.04 ab 0.035 obr 0.03 p yt 0.025 i 0.02 nse d 0.015 0.01 0.005 0 -0.9 -0.8 -0.6 -0.5 -0.4 -0.3 -0.1 0.01 0.14 0.27 0.4 0.53 0.66 0.79 0.92 1.05 1.18 1.31 1.44 1.57 1.7 RFE

e) model 5 0.05 0.045 li 0.04 abb 0.035 o 0.03 pr 0.025 yt is 0.02 ne 0.015 d 0.01 0.005 0 -0.9 -0.8 -0.6 -0.5 -0.4 -0.3 -0.1 0.01 0.14 0.27 0.4 0.53 0.66 0.79 0.92 1.05 1.18 1.31 1.44 1.57 1.7 RFE

h) model 8

0.05

0.045 li 0.04 b 0.035 a b o 0.03 r p y 0.025 t i 0.02 s n e 0.015 d 0.01 0.005 0 -0.9 -0.8 -0.6 -0.5 -0.4 -0.3 -0.1 0.01 0.14 0.27 0.4 0.53 0.66 0.79 0.92 1.05 1.18 1.31 1.44 1.57 1.7 RFE

c) model 3 0.05 0.045 li 0.04 b 0.035 a b o r 0.03 p 0.025 y t 0.02 i s n e 0.015 d 0.01 0.005 0 -0.9 -0.8 -0.6 -0.5 -0.4 -0.3 -0.1 0.01 0.14 0.27 0.4 0.53 0.66 0.79 0.92 1.05 1.18 1.31 1.44 1.57 1.7 RFE

f) model 6 0.05 0.045 li 0.04 b 0.035 a b 0.03 o r p 0.025 y t 0.02 i s n e 0.015 d 0.01 0.005 0 -0.9 -0.8 -0.6 -0.5 -0.4 -0.3 -0.1 0.01 0.14 0.27 0.4 0.53 0.66 0.79 0.92 1.05 1.18 1.31 1.44 1.57 1.7 RFE

i) model 9

Fig. 4 Nonparametric distributions of demand forecasting errors (x coordinate is RFE, and y coordinate is density probability)

samples for each model distribution fitting in Fig.3, and their sample size and nonparametric distributions are shown as Table 2 and Fig.4. It’s obvious that the samples number is larger than traditional methods. Because NIQR has more samples over traditional methods, it can fit the population distribution and estimate relative parameters more correctly. Therefore it can evaluate and correct the model forecasting error more effectively. The effectiveness of NIQR for error correction will be validated in the coming subsection.

4.4 Sample Size Selection of Correction and Error Correction Methods Comparison After chose the best demand forecasting model 9, the next 6 samples of this model are used for NIQR error correction method validation. In order to choose an optimum sample size for error correction, the NIQR-Q2s of RFE using 3 to 10 samples before current sample are used for error correction using Function 6. Then use forecasting Mean Square Error (MSE) to evaluate the effectiveness of NIQR error correction method with traditional median and mean error correction methods. The results are shown as Fig.5. It’s obvious that 6 samples with NIQR method which minimizes MSE is the optimum sample size for model 9 forecasting error corrections, and it maybe changed over time, so DOE can be carried on over a period of time. After sample size selection, error correction of future forecasting can be achieved via Function 6 too.

Nonparametric Inter-Quartile Range for Error Evaluation and Correction

499

1000 NIQR-Q2

950

median

mean

900 SME 850 800 750 700

3

4

5 6 7 8 sample size for error correction

9

10

Fig. 5 Sample size selection for model 9 error correction

4.5 Results from Numerical Experiments Comparisons 1. Model forecasting error evaluation and better model selection: NIQR method increases the samples for distribution fitting to ensure the quality of results by its discrete sampling function of independent variable X, so it can estimate the population distribution more correctly over traditional methods. 2. Error correction: From DOE of sample size selection, the following two conclusions can be drawn. One is that NIQR-Q2 is a more robust method for sample size selection of error correction over traditional methods no matter how many the sample size is. Another is it can get the optimum sample size which minimize forecasting MSE. The proposed NIQR can avoid large samples requirement, and fit the distribution with small sample size more correctly. So it’s suitable for products with short lifecycle. Dynamic model evaluation and error correction can be carried out by collecting real-time data over time which is a great breakthrough over traditional methods.

5 Conclusion In this paper, a new model forecasting error evaluation and error correction method titled Nonparametric Inter-Quartile Range (NIQR) is proposed, which combines the distribution fitting based on nonparametric kernel density estimation with cumulative probability distribution function. It’s appropriate for error evaluation and error correction with small sample size from products with short lifecycle. This is a breakthrough over traditional methods, which means NIQR can fit the distribution more reasonable by increasing its sample size using the discrete sampling function of nonparametric kernel density estimation, in order to evaluation and selection forecasting models more correctly. With optimum sample size for error correction, NIQR can correct the forecasting errors more effectively with minimal forecasting Mean Square Error. And this new method can be applied in other factors analysis besides demand forecasting.

500

W.-R. Li and B. Li

Acknowledgements. This work is sponsored by the National Natural Science Foundation of China under Grant No. 70701007/G0109, supported by the Technology Support Plan of Sichuan province under Grant No. 2008JY0060, and also supported by Youth Foundation of University of Electronic Science and Technology of China (UESTC).

References 1. Cakanyildirim, M., Roundy, R.O.: SeDFAM: Semiconductor Demand Forecast Accuracy Model. IIE Transactions (Institute of Industrial Engineers) 34, 449–465 (2002) 2. Liang, Y.Y., Chou, Y.: Option-based Capacity Planning for Semiconductor Manufacturing. In: 2003 IEEE International Symposium on Semiconductor Manufacturing (ISSM 2003), pp. 77–80 (2003) 3. Chou, Y., Cheng, C.T., Yang, F., Liang, Y.Y.: Evaluating Alternative Capacity Strategies in Semiconductor Manufacturing under Uncertain Demand and Price Scenarios. International Journal of Production Economics 105, 591–606 (2007) 4. Zhao, X., Liu, T., Zhou, B., Hu, Y.: The Smoothing Improvement and the Application of Grey Model GM (1, 1). Journal of Northeast Dianli University 26(4), 63–66 (2006) 5. Pang, Z., Niu, Y.: Generalized Predictive Control Based on Predictive Error Correction by Neural Networks. Journal of Qingdao Technological University 27(4), 92–96 (2006) 6. Hu, H., Sheng, W., Li, G., Chen, Z.: Improvement of Linear Trend Estimation Method. Consumption Guide 143, 136 (2006) 7. Wallace, J.H., Mark, L.S.: Factory Physics: Foundations of Manufacturing Management, 2nd edn. Copyright 2001,1999, 1995 by the McGraw-Hill Companies, Inc. Authorized for Beijing: Tsinghua University Publishing House 251–254, 380–389 (2002) 8. Shi, G., Yang, L., Gong, W.: Foundation of Quanlity Control and Reliability Engineering, pp. 74–84. Chemical industry Publishing House, Beijing (2005) 9. Ma, G.: Managing Statistics. Science publishing house, Beijing (2002) 10. Matthias, H.: Introduction to Nonparametric Econometrics. HEC Lausanne & FAME, 20–21 (2003) 11. Jack, J., John, D.: Eonometric Methods, 4th edn., pp. 370–375. Copyright by the McGraw-Hill Companies, Inc. Translation Copyright by China Economics Publishing House (2002)

Simulated Annealing and Crowd Dynamics Approaches for Intelligent Control Qingpeng Zhang 1

Abstract. This is a survey of two approaches to intelligent control. The approaches are based on the author’s previous and ongoing projects in the Maximum Clique Problem (MCP) and the Crowd Dynamics. The ideas came from computational intelligence and fluid dynamics. One is Simulated Annealing (SA) algorithm and TABU search’s applications in the combinatorial search of intelligent control. Another one is from Continuum Dynamics and this approach can achieve a kind of group intelligence. This article is originally produced as a proposal and then a survey, thus is incomplete and lacking of simulation data. Keywords: Simulated Annealing, Crowd Dynamics, Intelligent Control.

1 Introduction In 1970s, King-Sen Fu and George N. Saridis introduced the idea that Intelligent Control (IC) is the intersection of Artificial Intelligence (AI), Automatic Control (AC) and Operations Research (OR). In [1], it can be described as follows: IC = AI ∩ AC ∩ OR

(1)

This relations and interactions can be expressed in Fig.1. Since its emergence, due to its perspective fascinating traits, Intelligent Control has attracted countless talented scientists and engineers to do research in relevant fields. After 30 years’ endeavor, there are mainly 6 kinds of Intelligent Controller: 1) 2) 3) 4)

Hierarchically Intelligent Control System; Expert Control system; Fuzzy Control; Neural Networks (NN)-Based Control;

Qingpeng Zhang Key Laboratory of Image Processing and Intelligent Control, Department of Control Science and Engineering, Huazhong University of Science and Technology, Wuhan 430074, Hubei, China [email protected] H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 501–506. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

502

Q. Zhang

Fig. 1 [1] Interaction of AI, OR and Control Theory, and resulting intelligent control

5) 6)

Human-Simulated Intelligent Control; Integrated Intelligent Control which combines two or more above areas.

In this section, the author would like to list the basic definition and functions of Intelligent Control’s different aspects/layers. The definitions are mainly from the Task Force on Intelligent Control (IEEE Control System Society, 1993) [2] and the panel discussion held by Panos J. Antsaklis (Notre Dame) in the 38th Conference on Decision and Control, 1999. Although many years have passed, the ideas are still valid and full of vitality. Most of them are classical and the guideline for new researchers now. Ideas from recent publications are also involved.

1.1 Control Control is to direct a system to a preassigned goal or to maximize a preassigned measure of utility under a set of specifications. Control methodology is the set of techniques and procedures used to construct and implement a controller for a dynamical system. The general control problem is how to construct a controller, given a model of the plant, so that specifications on how we would like the closed loop system to behave holds. [1]

1.2 Intelligence and Artificial Intelligence We still do not have a satisfactory quantitative way to characterize the “intelligence” of a controller or of a system. [7] An acceptable definition is stated in [1]:

Simulated Annealing and Crowd Dynamics Approaches for Intelligent Control

503

Intelligence is a (control) tool of fighting complexity and it has emerged as a result of evolution. Intelligence grows via 1) growth of computational power and 2) accumulation of knowledge of how to sense. Artificial Intelligence is defined to be the study of mental faculties through the use of computational methods. To achieve AI, there are mainly two approaches: 1) Using NN to simulate the capabilities that human brain possesses entirely. 2) Employing other aspects of AI like the emulation of evolution, annealing process and the ant colony’s intelligence.

1.3 Intelligent System and Intelligent Control Intelligent System’s attribute and capability are clarified in [3]: An intelligent system should have the ability to act appropriately in an uncertain environment, where an appropriate action is that which increases the probability of success, and the success is the achievement of behavioral sub-goals that support the system’s ultimate goal. There are varies of definitions of Intelligent Control from different aspects and understandings, in [3], intelligent control is defined as a computationally efficient procedure of directing to a goal of a complex system with incomplete and inadequate representation and under incomplete specifications of how to do this. Researchers are attempting to combine extend theories and methods from control, computer science and operations research to form a brand new kind of control, which is able to solve problems of complex, nonlinear and uncertain systems. The biggest advantage and the reason why Intelligent Control is introduced are that Intelligent Control can address control problems that cannot be formulated in the language of conventional control. The controller and the system to be controlled are combined [2]. Compared to conventional control, the “Control” in intelligent control is more general and more like the word which is used in everyday language [2]. The research areas relevant to intelligent control, in addition to conventional control include planning, learning, combinatorial search, hybrid systems, fault diagnosis, reconfiguration, NN, Fuzzy Logic, etc. Among these areas, combinatorial search holds a significant position because computational complex is the central issue and search algorithms are the main tool of fighting complexity. A variety of modern optimization algorithms was developed from different domains (which are mentioned a little in Section 2.2 and will be discussed in detail in Section 3). In this paper, the topics are combinatorial search algorithms.

1.4 New Modern Optimization Algorithms Based Intelligent Control Intelligent control encompasses many fields from conventional control, such as optimal control, robust control, stochastic control, linear control, and nonlinear control, as well as the more recent fuzzy, genetic, and neuro–control technologies [2]. Compared to 1980s to 1990s, neural networks, genetic algorithms, fuzzy

504

Q. Zhang

logic, and adaptive critics are no longer new ideas. In contrast, more conventional linear and nonlinear control theories have been improved approvingly over the years. Therefore, combinations of these techniques are used to capitalize on each technique’s strong points and minimize overall drawbacks. [8] To researchers including me, the challenge is to juxtapose all the methods mentioned above to develop controllers stronger than any single one. In this essay, I want to introduce several new tools from two domains: Statistical Physics and Fluid Dynamics based on my experience and understanding. I name them as Simulated Annealing and Crowd Dynamics Based Intelligent Control.

2 Simulated Annealing and Crowd Dynamics Based Intelligent Control In [5], Prof. K.M. Pissano (Ohio State) stated that: Intelligent control achieves automation, according to the authors, via the emulation of biological intelligence. It either seeks to replace a human who performs a control task (e.g., a chemical process operator) or it borrows ideas from how biological systems solve problems and applies them to the solution of control problems (e.g., the use of neural networks for control). Admittedly, bio-inspired computing and automation are intuitive and have produced significant impetus to the development of AI and intelligent control. However, intelligent goal selecting & path planning, pattern recognition, optimization, and other attributes known as factors of intelligent control under unexpected & complex environments can be achieved not only by emulation of biological intelligence. How a metal freezes into a minimum energy crystalline structure? How the liquid form natural lanes when encountered obstacles? Such questions are answered by physicians and mathematicians. Mathematical or statistical models observed by them could give us a new way to reach the global optimum in a particular system. For the first question, Metropolis Model is invented by Metropolis in 1952-1953. It was introduced to combinatorial optimization in 1983 and became widely know as Simulated Annealing (SA) algorithm [9]. TABU local search can be combined with SA very easily. The combined algorithm can obtain excellent capability for combinatorial optimization (I will describe it in section 3.1). For the second question, Fluid Dynamics has provided exact model (section 3.2). In the author’s view, both SA-TABU combined algorithm and Fluid Dynamics can be applied in control engineering in a simple way. As the new control tools will use modern optimization algorithms that are not widely used in intelligent control, I would like to name the proposed control methods as New Modern Optimization Algorithms Based Intelligent Control.

2.1 Simulated Annealing and TABU Based Intelligent Control We can develop a new type of controller based on SA and TABU local search. SA exploits an analogy between the way in which a metal cools and freezes into a minimum energy crystalline structure and the search for an optimum in a more

Simulated Annealing and Crowd Dynamics Approaches for Intelligent Control

505

general system. It has been proved theoretically to be convergent to the global optimum with appropriate Markov chains; TABU search is another effective algorithm for combinatorial optimization because the tabu list can help the algorithm jump from local optimum. According to the author’s project of Maximum Clique Problem (please refer to [11]), both of them are able to find the global optimum with a much higher probability than other algorithms like NN, but consume longer running time. Thus I combined them to achieve faster speed. The results showed that although for a certain problem, each computing step is slower than SA alone due to the tabu list, the number of steps is greatly reduced (for the chance to succeed is higher) so that the overall speed is increasing obviously. (Detailed process and results can be found in the author’s unpublished paper [11].) The new combined algorithms can replace old algorithms for combinatorial search and optimization in intelligent control directly.

2.2 Continuum Model Based Group Intelligence The research on the simulation of crowd dynamics was inspired by the demand from a variety of fields and implemented to numerous disciplines. Other than computer graphics, the crowd models also have important applications in psychology, disaster management, and eventually intelligent control. Its application in intelligent control will focus on path planning and groups’ goals selection. Most research projects in this field are multi-agent based. However in large groups and complex environments, the computing cost will increase exponentially. In [10], a continuum based model is developed from fluid dynamics. The model treats a group of individuals as a continuum. By parallelizing the functions in fluid dynamics into the algorithms, we can naturally reproduce realistic crowds with limited computing cost compared to multi-agent based models. The successful simulation of real crowds can have important application in multi-robots systems, guidance and planning.

3 Expected Results 3.1 Results of Simulated Annealing and TABU Based Intelligent Control As what have been stated above, the larger chance of finding the global optimum is the major advantage of SA. The drawback of slow speed can be alleviated by combining other algorithms like NN, GA and TABU. The prospect of SA based controller is encouraging.

3.2 Results of Continuum Model Based Group Intelligence The ability to form natural flows and avoid obstacles with low computing cost will enable the very large complex multi-robots systems, transportation systems, air

506

Q. Zhang

traffic management systems and other intelligent control systems to choose optimal paths and reasonable goals more quickly. Thus it will reduce accident rate in multi-systems. This attribute is extremely important for transportation and air traffic management systems. To conclude, the prospect of continuum crowds’ application in control systems is promising. Acknowledgments. The author thanks the anonymous reviewer for their constructive comments and suggestions. Prof. Linqiang Pan and Dr. Zehui Shao have provided tutorial support to the author’s research. Prof. Tsung-Chow Su has opened the door of crowd dynamics for the author.

References 1. Saridis, G.N.: Intelligent Robotic Control. IEEE Transaction on Automation Control 28, 547–557 (1983) 2. Panos, A.: Defining Intelligent Control. Technical report, Task Force on Intelligent Control, IEEE Control System Society (1993) 3. Meystel, A.: Intelligent Control. Encyclopedia of Physics and Technology. Academic Press, London (1993) 4. Passino, K.M.: Bridging the Gap Between Conventional Intelligent Control. Special Issue on Intelligent Control. IEEE Control Systems Magazine 13, 12–18 (1993) 5. Passino, K.M., Samad, T.: Perspectives in Control: New Concepts and Applications. IEEE Press, NJ (2001) 6. Xing, W., Xie, J.X.: Modern Optimization Algorithms, 2nd edn. Tsinghua University Press, Beijing (2005) (in Chinese) 7. Antsaklis, P.J., Baras, J.S., Doyle, J.C., Ho, Y.C., Johnson, T.L.: At the Gates of the Millennium: Are We in Control? In: Proceeding of the 38th Conference on Decision and Control (1999) 8. Neidhoefer, J.C., Krishnakumar, K.: Intelligent Control for Near-Autonomous Aircraft Missions. IEEE Transactions on Systems, Man, and Cybernetics—Part A: System and Humans 31, 14–29 (2001) 9. Kirpatrick, S., Gelatt, C.D., Vecchi, M.P.: Optimization by Simulated Annealing. Science 220, 611–680 (1983) 10. Treuille, A., Cooper, S., Popovi´c, Z., Crowds, C.: IEEE tranactions on Graphics 25(3), SIGGRAPH (2006) 11. Zhang, Q.P.: A Combined Simulated Annealing Algorithm for the Maximum Clique Problem. Technical report, Huazhong University of Science and Technology (2008)

Accomplishing Station Keeping Mode for Attitude Orbit Control Subsystem Designed for T-SAT Montenegro Salomón and Amézquita Kendrick 1

Abstract. Station keeping mode (SKM) for Attitude Orbit Control Subsystem (AOCS); this mode is an operational approach regarding the attitude dynamics control. The essential features of the design methodology is researching the basic theory and then afterwards, make an iterative design approach using some premise/assumptions, implementing the system simulation through the Simulink Matlab software package, designing the required controllers, monitor and analyzing the responses until the design give the best results within the range required. Firstly the thruster configuration is designed to acquire the parameters of the torque on the satellite. Next, the controllers are based on the well known PID control law. Controllers optimize the attitude and are used during the maneuvers for SKM. Afterwards the simulation results are given, and presented to demonstrate the performance and validity of the AOCS design approach. Finally the simulation results demonstrate that all the requirements were accomplished and the Station Keeping Mode was successfully designed.

1 Introduction Communication Satellite systems have become a great interest in the last decades. This is due to the technology progress, the necessity to increase scientific possibilities and telecommunication performance capacity in the long-term, but also because they allow nation’s governments to develop a complete satellite system at a large. This also provides possibilities for telemedicine, tele-education, educational projects at moderate costs for this country, involving students and scientists in space missions and acquiring experience, knowledge and expertise for their future career. In the context of a worldwide communications network, the role of satellite communications systems is very important. Satellite communications links add capacity to existing communications capabilities and provide additional alternate routings for communications traffic. Satellite links, as one of several kinds of long-distance links, interconnects switching centers located strategically around the world. Montenegro Salomón . Amézquita Kendrick Beihang University Astronautics and Aeronautics, Beijing, China H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 507–516. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

508

M. Salomón and A. Kendrick

T-SAT Project Communication satellites are generally located in Geostationary Orbit (GEO) and used according the configuration and specific design requirements. However a GEO satellite is usually launched with a payload that allows different implementations, therefore is also necessary the design a reliant and efficient launch vehicle that allows the positioning in space of a rather large mass and dimensions satellite. Below is introduced the communication satellite project investigated defined as T-SAT. The T-SAT satellite developed model, represents a special project for students training and can be used in professional courses of satellite technology. The students are divided in several teams, each one working with specifics task regarding the complete design and simulation project. The aim of the experience is to accomplish the necessary work in a concurrent engineering design manner and make all the joints and interfaces necessaries in order to improve the development of the design. This situation give good experiences about the whole satellite design process and allows exercising in the practice all the knowledge learned during the professional training courses. It is primarily an educational project with the objective to provide a realistic learning environment and hands-on experience for undergraduates, graduates and staff, in the development of a communication satellite based in a conceptual platform. The T-SAT, is a virtual satellite, which needs to meet some necessities according to the clients requirements documentation in which are explained some points that must be covered by the AOCS. For this paper we regard only those that are related to the performance requirements in Station Keeping Mode (SKM).operation. Function Requirements • The subsystem shall meet the required performance for the correct and expected accuracy of pointing and functioning of the Payload subsystem. Performance Requirements The attitude and orbit control system (AOCS) is one of the key subsystems within a satellite, as it's heavily interfacing to all other subsystems, then is necessary to have devices that can accomplish all tasks and reach that making well performance, during this work the focusing will be on transfer orbit and geostationary orbit. Geostationary Orbit • The attitude error of station-keeping mode should be less than: Roll: ±0.08°, Pitch: ±0.08°, Yaw: ±0.20° • When the station position changes, the S/C will be a long term attitude biased to keep the beam coverage on the Earth unchanged, and in such cases the attitude error of normal mode could be 1.2 times of the nominal value. • Station keeping errors: W/E ±0.1°, N/S ±0.1°

Accomplishing SKM for Attitude Orbit Control Subsystem designed for T-SAT

509

This paper was generated investigating the assessment, possibilities and performance of AOCS subsystem, in order to check the attitude control during station keeping mode of the T-SAT and at the same time tried to improve this kind of control. The design of the controller of attitude actuators is the primary aim of this paper and therefore will be explained. Station Keeping Mode During acquisition of the ground station position in geostationary orbit , as well as North-South Station Keeping (NSSK) and East-West Station Keeping (EWSK), the SKM is used to keep earth-pointing three axis attitude for payload operation. The 10N thrusters in pairs are symmetrically fired in drift orbit, to execute final ground station position acquisition maneuvers (The S/C to be accelerated east or west) and inclination correction maneuvers (The S/C to be accelerated south or north). In SKM, Earth Sensors are used to measure roll and pitch angles and Sun Sensors are used to measure yaw angle. The control actuators are 10N thrusters while the Momentum Wheels speed is kept constant. Since propellant to be consumed in NSSK is about 80 percent of orbit correction propellant, the optimal time should be selected to execute NSSK firing maneuvers in order to save propellant, and to extend S/C service life. During the NSSK, it is necessary to switch from NM to SKM because of a larger ΔV requirements, and 10N thrusters are used to control three-axis attitude. The AOCS needs the attitude feedback information as well as angular rate feedback information. Therefore, the gyros need to be warmed up firstly before entering the station keeping mode. When the integration of the angular rates from the gyros is used for the yaw measurement, it is necessary to calibrate the gyros by ground commands to determine compensation values for gyro drift. When the gyros are not able to provide angular rates of the S/C accurately, and only measurements of S/C’s three-axis angles are used for the attitude control in SKM; AOCS shall operate in the back-up SKM with selected telecommand according to the solar array characteristics. Simulation During the Attitude simulation process some mathematical models were used to emulate the physical behavior of the satellite and the space environment. These models will be reviewed below. Attitude Premise/Assumption In order to perform the simulations some values were assumed to be: For Satellite’s Dynamic and Solar Disturbance Torque: • • • •

Orbital elements: a=42165.53 Km, i=0.6°, e=0.001, Ω=60°, ω=25°, λ=105.1° Inertial Values: Ix=3000, Iy=1000, Iz=3000 Center of mass position: cx=0.00, cy=0.00, cz=0.834 Mass: 1250; Launch time: 00:00 Jan 1st 2008

510

M. Salomón and A. Kendrick

For Thruster configuration design and simulation: • Body size: 2200×1700×1700 (mm) and Center of mass position: cx=0.00, cy=0.00, cz=0.834 For Station-keeping controller design and simulation: • Satellite’s dynamic model • Solar Disturbance torque model • Thruster model Satellite Mathematical Model: The simplified dynamics equations of a rigid spacecraft are: ψ

φ

In the equations, assume that the inertial matrix of the satellite is diagonal matrix, , ,

,.

is the coordinate of the angular momentum of

momentum wheels in body coordinate.

is the coordinate of the outer

torque applied to the satellite in body coordinate.

is the orbit angular rate, i.e. 0 . φ, the angular rate of orbit coordinate with respect to inertial coordinate is 0 θ and ψ are the roll, pitch and yaw attitude respectively of the satellite. Thrusters Mathematical Model: For station keeping simulations the thrusters mathematical model was developed taking into account all the torques produced by each thruster gathered in the a or b branch (See figure 1). Also, knowing that in NSSK the 6a and 7a thrusters are used to generate northward force so they are activated all along the NSSK maneuver. If the X axis Pseudo Rate Modulator (PRM) output is +1, +X torque should be generated. Therefore 7a should be turned off to generate +X torque. So this is called “off-modulation”. When X axis PRM output is -1, 6a should be turned off to generate –X torque. For EWSK, 2a and 3a are used to generate eastward force to gain eastward velocity increment. The off modulation is used when either 2a or 3a is needed to generate torque; in that case one is turned off meanwhile the other one is used to generate the appropriate torque. With these conditions and by using combinational logic the combinational functions were obtained.

Accomplishing SKM for Attitude Orbit Control Subsystem designed for T-SAT

511

Fig. 1 Thrusters branches Fig. 2 Single Thruster Model

Thruster Configuration Design and Simulation Thrusters’ forces and torques: A single thruster is able to generate a torque of 10N-m. And with respect of the satellite’s body coordinate frame the forces and torques generated depend on the position of the thrusters used.

512

M. Salomón and A. Kendrick

Torque generating strategy: In SKM the entire torque on the satellite is the sum of the components of the torque generated by each active thruster. Station-Keeping Controller Design and Simulation • Controller design: In station keeping mode the controllers are based on the well known PID control law: Controller

.

.

.

Where the derivative error is taken directly from the gyros output, figure 3. • PID Controller Simulink model Fig. 3 PID Controller

• PRM Simulink model: This model was given by experts. The output of the PRM can be 1, 0, or -1. The switch on point is 0.02 and the switch off value is 0.01. Fig. 4 PRM Simulink Model

• EW_SKM Simulink model: This model consists of the integration of the modules: satellite’s dynamic, solar disturbance torque, PID controllers, PRM, and EWSK Thrusters model. It can be seen in the following figure.

Accomplishing SKM for Attitude Orbit Control Subsystem designed for T-SAT

513

Fig. 5 East/West Station Keeping Attitude Matlab Simulink Model

• NS_SKM Simulink model: Basically same as the EW_SKM Simulink model but the thruster model is based on the NSSK torque generation requirements. • Controller parameter selection: All the controllers used in EW_SKM and NS_SKM were tuned up by the classical trial and error process. Station Keeping Simulation Results EWSK Attitude Simulation The initial conditions for this simulation were: 0.05 0.05 , 0.15

0 0 0

Fig. 6 First 50 seconds attitude response for EWSK

,

0 93.9693 and 0

0 0 0

Fig. 7 Steady state attitude for EWSK (Simulation time: 1hr.)

514

M. Salomón and A. Kendrick

Fig. 8 EWSK Velocity change response

NSSK Attitude Simulation The initial conditions for this simulation were the same that above.

Fig. 9 First 200 seconds attitude

Fig. 10 Steady state attitude (Simulation time: 1hr.)

Accomplishing SKM for Attitude Orbit Control Subsystem designed for T-SAT

515

Fig. 11 NSSK Velocity change response

2 Conclusions • The satellite’s attitude transient response in east/west station keeping can be seen in the figure 6, from this it can be noticed how the control system o brings the satellite’s attitude from the initial simulation values to ±0.02 in less than 15 seconds. This result shows that the control system manages the system transient response in east/west station keeping successfully. • The figure 7 shows the satellite’s attitude steady state during 1 hour of simulation. In this case the satellite’s attitude is kept within the required values gave to AOCS for station keeping mode along the simulation. • The last figure for EWSK exposes the evolution of the eastward velocity change during the east/west station keeping maneuver. The velocity change required was 0.1 ⁄ and, from the figure 8, this value is reached at 8.33 seconds after the maneuver beginning. • Also, in another graphics is easy to check that using these results, all the requirements for the T-SAT were accomplished satisfactory. • The figure 9 shows the satellite’s attitude transient response in north/south station keeping. The angles pitch and yaw are brought within the range o ±0.01 in less than 15 seconds but the roll angle has a slower response while the north/south station keeping maneuver is being performed. This is because the thrusters used to correct the roll error also generate the maneuver northward force. Once the maneuver is finished, in this case at 129.4 seconds, the roll angle responds quickly.

516

M. Salomón and A. Kendrick

• The satellite’s attitude steady state is kept within the range ±0.01 as it can be seen in the figure 10 • The northward velocity change during the north/south station keeping maneuver is shown in the figure 11. The velocity change required was 1.78 ⁄ . This value is reached at 129.4 seconds after the maneuver beginning. o

From the simulation results is remarkable that under the worst initial conditions, the attitude error of station-keeping mode is inside the required range. Therefore the angles roll, pitch and yaw are within required values, which means that the controllers performance fulfill the requirements for the AOCS.

References 1. Marcel, J.: Sidi, Spacecraft Dynamics and Control (Practical Engineering Approach). Cambridge University Press, Holbrook (2000) 2. Hoecherl, F., Wilke, M., Quirmbach, O., Surauer, M., Igenbergs, E.: A Toolset for model based AOCS-Design. Institute of Astronautics Technical University of Munich, Germany 3. Chobotov, V.: Spacecraft Attitude Dynamics and Control. Krieger Publishing Company, Malabar (1991) 4. Wertz, J.R.: Spacecraft Attitude Determination and Control. D. Reidel Publishing Company, Dordrecht (1980) 5. Hughes, P.C.: Spacecraft Attitude Dynamics. Dover Publications, Inc., Mineola (2004) 6. Schlutz, J.: Attitude and Orbit Control of Small Satellite Using Thruster System. University of Sydney, Australia (2005)

Nonlinear System Identification Based on Recurrent Wavelet Neural Network Fengyao Zhao, Liangming Hu, and Zongkun Li*

Abstract. Based on Elman network, the recurrent wavelet neural network (RWNN) is presented, and the extended kalman filter of RWNN is given in this paper. The recurrent wavelet neural network (RWNN) can be used in the nonlinear system identification successfully. Practical example shows that RWNN has a faster convergence as well as a better precision in calculation, and a good result on the nonlinear system identification is got, which means it has a broad prospect on application. Keywords: Elman network, Recurrent wavelet neural network(RWNN), Extended kalman filter(EKF), Nonlinear dynamical system, Identification.

1 Introduction Along with the development of system identification and control technology, the application of the recurrent neural network (RNN) is more and more popular. RNN has the feedforward and feedback connections contrasted with feedforward network, which provides it with nonlinear mapping capacity and dynamical characteristics, so it can be used to simulate the dynamical system and solve dynamic problems. One typical type of RNN is Elman network [1], In which past output values of hidden neurons feed back into themselves. With memory neurons of context layer, recurrent neural networks have the ability of identification on dynamical system. In function approximation, neural networks combine with wavelets have shown good results. Recently, wavelet neural networks have also been used in identification on dynamical system [2-4]. In this research, we integrate Elman network and wavelet neural network to construct a novel recurrent wavelet neural network (RWNN) to identify nonlinear dynamical system. At the same time, the paper offers an effective extended kalman filter algorithm to train it act as an independent system simulator. Fengyao Zhao . Liangming Hu . Zongkun Li School of Water Conservancy and Environment, Zhengzhou University, Zhengzhou 450002, China [email protected], [email protected], [email protected] *

H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 517–525. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

518

F. Zhao, L. Hu, and Z. Li

2 Recurrent Wavelet Neural Network (RWNN) Fig.1 depicts the architecture of recurrent wavelet neural network (RWNN) with three layers of neurons. The first layer consists of two different groups of neurons. They are the groups of external input neurons and internal input neurons also called context units. The second layer is hidden layer, and the third layer is output layer. The inputs to the context units are the outputs of the hidden neurons. The outputs of the context units and the external input neurons are fed to the hidden neurons. Context units are also known as memory units as they store the previous output of the hidden neurons. This recurrent memory gives the network dynamical properties. In Elman network a commonly used activation function is the sigmoid. The architecture of RWNN is similar to that of Elman network, but in its hidden layer the activation function is wavelet function. In every neuron of RWNN hidden layer, there are two accessional parameters ai and bi, which represent the dilation factor and the translation factor of the wavelet activation function separately. In Fig.1, the RWNN has n+1 input neurons and one output neuron, and it has m hidden neurons and the same number of context neurons. Let x(t) Rn denote the network exterior input vectors and y(t) R denote the network output vector at the discrete time t correspondingly, which offer the training vectors. Hi(t) Rm denotes the output of the hidden layer and xc(t) Rm denotes the output of the context layer. Note that x0(t) is not the real exterior input of the network. x0(t)=1, and wi 0 = θ1i is the bias of hidden neuron i.

∈

∈

∈

∈

Let feedback gain coefficient be α , which means:

xc ,i (t ) = α H i (t − 1) .

(1)

Fig. 1 The architecture of RWNN

yt

( )

output layer W (t) Hm(t)

H1(t)

hidden layer

vt

W (t)

( )

xc,1 t

( )

context layer

xc, t

input layer

m( )

x t-1 0(

)

x t-1 n(

)

Nonlinear System Identification Based on Recurrent Wavelet Neural Network

519

The following relations describing the RWNN can be obtained: m ⎧ 1 ⎪ y (t ) = ∑ Wi (t ) H i (t ) i =1 ⎪ ⎪⎪ hi (t ) − bi (t ) ) ⎨ H i (t ) = ϕ ( ai (t ) ⎪ n m ⎪ ⎪hi (t ) = ∑ Wij2 (t ) x j (t ) + α ∑ vik (t ) H k (t − 1) + θ1i (t ) ⎪⎩ j =1 k =1

(2)

In which Wi1 (t ) is the weight connecting hidden and output layers, Wij2 (t ) is the weight connecting input and hidden layers, and vik (t ) is the weight connecting

，

context and hidden layers. ϕ (•) is a wavelet function and we choose Morlet wavelet function in this paper. ai(t) is the dilation factor and bi(t) is the translation factor of the wavelet function. Let t ' = hi (t ) − bi (t ) the Morlet wavelet basis ai (t ) function is:

，

ϕ (t ' ) = cos(1.75t ' )e −t '

2

/2

.

(3)

3 Application of the Extended Kalman Filter to RWNN The use of extended kalman filter (EKF) for training recurrent networks was first explored by Matthews[5], and Williams provides a detailed analytical treatment of EKF training of recurrent networks[6,7]. It is well known that the EKF neural network training algorithm is superior to the standard back-propagation algorithm, and the EKF has been a typical and effective method to train recurrent neural networks [8]. The behavior of RWNN can be described by following nonlinear discrete-time system:

θ (t + 1) = θ (t ) + ω (t ) , y (t ) = h(θ (t )) + v (t ) In the training RWNN, the state vector θ(t) is

θ (t ) = [W (t )T , v(t )T , a (t )T , b(t )T ]T .

(4)

Let y(t) denote the network actual output of the RWNN and yd(t) denote the desired output at time step t. Define ξ (t ) to be the network error at time t

ξ (t ) = y d (t ) − y (t ) .

(5)

Assume that the network starts to operate at time step 1 and operates up to final time step N. Define the total squared error over each of iterations by

520

F. Zhao, L. Hu, and Z. Li N N 1 1 E = ∑ ( yd (t ) − y (t )) 2 =∑ (ξ (t ))2 . 2 t =1 t =1 2

(6)

The objective of learning algorithms is to minimize the total squared error E by adjusting the weight of the network. At the tth time step, the input signals and recurrent node outputs are propagated through the network, and the functions y(t) are computed, then error vector ξ (t ) can be calculated. The dynamic derivatives of each component of matrix H(t) are formed with respect to all trainable weight parameters of the network: T

H (t ) =

∂y (t ) ⎛ ∂y (t ) T ∂y (t ) T ∂y (t ) T ∂y (t ) T ⎞ = ⎜( ) ,( ) ,( ) ,( ) ⎟ ∂θ (t ) ⎝ ∂W (t ) ∂v(t ) ∂a(t ) ∂b(t ) ⎠

(7)

Then θˆ(t ) and P(t) are updated by the following global EKF recursion: A(t ) = [(η (t ) S (t ))−1 + H (t )T P (t ) H (t )]−1

(8)

K (t ) = P(t ) H (t ) A(t ) .

(9)

θˆ(t + 1) = θˆ(t ) + K (t )ξ (t )

(10)

P(t + 1) = P(t ) − K (t ) H (t )T P(t ) + Q(t )

(11)

Where η (t ) is a scalar leaming rate parameter which in conjunction with the weighting matrix S (t ) , establishes the learning rate. The matrices A(t), for which a matrix of network output size must be inverted. K(t) is known as Kalman gain, computed at each time step, and is used to update the weight vector and error covariance matrix. Finally, Q(t ) is a diagonal covariance matrix that provides a mechanism by which the effects of artificial process noise are included in the Kalman recursion. The presence of this matrix helps to avoid numerical divergence of the algorithm, and also helps the algorithm avoid poor local minima. In RWNN, each component of matrix H(t) are calculated by following formulas:

∂y (t ) = H i (t ) . ∂Wi1 (t )

(12)

∂H i (t ) . ∂y(t ) = Wi1 (t ) 2 ∂Wij (t ) ∂Wij2 (t )

(13)

∂y (t ) ∂H (t ) = Wi1 (t ) i . ∂vik (t ) ∂vik (t )

(14)

∂H (t ) ∂y (t ) = Wi1 (t ) i . ∂ai (t ) ∂ai (t )

(15)

Nonlinear System Identification Based on Recurrent Wavelet Neural Network

～ ，

∂H (t ) ∂y(t ) = Wi1 (t ) i . ∂bi (t ) ∂bi (t )

521

(16)

In equation (13) (16) the calculating of every partial derivative at each time t depends on the partial derivative at previous time (t-1), m ∂H i (t ) ∂H i (t ) ⎡ ∂H k (t − 1) ⎤ . = ⎢ x j (t ) + α ∑ vik (t ) ⎥ 2 Wij2 (t − 1) ⎦⎥ ∂Wij (t ) ∂hi (t ) ⎣⎢ ∂ k =1

(17)

m ∂H i (t ) ∂H i (t ) ⎡ ∂H k (t − 1) ⎤ . = ⎢α H k (t − 1) + α ∑ vik (t ) ⎥ ∂vik (t ) ∂hi (t ) ⎣ ∂vik (t − 1) ⎦ k =1

(18)

m ⎡ ∂ (t ') ∂H i (t ) ∂H (t − 1) ⎤ . = ϕ '(t ') ⎢ + α ∑ vik (t ) k ⎥ ∂ai (t ) ∂ai (t − 1) ⎦ k =1 ⎣ ∂ai (t )

(19)

m ⎡ ∂ (t ') ∂H i (t ) ∂H (t − 1) ⎤ . = ϕ '(t ') ⎢ + α ∑ vik (t ) k ⎥ b ( t ) ∂bi (t ) ∂ ∂bi (t − 1) ⎦ k =1 ⎣ i

(20)

，from equation (2) we can get

In equation (19) and (20)

ϕ ' (t ' ) = −1.75 sin(1.75t ' )e − t '

2

/2

− t ' cos(1.75t ' )e − t '

2

/2

.

(21)

、

h (t ) − bi (t ) . ∂ (t ') can easily get from and the partial derivative ∂ (t ') t'= i ai (t ) ∂ai (t ) ∂bi (t ) At the beginning of the algorithm, the initialization of xc,i(0) and partial derivatives can be set to 0, i.e., x c , i (t ) = 0 , ∂H i (t ) = 0 , ∂H i (t ) = 0 , ∂H i (t ) = 0 , ∂H i (t ) = 0 , ∀i, j, k, when t = 0. ∂bi (t ) ∂vik (t ) ∂ai (t ) ∂Wij2 (t )

Then, we can calculate partial derivatives in matrix H(t) according to the sequence t = 1, 2,..., N . In general, the total error E is diminishing with the increase of calculating iterations in the algorithm.

4 Simulation and Experimental Results In this section, simulation and experiments are conducted to illustrate the validity of the proposed RWNN. The following one-input-one-output nonlinear process model coming from reference [9] is used in the simulation:

⎧ y (t + 1) = f [ y (t )] + g[u (t )] + ω y (t + 1) ⎪ 2 ⎨ f [ y (t )] = 3[ y (t )] /(6.2 + 2 y (t ) + y (t )) ⎪ g[u (t )] = u 2 (t ) + u 3 (t ) ⎩

(22)

Where u(t) is the manipulated input variable and y(t) is the output of the system.ωy(t) is the noise of y(t) in the simulation. This is a strongly nonlinear

522

F. Zhao, L. Hu, and Z. Li

process, as the dynamic relation between y(t) and u(t) changes by both nonlinear functions f [•] and g[•] and it is a dynamic system where real time implementation is critical. Assume f [•] and g [•] are both unknown, a RWNN is built like Fig.1 as the system identifier with the number of the input, hidden, context and output layer neurons being 2, 6, 6 and 1, respectively. The network is first trained by u(t) and y(t), then forecasts ~y (t ) according to another testing variable u~(t ) , through which tests the RWNN and its algorithm’s effectiveness on the identification of nonlinear dynamical system. In the learning phase u(t) is defined as the following equation:

，

u (t ) = 1.0 + 0.6 sin(2kπ / 50) + 0.4 sin(2kπ / 75) .

(23)

u(t) is shown in Fig.2, and the desired output can get from equations (22). The noise is not considered in the simulation. The number of time step is N = 200 , and the learning result of RWNN is shown in Fig.3. The training of the network adopts the previous EKF algorithm. Utilizing equations (8) (11) a value less than 0.001 of total squared error E can be achieved about 200 training iterations. For the sake of analyzing the efficiency of RWNN, the learning result of Elman network is also presented in Fig.3 for comparison. In the research Elman network has the same number of the input, hidden, context and output layer neurons as RWNN, and the Elman network is also trained 200 iterations by EKF algorithm. It can be seen from Fig.3 that the learning result of RWNN is much better than that of Elman network.

～ ，

Fig. 2 Training input curve respect to time

Fig. 3 The results of neural network learning phase

Nonlinear System Identification Based on Recurrent Wavelet Neural Network

523

Fig. 4 Error curves of RWNN and Elman network

Fig. 4 depicts the total error curve of RWNN and Elman network with respect to the learning iterations. At the early 50 iterations, the error curve of RWNN descends faster, which means RWNN convergence more quickly. At the latter 150 iterations, the two curves descend both slowly, which is a general result existing in the training algorithm, but the error value of RWNN is obviously smaller than that of Elman network, which means RWNN have higher convergence precision. After training the network, input the testing function u~ (t ) as shown in Fig.5, and the testing output ~ y (t ) of network is illustrated in Fig.6. It can be seen that the RWNN identifying result tracks desired signal quite well, indicating that the

Fig. 5 Testing input curve respect to time

Fig. 6 The results of identifier during the testing phase

524

F. Zhao, L. Hu, and Z. Li

proposed RWNN algorithm has a good ability of identifying nonlinear dynamical system. The testing result of Elman network is also shown in Fig.6 and it is clearly not so good as RWNN. It can be seen from the simulation results that RWNN has the advantage over Elman network in terms of convergence speed and identification precision. RWNN performs rather well in identifying nonlinear dynamical system, which is valuable for practice application.

5 Discussion (i) Wavelet function has a favorable localizing characteristic. For some localizable regions of the inputting function space, only a few weights of the RWNN need to be regulated, so it can easily acquire the localizable character of dynamic system. Usually RWNN gains much better convergence precision with fewer neurons and with fewer neurons training iterations [3]. (ii) There are many wavelets with good property in the wavelet function family, such as Mayer wavelet, Morlet wavelet, Gauss wavelet, etc. Different wavelet can be chosen as the basis function of RWNN according to different problems. In the EKF learning algorithm, only the derivative ϕ ' (t ' ) should be modified with respect to different wavelet basic function. (iii) In this paper RWNN is based on multi-input-one-output structure, it can easily extend to the multi-input-multi-output structure.

6 Conclusions In this research, a new dynamic neural network RWNN integrating Elman network and the wavelet neural network is proposed, and the extended kalman filter(EKF) algorithm of training it is also given. Simulate performance of the RWNN on the identification of nonlinear dynamical system is excellent. The RWNN reveals an extremely fast convergence and a relative high accuracy in the process of identifying nonlinear dynamical system. It is believed that the proposed RWNN is a promising identifying strategy for nonlinear dynamical system. Acknowledgements. The authors would acknowledge financial support from National Science Foundation of China(50279003).

References 1. Elman, J.L.: Finding structure in Time. Cognitive Science 14, 179–211 (1990) 2. Oussar, Y., Rivals, I., Personnaz, L., et al.: Training Wavelet Networks for Nonlinear Dynamic Input-output Modeling. Neurocomputing 20, 173–188 (1998) 3. Liang, F., Tan, Y.: Using Wavelet Neural Network in Non-linear System Identification. Journal of Guilin Institute of Electronic Techology 20(1), 18–22 (2000) (in Chinese) 4. Srivastava, S., Singh, M., Hanmandlu, M., Jha, A.N.: New Fuzzy Wavelet Neural Networks for System Identifiation and Control. Applied Soft Computing 6, 1–17 (2005)

Nonlinear System Identification Based on Recurrent Wavelet Neural Network

525

5. Matthews, M.B.: Neural Network Nonlinear Adaptive Filtering Using- the Extended Kalman Filter Algorithm. In: Proceedings of the International Neural networks Conference, Paris, vol. I, pp. 115–119 (1990) 6. Williams, R.J.: Some Observations on the Use of the Extended Kalman Filter as a Recurrent Network Learning Algorithm. Technical Report NU-CCS-92-1. Northeastern University, College of Computer Science, Boston (1992) 7. Williams, R.J.: Training Recurrent Networks Using the Extended Kalman Filter. In: International Joint Conference on Neural Networks, VBaltimore, vol. IV, pp. 241–246 (1992) 8. Puskorius, G.V., Feldkamp, L.A.: Neurocontrol of Nonlinear Dynamical Systems with Kalman Filter Trained Recurrent Networks. IEEE Trans. on Neural Networks 5, 279– 297 (1994) 9. Yuan, X., Liu, S.: Study on Neural Network Simulation of Dynamic System. Water Resources and Hydropower Engineering 3, 38–42 (1998)

Approximation to Nonlinear Discrete-Time Systems by Recurrent Neural Networks Fengjun Li

Abstract. Neural networks are widely used to approximate nonlinear functions. In order to study its approximation capability, a approximating approach for nonlinear discrete-time systems is presented by using the concept of the time-variant recurrent neural networks (RNNs) and the theory of twodimensional systems. Both theory and simulations results show that the derived mathematical model of RNNs can approximate the nonlinear dynamical systems to any degree of accuracy. Keywords: Approximation, nonlinear discrete-time system, recurrent neural networks.

1 Introduction There are two types of connections in neural networks. Neural networks with only feedforward connections are called feedforward networks, and neural networks with arbitrary connections are often called recurrent networks. Many of the applications of neural networks, particulary in the areas of nonlinear system identification and control, reduce to the problem of approximating unknow functions of one or more variables from discrete measures [1-6]. A number of authors have established that multilayer feedforward neural networks, with a variety of activation functions, serve as universal approximators [7-9]. For example, in 1989 Hornik, Stinchcombe, and White[7] could show that any Borel-measurable function on a compact domain can be approximated by a three-layered feedforward network, i.e., a feedforward network with one hidden layer, with an arbitrary accuracy. In the same year, Cybenko [8] and Fengjun Li School of Mathematics and Computer Science Ningxia University, Yinchuan 750021, China [email protected] H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 527–534. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

528

F. Li

Funahashi[9] found similar results, each with different methods. Whereas the proof of Hornik, Stinchcombe, and White [7] is base on the Stone-Weierstrass theorem, Cybenko [8] makes in principle use of Hahn-Banach theorem. Funahashi [9] mainly applies the Irie-Miyake and the Kolmogov-Arnold-Sprecher theorem. More generally, continuous nonlinear functionals can be approximated with feedforward neural networks, and this can be used to directly approximate the output of dynamical systems [10]. The nonlinear dynamical behavior of recurrent networks is suitable for spatio-temporal information processing. Theoretical studies on recurrent networks have been mainly concerned with the stability of convergence of the trajectory to the equilibria [11]. Recurrent neural networks (RNNs) is one type of artificial neural networks with very successful applications [11-14]. Some work has already been done on the capability of RNNs to approximate measurable functions, e.g. [15]. This paper focuses on dynamical systems and prove that those can be approximated by RNNs in state space model form with an arbitrary accuracy. The remainder of this article is organized as follows. A shot introduction on dynamical systems and RNNs in state space model form is introduced in Section 2. The approximation idea and algorithm are given in Section 3. Using the previously obtained theoretical results, some numerical results are given in Section 4. Finally, in Section 5 we summarize our current research. Throughout the paper we will use boldface characters to stand for vectors, and regular characters for scalars.

2

Dynamical Systems and RNNs

A time-variant discrete dynamical system which can be described as a equation s(t + 1) = f (s(t), u(t), t), s ∈ Rn , u ∈ Rm , t ∈ R,

(1)

where s(t) ∈ RL and u(t) ∈ Rm are the state vectors and input vectors of neurons, respectively. (1) is the state transition that is a mapping from the present internal hidden state of the system s(t) and the influence of external inputs u(t) to the new state s(t + 1). The system can be viewed as a partially observable autoregressive dynamic state transition s(t) → s(t + 1) that is also driven by external forces u(t). Without the external inputs u(t), the system is called an autonomous system [15]. However, most real world systems are driven by a superposition of an autonomous development and external influences. If we assume that the state transition does not depend on s(t), we are back in the framework of feedforward neural networks[16]. However, the inclusion of the internal hidden dynamics makes the modeling task much harder, because allows varying inter-temporal dependencies. Theoretically, in the

Approximation to Nonlinear Discrete-Time Systems

529

current framework an event s(t+1) is explained by a superposition of external inputs u(t), u(t − 1), · · · from all the previous time steps. In this paper, we propose to map nonlinear time-variant discrete dynamical systems (1) by a type of recurrent neural networks in state space model form   sL (t + 1) = F A(t)sL (t) + B(t)u(t) , sL ∈ RL , u ∈ Rm , (2) where A(t) and B(t) are weight matrices of appropriate dimensions at time t,  T F (x) = f (x1 ), f (x2 ), · · · , f (xL ) (0 < F˙ (·) ≤ 1) is the so called activation function of the network which has continuous derivation. A major advantage of RNNs written in the form of of a state space model (2) is the explicit correspondence between equations and architecture.

3

Approximation Idea

For nonlinear time-variant discrete dynamical systems, the common case, we prove that their finite time trajectory can be approximated by the state vector of the output units of RNNs to any degree. Our approximation idea is that at every time in the definition of the nonlinear time-variant discrete dynamical systems, using time-variant RNNs (2) to approximate them. Therefore, we must confirm the weight matrices A(t) and B(t) at every time t. We will convert this approximation problem to the train problem of RNNs (2). For seasonable, the train algorithm must be simple, efficient and convergence quickly. We find in the process of train weight matrices in RNNs (2), each variable depends on two other variables—time t and iterated number k. Therefore, using the denotation way of two-dimensional system, (2) can be rewritten as   sL (t + 1, k) = F A(t, k)sL (t, k) + B(t, k)u(t) . (3) The train algorithm of weight matrices A(t) and B(t) is denoted as  A(t, k + 1) = A(t, k) + ΔA(t, k), B(t, k + 1) = B(t, k) + ΔB(t, k).

(4)

Next, we will confirm ΔA and ΔB in (4). We use the state vector of the output unites of RNNs to approximate the finite time trajectory of nonlinear time-variant discrete dynamical systems. Let e(t, k) = sL (t, k) − s(t),

(5)

sL (0, k) = sL (0) = s(0)

(6)

and A(t, 0), B(t, 0), t = 0, 1, · · ·, with random. If δ(t + 1, k) = sL (t, k + 1) − sL (t, k),

(7)

530

F. Li

then δ(1, k) = 0, and δ(t + 1, k) = G(t − 1, k)[A(t − 1, k − 1)sL (t − 1, k + 1) + B(t − 1, k + 1)u(t − 1) − A(t − 1, k)sL (t − 1, k) − B(t − 1, k)u(t − 1)] = G(t − 1, k)[A(t − 1, k − 1)G(t, k) + ΔA(t − 1, k)sL (t − 1, k + 1) + ΔB(t − 1, k)u(t − 1),

(8)

where G(t − 1, k) = diag(f˙(ξ1 ), f˙(ξ2 ), · · · , f˙(ξL )), ξ = (ξ1 , ξ2 , · · · , ξL )T is between G(t − 1, k)[A(t − 1, k − 1)sL (t − 1, k + 1) + B(t − 1, k + 1)u(t − 1) and A(t − 1, k)sL(t − 1, k) + B(t − 1, k)u(t − 1). From (2), we have e(t, k + 1) − e(t, k) = −G(t − 1, k)[A(t − 1, k)δ(t, k) + ΔA(t − 1, k) × sL (t − 1, k + 1) + ΔB(t − 1, k)u(t).

(9)

Let G(t − 1, k)[A(t − 1, k)δ(t, k) + ΔA(t − 1, k)sL (t − 1, k + 1) + ΔB(t − 1, k)u(t) = P (t, k)δ(t, k) + Q(t, k)e(t, k).

(10)

From (8), (9) and (10), we obtain the two dimensional discrete Roessor’s model [17] as follows:      δ(t, k) P (t, k) Q(t, k) δ(t + 1, k) , (11) = e(t, k) −P (t, k) I − Q(t, k) e(t, k + 1) where I is identity matrix of appropriate dimensions, P (t, k) and Q(t, k) are confirmed in the future which initial conditions are δ(1, k) = 0, k = 0, 1, · · · , and e(t, 0) = sL (t, 0) − s(t), t = 0, 1, · · · . As 0 < F˙ (·) ≤ 1, and ˙ 1 ), f˙(ξ2 ), · · · , f˙(ξL )) G(t − 1, k) = diag(f(ξ is nonsingular, so from (10), we have

Approximation to Nonlinear Discrete-Time Systems



ΔA(t − 1, k) ΔB(t − 1, k)



531

= G−1 (t − 1, k)[P (t, k)δ(t, k) + Q(t, k)e(t, k)]   "  −A(t − 1, k)δ(t, k) sL (t − 1, k + 1) u(t − 1)  ×

sL (t − 1, k + 1) u(t − 1)

 −1

  sL (t − 1, k + 1) u(t − 1) .(12)

From[17], we know that in two dimensional discrete time-variant system (11), the convergence property is only depend on Q(t, k), independent of P (t, k). Without loss of generality, we take P (t, k) = 0. Because of 0 < F˙ (·) ≤ 1, we choose Q(t, k) = G(t, k − 1). further adjust variant t, (12) can be written as   " ΔA(t − 1, k) = e(t + 1, k) − A(t, k)[sL (t, k + 1) − sL (t, k)] ΔB(t − 1, k)  −1     sL (t, k + 1) × sL (t, k + 1) u(t) u(t)   × sL (t, k + 1) u(t) " = e(t + 1, k) − A(t, k)[sL (t, k + 1) − sL (t, k)]   1 sL (t, k + 1) . (13) × 2 u(t) sL (t, k + 1) + u2 (t) With this, we know that the approximation algorithm of time-variant discrete RNNs consists of (4) and (13). At each time in the definition of the nonlinear time-variant discrete dynamical systems (1), applying (4) and (13) to perform iterated computation until error sufficiently small. Then turn to next time continue this process.

4

Numerical Results

In order to demonstrate our theoretical result, in this section, we give an example of approximation to the nonlinear time-variant discrete dynamical systems  2       0.1 0.05t 0.3 s1 (t + 1) s1 (t) u(t) (14) = + 0.2 −0.15t 0.1 s2 (t + 1) s2 (t) by time-variant discrete RNNs (3), and analyze the error bound of this approximation algorithm. Let initial conditions s1 (0) = 0, s2 (0) = 0.5 and take f (x) = 0.5x in (3). The number of iterated train algorithm (3) and (13) is fixed two at each time t. we select u(t) = 0.25 for a numerical computation to have numerical results in Figures 1 and 2. If we use

532

F. Li

Fig. 1 The trajectory of s1 (t) and its approximator

Fig. 2 The trajectory of s2 (t) and its approximator

Fig. 3 The relationship between the approximation error and iterated number

Approximation to Nonlinear Discrete-Time Systems

! E = ||si (t) − sL,i (t)||2 =

T

533

 12 (si (t) − sL,i (t)) dt , i = 1, 2 2

(15)

0

as a metric of this approximation, then We also analyze the relationship between error bound and iterated number (see Figure 3). On the basis of the numerical example, we can conclude that our numerical approximation results are in good agreement with theory.

5 Conclusion In approximation by neural networks, we have two main problems— “density” and “complexity”. In this paper, we study the density problem. We introduce nonlinear time-variant discrete dynamical systems and prove their finite time trajectory can be approximated by the state vector of the output units of RNNs to any degree. In a word, this paper obtained results can precisely characterize the approximation ability of RNNs and clarify the relationship among the rate of approximation, the number of hidden units and the properties of approximated systems. Acknowledgements. This work was supported by NSFC project under contract No. 70531030 and Ningxia university project under contract No. ZR200803.

References 1. Narendra, K.S., Parthasarathy, K.: Identification and Control of Dynamic Systerms Using Neural Networks 1, 4–27 (1990) 2. Chen, S., Billings, S.A.: Neural Networks for Nonlinear Dynamic Systerm Modeling and Identification. Int. J. Contr. 56, 319–346 (1992) 3. Funahashi, K.I., Nakamura, Y.: Approximation of Dynamical Systerms by Continuous Time Recurrent Nueral Networks. Neural Networks 6(6), 801–806 (1993) 4. Chen, T.P., Chen, H.: Approximation of Continuous Functoinal by Neural Networks with Application to Dynamic Systerms. IEEE Trans. Neural Networks 4, 910–918 (1993) 5. Liu, G.P., Kadirkamanathan, V., Billings, S.A.: Variable Neural Networks for Adaptive Control of Nonlinear Systems. IEEE Trans. Syst., Man, Cybern. C 29, 34–43 (1999) 6. Chen, T.P., Amari, S.I.: New Theorems on Global Convergence of Some Dynamical Systems. Neural Networks 14, 251–255 (2001) 7. Hornik, K., Stinchombe, M., White, H.: Multilayer Feedforward Networks Are Universal Approximator. Neural Networks 2, 359–366 (1989) 8. Cybenko, G.: Approximation by Superpositions of Sigmoidal Function. Math. of Control Signals, and System 2, 303–314 (1989)

534

F. Li

9. Funahashi, K.I.: On the Approximate Realization of Continuous Mappings by Neural Networks. Neural Networks 2, 183–192 (1989) 10. Sch¨ afer, A.M., Zimmermann, H.-G.: Recurrent Neural Networks are Universal Approximators. In: Kollias, S.D., Stafylopatis, A., Duch, W., Oja, E. (eds.) ICANN 2006. LNCS, vol. 4131, pp. 632–640. Springer, Heidelberg (2006) 11. G´egout, C.: Stable and Convergent Dynamics for Discrete-Time Recurrent Networks. Nonlinear Analysis 30(3), 1663–1668 (1997) 12. Garzon, M., Botelho, F.: Dynamical Approximation by Recrrent Neural Networks. Neurocomputing 29, 25–46 (1999) 13. Hammer, B.: On the Approximation Capability of Recrrent Neural Networks. Neurocomputing 31, 107–123 (2000) 14. Liang, J., Nikiforuk, P.N., Gupta, M.M.: Approximation of Discrete-Time State-Space Trajectories Using Dynamic Recurrent Neural Networks. IEEE Trans. Auto. Contr. 40(7), 1266–1270 (1995) 15. Zimmermann, H.G., Grothmann, R., Sch¨ afer, A.M., Tietz, C.: Identifcation and Forecasting of Large Dynamical Consistent Neural Networks. In: Haykin, S.J., Principe, T.S., McWhirter, J. (eds.) New Directons in Statistcal Signal Processing: From Systems to Brain. MIT Press, Cambridge (2006) 16. Immermann, H.G., Neuneier, R.: Neural Network Architectures for the Modeling of Dynamical Systems. In: Kolen, J.F., Kremer, S. (eds.) A Field Guide to Dynamical Recurrent Networks, pp. 311–350. IEEE Press, Los Alamitos (2001) 17. Kaczorek, T., Klamka, J.: Minimum Energy Control of 2-D Linear Systems with Variable Coefficients. Int. J. Control 44, 645–651 (1986)

Model-Free Control of Nonlinear Noise Processes Based on C-FLAN Yali Zhou, Qizhi Zhang, Xiaodong Li, and Woonseng Gan*

Abstract. In practical active noise control (ANC) systems, nonlinear active controllers may be required in cases where the actuators used in ANC systems or the structures to be controlled exhibit nonlinear characteristics. In this paper, a chebyschev functional link artificial neural network (C-FLANN) is used as a nonlinear controller. Compared with the multilayer perception (MLP) neural network, CFLANN exhibits a much simpler structure, less training computation and faster convergence. The simultaneous perturbation stochastic approximation (SPSA) algorithm instead of the usual back-propagation method is applied as a learning rule of the network to adapt to the time-varying plant. Unlike the back-propagation method, the SPSA method does not require an estimation of the secondary path. Computer simulations have been carried out to demonstrate that the proposed algorithm outperforms the standard filtered-x least mean square (FXLMS) algorithm when the ANC system exhibits nonlinear and time-varying characteristics. Keywords: ANC, Nonlinear, Model-free, C-FLANN.

1 Introduction In recent years, active noise control (ANC) has been a relevant area of interest for many researchers. Well-established methods and corresponding applications have been reported in the literature [1,2]. The principle of ANC is the destructive interference at a definite location of an incoming acoustical noise using a signal, usually generated by a loudspeaker, with equal amplitude and opposite phase. The block diagram of one of the most widely adopted schemes, i. e., the so-called single-channel feed-forward ANC scheme is shown in Fig.1. In this scheme, x ( n ) is Yali Zhou . Qizhi Zhang School of Automation, Beijing Information Science and Technology University, Beijing 100192, China

*

Xiaodong Li Institute of Acoustics, Academia Sinica, Beijing 100080, China Woonseng Gan School of EEE, Nanyang Technological University, Singapore 639798, Singapore [email protected] H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 535–541. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

536

Y. Zhou et al.

the noise sensed by a reference microphone, d p ( n ) is the signal at the location where the noise has to be attenuated, y(n) is the signal generated by the controller, d s ( n ) is the interfering signal received at the same location where an error micro-

phone collects the error signal e ( n ) = d p ( n ) + d s ( n ) . This signal is used, together with the input signal, to adapt the controller, which is an adaptive filter driven by a suitable adaptation algorithm. Moreover, in Fig.1, P(Z) is the transfer function characterizing the primary path between the noise source and the error microphone, while S(Z) is the transfer function of the secondary path between the controller output and the error microphone. The most common form of linear adaptive algorithm/architecture combination is a transversal finite impulse response (FIR) filter using the filtered-X Least Mean Square (FXLMS) algorithm [1,2]. To deal with the noise cancellation, previous knowledge or modeling of the secondary path is required. While presently linear controllers constitute a well-established solution for ANC, it has been recently pointed out that various nonlinear effects may influence the behavior of an ANC structure [3,4]. Nonlinearities may be present in the primary path when the noise is propagating with high sound pressure [5] and the nonlinearity of the air is taken into account. Nonlinearities can also occur in the secondary path due to the use of A/D and D/A converters, power amplifiers, loudspeakers and transducers. Overdriving the electronics or loudspeakers gives rise to relevant nonlinear effects [6]. To deal with these effects, different structures for nonlinear controllers have been recently proposed in the literature [3-11]. Among them, structures based on functional link artificial neural network (FLANN) have attracted much research attention. The basic principle of FLANN is to expand the dimensionality of the input signal space by using a set of independent functions [10]. Several mapping functions have been used to achieve this purpose, such as Legendre, Chebyshev, and trigonometric polynomials [11].

P(Z)

ds(n) y(n)

FM

e(n) +

C-FLANN controller

f0

Z-1

x(n-1)

S(Z) Z-1

x(n-2)

Functional Expansion

x(n)

XN x(n)

dp(n)+

f1

∑

SPSA Z-1

Fig. 1 The block diagram of an ANC system

x(n-N+1)

WM

fM-1

Fig. 2 The FLANN structure

y(n)

Model-Free Control of Nonlinear Noise Processes Based on C-FLAN

537

In this paper, a chebyschev functional link artificial neural network (CFLANN) is used as a nonlinear controller, at the same time, the simultaneous perturbation stochastic approximation (SPSA) algorithm is used to update all weights of the C-FLANN simultaneously. Unlike the back-propagation method, this method does not require an estimation of the secondary path. Compared with the multilayer perception (MLP) neural network, C-FLANN exhibits a much simpler structure, less training computation and faster convergence [6]. So it is easier to implement by hardware and improve the performance-price ratio of system.

2 Control Algorithm The structure of the FLANN controller is shown in Fig.2, at any time instant n, defining the input signal vector X ( n ) as X (n) = [ x(n), x(n − 1),

x(n − N + 1)]T .

(1)

Where N is the length of the input vector X(n), [.]T denotes transpose of a vector. Then the functionally expanded vector F(n) is F (n) = [ f 0(n ), f 1( n),

fM − 1(n)]T = [ FE ( x( n)), FE ( x ( n − 1)),

FE ( x ( n − N + 1))]T . (2)

Where FE(.) stands for functional expansion, M is the length of the functional enhanced vector F ( n ) . These M independent functions map the N-dimensional signal space into an Mdimensional space, that is RN → RM N < M . W ( n ) is the weight vector and is given by W (n) = [ w0( n), w1( n),

wM − 1( n)]T .

(3)

It is common knowledge that if the secondary path of the ANC system is completely unknown, it is impossible to use usual gradient method as a learning rule to update the controller coefficients [12]. In this case, an estimator of the gradient of the error function is needed. The SPSA which was introduced by J. C. Spall [13] is a well-known gradient approximation approach that relies on measurements of the objective function, not on measurements of the gradient of the objective function. The objective of the following analysis is to develop the C-FLANN-based SPSA algorithm to improve the noise cancellation capability of a nonlinear ANC system. Step 1: Define the error function Note that in ANC systems, each sampling error signal does not contain enough information as an evaluation function to be optimized. That is, the expectation of the error signal has to be used as the evaluation function. For practicality, the sum of the error signal for a certain interval is used to approximate the expectation of the error signal. Thus, the error function is defined as [14].

J ( y (n)) =

1 λ 2 1 λ e (n) = ∑ [ds (n) + dp (n)]2 . ∑ 2 t =1 2 t =1

(4)

538

Y. Zhou et al.

where t is the sampling number in a block interval, and number of one block interval.

λis the total sampling

Step 2: Functional expansion of the input pattern vector X ( n )

For functional expansion of the input pattern, we have chosen the Chebyshev expansion. Then the functionally expanded vector F ( n ) can be written as F(n) =[ f 0(n), f 1(n), fM −1(n)]T =[T0(n),T1(n),T2(n), TP −1(n),T0(n−1),T1(n−1),T2(n−1), TP −1(n −1)......,T0(n− N +1),T1(n − N +1),T2(n− N +1), TP −1(n− N +1)]T =[1, x(n),2x(n)2 −1, 2x(n)TP − 2(n) −TP − 3(n),1, x(n−1),2x(n −1)2 −1, 2x(n−1)TP − 2(n−1) −TP − 3(n −1) 1, x(n− N +1),2x(n− N +1)2 −1, 2x(n− N +1)T (n− N +1) −T (n − N +1)]T P− 2

(5)

P−3

Where P is the order of functional expansion, then M=N*P. Step 3: Compute the anti-noise signal ds(n) The output of the C-FLANN is M −1

y (n) = W (n)F (n) = ∑ [ wj (n) fj (n)] . T

j =0

(6)

Then the anti-noise signal ds(n) can be calculated using the following equation L −1

T

ds (n) = S (n) Y (n) = ∑ sj (n) y (n − j ) .

(7)

j =0

Where S ( n ) = ⎡⎣ s0 ( n ) s1 ( n )… sL−1 ( n )⎤⎦ , is the impulse response of the secondary path transfer function S(Z), L is the length of the secondary T path. Y ( n ) = ⎡⎣ y ( n ) y ( n − 1)… y ( n − L + 1)⎤⎦ . T

Step 4: Generation of SP vector The following perturbation vector Δ (n) is generated as independent Bernoulli random variables with outcomes of ±1 that gives small disturbances to all weights

Δ (n) = (Δ 0(n), Δ1(n),

ΔM − 1(n))T .

(8)

Step 5: Error function evaluations Obtain two measurements of the error function J (i) based on the SP:

J ( y (W (n) + ck Δ (n))) and J ( y (W (n) − ck Δ (n))) . Where ck is a positive scalar and represents a magnitude of the perturbation. Step 6: Gradient approximation

Generate the SP approximation to the unknown gradient

ΔW (n) =

∂J ( y (W (n))) as ∂W (n)

J ( y (W (n) + ck Δ (n))) − J ( y (W (n) − ck Δ (n))) Δ ( n) . ck

(9)

Model-Free Control of Nonlinear Noise Processes Based on C-FLAN

539

Step 7: Update the weight vector W(n) of the C-FLANN Weights of the C-FLANN are updated in the following manner:

W (n + 1) = W (n) − ak ΔW (n) .

(10)

Where ak is a positive learning coefficient. From (9) and (10), it can be seen that the weights of the C-FLANN controller is updated without the need to model the secondary path, so this algorithm is called mode-free(MF) control algorithm.

3 Simulation Studies To demonstrate the effectiveness of the proposed algorithm, computer simulations are performed on a nonlinear ANC system. And at the same time, a comparison between the proposed algorithm and the FXLMS algorithm is made. The noise to be attenuated is a sinusoidal signal at 300 Hz sampled at 3000 samples/s. For the C-FLANN-based SPSA algorithm the length of the input vector X(n) is 13(N=13),the chebyschev functional expansion of the input signal is of fourthorder type(P=4), then M=N*P=52. The total sampling number of one block interval is set as 100. ck and ak are set as 0.001 and 0.00007, respectively. For the FXLMS algorithm, a 13-tap finite impulse response (FIR) filter is used, the convergence coefficient for the FXLMS algorithm is set at μ=0.001. Case 1: In this case, the primary path is characterized by a nonlinear inputoutput relationship, which consists of a linear transfer function

λ

P '( Z ) = 0.8Z − 6 + 0.6 Z − 7 − 0.2Z − 8 − 0.5Z − 9 − 0.1Z − 10 + 0.4Z − 11 − 0.05Z − 12 . (11) followed by the nonlinear combination of its outputs

dp (n) = u (n − 2) + 0.6u 2(n − 2) .

(12)

where H −1

u (n) = ( P'(n)) X (n) = ∑ [ p ' j (n) xj (n)] . T

j =0

(13)

where P'(n) = [ p ' 0(n), p ' 1(n), p ' H − 1(n)]T is the impulse response of the linear transfer function P' ( Z ) of the primary-path. The secondary path is characterized by the non-minimum phase transfer function and is assumed to be time-invariant S ( Z ) = 0.3Z − 2 + 0.6Z − 3 + 0.1 Z − 4 − 0.4 Z − 5 − 0.1Z − 6 + 0.2Z − 7 + 0.1Z − 8 + 0.01Z − 9 + 0.001Z − 10

. (14)

The simulation results of the canceling error in the frequency domain are shown in Fig.3. From the simulation results shown in Fig.3, it can be seen that when the secondary path model is deterministic and time-invariant, the proposed algorithm and the FXLMS algorithm can reduce the 300Hz sinusoidal signal effectively. But

540

Fig. 3 The error signal spectrum for case 1

Y. Zhou et al.

Fig. 4 The error signal versus number of iterations for case 2

only the C-FLANN-based SPSA algorithm is effective on the harmonic noise signal caused by nonlinearity. Case 2: Next, we deal with a tracking problem. With the same set of parameters used for case 1, when the number of iterations reaches 25,000, the secondary path is altered by letting S ( z ) = − S ( z ) . The adaptation continues for 50000 iterations. The error signal in error microphone versus the number of iterations is shown in Fig.4. From the simulation results shown in Fig.4, it can be seen that when the secondary path model is time-varying, the C-FLANN-based SPSA algorithm has a good tracking ability of the secondary path. After a short transient phase, the system settles down to a steady-state response. Whereas the FXLMS algorithm cannot adapt itself to the change of the secondary path because of its model-based characteristics [1]. From the simulation results shown in Fig.4, it is concluded that the SPSAbased MF controller can eliminate the need of the modeling of the secondary path for the ANC system. So, such an approach has potential advantages in accommodating systems where the equations governing the system are unknown or with time-varying dynamics [15].

4 Conclusions In this paper, the SPSA algorithm is applied to the nonlinear controller equipped with C-FLANN. This approach optimizes error function without using derivative of the error function. Therefore, the presented ANC algorithm does not require any estimation of the secondary path. Computer simulations have been carried out to assess the performance of the proposed algorithm as a candidate for nonlinear ANC. Its performance, in terms of error power spectrum, has been compared to that of the standard FXLMS. It is shown that for the nonlinear control, in terms of the error power spectrum, the proposed algorithm outperforms the standard FXLMS algorithm when the ANC system exhibits nonlinear and time-varying characteristics. This observation implies that the SPSA-based C-FLANN algo-

Model-Free Control of Nonlinear Noise Processes Based on C-FLAN

541

rithm can eliminate the need of the modeling of the secondary path for the ANC system. Acknowledgments. This research is supported by Training Funds for Elitist of Beijing (20061D0500600164, 20051A0500603) and Funding Project for Academic Human Resources Development in Institutions of Higher Learning under the Jurisdiction of Beijing Municipality (PXM2008_014215_055942).

References 1. Kuo, S.M., Morgan, D.R.: Active Noise Control Systems—Algorithms and DSP Implementations. Wiley, New York (1996) 2. Nelson, P.A., Elliott, S.J.: Active Sound Control. Academic Press, London (1991) 3. Zhou, Y.L., Zhang, Q.Z., Li, X.D., Gan, W.S.: Analysis and DSP Implementation of an ANC System Using a Filtered-Error Neural Network. Journal of Sound and Vibration 285, 1–25 (2005) 4. Strauch, P., Mulgrew, B.: Active Control of Nonlinear Noise Processes in a Linear Duct. IEEE Trans. Signal Processing 46, 2404–2412 (1998) 5. Klippel, W.: Active Attenuation of Nonlinear Sound. Patent, US (1999) 6. Patra, J.C., Kot, A.C.: Nonlinear Dynamic System Identification Using Chebyshev Functional Link Artificial Neural Networks. IEEE Trans. Syst., Man, Cybernetics 32, 505–511 (2002) 7. Tan, L., Jiang, J.: Adaptive Volterra Filter for Active Control of Nonlinear Noise processes. IEEE Trans. Signal Processing 49, 1667–1676 (2001) 8. Tokhi, M.O., Wood, R.: Active Noise Control Using Radial Basis Function Networks. Contr. Eng. Practice 5, 1311–1322 (1997) 9. Snyder, S.D., Tanaka, N.: Active Control of Vibration Using a Neural Network. IEEE Trans. on Neural Networks 6, 819–828 (1995) 10. Zhang, J.W., Wang, K.Q., Yue, Q.: Data Fusion Algorithm based on Functional Link Arti-ficial Neural Networks. In: The 6th World Congress on Intelligent Control and Automation, China, pp. 2806–2810 (2006) 11. Weng, W.D., Yen, C.T.: Reduced-Decision Feedback FLANN Nonlinear Channel Equaliser for Digital Communication Systems. IEE Proc. Commu. 151, 305–311 (2004) 12. Maeda, Y., De Figueiredo, R.J.P.: Learning Rules for Neuro- Controller via Simultaneous Perturbation. IEEE Transactions on Neural Networks 8, 1119–1130 (1997) 13. Spall, J.C.: Multivariate Stochastic Approximation Using Simultaneous Perturbation Gradient Approximation. IEEE Transactions on Automatic Control 37, 332–341 (1992) 14. Maeda, Y., Yoshida, T.: An Active Noise Control without Estimation of Secondary Path. In: ACTIVE 1999, USA, pp. 985–994 (1999) 15. Spall, J.C., Cristin, J.A.: Model-free Control of Nonlinear Stochastic Systems with Discrete-time Measurement. IEEE Transactions on Automatic Control 43, 1198–1210 (1998)

An Empirical Study of the Artificial Neural Network for Currency Exchange Rate Time Series Prediction Pin-Chang Chen, Chih-Yao Lo, and Hung-Teng Chang*

Abstract. This paper applies an integrated artificial neural network approach to forecast foreign exchange rates between the US dollar and Chinese Renminbi. In order to obtain a better forecasting performance in foreign exchange rates, this study develops an integrated forecasting model, which applies the SPSS as data preprocess method and an artificial neural network application named Alyuda Neuro Intelligence as forecasting tool. The results of this study provide evidence on the effectiveness and efficiency of the integrated artificial neural network model. The findings of this study should contribute positively to the development of theory, methodology, and practice of using artificial neural network to develop a forecasting model with enhanced forecasting accuracy. Keywords: Artificial Intelligence, Neural Network, Forecasting, Exchange Rates.

1 Introduction Artificial neural network is one of the most famous approaches in forecasting financial time series. Artificial neural network is a universal and highly flexible function approximator for pattern recognition and classification. It requires no prior assumption on the behavior and function form of the related variables, but it still can capture the underlying dynamic and nonlinear relationships among variables. Several design factors significantly impact the forecasting accuracy of artificial neural network. These factors include selection of input variables, architecture of the neural network, and quantity of input data. The question of system architecture design has been widely researched, but the corresponding questions of input variables selection and how many input data is needed in producing a reliable forecasting model have not been adequately addressed. Artificial neural network belongs to the class of data driven approach, as opposed to model driven approach. The representation of data is a critical for a successful neural network Pin-Chang Chen . Chih-Yao Lo . Hung-Teng Chang Department of Information Management, Yu-Da College of Business, Miaoli County, Taiwan 361, R.O.C. {chenpc,jacklo,cht}@ydu.edu.tw

*

H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 543–549. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

544

P.-C. Chen, C.-Y. Lo, and H.-T. Chang

design. However, the movements of foreign exchange rates are generally nonstationary and quite random in nature. For most artificial neural networks, the price input is not a desirable set and it makes forecasting difficult. To overcome this problem, some types of data preprocess and data transformation are required to make the raw data become stationary and statistical property. In order to obtain a better forecasting performance in foreign exchange rates, the researcher of this study is motivated to develop an improved forecasting model. This study develops an integrated forecasting model, which applies the SPSS as data preprocess tool and an artificial neural network application named Alyuda Neuro Intelligence as forecasting model. This forecasting model is intended to take advantage of the SPSS, which can be used to select the correlated data and transform the original disorderly raw data into a dimensionless series in order to obtain an appropriate fundamental for the accurate mathematical relations. This forecasting model also takes advantage of the Alyuda Neuro Intelligence to provide forecasting results by using its neural network learning ability in nonlinear relationships inherent in the data. The proposed forecasting model is intended to outperform other forecasting models and obtain a better forecasting performance in foreign exchange rates forecasting.

2 Literature Review The forecasting of foreign exchange rates has been attracting increasing attention as a favorite domain to be examined by various methods. There exists a huge amount of related data and indicators with possible influence when examining a financial time series with the aim of forecasting [1]. Artificial neural network is one of the most famous approaches in forecasting financial time series. Artificial neural network is a universal and highly flexible function model for pattern recognition and classification. It requires no prior assumption on the behavior and function form of the related variables, but it still can capture the underlying dynamic and nonlinear relationships among variables [2]. There are several papers and articles that show the use of artificial neural network is promising in forecasting foreign exchange rates. Wu and Yang [3] investigated the relationship between exchange rate fluctuation and other economic indicators by applying the neural network to predict the yearly exchange rate of the New Taiwan currency. This study selected several economic indicators including the current account, consumer price index, GDP, and money supply, discount rate, as input to predict the exchange rate by the neural network system. The empirical experiment showed that the performance of neural networks was better than the other models discussed. The result demonstrated the predictive strength of the neural network and its potential for solving financial forecasting problems. Panda and Narasimhan [4] demonstrated a neural network to make one-stepahead prediction of weekly Indian rupee/US dollar exchange rate. It compared the forecasting accuracy of the neural network with linear autoregressive and random walk models. By using six forecasting evaluation criteria, this study found that the

An Empirical Study of the Artificial Neural Network

545

neural network had superior in-sample forecasting abilities than linear autoregressive and random walk models. The findings in the study have provided evidence against the efficient market hypothesis and suggested that there exist always a possibility of extracting information hidden in the exchange rate market. Gradojevic and Yang [5] employed a non-parametric method to forecast highfrequency Canadian/US dollar exchange rate. The introduction of a microstructure variable, order flow, substantially improved the predicting abilities of both linear and non-linear models. The non-linear models outperformed random walk and linear models based on a number of recursive out-of-sample forecasts. Two main criteria applied to evaluate model performance were root mean squared error (RMSE) and the ability to predict the direction of exchange rate moves. The artificial neural network (ANN) model was consistently better in RMSE to random walk and linear models for the various out-of-sample set sizes. Moreover, ANN performed better than other models in terms of percentage of correctly predicted exchange rate changes. The empirical results suggested that optimal ANN architecture was superior to random walk and any linear competing model for highfrequency exchange rate forecasting. Chen and Leung [6] evaluated and compared the performance of models based on two competing neural network architectures, the multi-layered feed forward neural network (MLFN) and general regression neural network (GRNN). This study suggested that the selection of proper architectural design might contribute directly to the success in neural network forecasting. An auxiliary experiment was developed and confirmed the possible synergetic effect from combining forecasts made by the two different network architectures. Jamal [7] applied a neural network model for representing the exchange rate of the United States with two of its major trading partners, Canada and the Euro countries. A one-period lagged exchange rate was used as the explanatory variable. The results showed that this model was quite successful in predicting the exchange rates studied. It also indicated that models of the relationship of an exchange rate with its explanatory variable might be updated as often as necessary for estimation and forecasting. Dunis and Williams [8] examined and analyzed the use of neural network regression (NNR) models in foreign exchange (FX) forecasting and trading models. The NNR models were benchmarked against traditional forecasting techniques to ascertain their potential added value as a forecasting and quantitative trading tool. In addition to evaluation the various models using traditional forecasting accuracy measures, they were also assessed using financial criteria. Having constructed a synthetic EUR/USD series for the period up to January 4, 1999, the models were developed using the same in-sample data, leaving the remainder for out-of-sample forecasting, October 1994 to May 2000, and May 2000 to July 2001, respectively. The out-of-sample period results were tested in terms of forecasting accuracy, and in terms of trading performance via a simulated trading strategy. Transaction costs were also taken into account. It concluded that NNR models had the ability to forecast EUR/USD returns for the period investigated, and added value as a forecasting and quantitative trading tool.

546

P.-C. Chen, C.-Y. Lo, and H.-T. Chang

In summary, artificial neural network belongs to the class of data driven approach, as opposed to model driven approach. The representation of data is a critical for a successful neural network design. The merit of artificial neural network is its effective and efficient learning ability in nonlinear relationships inherent in the data. Its ability to self-organize and to function without a pre-programmed knowledge base gives it an important advantage in forecasting sensitive information.

3 Methodology This study is intended to present both a theoretical viewpoint of the development of an artificial neural network, and a practical application of the proposed forecasting model to the forecasting of foreign exchange rates. The research methodology of this study consists of three stages. In the first stage, data collection and data preprocess will be conducted. In the second stage, data training, testing, and forecasting will be applied. In the third stage, forecasting performance will be evaluated by using the statistical measures. Stage One: Data Collection and Data Preprocess This study is intended to forecast the daily exchange rates between US dollar and Chinese Renminbi, which influenced by the daily movements of other different exchange rates including 21 different countries. This forecasting model uses the daily data from January 1, 2007 to December 31, 2007 for in-sample estimation, leaving the period from January 1, 2008 to January 31, 2008 for out-of-sample forecasting and evaluation. The foreign exchange rates data has been collected from the Federal Reserve Bank of New York database and organized by using Excel. In addition, the SPSS will be used to check the correlation between 22 different currencies. Some unrelated exchange rates will be removed and the rest of them will be used as input data. Data preprocess also will be applied, all input data will be transformed into the scale between -1 and +1 by using the artificial neural network application named Alyuda Neuro Intelligence. Stage Two: Data Training, Testing, and Forecasting There are many topologies possible for artificial neural networks. The backpropagation algorithm is one of the most commonly used neural network paradigms. The back-propagation algorithm is a general purpose learning algorithm, which consists of three basic steps: input signals go forward, errors are calculated from expected outputs, and these errors are sent to previous layers for weight adjustments. The back-propagation algorithm will be used to develop the artificial neural network model in this study. This study uses an artificial neural network application named Alyuda Neuro Intelligence as the foundation of a prediction model. After training and testing the input data, the forecast values obtained from this artificial neural network model will be considered as the results of this study.

An Empirical Study of the Artificial Neural Network

547

Stage Three: Evaluation of the Forecasting Performance To examine the performance of forecasting, it is necessary to evaluate the previously unseen data. It is likely to be the closest to a true forecasting or trading situation by using the direct comparison of out-of-sample data. Typically, forecasting models are optimized using a mathematical criterion, and subsequently analyzed using statistical measures. This study will use forecasting accuracy via statistical measures to evaluate the forecasting performance.

4 Data Analysis (1) Data Collection and Data Preprocess According to the Federal Reserve Bank of New York database, there are 22 different exchange rates have been collected (See Figure 1). By using the SPSS correlation, 10 unrelated exchange rates have been removed. The rest of 12 exchange rates have been used as input data for the artificial neural network forecasting model named Alyuda Neuro Intelligence. All input data has been preprocessed into the scale between -1 and +1.

Fig. 1 Data Collection

(2) Data Training, Testing, and Forecasting

Fig. 2 Data Forecasting

548

P.-C. Chen, C.-Y. Lo, and H.-T. Chang

The back-propagation algorithm has been used to run the data training and testing. By using the Alyuda Neuro Intelligence, the best network design has been searched as 12-28-1 architecture. The diagram shows that the target line and the output line are perfectly matched (See Figure 2). It means that this artificial neural network model has been well trained. (3) Evaluation of the Forecasting Performance The out-of-sample data has been collected in order to compare the forecasting values and the real exchange rates. The results demonstrated that the differences between the forecasting values and the real exchange rates are very little. The correlation is 0.998462 and R-squared is 0.99653 (See Figure 3). Those statistical measures have proved the forecasting accuracy of this artificial neural network forecasting model.

Fig. 3 Data Evaluation

5 Conclusion Nowadays the forecasting of foreign exchange rates is more a necessary than an experimental simulation. Since the future exchange rates are not certain, forecasts need to be made for speculation purposes that involve the spot and derivatives markets. Governments, international banks, multinational corporations, and individuals generate their expectations through all kind of resources to develop forecasting tools and advisory services to predict the short-term and long-term exchange rates movements. The foreign exchange market and the foreign exchange rates have affected every aspect of our daily personal and corporate financial lives, and influenced the economic and political destiny of every country. This study provides evidence on the effectiveness and efficiency of the integration of the SPSS and an artificial neural network model into a combined-expertise construction. The proposed model has processed the mix of variables effectively and produced estimations and forecasts efficiently. The findings of this study should contribute positively to the development of theory, methodology, and practice of using artificial neural network to develop a forecasting model with enhanced forecasting accuracy.

An Empirical Study of the Artificial Neural Network

549

References 1. Haykin, S.: Neural Networks: A comprehensive foundation, 2nd edn. Prentice Hall, Upper Saddle River (1999) 2. Medsker, L., Turban, E., Trippi, R.R.: Neural network fundamentals for financial analysts. In: Trippi, R.R., Turban, E. (eds.) Neural Networks In Finance and Investing, pp. 3–24. McGraw-Hill, New York (1996) 3. Wu, W., Yang, H.: Forecasting New Taiwan Dollar/United States Dollar exchange rate using neural network. The Business Review 7(1), 63–70 (2007) 4. Panda, C., Narasimhan, V.: Forecasting Exchange Rate Better with Artificial Neural Network. Journal of Policy Modeling 29(2), 227–236 (2007) 5. Gradojevic, N., Yang, J.: Non-linear, Non-parametric, Non-fundamental Exchange Rate Forecasting. Journal of Forecasting 25(4), 227–245 (2006) 6. Chen, A., Leung, M.: Performance evaluation of neural network architectures: the case of predicting foreign exchange. Journal of Forecasting 24(6), 403–420 7. Jamal, A.: The Neural Network and Exchange Rate Modeling. International Journal of Management 22(1), 28–31 (2005) 8. Dunis, C., Williams, M.: Modeling and Trading the Euro/US Dollar Exchange Rate: Do Neural Network Models Perform Better? Derivatives Use, Trading and Regulation 8(3), 211–240 (2002)

Grey Prediction with Markov-Chain for Crude Oil Production and Consumption in China Hongwei Ma and Zhaotong Zhang*

Abstract. The crude oil demand is growing rapidly in China, driven by its rapid industrialization and motorization. China has already become the second-largest oil importer nation in the world, after the United States. The dynamic GM (1,1) model of grey theory is used to develop the dynamic GM(1,1) model to forecast the crude oil consumption and production in China. In order to improve the forecast accuracy, the original GM (1, 1) models are improved by using Markov-chain. We analyze the data of the crude oil consumption and production from 1990 to 2006 in China, and forecast China’s crude oil consumption and production by this Grey-Markov forecasting model, which shows that the improved grey forecasting model is of more reliability and higher forecast accuracy than GM (1, 1). The forecast results indicate that China’s crude oil consumption and production will continue to increase rapidly in the period of 2007 to 2015. Keywords: Grey system theory, Markov-chain, Grey forecasting model.

1 Introduction Along with economic growth of nearly 10% per year over the last two decades, crude oil production and consumption has been rapidly increasing in China. By 2007, its crude oil consumption ranked the second in the world. In the meantime, along with the continuous economic development and the acceleration of industrialization and motorization processes, China’s crude oil consumption and production will increase even more rapidly. The energy system is an imperfect information system situated between the ‘Black Box’ and ‘White Box’. According to the definition, it is a typical grey system [1]. For such a system, the classic and modern control theories can not work well. The grey theory was developed to meet this demand and is a truly multidisciplinary and generic theory, which deals with the systems characterized Hongwei Ma . Zhaotong Zhang College of Engineering, Nanjing Agricultural University, Nanjing 210031, China *

H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 551–561. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

552

H. Ma and Z. Zhang

by lacking of information. The fields covered by the grey system theory include system analysis, data processing, modeling, prediction, decision-making and control. The accumulated generating operation (AGO) is one of the most important characteristics of grey system theory, and its main purpose is to reduce the randomness of data. In the recent 30 years, the studies on energy system forecasting models have already made great progress, and many forecasting models have been developed. Because all of these models take the energy system as the white system, and adopted the mathematical statistics and regression analysis to establish the forecasting models, we think that the forecasted results of these models can not reflect the real changing situation of energy system completely [2]. This paper attempts to apply grey theory to China’s crude oil consumption and production prediction. We believe that the model GM (1, 1) and Markov model could be integrated with each other to forecast crude oil consumption and production to enhance the predicting accuracy.

2 Research Methodology 2.1 GM (1,1) Grey Forecasting Model The key of grey system theory is the grey dynamics model, which is notable for its generating function and grey differential equation. The concepts of grey block and differential similarity are the natures to establish grey forecast model. This model is generally described as GM (M, N), where M is the rank of differential equation, and N is the number of variables [3]. The procedures of deriving a GM (1, N) grey model are displayed as follows. Step1: Suppose an original sequence with n entries is X i( 0 ) = { x i( 0 ) (1), x i( 0 ) ( 2), … , x i( 0 ) ( k ), … , x i( 0 ) ( n )}, i = 1,2,

, N.

(1)

where xi(0) (k ) is the value at time k .

Step2: Based on the original sequence X i(0) , a new sequence X i(1) can be generated by one time accumulated generating operation (1-AGO), which is X i( 0 ) = { x i( 0 ) (1), x i( 0 ) ( 2), … , xi( 0 ) ( k ), … , x i( 0 ) ( n )}, i = 1,2,

, N.

(2)

where k

xi(1) (k ) = ∑ xi(0) ( j )

(3)

j =1

(1)

Obviously, the new sequence X i is monotonous increasing. Compared with the original sequence X i(0) , the regularity of data is enhanced and the randomness of data is reduced.

Grey Prediction with Markov-Chain for Crude Oil Production

553

Step3: Because of the exponentially increasing nature of the entry, the sequence X i(1) is similar to the exponentially increasing nature of the solution for a first-order differential equation which is accordingly used to express the entry, i.e.,

dx1(1) + ax1(1) = b2 x2(1) + b3 x3(1) + … + bN xN(1) . dt

(4)

Equation (4) is called a GM (1, N) model, where a, b2 , b3 ,… , bN are model parameters. The model parameters can be determined by the least square regression. aˆ = [a, b2 , b3 ,… , bN ]T = (BT B)−1 BT Y ,

(5)

where

Y = ( x1( 0) (2), x1( 0 ) (3), … , x1( 0 ) (k ), … x1( 0 ) (n)) T , 1 (1) ⎤ ⎡ (1) x 2(1) (2), x 3(1) (2), … , x N(1) (2)⎥ ⎢ − 2 [ x1 (1) + x1 (2)], ⎥ ⎢ 1 ⎢ − [ x (1) (2) + x1(1) (3)], x 2(1) (3), x 3(1) (3), … , x N(1) (3) ⎥ B=⎢ 2 1 ⎥ …, …, …, …, … ⎥ ⎢ ⎥ ⎢ 1 (1) (1) (1) (1) (1) ⎢− 2 [ x1 (n − 1) + x1 (n)], x 2 (n), x 3 (n), … , x N (n)⎥ ⎦ ⎣

(6)

According to the differential equation theory, the solution for a GM (1, N) (shown as (4)) is xˆ1(1) ( k + 1) = [ x1(1) (0) −

1 N 1 N bi xi(1) ( k + 1)e − ak + ∑ bi xi(1) ( k + 1)] ∑ a i =2 a i =2 .

(7)

Using the operation of one time inverse accumulated generating operation (1-IAGO), the predicted value of the entry k + 1, xˆ1(0) (k + 1), of the original sequence X 1(0) can be obtained by (7):

xˆ1(0) (k + 1) = xˆ1(1) (k + 1) − xˆ1(1) (k ) .

(8)

If N = 1 , then GM (1, N) model will become a GM(1,1) model [4].

2.2 Partition of States by Markov-Chain Forecasting Model ˆ (k) that The values of X (0 ) (k + 1) are distributed in the region of the trend curve Y may be divided into a convenient number of contiguous intervals[5].When X ( 0 ) (k + 1) falls in interval i, one of S such intervals, it may be regarded as corresponding to a state ⊗i in an m order Markov unstable sequence, ⊗ i can be signified as follows:

554

H. Ma and Z. Zhang

⊗ i = [ ⊗ i1 ,⊗ i2 ] .

(9)

where i=1,2,…,S, S is the amount of states.

ˆ (k ) + A . ⊗i1 = Y i

(10)

ˆ (k ) + B . ⊗i2 = Y i

(11)

ˆ (k ) is a time function, so ⊗i1 and ⊗i2 will vary with the time series. It means Y that the state ⊗i is dynamic. To establish S (the amount of states), ⊗i1 and ⊗i2 , depends on the study object and the original data series.

2.3 Calculate the Transition Probability P For Markov-chain series, the transition probability from state ⊗i to state ⊗ j can be established using an equation as follows: Pij (m) =

M ij (m) Mi

,

( i, j = 1, 2,

(12)

,S)

where Pij (m) is the transition probability of state ⊗ j transferred from state ⊗i for m steps, m is the number of transition steps each time, M ij (m) is the number of original data of state ⊗ j transferred from state ⊗i for m steps, M i is the number of original data points in state ⊗i . These Pij (m) values can be presented as a transition probability matrix R (m ) . ⎡ p11 (m) p12 (m) ⎢ p (m) p (m) 21 22 R(m) = ⎢ ⎢ ⎢ ⎢⎣ pi1 (m) pi2 (m)

p1j (m) ⎤ p 2j (m) ⎥⎥ , ⎥ ⎥ pij (m) ⎥⎦

( i, j = 1, 2,

,S) .

(13)

The state transition probability Pij ( m) reflects the statistical law of each state transition in a system, which is the foundation of Markov probability matrix forecast [6].The future development of the system can be forecasted by studying the state transition probability matrix R (m) .Generally, it is necessary to observe the one-step transition matrix R(1) . Suppose the object to be forecasted is in (1≤q≤S),

row

q

in

max Pqj (1) = pql (1), ( j = 1, 2,

matrix

R(1)

should

be

considered.

⊗q If

, S ) , (1≤q≤S)), then what will most probably happen

in the system at the next moment is the transition from state ⊗q to state ⊗l . It is difficult to determine the future transition of the state, if two or more transition probabilities in the row q of the matrix R (1) are the same. Therefore the transition

Grey Prediction with Markov-Chain for Crude Oil Production

555

probability matrix of two-step transition matrix R(2) or multi-step transition matrix R (m) should be considered [7].

2.4 Calculate the Forecasting Data After the determination of the future-state transition of a system, i.e. the determination of Grey elements ⊗i1 , ⊗i2 , the changing interval of the forecast value is ˆ (k + 1) , is considered to between ⊗i1 and ⊗i2 . The most probable forecast value, Y be the middle value of the determined state interval, that is 1 ˆ + 1) = 1 (⊗ + ⊗ ) = Y(k) ˆ Y(k + (A i + Bi ) . i1 i2 2 2

(14)

3 Evaluation of Forecasting Accuracy The accuracy of forecasting is evaluated by a post-error test method. It uses four important indexes, the average relative error δ, the post-error ratio C, the probability of small error P, and the forecasting precision ρ. The details are described as follows: Given the original sequence

{

}

X i( 0) = xi(0) (1), xi(0) (2),…, xi( 0) (n) and its predicted equivalent

Xˆ (0) = { xˆ (0) (1), xˆ (0) (2),… , xˆ (0) (n)} , the forecasting errors are ε ( k ) = x (0) ( k ) − xˆ (0) ( k ).

k = 1, 2,… , n

the mean values and covariance for both the forecasted sequence and the forecasting errors, x ,

S1 , and ε , S 2 ,δ, ρ are x=

ε=

δ=

1 n (0) 1 n ( 0) x ( k ), S 1 = [ x ( k ) − x ] 2 , k = 1,2, ∑ ∑ n k =1 n k =1

1 n ( 0) ∑ ε (k ), n k =1

S2 =

1 n ( 0) [ε ( k ) − ε ] 2 , k = 1,2, ∑ n k =1

1 n 1 x ( 0 ) ( k ) − xˆ ( 0 ) (k ) , ρ = 1 − δ , ∑ ( 0) n k =1 x (k )

from (15) and (16), we can obtain

k = 1,2,

, n.

, n.

, n.

(15)

(16)

(17)

556

H. Ma and Z. Zhang

Table 1 The four grades of forecasting accuracy

grade1: very good grade2: good grade3: qualified

δ <0.01 <0.05 <0.10

P >0.95 >0.80 >0.70

C <0.35 <0.50 <0.65

ρ(%) >95 >90 >85

grade4: unqualifed

<0.20

>0.60

<0.80

>80

Grade

C = S 2 / S1 ,

P = p{ε (k ) − ε 〈 0.6745 S1 }

(18)

Based on these values, the accuracy grade of model can be judged according to the criteria list in Table 1. The smaller the C value is, the higher the accuracy grade of model has. On the other hand, the higher the P value, the higher the accuracy grade of model has. Because the post-error ratio C indicates the change rate of the forecasting error, a smaller C value explains a larger S1 value and a smaller S2 value, and P shows the probability of the relative bias for forecasting error, and the higher P value indicates that the proportions of data points with residual values less than the average residual by values smaller than 0.6745[8].

4 Model Checking and Forecast on Crude Oil Consumption and Production in China There are many factors which could influence the crude oil production and consumption, such as the industry structure, the economic growth, the governmental policy and so on. Some factors are clear, and others are not clear. So the time series of the crude oil production of China shows random fluctuations. As Table 2 shows, the historical data of the crude oil production from 1990 to 2005 is rising, but fluctuating randomly. So this paper forecast and analyzes the crude oil production of China by Grey-Markov forecasting model [9]. Table 2 The crude oil production of China from 1990 to 2006 (Unit: 10000 tons of SCE)

Number Year Amount Number Year Amount

1 1990 19745 .18 9 1998 22986 .25

2 1991 20130. 05 10 1999 22857. 16

3 1992 20271. 38 11 2000 23280. 51

4 1993 20768. 03 12 2001 23420. 70

5 1994 20896. 30 13 2002 23858. 05

6 1995 21419. 64 14 2003 24232. 16

7 1996 22544. 72 15 2004 25122. 45

8 1997 22906. 93 16 2005 25982. 68

Grey Prediction with Markov-Chain for Crude Oil Production

557

4.1 Build the GM (1, 1) Grey Forecasting Model Based on the historical data of the crude oil production in China from 1990 to 2002,a trend curve equation is built by GM(1,1) Grey forecasting model. It is as follows:

ˆ (k ) = X ˆ ( 0 ) ( k + 1) = 1255703 .674807e 0.015975k − 1235958 .494807 Y Where k is the series number of the year, and k=0 means that it is 1990.

4.2 Partition of States by Markov-Chain Forecasting Model According to the actual crude oil production data, four states, i.e., four contiguous ˆ (0) (k + 1) . The four state intervals intervals can be established about the curve of X can be got as follows:

ˆ ( k ) − 0.03 Y , ⊗1 : ⊗11 = Y

ˆ ( k ) − 0.01 Y ⊗12 = Y

ˆ ( k ) − 0.01Y , ⊗ 2 : ⊗ 21 = Y ˆ ⊗ 3 : ⊗ 31 = Y(k),

ˆ (k ) ⊗ 22 = Y

ˆ ⊗ 32 = Y(k) + 0.01Y

ˆ ( k ) + 0.01 Y , ⊗ 4 : ⊗ 41 = Y

ˆ ( k ) + 0.03 Y ⊗ 42 = Y

where Y denotes the average value of the historical crude oil production from 1990 to 2002.The historical data series, the regressed curve and the states intervals are showed in Fig.1. 25000

Crude Oil Production in China

24000 23000 22000 21000 20000 19000 1990

1992

1994

1996

1998

2000

2002

Fig. 1 The forecasting curve of the crude oil production in China

4.3 Calculate the Transition Probability P After observing the states intervals and historical data series, the number of the historical data in every interval can be got. They are as follows:

558

H. Ma and Z. Zhang

M1 = 3,

M 2 = 5,

M 3 = 1,

M4 = 3

Where Mi denotes the number of the historical data in the interval I, and i=1, 2, 3, 4. Then calculate the one-step transition probability to every states interval and present them in the transition matrix R(1) as follows: ⎡ ⎢ ⎢ ⎢ R (1 ) = ⎢ ⎢ ⎢ ⎢ ⎢ ⎣

1 3 3 5 0 0

2 3 1 5 1 1 1 3

0 0 0 0

⎤ 0 ⎥ 1 ⎥ ⎥ 5 ⎥ 0 ⎥ ⎥ 2 ⎥ ⎥ 3 ⎦

The crude oil production of 2002 year is in the ⊗1 state interval, so observe the first row of the transition matrix. Because max P1j = P12 ( j = 1, 2,3, 4) as the matrix R(1) shows, the crude oil production of 2003 year is most probably in the ⊗2 state interval, so we can calculate the most probable forecast value of 2003 year’s crude oil production by Eq (14) [10]. 1 ˆ = (⊗21 + ⊗22 ) = 24384.32 Y(14) 2

4.4 Comparison of Forecast Precision between the Grey-Markov Forecasting Model and the GM (1, 1) Forecasting Model As the above, we can get the forecast values from 2003 to 2006 calculated by Grey-Markov forecasting model and GM (1, 1) forecasting model [11]. From Table 2, we can get the reality amount of the crude oil production in China. We compared the forecast values between the two models and the results are showed in Table 3 and Table 4. Table 3 Comparison of forecast results with two different methods

Year

Reality

GM(1,1) value

Precision

Grey-Markov value Precision

2003

24232.16

24493.97

98.92%

24382.32

99.37%

2004

25122.45

24888.40

99.07%

25107.70

99.94%

2005

25982.68

25289.19

97.33%

25727.78

99.02%

2006

26394.09

25696.43

97.36%

26135.02

99.02%

Grey Prediction with Markov-Chain for Crude Oil Production

559

Table 4 Comparison of forecast results with two different methods

Type Crude Oil Production Type Crude Oil Production

δ 0.034 δ 0.021

GM(1,1) Model C P ρ(%) 0.195 1 96.6 Grey-Markov Model C P ρ(%) 0.096 1 98.6

Grade grade 2 Grade grade 1

Table 3 and Table 4 shows that the forecast values of Grey-Markov forecasting model are more precise and reliable than original GM(1,1) model, for this model makes full use of the information given by historical data, and increase greatly the forecasting precision of random fluctuating sequences [12].

4.5 The Future Forecast Results of Crude Oil Consumption and Production in China Calculate the forecast value from 2007 to 2015 by the Grey-Markov forecasting model following the above steps. Table 5 shows the forecasting values. The forecast results of crude oil consumption and production from 2007 to 2015 via the Grey-Markov forecasting model are shown in Table5 and Fig.2. As shown by prediction, the crude oil consumption and production in China will continue to increase rapidly for the next 10 years [13]. In addition, Table5 and Fig.2 show that from 2007 to 2015, the annual growth rate of crude oil production will reach 2.13%, while the annual growth rate of crude oil consumption will be 7.24%. The average crude oil consumption from 2007 to 2015 will reach 2.38 times as against the average crude oil production. In 2015, the crude oil consumption and production will Table 5 The future forecast results from 2007 to 2015(unit:10000 tons of SCE)

Year

Grey-Markov Forecast Value

2007

production 26483.0298

consumption 53046.9621

2008

26944.1947

56865.7286

2009

27413.3901

60959.4019

2010

27890.7559

65347.7724

2011

28376.4343

70052.0546

2012

28870.5702

75094.9906

2013

29373.3107

80500.9595

2014

29884.8058

86296.0956

2015

30405.2078

92508.4143

560

H. Ma and Z. Zhang

100000

Crude Oil Production and Consumption

90000 80000 70000 60000 50000 40000 30000 20000 10000 1990

1995

2000 Production

2005

2010

2015

Cons umption

Fig. 2 The forecasting results of the crude oil production in China

be 925.0841 MTCE and 304.0521 MTCE respectively, and the former will be 3.0425 times as against the later. So the crude oil in China will continue to depend on export heavily [14].

5 Conclusions This paper has applied the Markov-chain to develop the GM (1, 1) models to forecast the crude oil consumption and production in China. The GM (1, 1) models reflect the macroscopically regulation, the Markov-chain shows the vibration development of the microcosmic system, and both not only have the mutual advantage but also can make full use of the information obtained from the original time series [15]. The posteriori check results show that the Grey-Markov forecasting models have higher forecast accuracy than the original ones. Acknowledgments. This study was supported financially by the college of engineering, Nanjing agricultural university, and the agricultural machinery bureau of Jiangsu province (Project No.GXS06012).The authors would like to thank Prof. Wang Xiaohua at the college of engineering, Nanjing agricultural university for his valuable comments and kindly help.

References 1. Deng, J.L.: Grey System (Society and Economy). Publishing House of National Defense Industry, Wuhan (1985) 2. Wang, X.H., Feng, Z.M.: Rural household energy with the economic development in China: stages and characteristic indices. Energy Policy 29, 1391–1397 (2001) 3. He, Y., Bao, Y.D.: Grey-Markov forecasting model and its application. System Engineering-Theory & Practice 9, 59–63 (1992)

Grey Prediction with Markov-Chain for Crude Oil Production

561

4. Deng, J.L.: Control Problems of Grey System. Huazhong University of Science and Technology Press, Wuhan (1990) 5. Huang, M., He, Y., Cen, H.: Predictive analysis on electric-power supply and demand in China. Renewable Energy 32, 1165–1174 (2007) 6. Liu, S.F., Lin, Y.: An Introduction to Grey Systems: Foundation, Methodology and Application, pp. 97–111. IIGSS Academic Publisher, Slippery Rock (1998) 7. National Bureau of Statistic of China. China statistical year book. China Statistics Press, Beijing (2007) 8. Hu, D.L.: Applying the grey system theory to evaluate the competitive competence of enterprises. Journal of Scientific-Technical Progress and Strategy 20, 159–161 (2003) 9. He, Y.: A new forecasting model for agricultural commodities. Agric. Eng. Res. 60, 227–235 (1995) 10. CCICED: Energy Strategies and Techniques. Publishing House of China Environmental Science (1997) 11. Hsu, C.I., Wen, Y.H.: Improved grey prediction models for the trans-Pacific air passenger market. Transportation planning and Technology 22, 87–107 (1998) 12. Chao, H.W.: Predicting tourism demand using fuzzy time series and hybrid grey theory. Tourism Manage 25, 367–374 (2004) 13. Lee, C.: Grey system theory wins application on earthquake forecasting. Journal of Seismology 4, 27–31 (1986) 14. He, Y., Bao, Y.D.: Grey-Markov forecasting model and its application. Sys. Eng (Theory Practice) 9, 59–63 (1992) 15. Li, C.H.: Applying the grey prediction model to the global integrated circuit industry. Technological Forecasting and Social Change 70, 63–74 (2003)

Fabric Weave Identification Based on Cellular Neural Network Suyi Liu, Qian Wan, and Heng Zhang*

Abstract. Owing to the property of binary output it has a very important in identification of image processing in cellular neural networks (CNN). Warp and weft interlacing points of fabric image will be expressed as images of binary property in the paper. Adopting cellular neural network it makes a cellular element a warp interlacing point or a weft interlacing point, and calculates the relation of cellular element and neighbor cellular element, then it can identify the weave structure of fabric. It has made identification of weave structure on three original fabric weave contained plain weave, twill weave and satin weave, and obtained better result. Keywords: Cellular Neural Network, Fabric, Warp Interlacing Point, Weft Interlacing Point, Weave Identification.

1 Introduction Cellular neural network (CNN), which is network connecting with locals, consist of many cell elements. It has features of few data quantity, fast processing speed and good effect. So it is widely used in image processing, signal processing and other fields. The analysis and identification of fabric weave structure is an important part in textile industry production process. The main purpose of identification of fabric weave structure is that identify integrated structure of fabric sample, one cycle period at least, and structure character, then it can guide further production. At present the way of analysis and identification on fabric weave structure usually takes Fourier analytics, autocorrelation function method, and wavelets transform method. According to the similar feature between fabrics weave structure, by adopting CNN a cellular element is represented a weft interlacing point or a warp interlacing point, then it identifies the fabrics weave structure after calculating the relationship of cellular and adjacent cellular. Suyi Liu . Qian Wan . Heng Zhang College of Electronics and Information Engineering, Wuhan University of Science and Engineering, 430073 Wuhan, China [email protected] *

H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 563–569. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

564

S. Liu, Q. Wan, and H. Zhang

2 Cellular Neural Network and Its Application in Image Processing 2.1 Basic Theory of Cellular Neural Network Basic unit of cellular neural network is called artificial cell just connected with surrounding neurons, and it is a continuous non-linear dynamic system. Shown in figure1, it is a cellular neural network in 4 × 4 size. c(i, j ) means the cell in the

i th row and the j th column. Fig. 1 A structure diagram of 2 dimension cellular neural network

Cellular neural network, in M × N dimension, is constitutes by M × N cells

of c(i, j ) , 1 ≤ i ≤ M ,1 ≤ j ≤ N . And its dynamic process can be described by one-step nonlinear differential equation in the following. (1) State equation

v xij (t ) = −v xij (t ) +

∑ Aij, kl v ykl (t ) +

c ( k , l )∈N r (i , j )

∑ Bij;kl vukl (t ) +I ij . (1)

c ( k ,l )∈N ir (i ' j )

1 ≤ i ≤ m, 1 ≤ j ≤ n , where Nr (i, j ) is the near neighbors of a neuron c(i, j ) , and its radius is r. The definition is:

N i (i, j ) = {C (k , l ) max{ k − i , l − j } ≤ r} , 1 ≤ k ≤ m, 1 ≤ l ≤ n .

Aij , kl ——feedback coefficient between neuron and output; Bij , kl ——coefficient between neuron and output; I ij ——external biasing input. (2) Output equation

v yij (t ) = f (v xij (t )) = 0.5[ v xij (t ) + 1 − v xij (t ) − 1 ] .

(2)

Fabric Weave Identification Based on Cellular Neural Network

565

Fig. 2 Piecewise linear function

1 ≤ i ≤ m, 1 ≤ j ≤ n . The equation shows relation between output and state, where output function f (v ) is a piecewise linear function, shown in figure 2. (3) Constraint condition

v xij (t ) ≤ 1 vuij (t ) ≤ 1

.

(3)

(4) Parametric hypothesis

A(i, j; k , l ) = A(k , l ; i, j ) .

(4)

Literature [2] has proved CNN is stable. And it shows: Theorem 1 If network parameter meet

A(i, j; i, j ) >

1 . Rx

(5)

Then lim v xij (t ) > 1 , lim v yij (t ) = ±1 . t →∞ t →∞ The theorem 1 ensures cellular neural network has output of binary output. The quality has important significance for classification problem of image processing.

2.2 Cellular Neural Network Applied to Image Processing Applying CNN to process an image it needs the following disposal. (1) Differential equation transform into difference equation First order difference equation, is equivalent of equation 1, shows:

v xij (t + 1) − v xij (t ) = −v xij (t ) +

∑

∑

Aij , kl v ykl (t ) + Bij ; kl vukl (t ) +I ij c ( k , l )∈N r (i , j ) c ( k , l )∈N ir (i ' j )

Then

v xij (t + 1) =

∑

∑

Aij , kl v ykl (t ) + Bij; kl vukl (t ) +I ij . c ( k , l )∈N r (i , j ) c ( k , l )∈N ir (i ' j )

(6)

566

S. Liu, Q. Wan, and H. Zhang

(2) Range adjustment of pixel value In order to meet expression (3) of constraint condition of CNN, image pixel value [0,255] is linearly mapped to the range of [-1,1]. Its means to process as follows:

vukl =

2 vukl − 1 . 255

(7)

When CNN is applied to image processing, its size is identical with pending image. If pending image contains M × N pixel, CNN contains M × N cell neuron, where has one-to-one correspondence between pixel and cell. According to the expression (3) of constraint condition of CNN initial state value in processing, set initial state value of CNN at zero each point of network is substituted into expression (1), until the whole network stability, and then the output is binary, that is a classification pattern of corresponding image.

3 Analysis of Fabric Weaves Sort of fabric weaves is numerous. It includes original weave, fancy weave, compound weave, complex weave. The elementary weave, is the most basic weave form, includes plain weave, twill weave and satin weave. They are the basis of fancy weave and other weave, shown in Fig. 3. Thus it is very important to analyze and identify elementary weave.

(a) plain weave

(b) twill weave

(c)satin weave

Fig. 3 Schematic diagram of three original weaves

Fabric consists of interlacing points that are staggered by weft and warp. Fig 3(a) shows plain weave arrangement graph. To up, down, left and right direction-adjacent interlacing point of each warp interlacing point is weft interlacing point. In the same way, adjacent interlacing point of each weft interlacing point is warp interlacing point. So we can identify weave structure of the fabric with analyzing the relationship of the fabric weave points. Let property of interlacing point of fabric image be expressed as numerical value, weft interlacing points are expressed as “1” (it is represented as white in binary image), and warp interlacing points are expressed as “0” (it is represented

Fabric Weave Identification Based on Cellular Neural Network

567

as black in binary image). Filling with the original location and region of the interlacing points respectively by this way, the binary image of interlacing point points will be formed. By analyzing fabric weave image, we found that it is similar to the structure of the CNN. Therefore, by adopting CNN it made a cellular element a warp interlacing point or a weft interlacing point, further calculated the relation of cellular element and neighbor cellular element, and then weave structure of fabric can be identified.

4 The Identification of Fabric Original Weave Based on CNN 4.1 Design of Network Parameters Referred to cellular state equation (1), the dynamic mechanism of CNN includes output feedback and input controlling. Because the effect of output feedback is determined by template A, and the effect of input controlling is determined by template B, key of making image processing with CNN is how to find out a suitable template. When it chose r = 1 , the structure of C (i, j ) and eight neurons surrounded,

Nr (i, j ) ∈ R3×3 . Then A and B both are 3 × 3 templates, according to the feature of the fabric original weave, it choose a temple which likes form shown in:

namely

⎡0 0 0 ⎤ A = ⎢⎢0 a 0⎥⎥ ⎢⎣0 0 0⎥⎦

，

⎡− c − c − c ⎤ B = ⎢⎢− c b − c ⎥⎥ . ⎢⎣− c − c − c ⎥⎦

4.2 Identification of Fabric Original Weaves The paper takes plain weave, will weave and satin weave fabric as original images. Shown in fig 4, it is a bitmap which has 256 gray levels.

(a) plain weave Fig. 4 Fabric sample

(b) twill weave

(c) satin weave

568

S. Liu, Q. Wan, and H. Zhang

Through multiple adjustment calculations by the use of MATLAB software, we get network parameters: I=0.5 a=2,b=8,c=1, namely

，

⎡ 0 0 0⎤ ⎡− 1 − 1 − 1⎤ A = ⎢⎢0 2 0⎥⎥ , B = ⎢⎢− 1 8 − 1⎥⎥ . ⎢⎣0 0 0⎥⎦ ⎢⎣− 1 − 1 − 1⎥⎦ We can get the identification result of the fabric original weave based on the flow chart shown in fig 5. Network conditions of convergence are: If there are cells which their state between 1 and -1 in network, it manifests that network is not convergent, and then network will enter into cyclic iteration again. Otherwise it is convergent, that means it can jump out of cyclic iteration. Fig. 5 Flow chart

Start

Readin Image

Data Processing

Setting Initial Parameters Calculation and Iteration

No

Judging If Network Is Convergence Yes Output

End

Fabric Weave Identification Based on Cellular Neural Network

569

The identification results of fabric original weave refers to table 1. Table 1 Identification results of three original weaves of fabric Fabric categories

Plain weave Twill weave Satin weave

Weft interlacing point 1 2 4

Warp interlacing point 1 1 1

Meaning

a weft intersect a warp two weft intersect a warp four weft intersect a warp

5 Conclusion According to the similarity between fabrics weave structure and CNN structure in this paper, by adopting CNN it made a cellular element a warp interlacing point or a weft interlacing point, further calculated the relation of cellular element and neighbor cellular element, and then it identified weave structure of fabric. The paper has made demonstrated experiments of weave structure identification on three original fabric weaves contained plain weave, twill weave and satin weave. It got a good identifiable result, thus it is said that the way is effective.

References 1. Ravandi, S.A.H, Toriumi, K., Matsumoto, Y.: Fourier Transform Analysis of Plain Weave Fabric Appearance. Waterman, T.F. (ed.) Textile Research Journal 65, 65–69 (1995) 2. Li, Y.M., Xu, B.J., Gao, W.D.: Studies on Automatic Recognition System of Fabrics Weave Parameters. Beijing Textile 23(2), 54–57 (2002) 3. He, F., Li, L.Q., Xu, J.M.: Woven Fabric Density Measure Based on Adaptive Wavelets Transform. Journal of Textile Research 28(2), 32–35 (2007) 4. Chua, L.O.: CNN: A Paradigm for Complexity. Scientific Series On Nonlinear Science (1998) 5. Chua, L.O., Yang, L.: Cellular Neural Networks: Theory. IEEE Transactions on Circuits and Systems 35(10), 1257–1272 (1988) 6. Cai, B.X.: Fabric Structure and Design. Textile Industry Publishing House, Beijing (1990) 7. Chua, L.: CNN: A Paradigm for Complexity. World Scientific, Singapore (1998) 8. Wang, W.L., Li, Z.G., Xiao, Z.T.: Study on Image Processing Based on Cellular Neural Network. Computer Science. 35(4A), 159–161 (2008)

Cutting Force Prediction of High-Speed Milling Hardened Steel Based on BP Neural Networks Yuanling Chen, Weiren Long, Fanglan Ma, and Baolei Zhang*

Abstract. It is absolutely necessary to machine the complex mould cavity with micro-ball end mill in high speed machining. Because of the priceless and frangibility of the tool, it is significant to predict the cutting force in order to lessen the tool’s disrepair, to ensure the quality and to improve efficiency. This paper established the cutting force prediction model based on BP neural network when machining arc surface of hardened steel in high speed. According to the sample set of experimental results used to train and test the neural network, we realize the cutting force prediction and simulation through introducing the elastic grads’ decrease method to improve the speed of convergence and precision in the process of cutting. The practice showed that most of the error values are about 5% except some special individual. Obviously, to predict the cutting force is feasible in the process of non-stability and non-linear extremely of cutting through the non-linear neural network. Keywords: BP neural networks, Micro-ball end mill, Hardened steel, Cutting force, Prediction.

1 Introduction It is one of the powerful ways to research the dynamical trait in the process of milling based on the physics modeling and simulation. Many scholars established the milling force model through the cutting theory analysis, trial-system identification as well as differential method and established edge line’s or solid modeling of ball end mill considering the effect of tool’s flexible bend[1,2,3]. Because the tool is slim, it will turn backward and vibration easily leading to the cutting force and thickness of chip change accordingly. In fact, it is difficult to predict and measure the value of turn and trait through study the cutting force model in analysis and trial Yuanling Chen . Weiren Long . Fanglan Ma . Baolei Zhang College of Mechanical Engineering, Guangxi University, Nanning 530004, China [email protected]

*

H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 571–577. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

572

Y. Chen et al.

way. The forward neural network has better study ability and non-linear changing mechanism, it can model and simulate exact system effectively, which is difficult to establish with math formula. Many foreign scholars carried on researches and applications on the neural network in kinds of fields, such as the tool’s condition identification, the optimization of cutting condition, the prediction of surface’s roughness and control, and so on. They have made good achievements [4,5,6].

2 Theory of BP Neural Network The study of neural network can be regarded as a process of specific function approach, and the answer of the problems commonly not precision. So we must consider the accuracy and the function approaching ability. The paper adopts the forward neural network, namely BP. The need of BP neural network must to be met is monotonicity. The double bent function is the following:

f ( x) =

1 1 + e− x

.

(1)

The reverse spread of the vector ä is the base of BP. It adjusts the weight and threshold using square of the network’s error and the differential coefficient of the input of every layer in the network to reduce the error’s sum of squares which get expected outputs. So we usually choose the function of the linear neural cells. When the value of f ( x) is about 0 or 1, the update becomes much smaller leading to much more stability. So before training, we must make the sample’s inputs and outputs to be naturalized as a citizen in order to make the outputs between 0 and 1.

3 Experimental Setup 3.1 Experimental Conditions 3.1.1 Machine Tool, Cutter and Workpiece Material The experiment was carried on an YCM-V85A vertical spindle milling center (maximum rotating speed up to 8000RPM). The cutter is a 4-flutes hard alloy ball end mill with TiAlN-coated and its diameter is 2 millimeters. The workpiece material is hardened steel 45(HRC 52). 3.1.2 Force-Measured System The forces were measured using YDX-III9720 type dynamometer. The signals were amplified by charge amplifier, and then were transmitted to computer through signal-collected card cooperated with the software “general cutting forcemeasured system”.

Cutting Force Prediction of High-Speed Milling Hardened Steel

573

3.2 Experimental Project The experiment adopted single factor to machine arc surface with slot-milling method. That is, only changes levels of one factor. The experimental parameter’s setup was shown in the table 1. Table 1 Test parameters

Factors

Levels

Axial cutting depth Ad (mm)

0.07 0.08 0.09 0.10 0.11 0.12 0.13

Spindle speed n(r/min)

2387 3183 3978 4774 5570 6366

Feed F(mm/min)

50 100 150 200 250 300 350

Arc radius R(mm)

4.0 4.5 5.0 5.5 6.0

Tool’s Cantilever L(mm)

13 15 17 19 21

The cutter’s diameter

：φ=2mm；the cutter’s teeth：N=4；

4 Establishing the Cutting Force’s Neural Network Model The neural network adopted the toolbox in the MATLAB to design and simulate. For the cutter, the force Fx, Fy in the plane XY made the bent of the cutter, come into being the machine error, and the force Z-direction Fz make the cutter disrepair because of being compressed. The neural network can be applied in the prediction of Fx, Fy and Fz. Because the cutting force varied at any moment (as Fig.1), we choose the maximum of the force at one point ensuring the comparabilZ ity of the experimental data. The neural network can be divided into G 7RRO $ teacher-study, which offers teacher signal Y R accorded with input signal and studies the % connection between input and output, and $ no teacher-study, which studies by itself © accorded with the input signal. This study carried out teacher- train to the network ¥ through the cutting force , which gotten in the experiment on the condition that R changes the axial cutting depth, feed speed, rotate speed, surface radius and the suspended handle. The training samples Fig. 1 Milling along arc surface should be consisted of all the values which 7

)

H F H L S N U R :

574

Y. Chen et al.

Table 2 The samples of net training No. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Ad F (mm) (mm/min) 0.07 200 0.09 200 0.11 200 0.13 200 0.10 50 0.10 150 0.10 250 0.10 350 0.10 200 0.10 200 0.10 200 0.10 200 0.10 200 0.10 200 0.10 200 0.10 200

n (r/min) 6366 6366 6366 6366 6366 6366 6366 6366 5570 3978 2387 6366 6366 6366 6366 6366

R (mm) 4.0 4.0 4.0 4.0 4.0 4.0 4.0 4.0 4.0 4.0 4.0 5.0 6.0 4.0 4.0 4.0

L (mm) 19 19 19 19 19 19 19 19 19 19 19 19 19 13 17 21

Fx (N) 63.4 88.3 105.0 115.2 82.9 88.7 102.6 117.7 103.4 114.6 119.8 107.8 120.9 80.5 88.4 68.5

Fy (N) 39.5 47.6 49.2 48.9 37.5 43.8 46.6 46.1 44.1 46.0 47.7 52.3 59.7 47.2 56.9 40.6

Fz (N) 128.7 173.8 196.9 225.3 195.0 185.3 209.2 240.8 209.7 224.9 238.6 195.8 220.6 134.9 180.6 140.4

Fy (N) 45.4 45.8 43.1 42.3 46.6 40.6 50.7

Fz (N) 147.5 188.3 196.4 220.3 241.6 183.8 163.2

Table 3 The samples of net checking No. 17 18 19 20 21 22 23

Ad (mm) 0.08 0.10 0.10 0.10 0.10 0.10 0.10

F (mm/min) 200 200 100 200 200 200 200

n R (r/min) (mm) 6366 4.0 6366 4.0 6366 4.0 4774 4.0 3178 4.0 6366 4.5 6366 4.0

L (mm) 19 19 19 19 19 19 15

Fx (N) 76.8 93.9 88.3 109 118.1 100.4 75.6

6 7 8 3 1 ,

used to predict parameters, Viz. It should be covered the maximum and minimum and make the network to have enough samples to get better approaching precision and widely applied scope. 6 7 8 3 7 8 2

4.1 Multilayer Network Design and Training

 W XU S H W \ XD 2 /

U WH X \ S D Q/ ,

Fig 2 is the typical multilayer neur-al network’s sketch. In the study, there are 5 nerve cells in the input layer, 11 in the hidden layer and 3 in the output layer. The +LGGHQ double bent tangent function sigmoid is /D\HUV adopted in the input layer and middle layer, as well as the output layer adopted linear Fig. 2 The typical sketch of Multi-layer function. The samples in the table 2 are neural network

Cutting Force Prediction of High-Speed Milling Hardened Steel

575

used to train network through elastic grads’ decrease met-hod. Its flow chart as Fig.3. The training precision is kept 1e-5, the maximum training times is 1e4, the momentum coefficientäcan overcome the system’s vibration and the leaning rateócan control the pace of convergence. In this experiment, the momentum coefficient is 2.5 and leaning rate is 0.8, the cutting experimental parameters and results of the test 1-16 are used as training samples. The network is converged after 8045 steps. The result Error defined as the absolute value of experimental value which divided by the difference of the absolute value of prediction and the absolute value of experiment.

Start

The initialization of weight and threshold

Study patterns are offered to network

Middle layer’s input and output

Sj =

∑W

ij

，b

⋅aj −θ

j

j

= f (S j )

Deferent layer’s input and output

Lj =

∑V

t j

⋅bj − γ

， C

κ t

j

= f ( Lt )

YES

E <ε

END

NO The general errors and grads of output layer

d

k j

= ( y tk − C tk ) ⋅ f ′ ( L t )

、 Δν

= a ⋅ d tk ⋅ b j

Every unit’s input and output

、

e = [ ∑ d tk v jt ] ⋅ f ′ ( b i ) k t

Δ W = β ⋅ d tk ⋅ b j

Modify the weight and threshold with grads and optimize arithmetic

Fig. 3 The flow chart of BP network’s arithmetic

576

Y. Chen et al.

4.2 Results of Network Check The samples (17-23) which do not participate in the train can be used as test samples to predict in the trained BP network to verify its cover capability in the table 3. The predicted results and errors are as shown in the table 4, the predicted errors are within 10%, except that very few are much bigger which show that the network having better predicted capability can be used to predict the cutting force. Table 4 Testing results and errors Predicted value(N)

No. 1 2 3 4 5 6 7

Error (%)

Px

Py

Pz

Ex

Ey

Ez

76.7 97.2 84.8 109.3 118.9 105.0 82.4

44.3 49.0 36.3 44.3 47.8 44.6 50.9

154.8 186.2 185.2 220.5 229.2 191.3 142.2

0.16 3.48 3.99 0.23 0.70 4.57 9.02

2.46 6.93 15.80 4.65 2.58 9.96 0.31

4.94 1.09 5.72 0.11 5.15 4.06 12.85

5 Application Through designing, training and testing the network above, the application capability can satisfy our request and can be used to predict the cutting force. The cutting parameters which is used to prediction and the predicted results which we adopted the above network getting are shown in the table 5. Table 5 Predicted results of cutting force No. 1 2 3

Ad (mm) 0.12 0.10 0.10

F (mm/min) 200 300 200

N (r/min) 6366 6366 6366

R (mm) 4.0 4.0 5.5

L (mm) 19 19 19

Px (N) 110.8 111.6 111.6

Py (N) 49.0 46.2 62.2

Pz (N) 210.2 227.8 200.7

In order to verify the feasibility of the predicted cutting force based on the network above, we compare the experimental cutting force with the parameters above and predicted force, getting the errors shown in table 6. The fact which predicts error values is about 5% mostly except some special individual. The error for the individual predicting value is relatively large for which the main reason is caused by the following multiple factors when collecting the experimental data, the accumulation of the random error and the system error because of the influence from the outer factors, and the accumulation of the nettraining error. Due to the prediction of the nerve net is actually to seek the imminent function by the experimental data, and only in the condition that the structure of net is convergent, the result of the prediction is stable and reliable.

Cutting Force Prediction of High-Speed Milling Hardened Steel

577

Table 6 Experimental value and predicted error No. 1 2 3

The experimental value(N)

Error(%)

Fx

Fy

Fz

Ex

Ey

Ez

104.3 105.9 113.2

48.3 46.7 53.4

200.3 222.8 208.7

5.85 5.14 1.47

1.45 1.09 14.16

4.69 2.18 3.98

As for the above research, we can extend to other predict toward the cutting force in larger range, and choose in prior the cutting parameters and optimize the cutting process according to the extent of the predicting value.

6 Conclusions Through the above analysis, it gets the following conclusions. 1) 2) 3) 4)

Most of the error value is about 5% except some special individual. The multilayer net which is trained by the large-scale swatch can predict the cutting force well of which the result is stable and reliable. The research result shows that the predict of cutting force during the process of the non-linear nerve net and the unstable & the altitudinal non-linear nerve net is reliable. To calculate backward the cutting parameter by using the compare of the nerve net model can combine and optimize the cutting parameter.

Acknowledgement. The research is supported by the National Natural Science Foundation of China (50665001) and the project of Guangxi Graduate Student Education Innovation Plan.

References 1. Ma, W., Lin, Z., Chen, K.: The Cutting Force Model of Rigid Ball End Mill. Mechanical Science And Technology 17, 422–424 (1998) (in Chinese) 2. Ma, W., Wang, N.: The Study of Cutting Force Model Considering The Elastic Distortion of Ball End Mill. Journal of Nanjing University of Aeronautics & Astronautics 30, 633–640 (1998) (in Chinese) 3. Ni, Q., Li, C., Ruan, X.: Cutting Forces Simulation of Ball-End Milling Based on Solid Modeling. Journal of Shanghai Jiao Tong University 35, 1003–1007 (2001) (in Chinese) 4. Dimla, E., Dimla, J.R., Paul, M., Nigel, J.: Automatic Tool State Identification in a Metal Turning Operation Using MLP. Neural Networks and Multivariate Process Parameters 38, 343–352 (1998) 5. Franci, C., Uros, Z.: Approach to Optimization of Cutting Conditions by Using Artificial Neural Networks. Materials Processing Technology 173, 281–290 (2006) 6. Durmus, K.: Prediction and Control of Surface Roughness in CNC Lathe Using Artificial Neural Network. Materials Processing Technology 175, 321–329 (2008) 7. Ruan, X.: Artificial Intelligence’s Way Neural Calculational Science the Simulation of Brain’s Function Based on the Cell. National Defense Industry Publishing Company, Nanjing (2006) (in Chinese)

BP Neural Networks Based Soft Measurement of Rheological Properties of CWS Dong Xie and Changxi Li*

Abstract. Rheological properties of the CWS are very important for pumping and combustion, but online measurement of them is still a problem for CWS’s NonNewtonian fluid characteristics. Based on the special nonlinear mapping function of BP neural networks, a kind of soft measurement technology is put forward to predict the rheological characteristics of CWS. From the solids concentration and one or two apparent viscosities of the suspension at certain shear rate, the BP neural network can predicate the apparent viscosities at other shear rates with time dependence history. The result shows its effectiveness comparing with the Yield Power Low equation. Keywords: Neural networks, Coal water slurry, Rheology measurement, Soft measurement.

1 Introduction Coal-water slurries (CWSs), also referred to coal-water mixtures (CWMs), coalwater fuels (CWFs) and coal-water pastes (CWPs), are the most promising of all alternative coal-based fuels for replacing oil by coal due to their lower cost and similarity to oil with respect to convenience in transporting and handling. CWS is a high concentrated suspension mixed with coal, water and about 1% chemical additives. It needs an appropriate yield value to maintain its stability during storage, a low apparent viscosity to be combusted satisfactorily and a high coal concentration for economic use [1]. Therefore, the characterization of rheological behavior in CWS is important for slurry preparation, storage, transfer and atomization. In practical terms, the slurry should have a low viscosity at the moderate shear rates for pumping (10-200s-1). When CWS is transported over long distances of kilometers or more through pipes, its rheological properties will change greatly as the settling of solids. It is necessary to measure CWS’s Dong Xie and Changxi Li Automatic Control Science & Engineering Dept., Huazhong University of Science & Technology, Wuhan 430074, China [email protected]

*

H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 579–586. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

580

D. Xie and C. Li

rheological parameters in order to decide whether continue to pump or to do some pretreatment works first, such as chumming or heating, etc. Some studies have been directed towards relating rheological properties of CWS to atomization quality (5000-30000 s-1) [2,3]. It is found that low slurry viscosity leads to small droplet size during atomization of CWSs and hence an increase in the carbon conversion efficiency in a boiler or furnace. In order to implement automatic production, the online measurement technology for the rheological properties of CWS is in an urgently requirement. As a Non-Newtonian fluid, the characteristics of the CWSs make it complex to evaluate its detailed rheological properties. It is usually using sampling inspection offline. The aim of the present study was to investigate the possible ways of implement an online rheological measurement of the CWS by a soft measurement method based on a neural network prediction model.

2 Theory and Method 2.1 Rheological Characteristics of CWS In the former researches, it is concluded that CWSs generally possess yield points, and any single CWS can exhibit regions of Newtonian, dilatants and pseudo-plastic behavior. It is generally agreed that CWS exhibits a weak shearthinning behavior at low and intermediate shear rates, and becomes Newtonian at higher shear rate where atomization occurs [4]. And it also exhibits thixotropic and rheopectic behavior obviously [5]. Previous studies have revealed that the rheological characteristics of CWS would be varied with a number of factors [1,6]: 1) 2) 3) 4) 5) 6)

physicochemical properties of coal; volume fraction, Ø, of the suspension; particle size range and its distribution; interparticle interactions in the suspension, which are affected by the nature of surface groups, the pH and presence of electrolytes and chemical additives; temperature of the suspension; pretreatment of the suspension, such as microwave, ultrasonic wave and magnetic water;

In most of the case, the CWS suspension is so complex to get its rheological mechanism model and the constitutive equations. Even get one, it’s usually difficult to use. Therefore, empirical models are more commonly used. In order to mathematically describe the relationship between shear stress and shear rate, equations have been developed to fit the flow curves obtained by experiments. The Ostwal de Waele or Power Law as follows [1,7]

τ = kγ n .

(1)

is widely used. The τ is the shear stress, k is the consistency index, γ is the shear rate, n is the flow behavior index. The power law index (n) is a measure of the

581

Shear Stress(Pa)

Shear Stress(Pa)

BP Neural Networks Based Soft Measurement of Rheological Properties of CWS

Fig. 1 Time Dependency of Thixotropic & Rheopetic CWSs

deviation from Newtonian behavior of the fluid, i.e. n=1 for Newtonian and k=μ (viscosity); if n<1, the fluid is shear thinning, and if n>1, the fluid is dilatant. The more universally applicable equation is Yield Power Low [8]:

τ = τ 0 + kγ n .

(2)

where the τ and k are as described above, the τ0 is yield point. When τ0=0, it is the power law. When n=1, it is a Bingham plastic fluid and k=η (plastic viscosity). There are others equations such as Casson equation and Sisko equation [8,9]. However, all the equations described above deal only with the effect of shear rate on viscosity, while the shear rate history of CWS is not included. As a thixotropic fluid, the rheology can change with time as the structure of the fluid is altered. Fig.1 shows the effect of slowly change of the shear rate. For thixotropic CWS, the decreasing shear curve is lower than the increasing shear curve, and the viscosity of the sample decrease with time. The opposite behavior is seen with rheopectic fluid. Because of the time dependency characteristic of CWS, the Yield Power Lower would have big error. According to the raw data of [10], the average error of the cure fit by yield power lower could be near to 50%.

2.2 Rheological Measurement Viscosity and yield point are two of the most important rheological properties of CWMs. The most common measuring principle to measure rheological properties is the rotational rheometer which allows determining a list of properties in an accepted manner. The method is based on the measurement of the rotational torque when a defined speed is preset or the resulting speed is measured at a set torque level. It is obvious that we can only get apparent viscosity of the NonNewtonian fluid at a certain steady shear rate once a time. Using sinusoidal stress or a sudden jump, we could only get the quantity of the elastic properties which is not the most concerned. Other measure methods such as capillary rheometer have the same problem, we cannot get the apparent viscosity covered a shear rate range in a time short enough to implement online measurement. Even having several rheometers running at separate shear rates simultaneously, we would ignore the time dependency characteristic and have a big error.

582

D. Xie and C. Li

2.3 Soft Measurement and ANN Soft measurement is a method by modeling mathematic model for some parameters which are difficult to be detected directly. The mathematic model descripts the relationship between the parameters which are needed to be detected and some other parameters which are easy to get. By getting these easy-detected parameters instead, it is possible to achieve what we need. But it is always difficult to find the related parameters and the mathematic model. The artificial neural network (ANN) method such as back propagation (BP) neural nets has been a highly feasible and creative technique to solve traditional problems. It can map highly nonlinear input-output relationships; the experiences it obtains include not only those of human knowledge, but also those which may be still unknown to human experts. Another important function of ANN is the ability to interpolate and extrapolate from its experiences. Because of this, many researches on creating soft measurement model by ANN have carried out [11]. Because there are many factors influence the rheology of CWSs and the rheological properties is non-linear, it is possible to use 3-layer PB neural network to get the measurement result.

3 Numerical Example Analysis The experiment data came from [10]. There were 32 groups of rheology test data of CWS for autoclave sample. The rheological properties were determined based on the Yield Power Low equation (Eq.2) using the Haak RV 100 viscometer. In all the tests, every condition was fixed but the solids concentration which changed from 33.91 wt% to 55.39 wt%. In each test, it was got 10 apparent viscosities at 10 shear rates which were respectively 50Hz, 100Hz, 200Hz, 300Hz, 400Hz, 400Hz, 300Hz, 200Hz, 100Hz and 50Hz. The shear rate increased from 50Hz to 400Hz at first, and then reduced to the start point. For example in one test, the solids concentration was 43.51 wt%, the apparent viscosities were as Table 1 and Fig.2. It’s assumed that the physicochemical properties of coal were almost the same in all the samples; the particle size range and its distributions were similar; the interparticle interactions in the suspension were the same, and all the tests were carried out in the same temperature. So what only affects the rheology of the CWS was the solids concentration. To avoid the case that individual element was insensitive to the rheology, a point or two of the apparent viscosity was also selected as the input of the BP networks. Considering to the hysteresis loop of the rheology curve, when added one point of the apparent viscosity with all the shear rates, the one at 400Hz shear rate was selected, and the one at 50Hz was added for two points. It was easy to have rheometers running at these shear rate and read the viscosities at any time. The online concentration meter was also wildly used in industry. So if the ANN worked, the online measurement of the rheology of the CWS could be implemented by this soft measurement method. The shear rate sequence was also selected as the input. The output of the BP networks was the vector of 10 apparent viscosities at 10 shear rates.

BP Neural Networks Based Soft Measurement of Rheological Properties of CWS

583

Table 1 The apparent viscosities at 10 shear rates of 43.51 wt% CWS Shear rate(Hz) Viscosity(cP)

+50 +100 +200 +300 +400 -400 -300 -200 -100 -50 295.4 231.6 185.0 170.9 159.1 159.7 159.5 170.5 210.0 291.8

Apparent viscosity(cP)

300

250

200

150 50

100

150

200 250 Shear rate(Hz)

300

350

Fig. 2 The apparent viscosity at 10 shear rates

6 Train Validation Test

5

Square error

4

3

2

1

0

0

1

2

3

4 Epoching

5

6

Fig. 3 Error curve of the 12 elements BP neural network

7

8

400

584

D. Xie and C. Li

So two 3-layer frame were chosen as the neural networks structure, which were 12-30-10 and 13-30-10. 16 groups of rheological data were selected randomly to train the networks, 8 groups to validate the networks, and 8 left to test it. Normalization processing was applied to all the input data. The training function was chosen as the BP neural networks’ training algorithm. After learning training of the neural networks, the error curves of network training were close to the ideal level, as shown in Fig.3 and Fig.4. It was seen that the error curve of networks training was close to the ideal level quickly. The 13 elements BP neural network had a smaller error than the 12 elements one. The validation error of the 12 elements was a little big while the error of the 13elements one came to an acceptable level. The effect of the predicate of the BP neural networks and the Yield Power Low equation (Eq.(2)) fit to the raw data are showing as Fig.5 and Fig.6. The first 16 data are train data, from 17 to 24 are validation data and last 8 are test data. It is seen that both of the neural networks are trained well as they have little error in the first 16 data. But the 12 elements network has an error high to 37%, while the 13 elements one has 24%. However, comparing with the Yield Power Low equation, the 12 elements network has similar error, and error of the 13 elements network is distinct reduced. The BP networks have a better effect to evaluate the rheological characteristic of CWSs than the equation. The reasons why the error is still high up to 24% are that the training data is not big enough and that some factors influencing the rheological properties are not considered, such as pH and the ash rate of the suspension. After all, it is always difficult to get a precision value of rheological parameters. 7 Train Validation Test

6

Square error

5

4

3

2

1

0 0

1

2

3

4 5 Epoching

6

7

Fig. 4 Error curve of the 13 elements BP neural network

8

9

BP Neural Networks Based Soft Measurement of Rheological Properties of CWS

585

0.5 ANN Error YPL Error

0.45 0.4 0.35

Error

0.3 0.25 0.2 0.15 0.1 0.05 0

0

5

10

15 20 Data Groups

25

30

35

Fig. 5 Compare the error of the 12 elements ANN with the error of Yield Power Low equation 0.5 ANN Error YPL Error

0.45 0.4 0.35

Error

0.3 0.25 0.2 0.15 0.1 0.05 0

0

5

10

15 20 Data Groups

25

30

35

Fig. 6 Compare the error of the 13 elements ANN with the error of Yield Power Low equation

586

D. Xie and C. Li

4 Conclusion In this paper, a novel soft measurement method based on BP neural network for the rheological characteristic of CWS is presented. From the solids concentration and one or two apparent viscosities of the suspension at certain shear rate, the BP neural network can predicate the apparent viscosities at other shear rate with time dependence. The result show that the accuracy of the new soft measurement method exceeds that of the common used Yield Power Low equation. The online measurement of the rheological properties of CWS is possible to implement in this way. If we get more parameters of the CWS condition, such as the temperature, ash rate, particle size rate and its distribution, PH and the chemical additive concentrations, etc, it is probable to get an exacter model. And a Hopfield neural network would be more suitable than BP networks for the hysteresis [12]. All these need more study in the future.

References 1. Ogura, T., Tanoura, M., Hırakı, A.: Behaviour of Surfactants in a Highly Loaded CoalWater Slurry: 1. Effects of Surfactants Concentration on Its Properties. Beulleti of the Chemical Society of Japan 66, 1343–1349 (1993) 2. LaFlesh, R.C., Lachowicz, Y.V., McGown, J.G.: Combustion Characteristics of CoalWater Fuels. In: Proc. Eighth Int. Symp. On Coal Slurry Prep. and Utils., U.S. Dept. of Energy, P.E.T.C., Pittsburgh, PA, pp. 438–452 (1986) 3. Burdukov, A.P., Popov, V.I., Tomilov, V.G.: The Rheodynamics and Combustion of Coal-Water Mixtures. Fuel 14, 927–933 (2002) 4. Ken, D.K., Paul, D.: Dynamic Surface Tension of Coal-Water Slurry Fuels. Fuel 7, 295–300 (1995) 5. Lu, p., Zhang, M.: Rheology of Coal-Water Paste. Powder Technology 150, 189–195 (2005) 6. Roh, N.S., Shin, D.H., Kim, D.C., Kim, J.D.: Rheological Behavior of Coal Water Mixture: 2. Effect of Surfactants and Temperature. Fuel 74, 1313–1318 (1995) 7. Gürses, A., Açıkyıldız, M., Doğar, Ç.: An Investigation on Effecs of Various Parameters on Viscosities of Coal-Water Mixture Prepared with Erzurum-Aşkale Lignite Coal. Fuel Processing Technology 87, 821–827 (2006) 8. Raffi, M., Turian, Jamel, F., Attal, Sung, d., Liewis, E., Wedge, w.: Properties and Rheology of Coal-Water Mixtures Using Different Coals. Fuel 81, 2019–2033 (2002) 9. Trochet-Mignard, L., Taylor, P., Bognolo, G., Tadros, T.F.: Concentrated Coal-Water Suspensions Containing Nonionic Surfactants and Polyelectrolytes 2. Adsorption of Nonyl Phenyl Propylene Oxide-Ethylene Oxide on Coal and the Rheology of the Resulting Suspension. Colloids and Surfaces A: Physicochemical and Engineering Aspects 95, 37–42 (1995) 10. Chris, M., Mark, A., Brian, C.: Coal-Water Fuel Preparation and Gasification. ThailandTask 39, Topical Report. DE-FC21-93MC30098 (1996), http://www.OSTI.org 11. Xiong, G., Nybeg, T.R., Xu, X.: BP Based Soft Measurement of Flash Point of Lubrication Oil. In: Proceedings of the 3rd World Congress on Intelligent Control and Automation, vol. 2, pp. 1114–1118 (2000) 12. Sunil, B., Jerry, M.: The Hysteretic Hopfield Neural Network. IEEE Transactions on Neural Networks 11, 879–888 (2000)

A Parameters Self-adjusting ANN-PI Controller Based on Homotopy BP Algorithm Shuguang Liu and Mingyuan Liu*

Abstract. Back-Propagation neural network can record the fuzzy control rules efficiently, and utilize these experiences according to associative memory. But, all existing feedforward net learning algorithms have a local minimum problem inevitably. To solve above problem, a Homotopy continuation BP algorithm is adopted in this paper, which provides an effective method for BP network’s global convergence and is of very fast convergent speed. For some complex nonlinear control systems, a parameters self-adjusting fuzzy-PI controller is ever adopted effectively. Because ANN has strongly nonlinear mapping power, so we can use ANN based on Homopy BP algorithm to replace fuzzy segment to reconstruct a new ANN-PI controller, which has a faster dynamic response, higher control accuracy, better disturbance-resisting ability, less sensitive to parameter changes, and robustness. Keywords: Homotopy, neural network, parameters self-adjusting, fuzzy rules.

1 Introduction Because fuzzy control can simulate human being’s adjusting mentality and control strategy with a nice bit of intelligence, and its adjusting rules is independent of the accurate mathematics model and easy to be implemented, so we pay attention to it more and more[1-3]. In general, a control object’s mathematics model must be knew before designing a control system, however it is very difficult for us to obtain this model with good accuracy and convenient to design in many manufacturing processes. Traditional fuzzy controller is a kind of control system based on fuzzy rules, which is an induction of fuzzy information and conclusion of control experience. In people’s thinking, an abstract conception is corresponding to a fuzzy variable, whereas the experiential knowledge is a mapping of linking them. From Shuguang Liu Xi’an Polytechnic University, Xi’an 710048 China [email protected]

*

Mingyuan Liu Xi’an Jiaotong University, Xi’an 710049 China H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 587–595. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

588

S. Liu and M. Liu

view of physiology, people’s knowledge is some brain’s neural memory, which is carried out by different conjunction intension between neurons. According to this point, the fuzzy rules can be stored in manual neural network completely[4]. The learning procedure in BP neural network based on gradient descent method is always related to the initial value of all weights connecting neurons, only when these weights is appropriate, the network is convergent, otherwise it is not convergent. Moreover, BP network’s convergence speed is very slow, it’s iterative step is likely to be up to several thousands times so much as over ten thousand times, perhaps a satisfying result don’t be received. Homotopy BP algorithm is not only convergent in a large area, but also it has a quick convergence speed and strong ability of overcoming morbidity. When the gradient descent method is not convergent, it can find a satisfying solution[5]. Because of the simple ANN controller’s input quantitative factors and output proportional factors are fixed and can not be adjusted automatically, so its control accuracy and dynamic response speed, anti-jamming capability are certain limited and can’t be further improved. Although ANN-PI controller can speed up the dynamic response and improve control accuracy, but because of it’s fixed parameters, controller’s adaptive capacity is not excellent. In response to this problem, on the basis of ANN-PI controller we use the strategy of self-tuning parameters to design one kind of parameters’ self-adjusting ANN-PI controller for sizing machine warp unwinding tension autolevelling[6].

2 Homotopy BP Algorithm For the back-propagation (BP) neural network in fig.1, an error function can be defined by 2

⎤ 1 N3 1 N2 ⎡ 2 ε (k ) = ∑ yˆ j (k ) − y j (k ) = ∑ ⎢ yˆ j (k ) − f (∑ Wij (k ) xi (k ))⎥ (1) 2 j 2 j =1 ⎣ j =1 ⎦

[

where

]

yˆ is supposed to be the expected output, y denotes the actual output.

Fig. 1 A typical structure of three layers BP network

A Parameters Self-adjusting ANN-PI Controller Based on Homotopy BP Algorithm

589

In fact, BP algorithm is one grads algorithm, namely

Wij (k + 1) = Wij (k ) − μ∇ε (k ) = W (k ) − μ

∂ε (k ) ∂Wij (k )

(2)

By using the nonlinear optimal K-T condition, we can translate equation (2) into the zero problem of the continuous function f, i.e.

f ( x, W ) = 0 Let W and Y be a non-empty subset of glazed mapping, if

，

(3)

R n respectively G, F : W → Y is a

(t ,W ) ∈ [0, 1] × W satisfy

H : [t , W ] = (1 − t )G (W ) + tF (W ) .

(4)

H : [0, 1] × W → Y is called a linear Homotopy between G and F, else t ∈ [0, 1] is a Homotopy parameter, and the dimension of R n is n = N1 × N 2 + N 2 × N 3 .

then

Confirming zero of nonlinear question F is that a auxiliary zero-glazed mapping:

G : R n → R n is selected to construct Homotopy mapping between F and G: H : [0, 1] × R n → R n , if H (0,W ) = G (W ) and H (1,W ) = F (W ) , then zero of Homotopy H is coherent with that of G at t=0, and zero of H is also coherent with that of F at t=1. Let ∇ε ( k ) = ∇ε (W ) = 0 , the zero problem of equation (2) is equivalent to that of equation (3). So, a Homotopy function can be constructed as follows:

H (t ,W (t )) = (1 − t )(W (t ) − W (0)) + t∇ε (W ), t ∈ [0, 1]

(5)

F (W ) = ∇ε (W ) is a waited-solved problem, the auxiliary function G (W ) = W (t ) − W (0) is a simple problem, W(0) is some

where the objective function

initial values of weights. Zero set of Homotopy mapping for equation (4) implies that

H −1 (0) = {(t ,W ) H (t ,W ) = 0}, t ∈ [0, 1], W ∈ R n .

(6)

Let

H (t , W (t )) = 0, t ∈ [0, 1] ,

(7)

if equation (7) is continuously differentiable in an interval [0,1], there exists the renewed weights formula such that[5] W ( k + 1) = W (k ) + k = 0,1,

,L − k

1⎡ k k ∂ ⎤ (1 − ) I + ∇ε (W (k )) −1 ⎥ • [∇ε (W (k ) − (W (k ) − W (0))] (8) ⎢ L⎣ L L ∂W ⎦

590

S. Liu and M. Liu

where I is an identity matrix, L is an iterative step. For the arbitrary given W(0), an enough exact W(L) can be obtained after L iterative step such that −1

⎡ ∂ ⎤ W (k + 1) = W (k ) − μ ⎢ ∇ε (W (k ))⎥ • [∇ε (W (k )) − W (k )], k = L, L + 1, ∂ W ⎣ ⎦

(9)

3 Parameters Self-adjusting ANN-PI Controller Based on Homotopy BP Algorithm 3.1 Using Three Layers BP Network to Record the Fuzzy Rules Fuzzy controller’s control rules can be valued as follows:

if E = Ai and EC = B j then U = Cij , i = 1,2,

, m; j = 1,2,

, n.

(10)

Where Ai, Bj, Cij are the fuzzy sets respectively on domain X, Y, Z, denoting a set of fuzzy variables such as positive large (PL), positive middle (PM), negative large (NL), . Sentences (10) can be described as a fuzzy relationship R from X×Y to Z as follows:

…

R = ∨ Ai ∧ B j ∧ Cij

(11)

i, j

Which yields,

μ R ( x , y , z ) = ∨ μ A ( x ) ∧ μ B ( y ) ∧ μC ( z ) . i

i, j

j

ij

(12)

If output error and error change of the controlled object were fuzzy sets A and B, the control action would be gained by fuzzy reasoning composition rules as follows [3]:

U = ( A × B) R .

(13)

Thus,

μU ( z ) = ∨ μ R ( x, y , z ) ∧ μ A ( x ) ∧ μ B ( y ) .

(14)

x, y

A center of gravity method is sued to obtain controller’s output: n

U=

∑U i −1

i

⋅ μU (U i ) (15)

n

∑μ i =1

U

(U i )

Above fuzzy logic rules can be implemented by three layers BP neural network[4].

A Parameters Self-adjusting ANN-PI Controller Based on Homotopy BP Algorithm

591

Fig. 2 Three layers BP network controller

In fig.2, the 1st to 3rd layer of Homotopy network implements the fuzzy control rules ‘IF-THEN’, but the 3rd to 4th layer does a output adjudication. The numbers of three layer neural network’s input- and output- nodes are designed according to calculating accuracy, but that of middle hidden nodes are related on the recorded samples. In general, s hidden nodes can record s+1 different samples exactly[7].

3.2 Parameters Self-adjusting ANN-PI Controller As the proportional and integral action of linear PI controller can enable the control system to get higher accuracy and speed up the dynamic response, so introduction of PI structure to ANN will be able to improve the controller’s performance. The relationship of the input and output of ANN-PI controller is t

U (t ) = K P ⋅ U + K I ⋅ ∫ Udt 0

t

= K P ⋅ U i + K I ⋅ T ⋅ ∑U i

(16)

i =0

where T is sampling time, KP is the proportional gain, KI is the integral gain. To further improve the controller performance, literature[6] put forward the thought of K1, K2, KP, KI self-adjusting on the basis of ANN-PI controller, so that K1, K2, KP, KI can be accommodated by themselves with different of e and e . When e or e is larger, we take ‘rough-tuning’ strategy, i.e. decreasing K1, K2 to reduce the resolution of e and e , in the meanwhile enlarging KP, KI to output the larger control quantity. Conversely, when e or e is smaller, which closes to the steady-state system, we take the ‘fine-tuning’ strategy, i.e. enlarging K1, K2 to improve the resolution of e and e , in the meanwhile decreasing KP, KI to output the small control quantity. The self-adjustment operation process is that we first use the original K1, K2 to quantify e and e , and according to ANN2 we can get the

592

S. Liu and M. Liu

parameters enlarging (or decreasing) multiples n, after that we can calculate K1=K1n, K2=K2n, KI=KI/n, KP=KP/n as ANN1’s new parameters. The system diagram of parameters self-adjusting ANN-PI controller is shown as fig.3.

.

e

Fig. 3 Parameters self-adjusting ANN-PI controller

Fuzzy sets and domain of E, EC, U, N are respectively {PB,PM,PS,PO,NO,NS,NM,NB} and {+6,+5,+4,+3,+2,+1,+0,-0,-1,-2,-3,-4,-5,-6} {PB,PM,PS,O,NS,NM,NB} and {+6,+5,+4,+3,+2,+1,0,-1,-2,-3,-4,-5,-6} {PB,PM,PS,NO,NS,NM,NB} and {+7,+6,+5,+4,+3,+2,+1,0,-1,-2,-3,-4,-5,-6,-7} {AB,AM,AS,OK,CS,CM,CB} and {8,4,2,1,1/2,1/4,1/8} Fuzzy control rules table of ANN1 and parameters modified rules table of ANN2 show in tab.1 and tab.2 respectively. The inputs of ANN1 has 26 nodes, corresponding to all integers of E, EC from – 6 to +6, the outputs has 15 nodes corresponding to all integers of U from –7 to +7, the hidden nodes is 57. The inputs of ANN2 has 26 nodes the outputs has 7 nodes, the hidden nodes is 57. For ANN1 and ANN2, some arbitrary real numbers would be selected from [-1, 1] to initialize weights.

，

，

Table 1 Fuzzy control rules table EC

U

E

NB

NM

NS

O

PS

PM

PB

NB

PB

PB

PB

PB

PM

O

O

NM

PB

PB

PB

PB

PM

O

O

NS

PM

PM

PM

PS

O

NS

NS

NO

PM

PM

PS

O

NS

NM

NM

PO

PM

PM

PS

O

NS

NM

NM

PS

PS

PS

O

NS

NM

NM

NM

PM

O

O

NM

NB

NB

NB

NB

PB

O

O

NM

NB

NB

NB

NB

A Parameters Self-adjusting ANN-PI Controller Based on Homotopy BP Algorithm

593

Table 2 Parameters modified rules table EC

N

E

NB

NM

NS

O

PS

PM

NB

CB

CM

CS

OK

CS

CM

PB CB

NM

CM

CS

OK

OK

OK

CS

CM

NS

CS

OK

OK

AS

OK

OK

CS

NO

OK

OK

AM

AB

AM

OK

OK

PO

OK

OK

AM

AB

AM

OK

OK

PS

CS

OK

OK

AS

OK

OK

CS

PM

CM

CS

OK

OK

OK

CS

CM

PB

CB

CM

CS

OK

CS

CM

CB

4 Simulation and Application Simulations are carried in two cases: one is with the conventional fuzzy controller and the other is with the parameters of self-adjusting ANN-PI controller. Simulation model is selected as follow:

G (s) =

e − sτ 5( s + 5)

(17)

The step response curves of two different controllers imposed on equation (17) are shown in fig.4 at τ = 0 and τ = 3 . Simulating results shows that the parameters self-adjusting ANN-PI controller is better than the conventional fuzzy controller in aspects of the rise time, overshoot, settling time, accuracy and hysteresis. \





















W

1- the conventional fuzzy controller; 2- the parameters of self-adjusting ANN-PI controller. Fig. 4 Two kinds of controllers’ step response curves

In the textile process, the control problem of sizing machine warp unwinding tension is one of the problems of yarn tension levelling. The yarn’s mechanical characteristic has not yet been clear [8]. Taking sizing machine as an example, there

594

S. Liu and M. Liu

are many factors to impact the mechanical features of yarns, such as the changes of winding torque, the improper temperature in serous fluid and drying room, the changes of yarn roll and grouting roll rate, the bad moment of yarn roll and tension roll, the bad control of shaft brake, even as the braking zone between the wheel and brake pads for long-term use and wear, and so will bring on tension in the yarn. These reasons make some difficulties to establish accurate mathematical models. Furthermore, view from the research of unwinding aerodynamic control of electrical circuits used in the proportional valve, the proportional valve has the feature of hysteresis, which is a non-linear element, no doubt this will lead to non-linear in the axis unwinding. So, an ANN-PI control strategy based on Homotopy BP network for sizing machine warp unwinding tension control is more appropriate. The yarn tension control curve of 45 Tex. Count cotton is shown in fig.5. From the experimental results, the conventional fuzzy controller’s ascending time is about six sampling period (sampling time T=10ms [7]), while the parameters selfadjusting ANN-PI controller’s ascending time is about three sampling period, its response time is increased greatly. Compared with the conventional fuzzy controller, the parameters self-adjusting ANN-PI controller has small fluctuation for the response characteristic curve, good disturbance-resisting ability and high accuracy. To test the control system’s disturbance-resisting capability, the random interference research on the performance of the parameters self-adjusting ANN-PI controller should be carried out. A random interference f=Us•RND (Us = 0.4V, the time imposed on the system is 5 sampling periods) was added to the control Fig. 5 Unwinding tension control curve

1- the conventional fuzzy controller; 2- the parameters of self-adjusting ANN-PI controller.

Fig. 6 Self-adjusting parameters fuzzy-PI controller’s unwinding tension control curve

A Parameters Self-adjusting ANN-PI Controller Based on Homotopy BP Algorithm

595

system, like fig.6. The results showed that this control system has better disturbance-resisting capability, showing a strong adaptability to the parameters changes.

References 1. Zadeh, L.A.: Fuzzy Sets. Inform. Contr. 8, 338–353 (1965) 2. Rutherford, D.A., Bloore, G.C.: The Implementation of Fuzzy Algorithms for Control. Proc. IEEE 6, 572–573 (1976) 3. Zadeh, L.A.: Outline of a New Approach to the Analysis of Complex Systems and Decision Processes. IEEE Trans. Syst. Man. Cybern. 3, 28–44 (1973) 4. Chang, S.K.: On the Execution of Fuzzy Programs Using Finite-State Machines. IEEE Trans. Computer 61, 211–253 (1972) 5. Li, B.S., Liu, Z.J.: The Application of Fuzzy Set Theory to the Design of a Class Controllers. Acta Automatica Sinica 6, 25–32 (1980) 6. Hu, J.Y., Wu, Z.Q., Song, S.S.: Parameters Self-Adjusting Fuzzy-PI Controller. Information and Control 6, 28–33 (1997) 7. Liu, S.G., Wei, J.M.: Application of Fuzzy Control to Yarn Tension Autolevelling. Journal of Textile Research 15, 4–8 (1993)

Study on Optimization of the Laser Texturing Surface Morphology Parameters Based on ANN Zhigao Luo, Binbin Fan, Xiaodong Guo, Xiang Wang, and Ju Li*

Abstract. This paper analyses the important impact of laser texturing surface morphology on drawing forming. Based on the artificial neural network theory, the nonlinear mapping relations between the morphology parameters and the quality of drawing sheet was studied, the optimization model of laser texturing morphology parameters were established, and fixed on the optimization target, then the fitness function suitable for the genetic algorithm was determined, and the laser texturing morphology parameters were optimized. Taken BenDou for example, the multi-parameter numerical simulation was taken, and the morphology sharp parameters were obtained after the Genetic Algorithm. The results showed that parts with better formability, and noted that this method had better Optimizing guide. Keywords: Laser texturing, Drawing forming, the artificial neural network, the Genetic Algorithm, Numerical simulation.

1 Introduction In the drawing of sheet metal forming process, the main forming default of product is rupture and wrinkle. With the geometric parameters and the process parameters of mold setting in premise, and the main factor affect product quality is friction lubrication. The greater the friction, the greater movement of resistance of the sheet metal, the greater tension stress suffere in sheet metal; the smaller the friction, the sheet metal suffere the uneven tension stress, and prone to wrinkle defects. Laser texturing technology changes the friction between the sheet metal and the mold by changing the mold surface topography parameters, and indirectly improves the quality of the drawing forming. The process of drawing forming is a multi-parameter nonlinear interactions, and the non-linear mapping from the multiple-input to multiple-output is achieved by the artificial neural network, the intelligent design for laser texturing morphology is applied by artificial neural network in prior. This paper takes the drawing process of Zhigao Luo . Binbin Fan . Xiaodong Guo . Xiang Wang . Ju Li College of Mechanical Engineering, Jiangsu University, Zhenjiang 212013, China [email protected], [email protected] *

H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 597–603. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

598

Z. Luo et al.

Bendou as an example to study, establish the artificial neural network models between the Drawing quality and appearance of laser texturing parameters, at the same time the genetic algorithm is used to optimize the laser texturing morphology parameters.

2 Establish the Models of the Morphology Parameters 2.1 Confirm the Optimize of Variable Parameters The impact of drawing contains four factors, such as size and shape of the parts parameters, the shape of blank, process conditions, and sheet performance. In this paper, the drawing quality improved by the laser texturing morphology on the surface is considered in paper, so the relevant parameters of the morphology are selected: micro figure radius R, distribution density ρ, height h as the variable parameter. Optimize model of the variable is:

X = [R、ρ、h] .

(1)

as 1 show, as a result of the restrictions of the processing parameters of Laser texturing, the micro figure radius ranges from 250 to 425μm, the height is between 7.2 to 35.2μm, and the distribution density of micro figure affect the drawing process little , So optimization of the surface distribution density is between 6% to 30%.

2.2 Confirm of the Optimize Target The common defect in sheet metal forming process are rupture and wrinkle, the drawing performance indicators can usually be chosen as a measurement: (1) Rupture factor p, there are a variety of reasons for the rupture in sheet drawing, such as the rupture strength, the plastic rupture and so on. FLD sheet describes the limit strain under the different ways, which can determine the occurrence of rupture. The rupture factor is defined according to the windage D in FLD, as:

、 、h）= ( D + D

P( R ρ

where D is windage,

max

) / Dmax ,

(2)

Dmax is the absolute maximum of windage.

(2) Wrinkle factor Q, there are many external factors lead to wrinkle , such as stress wrinkle, shear stress wrinkle, uneven tensile stress wrinkle, bending stress wrinkle and so on, and in the final analysis, wrinkle is produced by shear stress in up and under surface, lead to instability plastic deformation of sheet. Q is defined as:

、 、h）= ε

Q(R ρ where strain.

ε2

2

− ε1 ,

is the up surface tangential strain,

ε1

(3)

is the down surface tangential

Study on Optimization of the Laser Texturing Surface Morphology Parameters

599

(3) The homogeneous of sheet thickness M, and the rupture and wrinkle are related to thickness, so , M is defined as:

M ( R、ρ、h）= J / J max , where J is the thickness distribution, J max is the max in J, where i

(4) 1

⎛ ⎞p J = ⎜ ∑ ( h i − h) p ⎟ , ⎝ 1 ⎠ n

n is the unit number, h is the thickness of the ith unit , h is the initial thickness, p is the strengthen factor. Above all, the research target is to avoid rupture and wrinkle in a reasonable windage and thickness uniformity. And the simple optimization objective function as follows:

、、）

f ( R ρ h = w1[( D + Dmax ) / Dmax ] + w2 ε 2 − ε 1 + w3 ( J / J max ) , (5) where

w1

、 w 、 w are the weighted coefficients. 2

3

2.3 Determination of Fitness Function In this paper, the design variables of the objective function was forecast by the artificial neural network model, and the proportion of linear method was used in the genetic algorithm fitness function. Fitness function is the only measurement to evaluate the genetic chromosome performance, the greater value the fitness function is, the closer the corresponding solution to the problem optimal solution. According to the analysis above, fitness function is constructed as follows:

、ρ、h) = f (f R(R、、ρρ、、h)h)

、、 、、

− f ( R ρ h) , ρ h) min max − f ( R

eval ( R

max

(6)

f ( R、ρ、h) is one of the individual objective function value, f ( R、ρ、h) max f ( R、ρ、h) min stand for the minimum and maximum of the objection function respectively, the smaller f ( R、ρ、h) is the greater fitwhere:

、

ness, and the better solutions for gene surface.

3 Method of Hybrid Optimization During the process of sheet metal drawing forming, the artificial network model is established by a limited time numerical simulation, and the relationship between the surface topography parameter and the quality is trained by the neural network. Different sheet metal forming result is quickly and largely received from this model quickly. The input node to be chosen is the diameter of micro figure R, density ρ and height h. The output node is the deviation D, tangential strain ε and the thickness J. The number computed by numerical simulation training is used as data sample, and another number is calculated as the test data at the same time. A

600

Z. Luo et al.

Begin

Numerical simulation

Establish the neural network model

Determine the group size

GA

Training neural network model Convergence judge

N

Objective function value Y Optimization parameter Fig. 1 Hybrid optimization program

、

suitable neural network parameters are chosen, such as layer the number of neurons and transfer function and so on. A neural network model is selected through sample, and then test data is used to be test, the model is accepted if the error is in the permitted scope of the license. The objective function is predicted based on the artificial neural network above, and the individual fitness value of real-time solution achieved in the optimal iterative process. By reducing the numerical simulation computation; The genetic algorithm optimization which is used as a optimization algorithm combine with MATLAB6.5 that improve the efficiency and accuracy of computing, and the laser texturing morphology optimization program is work out, which is used to completed the algorithm solution in optimize process.

4 Optimization Experiment 1) Computing model The designed computing model is a bendou with flange, and the slab is rectangular, the length is 300mm, width is 180mm, and drawing height is 60mm, 45 materials is selected to simulate as the sheet material, the thickness of sheet metal is 1.0mm, considering the asperity of friction conditions, 0.12 is taken as lubrication friction coefficient.

Study on Optimization of the Laser Texturing Surface Morphology Parameters

1

R

ȡ

D

2

h

601

İ

30

J

Fig. 2 The model of artificial neural network

2) modeling and training the artificial neural network In this paper, three neural network is used the input layer neurons is 3, that is the micro figure radius, the density and height; the hidden layer neurons is 30; and output neurons is 3, that is the deviation, Tangential Strain and thickness uniformity, as follow fig shows: All neurons are calculated by using sigmoid function, the minimum error expectations of neural network training parameters is 0.001; the largest number of cycle is 50000; smoothing factor is 0.1, the momentum factor is 0.25; space samples are taken from the orthogonal finite element simulation results of micro figure, with a total of 25, micro figure radius (in mm) values are 0.25,0.294,0.338,0.382,0.425. Densities are 6%, 12%, 18%, 24%, 30%. Height (in mm) values are 7.2,14.2,21.2,28.2,35.2. The system error reaches 0.001 after 23546 times iteration. Another 4 data are chosen to test, and the error between numerical simulation and training network show in the table. As show in the table, the error is in 5%, and indicates that the reliability of the network model.

，

3) Genetic algorithms optimization Choose 100 as the size of the population; crossover arithmetic uses two cross-cutting, cross-probability is 0.9; mutation operator use Gaussian variation and mutation probability is 0.1; the largest genetic algebra is 100; individual replacement percentage is 25%. A set of approximate solution are received after genetic Table 1 The error between numerical simulation and training network

number 1 2 3 4

Windage D 3.08% 0.57% 0.44% 0.61%

Tangential strainε 3.2% 0.64% 0.48% 0.83%

Thickness distribution J 1.67% 0.77% 0.60% 1.02%

602

Z. Luo et al.

(a)

(b)

Fig. 3 The comparison of result between Conventional drawing and drawing with laser texturing

cycle: R = 0.382mm, ρ = 24%, h = 21.2mm. The numerical simulation is carried by using this group, and the error is controlled in 1.5 percent, of which the value of D in the gross morphology reduced 11.3 percent, and the value of ε reduces 7.3 percent, the value of J reduced 3.5 percent. The results shown in the figure using the approximate solution to experiment, the rupture and wrinkle didn’t occur in the parts, so the quality of the parts are considered to be eligible in line with the requirements.

5 Conclusion (1) In this paper, the drawing mold surface topography parameters in bendou are optimized with the methods of combining numerical simulation artificial neural network and genetic algorithm, the results show that the parts could be drawing better than the conventional thickness distribution, and strain to cut the maximum deviation value in the optimization of laser texturing micro figure radius density and height. (2) Using powerful ability to function approximation of the artificial neural networks, the relationship between the drawing sheet metal forming parameters on the surface of the mold (micro figure radius, density and height) and the quality of forming by less numerical simulation, not only to ensure the accuracy, but also reduce the number of complex numerical simulation. On the basis of the neural network modeling, the sheet metal forming parameters were optimized with the overall search capabilities of genetic algorithm. the genetic algorithm optimization has good results through examples, and show the applicability of the model.

、

、

References 1. Pan, J.F., Zhong, Y.X., Yuan, C.L.: Process Parameters Optimization for Sheet Metal Forming During Drawing with a Multi-Objective Genetic Algorithm. Journal of Tsinghua University, Science and Technology 47(8), 1267–1269 (2007)

Study on Optimization of the Laser Texturing Surface Morphology Parameters

603

2. Samya, E., Jacques, T., Bassem, B.: Genetic Algorithms to Solve the Cover Printing Problem. Computers & Operations Research 34(11), 3346–3361 (2007) 3. Carlos, C.: Theoretical and Numerical Constraint-Handling Techniques Used with Evolutionary Algorithms: a Survey of the State of the Art. Computer methods in applied mechanics and engineering 191(11), 1245–1287 (2002) 4. Dai, H., Li, Z., Xia, J.: Processing Parameter Optimization and Experimental Study on Drawing Hole Forming Zhongguo Jixie Gongcheng. China Mechanical Engineering 17(15), 1627–1634 (2006) 5. Singh, S.K., Kumar, D.R.: Application of a Neural Network to Predict Thickness Strains and Finite Element Simulation of Hydro-Mechanical Deep Drawing. International Journal of Advanced Manufacturing Technology 25(1-2), 101–107 (2005) 6. Wang, X., Cao, L.: GA Theory and Application Software. Xi’an Jiaotong University Press (2002)

A Combined Newton Method for Second-Order Cone Programming Xiaoni Chi and Jin Peng

Abstract. Based on the Fischer-Burmeister smoothing function, a combined Newton method is proposed for solving the second-order cone programming (SOCP). The algorithm combines the techniques in both non-smoothing Newton methods and smoothing Newton methods. Without any restrictions regarding its starting points, the algorithm needs to perform at most one line search at each iteration and is shown to be globally convergent. Numerical results indicate the effectiveness of our algorithm. Keywords: Second-order cone programming, Non-smoothing Newton method, Smoothing Newton method, Global convergence.

1 Introduction The second-order cone programming (SOCP) problem is to minimize or maximize a linear function over the intersection of an affine space with the Cartesian product of a finite number of second-order cones. Consider the following SOCP problem [1] ( n ) n T (P ) min ci xi : Ai xi = b, xi ∈ Ki , i = 1, . . . , n i=1

i=1

and its dual problem (D) max

" bT y : ATi y + si = ci , si ∈ Ki , i = 1, . . . , n .

Xiaoni Chi · Jin Peng College of Mathematics and Information Science, Huanggang Normal University, Huangzhou 438000, China [email protected] H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 605–612. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

606

X. Chi and J. Peng

Here Ai ∈ Rm×ki , ci ∈ Rki , i = 1, . . . , n, b ∈ Rm are the data, and xi ∈ Ki , si ∈ Ki , i = 1, . . . , n, y ∈ Rm are the variables. The set Ki (i = 1, . . . , n) is the second-order cone (SOC) of dimension ki , i.e., Ki := {xi = (xi0 , xi1 ) ∈ R × Rki −1 : xi0 − xi1  ≥ 0}, where  ·  denotes the Euclidean norm. It is not difficult to show that the SOC Ki (i = 1, . . . , n) is self-dual. Let k = k1 + · · · + kn , K = K1 × · · · × Kn , A = (A1 , · · · , An ) ∈ Rm×k , c = (c1 , · · · , cn ) ∈ Rk , x = (x1 , · · · , xn ) ∈ K, s = (s1 , · · · , sn ) ∈ K, where we use x = (x1 , · · · , xn ) for the column vector x = (xT1 , · · · , xTn )T . Thus, problems (P) and (D) can be simply written as (P ) min {cT x : Ax = b, x ∈ K},

(1)

(D) max {bT y : AT y + s = c, s ∈ K}.

(2)

Without loss of generality we may assume that n = 1 and k = k1 in the following analysis, because our analysis can be easily extended to the general case. Then the sets of strictly feasible solutions of (1) and (2) are: F 0 (P ) = {x : Ax = b, x ∈ K 0 }, F 0 (D) = {(y, s) : AT y + s = c, s ∈ K 0 } respectively, where K 0 denotes the interior of the SOC K defined as K 0 := {x = (x0 , x1 ) ∈ R × Rk−1 : x0 − x1  > 0}. Throughout this paper, we assume that F 0 (P ) × F 0 (D) = ∅. Under this assumption, it can be shown that both (1) and (2) have optimal solutions and their optimal values coincide [1]. Recently great attention has been paid to the SOCP problems, since they have a wide range of engineering applications [2]. As novel algorithms for solving optimization problems, smoothing Newton methods [3, 4, 5] perform very well in both theory and practice. However, the smoothing Newton methods available are mostly for solving complementarity problems [3, 5] and variational inequality problems [4], whereas there is little work on smoothing Newton methods for the SOCP. Moreover, some algorithms [5] need to perform two or three line searches at each iteration. In this paper, we present a combined Newton method for the SOCP. By using the techniques in both non-smoothing Newton methods and

A Combined Newton Method for Second-Order Cone Programming

607

smoothing Newton methods, we give a novel and effective way to solve the SOCP problems. The rest of this paper is organized as follows. Section 2 presents our new algorithm. In Section 3, we analyze the convergence of the algorithm. Numerical results are given in Section 4 to illustrate the effectiveness of the algorithm. Section 5 concludes this paper.

2 Algorithm Description It is well-known that solving the SOCP problems (1) and (2) is equivalent to [1] finding (x, y, s) ∈ Rk × Rm × Rk such that ⎧ x ∈ K, ⎨ Ax = b, AT y + s = c, s ∈ K, (3) ⎩ x ◦ s = 0. Here the Euclidean Jordan algebra [1, 6] for the SOC K is the algebra defined by x ◦ s = (xT s, x0 s1 + s0 x1 ), ∀x, s ∈ Rk . The element e = (1, 0, · · · , 0) ∈ Rk is the unit element of this algebra. Let φ : Rk × Rk × R → Rk denote the Fischer-Burmeister smoothing function [7] 2 φ(x, s, μ) = x + s − x2 + s2 + 2μ2 e. By Proposition 4.2 in [7], the pointwise limit φ(x, s) = limμ→0+ φ(x, s, μ) satisfies φ(x, s) = 0 ⇐⇒ x ◦ s = 0, x ∈ K, s ∈ K. Let u := (x, y, s) ∈ Rk × Rm × Rk , z := (u, μ) ∈ Rm+2k+1 and define ⎞ ⎛ Ax − b Φ(u) := ⎝ AT y + s − c ⎠ , φ(x, s)

(4)

(5)

⎛

⎞ Ax − b ⎜ AT y + s − c ⎟ ⎟ H(z) := ⎜ ⎝ φ(x, s, μ) ⎠ . μ

(6)

Let ∂Φ stand for the generalized Jacobian of Φ in the sense of Clarke [8]. On account of (3), (4) and (6), z ∗ := (x∗ , y ∗ , s∗ , 0) is a root of H(z) = 0 if and only if (x∗ , y ∗ , s∗ ) is the optimal solution of (1) and (2).

608

X. Chi and J. Peng

Algorithm 1 Step 0. Choose σ, δ, η1 , η2 ∈ (0, 1), μ0 ∈ (0, ∞) and let z = (0, μ0 ) ∈ Rm+2k+1 . Let u0 := (x0 , y0 , s0 ) ∈ Rk × Rm × Rk be an arbitrary point and z0 := (u0 , μ0 ). Choose γ ∈ (0, 1) such that γμ0 < 1. Let β0 := Φ(u0 ). Set k := 0. Step 1. If H(zk ) = 0, stop. Otherwise, let ρ(zk ) = γ min{1, H(zk )2 }. Step 2. (Non-smooth Newton Step) Choose Vk ∈ ∂Φ(uk ). If Vk is singular, go to Step 3. Otherwise, compute Δ; uk ∈ Rm+2k from Vk Δ; uk = −Φ(uk ).

(7)

If Φ(uk + Δ; uk ) = 0, stop. If the following two conditions are satisfied: Φ(uk + Δ; uk ) ≤ η1 βk ,

(8)

uk , μk ) ≤ η2 H(zk ), H(uk + Δ;

(9)

set uk+1 := uk + Δ; uk , μk+1 := μk , βk+1 := Φ(uk + Δ; uk ), k := k + 1, and go to Step 1. Otherwise, go to Step 3. Step 3. (Smoothing Newton Step) Compute a solution Δzk := (Δuk , Δμk ) ∈ Rm+2k+1 of H(zk ) + H (zk )Δzk = ρ(zk )z.

(10)

Let λk = max{δ l | l = 0, 1, 2, . . .} such that H(zk + λk Δzk ) ≤ [1 − σ(1 − γμ0 )λk ]H(zk ),

(11)

and set zk+1 := zk + λk Δzk , βk+1 := Φ(uk + λk Δuk ), k := k + 1. Go to Step 1. By following the proof of Proposition 6.2 in [7] and using Corollary 5.4 in [7], we can obtain the following properties of the function H(z) defined as in (6) to establish the well-definedness of Algorithm 1. Theorem 1. (i) The function H(z) is globally Lipschitz continuous in Rm+2k+1 . If μ > 0, H(z) is continuously differentiable with its Jacobian ⎞ ⎛ A 0 0 0 ⎜ 0 AT I 0 ⎟ ⎟ H(z) = ⎜ ⎝ M (z) 0 N (z) P (z) ⎠ , 0 0 0 1 where −1 −1 M (z) = I − L−1 w Lx , N (z) = I − Lw Ls , P (z) = −2μLw e,

A Combined Newton Method for Second-Order Cone Programming

2 w := x2 + s2 + 2μ2 e,

 Lx :=

x0 xT1 x1 x0 I

609

 .

(ii) If A has full row rank, H (z) is nonsingular for any μ > 0. By Theorem 1(i), is is not difficult to show that if A has full row rank Algorithm 1 is well-defined.

3 Convergence Analysis Lemma 1. Suppose that A has full row rank and that {zk = (uk , μk )} is the iteration sequence generated by Algorithm 1. If Φ(uk ) = 0 for all k ≥ 0, we have that {zk } is an infinite iteration sequence, {μk } is monotonically decreasing and zk ∈ Ω for any k ≥ 0, where " Ω = z = (x, y, s, μ) ∈ Rm+2k+1 : μ ≥ ρ(z)μ0 . Proof. It is not difficult to show that if Φ(uk ) = 0 for all k ≥ 0, {zk } is an infinite iteration sequence with μk > 0. Next we prove zk ∈ Ω for any k ≥ 0 by induction on k. It is obvious that z0 ∈ Ω, because ρ(z0 ) ≤ γ < 1. Suppose that zk ∈ Ω. On account of Algorithm 1, the iteration sequence {zk } includes {z;k } generated by Step 2 and {zk } generated by Step 3. If the sequence {zk } is infinite, we prove zk+1 ∈ Ω and {μk } is monotonically decreasing by following the proof of Proposition 8 in [4] and using the relation (11). If the sequence {z;k } is infinite, we prove zk+1 ∈ Ω by considering the following three cases. Case (i) If H(zk ) > 1 and H(zk+1 ) > 1, we have ρ(zk+1 )μ0 = γμ0 = ρ(zk )μ0 ≤ μk = μk+1 . Case (ii) If H(zk ) > 1 and H(zk+1 ) ≤ 1, we obtain ρ(zk+1 )μ0 = γμ0 H(zk+1 )2 ≤ γμ0 = ρ(zk )μ0 ≤ μk = μk+1 . Case (iii) If H(zk ) ≤ 1, it follows from (9) that ρ(zk+1 )μ0 = γμ0 H(zk+1 )2 ≤ γμ0 H(zk )2 = ρ(zk )μ0 ≤ μk = μk+1 . Hence we have that zk ∈ Ω for any k ≥ 0 and {μk } is monotonically decreasing. ! Theorem 2. Suppose that A has full row rank and that z ∗ = (u∗ , μ∗ ) is an accumulation point of the iteration sequence {zk = (uk , μk )} generated by Algorithm 1. If Φ(uk ) = 0 for all k ≥ 0, at least one of Φ(u∗ ) = 0 and H(z ∗ ) = 0 holds. Proof. Since Φ(uk ) = 0 for all k ≥ 0, we have H(zk ) = 0 for all k ≥ 0. Thus {zk } is an infinite iteration sequence, which includes {z;k } generated by

610

X. Chi and J. Peng

Step 2 and {zk } generated by Step 3. We consider the following two cases separately. Case (i) If the sequence {zk } is finite, the sequence {z;k } is infinite. Then by Step 2 there exists a sufficiently large k0 such that zk = z;k for any k > k0 . Therefore, we have from (8) that Φ(uk ) ≤ η1k−k0 βk0 for any k > k0 , which implies Φ(u∗ ) = 0. Case (ii) If the sequence {zk } is infinite, by taking a subsequence if necessary, suppose that {zk } converges to z ∗ = (u∗ , μ∗ ) as k → ∞. In view of (9), (11) and Lemma 1 that the two sequences {H(zk )} and {μk } are monotonically decreasing. Then {H(zk )} and {μk } converge to H(z ∗ ) and μ∗ respectively as k → ∞. Furthermore, {H(zk )}, {μk } and {ρk } converge to H(z ∗ ), μ∗ and ρ∗ := γ min{1, H(z ∗)2 } respectively. If H(z ∗ ) = 0, we obtain the desired result. On the contrary, suppose that H(z ∗ ) > 0. It follows from Lemma 1 that 0 < ρ∗ μ0 ≤ μ∗ . Thus by (11) and Theorem 1, we have lim λk = 0, lim H(zk ) = H(z ∗ ), lim H (zk ) = H (z ∗ ).

k→∞

k→∞

k→∞

On the one hand, the Armijo condition (11) is not satisfied for λk := λk /δ by Step 3 in Algorithm 1, i.e., H(zk + λk Δzk ) − H(zk ) > −σ(1 − γμ0 )H(zk ). λk Taking the limit k → ∞ in the above inequality and using λk → 0, we obtain H(z ∗ )T H (z ∗ )Δz ∗ ≥ −σ(1 − γμ0 )H(z ∗ )2 .

(12)

On the other hand, we have from (10) H(z ∗ )T H (z ∗ )Δz ∗ = −H(z ∗ )2 + ρ∗ H(z ∗ )T z ≤ −H(z ∗)2 + γμ0 H(z ∗)2 . Substituting the last relation into (12) yields (1 − σ)(1 − γμ0 ) ≤ 0, which contradicts the fact σ ∈ (0, 1), γμ0 < 1. Hence we obtain H(z ∗ ) = 0. !

4 Numerical Results We implemented Algorithm 1 in MATLAB 7.0.1 to see the performance of the combined Newton method for the SOCP. All the experiments were performed on a Intel(R) Pentium(R) 4 CPU 3.00 GHz and 512 MB memory desktop computer under Windows XP.

A Combined Newton Method for Second-Order Cone Programming

611

Table 1 Performances of Algorithm 1 on SOCPs Algorithm 1 m

k

iter

cpu(s)

10

20

3

0.031

25

50

3

0.032

50

100

3

0.125

75

150

3

0.234

100

200

4

0.765

125

250

3

1.122

We use randomly generated problems with size n = 1 and k(= 2m) from 20 to 250. Let x0 = 0 ∈ Rk , y0 = 0 ∈ Rm , s0 = c be initial points. The parameters used in Algorithm 1 were as follows: σ = 0.25, δ = 0.95, μ0 = 0.75, η1 = 0.8, η2 = 0.99 and γ = 0.99. We used H(z) ≤ 10−6 as the stopping criterion. Table 1 shows the effectiveness of Algorithm 1.

5 Conclusions By using the techniques in both non-smoothing Newton methods and smoothing Newton methods, we propose a combined Newton method for the SOCP in this paper. Based on the Fischer-Burmeister smoothing function, our algorithm is shown to possess the following good properties: (i)our algorithm needs to perform at most one line search at each iteration; (ii) the algorithm does not have any restrictions regarding its starting points; (iii)our algorithm is shown to be globally convergent. Acknowledgements. This work is supported by the National Natural Science Foundation (Grant No. 70671050), the Major Research Program (Grant No. Z20082701) and the Group Innovation Project of Hubei Provincial Department of Education, the Excellent Youth Project of Hubei Provincial Department of Education, and the Doctorial Foundation of Huanggang Normal University (No. 08cd158), China.

References 1. Alizadeh, F., Goldfarb, D.: Second-order cone programming. Math. Program. 95, 3–51 (2003) 2. Lobo, M.S., Vandenberghe, L., Boyd, S., Lebret, H.: Applications of second-order cone programming. Linear Algebra Appl. 284, 193–228 (1998)

612

X. Chi and J. Peng

3. Chen, B., Xiu, N.: A global linear and local quadratic non-interior continuation method for nonlinear complementarity problems based on Chen-Mangasarian smoothing functions. SIAM J. Optim. 9, 605–623 (1999) 4. Qi, L., Sun, D., Zhou, G.: A new look at smoothing Newton methods for nonlinear complementarity problems and box constrained variational inequalities. Math. Program. 87, 1–35 (2000) 5. Tseng, P.: Analysis of a non-interior continuation method based on ChenMangasarian smoothing functions for complementarity problems. In: Reformulation –Nonsmooth, Piecewise Smooth, Semismooth and Smoothing Methods, pp. 381–404. Kluwer Academic Publishers, Boston (1999) 6. Faraut, J., Kor´ anyi, A.: Analysis on Symmetric Cones. Oxford University Press, Oxford (1994) 7. Fukushima, M., Luo, Z.Q., Tseng, P.: Smoothing functions for second-order-cone complementarity problems. SIAM J. Optim. 12, 436–460 (2001) 8. Clarke, F.H.: Optimization and nonsmooth analysis. John Wiley and Sons, New York (1983)

MES Scheduling Optimization and Simulation Based on CAPP/PPC Integration Yan Cao, Ning Liu, Lina Yang, and Yanli Yang*

Abstract. Confronted with the complexity of manufacturing environment, manufacture system, and manufacturing process, how to abase this complexity is an important problem to realize production scheduling and control effectively. Thereinto, CAPP and PPC are two main modules involved in production activities. Firstly, the existing approaches of CAPP/PPC integration are analyzed. Then, a conceptual model for concurrent distributed integration of CAPP and PPC is presented. The model can achieve the integration of CAPP and PPC and solve the production planning and control in a virtual enterprise. According to the conceptual model of the functional integration of CAPP/PPC, a method to realize the functional integration of CAPP/PPC is discussed. And then, according to the requirements of single piece and small batch production, MES scheduling optimization can be realized based on CAPP/PPC Integration. The structure of the scheduling sub-system of MES is also presented that is consisted of three core modules, namely operation planning, operation scheduling, and material tracing module. Finally, based on Witness, a simulation model of a specific workshop is established. Through the comparison of scheduling schemes, the optimal MES scheduling scheme is obtained. Keywords: CAPP, PPC, MES, Scheduling, Integration, Simulation. Yan Cao Advanced Manufacturing Engineering Institute, School of Mechatronic Engineering, Xi’an Technological University, Xi’an 710032, China [email protected]

*

Ning Liu China First Heavy Industries, Fularji 161042, China Lina Yang Xi’an University of Science and Technology, Xi’an 710054, China Yanli Yang Shenzhen University, Shenzhen 518060, China H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 613–622. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

614

Y. Cao et al.

1 Introduction Confronted with the complexity of manufacturing environment, manufacture system, and manufacturing process, how to abase the complexity is an important problem to realize workshop production scheduling and control effectively. Production scheduling and decision-making are a multiple objective optimization problem where local decision-making and optimization exist at each phase of production scheduling and control [1, 2]. The gap of CAPP and PPC often suffers from stochastic resource bottlenecks, non-availability of tools and personnel, breakdown of machine tools, and so on. The reasons that lead to the above phenomena are as follows: • Process planners assume unlimited and idle resources in a workshop. • CAPP and PPC have conflicting objectives. CAPP emphasizes the technological requirements of products, while PPC mainly considers the products’ time to market. • Not considering the dynamic production environment, the objective time delay between the production planning and its execution phase is inevitable. • CAPP may not consider the dynamic nature of workshop production scheduling and control. Hence, it can not predict the load status of the resources in a workshop. In recent years, with the development of information technology and the request of agile production, Manufacturing Execution System (MES), which is a management information system at the workshop layer, becomes a bridge that spans and fills up the gap between the upper layer of plan management and the ground floor layer of industrial control in an enterprise [3, 4, 5, 6]. Workshop scheduling is the core module of MES, which directly affects efficiency, operation, and management in workshop production. Effective scheduling methods can improve benefit and reduce cost [7, 8, 9, 10]. Moreover, MES scheduling performance can be greatly improved through CAPP/PPC integration.

2 CAPP/PPC Integration Approaches The integration of CAPP and workshop scheduling arose in the middle of the 1980s. After about ten years, the ideas of the integration of CAPP and PPC were suggested. So far, several approaches for integration of CAPP and PPC have been discussed in terms of nonlinear process planning, flexible process planning, closed loop process planning, dynamic process planning, alternative process planning, as well as just-in-time process planning. According to the review and analysis of references, there are mainly three types of general integration approaches.

MES Scheduling Optimization and Simulation Based on CAPP/PPC Integration

615

• Interfaced integration. • Modular integration. • Unified integration.

3 A Model of Concurrent Distributed CAPP/PPC Integration Research on integration of CAPP and PPC must be fulfilled from the point of view of enterprise integration. In the paper, a model of concurrent distributed CAPP/PPC integration is constructed. The structure is illustrated in Fig. 1. Hereinto, FU represents functional unit.

FU of alternative routes

CAPP

FU of space

FU of time

PPS

Fig. 1 A model of concurrent distributed integration

4 A Hierarchical Distributed Resource Model of a Virtual Enterprise A hierarchical distributed resource model of a virtual enterprise is shown in Fig. 2.

5 Realization Method of CAPP/PPC Integration The flow chart of the functional integration of CAPP/PPC is shown in Fig. 3. As capacity unbalance leads to long lead-time, high levels of work-in-process (WIP), as well as rising cost, it affects the synthetic objective to minimize cost and lead time.

616

Fig. 2 A hierarchical resource distribution model of a virtual enterprise

Y. Cao et al.

MES Scheduling Optimization and Simulation Based on CAPP/PPC Integration

617

Fig. 3 CAPP/PPC integration flow chart

6 MES Scheduling Optimization Based on CAPP/PPC Integration According to the requirements of single piece and small batch production, MES scheduling optimization can be realized based on CAPP/PPC Integration. Consequently, the information from MES can reflects production changes in time. And, production planning can responds to the changes rapidly. The data flow among production planning, MES, and production control is shown in Fig. 4.

618

Y. Cao et al.

Production planning Product demand BOM Order sheet

Task status System status

……

……

MES

Production plan Analysis report Inventory

Production progress Equipment status

……

……

Production control

Fig. 4 The data flow among production planning, MES, and production control

7 MES Scheduling Sub-system The scheduling sub-system of MES is consisted of three core modules, namely operation planning, operation scheduling, and material tracing module, as shown in Fig. 5. Operation scheduling adopts optimization algorithms, such as intelligent algorithm, heuristic algorithm, and so on.

8 Workshop Simulation for MES Scheduling Optimization Based on Witness In the paper, workshop operation simulation is adopted to seek the optimal MES scheduling scheme. Based on Witness, a simulation model of workshop operations is established. Through comparison of scheduling schemes, workshop operations are optimized. In a workshop where there are five groups of machines, three products are to be machined. The products need four, three, and five working procedures respectively. The initial simulation model is shown in Fig. 6. Its statistic information is shown in Table 1-3. Table 1 Product statistic information

Name A B C

No. Entered 3466 5940 2292

No. Shipped 3417 5889 2241

W.I.P. 49 51 51

Avg W.I.P. 22.03 47.57 27.86

Avg Time 1113.59 1403.09 2129.43

MES Scheduling Optimization and Simulation Based on CAPP/PPC Integration

Fig. 5 The functional modules of MES scheduling sub-system

619

620

Y. Cao et al.

Fig. 6 The initial simulation model Table 2 Machine statistic information

Name Machine1 Machine2 Machine3 Machine4 Machine5

% Idle 3.98 4.94 27.58 2.08 21.22

% Busy 96.02 95.06 72.42 97.92 78.78

No. Of Operations 11616 5682 11595 8144 5672

Table 3 Buffer statistic information

Name

Total In

Total Out

Now In

Max

Min

Avg Size

Avg Time

Buffers1 Buffers2 Buffers3 Buffers4 Buffers5

11620 5757 11607 8194 5682

11619 5684 11599 8147 5673

1 73 8 47 9

73 91 30 142 35

0 0 0 0 0

18.65 17.46 1.39 44.25 4.31

281.15 531.47 20.91 946.17 132.78

Table 4 Product statistic information after optimization

Name A B C

No. Entered 3409 5813 2286

No. Shipped 3403 5812 2277

W.I.P. 6 1 9

Avg W.I.P 4.90 6.77 4.91

Avg Time 251.66 204.17 376.28

MES Scheduling Optimization and Simulation Based on CAPP/PPC Integration

621

The system model after optimization is shown in Fig. 7. The statistic information after model optimization is shown in Table 4-6.

Fig. 7 The system model after optimization Table 5 Machine statistic information after optimization

Name Machine1 Machine2 Machine3 Machine4 Machine5

% Idle 29.03 36.04 28.37 27.37 22.80

% Busy 70.97 63.96 71.63 72.63 77.20

No. Of Operations 11497 5686 11496 8091 5684

Table 6 Buffer statistic information after optimization

Name Buffer1 Buffer2 Buffer3 Buffer4 Buffer5

Total In 11501 5693 11499 8091 5686

Total Out 11501 5689 11499 8091 5685

Now In 0 4 0 0 1

Max 14 15 15 18 21

Min 0 0 0 0 0

Avg Size 0.75 0.74 0.84 1.00 1.95

Avg Time 11.37 22.90 12.82 21.64 60.08

9 Conclusions According to the requirements of single piece and small batch production, a model for CAPP/PPC integration is presented, based on which MES scheduling performance can be optimized. Although the emphasis is focused on the application to a virtual enterprise, the approach can also be applied to other manufacturing enterprises. A scheduling sub-system of MES is presented that is consisted of three core modules, namely operation planning, operation scheduling, and material

622

Y. Cao et al.

tracing module. Based on Witness, a simulation model for a specific workshop is established. Through comparison of scheduling schemes, workshop operations are optimized. The future researches are mainly focused on intelligent scheduling algorithms to deal with the complexity and incidents in practical production more effectively. What is more, the functions of the scheduling sub-system of MES need to be perfected and extended. Acknowledgments. The paper is supported by Shaanxi Major Subject Construction Project and President Fund of Xi’an Technological University.

References 1. Zhang, X.D., Yan, H.S.: Integrated Optimization of Production Planning and Scheduling for Multi-Stage Workshop. Chinese Journal of Mechanical Engineering 41, 98– 105 (2005) 2. He, Y., Liu, F., Shi, J.L.: A Framework of Scheduling Models in Machining Workshop for Green Manufacturing. Journal of Advanced Manufacturing Systems 7, 319–322 (2008) 3. Wei, Y., Li, D.B., Yuan, M.H.: Research on Manufacturing Execution System for Agile Assembling Resource Reconfiguration Based on Agent. Machine Tool & Hydraulics 36, 44–47, 52 (2008) 4. Wang, Q.F., Liu, F., Huang, H.L.: Service-Oriented Reconfigurable Manufacturing Execution System for Discrete Workshop. Computer Integrated Manufacturing Systems 14, 737–743 (2008) 5. Pan, Y., Zhang, W.X.: Research on Multi-Agent-Based MES Structure of Discrete Manufacturing Industry. Application Research of Computers 26, 244–246, 249 (2009) 6. Cheng, Z.L., Fan, Y.Q.: Design of component-based flexible MES in metallurgical industry. Computer Integrated Manufacturing Systems 13, 490–496 (2007) 7. Zhou, W.K., Zhu, J.Y.: Application of Workflow Technology for Workshop Scheduling. Journal of Harbin Institute of Technology (New Series) 12, 105–110 (2005) 8. Chu, H.Y., Cao, Q.J., Fei, R.Y.: Production Scheduling Research for Workshop Based on Manufacturing Cell. Journal of Beijing University of Technology 32, 730–736 (2006) 9. Liu, M., Yan, J.W.: An Adaptively Annealing Genetic Algorithm Based Scheduling Method of Workshop Daily Operating Planning. Chinese Journal of Computers 30, 1164–1172 (2007) 10. Yu, X.Y., Sun, S.D., Chu, W.: Parallel Cooperative Evolutionary Genetic Algorithm for Multi-Workshop Planning and Scheduling Problems. Computer Integrated Manufacturing Systems 14, 991–1000 (2008)

An Improved Diversity Guided Particle Swarm Optimization Dongsheng Xu and Xiaoyan Ai*

Abstract. Particle swarm optimization (PSO) is a new population based stochastic search algorithm, which has shown good performance on well-known numerical test problems. However, on strongly multimodal test problems the PSO easily suffers from premature convergence. In this paper, an improved diversity guided PSO is proposed, namely IARPSO, which combines a diversity guided PSO (ARPSO) and a Cauchy mutation operator. The purpose of IARPSO is to enhance the global search ability of ARPSO by Conducting a Cauchy mutation on the global best particle. Experimental results on 6 multimodal functions with many local minima show that the IARPSO outperforms the standard PSO, ARPSO and ATRE-PSO on all test functions. Keywords: Particle swarm optimization (PSO), Diversity, Cauchy mutation, Function optimization.

1 Introduction Particle swarm optimization is a population based stochastic search technique first introduced by Kennedy and Eberhart in 1995 [1]. Since then it has been used to solve many optimization problems. Its performance has been compared with stochastic search algorithms such as simulated annealing (SA) and genetic algorithm (GA) [2]. Although PSO has shown good performance in many optimization problems, it easily suffers from local optima in multimodal optimization problems. A major problem with PSO in multimodal optimization is premature convergence, which results in great performance loss and sub-optimal solutions. In order to overcome the drawback of PSO, many techniques have been developed to improve its performance, such as ARPSO [3], ATRE-PSO [4] etc. Riget [3] has proposed a diversity-guided PSO, called ARPSO, which uses a diversity measure to control the search of swarm. The ARPSO defines an attraction phase and a repulsion phase. The former exists in the standard PSO algorithm, and the latter modifies the velocity updating formula of the standard PSO. In the Dongsheng Xu . Xiaoyan Ai Department of Information Technology, Yulin University, Yulin 719000, China [email protected], [email protected] *

H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 623–630. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

624

D. Xu and X. Ai

attraction phase the swarm is contracting, and consequently the diversity decreases. When the diversity drops below a predefined constant number dlow, the repulsion mechanism is applied for expanding swarm. Finally, when the diversity reaches to another predefined constant number dhigh, the ARPSO switches back to the attraction phase. The experimental results have shown that the ARPSO prevents premature convergence to a high degree. On the basis of ARPSO, Pant [4] has presented ATRE-PSO, in which a middle phase between attraction and repulsion is introduced. In ARPSO if the diversity is above dhigh then particles attract each other, and if it is below dlow, then the particles repel each other until they meet the required diversity dhigh. When the diversity is lying in between dlow and dhigh, the ATRE-PSO employs a middle phase called the phase of positive conflict. In this phase there is neither a complete attraction nor a complete repulsion. Each particle in swarm is attracted by its own previous best particle and is repelled by the global best particle. In this way the diversity may lie in a balance level, because there is neither total attraction nor total repulsion but a middle phase between the two. The simulation results have shown the middle phase could improve the exploring and exploiting abilities of ARPSO. In this paper, an improved ARPSO is proposed, namely IARPSO, which combines the ARPSO and a Cauchy mutation operator conducted on the global best particle. The Cauchy mutation is more likely to generate a larger jump than Gaussian mutation, which could help the global best particle jump out local optima. Experimental results on 6 well-known multimodal functions with many local minima show that the IARPSO outperforms the standard PSO, ARPSO and ATRE-PSO. The rest of the paper is organized as follows. In Section 2, the standard PSO, ARPSO, ATRE-PSO, and our proposed algorithm IARPSO are introduced. In Section 3, the benchmark functions, parameter settings, experimental results and discussions are presented. Conclusions as well as further works are given in Section 4.

2 An Improved Diversity Guided PSO (IARPSO) 2.1 The Standard PSO Like other evolutionary algorithms, PSO is also a population-based search algorithm and starts with an initial population of randomly generated solutions called particles [5]. Each particle in PSO has a velocity and a position. PSO remembers both the best position found by all particles and the best positions found by each particle in the search process. For a search problem in an n-dimensional space, a particle represents a potential solution. The velocity vij and position xij of the jth dimension of the ith particle are updated according to Eqs. (1) and (2): v (t + 1) = w ⋅ v (t ) + c1 ⋅ rand 1 ⋅ ( pbestij ( t ) − x (t )) + c2 ⋅ rand 2ij ⋅ ( gbest (t ) − x (t )) ij ij ij ij j ij

(1)

An Improved Diversity Guided Particle Swarm Optimization

625

xij (t + 1) = xij (t ) + vij (t + 1)

(2)

where i = 1,2,…, is the particle’s index, Xi = (xi1, xi2,…,xin) is the position of the ith particle; Vi = (vi1, vi2,…, vin) represents velocity of particle i. pbesti = (pbesti1, pbesti2,…, pbestin) is the best previous position yielding the best fitness value for the ith particle; and gbest = (gbest1, gbest2,…., gbestn) is the global best particle found by all particles so far. The inertia factor w was proposed by Shi and Eberhart [6], rand1ij and rand2ij are two random numbers independently generated within the range of [0,1], c1 and c2 are two learning factors which control the influence of the social and cognitive components, and t = 1,2,…, indicates the iterations.

2.2 The ARPSO Riget [3] has presented a diversity-guided PSO (ARPSO) which defines a repulsion phase based on a modified velocity updating model. The basic PSO algorithm only employs an attraction phase which attracts each particle. When the diversity in swarm drops below a predefined constant number dlow, the ARPSO switches to the repulsion phase, in which particles repel each other due to the modified velocity updating formula, and then the diversity increases. When the diversity reaches to another predefined constant number dhigh, the ARPSO switches back to the attraction phase. The switching search model is defined as [3]: v (t + 1) = w ⋅ v (t ) + dir ⋅ [c1 ⋅ rand 1 ⋅ ( pbestij (t ) − x (t )) + c2 ⋅ rand 2ij ⋅ ( gbest (t ) − x (t ))] ij ij ij ij j ij

(3)

where the sign dir satisfies:

⎧if (dir > 0 & & diversity < dlow) ⎨ ⎩if (dir < 0 & & diversity > dhigh)

dir = −1; dir = 1;

, dir is set to 1 at the beginning and

the diversity is computed as: diversity ( t ) =

S ⋅ ∑ S ⋅ L i =1 1

(

)

2 n ∑ x ij ( t ) − x j ( t ) j =1

S 1 x j (t ) = ⋅ ∑ x ij ( t ) S i =1

(4)

(5)

where |S| is the swarm size, |L| is the length of longest diagonal in the search space, n is the dimensional size, t = 1,2,…, indicates the iterations, xij is the jth value of the ith particle and x j is the jth value of the average point all particles in swarm.

2.3 The ATRE-PSO In the ARPSO [4], particles alternates attraction and repulsion phase when the diversity is above dhigh or below dlow. The diversity < dlow and diversity > dhigh may

626

D. Xu and X. Ai

not be the only two possibilities for deciding the search model of the swarm, but many times the diversity may lie in between dlow and dhigh. For this reason, Pant proposes an improved ARPSO algorithm, namely ATRE-PSO, which introduces a middle phase called the phase of positive conflict in the ARPSO. In the middle phase there is neither a complete attraction nor a complete repulsion. Each particle in swarm is attracted by its own previous best particle and is repelled by the global best particle. The middle phase is defined as [4]: v (t + 1) = w ⋅ v (t ) + c1 ⋅ rand 1 ⋅ ( pbestij (t ) − x ( t )) − c2 ⋅ rand 2ij ⋅ ( gbest (t ) − x (t )) ij ij ij ij j ij

(6)

2.4 Our Proposed Method IARPSO Although ARPSO enhances the diversity of swarm and prevents premature convergence in a certain extent, it still suffers from local optima in some multimodal functions. In this paper, an improved ARPSO is proposed, called IARPSO, which employs a Cauchy mutation operator. It is to hope that the long jump from the Cauchy mutation on the global best particle could improve the global search ability of ARPSO. The motion of particles is dominated by its previous best particles and the global best particle. So the changes of the global best particle will influence the movement of particles. In the IARPSO, a random number generated by a Cauchy mutation operator is added to the global best particle every generation. Such slight changes on the global best particle extend its search space, and may help particles jump to a better position and escape from local optima finally for its long flat tails. Some studies [7–8] have explained why Cauchy mutation performs better than traditional Gaussian mutation for most function optimization problems. The Cauchy mutation is more likely to generate larger jumps than Gaussian mutation and thus better in success. The Cauchy mutation operator is defined by [8]: gbest (t + 1) = gbest (t ) + cauchy ()

(7)

where cauchy() is a random number generated by a Cauchy distribution with a scale parameter t =1. The main steps of IARPSO are given in Table 1.

3 Experimental Studies on IARPSO 3.1 Test Functions 6 well-known multimodal functions with many local minima used in [7] have been chosen in our experimental studies. All the functions used in this paper are to be minimized. The description of the benchmark functions and their global optimum(s) are listed in Table 2.

An Improved Diversity Guided Particle Swarm Optimization

627

Table 1 The main steps of IARPSO

Begin PopSize = population size; P = current population; eval = the number of evaluations; MaxEvaluation = the maximum number of evaluations; diversity = the swarm diversity; while (eval < MaxEvaluation) Select a search model according to equation (3); for i = 1 to PopSize Calculate the velocity of particle Pi according to the selected search model; Update the position of particle Pi according to equation (2); Calculate the fitness value of particle Pi ; for end Mutate gbest according to equation (7); If gbest(t+1) is better than gbest(t) Update gbest with gbest(t+1) If end Update pbest, gbest in P if needed; Calculate the swarm diversity according to equation (4); eval++; while end End

Table 2 The 6 multimodal test functions used in our experimental studies, where n is the dimension of the functions, fmin is the minimum values of the function, and X ⊆ Rn is the search space Test Functions f1 = ∑in=1 − xi ⋅ sin

( ) xi

2 f2 = ∑in=1 [ xi − 10 cos(2π xi ) + 10] f3

=

− 2 0

− ex p

f4 =

⋅

ex p

⎛ ⎜ ⎝

1 4000

π

1 n

⎛ ⎜ ⎝

− 0 .2

⋅

1 n

∑ in = 1 c o s

(2 π

2 ∑ in = 1 x i

xi

2 ∑ in= 1 x i − ∏ in= 1 co s(

(

) ⎞⎟ ⎠

xi

+

⎞ ⎟ ⎠

2 0 +

) +1

i

⎪

u ( x , a , k , m ) = ⎨0, i

(

X

30

[–500, 500]

fmin

30

[–5.12, 5.12]

0

30

[–32, 32]

0

30

[–600, 600]

0

30

[–50, 50]

0

30

[–50, 50]

–12569.5

e

2 2 2 10 sin ( π y ) + ∑ in=−11 ( y -1) [1 + sin( π y )] + ( y -1) 1 i i +1 n nn 1 + ∑ i =1 u ( x ,10 ,100 ,4 ) , y = 1 + ( x + 1) i i 4 i ⎧ m ⎪ k ( xi − a ) , xi > a

f5 ( x ) =

n

)

− a ≤ xi ≤ a

⎪ ⎪⎩ k ( − xi − a ) m , xi < − a

f 6 ( x ) = 0 .1 sin 2 ( 3 π x1 ) + ∑ in=−11 ( x i -1) 2 [1 + sin 2 ( 3 π x i + 1 )] + 2 2 ( x n - 1) [(1 + sin ( 2 π x n )] + ∑ in= 1 u ( x ,5 ,1 0 0 ,4 ) i

)

–1.1428

628

D. Xu and X. Ai

3.2 Parameter Settings There are four variant PSO algorithms including the proposed IARPSO used in the following experiments. The algorithms and parameters settings are listed below: z z z z

The standard PSO (PSO); ARPSO [3]; ATRE-PSO [4]; Our approach (IARPSO).

For the standard PSO, w = 0.72984, c1 = c2 = 1.49618, and the maximum velocity Vmax is set to the half range of the search space on each dimension. For ARPSO, ATRE-PSO and IARPSO, w is linearly decreasing from 0.9 to 0.4, c1 = c2 = 2.0, dlow = 5.0e–6 and dhigh = 0.25 by the suggestions of [3–4]. For all algorithms, the population size is set to 10 and the maximum number of evaluations is set to 100,000 [8]. All the experiments in this paper are conducted 30 times with different random seeds, and the average results throughout the optimization runs are recorded.

3.3 Experimental Results and Discussions The results of comparison among PSO, ARPSO, ATRE-PSO and IARPSO are given in Table 3, where “Mean” indicates the mean best function values found in the last generation, and “Dev” stands for the standard deviation. The convergence characteristics in terms of the best fitness value of the median run of each algorithm for each test function are presented in Figure 1. From the above results, IARPSO achieve better performance than the standard PSO and ATRE-PSO on all test functions. IARPSO outperforms ARPSO on functions 1, 2, 3 and 4. On functions 5 and 6, ARPSO and IARPSO nearly have the same performance. Especially on function 1, the IARPSO shows good performance in global search ability, even though the test function has many local minima. That can be contributed to the Cauchy mutation, which is more likely to generate larger jump than Gaussian mutation and help the global best particle escape from Table 3 The comparison results among PSO, ARPSO, ATRE-PSO and HPSO PSO

F

ARPSO

ATRE-PSO

IARPSO

Mean

Dev

Mean

Dev

Mean

Dev

Mean

Dev

f1

–6093

462

–7259

568

–6508

763

–11055

324

f2

69.65

23.7

42.78

9.56

38.8

14.3

37.8

10.6

f3

6.42

3.51

1.47e–12

1.91e–13

2.74e–03

1.04e–03

9.84e–13

8.23e–13

f4

0.16

0.70

4.43e–02

3.06e–02

6.13e–02

4.27e–02

1.23e–02

5.37e–02

f5

1.25

0.79

3.02e–17

4.37e–25

2.18e–07

2.45e–07

3.02e–17

7.61e–26

f6

1.49

5.06

–1.142

4.63e–03

–1.139

3.14e–02

–1.142

3.79e–03

An Improved Diversity Guided Particle Swarm Optimization

f1

f2

f3

f4

f5

f6

629

Fig 1 The performance comparisons among PSO, ARPSO, ATRE-PSO and IARPSO

local optima. From the results of Figure 1, IARPSO converges faster than the other three PSO algorithms on all test functions.

4 Conclusions The purpose of IARPSO is to improve the global search ability of ARPSO by combining a Cauchy mutation operator. It is to hope that the long jump generated by the Cauchy mutation could help particles jump out local optima. Experimental studies on 6 well-known multimodal functions with many local minima show that the IARPSO outperforms PSO, ARPSO and ATRE-PSO on all test functions. However, there are still fewer cases where IARPSO falls in the local optima as what happens on function 2. It suggests that the proposed method might not enough to prevent from premature convergence in that case. Future work will focus on improving the performance of ARPSO further.

630

D. Xu and X. Ai

References 1. Kennedy, J., Eberhart, R.C.: Particle Swarm Optimization. In: Proceedings of IEEE International Conference on Neural Networks, pp. 1942–1948 (1995) 2. Eberhart, R.C., Shi, Y.: Comparison Between Genetic Algorithms and Particle Swarm Optimization. In: Porto, V.W., Waagen, D. (eds.) EP 1998. LNCS, vol. 1447, pp. 69– 73. Springer, Heidelberg (1998) 3. Riget, J., Vesterstom, J.S.: A Diversity-guided Particle Swarm Optimizer – the arPSO. Technical report, EVAlife, Denmark (2002) 4. Pant, M., Radha, T., Singh, V.P.: A Simple Diversity Guided Particle Swarm Optimization. In: Proceedings of Congress Evolutionary Computation, pp. 3294–3299 (2007) 5. Hu, X., Shi, Y., Eberhart, R.C.: Recent Advance in Particle Swarm. In: Proceedings of Congress Evolutionary Computation, pp. 90–97 (2004) 6. Shi, Y., Eberhart, R.C.: A Modified Particle Swarm Optimization. In: Proceedings of Congress Evolutionary Computation, pp. 69–73 (1998) 7. Yao, X., Liu, Y., Lin, G.: Evolutionary Programming Made Faster. IEEE Trans. Evol. Comput. 3, 82–102 (1999) 8. Wang, H., Liu, Y., Li, C.H., Zeng, S.Y.: A Hybrid Particle Swarm Algorithm with Cauchy Mutation. In: Proceedings of IEEE Swarm Intelligence Symposium, pp. 356– 360 (2007)

Research on Intelligent Diagnosis of Mechanical Fault Based on Ant Colony Algorithm Zhousuo Zhang, Wei Cheng, and Xiaoning Zhou*

Abstract. Ant colony algorithm is an evolutionary optimization algorithm that simulates the foraging behavior of ant in nature, and it is distributed, parallel, robust and based on positive feedback. Basic principle of ant colony algorithm is introduced, and an adaptive clustering algorithm based on multi-ants parallel mechanism is constructed in this paper. The multi-ants parallel and adaptive clustering algorithm is applied to fault classification of locomotive wheel-paired bearings, and the accuracy rate of classification is 87%. Research results show the algorithm is effective on practical fault diagnosis. Keywords: Intelligent diagnosis, Ant colony algorithm, Supervised learning, Unsupervised learning, Clustering.

1 Introduction Ant colony algorithm was firstly proposed by Dorigo M in 1991 [1], and then it was used in dealing with the traveling salesman problem (TSP) successfully [2]. And now it is widely used in clustering [3, 4] and feature extraction [5, 6]. As an evolutionary optimization algorithm, ant colony algorithm (ACO) has the advantages of positive feedback, distributed and parallel computing, robustness and so on [7, 8]. Therefore, it is promising to construct mechanical fault clustering models by combining ant colony algorithm and clustering methods for fault classification and recognition in the condition of lack of prior information. For example, the categories of training samples are unknown. This paper firstly introduces the basic principle of ant colony algorithm, and takes pheromone concentration as the main basis of classifying samples to construct a multi-ants parallel and adaptive clustering model base on ant colony Zhousuo Zhang . Wei Cheng . Xiaoning Zhou School of Mechanical Engineering, State Key Laboratory for Manufacturing Systems Engineering, Xi’an Jiaotong University, Xi’an 710049, China [email protected], [email protected], [email protected] *

H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 631–640. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

632

Z. Zhang, W. Cheng, and X. Zhou

algorithm and clustering algorithm. In the chapter 3 we give a detailed analysis and calculation process of the model. And in the chapter 4 we apply the clustering method to fault classification of locomotive wheel set bearings.

2 Basic Principles of Ant Colony Algorithm bi (t ) represent the number of ants that located in element i at time t , τ ij (t ) represent the information contents of path (i, j ) at time t , and n is the scale of TSP, m is the total number of ants, then

Let

n

m = ∑ bi (t ) ; Γ = {τ ij (t ) | ci , c j ⊂ C}

(1)

i =1

is the set of residual information contents in connections

lij between two elements

in set C at time t . At the initial time the information content of each path are equal,

and let τ ij (0) = const , thus the optimization of basic ant colony algorithm is implemented by means of directed graph g = (C , L, Γ) . Ant k ( k = 1, 2, , m) determines its transfer direction according to the information content of each path in its motion process. Let tabu list tabuk ( k = 1, 2, , m) record the city that ant k currently pass, the set will be dynamically adjusted along with the evolution process of tabuk . In the searching process, ants calculate state transition probability ac-

pijk (t ) represents the state transition probability from element(city) i to element(city) j of k ant k at time t , pij (t ) can be calculated as follow formula: cording to the information contents and heuristic information of each path.

⎧ [τ ij (t )]α ⋅ [ηik (t )]β , ⎪ α β ⎪ pijk (t ) = ⎨ ∑ [τ is (t )] ⋅ [ηis (t )] s ⊂ allowed k ⎪ 0, ⎪⎩

where

α

is an information heuristic factor,

if j ∈ allowed k

(2) otherwise

β

is an expectation heuristic factor,

allowed k = {C − tabuk } represents the next optional cities of ant k . α repre-

sents the relative importance of path, the bigger α is, the stronger of collaboration between ants will be. β represents the relative importance of visibility.

ηij (t )

is heuristic factor, its expression is η ij

= 1 d ij , dij represents the distance

between two adjacent cities. For ant k , the smaller dij is, the bigger be, and the bigger

k ij

p (t ) also will be.

ηij (t )

will

Research on Intelligent Diagnosis of Mechanical Fault

633

To avoid heuristic information submerged by residual information contents due to excess of residual pheromone, it is necessary to update the residual information contents after each ant finishes one step or travels all the n cities. Therefore, the information content of path (i, j ) at time t + n can be adjusted as follow formula: m

τ ij (t + n) = (1 − ρ ) ⋅τ ij (t ) + Δτ ij (t ) Δτ ij (t ) = ∑ Δτ ijk (t )

(3)

k =1

where ρ is an volatilization coefficient of the pheromone, and 1 − ρ is a residual factor of the pheromone. To avoid infinite accumulation of information contents, the value range of ρ should be ρ ⊂ [0,1) . At initial time, Δτ ij (0) = 0 .

Δτ ij (t ) represents the pheromone increment that ant k leaves at path (i, j ) in a cycle. One of the commonly used algorithms for updating pheromone is AntCycle algorithm: ⎧Q ⎪ , Δτ k (t ) = ⎨ Lk ij ⎪ 0, ⎩

where

（ ）

if ant k pass i, j in this cycle

(4)

otherwise

Lk represents the total length that ant k passed in this cycle, Q is the

strength of pheromone, which influences the convergence rate of the algorithm. And this algorithm is the basic algorithm of ant colony algorithm.

3 Multi-ants Parallel and Adaptive Clustering Algorithm This algorithm combines ant colony algorithm with clustering analysis, and takes pheromone concentration as the main basis of classifying samples [9]. In the initial stage this algorithm constructs some initial solutions randomly, and modifies the solutions according to local search mechanism. Then it constructs pheromone matrix according to the optimized solutions and executes new cycles. When the new solutions satisfy the suspension condition, the algorithm stops calculating. The main steps of this algorithm can be described as follows:

3.1 Coding Solutions and Establishing Objective Function

v dimensions has n objects {x1 , x2 , , xv } and R ants. The clustering analysis can be described as follows: R ants divide n objects into K classes and the clustering evaluation function J c is minimized. Supposing that dataset with

At first the algorithm codes solutions, and the coding methods can be described by means of simple strings such as s = {c1 , c2 , , cn } , where {ci | i = 1,…, n}

634

Z. Zhang, W. Cheng, and X. Zhou

is the class labels of object i . If the classes are marked with integer,

ci ∈1, 2,… , K . ci = c j represents that object xi , x j belong to the same class and ci

≠ c j represents that object xi , x j do not belong to the same class. Thus the

solution matrix S

= {s1 , s2,

, sR } can be constructed by the solution set of R

J c is defined as the square sum of Euclidean distance

ants. Evaluation function

between objects and clustering center of the objects, it can be described as follow formula: K

n

V

J c = MinF (ω , m) = ∑∑∑ ωij || xiv − m jv ||2

(5)

j =1 i =1 v =1

xiv is the vth attribute value of object i , m jv is the mean value of all the samples’ attribute v in cluster j . m is a K × n matrix of clustering center, and ωij is n × K matrix of relation weights:

where

⎧1, xi ∈ j ⎩0, xi ∉ j

ωij = ⎨

(6)

and m jv can be calculated according to the follow formula:

∑ ωx = ∑ ω N

m jv

i =1 ij iv N i =1

, j = 1,..., K , v = 1,..., n

(7)

ij

According to the formula (5), we can evaluate the solutions generated by ants repeatedly until function J c satisfies the condition, or J c is less than the minimum function value that is setup before. Then the optimum clustering center related to the minimum J c can be selected.

3.2 Constructing Initial Solution of Pheromone At first let all the ants’ solutions be empty. For pheromone matrix

tao with n × k

dimensions, the element taoij represents the pheromone concentration of sample

i relative to cluster j . All the elements of pheromone matrix are initialized to equal values before executing cycles. In this paper, all the initial values are 0.01. In the process of structuring initial solutions, the initial clustering centers are constructed for R ( R < n) ants by means of k-means algorithm, and the main

（1）The initial clustering centers are selected for each

steps of the algorithm are:

Research on Intelligent Diagnosis of Mechanical Fault

（）

635

ant randomly. 2 The distances between each ant(data) and K initial clustering centers are calculated. 3 each ant chooses the nearest clustering center, and allocates itself to the same cluster.

（）

3.3 Modifying Solutions Using Local Search Mechanism R ants construct some initial solutions, and search for some optimized solutions of all initial solutions locally. Thereupon, the quality of solutions is further improved. Thus at first we should calculate the values of evaluation function J c of each ant as the basis for judging the initial solutions accurate or not. The local search strategy is to vary solutions randomly in order to generate new solutions. If the new solutions are more accurate than the former solutions, the old solutions will be replaced by the new ones, otherwise the old solutions will be preserved.

3.4 Updating Pheromone Updating pheromone can dynamically reflect the information generated by the ants in motion. Therefore, pheromone matrix is updated according to optimized solutions generated by L ants after every cycle. Supposing that ρ ⊂ [0,1] is an pheromone volatilization coefficient, and it represents the volatile degree of pheromone as time lapses. And the formula of pheromone updating can be expressed as follows:

taoij (t + 1) = (1 − ρ ) × taoij (t ) + ∑ l =1 Δtaoijl L

⎧1 ⎪ , Δtaoijl = ⎨ Fl ⎪ 0, ⎩

object i belongs to cluster j

(8)

(9)

otherwise

The pheromone matrix updated will be the basis for the next iterative searching of ant colony, and new solutions are calculated by means of this method.

4 Engineering Applications of Multi-ants Parallel and Adaptive Clustering Algorithm Wheel set bearings are weak link in locomotives and are easy to malfunction, so it has important significance to classify faults correctly in engineering application. In this section we apply the multi-ants parallel and adaptive clustering algorithm to fault classification of locomotive wheel set bearings. And we make experiment on the JL-501A test-bench of locomotive wheel set bearings. To simulate the actual working condition, we use hydraulic system to drive and load, and set speed and load with the control cabinet. According to the above

636

Z. Zhang, W. Cheng, and X. Zhou Sony EX Data Acquisition Module

Experimental Control Cabinet and Bench

Data Recording Module

Fig. 1 Locomotive rolling bearing test system Moto

Support

Coupling

Axi Test Bearing Loadable Module Acceleration Sensor

Test Bench

Hydraulic Cylinder

Speed Sensor

Fig. 2 Structural diagram of experimental device

Early scratches

(a) Early fault of outer ring

Rolling body abrasion

(b) Rolling body fault

Fig. 3 Bearing fault physical picture

method, we realize continuously variable transmission and get a load power up to 20000N. The acceleration sensor is adsorbed on the bottom of load module. Although the vibration signals are weakened, it is consistent with the actual situation

Research on Intelligent Diagnosis of Mechanical Fault

637

most. Fig.1 shows the overall test system of locomotive rolling bearing, and Fig.2 gives a detailed structural diagram of experimental device. In the experiment, we acquire vibration acceleration signals in three different conditions: the bearings without faults (Normal condition), slight friction fault of outer ring, and compound faults of outer-ring peeling and rolling-element pitting. And Fig.3 (a) and (b) shows the physical pictures of outer ring and rolling-element fault. The detailed parameters of our experiment are listed in Table.1, and the original time-domain signals in these three conditions are shown in Fig.4. In our experiment, we chose three time domain characteristics, effective value, peak value and standard deviation, as the feature attributes to classify the data. Let signal x = ( x1 , x2 ,..., xT ) , T is the length of x , xˆ is the mean value of x , thus effective value

xrms , peak value x peak and standard deviation xs can be calcu-

lated as follows: Table 1 Experimental parameters of locomative wheel bearings Motor spindle speed (n/rpm)

Sampling frequency(Hz)

Normal condition

499

25.6k

Outer ring minor fault

506

25.6k

Outer ring and rolling fault

519

25.6k

Amplitude/V

Fault type

Amplitude/V

(a) Normal Condition

Amplitude/V

(b) Outer ring minor fault

(c) Outer ring and rolling compound fault

Fig. 4 Original Time-Domain Signals

638

Z. Zhang, W. Cheng, and X. Zhou

1

xrms

⎡ 1 T 2⎤2 =⎢ ∑ xi ⎥ ⎣ T − 1 i =1 ⎦

(10)

x peak = max( xi )

(11) 1

2 ⎡ 1 T 2⎤ ˆ xs = ⎢ ( x x ) − ∑ i ⎥ ⎦ ⎣ T − 1 i =1

(12)

For each condition we use 100 data samples: 50 of them are used as training samples to train classifier, and the others are used to test the performance of the classifier. The main steps and process for fault classification of locomotive bearings by means of the multi-ants parallel and adaptive clustering algorithm can be described as follows: 1) The number of ants is initialized as R=150/2=75, the initial solutions are emptied and the cycle number is initialized as NCmax = 10 . 2) Some initial clustering centers are constructed randomly and the number of the clustering centers is set such as k = 3 , then 75 clustering evaluation functions

J generated by 75 ants are calculated and the result is J max = 3.92 in our

experiment. 3) 10 solutions selected are more accurate according to the value of J and local search was made to obtain more optimized solutions. 4) Pheromone is updated according to the new solutions and new cycles are executed until the solutions satisfy the terminal conditions. 5) The fault categories of the data are classified according to the pheromone concentration matrix, and the average clustering center is calculated to construct the classifier. 6) 150 test samples are used to test the classifier. The above classifier is used to test 150 samples, and the results are shown in the above figures. Fig.5 (a) shows the fault classification 3D figures of the locomotive wheel set bearings, and it indicates the spatial classification of different types of faults. Fig.5 (b) shows that all the features are in the same plane and the distribution of every feature is more centralized, we can give good classification boundaries in x1-x2 plane. Fig.5 (c) and (d) indicate that projections in x2-x3 plane and x1-x3 plane are the same, and this is because that all the features are in the same plane and the 3D figure of the fault classification is symmetrical. According to the Fig.5, the faults are classified into three types, and 131 of 150 samples are classified correctly, which means we get a classification accurate of 87%. So it is effective for fault classification by means of multi-ants parallel and adaptive clustering algorithm.

Research on Intelligent Diagnosis of Mechanical Fault

(a) 3D figure of the fault classification

(b) Projections in x1-x2 plane

(c) Projections in x2-x3 plane

(d) Projections in x1-x3 plane

639

Compound fault of outer ring and rolling element Normal state Slight fault of outer ring Wrongly classified sample Fig. 5 3D and Projections of the locomotive wheel set bearings fault classification

5 Conclusion At present intelligent diagnosis methods that are commonly used are mostly belong to supervised methods. However, in some engineering applications, it is difficult to determine the categories of the training samples even the categories can not be determined. Therefore, unsupervised clustering learning methods are needed to deal with fault classifications in these conditions. This paper constructs a multi-ants parallel and adaptive clustering algorithm based on ant colony algorithm, and applies the algorithm to fault classification of locomotive wheel set bearings. The classification results show that 87% of the faults can be recognized correctly. So it has important engineering significance to classify faults by means of multi-ants clustering algorithm.

640

Z. Zhang, W. Cheng, and X. Zhou

Acknowledgements. This paper is supported by the National Natural Science Foundation of China“Research on Hybrid Intelligent Technique and Its Application in Fault Diagnosis Based on Granular Computing” (Grant No. 50875197) and the Project Sponsored by the Scientific Research Foundation for the Returned Overseas Chinese Scholars, State Education Ministry.

References 1. Colorni, A., Dorigo, M., Maniezzo, V.: Distributed Optimization by Ant Colonies. In: Proceedings of the 1st European Conference on Artificial Life, pp. 134–142 (1991) 2. Dorigo, M., Maniezzo, V., Colorni, A.: Ant System: Optimization by a Colony of Cooperating agents. IEEE Trans. Syst., Man, Cybern. B 26, 29–41 (1996) 3. Prakash, S., Jayaraman, V.: An Ant Colony Approach for Clustering. Analytica Chimica Acta 509, 187–195 (2004) 4. Mehemet, K., Ali, N.: A New Arrhythmia Clustering Technique Base on Ant Colony Optimization. Journal of Biomedical Informatics 41, 874–881 (2008) 5. Rahul, K., Screeram, R.: A Hybrid Approach for Feature Subset Selection Using Neural Networks and Ant Colony Optimization. Expert Systems with Applications 33, 49–60 (2007) 6. Hamidreza, R., Karim, F.: An Improved Feature Selection Method Based on Ant Colony Optimization (AOC) Evaluated on Face Recognition System. Applied Mathematics and Computation 205, 716–725 (2008) 7. Dorigo, M.: Optimization, Learning and Natural Algorithms. Ph.D. Dissertation, Department of Electronics, Politecnico di Milano, Italy (1992) 8. Liu, H.: Principle and Application of Ant Colony Algorithm. Science Press, Beijing (2005) 9. Zhang, W.: Adaptive Ant Colony Optimized Clustering Algorithm Based on Multi Agent Architecture. Computer Engineering and Applications 15, 17–19 (2005)

A New Supermemory Gradient Method without Line Search for Unconstrained Optimization June Liu, Huanbin Liu, and Yue Zheng

Abstract. In this paper, we present a new supermemory gradient method without line search for unconstrained optimization problems. The new method can guarantee a descent at each iteration. It sufficiently uses the previous multi-step iterative information at each iteration and avoids the storage and computation of matrices associated with the Hessian of objective functions, so that it is suitable to solve large scale optimization problems. We also prove its global convergence under some mild conditions. In addition, We analyze the linear convergence rate of the new method when the objective function is uniformly convex and twice continuously differentiable. Keywords: Unconstrained optimization, Memory gradient method, Global convergence, Convergence rate.

1 Introduction Consider the unconstrained optimization problem min f (x),

x ∈ Rn ,

(1)

where f (x) : Rn → R is continuously differentiable and its gradient is available. Usually we use the iterative method for solving (1) and its form is given by June Liu · Huanbin Liu · Yue Zheng Institute of Uncertain Systems, College of Mathematics and Information Science, Huanggang Normal University, Hubei 438000, China Yue Zheng School of Mathematics and Statistics, Wuhan University, Wuhan 430072, China [email protected],[email protected],[email protected] H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 641–647. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com 

642

J. Liu, H. Liu, and Y. Zheng

xk+1 = xk + αk dk ,

(2)

where αk is a positive step size and dk is a search direction of f (x) at xk . Throughout this paper, we denote f (xk ) by fk , ∇f (xk ) by gk and f (x∗ ) by f ∗ respectively. There are many traditional methods for solving (1) such as the Newton method, the quasi-Newton method, the steepest descent method and the conjugate gradient method. It is well-known that the Newton method and the quasi-Newton method are most effective iterative methods. However, they need store and update matrices associated with the Hessian of the objective function. As problem size and computational expense increase, so does the difficulty of finding solutions. To further reduce the computational expense associated with Newton-like methods, the conjugate gradient method has recently attracted more attention. The most important features of this method are that storage requirement are minimal and that computational expense are relatively small. Consequently, the conjugate gradient method is particularly useful when the dimension of (1) is large. Generally it has the form (2) with  , f or k = 1 −gk dk = (3) −gk + βk dk−1 , f or k ≥ 2 where βk is a scalar which determines the different conjugate gradient methods. The choice of βk should be such that (2)-(3) reduces to the linear conjugate gradient method in the case when f (x) is a strictly convex quadratic and αk is the exact 1-dimentional minimizer. Well-known formulas for βk are the Fletcher-Reeves(FR), Polak-Ribiere-Polyak(PRP), HestenesStiefel(HS), Conjugate Decent(CD) and Dai and Yuan(DY) formulas, which are given by gk 2 βkF R = , gk−1 2 βkP RP = βkHS =

gkT (gk − gk−1 ) , gk−1 2

gkT (gk − gk−1 ) , dTk−1 (gk − gk−1 )

βkCD = − βkDY = −

gk 2 , dTk−1 gk−1

gk 2 , dTk−1 (gk − gk−1 )

respectively. Their convergence properties have been reported by many authors [1]-[8].

A New Supermemory Gradient Method without Line Search

643

Similar methods to the conjugate gradient method are the memory gradient method or the supermemory gradient method. Not only are their basic idea simple but also they avoid computation and storage of some matrices, so that they are suitable to solve large scale optimization problems. The main difference between them is that the latter can use the information of the previous multi-step iterations more sufficiently and hence it is helpful to design algorithms with quick convergence rate. Many authors have studied the global convergence properties for memory gradient methods and yielded substantial results [9]-[12]. However, it is difficult or time-consuming to implement an exact line search for seeking step length in practical computation. We should use a suitable search direction and some available inexact line searches to choose a step size at each iteration for memory gradient methods and guarantee the global convergence. Generally, we often use Wolfe line search, Goldstein line search and Armijo line search for finding a step size. However for large scale problems, these line search rules need expensive computation of function and gradient values at each iteration. Recently, Sun and Zhang [13] proposed a particular choice of step size which means no line search and established the global convergence. Chen and Sun [14], Li and Chen [15], Yasushi Narushima [16], Yu [17] etc made further discussion from other aspects. Based on these, we will study the convergence properties of a new supermemory gradient method in this paper. We prove that, without any line searches, the new method can also guarantee a descent at each iteration. Finally, the global convergence and linear convergence rate are proved under some assumptions.

2 New Method For future reference, We formally assume the following. (H1) The objective function f (x) is bounded below on the level set Ω = {x ∈ Rn |f (x) ≤ f (x1 )}. (H2) In a neighborhood N of Ω, f (x) is differentiable and its gradient g(x) is Lipschitz continuous, namely, there exists a constant L > 0 such that g(x) − g(y) ≤ Lx − y, ∀x, y ∈ N.

(4)

Algorithm (A) Step 0 Given constants ε > 0, ρ ∈ ( 12 , 1), positive integer m and k := 1; Step 1 If gk  < ε, then stop! else go to Step 2; Step 2 Computer dk which satisfies the following formula ⎧ −gk , f or k ≤ m − 1 ⎨ m  (5) dk = dk−i , f or k ≥ m ⎩ −gk + βk i=1

644

where βk =

J. Liu, H. Liu, and Y. Zheng ρ gk ; m  dk−i

i=1

gT d

k k Step 3 Computer αk = − L d 2; k Step 4 Let xk+1 = xk + αk dk ; Step 5 Let k := k + 1 and go to Step 1.

Lemma 1. For all k ≥ 1, we have that gkT dk ≤ −(1 − ρ)gk 2 .

(6)

Proof. For obtaining (6), we consider the following two cases. Case (i):k ≤ m − 1. It is obvious that (6) holds. Case (ii):k ≥ m. gkT dk = −gk 2 + βk

m

gkT dk−i

i=1

≤ −gk 2 + βk gk 

m

dk−i 

i=1

= −(1 − ρ)gk 2 . We can immediately draw the conclusion of Lemma 1. Lemma 2. For all k ≥ 1, we have dk  ≤ (1 + ρ)gk .

(7)

Proof. If k ≤ m − 1, then dk  = gk  ≤ (1 + ρ)gk . If k ≥ m, then dk 2 = gk 2 − 2βk

m

gkT dk−i + βk2 

i=1

≤ gk 2 + 2βk gk 

m

m

dk−i 2

i=1 m 2 dk−i  + βk ( dk−i )2

i=1

i=1

= gk 2 + 2ρgk 2 + ρ2 gk 2 = (1 + ρ)2 gk 2 . Consequently, we can obtain dk  ≤ (1 + ρ)gk . Lemma 3. Suppose that the assumption (H1)-(H2) hold. Let {xk } be generated by Algorithm(A). Then we have f (xk ) − f (xk+1 ) ≥

1 (gkT dk )2 . 2L dk 2

(8)

A New Supermemory Gradient Method without Line Search

645

Proof. By mean value theorem and (H2) we have !

1

f (xk+1 ) − f (xk ) = αk !

g(xk + tαk dk )T dk dt 0

1

(g(xk + tαk dk ) − gk )T dk dt

= αk gkT dk + αk 0

! ≤

αk gkT dk

1

g(xk + tαk dk ) − gk dk  dt

+ αk 0

1 ≤ αk gkT dk + Lα2k dk 2 2 1 (gkT dk )2 =− . 2L dk 2

3 Convergence Analysis By using Lemma 1–3, we can easily obtain the following result. Theorem 1. Suppose that the assumption (H1)-(H2) hold. Let {xk } be generated by Algorithm(A). Then we have that lim gk  = 0.

k→∞

In order to analyze the convergence rate, we further assume that (H3)f (x) is uniformly convex and twice continuously differentiable. Lemma 4. Suppose that (H3) hold, then f (x) has the following properties (1)f (x) has a unique minimizer x∗ on Rn . (2)The level set Ω = {x ∈ Rn |f (x) ≤ f (x1 )} is bounded. (3)There exists 0 < m1 < m2 such that 1 1 m1 x − x∗ 2 ≤ f (x) − f (x∗ ) ≤ m2 x − x∗ 2 , 2 2 ∗ m1 x − x  ≤ g(x) ≤ m2 x − x∗ .

(9) (10)

(4) Assumptions (H1) and (H2) hold. Proof. The results of this lemma are obtained from [18]. Theorem 2. Suppose that (H3) holds and the Algorithm(A) generates an infinite sequence {xk }. Then xk convergence to x∗ at least R-linearly. Proof. Conform to Lemma 4, we may assume that L = m2 . It follows by Lemma 3 and from (6) and (7) that f (xk+1 − f (xk )) ≤ −

1 1−ρ 2 ( ) gk 2 . 2L 1 + ρ

646

J. Liu, H. Liu, and Y. Zheng

Combining the above inequality (9) and (10), we can obtain f (xk+1 ) − f (xk ) ≤ −( Define θ=(

m1 2 1 − ρ 2 ) ( ) (f (xk ) − f (x∗ )). m2 1+ρ

(11)

m1 2 1 − ρ 2 ) . ) ( m2 1+ρ

It is obvious that 0 < θ < 1. By (9) and (11), we have m1 xk − x∗ 2 ≤ f (xk ) − f (xk−1 ) + f (xk−1 ) − f (x∗ ) ≤ (1 − θ)(f (xk−1 ) − f (x∗ )) ≤ · · · ≤ (1 − θ)k−1 (f (x1 ) − f (x∗ )). which shows that xk convergence to x∗ at least R-linearly. This completes the proof.

4 Conclusion In this paper, we presented a new supermemory gradient method without line search for unconstrained optimization problems. The new method can guarantee a descent at each iteration. We proved its global convergence under mild conditions. We also analyzed the linear convergence rate of the new method when the objective function is uniformly convex and twice continuously differentiable. However more numerical test should be done for practical problems.

References 1. Dai, Y.H., Yuan, Y.X.: Convergence Properties of the Fletcher-Reeves Method. IMA J. Numer. Anal. 16, 155–164 (1996) 2. Hu, Y.F., Storey, C.: Global Convergence Result for Conjugate Gradient Methods. JOTA 71, 399–405 (1991) 3. Grippo, L., Lucidi, S.: A Globally Convergent Version of the Polak-Ribiere Conjugate Gradient Method. Math. Prog. 78, 375–391 (1997) 4. Fletcher, R.: Practical Methods of Optimization. Unconstrained Optimization, vol. 1. John Wiley Sons, New York (1987) 5. Dai, Y.H., Yuan, Y.X.: Convergence Properties of the Conjugate Descent Method. Advances in Mathematics 25, 552–562 (1996) 6. Dai, Y.H., Yuan, Y.X.: Nonlinear Conjugate Gradient Methods. Shanghai Scientific and Technical Publishers (2000) 7. Dai, Y.H., Yuan, Y.X.: A Nonlinear Conjugate Gradient Method with a Strong Global Convergence Property. SIAM J. Optim. 10, 177–182 (1999)

A New Supermemory Gradient Method without Line Search

647

8. Gilbert, J.C., Nocedal, J.: Global Convergence Properties of Conjugate Gradient Methods for Optimization. SIAM J. Optim. 2, 21–42 (1992) 9. Wolfe, M.A., Viazminsky, C.: Supermemory Descent Methods for Unconstrained Minimization. J. Optim. Theory Appl. 18, 455–468 (1976) 10. Zhenjun, S.: Supermemory Gradient Method for Unconstrained Optimization. J. Eng. Math. 17, 99–104 (2000) 11. Zhenjun, S.: A New Memory Gradient under Exact Line Search. Asia-Pacific J. Oper. Res. 20, 275–284 (2003) 12. Zhenjun, S.: A New Supermemory Gradient Method for Unconstrained Optimization. Advances in Mathematics 35, 265–274 (2006) 13. Sun, J., Zhang, J.: Global Convergence of Conjugate Gradient Methods without Line Search. Annals of Operations Research 103, 161–173 (2001) 14. Chen, X., Sun, J.: Global Convergence of a Two-parameter Family of Conjugate Gradient Methods without Line Search. Journal of Computational and Applied Mathematics 146, 37–45 (2002) 15. Li, X., Chen, X.: Global Convergence of Shortest-residual Family of Conjugate Gradient Methods without Line Search. Asia-Pacific Journal of Operational Research 22, 529–538 (2005) 16. Narushima, Y.: A Memory Gradient Method without Line Search for Unconstrained Optimization. SUT Journal of Mathematics 42, 191–206 (2006) 17. Yu, Z.S.: Global Convergence of a Memory Gradient Method without Line Search. J. Appl. Math. Comput. 26, 545–553 (2008) 18. Cohen, A.I.: Stepsize Analysis for Descent Methods. J. Optim. Theory Appl. 33, 187–205 (1981)

A Neural Network Approach for Solving Linear Bilevel Programming Problem Tiesong Hu, Bing Huang, and Xiang Zhang*

Abstract. A novel neural network approach is proposed for solving linear bilevel programming problem. The proposed neural network is proved to be Lyapunov stable and capable of generating optimal solution to the linear bilevel programming problem. The numerical result shows that the neural network approach is feasible and efficient. Keywords: Linear bilevel programming, Neural network, Asymptotic stability, Optimal solution.

1 Introduction Bilevel programming (BLP) problem arise in a wide variety of scientific and engineering applications including resource allocation, finance budget, price control, transportation and network design, and so on. It is characterized by the existence of two optimization problems in which the constraint region of the first-level problem is implicitly determined by another optimization problem. BLP problem has been proved to be NP-hard[1]. Since McCulloch and Pyne[2,3] utilized logical calculus to emulate nervous activities, there have been various types of analogue neural networks proposed for computation, and recent research has shown that the neural network approach has many computational advantages over traditional digital computers[4-9]. However, so far little has been published on application of neural networks to the BLP problem. In this paper, we present an approach of constructing neural network, which is different from the method given in [10,11] for the linear BLP problem. Here, following the Kuhn-Tucker optimality condition of the lower level problem, we reduce the linear BLP problem to a regular linear programming with complementary constraints. Then we propose a novel neural network approach for the linear programming problem with complementary constraints and get the approximate Tiesong Hu . Bing Huang . Xiang Zhang State Key Laboratory of Water Resource and Hydropower Engineering Science, Wuhan University, Wuhan 430072, China *

H. Wang et al. (Eds.): The Sixth ISNN 2009, AISC 56, pp. 649–658. springerlink.com © Springer-Verlag Berlin Heidelberg 2009

650

T. Hu, B. Huang, and X. Zhang

optimal solution of the linear BLP problem. It is noted that the neural network proposed here can also be used to solve the linear programming with complementary constraints. Towards these ends, the rest of the paper is organized as follows. In Section 2, we will firstly introduce the smoothing method for the linear programming with complementary constraints. Then in Section 3, we propose a neural network for solving the smoothed problem and derive the condition for asymptotic stability, solution feasibility and solution optimality. Numerical example are given in Section 4. Finally we conclude our paper.

2 Linear BLP Problem and Smoothing Method Let

x ∈ X ⊂ R n , y ∈ Y ⊂ R m , F : X × Y → R 1 , f : X × Y → R 1 . The

general model of the linear BLP can be written as [2]:

min F ( x, y ) = c1 x + d1 y x∈ X

A1 x + B1 y ≤ b1

s.t.

min f ( x, y ) = c2 x + d 2 y

(1)

y∈Y

s.t.

A2 x + B2 y ≤ b2

c1 , c 2 ∈ R n , d 1 , d 2 ∈ R m , b1 ∈ R p , b2 ∈ R q , A1 ∈ R p×n , B1 ∈ R p×m , A2 ∈ R q×n , B2 ∈ R q×m . According to the above introduction, we can replace the lower level problem with its Kuhn-Tucker optimality conditions, and get the following one-level problem.

min c1 x + d1 y s.t.

A1 x + B1 y ≤ b1 , A2 x + B2 y ≤ b2 , uB2 − v = − d 2 ,

(2)

u (b2 − A2 x − B2 y ) + vy = 0, x ≥ 0, y ≥ 0, u ≥ 0, v ≥ 0 where

u ∈ R q , v ∈ R m are (row) vector.

Problem (2) is called the mathematical program with complementary constraint, the regularity assumptions which are needed for successfully handling smooth optimization programs are never satisfied. Then, we reformulate problem (2) to the following non-smooth equivalent reformulation [12]:

A Neural Network Approach for Solving Linear Bilevel Programming Problem

651

min c1 x + d1 y A1 x + B1 y ≤ b1 ,

s.t.

A2 x + B2 y ≤ b2 , uB2 − v = − d 2 ,

(3)

−2 min(u , b2 − A2 x − B2 y ) = 0, −2 min(v, y ) = 0, x≥0 The “min” operator is applied component-wise to the vectors. Problem (3) is nonsmooth optimality problem, and it is not good to use neural network to solve problem (3). But fortunately, some researchers have presented smoothing method for problem (3). Following this smoothing method we can propose a novel neural network approach for the linear BLP problem. Let μ ∈ R be a parameter. Define the function φμ

: R 2 → R by

φμ ( a , b) = ( a − b ) 2 + 4 μ 2 − ( a + b) Then, we can have the following proposition. Proposition 1 [12]. For every μ ∈ R , we have

φμ (a, b) = 0 ⇔ a ≥ 0, b ≥ 0, ab = μ 2 Note also that for μ for every

μ≠0

every (a, b) ,

,

= 0 , we have φμ (a, b) = 0 ⇔ a ≥ 0, b ≥ 0, ab = 0 . While,

φμ ( a, b)

is differential for every

(a, b) . Moreover, for

lim φμ (a, b) = −2 min(a, b) . The function φμ (a, b) is therefore μ →0

a smooth perturbation of the complementary conditions. Then, problem (3) can be approximated by: min c1 x + d1 y s.t.

A1 x + B1 y ≤ b1 , uB2 − v = −d 2 , [ui − (b2 − A2 x − B2 y )i ]2 + 4μ 2 − ui − (b2 − A2 x − B2 y )i = 0, (v j − y j ) 2 + 4μ 2 − v j − y j = 0, x≥0

j = 1,… , m

i = 1,… , q

(4)

652

T. Hu, B. Huang, and X. Zhang

Using the problem (4), we overcome the difficulty that the problem (2) does not satisfy any regularity assumptions which are needed for successfully handling smooth optimization problems, and pave the way for using neural network approach to solve the problem (2). To simply the discussion, we introduce the following notations.

⎛ A x + B1 y − b1 ⎞ G ( x, y , u , v ) = ⎜ 1 ⎟, −x ⎝ ⎠ ⎛ ⎞ uB2 − v + d 2 ⎜ ⎟ H ( x, y, u, v) = ⎜ φμ (ui , (b2 − A2 x − B2 y )i ), i = 1,… , q ⎟ ⎜ ⎟ φμ (v j , y j ), j = 1,… , m ⎝ ⎠

Let x ' = ( x, y , u , v ) , we can write problem (4) equivalently as the following problem.

min f ( x ') = c1 x + d1 y

，

s.t. Gl ( x ') ≤ 0 l = 1,… , n + p (5) H k ( x ') = 0 k = 1,… , 2m + q Definition 2. Let x ' be a feasible point of problem (5) and L = {l : Gl ( x ' ) = 0, l = 1,… , n + p} . We say that x ' is a regular point of problem (5) if the gradients ∇H k ( x ') , ∇Gl ( x '), l ∈ L are linearly independent.

，

Similar to the main result in [3](Theorem 6.11), we can directly have the following theorem.

{( x ') } be a sequence of solutions to problem (5). Suppose the } converges to some x for μ → 0 . If x is the regular point of μ

Theorem 1. Let sequence

{( x ') μ

_

_

_

the problem (5), then

x solves the problem (2).

3 Neural Network for Linear BLP Problem The Lagrange function of problem (4) can be defined by

L( x ', Y , λ , μ ) = f ( x ') +

2m+q

n+q

k =1

l =1

∑ λk H k ( x ') + ∑ μl [Gl ( x ') + Yl 2 ]

where the term Y is slack variable, and the terms λ , μ are referred as Lagrange multiplier. Then, our aim now is to design a neural network that will settle down to the equilibrium, which is also a stationary point of the Lagrange function

A Neural Network Approach for Solving Linear Bilevel Programming Problem

653

L( x ', Y , λ , μ ) . The transient behavior of the neural network can be defined by the following equations.

⎧ dx ' ⎪ dt = −∇ x ' L( x ', Y , λ , μ ) ⎪ ⎪ dY = −∇ L( x ', Y , λ , μ ) Y ⎪ dt (LBPNN) ⎨ ⎪ d λ = ∇ L( x ', Y , λ , μ ) λ ⎪ dt ⎪ dμ ⎪ = ∇ μ L( x ', Y , λ , μ ) ⎩ dt

(6)

Now we will study the relationship between the equilibrium of LBPNN and the approximate optimal solution of the problem (1) for ε → 0 + . We have the following theorem. *

Theorem 2. Let (( x ')

, Y * , λ * , μ * ) be the equilibrium of the neural network (6),

*

and assume that ( x ') is a regular point of problem (4). Then the equilibrium of the neural network solves problem (4). Proof. The proof of theorem 2 can be divided into two steps. Firstly, same to the proof of theorem 3 in [6], we can get that μl > 0, l = 1,… , n + p and the equilibrium of the neural network is the Kuhn-Tucker point of the problem. Secondly, we note that it is easy to verify that ∇ x ' x ' L(( x ') 2

Let

*

, Y * , λ * , μ * ) is positive definite.

Z = {z | ∇H k ( x 'k ) z = 0, k = 1,… , 2m + q; ∇Gl ( x 'k ) z = 0, ∀l ∈ L} ,

we have that z ∇ x ' x ' L(( x ') T

2

*

, Y * , λ * , μ * ) z > 0 , then following the sufficiency

optimality conditions of second order for the problem (4), we can get that (( x ') , Y , λ , μ ) solves the problem (4). While for the neural network to be of practical sense, the neural network should be of asymptotically stable, so that the neural network will always converge to *

*

*

*

(( x ')* , Y * , λ * , μ * ) from an arbitrary initial point within the attraction domain *

of (( x ')

, Y * , λ * , μ * ) . With this in mind, we state and prove the following theo-

rem, which in other words represents the local stability of the network.

(( x ')* , Y * , λ * , μ * ) be the equilibrium of the neural network (6). * * * * * If ( x ') is the regular point of problem (4), then (( x ') , Y , λ , μ ) is an asymp-

Theorem 3. Let

totically stable point of the neural network.

654

T. Hu, B. Huang, and X. Zhang

Proof. Let the following equation denote the Lyapunov function of the network